diff --git a/CODE_OF_CONDUCT.md b/CODE_OF_CONDUCT.md
index 72304bee694..4992e54e7f6 100644
--- a/CODE_OF_CONDUCT.md
+++ b/CODE_OF_CONDUCT.md
@@ -1,7 +1,11 @@
 # TensorFlow Code of Conduct
 
-In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
-
+In the interest of fostering an open and welcoming environment, we as
+contributors and maintainers pledge to make participation in our project and our
+community a harassment-free experience for everyone, regardless of age, body
+size, disability, ethnicity, gender identity and expression, level of
+experience, nationality, personal appearance, race, religion, or sexual identity
+and orientation.
 
 ## Our Standards
 
diff --git a/ISSUE_TEMPLATE.md b/ISSUE_TEMPLATE.md
index 04cd8cb65ef..ffd1efc44ed 100644
--- a/ISSUE_TEMPLATE.md
+++ b/ISSUE_TEMPLATE.md
@@ -4,26 +4,31 @@ https://stackoverflow.com/questions/tagged/tensorflow
 
 If you open a GitHub issue, here is our policy:
 
-1. It must be a bug, a feature request, or a significant problem with documentation (for small docs fixes please send a PR instead).
-2. The form below must be filled out.
-3. It shouldn't be a TensorBoard issue. Those go [here](https://github.com/tensorflow/tensorboard/issues).
+1.  It must be a bug, a feature request, or a significant problem with the
+    documentation (for small docs fixes please send a PR instead).
+2.  The form below must be filled out.
+3.  It shouldn't be a TensorBoard issue. Those go
+    [here](https://github.com/tensorflow/tensorboard/issues).
 
 **Here's why we have that policy**: TensorFlow developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.
 
 ------------------------
 
 ### System information
-- **Have I written custom code (as opposed to using a stock example script provided in TensorFlow)**:
-- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
-- **Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device**:
-- **TensorFlow installed from (source or binary)**:
-- **TensorFlow version (use command below)**:
-- **Python version**:
-- **Bazel version (if compiling from source)**:
-- **GCC/Compiler version (if compiling from source)**:
-- **CUDA/cuDNN version**:
-- **GPU model and memory**:
-- **Exact command to reproduce**:
+
+-   **Have I written custom code (as opposed to using a stock example script
+    provided in TensorFlow)**:
+-   **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
+-   **Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue
+    happens on a mobile device**:
+-   **TensorFlow installed from (source or binary)**:
+-   **TensorFlow version (use command below)**:
+-   **Python version**:
+-   **Bazel version (if compiling from source)**:
+-   **GCC/Compiler version (if compiling from source)**:
+-   **CUDA/cuDNN version**:
+-   **GPU model and memory**:
+-   **Exact command to reproduce**:
 
 You can collect some of this information using our environment capture script:
 
diff --git a/RELEASE.md b/RELEASE.md
index 6f3aa94c203..f93626cc876 100644
--- a/RELEASE.md
+++ b/RELEASE.md
@@ -90,89 +90,150 @@ Coinciding with this change, new releases of [TensorFlow's Docker images](https:
 * The current TensorFlow release now **requires** [gast](https://pypi.org/project/gast/) version 0.3.3.
 
 ## Bug Fixes and Other Changes
-* `tf.data`:
-  * Removed `autotune_algorithm` from experimental optimization options.
-* TF Core:
-  * `tf.constant` always creates CPU tensors irrespective of the current device context.
-  * Eager `TensorHandles` maintain a list of mirrors for any copies to local or remote devices. This avoids any redundant copies due to op execution.
-  * For `tf.Tensor` & `tf.Variable`, `.experimental_ref()` is no longer experimental and is available as simply `.ref()`.
-  * `pfor/vectorized_map`: Added support for vectorizing 56 more ops. Vectorizing `tf.cond` is also supported now.
-  * Set as much partial shape as we can infer statically within the gradient impl of the gather op.
-  * Gradient of `tf.while_loop` emits `StatelessWhile` op if `cond` and body functions are stateless. This allows multiple gradients while ops to run in parallel under distribution strategy.
-  * Speed up `GradientTape` in eager mode by auto-generating list of op inputs/outputs which are unused and hence not cached for gradient functions.
-  * Support `back_prop=False` in `while_v2` but mark it as deprecated.
-  * Improve error message when attempting to use `None` in data-dependent control flow.
-  * Add `RaggedTensor.numpy()`.
-  * Update `RaggedTensor.__getitem__` to preserve uniform dimensions & allow indexing into uniform dimensions.
-  * Update `tf.expand_dims` to always insert the new dimension as a non-ragged dimension.
-  * Update `tf.embedding_lookup` to use `partition_strategy` and `max_norm` when `ids` is ragged.
-  * Allow `batch_dims==rank(indices)` in `tf.gather`.
-  * Add support for bfloat16 in `tf.print`.
-* `tf.distribute`:
-  * Support `embedding_column` with variable-length input features for `MultiWorkerMirroredStrategy`.
-* `tf.keras`:
-  * Added `experimental_aggregate_gradients` argument to `tf.keras.optimizer.Optimizer.apply_gradients`. This allows custom gradient aggregation and processing aggregated gradients in custom training loop.
-  * Allow `pathlib.Path` paths for loading models via Keras API.
-* `tf.function`/AutoGraph:
-  * AutoGraph is now available in `ReplicaContext.merge_call`, `Strategy.extended.update` and `Strategy.extended.update_non_slot`.
-  * Experimental support for shape invariants has been enabled in `tf.function`. See the API docs for `tf.autograph.experimental.set_loop_options` for additonal info.
-  * AutoGraph error messages now exclude frames corresponding to APIs internal to AutoGraph.
-  * Improve shape inference for `tf.function` input arguments to unlock more Grappler optimizations in TensorFlow 2.x.
-  * Improve automatic control dependency management of resources by allowing resource reads to occur in parallel and synchronizing only on writes.
-  * Fix execution order of multiple stateful calls to `experimental_run_v2` in `tf.function`.
-  * You can now iterate over `RaggedTensors` using a for loop inside `tf.function`.
-* `tf.lite`:
-  *  Migrated the `tf.lite` C inference API out of experimental into lite/c.
-  * Add an option to disallow `NNAPI` CPU / partial acceleration on Android 10
-  * TFLite Android AARs now include the C headers and APIs are required to use TFLite from native code.
-  * Refactors the delegate and delegate kernel sources to allow usage in the linter.
-  * Limit delegated ops to actually supported ones if a device name is specified or `NNAPI` CPU Fallback is disabled.
-  * TFLite now supports `tf.math.reciprocal1` op by lowering to `tf.div op`.
-  * TFLite's unpack op now supports boolean tensor inputs.
-  * Microcontroller and embedded code moved from experimental to main TensorFlow Lite folder
-  * Check for large TFLite tensors.
-  * Fix GPU delegate crash with C++17.
-  * Add 5D support to TFLite `strided_slice`.
-  * Fix error in delegation of `DEPTH_TO_SPACE` to `NNAPI` causing op not to be accelerated.
-  * Fix segmentation fault when running a model with LSTM nodes using `NNAPI` Delegate
-  * Fix `NNAPI` delegate failure when an operand for Maximum/Minimum operation is a scalar.
-  * Fix `NNAPI` delegate failure when Axis input for reduce operation is a scalar.
-  * Expose option to limit the number of partitions that will be delegated to `NNAPI`.
-  * If a target accelerator is specified, use its feature level to determine operations to delegate instead of SDK version.
-* `tf.random`:
-  * Various random number generation improvements:
-    * Add a fast path for default `random_uniform`
-    * `random_seed` documentation improvement.
-    * `RandomBinomial` broadcasts and appends the sample shape to the left rather than the right.
-  * Added `tf.random.stateless_binomial`, `tf.random.stateless_gamma`, `tf.random.stateless_poisson`
-  * `tf.random.stateless_uniform` now supports unbounded sampling of `int` types.
-* Math and Linear Algebra:
-  * Add `tf.linalg.LinearOperatorTridiag`.
-  * Add `LinearOperatorBlockLowerTriangular`
-  * Add broadcasting support to tf.linalg.triangular_solve[#26204](https://github.com/tensorflow/tensorflow/issues/26204), tf.math.invert_permutation.
-  * Add `tf.math.sobol_sample` op.
-  * Add `tf.math.xlog1py`.
-  * Add `tf.math.special.{dawsn,expi,fresnel_cos,fresnel_sin,spence}`.
-  * Add a Modified Discrete Cosine Transform (MDCT) and its inverse to `tf.signal`.
-* TPU Enhancements:
-  * Refactor `TpuClusterResolver` to move shared logic to a separate pip package.
-  * Support configuring TPU software version from cloud tpu client.
-  * Allowed TPU embedding weight decay factor to be multiplied by learning rate.
-* XLA Support:
-  * Add standalone XLA AOT runtime target + relevant .cc sources to pip package.
-  * Add check for memory alignment to MemoryAllocation::MemoryAllocation() on 32-bit ARM. This ensures a deterministic early exit instead of a hard to debug bus error later.
-  * `saved_model_cli aot_compile_cpu` allows you to compile saved models to XLA header+object files and include them in your C++ programs.
-  * Enable `Igamma`, `Igammac` for XLA.
-* Deterministic Op Functionality:
-  * XLA reduction emitter is deterministic when the environment variable `TF_DETERMINISTIC_OPS` is set to "true" or "1". This extends deterministic `tf.nn.bias_add` back-prop functionality (and therefore also deterministic back-prop of bias-addition in Keras layers) to include when XLA JIT complilation is enabled.
-  * Fix problem, when running on a CUDA GPU and when either environment variable `TF_DETERMINSTIC_OPS` or environment variable `TF_CUDNN_DETERMINISTIC` is set to "true" or "1", in which some layer configurations led to an exception with the message "No algorithm worked!"
-* Tracing and Debugging:
-  * Add source, destination name to `_send` traceme to allow easier debugging.
-  * Add traceme event to `fastpathexecute`.
-* Other:
-  * Fix an issue with AUC.reset_states for multi-label AUC [#35852](https://github.com/tensorflow/tensorflow/issues/35852)
-  * Fix the TF upgrade script to not delete files when there is a parsing error and the output mode is `in-place`.
-  * Move `tensorflow/core:framework/*_pyclif` rules to `tensorflow/core/framework:*_pyclif`.
+
+*   `tf.data`:
+    *   Removed `autotune_algorithm` from experimental optimization options.
+*   TF Core:
+    *   `tf.constant` always creates CPU tensors irrespective of the current
+        device context.
+    *   Eager `TensorHandles` maintain a list of mirrors for any copies to local
+        or remote devices. This avoids any redundant copies due to op execution.
+    *   For `tf.Tensor` & `tf.Variable`, `.experimental_ref()` is no longer
+        experimental and is available as simply `.ref()`.
+    *   `pfor/vectorized_map`: Added support for vectorizing 56 more ops.
+        Vectorizing `tf.cond` is also supported now.
+    *   Set as much partial shape as we can infer statically within the gradient
+        impl of the gather op.
+    *   Gradient of `tf.while_loop` emits `StatelessWhile` op if `cond` and body
+        functions are stateless. This allows multiple gradients while ops to run
+        in parallel under distribution strategy.
+    *   Speed up `GradientTape` in eager mode by auto-generating list of op
+        inputs/outputs which are unused and hence not cached for gradient
+        functions.
+    *   Support `back_prop=False` in `while_v2` but mark it as deprecated.
+    *   Improve error message when attempting to use `None` in data-dependent
+        control flow.
+    *   Add `RaggedTensor.numpy()`.
+    *   Update `RaggedTensor.__getitem__` to preserve uniform dimensions & allow
+        indexing into uniform dimensions.
+    *   Update `tf.expand_dims` to always insert the new dimension as a
+        non-ragged dimension.
+    *   Update `tf.embedding_lookup` to use `partition_strategy` and `max_norm`
+        when `ids` is ragged.
+    *   Allow `batch_dims==rank(indices)` in `tf.gather`.
+    *   Add support for bfloat16 in `tf.print`.
+*   `tf.distribute`:
+    *   Support `embedding_column` with variable-length input features for
+        `MultiWorkerMirroredStrategy`.
+*   `tf.keras`:
+    *   Added `experimental_aggregate_gradients` argument to
+        `tf.keras.optimizer.Optimizer.apply_gradients`. This allows custom
+        gradient aggregation and processing aggregated gradients in custom
+        training loop.
+    *   Allow `pathlib.Path` paths for loading models via Keras API.
+*   `tf.function`/AutoGraph:
+    *   AutoGraph is now available in `ReplicaContext.merge_call`,
+        `Strategy.extended.update` and `Strategy.extended.update_non_slot`.
+    *   Experimental support for shape invariants has been enabled in
+        `tf.function`. See the API docs for
+        `tf.autograph.experimental.set_loop_options` for additonal info.
+    *   AutoGraph error messages now exclude frames corresponding to APIs
+        internal to AutoGraph.
+    *   Improve shape inference for `tf.function` input arguments to unlock more
+        Grappler optimizations in TensorFlow 2.x.
+    *   Improve automatic control dependency management of resources by allowing
+        resource reads to occur in parallel and synchronizing only on writes.
+    *   Fix execution order of multiple stateful calls to `experimental_run_v2`
+        in `tf.function`.
+    *   You can now iterate over `RaggedTensors` using a for loop inside
+        `tf.function`.
+*   `tf.lite`:
+    *   Migrated the `tf.lite` C inference API out of experimental into lite/c.
+    *   Add an option to disallow `NNAPI` CPU / partial acceleration on Android
+        10
+    *   TFLite Android AARs now include the C headers and APIs are required to
+        use TFLite from native code.
+    *   Refactors the delegate and delegate kernel sources to allow usage in the
+        linter.
+    *   Limit delegated ops to actually supported ones if a device name is
+        specified or `NNAPI` CPU Fallback is disabled.
+    *   TFLite now supports `tf.math.reciprocal1` op by lowering to `tf.div op`.
+    *   TFLite's unpack op now supports boolean tensor inputs.
+    *   Microcontroller and embedded code moved from experimental to main
+        TensorFlow Lite folder
+    *   Check for large TFLite tensors.
+    *   Fix GPU delegate crash with C++17.
+    *   Add 5D support to TFLite `strided_slice`.
+    *   Fix error in delegation of `DEPTH_TO_SPACE` to `NNAPI` causing op not to
+        be accelerated.
+    *   Fix segmentation fault when running a model with LSTM nodes using
+        `NNAPI` Delegate
+    *   Fix `NNAPI` delegate failure when an operand for Maximum/Minimum
+        operation is a scalar.
+    *   Fix `NNAPI` delegate failure when Axis input for reduce operation is a
+        scalar.
+    *   Expose option to limit the number of partitions that will be delegated
+        to `NNAPI`.
+    *   If a target accelerator is specified, use its feature level to determine
+        operations to delegate instead of SDK version.
+*   `tf.random`:
+    *   Various random number generation improvements:
+    *   Add a fast path for default `random_uniform`
+    *   `random_seed` documentation improvement.
+    *   `RandomBinomial` broadcasts and appends the sample shape to the left
+        rather than the right.
+    *   Added `tf.random.stateless_binomial`, `tf.random.stateless_gamma`,
+        `tf.random.stateless_poisson`
+    *   `tf.random.stateless_uniform` now supports unbounded sampling of `int`
+        types.
+*   Math and Linear Algebra:
+    *   Add `tf.linalg.LinearOperatorTridiag`.
+    *   Add `LinearOperatorBlockLowerTriangular`
+    *   Add broadcasting support to
+        tf.linalg.triangular_solve[#26204](https://github.com/tensorflow/tensorflow/issues/26204),
+        tf.math.invert_permutation.
+    *   Add `tf.math.sobol_sample` op.
+    *   Add `tf.math.xlog1py`.
+    *   Add `tf.math.special.{dawsn,expi,fresnel_cos,fresnel_sin,spence}`.
+    *   Add a Modified Discrete Cosine Transform (MDCT) and its inverse to
+        `tf.signal`.
+*   TPU Enhancements:
+    *   Refactor `TpuClusterResolver` to move shared logic to a separate pip
+        package.
+    *   Support configuring TPU software version from cloud tpu client.
+    *   Allowed TPU embedding weight decay factor to be multiplied by learning
+        rate.
+*   XLA Support:
+    *   Add standalone XLA AOT runtime target + relevant .cc sources to pip
+        package.
+    *   Add check for memory alignment to MemoryAllocation::MemoryAllocation()
+        on 32-bit ARM. This ensures a deterministic early exit instead of a hard
+        to debug bus error later.
+    *   `saved_model_cli aot_compile_cpu` allows you to compile saved models to
+        XLA header+object files and include them in your C++ programs.
+    *   Enable `Igamma`, `Igammac` for XLA.
+*   Deterministic Op Functionality:
+    *   XLA reduction emitter is deterministic when the environment variable
+        `TF_DETERMINISTIC_OPS` is set to "true" or "1". This extends
+        deterministic `tf.nn.bias_add` back-prop functionality (and therefore
+        also deterministic back-prop of bias-addition in Keras layers) to
+        include when XLA JIT compilation is enabled.
+    *   Fix problem, when running on a CUDA GPU and when either environment
+        variable `TF_DETERMINSTIC_OPS` or environment variable
+        `TF_CUDNN_DETERMINISTIC` is set to "true" or "1", in which some layer
+        configurations led to an exception with the message "No algorithm
+        worked!"
+*   Tracing and Debugging:
+    *   Add source, destination name to `_send` traceme to allow easier
+        debugging.
+    *   Add traceme event to `fastpathexecute`.
+*   Other:
+    *   Fix an issue with AUC.reset_states for multi-label AUC
+        [#35852](https://github.com/tensorflow/tensorflow/issues/35852)
+    *   Fix the TF upgrade script to not delete files when there is a parsing
+        error and the output mode is `in-place`.
+    *   Move `tensorflow/core:framework/*_pyclif` rules to
+        `tensorflow/core/framework:*_pyclif`.
 
 ## Thanks to our Contributors
 
diff --git a/tensorflow/c/c_api.cc b/tensorflow/c/c_api.cc
index 132761da4bf..36a08c8cfc9 100644
--- a/tensorflow/c/c_api.cc
+++ b/tensorflow/c/c_api.cc
@@ -589,14 +589,16 @@ void TF_DeleteDeviceList(TF_DeviceList* list) { delete list; }
 
 TF_DeviceList* TF_SessionListDevices(TF_Session* session, TF_Status* status) {
   TF_DeviceList* response = new TF_DeviceList;
-  status->status = session->session->ListDevices(&response->response);
+  if (session && session->session)
+    status->status = session->session->ListDevices(&response->response);
   return response;
 }
 
 TF_DeviceList* TF_DeprecatedSessionListDevices(TF_DeprecatedSession* session,
                                                TF_Status* status) {
   TF_DeviceList* response = new TF_DeviceList;
-  status->status = session->session->ListDevices(&response->response);
+  if (session && session->session)
+    status->status = session->session->ListDevices(&response->response);
   return response;
 }
 
@@ -1384,6 +1386,7 @@ void TF_OperationGetAttrStringList(TF_Operation* oper, const char* attr_name,
     cpp_type v;                                                              \
     status->status =                                                         \
         tensorflow::GetNodeAttr(oper->node.attrs(), attr_name, &v);          \
+    if (!status->status.ok()) return;                                        \
     *value = static_cast<c_type>(v);                                         \
   }                                                                          \
   void func##List(TF_Operation* oper, const char* attr_name, c_type* values, \
@@ -2178,6 +2181,7 @@ TF_Session* TF_NewSession(TF_Graph* graph, const TF_SessionOptions* opt,
     }
     return new_session;
   } else {
+    LOG(ERROR) << status->status;
     DCHECK_EQ(nullptr, session);
     return nullptr;
   }
diff --git a/tensorflow/c/eager/BUILD b/tensorflow/c/eager/BUILD
index 407acfe1ca9..0c953db40b2 100644
--- a/tensorflow/c/eager/BUILD
+++ b/tensorflow/c/eager/BUILD
@@ -369,6 +369,7 @@ tf_cuda_cc_test(
     args = ["--heap_check=local"],
     extra_copts = tfe_xla_copts(),
     tags = [
+        "no_windows",
         "noasan",  # leaks gRPC server instances
     ],
     deps = [
@@ -400,7 +401,10 @@ tf_cuda_cc_test(
     # TODO(b/136478427): Figure out how to correctly shut the server down
     args = ["--heap_check=local"],
     extra_copts = tfe_xla_copts(),
-    tags = ["noasan"],  # leaks gRPC server instances
+    tags = [
+        "no_windows",
+        "noasan",  # leaks gRPC server instances
+    ],
     deps = [
         ":c_api",
         ":c_api_experimental",
@@ -430,7 +434,10 @@ tf_cuda_cc_test(
     # TODO(b/136478427): Figure out how to correctly shut the server down
     args = ["--heap_check=local"],
     extra_copts = tfe_xla_copts(),
-    tags = ["noasan"],  # leaks gRPC server instances
+    tags = [
+        "no_windows",
+        "noasan",  # leaks gRPC server instances
+    ],
     deps = [
         ":c_api",
         ":c_api_experimental",
diff --git a/tensorflow/c/eager/c_api.cc b/tensorflow/c/eager/c_api.cc
index 5a39c17e1d9..865f8061ae0 100644
--- a/tensorflow/c/eager/c_api.cc
+++ b/tensorflow/c/eager/c_api.cc
@@ -1397,23 +1397,17 @@ void TFE_ContextAddFunctionDef(TFE_Context* ctx,
         tensorflow::errors::InvalidArgument("Invalid FunctionDef proto");
     return;
   }
-  tensorflow::EagerContext* context =
-      tensorflow::ContextFromInterface(tensorflow::unwrap(ctx));
-  status->status = context->AddFunctionDef(function_def);
+  status->status = tensorflow::unwrap(ctx)->AddFunctionDef(function_def);
 }
 
 void TFE_ContextAddFunction(TFE_Context* ctx, TF_Function* function,
                             TF_Status* status) {
-  tensorflow::EagerContext* context =
-      tensorflow::ContextFromInterface(tensorflow::unwrap(ctx));
-  status->status = context->AddFunctionDef(function->fdef);
+  status->status = tensorflow::unwrap(ctx)->AddFunctionDef(function->fdef);
 }
 
 void TFE_ContextRemoveFunction(TFE_Context* ctx, const char* name,
                                TF_Status* status) {
-  tensorflow::EagerContext* context =
-      tensorflow::ContextFromInterface(tensorflow::unwrap(ctx));
-  status->status = context->RemoveFunction(name);
+  status->status = tensorflow::unwrap(ctx)->RemoveFunction(name);
 }
 
 unsigned char TFE_ContextHasFunction(TFE_Context* ctx, const char* name) {
diff --git a/tensorflow/c/eager/context_interface.h b/tensorflow/c/eager/context_interface.h
index 2861fa43b66..9cd0669ff68 100644
--- a/tensorflow/c/eager/context_interface.h
+++ b/tensorflow/c/eager/context_interface.h
@@ -104,6 +104,14 @@ class AbstractContextInterface {
   // Block until all pending nodes are finished.
   virtual Status AsyncWait() = 0;
 
+  // Add a function (serialized FunctionDef protocol buffer) so that it can
+  // be executed as an op. Return error if the function with the same name
+  // already exists.
+  virtual Status AddFunctionDef(const FunctionDef& fdef) = 0;
+  // Remove a function. 'func' argument is the name of a previously added
+  // FunctionDef. The name is in fdef.signature.name.
+  virtual Status RemoveFunction(const string& func) = 0;
+
  protected:
   virtual ~AbstractContextInterface() {}
 };
diff --git a/tensorflow/c/eager/parallel_device/BUILD b/tensorflow/c/eager/parallel_device/BUILD
index 6fce918aab1..0d0e5ffce10 100644
--- a/tensorflow/c/eager/parallel_device/BUILD
+++ b/tensorflow/c/eager/parallel_device/BUILD
@@ -12,28 +12,69 @@ package(
 # need a second rule that omits .cc files, in
 # tensorflow/python:_pywrap_parallel_device.
 filegroup(
-    name = "headers",
+    name = "lib_headers",
+    srcs = ["parallel_device_lib.h"],
+)
+
+filegroup(
+    name = "lib_sources",
+    srcs = ["parallel_device_lib.cc"],
+)
+
+filegroup(
+    name = "device_headers",
     srcs = ["parallel_device.h"],
+)
+
+filegroup(
+    name = "device_sources",
+    srcs = ["parallel_device.cc"],
+)
+
+filegroup(
+    name = "headers",
+    srcs = [
+        ":device_headers",
+        ":lib_headers",
+    ],
     visibility = ["//tensorflow/python:__pkg__"],
 )
 
 filegroup(
     name = "sources",
-    srcs = ["parallel_device.cc"],
+    srcs = [
+        ":device_sources",
+        ":lib_sources",
+    ],
     visibility = ["//tensorflow/python:__pkg__"],
 )
 
 cc_library(
     name = "parallel_device",
-    srcs = [":sources"],
-    hdrs = [":headers"],
+    srcs = [":device_sources"],
+    hdrs = [":device_headers"],
+    visibility = ["//tensorflow:internal"],
+    deps = [
+        ":parallel_device_lib",
+        "//tensorflow/c:c_api",
+        "//tensorflow/c/eager:c_api",
+        "//tensorflow/c/eager:c_api_experimental",
+        "@com_google_absl//absl/strings",
+        "@com_google_absl//absl/types:optional",
+        "@com_google_absl//absl/types:variant",
+    ],
+)
+
+cc_library(
+    name = "parallel_device_lib",
+    srcs = [":lib_sources"],
+    hdrs = [":lib_headers"],
     visibility = ["//tensorflow:internal"],
     deps = [
         "//tensorflow/c:c_api",
         "//tensorflow/c/eager:c_api",
         "//tensorflow/c/eager:c_api_experimental",
         "//tensorflow/core:lib",
-        "@com_google_absl//absl/strings",
         "@com_google_absl//absl/types:optional",
         "@com_google_absl//absl/types:variant",
     ],
diff --git a/tensorflow/c/eager/parallel_device/parallel_device.cc b/tensorflow/c/eager/parallel_device/parallel_device.cc
index 06669cfcab6..eec893e704d 100644
--- a/tensorflow/c/eager/parallel_device/parallel_device.cc
+++ b/tensorflow/c/eager/parallel_device/parallel_device.cc
@@ -23,25 +23,13 @@ limitations under the License.
 #include "tensorflow/c/c_api.h"
 #include "tensorflow/c/eager/c_api.h"
 #include "tensorflow/c/eager/c_api_experimental.h"
+#include "tensorflow/c/eager/parallel_device/parallel_device_lib.h"
 #include "tensorflow/c/tf_status.h"
-#include "tensorflow/core/lib/gtl/cleanup.h"
 
 namespace tensorflow {
-namespace eager {
+namespace parallel_device {
 namespace {
 
-// Functor for making unique_ptrs slightly more ergonomic. Using
-// decltype(delete_fn) in the unique_ptr's second template argument requires
-// passing a function pointer to delete_fn when constructing the unique_ptr.
-class TensorHandleDeleter {
- public:
-  void operator()(TFE_TensorHandle* to_delete) const {
-    TFE_DeleteTensorHandle(to_delete);
-  }
-};
-
-using TensorHandlePtr = std::unique_ptr<TFE_TensorHandle, TensorHandleDeleter>;
-
 class OpDeleter {
  public:
   void operator()(TFE_Op* to_delete) const { TFE_DeleteOp(to_delete); }
@@ -49,224 +37,43 @@ class OpDeleter {
 
 using OpPtr = std::unique_ptr<TFE_Op, OpDeleter>;
 
-class ExecutorDeleter {
- public:
-  void operator()(TFE_Executor* to_delete) const {
-    TFE_DeleteExecutor(to_delete);
-  }
-};
-
-using ExecutorPtr = std::unique_ptr<TFE_Executor, ExecutorDeleter>;
-
-class ParallelTensor;
-
 using MaybeParallelTensorOwned =
     absl::variant<std::unique_ptr<ParallelTensor>, TensorHandlePtr>;
-using MaybeParallelTensorUnowned =
-    absl::variant<ParallelTensor*, TFE_TensorHandle*>;
 
-// Creates a vector of `count` new executors (threads).
-std::vector<ExecutorPtr> MakeExecutors(size_t count) {
-  std::vector<ExecutorPtr> executors;
-  executors.reserve(count);
-  for (int i = 0; i < count; ++i) {
-    executors.emplace_back(TFE_NewExecutor(true /* is_async */));
-  }
-  return executors;
-}
-
-// A representation of the custom device passed in and out of the TFE custom
-// device APIs, providing context about the parallel device to
-// ParallelDeviceExecute.
-class ParallelDevice {
+// A ParallelDevice on its own is not registered with a TFE_Context, and so has
+// no device name (e.g. for `tf.device`). `NamedParallelDevice` associates a
+// name with it, which lets us pack its `ParallelTensor`s into TFE_TensorHandles
+// placed on the parallel device.
+class NamedParallelDevice {
  public:
-  ParallelDevice(const std::string& name,
-                 const std::vector<std::string>& devices);
-
-  // Helper to copy a tensor handle from another device once for each component
-  // of the ParallelDevice.
-  //
-  // Sets a bad status and returns a nullptr if `tensor` is already on the
-  // ParallelDevice, or if the individual copies fail.
-  std::unique_ptr<ParallelTensor> CopyToParallelDevice(TFE_Context* context,
-                                                       TFE_TensorHandle* tensor,
-                                                       TF_Status* status) const;
-
-  // A parallel tensor with scalar integers numbering component devices.
-  std::unique_ptr<ParallelTensor> DeviceIDs(TFE_Context* context,
-                                            TF_Status* status) const;
-
-  // Takes a description of a single operation being executed on the
-  // ParallelDevice, and in turn runs one operation per component device with
-  // its corresponding inputs from the input ParallelTensors (or
-  // implicitly-mirrored tensors on other devices). Wraps the resulting
-  // per-device and per-output TFE_TensorHandles into one ParallelTensor per
-  // output of the original operation.
-  //
-  // `inputs` are either ParallelTensors, i.e. already on the ParallelDevice, or
-  // un-replicated TFE_TensorHandles on other devices. TPUReplicatedInput
-  // requires non-parallel tensors, and TPUReplicatedOutput requires a parallel
-  // tensor, but other operations will implicitly broadcast non-parallel input
-  // tensors across the ParallelDevice's component devices.
-  //
-  // Two special-cased operations, TPUReplicatedInput and TPUReplicatedOutput,
-  // pack and un-pack parallel tensors respectively. Only TPUReplicatedOutput
-  // causes `Execute` to return non-parallel tensors.
-  //
-  // Attributes are forwarded to executed operations unmodified.
-  //
-  // The returned optional has a value if and only if `status` evaluates to
-  // TF_OK.
-  absl::optional<std::vector<MaybeParallelTensorOwned>> Execute(
-      TFE_Context* context, std::vector<MaybeParallelTensorUnowned> inputs,
-      const char* operation_name, const TFE_OpAttrs* attributes,
-      int expected_max_outputs, TF_Status* status) const;
-
-  // Implements the parallel case for `Execute`, where all of the outputs of the
-  // operation are ParallelTensors, and all inputs are either ParallelTensors or
-  // should be implicitly broadcast. This means the operation is not
-  // TPUReplicatedInput or TPUReplicatedOutput.
-  //
-  // The returned optional has a value if and only if `status` evaluates to
-  // TF_OK. Bad statuses are forwarded from underlying `TFE_Execute` calls, or
-  // if sanity checks on dtypes/metadata fail.
-  absl::optional<std::vector<std::unique_ptr<ParallelTensor>>>
-  ExecuteParallelOperation(TFE_Context* context,
-                           std::vector<MaybeParallelTensorUnowned> inputs,
-                           const char* operation_name,
-                           const TFE_OpAttrs* attributes,
-                           int expected_max_outputs, TF_Status* status) const;
-
-  const std::string& device_name() const { return device_name_; }
+  NamedParallelDevice(const std::string& name,
+                      std::unique_ptr<ParallelDevice> parallel_device)
+      : device_name_(name), parallel_device_(std::move(parallel_device)) {}
+  const std::string& name() const { return device_name_; }
+  const ParallelDevice& device() const { return *parallel_device_; }
 
  private:
-  // The name of the parallel device
-  // (e.g. "/job:localhost/replica:0/task:0/device:CUSTOM:0")
-  const std::string device_name_;
-  // A sequence of device names, indicating which devices replicated operations
-  // are forwarded to.
-  const std::vector<std::string> underlying_devices_;
-  // A sequence of TFE_Executors, one per device, for executing operations in
-  // parallel.
-  const std::vector<ExecutorPtr> executors_;
+  std::string device_name_;
+  std::unique_ptr<ParallelDevice> parallel_device_;
 };
 
-// The internal representation of a TFE_TensorHandle placed on a
-// ParallelDevice. Contains a tuple of tensors, one on each of the
-// `underlying_devices_` of the ParallelDevice.
-class ParallelTensor {
- public:
-  // Construct a ParallelTensor from TensorHandles placed on the component
-  // devices of a ParallelDevice.
-  static std::unique_ptr<ParallelTensor> FromTensorHandles(
-      const ParallelDevice& parallel_device,
-      std::vector<TensorHandlePtr> components, TF_Status* status);
-
-  // Helper to wrap a ParallelTensor into a TFE_TensorHandle which contains it.
-  static TensorHandlePtr AsTensorHandle(TFE_Context* context,
-                                        std::unique_ptr<ParallelTensor> t,
-                                        TF_Status* status);
-
-  size_t num_tensors() const { return tensors_.size(); }
-  TFE_TensorHandle* tensor(size_t index) const { return tensors_[index].get(); }
-
- private:
-  ParallelTensor(const ParallelDevice& device,
-                 std::vector<TensorHandlePtr> tensors,
-                 std::vector<int64_t> shape, const TF_DataType dtype)
-      : device_(device),
-        tensors_(std::move(tensors)),
-        shape_(std::move(shape)),
-        dtype_(dtype) {}
-
-  const ParallelDevice& device_;
-  const std::vector<TensorHandlePtr> tensors_;
-  const std::vector<int64_t> shape_;
-  const TF_DataType dtype_;
-};
-
-ParallelDevice::ParallelDevice(const std::string& name,
-                               const std::vector<std::string>& devices)
-    : device_name_(name),
-      underlying_devices_(devices),
-      executors_(MakeExecutors(underlying_devices_.size())) {}
-
-std::unique_ptr<ParallelTensor> ParallelDevice::CopyToParallelDevice(
-    TFE_Context* context, TFE_TensorHandle* tensor, TF_Status* status) const {
-  const char* current_device = TFE_TensorHandleDeviceName(tensor, status);
-  if (device_name_ == current_device) {
-    std::string message(absl::StrCat(
-        "Tried to copy a TensorHandle to its existing device: ", device_name_));
-    TF_SetStatus(status, TF_INTERNAL, message.c_str());
-    return nullptr;
-  }
-  std::vector<TensorHandlePtr> components;
-  components.reserve(underlying_devices_.size());
-  for (const std::string& underlying_device_name : underlying_devices_) {
-    TFE_TensorHandle* t = TFE_TensorHandleCopyToDevice(
-        tensor, context, underlying_device_name.c_str(), status);
-    if (TF_GetCode(status) != TF_OK) return nullptr;
-    components.emplace_back(t);
-  }
-  return ParallelTensor::FromTensorHandles(*this, std::move(components),
-                                           status);
-}
-
-std::unique_ptr<ParallelTensor> ParallelDevice::DeviceIDs(
-    TFE_Context* context, TF_Status* status) const {
-  // TODO(allenl): We could cache DeviceIDs (keyed by context).
-  std::vector<TensorHandlePtr> components;
-  components.reserve(underlying_devices_.size());
-  for (int device_index = 0; device_index < underlying_devices_.size();
-       ++device_index) {
-    int64_t* device_id = new int64_t;
-    *device_id = device_index;
-    std::unique_ptr<TF_Tensor, decltype(&TF_DeleteTensor)> tensor(
-        TF_NewTensor(
-            TF_INT64, /*dims=*/nullptr, /*num_dims=*/0, device_id,
-            sizeof(int64_t),
-            [](void* data, size_t, void* arg) {
-              delete reinterpret_cast<int64_t*>(data);
-            },
-            nullptr),
-        TF_DeleteTensor);
-    // TODO(allenl): Here and when executing regular operations, we could hold
-    // on to one TFE_Op per device and just call TFE_ResetOp to avoid parsing
-    // device names repeatedly.
-    OpPtr const_op(TFE_NewOp(context, "Const", status));
-    if (TF_GetCode(status) != TF_OK) return nullptr;
-    TFE_OpSetDevice(const_op.get(), underlying_devices_[device_index].c_str(),
-                    status);
-    if (TF_GetCode(status) != TF_OK) return nullptr;
-    TFE_OpSetAttrTensor(const_op.get(), "value", tensor.get(), status);
-    if (TF_GetCode(status) != TF_OK) return nullptr;
-    TFE_OpSetAttrType(const_op.get(), "dtype", TF_INT64);
-    TFE_TensorHandle* device_handle;
-    int num_outputs = 1;
-    TFE_Execute(const_op.get(), &device_handle, &num_outputs, status);
-    if (TF_GetCode(status) != TF_OK) return nullptr;
-    components.emplace_back(device_handle);
-    if (TF_GetCode(status) != TF_OK) return nullptr;
-  }
-  return ParallelTensor::FromTensorHandles(*this, std::move(components),
-                                           status);
-}
-
-absl::optional<std::vector<MaybeParallelTensorOwned>> ParallelDevice::Execute(
-    TFE_Context* context, std::vector<MaybeParallelTensorUnowned> inputs,
-    const char* operation_name, const TFE_OpAttrs* attributes,
-    int expected_max_outputs, TF_Status* status) const {
+absl::optional<std::vector<MaybeParallelTensorOwned>> ExecuteWithSpecialOps(
+    const ParallelDevice& parallel_device,
+    const std::string& parallel_device_name, TFE_Context* context,
+    std::vector<MaybeParallelTensorUnowned> inputs, const char* operation_name,
+    const TFE_OpAttrs* attributes, int expected_max_outputs,
+    TF_Status* status) {
   absl::optional<std::vector<MaybeParallelTensorOwned>> result;
   // TODO(allenl): We should remove "TPU" from these op names at the very least,
   // or consider other ways of packing/unpacking parallel tensors.
   if (operation_name == std::string("TPUReplicatedInput")) {
     // Special-cased operation for packing per-device tensors into one parallel
     // tensor.
-    if (inputs.size() != underlying_devices_.size()) {
+    if (inputs.size() != parallel_device.num_underlying_devices()) {
       std::string message(absl::StrCat(
-          "The parallel device ", device_name_, " expected ",
-          underlying_devices_.size(), " inputs to TPUReplicatedInput, but got ",
-          inputs.size()));
+          "The parallel device ", parallel_device_name, " expected ",
+          parallel_device.num_underlying_devices(),
+          " inputs to TPUReplicatedInput, but got ", inputs.size()));
       TF_SetStatus(status, TF_INVALID_ARGUMENT, message.c_str());
       return result;
     }
@@ -289,7 +96,7 @@ absl::optional<std::vector<MaybeParallelTensorOwned>> ParallelDevice::Execute(
     std::vector<MaybeParallelTensorOwned> result_content;
     result_content.reserve(1);
     result_content.push_back(ParallelTensor::FromTensorHandles(
-        *this, std::move(components), status));
+        parallel_device, std::move(components), status));
     if (TF_GetCode(status) != TF_OK) return result;
     result.emplace(std::move(result_content));
     return result;
@@ -300,10 +107,10 @@ absl::optional<std::vector<MaybeParallelTensorOwned>> ParallelDevice::Execute(
     TFE_OpAddAttrs(op.get(), attributes);
     int expected_outputs = TFE_OpGetOutputLength(op.get(), "outputs", status);
     if (TF_GetCode(status) != TF_OK) return result;
-    if (expected_outputs != underlying_devices_.size()) {
+    if (expected_outputs != parallel_device.num_underlying_devices()) {
       std::string message(absl::StrCat(
-          "The parallel device ", device_name_, " expected ",
-          underlying_devices_.size(),
+          "The parallel device ", parallel_device_name, " expected ",
+          parallel_device.num_underlying_devices(),
           " outputs for TPUReplicatedOutput, but got ", expected_outputs));
       TF_SetStatus(status, TF_INVALID_ARGUMENT, message.c_str());
       return result;
@@ -329,15 +136,15 @@ absl::optional<std::vector<MaybeParallelTensorOwned>> ParallelDevice::Execute(
   } else if (operation_name == std::string("DeviceID")) {
     std::vector<MaybeParallelTensorOwned> result_content;
     result_content.reserve(1);
-    result_content.push_back(DeviceIDs(context, status));
+    result_content.push_back(parallel_device.DeviceIDs(context, status));
     if (TF_GetCode(status) != TF_OK) return result;
     result.emplace(std::move(result_content));
     return result;
   }
   absl::optional<std::vector<std::unique_ptr<ParallelTensor>>>
       maybe_parallel_results(
-          ExecuteParallelOperation(context, std::move(inputs), operation_name,
-                                   attributes, expected_max_outputs, status));
+          parallel_device.Execute(context, std::move(inputs), operation_name,
+                                  attributes, expected_max_outputs, status));
   if (!maybe_parallel_results.has_value()) return result;
   std::vector<std::unique_ptr<ParallelTensor>> parallel_results(
       std::move(maybe_parallel_results.value()));
@@ -351,153 +158,6 @@ absl::optional<std::vector<MaybeParallelTensorOwned>> ParallelDevice::Execute(
   return result;
 }
 
-absl::optional<std::vector<std::unique_ptr<ParallelTensor>>>
-ParallelDevice::ExecuteParallelOperation(
-    TFE_Context* context, std::vector<MaybeParallelTensorUnowned> inputs,
-    const char* operation_name, const TFE_OpAttrs* attributes,
-    int expected_max_outputs, TF_Status* status) const {
-  absl::optional<std::vector<std::unique_ptr<ParallelTensor>>> result;
-  // Compute per-device per-output tensors
-  std::vector<std::vector<TensorHandlePtr>> per_device_output_tensors;
-  per_device_output_tensors.reserve(underlying_devices_.size());
-  // TODO(allenl): Add a TFE_ExecuteWithExecutor API so we don't have to keep
-  // setting the thread-local executor like this.
-  TFE_Executor* previous_executor(TFE_ContextGetExecutorForThread(context));
-  auto reset_executor = gtl::MakeCleanup([context, previous_executor]() {
-    TFE_ContextSetExecutorForThread(context, previous_executor);
-    TFE_DeleteExecutor(previous_executor);
-  });
-  int first_op_output_count;
-  for (int device_index = 0; device_index < underlying_devices_.size();
-       ++device_index) {
-    TFE_Executor* executor = executors_[device_index].get();
-    // Note that the `reset_executor` cleanup sets the thread's executor back to
-    // the value before this function ran.
-    TFE_ContextSetExecutorForThread(context, executor);
-    OpPtr op(TFE_NewOp(context, operation_name, status));
-    if (TF_GetCode(status) != TF_OK) return result;
-    TFE_OpSetDevice(op.get(), underlying_devices_[device_index].c_str(),
-                    status);
-    TFE_OpAddAttrs(op.get(), attributes);
-    for (int input_index = 0; input_index < inputs.size(); ++input_index) {
-      if (absl::holds_alternative<TFE_TensorHandle*>(inputs[input_index])) {
-        // Non-parallel tensors are implicitly broadcast, i.e. set as the input
-        // to each parallel operation.
-        //
-        // TODO(allenl): There may be smarter ways to do this copy in some
-        // cases, i.e. with a collective broadcast. We'll need to be careful
-        // about things that are taken as inputs on the host or on their
-        // existing device (for multi-device functions).
-        TFE_OpAddInput(op.get(),
-                       absl::get<TFE_TensorHandle*>(inputs[input_index]),
-                       status);
-        if (TF_GetCode(status) != TF_OK) return result;
-      } else {
-        // Parallel tensors are divided between operations by device.
-        TFE_OpAddInput(op.get(),
-                       absl::get<ParallelTensor*>(inputs[input_index])
-                           ->tensor(device_index),
-                       status);
-        if (TF_GetCode(status) != TF_OK) return result;
-      }
-    }
-    std::vector<TFE_TensorHandle*> op_outputs(expected_max_outputs);
-    int real_num_outputs = expected_max_outputs;
-    // For nested devices, the inner device sees the async executor we've
-    // set. Inner parallel devices will just overwrite this with their own and
-    // then set it back to ours before returning. This means parallel devices
-    // which consist of several aliased parallel devices would hypothetically
-    // deadlock if the outer parallel device ran one collective with a group
-    // size equal to the total number of aliased physical devices. Currently
-    // physical devices cannot participate in a single collective reduction
-    // multiple times, so this would fail earlier.
-    //
-    // TODO(allenl): Keep a map from outer executor to list of inner executors
-    // rather than a single list of executors so aliased nested parallel devices
-    // don't re-use an executor.
-    TFE_Execute(op.get(), op_outputs.data(), &real_num_outputs, status);
-    if (device_index == 0) {
-      first_op_output_count = real_num_outputs;
-    } else {
-      if (real_num_outputs != first_op_output_count) {
-        TF_SetStatus(status, TF_INTERNAL,
-                     "Parallel ops produced different numbers of tensors.");
-        return result;
-      }
-    }
-    if (TF_GetCode(status) != TF_OK) return result;
-    std::vector<TensorHandlePtr> this_outputs;
-    this_outputs.reserve(real_num_outputs);
-    for (int output_num = 0; output_num < real_num_outputs; ++output_num) {
-      this_outputs.emplace_back(op_outputs[output_num]);
-    }
-    per_device_output_tensors.push_back(std::move(this_outputs));
-  }
-  for (int device_index = 0; device_index < underlying_devices_.size();
-       ++device_index) {
-    TFE_Executor* executor = executors_[device_index].get();
-    // TODO(b/157523095): Syncing the executor here shouldn't be
-    // necessary. Currently async+remote is missing cross-executor
-    // coordination.
-    TFE_ExecutorWaitForAllPendingNodes(executor, status);
-    if (TF_GetCode(status) != TF_OK) return result;
-  }
-  // For each output of the original operation, pack the per-device
-  // TensorHandles we've computed into a single parallel TensorHandle.
-  std::vector<std::unique_ptr<ParallelTensor>> per_device_outputs;
-  per_device_outputs.reserve(first_op_output_count);
-  for (int i = 0; i < first_op_output_count; ++i) {
-    std::vector<TensorHandlePtr> components;
-    components.reserve(underlying_devices_.size());
-    for (int j = 0; j < underlying_devices_.size(); ++j) {
-      components.push_back(std::move(per_device_output_tensors[j][i]));
-    }
-    per_device_outputs.push_back(ParallelTensor::FromTensorHandles(
-        *this, std::move(components), status));
-    if (TF_GetCode(status) != TF_OK) return result;
-  }
-  result.emplace(std::move(per_device_outputs));
-  return result;
-}
-
-std::unique_ptr<ParallelTensor> ParallelTensor::FromTensorHandles(
-    const ParallelDevice& parallel_device,
-    std::vector<TensorHandlePtr> components, TF_Status* status) {
-  TF_DataType dtype = TFE_TensorHandleDataType(components[0].get());
-  std::vector<int64_t> shape(
-      TFE_TensorHandleNumDims(components[0].get(), status));
-  if (TF_GetCode(status) != TF_OK) return nullptr;
-  for (int i = 0; i < shape.size(); ++i) {
-    shape[i] = TFE_TensorHandleDim(components[0].get(), i, status);
-    if (TF_GetCode(status) != TF_OK) return nullptr;
-  }
-
-  // Verify that the TensorHandle's shape and dtype match all of the component
-  // shapes and dtypes.
-  for (TensorHandlePtr& component : components) {
-    for (int i = 0; i < shape.size(); ++i) {
-      int64_t tensor_dim = TFE_TensorHandleDim(component.get(), i, status);
-      if (TF_GetCode(status) != TF_OK) return nullptr;
-      if (tensor_dim != shape[i]) {
-        // TODO(allenl): Allow shapes to differ.
-        TF_SetStatus(status, TF_UNIMPLEMENTED,
-                     "Components of a ParallelTensor must currently all have "
-                     "the same shape");
-        return nullptr;
-      }
-      if (TFE_TensorHandleDataType(component.get()) != dtype) {
-        TF_SetStatus(status, TF_INTERNAL,
-                     "Components of a ParallelTensor must all have "
-                     "the same dtype");
-        return nullptr;
-      }
-    }
-  }
-
-  return std::unique_ptr<ParallelTensor>(new ParallelTensor(
-      parallel_device, std::move(components), std::move(shape), dtype));
-}
-
 // Used as an argument to TFE_NewTensorHandleFromDeviceMemory, indicating how
 // ParallelTensors wrapped in TFE_TensorHandles should be cleaned up once their
 // reference counts drop to zero.
@@ -505,17 +165,18 @@ void ParallelTensorDeallocator(void* data, size_t len, void* arg) {
   delete reinterpret_cast<ParallelTensor*>(data);
 }
 
-TensorHandlePtr ParallelTensor::AsTensorHandle(
-    TFE_Context* context, std::unique_ptr<ParallelTensor> t,
-    TF_Status* status) {
+TensorHandlePtr ParallelTensorToTensorHandle(
+    const std::string& parallel_device_name, TFE_Context* context,
+    std::unique_ptr<ParallelTensor> t, TF_Status* status) {
   // The resulting TensorHandle owns an opaque pointer to "device memory", which
   // for a ParallelDevice is really a ParallelTensor. When the TensorHandle is
   // deleted, it will call ParallelTensorDeallocator to free the struct.
   ParallelTensor* t_released = t.release();
+  const std::vector<int64_t>& shape(t_released->shape());
   return TensorHandlePtr(TFE_NewTensorHandleFromDeviceMemory(
-      context, t_released->device_.device_name().c_str(), t_released->dtype_,
-      t_released->shape_.data(), t_released->shape_.size(), t_released, 1,
-      &ParallelTensorDeallocator, nullptr, status));
+      context, parallel_device_name.c_str(), t_released->dtype(), shape.data(),
+      shape.size(), t_released, 1, &ParallelTensorDeallocator, nullptr,
+      status));
 }
 
 // For TFE_CustomDevice::copy_tensor_to_device in the parallel device
@@ -531,12 +192,14 @@ TensorHandlePtr ParallelTensor::AsTensorHandle(
 TFE_TensorHandle* CopyToParallelDevice(TFE_Context* context,
                                        TFE_TensorHandle* tensor,
                                        TF_Status* status, void* device_info) {
-  ParallelDevice* dev = reinterpret_cast<ParallelDevice*>(device_info);
+  NamedParallelDevice* named_device =
+      reinterpret_cast<NamedParallelDevice*>(device_info);
+  const ParallelDevice& dev = named_device->device();
   std::unique_ptr<ParallelTensor> parallel_tensor(
-      dev->CopyToParallelDevice(context, tensor, status));
+      dev.CopyToParallelDevice(context, tensor, status));
   if (TF_GetCode(status) != TF_OK) return nullptr;
-  return ParallelTensor::AsTensorHandle(context, std::move(parallel_tensor),
-                                        status)
+  return ParallelTensorToTensorHandle(named_device->name(), context,
+                                      std::move(parallel_tensor), status)
       .release();
 }
 
@@ -570,14 +233,15 @@ void ParallelDeviceExecute(TFE_Context* context, int num_inputs,
                            const TFE_OpAttrs* attributes, int* num_outputs,
                            TFE_TensorHandle** outputs, TF_Status* status,
                            void* device_info) {
-  ParallelDevice* dev = reinterpret_cast<ParallelDevice*>(device_info);
+  NamedParallelDevice* named_device =
+      reinterpret_cast<NamedParallelDevice*>(device_info);
   std::vector<MaybeParallelTensorUnowned> typed_inputs;
   typed_inputs.reserve(num_inputs);
   for (int i = 0; i < num_inputs; ++i) {
     const char* tensor_handle_device =
         TFE_TensorHandleDeviceName(inputs[i], status);
     if (TF_GetCode(status) != TF_OK) return;
-    if (dev->device_name() == tensor_handle_device) {
+    if (named_device->name() == tensor_handle_device) {
       // We assume that any tensors already placed on this device are
       // ParallelTensors.
       typed_inputs.emplace_back(reinterpret_cast<ParallelTensor*>(
@@ -589,8 +253,9 @@ void ParallelDeviceExecute(TFE_Context* context, int num_inputs,
   }
 
   absl::optional<std::vector<MaybeParallelTensorOwned>> maybe_typed_outputs(
-      dev->Execute(context, std::move(typed_inputs), operation_name, attributes,
-                   *num_outputs, status));
+      ExecuteWithSpecialOps(named_device->device(), named_device->name(),
+                            context, std::move(typed_inputs), operation_name,
+                            attributes, *num_outputs, status));
   if (TF_GetCode(status) != TF_OK) return;
   if (!maybe_typed_outputs.has_value()) {
     TF_SetStatus(status, TF_INTERNAL, "OK status but no value was returned.");
@@ -611,8 +276,8 @@ void ParallelDeviceExecute(TFE_Context* context, int num_inputs,
     if (absl::holds_alternative<TensorHandlePtr>(typed_output)) {
       outputs[i] = absl::get<TensorHandlePtr>(typed_output).release();
     } else {
-      outputs[i] = ParallelTensor::AsTensorHandle(
-                       context,
+      outputs[i] = ParallelTensorToTensorHandle(
+                       named_device->name(), context,
                        std::move(absl::get<std::unique_ptr<ParallelTensor>>(
                            typed_output)),
                        status)
@@ -629,7 +294,7 @@ void ParallelDeviceExecute(TFE_Context* context, int num_inputs,
 // device_info is passed in using a C-style generic. It must always be a
 // ParallelDevice.
 void DeleteParallelDevice(void* device_info) {
-  delete reinterpret_cast<ParallelDevice*>(device_info);
+  delete reinterpret_cast<NamedParallelDevice*>(device_info);
 }
 
 }  // namespace
@@ -648,8 +313,10 @@ void AllocateParallelDevice(const char* device_name,
        ++device_index) {
     underlying_devices_vector.push_back(underlying_devices[device_index]);
   }
-  *device_info = new ParallelDevice(device_name, underlying_devices_vector);
+  std::unique_ptr<ParallelDevice> parallel_device(
+      new ParallelDevice(underlying_devices_vector));
+  *device_info =
+      new NamedParallelDevice{device_name, std::move(parallel_device)};
 }
-
-}  // namespace eager
+}  // namespace parallel_device
 }  // namespace tensorflow
diff --git a/tensorflow/c/eager/parallel_device/parallel_device.h b/tensorflow/c/eager/parallel_device/parallel_device.h
index f448a4c5b83..b8e571b8aaf 100644
--- a/tensorflow/c/eager/parallel_device/parallel_device.h
+++ b/tensorflow/c/eager/parallel_device/parallel_device.h
@@ -21,7 +21,7 @@ limitations under the License.
 #include "tensorflow/c/eager/c_api_experimental.h"
 
 namespace tensorflow {
-namespace eager {
+namespace parallel_device {
 
 // Allocate a parallel device named `device_name` which forwards operations to
 // `underlying_devices`, maintaining "parallel tensors" with components placed
@@ -59,7 +59,7 @@ void AllocateParallelDevice(const char* device_name,
                             int num_underlying_devices,
                             TFE_CustomDevice* device, void** device_info);
 
-}  // namespace eager
+}  // namespace parallel_device
 }  // namespace tensorflow
 
 #endif  // TENSORFLOW_C_EAGER_PARALLEL_DEVICE_PARALLEL_DEVICE_H_
diff --git a/tensorflow/c/eager/parallel_device/parallel_device_lib.cc b/tensorflow/c/eager/parallel_device/parallel_device_lib.cc
new file mode 100644
index 00000000000..f56b8d8ac88
--- /dev/null
+++ b/tensorflow/c/eager/parallel_device/parallel_device_lib.cc
@@ -0,0 +1,251 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include "tensorflow/c/eager/parallel_device/parallel_device_lib.h"
+
+#include "tensorflow/core/lib/gtl/cleanup.h"
+
+namespace tensorflow {
+namespace parallel_device {
+namespace {
+
+class OpDeleter {
+ public:
+  void operator()(TFE_Op* to_delete) const { TFE_DeleteOp(to_delete); }
+};
+
+using OpPtr = std::unique_ptr<TFE_Op, OpDeleter>;
+
+// Creates a vector of `count` new executors (threads).
+std::vector<ExecutorPtr> MakeExecutors(size_t count) {
+  std::vector<ExecutorPtr> executors;
+  executors.reserve(count);
+  for (int i = 0; i < count; ++i) {
+    executors.emplace_back(TFE_NewExecutor(true /* is_async */));
+  }
+  return executors;
+}
+
+}  // namespace
+
+ParallelDevice::ParallelDevice(const std::vector<std::string>& devices)
+    : underlying_devices_(devices),
+      executors_(MakeExecutors(underlying_devices_.size())) {}
+
+std::unique_ptr<ParallelTensor> ParallelDevice::CopyToParallelDevice(
+    TFE_Context* context, TFE_TensorHandle* tensor, TF_Status* status) const {
+  std::vector<TensorHandlePtr> components;
+  components.reserve(underlying_devices_.size());
+  for (const std::string& underlying_device_name : underlying_devices_) {
+    TFE_TensorHandle* t = TFE_TensorHandleCopyToDevice(
+        tensor, context, underlying_device_name.c_str(), status);
+    if (TF_GetCode(status) != TF_OK) return nullptr;
+    components.emplace_back(t);
+  }
+  return ParallelTensor::FromTensorHandles(*this, std::move(components),
+                                           status);
+}
+
+std::unique_ptr<ParallelTensor> ParallelDevice::DeviceIDs(
+    TFE_Context* context, TF_Status* status) const {
+  // TODO(allenl): We could cache DeviceIDs (keyed by context).
+  std::vector<TensorHandlePtr> components;
+  components.reserve(underlying_devices_.size());
+  for (int device_index = 0; device_index < underlying_devices_.size();
+       ++device_index) {
+    int64_t* device_id = new int64_t;
+    *device_id = device_index;
+    std::unique_ptr<TF_Tensor, decltype(&TF_DeleteTensor)> tensor(
+        TF_NewTensor(
+            TF_INT64, /*dims=*/nullptr, /*num_dims=*/0, device_id,
+            sizeof(int64_t),
+            [](void* data, size_t, void* arg) {
+              delete reinterpret_cast<int64_t*>(data);
+            },
+            nullptr),
+        TF_DeleteTensor);
+    // TODO(allenl): Here and when executing regular operations, we could hold
+    // on to one TFE_Op per device and just call TFE_ResetOp to avoid parsing
+    // device names repeatedly.
+    OpPtr const_op(TFE_NewOp(context, "Const", status));
+    if (TF_GetCode(status) != TF_OK) return nullptr;
+    TFE_OpSetDevice(const_op.get(), underlying_devices_[device_index].c_str(),
+                    status);
+    if (TF_GetCode(status) != TF_OK) return nullptr;
+    TFE_OpSetAttrTensor(const_op.get(), "value", tensor.get(), status);
+    if (TF_GetCode(status) != TF_OK) return nullptr;
+    TFE_OpSetAttrType(const_op.get(), "dtype", TF_INT64);
+    TFE_TensorHandle* device_handle;
+    int num_outputs = 1;
+    TFE_Execute(const_op.get(), &device_handle, &num_outputs, status);
+    if (TF_GetCode(status) != TF_OK) return nullptr;
+    components.emplace_back(device_handle);
+    if (TF_GetCode(status) != TF_OK) return nullptr;
+  }
+  return ParallelTensor::FromTensorHandles(*this, std::move(components),
+                                           status);
+}
+
+absl::optional<std::vector<std::unique_ptr<ParallelTensor>>>
+ParallelDevice::Execute(TFE_Context* context,
+                        std::vector<MaybeParallelTensorUnowned> inputs,
+                        const char* operation_name,
+                        const TFE_OpAttrs* attributes, int expected_max_outputs,
+                        TF_Status* status) const {
+  absl::optional<std::vector<std::unique_ptr<ParallelTensor>>> result;
+  // Compute per-device per-output tensors
+  std::vector<std::vector<TensorHandlePtr>> per_device_output_tensors;
+  per_device_output_tensors.reserve(underlying_devices_.size());
+  // TODO(allenl): Add a TFE_ExecuteWithExecutor API so we don't have to keep
+  // setting the thread-local executor like this.
+  TFE_Executor* previous_executor(TFE_ContextGetExecutorForThread(context));
+  auto reset_executor =
+      tensorflow::gtl::MakeCleanup([context, previous_executor]() {
+        TFE_ContextSetExecutorForThread(context, previous_executor);
+        TFE_DeleteExecutor(previous_executor);
+      });
+  int first_op_output_count;
+  for (int device_index = 0; device_index < underlying_devices_.size();
+       ++device_index) {
+    TFE_Executor* executor = executors_[device_index].get();
+    // Note that the `reset_executor` cleanup sets the thread's executor back to
+    // the value before this function ran.
+    TFE_ContextSetExecutorForThread(context, executor);
+    OpPtr op(TFE_NewOp(context, operation_name, status));
+    if (TF_GetCode(status) != TF_OK) return result;
+    TFE_OpSetDevice(op.get(), underlying_devices_[device_index].c_str(),
+                    status);
+    TFE_OpAddAttrs(op.get(), attributes);
+    for (int input_index = 0; input_index < inputs.size(); ++input_index) {
+      if (absl::holds_alternative<TFE_TensorHandle*>(inputs[input_index])) {
+        // Non-parallel tensors are implicitly broadcast, i.e. set as the input
+        // to each parallel operation.
+        //
+        // TODO(allenl): There may be smarter ways to do this copy in some
+        // cases, i.e. with a collective broadcast. We'll need to be careful
+        // about things that are taken as inputs on the host or on their
+        // existing device (for multi-device functions).
+        TFE_OpAddInput(op.get(),
+                       absl::get<TFE_TensorHandle*>(inputs[input_index]),
+                       status);
+        if (TF_GetCode(status) != TF_OK) return result;
+      } else {
+        // Parallel tensors are divided between operations by device.
+        TFE_OpAddInput(op.get(),
+                       absl::get<ParallelTensor*>(inputs[input_index])
+                           ->tensor(device_index),
+                       status);
+        if (TF_GetCode(status) != TF_OK) return result;
+      }
+    }
+    std::vector<TFE_TensorHandle*> op_outputs(expected_max_outputs);
+    int real_num_outputs = expected_max_outputs;
+    // For nested devices, the inner device sees the async executor we've
+    // set. Inner parallel devices will just overwrite this with their own and
+    // then set it back to ours before returning. This means parallel devices
+    // which consist of several aliased parallel devices would hypothetically
+    // deadlock if the outer parallel device ran one collective with a group
+    // size equal to the total number of aliased physical devices. Currently
+    // physical devices cannot participate in a single collective reduction
+    // multiple times, so this would fail earlier.
+    //
+    // TODO(allenl): Keep a map from outer executor to list of inner executors
+    // rather than a single list of executors so aliased nested parallel devices
+    // don't re-use an executor.
+    TFE_Execute(op.get(), op_outputs.data(), &real_num_outputs, status);
+    if (device_index == 0) {
+      first_op_output_count = real_num_outputs;
+    } else {
+      if (real_num_outputs != first_op_output_count) {
+        TF_SetStatus(status, TF_INTERNAL,
+                     "Parallel ops produced different numbers of tensors.");
+        return result;
+      }
+    }
+    if (TF_GetCode(status) != TF_OK) return result;
+    std::vector<TensorHandlePtr> this_outputs;
+    this_outputs.reserve(real_num_outputs);
+    for (int output_num = 0; output_num < real_num_outputs; ++output_num) {
+      this_outputs.emplace_back(op_outputs[output_num]);
+    }
+    per_device_output_tensors.push_back(std::move(this_outputs));
+  }
+  for (int device_index = 0; device_index < underlying_devices_.size();
+       ++device_index) {
+    TFE_Executor* executor = executors_[device_index].get();
+    // TODO(b/157523095): Syncing the executor here shouldn't be
+    // necessary. Currently async+remote is missing cross-executor
+    // coordination.
+    TFE_ExecutorWaitForAllPendingNodes(executor, status);
+    if (TF_GetCode(status) != TF_OK) return result;
+  }
+  // For each output of the original operation, pack the per-device
+  // TensorHandles we've computed into a single parallel TensorHandle.
+  std::vector<std::unique_ptr<ParallelTensor>> per_device_outputs;
+  per_device_outputs.reserve(first_op_output_count);
+  for (int i = 0; i < first_op_output_count; ++i) {
+    std::vector<TensorHandlePtr> components;
+    components.reserve(underlying_devices_.size());
+    for (int j = 0; j < underlying_devices_.size(); ++j) {
+      components.push_back(std::move(per_device_output_tensors[j][i]));
+    }
+    per_device_outputs.push_back(ParallelTensor::FromTensorHandles(
+        *this, std::move(components), status));
+    if (TF_GetCode(status) != TF_OK) return result;
+  }
+  result.emplace(std::move(per_device_outputs));
+  return result;
+}
+
+std::unique_ptr<ParallelTensor> ParallelTensor::FromTensorHandles(
+    const ParallelDevice& parallel_device,
+    std::vector<TensorHandlePtr> components, TF_Status* status) {
+  TF_DataType dtype = TFE_TensorHandleDataType(components[0].get());
+  std::vector<int64_t> shape(
+      TFE_TensorHandleNumDims(components[0].get(), status));
+  if (TF_GetCode(status) != TF_OK) return nullptr;
+  for (int i = 0; i < shape.size(); ++i) {
+    shape[i] = TFE_TensorHandleDim(components[0].get(), i, status);
+    if (TF_GetCode(status) != TF_OK) return nullptr;
+  }
+
+  // Verify that the TensorHandle's shape and dtype match all of the component
+  // shapes and dtypes.
+  for (TensorHandlePtr& component : components) {
+    for (int i = 0; i < shape.size(); ++i) {
+      int64_t tensor_dim = TFE_TensorHandleDim(component.get(), i, status);
+      if (TF_GetCode(status) != TF_OK) return nullptr;
+      if (tensor_dim != shape[i]) {
+        // TODO(allenl): Allow shapes to differ.
+        TF_SetStatus(status, TF_UNIMPLEMENTED,
+                     "Components of a ParallelTensor must currently all have "
+                     "the same shape");
+        return nullptr;
+      }
+      if (TFE_TensorHandleDataType(component.get()) != dtype) {
+        TF_SetStatus(status, TF_INTERNAL,
+                     "Components of a ParallelTensor must all have "
+                     "the same dtype");
+        return nullptr;
+      }
+    }
+  }
+
+  return std::unique_ptr<ParallelTensor>(new ParallelTensor(
+      parallel_device, std::move(components), std::move(shape), dtype));
+}
+
+}  // namespace parallel_device
+}  // namespace tensorflow
diff --git a/tensorflow/c/eager/parallel_device/parallel_device_lib.h b/tensorflow/c/eager/parallel_device/parallel_device_lib.h
new file mode 100644
index 00000000000..377377bc9c1
--- /dev/null
+++ b/tensorflow/c/eager/parallel_device/parallel_device_lib.h
@@ -0,0 +1,141 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#ifndef TENSORFLOW_C_EAGER_PARALLEL_DEVICE_PARALLEL_DEVICE_LIB_H_
+#define TENSORFLOW_C_EAGER_PARALLEL_DEVICE_PARALLEL_DEVICE_LIB_H_
+
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "absl/types/optional.h"
+#include "absl/types/variant.h"
+#include "tensorflow/c/c_api.h"
+#include "tensorflow/c/eager/c_api.h"
+#include "tensorflow/c/eager/c_api_experimental.h"
+
+namespace tensorflow {
+namespace parallel_device {
+
+// Functor for making unique_ptrs slightly more ergonomic. Using
+// decltype(delete_fn) in the unique_ptr's second template argument requires
+// passing a function pointer to delete_fn when constructing the unique_ptr.
+class TensorHandleDeleter {
+ public:
+  void operator()(TFE_TensorHandle* to_delete) const {
+    TFE_DeleteTensorHandle(to_delete);
+  }
+};
+
+using TensorHandlePtr = std::unique_ptr<TFE_TensorHandle, TensorHandleDeleter>;
+
+class ExecutorDeleter {
+ public:
+  void operator()(TFE_Executor* to_delete) const {
+    TFE_DeleteExecutor(to_delete);
+  }
+};
+
+using ExecutorPtr = std::unique_ptr<TFE_Executor, ExecutorDeleter>;
+
+class ParallelTensor;
+
+using MaybeParallelTensorUnowned =
+    absl::variant<ParallelTensor*, TFE_TensorHandle*>;
+
+// Forwards operations to `devices`, maintaining ParallelTensor with components
+// placed on each underlying device.
+class ParallelDevice {
+ public:
+  explicit ParallelDevice(const std::vector<std::string>& devices);
+
+  // Helper to copy a tensor handle from another device once for each component
+  // of the ParallelDevice.
+  //
+  // Sets a bad status and returns a nullptr if `tensor` is already on the
+  // ParallelDevice, or if the individual copies fail.
+  std::unique_ptr<ParallelTensor> CopyToParallelDevice(TFE_Context* context,
+                                                       TFE_TensorHandle* tensor,
+                                                       TF_Status* status) const;
+
+  // A parallel tensor with scalar integers numbering component devices.
+  std::unique_ptr<ParallelTensor> DeviceIDs(TFE_Context* context,
+                                            TF_Status* status) const;
+
+  // The number of devices operations run on.
+  size_t num_underlying_devices() const { return underlying_devices_.size(); }
+
+  // Takes a description of a single operation being executed on the
+  // ParallelDevice, and in turn runs one operation per component device with
+  // its corresponding inputs from the input ParallelTensors (or
+  // implicitly-mirrored tensors on other devices). Wraps the resulting
+  // per-device and per-output TFE_TensorHandles into one ParallelTensor per
+  // output of the original operation.
+  //
+  // Attributes are forwarded to executed operations unmodified.
+  //
+  // The returned optional has a value if and only if `status` evaluates to
+  // TF_OK. Bad statuses are forwarded from underlying `TFE_Execute` calls, or
+  // if sanity checks on dtypes/metadata fail.
+  absl::optional<std::vector<std::unique_ptr<ParallelTensor>>> Execute(
+      TFE_Context* context, std::vector<MaybeParallelTensorUnowned> inputs,
+      const char* operation_name, const TFE_OpAttrs* attributes,
+      int expected_max_outputs, TF_Status* status) const;
+
+ private:
+  // A sequence of device names, indicating which devices replicated operations
+  // are forwarded to.
+  const std::vector<std::string> underlying_devices_;
+  // A sequence of TFE_Executors, one per device, for executing operations in
+  // parallel.
+  const std::vector<ExecutorPtr> executors_;
+};
+
+// Contains a tuple of tensors, one on each of the `underlying_devices_` of the
+// ParallelDevice.
+class ParallelTensor {
+ public:
+  // Construct a ParallelTensor from TensorHandles placed on the component
+  // devices of a ParallelDevice.
+  static std::unique_ptr<ParallelTensor> FromTensorHandles(
+      const ParallelDevice& parallel_device,
+      std::vector<TensorHandlePtr> components, TF_Status* status);
+
+  size_t num_tensors() const { return tensors_.size(); }
+  TFE_TensorHandle* tensor(size_t index) const { return tensors_[index].get(); }
+
+  // A generalization of the shapes of the underlying tensors.
+  const std::vector<int64_t>& shape() const { return shape_; }
+  TF_DataType dtype() const { return dtype_; }
+
+ private:
+  ParallelTensor(const ParallelDevice& device,
+                 std::vector<TensorHandlePtr> tensors,
+                 std::vector<int64_t> shape, const TF_DataType dtype)
+      : device_(device),
+        tensors_(std::move(tensors)),
+        shape_(std::move(shape)),
+        dtype_(dtype) {}
+
+  const ParallelDevice& device_;
+  const std::vector<TensorHandlePtr> tensors_;
+  const std::vector<int64_t> shape_;
+  const TF_DataType dtype_;
+};
+
+}  // namespace parallel_device
+}  // namespace tensorflow
+
+#endif  // TENSORFLOW_C_EAGER_PARALLEL_DEVICE_PARALLEL_DEVICE_LIB_H_
diff --git a/tensorflow/c/eager/parallel_device/parallel_device_testlib.h b/tensorflow/c/eager/parallel_device/parallel_device_testlib.h
index fdd21087949..3f917224187 100644
--- a/tensorflow/c/eager/parallel_device/parallel_device_testlib.h
+++ b/tensorflow/c/eager/parallel_device/parallel_device_testlib.h
@@ -165,7 +165,7 @@ void RegisterParallelDevice(
     TF_Status* status) {
   TFE_CustomDevice device;
   void* device_info;
-  tensorflow::eager::AllocateParallelDevice(
+  tensorflow::parallel_device::AllocateParallelDevice(
       device_name, underlying_devices.data(), underlying_devices.size(),
       &device, &device_info);
   TFE_RegisterCustomDevice(context, device, device_name, device_info, status);
diff --git a/tensorflow/compiler/aot/BUILD b/tensorflow/compiler/aot/BUILD
index fd4ae10595b..a57fef7ec10 100644
--- a/tensorflow/compiler/aot/BUILD
+++ b/tensorflow/compiler/aot/BUILD
@@ -67,13 +67,13 @@ cc_library(
         "//tensorflow/core:lib",
         "//tensorflow/core:lib_internal",
         "//tensorflow/core:protos_all_cc",
-        "@llvm-project//llvm:arm_code_gen",  # fixdeps: keep
-        "@llvm-project//llvm:powerpc_code_gen",  # fixdeps: keep
-        "@llvm-project//llvm:target",
-        "@llvm-project//llvm:x86_code_gen",  # fixdeps: keep
+        "@llvm-project//llvm:arm_target",  # fixdeps: keep
+        "@llvm-project//llvm:powerpc_target",  # fixdeps: keep
+        "@llvm-project//llvm:target_base",
+        "@llvm-project//llvm:x86_target",  # fixdeps: keep
         "//tensorflow/core:regexp_internal",
     ] + if_llvm_aarch64_available([
-        "//third_party/llvm/llvm-project/llvm:aarch64_target",  # fixdeps: keep
+        "@llvm-project//llvm:aarch64_target",  # fixdeps: keep
     ]),
 )
 
@@ -95,7 +95,7 @@ tf_cc_test(
         "//tensorflow/core/platform:resource_loader",
         "@com_google_absl//absl/strings",
         "@llvm-project//llvm:support",  # fixdeps: keep
-        "@llvm-project//llvm:x86_code_gen",  # fixdeps: keep
+        "@llvm-project//llvm:x86_target",  # fixdeps: keep
     ],
 )
 
@@ -109,12 +109,12 @@ cc_library(
     name = "llvm_targets",
     visibility = ["//tensorflow/python:__pkg__"],
     deps = [
-        "@llvm-project//llvm:arm_code_gen",  # fixdeps: keep
-        "@llvm-project//llvm:powerpc_code_gen",  # fixdeps: keep
-        "@llvm-project//llvm:target",
-        "@llvm-project//llvm:x86_code_gen",  # fixdeps: keep
+        "@llvm-project//llvm:arm_target",  # fixdeps: keep
+        "@llvm-project//llvm:powerpc_target",  # fixdeps: keep
+        "@llvm-project//llvm:target_base",
+        "@llvm-project//llvm:x86_target",  # fixdeps: keep
     ] + if_llvm_aarch64_available([
-        "//third_party/llvm/llvm-project/llvm:aarch64_target",  # fixdeps: keep
+        "@llvm-project//llvm:aarch64_target",  # fixdeps: keep
     ]),
 )
 
@@ -286,9 +286,9 @@ cc_library(
         "@com_google_absl//absl/memory",
         "@com_google_absl//absl/strings",
         "@com_google_absl//absl/types:span",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
-        "@llvm-project//llvm:target",
+        "@llvm-project//llvm:target_base",
     ],
 )
 
diff --git a/tensorflow/compiler/mlir/lite/flatbuffer_import.cc b/tensorflow/compiler/mlir/lite/flatbuffer_import.cc
index 59b0b07a2ed..185f31dd093 100644
--- a/tensorflow/compiler/mlir/lite/flatbuffer_import.cc
+++ b/tensorflow/compiler/mlir/lite/flatbuffer_import.cc
@@ -424,6 +424,10 @@ StatusOr<Operation*> BuildExternalConstOp(const tflite::TensorT& tensor,
 StatusOr<Operation*> BuildConstOp(const tflite::TensorT& tensor,
                                   const std::vector<uint8_t>& buffer,
                                   OpBuilder builder, Location loc) {
+  if (buffer.empty()) {
+    return errors::InvalidArgument("Constant's buffer may not be empty");
+  }
+
   TF_ASSIGN_OR_RETURN(auto type, GetTensorType(tensor, builder,
                                                /*shapeless_are_scalars=*/true,
                                                /*is_constant=*/true));
@@ -695,8 +699,6 @@ StatusOr<absl::flat_hash_set<const tflite::OperatorT*>> PruneSubgraph(
   for (int32_t output : output_indices) {
     if (auto& op = defining_op[output]) {
       queue.push_back(op);
-    } else {
-      return errors::InvalidArgument("Output tensor doesn't have defining op");
     }
   }
 
diff --git a/tensorflow/compiler/mlir/lite/ir/tfl_ops.cc b/tensorflow/compiler/mlir/lite/ir/tfl_ops.cc
index 0c384ebf9f3..7598bb9a3ee 100644
--- a/tensorflow/compiler/mlir/lite/ir/tfl_ops.cc
+++ b/tensorflow/compiler/mlir/lite/ir/tfl_ops.cc
@@ -46,28 +46,68 @@ namespace mlir {
 #include "tensorflow/compiler/mlir/lite/ir/tfl_structs.cc.inc"
 namespace TFL {
 
-// Returns true when the given two types have the same shape or broadcastable
-// shape within the given rank. If any given shapes are non-static, this method
-// returns true.
-bool IsBinaryOperandsHaveSameShapesOrBroadcastableShape(Type lhs, Type rhs,
-                                                        int max_bcast_rank) {
-  // Ignore shape checking on the non-static shapes for model compatibility.
-  auto lhs_shaped_type = lhs.dyn_cast<ShapedType>();
-  if (!lhs_shaped_type || !lhs_shaped_type.hasStaticShape()) return true;
-  auto rhs_shaped_type = rhs.dyn_cast<ShapedType>();
-  if (!rhs_shaped_type || !rhs_shaped_type.hasStaticShape()) return true;
+// Returns true when the given operand arguments have the same shape or
+// broadcastable shape within the given rank. If any given shapes are
+// non-static and maximum rank is within the given rank, this method returns
+// true.
+bool IsOperandsHaveSameShapesOrBroadcastableShape(Operation *op,
+                                                  ArrayRef<unsigned> indices,
+                                                  int max_bcast_rank) {
+  if (indices.empty()) return true;
 
-  if (lhs_shaped_type.getShape().equals(rhs_shaped_type.getShape()))
-    return true;
+  // First, it checks there are any inputs that has unknown rank.
+  bool has_unknown_shape_input = false;
+  bool has_same_shape = true;
+  bool reach_first_known_shape = false;
+  int64_t max_rank = -1;
 
+  ArrayRef<int64_t> pivot_shape;
+  SmallVector<int64_t, 4> current_shape;
   SmallVector<int64_t, 4> result_shape;
-  if (!OpTrait::util::getBroadcastedShape(lhs_shaped_type.getShape(),
-                                          rhs_shaped_type.getShape(),
-                                          result_shape)) {
-    return false;
+
+  for (unsigned index : indices) {
+    ShapedType shaped_type =
+        op->getOperand(index).getType().dyn_cast<ShapedType>();
+    if (!shaped_type || !shaped_type.hasRank()) {
+      // Marks that we have an unknown rank input.
+      has_unknown_shape_input = true;
+      continue;
+    }
+    max_rank = std::max(max_rank, shaped_type.getRank());
+    if (!shaped_type.hasStaticShape()) {
+      // Marks that we have an unknown shape input.
+      has_unknown_shape_input = true;
+      continue;
+    }
+
+    ArrayRef<int64_t> shape = shaped_type.getShape();
+    if (!reach_first_known_shape) {
+      pivot_shape = shape;
+      current_shape.assign(shape.begin(), shape.end());
+      reach_first_known_shape = true;
+      continue;
+    }
+
+    if (!pivot_shape.equals(shape)) {
+      has_same_shape = false;
+    }
+    //  Checks if all the inputs are broadcastable since they have not all the
+    //  same shapes.
+    if (!OpTrait::util::getBroadcastedShape(current_shape, shape,
+                                            result_shape)) {
+      return false;
+    }
+    current_shape = result_shape;
   }
-  return lhs_shaped_type.getRank() <= max_bcast_rank &&
-         rhs_shaped_type.getRank() <= max_bcast_rank;
+
+  // It will treat the unknown shape inputs as acceptable inputs for model
+  // compatibility unless there is an known rank that is bigger than the allowed
+  // broadcast maximum rank.
+  if (has_unknown_shape_input) return max_rank <= max_bcast_rank;
+
+  // If all the shape is known and same, CPU kernels are able to handle inputs
+  // regardless of dimension size.
+  return has_same_shape || max_rank <= max_bcast_rank;
 }
 
 //===----------------------------------------------------------------------===//
@@ -1882,7 +1922,7 @@ static LogicalResult Verify(TransposeConvOp op) {
 
   auto expected_output_type =
       RankedTensorType::get(output_shape, output_type.getElementType());
-  if (output_type != expected_output_type) {
+  if (failed(mlir::verifyCompatibleShape(output_type, expected_output_type))) {
     return op.emitOpError(llvm::formatv("expect output type {0}, got {1}",
                                         expected_output_type, output_type));
   }
@@ -2004,7 +2044,8 @@ static LogicalResult Verify(TransposeOp op) {
     }
     auto expected_output_type =
         RankedTensorType::get(transposed_shape, input_type.getElementType());
-    if (output_type != expected_output_type) {
+    if (failed(
+            mlir::verifyCompatibleShape(output_type, expected_output_type))) {
       return op.emitOpError(llvm::formatv("expect output type {0}, got {1}",
                                           expected_output_type, output_type));
     }
diff --git a/tensorflow/compiler/mlir/lite/ir/tfl_ops.td b/tensorflow/compiler/mlir/lite/ir/tfl_ops.td
index 48bc68e5c95..536960e347d 100644
--- a/tensorflow/compiler/mlir/lite/ir/tfl_ops.td
+++ b/tensorflow/compiler/mlir/lite/ir/tfl_ops.td
@@ -123,14 +123,13 @@ class TFL_RuntimePredOpTrait<string desc, Pred pred> :
   string tflRuntimeDescription = desc;
 }
 
-class TFL_BinaryOperandsHaveSameShapesOrBroadcastableShape<
-    int i, int j, int max_bcast_rank> :
-  TFL_RuntimePredOpTrait<"operand #" # i # " and operand #" # j #
-      " have the same shape or broadcastable shapes within the rank " #
-      max_bcast_rank,
-    CPred<"TFL::IsBinaryOperandsHaveSameShapesOrBroadcastableShape("
-              "$_op.getOperand(" # i # ").getType(), $_op.getOperand(" # j #
-              ").getType(), " # max_bcast_rank # ")">>;
+class TFL_OperandsHaveSameShapesOrBroadcastableShape<
+    list<int> indices, int max_bcast_rank> :
+  TFL_RuntimePredOpTrait<"operands do not have the same shape or "
+      "broadcastable shapes within the rank " # max_bcast_rank,
+    CPred<"TFL::IsOperandsHaveSameShapesOrBroadcastableShape("
+            "$_op, llvm::ArrayRef<unsigned>({" # StrJoinInt<indices>.result #
+            "}), " # max_bcast_rank # ")">>;
 
 // These additional types/type constraints here are used to decouple the ops
 // from runtime support for the ops. Prefer to use these types when defining
@@ -213,11 +212,20 @@ class TFL_OperandIsRankedAndHasDimPred<int n, int dim> : And<[
   CPred<"$_op.getOperand(" # n # ").getType().cast<ShapedType>().getRank() > "
   # dim>]>;
 
+// Returns true if the n-th operand is ranked and has a dimension length = size
+// at the rank dim.
 class TFL_OperandDimEquals<int n, int dim, int size> : And<[
   TFL_OperandIsRankedAndHasDimPred<n, dim>,
   CPred<"$_op.getOperand(" # n # ").getType().cast<ShapedType>()"
       ".getShape()[" # dim # " ] == " # size>]>;
 
+// Returns true if the n-th operand is ranked and has a dimension length <=
+// size at the rank dim.
+class TFL_OperandDimIsAtMost<int n, int dim, int size> : And<[
+  TFL_OperandIsRankedAndHasDimPred<n, dim>,
+  CPred<"$_op.getOperand(" # n # ").getType().cast<ShapedType>()"
+      ".getShape()[" # dim # " ] <= " # size>]>;
+
 // Returns true if the n-th operand has unknown rank or at least rank m.
 class TFL_OperandHasAtleastRank<int n, int m> :
   PredOpTrait<"operand " # n # " is " # m # "-D",
@@ -463,6 +471,7 @@ class TFL_ConvOp<string mnemonic, string opSummary, int index> :
 //===----------------------------------------------------------------------===//
 def TFL_AbsOp : TFL_Op<"abs", [
     NoSideEffect,
+    SameOperandsAndResultShape,
     SameOperandsAndResultType,
     NoQuantizableResult,
     TFL_GpuTargetOp]> {
@@ -482,7 +491,7 @@ an output element, this operation computes \\(y = |x|\\).
 }
 
 def TFL_AddOp : TFL_Op<"add", [
-    TFL_BinaryOperandsHaveSameShapesOrBroadcastableShape<0, 1, 5>,
+    TFL_OperandsHaveSameShapesOrBroadcastableShape<[0, 1], 5>,
     ResultsBroadcastableShape,
     NoSideEffect,
     Commutative,
@@ -669,7 +678,10 @@ def TFL_ArgMinOp : TFL_Op<"arg_min", [NoSideEffect]> {
     }]>;
 }
 
-def TFL_CeilOp: TFL_Op<"ceil", [NoSideEffect, SameOperandsAndResultType]> {
+def TFL_CeilOp: TFL_Op<"ceil", [
+    NoSideEffect,
+    SameOperandsAndResultShape,
+    SameOperandsAndResultType]> {
   let summary = "Ceil operator";
 
   let description = [{
@@ -818,6 +830,7 @@ def TFL_Conv2DOp : TFL_ConvOp<"conv_2d", "Convolution", 0> {
 
 def TFL_CosOp: TFL_Op<"cos", [
     NoSideEffect,
+    SameOperandsAndResultShape,
     SameOperandsAndResultType,
     NoQuantizableResult,
     TFL_GpuTargetOp]> {
@@ -1021,7 +1034,7 @@ def TFL_ScatterNdOp : TFL_Op<"scatter_nd", [
 def TFL_LessEqualOp : TFL_Op<"less_equal", [
     ResultsBroadcastableShape,
     BinaryOpSameElementTypeConstraint,
-    TFL_BinaryOperandsHaveSameShapesOrBroadcastableShape<0, 1, 4>,
+    TFL_OperandsHaveSameShapesOrBroadcastableShape<[0, 1], 4>,
     NoSideEffect,
     NoQuantizableResult]> {
   let summary = "Less_equal operator";
@@ -1082,7 +1095,7 @@ convolutional neural networks (NIPS 2012)](http://papers.nips.cc/paper/4824-imag
 }
 
 def TFL_GreaterEqualOp : TFL_Op<"greater_equal", [
-    TFL_BinaryOperandsHaveSameShapesOrBroadcastableShape<0, 1, 4>,
+    TFL_OperandsHaveSameShapesOrBroadcastableShape<[0, 1], 4>,
     ResultsBroadcastableShape,
     NoSideEffect,
     NoQuantizableResult]> {
@@ -1150,12 +1163,12 @@ innermost matrices.  These will be overwritten by the values in `diagonal`.
   }];
 
   let arguments = (ins
-    TensorOf<[F32, I8, I16, I32, I64, UI8, QI8, QI16, QUI8, TFL_Quint8]>:$input,
-    TensorOf<[F32, I8, I16, I32, I64, UI8, QI8, QI16, QUI8, TFL_Quint8]>:$diagonal
+    TFL_TensorOf<[F32, I8, I16, I32, I64, UI8, QI8, QI16, QUI8, TFL_Quint8]>:$input,
+    TFL_TensorOf<[F32, I8, I16, I32, I64, UI8, QI8, QI16, QUI8, TFL_Quint8]>:$diagonal
   );
 
   let results = (outs
-    TensorOf<[F32, I8, I16, I32, I64, UI8, QI8, QI16, QUI8, TFL_Quint8]>:$result
+    TFL_TensorOf<[F32, I8, I16, I32, I64, UI8, QI8, QI16, QUI8, TFL_Quint8]>:$result
   );
 
   let hasOptions = 0;
@@ -1273,7 +1286,7 @@ larger than 0.
 }
 
 def TFL_NotEqualOp : TFL_Op<"not_equal", [
-    TFL_BinaryOperandsHaveSameShapesOrBroadcastableShape<0, 1, 4>,
+    TFL_OperandsHaveSameShapesOrBroadcastableShape<[0, 1], 4>,
     BinaryOpSameElementTypeConstraint,
     ResultsBroadcastableShape,
     Commutative,
@@ -1309,7 +1322,7 @@ def TFL_DivOp : TFL_Op<"div", [
     // TODO(fengliuai): NoQuantizableResult is only correct for int8
     // quantization. update to handle Uint8 quantization.
     BinaryOpSameElementTypeConstraint,
-    TFL_BinaryOperandsHaveSameShapesOrBroadcastableShape<0, 1, 5>,
+    TFL_OperandsHaveSameShapesOrBroadcastableShape<[0, 1], 5>,
     ResultsBroadcastableShape,
     NoSideEffect,
     NoQuantizableResult,
@@ -1338,7 +1351,10 @@ def TFL_DivOp : TFL_Op<"div", [
   let hasFolder = 1;
 }
 
-def TFL_EluOp: TFL_Op<"elu", [NoSideEffect, SameOperandsAndResultType]> {
+def TFL_EluOp: TFL_Op<"elu", [
+    NoSideEffect,
+    SameOperandsAndResultShape,
+    SameOperandsAndResultType]> {
   let summary = "Exponential Linear Unit operator";
   let description = [{
     Computes the exponential linear
@@ -1374,10 +1390,11 @@ def TFL_EmbeddingLookupOp: TFL_Op<"embedding_lookup",
   let results = (outs TFL_TensorOf<[F32, I8, UI8]>:$output);
 }
 
-def TFL_EqualOp: TFL_Op<"equal", [Commutative, ResultsBroadcastableShape,
+def TFL_EqualOp: TFL_Op<"equal", [
+    Commutative,
     NoQuantizableResult,
     ResultsBroadcastableShape,
-    TFL_BinaryOperandsHaveSameShapesOrBroadcastableShape<0, 1, 4>,
+    TFL_OperandsHaveSameShapesOrBroadcastableShape<[0, 1], 4>,
     PredOpTrait<"Operands have same value type", TCopVTEtIsSameAs<0, 1>>]> {
   let summary = "Equal operator";
 
@@ -1516,7 +1533,10 @@ def TFL_FillOp: TFL_Op<"fill", [
   let hasOptions = 0;
 }
 
-def TFL_FloorOp: TFL_Op<"floor", [NoSideEffect, SameOperandsAndResultType]> {
+def TFL_FloorOp: TFL_Op<"floor", [
+    NoSideEffect,
+    SameOperandsAndResultShape,
+    SameOperandsAndResultType]> {
   let summary = "Floor operator";
 
   let description = [{
@@ -1534,7 +1554,7 @@ def TFL_FloorDivOp : TFL_Op<"floor_div", [
     BinaryOpSameElementTypeConstraint,
     PredOpTrait<"lhs and output must have same element type",
       TFL_TCresVTEtIsSameAsOp<0, 0>>,
-    TFL_BinaryOperandsHaveSameShapesOrBroadcastableShape<0, 1, 4>]> {
+    TFL_OperandsHaveSameShapesOrBroadcastableShape<[0, 1], 4>]> {
   let summary = "Floor div operator";
 
   let description = [{
@@ -1559,7 +1579,7 @@ def TFL_FloorModOp : TFL_Op<"floor_mod", [
     BinaryOpSameElementTypeConstraint,
     PredOpTrait<"lhs and output must have same element type",
       TFL_TCresVTEtIsSameAsOp<0, 0>>,
-    TFL_BinaryOperandsHaveSameShapesOrBroadcastableShape<0, 1, 4>]> {
+    TFL_OperandsHaveSameShapesOrBroadcastableShape<[0, 1], 4>]> {
   let summary = "Division reminder";
 
   let description = [{
@@ -1578,7 +1598,7 @@ def TFL_FloorModOp : TFL_Op<"floor_mod", [
 def TFL_GreaterOp : TFL_Op<"greater", [
     ResultsBroadcastableShape,
     BinaryOpSameElementTypeConstraint,
-    TFL_BinaryOperandsHaveSameShapesOrBroadcastableShape<0, 1, 4>,
+    TFL_OperandsHaveSameShapesOrBroadcastableShape<[0, 1], 4>,
     NoSideEffect,
     NoQuantizableResult]> {
   let summary = "Greater operator";
@@ -1670,7 +1690,7 @@ def TFL_LeakyReluOp: TFL_Op<"leaky_relu", [
 def TFL_LessOp : TFL_Op<"less", [
     ResultsBroadcastableShape,
     BinaryOpSameElementTypeConstraint,
-    TFL_BinaryOperandsHaveSameShapesOrBroadcastableShape<0, 1, 4>,
+    TFL_OperandsHaveSameShapesOrBroadcastableShape<[0, 1], 4>,
     NoSideEffect,
     NoQuantizableResult]> {
   let summary = "Less operator";
@@ -1710,7 +1730,10 @@ def TFL_LogicalAndOp : TFL_Op<"logical_and", [NoSideEffect]> {
   let printer = [{ return mlir::impl::printOneResultOp(getOperation(), p); }];
 }
 
-def TFL_LogicalNotOp : TFL_Op<"logical_not", [NoSideEffect, NoQuantizableResult]> {
+def TFL_LogicalNotOp : TFL_Op<"logical_not", [
+    NoSideEffect,
+    SameOperandsAndResultShape,
+    NoQuantizableResult]> {
   let summary = "Logical NOT operator";
 
   let description = [{
@@ -1794,6 +1817,7 @@ def TFL_LogisticOp: TFL_Op<"logistic", [
 
 def TFL_LogOp: TFL_Op<"log", [
     NoSideEffect,
+    SameOperandsAndResultShape,
     SameOperandsAndResultType,
     NoQuantizableResult,
     TFL_GpuTargetOp]> {
@@ -1884,6 +1908,7 @@ def TFL_MaxPool2DOp : TFL_Op<"max_pool_2d", [
 def TFL_MaximumOp : TFL_Op<"maximum", [
     ResultsBroadcastableShape,
     NoSideEffect,
+    TFL_OperandsHaveSameShapesOrBroadcastableShape<[0, 1], 5>,
     Commutative,
     SameOperandsAndResultsScale,
     TFL_GpuTargetOp]> {
@@ -2118,6 +2143,7 @@ def TFL_ReduceProdOp: TFL_Op<"reduce_prod", [
 def TFL_MinimumOp : TFL_Op<"minimum", [
     ResultsBroadcastableShape,
     NoSideEffect,
+    TFL_OperandsHaveSameShapesOrBroadcastableShape<[0, 1], 5>,
     Commutative,
     SameOperandsAndResultsScale,
     TFL_GpuTargetOp]> {
@@ -2145,7 +2171,7 @@ def TFL_MulOp : TFL_Op<"mul", [
     NoSideEffect,
     Commutative,
     BinaryOpSameElementTypeConstraint,
-    TFL_BinaryOperandsHaveSameShapesOrBroadcastableShape<0, 1, 5>,
+    TFL_OperandsHaveSameShapesOrBroadcastableShape<[0, 1], 5>,
     TFL_GpuTargetOp]> {
   let summary = "Multiplication operator";
 
@@ -2171,7 +2197,10 @@ def TFL_MulOp : TFL_Op<"mul", [
   let hasOptions = 1;
 }
 
-def TFL_NegOp: TFL_Op<"neg", [NoSideEffect, SameOperandsAndResultType]> {
+def TFL_NegOp: TFL_Op<"neg", [
+    NoSideEffect,
+    SameOperandsAndResultShape,
+    SameOperandsAndResultType]> {
   let summary = "Negation operator";
 
   let description = [{
@@ -2247,6 +2276,9 @@ def TFL_PadOp : TFL_Op<"pad", [
     TFL_OperandHasRankAtMost<0, 4>,
     TFL_OperandHasRank<1, 2>,
     TFL_OperandRankEquals1DimOfOperand<0, 1>,
+    PredOpTrait<"the first dim size of the padding argument must be at most 4",
+      Or<[TFL_OperandIsUnrankedPred<1>,
+          TFL_OperandDimIsAtMost<1, 0, 4>]>>,
     TFL_GpuTargetOp]> {
   let summary = "Padding operator";
 
@@ -2292,6 +2324,9 @@ def TFL_PadV2Op : TFL_Op<"padv2", [
     TFL_OperandHasRank<1, 2>,
     TFL_OperandHasRank<2, 0>,
     TFL_OperandRankEquals1DimOfOperand<0, 1>,
+    PredOpTrait<"the first dim size of the padding argument must be at most 4",
+      Or<[TFL_OperandIsUnrankedPred<1>,
+          TFL_OperandDimIsAtMost<1, 0, 4>]>>,
     PredOpTrait<"input and constant value operands must have same element type",
       TFL_TCopVTEtAreSameAt<0, 2>>]> {
   let summary = "Padding operator v2";
@@ -2333,10 +2368,12 @@ def TFL_PadV2Op : TFL_Op<"padv2", [
   let hasOptions = 1;
 }
 
-def TFL_PowOp : TFL_Op<"pow", [ResultsBroadcastableShape,
-                               NoSideEffect,
-                               NoQuantizableResult,
-                               TFL_GpuTargetOp]> {
+def TFL_PowOp : TFL_Op<"pow", [
+    ResultsBroadcastableShape,
+    NoSideEffect,
+    TFL_OperandsHaveSameShapesOrBroadcastableShape<[0, 1], 4>,
+    NoQuantizableResult,
+    TFL_GpuTargetOp]> {
   let summary = "Power operator";
 
   let description = [{
@@ -2360,7 +2397,7 @@ def TFL_PReluOp : TFL_Op<"prelu", [
     NoSideEffect,
     ResultsBroadcastableShape,
     TFL_GpuTargetOp,
-    TFL_BinaryOperandsHaveSameShapesOrBroadcastableShape<0, 1, 4>,
+    TFL_OperandsHaveSameShapesOrBroadcastableShape<[0, 1], 4>,
     BinaryOpSameElementTypeConstraint,
     PredOpTrait<"input and output must have the same element type",
       TFL_TCresVTEtIsSameAsOp<0, 0>>]> {
@@ -2671,8 +2708,9 @@ def TFL_SelectOp : TFL_Op<"select", [
 }
 
 def TFL_SelectV2Op : TFL_Op<"select_v2", [
+    ResultsBroadcastableShape,
     NoSideEffect,
-    TFL_BinaryOperandsHaveSameShapesOrBroadcastableShape<1, 2, 4>,
+    TFL_OperandsHaveSameShapesOrBroadcastableShape<[0, 1, 2], 4>,
     PredOpTrait<"operands have same element type", TCopVTEtIsSameAs<1, 2>>,
     PredOpTrait<"operands and result have same element type",
       TFL_TCresVTEtIsSameAsOp<0, 1>>]> {
@@ -2705,6 +2743,7 @@ def TFL_SelectV2Op : TFL_Op<"select_v2", [
 
 def TFL_SinOp: TFL_Op<"sin", [
     NoSideEffect,
+    SameOperandsAndResultShape,
     SameOperandsAndResultType,
     NoQuantizableResult,
     TFL_GpuTargetOp]> {
@@ -2752,6 +2791,7 @@ def TFL_SoftmaxOp : TFL_Op<"softmax", [
 
 def TFL_SqrtOp: TFL_Op<"sqrt", [
     NoSideEffect,
+    SameOperandsAndResultShape,
     SameOperandsAndResultType,
     NoQuantizableResult,
     TFL_GpuTargetOp]> {
@@ -2770,6 +2810,7 @@ def TFL_SqrtOp: TFL_Op<"sqrt", [
 
 def TFL_SquareOp: TFL_Op<"square", [
     NoSideEffect,
+    SameOperandsAndResultShape,
     SameOperandsAndResultType,
     NoQuantizableResult,
     TFL_GpuTargetOp]> {
@@ -2791,7 +2832,7 @@ def TFL_SquareOp: TFL_Op<"square", [
 def TFL_SubOp : TFL_Op<"sub", [
     ResultsBroadcastableShape,
     BinaryOpSameElementTypeConstraint,
-    TFL_BinaryOperandsHaveSameShapesOrBroadcastableShape<0, 1, 5>,
+    TFL_OperandsHaveSameShapesOrBroadcastableShape<[0, 1], 5>,
     NoSideEffect]> {
   let summary = "Subtraction operator";
 
@@ -2820,7 +2861,7 @@ def TFL_SubOp : TFL_Op<"sub", [
 // TODO(jpienaar): Expand the kernel implementation to support all types besides
 // I32 and F32.
 def TFL_SquaredDifferenceOp : TFL_Op<"squared_difference", [
-    TFL_BinaryOperandsHaveSameShapesOrBroadcastableShape<0, 1, 4>,
+    TFL_OperandsHaveSameShapesOrBroadcastableShape<[0, 1], 4>,
     SameOperandsAndResultElementType,
     ResultsBroadcastableShape,
     NoSideEffect,
@@ -3007,6 +3048,8 @@ def TFL_UnpackOp : TFL_Op<"unpack", [
 def TFL_ZerosLikeOp: TFL_Op<"zeros_like", [
     PredOpTrait<"input and output must have same element type",
       TFL_TCresVTEtIsSameAsOp<0, 0>>,
+    SameOperandsAndResultType,
+    SameOperandsAndResultShape,
     NoSideEffect]> {
   let summary = "ZerosLike operator";
 
@@ -3319,7 +3362,9 @@ def TFL_StridedSliceOp: TFL_Op<"strided_slice", [
 }
 
 def TFL_CastOp : TFL_Op<"cast", [
-    NoSideEffect, SameOperandsAndResultShape, NoQuantizableResult]> {
+    NoSideEffect,
+    SameOperandsAndResultShape,
+    NoQuantizableResult]> {
   let summary = "Cast operator";
 
   let description = [{
diff --git a/tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir/pruning_function_input_as_output.mlir b/tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir/pruning_function_input_as_output.mlir
new file mode 100644
index 00000000000..1540641fbc0
--- /dev/null
+++ b/tensorflow/compiler/mlir/lite/tests/flatbuffer2mlir/pruning_function_input_as_output.mlir
@@ -0,0 +1,16 @@
+// RUN: flatbuffer_translate -mlir-to-tflite-flatbuffer %s -o - | flatbuffer_translate -output-arrays=mul,div,exp --experimental-prune-unreachable-nodes-unconditionally --tflite-flatbuffer-to-mlir - -o - | FileCheck --dump-input-on-failure %s
+
+// CHECK: (%[[ARG:.*]]: tensor<4xf32>) -> (tensor<4xf32>, tensor<4xf32>, tensor<4xf32>)
+func @main(%arg0: tensor<4xf32>) -> tensor<4xf32> attributes {tf.entry_function = {inputs = "mul"}} {
+  %0 = "tfl.pseudo_const" () {value = dense<1.0> : tensor<4xf32>} : () -> tensor<4xf32> loc("Const")
+  %1 = "tfl.squared_difference"(%arg0, %0) {fused_activation_function = "NONE"} : (tensor<4xf32>, tensor<4xf32>) -> tensor<4xf32> loc("squared_difference")
+  // CHECK: %[[DIV:.*]] = tfl.div
+  %2 = "tfl.div"(%1, %arg0) {fused_activation_function = "NONE"} : (tensor<4xf32>, tensor<4xf32>) -> tensor<4xf32> loc("div")
+  // CHECK: %[[EXP:.*]] = "tfl.exp"
+  %3 = "tfl.exp"(%2) : (tensor<4xf32>) -> tensor<4xf32> loc("exp")
+  // tfl.neg should be pruned
+  // CHECK-NOT: "tfl.neg"
+  %4 = "tfl.neg"(%3) : (tensor<4xf32>) -> tensor<4xf32> loc("neg")
+  // CHECK: return %[[ARG]], %[[DIV]], %[[EXP]]
+  return %4 : tensor<4xf32>
+}
diff --git a/tensorflow/compiler/mlir/lite/tests/legalize-tf.mlir b/tensorflow/compiler/mlir/lite/tests/legalize-tf.mlir
index 15c73d2db2c..fb94a8212ac 100644
--- a/tensorflow/compiler/mlir/lite/tests/legalize-tf.mlir
+++ b/tensorflow/compiler/mlir/lite/tests/legalize-tf.mlir
@@ -1529,3 +1529,22 @@ func @matmul_batchv2_unknown_dim(%arg0: tensor<?x10x15xf32>, %arg1: tensor<15x17
 // CHECK-LABEL: matmul_batchv2_unknown_dim
 // CHECK: "tfl.batch_matmul"(%arg0, %arg1) {adj_x = false, adj_y = false} : (tensor<?x10x15xf32>, tensor<15x17xf32>) -> tensor<?x10x17xf32>
 }
+
+// -----
+
+func @select_v2_with_6d_broadcasting(%arg0: tensor<1x1x1x1x3x1xi1>, %arg1 : tensor<1x1x1x1x1x4xf32>, %arg2 : tensor<1x1x1x2x1x1xf32>) -> tensor<1x1x1x2x3x4xf32> {
+  %0 = "tf.SelectV2"(%arg0, %arg1, %arg2): (tensor<1x1x1x1x3x1xi1>, tensor<1x1x1x1x1x4xf32>, tensor<1x1x1x2x1x1xf32>) -> tensor<1x1x1x2x3x4xf32>
+  return %0 : tensor<1x1x1x2x3x4xf32>
+// CHECK-LABEL: select_v2_with_6d_broadcasting
+// CHECK: "tf.SelectV2"(%arg0, %arg1, %arg2)
+}
+
+// -----
+
+func @maximum_with_6d_broadcasting(%arg0: tensor<1x1x1x1x8x16xf32>, %arg1: tensor<8x16xf32>) -> tensor<1x1x1x1x8x16xf32> {
+  %0 = "tf.Maximum"(%arg0, %arg1) : (tensor<1x1x1x1x8x16xf32>, tensor<8x16xf32>) -> tensor<1x1x1x1x8x16xf32>
+  return %0 : tensor<1x1x1x1x8x16xf32>
+
+// CHECK-LABEL: maximum_with_6d_broadcasting
+// CHECK:  "tf.Maximum"(%arg0, %arg1)
+}
diff --git a/tensorflow/compiler/mlir/lite/tests/ops.mlir b/tensorflow/compiler/mlir/lite/tests/ops.mlir
index 3451f28380b..6ab3aa250af 100644
--- a/tensorflow/compiler/mlir/lite/tests/ops.mlir
+++ b/tensorflow/compiler/mlir/lite/tests/ops.mlir
@@ -794,6 +794,41 @@ func @testSelectWithUnsupportedType(%cond : tensor<?xi1>, %arg0 : tensor<?xi32>,
 
 // -----
 
+// CHECK-LABEL: testSelectV2
+func @testSelectV2(%cond : tensor<*xi1>, %arg0 : tensor<*xf32>, %arg1 : tensor<*xf32>) -> tensor<*xf32> {
+  // CHECK: "tfl.select_v2"(%arg0, %arg1, %arg2)
+  %0 = "tfl.select_v2"(%cond, %arg0, %arg1): (tensor<*xi1>, tensor<*xf32>, tensor<*xf32>) -> tensor<*xf32>
+  return %0 : tensor<*xf32>
+}
+
+// -----
+
+// CHECK-LABEL: testSelectV2WithHighDimInputs
+func @testSelectV2WithHighDimInputs(%cond : tensor<1x2x3x4x5x6xi1>, %arg0 : tensor<1x2x3x4x5x6xf32>, %arg1 : tensor<1x2x3x4x5x6xf32>) -> tensor<1x2x3x4x5x6xf32> {
+  // CHECK: "tfl.select_v2"(%arg0, %arg1, %arg2)
+  %0 = "tfl.select_v2"(%cond, %arg0, %arg1): (tensor<1x2x3x4x5x6xi1>, tensor<1x2x3x4x5x6xf32>, tensor<1x2x3x4x5x6xf32>) -> tensor<1x2x3x4x5x6xf32>
+  return %0 : tensor<1x2x3x4x5x6xf32>
+}
+
+// -----
+
+// CHECK-LABEL: testSelectV2With4DBroadcasting
+func @testSelectV2With4DBroadcasting(%cond : tensor<1x1x3x1xi1>, %arg0 : tensor<1x1x1x4xf32>, %arg1 : tensor<1x2x1x1xf32>) -> tensor<1x2x3x4xf32> {
+  // CHECK: "tfl.select_v2"(%arg0, %arg1, %arg2)
+  %0 = "tfl.select_v2"(%cond, %arg0, %arg1): (tensor<1x1x3x1xi1>, tensor<1x1x1x4xf32>, tensor<1x2x1x1xf32>) -> tensor<1x2x3x4xf32>
+  return %0 : tensor<1x2x3x4xf32>
+}
+
+// -----
+
+func @testSelectV2WithWrongBroadcastableArguments(%cond : tensor<3x4xi1>, %arg0 : tensor<2x3x4xf32>, %arg1 : tensor<4x3xf32>) -> tensor<2x3x4xf32> {
+  // expected-error @+1 {{'tfl.select_v2' op operands don't have broadcast-compatible shapes}}
+  %0 = "tfl.select_v2"(%cond, %arg0, %arg1): (tensor<3x4xi1>, tensor<2x3x4xf32>, tensor<4x3xf32>) -> tensor<2x3x4xf32>
+  return %0 : tensor<2x3x4xf32>
+}
+
+// -----
+
 // CHECK-LABEL: topk
 func @topk(%arg0: tensor<8xf32>, %arg1: tensor<i32>) -> (tensor<?xf32>, tensor<?xi32>) {
   %0, %1 = "tfl.topk_v2"(%arg0, %arg1) : (tensor<8xf32>, tensor<i32>) -> (tensor<?xf32>, tensor<?xi32>)
@@ -888,6 +923,27 @@ func @testPadWithInvalidPaddingsRank(tensor<2x1x3xf32>, tensor<1x3x2xi32>) -> te
 
 // -----
 
+func @testPadUnknownPaddings(tensor<2x1x3xf32>, tensor<*xi32>) -> tensor<? x f32> {
+^bb0(%arg0: tensor<2x1x3xf32>, %arg1: tensor<*xi32>):
+  %0 = "tfl.pad"(%arg0, %arg1) : (tensor<2x1x3xf32>, tensor<*xi32>) -> tensor<? x f32>
+  return %0#0 : tensor<? x f32>
+
+  // CHECK-LABEL: testPadUnknownPaddings
+  // CHECK:  "tfl.pad"(%arg0, %arg1) : (tensor<2x1x3xf32>, tensor<*xi32>) -> tensor<?xf32>
+  // CHECK:  return
+}
+
+// -----
+
+func @testPadUnsupportedPaddings(tensor<*xf32>, tensor<5x3xi32>) -> tensor<? x f32> {
+^bb0(%arg0: tensor<*xf32>, %arg1: tensor<5x3xi32>):
+  // expected-error @+1 {{'tfl.pad' op failed to verify that the first dim size of the padding argument must be at most 4}}
+  %0 = "tfl.pad"(%arg0, %arg1) : (tensor<*xf32>, tensor<5x3xi32>) -> tensor<? x f32>
+  return %0#0 : tensor<? x f32>
+}
+
+// -----
+
 // CHECK-LABEL: testPadQuantizedU8
 func @testPadQuantizedU8(%arg0: tensor<2x1x3x!quant.uniform<u8:f32, 0.1>>, %arg1: tensor<3x2xi32>) -> tensor<? x !quant.uniform<u8:f32, 0.1>> {
   // CHECK: "tfl.pad"(%arg0, %arg1)
@@ -958,6 +1014,29 @@ func @testPadV2WithInvalidConstantScalar(tensor<2x1x3xf32>, tensor<3x2xi32>) ->
 
 // -----
 
+func @testPadV2UnknownPaddings(tensor<2x1x3xf32>, tensor<*xi32>) -> tensor<? x f32> {
+^bb0(%arg0: tensor<2x1x3xf32>, %arg1: tensor<*xi32>):
+  %cst = constant dense<2.0> : tensor<f32>
+  %0 = "tfl.padv2"(%arg0, %arg1, %cst) : (tensor<2x1x3xf32>, tensor<*xi32>, tensor<f32>) -> tensor<? x f32>
+  return %0#0 : tensor<? x f32>
+
+  // CHECK-LABEL: testPadV2UnknownPaddings
+  // CHECK:  "tfl.padv2"(%arg0, %arg1, %cst) : (tensor<2x1x3xf32>, tensor<*xi32>, tensor<f32>) -> tensor<?xf32>
+  // CHECK:  return
+}
+
+// -----
+
+func @testPadV2UnsupportedPaddings(tensor<*xf32>, tensor<5x3xi32>) -> tensor<? x f32> {
+^bb0(%arg0: tensor<*xf32>, %arg1: tensor<5x3xi32>):
+  %cst = constant dense<2.0> : tensor<f32>
+  // expected-error @+1 {{'tfl.padv2' op failed to verify that the first dim size of the padding argument must be at most 4}}
+  %0 = "tfl.padv2"(%arg0, %arg1, %cst) : (tensor<*xf32>, tensor<5x3xi32>, tensor<f32>) -> tensor<? x f32>
+  return %0#0 : tensor<? x f32>
+}
+
+// -----
+
 func @packQuantizedU8(%arg0: tensor<2x!quant.uniform<u8:f32, 0.1>>, %arg1: tensor<2x!quant.uniform<u8:f32, 0.1>>) -> tensor<2x2x!quant.uniform<u8:f32, 0.1>> {
   // CHECK: "tfl.pack"(%arg0, %arg1) {axis = 0 : i32, values_count = 2 : i32}
   %0 = "tfl.pack"(%arg0, %arg1) {axis = 0 : i32, values_count = 2 : i32} : (tensor<2x!quant.uniform<u8:f32, 0.1>>, tensor<2x!quant.uniform<u8:f32, 0.1>>) -> tensor<2x2x!quant.uniform<u8:f32, 0.1>>
@@ -1322,6 +1401,14 @@ func @transpose(%arg0 : tensor<2x2xi32>, %arg1 : tensor<2xi32>) -> tensor<2x2xi3
   return %0 : tensor<2x2xi32>
 }
 
+// -----
+
+// CHECK-LABEL: transpose_with_output_that_has_dynamic_sizes
+func @transpose_with_output_that_has_dynamic_sizes(%arg0 : tensor<2x2xi32>, %arg1 : tensor<2xi32>) -> tensor<?x?xi32> {
+  // CHECK: "tfl.transpose"(%arg0, %arg1)
+  %0 = "tfl.transpose"(%arg0, %arg1) : (tensor<2x2xi32>, tensor<2xi32>) -> tensor<?x?xi32>
+  return %0 : tensor<?x?xi32>
+}
 
 // -----
 
@@ -2070,6 +2157,16 @@ func @testTransposeConv(%arg0: tensor<4xi32>, %arg1: tensor<32x4x4x128xf32>, %ar
 
 // -----
 
+// CHECK-LABEL: testTransposeConvWithOutputThatHasDynamicSizes
+func @testTransposeConvWithOutputThatHasDynamicSizes(%arg0: tensor<4xi32>, %arg1: tensor<32x4x4x128xf32>, %arg2: tensor<1x32x42x128xf32>) -> tensor<?x?x?x?xf32> {
+  // CHECK: "tfl.transpose_conv"(%arg0, %arg1, %arg2, %cst)
+  %cst = constant unit
+  %0 = "tfl.transpose_conv"(%arg0, %arg1, %arg2, %cst) {padding = "SAME", stride_h = 2 : i32, stride_w = 2 : i32} : (tensor<4xi32>, tensor<32x4x4x128xf32>, tensor<1x32x42x128xf32>, none) -> tensor<?x?x?x?xf32>
+  return %0 : tensor<?x?x?x?xf32>
+}
+
+// -----
+
 func @testConvolution2DTransposeBias(%arg0: tensor<32x4x4x128xf32>, %arg1: tensor<1x32x42x128xf32>, %arg2: tensor<4xi32>) -> tensor<1x64x84x32xf32> {
   // custom op for "tfl.convolution_2d_transpose_bias"(%arg0, %arg1, %arg2) {padding = "SAME", stride_h = 2 : i32, stride_w = 2 : i32} : (tensor<32x4x4x128xf32>, tensor<1x32x42x128xf32>, tensor<4xi32>) -> tensor<1x64x84x32xf32>
   %0 = "tfl.custom"(%arg0, %arg1, %arg2) {custom_option = opaque<"tfl", "0x010000000200000002000000"> : tensor<12xi8>, custom_code = "Convolution2DTransposeBias"} : (tensor<32x4x4x128xf32>, tensor<1x32x42x128xf32>, tensor<4xi32>) -> tensor<1x64x84x32xf32>
diff --git a/tensorflow/compiler/mlir/tensorflow/BUILD b/tensorflow/compiler/mlir/tensorflow/BUILD
index afc7065a4bc..40add34393b 100644
--- a/tensorflow/compiler/mlir/tensorflow/BUILD
+++ b/tensorflow/compiler/mlir/tensorflow/BUILD
@@ -426,6 +426,7 @@ cc_library(
         "transforms/layout_optimization.cc",
         "transforms/mark_function_visibility.cc",
         "transforms/materialize_mlir_passthrough_op.cc",
+        "transforms/op_fusion.cc",
         "transforms/optimize.cc",
         "transforms/optimize_global_tensors.cc",
         "transforms/parallel_execute_to_islands.cc",
@@ -436,6 +437,7 @@ cc_library(
         "transforms/replicate_to_island.cc",
         "transforms/resource_device_inference.cc",
         "transforms/resource_op_lifting.cc",
+        "transforms/rewrite_tpu_embedding_ops.cc",
         "transforms/shape_inference.cc",
         "transforms/shape_inference_pass.cc",
         "transforms/sink_constant.cc",
@@ -451,6 +453,7 @@ cc_library(
         "transforms/tpu_extract_head_tail_outside_compilation.cc",
         "transforms/tpu_extract_outside_compilation.cc",
         "transforms/tpu_merge_variables_with_execute.cc",
+        "transforms/tpu_outside_compilation_cluster.cc",
         "transforms/tpu_rewrite_pass.cc",
         "transforms/tpu_sharding_identification_pass.cc",
         "transforms/tpu_space_to_depth_pass.cc",
diff --git a/tensorflow/compiler/mlir/tensorflow/ir/tf_generated_ops.td b/tensorflow/compiler/mlir/tensorflow/ir/tf_generated_ops.td
index e4e77b485cd..a8667427a45 100644
--- a/tensorflow/compiler/mlir/tensorflow/ir/tf_generated_ops.td
+++ b/tensorflow/compiler/mlir/tensorflow/ir/tf_generated_ops.td
@@ -917,6 +917,30 @@ the feature dimension is the third-to-last.
   }];
 }
 
+def TF_BiasAddV1Op : TF_Op<"BiasAddV1", [NoSideEffect]> {
+  let summary = "Adds `bias` to `value`.";
+
+  let description = [{
+This is a deprecated version of BiasAdd and will be soon removed.
+
+This is a special case of `tf.add` where `bias` is restricted to be 1-D.
+Broadcasting is supported, so `value` may have any number of dimensions.
+  }];
+
+  let arguments = (ins
+    TensorOf<[BF16, F16, F32, F64, I16, I32, I64, I8, TF_Complex128, TF_Complex64, TF_Qint32, TF_Qint8, TF_Quint8, TF_Uint16, TF_Uint32, TF_Uint64, TF_Uint8]>:$value,
+    TensorOf<[BF16, F16, F32, F64, I16, I32, I64, I8, TF_Complex128, TF_Complex64, TF_Qint32, TF_Qint8, TF_Quint8, TF_Uint16, TF_Uint32, TF_Uint64, TF_Uint8]>:$bias
+  );
+
+  let results = (outs
+    TensorOf<[BF16, F16, F32, F64, I16, I32, I64, I8, TF_Complex128, TF_Complex64, TF_Qint32, TF_Qint8, TF_Quint8, TF_Uint16, TF_Uint32, TF_Uint64, TF_Uint8]>:$output
+  );
+
+  TF_DerivedOperandTypeAttr T = TF_DerivedOperandTypeAttr<0>;
+
+  let hasCanonicalizer = 1;
+}
+
 def TF_BitcastOp : TF_Op<"Bitcast", [NoSideEffect]> {
   let summary = [{
 Bitcasts a tensor from one type to another without copying data.
@@ -2946,19 +2970,23 @@ Fake-quantize the 'inputs' tensor, type float to 'outputs' tensor of same type.
   }];
 
   let description = [{
-Attributes `[min; max]` define the clamping range for the `inputs` data.
-`inputs` values are quantized into the quantization range (`[0; 2^num_bits - 1]`
-when `narrow_range` is false and `[1; 2^num_bits - 1]` when it is true) and
-then de-quantized and output as floats in `[min; max]` interval.
-`num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
+Attributes
+
+*   `[min; max]` define the clamping range for the `inputs` data.
+*   `inputs` values are quantized into the quantization range (
+`[0; 2^num_bits - 1]` when `narrow_range` is false and `[1; 2^num_bits - 1]`
+when it is true) and then de-quantized and output as floats in `[min; max]`
+interval.
+*   `num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
 
 Before quantization, `min` and `max` values are adjusted with the following
 logic.
 It is suggested to have `min <= 0 <= max`. If `0` is not in the range of values,
 the behavior can be unexpected:
-If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
-If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
-If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
+
+*   If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
+*   If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
+*   If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
 `min_adj = scale * round(min / scale)` and `max_adj = max + min_adj - min`.
 
 Quantization is called fake since the output is still in floating point.
@@ -2984,25 +3012,30 @@ Quantization is called fake since the output is still in floating point.
 
 def TF_FakeQuantWithMinMaxVarsOp : TF_Op<"FakeQuantWithMinMaxVars", [NoSideEffect]> {
   let summary = [{
-Fake-quantize the 'inputs' tensor of type float via global float scalars `min`
+Fake-quantize the 'inputs' tensor of type float via global float scalars
   }];
 
   let description = [{
-and `max` to 'outputs' tensor of same shape as `inputs`.
+Fake-quantize the `inputs` tensor of type float via global float scalars
+`min` and `max` to `outputs` tensor of same shape as `inputs`.
 
-`[min; max]` define the clamping range for the `inputs` data.
-`inputs` values are quantized into the quantization range (`[0; 2^num_bits - 1]`
-when `narrow_range` is false and `[1; 2^num_bits - 1]` when it is true) and
-then de-quantized and output as floats in `[min; max]` interval.
-`num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
+Attributes
+
+*   `[min; max]` define the clamping range for the `inputs` data.
+*   `inputs` values are quantized into the quantization range (
+`[0; 2^num_bits - 1]` when `narrow_range` is false and `[1; 2^num_bits - 1]`
+when it is true) and then de-quantized and output as floats in `[min; max]`
+interval.
+*   `num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
 
 Before quantization, `min` and `max` values are adjusted with the following
 logic.
 It is suggested to have `min <= 0 <= max`. If `0` is not in the range of values,
 the behavior can be unexpected:
-If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
-If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
-If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
+
+*   If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
+*   If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
+*   If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
 `min_adj = scale * round(min / scale)` and `max_adj = max + min_adj - min`.
 
 This operation has a gradient and thus allows for training `min` and `max`
@@ -3029,26 +3062,31 @@ values.
 
 def TF_FakeQuantWithMinMaxVarsPerChannelOp : TF_Op<"FakeQuantWithMinMaxVarsPerChannel", [NoSideEffect]> {
   let summary = [{
-Fake-quantize the 'inputs' tensor of type float and one of the shapes: `[d]`,
+Fake-quantize the 'inputs' tensor of type float via per-channel floats
   }];
 
   let description = [{
-`[b, d]` `[b, h, w, d]` via per-channel floats `min` and `max` of shape `[d]`
-to 'outputs' tensor of same shape as `inputs`.
+Fake-quantize the `inputs` tensor of type float per-channel and one of the
+shapes: `[d]`, `[b, d]` `[b, h, w, d]` via per-channel floats `min` and `max`
+of shape `[d]` to `outputs` tensor of same shape as `inputs`.
 
-`[min; max]` define the clamping range for the `inputs` data.
-`inputs` values are quantized into the quantization range (`[0; 2^num_bits - 1]`
-when `narrow_range` is false and `[1; 2^num_bits - 1]` when it is true) and
-then de-quantized and output as floats in `[min; max]` interval.
-`num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
+Attributes
+
+*   `[min; max]` define the clamping range for the `inputs` data.
+*   `inputs` values are quantized into the quantization range (
+`[0; 2^num_bits - 1]` when `narrow_range` is false and `[1; 2^num_bits - 1]`
+when it is true) and then de-quantized and output as floats in `[min; max]`
+interval.
+*   `num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
 
 Before quantization, `min` and `max` values are adjusted with the following
 logic.
 It is suggested to have `min <= 0 <= max`. If `0` is not in the range of values,
 the behavior can be unexpected:
-If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
-If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
-If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
+
+*   If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
+*   If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
+*   If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
 `min_adj = scale * round(min / scale)` and `max_adj = max + min_adj - min`.
 
 This operation has a gradient and thus allows for training `min` and `max`
@@ -10801,6 +10839,38 @@ def TF_ZerosLikeOp : TF_Op<"ZerosLike", [NoSideEffect, SameOperandsAndResultType
   TF_DerivedOperandTypeAttr T = TF_DerivedOperandTypeAttr<0>;
 }
 
+def TF__FusedConv2DOp : TF_Op<"_FusedConv2D", [NoSideEffect]> {
+  let summary = [{
+*NOTE*: Do not invoke this operator directly in Python. Grappler is
+  }];
+
+  let description = [{
+expected to create these operators.
+  }];
+
+  let arguments = (ins
+    TF_F32OrF64Tensor:$input,
+    TF_F32OrF64Tensor:$filter,
+    Variadic<TF_F32OrF64Tensor>:$args,
+
+    I64ArrayAttr:$strides,
+    TF_AnyStrAttrOf<["SAME", "VALID", "EXPLICIT"]>:$padding,
+    DefaultValuedAttr<I64ArrayAttr, "{}">:$explicit_paddings,
+    DefaultValuedAttr<TF_ConvnetDataFormatAttr, "NHWC">:$data_format,
+    DefaultValuedAttr<I64ArrayAttr, "{1, 1, 1, 1}">:$dilations,
+    DefaultValuedAttr<BoolAttr, "true">:$use_cudnn_on_gpu,
+    DefaultValuedAttr<StrArrayAttr, "{}">:$fused_ops,
+    DefaultValuedAttr<F32Attr, "0.0001f">:$epsilon
+  );
+
+  let results = (outs
+    TF_F32OrF64Tensor:$output
+  );
+
+  TF_DerivedOperandTypeAttr T = TF_DerivedOperandTypeAttr<0>;
+  TF_DerivedOperandSizeAttr num_args = TF_DerivedOperandSizeAttr<2>;
+}
+
 def TF__HostComputeMlirOp : TF_Op<"_HostComputeMlir", []> {
   let summary = "A host-side computation called from a TPU device.";
 
diff --git a/tensorflow/compiler/mlir/tensorflow/ir/tf_ops.cc b/tensorflow/compiler/mlir/tensorflow/ir/tf_ops.cc
index 8154e501f60..7b3e1508efb 100644
--- a/tensorflow/compiler/mlir/tensorflow/ir/tf_ops.cc
+++ b/tensorflow/compiler/mlir/tensorflow/ir/tf_ops.cc
@@ -755,6 +755,15 @@ static LogicalResult Verify(BiasAddGradOp op) {
   return success();
 }
 
+//===----------------------------------------------------------------------===//
+// BiasAddV1Op
+//===----------------------------------------------------------------------===//
+
+void BiasAddV1Op::getCanonicalizationPatterns(OwningRewritePatternList &results,
+                                              MLIRContext *context) {
+  results.insert<BiasAddV1ToBiasAdd>(context);
+}
+
 //===----------------------------------------------------------------------===//
 // BitcastOp
 //===----------------------------------------------------------------------===//
diff --git a/tensorflow/compiler/mlir/tensorflow/ir/tf_ops.td b/tensorflow/compiler/mlir/tensorflow/ir/tf_ops.td
index 5f3a1a5be35..d8675bb786f 100644
--- a/tensorflow/compiler/mlir/tensorflow/ir/tf_ops.td
+++ b/tensorflow/compiler/mlir/tensorflow/ir/tf_ops.td
@@ -358,6 +358,41 @@ def TF_LegacyCallOp : TF_Op<"LegacyCall",
   }];
 }
 
+def TF_ParseExampleOp : TF_Op<"ParseExample",
+                               [NoSideEffect,
+                                AttrSizedResultSegments,
+                                AttrSizedOperandSegments]> {
+
+  let summary =
+    "Transforms a vector of tf.Example protos (as strings) into typed tensors.";
+
+  let arguments = (ins
+    TF_StrTensor:$serialized,
+    TF_StrTensor:$names,
+    Variadic<TF_StrTensor>:$sparse_keys,
+    Variadic<TF_StrTensor>:$dense_keys,
+    Variadic<TensorOf<[F32, I64, TF_Str]>>:$dense_defaults,
+
+    TF_ShapeAttrArray:$dense_shapes,
+    I32ElementsAttr:$result_segment_sizes,
+    I32ElementsAttr:$operand_segment_sizes
+  );
+
+  let results = (outs
+    Variadic<I64Tensor>:$sparse_indices,                    // len(sparse_types)
+    Variadic<TensorOf<[F32, I64, TF_Str]>>:$sparse_values,  // len(sparse_types)
+    Variadic<I64Tensor>:$sparse_shapes,                     // len(sparse_types)
+    Variadic<TensorOf<[F32, I64, TF_Str]>>:$dense_values    // len(Tdense)
+  );
+
+  TF_DerivedOperandSizeAttr Nsparse = TF_DerivedOperandSizeAttr<2>;
+  TF_DerivedOperandSizeAttr Ndense = TF_DerivedOperandSizeAttr<3>;
+  TF_DerivedOperandTypeListAttr Tdense = TF_DerivedOperandTypeListAttr<4>;
+  TF_DerivedResultTypeListAttr sparse_types = TF_DerivedResultTypeListAttr<1>;
+
+  let verifier = ?;
+}
+
 def TF_ParseExampleV2Op : TF_Op<"ParseExampleV2",
                                 [NoSideEffect,
                                  AttrSizedResultSegments]> {
diff --git a/tensorflow/compiler/mlir/tensorflow/tests/annotate-parameter-replication.mlir b/tensorflow/compiler/mlir/tensorflow/tests/annotate-parameter-replication.mlir
index 743f0b43b69..38105986002 100644
--- a/tensorflow/compiler/mlir/tensorflow/tests/annotate-parameter-replication.mlir
+++ b/tensorflow/compiler/mlir/tensorflow/tests/annotate-parameter-replication.mlir
@@ -19,7 +19,7 @@ module attributes {tf.versions = {producer = 888 : i32}} {
 
   // CHECK-LABEL: func @_func
   // CHECK-SAME: %[[ARG0:.*]]: tensor<?xi32>,
-  // CHECK-SAME: %[[ARG1:.*]]: tensor<?xi32> {tf_device.is_same_data_across_replicas = true}
+  // CHECK-SAME: %[[ARG1:.*]]: tensor<?xi32> {xla_hlo.is_same_data_across_replicas}
   // CHECK-SAME: %[[ARG2:.*]]: tensor<?xi32>)
   func @_func(%arg0: tensor<?xi32>, %arg1: tensor<?xi32>, %arg2: tensor<?xi32>) -> tensor<?xi32> {
     %0 = "tf._D"(%arg0, %arg1) : (tensor<?xi32>, tensor<?xi32>) -> tensor<?xi32>
@@ -54,9 +54,9 @@ module attributes {tf.versions = {producer = 888 : i32}} {
   }
 
   // CHECK-LABEL: func @_func
-  // CHECK-SAME: %[[ARG0:.*]]: tensor<?xi32> {tf_device.is_same_data_across_replicas = true},
+  // CHECK-SAME: %[[ARG0:.*]]: tensor<?xi32> {xla_hlo.is_same_data_across_replicas},
   // CHECK-SAME: %[[ARG1:.*]]: tensor<?xi32>,
-  // CHECK-SAME: %[[ARG2:.*]]: tensor<!tf.resource<tensor<?xi32>>> {tf_device.is_same_data_across_replicas = true}
+  // CHECK-SAME: %[[ARG2:.*]]: tensor<!tf.resource<tensor<?xi32>>> {xla_hlo.is_same_data_across_replicas}
   func @_func(%arg0: tensor<?xi32>, %arg1: tensor<?xi32>, %arg2: tensor<!tf.resource<tensor<?xi32>>>) -> tensor<?xi32> {
     %0 = "tf._D"(%arg0, %arg1) : (tensor<?xi32>, tensor<?xi32>) -> tensor<?xi32>
     return %0 : tensor<?xi32>
@@ -78,7 +78,7 @@ module attributes {tf.versions = {producer = 888 : i32}} {
   }
 
   // CHECK-LABEL: func @_func
-  // CHECK-NOT: tf_device.is_same_data_across_replicas
+  // CHECK-NOT: xla_hlo.is_same_data_across_replicas
   func @_func(%arg0: tensor<?xi32>, %arg1: tensor<?xi32>) -> tensor<?xi32> {
     %0 = "tf._D"(%arg0, %arg1) : (tensor<?xi32>, tensor<?xi32>) -> tensor<?xi32>
     return %0 : tensor<?xi32>
diff --git a/tensorflow/compiler/mlir/tensorflow/tests/breakup-islands.mlir b/tensorflow/compiler/mlir/tensorflow/tests/breakup-islands.mlir
index 61e0772726c..290da5a6135 100644
--- a/tensorflow/compiler/mlir/tensorflow/tests/breakup-islands.mlir
+++ b/tensorflow/compiler/mlir/tensorflow/tests/breakup-islands.mlir
@@ -344,3 +344,31 @@ func @switchn_control_input(%arg1: tensor<i32>) {
   }
   return
 }
+
+// CHECK-LABEL: func @single_op_island_forward_block_arg
+// CHECK: %[[CONST:.*]], %{{.*}} = tf_executor.island wraps "tf.Const"
+// CHECK: tf_executor.fetch %[[CONST]], %arg0
+func @single_op_island_forward_block_arg(%arg0: tensor<?x?x?x?xbf16>) -> (tensor<2048xf32>, tensor<?x?x?x?xbf16>) {
+  %0:2 = tf_executor.graph {
+    %outputs:2, %control = tf_executor.island {
+      %1 = "tf.Const"() {value = dense<0.000000e+00> : tensor<2048xf32>} : () -> tensor<2048xf32>
+      tf_executor.yield %1, %arg0 : tensor<2048xf32>, tensor<?x?x?x?xbf16>
+    }
+    tf_executor.fetch %outputs#0, %outputs#1 : tensor<2048xf32>, tensor<?x?x?x?xbf16>
+  }
+  return %0#0, %0#1 : tensor<2048xf32>, tensor<?x?x?x?xbf16>
+}
+
+// CHECK-LABEL: func @single_op_island_duplicate_result
+// CHECK: %[[CONST:.*]], %{{.*}} = tf_executor.island wraps "tf.Const"
+// CHECK: tf_executor.fetch %[[CONST]], %[[CONST]]
+func @single_op_island_duplicate_result() -> (tensor<2048xf32>, tensor<2048xf32>) {
+  %0:2 = tf_executor.graph {
+    %outputs:2, %control = tf_executor.island {
+      %1 = "tf.Const"() {value = dense<0.000000e+00> : tensor<2048xf32>} : () -> tensor<2048xf32>
+      tf_executor.yield %1, %1 : tensor<2048xf32>, tensor<2048xf32>
+    }
+    tf_executor.fetch %outputs#0, %outputs#1 : tensor<2048xf32>, tensor<2048xf32>
+  }
+  return %0#0, %0#1 : tensor<2048xf32>, tensor<2048xf32>
+}
diff --git a/tensorflow/compiler/mlir/tensorflow/tests/canonicalize.mlir b/tensorflow/compiler/mlir/tensorflow/tests/canonicalize.mlir
index 958a2a0182e..3b499f404bf 100644
--- a/tensorflow/compiler/mlir/tensorflow/tests/canonicalize.mlir
+++ b/tensorflow/compiler/mlir/tensorflow/tests/canonicalize.mlir
@@ -34,6 +34,13 @@ func @testBatchMatMulV2ToMatMul(%arg0: tensor<4x3xf32>, %arg1: tensor<4x5xf32>)
 // CHECK: return %0
 }
 
+// CHECK-LABEL: testBiasAddV1ToBiasAdd
+func @testBiasAddV1ToBiasAdd(%arg0: tensor<*xf32>, %arg1: tensor<128xf32>) -> tensor<*xf32> {
+  // CHECK: "tf.BiasAdd"(%arg0, %arg1) {data_format = "NHWC"} : (tensor<*xf32>, tensor<128xf32>) -> tensor<*xf32>
+  %0 = "tf.BiasAddV1"(%arg0, %arg1) : (tensor<*xf32>, tensor<128xf32>) -> tensor<*xf32>
+  return %0: tensor<*xf32>
+}
+
 // CHECK-LABEL: func @testLeakyRelu
 func @testLeakyRelu(%arg0 : tensor<16xf32>) -> (tensor<16xf32>) {
   %2 = "tf.LeakyRelu"(%arg0) {alpha = 1.0 : f32} : (tensor<16xf32>) -> tensor<16xf32>
diff --git a/tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir/parse_example.pbtxt b/tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir/parse_example.pbtxt
index ec7f0117a8c..30396242d28 100644
--- a/tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir/parse_example.pbtxt
+++ b/tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir/parse_example.pbtxt
@@ -1,73 +1,12 @@
-# RUN: tf-mlir-translate -graphdef-to-mlir -tf-enable-shape-inference-on-import=false %s -tf-input-arrays=input0 -tf-input-data-types=DT_STRING -tf-input-shapes=32 -tf-output-arrays=ParseExample/ParseExampleV2:0,ParseExample/ParseExampleV2:7 -o - | FileCheck %s
+# RUN: tf-mlir-translate -graphdef-to-mlir -tf-enable-shape-inference-on-import=false -tf-output-arrays=result %s | FileCheck %s
 
-# CHECK: %[[parse_example:.*]]:8, %[[parse_example_control:.*]] = tf_executor.island wraps "tf.ParseExampleV2"(%arg0,
-# CHECK: result_segment_sizes = dense<[2, 2, 2, 2, 0, 0]> : vector<6xi32>
-# CHECK: tf_executor.fetch %[[parse_example]]#0, %[[parse_example]]#7 : tensor<*xi64>, tensor<*xf32>
+# CHECK: %[[output:.*]], %[[control:.*]]tf_executor.island wraps "tf.ParseExample"
+# CHECK: operand_segment_sizes = dense<[1, 1, 0, 1, 1]> : vector<5xi32>
+# CHECK: result_segment_sizes = dense<[0, 0, 0, 1]> : vector<4xi32>
+# CHECK: tf_executor.fetch %[[output]] : tensor<*xi64>
 
 node {
-  name: "input0"
-  op: "Placeholder"
-  attr {
-    key: "dtype"
-    value {
-      type: DT_STRING
-    }
-  }
-  attr {
-    key: "shape"
-    value {
-      shape {
-        unknown_rank: true
-      }
-    }
-  }
-}
-node {
-  name: "ParseExample/Const"
-  op: "Const"
-  attr {
-    key: "dtype"
-    value {
-      type: DT_FLOAT
-    }
-  }
-  attr {
-    key: "value"
-    value {
-      tensor {
-        dtype: DT_FLOAT
-        tensor_shape {
-          dim {
-          }
-        }
-      }
-    }
-  }
-}
-node {
-  name: "ParseExample/Const_1"
-  op: "Const"
-  attr {
-    key: "dtype"
-    value {
-      type: DT_FLOAT
-    }
-  }
-  attr {
-    key: "value"
-    value {
-      tensor {
-        dtype: DT_FLOAT
-        tensor_shape {
-          dim {
-          }
-        }
-      }
-    }
-  }
-}
-node {
-  name: "ParseExample/ParseExampleV2/names"
+  name: "serilaized"
   op: "Const"
   attr {
     key: "dtype"
@@ -82,14 +21,16 @@ node {
         dtype: DT_STRING
         tensor_shape {
           dim {
+            size: 1
           }
         }
+        string_val: ""
       }
     }
   }
 }
 node {
-  name: "ParseExample/ParseExampleV2/sparse_keys"
+  name: "Const"
   op: "Const"
   attr {
     key: "dtype"
@@ -104,17 +45,16 @@ node {
         dtype: DT_STRING
         tensor_shape {
           dim {
-            size: 2
+            size: 1
           }
         }
-        string_val: "feature_key3"
-        string_val: "feature_key4"
+        string_val: "value"
       }
     }
   }
 }
 node {
-  name: "ParseExample/ParseExampleV2/dense_keys"
+  name: "Const_1"
   op: "Const"
   attr {
     key: "dtype"
@@ -128,54 +68,57 @@ node {
       tensor {
         dtype: DT_STRING
         tensor_shape {
-          dim {
-            size: 2
-          }
         }
-        string_val: "feature_key1"
-        string_val: "feature_key2"
+        string_val: "value"
       }
     }
   }
 }
 node {
-  name: "ParseExample/ParseExampleV2/ragged_keys"
+  name: "Const_2"
   op: "Const"
   attr {
     key: "dtype"
     value {
-      type: DT_STRING
+      type: DT_INT64
     }
   }
   attr {
     key: "value"
     value {
       tensor {
-        dtype: DT_STRING
+        dtype: DT_INT64
         tensor_shape {
-          dim {
-          }
         }
+        int64_val: -1
       }
     }
   }
 }
 node {
-  name: "ParseExample/ParseExampleV2"
-  op: "ParseExampleV2"
-  input: "input0"
-  input: "ParseExample/ParseExampleV2/names"
-  input: "ParseExample/ParseExampleV2/sparse_keys"
-  input: "ParseExample/ParseExampleV2/dense_keys"
-  input: "ParseExample/ParseExampleV2/ragged_keys"
-  input: "ParseExample/Const"
-  input: "ParseExample/Const_1"
+  name: "result"
+  op: "ParseExample"
+  input: "serilaized"
+  input: "Const"
+  input: "Const_1"
+  input: "Const_2"
+  attr {
+    key: "Ndense"
+    value {
+      i: 1
+    }
+  }
+  attr {
+    key: "Nsparse"
+    value {
+      i: 0
+    }
+  }
   attr {
     key: "Tdense"
     value {
       list {
-        type: DT_FLOAT
-        type: DT_FLOAT
+        type: DT_INT64
       }
     }
   }
@@ -184,29 +127,10 @@ node {
     value {
       list {
         shape {
+          dim {
+            size: 1
+          }
         }
-        shape {
-        }
-      }
-    }
-  }
-  attr {
-    key: "num_sparse"
-    value {
-      i: 2
-    }
-  }
-  attr {
-    key: "ragged_split_types"
-    value {
-      list {
-      }
-    }
-  }
-  attr {
-    key: "ragged_value_types"
-    value {
-      list {
       }
     }
   }
@@ -214,12 +138,12 @@ node {
     key: "sparse_types"
     value {
       list {
-        type: DT_STRING
-        type: DT_INT64
       }
     }
   }
 }
-versions {
-  producer: 175
+library {
+}
+versions {
+  producer: 413
 }
diff --git a/tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir/parse_example_v2.pbtxt b/tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir/parse_example_v2.pbtxt
new file mode 100644
index 00000000000..ec7f0117a8c
--- /dev/null
+++ b/tensorflow/compiler/mlir/tensorflow/tests/graphdef2mlir/parse_example_v2.pbtxt
@@ -0,0 +1,225 @@
+# RUN: tf-mlir-translate -graphdef-to-mlir -tf-enable-shape-inference-on-import=false %s -tf-input-arrays=input0 -tf-input-data-types=DT_STRING -tf-input-shapes=32 -tf-output-arrays=ParseExample/ParseExampleV2:0,ParseExample/ParseExampleV2:7 -o - | FileCheck %s
+
+# CHECK: %[[parse_example:.*]]:8, %[[parse_example_control:.*]] = tf_executor.island wraps "tf.ParseExampleV2"(%arg0,
+# CHECK: result_segment_sizes = dense<[2, 2, 2, 2, 0, 0]> : vector<6xi32>
+# CHECK: tf_executor.fetch %[[parse_example]]#0, %[[parse_example]]#7 : tensor<*xi64>, tensor<*xf32>
+
+node {
+  name: "input0"
+  op: "Placeholder"
+  attr {
+    key: "dtype"
+    value {
+      type: DT_STRING
+    }
+  }
+  attr {
+    key: "shape"
+    value {
+      shape {
+        unknown_rank: true
+      }
+    }
+  }
+}
+node {
+  name: "ParseExample/Const"
+  op: "Const"
+  attr {
+    key: "dtype"
+    value {
+      type: DT_FLOAT
+    }
+  }
+  attr {
+    key: "value"
+    value {
+      tensor {
+        dtype: DT_FLOAT
+        tensor_shape {
+          dim {
+          }
+        }
+      }
+    }
+  }
+}
+node {
+  name: "ParseExample/Const_1"
+  op: "Const"
+  attr {
+    key: "dtype"
+    value {
+      type: DT_FLOAT
+    }
+  }
+  attr {
+    key: "value"
+    value {
+      tensor {
+        dtype: DT_FLOAT
+        tensor_shape {
+          dim {
+          }
+        }
+      }
+    }
+  }
+}
+node {
+  name: "ParseExample/ParseExampleV2/names"
+  op: "Const"
+  attr {
+    key: "dtype"
+    value {
+      type: DT_STRING
+    }
+  }
+  attr {
+    key: "value"
+    value {
+      tensor {
+        dtype: DT_STRING
+        tensor_shape {
+          dim {
+          }
+        }
+      }
+    }
+  }
+}
+node {
+  name: "ParseExample/ParseExampleV2/sparse_keys"
+  op: "Const"
+  attr {
+    key: "dtype"
+    value {
+      type: DT_STRING
+    }
+  }
+  attr {
+    key: "value"
+    value {
+      tensor {
+        dtype: DT_STRING
+        tensor_shape {
+          dim {
+            size: 2
+          }
+        }
+        string_val: "feature_key3"
+        string_val: "feature_key4"
+      }
+    }
+  }
+}
+node {
+  name: "ParseExample/ParseExampleV2/dense_keys"
+  op: "Const"
+  attr {
+    key: "dtype"
+    value {
+      type: DT_STRING
+    }
+  }
+  attr {
+    key: "value"
+    value {
+      tensor {
+        dtype: DT_STRING
+        tensor_shape {
+          dim {
+            size: 2
+          }
+        }
+        string_val: "feature_key1"
+        string_val: "feature_key2"
+      }
+    }
+  }
+}
+node {
+  name: "ParseExample/ParseExampleV2/ragged_keys"
+  op: "Const"
+  attr {
+    key: "dtype"
+    value {
+      type: DT_STRING
+    }
+  }
+  attr {
+    key: "value"
+    value {
+      tensor {
+        dtype: DT_STRING
+        tensor_shape {
+          dim {
+          }
+        }
+      }
+    }
+  }
+}
+node {
+  name: "ParseExample/ParseExampleV2"
+  op: "ParseExampleV2"
+  input: "input0"
+  input: "ParseExample/ParseExampleV2/names"
+  input: "ParseExample/ParseExampleV2/sparse_keys"
+  input: "ParseExample/ParseExampleV2/dense_keys"
+  input: "ParseExample/ParseExampleV2/ragged_keys"
+  input: "ParseExample/Const"
+  input: "ParseExample/Const_1"
+  attr {
+    key: "Tdense"
+    value {
+      list {
+        type: DT_FLOAT
+        type: DT_FLOAT
+      }
+    }
+  }
+  attr {
+    key: "dense_shapes"
+    value {
+      list {
+        shape {
+        }
+        shape {
+        }
+      }
+    }
+  }
+  attr {
+    key: "num_sparse"
+    value {
+      i: 2
+    }
+  }
+  attr {
+    key: "ragged_split_types"
+    value {
+      list {
+      }
+    }
+  }
+  attr {
+    key: "ragged_value_types"
+    value {
+      list {
+      }
+    }
+  }
+  attr {
+    key: "sparse_types"
+    value {
+      list {
+        type: DT_STRING
+        type: DT_INT64
+      }
+    }
+  }
+}
+versions {
+  producer: 175
+}
diff --git a/tensorflow/compiler/mlir/tensorflow/tests/legalize_hlo.mlir b/tensorflow/compiler/mlir/tensorflow/tests/legalize_hlo.mlir
index 198227bf5dd..00e35460f20 100644
--- a/tensorflow/compiler/mlir/tensorflow/tests/legalize_hlo.mlir
+++ b/tensorflow/compiler/mlir/tensorflow/tests/legalize_hlo.mlir
@@ -713,6 +713,16 @@ func @convert_dot_2d_2d(%arg0: tensor<1x256xf32>, %arg1: tensor<256x1xf32>) -> t
   return %0 : tensor<1x1xf32>
 }
 
+func @broadcast_in_dim_tf_style(%arg0: tensor<8x1x16xf32>) -> tensor<3x8x8x16xf32> {
+  %0 = "xla_hlo.broadcast_in_dim"(%arg0) {broadcast_dimensions = dense<[1, 2, 3]> : tensor<3xi64>, name = "broadcast.0"} : (tensor<8x1x16xf32>) -> tensor<3x8x8x16xf32>
+  return %0 : tensor<3x8x8x16xf32>
+}
+
+func @broadcast_in_dim_general_case(%arg0: tensor<3x1x16xf32>) -> tensor<3x8x8x16xf32> {
+  %0 = "xla_hlo.broadcast_in_dim"(%arg0) {broadcast_dimensions = dense<[0, 2, 3]> : tensor<3xi64>, name = "broadcast.0"} : (tensor<3x1x16xf32>) -> tensor<3x8x8x16xf32>
+  return %0 : tensor<3x8x8x16xf32>
+}
+
 // NOTE: Assertions have been autogenerated by utils/generate-test-checks.py
 
 // CHECK-LABEL:   func @biasAdd_NHWC(
@@ -1570,3 +1580,19 @@ func @convert_dot_2d_2d(%arg0: tensor<1x256xf32>, %arg1: tensor<256x1xf32>) -> t
 // CHECK:           [[VAL_394:%.*]] = "tf.MatMul"([[VAL_392]], [[VAL_393]]) {transpose_a = false, transpose_b = false} : (tensor<1x256xf32>, tensor<256x1xf32>) -> tensor<1x1xf32>
 // CHECK:           return [[VAL_394]] : tensor<1x1xf32>
 // CHECK:         }
+
+// CHECK-LABEL:   func @broadcast_in_dim_tf_style(
+// CHECK-SAME:                  [[VAL_395:%.*]]: tensor<8x1x16xf32>) -> tensor<3x8x8x16xf32> {
+// CHECK:           [[VAL_396:%.*]] = constant dense<[3, 8, 8, 16]> : tensor<4xi64>
+// CHECK:           [[VAL_397:%.*]] = "tf.BroadcastTo"([[VAL_395]], [[VAL_396]]) : (tensor<8x1x16xf32>, tensor<4xi64>) -> tensor<3x8x8x16xf32>
+// CHECK:           return [[VAL_397]] : tensor<3x8x8x16xf32>
+// CHECK:         }
+
+// CHECK-LABEL:   func @broadcast_in_dim_general_case(
+// CHECK-SAME:                  [[VAL_398:%.*]]: tensor<3x1x16xf32>) -> tensor<3x8x8x16xf32> {
+// CHECK:           [[VAL_399:%.*]] = constant dense<[3, 1, 1, 16]> : tensor<4xi64>
+// CHECK:           [[VAL_400:%.*]] = "tf.Reshape"([[VAL_398]], [[VAL_399]]) : (tensor<3x1x16xf32>, tensor<4xi64>) -> tensor<3x1x1x16xf32>
+// CHECK:           [[VAL_401:%.*]] = constant dense<[3, 8, 8, 16]> : tensor<4xi64>
+// CHECK:           [[VAL_402:%.*]] = "tf.BroadcastTo"([[VAL_400]], [[VAL_401]]) : (tensor<3x1x1x16xf32>, tensor<4xi64>) -> tensor<3x8x8x16xf32>
+// CHECK:           return [[VAL_402]] : tensor<3x8x8x16xf32>
+// CHECK:         }
diff --git a/tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef/parse_example.mlir b/tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef/parse_example.mlir
index 1a2c1446c27..a40840e3a7c 100644
--- a/tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef/parse_example.mlir
+++ b/tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef/parse_example.mlir
@@ -1,83 +1,61 @@
 // RUN: tf-mlir-translate -mlir-to-graphdef %s -o - | FileCheck %s --dump-input-on-failure
 
-module attributes {tf.versions = {bad_consumers = [], min_consumer = 0 : i32, producer = 175 : i32}} {
-  func @main(%arg0: tensor<32x!tf.string>) -> (tensor<?x2xi64>) attributes {tf.entry_function = {inputs = "input0", outputs = "ParseExample/ParseExampleV2"}} {
+// CHECK:      name: "tf.ParseExample"
+// CHECK-NEXT: op: "ParseExample"
+// CHECK-NEXT: input: "tf.Const3"
+// CHECK-NEXT: input: "tf.Const"
+// CHECK-NEXT: input: "tf.Const1"
+// CHECK-NEXT: input: "tf.Const2"
+// CHECK-NEXT: attr {
+// CHECK-NEXT:   key: "Ndense"
+// CHECK-NEXT:   value {
+// CHECK-NEXT:     i: 1
+// CHECK-NEXT:   }
+// CHECK-NEXT: }
+// CHECK-NEXT: attr {
+// CHECK-NEXT:   key: "Nsparse"
+// CHECK-NEXT:   value {
+// CHECK-NEXT:     i: 0
+// CHECK-NEXT:   }
+// CHECK-NEXT: }
+// CHECK-NEXT: attr {
+// CHECK-NEXT:   key: "Tdense"
+// CHECK-NEXT:   value {
+// CHECK-NEXT:     list {
+// CHECK-NEXT:       type: DT_INT64
+// CHECK-NEXT:     }
+// CHECK-NEXT:   }
+// CHECK-NEXT: }
+// CHECK-NEXT: attr {
+// CHECK:        key: "dense_shapes"
+// CHECK-NEXT:   value {
+// CHECK-NEXT:     list {
+// CHECK-NEXT:       shape {
+// CHECK-NEXT:         dim {
+// CHECK-NEXT:           size: 1
+// CHECK-NEXT:         }
+// CHECK-NEXT:       }
+// CHECK-NEXT:     }
+// CHECK-NEXT:   }
+// CHECK-NEXT: }
+// CHECK-NEXT: attr {
+// CHECK-NEXT:   key: "sparse_types"
+// CHECK-NEXT:   value {
+// CHECK-NEXT:     list {
+// CHECK-NEXT:     }
+// CHECK-NEXT:   }
+// CHECK-NEXT: }
 
+module attributes {tf.versions = {bad_consumers = [], min_consumer = 0 : i32, producer = 413 : i32}} {
+  func @main() -> tensor<*xi64> attributes {tf.entry_function = {control_outputs = "", inputs = "", outputs = "result"}} {
     %0 = tf_executor.graph {
-      %outputs, %control = tf_executor.island wraps "tf.Const"() {device = "", dtype = f32, value = dense<[]> : tensor<0xf32>} : () -> tensor<0xf32>
-      %outputs_0, %control_1 = tf_executor.island wraps "tf.Const"() {device = "", dtype = f32, value = dense<[]> : tensor<0xf32>} : () -> tensor<0xf32>
-      %outputs_2, %control_3 = tf_executor.island wraps "tf.Const"() {device = "", dtype = !tf.string, value = dense<""> : tensor<2x!tf.string>} : () -> tensor<2x!tf.string>
-      %outputs_4, %control_5 = tf_executor.island wraps "tf.Const"() {device = "", dtype = !tf.string, value = dense<""> : tensor<0x!tf.string>} : () -> tensor<0x!tf.string>
-      %outputs_6, %control_7 = tf_executor.island wraps "tf.Const"() {device = "", dtype = !tf.string, value = dense<""> : tensor<0x!tf.string>} : () -> tensor<0x!tf.string>
-      %outputs_8, %control_9 = tf_executor.island wraps "tf.Const"() {device = "", dtype = !tf.string, value = dense<""> : tensor<2x!tf.string>} : () -> tensor<2x!tf.string>
-
-      %outputs_10:8, %control_11 = tf_executor.island wraps "tf.ParseExampleV2"(%arg0, %outputs_4, %outputs_8, %outputs_2, %outputs_6, %outputs, %outputs_0) {Tdense = ["tfdtype$DT_FLOAT", "tfdtype$DT_FLOAT"], dense_shapes = [#tf.shape<>, #tf.shape<>], device = "", num_sparse = 2 : i64, ragged_split_types = [], ragged_value_types = [], result_segment_sizes = dense<[2, 2, 2, 2, 0, 0]> : vector<6xi32>, sparse_types = ["tfdtype$DT_STRING", "tfdtype$DT_INT64"]} : (tensor<32x!tf.string>, tensor<0x!tf.string>, tensor<2x!tf.string>, tensor<2x!tf.string>, tensor<0x!tf.string>, tensor<0xf32>, tensor<0xf32>) -> (tensor<?x2xi64>, tensor<?x2xi64>, tensor<?x!tf.string>, tensor<?xi64>, tensor<2xi64>, tensor<2xi64>, tensor<32xf32>, tensor<32xf32>) loc("ParseExample")
-      // CHECK:      name: "ParseExample"
-      // CHECK-NEXT: op: "ParseExampleV2"
-      // CHECK-NEXT: input: "input0"
-      // CHECK-NEXT: input: "tf.Const3"
-      // CHECK-NEXT: input: "tf.Const5"
-      // CHECK-NEXT: input: "tf.Const2"
-      // CHECK-NEXT: input: "tf.Const4"
-      // CHECK-NEXT: input: "tf.Const"
-      // CHECK-NEXT: input: "tf.Const1"
-      // CHECK-NEXT: attr {
-      // CHECK-NEXT:   key: "Tdense"
-      // CHECK-NEXT:     value {
-      // CHECK-NEXT:       list {
-      // CHECK-NEXT:         type: DT_FLOAT
-      // CHECK-NEXT:         type: DT_FLOAT
-      // CHECK-NEXT:       }
-      // CHECK-NEXT:     }
-      // CHECK-NEXT:   }
-      // CHECK-NEXT: attr {
-      // CHECK:        key: "dense_shapes"
-      // CHECK-NEXT:   value {
-      // CHECK-NEXT:     list {
-      // CHECK-NEXT:       shape {
-      // CHECK-NEXT:       }
-      // CHECK-NEXT:       shape {
-      // CHECK-NEXT:       }
-      // CHECK-NEXT:     }
-      // CHECK-NEXT:   }
-      // CHECK-NEXT: }
-      // CHECK-NEXT: attr {
-      // CHECK-NEXT:   key: "num_sparse"
-      // CHECK-NEXT:   value {
-      // CHECK-NEXT:     i: 2
-      // CHECK-NEXT:   }
-      // CHECK-NEXT: }
-      // CHECK-NEXT: attr {
-      // CHECK-NEXT:   key: "ragged_split_types"
-      // CHECK-NEXT:   value {
-      // CHECK-NEXT:     list {
-      // CHECK-NEXT:     }
-      // CHECK-NEXT:   }
-      // CHECK-NEXT: }
-      // CHECK-NEXT: attr {
-      // CHECK-NEXT:   key: "ragged_value_types"
-      // CHECK-NEXT:   value {
-      // CHECK-NEXT:     list {
-      // CHECK-NEXT:     }
-      // CHECK-NEXT:   }
-      // CHECK-NEXT: }
-      // CHECK-NEXT: attr {
-      // CHECK-NEXT:   key: "sparse_types"
-      // CHECK-NEXT:   value {
-      // CHECK-NEXT:     list {
-      // CHECK-NEXT:       type: DT_STRING
-      // CHECK-NEXT:       type: DT_INT64
-      // CHECK-NEXT:     }
-      // CHECK-NEXT:   }
-      // CHECK-NEXT: }
-
-      tf_executor.fetch %outputs_10#0 : tensor<?x2xi64>
+      %outputs, %control = tf_executor.island wraps "tf.Const"() {device = "", value = dense<"value"> : tensor<1x!tf.string>} : () -> tensor<1x!tf.string>
+      %outputs_0, %control_1 = tf_executor.island wraps "tf.Const"() {device = "", value = dense<"value"> : tensor<!tf.string>} : () -> tensor<!tf.string>
+      %outputs_2, %control_3 = tf_executor.island wraps "tf.Const"() {device = "", value = dense<-1> : tensor<i64>} : () -> tensor<i64>
+      %outputs_4, %control_5 = tf_executor.island wraps "tf.Const"() {device = "", value = dense<""> : tensor<1x!tf.string>} : () -> tensor<1x!tf.string>
+      %outputs_6, %control_7 = tf_executor.island wraps "tf.ParseExample"(%outputs_4, %outputs, %outputs_0, %outputs_2) {dense_shapes = [#tf.shape<1>], device = "", operand_segment_sizes = dense<[1, 1, 0, 1, 1]> : vector<5xi32>, result_segment_sizes = dense<[0, 0, 0, 1]> : vector<4xi32>} : (tensor<1x!tf.string>, tensor<1x!tf.string>, tensor<!tf.string>, tensor<i64>) -> tensor<*xi64>
+      tf_executor.fetch %outputs_6 : tensor<*xi64>
     }
-    return %0#0 : tensor<?x2xi64>
-    // CHECK:      name: "ParseExample/ParseExampleV2"
-    // CHECK-NEXT: op: "_Retval"
-    // CHECK-NEXT: input: "ParseExample"
-
+    return %0 : tensor<*xi64>
   }
 }
-
diff --git a/tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef/parse_example_v2.mlir b/tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef/parse_example_v2.mlir
new file mode 100644
index 00000000000..1a2c1446c27
--- /dev/null
+++ b/tensorflow/compiler/mlir/tensorflow/tests/mlir2graphdef/parse_example_v2.mlir
@@ -0,0 +1,83 @@
+// RUN: tf-mlir-translate -mlir-to-graphdef %s -o - | FileCheck %s --dump-input-on-failure
+
+module attributes {tf.versions = {bad_consumers = [], min_consumer = 0 : i32, producer = 175 : i32}} {
+  func @main(%arg0: tensor<32x!tf.string>) -> (tensor<?x2xi64>) attributes {tf.entry_function = {inputs = "input0", outputs = "ParseExample/ParseExampleV2"}} {
+
+    %0 = tf_executor.graph {
+      %outputs, %control = tf_executor.island wraps "tf.Const"() {device = "", dtype = f32, value = dense<[]> : tensor<0xf32>} : () -> tensor<0xf32>
+      %outputs_0, %control_1 = tf_executor.island wraps "tf.Const"() {device = "", dtype = f32, value = dense<[]> : tensor<0xf32>} : () -> tensor<0xf32>
+      %outputs_2, %control_3 = tf_executor.island wraps "tf.Const"() {device = "", dtype = !tf.string, value = dense<""> : tensor<2x!tf.string>} : () -> tensor<2x!tf.string>
+      %outputs_4, %control_5 = tf_executor.island wraps "tf.Const"() {device = "", dtype = !tf.string, value = dense<""> : tensor<0x!tf.string>} : () -> tensor<0x!tf.string>
+      %outputs_6, %control_7 = tf_executor.island wraps "tf.Const"() {device = "", dtype = !tf.string, value = dense<""> : tensor<0x!tf.string>} : () -> tensor<0x!tf.string>
+      %outputs_8, %control_9 = tf_executor.island wraps "tf.Const"() {device = "", dtype = !tf.string, value = dense<""> : tensor<2x!tf.string>} : () -> tensor<2x!tf.string>
+
+      %outputs_10:8, %control_11 = tf_executor.island wraps "tf.ParseExampleV2"(%arg0, %outputs_4, %outputs_8, %outputs_2, %outputs_6, %outputs, %outputs_0) {Tdense = ["tfdtype$DT_FLOAT", "tfdtype$DT_FLOAT"], dense_shapes = [#tf.shape<>, #tf.shape<>], device = "", num_sparse = 2 : i64, ragged_split_types = [], ragged_value_types = [], result_segment_sizes = dense<[2, 2, 2, 2, 0, 0]> : vector<6xi32>, sparse_types = ["tfdtype$DT_STRING", "tfdtype$DT_INT64"]} : (tensor<32x!tf.string>, tensor<0x!tf.string>, tensor<2x!tf.string>, tensor<2x!tf.string>, tensor<0x!tf.string>, tensor<0xf32>, tensor<0xf32>) -> (tensor<?x2xi64>, tensor<?x2xi64>, tensor<?x!tf.string>, tensor<?xi64>, tensor<2xi64>, tensor<2xi64>, tensor<32xf32>, tensor<32xf32>) loc("ParseExample")
+      // CHECK:      name: "ParseExample"
+      // CHECK-NEXT: op: "ParseExampleV2"
+      // CHECK-NEXT: input: "input0"
+      // CHECK-NEXT: input: "tf.Const3"
+      // CHECK-NEXT: input: "tf.Const5"
+      // CHECK-NEXT: input: "tf.Const2"
+      // CHECK-NEXT: input: "tf.Const4"
+      // CHECK-NEXT: input: "tf.Const"
+      // CHECK-NEXT: input: "tf.Const1"
+      // CHECK-NEXT: attr {
+      // CHECK-NEXT:   key: "Tdense"
+      // CHECK-NEXT:     value {
+      // CHECK-NEXT:       list {
+      // CHECK-NEXT:         type: DT_FLOAT
+      // CHECK-NEXT:         type: DT_FLOAT
+      // CHECK-NEXT:       }
+      // CHECK-NEXT:     }
+      // CHECK-NEXT:   }
+      // CHECK-NEXT: attr {
+      // CHECK:        key: "dense_shapes"
+      // CHECK-NEXT:   value {
+      // CHECK-NEXT:     list {
+      // CHECK-NEXT:       shape {
+      // CHECK-NEXT:       }
+      // CHECK-NEXT:       shape {
+      // CHECK-NEXT:       }
+      // CHECK-NEXT:     }
+      // CHECK-NEXT:   }
+      // CHECK-NEXT: }
+      // CHECK-NEXT: attr {
+      // CHECK-NEXT:   key: "num_sparse"
+      // CHECK-NEXT:   value {
+      // CHECK-NEXT:     i: 2
+      // CHECK-NEXT:   }
+      // CHECK-NEXT: }
+      // CHECK-NEXT: attr {
+      // CHECK-NEXT:   key: "ragged_split_types"
+      // CHECK-NEXT:   value {
+      // CHECK-NEXT:     list {
+      // CHECK-NEXT:     }
+      // CHECK-NEXT:   }
+      // CHECK-NEXT: }
+      // CHECK-NEXT: attr {
+      // CHECK-NEXT:   key: "ragged_value_types"
+      // CHECK-NEXT:   value {
+      // CHECK-NEXT:     list {
+      // CHECK-NEXT:     }
+      // CHECK-NEXT:   }
+      // CHECK-NEXT: }
+      // CHECK-NEXT: attr {
+      // CHECK-NEXT:   key: "sparse_types"
+      // CHECK-NEXT:   value {
+      // CHECK-NEXT:     list {
+      // CHECK-NEXT:       type: DT_STRING
+      // CHECK-NEXT:       type: DT_INT64
+      // CHECK-NEXT:     }
+      // CHECK-NEXT:   }
+      // CHECK-NEXT: }
+
+      tf_executor.fetch %outputs_10#0 : tensor<?x2xi64>
+    }
+    return %0#0 : tensor<?x2xi64>
+    // CHECK:      name: "ParseExample/ParseExampleV2"
+    // CHECK-NEXT: op: "_Retval"
+    // CHECK-NEXT: input: "ParseExample"
+
+  }
+}
+
diff --git a/tensorflow/compiler/mlir/tensorflow/tests/op_fusion.mlir b/tensorflow/compiler/mlir/tensorflow/tests/op_fusion.mlir
new file mode 100644
index 00000000000..4688d2ee712
--- /dev/null
+++ b/tensorflow/compiler/mlir/tensorflow/tests/op_fusion.mlir
@@ -0,0 +1,109 @@
+// RUN: tf-opt %s -op-fusion | FileCheck %s --dump-input-on-failure
+
+//===----------------------------------------------------------------------===//
+// Conv2D + BiasAdd + <Activation> fusions.
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: conv2DBiasAdd_noActivation
+func @conv2DBiasAdd_noActivation(%arg0: tensor<128xf32>, %arg1: tensor<1x1x3x128xf32>, %arg2: tensor<8x32x32x3xf32>) -> (tensor<*xf32>) {
+  // CHECK: %[[VAL_0:.*]] = "tf._FusedConv2D"(%arg2, %arg1, %arg0) {data_format = "NHWC", dilations = [1, 1, 1, 1], epsilon = 0.000000e+00 : f32, explicit_paddings = [], fused_ops = ["BiasAdd"], padding = "SAME", strides = [1, 1, 1, 1], use_cudnn_on_gpu = true} : (tensor<8x32x32x3xf32>, tensor<1x1x3x128xf32>, tensor<128xf32>) -> tensor<*xf32>
+  // CHECK: %[[VAL_1:.*]] = "tf.Identity"(%[[VAL_0]]) : (tensor<*xf32>) -> tensor<*xf32>
+  // CHECK: return %[[VAL_1]]
+  %0 = "tf.Conv2D"(%arg2, %arg1) {data_format = "NHWC", dilations = [1, 1, 1, 1], explicit_paddings = [], padding = "SAME", strides = [1, 1, 1, 1], use_cudnn_on_gpu = true} : (tensor<8x32x32x3xf32>, tensor<1x1x3x128xf32>) -> tensor<*xf32>
+  %1 = "tf.BiasAdd"(%0, %arg0) {data_format = "NHWC"} : (tensor<*xf32>, tensor<128xf32>) -> tensor<*xf32>
+  %2 = "tf.Identity"(%1) : (tensor<*xf32>) -> tensor<*xf32>
+  return %2 : tensor<*xf32>
+}
+
+// CHECK-LABEL: conv2DBiasAdd_reluActivation
+func @conv2DBiasAdd_reluActivation(%arg0: tensor<128xf32>, %arg1: tensor<1x1x3x128xf32>, %arg2: tensor<8x32x32x3xf32>) -> (tensor<*xf32>) {
+  // CHECK: %[[VAL_0:.*]] = "tf._FusedConv2D"(%arg2, %arg1, %arg0) {data_format = "NHWC", dilations = [1, 1, 1, 1], epsilon = 0.000000e+00 : f32, explicit_paddings = [], fused_ops = ["BiasAdd", "Relu"], padding = "SAME", strides = [1, 1, 1, 1], use_cudnn_on_gpu = true} : (tensor<8x32x32x3xf32>, tensor<1x1x3x128xf32>, tensor<128xf32>) -> tensor<*xf32>
+  // CHECK: %[[VAL_1:.*]] = "tf.Identity"(%[[VAL_0]]) : (tensor<*xf32>) -> tensor<*xf32>
+  // CHECK: return %[[VAL_1]]
+  %0 = "tf.Conv2D"(%arg2, %arg1) {data_format = "NHWC", dilations = [1, 1, 1, 1], explicit_paddings = [], padding = "SAME", strides = [1, 1, 1, 1], use_cudnn_on_gpu = true} : (tensor<8x32x32x3xf32>, tensor<1x1x3x128xf32>) -> tensor<*xf32>
+  %1 = "tf.BiasAdd"(%0, %arg0) {data_format = "NHWC"} : (tensor<*xf32>, tensor<128xf32>) -> tensor<*xf32>
+  %2 = "tf.Relu"(%1) : (tensor<*xf32>) -> tensor<*xf32>
+  %3 = "tf.Identity"(%2) : (tensor<*xf32>) -> tensor<*xf32>
+  return %3 : tensor<*xf32>
+}
+
+// CHECK-LABEL: conv2DBiasAdd_relu6Activation
+func @conv2DBiasAdd_relu6Activation(%arg0: tensor<128xf32>, %arg1: tensor<1x1x3x128xf32>, %arg2: tensor<8x32x32x3xf32>) -> (tensor<*xf32>) {
+  // CHECK: %[[VAL_0:.*]] = "tf._FusedConv2D"(%arg2, %arg1, %arg0) {data_format = "NHWC", dilations = [1, 1, 1, 1], epsilon = 0.000000e+00 : f32, explicit_paddings = [], fused_ops = ["BiasAdd", "Relu6"], padding = "SAME", strides = [1, 1, 1, 1], use_cudnn_on_gpu = true} : (tensor<8x32x32x3xf32>, tensor<1x1x3x128xf32>, tensor<128xf32>) -> tensor<*xf32>
+  // CHECK: %[[VAL_1:.*]] = "tf.Identity"(%[[VAL_0]]) : (tensor<*xf32>) -> tensor<*xf32>
+  // CHECK: return %[[VAL_1]]
+  %0 = "tf.Conv2D"(%arg2, %arg1) {data_format = "NHWC", dilations = [1, 1, 1, 1], explicit_paddings = [], padding = "SAME", strides = [1, 1, 1, 1], use_cudnn_on_gpu = true} : (tensor<8x32x32x3xf32>, tensor<1x1x3x128xf32>) -> tensor<*xf32>
+  %1 = "tf.BiasAdd"(%0, %arg0) {data_format = "NHWC"} : (tensor<*xf32>, tensor<128xf32>) -> tensor<*xf32>
+  %2 = "tf.Relu6"(%1) : (tensor<*xf32>) -> tensor<*xf32>
+  %3 = "tf.Identity"(%2) : (tensor<*xf32>) -> tensor<*xf32>
+  return %3 : tensor<*xf32>
+}
+
+// CHECK-LABEL: conv2DBiasAdd_eluActivation
+func @conv2DBiasAdd_eluActivation(%arg0: tensor<128xf32>, %arg1: tensor<1x1x3x128xf32>, %arg2: tensor<8x32x32x3xf32>) -> (tensor<*xf32>) {
+  // CHECK: %[[VAL_0:.*]] = "tf._FusedConv2D"(%arg2, %arg1, %arg0) {data_format = "NHWC", dilations = [1, 1, 1, 1], epsilon = 0.000000e+00 : f32, explicit_paddings = [], fused_ops = ["BiasAdd", "Elu"], padding = "SAME", strides = [1, 1, 1, 1], use_cudnn_on_gpu = true} : (tensor<8x32x32x3xf32>, tensor<1x1x3x128xf32>, tensor<128xf32>) -> tensor<*xf32>
+  // CHECK: %[[VAL_1:.*]] = "tf.Identity"(%[[VAL_0]]) : (tensor<*xf32>) -> tensor<*xf32>
+  // CHECK: return %[[VAL_1]]
+  %0 = "tf.Conv2D"(%arg2, %arg1) {data_format = "NHWC", dilations = [1, 1, 1, 1], explicit_paddings = [], padding = "SAME", strides = [1, 1, 1, 1], use_cudnn_on_gpu = true} : (tensor<8x32x32x3xf32>, tensor<1x1x3x128xf32>) -> tensor<*xf32>
+  %1 = "tf.BiasAdd"(%0, %arg0) {data_format = "NHWC"} : (tensor<*xf32>, tensor<128xf32>) -> tensor<*xf32>
+  %2 = "tf.Elu"(%1) : (tensor<*xf32>) -> tensor<*xf32>
+  %3 = "tf.Identity"(%2) : (tensor<*xf32>) -> tensor<*xf32>
+  return %3 : tensor<*xf32>
+}
+
+// CHECK-LABEL: conv2DBiasAdd_convMultipleUses
+func @conv2DBiasAdd_convMultipleUses(%arg0: tensor<128xf32>, %arg1: tensor<1x1x3x128xf32>, %arg2: tensor<8x32x32x3xf32>) -> (tensor<*xf32>, tensor<*xf32>) {
+  // CHECK-NOT: "tf._FusedConv2D"
+  %0 = "tf.Conv2D"(%arg2, %arg1) {data_format = "NHWC", dilations = [1, 1, 1, 1], explicit_paddings = [], padding = "SAME", strides = [1, 1, 1, 1], use_cudnn_on_gpu = true} : (tensor<8x32x32x3xf32>, tensor<1x1x3x128xf32>) -> tensor<*xf32>
+  %1 = "tf.BiasAdd"(%0, %arg0) {data_format = "NHWC"} : (tensor<*xf32>, tensor<128xf32>) -> tensor<*xf32>
+  %2 = "tf.Elu"(%1) : (tensor<*xf32>) -> tensor<*xf32>
+  %3 = "tf.Identity"(%2) : (tensor<*xf32>) -> tensor<*xf32>
+  %4 = "tf.Identity"(%0) : (tensor<*xf32>) -> tensor<*xf32>
+  return %3, %4 : tensor<*xf32>, tensor<*xf32>
+}
+
+// CHECK-LABEL: conv2DBiasAdd_biasAddMultipleUse
+func @conv2DBiasAdd_biasAddMultipleUse(%arg0: tensor<128xf32>, %arg1: tensor<1x1x3x128xf32>, %arg2: tensor<8x32x32x3xf32>) -> (tensor<*xf32>, tensor<*xf32>) {
+  // CHECK-DAG: %[[VAL:.*]] = "tf._FusedConv2D"(%arg2, %arg1, %arg0) {data_format = "NHWC", dilations = [1, 1, 1, 1], epsilon = 0.000000e+00 : f32, explicit_paddings = [], fused_ops = ["BiasAdd"], padding = "SAME", strides = [1, 1, 1, 1], use_cudnn_on_gpu = true} : (tensor<8x32x32x3xf32>, tensor<1x1x3x128xf32>, tensor<128xf32>) -> tensor<*xf32>
+  // CHECK-DAG: %[[VAL_0:.*]] = "tf.Elu"(%[[VAL]]) : (tensor<*xf32>) -> tensor<*xf32>
+  // CHECK-DAG: %[[VAL_1:.*]] = "tf.Identity"(%[[VAL_0]]) : (tensor<*xf32>) -> tensor<*xf32>
+  // CHECK-DAG: %[[VAL_2:.*]] = "tf.Identity"(%[[VAL]]) : (tensor<*xf32>) -> tensor<*xf32>
+  // CHECK: return %[[VAL_1]], %[[VAL_2]]
+  %0 = "tf.Conv2D"(%arg2, %arg1) {data_format = "NHWC", dilations = [1, 1, 1, 1], explicit_paddings = [], padding = "SAME", strides = [1, 1, 1, 1], use_cudnn_on_gpu = true} : (tensor<8x32x32x3xf32>, tensor<1x1x3x128xf32>) -> tensor<*xf32>
+  %1 = "tf.BiasAdd"(%0, %arg0) {data_format = "NHWC"} : (tensor<*xf32>, tensor<128xf32>) -> tensor<*xf32>
+  %2 = "tf.Elu"(%1) : (tensor<*xf32>) -> tensor<*xf32>
+  %3 = "tf.Identity"(%2) : (tensor<*xf32>) -> tensor<*xf32>
+  %4 = "tf.Identity"(%1) : (tensor<*xf32>) -> tensor<*xf32>
+  return %3, %4 : tensor<*xf32>, tensor<*xf32>
+}
+
+// CHECK-LABEL: conv2D_noFusion
+func @conv2D_noFusion(%arg0: tensor<128xf32>, %arg1: tensor<1x1x3x128xf32>, %arg2: tensor<8x32x32x3xf32>) -> (tensor<*xf32>) {
+  // CHECK-NOT: "tf._FusedConv2D"
+  %0 = "tf.Conv2D"(%arg2, %arg1) {data_format = "NHWC", dilations = [1, 1, 1, 1], explicit_paddings = [], padding = "SAME", strides = [1, 1, 1, 1], use_cudnn_on_gpu = true} : (tensor<8x32x32x3xf32>, tensor<1x1x3x128xf32>) -> tensor<*xf32>
+  %2 = "tf.Elu"(%0) : (tensor<*xf32>) -> tensor<*xf32>
+  %3 = "tf.Identity"(%2) : (tensor<*xf32>) -> tensor<*xf32>
+  return %3 : tensor<*xf32>
+}
+
+// CHECK-LABEL: conv2D_noFusion1
+func @conv2D_noFusion1(%arg0: tensor<*xf32>, %arg1: tensor<1x1x3x128xf32>, %arg2: tensor<8x32x32x3xf32>) -> (tensor<*xf32>) {
+  // CHECK-NOT: "tf._FusedConv2D"
+  %0 = "tf.Conv2D"(%arg2, %arg1) {data_format = "NHWC", dilations = [1, 1, 1, 1], explicit_paddings = [], padding = "SAME", strides = [1, 1, 1, 1], use_cudnn_on_gpu = true} : (tensor<8x32x32x3xf32>, tensor<1x1x3x128xf32>) -> tensor<*xf32>
+  // The result of the conv must be the first input to BiasAdd to be fusable.
+  %1 = "tf.BiasAdd"(%arg0, %0) {data_format = "NHWC"} : (tensor<*xf32>, tensor<*xf32>) -> tensor<*xf32>
+  %2 = "tf.Elu"(%1) : (tensor<*xf32>) -> tensor<*xf32>
+  %3 = "tf.Identity"(%2) : (tensor<*xf32>) -> tensor<*xf32>
+  return %3 : tensor<*xf32>
+}
+
+// CHECK-LABEL: conv2D_dataFormatMismatch
+func @conv2D_dataFormatMismatch(%arg0: tensor<128xf32>, %arg1: tensor<1x1x3x128xf32>, %arg2: tensor<8x32x32x3xf32>) -> (tensor<*xf32>) {
+  // CHECK-NOT: "tf._FusedConv2D"
+  %0 = "tf.Conv2D"(%arg2, %arg1) {data_format = "NHWC", dilations = [1, 1, 1, 1], explicit_paddings = [], padding = "SAME", strides = [1, 1, 1, 1], use_cudnn_on_gpu = true} : (tensor<8x32x32x3xf32>, tensor<1x1x3x128xf32>) -> tensor<*xf32>
+  // The result of the conv must be the first input to BiasAdd to be fusable.
+  %1 = "tf.BiasAdd"(%0, %arg0) {data_format = "NCHW"} : (tensor<*xf32>, tensor<128xf32>) -> tensor<*xf32>
+  %2 = "tf.Elu"(%1) : (tensor<*xf32>) -> tensor<*xf32>
+  %3 = "tf.Identity"(%2) : (tensor<*xf32>) -> tensor<*xf32>
+  return %3 : tensor<*xf32>
+}
diff --git a/tensorflow/compiler/mlir/tensorflow/tests/rewrite_tpu_embedding_ops.mlir b/tensorflow/compiler/mlir/tensorflow/tests/rewrite_tpu_embedding_ops.mlir
new file mode 100644
index 00000000000..e4e6304ef3b
--- /dev/null
+++ b/tensorflow/compiler/mlir/tensorflow/tests/rewrite_tpu_embedding_ops.mlir
@@ -0,0 +1,42 @@
+// RUN: tf-opt -tf-rewrite-tpu-embedding-ops %s | FileCheck %s --dump-input-on-failure
+
+// CHECK-LABEL: func @recv_tpu_embedding_activations
+func @recv_tpu_embedding_activations() -> (tensor<512x256xf32>) {
+  // CHECK: %[[DATA:.*]] = "tf._RecvTPUEmbeddingDeduplicationData"() {config = {{.*}}} : () -> tensor<!tf.variant>
+  // CHECK: %[[RESULT:.*]] = "tf._RecvTPUEmbeddingActivations"(%[[DATA]]) {config = {{.*}}} : (tensor<!tf.variant>) -> tensor<512x256xf32>
+  // CHECK: return %[[RESULT]]
+  // CHECK-NOT: tf.RecvTPUEmbeddingActivations
+  // CHECK-NOT: tf.SendTPUEmbeddingGradients
+
+  %0 = "tf.RecvTPUEmbeddingActivations"() {config = "\0A%\0A\0Dwatches_table\10\F5\03\18\80\02 \01*\0C\1A\00j\05\0D\00\00\80?\88\01\01\10\02\18\80\04 \01(\02"} : () -> tensor<512x256xf32>
+  return %0 : tensor<512x256xf32>
+}
+
+// CHECK-LABEL: func @send_tpu_embedding_gradients
+func @send_tpu_embedding_gradients(%arg0: tensor<512x256xf32>) -> () {
+  // CHECK: %[[DATA:.*]] = "tf._RecvTPUEmbeddingDeduplicationData"() {config = {{.*}}} : () -> tensor<!tf.variant>
+  // CHECK: "tf._SendTPUEmbeddingGradients"(%arg0, %[[DATA]]) {config = {{.*}}, operand_segment_sizes = dense<[1, 0, 1]> : vector<3xi32>} : (tensor<512x256xf32>, tensor<!tf.variant>) -> ()
+  // CHECK-NOT: tf.SendTPUEmbeddingGradients
+  // CHECK-NOT: tf.RecvTPUEmbeddingActivations
+
+  "tf.SendTPUEmbeddingGradients"(%arg0) {config = "\0A%\0A\0Dwatches_table\10\F5\03\18\80\02 \01*\0C\1A\00j\05\0D\00\00\80?\88\01\01\10\02\18\80\04 \01(\02", operand_segment_sizes = dense<[1, 0]> : vector<2xi32>} : (tensor<512x256xf32>) -> ()
+  return
+}
+
+// CHECK-LABEL: func @recv_send_ops
+func @recv_send_ops() -> () {
+  // CHECK: %[[DATA:.*]] = "tf._RecvTPUEmbeddingDeduplicationData"()
+  // CHECK: %[[ACTIVATIONS:.*]] = "tf._RecvTPUEmbeddingActivations"(%[[DATA]])
+  // CHECK: "tf._SendTPUEmbeddingGradients"(%[[ACTIVATIONS]], %[[DATA]])
+
+  %0 = "tf.RecvTPUEmbeddingActivations"() {config = "\0A%\0A\0Dwatches_table\10\F5\03\18\80\02 \01*\0C\1A\00j\05\0D\00\00\80?\88\01\01\10\02\18\80\04 \01(\02"} : () -> tensor<512x256xf32>
+  "tf.SendTPUEmbeddingGradients"(%0) {config = "\0A%\0A\0Dwatches_table\10\F5\03\18\80\02 \01*\0C\1A\00j\05\0D\00\00\80?\88\01\01\10\02\18\80\04 \01(\02", operand_segment_sizes = dense<[1, 0]> : vector<2xi32>} : (tensor<512x256xf32>) -> ()
+  return
+}
+
+// CHECK-LABEL: func @no_embedding_ops
+func @no_embedding_ops(%arg0: tensor<2x2xf32>) -> (tensor<2x2xf32>) {
+  // CHECK: tf.Add
+  %0 = "tf.Add"(%arg0, %arg0) : (tensor<2x2xf32>, tensor<2x2xf32>) -> tensor<2x2xf32>
+  return %0 : tensor<2x2xf32>
+}
diff --git a/tensorflow/compiler/mlir/tensorflow/tests/tpu_outside_compilation_cluster.mlir b/tensorflow/compiler/mlir/tensorflow/tests/tpu_outside_compilation_cluster.mlir
new file mode 100644
index 00000000000..68c0217b73d
--- /dev/null
+++ b/tensorflow/compiler/mlir/tensorflow/tests/tpu_outside_compilation_cluster.mlir
@@ -0,0 +1,250 @@
+// RUN: tf-opt %s -tf-tpu-outside-compilation-cluster | FileCheck %s --dump-input-on-failure
+
+// CHECK-LABEL: func @one_cluster_no_dependencies
+func @one_cluster_no_dependencies() {
+  // CHECK: "tf.opA"
+  // CHECK: "tf.opB"
+  // CHECK-SAME: _xla_outside_compilation = "{{[a-zA-Z_0-9]+}}"
+  // CHECK: "tf.opC"
+  "tf_device.cluster"() ( {
+    "tf.opA"() : () -> ()
+    "tf.opB"() {_xla_outside_compilation = "0"} : () -> ()
+    "tf.opC"() : () -> ()
+    tf_device.return
+  }) {cluster_attr = "cluster_attr"} : () -> ()
+  return
+}
+
+// CHECK-LABEL: func @one_cluster_with_one_op
+func @one_cluster_with_one_op() {
+  // CHECK: "tf.opA"
+  // CHECK-NEXT: "tf.opB"
+  // CHECK-SAME: _xla_outside_compilation = "{{[a-zA-Z_0-9]+}}"
+  // CHECK-NEXT: "tf.opC"
+  "tf_device.cluster"() ( {
+    %a = "tf.opA"() : () -> tensor<i32>
+    %b = "tf.opB"(%a) {_xla_outside_compilation = "0"} : (tensor<i32>) -> tensor<i32>
+    "tf.opC"(%b) : (tensor<i32>) -> ()
+    tf_device.return
+  }) {cluster_attr = "cluster_attr"} : () -> ()
+  return
+}
+
+// CHECK-LABEL: func @one_cluster_with_two_ops
+func @one_cluster_with_two_ops() {
+  // CHECK: "tf.opA"
+  // CHECK-NEXT: "tf.opB"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER2:[a-zA-Z_0-9]+]]"
+  // CHECK-NEXT: "tf.opC"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER2]]"
+  // CHECK-NEXT: "tf.opD"
+  "tf_device.cluster"() ( {
+    %a = "tf.opA"() : () -> tensor<i32>
+    %b = "tf.opB"(%a) {_xla_outside_compilation = "0"} : (tensor<i32>) -> tensor<i32>
+    %c = "tf.opC"(%b) {_xla_outside_compilation = "0"} : (tensor<i32>) -> tensor<i32>
+    "tf.opD"(%c) : (tensor<i32>) -> ()
+    tf_device.return
+  }) {cluster_attr = "cluster_attr"} : () -> ()
+  return
+}
+
+// CHECK-LABEL: func @one_cluster_with_three_ops
+func @one_cluster_with_three_ops() {
+  // CHECK: "tf.opA"
+  // CHECK: "tf.opB"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER3:[a-zA-Z_0-9]+]]"
+  // CHECK: "tf.opC"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER3]]"
+  // CHECK: "tf.opD"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER3]]"
+  // CHECK: "tf.opE"
+  "tf_device.cluster"() ( {
+    %a = "tf.opA"() : () -> tensor<i32>
+    %b = "tf.opB"(%a) {_xla_outside_compilation = "0"} : (tensor<i32>) -> tensor<i32>
+    %c = "tf.opC"(%b) {_xla_outside_compilation = "0"} : (tensor<i32>) -> tensor<i32>
+    %d = "tf.opD"(%b, %c) {_xla_outside_compilation = "0"} : (tensor<i32>, tensor<i32>) -> tensor<i32>
+    "tf.opE"(%d) : (tensor<i32>) -> ()
+    tf_device.return
+  }) {cluster_attr = "cluster_attr"} : () -> ()
+  return
+}
+
+// CHECK-LABEL: func @two_clusters_no_dependencies
+func @two_clusters_no_dependencies() {
+  // CHECK: "tf.opA"
+  // CHECK: "tf.opB"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER4:[a-zA-Z_0-9]+]]"
+  // CHECK: "tf.opC"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER4]]"
+  // CHECK: "tf.opD"
+  "tf_device.cluster"() ( {
+    "tf.opA"() : () -> ()
+    "tf.opB"() {_xla_outside_compilation = "0"} : () -> ()
+    "tf.opC"() {_xla_outside_compilation = "0"} : () -> ()
+    "tf.opD"() : () -> ()
+    tf_device.return
+  }) {cluster_attr = "cluster_attr"} : () -> ()
+  return
+}
+
+// CHECK-LABEL: func @two_clusters_with_one_op_each
+func @two_clusters_with_one_op_each() {
+  // CHECK: "tf.opA"
+  // CHECK-NEXT: "tf.opB"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER6:[a-zA-Z_0-9]+]]"
+  // CHECK-NEXT: "tf.opC"
+  // CHECK-NEXT: "tf.opD"
+  // CHECK-NOT: _xla_outside_compilation = "[[CLUSTER6]]"
+  // CHECK-SAME: _xla_outside_compilation = "{{[a-zA-Z_0-9]+}}"
+  // CHECK-NEXT: "tf.opE"
+  "tf_device.cluster"() ( {
+    %a = "tf.opA"() : () -> tensor<i32>
+    %b = "tf.opB"(%a) {_xla_outside_compilation = "0"} : (tensor<i32>) -> tensor<i32>
+    %c = "tf.opC"(%b) : (tensor<i32>) -> tensor<i32>
+    %d = "tf.opD"(%c) {_xla_outside_compilation = "0"} : (tensor<i32>) -> tensor<i32>
+    "tf.opE"(%d) : (tensor<i32>) -> ()
+    tf_device.return
+  }) {cluster_attr = "cluster_attr"} : () -> ()
+  return
+}
+
+// CHECK-LABEL: func @two_clusters_with_two_ops_each
+func @two_clusters_with_two_ops_each() {
+  // CHECK: "tf.opA"
+  // CHECK-NEXT: "tf.opB"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER8:[a-zA-Z_0-9]+]]"
+  // CHECK-NEXT: "tf.opC"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER8]]"
+  // CHECK-NEXT: "tf.opD"
+  // CHECK-NEXT: "tf.opE"
+  // CHECK-NOT: _xla_outside_compilation = "[[CLUSTER8]]"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER9:[a-zA-Z_0-9]+]]"
+  // CHECK-NEXT: "tf.opF"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER9]]"
+  // CHECK-NEXT: "tf.opG"
+  "tf_device.cluster"() ( {
+    %a = "tf.opA"() : () -> tensor<i32>
+    %b = "tf.opB"(%a) {_xla_outside_compilation = "0"} : (tensor<i32>) -> tensor<i32>
+    %c = "tf.opC"(%b) {_xla_outside_compilation = "0"} : (tensor<i32>) -> tensor<i32>
+    %d = "tf.opD"(%c) : (tensor<i32>) -> tensor<i32>
+    %e = "tf.opE"(%d) {_xla_outside_compilation = "0"} : (tensor<i32>) -> tensor<i32>
+    %f = "tf.opF"(%e) {_xla_outside_compilation = "0"} : (tensor<i32>) -> tensor<i32>
+    "tf.opG"(%f) : (tensor<i32>) -> ()
+    tf_device.return
+  }) {cluster_attr = "cluster_attr"} : () -> ()
+  return
+}
+
+// CHECK-LABEL: func @two_clusters_with_same_parent
+func @two_clusters_with_same_parent() {
+  // CHECK: "tf.opA"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER10:[a-zA-Z_0-9]+]]"
+  // CHECK-NEXT: "tf.opB"
+  // CHECK-NEXT: "tf.opC"
+  // CHECK-NOT: _xla_outside_compilation = "[[CLUSTER10]]"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER11:[a-zA-Z_0-9]+]]"
+  // CHECK-NEXT: "tf.opD"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER10]]"
+  // CHECK-NEXT: "tf.opE"
+  // CHECK-NEXT: "tf.opF"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER11]]"
+  // CHECK-NEXT: "tf.opG"
+  "tf_device.cluster"() ( {
+    %a = "tf.opA"() {_xla_outside_compilation = "0"} : () -> tensor<i32>
+    %b = "tf.opB"(%a) : (tensor<i32>) -> tensor<i32>
+    %c = "tf.opC"(%b) {_xla_outside_compilation = "0"} : (tensor<i32>) -> tensor<i32>
+    %d = "tf.opD"() {_xla_outside_compilation = "0"} : () -> tensor<i32>
+    %e = "tf.opE"(%d) : (tensor<i32>) -> tensor<i32>
+    %f = "tf.opF"(%e) {_xla_outside_compilation = "0"} : (tensor<i32>) -> tensor<i32>
+    %g = "tf.opG"(%c, %f) : (tensor<i32>, tensor<i32>) -> tensor<i32>
+    tf_device.return
+  }) {cluster_attr = "cluster_attr"} : () -> ()
+  return
+}
+
+// CHECK-LABEL: func @two_clusters_with_same_outside_compiled_parent
+func @two_clusters_with_same_outside_compiled_parent() {
+  // CHECK: "tf.opA"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER12:[a-zA-Z_0-9]+]]"
+  // CHECK-NEXT: "tf.opB"
+  // CHECK-NEXT: "tf.opC"
+  // CHECK-NOT: _xla_outside_compilation = "[[CLUSTER12]]"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER13:[a-zA-Z_0-9]+]]"
+  // CHECK-NEXT: "tf.opD"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER12]]"
+  // CHECK-NEXT: "tf.opE"
+  // CHECK-NEXT: "tf.opF"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER13]]"
+  // CHECK-NEXT: "tf.opG"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER13]]"
+  "tf_device.cluster"() ( {
+    %a = "tf.opA"() {_xla_outside_compilation = "0"} : () -> tensor<i32>
+    %b = "tf.opB"(%a) : (tensor<i32>) -> tensor<i32>
+    %c = "tf.opC"(%b) {_xla_outside_compilation = "0"} : (tensor<i32>) -> tensor<i32>
+    %d = "tf.opD"() {_xla_outside_compilation = "0"} : () -> tensor<i32>
+    %e = "tf.opE"(%d) : (tensor<i32>) -> tensor<i32>
+    %f = "tf.opF"(%e) {_xla_outside_compilation = "0"} : (tensor<i32>) -> tensor<i32>
+    %g = "tf.opG"(%c, %f) {_xla_outside_compilation = "0"} : (tensor<i32>, tensor<i32>) -> tensor<i32>
+    tf_device.return
+  }) {cluster_attr = "cluster_attr"} : () -> ()
+  return
+}
+
+// CHECK-LABEL: func @parent_with_a_non_outside_compiled_child
+func @parent_with_a_non_outside_compiled_child() {
+  // CHECK: "tf.opA"
+  // CHECK-NEXT: "tf.opB"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER14:[a-zA-Z_0-9]+]]"
+  // CHECK-NEXT: "tf.opC"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER14]]"
+  "tf_device.cluster"() ( {
+    %a = "tf.opA"() : () -> tensor<i32>
+    %b = "tf.opB"() {_xla_outside_compilation = "0"} : () -> tensor<i32>
+    %c = "tf.opC"(%a, %b) {_xla_outside_compilation = "0"} : (tensor<i32>, tensor<i32>) -> tensor<i32>
+    tf_device.return
+  }) {cluster_attr = "cluster_attr"} : () -> ()
+  return
+}
+
+// CHECK-LABEL: func @outside_compile_with_block
+func @outside_compile_with_block() {
+  // CHECK: "tf.opA"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER15:[a-zA-Z_0-9]+]]"
+  // CHECK-NEXT: "tf.opB"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER15]]"
+  // CHECK: "tf.opC"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER15]]"
+  "tf_device.cluster"() ( {
+    %a = "tf.opA"() {_xla_outside_compilation = "0"} : () -> tensor<i32>
+    %b = "tf.opB"() {_xla_outside_compilation = "0"} : () -> tensor<i32>
+    "tf_device.cluster" () ( {
+      tf_device.return
+    }) {cluster_attr = "cluster_attr"} : () -> ()
+    %c = "tf.opC"() {_xla_outside_compilation = "0"} : () -> tensor<i32>
+    tf_device.return
+  }) {cluster_attr = "cluster_attr"} : () -> ()
+  return
+}
+
+// CHECK-LABEL: func @two_clusters_with_one_op_each_with_indirect_dependency
+func @two_clusters_with_one_op_each_with_indirect_dependency() {
+  // CHECK: "tf.opA"
+  // CHECK-NEXT: "tf.opB"
+  // CHECK-SAME: _xla_outside_compilation = "[[CLUSTER16:[a-zA-Z_0-9]+]]"
+  // CHECK-NEXT: "tf.opC"
+  // CHECK-NEXT: "tf.opD"
+  // CHECK-NEXT: "tf.opE"
+  // CHECK-NOT: _xla_outside_compilation = "[[CLUSTER16]]"
+  // CHECK-SAME: _xla_outside_compilation = "{{[a-zA-Z_0-9]+}}"
+  // CHECK-NEXT: "tf.opF"
+  "tf_device.cluster"() ( {
+    %a = "tf.opA"() : () -> tensor<i32>
+    %b = "tf.opB"(%a) {_xla_outside_compilation = "0"} : (tensor<i32>) -> tensor<i32>
+    %c = "tf.opC"(%b) : (tensor<i32>) -> tensor<i32>
+    %d = "tf.opD"(%c) : (tensor<i32>) -> tensor<i32>
+    %e = "tf.opE"(%d) {_xla_outside_compilation = "0"} : (tensor<i32>) -> tensor<i32>
+    "tf.opF"(%e) : (tensor<i32>) -> ()
+    tf_device.return
+  }) {cluster_attr = "cluster_attr"} : () -> ()
+  return
+}
diff --git a/tensorflow/compiler/mlir/tensorflow/transforms/annotate_parameter_replication.cc b/tensorflow/compiler/mlir/tensorflow/transforms/annotate_parameter_replication.cc
index fb3ecfde771..6ba6f416c70 100644
--- a/tensorflow/compiler/mlir/tensorflow/transforms/annotate_parameter_replication.cc
+++ b/tensorflow/compiler/mlir/tensorflow/transforms/annotate_parameter_replication.cc
@@ -33,7 +33,7 @@ namespace TFDevice {
 
 namespace {
 
-constexpr char kReplicationAttr[] = "tf_device.is_same_data_across_replicas";
+constexpr char kReplicationAttr[] = "xla_hlo.is_same_data_across_replicas";
 constexpr char kMirroredVariableIndicesAttr[] = "_mirrored_variable_indices";
 
 // Analyzes the inputs to ClusterFuncOps in the module, and annotates their
@@ -83,8 +83,7 @@ void AnnotateParameterReplication::runOnOperation() {
         // Not a replication-invariant operand.
         continue;
       }
-      func.setArgAttr(entry.index(), kReplicationAttr,
-                      builder.getBoolAttr(true));
+      func.setArgAttr(entry.index(), kReplicationAttr, builder.getUnitAttr());
     }
   });
 }
diff --git a/tensorflow/compiler/mlir/tensorflow/transforms/canonicalize.td b/tensorflow/compiler/mlir/tensorflow/transforms/canonicalize.td
index cf09f8d64fb..43661902138 100644
--- a/tensorflow/compiler/mlir/tensorflow/transforms/canonicalize.td
+++ b/tensorflow/compiler/mlir/tensorflow/transforms/canonicalize.td
@@ -65,6 +65,12 @@ def BatchMatMulV2ToMatMul : Pat<(TF_BatchMatMulV2Op $x, $y, $adj_x, $adj_y),
                               (TF_MatMulOp $x, $y, $adj_x, $adj_y),
                               [(IsRank2Tensor $x), (IsRank2Tensor $y)]>;
 
+//===----------------------------------------------------------------------===//
+// BiasAddV1 op patterns.
+//===----------------------------------------------------------------------===//
+
+def BiasAddV1ToBiasAdd : Pat<(TF_BiasAddV1Op $arg0, $arg1),
+  (TF_BiasAddOp $arg0, $arg1, ConstantAttr<TF_ConvnetDataFormatAttr, "NHWC">)>;
 
 //===----------------------------------------------------------------------===//
 // Bitcast op patterns.
diff --git a/tensorflow/compiler/mlir/tensorflow/transforms/legalize_hlo.cc b/tensorflow/compiler/mlir/tensorflow/transforms/legalize_hlo.cc
index d635d605607..f6c00e8cb82 100644
--- a/tensorflow/compiler/mlir/tensorflow/transforms/legalize_hlo.cc
+++ b/tensorflow/compiler/mlir/tensorflow/transforms/legalize_hlo.cc
@@ -18,6 +18,7 @@ limitations under the License.
 #include <memory>
 #include <vector>
 
+#include "llvm/ADT/STLExtras.h"
 #include "llvm/ADT/SmallVector.h"
 #include "llvm/Support/raw_ostream.h"
 #include "mlir/Dialect/StandardOps/IR/Ops.h"  // from @llvm-project
@@ -159,6 +160,44 @@ Value ConvertDotOp(PatternRewriter &rewriter, Operation *old_op) {
   return reshape.getResult();
 }
 
+// Returns true if broadcast_dimensions obey Tensorflow convention, as in new
+// dimensions are added as prefix.
+bool IsTFStyleBroadcast(DenseIntElementsAttr broadcast_dimensions,
+                        Value output) {
+  // broadcast_dimensions is an increasing list by definition, thus it suffices
+  // to check the first element.
+  int64_t input_rank = broadcast_dimensions.getNumElements();
+  int64_t output_rank = output.getType().cast<ShapedType>().getRank();
+  return input_rank == 0 ||
+         (broadcast_dimensions.getValue({0}).cast<IntegerAttr>().getInt() ==
+          output_rank - input_rank);
+}
+
+// Returns the intermediate shape that input tensor should be reshaped to during
+// legalization of BroadcastInDimOp.
+ConstantOp ExpandedShape(PatternRewriter &rewriter, Value input,
+                         DenseIntElementsAttr broadcast_dimensions,
+                         Value output) {
+  // Initialize expanded shape with output rank and dimensions of 1.
+  SmallVector<Attribute, 4> expanded_shape(
+      output.getType().cast<ShapedType>().getRank(),
+      /*Value=*/rewriter.getI64IntegerAttr(1));
+
+  // Set dimension sizes specified by broadcast_dimensions.
+  ArrayRef<int64_t> input_shape = input.getType().cast<ShapedType>().getShape();
+  for (auto x : llvm::enumerate(broadcast_dimensions)) {
+    expanded_shape[x.value().getSExtValue()] =
+        rewriter.getI64IntegerAttr(input_shape[x.index()]);
+  }
+
+  // Create the expanded type wrapped in a ConstantOp.
+  auto attr_type =
+      RankedTensorType::get({static_cast<int64_t>(expanded_shape.size())},
+                            rewriter.getIntegerType(64));
+  auto attr = DenseElementsAttr::get(attr_type, expanded_shape);
+  return rewriter.create<ConstantOp>(output.getLoc(), attr_type, attr);
+}
+
 #include "tensorflow/compiler/mlir/tensorflow/transforms/generated_legalize_hlo.inc"
 
 /// Performs the lowering to XLA dialect.
diff --git a/tensorflow/compiler/mlir/tensorflow/transforms/legalize_hlo_patterns.td b/tensorflow/compiler/mlir/tensorflow/transforms/legalize_hlo_patterns.td
index e4b6a28d65f..df78aa97f01 100644
--- a/tensorflow/compiler/mlir/tensorflow/transforms/legalize_hlo_patterns.td
+++ b/tensorflow/compiler/mlir/tensorflow/transforms/legalize_hlo_patterns.td
@@ -29,6 +29,17 @@ def AreBroadcastCompatible : Constraint<CPred<"AreBroadcastCompatible($0, $1)">,
 // Return a constant op that carries the shape of the given value.
 def ShapeToConst : NativeCodeCall<"ShapeToConst($_builder, $0)">;
 
+// Check if broadcast dimensions match Tensorflow convention.
+def IsTFStyleBroadcast : Constraint<CPred<"IsTFStyleBroadcast($0, $1)">,
+    "new dimensions are added as prefix">;
+
+// Check if broadcast dimensions do not match Tensorflow convention.
+def IsNotTFStyleBroadcast : Constraint<Neg<CPred<"IsTFStyleBroadcast($0, $1)">>,
+    "new dimensions are inserted in intermediate positions">;
+
+// Return intermediate shape before broadcasting, wrapped in a constant op.
+def ExpandedShape : NativeCodeCall<"ExpandedShape($_builder, $0, $1, $2)">;
+
 def : Pat<(HLO_ConstOp $value), (TF_ConstOp $value)>;
 
 //===----------------------------------------------------------------------===//
@@ -38,6 +49,7 @@ def : Pat<(HLO_ConstOp $value), (TF_ConstOp $value)>;
 // context, getting to these ops may require some raising.
 //===----------------------------------------------------------------------===//
 
+// TODO(b/158025719): Properly handle broadcast_dimensions.
 foreach fromToBinPair = [[HLO_AddOp, HLOClient_BroadcastAddOp, TF_AddV2Op],
                          [HLO_DivOp, HLOClient_BroadcastDivOp, TF_DivOp],
                          [HLO_ShiftLeftOp, HLOClient_BroadcastShiftLeftOp, TF_LeftShiftOp],
@@ -80,6 +92,7 @@ def : Pat<(HLO_FloorOp (HLOClient_BroadcastDivOp $l, $r, $_)), (TF_FloorDivOp $l
           [(AreBroadcastCompatible $l, $r)]>;
 
 def : Pat<(HLO_ComplexOp $r, $i), (TF_ComplexOp $r, $i)>;
+
 //===----------------------------------------------------------------------===//
 // Unary op patterns.
 //===----------------------------------------------------------------------===//
@@ -112,6 +125,16 @@ def : Pat<(HLO_AbsOp TF_ComplexTensor:$arg), (TF_ComplexAbsOp $arg)>;
 
 def : Pat<(HLO_BroadcastOp $arg, $shape),
           (TF_BroadcastToOp $arg, (TF_ConstOp $shape))>;
+def : Pat<(HLO_BroadcastInDimOp:$output $input, $broadcast_dimensions),
+          (TF_BroadcastToOp $input, (ShapeToConst $output)),
+          [(IsTFStyleBroadcast $broadcast_dimensions, $output)]>;
+def : Pat<(HLO_BroadcastInDimOp:$output $input, $broadcast_dimensions),
+          (TF_BroadcastToOp
+              (TF_ReshapeOp
+                  $input,
+                  (ExpandedShape $input, $broadcast_dimensions, $output)),
+              (ShapeToConst $output)),
+          [(IsNotTFStyleBroadcast $broadcast_dimensions, $output)]>;
 def : Pat<(HLO_TransposeOp $arg, $permutation),
           (TF_TransposeOp $arg, (TF_ConstOp $permutation))>;
 def : Pat<(HLO_ReverseOp $op, $dims), (TF_ReverseV2Op $op, (TF_ConstOp $dims))>;
diff --git a/tensorflow/compiler/mlir/tensorflow/transforms/op_fusion.cc b/tensorflow/compiler/mlir/tensorflow/transforms/op_fusion.cc
new file mode 100644
index 00000000000..202783d1cc8
--- /dev/null
+++ b/tensorflow/compiler/mlir/tensorflow/transforms/op_fusion.cc
@@ -0,0 +1,174 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include <cstdio>
+#include <iostream>
+
+#include "llvm/ADT/StringRef.h"
+#include "mlir/IR/Attributes.h"  // from @llvm-project
+#include "mlir/IR/Diagnostics.h"  // from @llvm-project
+#include "mlir/IR/Function.h"  // from @llvm-project
+#include "mlir/IR/PatternMatch.h"  // from @llvm-project
+#include "mlir/Pass/Pass.h"  // from @llvm-project
+#include "mlir/Support/LLVM.h"  // from @llvm-project
+#include "mlir/Support/LogicalResult.h"  // from @llvm-project
+#include "tensorflow/compiler/mlir/tensorflow/ir/tf_ops.h"
+
+namespace mlir {
+
+namespace TF {
+
+namespace {
+
+// Note: This implements the fusions performed in the old Remapper Grappler
+// pass. That pass has specific cases for GPU and based on different
+// target configurations on both CPU and GPU (Intel MKL, ROCm, etc.). This MLIR
+// pass covers the general CPU case and at the moment does not account for any
+// specific target configurations.
+// TODO(b/158265178): Support GPU-specific fusions.
+// TODO(b/158266710): Support CPU MKL configurations.
+
+// Optimizes TF computations by fusing subgraphs/nodes onto more efficient
+// implementations to decrease the number of operations needed to perform a
+// computation.
+struct OpFusionPass : public PassWrapper<OpFusionPass, FunctionPass> {
+  void runOnFunction() override;
+};
+
+// Returns an op's name with the dialect prefix stripped off.
+StringRef GetOpNameWithoutDialect(Operation *op) {
+  return op->getName().getStringRef().split(".").second;
+}
+
+bool IsActivationFunction(Operation *op) {
+  return isa<EluOp>(op) || isa<ReluOp>(op) || isa<Relu6Op>(op);
+}
+
+// Finds and returns an activation op that uses the result of `op`. If there are
+// multiple such activations, one is returned (with no guarantee as to which
+// one). If there are no activation functions that use the output, returns
+// nullptr.
+Operation *GetActivation(Value op) {
+  for (auto &use : op.getUses()) {
+    if (IsActivationFunction(use.getOwner())) return use.getOwner();
+  }
+  return nullptr;
+}
+
+// Finds and returns a BiasAdd that uses the result of `op` as the `value`
+// input. If there are multiple such BiasAdds, one is returned (with no
+// guarantee as to which one). If there are no BiasAdds that use the output,
+// returns a null BiasAddOp.
+BiasAddOp GetBiasAdd(Value op) {
+  for (auto &use : op.getUses()) {
+    auto bias_add = dyn_cast_or_null<BiasAddOp>(use.getOwner());
+    // If it's a BiasAdd, check that the conv op is the first input.
+    if (bias_add && bias_add.value() == op) return bias_add;
+  }
+  // No BiasAddOps found among uses.
+  return BiasAddOp();
+}
+
+// Performs a fusion of the following pattern(s), if possible:
+//   Conv2D + BiasAdd + <Activation> -> _FusedConv2D
+//
+// Note that fusion with activation is preferred, but a Conv2D and BiasAdd can
+// also be replaced by a _FusedConv2D if there is no other activation function.
+// i.e., this class also supports the following fusion:
+//   Conv2D + BiasAdd -> _FusedConv2D
+//
+// TODO(b/158266331): Support fusing Conv2D + BiasAdd + a chain of activations.
+class FuseConv2DBiasAdd : public OpRewritePattern<Conv2DOp> {
+ public:
+  using OpRewritePattern::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(Conv2DOp op,
+                                PatternRewriter &rewriter) const override {
+    // If the convolution is used in multiple places, fusing it will only create
+    // more convolutions, which is slower.
+    if (!op.getResult().hasOneUse())
+      return rewriter.notifyMatchFailure(op, "result is used by multiple ops");
+
+    BiasAddOp bias_add = GetBiasAdd(op);
+    if (!bias_add) {
+      return rewriter.notifyMatchFailure(
+          op, "does not feed into a tf.BiasAdd/tf.BiasAddV1 op");
+    }
+
+    // Check that Conv and BiasAdd formats match.
+    if (op.data_format() != bias_add.data_format()) {
+      return rewriter.notifyMatchFailure(op, [&](Diagnostic &diag) {
+        diag << "data format does not match Conv2D data format ("
+             << bias_add.data_format() << " vs " << op.data_format() << ")";
+      });
+    }
+
+    SmallVector<Location, 3> locations{op.getLoc(), bias_add.getLoc()};
+    SmallVector<Attribute, 2> fused_ops{StringAttr::get(
+        GetOpNameWithoutDialect(bias_add), rewriter.getContext())};
+    Type result_type;
+
+    // BiasAdd may or may not feed into an activation function.
+    auto activation = GetActivation(bias_add);
+
+    // If there is an activation, only fuse it if this is the only op to use the
+    // result of the BiasAdd.
+    bool fuse_activation = activation && bias_add.output().hasOneUse();
+
+    // Include info about the activation function if applicable.
+    if (fuse_activation) {
+      locations.push_back(activation->getLoc());
+      fused_ops.push_back(StringAttr::get(GetOpNameWithoutDialect(activation),
+                                          rewriter.getContext()));
+      result_type = activation->getResultTypes().front();
+    } else {
+      result_type = bias_add.getResult().getType();
+    }
+
+    auto loc = rewriter.getFusedLoc(locations);
+    ArrayAttr fused_ops_attr = ArrayAttr::get(fused_ops, rewriter.getContext());
+    // Epsilon is used only in fusions with the BatchNorm op.
+    APFloat epsilon = APFloat(0.0f);
+    auto fused_op = rewriter.create<_FusedConv2DOp>(
+        loc, result_type, op.input(), op.filter(), bias_add.bias(),
+        op.strides(), op.padding(), op.explicit_paddings(), op.data_format(),
+        op.dilations(), op.use_cudnn_on_gpu(), fused_ops_attr, epsilon);
+    auto op_to_replace = fuse_activation ? activation : bias_add;
+    rewriter.replaceOp(op_to_replace, {fused_op});
+    return success();
+  }
+};
+
+void OpFusionPass::runOnFunction() {
+  OwningRewritePatternList patterns;
+  auto func = getFunction();
+  patterns.insert<FuseConv2DBiasAdd>(&getContext());
+
+  applyPatternsAndFoldGreedily(func, patterns);
+}
+
+}  // namespace
+
+std::unique_ptr<OperationPass<FuncOp>> CreateOpFusionPass() {
+  return std::make_unique<OpFusionPass>();
+}
+
+static PassRegistration<OpFusionPass> pass(
+    "op-fusion",
+    "Replaces commonly occurring subgraphs with optimized fused kernels");
+
+}  // namespace TF
+
+}  // namespace mlir
diff --git a/tensorflow/compiler/mlir/tensorflow/transforms/passes.h b/tensorflow/compiler/mlir/tensorflow/transforms/passes.h
index 93d7af96c1e..3973eb60707 100644
--- a/tensorflow/compiler/mlir/tensorflow/transforms/passes.h
+++ b/tensorflow/compiler/mlir/tensorflow/transforms/passes.h
@@ -52,6 +52,10 @@ std::unique_ptr<OperationPass<FuncOp>> CreateBatchMatMulToEinsumPass();
 // Optimizes Tensorflow graph.
 std::unique_ptr<OperationPass<FuncOp>> CreateTFOptimizePass();
 
+// Creates pass to rewrite RecvTPUEmbeddingActivationsOp and
+// SendTPUEmbeddingGradients ops to internal variants.
+std::unique_ptr<OperationPass<FuncOp>> CreateRewriteTPUEmbeddingOps();
+
 // Performs specific fusion for GPU targets.
 std::unique_ptr<OperationPass<FuncOp>> CreateGpuOpFusionPass();
 
@@ -142,6 +146,9 @@ CreateTensorArrayOpsDecompositionPass();
 
 // Create a pass that legalize HLO to TF dialect.
 std::unique_ptr<OperationPass<FuncOp>> CreateLegalizeHloToTfPass();
+
+// Creates a pass that performs fusion of common sequences of ops.
+std::unique_ptr<OperationPass<FuncOp>> CreateOpFusionPass();
 }  // namespace TF
 
 namespace TFControlFlow {
@@ -267,6 +274,10 @@ std::unique_ptr<OperationPass<FuncOp>> CreateTPUMergeVariablesWithExecutePass();
 // run-time according to compilation result.
 std::unique_ptr<OperationPass<ModuleOp>> CreateTPUVariableReformattingPass();
 
+// Creates a pass that groups outside compiled operations (CPU ops inside TPU
+// cluster) into clusters that can be extracted and run on the CPU.
+std::unique_ptr<OperationPass<FuncOp>> CreateTPUOutsideCompilationClusterPass();
+
 // Creates a pass that extracts outside compilation (CPU ops inside TPU cluster)
 // at head/tail of TPU cluster to run before/after TPU computation.
 std::unique_ptr<OperationPass<ModuleOp>>
diff --git a/tensorflow/compiler/mlir/tensorflow/transforms/rewrite_tpu_embedding_ops.cc b/tensorflow/compiler/mlir/tensorflow/transforms/rewrite_tpu_embedding_ops.cc
new file mode 100644
index 00000000000..527d249d937
--- /dev/null
+++ b/tensorflow/compiler/mlir/tensorflow/transforms/rewrite_tpu_embedding_ops.cc
@@ -0,0 +1,114 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include "llvm/ADT/SmallVector.h"
+#include "mlir/IR/Attributes.h"  // from @llvm-project
+#include "mlir/IR/StandardTypes.h"  // from @llvm-project
+#include "mlir/Pass/Pass.h"  // from @llvm-project
+#include "mlir/Pass/PassRegistry.h"  // from @llvm-project
+#include "tensorflow/compiler/mlir/tensorflow/ir/tf_ops.h"
+
+namespace mlir {
+namespace TF {
+
+namespace {
+
+// Rewrites RecvTPUEmbeddingActivationsOp and SendTPUEmbeddingGradients ops to
+// internal variants by introducing _RecvTPUEmbeddingDeduplicationData op.
+struct RewriteTPUEmbeddingOps
+    : public PassWrapper<RewriteTPUEmbeddingOps, FunctionPass> {
+  void runOnFunction() override;
+};
+
+// Rewrites the given op to `OpT` op after adding the given operand at the end.
+template <typename OpT>
+OpT AddOperandAndRewriteAs(Operation* op, Value operand, OpBuilder* builder) {
+  builder->setInsertionPoint(op);
+  auto operands = llvm::to_vector<4>(op->getOperands());
+  operands.push_back(operand);
+  auto new_op = builder->create<OpT>(op->getLoc(), op->getResultTypes(),
+                                     operands, op->getAttrs());
+  op->replaceAllUsesWith(new_op.getOperation()->getResults());
+  op->erase();
+  return new_op;
+}
+
+// Returns success if the function has at most one op of the template type and
+// assigns it to `result`, if present. If there are multiple such ops, returns
+// failure.
+template <typename OpT>
+LogicalResult GetOp(FuncOp func, OpT* result) {
+  *result = {};
+  for (auto op : func.getOps<OpT>()) {
+    if (*result) return op.emitError("should be unique within a function");
+    *result = op;
+  }
+  return success();
+}
+
+void RewriteTPUEmbeddingOps::runOnFunction() {
+  FuncOp func = getFunction();
+
+  RecvTPUEmbeddingActivationsOp recv_op;
+  if (failed(GetOp(func, &recv_op))) return signalPassFailure();
+
+  SendTPUEmbeddingGradientsOp send_op;
+  if (failed(GetOp(func, &send_op))) return signalPassFailure();
+
+  // No TPU embedding ops.
+  if (!recv_op && !send_op) return;
+
+  Location loc = recv_op ? recv_op.getLoc() : send_op.getLoc();
+  StringRef config = recv_op ? recv_op.config() : send_op.config();
+
+  // Create _RecvTPUEmbeddingDeduplicationData op.
+  OpBuilder builder(func.getBody());
+  auto output_ty = RankedTensorType::get({}, VariantType::get(&getContext()));
+  auto dedup_op = builder.create<_RecvTPUEmbeddingDeduplicationDataOp>(
+      loc, output_ty, config);
+
+  // Rewrite RecvTPUEmbeddingActivations op to the corresponding internal op.
+  if (recv_op)
+    AddOperandAndRewriteAs<_RecvTPUEmbeddingActivationsOp>(recv_op, dedup_op,
+                                                           &builder);
+
+  // Rewrite SendTPUEmbeddingGradients op to the corresponding internal op and
+  // then update the OperandSegmentSize attribute.
+  if (send_op) {
+    int32_t operand_sizes[] = {static_cast<int32_t>(send_op.N()),
+                               static_cast<int32_t>(send_op.NN()), 1};
+    auto attr_ty = VectorType::get(3, builder.getI32Type());
+    auto operand_size_attr = DenseIntElementsAttr::get(attr_ty, operand_sizes);
+
+    auto new_send_op = AddOperandAndRewriteAs<_SendTPUEmbeddingGradientsOp>(
+        send_op, dedup_op, &builder);
+    new_send_op.setAttr(new_send_op.getOperandSegmentSizeAttr(),
+                        operand_size_attr);
+  }
+}
+
+}  // anonymous namespace
+
+std::unique_ptr<OperationPass<FuncOp>> CreateRewriteTPUEmbeddingOps() {
+  return std::make_unique<RewriteTPUEmbeddingOps>();
+}
+
+static PassRegistration<RewriteTPUEmbeddingOps> pass(
+    "tf-rewrite-tpu-embedding-ops",
+    "Rewrites TPU embedding send/recv ops by adding TPU embedding "
+    "deduplication data");
+
+}  // namespace TF
+}  // namespace mlir
diff --git a/tensorflow/compiler/mlir/tensorflow/transforms/tpu_outside_compilation_cluster.cc b/tensorflow/compiler/mlir/tensorflow/transforms/tpu_outside_compilation_cluster.cc
new file mode 100644
index 00000000000..be01b7644ea
--- /dev/null
+++ b/tensorflow/compiler/mlir/tensorflow/transforms/tpu_outside_compilation_cluster.cc
@@ -0,0 +1,131 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include "llvm/ADT/SmallSet.h"
+#include "llvm/ADT/SmallVector.h"
+#include "llvm/ADT/StringRef.h"
+#include "llvm/Support/FormatVariadic.h"
+#include "mlir/IR/Attributes.h"  // from @llvm-project
+#include "mlir/IR/Operation.h"  // from @llvm-project
+#include "mlir/IR/Types.h"  // from @llvm-project
+#include "mlir/Support/LLVM.h"  // from @llvm-project
+#include "mlir/Support/LogicalResult.h"  // from @llvm-project
+#include "tensorflow/compiler/mlir/tensorflow/ir/tf_device.h"
+#include "tensorflow/compiler/mlir/tensorflow/transforms/passes.h"
+
+namespace mlir {
+namespace TFTPU {
+
+namespace {
+
+constexpr char kXlaOutsideCompilationAttr[] = "_xla_outside_compilation";
+
+struct TPUOutsideCompilationCluster
+    : public PassWrapper<TPUOutsideCompilationCluster, FunctionPass> {
+  void runOnFunction() override;
+};
+
+// Represents an outside compiled cluster. All ops that are added to the same
+// cluster will be extracted together in a later pass.
+class OutsideCompiledCluster {
+ public:
+  explicit OutsideCompiledCluster(int number)
+      : cluster_name_(llvm::formatv("cluster{0}", number).str()) {}
+
+  // Attempts to add an op to this cluster.
+  // This function requires all ops to be added before their uses.
+  bool AddOp(Operation* op) {
+    // Check if the op is safe to add before adding it.
+    bool add = IsSafeToAdd(op);
+    if (add) {
+      // Set the ops kXlaOutsideCompilationAttr to the cluster name.
+      op->setAttr(kXlaOutsideCompilationAttr,
+                  StringAttr::get(cluster_name_, op->getContext()));
+
+      // Since we are adding the op to the cluster, the op is no longer
+      // considered a user of this cluster.
+      users_.erase(op);
+    }
+
+    // Add this op's users to the cluster users.
+    users_.insert(op->user_begin(), op->user_end());
+    return add;
+  }
+
+ private:
+  // Checks if it is safe for an op to be merged into this cluster.
+  bool IsSafeToAdd(Operation* op) {
+    // If the op is not marked for outside compilation it doesn't belong in a
+    // cluster.
+    if (!op->getAttrOfType<StringAttr>(kXlaOutsideCompilationAttr))
+      return false;
+
+    // Checks to see if the op's operands are related to this
+    // clusters users. If they are related, then there is an op between this
+    // op and the cluster. Since ops are added before their uses, there
+    // is no way for the op in-between to ever be added to this cluster
+    // therefore there is no way this op can ever be added to the cluster.
+    for (const Value& value : op->getOperands()) {
+      Operation* op_operand = value.getDefiningOp();
+      if (op_operand && users_.find(op_operand) != users_.end()) return false;
+    }
+    return true;
+  }
+
+  // users_ stores the direct and indirect users of the outside compiled ops in
+  // this cluster. It does NOT store the outside compiled ops that are a part
+  // of this cluster that will be collectively extracted and run on the cpu.
+  // users_ is consulted when attempting to add a new outside compiled to the
+  // cluster. If the new op's operand(s) are already in users_, it means that
+  // the operand(s) were not added to the cluster so it is not safe to add the
+  // new op to the cluster either.
+  llvm::SmallPtrSet<Operation*, 8> users_;
+  std::string cluster_name_;
+};
+
+void TPUOutsideCompilationCluster::runOnFunction() {
+  llvm::SmallVector<OutsideCompiledCluster, 8> clusters;
+  int cluster_counter = 0;
+
+  getFunction().walk([&](tf_device::ClusterOp tpu_cluster) {
+    for (Operation& op : tpu_cluster.GetBody()) {
+      // Try to add the op to existing clusters.
+      bool added = false;
+      for (auto& cluster : clusters)
+        if ((added = cluster.AddOp(&op))) break;
+
+      // If the op cannot be added to existing clusters, create a new cluster.
+      if (!added) {
+        OutsideCompiledCluster new_cluster(cluster_counter++);
+        new_cluster.AddOp(&op);
+        clusters.push_back(new_cluster);
+      }
+    }
+  });
+}
+
+}  // anonymous namespace
+
+std::unique_ptr<OperationPass<FuncOp>>
+CreateTPUOutsideCompilationClusterPass() {
+  return std::make_unique<TPUOutsideCompilationCluster>();
+}
+
+static PassRegistration<TPUOutsideCompilationCluster> pass(
+    "tf-tpu-outside-compilation-cluster",
+    "Identifies clusters of operations assigned to outside compilation");
+
+}  // namespace TFTPU
+}  // namespace mlir
diff --git a/tensorflow/compiler/mlir/tensorflow/translate/breakup-islands.cc b/tensorflow/compiler/mlir/tensorflow/translate/breakup-islands.cc
index 3245e3b9e6a..7284626c46a 100644
--- a/tensorflow/compiler/mlir/tensorflow/translate/breakup-islands.cc
+++ b/tensorflow/compiler/mlir/tensorflow/translate/breakup-islands.cc
@@ -219,7 +219,7 @@ void BreakUpIslands::BreakUpIsland(
   }
 
   // Skip islands that are already only a single op.
-  if (hasSingleElement(island_body)) return;
+  if (island_op.WrapsSingleOp()) return;
 
   auto control_type = tf_executor::ControlType::get(&getContext());
   auto island_control_inputs = llvm::to_vector<4>(island_op.controlInputs());
diff --git a/tensorflow/compiler/mlir/tensorflow/translate/import_model.cc b/tensorflow/compiler/mlir/tensorflow/translate/import_model.cc
index 3aa700d3718..89188352677 100644
--- a/tensorflow/compiler/mlir/tensorflow/translate/import_model.cc
+++ b/tensorflow/compiler/mlir/tensorflow/translate/import_model.cc
@@ -3572,6 +3572,24 @@ StatusOr<mlir::OwningModuleRef> ConvertGraphToMlir(
                                    /*func_name=*/"main");
 }
 
+stream_executor::port::StatusOr<mlir::OwningModuleRef> ConvertFunctionToMlir(
+    mlir::StringRef name, const FunctionLibraryDefinition& flib_def,
+    mlir::MLIRContext* context) {
+  const tensorflow::FunctionDef* fdef = flib_def.Find(name.str());
+  if (fdef == nullptr)
+    return tensorflow::errors::NotFound("Cannot find function ", name.str());
+
+  std::unique_ptr<tensorflow::FunctionBody> fbody;
+  TF_RETURN_IF_ERROR(FunctionDefToBodyHelper(*fdef, tensorflow::AttrSlice(),
+                                             &flib_def, &fbody));
+
+  tensorflow::GraphDebugInfo dummy_debug_info;
+  tensorflow::GraphImportConfig specs;
+  specs.graph_as_function = true;
+  return GraphDefImporter::Convert(context, *fbody->graph, dummy_debug_info,
+                                   flib_def, specs, name);
+}
+
 StatusOr<mlir::OwningModuleRef> ConvertSavedModelToMlir(
     SavedModelV2Bundle* saved_model, mlir::MLIRContext* context,
     absl::Span<std::string> exported_names, bool add_default_attributes) {
diff --git a/tensorflow/compiler/mlir/tensorflow/translate/import_model.h b/tensorflow/compiler/mlir/tensorflow/translate/import_model.h
index bdb72345201..80001c44389 100644
--- a/tensorflow/compiler/mlir/tensorflow/translate/import_model.h
+++ b/tensorflow/compiler/mlir/tensorflow/translate/import_model.h
@@ -20,6 +20,7 @@ limitations under the License.
 
 #include "mlir/IR/MLIRContext.h"  // from @llvm-project
 #include "mlir/IR/Module.h"  // from @llvm-project
+#include "mlir/Support/LLVM.h"  // from @llvm-project
 #include "tensorflow/cc/saved_model/bundle_v2.h"
 #include "tensorflow/cc/saved_model/loader.h"
 #include "tensorflow/compiler/mlir/tensorflow/translate/mlir_roundtrip_flags.h"
@@ -45,6 +46,13 @@ stream_executor::port::StatusOr<mlir::OwningModuleRef> ConvertGraphToMlir(
     const FunctionLibraryDefinition& flib_def, const GraphImportConfig& specs,
     mlir::MLIRContext* context);
 
+// [Experimental]
+// Given a Function, returns a MLIR module containing the graph, expressed with
+// tf_executor dialect.
+stream_executor::port::StatusOr<mlir::OwningModuleRef> ConvertFunctionToMlir(
+    mlir::StringRef name, const FunctionLibraryDefinition& flib_def,
+    mlir::MLIRContext* context);
+
 // Given a SavedModel, returns a MLIR module containing the functions, expressed
 // with tf_executor dialect.
 stream_executor::port::StatusOr<mlir::OwningModuleRef> ConvertSavedModelToMlir(
diff --git a/tensorflow/compiler/mlir/tensorflow/utils/compile_mlir_util_test.cc b/tensorflow/compiler/mlir/tensorflow/utils/compile_mlir_util_test.cc
index 91640aff437..cc724ba36f8 100644
--- a/tensorflow/compiler/mlir/tensorflow/utils/compile_mlir_util_test.cc
+++ b/tensorflow/compiler/mlir/tensorflow/utils/compile_mlir_util_test.cc
@@ -184,7 +184,7 @@ TEST(CompileSerializedMlirToXlaHloTest, CompileTimeConstantFoldedSuccess) {
   // only be lowered when tf.Shape is folded into a constant.
   constexpr char mlir_module[] = R"(
     module attributes {tf.versions = {producer = 179 : i32}} {
-      func @main(%arg0: tensor<10x19xf32>, %arg1: tensor<19x10xf32> {tf_device.is_same_data_across_replicas = true}) -> tensor<10x19xf32> {
+      func @main(%arg0: tensor<10x19xf32>, %arg1: tensor<19x10xf32> {xla_hlo.is_same_data_across_replicas}) -> tensor<10x19xf32> {
         %0 = "tf.Shape"(%arg0) : (tensor<10x19xf32>) -> tensor<2xi64>
         %1 = "tf.Reshape"(%arg1, %0) : (tensor<19x10xf32>, tensor<2xi64>) -> tensor<10x19xf32>
         return %1 : tensor<10x19xf32>
diff --git a/tensorflow/compiler/mlir/tensorflow/utils/export_utils.cc b/tensorflow/compiler/mlir/tensorflow/utils/export_utils.cc
index 4877cbc4a44..26552087173 100644
--- a/tensorflow/compiler/mlir/tensorflow/utils/export_utils.cc
+++ b/tensorflow/compiler/mlir/tensorflow/utils/export_utils.cc
@@ -379,13 +379,13 @@ Status ConvertAttributes(
         func_call_attrs[string(name)] = value;
         continue;
       }
-      case mlir::StandardAttributes::Bool:
-        TF_RETURN_IF_ERROR(
-            ConvertAttribute(attr.cast<mlir::BoolAttr>(), &value));
-        break;
       case mlir::StandardAttributes::Integer:
-        TF_RETURN_IF_ERROR(
-            ConvertAttribute(attr.cast<mlir::IntegerAttr>(), &value));
+        if (auto boolAttr = attr.dyn_cast<mlir::BoolAttr>()) {
+          TF_RETURN_IF_ERROR(ConvertAttribute(boolAttr, &value));
+        } else {
+          TF_RETURN_IF_ERROR(
+              ConvertAttribute(attr.cast<mlir::IntegerAttr>(), &value));
+        }
         break;
       case mlir::StandardAttributes::Float:
         TF_RETURN_IF_ERROR(
diff --git a/tensorflow/compiler/mlir/tools/kernel_gen/BUILD b/tensorflow/compiler/mlir/tools/kernel_gen/BUILD
index 27a8dbd2809..940a9d21eeb 100644
--- a/tensorflow/compiler/mlir/tools/kernel_gen/BUILD
+++ b/tensorflow/compiler/mlir/tools/kernel_gen/BUILD
@@ -21,6 +21,7 @@ cc_library(
         "@llvm-project//mlir:StandardOps",
         "@llvm-project//mlir:TargetNVVMIR",
         "@llvm-project//mlir:Transforms",
+        "//tensorflow/compiler/mlir/tensorflow",
         "//tensorflow/compiler/mlir/xla:hlo",
         "//tensorflow/compiler/mlir/xla:lhlo",
         "//tensorflow/compiler/mlir/xla:xla_legalize_tf",
diff --git a/tensorflow/compiler/mlir/tools/kernel_gen/cubin_creator.cc b/tensorflow/compiler/mlir/tools/kernel_gen/cubin_creator.cc
index f47485d0214..d1e5c09cf59 100644
--- a/tensorflow/compiler/mlir/tools/kernel_gen/cubin_creator.cc
+++ b/tensorflow/compiler/mlir/tools/kernel_gen/cubin_creator.cc
@@ -33,6 +33,7 @@ limitations under the License.
 #include "mlir/Dialect/GPU/GPUDialect.h"  // from @llvm-project
 #include "mlir/Dialect/LLVMIR/LLVMDialect.h"  // from @llvm-project
 #include "mlir/Dialect/StandardOps/IR/Ops.h"  // from @llvm-project
+#include "mlir/IR/Dialect.h"  // from @llvm-project
 #include "mlir/IR/Function.h"  // from @llvm-project
 #include "mlir/IR/Operation.h"  // from @llvm-project
 #include "mlir/IR/StandardTypes.h"  // from @llvm-project
@@ -42,6 +43,7 @@ limitations under the License.
 #include "mlir/Pass/PassManager.h"  // from @llvm-project
 #include "mlir/Target/NVVMIR.h"  // from @llvm-project
 #include "mlir/Transforms/DialectConversion.h"  // from @llvm-project
+#include "tensorflow/compiler/mlir/tensorflow/ir/tf_ops.h"
 #include "tensorflow/compiler/mlir/xla/ir/hlo_ops.h"
 #include "tensorflow/compiler/mlir/xla/transforms/passes.h"
 #include "tensorflow/compiler/mlir/xla/transforms/rewriters.h"
@@ -216,14 +218,22 @@ Status PropagateStaticShapeKnowledgeToKernel(
   }
   return Status::OK();
 }
+
+void RegisterDialects() {
+  static bool init_once = []() {
+    mlir::registerDialect<mlir::TF::TensorFlowDialect>();
+    return true;
+  }();
+  (void)init_once;
+}
 }  // namespace
 
 StatusOr<std::vector<uint8_t>> tensorflow::kernel_gen::GenerateCubinForTfCode(
     llvm::StringRef tf_code, std::pair<int32_t, int32_t> compute_capability,
     llvm::ArrayRef<uint32_t> tile_sizes, llvm::ArrayRef<uint32_t> same_shape,
     llvm::ArrayRef<uint32_t> unroll_factors) {
+  RegisterDialects();
   mlir::MLIRContext context;
-  context.allowUnregisteredDialects();  // TODO(b/152572127)
   mlir::OwningModuleRef module = mlir::parseSourceString(tf_code, &context);
 
   TF_RETURN_IF_ERROR(LowerTfOpToLhloWithDynamicShapes(module.get()));
diff --git a/tensorflow/compiler/mlir/tools/kernel_gen/tf_to_cubin.cc b/tensorflow/compiler/mlir/tools/kernel_gen/tf_to_cubin.cc
index 8edc567e777..66fcabde0ac 100644
--- a/tensorflow/compiler/mlir/tools/kernel_gen/tf_to_cubin.cc
+++ b/tensorflow/compiler/mlir/tools/kernel_gen/tf_to_cubin.cc
@@ -47,6 +47,7 @@ bool ParseStringList(std::string string_list, std::vector<uint32_t>* result) {
 }  // namespace
 
 int main(int argc, char** argv) {
+  std::string input_file = "foo.mlir";
   std::string output_file = "foo.bin";
   int32_t architecture = 50;
   std::vector<uint32_t> tile_sizes;
@@ -75,6 +76,7 @@ int main(int argc, char** argv) {
   };
 
   std::vector<tensorflow::Flag> flag_list = {
+      tensorflow::Flag("input", &input_file, "input file"),
       tensorflow::Flag("output", &output_file, "output file"),
       tensorflow::Flag("arch", &architecture,
                        "target architecture (e.g. 50 for sm_50)"),
@@ -94,8 +96,16 @@ int main(int argc, char** argv) {
   std::pair<int32_t, int32_t> compute_capability(architecture / 10,
                                                  architecture % 10);
 
+  std::string tf_code;
+  auto read_status = tensorflow::ReadFileToString(tensorflow::Env::Default(),
+                                                  input_file, &tf_code);
+  if (!read_status.ok()) {
+    LOG(ERROR) << read_status;
+    return 1;
+  }
+
   auto cubin = tensorflow::kernel_gen::GenerateCubinForTfCode(
-      argv[1], compute_capability, tile_sizes, same_shape, unroll_factors);
+      tf_code, compute_capability, tile_sizes, same_shape, unroll_factors);
 
   if (!cubin.ok()) {
     LOG(ERROR) << cubin.status();
diff --git a/tensorflow/compiler/mlir/xla/ir/hlo_ops.td b/tensorflow/compiler/mlir/xla/ir/hlo_ops.td
index 97b8e1c1863..c82322b798f 100644
--- a/tensorflow/compiler/mlir/xla/ir/hlo_ops.td
+++ b/tensorflow/compiler/mlir/xla/ir/hlo_ops.td
@@ -634,7 +634,7 @@ def HLO_GetTupleElementOp: HLO_Op<"get_tuple_element", [NoSideEffect]>, BASE_HLO
     I32Attr:$index
   );
 
-  let results = (outs HLO_TensorOrTuple);
+  let results = (outs HLO_TensorOrTokenOrTuple);
 
   let hasFolder = 1;
 
diff --git a/tensorflow/compiler/mlir/xla/ir/hlo_ops_base.td b/tensorflow/compiler/mlir/xla/ir/hlo_ops_base.td
index bad1bf16ec3..408a81c48af 100644
--- a/tensorflow/compiler/mlir/xla/ir/hlo_ops_base.td
+++ b/tensorflow/compiler/mlir/xla/ir/hlo_ops_base.td
@@ -62,6 +62,8 @@ def HLO_Tuple : NestedTupleOf<[HLO_Tensor, HLO_Token]>;
 
 def HLO_TensorOrTuple : AnyTypeOf<[HLO_Tensor, HLO_Tuple]>;
 
+def HLO_TensorOrTokenOrTuple : AnyTypeOf<[HLO_Tensor, HLO_Token, HLO_Tuple]>;
+
 def HLO_DimensionValue : AnyTypeOf<[Index, HLO_Pred, HLO_Int]>;
 
 // Dynamic representation of a shape vector as a tensor.
diff --git a/tensorflow/compiler/mlir/xla/mlir_hlo_to_hlo.cc b/tensorflow/compiler/mlir/xla/mlir_hlo_to_hlo.cc
index 3c670ef0c6e..1c25625802f 100644
--- a/tensorflow/compiler/mlir/xla/mlir_hlo_to_hlo.cc
+++ b/tensorflow/compiler/mlir/xla/mlir_hlo_to_hlo.cc
@@ -73,7 +73,8 @@ constexpr char kPaddingMapAttr[] = "xla_hlo.padding_map";
 constexpr char kShapeIndicesAttr[] = "shape_indices";
 constexpr char kPaddingArgIndicesAttr[] = "padding_arg_indices";
 constexpr char kShardingAttr[] = "xla_hlo.sharding";
-constexpr char kRepicationAttr[] = "tf_device.is_same_data_across_replicas";
+constexpr char kFrontendAttributesAttr[] = "xla_hlo.frontend_attributes";
+constexpr char kRepicationAttr[] = "xla_hlo.is_same_data_across_replicas";
 
 // Passes through everything except for unique_ptr, on which it calls get().
 // This exists to allow the generated code to call XLA functions that take a raw
@@ -381,21 +382,41 @@ static xla::ScatterDimensionNumbers Convert_scatter_dimension_numbers(
   return output;
 }
 
+// Extracts sharding from attribute string.
+static absl::optional<xla::OpSharding> CreateOpShardingFromStringRef(
+    llvm::StringRef sharding) {
+  xla::OpSharding sharding_proto;
+  if (!sharding_proto.ParseFromString(sharding.str())) return absl::nullopt;
+  return sharding_proto;
+}
+
 // Returns an OpSharding proto from the "sharding" attribute of the op. If the
 // op doesn't have a sharding attribute or the sharding attribute is invalid,
 // returns absl::nullopt.
 static absl::optional<xla::OpSharding> CreateOpShardingFromAttribute(
     mlir::Operation* op) {
   auto sharding = op->getAttrOfType<mlir::StringAttr>(kShardingAttr);
-  if (!sharding) {
-    return absl::nullopt;
-  }
-  ::xla::OpSharding sharding_proto;
-  if (!::tensorflow::protobuf::TextFormat::ParseFromString(
-          sharding.getValue().str(), &sharding_proto)) {
-    return absl::nullopt;
-  }
-  return sharding_proto;
+  if (!sharding) return absl::nullopt;
+  return CreateOpShardingFromStringRef(sharding.getValue());
+}
+
+// Returns a FrontendAttributes proto from the "frontend_attributes" attribute
+// of the op. An empty FrontendAttributes proto is returned if an op does not
+// have frontend attributes.
+static xla::FrontendAttributes CreateOpFrontendAttributesFromAttribute(
+    mlir::Operation* op) {
+  xla::FrontendAttributes frontend_attributes;
+  auto frontend_attributes_dict =
+      op->getAttrOfType<mlir::DictionaryAttr>(kFrontendAttributesAttr);
+
+  if (!frontend_attributes_dict) return frontend_attributes;
+
+  for (const auto& attr : frontend_attributes_dict)
+    if (auto value_str_attr = attr.second.dyn_cast<mlir::StringAttr>())
+      frontend_attributes.mutable_map()->insert(
+          {attr.first.str(), value_str_attr.getValue().str()});
+
+  return frontend_attributes;
 }
 
 // Checks if all shardings are set.
@@ -407,14 +428,6 @@ static bool AllOptionalShardingsAreSet(
                       });
 }
 
-// Extracts sharding from attribute string.
-static absl::optional<xla::OpSharding> CreateOpShardingFromStringRef(
-    llvm::StringRef sharding) {
-  xla::OpSharding sharding_proto;
-  if (!sharding_proto.ParseFromString(sharding.str())) return absl::nullopt;
-  return sharding_proto;
-}
-
 // Extracts argument and result shardings from function.
 static void ExtractShardingsFromFunction(
     mlir::FuncOp function,
@@ -1144,8 +1157,8 @@ LogicalResult ConvertToHloModule::RunOnFunction(mlir::FuncOp f) {
     bool any_arg_replicated = false;
     entry_args_same_across_replicas.reserve(f.getNumArguments());
     for (int64_t i = 0; i < f.getNumArguments(); ++i) {
-      auto attr = f.getArgAttrOfType<mlir::BoolAttr>(i, kRepicationAttr);
-      entry_args_same_across_replicas.push_back(attr && attr.getValue());
+      auto attr = f.getArgAttrOfType<mlir::UnitAttr>(i, kRepicationAttr);
+      entry_args_same_across_replicas.push_back(attr != nullptr);
       any_arg_replicated |= entry_args_same_across_replicas.back();
       // Pass the alias info to the builder so that it will build the alias info
       // into the resulting HloModule.
diff --git a/tensorflow/compiler/mlir/xla/operator_writer_gen.cc b/tensorflow/compiler/mlir/xla/operator_writer_gen.cc
index 44af7ca75bb..27cd7e21147 100644
--- a/tensorflow/compiler/mlir/xla/operator_writer_gen.cc
+++ b/tensorflow/compiler/mlir/xla/operator_writer_gen.cc
@@ -138,6 +138,13 @@ static bool OperatorWritersMain(raw_ostream& os, RecordKeeper& records) {
   os << "  xla::XlaScopedShardingAssignment sharding(lowering_context.builder, "
         "CreateOpShardingFromAttribute(op));\n\n";
 
+  // Create a scoped object to assign frontend attributes to generated XLA ops.
+  // Any HLO can have an attribute of "frontend_attributes", which are used to
+  // pass hints / configuration options.
+  os << "  xla::XlaScopedFrontendAttributesAssignment "
+        "frontend_attributes(lowering_context.builder, "
+        "CreateOpFrontendAttributesFromAttribute(op));\n\n";
+
   // Retrieve all the definitions derived from HLO_Op and sort by record name.
   for (const auto* def : records.getAllDerivedDefinitions("HLO_Op")) {
     // Skip operations that have a custom exporter.
diff --git a/tensorflow/compiler/mlir/xla/tests/legalize-tf.mlir b/tensorflow/compiler/mlir/xla/tests/legalize-tf.mlir
index 5a175b1774a..b01c84f9517 100644
--- a/tensorflow/compiler/mlir/xla/tests/legalize-tf.mlir
+++ b/tensorflow/compiler/mlir/xla/tests/legalize-tf.mlir
@@ -845,7 +845,19 @@ func @infeed_dequeue_tuple() -> (tensor<3xi32>, tensor<4xf32>) {
 func @infeed_dequeue_tuple_sharding() -> tensor<8xi32> {
   // CHECK: "xla_hlo.infeed"
   // An additional sharding is added at the end to account for token result.
-  // CHECK-SAME: xla_hlo.sharding = "type: TUPLE\0Atuple_shardings {\0A type: MAXIMAL\0A tile_assignment_dimensions: 1\0A tile_assignment_devices: 0\0A}\0Atuple_shardings {\0A type: MAXIMAL\0A tile_assignment_dimensions: 1\0A tile_assignment_devices: 0\0A}\0A"
+  // Proto debug string:
+  //   type: TUPLE
+  //   tuple_shardings {
+  //     type: MAXIMAL
+  //     tile_assignment_dimensions: 1
+  //     tile_assignment_devices: 0
+  //   }
+  //   tuple_shardings {
+  //     type: MAXIMAL
+  //     tile_assignment_dimensions: 1
+  //     tile_assignment_devices: 0
+  //   }
+  // CHECK-SAME: xla_hlo.sharding = "\08\02*\08\08\01\1A\01\01\22\01\00*\08\08\01\1A\01\01\22\01\00"
   %0 = "tf.InfeedDequeueTuple"() {_XlaSharding = "\08\02*\08\08\01\1A\01\01\22\01\00"} : () -> tensor<8xi32>
   return %0 : tensor<8xi32>
 }
@@ -1526,6 +1538,67 @@ func @rfft_1D(%arg0: tensor<8xf32>) -> tensor<8xcomplex<f32>> {
   return %0 : tensor<8xcomplex<f32>>
 }
 
+//===----------------------------------------------------------------------===//
+// Shape op legalization.
+//===----------------------------------------------------------------------===//
+
+// CHECK-LABEL: func @shape_1D
+func @shape_1D(%arg0: tensor<?xf32>) -> tensor<1xi32> {
+  // CHECK-DAG: [[SHAPE:%.+]] = shape.shape_of %arg0
+  // CHECK-DAG: [[EXTENT:%.+]] = shape.get_extent [[SHAPE]], 0
+  // CHECK-DAG: [[TO_INDEX:%.+]] = shape.size_to_index [[EXTENT]]
+  // CHECK-DAG: [[CAST:%.+]] = index_cast [[TO_INDEX]]
+  // CHECK-DAG: [[TENSOR:%.+]] = tensor_from_elements([[CAST]])
+  // CHECK-DAG: [[RESHAPE:%.+]] = "xla_hlo.reshape"([[TENSOR]])
+  // CHECK-DAG: [[CONCAT:%.+]] = "xla_hlo.concatenate"([[RESHAPE]]) {dimension = 0 : i64}
+  %0 = "tf.Shape"(%arg0) : (tensor<?xf32>) -> tensor<1xi32>
+
+  // CHECK: return [[CONCAT]]
+  return %0 : tensor<1xi32>
+}
+
+// CHECK-LABEL: func @shape_2D
+func @shape_2D(%arg0: tensor<?x?xf32>) -> tensor<2xi32> {
+  // CHECK-DAG: [[SHAPE:%.+]] = shape.shape_of %arg0
+  // CHECK-DAG: [[EXTENT0:%.+]] = shape.get_extent [[SHAPE]], 0
+  // CHECK-DAG: [[EXTENT1:%.+]] = shape.get_extent [[SHAPE]], 1
+  // CHECK-DAG: [[TO_INDEX0:%.+]] = shape.size_to_index [[EXTENT0]]
+  // CHECK-DAG: [[TO_INDEX1:%.+]] = shape.size_to_index [[EXTENT1]]
+  // CHECK-DAG: [[CAST0:%.+]] = index_cast [[TO_INDEX0]]
+  // CHECK-DAG: [[CAST1:%.+]] = index_cast [[TO_INDEX1]]
+  // CHECK-DAG: [[TENSOR0:%.+]] = tensor_from_elements([[CAST0]])
+  // CHECK-DAG: [[TENSOR1:%.+]] = tensor_from_elements([[CAST1]])
+  // CHECK-DAG: [[RESHAPE0:%.+]] = "xla_hlo.reshape"([[TENSOR0]])
+  // CHECK-DAG: [[RESHAPE1:%.+]] = "xla_hlo.reshape"([[TENSOR1]])
+  // CHECK-DAG: [[CONCAT:%.+]] = "xla_hlo.concatenate"([[RESHAPE0]], [[RESHAPE1]]) {dimension = 0 : i64}
+  %0 = "tf.Shape"(%arg0) : (tensor<?x?xf32>) -> tensor<2xi32>
+
+  // CHECK: return [[CONCAT]]
+  return %0 : tensor<2xi32>
+}
+
+// CHECK-LABEL: func @shape_with_const
+func @shape_with_const(%arg0: tensor<?x3xf32>) -> tensor<2xi32> {
+  // CHECK-DAG: [[SHAPE:%.+]] = shape.shape_of %arg0
+  // CHECK-DAG: [[EXTENT:%.+]] = shape.get_extent [[SHAPE]], 0
+  // CHECK-DAG: [[TO_INDEX:%.+]] = shape.size_to_index [[EXTENT]]
+  // CHECK-DAG: [[CAST:%.+]] = index_cast [[TO_INDEX]]
+  // CHECK-DAG: [[TENSOR:%.+]] = tensor_from_elements([[CAST]])
+  // CHECK-DAG: [[RESHAPE:%.+]] = "xla_hlo.reshape"([[TENSOR]])
+  // CHECK-DAG: [[CONST:%.+]] = xla_hlo.constant dense<3>
+  // CHECK-DAG: [[CONCAT:%.+]] = "xla_hlo.concatenate"([[RESHAPE]], [[CONST]]) {dimension = 0 : i64}
+  %0 = "tf.Shape"(%arg0) : (tensor<?x3xf32>) -> tensor<2xi32>
+
+  // CHECK: return [[CONCAT]]
+  return %0 : tensor<2xi32>
+}
+
+// CHECK-LABEL: func @shape_rankless
+func @shape_rankless(%arg0: tensor<*xf32>) -> tensor<?xi32> {
+  %0 = "tf.Shape"(%arg0) : (tensor<*xf32>) -> tensor<?xi32>
+  return %0 : tensor<?xi32>
+}
+
 //===----------------------------------------------------------------------===//
 // Transpose op legalization.
 //===----------------------------------------------------------------------===//
@@ -2861,6 +2934,50 @@ func @range(%arg0: tensor<f32>, %arg1: tensor<f32>) -> tensor<5xf32> {
   return %3 : tensor<5xf32>
 }
 
+// CHECK-LABEL: func @range_dynamic
+// CHECK-SAME: [[START:%.*]]: tensor<f32>, [[DELTA:%.*]]: tensor<f32>
+func @range_dynamic(%arg0: tensor<f32>, %arg1: tensor<f32>, %arg2: tensor<f32>) -> tensor<?xf32> {
+  // CHECK-DAG: [[SUB:%.+]] = xla_hlo.subtract %arg1, %arg0
+  // CHECK-DAG: [[ABS1:%.+]] = "xla_hlo.abs"([[SUB]])
+  // CHECK-DAG: [[CONVERT1:%.+]] = "xla_hlo.convert"([[ABS1]])
+  // CHECK-DAG: [[CONVERT2:%.+]] = "xla_hlo.convert"(%arg2)
+  // CHECK-DAG: [[DIV:%.+]] = xla_hlo.divide [[CONVERT1]], [[CONVERT2]]
+  // CHECK-DAG: [[CEIL:%.+]] = "xla_hlo.ceil"([[DIV]])
+  // CHECK-DAG: [[CONVERT3:%.+]] = "xla_hlo.convert"([[CEIL]])
+  // CHECK-DAG: [[RESHAPE:%.+]] = "xla_hlo.reshape"([[CONVERT3]])
+  // CHECK-DAG: [[IOTA:%.+]] = "xla_hlo.dynamic_iota"([[RESHAPE]]) {iota_dimension = 0 : i64}
+  // CHECK-DAG: [[CONVERT3:%.+]] = "xla_hlo.convert"(%arg0)
+  // CHECK-DAG: [[CONVERT4:%.+]] = "xla_hlo.convert"(%arg2)
+  // CHECK-DAG: [[MUL:%.+]] = xla_chlo.broadcast_multiply [[IOTA]], [[CONVERT4]] {broadcast_dimensions = dense<[]> : tensor<0xi64>}
+  // CHECK-DAG: [[ADD:%.+]] = xla_chlo.broadcast_add [[MUL]], [[CONVERT3]] {broadcast_dimensions = dense<[]> : tensor<0xi64>}
+  %2 = "tf.Range"(%arg0, %arg1, %arg2) {Tidx = "tfdtype$DT_FLOAT", device = "", name = "range"} : (tensor<f32>, tensor<f32>, tensor<f32>) -> tensor<?xf32>
+
+  // CHECK: return [[ADD]]
+  return %2 : tensor<?xf32>
+}
+
+// CHECK-LABEL: func @range_int_dynamic
+// CHECK-SAME: [[START:%.*]]: tensor<i32>, [[DELTA:%.*]]: tensor<i32>
+func @range_int_dynamic(%arg0: tensor<i32>, %arg1: tensor<i32>, %arg2: tensor<i32>) -> tensor<?xi32> {
+  // CHECK-DAG: [[SUB:%.+]] = xla_hlo.subtract %arg1, %arg0
+  // CHECK-DAG: [[ABS1:%.+]] = "xla_hlo.abs"([[SUB]])
+  // CHECK-DAG: [[CONVERT1:%.+]] = "xla_hlo.convert"([[ABS1]])
+  // CHECK-DAG: [[CONVERT2:%.+]] = "xla_hlo.convert"(%arg2)
+  // CHECK-DAG: [[DIV:%.+]] = xla_hlo.divide [[CONVERT1]], [[CONVERT2]]
+  // CHECK-DAG: [[CEIL:%.+]] = "xla_hlo.ceil"([[DIV]])
+  // CHECK-DAG: [[CONVERT3:%.+]] = "xla_hlo.convert"([[CEIL]])
+  // CHECK-DAG: [[RESHAPE:%.+]] = "xla_hlo.reshape"([[CONVERT3]])
+  // CHECK-DAG: [[IOTA:%.+]] = "xla_hlo.dynamic_iota"([[RESHAPE]]) {iota_dimension = 0 : i64}
+  // CHECK-DAG: [[CONVERT3:%.+]] = "xla_hlo.convert"(%arg0)
+  // CHECK-DAG: [[CONVERT4:%.+]] = "xla_hlo.convert"(%arg2)
+  // CHECK-DAG: [[MUL:%.+]] = xla_chlo.broadcast_multiply [[IOTA]], [[CONVERT4]] {broadcast_dimensions = dense<[]> : tensor<0xi64>}
+  // CHECK-DAG: [[ADD:%.+]] = xla_chlo.broadcast_add [[MUL]], [[CONVERT3]] {broadcast_dimensions = dense<[]> : tensor<0xi64>}
+  %2 = "tf.Range"(%arg0, %arg1, %arg2) {Tidx = "tfdtype$DT_FLOAT", device = "", name = "range"} : (tensor<i32>, tensor<i32>, tensor<i32>) -> tensor<?xi32>
+
+  // CHECK: return [[ADD]]
+  return %2 : tensor<?xi32>
+}
+
 // CHECK-LABEL: func @linspace_static
 // CHECK-SAME: [[START:%.*]]: tensor<f32>, [[STOP:%.*]]: tensor<f32>
 func @linspace_static(%arg0: tensor<f32>, %arg1: tensor<f32>) -> tensor<4xf32> {
diff --git a/tensorflow/compiler/mlir/xla/tests/lhlo-legalize-select-and-scatter.mlir b/tensorflow/compiler/mlir/xla/tests/lhlo-legalize-select-and-scatter.mlir
index c640b395f4d..3bc5e974ae9 100644
--- a/tensorflow/compiler/mlir/xla/tests/lhlo-legalize-select-and-scatter.mlir
+++ b/tensorflow/compiler/mlir/xla/tests/lhlo-legalize-select-and-scatter.mlir
@@ -41,12 +41,12 @@ func @select_and_scatter(%arg: memref<112x112xf32>,
 // CHECK:  [[C56:%.*]] = constant 56 : index
 // CHECK:  [[C1:%.*]] = constant 1 : index
 // CHECK:  [[C0_F32:%.*]] = constant 0.000000e+00 : f32
-// CHECK:  [[CFALSE:%.*]] = constant 0 : i1
+// CHECK:  [[CFALSE:%.*]] = constant false
 // CHECK:  [[C3:%.*]] = constant 3 : index
 // CHECK:  [[C2:%.*]] = constant 2 : index
 // CHECK:  [[C0:%.*]] = constant 0 : index
 // CHECK:  [[C112:%.*]] = constant 112 : index
-// CHECK:  [[CTRUE:%.*]] = constant 1 : i1
+// CHECK:  [[CTRUE:%.*]] = constant true
 
 // Parallel loop to initialize the output buffer.
 // CHECK:    [[INIT:%.*]] = load [[INIT_BUF]][] : memref<f32>
diff --git a/tensorflow/compiler/mlir/xla/tests/lhlo-legalize-to-linalg.mlir b/tensorflow/compiler/mlir/xla/tests/lhlo-legalize-to-linalg.mlir
index 626e905695c..0d5a48e7b71 100644
--- a/tensorflow/compiler/mlir/xla/tests/lhlo-legalize-to-linalg.mlir
+++ b/tensorflow/compiler/mlir/xla/tests/lhlo-legalize-to-linalg.mlir
@@ -228,32 +228,54 @@ func @dynamic_broadcast_in_dim(%operand: memref<?x?x?xf32>,
 
 // -----
 
-// CHECK-DAG: #[[OPERAND_MAP:.*]] = affine_map<(d0, d1, d2, d3, d4) -> (d4, d0, 0)>
-// CHECK-DAG: #[[RESULT_MAP:.*]] = affine_map<(d0, d1, d2, d3, d4) -> (d0, d1, d2, d3, d4)>
-// CHECK-LABEL: func @broadcast_in_dim_with_expansion
-func @broadcast_in_dim_with_expansion(%operand: memref<5x7x1xf32>,
-                                      %result: memref<7x10x6x4x5xf32>) {
+// CHECK-DAG: #[[OPERAND_MAP:.*]] = affine_map<(d0, d1) -> (d0)>
+// CHECK-DAG: #[[RESULT_MAP:.*]] = affine_map<(d0, d1) -> (d0, d1)>
+// CHECK-LABEL: func @static_broadcast_in_dim_no_expansion
+func @static_broadcast_in_dim_no_expansion(%operand: memref<5xf32>,
+                                           %result: memref<5x10xf32>) {
   "xla_lhlo.broadcast_in_dim"(%operand, %result) {
-    broadcast_dimensions = dense<[4,0,2]> : tensor<3xi64>
-  } : (memref<5x7x1xf32>, memref<7x10x6x4x5xf32>) -> ()
+    broadcast_dimensions = dense<[0]> : tensor<1xi64>
+  } : (memref<5xf32>, memref<5x10xf32>) -> ()
   return
 }
+// CHECK-NOT: linalg.reshape
 // CHECK: linalg.generic {{{.*}}indexing_maps = [#[[OPERAND_MAP]], #[[RESULT_MAP]]]
 // CHECK-NEXT: ^bb0(%[[OPERAND:.*]]: f32, %[[RESULT:.*]]: f32):
 // CHECK-NEXT:   linalg.yield %[[OPERAND]] : f32
 
 // -----
 
-// CHECK-DAG: #[[RESULT_MAP_0:.*]] = affine_map<(d0, d1, d2) -> ()>
+// CHECK-DAG: #[[REASSOCIATION:.*]] = affine_map<(d0, d1) -> (d0, d1)>
+// CHECK-DAG: #[[OPERAND_MAP:.*]] = affine_map<(d0, d1, d2) -> (d0)>
 // CHECK-DAG: #[[RESULT_MAP:.*]] = affine_map<(d0, d1, d2) -> (d0, d1, d2)>
-// CHECK-LABEL: func @broadcast_in_dim_scalar
-func @broadcast_in_dim_scalar(%operand: memref<f32>,
-                              %result: memref<7x10x6xf32>) {
+// CHECK-LABEL: func @static_broadcast_in_dim_expansion
+func @static_broadcast_in_dim_expansion(%operand: memref<1x5xf32>,
+                                        %result: memref<5x10x100xf32>) {
   "xla_lhlo.broadcast_in_dim"(%operand, %result) {
-    broadcast_dimensions = dense<[]> : tensor<0xi64>
-  } : (memref<f32>, memref<7x10x6xf32>) -> ()
+    broadcast_dimensions = dense<[2, 0]> : tensor<2xi64>
+  } : (memref<1x5xf32>, memref<5x10x100xf32>) -> ()
   return
 }
+// CHECK: %[[RESHAPED_ARG:.*]] = linalg.reshape %{{.*}}#[[REASSOCIATION]]]
+// CHECK-SAME:                   memref<1x5xf32> into memref<5xf32>
+// CHECK: linalg.generic {{{.*}}indexing_maps =
+// CHECK-SAME:       [#[[OPERAND_MAP]], #[[RESULT_MAP]]]{{.*}} %[[RESHAPED_ARG]]
+// CHECK-NEXT: ^bb0(%[[OPERAND:.*]]: f32, %[[RESULT:.*]]: f32):
+// CHECK-NEXT:   linalg.yield %[[OPERAND]] : f32
+
+// -----
+
+// CHECK-DAG: #[[RESULT_MAP_0:.*]] = affine_map<(d0, d1) -> ()>
+// CHECK-DAG: #[[RESULT_MAP:.*]] = affine_map<(d0, d1) -> (d0, d1)>
+// CHECK-LABEL: func @static_broadcast_in_dim_scalar
+func @static_broadcast_in_dim_scalar(%operand: memref<f32>,
+                                     %result: memref<5x10xf32>) {
+  "xla_lhlo.broadcast_in_dim"(%operand, %result) {
+    broadcast_dimensions = dense<[]> : tensor<0xi64>
+  } : (memref<f32>, memref<5x10xf32>) -> ()
+  return
+}
+// CHECK-NOT: linalg.reshape
 // CHECK: linalg.generic {{{.*}}indexing_maps = [#[[RESULT_MAP_0]], #[[RESULT_MAP]]]
 // CHECK-NEXT: ^bb0(%[[CONST:.*]]: f32, %[[RESULT:.*]]: f32):
 // CHECK-NEXT:   linalg.yield %[[CONST]] : f32
@@ -262,19 +284,39 @@ func @broadcast_in_dim_scalar(%operand: memref<f32>,
 
 // CHECK-DAG: #[[OPERAND_MAP:.+]] = affine_map<(d0, d1) -> (d0)>
 // CHECK-DAG: #[[RESULT_MAP:.+]] = affine_map<(d0, d1) -> (d0, d1)>
-// CHECK-LABEL: func @broadcast_in_dim_with_one_to_one
-func @broadcast_in_dim_with_one_to_one(%operand: memref<1xf32>, %result: memref<1x5xf32>) {
+// CHECK-LABEL: func @static_broadcast_in_dim_with_one_to_one
+func @static_broadcast_in_dim_with_one_to_one(%operand: memref<1xf32>,
+                                              %result: memref<1x5xf32>) {
   "xla_lhlo.broadcast_in_dim"(%operand, %result) {
     broadcast_dimensions = dense<[0]> : tensor<1xi64>
   } : (memref<1xf32>, memref<1x5xf32>) -> ()
   return
 }
+// CHECK-NOT: linalg.reshape
 // CHECK: linalg.generic {{{.*}}indexing_maps = [#[[OPERAND_MAP]], #[[RESULT_MAP]]]
 // CHECK-NEXT: ^bb0(%[[OPERAND:.+]]: f32, %{{.+}}: f32):
 // CHECK-NEXT:   linalg.yield %[[OPERAND]] : f32
 
 // -----
 
+// CHECK-DAG: #[[RESULT_MAP:.+]] = affine_map<(d0, d1) -> (d0, d1)>
+// CHECK-LABEL: func @static_broadcast_in_dim_with_one_to_many
+func @static_broadcast_in_dim_with_one_to_many(%operand: memref<1xf32>,
+                                               %result: memref<5x5xf32>) {
+  "xla_lhlo.broadcast_in_dim"(%operand, %result) {
+    broadcast_dimensions = dense<[1]> : tensor<1xi64>
+  } : (memref<1xf32>, memref<5x5xf32>) -> ()
+  return
+}
+// CHECK-NOT: linalg.reshape
+// CHECK: %[[C0:.*]] = constant 0 : index
+// CHECK: %[[VALUE:.*]] = load %{{.*}}[[C0]]
+// CHECK: linalg.generic {{{.*}}indexing_maps = [#[[RESULT_MAP]]]
+// CHECK-NEXT: ^bb0(%{{.+}}: f32):
+// CHECK-NEXT:   linalg.yield %[[VALUE]] : f32
+
+// -----
+
 // CHECK-LABEL: func @constant
 func @constant(%value: memref<i32>) {
   "xla_lhlo.constant"(%value) {
diff --git a/tensorflow/compiler/mlir/xla/tests/lhlo-legalize-to-parallel-loops.mlir b/tensorflow/compiler/mlir/xla/tests/lhlo-legalize-to-parallel-loops.mlir
index 32c367f97d6..f079e6feec3 100644
--- a/tensorflow/compiler/mlir/xla/tests/lhlo-legalize-to-parallel-loops.mlir
+++ b/tensorflow/compiler/mlir/xla/tests/lhlo-legalize-to-parallel-loops.mlir
@@ -150,7 +150,7 @@ func @reduce_window(%arg: memref<112x112xf32>,
 // CHECK-SAME:      [[OPERAND_BUF:%.*]]: memref<112x112xf32>,
 // CHECK-SAME:      [[INIT_BUF:%.*]]: memref<f32>,
 // CHECK-SAME:      [[RESULT_BUF:%.*]]: memref<56x56xf32>) {
-// CHECK-DAG:  [[IN_BOUNDS:%.*]] = constant 1 : i1
+// CHECK-DAG:  [[IN_BOUNDS:%.*]] = constant true
 // CHECK-DAG:  [[C0:%.*]] = constant 0 : index
 // CHECK-DAG:  [[C1:%.*]] = constant 1 : index
 // CHECK-DAG:  [[C2:%.*]] = constant 2 : index
diff --git a/tensorflow/compiler/mlir/xla/tests/ops.mlir b/tensorflow/compiler/mlir/xla/tests/ops.mlir
index 0a69ee93aee..38964fb56c2 100644
--- a/tensorflow/compiler/mlir/xla/tests/ops.mlir
+++ b/tensorflow/compiler/mlir/xla/tests/ops.mlir
@@ -830,6 +830,13 @@ func @get_tuple_element(%arg0: tuple<tensor<f32>, tensor<i32>>) -> tensor<f32> {
 
 // -----
 
+func @get_tuple_element_token(%arg0: tuple<tensor<f32>, !xla_hlo.token>) -> !xla_hlo.token {
+  %0 = "xla_hlo.get_tuple_element"(%arg0) {index = 1 : i32} : (tuple<tensor<f32>, !xla_hlo.token>) -> !xla_hlo.token
+  return %0 : !xla_hlo.token
+}
+
+// -----
+
 func @get_tuple_element_bad_type(%arg0: tuple<tensor<f32>, tensor<i32>>) -> tensor<i32> {
   // expected-error@+1 {{has return type tensor<i32>, but expected tensor<f32>}}
   %0 = "xla_hlo.get_tuple_element"(%arg0) {index = 0 : i32} : (tuple<tensor<f32>, tensor<i32>>) -> tensor<i32>
diff --git a/tensorflow/compiler/mlir/xla/tests/translate/export.mlir b/tensorflow/compiler/mlir/xla/tests/translate/export.mlir
index 20b43e8633d..576c2f89d0b 100644
--- a/tensorflow/compiler/mlir/xla/tests/translate/export.mlir
+++ b/tensorflow/compiler/mlir/xla/tests/translate/export.mlir
@@ -963,9 +963,19 @@ func @main(%input0: tensor<16x16xf32>, %input1: tensor<16x16xi32>) {
 
 // -----
 
+// The following op sharding is used:
+// Proto debug string:
+//   type: OTHER
+//   tile_assignment_dimensions: 1
+//   tile_assignment_dimensions: 2
+//   tile_assignment_devices: 0
+//   tile_assignment_devices: 1
+// Serialized string:
+//   "\08\03\1A\02\01\02\22\02\00\01"
+
 // CHECK:  HloModule
 func @main(%arg0: tensor<16x16xf32>) -> tensor<16x16xf32> {
-  %0 = "xla_hlo.custom_call"(%arg0) {backend_config = "", call_target_name = "Sharding", xla_hlo.sharding = "type: OTHER\ntile_assignment_dimensions: 1\ntile_assignment_dimensions: 2\ntile_assignment_devices: 0\ntile_assignment_devices: 1"} : (tensor<16x16xf32>) -> tensor<16x16xf32>
+  %0 = "xla_hlo.custom_call"(%arg0) {backend_config = "", call_target_name = "Sharding", xla_hlo.sharding = "\08\03\1A\02\01\02\22\02\00\01"} : (tensor<16x16xf32>) -> tensor<16x16xf32>
   return %0 : tensor<16x16xf32>
 }
 
@@ -978,7 +988,7 @@ func @main(%arg0: tensor<16x16xf32>) -> tensor<16x16xf32> {
 // Tests that the exported HLO module keeps parameter replication annotation.
 
 // CHECK:  HloModule
-func @main(%arg0: tensor<16x16xf32>, %arg1: tensor<16x16xf32> {tf_device.is_same_data_across_replicas = true}) -> tensor<16x16xf32> {
+func @main(%arg0: tensor<16x16xf32>, %arg1: tensor<16x16xf32> {xla_hlo.is_same_data_across_replicas}) -> tensor<16x16xf32> {
   %0 = "xla_hlo.add"(%arg0, %arg1) : (tensor<16x16xf32>, tensor<16x16xf32>) -> tensor<16x16xf32>
   return %0 : tensor<16x16xf32>
 }
@@ -1008,19 +1018,19 @@ func @main(%arg0: tensor<2xcomplex<f32>>, %arg1: tensor<2xcomplex<f64>>) -> (ten
 // -----
 
 // CHECK:  HloModule
-func @main(%arg0: tensor<4xui8>) -> (tensor<4xui8>) {
+func @main(%arg0: tensor<4xui8>) -> tensor<4xui8> {
   %0 = "xla_hlo.not"(%arg0) : (tensor<4xui8>) -> tensor<4xui8>
   return %0 : tensor<4xui8>
 }
 
 // CHECK: ENTRY
 // CHECK: %[[ARG0:.*]] = u8[4] parameter(0)
-//  ROOT %[[RESULT:.*]] = u8[4] not(u8[4] %[[ARG0]])
+// CHECK: ROOT %[[RESULT:.*]] = u8[4] not(u8[4] %[[ARG0]])
 
 // -----
 
 // CHECK:  HloModule
-func @main(%arg0: tensor<4xi32>) -> (tensor<*xi32>) {
+func @main(%arg0: tensor<4xi32>) -> tensor<*xi32> {
   %0 = "xla_hlo.not"(%arg0) : (tensor<4xi32>) -> tensor<4xi32>
   %1 = tensor_cast %0 : tensor<4xi32> to tensor<*xi32>
   return %1 : tensor<*xi32>
@@ -1028,4 +1038,52 @@ func @main(%arg0: tensor<4xi32>) -> (tensor<*xi32>) {
 
 // CHECK: ENTRY
 // CHECK: %[[ARG0:.*]] = s32[4] parameter(0)
-//  ROOT %[[RESULT:.*]] = s32[4] not(s32[4] %[[ARG0]])
+// CHECK: ROOT %[[RESULT:.*]] = s32[4] not(s32[4] %[[ARG0]])
+
+// -----
+
+// Tests ops with different frontend attributes have such attributes set
+// correctly in HloModule as frontend_attributes.
+
+// CHECK:  HloModule
+func @main(%arg: tensor<3x4xf32>, %token: !xla_hlo.token) -> tuple<tensor<3x4xf32>, !xla_hlo.token> {
+  %0 = "xla_hlo.send"(%arg, %token) {channel_id = {handle = 1 : i64, type = 2 : i64}, is_host_transfer = true, xla_hlo.frontend_attributes = {_xla_host_transfer_original_type = "f32", _xla_host_transfer_rendezvous = "channel_dtoh_0"}} : (tensor<3x4xf32>, !xla_hlo.token) -> !xla_hlo.token
+  %1 = "xla_hlo.recv"(%0) {channel_id = {handle = 2 : i64, type = 3 : i64}, is_host_transfer = true, xla_hlo.frontend_attributes = {_xla_host_transfer_original_type = "f32", _xla_host_transfer_rendezvous = "channel_htod_0"}} : (!xla_hlo.token) -> tuple<tensor<3x4xf32>, !xla_hlo.token>
+  return %1 : tuple<tensor<3x4xf32>, !xla_hlo.token>
+}
+
+// CHECK:  ENTRY
+// CHECK:  %[[SEND:.*]] = (f32[3,4], u32[], token[]) send
+// CHECK-SAME: frontend_attributes={_xla_host_transfer_original_type="f32",_xla_host_transfer_rendezvous="channel_dtoh_0"}
+// CHECK:  %[[SEND_DONE:.*]] = token[] send-done((f32[3,4], u32[], token[]) %[[SEND]])
+// CHECK-SAME: frontend_attributes={_xla_host_transfer_original_type="f32",_xla_host_transfer_rendezvous="channel_dtoh_0"}
+// CHECK:  %[[RECV:.*]] = (f32[3,4], u32[], token[]) recv(token[] %[[SEND_DONE]])
+// CHECK-SAME: frontend_attributes={_xla_host_transfer_original_type="f32",_xla_host_transfer_rendezvous="channel_htod_0"}
+// CHECK:  ROOT %{{.*}} = (f32[3,4], token[]) recv-done((f32[3,4], u32[], token[]) %[[RECV]])
+// CHECK-SAME: frontend_attributes={_xla_host_transfer_original_type="f32",_xla_host_transfer_rendezvous="channel_htod_0"}
+
+// -----
+
+// Tests ops with empty frontend attributes do not have frontend_attributes
+// populated in HloModule.
+
+// CHECK:  HloModule
+func @main(%arg: tensor<3x4xf32>, %token: !xla_hlo.token) -> !xla_hlo.token {
+  %0 = "xla_hlo.send"(%arg, %token) {channel_id = {handle = 1 : i64, type = 2 : i64}, is_host_transfer = true, xla_hlo.frontend_attributes = {}} : (tensor<3x4xf32>, !xla_hlo.token) -> !xla_hlo.token
+  return %0 : !xla_hlo.token
+}
+
+// CHECK-NOT:  frontend_attributes
+
+// -----
+
+// Tests ops with no frontend attributes do not have frontend_attributes
+// populated in HloModule.
+
+// CHECK:  HloModule
+func @main(%arg: tensor<3x4xf32>, %token: !xla_hlo.token) -> !xla_hlo.token {
+  %0 = "xla_hlo.send"(%arg, %token) {channel_id = {handle = 1 : i64, type = 2 : i64}, is_host_transfer = true} : (tensor<3x4xf32>, !xla_hlo.token) -> !xla_hlo.token
+  return %0 : !xla_hlo.token
+}
+
+// CHECK-NOT:  frontend_attributes
diff --git a/tensorflow/compiler/mlir/xla/transforms/legalize_tf.cc b/tensorflow/compiler/mlir/xla/transforms/legalize_tf.cc
index dd7a28a1388..0f26c380162 100644
--- a/tensorflow/compiler/mlir/xla/transforms/legalize_tf.cc
+++ b/tensorflow/compiler/mlir/xla/transforms/legalize_tf.cc
@@ -2764,6 +2764,86 @@ class ConvertRangeOp : public OpRewritePattern<TF::RangeOp> {
   }
 };
 
+// Converts RangeOp for cases with the length is a dynamic value. The shape of
+// the resulting tensor computed, then the start and delta is used with the
+// dynamic_iota value to compute the final range value.
+//
+// For example, the resulting range op value:
+//   %range = "tf.range"(%start, %limit, %delta)
+//
+// Is converted to the following.
+//   %start + %delta * iota(ceil(abs((%limit - %start) / %delta))
+//
+// Implementation is defined in C++ due to the complicated type behavior.
+class ConvertDynamicRangeOp : public OpRewritePattern<TF::RangeOp> {
+  using OpRewritePattern<TF::RangeOp>::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(TF::RangeOp op,
+                                PatternRewriter &rewriter) const override {
+    auto result = op.getResult();
+    auto result_type = result.getType().cast<ShapedType>();
+    if (result_type.hasStaticShape()) {
+      return failure();
+    }
+
+    Value start = op.start();
+    Value delta = op.delta();
+    Value limit = op.limit();
+
+    // To compute the length we need to use floating point calculations so that
+    // ceil can be computed for the number of steps.
+    auto compute_element_type =
+        getElementTypeOrSelf(start.getType()).isa<FloatType>()
+            ? getElementTypeOrSelf(start.getType())
+            : rewriter.getF64Type();
+    auto compute_type = RankedTensorType::get(
+        limit.getType().cast<ShapedType>().getShape(), compute_element_type);
+
+    // Compute the length of the sequence we are going to need. This includes
+    // some conversion to float for the operations.
+    //
+    // %size = ceil(abs((%limit - %start) / %delta))
+    auto range = rewriter.create<xla_hlo::SubOp>(op.getLoc(), limit, start);
+    auto abs = rewriter.create<xla_hlo::AbsOp>(op.getLoc(), range);
+
+    // Delta is not necessarily the same type as start and limit.
+    auto abs_cast =
+        rewriter.create<xla_hlo::ConvertOp>(op.getLoc(), compute_type, abs);
+    auto delta_cast =
+        rewriter.create<xla_hlo::ConvertOp>(op.getLoc(), compute_type, delta);
+
+    // Compute the total number of integer steps and convert to the HLO
+    // dimension tensor.
+    auto normalized =
+        rewriter.create<xla_hlo::DivOp>(op.getLoc(), abs_cast, delta_cast);
+    auto ceil = rewriter.create<xla_hlo::CeilOp>(op.getLoc(), normalized);
+    auto steps = rewriter.create<xla_hlo::ConvertOp>(
+        op.getLoc(), RankedTensorType::get({}, rewriter.getI64Type()), ceil);
+    auto reshape = rewriter.create<xla_hlo::ReshapeOp>(
+        op.getLoc(), RankedTensorType::get({1}, rewriter.getI64Type()), steps);
+
+    // Using the resulting length compute the correct range value:
+    //
+    // %range = %start + %delta * iota(%size)
+    auto out_scalar_type =
+        RankedTensorType::get({}, getElementTypeOrSelf(result_type));
+    auto start_out_cast = rewriter.create<xla_hlo::ConvertOp>(
+        op.getLoc(), out_scalar_type, start);
+    auto delta_out_cast = rewriter.create<xla_hlo::ConvertOp>(
+        op.getLoc(), out_scalar_type, delta);
+
+    auto iota = rewriter.create<DynamicIotaOp>(
+        op.getLoc(), result_type, reshape, rewriter.getI64IntegerAttr(0));
+    auto scaled = rewriter.create<xla_chlo::BroadcastMulOp>(
+        op.getLoc(), result_type, iota, delta_out_cast,
+        xla::getBroadcastDimensionsAttr(&rewriter, iota, delta_cast));
+    rewriter.replaceOpWithNewOp<xla_chlo::BroadcastAddOp>(
+        op, result_type, scaled, start_out_cast,
+        xla::getBroadcastDimensionsAttr(&rewriter, scaled, start_out_cast));
+    return success();
+  }
+};
+
 ElementsAttr ConvertAxisAttr(Value val, ElementsAttr attr, Builder *builder) {
   auto int_attr = attr.cast<DenseIntElementsAttr>();
   auto type = val.getType().cast<ShapedType>();
@@ -3806,17 +3886,15 @@ class ConvertInfeedDequeueTupleOp
 
       // Token is a control signal and not a real data, so arbitrarily assign
       // the token to device 0.
-      if (sharding_proto.type() == ::xla::OpSharding::TUPLE)
+      if (sharding_proto.type() == ::xla::OpSharding::TUPLE) {
         *sharding_proto.add_tuple_shardings() =
             ::xla::sharding_builder::AssignDevice(0);
-
-      std::string sharding_str;
-      if (!::tensorflow::protobuf::TextFormat::PrintToString(sharding_proto,
-                                                             &sharding_str))
-        return failure();
-
-      data_and_token.setAttr(kShardingAttr,
-                             rewriter.getStringAttr(sharding_str));
+        data_and_token.setAttr(
+            kShardingAttr,
+            rewriter.getStringAttr(sharding_proto.SerializeAsString()));
+      } else {
+        data_and_token.setAttr(kShardingAttr, op._XlaShardingAttr());
+      }
     }
 
     // The infeed instruction produces a tuple of the infeed data and a token
@@ -4359,21 +4437,12 @@ class ConvertXlaShardingOp : public OpRewritePattern<TF::XlaShardingOp> {
     // using a string.
     if (!op._XlaSharding().hasValue()) return failure();
 
-    // _XlaSharding attribute in TF is a serialized string of the OpSharding
-    // proto, so convert to a text form here.
-    ::xla::OpSharding sharding_proto;
-    std::string sharding_str;
-    if (!sharding_proto.ParseFromString(op._XlaSharding().getValue().str()) ||
-        !::tensorflow::protobuf::TextFormat::PrintToString(sharding_proto,
-                                                           &sharding_str))
-      return failure();
-
     auto custom_call = rewriter.create<xla_hlo::CustomCallOp>(
         op.getLoc(), op.getType(), op.input(),
         /*call_target_name=*/rewriter.getStringAttr("Sharding"),
         /*has_side_effect=*/rewriter.getBoolAttr(false),
         /*backend_config=*/rewriter.getStringAttr(""));
-    custom_call.setAttr(kShardingAttr, rewriter.getStringAttr(sharding_str));
+    custom_call.setAttr(kShardingAttr, op._XlaShardingAttr());
     rewriter.replaceOp(op, custom_call.getResult());
 
     return success();
@@ -4541,6 +4610,60 @@ class ConvertCumsumOp : public OpRewritePattern<TF::CumsumOp> {
   }
 };
 
+// Converts the Tensorflow ShapeOp to a sequence of Shape dialect and Standard
+// dialect lowerings. This involves extracting the shape type, extracting and
+// converting each dimension to a known integer type, and repacking into a final
+// tensor.
+class ConvertShapeOp : public OpRewritePattern<TF::ShapeOp> {
+ public:
+  using OpRewritePattern::OpRewritePattern;
+
+  LogicalResult matchAndRewrite(TF::ShapeOp op,
+                                PatternRewriter &rewriter) const override {
+    Value input = op.input();
+    auto input_ty = input.getType().dyn_cast<RankedTensorType>();
+    // If the shape is static it can be canonicalized.
+    if (!input_ty || input_ty.hasStaticShape()) {
+      return failure();
+    }
+
+    auto result_ty = op.getResult().getType().cast<RankedTensorType>();
+    auto element_ty = result_ty.getElementType();
+
+    int64_t rank = input_ty.getRank();
+    auto shape_op = rewriter.create<shape::ShapeOfOp>(op.getLoc(), input);
+
+    auto index_ty = RankedTensorType::get({1}, element_ty);
+    llvm::SmallVector<Value, 4> dim_values;
+    for (int64_t i = 0; i < rank; ++i) {
+      if (!input_ty.isDynamicDim(i)) {
+        auto dim_attr = DenseElementsAttr::get(
+            index_ty,
+            rewriter.getIntegerAttr(element_ty, input_ty.getDimSize(i)));
+        auto index = rewriter.create<xla_hlo::ConstOp>(op.getLoc(), dim_attr);
+        dim_values.push_back(index);
+        continue;
+      }
+
+      auto extent_op = rewriter.create<shape::GetExtentOp>(
+          op.getLoc(), shape_op, rewriter.getI64IntegerAttr(i));
+      auto index_op = rewriter.create<shape::SizeToIndexOp>(
+          op.getLoc(), rewriter.getIndexType(), extent_op);
+      auto int_op =
+          rewriter.create<IndexCastOp>(op.getLoc(), element_ty, index_op);
+      auto from_tensor = rewriter.create<TensorFromElementsOp>(
+          op.getLoc(), int_op.getResult());
+      auto reshape_op =
+          rewriter.create<ReshapeOp>(op.getLoc(), index_ty, from_tensor);
+      dim_values.push_back(reshape_op);
+    }
+
+    rewriter.replaceOpWithNewOp<ConcatenateOp>(op, result_ty, dim_values,
+                                               rewriter.getI64IntegerAttr(0));
+    return success();
+  }
+};
+
 // Converts a TF QR op to HLO.
 class ConvertQrOp : public OpRewritePattern<TF::QrOp> {
  public:
@@ -5119,8 +5242,9 @@ LogicalResult legalizeTF(Operation *op, bool allow_partial_conversion,
       ConvertMaxOp, ConvertMinOp, ConvertAvgPoolOp, ConvertMaxPool2DOp,
       ConvertMaxPool3DOp, ConvertMaxPool2DGradOp, ConvertMaxPool3DGradOp,
       ConvertMeanOp, ConvertOneHotOp, ConvertOutfeedEnqueueTupleOp,
-      ConvertProdOp, ConvertQrOp, ConvertRangeOp, ConvertSelectV2Op,
-      ConvertSigmoidOp, ConvertSizeOp, ConvertSoftmaxOp<TF::LogSoftmaxOp, true>,
+      ConvertProdOp, ConvertQrOp, ConvertDynamicRangeOp, ConvertRangeOp,
+      ConvertSelectV2Op, ConvertSigmoidOp, ConvertShapeOp, ConvertSizeOp,
+      ConvertSoftmaxOp<TF::LogSoftmaxOp, true>,
       ConvertSoftmaxOp<TF::SoftmaxOp, false>, ConvertSplitOp, ConvertSplitVOp,
       ConvertStridedSliceOp, ConvertStridedSliceGradOp, ConvertSumOp,
       ConvertTensorScatterUpdateOp, ConvertTileOp, ConvertTopKV2Op,
diff --git a/tensorflow/compiler/mlir/xla/transforms/xla_legalize_to_linalg.cc b/tensorflow/compiler/mlir/xla/transforms/xla_legalize_to_linalg.cc
index 8b2da690041..547861a93e6 100644
--- a/tensorflow/compiler/mlir/xla/transforms/xla_legalize_to_linalg.cc
+++ b/tensorflow/compiler/mlir/xla/transforms/xla_legalize_to_linalg.cc
@@ -271,16 +271,17 @@ class BroadcastConverter
   }
 };
 
-template <typename OpTy, bool isLHLO = true>
-class BroadcastInDimConverter
-    : public DataMovementOpConverter<BroadcastInDimConverter<OpTy, isLHLO>,
-                                     OpTy, isLHLO> {
+class HloBroadcastInDimConverter
+    : public DataMovementOpConverter<HloBroadcastInDimConverter,
+                                     xla_hlo::BroadcastInDimOp, false> {
  public:
-  using DataMovementOpConverter<BroadcastInDimConverter<OpTy, isLHLO>, OpTy,
-                                isLHLO>::DataMovementOpConverter;
+  using DataMovementOpConverter<HloBroadcastInDimConverter,
+                                xla_hlo::BroadcastInDimOp,
+                                false>::DataMovementOpConverter;
 
-  static ArrayAttr getIndexingMapsAttr(OpTy broadcastOp, Builder* b) {
-    auto resultType = getXLAOpResultType<isLHLO>(broadcastOp);
+  static ArrayAttr getIndexingMapsAttr(xla_hlo::BroadcastInDimOp broadcastOp,
+                                       Builder* b) {
+    auto resultType = getXLAOpResultType<false>(broadcastOp);
     auto operandType =
         broadcastOp.operand().getType().template cast<ShapedType>();
     unsigned nloops = resultType.getRank();
@@ -302,8 +303,6 @@ class BroadcastInDimConverter
         int size = broadcastDim.value().getSExtValue();
         bool expansion_needed = operandShape[broadcastDim.index()] == 1 &&
                                 resultType.getShape()[size] != 1;
-        // TODO(pifon): Add support for args with dynamic shapes for the case
-        // when a dimension of size 1 is broadcasted into dim of size N.
         dimExprs.push_back(expansion_needed ? b->getAffineConstantExpr(0)
                                             : b->getAffineDimExpr(size));
       }
@@ -314,6 +313,181 @@ class BroadcastInDimConverter
   }
 };
 
+class LhloBroadcastInDimConverter
+    : public OpConversionPattern<xla_lhlo::BroadcastInDimOp> {
+ public:
+  using OpConversionPattern<xla_lhlo::BroadcastInDimOp>::OpConversionPattern;
+
+  LogicalResult matchAndRewrite(
+      xla_lhlo::BroadcastInDimOp op, ArrayRef<Value> args,
+      ConversionPatternRewriter& rewriter) const final {
+    xla_lhlo::BroadcastInDimOpOperandAdaptor operand_adaptor(args);
+    auto result_type = operand_adaptor.output().getType().cast<MemRefType>();
+    auto result_shape = result_type.getShape();
+
+    auto operand_and_dims = InsertReshapeIfNecessary(op, args, rewriter);
+
+    Value operand = std::get<0>(operand_and_dims);
+    auto broadcast_dims = std::get<1>(operand_and_dims);
+
+    auto loc = op.getLoc();
+    auto nloops = result_type.getRank();
+    auto operand_type = operand.getType().cast<MemRefType>();
+
+    // For a degenerate case, i.e. broadcasting with expansion of
+    // memref<1xELEMENT_TYPE>, the operand is not passed to `linalg.generic`.
+    // Instead the value is loaded and used directly in `linalg.yield`.
+    if (operand_type.getRank() == 1 &&
+        operand_type.getDimSize(0) <
+            result_type.getDimSize(broadcast_dims.front())) {
+      Value zero = rewriter.create<ConstantIndexOp>(loc, 0);
+      Value val =
+          rewriter.create<LoadOp>(loc, operand, llvm::makeArrayRef({zero}));
+      auto linalgOp = rewriter.create<linalg::GenericOp>(
+          loc, llvm::None, llvm::makeArrayRef(operand_adaptor.output()),
+          rewriter.getI64IntegerAttr(0), rewriter.getI64IntegerAttr(1),
+          rewriter.getAffineMapArrayAttr(
+              {rewriter.getMultiDimIdentityMap(nloops)}),
+          GetNParallelLoopsAttrs(nloops, &rewriter),
+          /*doc=*/nullptr, /*library_call=*/nullptr);
+
+      auto* region = &linalgOp.region();
+      auto* block = rewriter.createBlock(region, region->end());
+      block->addArgument(result_type.getElementType());
+
+      rewriter.setInsertionPointToEnd(block);
+      rewriter.create<linalg::YieldOp>(loc, val);
+    } else {
+      ArrayAttr indexingMapsAttr = getIndexingMapsAttr(
+          op, broadcast_dims, result_shape, operand_type, &rewriter);
+
+      OpBuilder::InsertionGuard linalgOpGuard(rewriter);
+      auto linalgOp = rewriter.create<linalg::GenericOp>(
+          loc, llvm::None,
+          llvm::makeArrayRef({operand, operand_adaptor.output()}),
+          rewriter.getI64IntegerAttr(1), rewriter.getI64IntegerAttr(1),
+          indexingMapsAttr, GetNParallelLoopsAttrs(nloops, &rewriter),
+          /*doc=*/nullptr, /*library_call=*/nullptr);
+
+      auto* region = &linalgOp.region();
+      auto* block = rewriter.createBlock(region, region->end());
+      block->addArguments(operand_type.getElementType());
+      block->addArgument(result_type.getElementType());
+
+      rewriter.setInsertionPointToEnd(block);
+      rewriter.create<linalg::YieldOp>(loc, block->getArgument(0));
+    }
+    rewriter.replaceOp(op, llvm::None);
+    return success();
+  }
+
+  // Inserts 'linalg.reshape' if there is a size-1 dim expansion.
+  std::pair<Value, SmallVector<int64_t, 2>> InsertReshapeIfNecessary(
+      xla_lhlo::BroadcastInDimOp op, ArrayRef<Value> args,
+      ConversionPatternRewriter& rewriter) const {
+    xla_lhlo::BroadcastInDimOpOperandAdaptor operand_adaptor(args);
+    Value operand = operand_adaptor.operand();
+    auto operand_type = operand_adaptor.operand().getType().cast<MemRefType>();
+    auto operand_shape = operand_type.getShape();
+
+    Value result = operand_adaptor.output();
+    auto result_type = result.getType().cast<MemRefType>();
+    auto result_shape = result_type.getShape();
+
+    SmallVector<int64_t, 2> operand_strides;
+    int64_t operand_offset;
+    if (failed(getStridesAndOffset(operand_type, operand_strides,
+                                   operand_offset))) {
+      op.emitOpError() << "Failed to get offset and strides.";
+    }
+
+    SmallVector<int64_t, 2> new_shape, new_strides, broadcast_dims;
+    SmallVector<SmallVector<AffineExpr, 2>, 4> collapsed_dims_list;
+    SmallVector<AffineExpr, 2> collapsed_dims;
+    for (const auto& item :
+         enumerate(op.broadcast_dimensions().getIntValues())) {
+      size_t index = item.index();
+      int dim = item.value().getSExtValue();
+
+      collapsed_dims.push_back(rewriter.getAffineDimExpr(index));
+
+      bool expansion_needed =
+          operand_shape[index] == 1 && result_shape[dim] != 1;
+      if (expansion_needed) {
+        continue;
+      }
+      new_shape.push_back(operand_shape[index]);
+      new_strides.push_back(operand_strides[index]);
+      broadcast_dims.push_back(dim);
+
+      collapsed_dims_list.push_back(collapsed_dims);
+      collapsed_dims.clear();
+    }
+    // If `collapsed_dims_list` is empty, then the memref has shape [1, ..., 1]
+    // and all dimensions need expansion. Such memref will be reshaped to a 1D
+    // memref with a single element. New shape and strides needs to be updated
+    // accordingly.
+    if (collapsed_dims_list.empty()) {
+      collapsed_dims_list.push_back({});
+      new_shape.push_back(1);
+      new_strides.push_back(1);
+      broadcast_dims.push_back(0);
+    }
+    for (const auto& dims : collapsed_dims) {
+      collapsed_dims_list.back().push_back(dims);
+    }
+
+    // `linalg.reshape` is inserted only if necessary, i.e. when the rank can be
+    // reduced.
+    if (new_shape.size() < operand_shape.size()) {
+      SmallVector<ArrayRef<AffineExpr>, 4> reassociation_maps;
+      for (const auto& dims : collapsed_dims_list)
+        reassociation_maps.push_back(dims);
+      auto new_memref_type = MemRefType::get(
+          new_shape, operand_type.getElementType(),
+          makeStridedLinearLayoutMap(new_strides, operand_offset,
+                                     rewriter.getContext()));
+      operand = rewriter.create<linalg::ReshapeOp>(op.getLoc(), new_memref_type,
+                                                   operand_adaptor.operand(),
+                                                   reassociation_maps);
+    }
+    return std::make_pair(operand, broadcast_dims);
+  }
+
+  ArrayAttr getIndexingMapsAttr(xla_lhlo::BroadcastInDimOp op,
+                                ArrayRef<int64_t> broadcastDims,
+                                ArrayRef<int64_t> resultShape,
+                                MemRefType operandType, Builder* b) const {
+    unsigned nloops = resultShape.size();
+
+    // The input is a scalar, i.e. this is a scalar broadcast op.
+    if (operandType.getRank() == 0) {
+      return b->getAffineMapArrayAttr(
+          {AffineMap::get(nloops, /*symbolCount=*/0, b->getContext()),
+           b->getMultiDimIdentityMap(nloops)});
+    }
+
+    auto operandShape = operandType.getShape();
+    SmallVector<AffineExpr, 4> dimExprs;
+    dimExprs.reserve(nloops);
+
+    for (const auto& broadcastDim : llvm::enumerate(broadcastDims)) {
+      int size = broadcastDim.value();
+      bool expansion_needed =
+          operandShape[broadcastDim.index()] == 1 && resultShape[size] != 1;
+      if (expansion_needed) {
+        op.emitOpError(
+            "BroadcastInDimOp lowering to Linalg does not support size-1 "
+            "dimensions expansion.");
+      }
+      dimExprs.push_back(b->getAffineDimExpr(size));
+    }
+    return b->getAffineMapArrayAttr(
+        {AffineMap::get(nloops, /*symbolCount=*/0, dimExprs, b->getContext()),
+         b->getMultiDimIdentityMap(nloops)});
+  }
+};
+
 /// Pattern for the special case where reshape is adding or removing a dimension
 /// of size 1. These can be lowered to a linalg.generic op.
 ///
@@ -639,9 +813,9 @@ void populateLHLOToLinalgConversionPattern(MLIRContext* context,
                                            OwningRewritePatternList* patterns) {
   // clang-format off
   patterns->insert<BroadcastConverter<xla_lhlo::BroadcastOp>,
-                   BroadcastInDimConverter<xla_lhlo::BroadcastInDimOp>,
                    ConstConverter,
                    IotaConverter,
+                   LhloBroadcastInDimConverter,
                    PointwiseToLinalgConverter<xla_lhlo::AbsOp>,
                    PointwiseToLinalgConverter<xla_lhlo::AddOp>,
                    PointwiseToLinalgConverter<xla_lhlo::AndOp>,
@@ -742,7 +916,7 @@ namespace xla_hlo {
 void populateHLOToLinalgConversionPattern(MLIRContext* context,
                                           OwningRewritePatternList* patterns) {
   patterns->insert<BroadcastConverter<xla_hlo::BroadcastOp, false>,
-                   BroadcastInDimConverter<xla_hlo::BroadcastInDimOp, false>,
+                   HloBroadcastInDimConverter,
                    PointwiseToLinalgConverter<xla_hlo::AbsOp, false>,
                    PointwiseToLinalgConverter<xla_hlo::AddOp, false>,
                    PointwiseToLinalgConverter<xla_hlo::AndOp, false>,
diff --git a/tensorflow/compiler/tests/BUILD b/tensorflow/compiler/tests/BUILD
index ea4ba8dab6b..91b3ecdfedb 100644
--- a/tensorflow/compiler/tests/BUILD
+++ b/tensorflow/compiler/tests/BUILD
@@ -1162,7 +1162,7 @@ tf_xla_py_test(
 
 tf_xla_py_test(
     name = "scan_ops_test",
-    size = "small",
+    size = "medium",
     srcs = ["scan_ops_test.py"],
     python_version = "PY3",
     tags = [
diff --git a/tensorflow/compiler/tf2tensorrt/BUILD b/tensorflow/compiler/tf2tensorrt/BUILD
index 3d3eab51268..4a8599e29f6 100644
--- a/tensorflow/compiler/tf2tensorrt/BUILD
+++ b/tensorflow/compiler/tf2tensorrt/BUILD
@@ -555,6 +555,7 @@ cc_library(
     hdrs = ["convert/utils.h"],
     copts = tf_copts(),
     deps = [
+        "@com_google_absl//absl/algorithm:container",
         "//tensorflow/core:framework",
         "//tensorflow/core:lib_proto_parsing",
         "//tensorflow/core:lib",
diff --git a/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc b/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc
index 8ca7c4cdf8f..20ee5ffd8f8 100644
--- a/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc
+++ b/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.cc
@@ -445,16 +445,32 @@ nvinfer1::ITensor* Converter::CreateConstantLayer(
   return trt_tensor;
 }
 
+// Creates a scalar constant and fills with value.
+template <typename T>
+Status CreateScalarConstant(
+    OpConverterParams* params, T value, nvinfer1::ITensor** tensor,
+    nvinfer1::DataType trt_type = nvinfer1::DataType::kINT32,
+    const nvinfer1::Dims& dims = {1, {1}}) {
+  TRT_ShapedWeights weights =
+      params->weight_store->GetTempWeights(trt_type, dims);
+  TF_RETURN_IF_ERROR(weights.SetValues(value));
+  *tensor = params->converter->CreateConstantLayer(weights, dims);
+  TFTRT_RETURN_ERROR_IF_NULLPTR(*tensor, params->node_def.name());
+  params->converter->ProvideQuantizationRange(*tensor, value, value);
+  return Status::OK();
+}
+
+// Creates a constant with the same rank as dims, where each dimension has
+// size = 1.
 Status CreateBroadcastableScalarConstant(OpConverterParams* params, float value,
                                          const nvinfer1::Dims& dims,
                                          nvinfer1::ITensor** tensor,
                                          const char* dtype_attr_name = "T") {
-  nvinfer1::DataType trt_dtype =
-      nvinfer1::DataType::kFLOAT;  // Default to FP32.
+  nvinfer1::DataType trt_type = nvinfer1::DataType::kFLOAT;  // Default to FP32.
   TFAttrs attrs(params->node_def);
   if (attrs.count(dtype_attr_name)) {
     DataType dtype = attrs.get<DataType>(dtype_attr_name);
-    TF_RETURN_IF_ERROR(TfDataTypeToTrt(dtype, &trt_dtype));
+    TF_RETURN_IF_ERROR(TfDataTypeToTrt(dtype, &trt_type));
   }
 
   // In order to be broadcastable, the number of dims has to match.
@@ -462,24 +478,8 @@ Status CreateBroadcastableScalarConstant(OpConverterParams* params, float value,
   for (int i = 0; i < broadcastable_dims.nbDims; i++) {
     broadcastable_dims.d[i] = 1;
   }
-  TRT_ShapedWeights weights =
-      params->weight_store->GetTempWeights(trt_dtype, broadcastable_dims);
-  void* raw_ptr = weights.GetValues();
-  switch (trt_dtype) {
-    case nvinfer1::DataType::kFLOAT:
-      static_cast<float*>(raw_ptr)[0] = value;
-      break;
-    case nvinfer1::DataType::kHALF:
-      static_cast<Eigen::half*>(raw_ptr)[0] = Eigen::half(value);
-      break;
-    default:
-      return errors::InvalidArgument("Unsupported data type ",
-                                     DebugString(trt_dtype));
-  }
-  *tensor = params->converter->CreateConstantLayer(weights, broadcastable_dims);
-  TFTRT_RETURN_ERROR_IF_NULLPTR(*tensor, params->node_def.name());
-  params->converter->ProvideQuantizationRange(*tensor, value, value);
-  return Status::OK();
+  return CreateScalarConstant(params, value, tensor, trt_type,
+                              broadcastable_dims);
 }
 
 // Convert an axis from TF format to TRT format while validating. TF format
@@ -663,6 +663,31 @@ nvinfer1::Weights TRT_ShapedWeights::GetTrtWeights() const {
   return nvinfer1::Weights{type_, GetValues(), count()};
 }
 
+template <typename T>
+Status TRT_ShapedWeights::SetValues(T value) {
+  switch (type_) {
+    case nvinfer1::DataType::kFLOAT: {
+      float* ptr = tensor_.flat<float>().data();
+      std::fill(ptr, ptr + count(), value);
+      break;
+    }
+    case nvinfer1::DataType::kHALF: {
+      Eigen::half* ptr = tensor_.flat<Eigen::half>().data();
+      std::fill(ptr, ptr + count(), Eigen::half(value));
+      break;
+    }
+    case nvinfer1::DataType::kINT32: {
+      int32* ptr = tensor_.flat<int32>().data();
+      std::fill(ptr, ptr + count(), value);
+      break;
+    }
+    default:
+      return errors::InvalidArgument("Unsupported data type ",
+                                     tensorflow::tensorrt::DebugString(type_));
+  }
+  return Status::OK();
+}
+
 size_t TRT_ShapedWeights::size_bytes() const {
   size_t data_type_size = -1;
   switch (type_) {
@@ -1297,10 +1322,14 @@ Status Converter::AddInputTensor(const string& name, nvinfer1::DataType dtype,
   // We verify the batch size only for the input nodes, and rely on individual
   // op converter to ensure the batch size of the outputs is not changed.
   // TODO(laigd): we need to test this properties.
-  Status status = MaybeUpdateBatchSize(batch_size);
-  if (!status.ok()) {
-    return Status(status.code(), StrCat("Batch size doesn't match for tensor ",
-                                        name, ": ", status.error_message()));
+  Status status;
+  if (use_implicit_batch_) {
+    status = MaybeUpdateBatchSize(batch_size);
+    if (!status.ok()) {
+      return Status(status.code(),
+                    StrCat("Batch size doesn't match for tensor ", name, ": ",
+                           status.error_message()));
+    }
   }
   nvinfer1::ITensor* tensor = network()->addInput(name.c_str(), dtype, dims);
   if (tensor == nullptr) {
@@ -1893,32 +1922,34 @@ Status Converter::GetInputs(const NodeDef& node_def,
   return Status::OK();
 }
 
+enum class TrtInputArg { kTensor = 1, kWeight = 2, kBoth = 3 };
+
 // Checks that the number of inputs match, and enforces that the inputs marked
-// as true are constant weights. true means that the input must be a weight,
-// while false means the input must be a tensor. In the future, false will mean
-// the input can be a tensor or weight.
+// as weights are constant. Inputs are allowed to be both weight and tensor.
 Status CheckInputsWeights(
     const OpConverterParams& params,
-    const std::vector<std::pair<string, bool>>& inputs_is_weight) {
+    const std::vector<std::pair<string, TrtInputArg>>& expected_inputs) {
   const auto& inputs = params.inputs;
   const auto& node_def = params.node_def;
-  if (inputs.size() != inputs_is_weight.size()) {
+  if (inputs.size() != expected_inputs.size()) {
     return errors::InvalidArgument(
         node_def.op(), " got ", inputs.size(), " inputs but expected ",
-        inputs_is_weight.size(), ", at ", node_def.name());
+        expected_inputs.size(), ", at ", node_def.name());
   }
   for (int i = 0; i < inputs.size(); i++) {
-    if (inputs_is_weight[i].second && inputs.at(i).is_tensor()) {
-      return errors::Unimplemented("The input \"", inputs_is_weight[i].first,
+    if (expected_inputs[i].second == TrtInputArg::kWeight &&
+        inputs.at(i).is_tensor()) {
+      return errors::Unimplemented("The input \"", expected_inputs[i].first,
                                    "\" for ", node_def.op(),
                                    " must be a constant, at ", node_def.name());
     }
-    // TODO(tmorris): Remove this check and provide a method to automatically
+    // TODO(tfeher): Remove this check and provide a method to automatically
     // retrieve an input as a tensor, converting via CreateConstantLayer if it
     // was originally a weight. We will want a caching mechanism to prevent many
     // duplicate constants from being created.
-    if (!inputs_is_weight[i].second && inputs.at(i).is_weights()) {
-      return errors::Unimplemented("The input \"", inputs_is_weight[i].first,
+    if (expected_inputs[i].second == TrtInputArg::kTensor &&
+        inputs.at(i).is_weights()) {
+      return errors::Unimplemented("The input \"", expected_inputs[i].first,
                                    "\" for ", node_def.op(),
                                    " must be a tensor, at ", node_def.name());
     }
@@ -1926,6 +1957,23 @@ Status CheckInputsWeights(
   return Status::OK();
 }
 
+// Checks that the number of inputs match, and enforces that the inputs marked
+// as true are constant weights. true means that the input must be a weight,
+// while false means the input must be a tensor.
+Status CheckInputsWeights(
+    const OpConverterParams& params,
+    const std::vector<std::pair<string, bool>>& inputs_is_weight) {
+  std::vector<std::pair<string, TrtInputArg>> expected_inputs;
+  expected_inputs.reserve(inputs_is_weight.size());
+  std::transform(
+      inputs_is_weight.begin(), inputs_is_weight.end(),
+      std::back_inserter(expected_inputs), [](std::pair<string, bool> x) {
+        return std::make_pair(
+            x.first, x.second ? TrtInputArg::kWeight : TrtInputArg::kTensor);
+      });
+  return CheckInputsWeights(params, expected_inputs);
+}
+
 Status GetNodeDefTfType(const NodeDef& node_def, DataType* tf_type,
                         const char* type_attr_name) {
   TFAttrs attrs(node_def);
@@ -2451,53 +2499,114 @@ Status ConvertExpandDims(OpConverterParams* params) {
   // ExpandDim's ability to add an axis at end of the shape.
   int trt_axis;
   TF_RETURN_IF_ERROR(ConvertAxis(axis[0], dims.nbDims + 1, node_def.name(),
-                                 /*use_implicit_batch=*/true, &trt_axis));
+                                 params->use_implicit_batch, &trt_axis));
   if (params->validation_only) return Status::OK();
-
-  // ExpandDims: Insert new dim of size 1.
-  input_dims.insert(input_dims.begin() + trt_axis, 1);
-  // Reshape tensor.
-  nvinfer1::Dims new_dims;
-  TF_RETURN_IF_ERROR(TensorShapeArrayToTrtDims(input_dims, &new_dims));
   nvinfer1::ITensor* output_tensor = nullptr;
-  TF_RETURN_IF_ERROR(params->converter->PrepareTensorForShape(
-      input_tensor, new_dims, /*validation_only=*/false, &output_tensor));
+
+  if (!params->use_implicit_batch && !HasStaticShape(input_dims)) {
+    TF_RETURN_IF_ERROR(params->converter->DynamicExpandDims(
+        input_tensor.tensor(), dims, trt_axis, params, &output_tensor));
+  } else {
+    // ExpandDims: Insert new dim of size 1.
+    input_dims.insert(input_dims.begin() + trt_axis, 1);
+    // Reshape tensor.
+    nvinfer1::Dims new_dims;
+    TF_RETURN_IF_ERROR(TensorShapeArrayToTrtDims(input_dims, &new_dims));
+    TF_RETURN_IF_ERROR(params->converter->PrepareTensorForShape(
+        input_tensor, new_dims, /*validation_only=*/false, &output_tensor));
+  }
   params->outputs->push_back(TRT_TensorOrWeights(output_tensor));
   return Status::OK();
 }
 
+Status Converter::DynamicReshape(nvinfer1::ITensor* input,
+                                 std::vector<std::pair<int, int>> slices,
+                                 OpConverterParams* params,
+                                 nvinfer1::ITensor** output,
+                                 std::vector<int> size_for_added_dims) {
+  *output = nullptr;
+  // DynamicReshape relies on INetworkDefinition::addShape that was introduced
+  // in TensorRT 6.
+#if IS_TRT_VERSION_GE(6, 0, 0, 0)
+  if (params->validation_only) {
+    return errors::Internal(
+        "DynamicReshape should not be used during validation");
+  }
+  nvinfer1::ITensor* shape = network()->addShape(*input)->getOutput(0);
+  // Build new shape = shape[:trt_axis] + [1] + shape[trt_axis:]
+  std::vector<nvinfer1::ITensor const*> concat_inputs;
+  for (int i = 0; i < std::max(slices.size(), size_for_added_dims.size());
+       i++) {
+    nvinfer1::ITensor* tensor;
+    // maybe_add_a_dimension(i);
+    if (i < size_for_added_dims.size() && size_for_added_dims[i] >= 0) {
+      TF_RETURN_IF_ERROR(
+          CreateScalarConstant(params, size_for_added_dims[i], &tensor));
+      concat_inputs.push_back(tensor);
+    }
+    if (i < slices.size()) {
+      concat_inputs.push_back(
+          network()
+              ->addSlice(*shape, {1, {slices[i].first}},
+                         {1, {slices[i].second - slices[i].first}}, {1, {1}})
+              ->getOutput(0));
+    }
+  }
+  nvinfer1::IConcatenationLayer* concat_layer = network()->addConcatenation(
+      const_cast<nvinfer1::ITensor* const*>(concat_inputs.data()),
+      concat_inputs.size());
+  concat_layer->setAxis(0);
+  nvinfer1::ITensor* new_shape = concat_layer->getOutput(0);
+  // Reshape input using new shape
+  nvinfer1::IShuffleLayer* shuffle = network()->addShuffle(*input);
+  shuffle->setInput(1, *new_shape);
+  *output = shuffle->getOutput(0);
+  return Status::OK();
+#else
+  return errors::Unavailable(
+      "Dynamic shape input requires TensorRT 6 or above");
+#endif
+}
+
+Status Converter::DynamicExpandDims(nvinfer1::ITensor* input,
+                                    const nvinfer1::Dims& dims, int axis,
+                                    OpConverterParams* params,
+                                    nvinfer1::ITensor** output) {
+  if (params->validation_only) {
+    *output = nullptr;
+    return errors::Internal(
+        "DynamicExpandDims should not be used during validation");
+  }
+  std::vector<std::pair<int, int>> slices;
+  std::vector<int> extra_dims;
+  if (axis != 0) {
+    slices.push_back(std::pair<int, int>{0, axis});
+    extra_dims.push_back(-1);
+  }
+  extra_dims.push_back(1);
+  if (axis != dims.nbDims) {
+    slices.push_back(std::pair<int, int>{axis, dims.nbDims});
+  }
+  return DynamicReshape(input, slices, params, output, extra_dims);
+}
+
 Status Converter::SqueezeTensor(nvinfer1::ITensor* input,
                                 std::vector<int>* input_dims,
+                                OpConverterParams* params,
                                 nvinfer1::ITensor** output) {
-#if IS_TRT_VERSION_GE(6, 0, 0, 0)
   // If the remaining dimensions of a squeeze operation have dynamic sizes, we
   // need to use TRT ops to build the result shape for the squeeze operation.
   // This is because IShuffleLayer::setReshapeDimensions treats -1 as a special
   // value.
-  if (absl::c_any_of(*input_dims, [](int i) { return i == -1; })) {
-    nvinfer1::ITensor* shape = network()->addShape(*input)->getOutput(0);
-    std::vector<nvinfer1::ITensor const*> concat_inputs;
+  if (!params->use_implicit_batch && !HasStaticShape(*input_dims)) {
+    std::vector<std::pair<int, int>> slices;
     for (int i = 0; i < input_dims->size(); i++) {
-      // If input dim wasn't set to 0 earlier, we include it in new shape.
       if (input_dims->at(i) != 0) {
-        concat_inputs.push_back(
-            network()
-                ->addSlice(*shape, {1, {i}}, {1, {1}}, {1, {1}})
-                ->getOutput(0));
+        slices.push_back(std::pair<int, int>(i, i + 1));
       }
     }
-    nvinfer1::IConcatenationLayer* concat_layer = network()->addConcatenation(
-        const_cast<nvinfer1::ITensor* const*>(concat_inputs.data()),
-        concat_inputs.size());
-    concat_layer->setAxis(0);
-    nvinfer1::ITensor* new_shape = concat_layer->getOutput(0);
-    // Reshape input using new shape
-    nvinfer1::IShuffleLayer* shuffle = network()->addShuffle(*input);
-    shuffle->setInput(1, *new_shape);
-    *output = shuffle->getOutput(0);
-    return Status::OK();
+    return DynamicReshape(input, slices, params, output);
   }
-#endif
   // Remove all dims which are equal to 0.
   input_dims->erase(std::remove(input_dims->begin(), input_dims->end(), 0),
                     input_dims->end());
@@ -2564,7 +2673,7 @@ Status ConvertSqueeze(OpConverterParams* params) {
 
   nvinfer1::ITensor* output_tensor = nullptr;
   TF_RETURN_IF_ERROR(params->converter->SqueezeTensor(
-      input_tensor.tensor(), &input_dims, &output_tensor));
+      input_tensor.tensor(), &input_dims, params, &output_tensor));
   params->outputs->push_back(TRT_TensorOrWeights(output_tensor));
   return Status::OK();
 }
@@ -4149,8 +4258,13 @@ Status ConvertBinary(OpConverterParams* params) {
                                    " inputs but expected 2, at ",
                                    node_def.name());
   }
-  TF_RETURN_IF_ERROR(
-      AllowDataTypes(*params, {DataType::DT_FLOAT, DataType::DT_HALF}));
+#if IS_TRT_VERSION_GE(6, 0, 0, 0)
+  std::set<DataType> allowed_types{DataType::DT_FLOAT, DataType::DT_HALF,
+                                   DataType::DT_INT32};
+#else
+  std::set<DataType> allowed_types{DataType::DT_FLOAT, DataType::DT_HALF};
+#endif
+  TF_RETURN_IF_ERROR(AllowDataTypes(*params, allowed_types));
 
   // Constant folding should have been done by TensorFlow
   if (inputs.at(0).is_weights() && inputs.at(1).is_weights()) {
@@ -4896,24 +5010,14 @@ Status ConvertGather(OpConverterParams* params) {
   const auto& node_def = params->node_def;
   // TODO(tmorris): Use CheckInputsWeights by changing bool to enum with an
   // option for an input to be either tensor or weight.
-  if (inputs.size() != 3) {
-    return errors::InvalidArgument("GatherV2 got ", inputs.size(),
-                                   " inputs but expected 3, at ",
-                                   node_def.name());
-  }
+  TF_RETURN_IF_ERROR(
+      CheckInputsWeights(*params, {{"params", TrtInputArg::kBoth},
+                                   {"indices", TrtInputArg::kTensor},
+                                   {"axis", TrtInputArg::kWeight}}));
+
   const auto& params_input = inputs.at(0);
   const auto& indices_input = inputs.at(1);
   const auto& axis_input = inputs.at(2);
-  if (!axis_input.is_weights()) {
-    return errors::Unimplemented(
-        "The input \"axis\" for GatherV2 must be a constant, at ",
-        node_def.name());
-  }
-  if (!indices_input.is_tensor()) {
-    return errors::Unimplemented(
-        "The input \"indices\" for GatherV2 must be a tensor, at ",
-        node_def.name());
-  }
 
   TF_RETURN_IF_ERROR(AllowDataTypes(
       *params, {DataType::DT_FLOAT, DataType::DT_HALF, DataType::DT_INT32},
@@ -4927,14 +5031,16 @@ Status ConvertGather(OpConverterParams* params) {
                                    node_def.name());
   }
   int trt_axis = 0;
-  TF_RETURN_IF_ERROR(ConvertAxis(axis[0], params_input.GetTrtDims().nbDims,
-                                 node_def.name(), params_input.is_tensor(),
-                                 &trt_axis));
-  if (params_input.is_weights() && trt_axis != 0) {
+  TF_RETURN_IF_ERROR(ConvertAxis(
+      axis[0], params_input.GetTrtDims().nbDims, node_def.name(),
+      params->use_implicit_batch && params_input.is_tensor(), &trt_axis));
+  if (params->use_implicit_batch && params_input.is_weights() &&
+      trt_axis != 0) {
     return errors::Unimplemented(
         "The input axis must be zero when params is a weight.");
   }
-  if (params_input.is_tensor() && indices_input.batch_size() != 1) {
+  if (params->use_implicit_batch && params_input.is_tensor() &&
+      indices_input.batch_size() != 1) {
     return errors::Unimplemented(
         "Indices must have a batch size of 1 when params is a tensor.");
   }
@@ -4943,10 +5049,13 @@ Status ConvertGather(OpConverterParams* params) {
   // where "+ 1" adds the batch dim. If params is a weight, the TRT rank matches
   // the TF rank so we don't have to add + 1.
   const int params_tf_rank =
-      params_input.GetTrtDims().nbDims + (params_input.is_tensor() ? 1 : 0);
-  const int indices_tf_rank = indices_input.GetTrtDims().nbDims + 1;
+      params_input.GetTrtDims().nbDims +
+      (params->use_implicit_batch && params_input.is_tensor() ? 1 : 0);
+  const int indices_tf_rank =
+      indices_input.GetTrtDims().nbDims + (params->use_implicit_batch ? 1 : 0);
   const int tf_gather_output_rank = params_tf_rank + indices_tf_rank - 1;
-  if (tf_gather_output_rank > nvinfer1::Dims::MAX_DIMS + 1) {
+  if (tf_gather_output_rank >
+      nvinfer1::Dims::MAX_DIMS + (params->use_implicit_batch ? 1 : 0)) {
     return errors::InvalidArgument(
         "Result of gather has dimension greater than ",
         nvinfer1::Dims::MAX_DIMS + 1);
@@ -4978,7 +5087,8 @@ Status ConvertGather(OpConverterParams* params) {
   // because of the implicit batch dim in the indices (see the above note).
   const int expected_trt_output_rank =
       tf_gather_output_rank - (params_input.is_tensor() ? 2 : 1);
-  if (trt_gather_output_dims.nbDims != expected_trt_output_rank) {
+  if (params->use_implicit_batch &&
+      trt_gather_output_dims.nbDims != expected_trt_output_rank) {
     return errors::Internal(
         "Get unexpected output dimensions of IGatherLayer. Expect nbDims: ",
         expected_trt_output_rank,
@@ -4986,7 +5096,7 @@ Status ConvertGather(OpConverterParams* params) {
   }
   // Reshape the output so after adding the implicit batch dim it'll match the
   // output shape of TF GatherV2.
-  if (params_input.is_tensor()) {
+  if (params->use_implicit_batch && params_input.is_tensor()) {
     for (int i = trt_gather_output_dims.nbDims; i > trt_axis; --i) {
       trt_gather_output_dims.d[i] = trt_gather_output_dims.d[i - 1];
     }
diff --git a/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.h b/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.h
index 2fe8eec9675..7a1276c645c 100644
--- a/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.h
+++ b/tensorflow/compiler/tf2tensorrt/convert/convert_nodes.h
@@ -185,6 +185,10 @@ class TRT_ShapedWeights {
     return const_cast<char*>(tensor_.tensor_data().data());
   }
 
+  // Fills all the weight values with value.
+  template <typename T>
+  Status SetValues(T value);
+
   int64_t count() const;
 
   size_t size_bytes() const;
@@ -529,12 +533,62 @@ class Converter {
                                const bool validation_only,
                                nvinfer1::ITensor** tensor);
 
+  // Reshapes a dynamic shape tensor by removing or adding dimensions of size 1,
+  // and/or permuting the dimensions. The new shape is derived from the shape of
+  // the input tensor according to the slices and size_for_added_dims arguments.
+  //
+  // If there would be at most one unknown dimension, we could set the new shape
+  // using IShuffleLayer::setReshapeDimensions, which treats -1 as a special
+  // value (the same way as TF). In general, we can have more than one unknown
+  // dimensions, and we have to manipulate the shape tensors during runtime to
+  // define the new shape. This helper function defines the necessary shape
+  // inference layers and calls reshape using the calculated new shape.
+  //
+  // Example:
+  //
+  // Assume that we want to reshape a tensor from shape {A,B,C,D} to {C,D,A,B}
+  // (no transpose, just change the shape). In dynamic shape mode, the A,B,C,D
+  // values are not necessarily known at conversion time, they can be all -1. We
+  // can only define the new shape at runtime, when the actual shape is already
+  // known. To define the new shape:
+  // - We use an IShapeLayer to retrieve a shape tensor with the {A,B,C,D}
+  //   values.
+  // - Create two slices {C,D} and {A,B} of the shape tensor.
+  // - Concatenate these slices {C,D,A,B},
+  // - Set the {C,D,A,B} shape tensor as an input shape tensor for
+  // IShuffleLayer.
+  //
+  // This can be achieved by calling DynamicReshape(input, {{2,4},{0,2}},
+  // params).
+  //
+  // Before each slice we can insert a new dim if the corresponding
+  // size_for_added_dims element is not negative. The size_for_added_dims array
+  // can have more than slices.size() elements, in order to insert a dimension
+  // ater the last slice.
+  //
+  // Parameters:
+  // input - input tensor
+  // slices - [start, end) pairs of slices
+  // params - conversion parameters
+  // output - reshaped tensor
+  // size_for_added_dims - size of dimension inserted right before slice[i]. We
+  //   only insert a new dim if size_for_added_dims[i] >= 0.
+  Status DynamicReshape(nvinfer1::ITensor* input,
+                        std::vector<std::pair<int, int>> slices,
+                        OpConverterParams* params, nvinfer1::ITensor** output,
+                        std::vector<int> size_for_added_dims = {});
+
+  // Inserts a singleton dimension at axis for a dynamic shape tensor.
+  Status DynamicExpandDims(nvinfer1::ITensor* input, const nvinfer1::Dims& dims,
+                           int axis, OpConverterParams* params,
+                           nvinfer1::ITensor** output);
+
   // Helper function to add a squeeze op to the network.
   //
   // The input_dims argument stores the TRT dimensions of the input tensor,
   // where the dimensions to be squeezed are replaced by 0.
   Status SqueezeTensor(nvinfer1::ITensor* input, std::vector<int>* input_dims,
-                       nvinfer1::ITensor** output);
+                       OpConverterParams* params, nvinfer1::ITensor** output);
 
   // Creates an IConstantLayer using 'weights' whose dimensions are specified by
   // 'dims', and returns the output ITensor.
@@ -632,6 +686,9 @@ Status GetTrtBroadcastShape(const TRT_TensorOrWeights& operand_l,
 const std::unordered_map<string, nvinfer1::UnaryOperation>* UnaryOperationMap();
 // Map of all supported ActivationTypes
 const std::unordered_map<string, nvinfer1::ActivationType>* ActivationTypeMap();
+// Map of all supported BinaryOperations
+const std::unordered_map<string, nvinfer1::ElementWiseOperation>*
+BinaryOperationMap();
 
 }  // namespace convert
 }  // namespace tensorrt
diff --git a/tensorflow/compiler/tf2tensorrt/convert/convert_nodes_test.cc b/tensorflow/compiler/tf2tensorrt/convert/convert_nodes_test.cc
index f9b0cafe253..450831910f6 100644
--- a/tensorflow/compiler/tf2tensorrt/convert/convert_nodes_test.cc
+++ b/tensorflow/compiler/tf2tensorrt/convert/convert_nodes_test.cc
@@ -1359,25 +1359,25 @@ class OpConverterTest : public ::testing::Test {
   }
 
   // Constructs a tensor with given values (vals). The tensor type is defined by
-  // the tf_dtype argument, its shape is given by input_dims. The tensor is
+  // the tf_type argument, its shape is given by input_dims. The tensor is
   // constructed using the allocator of OpConverterTest in Unified Memory.
   template <typename T>
   Tensor AsTensor(std::vector<T> vals, const std::vector<int> input_dims,
-                  DataType tf_dtype) {
-    Tensor ret(allocator_.get(), tf_dtype, {static_cast<int64>(vals.size())});
-    if (tf_dtype == DT_FLOAT) {
+                  DataType tf_type) {
+    Tensor ret(allocator_.get(), tf_type, {static_cast<int64>(vals.size())});
+    if (tf_type == DT_FLOAT) {
       auto conv_vals = CastTestVector<T, float>(vals);
       std::copy_n(conv_vals.data(), conv_vals.size(), ret.flat<float>().data());
-    } else if (tf_dtype == DT_HALF) {
+    } else if (tf_type == DT_HALF) {
       auto conv_vals = CastTestVector<T, Eigen::half>(vals);
       std::copy_n(conv_vals.data(), conv_vals.size(),
                   ret.flat<Eigen::half>().data());
-    } else if (tf_dtype == DT_INT32) {
+    } else if (tf_type == DT_INT32) {
       auto conv_vals = CastTestVector<T, int32>(vals);
       std::copy_n(conv_vals.data(), conv_vals.size(), ret.flat<int32>().data());
     } else {
       LOG(FATAL) << "Cannot create tensor with type "
-                 << DataTypeString(tf_dtype);
+                 << DataTypeString(tf_type);
     }
     TensorShape shape;
     TF_EXPECT_OK(TensorShapeUtils::MakeShape(input_dims, &shape));
@@ -1394,9 +1394,9 @@ class OpConverterTest : public ::testing::Test {
 
   // Constructs a flat tensor in Unified Memory.
   template <typename T>
-  Tensor ConstructTensor(int data_size, const T& value, DataType tf_dtype) {
+  Tensor ConstructTensor(int data_size, const T& value, DataType tf_type) {
     std::vector<T> values(data_size, value);
-    return AsTensor<T>(values, {data_size}, tf_dtype);
+    return AsTensor<T>(values, {data_size}, tf_type);
   }
 
   void CheckDataTypeMatches(const DataVec& datas) {
@@ -1405,10 +1405,10 @@ class OpConverterTest : public ::testing::Test {
       ASSERT_NE(-1, input_index);
       const nvinfer1::DataType trt_dtype =
           engine_->getBindingDataType(input_index);
-      const DataType tf_dtype = TrtDataTypeToTf(trt_dtype);
-      ASSERT_EQ(data.tensor.dtype(), tf_dtype)
+      const DataType tf_type = TrtDataTypeToTf(trt_dtype);
+      ASSERT_EQ(data.tensor.dtype(), tf_type)
           << DataTypeString(data.tensor.dtype()) << " vs. "
-          << DataTypeString(tf_dtype);
+          << DataTypeString(tf_type);
     }
   }
 
@@ -1489,12 +1489,13 @@ class OpConverterTest : public ::testing::Test {
   // dimension is included in dims (ie for an NCHW tensor dims = {N, C, H, W}).
   void AddTestTensorWithTFDims(
       const string& name, const std::vector<int32>& dims,
-      nvinfer1::DataType trt_dtype = nvinfer1::DataType::kFLOAT) {
-    DataType tf_dtype = TrtDataTypeToTf(trt_dtype);
+      nvinfer1::DataType trt_type = nvinfer1::DataType::kFLOAT,
+      Status add_input_status = Status::OK()) {
+    DataType tf_type = TrtDataTypeToTf(trt_type);
     ops::Placeholder::Attrs attrs;
     TF_EXPECT_OK(TensorShapeUtils::MakeShape(dims, &attrs.shape_));
 
-    auto input = ops::Placeholder(scope_.WithOpName(name), tf_dtype, attrs);
+    auto input = ops::Placeholder(scope_.WithOpName(name), tf_type, attrs);
     node_inputs_[name] = input.output;
 
     // Add a real ITensor for conversion conditionally.
@@ -1502,8 +1503,9 @@ class OpConverterTest : public ::testing::Test {
         TensorShapeToTrtDims(attrs.shape_, converter_->use_implicit_batch());
     if (!converter_->use_implicit_batch() || HasStaticShape(trt_dims)) {
       int batch_size = dims[0];
-      TF_EXPECT_OK(
-          converter_->AddInputTensor(name, trt_dtype, trt_dims, batch_size));
+      Status status =
+          converter_->AddInputTensor(name, trt_type, trt_dims, batch_size);
+      ASSERT_EQ(add_input_status, status);
     }
   }
 
@@ -1552,24 +1554,23 @@ class OpConverterTest : public ::testing::Test {
         converter_->AddTensorOrWeights(name, TRT_TensorOrWeights{weights}));
   }
 
-  template <typename T>
+  template <typename T = int32>
   void AddTestWeights(const string& name, const std::vector<int>& dims,
-                      const std::vector<T>& values, DataType tf_dtype) {
-    if (tf_dtype == DT_FLOAT) {
+                      const std::vector<T>& values, DataType tf_type) {
+    if (tf_type == DT_FLOAT) {
       AddTestWeights(name, dims, CastTestVector<T, float>(values));
-    } else if (tf_dtype == DT_HALF) {
+    } else if (tf_type == DT_HALF) {
       AddTestWeights(name, dims, CastTestVector<T, Eigen::half>(values));
-    } else if (tf_dtype == DT_INT32) {
+    } else if (tf_type == DT_INT32) {
       AddTestWeights(name, dims, CastTestVector<T, int32>(values));
     } else {
       FAIL() << "Cannot create test weights with type "
-             << DataTypeString(tf_dtype);
+             << DataTypeString(tf_type);
     }
   }
 
   // Test validation in validation-only mode.
-  void RunValidation(const Node* node, error::Code expected_code = error::OK,
-                     const char* expected_msg_substr = nullptr) {
+  Status RunValidation(const Node* node) {
     grappler::GrapplerItem item;
     TF_EXPECT_OK(scope_.ToGraphDef(&item.graph));
     grappler::GraphProperties graph_properties(item);
@@ -1578,8 +1579,7 @@ class OpConverterTest : public ::testing::Test {
     TrtNodeValidator validator(graph_properties, converter_->precision_mode(),
                                /*use_calibration=*/false,
                                converter_->use_implicit_batch());
-    ExpectStatus(validator.IsTensorRTCandidate(node), expected_code,
-                 expected_msg_substr);
+    return validator.IsTensorRTCandidate(node);
   }
 
   void RunConversion(const Node* node, error::Code expected_code = error::OK,
@@ -1610,9 +1610,11 @@ class OpConverterTest : public ::testing::Test {
       graph->AddEdge(input.node(), input.index(), node, i);
     }
 
-    RunValidation(node, expected_code, expected_msg_substr);
-    if (should_run_conversion) {
+    status = RunValidation(node);
+    if (should_run_conversion && status.ok()) {
       RunConversion(node, expected_code, expected_msg_substr);
+    } else {
+      ExpectStatus(status, expected_code, expected_msg_substr);
     }
   }
 
@@ -1717,7 +1719,7 @@ class ParameterizedOpConverterTestBase
  public:
   ParameterizedOpConverterTestBase()
       : trt_mode(std::get<0>(GetParam())),
-        tf_dtype(std::get<1>(GetParam())),
+        tf_type(std::get<1>(GetParam())),
         converter_precision(std::get<2>(GetParam())) {}
 
   void Reset() {
@@ -1744,11 +1746,15 @@ class ParameterizedOpConverterTestBase
   //   be empty, in that case the partial_input_shape will be set automatically
   //   depending on the trt_mode argument. (This argument also includes explicit
   //   batch dim).
+  // - add_input_status adding ITensor to the network can fail in implicit batch
+  //   mode if the batch size is inconsistent. Using the add_input_status arg we
+  //   can test such errors.
   //
-  template <typename T>
+  template <typename T = int>
   void AddTestTensor(const string& name, const std::vector<int32>& dims,
-                     DataType tf_dtype, const std::vector<T>& values,
-                     const std::vector<int32>& partial_input_shape_dims = {}) {
+                     DataType tf_type, const std::vector<T>& values,
+                     const std::vector<int32>& partial_input_shape_dims = {},
+                     Status add_input_status = Status::OK()) {
     std::vector<int32> partial_shape;
     if (!partial_input_shape_dims.empty()) {
       partial_shape = partial_input_shape_dims;
@@ -1761,24 +1767,25 @@ class ParameterizedOpConverterTestBase
         partial_shape = dims;
       }
     }
-    AddTestTensorWithTFDims(name, partial_shape, TfDataTypeToTrt(tf_dtype));
+    AddTestTensorWithTFDims(name, partial_shape, TfDataTypeToTrt(tf_type),
+                            add_input_status);
     if (!values.empty()) {
       VLOG(2) << "Adding test tensor: " << name << " "
-              << DataTypeString(tf_dtype);
-      InputOutputData data{name, AsTensor(values, dims, tf_dtype)};
+              << DataTypeString(tf_type);
+      InputOutputData data{name, AsTensor(values, dims, tf_type)};
       VLOG(2) << "Added tensor: " << data.name
               << DataTypeString(data.tensor.dtype());
       input_data_.push_back(data);
     }
   }
 
-  // Adds test tensor (same as above) but with the default tf_dtype defined by
+  // Adds test tensor (same as above) but with the default tf_type defined by
   // the test params.
+  template <typename T = int>
   void AddTestTensor(const string& name, const std::vector<int32>& dims,
-                     const std::vector<float>& values = {},
+                     const std::vector<T>& values = {},
                      const std::vector<int32>& partial_input_shape_dims = {}) {
-    AddTestTensor<float>(name, dims, tf_dtype, values,
-                         partial_input_shape_dims);
+    AddTestTensor<T>(name, dims, tf_type, values, partial_input_shape_dims);
   }
 
   // Builds and runs the converted network. Checks output tensor shape. Tests
@@ -1797,7 +1804,7 @@ class ParameterizedOpConverterTestBase
           TensorShapeUtils::MakeShape(expected_output_dims[i], &shape));
       string out_name = (n_output == 1) ? name : StrCat(name, ":", i);
       InputOutputData data{out_name,
-                           ConstructTensor(shape.num_elements(), 0, tf_dtype)};
+                           ConstructTensor(shape.num_elements(), 0, tf_type)};
       output_data.push_back(data);
     }
     ASSERT_FALSE(input_data_.empty());
@@ -1836,7 +1843,7 @@ class ParameterizedOpConverterTestBase
 
  protected:
   const TrtTestMode trt_mode;
-  const DataType tf_dtype;
+  const DataType tf_type;
   const TrtPrecisionMode converter_precision;
   DataVec input_data_;
 };
@@ -1869,6 +1876,15 @@ INSTANTIATE_TEST_CASE_P(
     ::testing::Combine(::testing::ValuesIn(ValidTrtModes),
                        ::testing::Values(DT_FLOAT, DT_HALF),
                        ::testing::Values(TrtPrecisionMode::FP32)));
+
+// Base class for tests that need to be tested for FP32, FP16, and INT32
+class OpConverterTest3 : public ParameterizedOpConverterTestBase {};
+INSTANTIATE_TEST_CASE_P(
+    OpConvTestInstantiation3, OpConverterTest3,
+    ::testing::Combine(::testing::ValuesIn(ValidTrtModes),
+                       ::testing::Values(DT_FLOAT, DT_HALF, DT_INT32),
+                       ::testing::Values(TrtPrecisionMode::FP32)));
+
 template <typename T>
 void CopyTensorElements(const Tensor& tensor, protobuf::RepeatedField<T>* out) {
   out->Clear();
@@ -2009,7 +2025,7 @@ TEST_F(OpConverterTest, ConvertConst) {
 TEST_P(OpConverterTest1, ConvertTranspose) {
   // Get the NodeDef for Transpose.
   Scope s = Scope::NewRootScope();
-  auto input = ops::Placeholder(s.WithOpName("input"), tf_dtype);
+  auto input = ops::Placeholder(s.WithOpName("input"), tf_type);
   auto weights = ops::Placeholder(s.WithOpName("weights"), DT_INT32);
   auto transpose = ops::Transpose(s.WithOpName("my_transpose"), input, weights);
   const NodeDef& node_def = transpose.operation.node()->def();
@@ -2408,10 +2424,10 @@ TEST_P(OpConverterTest2, ConvertBiasAdd) {
   // DT_INT32 type here. DT_FLOAT and DT_HALF are tested.
   // Get the NodeDef for BiasAdd.
   auto get_biasadd_nodedef = [](const string& data_format,
-                                DataType tf_dtype) -> NodeDef {
+                                DataType tf_type) -> NodeDef {
     Scope s = Scope::NewRootScope();
-    auto input = ops::Placeholder(s.WithOpName("input"), tf_dtype);
-    auto weights = ops::Placeholder(s.WithOpName("weights"), tf_dtype);
+    auto input = ops::Placeholder(s.WithOpName("input"), tf_type);
+    auto weights = ops::Placeholder(s.WithOpName("weights"), tf_type);
     const auto biasadd_attrs = ops::BiasAdd::DataFormat(data_format);
     auto biasadd =
         ops::BiasAdd(s.WithOpName("my_biasadd"), input, weights, biasadd_attrs);
@@ -2421,7 +2437,7 @@ TEST_P(OpConverterTest2, ConvertBiasAdd) {
   for (const string& data_format : {"NHWC", "NCHW"}) {
     for (const int trt_input_rank : {1, 2, 3, 4}) {
       Reset();
-      NodeDef node_def = get_biasadd_nodedef(data_format, tf_dtype);
+      NodeDef node_def = get_biasadd_nodedef(data_format, tf_type);
 
       // Add input, dims_array will be like {2, 1, ..., 1, 3}
       std::vector<int32> dims_array(trt_input_rank + 1, 1);
@@ -2443,7 +2459,7 @@ TEST_P(OpConverterTest2, ConvertBiasAdd) {
       for (int i = 0; i < channel_size; ++i) {
         bias[i] = i + 1;  // bias will be {1, 2, 3, ...}
       }
-      AddTestWeights("weights", {channel_size}, bias, tf_dtype);
+      AddTestWeights("weights", {channel_size}, bias, tf_type);
 
       // Build and run the engine.
       std::vector<float> output_data;
@@ -2468,103 +2484,18 @@ TEST_P(OpConverterTest2, ConvertBiasAdd) {
 }
 
 template <typename OpType>
-NodeDef GetBinaryOpNodeDef(const string& input_name_l,
-                           const string& input_name_r, DataType dtype) {
+NodeDef GetBinaryOpNodeDef(DataType dtype) {
   Scope s = Scope::NewRootScope();
-  auto input_l = ops::Placeholder(s.WithOpName(input_name_l), dtype);
-  auto input_r = ops::Placeholder(s.WithOpName(input_name_r), dtype);
+  auto input_l = ops::Placeholder(s.WithOpName("input1"), dtype);
+  auto input_r = ops::Placeholder(s.WithOpName("input2"), dtype);
   auto op = OpType(s.WithOpName("my_binary"), input_l, input_r);
   return op.operation.node()->def();
 }
 
-template <typename OpType, DataType dtype>
-void TestBinaryOp(OpConverterTest* test, bool operand_1_is_tensor,
-                  bool operand_2_is_tensor) {
-  typedef typename EnumToDataType<dtype>::Type CType;
-  test->Reset();
-  const NodeDef node_def =
-      GetBinaryOpNodeDef<OpType>("input1", "input2", dtype);
-  if (operand_1_is_tensor) {
-    test->AddTestTensor("input1", /*dims=*/{1, 2}, /*batch_size=*/2,
-                        TfDataTypeToTrt(dtype));
-  } else {
-    test->AddTestWeights("input1", /*dims=*/{1, 2},
-                         /*values=*/std::vector<CType>{CType(3), CType(6)});
-  }
-  if (operand_2_is_tensor) {
-    test->AddTestTensor("input2", /*dims=*/{2, 1}, /*batch_size=*/2,
-                        TfDataTypeToTrt(dtype));
-  } else {
-    test->AddTestWeights("input2", /*dims=*/{2, 1},
-                         /*values=*/std::vector<CType>{CType(2), CType(3)});
-  }
-  test->RunValidationAndConversion(node_def);
-
-  DataVec input_data;
-  if (operand_1_is_tensor) {
-    input_data.push_back(
-        {"input1",
-         test->AsTensor<CType>({CType(3), CType(6), CType(3), CType(6)})});
-  }
-  if (operand_2_is_tensor) {
-    input_data.push_back(
-        {"input2",
-         test->AsTensor<CType>({CType(2), CType(3), CType(2), CType(3)})});
-  }
-  DataVec output_data{{"my_binary", test->ConstructTensor<CType>(8)}};
-  // Check output dims.
-  TRT_TensorOrWeights output;
-  TF_EXPECT_OK(test->GetTensorOrWeights("my_binary", &output));
-  ASSERT_TRUE(output.is_tensor());
-  ExpectTrtDimsEqualsArray({2, 2}, output.tensor()->getDimensions());
-  // After broadcasting first input becomes {3, 6, 3, 6} and second input
-  // becomes {2, 3, 2, 3}.
-  TF_EXPECT_OK(test->BuildAndRun(input_data, &output_data, /*batch_size=*/2));
-  if (node_def.op() == "Add") {
-    EXPECT_THAT(
-        GetSpanForData<CType>(output_data[0]),
-        ElementsAreArray(CastTestVector<int, CType>({5, 8, 6, 9, 5, 8, 6, 9})));
-  } else if (node_def.op() == "Sub") {
-    EXPECT_THAT(
-        GetSpanForData<CType>(output_data[0]),
-        ElementsAreArray(CastTestVector<int, CType>({1, 4, 0, 3, 1, 4, 0, 3})));
-  } else if (node_def.op() == "Mul") {
-    EXPECT_THAT(GetSpanForData<CType>(output_data[0]),
-                ElementsAreArray(
-                    CastTestVector<int, CType>({6, 12, 9, 18, 6, 12, 9, 18})));
-  } else if (node_def.op() == "Div") {
-    EXPECT_THAT(GetSpanForData<CType>(output_data[0]),
-                ElementsAreArray(CastTestVector<float, CType>(
-                    {1.5, 3, 1, 2, 1.5, 3, 1, 2})));
-  } else if (node_def.op() == "RealDiv") {
-    EXPECT_THAT(GetSpanForData<CType>(output_data[0]),
-                ElementsAreArray(CastTestVector<float, CType>(
-                    {1.5, 3, 1, 2, 1.5, 3, 1, 2})));
-  } else if (node_def.op() == "FloorDiv") {
-    EXPECT_THAT(GetSpanForData<CType>(output_data[0]),
-                ElementsAreArray(
-                    CastTestVector<float, CType>({1, 3, 1, 2, 1, 3, 1, 2})));
-  } else if (node_def.op() == "Minimum") {
-    EXPECT_THAT(
-        GetSpanForData<CType>(output_data[0]),
-        ElementsAreArray(CastTestVector<int, CType>({2, 2, 3, 3, 2, 2, 3, 3})));
-  } else if (node_def.op() == "Maximum") {
-    EXPECT_THAT(
-        GetSpanForData<CType>(output_data[0]),
-        ElementsAreArray(CastTestVector<int, CType>({3, 6, 3, 6, 3, 6, 3, 6})));
-  } else if (node_def.op() == "Pow") {
-    ExpectArrayNear(
-        CastTestVector<int, CType>({9, 36, 27, 216, 9, 36, 27, 216}),
-        GetSpanForData<CType>(output_data[0]));
-  } else {
-    ASSERT_TRUE(false);
-  }
-}
-
-TEST_F(OpConverterTest, ConvertBinary) {
-  AttrValue dtype;
-  dtype.set_type(DT_FLOAT);
+TEST_P(OpConverterTest2, ConvertBinary) {
   {
+    AttrValue dtype;
+    dtype.set_type(tf_type);
     // Both inputs are weights.
     Reset();
     NodeDef node_def =
@@ -2577,46 +2508,56 @@ TEST_F(OpConverterTest, ConvertBinary) {
         "both input as constant at: my_add");
   }
 
+  using OpFunc = std::function<NodeDef(DataType)>;
+  std::map<std::string, std::pair<OpFunc, std::vector<float>>> op_test_info;
+#define ADD_OP(name, op, v1, v2, v3, v4, v5, v6, v7, v8) \
+  op_test_info[name] =                                   \
+      std::make_pair(GetBinaryOpNodeDef<op>,             \
+                     std::vector<float>(v1, v2, v3, v4, v5, v6, v7, v8))
+  ADD_OP("Add", ops::Add, {5, 8, 6, 9, 5, 8, 6, 9});
+  ADD_OP("AddV2", ops::AddV2, {5, 8, 6, 9, 5, 8, 6, 9});
+  ADD_OP("Sub", ops::Sub, {1, 4, 0, 3, 1, 4, 0, 3});
+  ADD_OP("Mul", ops::Mul, {6, 12, 9, 18, 6, 12, 9, 18});
+  ADD_OP("Div", ops::Div, {1.5, 3, 1, 2, 1.5, 3, 1, 2});
+  ADD_OP("RealDiv", ops::RealDiv, {1.5, 3, 1, 2, 1.5, 3, 1, 2});
+  ADD_OP("FloorDiv", ops::FloorDiv, {1, 3, 1, 2, 1, 3, 1, 2});
+  ADD_OP("Minimum", ops::Minimum, {2, 2, 3, 3, 2, 2, 3, 3});
+  ADD_OP("Maximum", ops::Maximum, {3, 6, 3, 6, 3, 6, 3, 6});
+  ADD_OP("Pow", ops::Pow, {9, 36, 27, 216, 9, 36, 27, 216});
+#undef ADD_OP
+  // Add all ops supported by ConvertBinary.
+  auto* supported_ops = BinaryOperationMap();
   // Test combinations of tensor vs weight inputs (except when both inputs are
   // weights).
   for (const bool operand_1_is_tensor : {true, false}) {
     for (const bool operand_2_is_tensor : {true, false}) {
       if (!operand_1_is_tensor && !operand_2_is_tensor) continue;
-      // FP32 tests
-      TestBinaryOp<ops::Add, DT_FLOAT>(this, operand_1_is_tensor,
-                                       operand_2_is_tensor);
-      TestBinaryOp<ops::Sub, DT_FLOAT>(this, operand_1_is_tensor,
-                                       operand_2_is_tensor);
-      TestBinaryOp<ops::Mul, DT_FLOAT>(this, operand_1_is_tensor,
-                                       operand_2_is_tensor);
-      TestBinaryOp<ops::Div, DT_FLOAT>(this, operand_1_is_tensor,
-                                       operand_2_is_tensor);
-      TestBinaryOp<ops::RealDiv, DT_FLOAT>(this, operand_1_is_tensor,
-                                           operand_2_is_tensor);
-      TestBinaryOp<ops::Minimum, DT_FLOAT>(this, operand_1_is_tensor,
-                                           operand_2_is_tensor);
-      TestBinaryOp<ops::Maximum, DT_FLOAT>(this, operand_1_is_tensor,
-                                           operand_2_is_tensor);
-      TestBinaryOp<ops::Pow, DT_FLOAT>(this, operand_1_is_tensor,
-                                       operand_2_is_tensor);
-      // FP16 tests
-      // TODO(tmorris): Use templates to avoid duplication.
-      TestBinaryOp<ops::Add, DT_HALF>(this, operand_1_is_tensor,
-                                      operand_2_is_tensor);
-      TestBinaryOp<ops::Sub, DT_HALF>(this, operand_1_is_tensor,
-                                      operand_2_is_tensor);
-      TestBinaryOp<ops::Mul, DT_HALF>(this, operand_1_is_tensor,
-                                      operand_2_is_tensor);
-      TestBinaryOp<ops::Div, DT_HALF>(this, operand_1_is_tensor,
-                                      operand_2_is_tensor);
-      TestBinaryOp<ops::RealDiv, DT_HALF>(this, operand_1_is_tensor,
-                                          operand_2_is_tensor);
-      TestBinaryOp<ops::Minimum, DT_HALF>(this, operand_1_is_tensor,
-                                          operand_2_is_tensor);
-      TestBinaryOp<ops::Maximum, DT_HALF>(this, operand_1_is_tensor,
-                                          operand_2_is_tensor);
-      TestBinaryOp<ops::Pow, DT_HALF>(this, operand_1_is_tensor,
-                                      operand_2_is_tensor);
+      for (auto& iter : *supported_ops) {
+        string op_name = iter.first;
+        SCOPED_TRACE(StrCat(op_name, "_", operand_1_is_tensor ? "T" : "W",
+                            operand_2_is_tensor ? "T" : "W"));
+        Reset();
+        if (!op_test_info.count(op_name)) {
+          FAIL() << "Binary op test map does not contain op " << op_name;
+        }
+        NodeDef node_def = op_test_info[op_name].first(tf_type);
+        std::vector<std::string> input_names;
+        std::vector<std::vector<int>> input_dims;
+        std::vector<std::vector<float>> input_values;
+        if (operand_1_is_tensor) {
+          AddTestTensor("input1", {2, 1, 2}, {3, 6, 3, 6});
+        } else {
+          AddTestWeights("input1", {1, 2}, std::vector<float>{3, 6}, tf_type);
+        }
+        if (operand_2_is_tensor) {
+          AddTestTensor("input2", {2, 2, 1}, {2, 3, 2, 3});
+        } else {
+          AddTestWeights("input2", {2, 1}, std::vector<float>{2, 3}, tf_type);
+        }
+        TestOpConverter("my_binary", node_def, {2, 2, 2}, Status::OK(),
+                        Status::OK(),
+                        ElementsAreArray(op_test_info[op_name].second));
+      }
     }
   }
 }
@@ -2966,17 +2907,17 @@ TEST_F(OpConverterTest, ConvertCombinedNMS) {
 #endif  // IS_TRT_VERSION_GE(5, 1, 0, 0)
 
 template <typename T>
-NodeDef CreateUnaryOp(DataType tf_dtype) {
+NodeDef CreateUnaryOp(DataType tf_type) {
   Scope s = Scope::NewRootScope();
-  auto input = ops::Placeholder(s.WithOpName("input"), tf_dtype);
+  auto input = ops::Placeholder(s.WithOpName("input"), tf_type);
   return T(s.WithOpName("my_unary"), input).operation.node()->def();
 }
 
 constexpr float kLeakyReluAlpha = 0.2f;
 template <>
-NodeDef CreateUnaryOp<ops::internal::LeakyRelu>(DataType tf_dtype) {
+NodeDef CreateUnaryOp<ops::internal::LeakyRelu>(DataType tf_type) {
   Scope s = Scope::NewRootScope();
-  auto input = ops::Placeholder(s.WithOpName("input"), tf_dtype);
+  auto input = ops::Placeholder(s.WithOpName("input"), tf_type);
   return ops::internal::LeakyRelu(
              s.WithOpName("my_unary"), input,
              ops::internal::LeakyRelu::Alpha(kLeakyReluAlpha))
@@ -2988,7 +2929,7 @@ TEST_P(OpConverterTest1, ConvertActivation) {
   {
     // Input is weights, should fail.
     Reset();
-    const NodeDef& node_def = CreateUnaryOp<ops::Relu>(tf_dtype);
+    const NodeDef& node_def = CreateUnaryOp<ops::Relu>(tf_type);
     AddTestWeights<int32>("input", {1, 2, 3}, {-3, -2, -1, 0, 1, 2});
     RunValidationAndConversion(
         node_def, error::UNIMPLEMENTED,
@@ -3045,7 +2986,7 @@ TEST_P(OpConverterTest1, ConvertActivation) {
       FAIL() << "Activation op test map does not contain op " << op_name;
     }
     Reset();
-    NodeDef node_def = op_map[op_name].first(tf_dtype);
+    NodeDef node_def = op_map[op_name].first(tf_type);
     const std::vector<float> input = {-100, -2, -1, 0, 1, 88};
     AddTestTensor("input", p.input_dims, input);
 
@@ -3070,10 +3011,10 @@ TEST_P(OpConverterTest1, ConvertActivation) {
   }
 }
 
-TEST_F(OpConverterTest, ConvertExpandDims) {
+TEST_P(OpConverterTest1, ConvertExpandDims) {
   // Get the NodeDef for ExpandDims.
   Scope s = Scope::NewRootScope();
-  auto input = ops::Placeholder(s.WithOpName("input"), DT_FLOAT);
+  auto input = ops::Placeholder(s.WithOpName("input"), tf_type);
   auto weights = ops::Placeholder(s.WithOpName("weights"), DT_INT32);
   auto expanddims =
       ops::ExpandDims(s.WithOpName("my_expanddims"), input, weights);
@@ -3090,85 +3031,60 @@ TEST_F(OpConverterTest, ConvertExpandDims) {
   {
     // Axis is a tensor, should fail.
     Reset();
-    AddTestTensor("input", {1, 2, 3});
+    AddTestTensor("input", {3, 2, 1});
     AddTestTensor("weights", {3});
     RunValidationAndConversion(node_def, error::UNIMPLEMENTED,
                                "The input \"axis\" for ExpandDims must be a "
                                "constant, at my_expanddims");
   }
-  {
-    // Add dim at batch dimension, should fail.
-    Reset();
-    AddTestTensor("input", {1, 2, 3});
-    AddTestWeights<int32>("weights", {1}, {0});
-    RunValidationAndConversion(
-        node_def, error::UNIMPLEMENTED,
-        "TensorRT does not allow manipulation of the batch dimension, at "
-        "my_expanddims");
-  }
-  {
-    // Add dim at batch dimension via negative axis, should fail.
-    Reset();
-    AddTestTensor("input", {1, 2, 3});
-    // Input is rank 4 (batch dim included)
-    AddTestWeights<int32>("weights", {1}, {-5});
-    RunValidationAndConversion(
-        node_def, error::UNIMPLEMENTED,
-        "TensorRT does not allow manipulation of the batch dimension, at "
-        "my_expanddims");
-  }
-  {
-    // Axis > rank(input), should fail.
-    Reset();
-    AddTestTensor("input", {1, 2, 3});
-    // Input is rank 4 (batch dim included)
-    AddTestWeights<int32>("weights", {1}, {5});
-    RunValidationAndConversion(
-        node_def, error::INVALID_ARGUMENT,
-        "Axis value of 5 is out of bounds, must be in range [-5, 5), at "
-        "my_expanddims");
-  }
-  {
-    // Axis < -rank(input)-1, should fail.
-    Reset();
-    AddTestTensor("input", {1, 2, 3});
-    // Input is rank 4 (batch dim included)
-    AddTestWeights<int32>("weights", {1}, {-6});
-    RunValidationAndConversion(
-        node_def, error::INVALID_ARGUMENT,
-        "Axis value of -6 is out of bounds, must be in range [-5, 5), at "
-        "my_expanddims");
-  }
-
-  struct TestParams {
-    std::vector<int> input_dims;
-    int axis;
-    std::vector<int> expected_output_dims;
+  std::vector<TestParamBase> test_params = {
+      TestParamBase{{1, 1, 2, 3},
+                    {},
+                    {1, 1, 1, 2, 3},
+                    {0},
+                    trt_mode == TrtTestMode::kImplicitBatch
+                        ? Status(error::UNIMPLEMENTED,
+                                 "TensorRT does not allow manipulation of the "
+                                 "batch dimension, at my_expanddims")
+                        : Status::OK()},
+      TestParamBase{{1, 1, 2, 3},
+                    {},
+                    {1, 1, 1, 2, 3},
+                    {-5},
+                    trt_mode == TrtTestMode::kImplicitBatch
+                        ? Status(error::UNIMPLEMENTED,
+                                 "TensorRT does not allow manipulation of the "
+                                 "batch dimension, at my_expanddims")
+                        : Status::OK()},
+      TestParamBase{{1, 1, 2, 3},
+                    {},
+                    {},
+                    {5},
+                    Status(error::INVALID_ARGUMENT,
+                           "Axis value of 5 is out of bounds, must be in range"
+                           " [-5, 5), at my_expanddims")},
+      TestParamBase{{1, 1, 2, 3},
+                    {},
+                    {},
+                    {-6},
+                    Status(error::INVALID_ARGUMENT,
+                           "Axis value of -6 is out of bounds, must be in range"
+                           " [-5, 5), at my_expanddims")},
+      TestParamBase{{1, 2, 3}, {}, {1, 1, 2, 3}, {1}},
+      TestParamBase{{1, 2, 3}, {}, {1, 1, 2, 3}, {-3}},
+      TestParamBase{{1, 2, 3}, {}, {1, 2, 3, 1}, {3}},
+      TestParamBase{{1, 2, 3}, {}, {1, 2, 3, 1}, {-1}},
+      TestParamBase{{1, 2, 3}, {}, {1, 2, 1, 3}, {2}},
+      TestParamBase{{1, 2, 3}, {}, {1, 2, 1, 3}, {-2}},
+      TestParamBase{{1, 6}, {}, {1, 1, 6}, {1}},
+      TestParamBase{{1, 6}, {}, {1, 6, 1}, {-1}},
   };
-
-  // Ok.
-  std::vector<TestParams> ok_params = {
-      TestParams{{2, 3}, 1, {1, 2, 3}}, TestParams{{2, 3}, -3, {1, 2, 3}},
-      TestParams{{2, 3}, 3, {2, 3, 1}}, TestParams{{2, 3}, -1, {2, 3, 1}},
-      TestParams{{2, 3}, 2, {2, 1, 3}}, TestParams{{2, 3}, -2, {2, 1, 3}},
-      TestParams{{6}, 1, {1, 6}},       TestParams{{6}, -1, {6, 1}},
-  };
-  for (int i = 0; i < ok_params.size(); ++i) {
+  for (auto p : test_params) {
     Reset();
-    AddTestTensor("input", ok_params[i].input_dims);
-    AddTestWeights<int32>("weights", {1}, {ok_params[i].axis});
-    RunValidationAndConversion(node_def);
-    TRT_TensorOrWeights output;
-    TF_EXPECT_OK(GetTensorOrWeights("my_expanddims", &output));
-    ASSERT_TRUE(output.is_tensor());
-    ExpectTrtDimsEqualsArray(ok_params[i].expected_output_dims,
-                             output.tensor()->getDimensions());
-
-    const DataVec input_data{{"input", AsTensor<float>({1, 2, 3, 4, 5, 6})}};
-    DataVec output_data{{"my_expanddims", ConstructTensor<float>(6)}};
-    TF_EXPECT_OK(BuildAndRun(input_data, &output_data));
-    EXPECT_THAT(GetSpanForData<float>(output_data[0]),
-                ElementsAre(1, 2, 3, 4, 5, 6));
+    AddTestTensor("input", p.input_dims, {1, 2, 3, 4, 5, 6});
+    AddTestWeights<int32>("weights", {1}, {p.param[0]});
+    TestOpConverter("my_expanddims", node_def, p.expected_output_dims, p.status,
+                    p.runtime_status, ElementsAreArray({1, 2, 3, 4, 5, 6}));
   }
 }
 
@@ -3176,9 +3092,9 @@ TEST_P(OpConverterTest1, ConvertSqueeze) {
   const bool use_implicit_batch = (trt_mode == TrtTestMode::kImplicitBatch);
   // Get the NodeDef for Squeeze.
   auto get_squeeze_nodedef = [](std::vector<int> axes,
-                                DataType tf_dtype) -> NodeDef {
+                                DataType tf_type) -> NodeDef {
     Scope s = Scope::NewRootScope();
-    auto input = ops::Placeholder(s.WithOpName("input"), tf_dtype);
+    auto input = ops::Placeholder(s.WithOpName("input"), tf_type);
     if (!axes.empty()) {
       ops::Squeeze::Attrs squeeze_attrs;
       squeeze_attrs.axis_ = gtl::ArraySlice<int>(axes);  // non-absl ok
@@ -3270,7 +3186,7 @@ TEST_P(OpConverterTest1, ConvertSqueeze) {
   for (TestParamBase p : test_params) {
     SCOPED_TRACE(p);
     Reset();
-    NodeDef node_def = get_squeeze_nodedef(p.param, tf_dtype);
+    NodeDef node_def = get_squeeze_nodedef(p.param, tf_type);
     AddTestTensor("input", p.input_dims, {1, 2, 3, 4, 5, 6},
                   p.partial_input_dims);
     TestOpConverter("my_squeeze", node_def, p.expected_output_dims, p.status,
@@ -4025,14 +3941,14 @@ TEST_F(OpConverterTest, ConvertSlice) {
 
 TEST_P(OpConverterTest1, ConvertConv2D) {
   // Get nodedef for Conv2D layer.
-  DataType tf_type = tf_dtype;
+  DataType tf_type_loc = tf_type;
   auto get_conv2d_nodedef =
-      [tf_type](std::vector<int> strides = {1, 1, 1, 1},
-                string padding = "SAME", string data_format = "NCHW",
-                std::vector<int> dilations = {1, 1, 1, 1}) -> NodeDef {
+      [tf_type_loc](std::vector<int> strides = {1, 1, 1, 1},
+                    string padding = "SAME", string data_format = "NCHW",
+                    std::vector<int> dilations = {1, 1, 1, 1}) -> NodeDef {
     Scope s = Scope::NewRootScope();
-    auto input = ops::Placeholder(s.WithOpName("input"), tf_type);
-    auto filter = ops::Placeholder(s.WithOpName("weights"), tf_type);
+    auto input = ops::Placeholder(s.WithOpName("input"), tf_type_loc);
+    auto filter = ops::Placeholder(s.WithOpName("weights"), tf_type_loc);
     ops::Conv2D::Attrs attrs =
         ops::Conv2D::Attrs().DataFormat(data_format).Dilations(dilations);
     auto conv2d = ops::Conv2D(s.WithOpName("my_conv2d"), input, filter, strides,
@@ -4233,8 +4149,8 @@ TEST_P(OpConverterTest1, ConvertConv2D) {
       partial_input_shape[channel_id] = ok_params[i].input_dims[channel_id];
     }
 
-    AddTestTensor("input", ok_params[i].input_dims, tf_dtype,
-                  ok_params[i].input, partial_input_shape);
+    AddTestTensor("input", ok_params[i].input_dims, tf_type, ok_params[i].input,
+                  partial_input_shape);
     AddTestWeights<float>("weights", ok_params[i].filter_dims,
                           ok_params[i].filter);
 
@@ -4938,17 +4854,34 @@ TEST_F(OpConverterTest, ConvertTopK) {
   }
 }
 
-template <DataType dtype>
-void TestConvertGather(OpConverterTest* test) {
-  typedef typename EnumToDataType<dtype>::Type CType;
-
+TEST_P(OpConverterTest3, ConvertGather) {
   // Get the NodeDef for GatherV2.
   Scope s = Scope::NewRootScope();
-  auto params = ops::Placeholder(s.WithOpName("params"), dtype);
+  auto params = ops::Placeholder(s.WithOpName("params"), tf_type);
   auto indices = ops::Placeholder(s.WithOpName("indices"), DT_INT32);
   auto axis = ops::Placeholder(s.WithOpName("axis"), DT_INT32);
   auto gather = ops::GatherV2(s.WithOpName("my_gather"), params, indices, axis);
   const NodeDef& node_def = gather.operation.node()->def();
+  {
+    // Axis is a tensor, should fail.
+    Reset();
+    AddTestTensor("params", {1, 1, 2, 3}, tf_type, {});
+    AddTestTensor("indices", {1, 2}, DT_INT32, {});
+    AddTestTensor("axis", {1}, DT_INT32, {});
+    RunValidationAndConversion(
+        node_def, error::UNIMPLEMENTED,
+        "The input \"axis\" for GatherV2 must be a constant, at my_gather");
+  }
+  {
+    // Axis is out of bounds, should fail.
+    Reset();
+    AddTestTensor("params", {1, 1, 2, 3});
+    AddTestTensor("indices", {1, 2}, DT_INT32, {});
+    AddTestWeights<int32>("axis", {1}, {4});
+    RunValidationAndConversion(node_def, error::INVALID_ARGUMENT,
+                               "Axis value of 4 is out of bounds, must be in "
+                               "range [-4, 4), at my_gather");
+  }
 
   struct TestParams {
     // TF shape of the input 'params' (including batch dimension).
@@ -4961,12 +4894,74 @@ void TestConvertGather(OpConverterTest* test) {
     std::vector<int> expected_output_shape;
     std::vector<int> expected_output;
     bool params_is_tensor;
+    Status status;
+    Status runtime_status;
+    Status add_index_status;
   };
 
   // Input is the same {1, 2, 3, 4, 5, 6} for all cases.
-  const std::vector<CType> params_input = {CType(1), CType(2), CType(3),
-                                           CType(4), CType(5), CType(6)};
-  std::vector<TestParams> ok_params = {
+  const std::vector<int> params_input = {1, 2, 3, 4, 5, 6};
+  std::vector<TestParams> test_params = {
+      // Axis is batch dimension, should fail in implicit batch mode.
+      TestParams{/*params_shape=*/{2, 1, 1, 3},
+                 /*indices_shape=*/{2},
+                 /*indices=*/{1, 0},
+                 /*axis=*/0,
+                 /*expected_output_shape=*/{2, 1, 1, 3},
+                 /*expected_output=*/{4, 5, 6, 1, 2, 3},
+                 /*params_is_tensor=*/true,
+                 trt_mode == TrtTestMode::kImplicitBatch
+                     ? Status{error::UNIMPLEMENTED,
+                              "TensorRT does not allow manipulation of the"
+                              " batch dimension, at my_gather"}
+                     : Status::OK()},
+      // Batch size of indices is not 1 when params is a tensor.
+      TestParams{/*params_shape=*/{2, 1, 3},
+                 /*indices_shape=*/{2, 1},
+                 /*indices=*/{2, 0},
+                 /*axis=*/2,
+                 /*expected_output_shape=*/{2, 1, 2, 1},
+                 /*expected_output=*/{3, 1, 6, 4},
+                 /*params_is_tensor=*/true,
+                 trt_mode == TrtTestMode::kImplicitBatch
+                     ? Status{error::UNIMPLEMENTED,
+                              "Indices must have a batch size of 1 when params"
+                              " is a tensor."}
+                     : Status::OK()},
+      // Axis is not zero when params is a weight, should fail in implicit batch
+      // mode.
+      TestParams{/*params_shape=*/{2, 1, 3},
+                 /*indices_shape=*/{2},
+                 /*indices=*/{1, 2},
+                 /*axis=*/2,
+                 /*expected_output_shape=*/{2, 1, 2},
+                 /*expected_output=*/{2, 3, 5, 6},
+                 /*params_is_tensor=*/false,
+                 trt_mode == TrtTestMode::kImplicitBatch
+                     ? Status{error::UNIMPLEMENTED,
+                              "The input axis must be zero when params is a"
+                              " weight."}
+                     : Status::OK()},
+      // Params with only batch dimension.
+      TestParams{/*params_shape=*/{6},
+                 /*indices_shape=*/{2},
+                 /*indices=*/{1, 3},
+                 /*axis=*/0,
+                 /*expected_output_shape=*/{2},
+                 /*expected_output=*/{2, 4},
+                 /*params_is_tensor=*/true,
+                 trt_mode == TrtTestMode::kImplicitBatch  // conversion_status
+                     ? Status{error::UNIMPLEMENTED,
+                              "TensorRT does not allow manipulation of the "
+                              "batch dimension, at my_gather"}
+                     : Status::OK(),
+                 Status::OK(),                            // runtime_status
+                 trt_mode == TrtTestMode::kImplicitBatch  // add_index_status
+                     ? Status{error::INVALID_ARGUMENT,
+                              "Batch size doesn't match for tensor indices: "
+                              "Provided batch size does not match converter "
+                              "batch size: 2 vs 6"}
+                     : Status::OK()},
       // Vector indices, and output rank is rank(params).
       TestParams{
           /*params_shape=*/{1, 1, 2, 3},
@@ -4986,7 +4981,8 @@ void TestConvertGather(OpConverterTest* test) {
           /*expected_output=*/{4, 5, 6},
           /*params_is_tensor=*/true,
       },
-      // Indices with rank>1, and output rank is rank(params)+rank(indices)-1.
+      // Indices with rank>1, and output rank is rank(params) + rank(indices) -
+      // 1
       TestParams{
           /*params_shape=*/{1, 1, 2, 3},
           /*indices_shape=*/{1, 1},
@@ -5070,125 +5066,22 @@ void TestConvertGather(OpConverterTest* test) {
       },
   };
 
-  // Ok.
-  for (int i = 0; i < ok_params.size(); i++) {
-    test->Reset();
-    const auto& params_shape = ok_params[i].params_shape;
-    if (ok_params[i].params_is_tensor) {
-      std::vector<int> params_dims(params_shape.begin() + 1,
-                                   params_shape.end());
-      test->AddTestTensor("params", params_dims, params_shape[0],
-                          TfDataTypeToTrt(dtype));
+  for (auto p : test_params) {
+    Reset();
+    if (p.params_is_tensor) {
+      AddTestTensor("params", p.params_shape, params_input);
     } else {
-      test->AddTestWeights<CType>("params", params_shape, params_input);
+      AddTestWeights("params", p.params_shape, params_input, tf_type);
     }
-
-    const auto& indices_shape = ok_params[i].indices_shape;
-    test->AddTestTensor(
-        "indices",
-        std::vector<int>(indices_shape.begin() + 1, indices_shape.end()),
-        indices_shape[0], nvinfer1::DataType::kINT32);
-    test->AddTestWeights<int32>("axis", {1}, {ok_params[i].axis});
-    test->RunValidationAndConversion(node_def);
-    TRT_TensorOrWeights output;
-    TF_EXPECT_OK(test->GetTensorOrWeights("my_gather", &output));
-    ASSERT_TRUE(output.is_tensor());
-
-    const auto& expected_output_shape = ok_params[i].expected_output_shape;
-    const auto& expected_output = ok_params[i].expected_output;
-    ASSERT_EQ(expected_output.size(),
-              TrtWeightDimsNumElements(GetTestDims(expected_output_shape)));
-    const std::vector<int> expected_output_dims(
-        expected_output_shape.begin() + 1, expected_output_shape.end());
-    ExpectTrtDimsEqualsArray(expected_output_dims,
-                             output.tensor()->getDimensions());
-
-    // Create input in CType and convert expected output to CType.
-    std::vector<CType> converted_expected_output(expected_output.begin(),
-                                                 expected_output.end());
-
-    DataVec input_data;
-    if (ok_params[i].params_is_tensor) {
-      input_data = {{"params", test->AsTensor<CType>(params_input)},
-                    {"indices", test->AsTensor<int32>(ok_params[i].indices)}};
-    } else {
-      input_data = {{"indices", test->AsTensor<int32>(ok_params[i].indices)}};
-    }
-    DataVec output_data{
-        {"my_gather", test->ConstructTensor<CType>(expected_output.size())}};
-    TF_EXPECT_OK(test->BuildAndRun(input_data, &output_data,
-                                   /*batch_size=*/expected_output_shape[0]));
-    EXPECT_THAT(GetSpanForData<CType>(output_data[0]),
-                ElementsAreArray(converted_expected_output));
+    AddTestTensor("indices", p.indices_shape, DT_INT32, p.indices, {},
+                  p.add_index_status);
+    AddTestWeights<int32>("axis", {1}, {p.axis});
+    TestOpConverter("my_gather", node_def, p.expected_output_shape, p.status,
+                    p.runtime_status, ElementsAreArray(p.expected_output));
   }
 }
 
-TEST_F(OpConverterTest, ConvertGather) {
-  // Get the NodeDef for GatherV2.
-  Scope s = Scope::NewRootScope();
-  auto params = ops::Placeholder(s.WithOpName("params"), DT_FLOAT);
-  auto indices = ops::Placeholder(s.WithOpName("indices"), DT_INT32);
-  auto axis = ops::Placeholder(s.WithOpName("axis"), DT_INT32);
-  auto gather = ops::GatherV2(s.WithOpName("my_gather"), params, indices, axis);
-  const NodeDef& node_def = gather.operation.node()->def();
-  {
-    // Axis is a tensor, should fail.
-    Reset();
-    AddTestTensor("params", {1, 2, 3});
-    AddTestTensor("indices", {2});
-    AddTestTensor("axis", {1});
-    RunValidationAndConversion(
-        node_def, error::UNIMPLEMENTED,
-        "The input \"axis\" for GatherV2 must be a constant, at my_gather");
-  }
-  {
-    // Axis is out of bounds, should fail.
-    Reset();
-    AddTestTensor("params", {1, 2, 3});
-    AddTestTensor("indices", {2});
-    AddTestWeights<int32>("axis", {1}, {4});
-    RunValidationAndConversion(node_def, error::INVALID_ARGUMENT,
-                               "Axis value of 4 is out of bounds, must be in "
-                               "range [-4, 4), at my_gather");
-  }
-  {
-    // Axis is batch dimension, should fail.
-    Reset();
-    AddTestTensor("params", {1, 2, 3});
-    AddTestTensor("indices", {2});
-    AddTestWeights<int32>("axis", {1}, {0});
-    RunValidationAndConversion(node_def, error::UNIMPLEMENTED,
-                               "TensorRT does not allow manipulation of the "
-                               "batch dimension, at my_gather");
-  }
-  {
-    // Axis is not zero when params is a weight, should fail.
-    Reset();
-    AddTestWeights<int32>("params", {1, 3}, {1, 2, 3});
-    AddTestTensor("indices", {2});
-    AddTestWeights<int32>("axis", {1}, {1});
-    RunValidationAndConversion(
-        node_def, error::UNIMPLEMENTED,
-        "The input axis must be zero when params is a weight.");
-  }
-  {
-    // Batch size of indices is not 1 when params is a tensor.
-    Reset();
-    AddTestTensor("params", {1, 2, 3}, /*batch_size=*/2);
-    AddTestTensor("indices", {2}, /*batch_size=*/2);
-    AddTestWeights<int32>("axis", {1}, {1});
-    RunValidationAndConversion(
-        node_def, error::UNIMPLEMENTED,
-        "Indices must have a batch size of 1 when params is a tensor.");
-  }
-
-  Reset();
-  TestConvertGather<DT_FLOAT>(this);
-  TestConvertGather<DT_HALF>(this);
-  TestConvertGather<DT_INT32>(this);
-}
-
-NodeDef CreateCastOp(DataType tf_dtype) {
+NodeDef CreateCastOp(DataType tf_type) {
   Scope s = Scope::NewRootScope();
   auto input = ops::Placeholder(s.WithOpName("input"), DT_HALF);
   return ops::Cast(s.WithOpName("my_unary"), input, DT_FLOAT)
@@ -5200,7 +5093,7 @@ TEST_P(OpConverterTest1, ConvertUnary) {
   {
     // Input is weights, should fail.
     Reset();
-    const NodeDef node_def = CreateUnaryOp<ops::Neg>(tf_dtype);
+    const NodeDef node_def = CreateUnaryOp<ops::Neg>(tf_type);
     AddTestWeights<float>("input", {1, 2, 3}, {-3, -2, -1, 0, 1, 2});
     RunValidationAndConversion(
         node_def, error::UNIMPLEMENTED,
@@ -5256,7 +5149,7 @@ TEST_P(OpConverterTest1, ConvertUnary) {
     if (!op_map.count(op_name)) {
       FAIL() << "Unary op test map does not contain op " << op_name;
     }
-    NodeDef node_def = op_map[op_name].first(tf_dtype);
+    NodeDef node_def = op_map[op_name].first(tf_type);
 
     // TODO(bixia): we assume this test is only instantiated for DT_FLOAT for
     // now. Need to find a better way to express input and output types.
@@ -5264,10 +5157,10 @@ TEST_P(OpConverterTest1, ConvertUnary) {
     // TODO(tfeher): improve tests by defining an expected output data type and
     // check that. Currently only the shape and values of the output are
     // checked.
-    DataType input_tf_dtype = op_name == "Cast" ? DT_HALF : tf_dtype;
+    DataType input_tf_type = op_name == "Cast" ? DT_HALF : tf_type;
 
     std::vector<float> input_values{-0.9f, 0.6f, 0.0f, -3.5f, 100.0f, 2.9f};
-    AddTestTensor("input", p.input_dims, input_tf_dtype, input_values);
+    AddTestTensor("input", p.input_dims, input_tf_type, input_values);
     std::vector<float> output;
     std::transform(input_values.begin(), input_values.end(),
                    std::back_inserter(output), op_map[op_name].second);
diff --git a/tensorflow/compiler/tf2tensorrt/convert/utils.h b/tensorflow/compiler/tf2tensorrt/convert/utils.h
index 43697573bbd..775616ff7aa 100644
--- a/tensorflow/compiler/tf2tensorrt/convert/utils.h
+++ b/tensorflow/compiler/tf2tensorrt/convert/utils.h
@@ -19,6 +19,7 @@ limitations under the License.
 #include <memory>
 #include <vector>
 
+#include "absl/algorithm/container.h"
 #include "tensorflow/core/framework/tensor_shape.h"
 #include "tensorflow/core/lib/core/status.h"
 
@@ -88,6 +89,10 @@ inline bool HasStaticShape(const nvinfer1::Dims& dims) {
   return true;
 }
 
+inline bool HasStaticShape(std::vector<int> dims) {
+  return !absl::c_any_of(dims, [](int i) { return i < 0; });
+}
+
 template <typename TensorShapeType>
 inline nvinfer1::Dims TensorShapeToTrtDims(const TensorShapeType& shape,
                                            bool ignore_first_dim) {
diff --git a/tensorflow/compiler/xla/pjrt/pjrt_client.cc b/tensorflow/compiler/xla/pjrt/pjrt_client.cc
index 80fd0e0b658..b09bc4ec724 100644
--- a/tensorflow/compiler/xla/pjrt/pjrt_client.cc
+++ b/tensorflow/compiler/xla/pjrt/pjrt_client.cc
@@ -187,6 +187,7 @@ PjRtClient::PjRtClient(
       CHECK(local_devices_[idx] == nullptr) << idx;
       local_devices_[idx] = device.get();
     }
+    device->client_ = this;
   }
   for (int idx = 0; idx < local_devices_.size(); ++idx) {
     CHECK(local_devices_[idx] != nullptr) << idx;
diff --git a/tensorflow/compiler/xla/pjrt/pjrt_client.h b/tensorflow/compiler/xla/pjrt/pjrt_client.h
index 775b44c7073..043495093d4 100644
--- a/tensorflow/compiler/xla/pjrt/pjrt_client.h
+++ b/tensorflow/compiler/xla/pjrt/pjrt_client.h
@@ -47,6 +47,8 @@ limitations under the License.
 
 namespace xla {
 
+class PjRtClient;
+
 class Device {
  public:
   explicit Device(int id, std::unique_ptr<LocalDeviceState> local_device_state,
@@ -86,12 +88,17 @@ class Device {
 
   virtual std::string DebugString() const;
 
+  PjRtClient* client() const { return client_; }
+
  private:
+  friend class PjRtClient;
+
   const int id_;
   const std::unique_ptr<LocalDeviceState> local_device_state_;
   const int host_id_;
   const std::string platform_name_;
   const std::string device_kind_;
+  PjRtClient* client_ = nullptr;
 };
 
 // Forward declaration.
@@ -113,7 +120,7 @@ using PjRtCrossHostRecvNotifier =
 //
 // It is the responsibility of the client of this API to keep the PjRtClient
 // alive as long as any of the other runtime objects are alive.
-class PjRtClient : public std::enable_shared_from_this<PjRtClient> {
+class PjRtClient {
  public:
   // `allocator` may null, in which case the platform default allocator is used.
   explicit PjRtClient(
diff --git a/tensorflow/compiler/xla/python/BUILD b/tensorflow/compiler/xla/python/BUILD
index 7913c2d9dd4..67ab4e19cb9 100644
--- a/tensorflow/compiler/xla/python/BUILD
+++ b/tensorflow/compiler/xla/python/BUILD
@@ -123,6 +123,25 @@ cc_library(
     ],
 )
 
+cc_library(
+    name = "traceback",
+    srcs = ["traceback.cc"],
+    hdrs = ["traceback.h"],
+    copts = [
+        "-fexceptions",
+        "-fno-strict-aliasing",
+    ],
+    features = ["-use_header_modules"],
+    deps = [
+        ":python_ref_manager",
+        "//tensorflow/core:lib",
+        "@com_google_absl//absl/container:inlined_vector",
+        "@com_google_absl//absl/strings",
+        "@com_google_absl//absl/strings:str_format",
+        "@pybind11",
+    ],
+)
+
 cc_library(
     name = "bfloat16",
     srcs = ["bfloat16.cc"],
@@ -159,6 +178,37 @@ py_test(
     ] + xla_py_test_deps(),
 )
 
+cc_library(
+    name = "py_client",
+    srcs = [
+        "py_buffer.cc",
+        "py_client.cc",
+        "py_executable.cc",
+    ],
+    hdrs = [
+        "py_buffer.h",
+        "py_client.h",
+        "py_executable.h",
+    ],
+    copts = [
+        "-fexceptions",
+        "-fno-strict-aliasing",
+    ],
+    features = ["-use_header_modules"],
+    deps = [
+        ":python_ref_manager",
+        ":traceback",
+        ":types",
+        "//tensorflow/compiler/xla:statusor",
+        "//tensorflow/compiler/xla:types",
+        "//tensorflow/compiler/xla/pjrt:pjrt_client",
+        "@com_google_absl//absl/algorithm:container",
+        "@com_google_absl//absl/types:optional",
+        "@com_google_absl//absl/types:span",
+        "@pybind11",
+    ],
+)
+
 cc_library(
     name = "dlpack",
     srcs = ["dlpack.cc"],
@@ -169,6 +219,8 @@ cc_library(
     ],
     features = ["-use_header_modules"],
     deps = [
+        ":py_client",
+        ":traceback",
         "//tensorflow/compiler/xla:types",
         "//tensorflow/compiler/xla:util",
         "//tensorflow/compiler/xla/pjrt:pjrt_client",
@@ -233,7 +285,7 @@ cc_library(
 )
 
 tf_cc_test(
-    name = "cpu_outfeed_receiver_test",
+    name = "outfeed_receiver_test_cpu",
     size = "small",
     srcs = ["outfeed_receiver_test.cc"],
     deps = [
@@ -261,9 +313,11 @@ cc_library(
     features = ["-use_header_modules"],
     deps = [
         ":outfeed_receiver",
+        ":py_client",
         ":types",
         "//tensorflow/compiler/xla/client:xla_builder",
         "//tensorflow/compiler/xla/pjrt:pjrt_client",
+        "@com_google_absl//absl/algorithm:container",
         "@com_google_absl//absl/memory",
         "@com_google_absl//absl/synchronization",
         "@pybind11",
@@ -290,8 +344,10 @@ pybind_extension(
         ":bfloat16",
         ":dlpack",
         ":ops",
+        ":py_client",
         ":python_ref_manager",
         ":outfeed_receiver_py",
+        ":traceback",
         ":types",
         "@com_google_absl//absl/base",
         "@com_google_absl//absl/hash",
@@ -307,6 +363,7 @@ pybind_extension(
         "//tensorflow/compiler/xla:shape_util",
         "//tensorflow/compiler/xla:status",
         "//tensorflow/compiler/xla:statusor",
+        "//tensorflow/compiler/xla:types",
         "//tensorflow/compiler/xla:util",
         "//tensorflow/compiler/xla:xla_data_proto_cc",
         "//tensorflow/compiler/xla/client:client_library",
diff --git a/tensorflow/compiler/xla/python/dlpack.cc b/tensorflow/compiler/xla/python/dlpack.cc
index d37d480607a..4fc17172ea7 100644
--- a/tensorflow/compiler/xla/python/dlpack.cc
+++ b/tensorflow/compiler/xla/python/dlpack.cc
@@ -23,7 +23,9 @@ limitations under the License.
 #include "absl/strings/str_join.h"
 #include "absl/types/span.h"
 #include "include/dlpack/dlpack.h"  // from @dlpack
+#include "tensorflow/compiler/xla/pjrt/pjrt_client.h"
 #include "tensorflow/compiler/xla/pjrt/tracked_device_buffer.h"
+#include "tensorflow/compiler/xla/python/traceback.h"
 #include "tensorflow/compiler/xla/types.h"
 #include "tensorflow/compiler/xla/util.h"
 #include "tensorflow/stream_executor/cuda/cuda_platform_id.h"
@@ -239,12 +241,12 @@ StatusOr<Device*> DeviceForDLContext(const PjRtClient& client,
 
 }  // namespace
 
-StatusOr<py::capsule> BufferToDLPackManagedTensor(PjRtBuffer* buffer) {
-  auto pack = absl::make_unique<DLPackTensor>();
+StatusOr<py::capsule> BufferToDLPackManagedTensor(PyBuffer* buffer) {
+  auto pack = std::make_unique<DLPackTensor>();
   // Block on outstanding operations, so that it is safe to read or mutate the
   // returned buffer.
   StatusOr<std::shared_ptr<TrackedDeviceBuffer>> buffer_or =
-      buffer->Release(/*wait_for_operations_to_complete=*/true);
+      buffer->buffer()->Release(/*wait_for_operations_to_complete=*/true);
   if (!buffer_or.ok()) {
     return InvalidArgument(
         "Buffer synchronization failed converting to DLPack tensor: %s",
@@ -258,22 +260,25 @@ StatusOr<py::capsule> BufferToDLPackManagedTensor(PjRtBuffer* buffer) {
   pack->tensor.manager_ctx = pack.get();
   pack->tensor.deleter = DLPackTensorDeleter;
   DLTensor& dt = pack->tensor.dl_tensor;
-  if (buffer->on_device_shape().IsTuple()) {
+  if (buffer->buffer()->on_device_shape().IsTuple()) {
     return Unimplemented(
         "unsafe_buffer_pointer is not implemented for tuple "
         "buffers.");
   }
   TF_RET_CHECK(pack->buffer->device_memory().size() == 1);
   dt.data = pack->buffer->device_memory().front().opaque();
-  TF_ASSIGN_OR_RETURN(dt.ctx, DLContextForDevice(*buffer->device()));
-  dt.ctx.device_id = buffer->device()->local_device_state()->device_ordinal();
-  dt.ndim = buffer->on_host_shape().dimensions_size();
-  TF_ASSIGN_OR_RETURN(dt.dtype, PrimitiveTypeToDLDataType(
-                                    buffer->on_host_shape().element_type()));
+  TF_ASSIGN_OR_RETURN(dt.ctx, DLContextForDevice(*buffer->buffer()->device()));
+  dt.ctx.device_id =
+      buffer->buffer()->device()->local_device_state()->device_ordinal();
+  dt.ndim = buffer->buffer()->on_host_shape().dimensions_size();
+  TF_ASSIGN_OR_RETURN(dt.dtype,
+                      PrimitiveTypeToDLDataType(
+                          buffer->buffer()->on_host_shape().element_type()));
 
-  pack->shape = std::vector<int64>(buffer->on_host_shape().dimensions().begin(),
-                                   buffer->on_host_shape().dimensions().end());
-  pack->strides = StridesForShape(buffer->on_host_shape());
+  pack->shape =
+      std::vector<int64>(buffer->buffer()->on_host_shape().dimensions().begin(),
+                         buffer->buffer()->on_host_shape().dimensions().end());
+  pack->strides = StridesForShape(buffer->buffer()->on_host_shape());
   dt.shape = reinterpret_cast<std::int64_t*>(pack->shape.data());
   dt.strides = reinterpret_cast<std::int64_t*>(pack->strides.data());
   dt.byte_offset = 0;
@@ -293,8 +298,8 @@ StatusOr<py::capsule> BufferToDLPackManagedTensor(PjRtBuffer* buffer) {
   return capsule;
 }
 
-StatusOr<std::unique_ptr<PjRtBuffer>> DLPackManagedTensorToBuffer(
-    const pybind11::capsule& tensor, PjRtClient* client) {
+StatusOr<std::unique_ptr<PyBuffer>> DLPackManagedTensorToBuffer(
+    const pybind11::capsule& tensor, std::shared_ptr<PyClient> client) {
   if (absl::string_view(tensor.name()) != kDlTensorCapsuleName) {
     return InvalidArgument(
         "DLPack tensor must be a capsule with name \"dltensor\", got \"%s\". "
@@ -307,8 +312,9 @@ StatusOr<std::unique_ptr<PjRtBuffer>> DLPackManagedTensorToBuffer(
         "Number of dimensions in DLManagedTensor must be nonnegative, got %d",
         dlmt->dl_tensor.ndim);
   }
-  TF_ASSIGN_OR_RETURN(Device * device,
-                      DeviceForDLContext(*client, dlmt->dl_tensor.ctx));
+  TF_ASSIGN_OR_RETURN(
+      Device * device,
+      DeviceForDLContext(*client->pjrt_client(), dlmt->dl_tensor.ctx));
   absl::Span<int64 const> dimensions(
       reinterpret_cast<int64*>(dlmt->dl_tensor.shape), dlmt->dl_tensor.ndim);
   TF_ASSIGN_OR_RETURN(PrimitiveType element_type,
@@ -344,8 +350,10 @@ StatusOr<std::unique_ptr<PjRtBuffer>> DLPackManagedTensorToBuffer(
   // capsule it cannot be used again.
   PyCapsule_SetName(tensor.ptr(), "used_dltensor");
   PyCapsule_SetDestructor(tensor.ptr(), nullptr);
-  return absl::make_unique<PjRtBuffer>(shape, shape, std::move(device_buffer),
-                                       client, device);
+  auto pjrt_buffer = std::make_unique<PjRtBuffer>(
+      shape, shape, std::move(device_buffer), client->pjrt_client(), device);
+  return std::make_unique<PyBuffer>(std::move(client), std::move(pjrt_buffer),
+                                    Traceback::Get());
 }
 
 }  // namespace xla
diff --git a/tensorflow/compiler/xla/python/dlpack.h b/tensorflow/compiler/xla/python/dlpack.h
index 6766bbe93b1..7200997cf27 100644
--- a/tensorflow/compiler/xla/python/dlpack.h
+++ b/tensorflow/compiler/xla/python/dlpack.h
@@ -17,14 +17,15 @@ limitations under the License.
 #define TENSORFLOW_COMPILER_XLA_PYTHON_DLPACK_H_
 
 #include "pybind11/pybind11.h"
-#include "tensorflow/compiler/xla/pjrt/pjrt_client.h"
+#include "tensorflow/compiler/xla/python/py_buffer.h"
+#include "tensorflow/compiler/xla/python/py_client.h"
 
 namespace xla {
 
-StatusOr<pybind11::capsule> BufferToDLPackManagedTensor(PjRtBuffer* buffer);
+StatusOr<pybind11::capsule> BufferToDLPackManagedTensor(PyBuffer* buffer);
 
-StatusOr<std::unique_ptr<PjRtBuffer>> DLPackManagedTensorToBuffer(
-    const pybind11::capsule& tensor, PjRtClient* client);
+StatusOr<std::unique_ptr<PyBuffer>> DLPackManagedTensorToBuffer(
+    const pybind11::capsule& tensor, std::shared_ptr<PyClient> client);
 
 }  // namespace xla
 
diff --git a/tensorflow/compiler/xla/python/outfeed_receiver.cc b/tensorflow/compiler/xla/python/outfeed_receiver.cc
index 0be4167c397..7c029ca7d19 100644
--- a/tensorflow/compiler/xla/python/outfeed_receiver.cc
+++ b/tensorflow/compiler/xla/python/outfeed_receiver.cc
@@ -98,23 +98,17 @@ uint32_t constexpr kOutfeedHeaderStart = 271828;
 // Special consumer IDs, without outfeed payload.
 uint32_t constexpr kOutfeedCidShutdown = 0;
 
-// A Device and its PjRtClient.
-struct DeviceWithClient {
-  Device* device;
-  std::shared_ptr<PjRtClient> client;
-};
-
 // Encapsulates data received from a device outfeed.
 class OutfeedData {
  public:
-  OutfeedData(DeviceWithClient device_client, uint32_t consumer_id, Shape shape)
-      : device_client_(device_client),
+  OutfeedData(Device* device, uint32_t consumer_id, Shape shape)
+      : device_(device),
         consumer_id_(consumer_id),
         shape_(shape),
         literal_(nullptr),
         literal_size_bytes_(0) {}
 
-  DeviceWithClient device_client() { return device_client_; }
+  Device* device() { return device_; }
   uint32_t consumer_id() const { return consumer_id_; }
   Shape shape() const { return shape_; }
   std::unique_ptr<Literal> literal() {
@@ -129,7 +123,7 @@ class OutfeedData {
   std::string DebugString() const;
 
  private:
-  DeviceWithClient device_client_;
+  Device* device_;
   uint32_t consumer_id_;
   Shape shape_;
   std::unique_ptr<Literal> literal_;
@@ -150,15 +144,14 @@ void OutfeedData::SetLiteral(std::unique_ptr<Literal> literal) {
 }
 
 std::string OutfeedData::DebugString() const {
-  return absl::StrFormat("dev=%s; cons=%d; shape=%s",
-                         device_client_.device->DebugString(), consumer_id_,
-                         shape_.ToString());
+  return absl::StrFormat("dev=%s; cons=%d; shape=%s", device_->DebugString(),
+                         consumer_id_, shape_.ToString());
 }
 
 class OutfeedReceiverImpl {
  public:
   OutfeedReceiverImpl(OutfeedReceiver::Callback callback,
-                      std::vector<std::shared_ptr<PjRtClient>> clients,
+                      absl::Span<PjRtClient* const> clients,
                       ssize_t max_callback_queue_size_bytes);
 
   OutfeedReceiverImpl(const OutfeedReceiverImpl&) = delete;
@@ -206,8 +199,8 @@ class OutfeedReceiverImpl {
   void Shutdown();
 
   OutfeedReceiver::Callback callback_;
-  // The devices on which we are listening, with their clients.
-  std::vector<DeviceWithClient> devices_;
+  // The devices on which we are listening.
+  std::vector<Device*> devices_;
   // Maximum bytes capacity of the callback queue.
   uint64_t max_callback_queue_size_bytes_;
 
@@ -232,14 +225,13 @@ class OutfeedReceiverImpl {
 };
 
 OutfeedReceiverImpl::OutfeedReceiverImpl(
-    OutfeedReceiver::Callback callback,
-    std::vector<std::shared_ptr<PjRtClient>> clients,
+    OutfeedReceiver::Callback callback, absl::Span<PjRtClient* const> clients,
     ssize_t max_callback_queue_size_bytes) {
   callback_ = callback;
   max_callback_queue_size_bytes_ = max_callback_queue_size_bytes;
   for (const auto& client : clients) {
     for (const auto& device : client->devices()) {
-      devices_.push_back(DeviceWithClient{device.get(), client});
+      devices_.push_back(device.get());
     }
   }
   CHECK_GT(devices_.size(), 0);
@@ -291,11 +283,11 @@ void OutfeedReceiverImpl::DeviceListenerThreadLoop(int device_idx) {
     absl::MutexLock lock(&mu_);
     ++num_listening_threads_;
   }
-  DeviceWithClient device_client = devices_[device_idx];
+  Device* device = devices_[device_idx];
   while (true) {
     Shape header_shape = ShapeUtil::MakeShape(U32, {kOutfeedHeaderWords});
     std::unique_ptr<Literal> header =
-        ReceiveRawFromOutfeed(device_client.device, header_shape).ValueOrDie();
+        ReceiveRawFromOutfeed(device, header_shape).ValueOrDie();
     absl::Span<uint32_t> header_data = header->data<uint32>();
     CHECK_EQ(header_data.size(), kOutfeedHeaderWords);
     CHECK_EQ(header_data[0], kOutfeedHeaderStart);
@@ -306,18 +298,17 @@ void OutfeedReceiverImpl::DeviceListenerThreadLoop(int device_idx) {
       auto registered_shape = shape_registry_.find(consumer_id);
       if (registered_shape == shape_registry_.end()) {
         LOG(FATAL)
-            << "[" << device_client.device->DebugString()
+            << "[" << device->DebugString()
             << "] Cannot find registered shape for consumer ID " << consumer_id
             << ". Perhaps the code was compiled with a different instance "
             << "of OutfeedReceiver.";
       }
       shape = registered_shape->second;
     }
-    auto received =
-        absl::make_unique<OutfeedData>(device_client, consumer_id, shape);
+    auto received = absl::make_unique<OutfeedData>(device, consumer_id, shape);
     VLOG(2) << "Listener received header " << received->DebugString();
     if (consumer_id == kOutfeedCidShutdown) {
-      VLOG(2) << "[" << device_client.device->DebugString()
+      VLOG(2) << "[" << device->DebugString()
               << "] Listener received shutdown header";
       absl::MutexLock lock(&mu_);
       --num_listening_threads_;
@@ -328,7 +319,7 @@ void OutfeedReceiverImpl::DeviceListenerThreadLoop(int device_idx) {
       return;
     }
     std::unique_ptr<Literal> data =
-        ReceiveRawFromOutfeed(device_client.device, shape).ValueOrDie();
+        ReceiveRawFromOutfeed(device, shape).ValueOrDie();
     received->SetLiteral(std::move(data));
     absl::MutexLock lock(&mu_);
     EnqueueReceivedData(std::move(received));
@@ -392,15 +383,14 @@ void OutfeedReceiverImpl::CallbackThreadLoop() {
     }
     {
       tensorflow::profiler::TraceMe traceme("OutfeedReceiver::Callback");
-      DeviceWithClient device_client = received->device_client();
-      callback_(device_client.device, std::move(device_client.client),
-                received->consumer_id(), received->literal());
+      callback_(received->device(), received->consumer_id(),
+                received->literal());
     }
   }
 }
 
 Status OutfeedReceiverImpl::SendShutdownOutfeedHeader(int device_idx) {
-  const Device* device = devices_[device_idx].device;
+  const Device* device = devices_[device_idx];
   constexpr int consumer_id = kOutfeedCidShutdown;
   VLOG(2) << "[" << device->DebugString()
           << "] SendSpecialHeader cons=" << consumer_id;
@@ -421,7 +411,7 @@ Status OutfeedReceiverImpl::SendShutdownOutfeedHeader(int device_idx) {
 
   TF_ASSIGN_OR_RETURN(
       std::unique_ptr<PjRtExecutable> executable,
-      PjRtExecutable::Compile(computation, devices_[device_idx].client.get(),
+      PjRtExecutable::Compile(computation, devices_[device_idx]->client(),
                               std::move(compile_options)));
   ExecuteOptions execute_options;
   TF_ASSIGN_OR_RETURN(std::vector<std::unique_ptr<PjRtBuffer>> output_buffers,
@@ -468,11 +458,11 @@ StatusOr<XlaOp> OutfeedReceiverImpl::AddOutfeedToBuilder(
   return token;
 }
 
-OutfeedReceiver::OutfeedReceiver(
-    Callback callback, std::vector<std::shared_ptr<PjRtClient>> clients,
-    ssize_t max_callback_queue_size_bytes) {
+OutfeedReceiver::OutfeedReceiver(Callback callback,
+                                 absl::Span<PjRtClient* const> clients,
+                                 ssize_t max_callback_queue_size_bytes) {
   p_impl_ = absl::make_unique<OutfeedReceiverImpl>(
-      callback, std::move(clients), max_callback_queue_size_bytes);
+      callback, clients, max_callback_queue_size_bytes);
 }
 
 OutfeedReceiver::~OutfeedReceiver() {}
diff --git a/tensorflow/compiler/xla/python/outfeed_receiver.h b/tensorflow/compiler/xla/python/outfeed_receiver.h
index a0fdfcd36f0..a8dcc559810 100644
--- a/tensorflow/compiler/xla/python/outfeed_receiver.h
+++ b/tensorflow/compiler/xla/python/outfeed_receiver.h
@@ -31,10 +31,9 @@ class OutfeedReceiverImpl;
 // Implements a multithreaded receiver of outfeeds from devices.
 class OutfeedReceiver {
  public:
-  // A callback takes: device, client (for the device), consumer id, received.
-  // The client pointer should be alive while the device is used.
-  using Callback = std::function<void(Device*, std::shared_ptr<PjRtClient>,
-                                      uint32_t, std::shared_ptr<Literal>)>;
+  // A callback takes: device, consumer id, received.
+  using Callback =
+      std::function<void(Device*, uint32_t, std::shared_ptr<Literal>)>;
 
   // Constructs the receiver for the given clients and callback function.
   //
@@ -45,8 +44,7 @@ class OutfeedReceiver {
   //   max_callback_queue_size_bytes: the maximum number of bytes for all
   //     received outfeeds queued to be processed. When this limit is reached
   //     we pause receiving outfeeds from devices.
-  OutfeedReceiver(Callback callback,
-                  std::vector<std::shared_ptr<PjRtClient>> clients,
+  OutfeedReceiver(Callback callback, absl::Span<PjRtClient* const> clients,
                   ssize_t max_callback_queue_size_bytes);
 
   OutfeedReceiver(const OutfeedReceiver&) = delete;
diff --git a/tensorflow/compiler/xla/python/outfeed_receiver_py.cc b/tensorflow/compiler/xla/python/outfeed_receiver_py.cc
index a6256cfe86c..67926d5bc6b 100644
--- a/tensorflow/compiler/xla/python/outfeed_receiver_py.cc
+++ b/tensorflow/compiler/xla/python/outfeed_receiver_py.cc
@@ -17,6 +17,7 @@ limitations under the License.
 
 #include <memory>
 
+#include "absl/algorithm/container.h"
 #include "absl/memory/memory.h"
 #include "absl/synchronization/mutex.h"
 #include "pybind11/functional.h"
@@ -24,6 +25,7 @@ limitations under the License.
 #include "tensorflow/compiler/xla/client/xla_builder.h"
 #include "tensorflow/compiler/xla/pjrt/pjrt_client.h"
 #include "tensorflow/compiler/xla/python/outfeed_receiver.h"
+#include "tensorflow/compiler/xla/python/py_client.h"
 #include "tensorflow/compiler/xla/python/types.h"
 
 namespace xla {
@@ -41,17 +43,22 @@ class OutfeedReceiverForPython {
       std::function<void(ClientAndPtr<Device>, uint32_t, pybind11::object)>;
 
   OutfeedReceiverForPython(CallbackToPython callback_python,
-                           std::vector<std::shared_ptr<PjRtClient>> clients,
-                           ssize_t max_callback_queue_size_bytes) {
-    callback_python_ = callback_python;
-    outfeed_receiver_shutting_down_ = false;
+                           std::vector<std::shared_ptr<PyClient>> clients,
+                           ssize_t max_callback_queue_size_bytes)
+      : callback_python_(std::move(callback_python)),
+        clients_(std::move(clients)) {
     OutfeedReceiver::Callback callback =
-        [this](Device* device, std::shared_ptr<PjRtClient> client,
-               uint32_t consumer_id, std::shared_ptr<Literal> literal) {
-          this->Callback(device, client, consumer_id, literal);
+        [this](Device* device, uint32_t consumer_id,
+               std::shared_ptr<Literal> literal) {
+          this->Callback(device, consumer_id, std::move(literal));
         };
+    std::vector<PjRtClient*> client_ptrs(clients.size());
+    absl::c_transform(clients_, client_ptrs.begin(),
+                      [](const std::shared_ptr<PyClient>& client) {
+                        return client->pjrt_client();
+                      });
     outfeed_receiver_ = absl::make_unique<OutfeedReceiver>(
-        callback, std::move(clients), max_callback_queue_size_bytes);
+        callback, client_ptrs, max_callback_queue_size_bytes);
   }
   OutfeedReceiverForPython(const OutfeedReceiverForPython&) = delete;
   OutfeedReceiverForPython& operator=(const OutfeedReceiverForPython&) = delete;
@@ -79,8 +86,8 @@ class OutfeedReceiverForPython {
                                                   arrays);
   }
 
-  void Callback(Device* device, std::shared_ptr<PjRtClient> client,
-                uint32_t consumer_id, std::shared_ptr<Literal> literal) {
+  void Callback(Device* device, uint32_t consumer_id,
+                std::shared_ptr<Literal> literal) {
     {
       absl::MutexLock lock(&mu_);
       if (outfeed_receiver_shutting_down_) {
@@ -88,19 +95,26 @@ class OutfeedReceiverForPython {
         return;
       }
     }
+    // We expect the number of clients to be small, so an O(n) search is fine.
+    auto it = absl::c_find_if(
+        clients_, [device](const std::shared_ptr<PyClient>& client) {
+          return client->pjrt_client() == device->client();
+        });
+    CHECK(it != clients_.end());
     py::gil_scoped_acquire gil_acquire;  // Need GIL also for LiteralToPython
     py::object literal_python =
         LiteralToPython(std::move(literal)).ValueOrDie();
     // The callback_ should handle all exceptions in user-code. If we get
     // an exception here, it is a bug in the callback and we should stop.
-    callback_python_(WrapWithClient<Device>(std::move(client), device),
-                     consumer_id, std::move(literal_python));
+    callback_python_(WrapWithClient<Device>(*it, device), consumer_id,
+                     std::move(literal_python));
   }
 
  private:
   CallbackToPython callback_python_;
   absl::Mutex mu_;
-  bool outfeed_receiver_shutting_down_ TF_GUARDED_BY(mu_);
+  bool outfeed_receiver_shutting_down_ TF_GUARDED_BY(mu_) = false;
+  std::vector<std::shared_ptr<PyClient>> clients_;
   std::unique_ptr<OutfeedReceiver> outfeed_receiver_;
 };
 
@@ -112,7 +126,7 @@ void BuildOutfeedReceiverSubmodule(py::module* m) {
   outfeed_receiver.def(
       "start",
       [](OutfeedReceiverForPython::CallbackToPython callback_to_python,
-         std::vector<std::shared_ptr<PjRtClient>> clients,
+         std::vector<std::shared_ptr<PyClient>> clients,
          ssize_t max_callback_queue_size_bytes)
           -> std::unique_ptr<OutfeedReceiverForPython> {
         auto server = absl::make_unique<OutfeedReceiverForPython>(
diff --git a/tensorflow/compiler/xla/python/outfeed_receiver_test.cc b/tensorflow/compiler/xla/python/outfeed_receiver_test.cc
index ea84b4e18d6..e8a5063b70b 100644
--- a/tensorflow/compiler/xla/python/outfeed_receiver_test.cc
+++ b/tensorflow/compiler/xla/python/outfeed_receiver_test.cc
@@ -75,14 +75,14 @@ class Accumulator {
 TEST(OutfeedReceiverTest, ReceiveOutfeedSimple) {
   TF_ASSERT_OK_AND_ASSIGN(std::shared_ptr<PjRtClient> cpu_client,
                           GetCpuClient(true));
-  std::vector<std::shared_ptr<PjRtClient>> clients{cpu_client};
+  std::vector<PjRtClient*> clients{cpu_client.get()};
 
   auto receiver = absl::make_unique<Accumulator>();
-  OutfeedReceiver::Callback callback =
-      [&receiver](Device* device, std::shared_ptr<PjRtClient> client,
-                  uint32_t consumer_id, std::shared_ptr<Literal> data) {
-        receiver->Receive(consumer_id, data);
-      };
+  OutfeedReceiver::Callback callback = [&receiver](
+                                           Device* device, uint32_t consumer_id,
+                                           std::shared_ptr<Literal> data) {
+    receiver->Receive(consumer_id, data);
+  };
   auto outfeed_receiver =
       std::make_shared<OutfeedReceiver>(callback, clients, 128);
   outfeed_receiver->Start();
@@ -108,14 +108,14 @@ TEST(OutfeedReceiverTest, ReceiveOutfeedSimple) {
 TEST(OutfeedReceiverTest, ReceiveOutfeedTwoComputations) {
   TF_ASSERT_OK_AND_ASSIGN(std::shared_ptr<PjRtClient> cpu_client,
                           GetCpuClient(true));
-  std::vector<std::shared_ptr<PjRtClient>> clients{cpu_client};
+  std::vector<PjRtClient*> clients{cpu_client.get()};
 
   auto receiver = absl::make_unique<Accumulator>();
-  OutfeedReceiver::Callback callback =
-      [&receiver](Device* device, std::shared_ptr<PjRtClient> client,
-                  uint32_t consumer_id, std::shared_ptr<Literal> data) {
-        receiver->Receive(consumer_id, data);
-      };
+  OutfeedReceiver::Callback callback = [&receiver](
+                                           Device* device, uint32_t consumer_id,
+                                           std::shared_ptr<Literal> data) {
+    receiver->Receive(consumer_id, data);
+  };
   auto outfeed_receiver =
       std::make_shared<OutfeedReceiver>(callback, clients, 128);
   outfeed_receiver->Start();
@@ -153,14 +153,14 @@ TEST(OutfeedReceiverTest, ReceiveOutfeedTwoComputations) {
 TEST(OutfeedReceiverTest, ReceiveOutfeedTwoOutfeed) {
   TF_ASSERT_OK_AND_ASSIGN(std::shared_ptr<PjRtClient> cpu_client,
                           GetCpuClient(true));
-  std::vector<std::shared_ptr<PjRtClient>> clients{cpu_client};
+  std::vector<PjRtClient*> clients{cpu_client.get()};
 
   auto receiver = absl::make_unique<Accumulator>();
-  OutfeedReceiver::Callback callback =
-      [&receiver](Device* device, std::shared_ptr<PjRtClient> client,
-                  uint32_t consumer_id, std::shared_ptr<Literal> data) {
-        receiver->Receive(consumer_id, data);
-      };
+  OutfeedReceiver::Callback callback = [&receiver](
+                                           Device* device, uint32_t consumer_id,
+                                           std::shared_ptr<Literal> data) {
+    receiver->Receive(consumer_id, data);
+  };
   auto outfeed_receiver =
       std::make_shared<OutfeedReceiver>(callback, clients, 128);
   outfeed_receiver->Start();
@@ -196,14 +196,14 @@ TEST(OutfeedReceiverTest, ReceiveOutfeedTwoOutfeed) {
 TEST(OutfeedReceiverTest, DifferentShapeForConsumerIdError) {
   TF_ASSERT_OK_AND_ASSIGN(std::shared_ptr<PjRtClient> cpu_client,
                           GetCpuClient(true));
-  std::vector<std::shared_ptr<PjRtClient>> clients{cpu_client};
+  std::vector<PjRtClient*> clients{cpu_client.get()};
 
   auto receiver = absl::make_unique<Accumulator>();
-  OutfeedReceiver::Callback callback =
-      [&receiver](Device* device, std::shared_ptr<PjRtClient> client,
-                  uint32_t consumer_id, std::shared_ptr<Literal> data) {
-        receiver->Receive(consumer_id, data);
-      };
+  OutfeedReceiver::Callback callback = [&receiver](
+                                           Device* device, uint32_t consumer_id,
+                                           std::shared_ptr<Literal> data) {
+    receiver->Receive(consumer_id, data);
+  };
   auto outfeed_receiver =
       std::make_shared<OutfeedReceiver>(callback, clients, 128);
   outfeed_receiver->Start();
@@ -230,14 +230,14 @@ TEST(OutfeedReceiverTest, DifferentShapeForConsumerIdError) {
 TEST(OutfeedReceiverTest, InvalidConsumerIdError) {
   TF_ASSERT_OK_AND_ASSIGN(std::shared_ptr<PjRtClient> cpu_client,
                           GetCpuClient(true));
-  std::vector<std::shared_ptr<PjRtClient>> clients{cpu_client};
+  std::vector<PjRtClient*> clients{cpu_client.get()};
 
   auto receiver = absl::make_unique<Accumulator>();
-  OutfeedReceiver::Callback callback =
-      [&receiver](Device* device, std::shared_ptr<PjRtClient> client,
-                  uint32_t consumer_id, std::shared_ptr<Literal> data) {
-        receiver->Receive(consumer_id, data);
-      };
+  OutfeedReceiver::Callback callback = [&receiver](
+                                           Device* device, uint32_t consumer_id,
+                                           std::shared_ptr<Literal> data) {
+    receiver->Receive(consumer_id, data);
+  };
   auto outfeed_receiver =
       std::make_shared<OutfeedReceiver>(callback, clients, 128);
   outfeed_receiver->Start();
diff --git a/tensorflow/compiler/xla/python/py_buffer.cc b/tensorflow/compiler/xla/python/py_buffer.cc
new file mode 100644
index 00000000000..fb858d98aa5
--- /dev/null
+++ b/tensorflow/compiler/xla/python/py_buffer.cc
@@ -0,0 +1,218 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include "tensorflow/compiler/xla/python/py_buffer.h"
+
+#include "tensorflow/compiler/xla/pjrt/pjrt_client.h"
+#include "tensorflow/compiler/xla/python/python_ref_manager.h"
+#include "tensorflow/compiler/xla/python/types.h"
+
+namespace xla {
+
+namespace py = pybind11;
+
+PyBuffer::PyBuffer(std::shared_ptr<PyClient> client,
+                   std::unique_ptr<PjRtBuffer> buffer,
+                   std::unique_ptr<Traceback> traceback)
+    : client_(std::move(client)),
+      buffer_(std::move(buffer)),
+      traceback_(std::move(traceback)) {}
+
+ClientAndPtr<Device> PyBuffer::device() const {
+  return WrapWithClient(client_, buffer_->device());
+}
+
+StatusOr<std::unique_ptr<PyBuffer>> PyBuffer::CopyToDevice(
+    const ClientAndPtr<Device>& dst_device) const {
+  CHECK(dst_device.get() != nullptr);
+  GlobalPyRefManager()->CollectGarbage();
+  auto traceback = Traceback::Get();
+  py::gil_scoped_release gil_release;
+  TF_ASSIGN_OR_RETURN(std::unique_ptr<PjRtBuffer> out,
+                      buffer_->CopyToDevice(dst_device.get()));
+  return std::make_unique<PyBuffer>(dst_device.client, std::move(out),
+                                    std::move(traceback));
+}
+
+Status PyBuffer::BlockHostUntilReady() {
+  GlobalPyRefManager()->CollectGarbage();
+  py::gil_scoped_release gil_release;
+  return buffer_->BlockHostUntilReady();
+}
+
+StatusOr<std::uintptr_t> PyBuffer::UnsafeBufferPointer() const {
+  TF_ASSIGN_OR_RETURN(ShapedBuffer shaped_buffer, buffer_->AsShapedBuffer());
+  if (shaped_buffer.on_device_shape().IsTuple()) {
+    return Unimplemented(
+        "unsafe_buffer_pointer is not implemented for tuple "
+        "buffers.");
+  }
+  return absl::bit_cast<std::uintptr_t>(shaped_buffer.root_buffer().opaque());
+}
+
+StatusOr<py::dict> PyBuffer::CudaArrayInterface() const {
+  if (buffer_->device()->local_device_state()->executor()->platform_kind() !=
+      se::PlatformKind::kCuda) {
+    return InvalidArgument(
+        "__cuda_array_interface__ is only defined for NVidia GPU buffers.");
+  }
+  if (!buffer_->on_device_shape().IsArray()) {
+    return InvalidArgument(
+        "__cuda_array_interface__ is only defined for array buffers.");
+  }
+  if (buffer_->on_host_shape().element_type() == BF16) {
+    return InvalidArgument(
+        "__cuda_array_interface__ is not supported for bfloat16 buffers.");
+  }
+  TF_RET_CHECK(
+      LayoutUtil::IsMonotonicWithDim0Major(buffer_->on_host_shape().layout()));
+  TF_ASSIGN_OR_RETURN(ShapedBuffer shaped_buffer, buffer_->AsShapedBuffer());
+
+  py::dict result;
+  result["shape"] = IntSpanToTuple(shaped_buffer.on_host_shape().dimensions());
+  TF_ASSIGN_OR_RETURN(py::str typestr,
+                      TypeDescriptorForPrimitiveType(
+                          shaped_buffer.on_host_shape().element_type()));
+  result["typestr"] = std::move(typestr);
+  py::tuple data(2);
+  data[0] = py::int_(
+      absl::bit_cast<std::uintptr_t>(shaped_buffer.root_buffer().opaque()));
+  data[1] = py::bool_(true);  // read-only
+  result["data"] = std::move(data);
+  result["version"] = py::int_(2);
+  return result;
+}
+
+// PEP 3118 buffer protocol implementation.
+
+namespace {
+
+// Extra data to be kept alive by the consumer of the buffer protocol.
+struct ExtraBufferInfo {
+  explicit ExtraBufferInfo(PjRtBuffer::ScopedHold device_buffer)
+      : device_buffer(std::move(device_buffer)) {}
+
+  std::string format;
+  std::vector<Py_ssize_t> strides;
+  // We keep a reference to the TrackedDeviceBuffer that backs the
+  // PjRtBuffer. This prevents a use-after-free in the event that Delete() is
+  // called on a buffer with an live buffer protocol view. It does however mean
+  // that Delete() sometimes won't actually delete immediately.
+  PjRtBuffer::ScopedHold device_buffer;
+};
+
+int PjRtBufferGetBuffer(PyObject* exporter, Py_buffer* view, int flags) {
+  auto& buffer =
+      *py::reinterpret_borrow<py::object>(exporter).cast<PyBuffer&>().buffer();
+  Status status = [&]() {
+    // Py_buffer objects are POD C structures, so we don't need to hold the GIL.
+    // Additionally we call BlockHostUntilReady() below, which may block.
+    py::gil_scoped_release gil_release;
+
+    if (buffer.device()->platform_name() != "cpu") {
+      return InvalidArgument(
+          "Python buffer protocol is only defined for CPU buffers.");
+    }
+    if (!buffer.on_device_shape().IsArray()) {
+      return InvalidArgument(
+          "Python buffer protocol is only defined for array buffers.");
+    }
+    // If we allowed exports of formatted BF16 buffers, consumers would get
+    // confused about the type because there is no way to describe BF16 to
+    // Python.
+    if (buffer.on_host_shape().element_type() == BF16 &&
+        ((flags & PyBUF_FORMAT) == PyBUF_FORMAT)) {
+      return InvalidArgument(
+          "bfloat16 buffer format not supported by Python buffer protocol.");
+    }
+    if ((flags & PyBUF_WRITEABLE) == PyBUF_WRITEABLE) {
+      return InvalidArgument("XLA buffers are read-only.");
+    }
+    PjRtBuffer::ScopedHold device_buffer(
+        buffer.GetBufferWithExternalReference());
+    if (!device_buffer.status().ok()) {
+      return InvalidArgument("Deleted buffer used in buffer protocol.");
+    }
+    const Shape& shape = buffer.on_host_shape();
+    if (((flags & PyBUF_C_CONTIGUOUS) == PyBUF_C_CONTIGUOUS ||
+         (flags & PyBUF_STRIDES) == PyBUF_ND) &&
+        !LayoutUtil::IsMonotonicWithDim0Major(shape.layout())) {
+      return InvalidArgument("Buffer is not in C-contiguous layout.");
+    } else if ((flags & PyBUF_F_CONTIGUOUS) == PyBUF_F_CONTIGUOUS &&
+               !LayoutUtil::IsMonotonicWithDim0Minor(shape.layout())) {
+      return InvalidArgument("Buffer is not in F-contiguous layout.");
+    } else if ((flags & PyBUF_ANY_CONTIGUOUS) == PyBUF_ANY_CONTIGUOUS &&
+               !LayoutUtil::IsMonotonicWithDim0Major(shape.layout()) &&
+               !LayoutUtil::IsMonotonicWithDim0Minor(shape.layout())) {
+      return InvalidArgument("Buffer is not in contiguous layout.");
+    }
+    std::memset(view, 0, sizeof(Py_buffer));
+    CHECK_EQ(device_buffer->device_memory().size(), 1);
+    view->buf =
+        const_cast<void*>(device_buffer->device_memory().front().opaque());
+    auto extra = absl::make_unique<ExtraBufferInfo>(std::move(device_buffer));
+    view->itemsize = ShapeUtil::ByteSizeOfPrimitiveType(shape.element_type());
+    view->len = ShapeUtil::ByteSizeOf(shape);
+    view->readonly = 1;
+    if ((flags & PyBUF_FORMAT) == PyBUF_FORMAT) {
+      TF_ASSIGN_OR_RETURN(extra->format, FormatDescriptorForPrimitiveType(
+                                             shape.element_type()));
+      view->format = const_cast<char*>(extra->format.c_str());
+    }
+    if ((flags & PyBUF_ND) == PyBUF_ND) {
+      view->ndim = shape.dimensions_size();
+      static_assert(sizeof(int64) == sizeof(Py_ssize_t),
+                    "Py_ssize_t must be 64 bits");
+      if (view->ndim != 0) {
+        view->shape = reinterpret_cast<Py_ssize_t*>(
+            const_cast<int64*>(shape.dimensions().data()));
+        if ((flags & PyBUF_STRIDES) == PyBUF_STRIDES) {
+          extra->strides = ByteStridesForShape(shape);
+          view->strides = extra->strides.data();
+        }
+      }
+    }
+    TF_RETURN_IF_ERROR(buffer.BlockHostUntilReady());
+    view->internal = extra.release();
+    return Status::OK();
+  }();
+  if (!status.ok()) {
+    PyErr_SetString(PyExc_BufferError, status.ToString().c_str());
+    return -1;
+  }
+  view->obj = exporter;
+  Py_INCREF(view->obj);
+  return 0;
+}
+
+void PjRtBufferReleaseBuffer(PyObject*, Py_buffer* buffer) {
+  auto extra = static_cast<ExtraBufferInfo*>(buffer->internal);
+  delete extra;
+}
+
+PyBufferProcs PjRtBufferProcs = []() {
+  PyBufferProcs procs;
+  procs.bf_getbuffer = &PjRtBufferGetBuffer;
+  procs.bf_releasebuffer = &PjRtBufferReleaseBuffer;
+  return procs;
+}();
+
+}  // namespace
+
+/*static*/ PyBufferProcs* PyBuffer::BufferProtocol() {
+  return &PjRtBufferProcs;
+}
+
+}  // namespace xla
diff --git a/tensorflow/compiler/xla/python/py_buffer.h b/tensorflow/compiler/xla/python/py_buffer.h
new file mode 100644
index 00000000000..9fc5ae48082
--- /dev/null
+++ b/tensorflow/compiler/xla/python/py_buffer.h
@@ -0,0 +1,73 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#ifndef TENSORFLOW_COMPILER_XLA_PYTHON_PY_BUFFER_H_
+#define TENSORFLOW_COMPILER_XLA_PYTHON_PY_BUFFER_H_
+
+#include <memory>
+#include <vector>
+
+#include "tensorflow/compiler/xla/python/py_client.h"
+#include "tensorflow/compiler/xla/python/traceback.h"
+#include "tensorflow/compiler/xla/statusor.h"
+#include "tensorflow/compiler/xla/types.h"
+
+namespace xla {
+
+// Python wrapper around PjRtBuffer. We use a wrapper class:
+// a) to keep the PjRtClient alive via a std::shared_ptr<>
+// b) to add Python-specific functionality.
+class PyBuffer {
+ public:
+  PyBuffer(std::shared_ptr<PyClient> client, std::unique_ptr<PjRtBuffer> buffer,
+           std::unique_ptr<Traceback> traceback);
+
+  std::shared_ptr<PyClient> client() const { return client_; }
+  PjRtBuffer* buffer() const { return buffer_.get(); }
+
+  ClientAndPtr<Device> device() const;
+  const std::string& platform_name() const { return buffer_->platform_name(); }
+  bool is_deleted() const { return buffer_->IsDeleted(); }
+
+  StatusOr<std::unique_ptr<PyBuffer>> CopyToDevice(
+      const ClientAndPtr<Device>& dst_device) const;
+
+  void Delete() { return buffer_->Delete(); }
+
+  Status BlockHostUntilReady();
+  Status CopyToHostAsync() { return buffer_->CopyToHostAsync(); }
+
+  const Shape& shape() { return buffer_->on_host_shape(); }
+
+  StatusOr<std::uintptr_t> UnsafeBufferPointer() const;
+
+  // Implementation of the CUDA array interface for sharing GPU buffers with
+  // other Python libraries.
+  StatusOr<pybind11::dict> CudaArrayInterface() const;
+
+  // PEP 3118 Python buffer protocol implementation.
+  static PyBufferProcs* BufferProtocol();
+
+  Traceback* traceback() { return traceback_.get(); }
+
+ private:
+  std::shared_ptr<PyClient> client_;
+  std::unique_ptr<PjRtBuffer> buffer_;
+  std::unique_ptr<Traceback> traceback_;
+};
+
+}  // namespace xla
+
+#endif  // TENSORFLOW_COMPILER_XLA_PYTHON_PY_BUFFER_H_
diff --git a/tensorflow/compiler/xla/python/py_client.cc b/tensorflow/compiler/xla/python/py_client.cc
new file mode 100644
index 00000000000..d5a38fc30a5
--- /dev/null
+++ b/tensorflow/compiler/xla/python/py_client.cc
@@ -0,0 +1,130 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include "tensorflow/compiler/xla/python/py_client.h"
+
+#include "tensorflow/compiler/xla/python/py_buffer.h"
+#include "tensorflow/compiler/xla/python/py_executable.h"
+#include "tensorflow/compiler/xla/python/python_ref_manager.h"
+#include "tensorflow/compiler/xla/python/traceback.h"
+#include "tensorflow/compiler/xla/python/types.h"
+
+namespace xla {
+
+namespace py = pybind11;
+
+PyClient::PyClient(std::shared_ptr<PjRtClient> pjrt_client)
+    : pjrt_client_(std::move(pjrt_client)) {}
+
+std::vector<ClientAndPtr<Device>> PyClient::Devices() {
+  std::vector<ClientAndPtr<Device>> devices;
+  devices.reserve(pjrt_client_->devices().size());
+  for (const auto& device : pjrt_client_->devices()) {
+    devices.push_back(WrapWithClient(shared_from_this(), device.get()));
+  }
+  return devices;
+}
+
+std::vector<ClientAndPtr<Device>> PyClient::LocalDevices() {
+  std::vector<ClientAndPtr<Device>> devices;
+  devices.reserve(pjrt_client_->local_devices().size());
+  for (Device* device : pjrt_client_->local_devices()) {
+    devices.push_back(WrapWithClient(shared_from_this(), device));
+  }
+  return devices;
+}
+
+StatusOr<std::vector<std::vector<ClientAndPtr<Device>>>>
+PyClient::GetDefaultDeviceAssignment(int num_replicas, int num_partitions) {
+  TF_ASSIGN_OR_RETURN(
+      DeviceAssignment device_assignment,
+      pjrt_client_->GetDefaultDeviceAssignment(num_replicas, num_partitions));
+  std::vector<std::vector<ClientAndPtr<Device>>> result;
+  result.resize(num_replicas);
+  for (int r = 0; r < num_replicas; ++r) {
+    result[r].resize(num_partitions);
+    for (int p = 0; p < num_partitions; ++p) {
+      int device_id = device_assignment(r, p);
+      auto iter = pjrt_client_->id_to_device().find(device_id);
+      CHECK(iter != pjrt_client_->id_to_device().end()) << device_id;
+      result[r][p] = WrapWithClient(shared_from_this(), iter->second);
+    }
+  }
+  return result;
+}
+
+StatusOr<std::vector<ClientAndPtr<Device>>>
+PyClient::GetDefaultDeviceAssignment1D(int num_replicas) {
+  TF_ASSIGN_OR_RETURN(DeviceAssignment device_assignment,
+                      pjrt_client_->GetDefaultDeviceAssignment(
+                          num_replicas, /*num_partitions=*/1));
+  std::vector<ClientAndPtr<Device>> result;
+  for (int i = 0; i < num_replicas; ++i) {
+    int device_id = device_assignment(i, 0);
+    auto iter = pjrt_client_->id_to_device().find(device_id);
+    CHECK(iter != pjrt_client_->id_to_device().end()) << device_id;
+    result.push_back(WrapWithClient(shared_from_this(), iter->second));
+  }
+  return result;
+}
+
+StatusOr<std::unique_ptr<PyBuffer>> PyClient::BufferFromPyal(
+    const pybind11::object& argument, Device* device, bool force_copy) {
+  if (device == nullptr) {
+    TF_RET_CHECK(!pjrt_client_->local_devices().empty());
+    device = pjrt_client_->local_devices().front();
+  }
+  CHECK(device != nullptr);
+  auto iter = pjrt_client_->id_to_device().find(device->id());
+  if (iter->second != device) {
+    return InvalidArgument("Cannot copy value to device '%s' with '%s' backend",
+                           device->DebugString(),
+                           pjrt_client_->platform_name());
+  }
+  GlobalPyRefManager()->CollectGarbage();
+
+  absl::optional<CastToArrayResult> c = CastToArray(argument);
+  if (!c) {
+    return InvalidArgument("from_python argument must be an array.");
+  }
+
+  TF_ASSIGN_OR_RETURN(PythonBufferTree tree, GetPythonBufferTree(argument));
+  std::shared_ptr<PythonRefManager::ManagedPyObjects> py_buffer_ref =
+      GlobalPyRefManager()->ManageReference(std::move(c->array));
+
+  auto traceback = Traceback::Get();
+
+  py::gil_scoped_release gil_release;
+  TF_ASSIGN_OR_RETURN(
+      std::unique_ptr<PjRtBuffer> buffer,
+      PjRtBuffer::FromHostBuffer(c->buf_ptr, c->shape, force_copy,
+                                 std::move(py_buffer_ref), pjrt_client_.get(),
+                                 device));
+  return std::make_unique<PyBuffer>(shared_from_this(), std::move(buffer),
+                                    std::move(traceback));
+}
+
+StatusOr<std::unique_ptr<PyExecutable>> PyClient::Compile(
+    const XlaComputation& computation, CompileOptions options) {
+  auto traceback = Traceback::Get();
+  py::gil_scoped_release gil_release;
+  TF_ASSIGN_OR_RETURN(std::unique_ptr<PjRtExecutable> executable,
+                      PjRtExecutable::Compile(computation, pjrt_client_.get(),
+                                              std::move(options)));
+  return std::make_unique<PyExecutable>(
+      shared_from_this(), std::move(executable), std::move(traceback));
+}
+
+}  // namespace xla
diff --git a/tensorflow/compiler/xla/python/py_client.h b/tensorflow/compiler/xla/python/py_client.h
new file mode 100644
index 00000000000..112568c5961
--- /dev/null
+++ b/tensorflow/compiler/xla/python/py_client.h
@@ -0,0 +1,136 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#ifndef TENSORFLOW_COMPILER_XLA_PYTHON_PY_CLIENT_H_
+#define TENSORFLOW_COMPILER_XLA_PYTHON_PY_CLIENT_H_
+
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "absl/types/optional.h"
+#include "pybind11/pybind11.h"
+#include "tensorflow/compiler/xla/pjrt/pjrt_client.h"
+#include "tensorflow/compiler/xla/statusor.h"
+#include "tensorflow/compiler/xla/types.h"
+
+namespace xla {
+
+class PyBuffer;
+class PyClient;
+class PyExecutable;
+
+// Custom holder types.
+//
+// We must keep the PyClient object alive as long as any of the runtime
+// objects are alive. Since we don't have a lot of control over Python
+// destructor ordering, we keep the PyClient object as a std::shared_ptr<>,
+// and ensure that each Python runtime object holds a reference to the
+// PyClient. An alternative design would be to keep a single global
+// singleton PyClient, although this seems less flexible, especially for
+// writing tests.
+//
+// To maintain PyClient references, we define pybind11 holder classes that
+// are custom smart pointers that also keep a reference to a PyClient.
+// pybind11 has a `keep_alive` feature that has a similar goal, but it doesn't
+// seem sufficiently flexible to describe ownership relationships in cases where
+// the ownership doesn't pertain to a direct argument or return value of a
+// function. Another alternative to the holder classes would be to create proxy
+// objects that contain both a reference and a runtime class; holder classes
+// seem less tedious to define.
+
+// A pair of a PyClient reference and an unowned pointer to T.
+template <typename T>
+struct ClientAndPtr {
+  ClientAndPtr() = default;
+  // pybind11 requires that we define a constructor that takes a raw pointer,
+  // but it should be unreachable.
+  explicit ClientAndPtr(T*) {
+    LOG(FATAL) << "ClientAndPtr should constructed via WrapWithClient.";
+  }
+
+  ClientAndPtr(const ClientAndPtr&) = default;
+  ClientAndPtr(ClientAndPtr&&) = default;
+  ClientAndPtr& operator=(const ClientAndPtr&) = default;
+  ClientAndPtr& operator=(ClientAndPtr&&) = default;
+
+  std::shared_ptr<PyClient> client;
+  T* contents;
+
+  T* get() const { return contents; }
+  T* operator->() const { return contents; }
+  T& operator*() const { return *contents; }
+};
+
+// By defining a templated helper function, we can use return type deduction
+// and avoid specifying types at the caller.
+template <typename T>
+ClientAndPtr<T> WrapWithClient(std::shared_ptr<PyClient> client, T* contents) {
+  ClientAndPtr<T> result;
+  result.client = std::move(client);
+  result.contents = contents;
+  return result;
+}
+
+// Python wrapper around PjRtClient.
+// We use a wrapper class to add Python-specific functionality.
+class PyClient : public std::enable_shared_from_this<PyClient> {
+ public:
+  explicit PyClient(std::shared_ptr<PjRtClient> pjrt_client);
+
+  PjRtClient* pjrt_client() const { return pjrt_client_.get(); }
+
+  const std::string& platform_name() const {
+    return pjrt_client_->platform_name();
+  }
+  int local_device_count() const { return pjrt_client_->local_device_count(); }
+  int device_count() const { return pjrt_client_->device_count(); }
+  int host_id() const { return pjrt_client_->host_id(); }
+
+  std::vector<ClientAndPtr<Device>> Devices();
+  std::vector<ClientAndPtr<Device>> LocalDevices();
+
+  StatusOr<std::vector<std::vector<ClientAndPtr<Device>>>>
+  GetDefaultDeviceAssignment(int num_replicas, int num_partitions);
+
+  // TODO(skye): delete after all callers can handle 2D output
+  StatusOr<std::vector<ClientAndPtr<Device>>> GetDefaultDeviceAssignment1D(
+      int num_replicas);
+
+  StatusOr<ChannelHandle> CreateChannelHandle() {
+    return pjrt_client_->client()->CreateChannelHandle();
+  }
+  StatusOr<ChannelHandle> CreateDeviceToHostChannelHandle() {
+    return pjrt_client_->client()->CreateDeviceToHostChannelHandle();
+  }
+  StatusOr<ChannelHandle> CreateHostToDeviceChannelHandle() {
+    return pjrt_client_->client()->CreateHostToDeviceChannelHandle();
+  }
+
+  StatusOr<std::unique_ptr<PyBuffer>> BufferFromPyal(
+      const pybind11::object& argument, Device* device, bool force_copy);
+
+  StatusOr<std::unique_ptr<PyExecutable>> Compile(
+      const XlaComputation& computation, CompileOptions options);
+
+ private:
+  std::shared_ptr<PjRtClient> pjrt_client_;
+};
+
+}  // namespace xla
+
+PYBIND11_DECLARE_HOLDER_TYPE(T, xla::ClientAndPtr<T>);
+
+#endif  // TENSORFLOW_COMPILER_XLA_PYTHON_PY_CLIENT_H_
diff --git a/tensorflow/compiler/xla/python/py_executable.cc b/tensorflow/compiler/xla/python/py_executable.cc
new file mode 100644
index 00000000000..930d9c4cbda
--- /dev/null
+++ b/tensorflow/compiler/xla/python/py_executable.cc
@@ -0,0 +1,101 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include "tensorflow/compiler/xla/python/py_executable.h"
+
+#include "absl/algorithm/container.h"
+
+namespace xla {
+
+namespace py = pybind11;
+
+PyExecutable::PyExecutable(std::shared_ptr<PyClient> client,
+                           std::unique_ptr<PjRtExecutable> executable,
+                           std::unique_ptr<Traceback> traceback)
+    : client_(std::move(client)),
+      executable_(std::move(executable)),
+      traceback_(std::move(traceback)) {}
+
+std::vector<ClientAndPtr<Device>> PyExecutable::LocalDevices() const {
+  std::vector<ClientAndPtr<Device>> devices;
+  devices.reserve(executable_->local_devices().size());
+  for (Device* device : executable_->local_devices()) {
+    devices.push_back(WrapWithClient(client_, device));
+  }
+  return devices;
+}
+
+StatusOr<std::vector<std::unique_ptr<PyBuffer>>> PyExecutable::Execute(
+    absl::Span<PyBuffer* const> args) {
+  auto traceback = Traceback::Get();
+  py::gil_scoped_release gil_release;
+  ExecuteOptions options;
+  options.untuple_result = true;
+  std::vector<PjRtBuffer*> arg_buffers(args.size());
+  absl::c_transform(args, arg_buffers.begin(),
+                    [](PyBuffer* buf) { return buf->buffer(); });
+  TF_ASSIGN_OR_RETURN(std::vector<std::unique_ptr<PjRtBuffer>> output_buffers,
+                      executable_->Execute(arg_buffers, options));
+  std::vector<std::unique_ptr<PyBuffer>> outputs;
+  outputs.reserve(output_buffers.size());
+  for (auto& buffer : output_buffers) {
+    outputs.push_back(std::make_unique<PyBuffer>(client_, std::move(buffer),
+                                                 std::move(traceback)));
+  }
+  return outputs;
+}
+
+StatusOr<std::vector<std::vector<std::unique_ptr<PyBuffer>>>>
+PyExecutable::ExecuteOnLocalDevices(
+    absl::Span<const std::vector<PyBuffer*>> args) {
+  auto traceback = Traceback::Get();
+  py::gil_scoped_release gil_release;
+  ExecuteOptions options;
+  options.untuple_result = true;
+  std::vector<std::vector<PjRtBuffer*>> arg_buffers(args.size());
+  for (int computation = 0; computation < args.size(); ++computation) {
+    arg_buffers[computation].resize(args[computation].size());
+    absl::c_transform(args[computation], arg_buffers[computation].begin(),
+                      [](PyBuffer* buf) { return buf->buffer(); });
+  }
+  TF_ASSIGN_OR_RETURN(
+      std::vector<std::vector<std::unique_ptr<PjRtBuffer>>> output_buffers,
+      executable_->ExecuteOnLocalDevices(arg_buffers, options));
+  std::vector<std::vector<std::unique_ptr<PyBuffer>>> outputs;
+  outputs.resize(output_buffers.size());
+  for (int computation = 0; computation < output_buffers.size();
+       ++computation) {
+    for (auto& buffer : output_buffers[computation]) {
+      outputs[computation].push_back(std::make_unique<PyBuffer>(
+          client_, std::move(buffer), std::move(traceback)));
+    }
+  }
+  return outputs;
+}
+
+StatusOr<std::vector<std::shared_ptr<HloModule>>> PyExecutable::HloModules()
+    const {
+  std::vector<std::shared_ptr<HloModule>> modules;
+  modules.reserve(executable_->executables().size());
+  for (const auto& local_exec : executable_->executables()) {
+    if (!local_exec->executable()->has_module()) {
+      return InvalidArgument("Executable does not have HLO modules.");
+    }
+    modules.push_back(local_exec->executable()->shared_module());
+  }
+  return std::move(modules);
+}
+
+}  // namespace xla
diff --git a/tensorflow/compiler/xla/python/py_executable.h b/tensorflow/compiler/xla/python/py_executable.h
new file mode 100644
index 00000000000..c01ebc8ec82
--- /dev/null
+++ b/tensorflow/compiler/xla/python/py_executable.h
@@ -0,0 +1,74 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#ifndef TENSORFLOW_COMPILER_XLA_PYTHON_PY_EXECUTABLE_H_
+#define TENSORFLOW_COMPILER_XLA_PYTHON_PY_EXECUTABLE_H_
+
+#include <memory>
+#include <utility>
+#include <vector>
+
+#include "absl/types/span.h"
+#include "tensorflow/compiler/xla/pjrt/pjrt_client.h"
+#include "tensorflow/compiler/xla/python/py_buffer.h"
+#include "tensorflow/compiler/xla/python/py_client.h"
+#include "tensorflow/compiler/xla/python/traceback.h"
+#include "tensorflow/compiler/xla/statusor.h"
+#include "tensorflow/compiler/xla/types.h"
+
+namespace xla {
+
+// Python wrapper around PjRtExecutable. We use a wrapper class:
+// a) to keep the PyClient alive via a std::shared_ptr<>
+// b) to add Python-specific functionality.
+class PyExecutable {
+ public:
+  PyExecutable(std::shared_ptr<PyClient> client,
+               std::unique_ptr<PjRtExecutable> executable,
+               std::unique_ptr<Traceback> traceback);
+
+  std::shared_ptr<PyClient> client() const { return client_; }
+
+  const std::vector<std::pair<int, int>>& local_logical_device_ids() const {
+    return executable_->local_logical_device_ids();
+  }
+
+  std::vector<ClientAndPtr<Device>> LocalDevices() const;
+
+  int64 SizeOfGeneratedCodeInBytes() const {
+    return executable_->SizeOfGeneratedCodeInBytes();
+  }
+
+  void Delete() { return executable_->Delete(); }
+
+  StatusOr<std::vector<std::unique_ptr<PyBuffer>>> Execute(
+      absl::Span<PyBuffer* const> args);
+
+  StatusOr<std::vector<std::vector<std::unique_ptr<PyBuffer>>>>
+  ExecuteOnLocalDevices(absl::Span<const std::vector<PyBuffer*>> args);
+
+  StatusOr<std::vector<std::shared_ptr<HloModule>>> HloModules() const;
+
+  Traceback* traceback() { return traceback_.get(); }
+
+ private:
+  std::shared_ptr<PyClient> client_;
+  std::unique_ptr<PjRtExecutable> executable_;
+  std::unique_ptr<Traceback> traceback_;
+};
+
+}  // namespace xla
+
+#endif  // TENSORFLOW_COMPILER_XLA_PYTHON_PY_EXECUTABLE_H_
diff --git a/tensorflow/compiler/xla/python/python_ref_manager.cc b/tensorflow/compiler/xla/python/python_ref_manager.cc
index cf449801205..2083f5acbc7 100644
--- a/tensorflow/compiler/xla/python/python_ref_manager.cc
+++ b/tensorflow/compiler/xla/python/python_ref_manager.cc
@@ -50,6 +50,22 @@ PythonRefManager::ManageReferences(absl::Span<py::object> objects) {
   return std::make_shared<ManagedPyObjects>(this, objects);
 }
 
+void PythonRefManager::AddGarbage(absl::Span<py::object> garbage) {
+  absl::MutexLock lock(&mu_);
+  for (py::object& o : garbage) {
+    python_garbage_.push_back(std::move(o));
+  }
+}
+
+void PythonRefManager::AddGarbage(
+    absl::Span<std::pair<PyCodeObject*, int> const> garbage) {
+  absl::MutexLock lock(&mu_);
+  for (const auto& o : garbage) {
+    python_garbage_.push_back(py::reinterpret_steal<py::object>(
+        reinterpret_cast<PyObject*>(o.first)));
+  }
+}
+
 void PythonRefManager::CollectGarbage() {
   // TODO(phawkins): we should CHECK(PyGILState_Check());
   std::deque<pybind11::object> garbage;
diff --git a/tensorflow/compiler/xla/python/python_ref_manager.h b/tensorflow/compiler/xla/python/python_ref_manager.h
index 0ad533c695f..7af2d64fc8f 100644
--- a/tensorflow/compiler/xla/python/python_ref_manager.h
+++ b/tensorflow/compiler/xla/python/python_ref_manager.h
@@ -66,6 +66,10 @@ class PythonRefManager {
   std::shared_ptr<ManagedPyObjects> ManageReferences(
       absl::Span<pybind11::object> objects);
 
+  // Adds garbage objects to the manager.
+  void AddGarbage(absl::Span<pybind11::object> garbage);
+  void AddGarbage(absl::Span<std::pair<PyCodeObject*, int> const> garbage);
+
   // Releases the contents of python_garbage_. Requires that the GIL is held.
   // The client calls this method during API entry points where the GIL is held
   // to free any garbage that has accumulated.
diff --git a/tensorflow/compiler/xla/python/tpu_driver/client/tpu_client_extension.cc b/tensorflow/compiler/xla/python/tpu_driver/client/tpu_client_extension.cc
index f44d69656e6..9a794b79c5c 100644
--- a/tensorflow/compiler/xla/python/tpu_driver/client/tpu_client_extension.cc
+++ b/tensorflow/compiler/xla/python/tpu_driver/client/tpu_client_extension.cc
@@ -173,9 +173,13 @@ PYBIND11_MODULE(tpu_client_extension, m) {
       .def("shape", &PyTpuBuffer::on_host_shape)
       .def("device", &PyTpuBuffer::device)
       .def("platform", &PyTpuBuffer::platform_name)
-      .def("is_deleted", [](const PyTpuBuffer& buffer) {
-        return buffer.DeviceBuffer() == nullptr;
-      });
+      .def("is_deleted",
+           [](const PyTpuBuffer& buffer) {
+             return buffer.DeviceBuffer() == nullptr;
+           })
+      // TODO(phawkins): implement traceback support.
+      .def_property_readonly("traceback",
+                             [](PyTpuBuffer*) { return py::none(); });
 
   py::class_<PyTpuExecutable>(m, "TpuExecutable")
       .def("local_logical_device_ids",
@@ -193,7 +197,10 @@ PYBIND11_MODULE(tpu_client_extension, m) {
       .def("execute", &PyTpuExecutable::Execute,
            py::call_guard<py::gil_scoped_release>(), py::arg("arguments"))
       .def("execute_on_local_devices", &PyTpuExecutable::ExecuteOnLocalDevices,
-           py::call_guard<py::gil_scoped_release>(), py::arg("arguments"));
+           py::call_guard<py::gil_scoped_release>(), py::arg("arguments"))
+      // TODO(phawkins): implement traceback support.
+      .def_property_readonly("traceback",
+                             [](PyTpuExecutable*) { return py::none(); });
 
   py::class_<TpuDevice, Device, std::shared_ptr<TpuDevice>>(m, "TpuDevice")
       .def_property_readonly("coords", &TpuDevice::coords)
diff --git a/tensorflow/compiler/xla/python/traceback.cc b/tensorflow/compiler/xla/python/traceback.cc
new file mode 100644
index 00000000000..8e4e78bb540
--- /dev/null
+++ b/tensorflow/compiler/xla/python/traceback.cc
@@ -0,0 +1,81 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include "tensorflow/compiler/xla/python/traceback.h"
+
+#include "absl/strings/str_format.h"
+#include "absl/strings/str_join.h"
+#include "pybind11/pytypes.h"
+#include "tensorflow/compiler/xla/python/python_ref_manager.h"
+#include "tensorflow/core/platform/logging.h"
+
+namespace xla {
+
+namespace py = pybind11;
+
+bool Traceback::enabled_ = false;
+
+Traceback::~Traceback() {
+  // We want Traceback objects to be safe to destroy without holding the
+  // GIL, so we defer destruction of the strings.
+  GlobalPyRefManager()->AddGarbage(frames_);
+}
+
+std::string Traceback::Frame::ToString() const {
+  return absl::StrFormat("%s;%s:%d", function_name, file_name, line_num);
+}
+
+std::string Traceback::ToString() const {
+  std::vector<std::string> frame_strs;
+  frame_strs.reserve(frames_.size());
+  for (const Frame& frame : Frames()) {
+    frame_strs.push_back(frame.ToString());
+  }
+  return absl::StrJoin(frame_strs, "\n");
+}
+
+std::vector<Traceback::Frame> Traceback::Frames() const {
+  // We require the GIL because we manipulate Python strings.
+  CHECK(PyGILState_Check());
+  std::vector<Traceback::Frame> frames;
+  frames.reserve(frames_.size());
+  for (const auto& frame : frames_) {
+    frames.push_back(Frame{
+        std::string(py::reinterpret_borrow<py::str>(frame.first->co_filename)),
+        std::string(py::reinterpret_borrow<py::str>(frame.first->co_name)),
+        frame.first->co_firstlineno,
+        PyCode_Addr2Line(frame.first, frame.second)});
+  }
+  return frames;
+}
+
+std::unique_ptr<Traceback> Traceback::Get() {
+  DCHECK(PyGILState_Check());
+  if (!enabled_) {
+    return nullptr;
+  }
+  auto tb = std::make_unique<Traceback>();
+  const PyThreadState* thread_state = PyThreadState_GET();
+  for (PyFrameObject* py_frame = thread_state->frame; py_frame != nullptr;
+       py_frame = py_frame->f_back) {
+    Py_INCREF(py_frame->f_code);
+    tb->frames_.emplace_back(py_frame->f_code, py_frame->f_lasti);
+  }
+  return tb;
+}
+
+void Traceback::SetEnabled(bool enabled) { enabled_ = enabled; }
+
+}  // namespace xla
diff --git a/tensorflow/compiler/xla/python/traceback.h b/tensorflow/compiler/xla/python/traceback.h
new file mode 100644
index 00000000000..73b3566bf95
--- /dev/null
+++ b/tensorflow/compiler/xla/python/traceback.h
@@ -0,0 +1,69 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#ifndef TENSORFLOW_COMPILER_XLA_PYTHON_TRACEBACK_H_
+#define TENSORFLOW_COMPILER_XLA_PYTHON_TRACEBACK_H_
+
+#include <memory>
+#include <string>
+#include <vector>
+
+#include "absl/container/inlined_vector.h"
+#include "pybind11/pybind11.h"
+
+namespace xla {
+
+// Represents a Python traceback.
+class Traceback {
+ public:
+  // Require GIL.
+  static std::unique_ptr<Traceback> Get();
+
+  // Require GIL.
+  static bool enabled() { return enabled_; }
+  // Require GIL.
+  static void SetEnabled(bool enabled);
+
+  Traceback() = default;
+  ~Traceback();
+
+  Traceback(const Traceback&) = delete;
+  Traceback(Traceback&&) = delete;
+  Traceback& operator=(const Traceback&) = delete;
+  Traceback& operator=(Traceback&&) = delete;
+
+  // Requires the GIL be held.
+  std::string ToString() const;
+
+  struct Frame {
+    pybind11::str file_name;
+    pybind11::str function_name;
+    int function_start_line;
+    int line_num;
+
+    std::string ToString() const;
+  };
+  std::vector<Frame> Frames() const;
+
+ private:
+  absl::InlinedVector<std::pair<PyCodeObject*, int>, 32> frames_;
+
+  // Protected by GIL.
+  static bool enabled_;
+};
+
+}  // namespace xla
+
+#endif  // TENSORFLOW_COMPILER_XLA_PYTHON_TRACEBACK_H_
diff --git a/tensorflow/compiler/xla/python/types.h b/tensorflow/compiler/xla/python/types.h
index 673f403d91e..6f4684f2ceb 100644
--- a/tensorflow/compiler/xla/python/types.h
+++ b/tensorflow/compiler/xla/python/types.h
@@ -26,7 +26,6 @@ limitations under the License.
 #include "pybind11/pybind11.h"
 #include "pybind11/stl.h"
 #include "tensorflow/compiler/xla/literal.h"
-#include "tensorflow/compiler/xla/pjrt/pjrt_client.h"
 #include "tensorflow/compiler/xla/shape.h"
 #include "tensorflow/compiler/xla/status.h"
 #include "tensorflow/compiler/xla/statusor.h"
@@ -36,97 +35,6 @@ limitations under the License.
 
 namespace xla {
 
-// Custom holder types.
-//
-// We must keep the PjRtClient object alive as long as any of the runtime
-// objects are alive. Since we don't have a lot of control over Python
-// destructor ordering, we keep the PjRtClient object as a std::shared_ptr<>,
-// and ensure that each Python runtime object holds a reference to the
-// PjRtClient. An alternative design would be to keep a single global
-// singleton PjRtClient, although this seems less flexible, especially for
-// writing tests.
-//
-// To maintain PjRtClient references, we define pybind11 holder classes that
-// are custom smart pointers that also keep a reference to a PjRtClient.
-// pybind11 has a `keep_alive` feature that has a similar goal, but it doesn't
-// seem sufficiently flexible to describe ownership relationships in cases where
-// the ownership doesn't pertain to a direct argument or return value of a
-// function. Another alternative to the holder classes would be to create proxy
-// objects that contain both a reference and a runtime class; holder classes
-// seem less tedious to define.
-
-// A pair of a PjRtClient reference and an unowned pointer to T.
-template <typename T>
-struct ClientAndPtr {
-  ClientAndPtr() = default;
-  // pybind11 requires that we define a constructor that takes a raw pointer,
-  // but it should be unreachable.
-  explicit ClientAndPtr(T*) {
-    LOG(FATAL) << "ClientAndPtr should constructed via WrapWithClient.";
-  }
-
-  ClientAndPtr(const ClientAndPtr&) = default;
-  ClientAndPtr(ClientAndPtr&&) = default;
-  ClientAndPtr& operator=(const ClientAndPtr&) = default;
-  ClientAndPtr& operator=(ClientAndPtr&&) = default;
-
-  std::shared_ptr<PjRtClient> client;
-  T* contents;
-
-  T* get() const { return contents; }
-  T* operator->() const { return contents; }
-  T& operator*() const { return *contents; }
-};
-
-// By defining a templated helper function, we can use return type deduction
-// and avoid specifying types at the caller.
-template <typename T>
-ClientAndPtr<T> WrapWithClient(std::shared_ptr<PjRtClient> client,
-                               T* contents) {
-  ClientAndPtr<T> result;
-  result.client = std::move(client);
-  result.contents = contents;
-  return result;
-}
-
-// A pair of a PjRtClient reference and an owned pointer to T.
-template <typename T>
-struct ClientAndUniquePtr {
-  ClientAndUniquePtr() = default;
-  // pybind11 requires that we define a constructor that takes a raw pointer,
-  // but it should be unreachable.
-  explicit ClientAndUniquePtr(T*) {
-    LOG(FATAL) << "ClientAndUniquePtr should constructed via WrapWithClient.";
-  }
-  ClientAndUniquePtr(const ClientAndUniquePtr&) = delete;
-  ClientAndUniquePtr(ClientAndUniquePtr&&) = default;
-  ClientAndUniquePtr& operator=(const ClientAndUniquePtr&) = delete;
-  ClientAndUniquePtr& operator=(ClientAndUniquePtr&&) = default;
-
-  std::shared_ptr<PjRtClient> client;
-  std::unique_ptr<T> contents;
-
-  T* get() const { return contents.get(); }
-  T* operator->() const { return contents.get(); }
-  T& operator*() const { return *contents; }
-};
-
-template <typename T>
-ClientAndUniquePtr<T> WrapWithClient(std::shared_ptr<PjRtClient> client,
-                                     std::unique_ptr<T> contents) {
-  ClientAndUniquePtr<T> result;
-  result.client = std::move(client);
-  result.contents = std::move(contents);
-  return result;
-}
-
-}  // namespace xla
-
-PYBIND11_DECLARE_HOLDER_TYPE(T, xla::ClientAndPtr<T>);
-PYBIND11_DECLARE_HOLDER_TYPE(T, xla::ClientAndUniquePtr<T>);
-
-namespace xla {
-
 // Initializes the NumPy API for the use of the types module.
 bool InitializeNumpyAPIForTypes();
 
diff --git a/tensorflow/compiler/xla/python/xla.cc b/tensorflow/compiler/xla/python/xla.cc
index 6ebe5e85245..3bb00717617 100644
--- a/tensorflow/compiler/xla/python/xla.cc
+++ b/tensorflow/compiler/xla/python/xla.cc
@@ -29,6 +29,7 @@ limitations under the License.
 #include "pybind11/numpy.h"
 #include "pybind11/pybind11.h"
 #include "pybind11/pytypes.h"
+#include "pybind11/stl_bind.h"
 #include "tensorflow/compiler/xla/client/client_library.h"
 #include "tensorflow/compiler/xla/client/local_client.h"
 #include "tensorflow/compiler/xla/client/xla_builder.h"
@@ -45,7 +46,10 @@ limitations under the License.
 #include "tensorflow/compiler/xla/python/dlpack.h"
 #include "tensorflow/compiler/xla/python/ops.h"
 #include "tensorflow/compiler/xla/python/outfeed_receiver_py.h"
+#include "tensorflow/compiler/xla/python/py_buffer.h"
+#include "tensorflow/compiler/xla/python/py_executable.h"
 #include "tensorflow/compiler/xla/python/python_ref_manager.h"
+#include "tensorflow/compiler/xla/python/traceback.h"
 #include "tensorflow/compiler/xla/python/types.h"
 #include "tensorflow/compiler/xla/service/custom_call_target_registry.h"
 #include "tensorflow/compiler/xla/service/hlo_graph_dumper.h"
@@ -156,154 +160,6 @@ Status PyRegisterCustomCallTarget(const std::string& fn_name,
   return Status::OK();
 }
 
-// PEP 3118 buffer protocol implementation.
-
-// Extra data to be kept alive by the consumer of the buffer protocol.
-struct ExtraBufferInfo {
-  explicit ExtraBufferInfo(PjRtBuffer::ScopedHold device_buffer)
-      : device_buffer(std::move(device_buffer)) {}
-
-  std::string format;
-  std::vector<Py_ssize_t> strides;
-  // We keep a reference to the TrackedDeviceBuffer that backs the
-  // PjRtBuffer. This prevents a use-after-free in the event that Delete() is
-  // called on a buffer with an live buffer protocol view. It does however mean
-  // that Delete() sometimes won't actually delete immediately.
-  PjRtBuffer::ScopedHold device_buffer;
-};
-
-int PjRtBufferGetBuffer(PyObject* exporter, Py_buffer* view, int flags) {
-  auto& buffer =
-      py::reinterpret_borrow<py::object>(exporter).cast<PjRtBuffer&>();
-  Status status = [&]() {
-    // Py_buffer objects are POD C structures, so we don't need to hold the GIL.
-    // Additionally we call BlockHostUntilReady() below, which may block.
-    py::gil_scoped_release gil_release;
-
-    if (buffer.device()->platform_name() != "cpu") {
-      return InvalidArgument(
-          "Python buffer protocol is only defined for CPU buffers.");
-    }
-    if (!buffer.on_device_shape().IsArray()) {
-      return InvalidArgument(
-          "Python buffer protocol is only defined for array buffers.");
-    }
-    // If we allowed exports of formatted BF16 buffers, consumers would get
-    // confused about the type because there is no way to describe BF16 to
-    // Python.
-    if (buffer.on_host_shape().element_type() == BF16 &&
-        ((flags & PyBUF_FORMAT) == PyBUF_FORMAT)) {
-      return InvalidArgument(
-          "bfloat16 buffer format not supported by Python buffer protocol.");
-    }
-    if ((flags & PyBUF_WRITEABLE) == PyBUF_WRITEABLE) {
-      return InvalidArgument("XLA buffers are read-only.");
-    }
-    PjRtBuffer::ScopedHold device_buffer(
-        buffer.GetBufferWithExternalReference());
-    if (!device_buffer.status().ok()) {
-      return InvalidArgument("Deleted buffer used in buffer protocol.");
-    }
-    const Shape& shape = buffer.on_host_shape();
-    if (((flags & PyBUF_C_CONTIGUOUS) == PyBUF_C_CONTIGUOUS ||
-         (flags & PyBUF_STRIDES) == PyBUF_ND) &&
-        !LayoutUtil::IsMonotonicWithDim0Major(shape.layout())) {
-      return InvalidArgument("Buffer is not in C-contiguous layout.");
-    } else if ((flags & PyBUF_F_CONTIGUOUS) == PyBUF_F_CONTIGUOUS &&
-               !LayoutUtil::IsMonotonicWithDim0Minor(shape.layout())) {
-      return InvalidArgument("Buffer is not in F-contiguous layout.");
-    } else if ((flags & PyBUF_ANY_CONTIGUOUS) == PyBUF_ANY_CONTIGUOUS &&
-               !LayoutUtil::IsMonotonicWithDim0Major(shape.layout()) &&
-               !LayoutUtil::IsMonotonicWithDim0Minor(shape.layout())) {
-      return InvalidArgument("Buffer is not in contiguous layout.");
-    }
-    std::memset(view, 0, sizeof(Py_buffer));
-    CHECK_EQ(device_buffer->device_memory().size(), 1);
-    view->buf =
-        const_cast<void*>(device_buffer->device_memory().front().opaque());
-    auto extra = absl::make_unique<ExtraBufferInfo>(std::move(device_buffer));
-    view->itemsize = ShapeUtil::ByteSizeOfPrimitiveType(shape.element_type());
-    view->len = ShapeUtil::ByteSizeOf(shape);
-    view->readonly = 1;
-    if ((flags & PyBUF_FORMAT) == PyBUF_FORMAT) {
-      TF_ASSIGN_OR_RETURN(extra->format, FormatDescriptorForPrimitiveType(
-                                             shape.element_type()));
-      view->format = const_cast<char*>(extra->format.c_str());
-    }
-    if ((flags & PyBUF_ND) == PyBUF_ND) {
-      view->ndim = shape.dimensions_size();
-      static_assert(sizeof(int64) == sizeof(Py_ssize_t),
-                    "Py_ssize_t must be 64 bits");
-      if (view->ndim != 0) {
-        view->shape = reinterpret_cast<Py_ssize_t*>(
-            const_cast<int64*>(shape.dimensions().data()));
-        if ((flags & PyBUF_STRIDES) == PyBUF_STRIDES) {
-          extra->strides = ByteStridesForShape(shape);
-          view->strides = extra->strides.data();
-        }
-      }
-    }
-    TF_RETURN_IF_ERROR(buffer.BlockHostUntilReady());
-    view->internal = extra.release();
-    return Status::OK();
-  }();
-  if (!status.ok()) {
-    PyErr_SetString(PyExc_BufferError, status.ToString().c_str());
-    return -1;
-  }
-  view->obj = exporter;
-  Py_INCREF(view->obj);
-  return 0;
-}
-
-void PjRtBufferReleaseBuffer(PyObject*, Py_buffer* buffer) {
-  auto extra = static_cast<ExtraBufferInfo*>(buffer->internal);
-  delete extra;
-}
-
-PyBufferProcs PjRtBufferProcs = []() {
-  PyBufferProcs procs;
-  procs.bf_getbuffer = &PjRtBufferGetBuffer;
-  procs.bf_releasebuffer = &PjRtBufferReleaseBuffer;
-  return procs;
-}();
-
-// Implementation of the CUDA array interface for sharing GPU buffers with other
-// Python libraries.
-StatusOr<py::dict> PjRtBufferCudaArrayInterface(const PjRtBuffer& buffer) {
-  if (buffer.device()->local_device_state()->executor()->platform_kind() !=
-      se::PlatformKind::kCuda) {
-    return InvalidArgument(
-        "__cuda_array_interface__ is only defined for NVidia GPU buffers.");
-  }
-  if (!buffer.on_device_shape().IsArray()) {
-    return InvalidArgument(
-        "__cuda_array_interface__ is only defined for array buffers.");
-  }
-  if (buffer.on_host_shape().element_type() == BF16) {
-    return InvalidArgument(
-        "__cuda_array_interface__ is not supported for bfloat16 buffers.");
-  }
-  TF_RET_CHECK(
-      LayoutUtil::IsMonotonicWithDim0Major(buffer.on_host_shape().layout()));
-  TF_ASSIGN_OR_RETURN(ShapedBuffer shaped_buffer, buffer.AsShapedBuffer());
-
-  py::dict result;
-  result["shape"] = IntSpanToTuple(shaped_buffer.on_host_shape().dimensions());
-  TF_ASSIGN_OR_RETURN(py::str typestr,
-                      TypeDescriptorForPrimitiveType(
-                          shaped_buffer.on_host_shape().element_type()));
-  result["typestr"] = std::move(typestr);
-  py::tuple data(2);
-  data[0] = py::int_(
-      absl::bit_cast<std::uintptr_t>(shaped_buffer.root_buffer().opaque()));
-  data[1] = py::bool_(true);  // read-only
-  result["data"] = std::move(data);
-  result["version"] = py::int_(2);
-  return result;
-}
-
-
 void BuildProfilerSubmodule(py::module* m) {
   py::module profiler =
       m->def_submodule("profiler", "TensorFlow profiler integration");
@@ -645,293 +501,148 @@ PYBIND11_MODULE(xla_extension, m) {
       .value("PLATFORM", GpuAllocatorConfig::Kind::kPlatform)
       .value("BFC", GpuAllocatorConfig::Kind::kBFC);
 
-  py::class_<PjRtClient, std::shared_ptr<PjRtClient>> py_local_client(
-      m, "LocalClient");
-  py_local_client.def_property_readonly("platform", &PjRtClient::platform_name)
-      .def("device_count", &PjRtClient::device_count)
-      .def("local_device_count", &PjRtClient::local_device_count)
-      .def("devices",
-           [](std::shared_ptr<PjRtClient> client) {
-             std::vector<ClientAndPtr<Device>> devices;
-             devices.reserve(client->devices().size());
-             for (const auto& device : client->devices()) {
-               devices.push_back(WrapWithClient(client, device.get()));
-             }
-             return devices;
-           })
-      .def("local_devices",
-           [](std::shared_ptr<PjRtClient> client) {
-             std::vector<ClientAndPtr<Device>> devices;
-             devices.reserve(client->local_devices().size());
-             for (Device* device : client->local_devices()) {
-               devices.push_back(WrapWithClient(client, device));
-             }
-             return devices;
-           })
-      .def("host_id", &PjRtClient::host_id)
+  py::class_<PyClient, std::shared_ptr<PyClient>> py_local_client(m, "Client");
+  py_local_client.def_property_readonly("platform", &PyClient::platform_name)
+      .def("device_count", &PyClient::device_count)
+      .def("local_device_count", &PyClient::local_device_count)
+      .def("devices", &PyClient::Devices)
+      .def("local_devices", &PyClient::LocalDevices)
+      .def("host_id", &PyClient::host_id)
       .def("get_default_device_assignment",
-           [](std::shared_ptr<PjRtClient> client, int num_replicas,
-              int num_partitions)
-               -> StatusOr<std::vector<std::vector<ClientAndPtr<Device>>>> {
-             TF_ASSIGN_OR_RETURN(DeviceAssignment device_assignment,
-                                 client->GetDefaultDeviceAssignment(
-                                     num_replicas, num_partitions));
-             std::vector<std::vector<ClientAndPtr<Device>>> result;
-             result.resize(num_replicas);
-             for (int r = 0; r < num_replicas; ++r) {
-               result[r].resize(num_partitions);
-               for (int p = 0; p < num_partitions; ++p) {
-                 int device_id = device_assignment(r, p);
-                 auto iter = client->id_to_device().find(device_id);
-                 CHECK(iter != client->id_to_device().end()) << device_id;
-                 result[r][p] = WrapWithClient(client, iter->second);
-               }
-             }
-             return result;
-           })
+           &PyClient::GetDefaultDeviceAssignment)
       // TODO(skye): delete after all callers can handle 2D output
       .def("get_default_device_assignment",
-           [](std::shared_ptr<PjRtClient> client,
-              int num_replicas) -> StatusOr<std::vector<ClientAndPtr<Device>>> {
-             TF_ASSIGN_OR_RETURN(DeviceAssignment device_assignment,
-                                 client->GetDefaultDeviceAssignment(
-                                     num_replicas, /*num_partitions=*/1));
-             std::vector<ClientAndPtr<Device>> result;
-             for (int i = 0; i < num_replicas; ++i) {
-               int device_id = device_assignment(i, 0);
-               auto iter = client->id_to_device().find(device_id);
-               CHECK(iter != client->id_to_device().end()) << device_id;
-               result.push_back(WrapWithClient(client, iter->second));
-             }
-             return result;
-           })
-      .def("create_channel_handle",
-           [](PjRtClient* client) {
-             return client->client()->CreateChannelHandle();
-           })
+           &PyClient::GetDefaultDeviceAssignment1D)
+      .def("create_channel_handle", &PyClient::CreateChannelHandle)
       .def("create_device_to_host_channel_handle",
-           [](PjRtClient* client) {
-             return client->client()->CreateDeviceToHostChannelHandle();
-           })
-      .def("create_host_to_device_channel_handle", [](PjRtClient* client) {
-        return client->client()->CreateHostToDeviceChannelHandle();
-      });
-  py_local_client.def(
-      "buffer_from_pyval",
-      [](std::shared_ptr<PjRtClient> client, const pybind11::object& argument,
-         Device* device,
-         bool force_copy) -> StatusOr<ClientAndUniquePtr<PjRtBuffer>> {
-        if (device == nullptr) {
-          TF_RET_CHECK(!client->local_devices().empty());
-          device = client->local_devices().front();
-        }
-        CHECK(device != nullptr);
-        auto iter = client->id_to_device().find(device->id());
-        if (iter->second != device) {
-          return InvalidArgument(
-              "Cannot copy value to device '%s' with '%s' backend",
-              device->DebugString(), client->platform_name());
-        }
-        GlobalPyRefManager()->CollectGarbage();
+           &PyClient::CreateDeviceToHostChannelHandle)
+      .def("create_host_to_device_channel_handle",
+           &PyClient::CreateHostToDeviceChannelHandle)
+      .def("buffer_from_pyval", &PyClient::BufferFromPyal, py::arg("argument"),
+           py::arg("device") = nullptr, py::arg("force_copy") = false)
+      .def("compile", &PyClient::Compile, py::arg("computation"),
+           py::arg("compile_options") = CompileOptions());
 
-        absl::optional<CastToArrayResult> c = CastToArray(argument);
-        if (!c) {
-          return InvalidArgument("from_python argument must be an array.");
-        }
-
-        TF_ASSIGN_OR_RETURN(PythonBufferTree tree,
-                            GetPythonBufferTree(argument));
-        std::shared_ptr<PythonRefManager::ManagedPyObjects> py_buffer_ref =
-            GlobalPyRefManager()->ManageReference(std::move(c->array));
-
-        py::gil_scoped_release gil_release;
+  m.def(
+      "get_cpu_client",
+      [](bool asynchronous) -> StatusOr<std::shared_ptr<PyClient>> {
+        TF_ASSIGN_OR_RETURN(std::shared_ptr<PjRtClient> client,
+                            GetCpuClient(asynchronous));
+        return std::make_shared<PyClient>(std::move(client));
+      },
+      py::arg("asynchronous") = true);
+  m.def("get_interpreter_client", []() -> StatusOr<std::shared_ptr<PyClient>> {
+    TF_ASSIGN_OR_RETURN(std::shared_ptr<PjRtClient> client,
+                        GetInterpreterClient());
+    return std::make_shared<PyClient>(std::move(client));
+  });
+  m.def(
+      "get_nvidia_gpu_client",
+      [](bool asynchronous, const GpuAllocatorConfig& allocator_config,
+         std::shared_ptr<DistributedRuntimeClient> distributed_client,
+         int node_id) -> StatusOr<std::shared_ptr<PyClient>> {
         TF_ASSIGN_OR_RETURN(
-            std::unique_ptr<PjRtBuffer> buffer,
-            PjRtBuffer::FromHostBuffer(c->buf_ptr, c->shape, force_copy,
-                                       std::move(py_buffer_ref), client.get(),
-                                       device));
-        return WrapWithClient(std::move(client), std::move(buffer));
+            std::shared_ptr<PjRtClient> client,
+            GetNvidiaGpuClient(asynchronous, allocator_config,
+                               std::move(distributed_client), node_id));
+        return std::make_shared<PyClient>(std::move(client));
       },
-      py::arg("argument"), py::arg("device") = nullptr,
-      py::arg("force_copy") = false);
-  py_local_client.def(
-      "compile",
-      [](std::shared_ptr<PjRtClient> client, const XlaComputation& computation,
-         CompileOptions options)
-          -> StatusOr<ClientAndUniquePtr<PjRtExecutable>> {
-        py::gil_scoped_release gil_release;
-        TF_ASSIGN_OR_RETURN(std::unique_ptr<PjRtExecutable> executable,
-                            PjRtExecutable::Compile(computation, client.get(),
-                                                    std::move(options)));
-        return WrapWithClient(std::move(client), std::move(executable));
-      },
-      py::arg("computation"), py::arg("compile_options") = CompileOptions());
+      py::arg("asynchronous") = true,
+      py::arg("allocator_config") = GpuAllocatorConfig(),
+      py::arg("distributed_client") = nullptr, py::arg("node_id") = 0);
 
-  m.def("get_cpu_client", &GetCpuClient, py::arg("asynchronous") = true);
-  m.def("get_interpreter_client", &GetInterpreterClient);
-  m.def("get_nvidia_gpu_client", &GetNvidiaGpuClient,
-        py::arg("asynchronous") = true,
-        py::arg("allocator_config") = GpuAllocatorConfig(),
-        py::arg("distributed_client") = nullptr, py::arg("node_id") = 0);
+  py::class_<Traceback::Frame>(m, "Frame")
+      .def_readonly("file_name", &Traceback::Frame::file_name)
+      .def_readonly("function_name", &Traceback::Frame::function_name)
+      .def_readonly("function_start_line",
+                    &Traceback::Frame::function_start_line)
+      .def_readonly("line_num", &Traceback::Frame::line_num)
+      .def("__repr__", [](const Traceback::Frame& frame) {
+        return absl::StrFormat("%s;%s:%d", frame.function_name, frame.file_name,
+                               frame.line_num);
+      });
 
-  py::class_<PjRtBuffer, ClientAndUniquePtr<PjRtBuffer>> buffer(
-      m, "PyLocalBuffer");
-  buffer
-      .def("copy_to_device",
-           [](PjRtBuffer* buffer, const ClientAndPtr<Device>& dst_device)
-               -> StatusOr<ClientAndUniquePtr<PjRtBuffer>> {
-             CHECK(dst_device.get() != nullptr);
-             GlobalPyRefManager()->CollectGarbage();
-             py::gil_scoped_release gil_release;
-             TF_ASSIGN_OR_RETURN(std::unique_ptr<PjRtBuffer> out,
-                                 buffer->CopyToDevice(dst_device.get()));
-             return WrapWithClient(dst_device.client, std::move(out));
-           })
-      .def("delete", &PjRtBuffer::Delete)
-      .def("block_host_until_ready",
-           [](PjRtBuffer* buffer) {
-             GlobalPyRefManager()->CollectGarbage();
-             py::gil_scoped_release gil_release;
-             return buffer->BlockHostUntilReady();
-           })
-      .def("copy_to_host_async", &PjRtBuffer::CopyToHostAsync,
+  py::class_<Traceback> traceback(m, "Traceback",
+                                  "Represents a Python stack trace.");
+  traceback.def_property_static(
+      "enabled", [](py::object /* cls */) { return Traceback::enabled(); },
+      [](py::object /* cls */, bool enabled) {
+        return Traceback::SetEnabled(enabled);
+      });
+  traceback.def_static(
+      "get_traceback", []() { return Traceback::Get(); },
+      R"doc(
+    Returns a :class:`Traceback` for the current thread.
+
+    If ``Traceback.enabled`` is ``True``, returns a :class:`Traceback` object
+    that describes the Python stack of the calling thread. Stack trace
+    collection has a small overhead, so it is disabled by default. If traceback
+    collection is disabled, returns ``None``.
+    )doc");
+  traceback.def_property_readonly("frames", &Traceback::Frames);
+  traceback.def("__str__", &Traceback::ToString);
+
+  py::class_<PyBuffer, std::unique_ptr<PyBuffer>> buffer(m, "Buffer");
+  // TODO(phawkins): alias for backward compatibility. Remove after JAX no
+  // longer uses this name.
+  m.add_object("PyLocalBuffer", buffer);
+  buffer.def("copy_to_device", &PyBuffer::CopyToDevice)
+      .def("delete", &PyBuffer::Delete)
+      .def("block_host_until_ready", &PyBuffer::BlockHostUntilReady)
+      .def("copy_to_host_async", &PyBuffer::CopyToHostAsync,
            py::call_guard<py::gil_scoped_release>())
       .def(
           "to_py",
           [](py::object buffer_obj) -> StatusOr<py::object> {
             GlobalPyRefManager()->CollectGarbage();
-            PjRtBuffer* buffer = buffer_obj.cast<PjRtBuffer*>();
-            LocalDeviceState* state = buffer->device()->local_device_state();
+            PyBuffer* buffer = buffer_obj.cast<PyBuffer*>();
+            LocalDeviceState* state =
+                buffer->buffer()->device()->local_device_state();
             if (state->executor()->platform_kind() == se::PlatformKind::kHost &&
-                buffer->on_device_shape().IsArray() &&
-                buffer->on_device_shape().element_type() != BF16) {
+                buffer->buffer()->on_device_shape().IsArray() &&
+                buffer->buffer()->on_device_shape().element_type() != BF16) {
               py::object out = py::reinterpret_steal<py::object>(
                   PyArray_FROM_O(buffer_obj.ptr()));
               CHECK(out.ptr() != nullptr)
-                  << buffer->on_host_shape().ToString(/*print_layout=*/true);
+                  << buffer->buffer()->on_host_shape().ToString(
+                         /*print_layout=*/true);
               return out;
             }
             std::shared_ptr<Literal> literal;
             {
               py::gil_scoped_release gil_release;
-              TF_ASSIGN_OR_RETURN(literal, buffer->ToLiteral());
+              TF_ASSIGN_OR_RETURN(literal, buffer->buffer()->ToLiteral());
             }
             return LiteralToPython(std::move(literal));
           })
-      .def("shape", &PjRtBuffer::on_host_shape)
-      .def_property_readonly("client",
-                             [](const PjRtBuffer& buffer) {
-                               return buffer.client()->shared_from_this();
-                             })
-      .def("device",
-           [](const PjRtBuffer& buffer) {
-             return WrapWithClient(buffer.client()->shared_from_this(),
-                                   buffer.device());
-           })
-      .def("platform", &PjRtBuffer::platform_name)
-      .def("is_deleted", [](PjRtBuffer* buffer) { return buffer->IsDeleted(); })
-      .def("unsafe_buffer_pointer",
-           [](const PjRtBuffer& buffer) -> StatusOr<std::uintptr_t> {
-             TF_ASSIGN_OR_RETURN(ShapedBuffer shaped_buffer,
-                                 buffer.AsShapedBuffer());
-             if (shaped_buffer.on_device_shape().IsTuple()) {
-               return Unimplemented(
-                   "unsafe_buffer_pointer is not implemented for tuple "
-                   "buffers.");
-             }
-             return absl::bit_cast<std::uintptr_t>(
-                 shaped_buffer.root_buffer().opaque());
-           })
+      .def("shape", &PyBuffer::shape)
+      .def_property_readonly("client", &PyBuffer::client)
+      .def("device", &PyBuffer::device)
+      .def("platform", &PyBuffer::platform_name)
+      .def("is_deleted", &PyBuffer::is_deleted)
+      .def("unsafe_buffer_pointer", &PyBuffer::UnsafeBufferPointer)
       .def_property_readonly("__cuda_array_interface__",
-                             &PjRtBufferCudaArrayInterface);
+                             &PyBuffer::CudaArrayInterface)
+      .def_property_readonly("traceback", &PyBuffer::traceback);
 
   // pybind11's implementation of the buffer protocol doesn't allow for correct
   // error handling. We bypass it and implement the buffer protocol ourselves.
   PyTypeObject* buffer_type = reinterpret_cast<PyTypeObject*>(buffer.ptr());
-  buffer_type->tp_as_buffer = &PjRtBufferProcs;
+  buffer_type->tp_as_buffer = PyBuffer::BufferProtocol();
 
-  py::class_<PjRtExecutable, ClientAndUniquePtr<PjRtExecutable>> executable(
-      m, "LocalExecutable");
-  executable
-      .def_property_readonly("client",
-                             [](const PjRtExecutable& executable) {
-                               return executable.client()->shared_from_this();
-                             })
-      .def("local_logical_device_ids",
-           &PjRtExecutable::local_logical_device_ids)
-      .def("local_devices",
-           [](const PjRtExecutable& executable) {
-             std::vector<ClientAndPtr<Device>> devices;
-             devices.reserve(executable.local_devices().size());
-             for (Device* device : executable.local_devices()) {
-               devices.push_back(WrapWithClient(
-                   executable.client()->shared_from_this(), device));
-             }
-             return devices;
-           })
+  py::class_<PyExecutable, std::unique_ptr<PyExecutable>> executable(
+      m, "Executable");
+  executable.def_property_readonly("client", &PyExecutable::client)
+      .def("local_logical_device_ids", &PyExecutable::local_logical_device_ids)
+      .def("local_devices", &PyExecutable::LocalDevices)
       .def("size_of_generated_code_in_bytes",
-           &PjRtExecutable::SizeOfGeneratedCodeInBytes)
-      .def("delete", &PjRtExecutable::Delete)
-      .def(
-          "execute",
-          [](const PjRtExecutable& executable,
-             absl::Span<PjRtBuffer* const> args)
-              -> StatusOr<std::vector<ClientAndUniquePtr<PjRtBuffer>>> {
-            py::gil_scoped_release gil_release;
-            ExecuteOptions options;
-            options.untuple_result = true;
-            TF_ASSIGN_OR_RETURN(
-                std::vector<std::unique_ptr<PjRtBuffer>> output_buffers,
-                executable.Execute(args, options));
-            std::vector<ClientAndUniquePtr<PjRtBuffer>> outputs;
-            outputs.reserve(output_buffers.size());
-            for (auto& buffer : output_buffers) {
-              outputs.push_back(WrapWithClient(
-                  executable.client()->shared_from_this(), std::move(buffer)));
-            }
-            return outputs;
-          },
-          py::arg("arguments"))
-      .def(
-          "execute_on_local_devices",
-          [](const PjRtExecutable& executable,
-             absl::Span<const std::vector<PjRtBuffer*>> args)
-              -> StatusOr<
-                  std::vector<std::vector<ClientAndUniquePtr<PjRtBuffer>>>> {
-            py::gil_scoped_release gil_release;
-            ExecuteOptions options;
-            options.untuple_result = true;
-            TF_ASSIGN_OR_RETURN(
-                std::vector<std::vector<std::unique_ptr<PjRtBuffer>>>
-                    output_buffers,
-                executable.ExecuteOnLocalDevices(args, options));
-            std::vector<std::vector<ClientAndUniquePtr<PjRtBuffer>>> outputs;
-            outputs.resize(output_buffers.size());
-            for (int computation = 0; computation < output_buffers.size();
-                 ++computation) {
-              for (auto& buffer : output_buffers[computation]) {
-                outputs[computation].push_back(
-                    WrapWithClient(executable.client()->shared_from_this(),
-                                   std::move(buffer)));
-              }
-            }
-            return outputs;
-          },
-          py::arg("arguments"))
-      .def(
-          "hlo_modules",
-          [](const PjRtExecutable& executable)
-              -> StatusOr<std::vector<std::shared_ptr<HloModule>>> {
-            std::vector<std::shared_ptr<HloModule>> modules;
-            modules.reserve(executable.executables().size());
-            for (const auto& local_exec : executable.executables()) {
-              if (!local_exec->executable()->has_module()) {
-                return InvalidArgument("Executable does not have HLO modules.");
-              }
-              modules.push_back(local_exec->executable()->shared_module());
-            }
-            return std::move(modules);
-          });
+           &PyExecutable::SizeOfGeneratedCodeInBytes)
+      .def("delete", &PyExecutable::Delete)
+      .def("execute", &PyExecutable::Execute, py::arg("arguments"))
+      .def("execute_on_local_devices", &PyExecutable::ExecuteOnLocalDevices,
+           py::arg("arguments"))
+      .def("hlo_modules", &PyExecutable::HloModules)
+      .def_property_readonly("traceback", &PyExecutable::traceback);
 
   py::class_<DebugOptions>(m, "DebugOptions")
       .def("__repr__", &DebugOptions::DebugString)
@@ -1129,14 +840,7 @@ PYBIND11_MODULE(xla_extension, m) {
            });
 
   m.def("buffer_to_dlpack_managed_tensor", BufferToDLPackManagedTensor);
-  m.def("dlpack_managed_tensor_to_buffer",
-        [](const py::capsule& tensor, std::shared_ptr<PjRtClient> client)
-            -> StatusOr<ClientAndUniquePtr<PjRtBuffer>> {
-          TF_ASSIGN_OR_RETURN(
-              std::unique_ptr<PjRtBuffer> buffer,
-              DLPackManagedTensorToBuffer(tensor, client.get()));
-          return WrapWithClient(std::move(client), std::move(buffer));
-        });
+  m.def("dlpack_managed_tensor_to_buffer", DLPackManagedTensorToBuffer);
 
   py::enum_<PrecisionConfig::Precision>(m, "PrecisionConfig_Precision")
       .value("DEFAULT", PrecisionConfig::DEFAULT)
@@ -1179,6 +883,8 @@ PYBIND11_MODULE(xla_extension, m) {
 
   m.def("get_distributed_runtime_service", &GetDistributedRuntimeService);
   m.def("get_distributed_runtime_client", &GetDistributedRuntimeClient);
+
+  m.def("collect_garbage", []() { GlobalPyRefManager()->CollectGarbage(); });
 }  // NOLINT(readability/fn_size)
 
 }  // namespace xla
diff --git a/tensorflow/compiler/xla/python/xla_client.py b/tensorflow/compiler/xla/python/xla_client.py
index 3085715bf12..c44fdaee0b9 100644
--- a/tensorflow/compiler/xla/python/xla_client.py
+++ b/tensorflow/compiler/xla/python/xla_client.py
@@ -19,7 +19,9 @@ from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function
 
+import atexit
 import collections
+import contextlib
 import enum  # pylint: disable=g-bad-import-order
 import inspect
 import os
@@ -406,6 +408,9 @@ def window_padding_type_to_pad_values(padding_type, lhs_dims, rhs_dims,
 XlaBuilder = _xla.XlaBuilder
 XlaComputation = _xla.XlaComputation
 FftType = _xla.FftType
+Client = _xla.Client
+Buffer = _xla.Buffer
+Executable = _xla.Executable
 
 
 def register_custom_call_target(name, fn, platform='cpu'):
@@ -661,3 +666,22 @@ def make_replica_groups(replica_groups):
         _make_replica_group_proto(group) for group in replica_groups
     ]
   return replica_groups_protos
+
+
+Traceback = _xla.Traceback
+
+
+@contextlib.contextmanager
+def tracebacks(enabled=True):
+  """Context manager that enables or disables traceback collection."""
+  saved = Traceback.enabled
+  Traceback.enabled = enabled
+  try:
+    yield
+  finally:
+    Traceback.enabled = saved
+
+
+# Perform one last garbage collection of deferred Python references. This is
+# mostly to keep ASAN happy.
+atexit.register(_xla.collect_garbage)
diff --git a/tensorflow/compiler/xla/python/xla_client_test.py b/tensorflow/compiler/xla/python/xla_client_test.py
index 000db2cb16b..0fc0bcae954 100644
--- a/tensorflow/compiler/xla/python/xla_client_test.py
+++ b/tensorflow/compiler/xla/python/xla_client_test.py
@@ -2038,6 +2038,67 @@ def TestFactory(xla_backend, cloud_tpu=False):
       del server
 
   tests.append(ProfilerTest)
+
+  class TracebackTest(absltest.TestCase):
+
+    def setUp(self):
+      super(TracebackTest, self).setUp()
+      self.backend = xla_backend()
+
+    def testNoTracebacksIfDisabled(self):
+      with xla_client.tracebacks(enabled=False):
+        self.assertEqual(None, xla_client.Traceback.get_traceback())
+        buffer = self.backend.buffer_from_pyval(np.array(7, np.int32))
+        self.assertEqual(None, buffer.traceback)
+
+        b = xla_client.XlaBuilder("computation")
+        ops.Add(ops.Constant(b, np.int32(1)), ops.Constant(b, np.int32(2)))
+        e = self.backend.compile(b.build())
+        self.assertEqual(None, e.traceback)
+
+    def assertIsTracebackContaining(self, tb, function):
+      self.assertIsInstance(tb, xla_client.Traceback)
+      self.assertIn(function, str(tb))
+      self.assertTrue(any(f.function_name == function for f in tb.frames))
+
+    def testTracebacks(self):
+      with xla_client.tracebacks(enabled=True):
+        tb = xla_client.Traceback.get_traceback()
+        self.assertIsTracebackContaining(tb, "testTracebacks")
+
+        # Tracebacks are not implemented on the TPU driver extension's variant
+        # of buffers and executables.
+        if not isinstance(self.backend, xla_client.Client):
+          return
+
+        buffer = self.backend.buffer_from_pyval(np.array(7, np.int32))
+        self.assertIsTracebackContaining(buffer.traceback, "testTracebacks")
+
+        b = xla_client.XlaBuilder("computation")
+        ops.Add(ops.Constant(b, np.int32(1)), ops.Constant(b, np.int32(2)))
+        e = self.backend.compile(b.build())
+        self.assertIsTracebackContaining(e.traceback, "testTracebacks")
+
+    def testNestedFunction(self):
+
+      def AFunction():
+
+        def AnotherFunction():
+          return xla_client.Traceback.get_traceback()
+
+        return AnotherFunction()
+
+      with xla_client.tracebacks(enabled=True):
+        tb = AFunction()
+        self.assertIsInstance(tb, xla_client.Traceback)
+        frames = tb.frames
+        i = next(
+            i for (i, f) in enumerate(frames) if f.function_name == "AFunction")
+        self.assertEqual(frames[i - 1].function_name, "AnotherFunction")
+        self.assertEqual(frames[i + 1].function_name, "testNestedFunction")
+
+  tests.append(TracebackTest)
+
   return tests
 
 
diff --git a/tensorflow/compiler/xla/service/BUILD b/tensorflow/compiler/xla/service/BUILD
index d08147d36dd..4fc052ce17a 100644
--- a/tensorflow/compiler/xla/service/BUILD
+++ b/tensorflow/compiler/xla/service/BUILD
@@ -1193,7 +1193,7 @@ cc_library(
     deps = [
         ":compiler",
         "//tensorflow/core:lib_internal",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
 )
 
@@ -2297,42 +2297,6 @@ tf_cc_test(
     ],
 )
 
-cc_library(
-    name = "depthwise_convolution_converter",
-    srcs = ["depthwise_convolution_converter.cc"],
-    hdrs = ["depthwise_convolution_converter.h"],
-    deps = [
-        ":hlo",
-        ":hlo_pass",
-        "//tensorflow/compiler/xla:literal",
-        "//tensorflow/compiler/xla:literal_util",
-        "//tensorflow/compiler/xla:shape_util",
-        "//tensorflow/compiler/xla:status_macros",
-        "//tensorflow/compiler/xla:types",
-        "//tensorflow/compiler/xla:util",
-        "//tensorflow/compiler/xla:xla_data_proto_cc",
-        "//tensorflow/core:lib",
-        "@com_google_absl//absl/memory",
-        "@com_google_absl//absl/strings",
-    ],
-)
-
-tf_cc_test(
-    name = "depthwise_convolution_converter_test",
-    size = "small",
-    srcs = ["depthwise_convolution_converter_test.cc"],
-    deps = [
-        ":depthwise_convolution_converter",
-        ":hlo",
-        ":hlo_matchers",
-        ":hlo_parser",
-        "//tensorflow/compiler/xla:test",
-        "//tensorflow/compiler/xla:types",
-        "//tensorflow/compiler/xla/tests:hlo_test_base",
-        "//tensorflow/compiler/xla/tests:xla_internal_test_main",
-    ],
-)
-
 cc_library(
     name = "while_loop_analysis",
     srcs = ["while_loop_analysis.cc"],
@@ -3942,7 +3906,7 @@ cc_library(
         "@com_google_absl//absl/container:flat_hash_map",
         "@com_google_absl//absl/strings",
         "@com_google_absl//absl/types:span",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:transform_utils",
     ],
 )
diff --git a/tensorflow/compiler/xla/service/cpu/BUILD b/tensorflow/compiler/xla/service/cpu/BUILD
index 3460e65b0a2..a6a06f09fe0 100644
--- a/tensorflow/compiler/xla/service/cpu/BUILD
+++ b/tensorflow/compiler/xla/service/cpu/BUILD
@@ -181,15 +181,15 @@ cc_library(
         "//tensorflow/compiler/xla/service/llvm_ir:llvm_util",
         "//tensorflow/core:lib",
         "//tensorflow/core:stream_executor_no_cuda",
-        "@llvm-project//llvm:core",
-        "@llvm-project//llvm:mc",
+        "@llvm-project//llvm:ir",
+        "@llvm-project//llvm:machine_code",
         "@llvm-project//llvm:object",
         "@llvm-project//llvm:support",
-        "@llvm-project//llvm:target",
-        "@llvm-project//llvm:x86_code_gen",  # fixdeps: keep
+        "@llvm-project//llvm:target_base",
+        "@llvm-project//llvm:x86_target",  # fixdeps: keep
     ] + select({
         "//tensorflow:linux_ppc64le": [
-            "@llvm-project//llvm:powerpc_code_gen",  # fixdeps: keep
+            "@llvm-project//llvm:powerpc_target",  # fixdeps: keep
         ],
         "//conditions:default": [
         ],
@@ -223,11 +223,11 @@ cc_library(
         ":runtime_single_threaded_matmul",
         "@com_google_absl//absl/memory",
         "@llvm-project//llvm:execution_engine",
-        "@llvm-project//llvm:core",
-        "@llvm-project//llvm:mc",  # fixdeps: keep
-        "@llvm-project//llvm:orc_jit",
+        "@llvm-project//llvm:ir",
+        "@llvm-project//llvm:machine_code",  # fixdeps: keep
+        "@llvm-project//llvm:orcjit",
         "@llvm-project//llvm:support",
-        "@llvm-project//llvm:target",  # fixdeps: keep
+        "@llvm-project//llvm:target_base",  # fixdeps: keep
         "//tensorflow/compiler/xla/service:custom_call_target_registry",
         "//tensorflow/compiler/xla:types",
         "//tensorflow/compiler/xla:util",
@@ -307,7 +307,7 @@ cc_library(
         "@com_google_absl//absl/strings",
         "@com_google_absl//absl/strings:str_format",
         "@com_google_absl//absl/types:span",
-        "@llvm-project//llvm:orc_jit",
+        "@llvm-project//llvm:orcjit",
     ],
 )
 
@@ -365,10 +365,10 @@ cc_library(
         "@com_google_absl//absl/strings",
         "@com_google_absl//absl/strings:str_format",
         "@com_google_absl//absl/types:span",
-        "@llvm-project//llvm:code_gen",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:codegen",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
-        "@llvm-project//llvm:target",
+        "@llvm-project//llvm:target_base",
         "@llvm-project//mlir:IR",
     ],
 )
@@ -385,7 +385,7 @@ cc_library(
         "//tensorflow/core:lib",
         "@com_google_absl//absl/container:flat_hash_map",
         "@llvm-project//llvm:analysis",
-        "@llvm-project//llvm:target",
+        "@llvm-project//llvm:target_base",
     ],
 )
 
@@ -414,7 +414,7 @@ cc_library(
         "//tensorflow/compiler/xla/service/llvm_ir:llvm_util",
         "@com_google_absl//absl/strings",
         "@com_google_absl//absl/types:span",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
 )
 
@@ -430,7 +430,7 @@ cc_library(
         "//tensorflow/compiler/xla/service/llvm_ir:loop_emitter",
         "//tensorflow/core:lib",
         "@com_google_absl//absl/strings:str_format",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
 )
 
@@ -446,7 +446,7 @@ cc_library(
         "//tensorflow/compiler/xla/service/llvm_ir:kernel_support_library",
         "//tensorflow/compiler/xla/service/llvm_ir:llvm_util",
         "//tensorflow/core:lib",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
 )
 
@@ -478,7 +478,7 @@ cc_library(
         "//tensorflow/compiler/xla/service/llvm_ir:llvm_util",
         "//tensorflow/core:lib",
         "@com_google_absl//absl/strings",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//mlir:EDSC",
         "@llvm-project//mlir:IR",
         "@llvm-project//mlir:LinalgOps",
@@ -521,12 +521,12 @@ cc_library(
         "//tensorflow/core:lib",
         "@com_google_absl//absl/memory",
         "@llvm-project//llvm:analysis",
-        "@llvm-project//llvm:core",
-        "@llvm-project//llvm:ipo",
-        "@llvm-project//llvm:mc",
+        "@llvm-project//llvm:ipo_transforms",
+        "@llvm-project//llvm:ir",
+        "@llvm-project//llvm:machine_code",
         "@llvm-project//llvm:object",
         "@llvm-project//llvm:support",
-        "@llvm-project//llvm:target",
+        "@llvm-project//llvm:target_base",
     ],
 )
 
@@ -584,7 +584,7 @@ cc_library(
         "//tensorflow/compiler/xla/service/llvm_ir:llvm_util",
         "//tensorflow/compiler/xla/service/llvm_ir:math_ops",
         "//tensorflow/core:lib",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:transform_utils",
     ],
 )
@@ -853,7 +853,7 @@ cc_library(
         "//tensorflow/compiler/xla:shape_util",
         "//tensorflow/compiler/xla:window_util",
         "//tensorflow/compiler/xla/service:hlo",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
 )
 
@@ -1046,7 +1046,7 @@ cc_library(
         "//tensorflow/core:lib",
         "@com_google_absl//absl/algorithm:container",
         "@com_google_absl//absl/types:span",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
@@ -1074,9 +1074,9 @@ tf_cc_test(
         "//tensorflow/compiler/xla:test",
         "//tensorflow/compiler/xla/tests:hlo_test_base",
         "//tensorflow/compiler/xla/tests:xla_internal_test_main",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
-        "@llvm-project//llvm:target",
+        "@llvm-project//llvm:target_base",
     ],
 )
 
@@ -1088,8 +1088,8 @@ cc_library(
         "//tensorflow/compiler/mlir/xla:hlo_utils",
         "//tensorflow/compiler/xla:shape_util",
         "//tensorflow/compiler/xla:status",
-        "@llvm-project//llvm:core",
-        "@llvm-project//llvm:ipo",
+        "@llvm-project//llvm:ipo_transforms",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:linker",
         "@llvm-project//mlir:IR",
         "@llvm-project//mlir:LLVMTransforms",
diff --git a/tensorflow/compiler/xla/service/cpu/cpu_executable.cc b/tensorflow/compiler/xla/service/cpu/cpu_executable.cc
index f031daecb1f..d9a328a326e 100644
--- a/tensorflow/compiler/xla/service/cpu/cpu_executable.cc
+++ b/tensorflow/compiler/xla/service/cpu/cpu_executable.cc
@@ -376,5 +376,9 @@ const InstructionValueSet& CpuExecutable::GetRootValueSet() const {
       module().entry_computation()->root_instruction());
 }
 
+int64 CpuExecutable::SizeOfGeneratedCodeInBytes() const {
+  return jit_->SizeOfGeneratedCodeInBytes();
+}
+
 }  // namespace cpu
 }  // namespace xla
diff --git a/tensorflow/compiler/xla/service/cpu/cpu_executable.h b/tensorflow/compiler/xla/service/cpu/cpu_executable.h
index 4ec688c1016..310e30e41f5 100644
--- a/tensorflow/compiler/xla/service/cpu/cpu_executable.h
+++ b/tensorflow/compiler/xla/service/cpu/cpu_executable.h
@@ -81,6 +81,8 @@ class CpuExecutable : public Executable {
 
   const BufferAssignment& buffer_assignment() const { return *assignment_; }
 
+  int64 SizeOfGeneratedCodeInBytes() const override;
+
  private:
   // Creates an array suitable for passing as the "buffer_table" argument to the
   // JIT compiled function pointer.
diff --git a/tensorflow/compiler/xla/service/cpu/simple_orc_jit.cc b/tensorflow/compiler/xla/service/cpu/simple_orc_jit.cc
index 4cc9e373b3e..c38ee486af3 100644
--- a/tensorflow/compiler/xla/service/cpu/simple_orc_jit.cc
+++ b/tensorflow/compiler/xla/service/cpu/simple_orc_jit.cc
@@ -163,6 +163,7 @@ void SimpleOrcJIT::NotifyObjectFinalized(
   uint64_t key = static_cast<uint64_t>(
       reinterpret_cast<uintptr_t>(object.getData().data()));
   gdb_jit_event_listener_->notifyObjectLoaded(key, object, object_info);
+  size_of_generated_code_in_bytes_ += object.getData().size();
 }
 
 void SimpleOrcJIT::NotifyObjectFreed(const llvm::object::ObjectFile& object) {
diff --git a/tensorflow/compiler/xla/service/cpu/simple_orc_jit.h b/tensorflow/compiler/xla/service/cpu/simple_orc_jit.h
index 66333fb65c0..9c470edbac2 100644
--- a/tensorflow/compiler/xla/service/cpu/simple_orc_jit.h
+++ b/tensorflow/compiler/xla/service/cpu/simple_orc_jit.h
@@ -88,6 +88,10 @@ class SimpleOrcJIT {
       const llvm::TargetOptions& target_options,
       llvm::CodeGenOpt::Level opt_level);
 
+  int64 SizeOfGeneratedCodeInBytes() const {
+    return size_of_generated_code_in_bytes_;
+  }
+
  private:
   llvm::JITSymbol ResolveRuntimeSymbol(const std::string& name);
 
@@ -103,6 +107,7 @@ class SimpleOrcJIT {
   std::shared_ptr<llvm::orc::SymbolResolver> symbol_resolver_;
   ObjLayerT object_layer_;
   CompileLayerT compile_layer_;
+  int64 size_of_generated_code_in_bytes_ = 0;
 
   // Non owning pointer to a JIT event listener that registers the JIT events
   // with an attached GDB.
diff --git a/tensorflow/compiler/xla/service/cpu/tests/BUILD b/tensorflow/compiler/xla/service/cpu/tests/BUILD
index 1ac8509cdb1..e6261dbfb55 100644
--- a/tensorflow/compiler/xla/service/cpu/tests/BUILD
+++ b/tensorflow/compiler/xla/service/cpu/tests/BUILD
@@ -108,7 +108,7 @@ tf_cc_test(
         "//tensorflow/core:test",
         "//tensorflow/core:test_main",
         "@com_google_absl//absl/memory",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
 )
 
@@ -123,9 +123,9 @@ tf_cc_test(
         "//tensorflow/core:test",
         "//tensorflow/core:test_main",
         "@com_google_absl//absl/strings",
-        "@llvm-project//llvm:arm_code_gen",  # fixdeps: keep
-        "@llvm-project//llvm:target",
-        "@llvm-project//llvm:x86_code_gen",  # fixdeps: keep
+        "@llvm-project//llvm:arm_target",  # fixdeps: keep
+        "@llvm-project//llvm:target_base",
+        "@llvm-project//llvm:x86_target",  # fixdeps: keep
     ],
 )
 
@@ -155,9 +155,9 @@ tf_cc_test(
         "//tensorflow/core:test",
         "//tensorflow/core:test_main",
         "@com_google_absl//absl/strings",
-        "@llvm-project//llvm:arm_code_gen",  # fixdeps: keep
-        "@llvm-project//llvm:target",
-        "@llvm-project//llvm:x86_code_gen",  # fixdeps: keep
+        "@llvm-project//llvm:arm_target",  # fixdeps: keep
+        "@llvm-project//llvm:target_base",
+        "@llvm-project//llvm:x86_target",  # fixdeps: keep
     ],
 )
 
@@ -259,8 +259,8 @@ tf_cc_test(
         "//tensorflow/core:test",
         "//tensorflow/core:test_main",
         "@com_google_absl//absl/strings",
-        "@llvm-project//llvm:arm_code_gen",  # fixdeps: keep
-        "@llvm-project//llvm:target",
-        "@llvm-project//llvm:x86_code_gen",  # fixdeps: keep
+        "@llvm-project//llvm:arm_target",  # fixdeps: keep
+        "@llvm-project//llvm:target_base",
+        "@llvm-project//llvm:x86_target",  # fixdeps: keep
     ],
 )
diff --git a/tensorflow/compiler/xla/service/depthwise_convolution_converter.cc b/tensorflow/compiler/xla/service/depthwise_convolution_converter.cc
deleted file mode 100644
index ad4d8118835..00000000000
--- a/tensorflow/compiler/xla/service/depthwise_convolution_converter.cc
+++ /dev/null
@@ -1,212 +0,0 @@
-/* Copyright 2019 The TensorFlow Authors. All Rights Reserved.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-==============================================================================*/
-
-#include "tensorflow/compiler/xla/service/depthwise_convolution_converter.h"
-
-#include <memory>
-#include <vector>
-
-#include "absl/memory/memory.h"
-#include "tensorflow/compiler/xla/literal.h"
-#include "tensorflow/compiler/xla/literal_util.h"
-#include "tensorflow/compiler/xla/service/dfs_hlo_visitor_with_default.h"
-#include "tensorflow/compiler/xla/service/hlo_computation.h"
-#include "tensorflow/compiler/xla/service/hlo_instruction.h"
-#include "tensorflow/compiler/xla/service/hlo_opcode.h"
-#include "tensorflow/compiler/xla/shape_util.h"
-#include "tensorflow/compiler/xla/status_macros.h"
-#include "tensorflow/compiler/xla/types.h"
-#include "tensorflow/compiler/xla/util.h"
-#include "tensorflow/compiler/xla/xla_data.pb.h"
-#include "tensorflow/core/lib/core/errors.h"
-#include "tensorflow/core/lib/core/status.h"
-#include "tensorflow/core/platform/logging.h"
-
-namespace xla {
-
-namespace {
-
-class ConvolutionVisitor : public DfsHloVisitorWithDefault {
- public:
-  // Default visitor action is to do nothing and return OK.
-  Status DefaultAction(HloInstruction* /*hlo_instruction*/) override {
-    return Status::OK();
-  }
-
-  Status HandleConvolution(HloInstruction* convolution) override;
-
-  Status HandleBackwardFilterBatchGroupConvolution(HloInstruction* convolution);
-
-  // Runs the visitor on a computation.
-  static bool Run(HloComputation* computation,
-                  std::function<bool(HloInstruction*)> is_cost_viable);
-
-  // Returns whether any convolution ops were rewritten.
-  const bool changed() const { return changed_; }
-
-  ~ConvolutionVisitor() override = default;
-
- private:
-  explicit ConvolutionVisitor(
-      HloComputation* computation,
-      std::function<bool(HloInstruction*)> is_cost_viable)
-      : computation_(computation), is_cost_viable_(is_cost_viable) {}
-
-  // Current HloComputation instance the ConvolutionVisitor is traversing.
-  HloComputation* computation_;
-
-  // Whether rewrite has occurred.
-  bool changed_ = false;
-
-  std::function<bool(HloInstruction*)> is_cost_viable_;
-};
-
-bool ConvolutionVisitor::Run(
-    HloComputation* computation,
-    std::function<bool(HloInstruction*)> is_cost_viable) {
-  ConvolutionVisitor visitor(computation, is_cost_viable);
-  TF_CHECK_OK(computation->Accept(&visitor));
-  return visitor.changed_;
-}
-
-namespace {
-Shape SwapInputOutputFeatureDims(const Shape& shape, int64 input_feature_dim,
-                                 int64 output_feature_dim) {
-  int64 num_dims = shape.dimensions_size();
-  CHECK_GE(num_dims, 2);
-  Shape transformed_shape = shape;
-  transformed_shape.set_dimensions(input_feature_dim,
-                                   shape.dimensions(output_feature_dim));
-  transformed_shape.set_dimensions(output_feature_dim,
-                                   shape.dimensions(input_feature_dim));
-  return transformed_shape;
-}
-}  // namespace
-
-// This function handles batch_group_counts which are relevant only for
-// depthwise backprop filter convolutions.
-Status ConvolutionVisitor::HandleBackwardFilterBatchGroupConvolution(
-    HloInstruction* convolution) {
-  auto dim_numbers = convolution->convolution_dimension_numbers();
-  auto lhs = convolution->mutable_operand(0);
-  auto rhs = convolution->mutable_operand(1);
-  int64 num_groups = convolution->batch_group_count();
-  int64 input_batch_dimension = dim_numbers.input_batch_dimension();
-  int64 input_batch = lhs->shape().dimensions(input_batch_dimension);
-
-  // TODO(b/139748189): Support 'num_grous' > 1 when input_batch !=
-  // num_groups.
-  if (num_groups == 1 || input_batch != num_groups) {
-    return Status::OK();
-  }
-
-  VLOG(2) << "Dealing with batch_group_count " << num_groups
-          << " for convolution " << convolution->ToString() << "\n";
-
-  int64 output_batch_dimension = dim_numbers.output_batch_dimension();
-  int64 output_feature_dimension = dim_numbers.output_feature_dimension();
-
-  // When mapping depthwise conv backward filter to batch grouped convolution,
-  // tf2xla bridge needs to swap the output batch and feature dimension. Since
-  // we want to use grouped convolution APIs, this swap needs to be reverted.
-  dim_numbers.set_output_batch_dimension(output_feature_dimension);
-  dim_numbers.set_output_feature_dimension(output_batch_dimension);
-
-  if (!is_cost_viable_(convolution)) {
-    Shape transformed_filter_grad_shape = SwapInputOutputFeatureDims(
-        convolution->shape(), dim_numbers.output_batch_dimension(),
-        dim_numbers.output_feature_dimension());
-
-    int64 input_feature_dimension = dim_numbers.input_feature_dimension();
-    int64 input_feature = lhs->shape().dimensions(input_feature_dimension);
-
-    auto add = [&](std::unique_ptr<HloInstruction> inst) {
-      return computation_->AddInstruction(std::move(inst));
-    };
-    // Reshape batch_dim C -> [G, C/G] - Batch and feature dims have been
-    // swapped in tf2xla bridge
-    std::vector<int64> reshape_dims = SpanToVector(lhs->shape().dimensions());
-    reshape_dims[input_batch_dimension] =
-        reshape_dims[input_batch_dimension] / num_groups;
-    reshape_dims.insert(reshape_dims.begin() + input_batch_dimension,
-                        num_groups);
-    lhs = add(HloInstruction::CreateReshape(
-        ShapeUtil::MakeShape(lhs->shape().element_type(), reshape_dims), lhs));
-
-    // Transpose G to the axis before N, For eg: [G, C/G, H, W, N ] -> [C/G, H,
-    // W, G, N]
-    std::vector<int64> transpose_dims(lhs->shape().dimensions_size());
-    std::iota(transpose_dims.begin(), transpose_dims.end(), 0);
-    transpose_dims.erase(transpose_dims.begin() + input_batch_dimension);
-    transpose_dims.insert(transpose_dims.begin() + input_feature_dimension,
-                          input_batch_dimension);
-    std::vector<int64> transpose_reshape_dims =
-        ComposePermutations(lhs->shape().dimensions(), transpose_dims);
-    lhs = add(HloInstruction::CreateTranspose(
-        ShapeUtil::MakeShape(lhs->shape().element_type(),
-                             transpose_reshape_dims),
-        lhs, transpose_dims));
-
-    // Merge [G,N] -> [N*G]
-    Shape new_shape = lhs->shape();
-    new_shape.DeleteDimension(input_feature_dimension);
-    new_shape.set_dimensions(input_feature_dimension,
-                             input_feature * num_groups);
-    lhs = add(HloInstruction::CreateReshape(new_shape, lhs));
-
-    std::vector<HloInstruction*> new_operands = {lhs, rhs};
-    auto new_conv = convolution->CloneWithNewOperands(
-        transformed_filter_grad_shape, new_operands);
-    new_conv->set_feature_group_count(num_groups);
-    new_conv->set_batch_group_count(1);
-    new_conv->set_convolution_dimension_numbers(dim_numbers);
-    auto new_convolution = computation_->AddInstruction(std::move(new_conv));
-
-    // Another reshape is required since the filter grad shape as a result of
-    // the 'new convolution` will be [kh, kw, C_i/G = 1, C_o = C_i = G ] but the
-    // expected shape is [kh, kw, C_i = G, DM=1] assuming the Depth-Multiplier
-    // (DM) is 1 and number of input features = G as required by the depthwise
-    // conv semantics
-    auto reshape =
-        HloInstruction::CreateReshape(convolution->shape(), new_convolution);
-    TF_RETURN_IF_ERROR(computation_->ReplaceWithNewInstruction(
-        convolution, std::move(reshape)));
-    changed_ = true;
-  }
-
-  return Status::OK();
-}
-
-Status ConvolutionVisitor::HandleConvolution(HloInstruction* convolution) {
-  return HandleBackwardFilterBatchGroupConvolution(convolution);
-}
-
-}  // namespace
-
-StatusOr<bool> DepthwiseConvolutionConverter::Run(HloModule* module) {
-  XLA_VLOG_LINES(2, "DepthwiseConvolutionConverter::Run(), before:\n" +
-                        module->ToString());
-  bool changed = false;
-  for (auto* comp : module->MakeNonfusionComputations()) {
-    if (ConvolutionVisitor::Run(comp, is_cost_viable_)) {
-      changed = true;
-    }
-  }
-  XLA_VLOG_LINES(
-      2, "DepthwiseConvolutionConverter::Run(), after:\n" + module->ToString());
-  return changed;
-}
-
-}  // namespace xla
diff --git a/tensorflow/compiler/xla/service/depthwise_convolution_converter.h b/tensorflow/compiler/xla/service/depthwise_convolution_converter.h
deleted file mode 100644
index a71b2b0d45d..00000000000
--- a/tensorflow/compiler/xla/service/depthwise_convolution_converter.h
+++ /dev/null
@@ -1,49 +0,0 @@
-/* Copyright 2019 The TensorFlow Authors. All Rights Reserved.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-==============================================================================*/
-
-#ifndef TENSORFLOW_COMPILER_XLA_SERVICE_DEPTHWISE_CONVOLUTION_CONVERTER_H_
-#define TENSORFLOW_COMPILER_XLA_SERVICE_DEPTHWISE_CONVOLUTION_CONVERTER_H_
-
-#include <functional>
-
-#include "absl/strings/string_view.h"
-#include "tensorflow/compiler/xla/service/hlo_module.h"
-#include "tensorflow/compiler/xla/service/hlo_pass_interface.h"
-#include "tensorflow/compiler/xla/status_macros.h"
-
-namespace xla {
-
-class DepthwiseConvolutionConverter : public HloModulePass {
- public:
-  explicit DepthwiseConvolutionConverter(
-      std::function<bool(HloInstruction*)> is_cost_viable)
-      : is_cost_viable_(is_cost_viable) {}
-
-  absl::string_view name() const override {
-    return "depthwise-convolution-converter";
-  }
-
-  // Run convolution rewriting on the given computation. Returns whether the
-  // computation was changed.
-  StatusOr<bool> Run(HloModule* module) override;
-
-  // Lambda containing cost model that decides whether to expand
-  // batch_group_count.
-  std::function<bool(HloInstruction*)> is_cost_viable_;
-};
-
-}  // namespace xla
-
-#endif  // TENSORFLOW_COMPILER_XLA_SERVICE_DEPTHWISE_CONVOLUTION_CONVERTER_H_
diff --git a/tensorflow/compiler/xla/service/depthwise_convolution_converter_test.cc b/tensorflow/compiler/xla/service/depthwise_convolution_converter_test.cc
deleted file mode 100644
index e9943b7e572..00000000000
--- a/tensorflow/compiler/xla/service/depthwise_convolution_converter_test.cc
+++ /dev/null
@@ -1,115 +0,0 @@
-/* Copyright 2019 The TensorFlow Authors. All Rights Reserved.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-==============================================================================*/
-
-#include "tensorflow/compiler/xla/service/depthwise_convolution_converter.h"
-
-#include <memory>
-#include <string>
-
-#include "tensorflow/compiler/xla/service/hlo_computation.h"
-#include "tensorflow/compiler/xla/service/hlo_instruction.h"
-#include "tensorflow/compiler/xla/service/hlo_matchers.h"
-#include "tensorflow/compiler/xla/service/hlo_opcode.h"
-#include "tensorflow/compiler/xla/service/hlo_parser.h"
-#include "tensorflow/compiler/xla/test.h"
-#include "tensorflow/compiler/xla/tests/hlo_test_base.h"
-#include "tensorflow/compiler/xla/types.h"
-
-namespace xla {
-namespace {
-
-using DepthwiseConvolutionConverterTest = HloTestBase;
-
-TEST_F(DepthwiseConvolutionConverterTest,
-       ConvertBatchGroupCountToFeatureGroupCount) {
-  string hlo_string = R"(HloModule Convolve1D1Window_0_module
-
-ENTRY %Convolve1D1Window_0.v3 (input: f32[16,19,19,512]{3,2,1,0}, filter: f32[16,19,19,512]{3,2,1,0}) -> f32[3,3,512,1]{3,2,1,0} {
-  %input = f32[16,19,19,512]{3,2,1,0} parameter(0)
-  %filter = f32[16,19,19,512]{3,2,1,0} parameter(1)
-  ROOT %convolution = f32[3,3,512,1]{3,2,1,0} convolution(f32[16,19,19,512]{3,2,1,0} %input, f32[16,19,19,512]{3,2,1,0} %filter), window={size=19x19 pad=1_1x1_1}, dim_labels=f01b_i01o->01fb, batch_group_count=512
-  })";
-  TF_ASSERT_OK_AND_ASSIGN(std::unique_ptr<HloModule> module,
-                          ParseAndReturnVerifiedModule(hlo_string));
-
-  auto computation = module->entry_computation();
-  HloInstruction* root = computation->root_instruction();
-  auto batch_group_count = root->batch_group_count();
-  EXPECT_EQ(root->opcode(), HloOpcode::kConvolution);
-  auto conv_dim_num = root->convolution_dimension_numbers();
-  int64 out_batch_dim = conv_dim_num.output_batch_dimension();
-  int64 out_feature_dim = conv_dim_num.output_feature_dimension();
-  auto cost_model = [](HloInstruction*) { return false; };
-  DepthwiseConvolutionConverter converter(cost_model);
-  ASSERT_TRUE(converter.Run(module.get()).ValueOrDie());
-  root = computation->root_instruction();
-  // Verify that the convolution is replaced by a reshape.
-  EXPECT_EQ(root->opcode(), HloOpcode::kReshape)
-      << HloOpcodeString(root->opcode()) << " vs Reshape";
-
-  // Verify that the operand to the reshape is the new convolution
-  // with feature_group_count = batch_group_count
-  auto new_conv = root->operand(0);
-  EXPECT_EQ(new_conv->opcode(), HloOpcode::kConvolution)
-      << HloOpcodeString(new_conv->opcode()) << " vs Convolution";
-  EXPECT_EQ(new_conv->feature_group_count(), batch_group_count);
-  // Verify that the output_batch_dim and output_feature_dim
-  // have been swapped back (tf2xla swaps these dimensions to make use
-  // of batch_group convolution for computing filter grad for depthwise
-  // convolutions)
-  EXPECT_EQ(new_conv->convolution_dimension_numbers().output_batch_dimension(),
-            out_feature_dim);
-  EXPECT_EQ(
-      new_conv->convolution_dimension_numbers().output_feature_dimension(),
-      out_batch_dim);
-
-  // Verify that the operand to conv is a reshape
-  auto reshape_1 = new_conv->operand(0);
-  EXPECT_EQ(reshape_1->opcode(), HloOpcode::kReshape)
-      << HloOpcodeString(reshape_1->opcode()) << " vs Reshape";
-
-  // Verify that the operand to reshape_1 is transpose
-  auto transpose = reshape_1->operand(0);
-  EXPECT_EQ(transpose->opcode(), HloOpcode::kTranspose)
-      << HloOpcodeString(transpose->opcode()) << " vs Transpose";
-
-  // Verify that the operand to transpose is reshape
-  auto reshape_2 = transpose->operand(0);
-  EXPECT_EQ(reshape_2->opcode(), HloOpcode::kReshape)
-      << HloOpcodeString(reshape_2->opcode()) << " vs Reshape";
-}
-
-TEST_F(DepthwiseConvolutionConverterTest,
-       OutputFeatureNotEqualBatchGroupCount) {
-  string hlo_string = R"(HloModule Convolve1D1Window_0_module
-  ENTRY %Convolve1D1Window_0.v3 (input: f32[4,6,6,48]{3,2,1,0}, filter: f32[4,6,6,96]{3,2,1,0}) -> f32[1,1,96,1]{3,2,1,0} {
-    %input = f32[4,6,6,48]{3,2,1,0} parameter(0)
-    %filter = f32[4,6,6,96]{3,2,1,0} parameter(1)
-
-    ROOT %convolution = f32[1,1,96,1]{3,2,1,0} convolution(f32[4,6,6,48]{3,2,1,0} %input, f32[4,6,6,96]{3,2,1,0} %filter), window={size=6x6 stride=2x2}, dim_labels=f01b_i01o->01fb, batch_group_count=48
-  })";
-  TF_ASSERT_OK_AND_ASSIGN(std::unique_ptr<HloModule> module,
-                          ParseAndReturnVerifiedModule(hlo_string));
-
-  auto computation = module->entry_computation();
-  HloInstruction* root = computation->root_instruction();
-  EXPECT_EQ(root->opcode(), HloOpcode::kConvolution);
-  auto cost_model = [](HloInstruction*) { return false; };
-  DepthwiseConvolutionConverter converter(cost_model);
-  ASSERT_TRUE(converter.Run(module.get()).ValueOrDie());
-}
-
-}  // namespace
-}  // namespace xla
diff --git a/tensorflow/compiler/xla/service/executable.cc b/tensorflow/compiler/xla/service/executable.cc
index f41c4b77cd1..0ab4a223916 100644
--- a/tensorflow/compiler/xla/service/executable.cc
+++ b/tensorflow/compiler/xla/service/executable.cc
@@ -245,6 +245,6 @@ StatusOr<ExecutionOutput> Executable::ExecuteAsyncOnStreamWrapper(
   return return_value;
 }
 
-int64 Executable::SizeOfGeneratedCodeInBytes() { return -1; }
+int64 Executable::SizeOfGeneratedCodeInBytes() const { return -1; }
 
 }  // namespace xla
diff --git a/tensorflow/compiler/xla/service/executable.h b/tensorflow/compiler/xla/service/executable.h
index 5d7bd26b01e..e6b26b4fdae 100644
--- a/tensorflow/compiler/xla/service/executable.h
+++ b/tensorflow/compiler/xla/service/executable.h
@@ -318,7 +318,7 @@ class Executable {
   // not supported by the executable.
   //
   // Does not include the size of used libraries (e.g. cuDNN, Eigen, etc.).
-  virtual int64 SizeOfGeneratedCodeInBytes();
+  virtual int64 SizeOfGeneratedCodeInBytes() const;
 
   // Dumping helpers.
   void set_hlo_proto(std::unique_ptr<xla::HloProto> hlo_proto) {
diff --git a/tensorflow/compiler/xla/service/gpu/BUILD b/tensorflow/compiler/xla/service/gpu/BUILD
index d6c33ddc855..2213abd82bc 100644
--- a/tensorflow/compiler/xla/service/gpu/BUILD
+++ b/tensorflow/compiler/xla/service/gpu/BUILD
@@ -205,7 +205,7 @@ cc_library(
         "@com_google_absl//absl/container:flat_hash_set",
         "@com_google_absl//absl/strings",
         "@com_google_absl//absl/types:span",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
 )
 
@@ -219,7 +219,7 @@ cc_library(
         "//tensorflow/core:lib",
         "@com_google_absl//absl/strings",
         "@com_google_absl//absl/types:span",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
@@ -309,7 +309,7 @@ cc_library(
         "@com_google_absl//absl/strings",
         "@com_google_absl//absl/types:optional",
         "@com_google_absl//absl/types:span",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
@@ -328,7 +328,7 @@ cc_library(
         "//tensorflow/compiler/xla/service/llvm_ir:llvm_util",
         "//tensorflow/compiler/xla/service/llvm_ir:loop_emitter",
         "//tensorflow/core:lib",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
 )
 
@@ -357,7 +357,7 @@ cc_library(
         "//tensorflow/core:lib",
         "@com_google_absl//absl/strings",
         "@com_google_absl//absl/types:span",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
@@ -647,7 +647,7 @@ cc_library(
         "//tensorflow/core:stream_executor_no_cuda",
         "//tensorflow/stream_executor:device_description",
         "@com_google_absl//absl/algorithm:container",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
 )
 
@@ -1136,7 +1136,7 @@ cc_library(
         "//tensorflow/core:lib",
         "//tensorflow/core:stream_executor_no_cuda",
         "@com_google_absl//absl/memory",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
     alwayslink = True,  # Contains per-platform transfer manager registration
 )
@@ -1191,8 +1191,6 @@ cc_library(
         "//tensorflow/compiler/xla/service:call_inliner",
         "//tensorflow/compiler/xla/service:conditional_simplifier",
         "//tensorflow/compiler/xla/service:convolution_4d_expander",
-        "//tensorflow/compiler/xla/service:convolution_group_converter",
-        "//tensorflow/compiler/xla/service:depthwise_convolution_converter",
         "//tensorflow/compiler/xla/service:dot_decomposer",
         "//tensorflow/compiler/xla/service:dump",
         "//tensorflow/compiler/xla/service:dynamic_index_splitter",
@@ -1237,7 +1235,7 @@ cc_library(
         "@com_google_absl//absl/strings",
         "@com_google_absl//absl/types:optional",
         "@com_google_absl//absl/types:span",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
 )
 
diff --git a/tensorflow/compiler/xla/service/gpu/gpu_compiler.cc b/tensorflow/compiler/xla/service/gpu/gpu_compiler.cc
index 5f6dfd7d3a5..78ef59236f7 100644
--- a/tensorflow/compiler/xla/service/gpu/gpu_compiler.cc
+++ b/tensorflow/compiler/xla/service/gpu/gpu_compiler.cc
@@ -37,8 +37,6 @@ limitations under the License.
 #include "tensorflow/compiler/xla/service/call_inliner.h"
 #include "tensorflow/compiler/xla/service/conditional_simplifier.h"
 #include "tensorflow/compiler/xla/service/convolution_4d_expander.h"
-#include "tensorflow/compiler/xla/service/convolution_group_converter.h"
-#include "tensorflow/compiler/xla/service/depthwise_convolution_converter.h"
 #include "tensorflow/compiler/xla/service/dot_decomposer.h"
 #include "tensorflow/compiler/xla/service/dump.h"
 #include "tensorflow/compiler/xla/service/dynamic_index_splitter.h"
@@ -152,23 +150,6 @@ Status GpuCompiler::OptimizeHloModule(
 
     pipeline.AddPass<Convolution4DExpander>();
 
-    auto cost_model = [](HloInstruction* conv) {
-      auto operand = conv->operand(0);
-      return operand->shape().dimensions(conv->convolution_dimension_numbers()
-                                             .input_batch_dimension()) ==
-             conv->batch_group_count();
-    };
-    pipeline.AddPass<DepthwiseConvolutionConverter>(cost_model);
-
-    // We use the ConvolutionGroupConverter to convert backprops of filter
-    // grouped convolutions into non-grouped equivalents.
-    auto batch_group_cost_model = [](HloInstruction*) { return false; };
-
-    pipeline.AddPass<ConvolutionGroupConverter>(
-        batch_group_cost_model,
-        /*convert_batch_groups_only=*/true,
-        /*filter_expansion=*/true);
-
     // Expand the sort op to support stable sorting if required.
     pipeline.AddPass<StableSortExpander>();
     // Convert BF16 operations to F32 operations so that the GPU backend can
diff --git a/tensorflow/compiler/xla/service/gpu/gpu_conv_rewriter.cc b/tensorflow/compiler/xla/service/gpu/gpu_conv_rewriter.cc
index 4a4448f668c..9d34bb39ba8 100644
--- a/tensorflow/compiler/xla/service/gpu/gpu_conv_rewriter.cc
+++ b/tensorflow/compiler/xla/service/gpu/gpu_conv_rewriter.cc
@@ -64,6 +64,59 @@ HloInstruction* CreateGpuConv(const char* call_target, const Shape& shape,
   return custom_call;
 }
 
+HloInstruction* ConvertBatchGroupedToFeatureGroupedConvolution(
+    HloInstruction* conv) {
+  CHECK_EQ(conv->feature_group_count(), 1);
+  int64 num_groups = conv->batch_group_count();
+  auto dim_numbers = conv->convolution_dimension_numbers();
+  auto lhs = conv->mutable_operand(0);
+  auto rhs = conv->mutable_operand(1);
+
+  int64 input_batch_dimension = dim_numbers.input_batch_dimension();
+
+  Shape output_shape = conv->shape();
+  int64 input_feature_dimension = dim_numbers.input_feature_dimension();
+  int64 input_feature = lhs->shape().dimensions(input_feature_dimension);
+
+  HloComputation* computation = lhs->parent();
+  auto add = [&](std::unique_ptr<HloInstruction> inst) {
+    return computation->AddInstruction(std::move(inst));
+  };
+  // Reshape batch_dim N -> [G, N/G]
+  std::vector<int64> reshape_dims = SpanToVector(lhs->shape().dimensions());
+  reshape_dims[input_batch_dimension] =
+      reshape_dims[input_batch_dimension] / num_groups;
+  reshape_dims.insert(reshape_dims.begin() + input_batch_dimension, num_groups);
+  lhs = add(HloInstruction::CreateReshape(
+      ShapeUtil::MakeShape(lhs->shape().element_type(), reshape_dims), lhs));
+
+  // Transpose G to the axis before C, For eg: [G, N/G, H, W, C ] -> [N/G, H,
+  // W, G, C]
+  std::vector<int64> transpose_dims(lhs->shape().dimensions_size());
+  std::iota(transpose_dims.begin(), transpose_dims.end(), 0);
+  transpose_dims.erase(transpose_dims.begin() + input_batch_dimension);
+  transpose_dims.insert(transpose_dims.begin() + input_feature_dimension,
+                        input_batch_dimension);
+  std::vector<int64> transpose_reshape_dims =
+      ComposePermutations(lhs->shape().dimensions(), transpose_dims);
+  lhs = add(HloInstruction::CreateTranspose(
+      ShapeUtil::MakeShape(lhs->shape().element_type(), transpose_reshape_dims),
+      lhs, transpose_dims));
+
+  // Merge [G,C] -> [C*G]
+  Shape new_shape = lhs->shape();
+  new_shape.DeleteDimension(input_feature_dimension);
+  new_shape.set_dimensions(input_feature_dimension, input_feature * num_groups);
+  lhs = add(HloInstruction::CreateReshape(new_shape, lhs));
+
+  std::vector<HloInstruction*> new_operands = {lhs, rhs};
+  auto new_conv = conv->CloneWithNewOperands(output_shape, new_operands);
+  new_conv->set_feature_group_count(num_groups);
+  new_conv->set_batch_group_count(1);
+  new_conv->set_convolution_dimension_numbers(dim_numbers);
+  return computation->AddInstruction(std::move(new_conv));
+}
+
 bool CanImplementAsGpuForwardConv(HloInstruction* conv) {
   const ConvolutionDimensionNumbers& dnums =
       conv->convolution_dimension_numbers();
@@ -91,9 +144,20 @@ bool CanImplementAsGpuForwardConv(HloInstruction* conv) {
 // Precondition: "conv" is a kConvolution.
 std::tuple<bool, Window, ConvolutionDimensionNumbers, HloInstruction*>
 MatchBackwardFilter(HloInstruction* conv) {
+  VLOG(2) << "Trying to match convolution backward filter.";
   const auto no_match_result =
       std::make_tuple(false, Window(), ConvolutionDimensionNumbers(), nullptr);
 
+  if (conv->feature_group_count() > 1) {
+    VLOG(1) << conv->ToString()
+            << " is a forward convolution. All grouped backward filters are "
+               "mapped to batch grouped convolutions in tf2xla bridge. Hence "
+               "backward filter "
+               "convolutions cannot have feature groups greater than 1 at this "
+               "point. No need to fold to backward filter.";
+    return no_match_result;
+  }
+
   // Step 1: match the instruction pattern without considering the paddings and
   // dimension numbers just yet. We may need some generic pattern matcher
   // similar to third_party/llvm/llvm/include/llvm/IR/PatternMatch.h
@@ -122,7 +186,6 @@ MatchBackwardFilter(HloInstruction* conv) {
   auto output_batch_dim = conv_dnums.output_batch_dimension();
   auto output_feature_dim = conv_dnums.output_feature_dimension();
   auto output_spatial_dims = conv_dnums.output_spatial_dimensions();
-
   for (const WindowDimension& window_dim : conv->window().dimensions()) {
     if (window_dim.stride() != 1) {
       VLOG(1) << "Forward convolution's window "
@@ -150,16 +213,7 @@ MatchBackwardFilter(HloInstruction* conv) {
       !window_util::HasWindowDilation(conv->window())) {
     VLOG(1) << conv->ToString()
             << " is a regular forward convolution. No need "
-               "to fold it to a backward filter convolution.";
-    return no_match_result;
-  }
-  auto rhs_in =
-      conv->mutable_operand(1)->shape().dimensions(kernel_input_feature_dim);
-  if (conv->feature_group_count() > 1 && rhs_in == 1 &&
-      input_batch_dim == output_batch_dim) {
-    VLOG(1) << conv->ToString()
-            << " is a depthwise forward convolution. No need to fold to "
-               "backward filter.";
+               "to fold it to a backward filter convolution....";
     return no_match_result;
   }
 
@@ -256,67 +310,14 @@ MatchBackwardFilter(HloInstruction* conv) {
   }
 
   HloInstruction* lhs = conv->mutable_operand(0);
-  if (conv->feature_group_count() == 1) {
-    return std::make_tuple(true, backward_conv_window, backward_conv_dnums,
-                           lhs);
-  }
-
-  int64 input_batch_dimension = backward_conv_dnums.input_batch_dimension();
-  int64 input_feature_dimension = backward_conv_dnums.input_feature_dimension();
-
-  int64 input_batch = lhs->shape().dimensions(input_batch_dimension);
-  int64 input_feature = lhs->shape().dimensions(input_feature_dimension);
-
-  // Reshape batch_dim G*N -> [G,N]
-  std::vector<int64> reshape_dims = SpanToVector(lhs->shape().dimensions());
-  auto num_groups = conv->feature_group_count();
-  CHECK_EQ(input_batch % num_groups, 0)
-      << "Input batch should be an exact multiple of feature group count";
-  reshape_dims[input_batch_dimension] =
-      reshape_dims[input_batch_dimension] / num_groups;
-  reshape_dims.insert(reshape_dims.begin() + input_batch_dimension, num_groups);
-
-  HloComputation* c = conv->parent();
-  HloInstruction* lhs_reshape_1 =
-      c->AddInstruction(HloInstruction::CreateReshape(
-          ShapeUtil::MakeShape(lhs->shape().element_type(), reshape_dims),
-          lhs));
-
-  // Transpose G to the axis before C/G, For eg: [G, N, C/G, H, W] -> [N, G,
-  // C/G, H, W]
-  std::vector<int64> transpose_dims(lhs_reshape_1->shape().dimensions_size());
-  std::iota(transpose_dims.begin(), transpose_dims.end(), 0);
-  transpose_dims.erase(transpose_dims.begin() + input_batch_dimension);
-  transpose_dims.insert(transpose_dims.begin() + input_feature_dimension,
-                        input_batch_dimension);
-  std::vector<int64> transpose_reshape_dims =
-      SpanToVector(lhs_reshape_1->shape().dimensions());
-  transpose_reshape_dims.erase(transpose_reshape_dims.begin() +
-                               input_batch_dimension);
-  transpose_reshape_dims.insert(
-      transpose_reshape_dims.begin() + input_feature_dimension, num_groups);
-
-  HloInstruction* lhs_transpose =
-      c->AddInstruction(HloInstruction::CreateTranspose(
-          ShapeUtil::MakeShape(lhs_reshape_1->shape().element_type(),
-                               transpose_reshape_dims),
-          lhs_reshape_1, transpose_dims));
-
-  // Merge [G,C/G] -> [C]
-  Shape new_shape = lhs_transpose->shape();
-  new_shape.DeleteDimension(input_feature_dimension);
-  new_shape.set_dimensions(input_feature_dimension,
-                           input_feature * conv->feature_group_count());
-  HloInstruction* lhs_reshape_2 = c->AddInstruction(
-      HloInstruction::CreateReshape(new_shape, lhs_transpose));
-  return std::make_tuple(true, backward_conv_window, backward_conv_dnums,
-                         lhs_reshape_2);
+  return std::make_tuple(true, backward_conv_window, backward_conv_dnums, lhs);
 }
 
 // Try to match a backward input pattern that contains "conv".
 // Precondition: "conv" is a kConvolution.
 std::tuple<bool, Window, ConvolutionDimensionNumbers, HloInstruction*>
 MatchBackwardInput(HloInstruction* conv) {
+  VLOG(2) << "Trying to match convolution backward input.";
   const auto no_match_result =
       std::make_tuple(false, Window(), ConvolutionDimensionNumbers(), nullptr);
 
@@ -639,7 +640,7 @@ static StatusOr<HloInstruction*> CreateCustomCallHelper(HloInstruction* conv) {
   if (match) {
     return CreateGpuConv(kCudnnConvBackwardFilterCallTarget, conv->shape(), lhs,
                          conv->mutable_operand(1), window, dnums,
-                         conv->feature_group_count(), conv->metadata());
+                         conv->batch_group_count(), conv->metadata());
   }
 
   // If all else fails, try a forward convolution.
@@ -682,6 +683,11 @@ static StatusOr<HloInstruction*> CreateCustomCallHelper(HloInstruction* conv) {
       TF_RETURN_IF_ERROR(
           conv->parent()->RemoveInstructionAndUnusedOperands(kernel_convert));
     }
+
+    if (conv->batch_group_count() > 1) {
+      conv = ConvertBatchGroupedToFeatureGroupedConvolution(conv);
+    }
+
     return CreateGpuConv(kCudnnConvForwardCallTarget, conv->shape(),
                          conv->mutable_operand(0), conv->mutable_operand(1),
                          conv->window(), conv->convolution_dimension_numbers(),
@@ -736,11 +742,13 @@ StatusOr<bool> RunOnComputation(HloComputation* computation) {
 }  // namespace
 
 StatusOr<bool> GpuConvRewriter::Run(HloModule* module) {
+  XLA_VLOG_LINES(2, "GpuConvRewriter::Run(), before:\n" + module->ToString());
   bool changed = false;
   for (HloComputation* computation : module->MakeNonfusionComputations()) {
     TF_ASSIGN_OR_RETURN(bool result, RunOnComputation(computation));
     changed |= result;
   }
+  XLA_VLOG_LINES(2, "GpuConvRewriter::Run(), after:\n" + module->ToString());
   return changed;
 }
 
diff --git a/tensorflow/compiler/xla/service/gpu/gpu_conv_rewriter_test.cc b/tensorflow/compiler/xla/service/gpu/gpu_conv_rewriter_test.cc
index b9c7d578d77..b10c33da2fd 100644
--- a/tensorflow/compiler/xla/service/gpu/gpu_conv_rewriter_test.cc
+++ b/tensorflow/compiler/xla/service/gpu/gpu_conv_rewriter_test.cc
@@ -135,86 +135,6 @@ TEST_F(GpuConvRewriterTest, BackwardFilterConvolve) {
       << md_after_opt.DebugString() << " vs " << metadata.DebugString();
 }
 
-TEST_F(GpuConvRewriterTest, BackwardFilterGroupConvolve) {
-  // In a nutshell, before pass:
-  // Input->batch_dim: 3 input_shape(3) = 4
-  // Input->feature_dim: 0 input_shape(0) = 32
-  // Kernel(gradient)->kernel_input_feature_dim (gradient_batch_dimension): 0
-  // Kernel(gradient)->kernel_output_feature_dim (gradient_feature_dimension): 3
-  // Output(dkernel)->output_batch_dim (dkernel_input_feature_dim): 2
-  // Output(dkernel)->output_feature_dim (dkernel_output_feature_dim): 3
-
-  // After pass: All shapes and dimension layout is brought
-  // back to normal as would be acceptable by cudnn
-  // Input->batch_dim: 0 input_shape(0) = 8
-  // Input->feature_dim: 3 input_shape(3) = 16
-  // Kernel(gradient)->kernel_input_feature_dim (gradient_batch_dimension): 2
-  // Kernel(gradient)->kernel_output_feature_dim (gradient_feature_dimension): 3
-  // Output(dkernel)->output_batch_dim (dkernel_input_feature_dim): 0
-  // Output(dkernel)->output_feature_dim (dkernel_output_feature_dim): 3
-  HloComputation::Builder builder(TestName());
-  HloInstruction* activations =
-      builder.AddInstruction(HloInstruction::CreateParameter(
-          0, ShapeUtil::MakeShape(F32, {32, 1, 3, 4}), "activations"));
-  HloInstruction* gradients =
-      builder.AddInstruction(HloInstruction::CreateParameter(
-          1, ShapeUtil::MakeShape(F32, {8, 1, 2, 16}), "gradients"));
-  Window conv_window = default_conv_window_;
-  conv_window.mutable_dimensions(1)->set_size(2);
-  conv_window.mutable_dimensions(1)->set_window_dilation(2);
-  auto* conv = builder.AddInstruction(HloInstruction::CreateConvolve(
-      ShapeInference::InferConvolveShape(
-          activations->shape(), gradients->shape(), /*feature_group_count=*/4,
-          /*batch_group_count=*/1, conv_window,
-          tf_default_dnums_for_backward_filter_)
-          .ConsumeValueOrDie(),
-      activations, gradients, /*feature_group_count=*/4,
-      /*batch_group_count=*/1, conv_window,
-      tf_default_dnums_for_backward_filter_, DefaultPrecisionConfig(2)));
-  OpMetadata metadata;
-  metadata.set_op_name("bar");
-  conv->set_metadata(metadata);
-  auto module = CreateNewVerifiedModule();
-  HloComputation* entry_computation =
-      module->AddEntryComputation(builder.Build());
-  EXPECT_TRUE(RunPass(module.get()));
-  ASSERT_THAT(entry_computation->root_instruction(),
-              op::GetTupleElement(
-                  op::CustomCall(kCudnnConvBackwardFilterCallTarget), 0));
-  // Check that metadata was preserved.
-  const auto& md_after_opt =
-      entry_computation->root_instruction()->operand(0)->metadata();
-  EXPECT_TRUE(protobuf_util::ProtobufEquals(md_after_opt, metadata))
-      << md_after_opt.DebugString() << " vs " << metadata.DebugString();
-  const HloInstruction* custom_call =
-      entry_computation->root_instruction()->operand(0);
-  const ConvolutionDimensionNumbers conv_dim =
-      custom_call->convolution_dimension_numbers();
-  const auto lhs_a = custom_call->operand(0);
-  const auto input_shape = lhs_a->shape();
-  // The input (lhs) batch_dim(dim 0 in the original NHWC layout) gets mapped to
-  // be the feature_dim(dim 3) with a value of N*g = 32 in tf2xla. As described
-  // in conv_grad_ops.h, this swap is required to implement backprop using fwd
-  // conv. After the pass the batch_dim gets remapped to dim 0. The batch_dim
-  // value gets scaled to N = N*g/g = 32/4 = 8 to be compatible with cudnn
-  EXPECT_EQ(0, conv_dim.input_batch_dimension());
-  EXPECT_EQ(8, input_shape.dimensions(conv_dim.input_batch_dimension()));
-  // Similarly, the input (lhs) feature_dim(dim 3 in the original NHWC layout)
-  // gets mapped to be the batch_dim(dim 0) with a value of C/g = 4 in tf2xla.
-  // After the pass the batch_dim gets remapped to dim 0. The feature_dim value
-  // gets scaled to C = C/g*g = 4*4 = 16 to be compatible with cudnn
-  EXPECT_EQ(3, conv_dim.input_feature_dimension());
-  EXPECT_EQ(16, input_shape.dimensions(conv_dim.input_feature_dimension()));
-  // Similarly, the feature and batch dims of the incoming gradients (used as
-  // rhs) and the in/out dims of the output of convolution i.e, dgrad have been
-  // been modified in tf2xla (as described in conv_grad_ops.h). This pass remaps
-  // everything back for the layout to be compatible with cudnn backprop APIs.
-  EXPECT_EQ(2, conv_dim.kernel_input_feature_dimension());
-  EXPECT_EQ(3, conv_dim.kernel_output_feature_dimension());
-  EXPECT_EQ(0, conv_dim.output_batch_dimension());
-  EXPECT_EQ(3, conv_dim.output_feature_dimension());
-}
-
 TEST_F(GpuConvRewriterTest,
        BackwardFilterConvolveEquivalentToForwardConvolution) {
   HloComputation::Builder builder(TestName());
diff --git a/tensorflow/compiler/xla/service/gpu/gpu_executable.cc b/tensorflow/compiler/xla/service/gpu/gpu_executable.cc
index a240a9dc65e..bf65df20544 100644
--- a/tensorflow/compiler/xla/service/gpu/gpu_executable.cc
+++ b/tensorflow/compiler/xla/service/gpu/gpu_executable.cc
@@ -510,13 +510,21 @@ const InstructionValueSet& GpuExecutable::GetRootValueSet() const {
       module().entry_computation()->root_instruction());
 }
 
-int64 GpuExecutable::SizeOfGeneratedCodeInBytes() {
+int64 GpuExecutable::SizeOfGeneratedCodeInBytes() const {
   // Non-empty PTX but empty cubin: compilation must have failed, return
   // "unknown".
   if (binary().empty() && !text_.empty()) {
     return -1;
   }
-  return binary().size();
+  int64 size = binary().size();
+  for (BufferAllocation::Index i = 0; i < assignment_->Allocations().size();
+       ++i) {
+    const BufferAllocation& allocation = assignment_->GetAllocation(i);
+    if (allocation.is_constant()) {
+      size += allocation.size();
+    }
+  }
+  return size;
 }
 
 }  // namespace gpu
diff --git a/tensorflow/compiler/xla/service/gpu/gpu_executable.h b/tensorflow/compiler/xla/service/gpu/gpu_executable.h
index 66a4e605821..0da446c9739 100644
--- a/tensorflow/compiler/xla/service/gpu/gpu_executable.h
+++ b/tensorflow/compiler/xla/service/gpu/gpu_executable.h
@@ -61,7 +61,7 @@ class GpuExecutable : public Executable {
                 std::unique_ptr<HloProfileIndexMap> hlo_profile_index_map);
   ~GpuExecutable() override;
 
-  int64 SizeOfGeneratedCodeInBytes() override;
+  int64 SizeOfGeneratedCodeInBytes() const override;
 
   // This should be called after set_ir_module_string.
   const string& ir_module_string() const { return ir_module_string_; }
diff --git a/tensorflow/compiler/xla/service/gpu/ir_emitter.cc b/tensorflow/compiler/xla/service/gpu/ir_emitter.cc
index aa8a6215cc7..5b1a0d7b2bf 100644
--- a/tensorflow/compiler/xla/service/gpu/ir_emitter.cc
+++ b/tensorflow/compiler/xla/service/gpu/ir_emitter.cc
@@ -236,12 +236,25 @@ bool IrEmitter::MaybeEmitDirectAtomicOperation(
   if (root_opcode == HloOpcode::kAdd) {
     llvm::Triple target_triple = llvm::Triple(module_->getTargetTriple());
     // NVPTX supports atomicAdd on F32 and integer types.
-    if (target_triple.isNVPTX() && element_type == F32) {
-      // F32 + F32
-      AtomicRMW(llvm::AtomicRMWInst::FAdd, output_address, source,
-                llvm::AtomicOrdering::SequentiallyConsistent);
-      return true;
+    if (target_triple.isNVPTX()) {
+      // "atom.add.f64 requires sm_60 or higher."
+      // https://docs.nvidia.com/cuda/parallel-thread-execution/index.html#parallel-synchronization-and-communication-instructions-atom
+      int cc_major = 0, cc_minor = 0;
+      ir_emitter_context_->device_description().cuda_compute_capability(
+          &cc_major, &cc_minor);
+
+      bool f64_atomic_add_supported = cc_major >= 6;
+
+      bool atomic_add_supported =
+          element_type == F32 ||
+          (f64_atomic_add_supported && element_type == F64);
+      if (atomic_add_supported) {
+        AtomicRMW(llvm::AtomicRMWInst::FAdd, output_address, source,
+                  llvm::AtomicOrdering::SequentiallyConsistent);
+        return true;
+      }
     }
+
     if (is_atomic_integral) {
       // integral + integral
       AtomicRMW(llvm::AtomicRMWInst::Add, output_address, source,
diff --git a/tensorflow/compiler/xla/service/gpu/kernel_thunk.cc b/tensorflow/compiler/xla/service/gpu/kernel_thunk.cc
index d976b5d8d4d..f0c7e285734 100644
--- a/tensorflow/compiler/xla/service/gpu/kernel_thunk.cc
+++ b/tensorflow/compiler/xla/service/gpu/kernel_thunk.cc
@@ -69,6 +69,23 @@ void KernelThunk::SetLaunchDimensions(const LaunchDimensions& launch_dims) {
   launch_dimensions_ = launch_dims;
 }
 
+static void PrintBufferContents(
+    se::Stream* stream, absl::Span<const se::DeviceMemoryBase> buffer_args) {
+  int input_idx = 0;
+  for (const se::DeviceMemoryBase& buf : buffer_args) {
+    auto host_buffer = absl::make_unique<char[]>(buf.size());
+    CHECK(stream->ThenMemcpy(host_buffer.get(), buf, buf.size()).ok());
+    CHECK(stream->BlockHostUntilDone().ok());
+
+    std::string buffer_contents;
+    for (int i = 0; i < buf.size(); i++) {
+      absl::StrAppendFormat(&buffer_contents, "%x ",
+                            static_cast<unsigned>(host_buffer[i]));
+    }
+    VLOG(100) << "BUF(" << input_idx++ << ") = " << buffer_contents;
+  }
+}
+
 Status KernelThunk::ExecuteOnStream(const ExecuteParams& params) {
   // Load the kernel.
   se::StreamExecutor* executor = params.stream->parent();
@@ -93,6 +110,11 @@ Status KernelThunk::ExecuteOnStream(const ExecuteParams& params) {
             << buf.size() << "B)";
     buffer_args.push_back(buf);
   }
+
+  if (VLOG_IS_ON(100)) {
+    PrintBufferContents(params.stream, buffer_args);
+  }
+
   auto op_profiler =
       params.profiler->MakeScopedInstructionProfiler(hlo_instruction());
   return ExecuteKernelOnStream(*kernel, buffer_args,
diff --git a/tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/BUILD b/tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/BUILD
index 1419a4f792d..b74eac1919d 100644
--- a/tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/BUILD
+++ b/tensorflow/compiler/xla/service/gpu/llvm_gpu_backend/BUILD
@@ -43,16 +43,16 @@ cc_library(
         "@llvm-project//llvm:analysis",
         "@llvm-project//llvm:bit_reader",
         "@llvm-project//llvm:bit_writer",
-        "@llvm-project//llvm:code_gen",
-        "@llvm-project//llvm:core",
-        "@llvm-project//llvm:ipo",
+        "@llvm-project//llvm:codegen",
+        "@llvm-project//llvm:ipo_transforms",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:ir_reader",
         "@llvm-project//llvm:linker",
-        "@llvm-project//llvm:nvptx_code_gen",  # buildcleaner: keep
-        "@llvm-project//llvm:objc_arc",  # buildcleaner: keep
-        "@llvm-project//llvm:scalar",
+        "@llvm-project//llvm:nvptx_target",  # buildcleaner: keep
+        "@llvm-project//llvm:objcarc_transforms",  # buildcleaner: keep
+        "@llvm-project//llvm:scalar_transforms",
         "@llvm-project//llvm:support",
-        "@llvm-project//llvm:target",
+        "@llvm-project//llvm:target_base",
     ],
 )
 
@@ -70,7 +70,7 @@ tf_cc_test(
         "//tensorflow/core:lib",
         "//tensorflow/core:test",
         "//tensorflow/core/platform:resource_loader",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
diff --git a/tensorflow/compiler/xla/service/gpu/tests/gpu_atomic_test.cc b/tensorflow/compiler/xla/service/gpu/tests/gpu_atomic_test.cc
index a54c0e5ae44..bd08b6189cf 100644
--- a/tensorflow/compiler/xla/service/gpu/tests/gpu_atomic_test.cc
+++ b/tensorflow/compiler/xla/service/gpu/tests/gpu_atomic_test.cc
@@ -80,6 +80,72 @@ CHECK-NOT: store atomic{{.*}}unordered, align 4
 )");
 }
 
+TEST_F(GpuAtomicTest, TestAddAtomicF32) {
+  const char* hlo_string = R"(
+    HloModule TensorFlowScatterV1
+
+    update_f32 (lhs: f32[], rhs: f32[]) -> f32[] {
+      lhs = f32[] parameter(0)
+      rhs = f32[] parameter(1)
+      ROOT add = f32[] add(lhs, rhs)
+    }
+
+    ENTRY main {
+      operand = f32[3,3] parameter(0)
+      indices = s32[2] parameter(1)
+      updates = f32[2,3] parameter(2)
+      ROOT scatter = f32[3,3] scatter(operand, indices, updates),
+          to_apply=update_f32,
+          update_window_dims={1},
+          inserted_window_dims={0},
+          scatter_dims_to_operand_dims={0},
+          index_vector_dim=1, unique_indices=false
+    }
+)";
+
+  CompileAndVerifyIr(hlo_string, R"(
+CHECK: atomicrmw fadd float* %[[ADDR:.*]], float %[[VALUE:.*]] seq_cst
+)");
+}
+
+TEST_F(GpuAtomicTest, TestAddAtomicF64) {
+  const se::DeviceDescription& device_description =
+      backend().default_stream_executor()->GetDeviceDescription();
+  int cc_major = 0, cc_minor = 0;
+  device_description.cuda_compute_capability(&cc_major, &cc_minor);
+
+  // Atomic add required sm_60 or above.
+  if (cc_major < 6) {
+    return;
+  }
+
+  const char* hlo_string = R"(
+    HloModule TensorFlowScatterV1
+
+    update_f64 (lhs: f64[], rhs: f64[]) -> f64[] {
+      lhs = f64[] parameter(0)
+      rhs = f64[] parameter(1)
+      ROOT add = f64[] add(lhs, rhs)
+    }
+
+    ENTRY main {
+      operand = f64[3,3] parameter(0)
+      indices = s32[2] parameter(1)
+      updates = f64[2,3] parameter(2)
+      ROOT scatter = f64[3,3] scatter(operand, indices, updates),
+          to_apply=update_f64,
+          update_window_dims={1},
+          inserted_window_dims={0},
+          scatter_dims_to_operand_dims={0},
+          index_vector_dim=1, unique_indices=false
+    }
+)";
+
+  CompileAndVerifyIr(hlo_string, R"(
+CHECK: atomicrmw fadd double* %[[ADDR:.*]], double %[[VALUE:.*]] seq_cst
+)");
+}
+
 }  // namespace
 }  // namespace gpu
 }  // namespace xla
diff --git a/tensorflow/compiler/xla/service/heap_simulator.h b/tensorflow/compiler/xla/service/heap_simulator.h
index cb492eaff90..d3b781ded0c 100644
--- a/tensorflow/compiler/xla/service/heap_simulator.h
+++ b/tensorflow/compiler/xla/service/heap_simulator.h
@@ -59,6 +59,10 @@ class HeapSimulator {
     int64 chunk_end() const { return offset + size; }
 
     bool OverlapsWith(Chunk other_chunk) const;
+
+    bool operator==(const Chunk& other) const {
+      return offset == other.offset && size == other.size;
+    }
   };
 
   // Result represents the result of the heap simulation.
diff --git a/tensorflow/compiler/xla/service/hlo_instruction.cc b/tensorflow/compiler/xla/service/hlo_instruction.cc
index c02100debc3..8eeb45731d1 100644
--- a/tensorflow/compiler/xla/service/hlo_instruction.cc
+++ b/tensorflow/compiler/xla/service/hlo_instruction.cc
@@ -2819,6 +2819,8 @@ bool HloInstruction::IsFusible() const {
     case HloOpcode::kReduce:
     case HloOpcode::kReduceWindow:
       return true;
+    case HloOpcode::kRng:
+      return user_count() <= 1;
     // Side effecting instructions cannot be fused.
     default:
       return !HasSideEffect();
diff --git a/tensorflow/compiler/xla/service/hlo_instructions.cc b/tensorflow/compiler/xla/service/hlo_instructions.cc
index 9c5a66f0040..bcc00d806da 100644
--- a/tensorflow/compiler/xla/service/hlo_instructions.cc
+++ b/tensorflow/compiler/xla/service/hlo_instructions.cc
@@ -1425,9 +1425,25 @@ void HloFusionInstruction::MergeFusionInstruction(
         unfused_instructions.empty());
   // Replace instruction_to_merge use of 'this' with unfused_root.
   TF_CHECK_OK(instruction_to_merge->ReplaceUseWith(this, unfused_root));
-  // Fuse 'unfused_instructions' into 'this'.
+
+  // Build a dummy root for the cloned fusion as we may remove the original root
+  // in the fusion process.
+  if (!unfused_instructions.empty()) {
+    HloComputation* computation = unfused_root->parent();
+    auto* dummy_root = computation->AddInstruction(
+        HloInstruction::CreateConstant(LiteralUtil::Zero(U32)));
+    computation->set_root_instruction(dummy_root,
+                                      /*accept_different_shape=*/true);
+  }
+
+  // Fuse 'unfused_instructions' into 'this'. Everytime we fuse an instruction
+  // we remove it from the closed fusion node. This is so that we don't add
+  // extra users to the producer of that instruction (we use user count to
+  // decide if a side-effectful instruction is fusible).
   for (auto& instruction : unfused_instructions) {
-    FuseInstruction(instruction);
+    auto* fused = FuseInstruction(instruction);
+    TF_CHECK_OK(instruction->ReplaceAllUsesWith(fused));
+    TF_CHECK_OK(instruction->parent()->RemoveInstruction(instruction));
   }
   CHECK_EQ(0, cloned_fusion->user_count());
   TF_CHECK_OK(parent()->parent()->RemoveEmbeddedComputation(
@@ -1802,7 +1818,8 @@ bool HloRngInstruction::IdenticalSlowPath(
     const HloInstruction& other,
     const std::function<bool(const HloComputation*, const HloComputation*)>&
         eq_computations) const {
-  return true;
+  const auto& casted_other = static_cast<const HloRngInstruction&>(other);
+  return distribution_ == casted_other.distribution_;
 }
 
 std::unique_ptr<HloInstruction> HloRngInstruction::CloneWithNewOperandsImpl(
diff --git a/tensorflow/compiler/xla/service/hlo_memory_scheduler.cc b/tensorflow/compiler/xla/service/hlo_memory_scheduler.cc
index bda297540ff..c8f996465b8 100644
--- a/tensorflow/compiler/xla/service/hlo_memory_scheduler.cc
+++ b/tensorflow/compiler/xla/service/hlo_memory_scheduler.cc
@@ -27,6 +27,7 @@ limitations under the License.
 #include "tensorflow/compiler/xla/service/dfs_hlo_visitor_with_default.h"
 #include "tensorflow/compiler/xla/service/heap_simulator.h"
 #include "tensorflow/compiler/xla/service/hlo_computation.h"
+#include "tensorflow/compiler/xla/service/hlo_opcode.h"
 #include "tensorflow/compiler/xla/service/hlo_schedule.h"
 #include "tensorflow/compiler/xla/service/tuple_points_to_analysis.h"
 #include "tensorflow/compiler/xla/shape_util.h"
@@ -211,6 +212,18 @@ class ListScheduler {
   // improve accounting for subcomputation memory (b/65409243).
   int64 BytesFreedIfScheduled(const ReadyListEntry& entry) {
     auto instruction = entry.instruction;
+    auto opcode = instruction->opcode();
+
+    // Scheduling the outfeed early and the infeed late gives more time to the
+    // communicating processor to do its work.
+    if (opcode == HloOpcode::kOutfeed &&
+        !instruction->outfeed_config().empty()) {
+      return INT_MAX;
+    }
+    if (opcode == HloOpcode::kInfeed && !instruction->infeed_config().empty()) {
+      return INT_MIN;
+    }
+
     int64 freed_bytes = 0;
     for (const auto& kv : entry.used_buffer_unscheduled_use_counts) {
       auto buffer = kv->first;
@@ -232,7 +245,6 @@ class ListScheduler {
       }
     }
     int64 bytes_defined;
-    auto opcode = instruction->opcode();
     if (max_subcomputation_bytes > 0 &&
         (opcode == HloOpcode::kWhile || opcode == HloOpcode::kCall ||
          opcode == HloOpcode::kConditional)) {
diff --git a/tensorflow/compiler/xla/service/instruction_fusion_test.cc b/tensorflow/compiler/xla/service/instruction_fusion_test.cc
index f4309ea09ae..f3943c2c21e 100644
--- a/tensorflow/compiler/xla/service/instruction_fusion_test.cc
+++ b/tensorflow/compiler/xla/service/instruction_fusion_test.cc
@@ -184,6 +184,7 @@ TEST_F(InstructionFusionTest, AvoidDuplicationIfNotAllFusibleRecursively) {
     abs1 = f32[] abs(add)
     rng = f32[] rng(p1, abs1), distribution=rng_uniform
     abs2 = f32[] abs(rng)
+    abs3 = f32[] abs(rng)    
     ROOT root = f32[] subtract(abs2, add)
   })")
                     .ValueOrDie();
@@ -375,6 +376,7 @@ TEST_F(InstructionFusionTest, AllowBinarySameValueOperandsDuplication) {
     abs1 = f32[] abs(add)
     rng = f32[] rng(p0, abs1), distribution=rng_uniform
     abs2 = f32[] abs(rng)
+    abs3 = f32[] abs(rng)
     ROOT root = f32[] subtract(abs2, add)
   })")
                     .ValueOrDie();
diff --git a/tensorflow/compiler/xla/service/layout_assignment.cc b/tensorflow/compiler/xla/service/layout_assignment.cc
index 82c30f1a710..307fd82069e 100644
--- a/tensorflow/compiler/xla/service/layout_assignment.cc
+++ b/tensorflow/compiler/xla/service/layout_assignment.cc
@@ -952,7 +952,11 @@ Status LayoutAssignment::CheckLayouts(HloModule* module) {
                          .IgnoreDynamicDimension()
                          .MinorToMajorOnlyInLayout()(instruction_subshape,
                                                      buffer->shape()) &&
-                    instruction->opcode() != HloOpcode::kBitcast) {
+                    // TODO(mingyao): Use explicit linear layout tiling to
+                    // detect and allow special bitcast.
+                    instruction->opcode() != HloOpcode::kBitcast &&
+                    instruction->opcode() != HloOpcode::kGetTupleElement &&
+                    instruction->opcode() != HloOpcode::kTuple) {
                   return InternalError(
                       "Layout of instruction %s at index {%s} does not match "
                       "source LogicalBuffer %s: %s vs %s",
diff --git a/tensorflow/compiler/xla/service/llvm_ir/BUILD b/tensorflow/compiler/xla/service/llvm_ir/BUILD
index cabcc8e06ee..a9491470e8f 100644
--- a/tensorflow/compiler/xla/service/llvm_ir/BUILD
+++ b/tensorflow/compiler/xla/service/llvm_ir/BUILD
@@ -42,7 +42,7 @@ cc_library(
         "@com_google_absl//absl/container:flat_hash_map",
         "@com_google_absl//absl/container:flat_hash_set",
         "@com_google_absl//absl/strings",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
 )
 
@@ -79,9 +79,9 @@ cc_library(
         "@com_google_absl//absl/base",
         "@com_google_absl//absl/strings",
         "@com_google_absl//absl/types:span",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
-        "@llvm-project//llvm:target",
+        "@llvm-project//llvm:target_base",
         "@llvm-project//llvm:transform_utils",
     ],
 )
@@ -101,7 +101,7 @@ cc_library(
         "@com_google_absl//absl/algorithm:container",
         "@com_google_absl//absl/strings",
         "@com_google_absl//absl/types:span",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
 )
 
@@ -119,7 +119,7 @@ cc_library(
         "@com_google_absl//absl/algorithm:container",
         "@com_google_absl//absl/strings",
         "@com_google_absl//absl/types:span",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
 )
 
@@ -137,7 +137,7 @@ cc_library(
         "//tensorflow/compiler/xla:xla_data_proto_cc",
         "//tensorflow/core:lib",
         "@com_google_absl//absl/strings:str_format",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
 )
 
@@ -162,7 +162,7 @@ cc_library(
         "@com_google_absl//absl/container:flat_hash_set",
         "@com_google_absl//absl/types:optional",
         "@com_google_absl//absl/types:span",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
 )
 
@@ -201,7 +201,7 @@ cc_library(
         "//tensorflow/core:lib",
         "@com_google_absl//absl/strings",
         "@com_google_absl//absl/types:span",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
@@ -218,7 +218,7 @@ cc_library(
         "//tensorflow/compiler/xla:xla_data_proto_cc",
         "//tensorflow/core:lib",
         "@com_google_absl//absl/types:span",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
 )
 
@@ -230,7 +230,7 @@ cc_library(
         ":llvm_loop",
         ":llvm_util",
         "@com_google_absl//absl/strings",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
 )
 
@@ -250,7 +250,7 @@ cc_library(
     hdrs = ["math_ops.h"],
     deps = [
         ":llvm_util",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
 )
 
@@ -259,6 +259,6 @@ cc_library(
     srcs = [],
     hdrs = ["ir_builder_mixin.h"],
     deps = [
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
 )
diff --git a/tensorflow/compiler/xla/service/memory_space_assignment.cc b/tensorflow/compiler/xla/service/memory_space_assignment.cc
index 0ed72f51754..a28e71f6cc3 100644
--- a/tensorflow/compiler/xla/service/memory_space_assignment.cc
+++ b/tensorflow/compiler/xla/service/memory_space_assignment.cc
@@ -31,18 +31,31 @@ const int kWhileExecutionCount = 5;
 }  // namespace
 
 float MemorySpaceAssignmentCostAnalysis::GetAlternateMemoryBenefit(
-    const HloInstruction& instruction,
-    float elapsed_time_due_to_alternate_mem) const {
+    const HloInstruction& instruction, float elapsed_time_due_to_alternate_mem,
+    MemorySpaceAssignmentCostAnalysis::Cache* cache) const {
   float elapsed_time_due_to_compute =
       GetInstructionElapsedDueToCompute(instruction);
   float elapsed_time_due_to_memory =
       GetInstructionElapsedDueToMemory(instruction);
   if (elapsed_time_due_to_memory > elapsed_time_due_to_compute) {
     // Memory bound, return how much alternate memory is better.
-    int while_nest_level = CalculateWhileLoopNestLevel(&instruction);
+    float while_nest_multiplier;
+    if (cache) {
+      // If there is a cache provided, memoize the while nest multiplier.
+      auto it = cache->while_nest_multiplier.find(&instruction);
+      if (it != cache->while_nest_multiplier.end()) {
+        while_nest_multiplier = it->second;
+      } else {
+        while_nest_multiplier = tensorflow::MathUtil::IPow<float>(
+            kWhileExecutionCount, CalculateWhileLoopNestLevel(&instruction));
+        cache->while_nest_multiplier[&instruction] = while_nest_multiplier;
+      }
+    } else {
+      while_nest_multiplier = tensorflow::MathUtil::IPow<float>(
+          kWhileExecutionCount, CalculateWhileLoopNestLevel(&instruction));
+    }
     return (elapsed_time_due_to_memory - elapsed_time_due_to_alternate_mem) *
-           tensorflow::MathUtil::IPow<float>(kWhileExecutionCount,
-                                             while_nest_level);
+           while_nest_multiplier;
   } else {
     // Compute bound, return how far off are we to memory boundedness.
     return elapsed_time_due_to_memory - elapsed_time_due_to_compute;
@@ -50,18 +63,21 @@ float MemorySpaceAssignmentCostAnalysis::GetAlternateMemoryBenefit(
 }
 
 float MemorySpaceAssignmentCostAnalysis::GetMemoryBoundedness(
-    const GlobalDecreasingSizeBestFitHeap::BufferInterval& interval) const {
+    const GlobalDecreasingSizeBestFitHeap::BufferInterval& interval,
+    MemorySpaceAssignmentCostAnalysis::Cache* cache) const {
   const HloInstruction& defining_instruction =
       *interval.buffer->defining_instruction();
   float alternate_mem_benefit = GetAlternateMemoryBenefit(
       defining_instruction,
       GetInstructionElapsedDueToMemory(defining_instruction,
                                        /*operand_in_alternate_mem=*/{},
-                                       /*output_in_alternate_mem=*/true));
+                                       /*output_in_alternate_mem=*/true),
+      cache);
   for (const HloUse& use : interval.buffer->uses()) {
     float use_alternate_mem_benefit = GetAlternateMemoryBenefit(
         *use.instruction,
-        GetInstructionElapsedDueToMemory(*use.instruction, use.operand_number));
+        GetInstructionElapsedDueToMemory(*use.instruction, use.operand_number),
+        cache);
     // If the benefit is positive (memory bound), add it to this buffer's
     // benefit. If the benefit is negative (compute bound), calculate the
     // maximum.
@@ -221,9 +237,7 @@ CostAnalysisPrefetchIntervalPicker::CostAnalysisPrefetchIntervalPicker(
     const MemorySpaceAssignmentCostAnalysis& cost_analysis,
     float min_async_copy_to_overlap_ratio,
     float max_async_copy_to_overlap_ratio)
-    : elapsed_time_(
-          cost_analysis.hlo_live_range().instruction_schedule().size(), 0.0),
-      while_nest_level_(
+    : while_nest_level_(
           cost_analysis.hlo_live_range().instruction_schedule().size(), 0),
       cost_analysis_(cost_analysis),
       min_async_copy_to_overlap_ratio_(min_async_copy_to_overlap_ratio),
@@ -232,19 +246,46 @@ CostAnalysisPrefetchIntervalPicker::CostAnalysisPrefetchIntervalPicker(
       &cost_analysis_.hlo_live_range().instruction_schedule();
 
   // Create a vector of elapsed times and while nesting levels of HLO
-  // instructions.
+  // instructions. The elapsed times are multiplied by pow(kWhileExecutionCount,
+  // nest_level) to account for executing the HLOs multiple times in while
+  // loops.
+  std::vector<float> instructions_elapsed_time(instruction_schedule_->size(),
+                                               0.0);
   for (const auto& instruction_and_logical_time : *instruction_schedule_) {
     float elapsed_time = cost_analysis_.cost_analysis().optimal_seconds(
         *instruction_and_logical_time.first);
     int64 logical_time = instruction_and_logical_time.second;
-    if (logical_time >= elapsed_time_.size()) {
-      elapsed_time_.resize(logical_time + 1, 0.0);
+    if (logical_time >= instructions_elapsed_time.size()) {
+      instructions_elapsed_time.resize(logical_time + 1, 0.0);
       while_nest_level_.resize(logical_time + 1, 0);
     }
-    elapsed_time_[logical_time] = elapsed_time;
-    while_nest_level_[logical_time] =
-        cost_analysis_.CalculateWhileLoopNestLevel(
-            instruction_and_logical_time.first);
+    int nest_level = cost_analysis_.CalculateWhileLoopNestLevel(
+        instruction_and_logical_time.first);
+    while_nest_level_[logical_time] = nest_level;
+    instructions_elapsed_time[logical_time] =
+        elapsed_time *
+        tensorflow::MathUtil::IPow<float>(kWhileExecutionCount, nest_level);
+  }
+  // As an optimization, create a cumulative sum vector of elapsed time.
+  float cumsum = 0.0;
+  elapsed_time_cumsum_.reserve(instructions_elapsed_time.size());
+  for (float elapsed_time : instructions_elapsed_time) {
+    cumsum += elapsed_time;
+    elapsed_time_cumsum_.push_back(cumsum);
+  }
+  // To be able to accurately determine the minimum nest level between a start
+  // time and an end time efficiently, populate a data structure that stores the
+  // closest nest level change index.
+  int prev_nest_level = 0;
+  int change_idx = -1;
+  while_nest_level_change_.reserve(instructions_elapsed_time.size());
+  for (int i = 0; i < while_nest_level_.size(); ++i) {
+    int nest_level = while_nest_level_[i];
+    if (nest_level != prev_nest_level) {
+      prev_nest_level = nest_level;
+      change_idx = i - 1;
+    }
+    while_nest_level_change_.push_back(change_idx);
   }
 }
 
@@ -256,7 +297,8 @@ bool CostAnalysisPrefetchIntervalPicker::CanAllocateInAlternateMemoryNoCopy(
   float async_copy_elapsed = cost_analysis_.GetAsyncCopyElapsed(shape);
   float logical_interval_elapsed =
       GetLogicalIntervalElapsed(start_time, end_time);
-  return max_async_copy_to_overlap_ratio_ * async_copy_elapsed >
+  return max_async_copy_to_overlap_ratio_ * max_overlap_multiplier_ *
+             async_copy_elapsed >
          logical_interval_elapsed;
 }
 
@@ -291,11 +333,20 @@ void CostAnalysisPrefetchIntervalPicker::Begin(const HloUse& use,
   // Find the earliest time we're allowed to start prefetching.
   for (current_logical_prefetch_time_ = start_time;
        current_logical_prefetch_time_ < end_logical_time_ &&
-       max_async_copy_to_overlap_ratio_ * async_copy_elapsed_ <
+       max_async_copy_to_overlap_ratio_ * max_overlap_multiplier_ *
+               async_copy_elapsed_ <
            GetLogicalIntervalElapsed(current_logical_prefetch_time_,
                                      end_logical_time_);
        ++current_logical_prefetch_time_) {
   }
+  // If the first prefetch interval violates the min overlap (this can happen if
+  // there is an HLO that has a very long estimated execution time), we go
+  // earlier in time until we no longer violate the min overlap (we start
+  // prefetching before the HLO that has the very long estimated execution
+  // time).
+  while (Done() && current_logical_prefetch_time_ > start_time) {
+    --current_logical_prefetch_time_;
+  }
 }
 
 int64 CostAnalysisPrefetchIntervalPicker::Next() {
@@ -316,19 +367,37 @@ bool CostAnalysisPrefetchIntervalPicker::Done() const {
          logical_interval_elapsed + inst_elapsed_reduction_;
 }
 
+void CostAnalysisPrefetchIntervalPicker::SetRetryNumber(int retry_number) {
+  // Use twice as large max overlap limit in each retry.
+  max_overlap_multiplier_ = 1 << retry_number;
+}
+
+int CostAnalysisPrefetchIntervalPicker::GetMinWhileNestLevel(
+    int64 start_time, int64 end_time) const {
+  int min_nest_level =
+      std::min(while_nest_level_[start_time], while_nest_level_[end_time]);
+  int change_idx = while_nest_level_change_[end_time];
+  while (change_idx >= start_time) {
+    min_nest_level = std::min(min_nest_level, while_nest_level_[change_idx]);
+    change_idx = while_nest_level_change_[change_idx];
+  }
+  return min_nest_level;
+}
+
 float CostAnalysisPrefetchIntervalPicker::GetLogicalIntervalElapsed(
     int64 start_time, int64 end_time) const {
-  int interval_nest_level =
-      std::min(while_nest_level_[start_time], while_nest_level_[end_time]);
-  float total_elapsed = 0;
-  for (int i = start_time + 1; i < end_time; ++i) {
-    total_elapsed +=
-        elapsed_time_[i] *
-        tensorflow::MathUtil::IPow<float>(
-            kWhileExecutionCount,
-            std::max(0, while_nest_level_[i] - interval_nest_level));
+  CHECK_LE(start_time, end_time);
+  if (start_time == end_time) {
+    return 0.0;
   }
-  return total_elapsed;
+  // Since elapsed_time_cumsum_ is already weighed by the while loop nesting
+  // level, normalize the elapsed time by dividing with the nesting factor of
+  // the interval (start and end times).
+  int interval_nest_level = GetMinWhileNestLevel(start_time, end_time);
+  return (elapsed_time_cumsum_[end_time - 1] -
+          elapsed_time_cumsum_[start_time]) /
+         tensorflow::MathUtil::IPow<float>(kWhileExecutionCount,
+                                           interval_nest_level);
 }
 
 std::string CostAnalysisPrefetchIntervalPicker::ToDebugString() const {
@@ -337,7 +406,9 @@ std::string CostAnalysisPrefetchIntervalPicker::ToDebugString() const {
   return absl::StrCat(
       "Async copy elapsed (s) = ", async_copy_elapsed_,
       ", inst elapsed reduction (s) = ", inst_elapsed_reduction_,
-      ", logical interval elapsed (s) = ", logical_interval_elapsed);
+      ", logical interval elapsed (s) = ", logical_interval_elapsed,
+      ", interval = (", current_logical_prefetch_time_, ", ", end_logical_time_,
+      ")");
 }
 
 std::string CostAnalysisPrefetchIntervalPicker::ToNoCopyDebugString(
@@ -557,11 +628,12 @@ bool AlternateMemoryBestFitHeap::IsUseAllowedInAlternateMemory(
     int64 root_time = instruction_schedule.at(while_body->root_instruction());
     int64 min_use_time = root_time;
     for (const HloUse& parameter_use : parameter_value->uses()) {
+      int64 use_time = instruction_schedule.at(parameter_use.instruction);
       if (parameter_use.instruction->opcode() != HloOpcode::kGetTupleElement &&
           parameter_use.instruction->opcode() != HloOpcode::kTuple &&
-          parameter_use.instruction->opcode() != HloOpcode::kBitcast) {
-        min_use_time = std::min(
-            min_use_time, instruction_schedule.at(parameter_use.instruction));
+          parameter_use.instruction->opcode() != HloOpcode::kBitcast &&
+          use_time > parameter_time) {
+        min_use_time = std::min(min_use_time, use_time);
       }
     }
     // If there is no use of this buffer inside the while loop, there is no need
@@ -571,21 +643,13 @@ bool AlternateMemoryBestFitHeap::IsUseAllowedInAlternateMemory(
               << "use time = " << min_use_time << ", root time = " << root_time;
       return false;
     }
-    HloValue* root_value =
-        &alias_analysis_.dataflow_analysis().GetUniqueValueAt(
-            while_body->root_instruction(), use.operand_index);
-    int64 root_definition_time =
-        instruction_schedule.at(root_value->defining_instruction());
-    const Shape& shape = root_value->shape();
+    const Shape& shape = parameter_value->shape();
     // Allow the buffer in alternate memory if the buffer has a short live range
     // either at the beginning or end of the while loop body.
     if (!options_.prefetch_interval_picker->CanAllocateInAlternateMemoryNoCopy(
-            shape, parameter_time, min_use_time) &&
-        !options_.prefetch_interval_picker->CanAllocateInAlternateMemoryNoCopy(
-            shape, root_definition_time, root_time)) {
+            shape, parameter_time, min_use_time)) {
       VLOG(4) << "While allocation not allowed in alternate memory. "
               << "use time = " << min_use_time
-              << ", def time = " << root_definition_time
               << ", root time = " << root_time;
       return false;
     }
@@ -740,14 +804,12 @@ void AlternateMemoryBestFitHeap::AppendAllocationInfoDebugString(
   }
 }
 
-void AlternateMemoryBestFitHeap::DumpIfEnabled(
-    absl::string_view buffer_info_str,
-    absl::string_view allocation_info_str) const {
+void AlternateMemoryBestFitHeap::DumpDebugStringsIfEnabled() const {
   if (!options_.dump_fn) {
     return;
   }
-  options_.dump_fn("bufferinfo", buffer_info_str);
-  options_.dump_fn("allocinfo", allocation_info_str);
+  options_.dump_fn("bufferinfo", buffer_info_str_);
+  options_.dump_fn("allocinfo", allocation_info_str_);
 }
 
 HeapSimulator::Result AlternateMemoryBestFitHeap::Finish() {
@@ -769,9 +831,6 @@ HeapSimulator::Result AlternateMemoryBestFitHeap::Finish() {
     }
   }
 
-  std::string buffer_info_str;
-  std::string allocation_info_str;
-
   for (auto& interval : sorted_buffer_intervals) {
     if (!interval.need_allocation) {
       continue;
@@ -795,12 +854,6 @@ HeapSimulator::Result AlternateMemoryBestFitHeap::Finish() {
     }
 
     auto colocated_intervals = GetSortedColocatedIntervals(interval);
-    // Create AllocationValues for all the
-    // colocated intervals.
-    std::vector<AllocationValue> allocation_values;
-    for (const auto& colocated_interval : colocated_intervals) {
-      CreateAllocationValues(colocated_interval->buffer, &allocation_values);
-    }
 
     if (AreIntervalsReservedInAlternateMemory(colocated_intervals)) {
       VLOG(3) << "Interval " << interval.buffer->ToShortString()
@@ -843,213 +896,234 @@ HeapSimulator::Result AlternateMemoryBestFitHeap::Finish() {
       continue;
     }
 
-    const auto& instruction_schedule = hlo_live_range_.instruction_schedule();
+    AppendBufferInfoDebugString(interval, &buffer_info_str_);
 
-    // TODO(berkin): For now, place the phi values due to conditionals in
-    // default memory.
-    for (const BufferInterval* colocated_interval : colocated_intervals) {
-      const HloValue* value = colocated_interval->buffer;
-      for (const auto& position : value->positions()) {
-        if (position.instruction->opcode() == HloOpcode::kConditional) {
-          VLOG(3) << "Adding required assignment for condition output: "
-                  << value->ToShortString();
-          AddRequiredAssignment(position.instruction, position.index,
-                                MemorySpace::kDefault);
-          for (const HloComputation* called_computation :
-               position.instruction->called_computations()) {
-            AddRequiredAssignment(called_computation->root_instruction(),
-                                  position.index, MemorySpace::kDefault);
-          }
-        }
-      }
-    }
-
-    AppendBufferInfoDebugString(interval, &buffer_info_str);
-
-    // Data structure to contain the preferred offset for a given computation.
-    // We ensure that the same offset will be allocated outside the while loop
-    // as well as inside the while loop.
-    absl::flat_hash_map<const HloComputation*, int64>
-        preferred_offset_for_computation;
-    bool allocation_success = true;
-    for (auto& allocation_value : allocation_values) {
-      int64 definition_time =
-          instruction_schedule.at(allocation_value.defining_instruction());
-
-      absl::optional<int64> preferred_offset;
-      auto preferred_offset_it =
-          preferred_offset_for_computation.find(allocation_value.computation());
-      if (preferred_offset_it != preferred_offset_for_computation.end()) {
-        preferred_offset = preferred_offset_it->second;
-      }
-
-      // Iterate over the uses.
-      for (int use_idx = 0; use_idx < allocation_value.uses().size();
-           ++use_idx) {
-        const HloUse& use = allocation_value.uses().at(use_idx);
-        int64 use_time = instruction_schedule.at(use.instruction);
-        int64 latest_prefetch_time = use_time;
-        bool allow_no_copy_alternate_mem_allocation = true;
-        absl::optional<int64> earliest_prefetch_time = absl::nullopt;
-
-        // Sequential calls include kWhile, kCall, and kConditional opcodes.
-        bool is_sequential_call =
-            (GetInstructionCallContext(use.instruction->opcode()) ==
-             CallContext::kSequential);
-        if (is_sequential_call) {
-          for (const HloComputation* called_computation :
-               use.instruction->called_computations()) {
-            const HloLiveRange::TimeBound& computation_span =
-                hlo_live_range_.computation_span_times().at(called_computation);
-            latest_prefetch_time =
-                std::min(computation_span.start, latest_prefetch_time);
-          }
-          if (use.instruction->opcode() == HloOpcode::kWhile) {
-            // Given an example while loop and flattened schedule (logical times
-            // shown on the left):
-            //
-            // 0:  a = ...
-            // 1:  ...
-            //     cond {
-            // 2:   p = param(0)
-            // 3:   ...
-            //     }
-            //     body {
-            // 4:   p = param(0)
-            // 5:   ...
-            // 6:   ROOT ...
-            //     }
-            // 7:  w = while(a), body=body, cond=cond
-            //
-            // When processing "a" (time 0) and its while use (time 7), we
-            // update the interval to time 0-4. This is so that the remaining
-            // interval (5-6) can be allocated separately and this buffer
-            // doesn't waste alternate memory space within the while loop body.
-            HloComputation* while_body = use.instruction->while_body();
-            // We require while body ROOTs to be the last in the schedule.
-            CHECK_EQ(
-                instruction_schedule.at(while_body->root_instruction()) + 1,
-                instruction_schedule.at(use.instruction))
-                << "While body ROOTs need to be the last in the schedule!  "
-                   "Please run RootInstructionSinker.";
-            // Replace the use time with the parameter time so that we can
-            // decide on alternate memory allocations within the while loop body
-            // when we look at uses within the while loop body.
-            use_time =
-                instruction_schedule.at(while_body->parameter_instruction(0));
-          } else if (use.instruction->opcode() == HloOpcode::kConditional) {
-            // Replace the use time with the earliest parameter of called
-            // computations.
-            for (const HloComputation* called_computation :
-                 use.instruction->called_computations()) {
-              use_time = std::min(
-                  use_time, instruction_schedule.at(
-                                called_computation->parameter_instruction(0)));
-            }
-          }
-        }
-
-        // Add a required assignment in default memory if the use not allowed in
-        // alternate memory.
-        if (!IsUseAllowedInAlternateMemory(allocation_value, use)) {
-          AddRequiredAssignment(allocation_value.value(), use.instruction,
-                                MemorySpace::kDefault, use_time);
-        } else if (use_idx > 0) {
-          // We allow buffers in alternate memory that are passed into
-          // conditionals to give up their alternate memory allocation inside
-          // the called computation. This means that if a conditional operator
-          // has an alternate memory allocation, subsequent uses cannot use the
-          // same alternate memory allocation in order not to clobber data. So
-          // we force default memory allocation for these subsequent uses.
-          const HloUse& previous_use = allocation_value.uses().at(use_idx - 1);
-          if (previous_use.instruction->opcode() == HloOpcode::kConditional &&
-              previous_use.instruction != use.instruction) {
-            allow_no_copy_alternate_mem_allocation = false;
-            earliest_prefetch_time =
-                instruction_schedule.at(previous_use.instruction);
-            VLOG(3) << "Previous use (" << previous_use.ToString()
-                    << ") of use (" << use.ToString()
-                    << ") is a conditional, so this use will need to evict. "
-                    << "Earliest prefetch time = " << *earliest_prefetch_time;
-          }
-        }
-
-        // Bitcasts don't define buffers and don't directly consume buffers.
-        // Skip allocating buffers for bitcast uses. The uses that feed from
-        // bitcasts will be handled specially.
-        if (use.instruction->opcode() != HloOpcode::kBitcast) {
-          AllocationRequest request;
-          // Rarely, (e.g., when conditional true and false parameters are the
-          // same), definition time can be the time of the conditional and use
-          // time is the parameter use, which is less.
-          request.start_time = std::min(definition_time, use_time);
-          request.end_time = use_time;
-          request.latest_prefetch_time = latest_prefetch_time;
-          request.size = interval.size;
-          request.allow_no_copy_alternate_mem_allocation =
-              allow_no_copy_alternate_mem_allocation;
-          request.earliest_prefetch_time = earliest_prefetch_time;
-          request.preferred_offset = preferred_offset;
-          request.use = use;
-          request.allocation_value = &allocation_value;
-          if (!FindAllocation(request)) {
-            // If the allocation finding failed (e.g., due to running out of
-            // asynchronous copies), then fall back to allocating the buffer
-            // entirely in the default memory.
-            UncommitPendingChunks();
-            allocation_success = false;
-            break;
-          }
-
-          // If there are multiple uses, they can try using the memory
-          // allocation already at the alternate memory.
-          definition_time = instruction_schedule.at(use.instruction);
-        }
-
-        // If the use has been a sequential call (e.g. a while loop), the other
-        // colocated intervals must alias with this allocation.
-        if (is_sequential_call) {
-          MemorySpaceAssignment::Allocation* aliased_allocation =
-              GetLiveAllocationAt(*allocation_value.allocation_sequence(),
-                                  use_time);
-          AddAliasedRequiredAssignmentsForSequentialCall(use,
-                                                         aliased_allocation);
-          // Remember the preferred offset to be used inside while loop body
-          // computations.
-          if (aliased_allocation->memory_space() == MemorySpace::kAlternate &&
-              use.instruction->opcode() == HloOpcode::kWhile) {
-            preferred_offset_for_computation[use.instruction->while_body()] =
-                aliased_allocation->chunk().offset;
-          }
-        }
-      }
-      if (!allocation_success) {
+    // Retry allocating this value with larger limits if allocation fails.
+    for (int retry_number = 0; retry_number < options_.max_retries;
+         retry_number++) {
+      final_retry_ = (retry_number == options_.max_retries - 1);
+      options_.prefetch_interval_picker->SetRetryNumber(retry_number);
+      bool success = AllocateColocatedIntervals(colocated_intervals);
+      if (success) {
         break;
       }
+      VLOG(2) << "Couldn't allocate. Retry number " << retry_number;
     }
-    if (allocation_success) {
-      for (AllocationValue& allocation_value : allocation_values) {
-        for (auto& allocation : *allocation_value.allocation_sequence()) {
-          AppendAllocationInfoDebugString(interval, *allocation,
-                                          &allocation_info_str);
-          allocations_->push_back(std::move(allocation));
-        }
-      }
-    }
-
-    pending_chunks_.clear();
-    pending_async_copies_.clear();
   }
 
   VLOG(3) << "Debug buffer info: ";
-  VLOG(3) << buffer_info_str;
+  VLOG(3) << buffer_info_str_;
   VLOG(3) << "Debug allocation info: ";
-  VLOG(3) << allocation_info_str;
-  DumpIfEnabled(buffer_info_str, allocation_info_str);
+  VLOG(3) << allocation_info_str_;
+  DumpDebugStringsIfEnabled();
 
   return result_;
 }
 
+bool AlternateMemoryBestFitHeap::AllocateColocatedIntervals(
+    const std::vector<const AlternateMemoryBestFitHeap::BufferInterval*>&
+        colocated_intervals) {
+  // TODO(berkin): For now, place the phi values due to conditionals in
+  // default memory.
+  for (const BufferInterval* colocated_interval : colocated_intervals) {
+    const HloValue* value = colocated_interval->buffer;
+    for (const auto& position : value->positions()) {
+      if (position.instruction->opcode() == HloOpcode::kConditional) {
+        VLOG(3) << "Adding required assignment for condition output: "
+                << value->ToShortString();
+        AddRequiredAssignment(position.instruction, position.index,
+                              MemorySpace::kDefault);
+        for (const HloComputation* called_computation :
+             position.instruction->called_computations()) {
+          AddRequiredAssignment(called_computation->root_instruction(),
+                                position.index, MemorySpace::kDefault);
+        }
+      }
+    }
+  }
+
+  // Create AllocationValues for all the colocated intervals.
+  std::vector<AllocationValue> allocation_values;
+  for (const auto& colocated_interval : colocated_intervals) {
+    CreateAllocationValues(colocated_interval->buffer, &allocation_values);
+  }
+  const auto& instruction_schedule = hlo_live_range_.instruction_schedule();
+
+  // Data structure to contain the preferred offset for a given computation.
+  // We ensure that the same offset will be allocated outside the while loop
+  // as well as inside the while loop.
+  absl::flat_hash_map<const HloComputation*, int64>
+      preferred_offset_for_computation;
+
+  bool allocation_success = true;
+  for (auto& allocation_value : allocation_values) {
+    int64 definition_time =
+        instruction_schedule.at(allocation_value.defining_instruction());
+
+    absl::optional<int64> preferred_offset;
+    auto preferred_offset_it =
+        preferred_offset_for_computation.find(allocation_value.computation());
+    if (preferred_offset_it != preferred_offset_for_computation.end()) {
+      preferred_offset = preferred_offset_it->second;
+    }
+
+    // Iterate over the uses.
+    for (int use_idx = 0; use_idx < allocation_value.uses().size(); ++use_idx) {
+      const HloUse& use = allocation_value.uses().at(use_idx);
+      int64 use_time = instruction_schedule.at(use.instruction);
+      int64 latest_prefetch_time = use_time;
+      bool allow_no_copy_alternate_mem_allocation = true;
+      absl::optional<int64> earliest_prefetch_time = absl::nullopt;
+
+      // Sequential calls include kWhile, kCall, and kConditional opcodes.
+      bool is_sequential_call =
+          (GetInstructionCallContext(use.instruction->opcode()) ==
+           CallContext::kSequential);
+      if (is_sequential_call) {
+        for (const HloComputation* called_computation :
+             use.instruction->called_computations()) {
+          const HloLiveRange::TimeBound& computation_span =
+              hlo_live_range_.computation_span_times().at(called_computation);
+          latest_prefetch_time =
+              std::min(computation_span.start, latest_prefetch_time);
+        }
+        if (use.instruction->opcode() == HloOpcode::kWhile) {
+          // Given an example while loop and flattened schedule (logical times
+          // shown on the left):
+          //
+          // 0:  a = ...
+          // 1:  ...
+          //     cond {
+          // 2:   p = param(0)
+          // 3:   ...
+          //     }
+          //     body {
+          // 4:   p = param(0)
+          // 5:   ...
+          // 6:   ROOT ...
+          //     }
+          // 7:  w = while(a), body=body, cond=cond
+          //
+          // When processing "a" (time 0) and its while use (time 7), we update
+          // the interval to time 0-4. This is so that the remaining interval
+          // (5-6) can be allocated separately and this buffer doesn't waste
+          // alternate memory space within the while loop body.
+          HloComputation* while_body = use.instruction->while_body();
+          // We require while body ROOTs to be the last in the schedule.
+          CHECK_EQ(instruction_schedule.at(while_body->root_instruction()) + 1,
+                   instruction_schedule.at(use.instruction))
+              << "While body ROOTs need to be the last in the schedule!  "
+                 "Please run RootInstructionSinker.";
+          // Replace the use time with the parameter time so that we can decide
+          // on alternate memory allocations within the while loop body when we
+          // look at uses within the while loop body.
+          use_time =
+              instruction_schedule.at(while_body->parameter_instruction(0));
+        } else if (use.instruction->opcode() == HloOpcode::kConditional) {
+          // Replace the use time with the earliest parameter of called
+          // computations.
+          for (const HloComputation* called_computation :
+               use.instruction->called_computations()) {
+            use_time = std::min(
+                use_time, instruction_schedule.at(
+                              called_computation->parameter_instruction(0)));
+          }
+        }
+      }
+
+      // Add a required assignment in default memory if the use not allowed in
+      // alternate memory.
+      if (!IsUseAllowedInAlternateMemory(allocation_value, use)) {
+        AddRequiredAssignment(allocation_value.value(), use.instruction,
+                              MemorySpace::kDefault, use_time);
+      } else if (use_idx > 0) {
+        // We allow buffers in alternate memory that are passed into
+        // conditionals to give up their alternate memory allocation inside the
+        // called computation. This means that if a conditional operator has an
+        // alternate memory allocation, subsequent uses cannot use the same
+        // alternate memory allocation in order not to clobber data. So we force
+        // default memory allocation for these subsequent uses.
+        const HloUse& previous_use = allocation_value.uses().at(use_idx - 1);
+        if (previous_use.instruction->opcode() == HloOpcode::kConditional &&
+            previous_use.instruction != use.instruction) {
+          allow_no_copy_alternate_mem_allocation = false;
+          earliest_prefetch_time =
+              instruction_schedule.at(previous_use.instruction);
+          VLOG(3) << "Previous use (" << previous_use.ToString() << ") of use ("
+                  << use.ToString()
+                  << ") is a conditional, so this use will need to evict. "
+                  << "Earliest prefetch time = " << *earliest_prefetch_time;
+        }
+      }
+
+      // Bitcasts don't define buffers and don't directly consume buffers.  Skip
+      // allocating buffers for bitcast uses. The uses that feed from bitcasts
+      // will be handled specially.
+      if (use.instruction->opcode() != HloOpcode::kBitcast) {
+        AllocationRequest request;
+        // Rarely, (e.g., when conditional true and false parameters are the
+        // same), definition time can be the time of the conditional and use
+        // time is the parameter use, which is less.
+        request.start_time = std::min(definition_time, use_time);
+        request.end_time = use_time;
+        request.latest_prefetch_time = latest_prefetch_time;
+        request.size = colocated_intervals[0]->size;
+        request.allow_no_copy_alternate_mem_allocation =
+            allow_no_copy_alternate_mem_allocation;
+        request.earliest_prefetch_time = earliest_prefetch_time;
+        request.preferred_offset = preferred_offset;
+        request.use = use;
+        request.allocation_value = &allocation_value;
+        if (!AllocateSegment(request)) {
+          // If the allocation finding failed (e.g., due to running out of
+          // asynchronous copies), then fall back to allocating the buffer
+          // entirely in the default memory.
+          UncommitPendingChunks();
+          allocation_success = false;
+          break;
+        }
+
+        // If there are multiple uses, they can try using the memory allocation
+        // already at the alternate memory.
+        definition_time = instruction_schedule.at(use.instruction);
+      }
+
+      // If the use has been a sequential call (e.g. a while loop), the other
+      // colocated intervals must alias with this allocation.
+      if (is_sequential_call) {
+        MemorySpaceAssignment::Allocation* aliased_allocation =
+            GetLiveAllocationAt(*allocation_value.allocation_sequence(),
+                                use_time);
+        AddAliasedRequiredAssignmentsForSequentialCall(use, aliased_allocation);
+        // Remember the preferred offset to be used inside while loop body
+        // computations.
+        if (aliased_allocation->memory_space() == MemorySpace::kAlternate &&
+            use.instruction->opcode() == HloOpcode::kWhile) {
+          preferred_offset_for_computation[use.instruction->while_body()] =
+              aliased_allocation->chunk().offset;
+        }
+      }
+    }
+    if (!allocation_success) {
+      break;
+    }
+  }
+  if (allocation_success) {
+    for (AllocationValue& allocation_value : allocation_values) {
+      for (auto& allocation : *allocation_value.allocation_sequence()) {
+        AppendAllocationInfoDebugString(*colocated_intervals[0], *allocation,
+                                        &allocation_info_str_);
+        allocations_->push_back(std::move(allocation));
+      }
+    }
+  }
+
+  pending_chunks_.clear();
+  pending_async_copies_.clear();
+  pending_required_assignments_.clear();
+  return allocation_success;
+}
+
 bool operator<(const AsynchronousCopy& a, const AsynchronousCopy& b) {
   return (a.start_time < b.start_time && a.end_time <= b.end_time) ||
          (a.start_time <= b.start_time && a.end_time < b.end_time);
@@ -1135,6 +1209,7 @@ void AlternateMemoryBestFitHeap::AllocateCrossProgramPrefetchBuffer(
 
   pending_chunks_.clear();
   pending_async_copies_.clear();
+  pending_required_assignments_.clear();
 }
 
 void AlternateMemoryBestFitHeap::AddAliasedRequiredAssignmentsForSequentialCall(
@@ -1197,7 +1272,9 @@ void AlternateMemoryBestFitHeap::AddRequiredAssignment(
     VLOG(3) << "Adding required assignment: " << value->ToShortString()
             << " at " << time << " at "
             << (memory_space == MemorySpace::kDefault ? "def" : "alt");
-    required_assignments_[value].push_back({memory_space, time, chunk});
+    RequiredMemoryAssignment required_assignment{memory_space, time, chunk};
+    required_assignments_[value].push_back(required_assignment);
+    pending_required_assignments_.push_back({value, required_assignment});
   }
 }
 
@@ -1316,8 +1393,30 @@ void AlternateMemoryBestFitHeap::UncommitPendingChunks() {
                                      kDummyChunk);
     }
   }
+  for (const auto& value_and_required_assignment :
+       pending_required_assignments_) {
+    auto& required_assignment_vector =
+        required_assignments_[value_and_required_assignment.first];
+    const RequiredMemoryAssignment& required_assignment =
+        value_and_required_assignment.second;
+    VLOG(3) << "Removing required assignment: "
+            << (required_assignment.memory_space == MemorySpace::kDefault
+                    ? "def"
+                    : "alt")
+            << " time = " << required_assignment.time << " off = "
+            << (required_assignment.chunk ? required_assignment.chunk->offset
+                                          : -1);
+    for (auto it = required_assignment_vector.begin();
+         it != required_assignment_vector.end(); ++it) {
+      if (*it == value_and_required_assignment.second) {
+        required_assignment_vector.erase(it);
+        break;
+      }
+    }
+  }
   pending_chunks_.clear();
   pending_async_copies_.clear();
+  pending_required_assignments_.clear();
 }
 
 void AlternateMemoryBestFitHeap::AddToPendingChunks(
@@ -1348,7 +1447,7 @@ AlternateMemoryBestFitHeap::RequiredMemoryAssignmentAt(const HloValue* buffer,
   return required_assignment_at_time;
 }
 
-bool AlternateMemoryBestFitHeap::FindAllocation(
+bool AlternateMemoryBestFitHeap::AllocateSegment(
     const AllocationRequest& request) {
   auto allocation_sequence = request.allocation_value->allocation_sequence();
   // start_time == end_time is a special case where the value is consumed
@@ -1462,6 +1561,12 @@ bool AlternateMemoryBestFitHeap::FindAllocation(
   if (Prefetch(request, **prev_allocation_in_default_mem_it)) {
     return true;
   }
+  if (!final_retry_ && prefetch_failed_due_to_async_copy_) {
+    // If prefetching failed due to asynchronous copy and we're not in our final
+    // try, return false (failure) so that we can retry this interval with
+    // larger limits.
+    return false;
+  }
 
   // If the end assignment was required to be in alternate memory but that
   // wasn't possible, then this allocation is invalid.
@@ -1773,6 +1878,10 @@ bool AlternateMemoryBestFitHeap::Prefetch(
   BufferInterval alternate_mem_interval;
   alternate_mem_interval.buffer = request.allocation_value->value();
   alternate_mem_interval.size = request.size;
+  // If any of the prefetch intervals couldn't be used due to number of
+  // outstanding async copy limit or async copy ordering, set
+  // prefetch_failed_due_to_async_copy_.
+  prefetch_failed_due_to_async_copy_ = false;
   while (!options_.prefetch_interval_picker->Done()) {
     alternate_mem_interval.start = options_.prefetch_interval_picker->Next();
     CHECK_LT(alternate_mem_interval.start, request.latest_prefetch_time);
@@ -1780,15 +1889,17 @@ bool AlternateMemoryBestFitHeap::Prefetch(
             << alternate_mem_interval.start << ", " << request.end_time << ")";
     // If this additional asynchronous copy would violate the limit, try a
     // different interval.
+    if (ViolatesAsyncCopyOrdering(alternate_mem_interval.start,
+                                  request.latest_prefetch_time)) {
+      VLOG(4) << "This would violate asynchronous copy ordering.";
+      prefetch_failed_due_to_async_copy_ = true;
+      continue;
+    }
     if (ViolatesMaximumOutstandingAsyncCopies(alternate_mem_interval.start,
                                               request.latest_prefetch_time,
                                               /*is_prefetch=*/true)) {
       VLOG(4) << "This would violate the outstanding async copy limit.";
-      continue;
-    }
-    if (ViolatesAsyncCopyOrdering(alternate_mem_interval.start,
-                                  request.latest_prefetch_time)) {
-      VLOG(4) << "This would violate asynchronous copy ordering.";
+      prefetch_failed_due_to_async_copy_ = true;
       continue;
     }
 
@@ -1812,6 +1923,7 @@ bool AlternateMemoryBestFitHeap::Prefetch(
 
       request.allocation_value->allocation_sequence()->back()->AddUse(
           request.use);
+      prefetch_failed_due_to_async_copy_ = false;
       return true;
     }
   }
@@ -1890,10 +2002,12 @@ MemorySpaceAssignment::CalculateAsyncCopyStats() const {
 
 /*static*/ MemorySpaceAssignment::BufferIntervalCompare
 MemorySpaceAssignment::GetMemoryBoundednessBufferIntervalCompare(
-    const MemorySpaceAssignmentCostAnalysis& cost_analysis) {
-  return [&](const BufferInterval& x, const BufferInterval& y) {
-    float x_memory_boundedness = cost_analysis.GetMemoryBoundedness(x);
-    float y_memory_boundedness = cost_analysis.GetMemoryBoundedness(y);
+    const MemorySpaceAssignmentCostAnalysis& cost_analysis,
+    MemorySpaceAssignmentCostAnalysis::Cache* cache) {
+  return [cost_analysis, cache](const BufferInterval& x,
+                                const BufferInterval& y) {
+    float x_memory_boundedness = cost_analysis.GetMemoryBoundedness(x, cache);
+    float y_memory_boundedness = cost_analysis.GetMemoryBoundedness(y, cache);
     if (x_memory_boundedness != y_memory_boundedness) {
       return x_memory_boundedness > y_memory_boundedness;
     }
diff --git a/tensorflow/compiler/xla/service/memory_space_assignment.h b/tensorflow/compiler/xla/service/memory_space_assignment.h
index 3f59abfd28e..c75457dd48e 100644
--- a/tensorflow/compiler/xla/service/memory_space_assignment.h
+++ b/tensorflow/compiler/xla/service/memory_space_assignment.h
@@ -78,6 +78,12 @@ class PresetAssignments {
 // bandwidths of different memory spaces.
 class MemorySpaceAssignmentCostAnalysis {
  public:
+  // An optional Cache object may be provided to some of the methods below to
+  // speed up the lookup.
+  struct Cache {
+    absl::flat_hash_map<const HloInstruction*, float> while_nest_multiplier;
+  };
+
   MemorySpaceAssignmentCostAnalysis(
       const HloCostAnalysis& cost_analysis,
       float async_copy_bandwidth_bytes_per_second,
@@ -97,15 +103,16 @@ class MemorySpaceAssignmentCostAnalysis {
   // alternate memory would help if the op is memory bound, or otherwise how far
   // off is the op to memory boundedness. The larger this number, the higher
   // priority it will be placed in the alternate memory.
-  float GetAlternateMemoryBenefit(
-      const HloInstruction& instruction,
-      float elapsed_time_due_to_alternate_mem) const;
+  float GetAlternateMemoryBenefit(const HloInstruction& instruction,
+                                  float elapsed_time_due_to_alternate_mem,
+                                  Cache* cache = nullptr) const;
 
   // Returns a heuristic value of memory boundedness for the given
   // BufferInterval.  The larger this number, the higher priority it will be
   // placed in the alternate memory.
   float GetMemoryBoundedness(
-      const GlobalDecreasingSizeBestFitHeap::BufferInterval& interval) const;
+      const GlobalDecreasingSizeBestFitHeap::BufferInterval& interval,
+      Cache* cache = nullptr) const;
 
   // Returns the elapsed time in seconds due to compute only.
   float GetInstructionElapsedDueToCompute(
@@ -183,6 +190,10 @@ class PrefetchIntervalPicker {
   // Returns true if the available prefetch intervals have been exhausted.
   virtual bool Done() const = 0;
 
+  // The retry number can be used to modify the interval picking policies. The
+  // first attempt will have a retry_number of 0, then 1, etc.
+  virtual void SetRetryNumber(int retry_number) {}
+
   // Returns a debug string for the current state of the prefetch interval
   // picker.
   virtual std::string ToDebugString() const = 0;
@@ -269,6 +280,8 @@ class CostAnalysisPrefetchIntervalPicker : public PrefetchIntervalPicker {
   int64 Next() override;
   bool Done() const override;
 
+  void SetRetryNumber(int retry_number) override;
+
   std::string ToDebugString() const override;
   std::string ToNoCopyDebugString(const Shape& shape, int64 start_time,
                                   int64 end_time) const override;
@@ -282,14 +295,22 @@ class CostAnalysisPrefetchIntervalPicker : public PrefetchIntervalPicker {
   // corresponds to the instruction schedule.
   float GetLogicalIntervalElapsed(int64 start_time, int64 end_time) const;
 
+  // Finds the minimum nest level in the given interval.
+  int GetMinWhileNestLevel(int64 start_time, int64 end_time) const;
+
   // For each instruction in the flattened schedule, maintain their elapsed time
-  // and while nesting level.
-  std::vector<float> elapsed_time_;
+  // (in cumulative sum) and while nesting level.
+  std::vector<float> elapsed_time_cumsum_;
   std::vector<int> while_nest_level_;
+  // Maintain the index of the most recent (before this instruction) nest level
+  // change in order to efficiently determine the minimum nest level in an
+  // interval.
+  std::vector<int> while_nest_level_change_;
 
   const MemorySpaceAssignmentCostAnalysis& cost_analysis_;
   float min_async_copy_to_overlap_ratio_;
   float max_async_copy_to_overlap_ratio_;
+  float max_overlap_multiplier_ = 1.0;
 
   float async_copy_elapsed_;
   float inst_elapsed_reduction_;
@@ -348,6 +369,11 @@ class MemorySpaceAssignment {
     int64 max_outstanding_prefetches = -1;
     int64 max_outstanding_evictions = -1;
 
+    // Specifies the maximum number of retries that will be performed for each
+    // value in case prefetching failed due to running out of asynchronous
+    // copies or asynchronous copy ordering.
+    int64 max_retries = 1;
+
     // If true, tries allocating buffers across (e.g., before and inside a while
     // loop body) sequential calls (kWhile, kCall, and kConditional).
     bool allocate_across_sequential_calls = false;
@@ -645,7 +671,8 @@ class MemorySpaceAssignment {
   StatusOr<AsyncCopyStats> CalculateAsyncCopyStats() const;
 
   static BufferIntervalCompare GetMemoryBoundednessBufferIntervalCompare(
-      const MemorySpaceAssignmentCostAnalysis& cost_analysis);
+      const MemorySpaceAssignmentCostAnalysis& cost_analysis,
+      MemorySpaceAssignmentCostAnalysis::Cache* cache = nullptr);
 
   // Verify that the memory space assignment is free of overlapping buffers and
   // export heap simulator trace to be used by buffer_assignment.
@@ -741,6 +768,11 @@ struct RequiredMemoryAssignment {
   MemorySpaceAssignment::MemorySpace memory_space;
   int64 time;
   absl::optional<HeapSimulator::Chunk> chunk;
+
+  bool operator==(const RequiredMemoryAssignment& other) const {
+    return memory_space == other.memory_space && time == other.time &&
+           chunk == other.chunk;
+  }
 };
 
 // A struct representing an asynchronous copy with its logical start and end
@@ -875,7 +907,15 @@ class AlternateMemoryBestFitHeap : public GlobalDecreasingSizeBestFitHeap {
   void CreateAllocationValues(const HloValue* value,
                               std::vector<AllocationValue>* allocation_values);
 
-  // Finds an allocation for the given interval.
+  // Finds allocations for colocated intervals. Colocated intervals consist of
+  // one or more BufferIntervals, each with a different HloValue. All of the
+  // intervals within colocated intervals have a must-alias relationship with
+  // each other. Returns true if allocation succeeded.
+  bool AllocateColocatedIntervals(
+      const std::vector<const BufferInterval*>& colocated_intervals);
+
+  // Finds an allocation for an allocation request for a segment (see the
+  // documentation for AllocationRequest above how a segment is defined).
   //
   // It performs three things in the following order:
   //  1- Allocate the allocation request entirely in the alternate memory, if
@@ -889,7 +929,7 @@ class AlternateMemoryBestFitHeap : public GlobalDecreasingSizeBestFitHeap {
   // false. This means we could not find a suitable allocation, so all previous
   // allocations for this buffer must be removed and allocated in the default
   // memory. Otherwise, this method returns true.
-  bool FindAllocation(const AllocationRequest& request);
+  bool AllocateSegment(const AllocationRequest& request);
 
   // Try allocating in alternate memory without any copies. Returns true if
   // successful.
@@ -985,8 +1025,7 @@ class AlternateMemoryBestFitHeap : public GlobalDecreasingSizeBestFitHeap {
       const BufferInterval& interval,
       const MemorySpaceAssignment::Allocation& allocation,
       std::string* debug_str) const;
-  void DumpIfEnabled(absl::string_view buffer_info_str,
-                     absl::string_view allocation_info_str) const;
+  void DumpDebugStringsIfEnabled() const;
 
   // Returns the available heap size in the alternate memory.
   int64 available_heap_size() const {
@@ -1004,12 +1043,20 @@ class AlternateMemoryBestFitHeap : public GlobalDecreasingSizeBestFitHeap {
   AsynchronousCopyOrdering async_copy_ordering_;
   std::vector<std::pair<BufferInterval, ChunkCandidate>> pending_chunks_;
   std::vector<AsynchronousCopy> pending_async_copies_;
+  std::vector<std::pair<const HloValue*, RequiredMemoryAssignment>>
+      pending_required_assignments_;
   // This map contains required memory assignments for HloValues (e.g., input
   // and outputs).
   absl::flat_hash_map<const HloValue*, std::vector<RequiredMemoryAssignment>>
       required_assignments_;
   // Number of bytes reserved in alternate memory space.
   int64 reserved_in_bytes_ = 0;
+  // Variables to control allocation retries.
+  bool final_retry_;
+  bool prefetch_failed_due_to_async_copy_;
+  // Debug strings.
+  std::string buffer_info_str_;
+  std::string allocation_info_str_;
 };
 
 }  // namespace xla
diff --git a/tensorflow/compiler/xla/service/memory_space_assignment_test.cc b/tensorflow/compiler/xla/service/memory_space_assignment_test.cc
index 23b311730f8..9c6b42cac91 100644
--- a/tensorflow/compiler/xla/service/memory_space_assignment_test.cc
+++ b/tensorflow/compiler/xla/service/memory_space_assignment_test.cc
@@ -68,7 +68,7 @@ class MemorySpaceAssignmentTest : public HloTestBase,
     return AssignMemorySpace(
         module, /*max_outstanding_async_copies=*/-1,
         MemorySpaceAssignment::GetMemoryBoundednessBufferIntervalCompare(
-            cost_analysis),
+            cost_analysis, &cache_),
         &prefetch_interval_picker);
   }
 
@@ -285,6 +285,8 @@ class MemorySpaceAssignmentTest : public HloTestBase,
     TF_CHECK_OK(module->set_schedule(schedule));
     return module;
   }
+
+  MemorySpaceAssignmentCostAnalysis::Cache cache_;
 };
 
 TEST_P(MemorySpaceAssignmentTest, ParameterOnly) {
diff --git a/tensorflow/compiler/xla/service/mlir_gpu/BUILD b/tensorflow/compiler/xla/service/mlir_gpu/BUILD
index 07655a61074..633fef673fd 100644
--- a/tensorflow/compiler/xla/service/mlir_gpu/BUILD
+++ b/tensorflow/compiler/xla/service/mlir_gpu/BUILD
@@ -66,7 +66,7 @@ cc_library(
         "//tensorflow/compiler/xla/service:compiler",
         "//tensorflow/compiler/xla/service/gpu:target_constants",
         "//tensorflow/core:stream_executor_no_cuda",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//mlir:IR",
         "@llvm-project//mlir:LLVMDialect",
     ],
diff --git a/tensorflow/compiler/xla/tests/BUILD b/tensorflow/compiler/xla/tests/BUILD
index c8a242c156a..f275bfd12ca 100644
--- a/tensorflow/compiler/xla/tests/BUILD
+++ b/tensorflow/compiler/xla/tests/BUILD
@@ -2143,7 +2143,7 @@ tf_cc_test(
         "//tensorflow/core:test_main",
         "//tensorflow/stream_executor",
         "@com_google_absl//absl/memory",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
     ],
 )
 
diff --git a/tensorflow/compiler/xla/tests/local_client_execute_test.cc b/tensorflow/compiler/xla/tests/local_client_execute_test.cc
index 7d804e18ed3..fab1a53611f 100644
--- a/tensorflow/compiler/xla/tests/local_client_execute_test.cc
+++ b/tensorflow/compiler/xla/tests/local_client_execute_test.cc
@@ -806,6 +806,29 @@ XLA_TEST_F(LocalClientExecuteTest, CompilePartitionedExecutable) {
   EXPECT_EQ(2, executables.size());
 }
 
+XLA_TEST_F(LocalClientExecuteTest,
+           DISABLED_ON_INTERPRETER(SizeOfGeneratedCodeInBytes)) {
+  XlaBuilder builder(TestName());
+  auto x = Parameter(&builder, 0, ShapeUtil::MakeShape(F32, {}), "x");
+  constexpr int size = 100000;
+  TF_ASSERT_OK_AND_ASSIGN(auto literal,
+                          LiteralUtil::CreateRandomLiteral<F32>(
+                              ShapeUtil::MakeShape(F32, {size}), 0.0, 1.0));
+  auto y = ConstantLiteral(&builder, literal);
+  Add(x, y);
+
+  Shape argument_layout =
+      ShapeUtil::MakeShapeWithLayout(F32, /*dimensions=*/{}, {});
+  TF_ASSERT_OK_AND_ASSIGN(
+      auto executables,
+      local_client_->Compile(builder.Build().ValueOrDie(), {&argument_layout},
+                             ExecutableBuildOptions()));
+  EXPECT_EQ(1, executables.size());
+  // The executable should be at least as large as the constant it contains.
+  EXPECT_GT(executables.front()->executable()->SizeOfGeneratedCodeInBytes(),
+            int64{sizeof(float) * size});
+}
+
 XLA_TEST_F(LocalClientExecuteTest, ShapeBufferToLiteralConversion) {
   // Test copying Literals to the device as ShapedBuffers, then copying them
   // back again to Literals.
diff --git a/tensorflow/core/api_def/base_api/api_def_FakeQuantWithMinMaxArgs.pbtxt b/tensorflow/core/api_def/base_api/api_def_FakeQuantWithMinMaxArgs.pbtxt
index aea945e4252..e242bd0fc3f 100644
--- a/tensorflow/core/api_def/base_api/api_def_FakeQuantWithMinMaxArgs.pbtxt
+++ b/tensorflow/core/api_def/base_api/api_def_FakeQuantWithMinMaxArgs.pbtxt
@@ -2,21 +2,26 @@ op {
   graph_op_name: "FakeQuantWithMinMaxArgs"
   summary: "Fake-quantize the \'inputs\' tensor, type float to \'outputs\' tensor of same type."
   description: <<END
-Attributes `[min; max]` define the clamping range for the `inputs` data.
-`inputs` values are quantized into the quantization range (`[0; 2^num_bits - 1]`
-when `narrow_range` is false and `[1; 2^num_bits - 1]` when it is true) and
-then de-quantized and output as floats in `[min; max]` interval.
-`num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
+Attributes
+
+*   `[min; max]` define the clamping range for the `inputs` data.
+*   `inputs` values are quantized into the quantization range (
+`[0; 2^num_bits - 1]` when `narrow_range` is false and `[1; 2^num_bits - 1]`
+when it is true) and then de-quantized and output as floats in `[min; max]`
+interval.
+*   `num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
 
 Before quantization, `min` and `max` values are adjusted with the following
 logic.
 It is suggested to have `min <= 0 <= max`. If `0` is not in the range of values,
 the behavior can be unexpected:
-If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
-If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
-If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
+
+*   If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
+*   If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
+*   If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
 `min_adj = scale * round(min / scale)` and `max_adj = max + min_adj - min`.
 
 Quantization is called fake since the output is still in floating point.
 END
 }
+
diff --git a/tensorflow/core/api_def/base_api/api_def_FakeQuantWithMinMaxVars.pbtxt b/tensorflow/core/api_def/base_api/api_def_FakeQuantWithMinMaxVars.pbtxt
index dda0755f239..9b4a7eb0ae1 100644
--- a/tensorflow/core/api_def/base_api/api_def_FakeQuantWithMinMaxVars.pbtxt
+++ b/tensorflow/core/api_def/base_api/api_def_FakeQuantWithMinMaxVars.pbtxt
@@ -1,22 +1,27 @@
 op {
   graph_op_name: "FakeQuantWithMinMaxVars"
-  summary: "Fake-quantize the \'inputs\' tensor of type float via global float scalars `min`"
+  summary: "Fake-quantize the \'inputs\' tensor of type float via global float scalars"
   description: <<END
-and `max` to 'outputs' tensor of same shape as `inputs`.
+Fake-quantize the `inputs` tensor of type float via global float scalars
+`min` and `max` to `outputs` tensor of same shape as `inputs`.
 
-`[min; max]` define the clamping range for the `inputs` data.
-`inputs` values are quantized into the quantization range (`[0; 2^num_bits - 1]`
-when `narrow_range` is false and `[1; 2^num_bits - 1]` when it is true) and
-then de-quantized and output as floats in `[min; max]` interval.
-`num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
+Attributes
+
+*   `[min; max]` define the clamping range for the `inputs` data.
+*   `inputs` values are quantized into the quantization range (
+`[0; 2^num_bits - 1]` when `narrow_range` is false and `[1; 2^num_bits - 1]`
+when it is true) and then de-quantized and output as floats in `[min; max]`
+interval.
+*   `num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
 
 Before quantization, `min` and `max` values are adjusted with the following
 logic.
 It is suggested to have `min <= 0 <= max`. If `0` is not in the range of values,
 the behavior can be unexpected:
-If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
-If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
-If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
+
+*   If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
+*   If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
+*   If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
 `min_adj = scale * round(min / scale)` and `max_adj = max + min_adj - min`.
 
 This operation has a gradient and thus allows for training `min` and `max`
diff --git a/tensorflow/core/api_def/base_api/api_def_FakeQuantWithMinMaxVarsPerChannel.pbtxt b/tensorflow/core/api_def/base_api/api_def_FakeQuantWithMinMaxVarsPerChannel.pbtxt
index 88bfdbbc076..52c050f6f29 100644
--- a/tensorflow/core/api_def/base_api/api_def_FakeQuantWithMinMaxVarsPerChannel.pbtxt
+++ b/tensorflow/core/api_def/base_api/api_def_FakeQuantWithMinMaxVarsPerChannel.pbtxt
@@ -1,23 +1,28 @@
 op {
   graph_op_name: "FakeQuantWithMinMaxVarsPerChannel"
-  summary: "Fake-quantize the \'inputs\' tensor of type float and one of the shapes: `[d]`,"
+  summary: "Fake-quantize the \'inputs\' tensor of type float via per-channel floats"
   description: <<END
-`[b, d]` `[b, h, w, d]` via per-channel floats `min` and `max` of shape `[d]`
-to 'outputs' tensor of same shape as `inputs`.
+Fake-quantize the `inputs` tensor of type float per-channel and one of the
+shapes: `[d]`, `[b, d]` `[b, h, w, d]` via per-channel floats `min` and `max`
+of shape `[d]` to `outputs` tensor of same shape as `inputs`.
 
-`[min; max]` define the clamping range for the `inputs` data.
-`inputs` values are quantized into the quantization range (`[0; 2^num_bits - 1]`
-when `narrow_range` is false and `[1; 2^num_bits - 1]` when it is true) and
-then de-quantized and output as floats in `[min; max]` interval.
-`num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
+Attributes
+
+*   `[min; max]` define the clamping range for the `inputs` data.
+*   `inputs` values are quantized into the quantization range (
+`[0; 2^num_bits - 1]` when `narrow_range` is false and `[1; 2^num_bits - 1]`
+when it is true) and then de-quantized and output as floats in `[min; max]`
+interval.
+*   `num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
 
 Before quantization, `min` and `max` values are adjusted with the following
 logic.
 It is suggested to have `min <= 0 <= max`. If `0` is not in the range of values,
 the behavior can be unexpected:
-If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
-If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
-If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
+
+*   If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
+*   If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
+*   If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
 `min_adj = scale * round(min / scale)` and `max_adj = max + min_adj - min`.
 
 This operation has a gradient and thus allows for training `min` and `max`
diff --git a/tensorflow/core/api_def/python_api/api_def_Minimum.pbtxt b/tensorflow/core/api_def/python_api/api_def_Minimum.pbtxt
index e7f90893fce..80dcb473bd4 100644
--- a/tensorflow/core/api_def/python_api/api_def_Minimum.pbtxt
+++ b/tensorflow/core/api_def/python_api/api_def_Minimum.pbtxt
@@ -7,11 +7,29 @@ op {
     name: "minimum"
   }
   description: <<END
-Example:
+Both inputs are number-type tensors (except complex).  `minimum` expects that
+both tensors have the same `dtype`.
+
+Examples:
+
 >>> x = tf.constant([0., 0., 0., 0.])
 >>> y = tf.constant([-5., -2., 0., 3.])
 >>> tf.math.minimum(x, y)
 <tf.Tensor: shape=(4,), dtype=float32, numpy=array([-5., -2., 0., 0.], dtype=float32)>
 
+Note that `minimum` supports broadcast semantics.
+
+>>> x = tf.constant([-5., 0., 0., 0.])
+>>> y = tf.constant([-3.])
+>>> tf.math.minimum(x, y)
+<tf.Tensor: shape=(4,), dtype=float32, numpy=array([-5., -3., -3., -3.], dtype=float32)>
+
+If inputs are not tensors, they will be converted to tensors.  See
+`tf.convert_to_tensor`.
+
+>>> x = tf.constant([-3.], dtype=tf.float32)
+>>> tf.math.minimum([-5], x)
+<tf.Tensor: shape=(1,), dtype=float32, numpy=array([-5.], dtype=float32)>
+
 END
 }
diff --git a/tensorflow/core/common_runtime/eager/context.h b/tensorflow/core/common_runtime/eager/context.h
index cceb883a965..e902ede7f15 100644
--- a/tensorflow/core/common_runtime/eager/context.h
+++ b/tensorflow/core/common_runtime/eager/context.h
@@ -275,7 +275,7 @@ class EagerContext : public AbstractContextInterface, public core::RefCounted {
 
   // Add the given `fdef` to the local FunctionLibraryDefinition. And add an
   // entry to the KernelAndDevice cache for it if it's not exist.
-  Status AddFunctionDef(const FunctionDef& fdef);
+  Status AddFunctionDef(const FunctionDef& fdef) override;
   // `library` contains all FunctionDefs and GradientDefs to expand `fdef`. Add
   // it to the local FunctionLibraryDefinition as well, but no need to add it
   // to the KernelAndDevice cache since they won't be executed as
@@ -286,7 +286,7 @@ class EagerContext : public AbstractContextInterface, public core::RefCounted {
 
   const FunctionDef* GetFunctionDef(const string& function_name);
 
-  Status RemoveFunction(const string& func);
+  Status RemoveFunction(const string& func) override;
 
   // Wait for pending nodes to be finished in local executors (including context
   // default executor and thread executors) and executors on remote workers.
diff --git a/tensorflow/core/common_runtime/eager/execute.cc b/tensorflow/core/common_runtime/eager/execute.cc
index e7d75517c15..5be0ad8db57 100644
--- a/tensorflow/core/common_runtime/eager/execute.cc
+++ b/tensorflow/core/common_runtime/eager/execute.cc
@@ -477,10 +477,11 @@ Status GetOrCreateKernelAndDevice(
           SupportedDeviceTypesForNode(*device_type_list, ndef, &supported_devs,
                                       &ctx.HostCPU()->parsed_name()));
       if (supported_devs.empty()) {
-        return errors::NotFound("Could not find valid device for node.\nNode:",
-                                FormatNodeDefForError(ndef),
-                                "\nAll kernels registered for op ", ndef.op(),
-                                " :\n", KernelsRegisteredForOp(ndef.op()));
+        return errors::NotFound(
+            "Could not find device for node: ",
+            errors::FormatNodeNameForError(ndef.name()), " = ", ndef.op(), "[",
+            SummarizeAttrs(ndef), "]", "\nAll kernels registered for op ",
+            ndef.op(), ":\n", KernelsRegisteredForOp(ndef.op()));
       }
       TF_RETURN_IF_ERROR(ctx.SelectDevice(op->GetDeviceParsedName(),
                                           supported_devs, DT_INVALID, &device));
@@ -537,7 +538,8 @@ Status GetOrCreateKernelAndDevice(
           ctx.GetCollectiveExecutorHandle(), ctx.HostCPU()));
     }
 
-    TF_RETURN_IF_ERROR(kernel->Init(ndef, graph_collector));
+    TF_RETURN_IF_ERROR(
+        kernel->Init({ctx.LogDevicePlacement()}, ndef, graph_collector));
 
     if (op->is_function()) {
       ctx.AddKernelToCache(cache_key, kernel.get());
diff --git a/tensorflow/core/common_runtime/eager/kernel_and_device.cc b/tensorflow/core/common_runtime/eager/kernel_and_device.cc
index 1e69f9d1767..48dd6a718e8 100644
--- a/tensorflow/core/common_runtime/eager/kernel_and_device.cc
+++ b/tensorflow/core/common_runtime/eager/kernel_and_device.cc
@@ -99,7 +99,7 @@ KernelAndDeviceFunc::~KernelAndDeviceFunc() {
   }
 }
 
-Status KernelAndDeviceOp::Init(const NodeDef& ndef,
+Status KernelAndDeviceOp::Init(const Context& ctx, const NodeDef& ndef,
                                GraphCollector* graph_collector) {
   OpKernel* k = nullptr;
   if (flr_ == nullptr) {
@@ -131,7 +131,8 @@ Status KernelAndDeviceOp::Init(const NodeDef& ndef,
   return Status::OK();
 }
 
-Status KernelAndDeviceFunc::InstantiateFunc(const NodeDef& ndef,
+Status KernelAndDeviceFunc::InstantiateFunc(const Context& ctx,
+                                            const NodeDef& ndef,
                                             GraphCollector* graph_collector) {
   const OpDef* op_def = nullptr;
   const FunctionDef* function_def;
@@ -209,14 +210,16 @@ Status KernelAndDeviceFunc::InstantiateFunc(const NodeDef& ndef,
       ->mutable_optimizer_options()
       ->set_do_function_inlining(true);
 
+  options.config_proto.set_log_device_placement(ctx.log_device_placement);
+
   TF_RETURN_IF_ERROR(
       pflr_->Instantiate(ndef.op(), AttrSlice(ndef), options, &handle_));
   return pflr_->IsCrossProcess(handle_, &is_cross_process_);
 }
 
-Status KernelAndDeviceFunc::Init(const NodeDef& ndef,
+Status KernelAndDeviceFunc::Init(const Context& ctx, const NodeDef& ndef,
                                  GraphCollector* graph_collector) {
-  TF_RETURN_IF_ERROR(InstantiateFunc(ndef, graph_collector));
+  TF_RETURN_IF_ERROR(InstantiateFunc(ctx, ndef, graph_collector));
   return pflr_->GetOutputDevices(handle_, &output_devices_);
 }
 
diff --git a/tensorflow/core/common_runtime/eager/kernel_and_device.h b/tensorflow/core/common_runtime/eager/kernel_and_device.h
index d2c54322513..87c2d7a5510 100644
--- a/tensorflow/core/common_runtime/eager/kernel_and_device.h
+++ b/tensorflow/core/common_runtime/eager/kernel_and_device.h
@@ -93,11 +93,16 @@ class EagerKernelArgs : public FunctionArgsInterface {
 // https://www.tensorflow.org/code/tensorflow/core/kernels/ops_testutil.h
 class KernelAndDevice : public core::RefCounted {
  public:
+  struct Context {
+    bool log_device_placement = false;
+  };
+
   // Populates this with a kernel appropriate for 'ndef'.
   //
   // The provided FunctionLibraryRuntime MUST outlive all calls to
   // Run() on the returned KernelAndDevice.
-  virtual Status Init(const NodeDef& ndef, GraphCollector* graph_collector) = 0;
+  virtual Status Init(const Context& ctx, const NodeDef& ndef,
+                      GraphCollector* graph_collector) = 0;
 
   // Non-multi-device functions are run using regular CallOp and look like
   // primitive operations from KernelAndDevice perspective.
@@ -194,7 +199,8 @@ class KernelAndDeviceOp final : public KernelAndDevice {
 
   ~KernelAndDeviceOp() override {}
 
-  Status Init(const NodeDef& ndef, GraphCollector* graph_collector) override;
+  Status Init(const Context& ctx, const NodeDef& ndef,
+              GraphCollector* graph_collector) override;
 
   Status Run(ScopedStepContainer* step_container, const EagerKernelArgs& inputs,
              std::vector<Tensor>* outputs,
@@ -282,9 +288,11 @@ class KernelAndDeviceFunc : public KernelAndDevice {
 
   bool IsFunction() override { return true; };
 
-  Status InstantiateFunc(const NodeDef& ndef, GraphCollector* graph_collector);
+  Status InstantiateFunc(const Context& ctx, const NodeDef& ndef,
+                         GraphCollector* graph_collector);
 
-  Status Init(const NodeDef& ndef, GraphCollector* graph_collector) override;
+  Status Init(const Context& ctx, const NodeDef& ndef,
+              GraphCollector* graph_collector) override;
 
   Status Run(ScopedStepContainer* step_container, const EagerKernelArgs& inputs,
              std::vector<Tensor>* outputs,
diff --git a/tensorflow/core/common_runtime/eager/kernel_and_device_test.cc b/tensorflow/core/common_runtime/eager/kernel_and_device_test.cc
index aa097388d11..a7aac4a8f6d 100644
--- a/tensorflow/core/common_runtime/eager/kernel_and_device_test.cc
+++ b/tensorflow/core/common_runtime/eager/kernel_and_device_test.cc
@@ -122,7 +122,7 @@ void BM_KernelAndDeviceInit(int iters) {
                       nullptr, env.cpu_device());
   tensorflow::testing::StartTiming();
   for (int i = 0; i < iters; ++i) {
-    TF_CHECK_OK(k.Init(ndef, nullptr));
+    TF_CHECK_OK(k.Init({}, ndef, nullptr));
   }
 }
 BENCHMARK(BM_KernelAndDeviceInit);
@@ -143,7 +143,7 @@ void BM_KernelAndDeviceRun(int iters) {
   TestEnv env;
   KernelAndDeviceOp k(nullptr, false, env.function_library_runtime(), nullptr,
                       nullptr, env.cpu_device());
-  TF_CHECK_OK(k.Init(ndef, nullptr));
+  TF_CHECK_OK(k.Init({}, ndef, nullptr));
   const EagerKernelArgs args(std::move(inputs));
   tensorflow::testing::StartTiming();
   for (int i = 0; i < iters; ++i) {
diff --git a/tensorflow/core/common_runtime/eager/tensor_handle.cc b/tensorflow/core/common_runtime/eager/tensor_handle.cc
index dbfc5639017..9b82c556cd0 100644
--- a/tensorflow/core/common_runtime/eager/tensor_handle.cc
+++ b/tensorflow/core/common_runtime/eager/tensor_handle.cc
@@ -342,17 +342,11 @@ Status TensorHandle::CreatePackedHandle(std::vector<TensorHandle*>&& handles,
     }
   }
 
-  Device* device;
-  if (devices.size() == 1) {
-    device = absl::get<Device*>(handles.at(0)->DeviceOrHostCPU(*ctx));
-  } else {
-    CompositeDevice* composite_device = nullptr;
-    TF_RETURN_IF_ERROR(
-        ctx->FindOrCreateCompositeDevice(devices, &composite_device));
-    device = composite_device;
-  }
+  CompositeDevice* composite_device = nullptr;
+  TF_RETURN_IF_ERROR(
+      ctx->FindOrCreateCompositeDevice(devices, &composite_device));
   *packed_handle =
-      new TensorHandle(std::move(handles), device, dtype, shape, ctx);
+      new TensorHandle(std::move(handles), composite_device, dtype, shape, ctx);
   (*packed_handle)->SetResourceHandleInfo(std::move(resource_handle_info));
   return Status::OK();
 }
diff --git a/tensorflow/core/common_runtime/eager/tensor_handle_test.cc b/tensorflow/core/common_runtime/eager/tensor_handle_test.cc
index 13b634bbec4..28092c0a604 100644
--- a/tensorflow/core/common_runtime/eager/tensor_handle_test.cc
+++ b/tensorflow/core/common_runtime/eager/tensor_handle_test.cc
@@ -219,6 +219,39 @@ TEST_F(PackedTensorHandleTest, PackedHandle) {
   packed_handle->Unref();
 }
 
+TEST_F(PackedTensorHandleTest, PackedSingleHandle) {
+  tensorflow::DataType dtype = DT_RESOURCE;
+  TensorShape shape = {};
+
+  Tensor t(dtype, shape);
+  Device* d = ListDevices().at(0);
+  TensorHandle* h =
+      TensorHandle::CreateLocalHandle(std::move(t), d, d, d, context());
+  std::vector<TensorHandle*> handles = {h};
+
+  TensorHandle* packed_handle = nullptr;
+  TF_EXPECT_OK(TensorHandle::CreatePackedHandle(std::move(handles), context(),
+                                                &packed_handle));
+  h->Unref();
+
+  EXPECT_EQ(packed_handle->Type(), TensorHandle::PACKED);
+  EXPECT_EQ(packed_handle->dtype, dtype);
+  TensorShape packed_shape;
+  TF_ASSERT_OK(packed_handle->Shape(&packed_shape));
+  EXPECT_EQ(packed_shape, shape);
+
+  CompositeDevice* device = reinterpret_cast<CompositeDevice*>(
+      absl::get<Device*>(packed_handle->device()));
+  EXPECT_EQ(device->name(), "/job:worker/replica:0/task:0/device:COMPOSITE:0");
+  EXPECT_EQ(device->underlying_devices()->size(), 1);
+  EXPECT_EQ(packed_handle->NumPackedHandles(), 1);
+  TensorHandle* h0 = nullptr;
+  TF_ASSERT_OK(packed_handle->ExtractPackedHandle(0, &h0));
+  EXPECT_EQ(absl::get<Device*>(h0->device()), d);
+  EXPECT_TRUE(IsReady(packed_handle));
+  packed_handle->Unref();
+}
+
 TEST(TensorHandle_ResourceDeviceTest, OnLocalDevice) {
   std::unique_ptr<Device> d0(
       CreateDevice("CPU", "/job:localhost/replica:0/task:0/device:CPU:0"));
diff --git a/tensorflow/core/common_runtime/session.cc b/tensorflow/core/common_runtime/session.cc
index 05c8432d4fe..e4fba76f3ac 100644
--- a/tensorflow/core/common_runtime/session.cc
+++ b/tensorflow/core/common_runtime/session.cc
@@ -78,7 +78,7 @@ Status NewSession(const SessionOptions& options, Session** out_session) {
   Status s = SessionFactory::GetFactory(options, &factory);
   if (!s.ok()) {
     *out_session = nullptr;
-    LOG(ERROR) << s;
+    LOG(ERROR) << "Failed to get session factory: " << s;
     return s;
   }
   // Starts exporting metrics through a platform-specific monitoring API (if
@@ -88,6 +88,7 @@ Status NewSession(const SessionOptions& options, Session** out_session) {
   s = factory->NewSession(options, out_session);
   if (!s.ok()) {
     *out_session = nullptr;
+    LOG(ERROR) << "Failed to create session: " << s;
   }
   return s;
 }
diff --git a/tensorflow/core/data/service/BUILD b/tensorflow/core/data/service/BUILD
index 6b2341d2b6d..bebd179401e 100644
--- a/tensorflow/core/data/service/BUILD
+++ b/tensorflow/core/data/service/BUILD
@@ -253,7 +253,6 @@ cc_library(
         ":grpc_master_impl",
         ":grpc_util",
         ":grpc_worker_impl",
-        ":local_credentials_factory",
         "//tensorflow/core:lib",
         "//tensorflow/core:tensorflow",
         tf_grpc_cc_dependency(),
diff --git a/tensorflow/core/distributed_runtime/cluster_function_library_runtime_test.cc b/tensorflow/core/distributed_runtime/cluster_function_library_runtime_test.cc
index ba988e83eef..527c1901e3f 100644
--- a/tensorflow/core/distributed_runtime/cluster_function_library_runtime_test.cc
+++ b/tensorflow/core/distributed_runtime/cluster_function_library_runtime_test.cc
@@ -36,9 +36,11 @@ class ClusterFunctionLibraryRuntimeTest : public ::testing::Test {
     TF_CHECK_OK(spec.AddHostPortsJob("localhost", cluster_->targets()));
     ChannelCreationFunction channel_func =
         ConvertToChannelCreationFunction(NewHostPortGrpcChannel);
+    grpc_worker_env_.reset(CreateGrpcWorkerEnv());
+    std::shared_ptr<GrpcChannelCache> channel_cache(
+        NewGrpcChannelCache(spec, channel_func));
     std::unique_ptr<WorkerCacheInterface> worker_cache(
-        NewGrpcWorkerCache(std::shared_ptr<GrpcChannelCache>(
-            NewGrpcChannelCache(spec, channel_func))));
+        NewGrpcWorkerCache(channel_cache, grpc_worker_env_.get()));
 
     worker_session_.reset(new WorkerSession(
         "cluster_test_session", "/job:localhost/replica:0/task:0",
@@ -110,6 +112,7 @@ class ClusterFunctionLibraryRuntimeTest : public ::testing::Test {
   std::unique_ptr<test::TestCluster> cluster_;
   std::unique_ptr<WorkerSession> worker_session_;
   std::unique_ptr<ClusterFunctionLibraryRuntime> cluster_flr_;
+  std::unique_ptr<GrpcWorkerEnv> grpc_worker_env_;
 };
 
 TEST_F(ClusterFunctionLibraryRuntimeTest, ConstructFunctionGraph) {
diff --git a/tensorflow/core/distributed_runtime/collective_rma_distributed.cc b/tensorflow/core/distributed_runtime/collective_rma_distributed.cc
index 73fb9d9cf64..c645c74f903 100644
--- a/tensorflow/core/distributed_runtime/collective_rma_distributed.cc
+++ b/tensorflow/core/distributed_runtime/collective_rma_distributed.cc
@@ -66,7 +66,7 @@ void PopulateTensorFromExtra(const RecvBufRespExtra& extra,
                              Tensor* cpu_tensor) {
   char* head = reinterpret_cast<char*>(DMAHelper::base(cpu_tensor));
   for (const auto& tensor_content_chunk : extra.tensor_content()) {
-    memcpy(head, tensor_content_chunk.data(),
+    memcpy(head, std::string(tensor_content_chunk).data(),
            tensor_content_chunk.size());
     head += tensor_content_chunk.size();
   }
diff --git a/tensorflow/core/distributed_runtime/eager/eager_service_impl_test.cc b/tensorflow/core/distributed_runtime/eager/eager_service_impl_test.cc
index cf430d78617..7a315ca1ea5 100644
--- a/tensorflow/core/distributed_runtime/eager/eager_service_impl_test.cc
+++ b/tensorflow/core/distributed_runtime/eager/eager_service_impl_test.cc
@@ -920,7 +920,7 @@ TEST_F(FunctionWithRemoteInputsTest, KernelAndDeviceFuncTest) {
 
   // Instantiate MatMulFunction on remote_device.
   const NodeDef node_def = MatMulFunctionNodeDef();
-  TF_ASSERT_OK(kernel->InstantiateFunc(node_def, nullptr));
+  TF_ASSERT_OK(kernel->InstantiateFunc({}, node_def, nullptr));
 
   // Run MatMulFunction on remote_device.
   gtl::InlinedVector<TensorValue, 4> input_tensors = {TensorValue()};
@@ -967,7 +967,7 @@ TEST_F(FunctionWithRemoteInputsTest, KernelAndDeviceFuncAsyncTest) {
 
   // Instantiate MatMulFunction on remote_device.
   const NodeDef node_def = MatMulFunctionNodeDef();
-  TF_ASSERT_OK(kernel->InstantiateFunc(node_def, nullptr));
+  TF_ASSERT_OK(kernel->InstantiateFunc({}, node_def, nullptr));
 
   // Run MatMulFunction on remote_device.
   gtl::InlinedVector<TensorValue, 4> input_tensors = {TensorValue()};
diff --git a/tensorflow/core/distributed_runtime/eager/remote_copy_node.cc b/tensorflow/core/distributed_runtime/eager/remote_copy_node.cc
index 090417863f3..41027d43188 100644
--- a/tensorflow/core/distributed_runtime/eager/remote_copy_node.cc
+++ b/tensorflow/core/distributed_runtime/eager/remote_copy_node.cc
@@ -58,7 +58,8 @@ Status CreateUncachedKernelAndDeviceOp(
                                       ctx.HostCPU()));
 
   const NodeDef& ndef = op->MutableAttrs()->BuildNodeDef();
-  return kernel->get()->Init(ndef, /*graph_collector=*/nullptr);
+  return kernel->get()->Init({ctx.LogDevicePlacement()}, ndef,
+                             /*graph_collector=*/nullptr);
 }
 
 // This gets a unique wire ID. We add a random identifier so that if the
diff --git a/tensorflow/core/distributed_runtime/remote_device_test.cc b/tensorflow/core/distributed_runtime/remote_device_test.cc
index 62082aa6d59..ff1acd9c015 100644
--- a/tensorflow/core/distributed_runtime/remote_device_test.cc
+++ b/tensorflow/core/distributed_runtime/remote_device_test.cc
@@ -39,6 +39,7 @@ class RemoteDeviceTest : public ::testing::Test {
   WorkerInterface* wi_;
   std::vector<Device*> devices_;
   std::unique_ptr<test::TestCluster> cluster_;
+  std::unique_ptr<GrpcWorkerEnv> grpc_worker_env_;
 
   RemoteDeviceTest() {
     SessionOptions options;
@@ -51,7 +52,9 @@ class RemoteDeviceTest : public ::testing::Test {
         ConvertToChannelCreationFunction(NewHostPortGrpcChannel);
     std::shared_ptr<GrpcChannelCache> channel_cache(
         NewGrpcChannelCache(spec, channel_func));
-    worker_cache_.reset(NewGrpcWorkerCache(channel_cache));
+    grpc_worker_env_.reset(CreateGrpcWorkerEnv());
+    worker_cache_.reset(
+        NewGrpcWorkerCache(channel_cache, grpc_worker_env_.get()));
     remote_name_ = "/job:localhost/replica:0/task:0";
     wi_ = worker_cache_->GetOrCreateWorker(remote_name_);
   }
@@ -82,11 +85,13 @@ class RemoteDeviceTest : public ::testing::Test {
 
 TEST_F(RemoteDeviceTest, GetStatus) {
   // We know what the testlib's fake server does.
-  EXPECT_EQ(devices_[0]->name(), strings::StrCat(remote_name_, "/cpu:0"));
+  EXPECT_EQ(devices_[0]->name(),
+            strings::StrCat(remote_name_, "/device:CPU:0"));
   EXPECT_EQ(devices_[0]->attributes().device_type(),
             DeviceType(DEVICE_CPU).type());
   EXPECT_EQ(devices_[0]->attributes().memory_limit(), 256 << 20);
-  EXPECT_EQ(devices_[1]->name(), strings::StrCat(remote_name_, "/cpu:1"));
+  EXPECT_EQ(devices_[1]->name(),
+            strings::StrCat(remote_name_, "/device:CPU:1"));
   EXPECT_EQ(devices_[1]->attributes().memory_limit(), 256 << 20);
 }
 
diff --git a/tensorflow/core/distributed_runtime/rpc/BUILD b/tensorflow/core/distributed_runtime/rpc/BUILD
index 60d7172c2fc..02dcfa86dd8 100644
--- a/tensorflow/core/distributed_runtime/rpc/BUILD
+++ b/tensorflow/core/distributed_runtime/rpc/BUILD
@@ -165,6 +165,26 @@ cc_library(
         "//tensorflow/core/distributed_runtime:worker_cache_partial",
         "//tensorflow/core/distributed_runtime:worker_interface",
         "//tensorflow/core/distributed_runtime/rpc/eager:grpc_eager_client",
+        "//tensorflow/core/util:env_var",
+    ],
+)
+
+tf_cc_test(
+    name = "grpc_worker_cache_test",
+    size = "small",
+    srcs = [
+        "grpc_worker_cache_test.cc",
+    ],
+    deps = [
+        ":grpc_worker_cache",
+        "//tensorflow/c:tf_status_headers",
+        "//tensorflow/core:test",
+        "//tensorflow/core:test_main",
+        "//tensorflow/core/distributed_runtime:test_utils",
+        "//tensorflow/core/distributed_runtime/rpc:grpc_channel",
+        "//tensorflow/core/platform:env",
+        "//tensorflow/core/platform:status",
+        "//tensorflow/core/platform:strcat",
     ],
 )
 
diff --git a/tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc b/tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc
index 6523d2fb4dd..c0b4d0ef6ec 100644
--- a/tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc
+++ b/tensorflow/core/distributed_runtime/rpc/grpc_server_lib.cc
@@ -87,23 +87,6 @@ RendezvousMgrInterface* NewRpcRendezvousMgr(const WorkerEnv* env) {
   return new RpcRendezvousMgr(env);
 }
 
-std::unique_ptr<GrpcWorkerEnv> CreateGrpcWorkerEnv() {
-  int num_cpus = port::NumSchedulableCPUs();
-  int64 num_completion_queues;
-  Status status = ReadInt64FromEnvVar("TF_GRPC_WORKER_CACHE_QUEUES", 64,
-                                      &num_completion_queues);
-  if (!status.ok()) {
-    LOG(ERROR) << "Error parsing TF_GRPC_WORKER_CACHE_QUEUES: " << status;
-  }
-  int64 num_threads;
-  status = ReadInt64FromEnvVar("TF_GRPC_WORKER_CACHE_THREADS", num_cpus,
-                               &num_threads);
-  if (!status.ok()) {
-    LOG(ERROR) << "Error parsing TF_GRPC_WORKER_CACHE_THREADS: " << status;
-  }
-  return absl::make_unique<GrpcWorkerEnv>(num_completion_queues, num_threads);
-}
-
 }  // namespace
 
 GrpcServer::GrpcServer(const ServerDef& server_def, Env* env)
@@ -275,7 +258,7 @@ Status GrpcServer::Init(const GrpcServerOptions& opts) {
     return errors::Unknown("Could not start gRPC server");
   }
   // Create the execution environment for the GRPC workers cache.
-  grpc_worker_env_ = CreateGrpcWorkerEnv();
+  grpc_worker_env_.reset(CreateGrpcWorkerEnv());
 
   WorkerCacheInterface* worker_cache;
   WorkerCacheFactoryOptions worker_cache_factory_options(server_def_);
@@ -401,7 +384,7 @@ Status GrpcServer::WorkerCacheFactory(const WorkerCacheFactoryOptions& options,
                                    " differs from expected port ", bound_port_);
   }
   *worker_cache = NewGrpcWorkerCacheWithLocalWorker(
-      channel_cache, worker_impl(), name_prefix, grpc_worker_env_.get());
+      channel_cache, grpc_worker_env(), worker_impl(), name_prefix);
   return Status::OK();
 }
 
diff --git a/tensorflow/core/distributed_runtime/rpc/grpc_server_lib.h b/tensorflow/core/distributed_runtime/rpc/grpc_server_lib.h
index 0474c5a517f..1e8fe35b5b4 100644
--- a/tensorflow/core/distributed_runtime/rpc/grpc_server_lib.h
+++ b/tensorflow/core/distributed_runtime/rpc/grpc_server_lib.h
@@ -137,6 +137,7 @@ class GrpcServer : public ServerInterface {
 
   const ServerDef& server_def() const { return server_def_; }
   GrpcWorker* worker_impl() const { return worker_impl_.get(); }
+  GrpcWorkerEnv* grpc_worker_env() const { return grpc_worker_env_.get(); }
 
  private:
   Env* env_;
diff --git a/tensorflow/core/distributed_runtime/rpc/grpc_worker_cache.cc b/tensorflow/core/distributed_runtime/rpc/grpc_worker_cache.cc
index 1d75728ddd2..cb4458a2ca4 100644
--- a/tensorflow/core/distributed_runtime/rpc/grpc_worker_cache.cc
+++ b/tensorflow/core/distributed_runtime/rpc/grpc_worker_cache.cc
@@ -15,24 +15,20 @@ limitations under the License.
 
 #include "tensorflow/core/distributed_runtime/rpc/grpc_worker_cache.h"
 
-#include <unordered_map>
-
 #include "tensorflow/core/distributed_runtime/rpc/eager/grpc_eager_client.h"
 #include "tensorflow/core/distributed_runtime/rpc/grpc_remote_worker.h"
 #include "tensorflow/core/distributed_runtime/rpc/grpc_util.h"
 #include "tensorflow/core/distributed_runtime/worker_cache_logger.h"
 #include "tensorflow/core/distributed_runtime/worker_cache_partial.h"
 #include "tensorflow/core/distributed_runtime/worker_interface.h"
+#include "tensorflow/core/platform/cpu_info.h"
 #include "tensorflow/core/platform/mutex.h"
+#include "tensorflow/core/util/env_var.h"
 
 namespace tensorflow {
 
 namespace {
 
-// TODO(ncteisen): consider adding a config var or flag for this
-static const size_t kGrpcWorkerCacheThreadCount = 8;
-static const size_t kNumCallbackThreads = 10;
-
 class GrpcWorkerCache : public WorkerCachePartial {
  public:
   explicit GrpcWorkerCache(std::shared_ptr<GrpcChannelCache> channel_cache,
@@ -43,13 +39,7 @@ class GrpcWorkerCache : public WorkerCachePartial {
         local_worker_(local_worker),
         channel_cache_(channel_cache),
         worker_env_(worker_env),
-        next_round_robin_assignment_(0) {
-    if (worker_env_ == nullptr) {
-      worker_env_ptr_ = absl::make_unique<GrpcWorkerEnv>(
-          kGrpcWorkerCacheThreadCount, kNumCallbackThreads);
-      worker_env_ = worker_env_ptr_.get();
-    }
-  }
+        next_round_robin_assignment_(0) {}
 
   void ListWorkers(std::vector<string>* workers) const override {
     channel_cache_->ListWorkers(workers);
@@ -118,8 +108,7 @@ class GrpcWorkerCache : public WorkerCachePartial {
   WorkerInterface* const local_worker_;  // Not owned.
   std::shared_ptr<GrpcChannelCache> channel_cache_;
   WorkerCacheLogger logger_;
-  GrpcWorkerEnv* worker_env_;  // Not owned, if worker_env_ptr_ is nullptr.
-  std::unique_ptr<GrpcWorkerEnv> worker_env_ptr_;
+  GrpcWorkerEnv* worker_env_;  // Not owned
 
   mutex assignment_mu_;
   std::unordered_map<std::string, size_t> target_assignments_
@@ -154,13 +143,32 @@ GrpcWorkerEnv::GrpcWorkerCacheThread::~GrpcWorkerCacheThread() {
   thread_.reset();
 }
 
-WorkerCacheInterface* NewGrpcWorkerCache(std::shared_ptr<GrpcChannelCache> cc) {
-  return new GrpcWorkerCache(cc, nullptr, "", nullptr);
+GrpcWorkerEnv* CreateGrpcWorkerEnv() {
+  int num_cpus = port::NumSchedulableCPUs();
+  int64 num_completion_queues;
+  Status status = ReadInt64FromEnvVar("TF_GRPC_WORKER_CACHE_QUEUES", 64,
+                                      &num_completion_queues);
+  if (!status.ok()) {
+    LOG(ERROR) << "Error parsing TF_GRPC_WORKER_CACHE_QUEUES: " << status;
+  }
+  int64 num_threads;
+  status = ReadInt64FromEnvVar("TF_GRPC_WORKER_CACHE_THREADS", num_cpus,
+                               &num_threads);
+  if (!status.ok()) {
+    LOG(ERROR) << "Error parsing TF_GRPC_WORKER_CACHE_THREADS: " << status;
+  }
+  return new GrpcWorkerEnv(num_completion_queues, num_threads);
+}
+
+WorkerCacheInterface* NewGrpcWorkerCache(std::shared_ptr<GrpcChannelCache> cc,
+                                         GrpcWorkerEnv* worker_env) {
+  return new GrpcWorkerCache(cc, /*local_worker=*/nullptr, /*local_target=*/"",
+                             worker_env);
 }
 
 WorkerCacheInterface* NewGrpcWorkerCacheWithLocalWorker(
-    std::shared_ptr<GrpcChannelCache> cc, WorkerInterface* local_worker,
-    const string& local_target, GrpcWorkerEnv* worker_env) {
+    std::shared_ptr<GrpcChannelCache> cc, GrpcWorkerEnv* worker_env,
+    WorkerInterface* local_worker, const string& local_target) {
   return new GrpcWorkerCache(cc, local_worker, local_target, worker_env);
 }
 
diff --git a/tensorflow/core/distributed_runtime/rpc/grpc_worker_cache.h b/tensorflow/core/distributed_runtime/rpc/grpc_worker_cache.h
index ef54f3e9b97..2dfbc79aa8a 100644
--- a/tensorflow/core/distributed_runtime/rpc/grpc_worker_cache.h
+++ b/tensorflow/core/distributed_runtime/rpc/grpc_worker_cache.h
@@ -16,8 +16,6 @@ limitations under the License.
 #ifndef TENSORFLOW_CORE_DISTRIBUTED_RUNTIME_RPC_GRPC_WORKER_CACHE_H_
 #define TENSORFLOW_CORE_DISTRIBUTED_RUNTIME_RPC_GRPC_WORKER_CACHE_H_
 
-#include <memory>
-
 #include "tensorflow/core/distributed_runtime/rpc/grpc_channel.h"
 #include "tensorflow/core/distributed_runtime/rpc/grpc_client_cq_tag.h"
 #include "tensorflow/core/distributed_runtime/worker_cache.h"
@@ -62,12 +60,17 @@ class GrpcWorkerEnv {
   std::vector<GrpcWorkerCacheThread> threads_;
 };
 
+// Create a GrpcWorkerEnv instance that can be used as argument to create
+// gRPC worker cache. Caller should take the ownership of the returned instance.
+GrpcWorkerEnv* CreateGrpcWorkerEnv();
+
 // The returned WorkerCacheInterface object takes the ownership of "cc".
-WorkerCacheInterface* NewGrpcWorkerCache(std::shared_ptr<GrpcChannelCache> cc);
+WorkerCacheInterface* NewGrpcWorkerCache(std::shared_ptr<GrpcChannelCache> cc,
+                                         GrpcWorkerEnv* worker_env);
 
 WorkerCacheInterface* NewGrpcWorkerCacheWithLocalWorker(
-    std::shared_ptr<GrpcChannelCache> cc, WorkerInterface* local_worker,
-    const string& local_target, GrpcWorkerEnv* worker_env);
+    std::shared_ptr<GrpcChannelCache> cc, GrpcWorkerEnv* worker_env,
+    WorkerInterface* local_worker, const string& local_target);
 
 }  // namespace tensorflow
 #endif  // TENSORFLOW_CORE_DISTRIBUTED_RUNTIME_RPC_GRPC_WORKER_CACHE_H_
diff --git a/tensorflow/core/distributed_runtime/rpc/grpc_worker_cache_test.cc b/tensorflow/core/distributed_runtime/rpc/grpc_worker_cache_test.cc
new file mode 100644
index 00000000000..ff32fa91205
--- /dev/null
+++ b/tensorflow/core/distributed_runtime/rpc/grpc_worker_cache_test.cc
@@ -0,0 +1,87 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include "tensorflow/core/distributed_runtime/rpc/grpc_worker_cache.h"
+
+#include "tensorflow/c/tf_status.h"
+#include "tensorflow/core/distributed_runtime/rpc/grpc_channel.h"
+#include "tensorflow/core/distributed_runtime/test_utils.h"
+#include "tensorflow/core/lib/core/status_test_util.h"
+#include "tensorflow/core/platform/status.h"
+#include "tensorflow/core/platform/strcat.h"
+#include "tensorflow/core/platform/test.h"
+#include "tensorflow/core/platform/threadpool.h"
+
+namespace tensorflow {
+
+TEST(GrpcWorkerCacheTest, NewGrpcWorkerCache) {
+  GrpcChannelSpec spec;
+  TF_ASSERT_OK(spec.AddHostPortsJob("worker", {"a:0", "b:1", "c:2"}));
+  ChannelCreationFunction channel_func =
+      ConvertToChannelCreationFunction(NewHostPortGrpcChannel);
+  auto channel_cache = std::shared_ptr<GrpcChannelCache>(
+      NewGrpcChannelCache(spec, channel_func));
+  std::unique_ptr<GrpcWorkerEnv> grpc_worker_env(CreateGrpcWorkerEnv());
+
+  // We created a job with 3 tasks. Getting the task 0, 1, 2 should return valid
+  // worker interfaces, and getting other tasks should return nullptr.
+  std::unique_ptr<WorkerCacheInterface> worker_cache(
+      NewGrpcWorkerCache(channel_cache, grpc_worker_env.get()));
+  WorkerInterface* wi;
+  wi = worker_cache->GetOrCreateWorker("/job:worker/replica:0/task:0");
+  EXPECT_NE(wi, nullptr);
+  worker_cache->ReleaseWorker("/job:worker/replica:0/task:0", wi);
+  wi = worker_cache->GetOrCreateWorker("/job:worker/replica:0/task:1");
+  EXPECT_NE(wi, nullptr);
+  worker_cache->ReleaseWorker("/job:worker/replica:0/task:1", wi);
+  wi = worker_cache->GetOrCreateWorker("/job:worker/replica:0/task:2");
+  EXPECT_NE(wi, nullptr);
+  worker_cache->ReleaseWorker("/job:worker/replica:0/task:2", wi);
+  wi = worker_cache->GetOrCreateWorker("/job:worker/replica:0/task:3");
+  EXPECT_EQ(wi, nullptr);
+
+  // Test creating a worker cache instance with local worker, and getting the
+  // worker instance with the specified local target.
+  std::unique_ptr<TestWorkerInterface> local_wi;
+  worker_cache.reset(NewGrpcWorkerCacheWithLocalWorker(
+      channel_cache, grpc_worker_env.get(), local_wi.get(), "local_target"));
+  wi = worker_cache->GetOrCreateWorker("local_target");
+  EXPECT_EQ(wi, local_wi.get());
+}
+
+TEST(GrpcWorkerCacheTest, DestructWorkerCacheInThreadPool) {
+  GrpcChannelSpec spec;
+  TF_ASSERT_OK(spec.AddHostPortsJob("worker", {"a:1", "b:2", "c:3"}));
+  ChannelCreationFunction channel_func =
+      ConvertToChannelCreationFunction(NewHostPortGrpcChannel);
+  auto channel_cache = std::shared_ptr<GrpcChannelCache>(
+      NewGrpcChannelCache(spec, channel_func));
+  std::unique_ptr<GrpcWorkerEnv> grpc_worker_env(CreateGrpcWorkerEnv());
+
+  // The GrpcWorkerEnv threadpool is used for worker interfaces for gRPC
+  // completion queue callbacks. Test worker cache destruction inside the
+  // callbacks that runs in the GrpcWorkerEnv threadpool.
+  WorkerCacheInterface* worker_cache =
+      NewGrpcWorkerCache(channel_cache, grpc_worker_env.get());
+  thread::ThreadPool* tp = grpc_worker_env->GetThreadPool();
+  Notification n;
+  tp->Schedule([worker_cache, &n] {
+    delete worker_cache;
+    n.Notify();
+  });
+  n.WaitForNotification();
+}
+
+}  // namespace tensorflow
diff --git a/tensorflow/core/framework/allocator_registry.cc b/tensorflow/core/framework/allocator_registry.cc
index 099c4bacc86..543f35a3254 100644
--- a/tensorflow/core/framework/allocator_registry.cc
+++ b/tensorflow/core/framework/allocator_registry.cc
@@ -113,7 +113,7 @@ SubAllocator* AllocatorFactoryRegistry::GetSubAllocator(int numa_node) {
       CHECK_LE(numa_node, port::NUMANumNodes());
       index = 1 + numa_node;
     }
-    if (best_entry->sub_allocators.size() < (index + 1)) {
+    if (best_entry->sub_allocators.size() < static_cast<size_t>(index + 1)) {
       best_entry->sub_allocators.resize(index + 1);
     }
     if (!best_entry->sub_allocators[index].get()) {
diff --git a/tensorflow/core/framework/cpu_allocator_impl.cc b/tensorflow/core/framework/cpu_allocator_impl.cc
index 1faee8986b8..814233074fb 100644
--- a/tensorflow/core/framework/cpu_allocator_impl.cc
+++ b/tensorflow/core/framework/cpu_allocator_impl.cc
@@ -75,7 +75,7 @@ class CPUAllocator : public Allocator {
   string Name() override { return "cpu"; }
 
   void* AllocateRaw(size_t alignment, size_t num_bytes) override {
-    if (num_bytes > LargeAllocationWarningBytes() &&
+    if (num_bytes > static_cast<size_t>(LargeAllocationWarningBytes()) &&
         single_allocation_warning_count_ < kMaxSingleAllocationWarnings) {
       ++single_allocation_warning_count_;
       LOG(WARNING) << "Allocation of " << num_bytes << " exceeds "
diff --git a/tensorflow/core/framework/dataset.cc b/tensorflow/core/framework/dataset.cc
index e99fc951391..ad28d82f8dc 100644
--- a/tensorflow/core/framework/dataset.cc
+++ b/tensorflow/core/framework/dataset.cc
@@ -336,8 +336,7 @@ bool GraphDefBuilderWrapper::HasAttr(const string& name,
 }
 
 Status IteratorBase::InitializeBase(IteratorContext* ctx,
-                                    const IteratorBase* parent,
-                                    const string& output_prefix) {
+                                    const IteratorBase* parent) {
   parent_ = parent;
   id_ =
       Hash64CombineUnordered(Hash64(prefix()), reinterpret_cast<uint64>(this));
@@ -349,9 +348,8 @@ Status IteratorBase::InitializeBase(IteratorContext* ctx,
     auto factory = [ctx, this](model::Node::Args args) {
       return CreateNode(ctx, std::move(args));
     };
-    model->AddNode(std::move(factory), prefix(), output_prefix, &node_);
-    cleanup_fns_.push_back(
-        [model, prefix = prefix()]() { model->RemoveNode(prefix); });
+    model->AddNode(std::move(factory), prefix(), parent->model_node(), &node_);
+    cleanup_fns_.push_back([this, model]() { model->RemoveNode(node_); });
   }
   return Status::OK();
 }
@@ -418,7 +416,7 @@ Status DatasetBase::MakeIterator(
     const string& output_prefix,
     std::unique_ptr<IteratorBase>* iterator) const {
   *iterator = MakeIteratorInternal(output_prefix);
-  Status s = (*iterator)->InitializeBase(ctx, parent, output_prefix);
+  Status s = (*iterator)->InitializeBase(ctx, parent);
   if (s.ok()) {
     s.Update((*iterator)->Initialize(ctx));
   }
diff --git a/tensorflow/core/framework/dataset.h b/tensorflow/core/framework/dataset.h
index 3635cf7c4ba..ef58c502613 100644
--- a/tensorflow/core/framework/dataset.h
+++ b/tensorflow/core/framework/dataset.h
@@ -610,6 +610,9 @@ class IteratorBase {
   // properly propagate errors.
   virtual Status Initialize(IteratorContext* ctx) { return Status::OK(); }
 
+  // Performs initialization of the base iterator.
+  Status InitializeBase(IteratorContext* ctx, const IteratorBase* parent);
+
   // Saves the state of this iterator.
   virtual Status Save(SerializationContext* ctx, IteratorStateWriter* writer) {
     return SaveInternal(ctx, writer);
@@ -673,10 +676,6 @@ class IteratorBase {
   friend class DatasetBase;
   friend class DatasetBaseIterator;  // for access to `node_`
 
-  // Performs initialization of the base iterator.
-  Status InitializeBase(IteratorContext* ctx, const IteratorBase* parent,
-                        const string& output_prefix);
-
   std::vector<std::function<void()>> cleanup_fns_;
   std::shared_ptr<model::Node> node_ = nullptr;
   const IteratorBase* parent_ = nullptr;  // Not owned.
diff --git a/tensorflow/core/framework/load_library.cc b/tensorflow/core/framework/load_library.cc
index c223eac4722..ab08d644074 100644
--- a/tensorflow/core/framework/load_library.cc
+++ b/tensorflow/core/framework/load_library.cc
@@ -107,7 +107,7 @@ Status LoadLibrary(const char* library_filename, void** result,
   if (env->GetSymbolFromLibrary(library.handle, "TfTpu_Initialize",
                                 &unused_symbol)
           .ok()) {
-    TF_RETURN_IF_ERROR(tensorflow::tpu::InitializeTPULibrary(library.handle));
+    TF_RETURN_IF_ERROR(tensorflow::tpu::InitializeTpuLibrary(library.handle));
   }
 #endif  // IS_MOBILE_PLATFORM
 
diff --git a/tensorflow/core/framework/model.cc b/tensorflow/core/framework/model.cc
index 658be94b9bb..a88950e4abc 100644
--- a/tensorflow/core/framework/model.cc
+++ b/tensorflow/core/framework/model.cc
@@ -1089,22 +1089,18 @@ Node::NodeVector Node::CollectNodes(TraversalOrder order) const
   NodeVector node_vector;
   std::list<std::shared_ptr<Node>> temp_list;
 
-  {
-    for (auto& input : inputs_) {
-      node_vector.push_back(input);
-      temp_list.push_back(input);
-    }
+  for (auto& input : inputs_) {
+    node_vector.push_back(input);
+    temp_list.push_back(input);
   }
 
   while (!temp_list.empty()) {
     auto cur_node = temp_list.front();
     temp_list.pop_front();
-    {
-      tf_shared_lock l(cur_node->mu_);
-      for (auto& input : cur_node->inputs_) {
-        node_vector.push_back(input);
-        temp_list.push_back(input);
-      }
+    tf_shared_lock l(cur_node->mu_);
+    for (auto& input : cur_node->inputs_) {
+      node_vector.push_back(input);
+      temp_list.push_back(input);
     }
   }
 
@@ -1222,46 +1218,41 @@ void Node::TotalMaximumBufferedBytesHelper(
 }
 
 void Model::AddNode(Node::Factory factory, const string& name,
-                    const string& output_name,
+                    std::shared_ptr<Node> parent,
                     std::shared_ptr<Node>* out_node) {
-  // The name captures the sequence of iterators joined by `::`. We use the full
-  // sequence as the key in the lookup table, but only the last element of the
-  // sequence as the name node.
-  std::vector<string> tokens =
-      str_util::Split(name, ':', str_util::SkipEmpty());
-  // The output name might contain an index. We need to strip it to make it
-  // possible for the model to successfully identify the output node.
-  string sanitized_output_name = output_name;
-  if (str_util::EndsWith(output_name, "]")) {
-    sanitized_output_name = output_name.substr(0, output_name.rfind('['));
-  }
-  std::shared_ptr<Node> output;
+  // The name captures the sequence of iterators joined by `::`. We only use the
+  // last element of the sequence as the name node.
+  auto node_name = str_util::Split(name, ':', str_util::SkipEmpty()).back();
   mutex_lock l(mu_);
-  auto it = lookup_table_.find(sanitized_output_name);
-  if (it != lookup_table_.end()) {
-    output = it->second;
-  }
-  std::shared_ptr<Node> node = factory({id_counter_++, tokens.back(), output});
+  std::shared_ptr<Node> node = factory({id_counter_++, node_name, parent});
   if (!output_) {
     output_ = node;
   }
-  if (output) {
+  if (parent) {
     VLOG(3) << "Adding " << node->long_name() << " as input for "
-            << output->long_name();
-    output->add_input(node);
+            << parent->long_name();
+    parent->add_input(node);
   } else {
     VLOG(3) << "Adding " << node->long_name();
   }
   collect_resource_usage_ =
       collect_resource_usage_ || node->has_tunable_parameters();
-  lookup_table_.insert(std::make_pair(name, node));
-  *out_node = node;
+  *out_node = std::move(node);
 }
 
 void Model::FlushMetrics() {
-  tf_shared_lock l(mu_);
-  for (const auto& pair : lookup_table_) {
-    pair.second->FlushMetrics();
+  std::deque<std::shared_ptr<Node>> queue;
+  {
+    tf_shared_lock l(mu_);
+    if (output_) queue.push_back(output_);
+  }
+  while (!queue.empty()) {
+    auto node = queue.front();
+    queue.pop_front();
+    node->FlushMetrics();
+    for (auto input : node->inputs()) {
+      queue.push_back(input);
+    }
   }
 }
 
@@ -1277,16 +1268,14 @@ void Model::Optimize(AutotuneAlgorithm algorithm, int64 cpu_budget,
   }
 }
 
-void Model::RemoveNode(const string& name) {
+void Model::RemoveNode(std::shared_ptr<Node> node) {
   mutex_lock l(mu_);
-  auto node = gtl::FindOrNull(lookup_table_, name);
   if (node) {
-    if ((*node)->output()) {
-      (*node)->output()->remove_input(*node);
+    if (node->output()) {
+      node->output()->remove_input(node);
     }
-    VLOG(3) << "Removing " << (*node)->long_name();
+    VLOG(3) << "Removing " << node->long_name();
   }
-  lookup_table_.erase(name);
 }
 
 absl::flat_hash_map<string, std::shared_ptr<Parameter>>
diff --git a/tensorflow/core/framework/model.h b/tensorflow/core/framework/model.h
index e325056f0c4..12a0abae5bc 100644
--- a/tensorflow/core/framework/model.h
+++ b/tensorflow/core/framework/model.h
@@ -274,6 +274,7 @@ class Node {
 
   // Records that a node thread has stopped executing.
   void record_stop(int64 time_nanos) TF_LOCKS_EXCLUDED(mu_) {
+    // TODO(jsimsa): Use DCHECK_NE(work_start_, 0) here.
     if (work_start_ != 0) {
       processing_time_ += time_nanos - work_start_;
       work_start_ = 0;
@@ -598,10 +599,9 @@ class Model {
   // Indicates whether to collect resource usage.
   bool collect_resource_usage() const { return collect_resource_usage_; }
 
-  // Adds a node with the given name and given output. The method returns
-  // a pointer to the node but does not transfer ownership.
+  // Adds a node with the given name and given parent.
   void AddNode(Node::Factory factory, const string& name,
-               const string& output_name, std::shared_ptr<Node>* out_node)
+               std::shared_ptr<Node> parent, std::shared_ptr<Node>* out_node)
       TF_LOCKS_EXCLUDED(mu_);
 
   // Flushes metrics record by the model.
@@ -612,7 +612,7 @@ class Model {
       TF_LOCKS_EXCLUDED(mu_);
 
   // Removes the given node.
-  void RemoveNode(const string& name) TF_LOCKS_EXCLUDED(mu_);
+  void RemoveNode(std::shared_ptr<Node> node) TF_LOCKS_EXCLUDED(mu_);
 
  private:
   // Collects tunable parameters in the tree rooted in the given node, returning
@@ -670,8 +670,6 @@ class Model {
   mutex mu_;
   int64 id_counter_ TF_GUARDED_BY(mu_) = 1;
   std::shared_ptr<Node> output_ TF_GUARDED_BY(mu_);
-  absl::flat_hash_map<string, std::shared_ptr<Node>> lookup_table_
-      TF_GUARDED_BY(mu_);
 
   // Indicates whether the modeling framework should collect resource usage
   // (e.g. CPU, memory). The logic for collecting this information assumes that
diff --git a/tensorflow/core/framework/op_kernel.cc b/tensorflow/core/framework/op_kernel.cc
index 2e7747380b4..abf73cb57df 100644
--- a/tensorflow/core/framework/op_kernel.cc
+++ b/tensorflow/core/framework/op_kernel.cc
@@ -1497,6 +1497,13 @@ Status SupportedDeviceTypesForNode(
         }
       }
     }
+
+    // If we were unable to find any valid devices let's validate if the node is
+    // even valid.
+    if (prioritized_device_types->empty()) {
+      TF_RETURN_IF_ERROR(ValidateNodeDef(def, op_reg_data->op_def));
+    }
+
     std::sort(prioritized_device_types->begin(),
               prioritized_device_types->end(),
               [](const std::pair<DeviceType, int32>& a,
diff --git a/tensorflow/core/framework/register_types.h b/tensorflow/core/framework/register_types.h
index 47aab2efb61..bc3e5e1743b 100644
--- a/tensorflow/core/framework/register_types.h
+++ b/tensorflow/core/framework/register_types.h
@@ -179,13 +179,14 @@ limitations under the License.
       TF_CALL_int64(m) TF_CALL_uint16(m) TF_CALL_int16(m) TF_CALL_uint8(m) \
           TF_CALL_int8(m)
 
-// Call "m" for all number types, including complex64 and complex128.
+#define TF_CALL_COMPLEX_TYPES(m) TF_CALL_complex64(m) TF_CALL_complex128(m)
+
+// Call "m" for all number types, including complex types
 #define TF_CALL_NUMBER_TYPES(m) \
-  TF_CALL_REAL_NUMBER_TYPES(m) TF_CALL_complex64(m) TF_CALL_complex128(m)
+  TF_CALL_REAL_NUMBER_TYPES(m) TF_CALL_COMPLEX_TYPES(m)
 
 #define TF_CALL_NUMBER_TYPES_NO_INT32(m) \
-  TF_CALL_REAL_NUMBER_TYPES_NO_INT32(m)  \
-  TF_CALL_complex64(m) TF_CALL_complex128(m)
+  TF_CALL_REAL_NUMBER_TYPES_NO_INT32(m) TF_CALL_COMPLEX_TYPES(m)
 
 #define TF_CALL_POD_TYPES(m) TF_CALL_NUMBER_TYPES(m) TF_CALL_bool(m)
 
@@ -202,8 +203,7 @@ limitations under the License.
 
 // Call "m" on all types supported on GPU.
 #define TF_CALL_GPU_ALL_TYPES(m) \
-  TF_CALL_GPU_NUMBER_TYPES(m)    \
-  TF_CALL_bool(m) TF_CALL_complex64(m) TF_CALL_complex128(m)
+  TF_CALL_GPU_NUMBER_TYPES(m) TF_CALL_COMPLEX_TYPES(m) TF_CALL_bool(m)
 
 #define TF_CALL_GPU_NUMBER_TYPES_NO_HALF(m) TF_CALL_float(m) TF_CALL_double(m)
 
@@ -213,11 +213,10 @@ limitations under the License.
   TF_CALL_qint8(m) TF_CALL_quint8(m) TF_CALL_qint32(m)
 
 // Types used for save and restore ops.
-#define TF_CALL_SAVE_RESTORE_TYPES(m)                                     \
-  TF_CALL_INTEGRAL_TYPES(m)                                               \
-  TF_CALL_half(m) TF_CALL_float(m) TF_CALL_double(m) TF_CALL_complex64(m) \
-      TF_CALL_complex128(m) TF_CALL_bool(m) TF_CALL_tstring(m)            \
-          TF_CALL_QUANTIZED_TYPES(m)
+#define TF_CALL_SAVE_RESTORE_TYPES(m)      \
+  TF_CALL_REAL_NUMBER_TYPES_NO_BFLOAT16(m) \
+  TF_CALL_COMPLEX_TYPES(m)                 \
+  TF_CALL_QUANTIZED_TYPES(m) TF_CALL_bool(m) TF_CALL_tstring(m)
 
 #ifdef TENSORFLOW_SYCL_NO_DOUBLE
 #define TF_CALL_SYCL_double(m)
diff --git a/tensorflow/core/framework/tensor_shape.cc b/tensorflow/core/framework/tensor_shape.cc
index f4b440e9cd1..8040a316a45 100644
--- a/tensorflow/core/framework/tensor_shape.cc
+++ b/tensorflow/core/framework/tensor_shape.cc
@@ -182,7 +182,7 @@ void TensorShapeBase<Shape>::InitDims(gtl::ArraySlice<int64> dim_sizes) {
 
   // Allow sizes that are under kint64max^0.25 so that 4-way multiplication
   // below cannot overflow.
-  static const uint64 kMaxSmall = 0xd744;
+  static const int64 kMaxSmall = 0xd744;
   static_assert(kMaxSmall * kMaxSmall * kMaxSmall * kMaxSmall <= kint64max,
                 "bad overflow check");
   bool large_size = false;
diff --git a/tensorflow/core/grappler/optimizers/mkl_remapper_test.cc b/tensorflow/core/grappler/optimizers/mkl_remapper_test.cc
index cf1953fcdb2..77929bea257 100644
--- a/tensorflow/core/grappler/optimizers/mkl_remapper_test.cc
+++ b/tensorflow/core/grappler/optimizers/mkl_remapper_test.cc
@@ -23,6 +23,19 @@ limitations under the License.
 #include "tensorflow/core/lib/core/status_test_util.h"
 #include "tensorflow/core/platform/test.h"
 
+#define REGISTER_TEST_FLOAT32(TEST) REGISTER_TEST(TEST, DT_FLOAT, Float32Input);
+
+#ifdef ENABLE_INTEL_MKL_BFLOAT16
+#define REGISTER_TEST_BFLOAT16(TEST) \
+  REGISTER_TEST(TEST, DT_BFLOAT16, BFloat16Input);
+
+#define REGISTER_TEST_ALL_TYPES(TEST) \
+  REGISTER_TEST_FLOAT32(TEST);        \
+  REGISTER_TEST_BFLOAT16(TEST);
+#else
+#define REGISTER_TEST_ALL_TYPES(TEST) REGISTER_TEST_FLOAT32(TEST);
+#endif  // ENABLE_INTEL_MKL_BFLOAT16
+
 namespace tensorflow {
 namespace grappler {
 
@@ -207,93 +220,99 @@ CREATE_CONV2DFUSION_ADD_BCAST_TEST(AddV2);
 #undef CREATE_CONV2DFUSION_ADD_BCAST_TEST
 #undef CREATE_CONV2DFUSION_TEST
 
-TEST_F(MklRemapperTest, FuseDepthwiseConv2DWithBiasAndActivation) {
-  using ::tensorflow::ops::Placeholder;
-
-  for (const string& activation : {"Relu", "Relu6", "Elu", "None"}) {
-    tensorflow::Scope s = tensorflow::Scope::NewRootScope();
-
-    auto input_shape = Placeholder::Shape({8, 32, 32, 3});
-    auto filter_shape = Placeholder::Shape({1, 1, 3, 1});
-    auto bias_shape = Placeholder::Shape({3});
-
-    auto input = Placeholder(s.WithOpName("input"), DT_FLOAT, input_shape);
-    auto filter = Placeholder(s.WithOpName("filter"), DT_FLOAT, filter_shape);
-    auto bias = Placeholder(s.WithOpName("bias"), DT_FLOAT, bias_shape);
-
-    std::vector<int> strides = {1, 1, 1, 1};
-    auto conv = ops::DepthwiseConv2dNative(s.WithOpName("depthwise_conv"),
-                                           input, filter, strides, "SAME");
-    auto bias_add = ops::BiasAdd(s.WithOpName("bias_add"), conv, bias);
-
-    ops::Identity fetch = [&]() -> ops::Identity {
-      auto activate = s.WithOpName("activation");
-      auto fetch = s.WithOpName("fetch");
-
-      if (activation == "Relu") {
-        return ops::Identity(fetch, ops::Relu(activate, bias_add));
-      } else if (activation == "Relu6") {
-        return ops::Identity(fetch, ops::Relu6(activate, bias_add));
-      } else if (activation == "Elu") {
-        return ops::Identity(fetch, ops::Elu(activate, bias_add));
-      }
-
-      DCHECK(activation == "None");
-      return ops::Identity(fetch, bias_add);
-    }();
-
-    auto input_t = GenerateRandomTensor<DT_FLOAT>({8, 32, 32, 3});
-    auto filter_t = GenerateRandomTensor<DT_FLOAT>({1, 1, 3, 1});
-    auto bias_t = GenerateRandomTensor<DT_FLOAT>({3});
-
-    GrapplerItem item;
-    item.fetch = {"fetch"};
-    item.feed = {{"input", input_t}, {"filter", filter_t}, {"bias", bias_t}};
-    TF_CHECK_OK(s.ToGraphDef(&item.graph));
-
-    // Place all nodes on CPU.
-    for (int i = 0; i < item.graph.node_size(); ++i) {
-      item.graph.mutable_node(i)->set_device("/device:CPU:0");
-    }
-
-    Remapper optimizer(RewriterConfig::ON);
-    GraphDef output;
-    TF_CHECK_OK(optimizer.Optimize(nullptr, item, &output));
-
-    int found = 0;
-    for (const NodeDef& node : output.node()) {
-      if (node.name() != "bias_add" && node.name() != "activation") continue;
-
-      EXPECT_EQ(node.op(), "_FusedDepthwiseConv2dNative");
-      ASSERT_EQ(node.input_size(), 3);
-      EXPECT_EQ(node.input(0), "input");
-      EXPECT_EQ(node.input(1), "filter");
-
-      EXPECT_EQ(node.attr().at("num_args").i(), 1);
-      EXPECT_EQ(node.input(2), "bias");
-
-      const auto fused_ops = node.attr().at("fused_ops").list().s();
-      if (node.name() == "bias_add") {
-        ASSERT_EQ(fused_ops.size(), 1);
-        EXPECT_EQ(fused_ops[0], "BiasAdd");
-        found++;
-      }
-      if (node.name() == "activation") {
-        ASSERT_EQ(fused_ops.size(), 2);
-        EXPECT_EQ(fused_ops[0], "BiasAdd");
-        EXPECT_EQ(fused_ops[1], activation);
-        found++;
-      }
-    }
-    EXPECT_EQ(found, 1);
-
-    auto tensors_expected = EvaluateNodes(item.graph, item.fetch, item.feed);
-    ASSERT_EQ(tensors_expected.size(), 1);
-    auto tensors = EvaluateNodes(output, item.fetch, item.feed);
-    ASSERT_EQ(tensors.size(), 1);
-    test::ExpectTensorNear<float>(tensors[0], tensors_expected[0], 1e-6);
+#define REGISTER_TEST(NAME, T, INPUT)                                         \
+  TEST_F(MklRemapperTest, NAME##_##T) {                                       \
+    using ::tensorflow::ops::Placeholder;                                     \
+                                                                              \
+    for (const string& activation : {"Relu", "Relu6", "Elu", "None"}) {       \
+      tensorflow::Scope s = tensorflow::Scope::NewRootScope();                \
+                                                                              \
+      auto input_shape = Placeholder::Shape({8, 32, 32, 3});                  \
+      auto filter_shape = Placeholder::Shape({1, 1, 3, 1});                   \
+      auto bias_shape = Placeholder::Shape({3});                              \
+                                                                              \
+      auto input = Placeholder(s.WithOpName("input"), DT_FLOAT, input_shape); \
+      auto filter =                                                           \
+          Placeholder(s.WithOpName("filter"), DT_FLOAT, filter_shape);        \
+      auto bias = Placeholder(s.WithOpName("bias"), DT_FLOAT, bias_shape);    \
+                                                                              \
+      std::vector<int> strides = {1, 1, 1, 1};                                \
+      auto conv = ops::DepthwiseConv2dNative(s.WithOpName("depthwise_conv"),  \
+                                             input, filter, strides, "SAME"); \
+      auto bias_add = ops::BiasAdd(s.WithOpName("bias_add"), conv, bias);     \
+                                                                              \
+      ops::Identity fetch = [&]() -> ops::Identity {                          \
+        auto activate = s.WithOpName("activation");                           \
+        auto fetch = s.WithOpName("fetch");                                   \
+                                                                              \
+        if (activation == "Relu") {                                           \
+          return ops::Identity(fetch, ops::Relu(activate, bias_add));         \
+        } else if (activation == "Relu6") {                                   \
+          return ops::Identity(fetch, ops::Relu6(activate, bias_add));        \
+        } else if (activation == "Elu") {                                     \
+          return ops::Identity(fetch, ops::Elu(activate, bias_add));          \
+        }                                                                     \
+                                                                              \
+        DCHECK(activation == "None");                                         \
+        return ops::Identity(fetch, bias_add);                                \
+      }();                                                                    \
+                                                                              \
+      auto input_t = GenerateRandomTensor<DT_FLOAT>({8, 32, 32, 3});          \
+      auto filter_t = GenerateRandomTensor<DT_FLOAT>({1, 1, 3, 1});           \
+      auto bias_t = GenerateRandomTensor<DT_FLOAT>({3});                      \
+                                                                              \
+      GrapplerItem item;                                                      \
+      item.fetch = {"fetch"};                                                 \
+      item.feed = {                                                           \
+          {"input", input_t}, {"filter", filter_t}, {"bias", bias_t}};        \
+      TF_CHECK_OK(s.ToGraphDef(&item.graph));                                 \
+                                                                              \
+      for (int i = 0; i < item.graph.node_size(); ++i) {                      \
+        item.graph.mutable_node(i)->set_device("/device:CPU:0");              \
+      }                                                                       \
+                                                                              \
+      Remapper optimizer(RewriterConfig::ON);                                 \
+      GraphDef output;                                                        \
+      TF_CHECK_OK(optimizer.Optimize(nullptr, item, &output));                \
+                                                                              \
+      int found = 0;                                                          \
+      for (const NodeDef& node : output.node()) {                             \
+        if (node.name() != "bias_add" && node.name() != "activation")         \
+          continue;                                                           \
+                                                                              \
+        EXPECT_EQ(node.op(), "_FusedDepthwiseConv2dNative");                  \
+        ASSERT_EQ(node.input_size(), 3);                                      \
+        EXPECT_EQ(node.input(0), "input");                                    \
+        EXPECT_EQ(node.input(1), "filter");                                   \
+                                                                              \
+        EXPECT_EQ(node.attr().at("num_args").i(), 1);                         \
+        EXPECT_EQ(node.input(2), "bias");                                     \
+                                                                              \
+        const auto fused_ops = node.attr().at("fused_ops").list().s();        \
+        if (node.name() == "bias_add") {                                      \
+          ASSERT_EQ(fused_ops.size(), 1);                                     \
+          EXPECT_EQ(fused_ops[0], "BiasAdd");                                 \
+          found++;                                                            \
+        }                                                                     \
+        if (node.name() == "activation") {                                    \
+          ASSERT_EQ(fused_ops.size(), 2);                                     \
+          EXPECT_EQ(fused_ops[0], "BiasAdd");                                 \
+          EXPECT_EQ(fused_ops[1], activation);                                \
+          found++;                                                            \
+        }                                                                     \
+      }                                                                       \
+      EXPECT_EQ(found, 1);                                                    \
+                                                                              \
+      auto tensors_expected =                                                 \
+          EvaluateNodes(item.graph, item.fetch, item.feed);                   \
+      ASSERT_EQ(tensors_expected.size(), 1);                                  \
+      auto tensors = EvaluateNodes(output, item.fetch, item.feed);            \
+      ASSERT_EQ(tensors.size(), 1);                                           \
+      test::ExpectTensorNear<float>(tensors[0], tensors_expected[0], 1e-6);   \
+    }                                                                         \
   }
-}
+REGISTER_TEST_ALL_TYPES(FuseDepthwiseConv2DWithBiasAndActivation);
+#undef REGISTER_TEST
 
 #ifdef ENABLE_MKLDNN_V1
 TEST_F(MklRemapperTest, FuseBatchNormWithRelu) {
diff --git a/tensorflow/core/kernels/BUILD b/tensorflow/core/kernels/BUILD
index 47b2fa44f57..fd6a8ab1cf6 100644
--- a/tensorflow/core/kernels/BUILD
+++ b/tensorflow/core/kernels/BUILD
@@ -6671,6 +6671,7 @@ filegroup(
         "spectrogram.h",
         "stateless_random_ops.h",
         "string_util.h",
+        "string_to_hash_bucket_op.h",
         "tensor_array.h",
         "tile_functor.h",
         "tile_ops_cpu_impl.h",
@@ -6735,6 +6736,7 @@ filegroup(
         "cwise_op_floor_mod.cc",
         "cwise_op_greater.cc",
         "cwise_op_greater_equal.cc",
+        "cwise_op_imag.cc",
         "cwise_op_invert.cc",
         "cwise_op_isfinite.cc",
         "cwise_op_isnan.cc",
@@ -6751,6 +6753,7 @@ filegroup(
         "cwise_op_mul_2.cc",
         "cwise_op_neg.cc",
         "cwise_op_pow.cc",
+        "cwise_op_real.cc",
         "cwise_op_reciprocal.cc",
         "cwise_op_right_shift.cc",
         "cwise_op_round.cc",
@@ -6902,6 +6905,7 @@ filegroup(
         "stateless_random_ops.cc",
         "string_join_op.cc",
         "string_util.cc",
+        "string_to_hash_bucket_op.cc",
         "summary_op.cc",
         "tensor_array.cc",
         "tensor_array_ops.cc",
diff --git a/tensorflow/core/kernels/aggregate_ops.cc b/tensorflow/core/kernels/aggregate_ops.cc
index 511a5f77a66..79062aee156 100644
--- a/tensorflow/core/kernels/aggregate_ops.cc
+++ b/tensorflow/core/kernels/aggregate_ops.cc
@@ -48,11 +48,10 @@ REGISTER_ADDN_CPU(Variant);
 #if (defined(GOOGLE_CUDA) && GOOGLE_CUDA) || \
     (defined(TENSORFLOW_USE_ROCM) && TENSORFLOW_USE_ROCM)
 #define REGISTER_ADDN_GPU(type) REGISTER_ADDN(type, GPU)
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_ADDN_GPU);
 TF_CALL_int64(REGISTER_ADDN_GPU);
-TF_CALL_complex64(REGISTER_ADDN_GPU);
-TF_CALL_complex128(REGISTER_ADDN_GPU);
 TF_CALL_variant(REGISTER_ADDN_GPU);
+TF_CALL_GPU_NUMBER_TYPES(REGISTER_ADDN_GPU);
+TF_CALL_COMPLEX_TYPES(REGISTER_ADDN_GPU);
 #undef REGISTER_ADDN_GPU
 
 // A special GPU kernel for int32.
diff --git a/tensorflow/core/kernels/aggregate_ops_gpu.cu.cc b/tensorflow/core/kernels/aggregate_ops_gpu.cu.cc
index 85bdc2447cd..c73323a795a 100644
--- a/tensorflow/core/kernels/aggregate_ops_gpu.cu.cc
+++ b/tensorflow/core/kernels/aggregate_ops_gpu.cu.cc
@@ -154,10 +154,9 @@ struct Add9Functor<GPUDevice, T> {
   template struct functor::Add8pFunctor<GPUDevice, type>; \
   template struct functor::Add9Functor<GPUDevice, type>;
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_FUNCTORS);
 TF_CALL_int64(REGISTER_FUNCTORS);
-TF_CALL_complex64(REGISTER_FUNCTORS);
-TF_CALL_complex128(REGISTER_FUNCTORS);
+TF_CALL_GPU_NUMBER_TYPES(REGISTER_FUNCTORS);
+TF_CALL_COMPLEX_TYPES(REGISTER_FUNCTORS);
 
 #undef REGISTER_FUNCTORS
 
diff --git a/tensorflow/core/kernels/batch_matmul_op_complex.cc b/tensorflow/core/kernels/batch_matmul_op_complex.cc
index 2cf163be0d4..bc36b95d6a1 100644
--- a/tensorflow/core/kernels/batch_matmul_op_complex.cc
+++ b/tensorflow/core/kernels/batch_matmul_op_complex.cc
@@ -17,12 +17,10 @@ limitations under the License.
 
 namespace tensorflow {
 
-TF_CALL_complex64(REGISTER_BATCH_MATMUL_CPU);
-TF_CALL_complex128(REGISTER_BATCH_MATMUL_CPU);
+TF_CALL_COMPLEX_TYPES(REGISTER_BATCH_MATMUL_CPU);
 
 #if GOOGLE_CUDA || TENSORFLOW_USE_ROCM
-TF_CALL_complex64(REGISTER_BATCH_MATMUL_GPU);
-TF_CALL_complex128(REGISTER_BATCH_MATMUL_GPU);
+TF_CALL_COMPLEX_TYPES(REGISTER_BATCH_MATMUL_GPU);
 #endif  // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
 
 }  // namespace tensorflow
diff --git a/tensorflow/core/kernels/cholesky_op.cc b/tensorflow/core/kernels/cholesky_op.cc
index 71e2604afce..ff8fd08f228 100644
--- a/tensorflow/core/kernels/cholesky_op.cc
+++ b/tensorflow/core/kernels/cholesky_op.cc
@@ -87,8 +87,7 @@ namespace functor {
   extern template struct MatrixBandPartFunctor<GPUDevice, T>;
 
 TF_CALL_GPU_NUMBER_TYPES(DECLARE_GPU_SPEC);
-TF_CALL_complex64(DECLARE_GPU_SPEC);
-TF_CALL_complex128(DECLARE_GPU_SPEC);
+TF_CALL_COMPLEX_TYPES(DECLARE_GPU_SPEC);
 }  // namespace functor
 
 template <class Scalar>
diff --git a/tensorflow/core/kernels/concat_lib.h b/tensorflow/core/kernels/concat_lib.h
index 1d6d219f7ba..35da7afe3f5 100644
--- a/tensorflow/core/kernels/concat_lib.h
+++ b/tensorflow/core/kernels/concat_lib.h
@@ -64,15 +64,12 @@ void ConcatGPU(
           inputs_flat,                                                        \
       Tensor* output, typename TTypes<T, 2>::Tensor* output_flat);
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER);
-TF_CALL_complex64(REGISTER);
-TF_CALL_complex128(REGISTER);
 TF_CALL_int32(REGISTER);  // Needed for TensorLists.
 TF_CALL_int64(REGISTER);
 TF_CALL_int16(REGISTER);
 TF_CALL_bfloat16(REGISTER);
-TF_CALL_bool(REGISTER);
 TF_CALL_uint8(REGISTER);
+TF_CALL_GPU_ALL_TYPES(REGISTER);
 #undef REGISTER
 #endif  // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
 
diff --git a/tensorflow/core/kernels/concat_lib_gpu.cc b/tensorflow/core/kernels/concat_lib_gpu.cc
index 154b1a31c3a..de029397847 100644
--- a/tensorflow/core/kernels/concat_lib_gpu.cc
+++ b/tensorflow/core/kernels/concat_lib_gpu.cc
@@ -98,15 +98,12 @@ void ConcatGPU(
           inputs_flat,                                                        \
       Tensor* output, typename TTypes<T, 2>::Tensor* output_flat);
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER);
-TF_CALL_complex64(REGISTER);
-TF_CALL_complex128(REGISTER);
 TF_CALL_int32(REGISTER);  // Needed for TensorLists.
 TF_CALL_int64(REGISTER);
 TF_CALL_int16(REGISTER);
 TF_CALL_bfloat16(REGISTER);
-TF_CALL_bool(REGISTER);
 TF_CALL_uint8(REGISTER);
+TF_CALL_GPU_ALL_TYPES(REGISTER);
 
 #undef REGISTER
 
diff --git a/tensorflow/core/kernels/concat_lib_gpu.h b/tensorflow/core/kernels/concat_lib_gpu.h
index 2db66a7c5a8..a83b6b63d9b 100644
--- a/tensorflow/core/kernels/concat_lib_gpu.h
+++ b/tensorflow/core/kernels/concat_lib_gpu.h
@@ -66,15 +66,12 @@ void ConcatGPUImpl(const Eigen::GpuDevice& d,
       const GpuDeviceArrayStruct<int64>& ptr_offsets, bool fixed_size,        \
       int split_size, typename TTypes<T, 2>::Matrix* output);
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER);
-TF_CALL_complex64(REGISTER);
-TF_CALL_complex128(REGISTER);
 TF_CALL_int32(REGISTER);  // Needed for TensorLists.
 TF_CALL_int64(REGISTER);
 TF_CALL_int16(REGISTER);
 TF_CALL_bfloat16(REGISTER);
-TF_CALL_bool(REGISTER);
 TF_CALL_uint8(REGISTER);
+TF_CALL_GPU_ALL_TYPES(REGISTER);
 #undef REGISTER
 
 }  // namespace tensorflow
diff --git a/tensorflow/core/kernels/concat_lib_gpu_impl.cu.cc b/tensorflow/core/kernels/concat_lib_gpu_impl.cu.cc
index 8f5458c9b56..859aca2f932 100644
--- a/tensorflow/core/kernels/concat_lib_gpu_impl.cu.cc
+++ b/tensorflow/core/kernels/concat_lib_gpu_impl.cu.cc
@@ -201,45 +201,33 @@ void ConcatGPUImpl(const Eigen::GpuDevice& gpu_device,
       const GpuDeviceArrayStruct<int64>& ptr_offsets, bool fixed_size, \
       int split_size, typename TTypes<T, 2>::Matrix* output);
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPUCONCAT32);
-TF_CALL_complex64(REGISTER_GPUCONCAT32);
-TF_CALL_complex128(REGISTER_GPUCONCAT32);
 TF_CALL_int32(REGISTER_GPUCONCAT32);  // Needed for TensorLists.
 TF_CALL_int64(REGISTER_GPUCONCAT32);
 TF_CALL_int16(REGISTER_GPUCONCAT32);
 TF_CALL_uint8(REGISTER_GPUCONCAT32);
-REGISTER_GPUCONCAT32(bfloat16);
-REGISTER_GPUCONCAT32(bool);
+TF_CALL_bfloat16(REGISTER_GPUCONCAT32);
+TF_CALL_GPU_ALL_TYPES(REGISTER_GPUCONCAT32);
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPUCONCAT64);
-TF_CALL_complex64(REGISTER_GPUCONCAT64);
-TF_CALL_complex128(REGISTER_GPUCONCAT64);
 TF_CALL_int32(REGISTER_GPUCONCAT64);  // Needed for TensorLists.
 TF_CALL_int64(REGISTER_GPUCONCAT64);
 TF_CALL_int16(REGISTER_GPUCONCAT64);
 TF_CALL_uint8(REGISTER_GPUCONCAT64);
-REGISTER_GPUCONCAT64(bfloat16);
-REGISTER_GPUCONCAT64(bool);
+TF_CALL_bfloat16(REGISTER_GPUCONCAT64);
+TF_CALL_GPU_ALL_TYPES(REGISTER_GPUCONCAT64);
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU32);
-TF_CALL_complex64(REGISTER_GPU32);
-TF_CALL_complex128(REGISTER_GPU32);
 TF_CALL_int32(REGISTER_GPU32);  // Needed for TensorLists.
 TF_CALL_int64(REGISTER_GPU32);
 TF_CALL_int16(REGISTER_GPU32);
 TF_CALL_uint8(REGISTER_GPU32);
-REGISTER_GPU32(bfloat16);
-REGISTER_GPU32(bool);
+TF_CALL_bfloat16(REGISTER_GPU32);
+TF_CALL_GPU_ALL_TYPES(REGISTER_GPU32);
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU64);
-TF_CALL_complex64(REGISTER_GPU64);
-TF_CALL_complex128(REGISTER_GPU64);
 TF_CALL_int32(REGISTER_GPU64);  // Needed for TensorLists.
 TF_CALL_int64(REGISTER_GPU64);
 TF_CALL_int16(REGISTER_GPU64);
 TF_CALL_uint8(REGISTER_GPU64);
-REGISTER_GPU64(bfloat16);
-REGISTER_GPU64(bool);
+TF_CALL_bfloat16(REGISTER_GPU64);
+TF_CALL_GPU_ALL_TYPES(REGISTER_GPU64);
 
 #undef REGISTER_GPUCONCAT32
 #undef REGISTER_GPUCONCAT64
diff --git a/tensorflow/core/kernels/concat_op.cc b/tensorflow/core/kernels/concat_op.cc
index 19ed267b441..be3e9a67c5f 100644
--- a/tensorflow/core/kernels/concat_op.cc
+++ b/tensorflow/core/kernels/concat_op.cc
@@ -227,13 +227,10 @@ REGISTER_CONCAT(uint64);
                               .HostMemory("axis"),       \
                           ConcatV2Op<GPUDevice, type>)
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
-REGISTER_GPU(bfloat16);
+TF_CALL_bfloat16(REGISTER_GPU);
 TF_CALL_uint8(REGISTER_GPU);
-TF_CALL_complex64(REGISTER_GPU);
-TF_CALL_complex128(REGISTER_GPU);
 TF_CALL_int64(REGISTER_GPU);
-REGISTER_GPU(bool);
+TF_CALL_GPU_ALL_TYPES(REGISTER_GPU);
 #undef REGISTER_GPU
 
 // A special GPU kernel for int32.
diff --git a/tensorflow/core/kernels/cubin_headers/BUILD b/tensorflow/core/kernels/cubin_headers/BUILD
index 1d9b98543f1..a7f810eeded 100644
--- a/tensorflow/core/kernels/cubin_headers/BUILD
+++ b/tensorflow/core/kernels/cubin_headers/BUILD
@@ -6,18 +6,8 @@ package(
     licenses = ["notice"],  # Apache 2.0
 )
 
-bias_add_kernel = """
-func @bias_add(%arg0: tensor<?x?xf99>,
-         %arg1: tensor<?xf99>) -> tensor<?x?xf99> {
-  %0 = "tf.BiasAdd"(%arg0, %arg1) { T = "tfdtype$DT_TYPE" }
-    : (tensor<?x?xf99>, tensor<?xf99>) -> tensor<?x?xf99>
-  return %0 : tensor<?x?xf99>
-}
-"""
-
 gen_kernel_library(
     name = "bias_add",
-    op = bias_add_kernel,
     same_shape = "0,2",
     tile_size = "16x16",
     types = [
@@ -27,17 +17,8 @@ gen_kernel_library(
     ],
 )
 
-relu_kernel = """
-func @relu(%arg0: tensor<?xf99>) -> tensor<?xf99> {
-  %0 = "tf.Relu"(%arg0) { T = "tfdtype$DT_TYPE" }
-    : (tensor<?xf99>) -> tensor<?xf99>
-  return %0 : tensor<?xf99>
-}
-"""
-
 gen_kernel_library(
     name = "relu",
-    op = relu_kernel,
     same_shape = "0,1",
     tile_size = "256",
     types = [
@@ -47,17 +28,8 @@ gen_kernel_library(
     ],
 )
 
-tanh_kernel = """
-func @tanh(%arg0: tensor<?xf99>) -> tensor<?xf99> {
-  %0 = "tf.Tanh"(%arg0) { T = "tfdtype$DT_TYPE" }
-    : (tensor<?xf99>) -> tensor<?xf99>
-  return %0 : tensor<?xf99>
-}
-"""
-
 gen_kernel_library(
     name = "tanh",
-    op = tanh_kernel,
     tile_size = "256",
     types = [
         "f32",
diff --git a/tensorflow/core/kernels/cubin_headers/bias_add.mlir.tmpl b/tensorflow/core/kernels/cubin_headers/bias_add.mlir.tmpl
new file mode 100644
index 00000000000..edbee639143
--- /dev/null
+++ b/tensorflow/core/kernels/cubin_headers/bias_add.mlir.tmpl
@@ -0,0 +1,6 @@
+func @bias_add(%arg0: tensor<?x?xf99>,
+         %arg1: tensor<?xf99>) -> tensor<?x?xf99> {
+  %0 = "tf.BiasAdd"(%arg0, %arg1) { T = "tfdtype$DT_TYPE" }
+    : (tensor<?x?xf99>, tensor<?xf99>) -> tensor<?x?xf99>
+  return %0 : tensor<?x?xf99>
+}
diff --git a/tensorflow/core/kernels/cubin_headers/build_defs.bzl b/tensorflow/core/kernels/cubin_headers/build_defs.bzl
index bd19d7edec3..ffeb06e6b36 100644
--- a/tensorflow/core/kernels/cubin_headers/build_defs.bzl
+++ b/tensorflow/core/kernels/cubin_headers/build_defs.bzl
@@ -27,13 +27,14 @@ def _gen_kernel_image_hdr_impl(ctx):
         filename = "%s.%s.cubin" % (name, arch)
         cubin = ctx.actions.declare_file(filename)
         ctx.actions.run(
+            inputs = [ctx.file.mlir_op],
             outputs = [cubin],
             executable = ctx.executable._tool,
             arguments = same_shape + [
                 "--tile_sizes=%s" % tile_sizes,
                 "--arch=%s" % arch.split("_")[1],
+                "--input=%s" % ctx.file.mlir_op.path,
                 "--output=%s" % cubin.path,
-                ctx.attr.op,
             ],
             mnemonic = "compile",
         )
@@ -70,7 +71,7 @@ _gen_kernel_image_hdr_rule = rule(
     implementation = _gen_kernel_image_hdr_impl,
     output_to_genfiles = True,
     attrs = {
-        "op": attr.string(mandatory = True),
+        "mlir_op": attr.label(mandatory = True, allow_single_file = True),
         "tile_size": attr.string(mandatory = True),
         "same_shape": attr.string(),
         "out": attr.output(mandatory = True),
@@ -87,12 +88,12 @@ _gen_kernel_image_hdr_rule = rule(
     },
 )
 
-def _gen_kernel_image_hdr(name, op, tile_size, tags = [], same_shape = None):
+def _gen_kernel_image_hdr(name, mlir_op, tile_size, tags = [], same_shape = None):
     """Generates a C header with fatbin data from a Tensorflow op."""
     if cuda_gpu_architectures():
         _gen_kernel_image_hdr_rule(
             name = name,
-            op = op,
+            mlir_op = mlir_op,
             tile_size = tile_size,
             same_shape = same_shape,
             out = "%s.h" % name,
@@ -101,17 +102,61 @@ def _gen_kernel_image_hdr(name, op, tile_size, tags = [], same_shape = None):
             tags = tags,
         )
 
-def gen_kernel_library(name, op, types, tile_size, tags = [], same_shape = None):
+def _gen_mlir_op_impl(ctx):
+    type_to_dtype = {
+        "f16": "DT_HALF",
+        "f32": "DT_FLOAT",
+        "f64": "DT_DOUBLE",
+    }
+    ctx.actions.run_shell(
+        inputs = [ctx.file.template],
+        outputs = [ctx.outputs.out],
+        command = "cat %s | sed s/f99/%s/g | sed s/DT_DTYPE/%s/g > %s" % (
+            ctx.file.template.path,
+            ctx.attr.type,
+            type_to_dtype[ctx.attr.type],
+            ctx.outputs.out.path,
+        ),
+    )
+
+_gen_mlir_op_rule = rule(
+    implementation = _gen_mlir_op_impl,
+    output_to_genfiles = True,
+    attrs = {
+        "template": attr.label(mandatory = True, allow_single_file = True),
+        "type": attr.string(mandatory = True),
+        "out": attr.output(mandatory = True),
+    },
+)
+
+def _gen_mlir_op(name, type):
+    _gen_mlir_op_rule(
+        name = "generate_{name}_{type}_mlir".format(name = name, type = type),
+        template = "{name}.mlir.tmpl".format(name = name),
+        type = type,
+        out = "{name}_{type}.mlir".format(name = name, type = type),
+    )
+
+def gen_kernel_library(name, types, tile_size, tags = [], same_shape = None):
+    """ Generate a library with kernels for a specific tensorflow op.
+
+    Args:
+      name: The name of the tensorflow op.
+      types: The types ("f16", "f32", "f64") for which a kernel should be generated.
+      tile_size: The tiling specification, e.g. "16x16".
+      tags: The tags which should be added to the library.
+      same_shape: The information about which shapes are the same, e.g. "0,1".
+    """
+
     if cuda_gpu_architectures():
-        type_to_dtype = {
-            "f16": "DT_HALF",
-            "f32": "DT_FLOAT",
-            "f64": "DT_DOUBLE",
-        }
         for type in types:
+            _gen_mlir_op(
+                name = name,
+                type = type,
+            )
             _gen_kernel_image_hdr(
                 name = "{name}_{type}_kernel".format(name = name, type = type),
-                op = op.replace("f99", type).replace("DT_TYPE", type_to_dtype[type]),
+                mlir_op = "{name}_{type}.mlir".format(name = name, type = type),
                 tile_size = tile_size,
                 tags = tags,
                 same_shape = same_shape,
diff --git a/tensorflow/core/kernels/cubin_headers/relu.mlir.tmpl b/tensorflow/core/kernels/cubin_headers/relu.mlir.tmpl
new file mode 100644
index 00000000000..1b761c3736e
--- /dev/null
+++ b/tensorflow/core/kernels/cubin_headers/relu.mlir.tmpl
@@ -0,0 +1,5 @@
+func @relu(%arg0: tensor<?xf99>) -> tensor<?xf99> {
+  %0 = "tf.Relu"(%arg0) { T = "tfdtype$DT_TYPE" }
+    : (tensor<?xf99>) -> tensor<?xf99>
+  return %0 : tensor<?xf99>
+}
diff --git a/tensorflow/core/kernels/cubin_headers/tanh.mlir.tmpl b/tensorflow/core/kernels/cubin_headers/tanh.mlir.tmpl
new file mode 100644
index 00000000000..bc044153c61
--- /dev/null
+++ b/tensorflow/core/kernels/cubin_headers/tanh.mlir.tmpl
@@ -0,0 +1,5 @@
+func @tanh(%arg0: tensor<?xf99>) -> tensor<?xf99> {
+  %0 = "tf.Tanh"(%arg0) { T = "tfdtype$DT_DTYPE" }
+    : (tensor<?xf99>) -> tensor<?xf99>
+  return %0 : tensor<?xf99>
+}
diff --git a/tensorflow/core/kernels/data/cache_dataset_ops.cc b/tensorflow/core/kernels/data/cache_dataset_ops.cc
index 9a1a4ee1ed3..b4d0f9e5ab3 100644
--- a/tensorflow/core/kernels/data/cache_dataset_ops.cc
+++ b/tensorflow/core/kernels/data/cache_dataset_ops.cc
@@ -130,12 +130,11 @@ class CacheDatasetOp::FileDatasetBase : public DatasetBase {
       } else {
         mode_ = Mode::write;
       }
-      InitializeIterator();
     }
 
     Status Initialize(IteratorContext* ctx) override {
       mutex_lock l(mu_);
-      return iterator_->Initialize(ctx);
+      return InitializeIterator(ctx);
     }
 
     Status GetNextInternal(IteratorContext* ctx,
@@ -180,8 +179,7 @@ class CacheDatasetOp::FileDatasetBase : public DatasetBase {
             << "mistake, please remove the above file and try running again.";
         mode_ = Mode::read;
       }
-      InitializeIterator();
-      TF_RETURN_IF_ERROR(iterator_->Initialize(ctx));
+      TF_RETURN_IF_ERROR(InitializeIterator(ctx));
       return RestoreInput(ctx, reader, iterator_);
     }
 
@@ -560,15 +558,16 @@ class CacheDatasetOp::FileDatasetBase : public DatasetBase {
       bool iterator_restored_ TF_GUARDED_BY(mu_);
     };  // FileReaderIterator
 
-    void InitializeIterator() TF_EXCLUSIVE_LOCKS_REQUIRED(mu_) {
-      // We intentionally use the same prefix for both `FileReaderIterator`
-      // and `FileWriterIterator`. Since at any time there will be at most
-      // one of them alive, there should be no conflicts. This allows both
-      // iterators to use a common key for `cur_index`. We leverage this
-      // in the corner case when this iterator is restored from an old
-      // checkpoint in `write` mode and the cache has been completely
-      // flushed to disk since then. In that case we simply build a
-      // `FileReaderIterator` and seek to the `cur_index`.
+    Status InitializeIterator(IteratorContext* ctx)
+        TF_EXCLUSIVE_LOCKS_REQUIRED(mu_) {
+      // We intentionally use the same prefix for both `FileReaderIterator` and
+      // `FileWriterIterator`. Since at any time there will be at most one of
+      // them alive, there should be no conflicts. This allows both iterators to
+      // use a common key for `cur_index`. We leverage this in the corner case
+      // when this iterator is restored from an old checkpoint in `write` mode
+      // and the cache has been completely flushed to disk since then. In that
+      // case we simply build a `FileReaderIterator` and seek to the
+      // `cur_index`.
       switch (mode_) {
         case Mode::read:
           iterator_ =
@@ -580,6 +579,8 @@ class CacheDatasetOp::FileDatasetBase : public DatasetBase {
               absl::make_unique<FileWriterIterator>(FileWriterIterator::Params{
                   dataset(), strings::StrCat(prefix(), kImpl)});
       }
+      TF_RETURN_IF_ERROR(iterator_->InitializeBase(ctx, this));
+      return iterator_->Initialize(ctx);
     }
 
     mutex mu_;
@@ -741,22 +742,14 @@ class CacheDatasetOp::MemoryDatasetBase : public DatasetBase {
 
     Status Initialize(IteratorContext* ctx) override {
       mutex_lock l(mu_);
-      InitializeIterator();
-      return iterator_->Initialize(ctx);
+      return InitializeIterator(ctx);
     }
 
     Status GetNextInternal(IteratorContext* ctx,
                            std::vector<Tensor>* out_tensors,
                            bool* end_of_sequence) override {
       mutex_lock l(mu_);
-      // TODO(b/154341936): Explicitly stopping and starting this iterator
-      // should not be necessary, but the `kImpl` added to the prefix passed
-      // to `iterator_` when it was created prevents the model from identifying
-      // this iterator as the output of `iterator_`.
-      RecordStop(ctx);
-      Status s = iterator_->GetNext(ctx, out_tensors, end_of_sequence);
-      RecordStart(ctx);
-      return s;
+      return iterator_->GetNext(ctx, out_tensors, end_of_sequence);
     }
 
    protected:
@@ -789,8 +782,7 @@ class CacheDatasetOp::MemoryDatasetBase : public DatasetBase {
                          [this](const string& s) { return full_name(s); }));
         cache_->Complete(std::move(temp_cache));
       }
-      InitializeIterator();
-      TF_RETURN_IF_ERROR(iterator_->Initialize(ctx));
+      TF_RETURN_IF_ERROR(InitializeIterator(ctx));
       return RestoreInput(ctx, reader, iterator_);
     }
 
@@ -946,7 +938,8 @@ class CacheDatasetOp::MemoryDatasetBase : public DatasetBase {
       size_t index_ TF_GUARDED_BY(mu_);
     };  // MemoryReaderIterator
 
-    void InitializeIterator() TF_EXCLUSIVE_LOCKS_REQUIRED(mu_) {
+    Status InitializeIterator(IteratorContext* ctx)
+        TF_EXCLUSIVE_LOCKS_REQUIRED(mu_) {
       if (cache_->IsCompleted()) {
         iterator_ = absl::make_unique<MemoryReaderIterator>(
             MemoryReaderIterator::Params{dataset(),
@@ -958,6 +951,8 @@ class CacheDatasetOp::MemoryDatasetBase : public DatasetBase {
                                          strings::StrCat(prefix(), kImpl)},
             cache_);
       }
+      TF_RETURN_IF_ERROR(iterator_->InitializeBase(ctx, this));
+      return iterator_->Initialize(ctx);
     }
 
     mutex mu_;
diff --git a/tensorflow/core/kernels/data/dataset_utils.cc b/tensorflow/core/kernels/data/dataset_utils.cc
index 15d6438bd02..3161004b7ab 100644
--- a/tensorflow/core/kernels/data/dataset_utils.cc
+++ b/tensorflow/core/kernels/data/dataset_utils.cc
@@ -455,6 +455,16 @@ Status RegisterCancellationCallback(CancellationManager* cancellation_manager,
   return Status::OK();
 }
 
+Status VerifyTypeMatch(const DataType& expected, const DataType& received,
+                       int index) {
+  if (expected != received) {
+    return errors::InvalidArgument("Data type mismatch at component ", index,
+                                   ": expected ", DataTypeString(expected),
+                                   " but got ", DataTypeString(received), ".");
+  }
+  return Status::OK();
+}
+
 Status VerifyTypesMatch(const DataTypeVector& expected,
                         const DataTypeVector& received) {
   if (expected.size() != received.size()) {
@@ -463,12 +473,30 @@ Status VerifyTypesMatch(const DataTypeVector& expected,
         " types but got ", received.size(), ".");
   }
   for (size_t i = 0; i < expected.size(); ++i) {
-    if (expected[i] != received[i]) {
-      return errors::InvalidArgument("Data type mismatch at component ", i,
-                                     ": expected ", DataTypeString(expected[i]),
-                                     " but got ", DataTypeString(received[i]),
-                                     ".");
-    }
+    TF_RETURN_IF_ERROR(VerifyTypeMatch(expected[i], received[i], i));
+  }
+  return Status::OK();
+}
+
+Status VerifyTypesMatch(const DataTypeVector& expected,
+                        const std::vector<Tensor>& received) {
+  if (expected.size() != received.size()) {
+    return errors::InvalidArgument(
+        "Number of components does not match: expected ", expected.size(),
+        " types but got ", received.size(), ".");
+  }
+  for (size_t i = 0; i < expected.size(); ++i) {
+    TF_RETURN_IF_ERROR(VerifyTypeMatch(expected[i], received[i].dtype(), i));
+  }
+  return Status::OK();
+}
+
+Status VerifyShapeCompatible(const PartialTensorShape& expected,
+                             const PartialTensorShape& received, int index) {
+  if (!expected.IsCompatibleWith(received)) {
+    return errors::InvalidArgument("Incompatible shapes at component ", index,
+                                   ": expected ", expected.DebugString(),
+                                   " but got ", received.DebugString(), ".");
   }
   return Status::OK();
 }
@@ -481,12 +509,22 @@ Status VerifyShapesCompatible(const std::vector<PartialTensorShape>& expected,
         " shapes but got ", received.size(), ".");
   }
   for (size_t i = 0; i < expected.size(); ++i) {
-    if (!expected[i].IsCompatibleWith(received[i])) {
-      return errors::InvalidArgument("Incompatible shapes at component ", i,
-                                     ": expected ", expected[i].DebugString(),
-                                     " but got ", received[i].DebugString(),
-                                     ".");
-    }
+    TF_RETURN_IF_ERROR(VerifyShapeCompatible(expected[i], received[i], i));
+  }
+
+  return Status::OK();
+}
+
+Status VerifyShapesCompatible(const std::vector<PartialTensorShape>& expected,
+                              const std::vector<Tensor>& received) {
+  if (expected.size() != received.size()) {
+    return errors::InvalidArgument(
+        "Number of components does not match: expected ", expected.size(),
+        " shapes but got ", received.size(), ".");
+  }
+  for (size_t i = 0; i < expected.size(); ++i) {
+    TF_RETURN_IF_ERROR(
+        VerifyShapeCompatible(expected[i], received[i].shape(), i));
   }
 
   return Status::OK();
diff --git a/tensorflow/core/kernels/data/dataset_utils.h b/tensorflow/core/kernels/data/dataset_utils.h
index 70ca70176e8..ac087360fd0 100644
--- a/tensorflow/core/kernels/data/dataset_utils.h
+++ b/tensorflow/core/kernels/data/dataset_utils.h
@@ -94,11 +94,17 @@ Status RegisterCancellationCallback(CancellationManager* cancellation_manager,
 Status VerifyTypesMatch(const DataTypeVector& expected,
                         const DataTypeVector& received);
 
+Status VerifyTypesMatch(const DataTypeVector& expected,
+                        const std::vector<Tensor>& received);
+
 // Returns Status::OK() if `expected` and `received` shapes are compatible,
 // errors::InvalidArgument otherwise.
 Status VerifyShapesCompatible(const std::vector<PartialTensorShape>& expected,
                               const std::vector<PartialTensorShape>& received);
 
+Status VerifyShapesCompatible(const std::vector<PartialTensorShape>& expected,
+                              const std::vector<Tensor>& received);
+
 // Returns a stable hash of the subgraph rooted at the given node.
 //
 // NOTE: There is currently no guarantee that the hash of a subgraph will stay
diff --git a/tensorflow/core/kernels/data/experimental/snapshot_dataset_op.cc b/tensorflow/core/kernels/data/experimental/snapshot_dataset_op.cc
index f9a20f44b15..157c758bc20 100644
--- a/tensorflow/core/kernels/data/experimental/snapshot_dataset_op.cc
+++ b/tensorflow/core/kernels/data/experimental/snapshot_dataset_op.cc
@@ -65,8 +65,6 @@ namespace experimental {
 
 // ==== Snapshot Implementation ====
 
-namespace {
-
 /* The current snapshot on-disk layout is as follows:
  *   /user/specified/path/
  *     - graphhash1/
@@ -95,25 +93,6 @@ namespace {
  *       ...
  */
 
-constexpr const char* const kShardDirectorySuffix = ".shard";
-
-inline tstring HashDirectory(const tstring& path, const uint64 hash) {
-  return io::JoinPath(path, absl::StrFormat("%d", hash));
-}
-
-inline tstring RunDirectory(const tstring& hash_directory,
-                            const uint64 run_id) {
-  return io::JoinPath(hash_directory, absl::StrFormat("%d", run_id));
-}
-
-inline tstring SnapshotShardDirectory(const tstring& run_directory,
-                                      const int64 snapshot_index) {
-  return io::JoinPath(run_directory, absl::StrFormat("%08d%s", snapshot_index,
-                                                     kShardDirectorySuffix));
-}
-
-}  // namespace
-
 class SnapshotDatasetV2Op::Dataset : public DatasetBase {
  public:
   Dataset(OpKernelContext* ctx, const DatasetBase* input, uint64 hash,
@@ -248,118 +227,19 @@ class SnapshotDatasetV2Op::Dataset::Iterator::Writer
                          IteratorStateReader* reader) override;
 
  private:
-  struct BufferElement {
-    std::vector<Tensor> value;
-    bool end_of_sequence = false;
-  };
-
-  class WriterThread {
-   public:
-    explicit WriterThread(Writer* writer_iterator, Env* env, int64 file_index,
-                          const tstring& shard_directory,
-                          uint64 current_checkpoint_id) {
-      thread_ = absl::WrapUnique(env->StartThread(
-          ThreadOptions(), absl::StrCat("snapshot_writer_thread_", file_index),
-          [this, writer_iterator, env, shard_directory, current_checkpoint_id] {
-            RunWriterThread(writer_iterator, env, shard_directory,
-                            current_checkpoint_id);
-          }));
-    }
-
-    void EnqueueTensors(const std::vector<Tensor>& tensors)
-        TF_LOCKS_EXCLUDED(deque_mu_) {
-      // Copy the Tensor to the deque for writing.
-      mutex_lock l(deque_mu_);
-      BufferElement be;
-      be.value = tensors;
-      deque_.push_back(std::move(be));
-    }
-
-    void DequeueTensors(BufferElement* be) TF_LOCKS_EXCLUDED(deque_mu_) {
-      mutex_lock l(deque_mu_);
-      deque_mu_.Await(
-          tensorflow::Condition(this, &WriterThread::DequeIsNotEmpty));
-      *be = deque_.front();
-      deque_.pop_front();
-    }
-
-    void StopThread() TF_LOCKS_EXCLUDED(deque_mu_) {
-      mutex_lock l(deque_mu_);
-      BufferElement be;
-      be.end_of_sequence = true;
-      deque_.push_back(std::move(be));
-    }
-
-    void RunWriterThread(Writer* writer_iterator, Env* env,
-                         const tstring& shard_directory,
-                         uint64 current_checkpoint_id) {
-      Status s = WriterThreadFn(writer_iterator, env, shard_directory,
-                                current_checkpoint_id);
-      if (!s.ok()) {
-        mutex_lock l(writer_iterator->mu_);
-        writer_iterator->writer_status_ = s;
-      }
-    }
-
-   private:
-    bool DequeIsNotEmpty() TF_EXCLUSIVE_LOCKS_REQUIRED(deque_mu_) {
-      return !deque_.empty();
-    }
-
-    Status WriterThreadFn(Writer* writer_iterator, Env* env,
-                          const tstring& shard_directory,
-                          uint64 current_checkpoint_id) {
-      std::unique_ptr<snapshot_util::Writer> writer;
-      TF_RETURN_IF_ERROR(env->RecursivelyCreateDir(shard_directory));
-
-      TF_RETURN_IF_ERROR(snapshot_util::Writer::Create(
-          env,
-          snapshot_util::GetCurrentCheckpointFile(shard_directory,
-                                                  current_checkpoint_id),
-          writer_iterator->dataset()->compression_, kSnapshotFileFormatVersion,
-          writer_iterator->dataset()->output_dtypes(), &writer));
-
-      while (true) {
-        BufferElement be;
-        DequeueTensors(&be);
-
-        if (be.end_of_sequence) {
-          TF_RETURN_IF_ERROR(writer->Close());
-          break;
-        }
-
-        TF_RETURN_IF_ERROR(writer->WriteTensors(be.value));
-      }
-      return Status::OK();
-    }
-
-    // If both the writer `mu_` and this `deque_mu_` needs to be acquired, the
-    // writer `mu_` must be acquired first.
-    mutex deque_mu_;
-
-    std::deque<BufferElement> deque_ TF_GUARDED_BY(deque_mu_);
-
-    // This has to be last. During destruction, we need to make sure that
-    // thread_ is destroyed first as the thread destructor blocks on thread
-    // completion. If there are other member variables after this, they may get
-    // destroyed first before the thread finishes, potentially causing the
-    // thread to access invalid memory.
-    std::unique_ptr<Thread> thread_;
-  };
-
-  Status GetShardIndex(std::vector<Tensor>* tensors, int64* shard_index)
-      TF_EXCLUSIVE_LOCKS_REQUIRED(mu_);
+  Status GetShardIndex(IteratorContext* ctx, const std::vector<Tensor>& tensors,
+                       int64* shard_index) TF_EXCLUSIVE_LOCKS_REQUIRED(mu_);
 
   Status WriteMetadataFile(Env* env, bool finalized)
       TF_EXCLUSIVE_LOCKS_REQUIRED(mu_);
 
-  void StopWriterThreads(bool mark_closed) TF_EXCLUSIVE_LOCKS_REQUIRED(mu_);
+  void SignalEOF(bool mark_closed) TF_EXCLUSIVE_LOCKS_REQUIRED(mu_);
 
   mutex mu_;
   std::unique_ptr<IteratorBase> input_impl_ TF_GUARDED_BY(mu_);
 
-  absl::flat_hash_map<int64, std::unique_ptr<WriterThread>> writer_threads_
-      TF_GUARDED_BY(mu_);
+  absl::flat_hash_map<int64, std::unique_ptr<snapshot_util::AsyncWriter>>
+      writers_ TF_GUARDED_BY(mu_);
   Status writer_status_ TF_GUARDED_BY(mu_);
   bool writers_closed_ TF_GUARDED_BY(mu_);
 
@@ -495,7 +375,8 @@ Status SnapshotDatasetV2Op::Dataset::AsGraphDefInternal(
 SnapshotDatasetV2Op::Dataset::Iterator::Iterator(const Params& params)
     : DatasetIterator<Dataset>(params),
       index_(0),
-      hash_dir_(HashDirectory(dataset()->path_, dataset()->hash_)) {}
+      hash_dir_(
+          snapshot_util::HashDirectory(dataset()->path_, dataset()->hash_)) {}
 
 Status SnapshotDatasetV2Op::Dataset::Iterator::Initialize(
     IteratorContext* ctx) {
@@ -536,16 +417,8 @@ Status SnapshotDatasetV2Op::Dataset::Iterator::GetNextInternal(
   if (iterator_ == nullptr) {
     TF_RETURN_IF_ERROR(InitializeIterator(ctx, nullptr));
   }
-  // TODO(b/154341936): Explicitly stopping and starting this iterator
-  // should not be necessary, but the additional
-  // `{Reader,Writer,Passthrough}::kIteratorName` added to the prefix passed to
-  // `iterator_` when it was created prevents the model from identifying this
-  // iterator as the output of `iterator_`.
-  RecordStop(ctx);
-  Status s = iterator_->GetNext(ctx, out_tensors, end_of_sequence);
   index_++;
-  RecordStart(ctx);
-  return s;
+  return iterator_->GetNext(ctx, out_tensors, end_of_sequence);
 }
 
 Status SnapshotDatasetV2Op::Dataset::Iterator::InitializeIterator(
@@ -565,8 +438,8 @@ Status SnapshotDatasetV2Op::Dataset::Iterator::InitializeIterator(
 
     experimental::SnapshotMetadataRecord metadata;
     bool file_exists;
-    TF_RETURN_IF_ERROR(
-        snapshot_util::ReadMetadataFile(hash_dir_, &metadata, &file_exists));
+    TF_RETURN_IF_ERROR(snapshot_util::ReadMetadataFile(
+        ctx->env(), hash_dir_, &metadata, &file_exists));
     if (!file_exists) {
       return errors::DataLoss("Snapshot metadata file in ", hash_dir_,
                               " does not exist any more.");
@@ -581,8 +454,8 @@ Status SnapshotDatasetV2Op::Dataset::Iterator::InitializeIterator(
   } else {
     experimental::SnapshotMetadataRecord metadata;
     bool file_exists;
-    TF_RETURN_IF_ERROR(
-        snapshot_util::ReadMetadataFile(hash_dir_, &metadata, &file_exists));
+    TF_RETURN_IF_ERROR(snapshot_util::ReadMetadataFile(
+        ctx->env(), hash_dir_, &metadata, &file_exists));
 
     // `pending_snapshot_expiry_seconds` is a legacy option where we would not
     // write snapshots that we think were still on-going. We decided that this
@@ -611,6 +484,7 @@ Status SnapshotDatasetV2Op::Dataset::Iterator::InitializeIterator(
           dataset(), absl::StrCat(prefix(), Passthrough::kIteratorName)});
       break;
   }
+  TF_RETURN_IF_ERROR(iterator_->InitializeBase(ctx, this));
   return iterator_->Initialize(ctx);
 }
 
@@ -627,25 +501,27 @@ Status SnapshotDatasetV2Op::Dataset::Iterator::Reader::Initialize(
   TF_RETURN_IF_ERROR(
       dataset()->reader_func_->Instantiate(ctx, &instantiated_reader_func_));
 
-  tstring hash_dir = HashDirectory(dataset()->path_, dataset()->hash_);
+  auto hash_dir =
+      snapshot_util::HashDirectory(dataset()->path_, dataset()->hash_);
   bool metadata_file_exists;
   experimental::SnapshotMetadataRecord metadata;
-  TF_RETURN_IF_ERROR(snapshot_util::ReadMetadataFile(hash_dir, &metadata,
-                                                     &metadata_file_exists));
+  TF_RETURN_IF_ERROR(snapshot_util::ReadMetadataFile(
+      ctx->env(), hash_dir, &metadata, &metadata_file_exists));
 
-  auto run_dir = io::JoinPath(hash_dir, metadata.run_id());
+  auto run_dir = snapshot_util::RunDirectory(hash_dir, metadata.run_id());
 
   std::vector<std::string> snapshot_shard_dirs;
   TF_RETURN_IF_ERROR(ctx->env()->GetMatchingPaths(
-      io::JoinPath(run_dir,
-                   absl::StrFormat("%s%s", "*", kShardDirectorySuffix)),
+      io::JoinPath(
+          run_dir,
+          absl::StrFormat("%s%s", "*", snapshot_util::kShardDirectorySuffix)),
       &snapshot_shard_dirs));
   std::sort(snapshot_shard_dirs.begin(), snapshot_shard_dirs.end());
 
   DatasetBase* dataset_of_snapshot_files;
   TF_RETURN_IF_ERROR(snapshot_util::Reader::MakeNestedDataset(
       ctx->env(), snapshot_shard_dirs, dataset()->compression_,
-      kSnapshotFileFormatVersion, dataset()->output_dtypes(),
+      metadata.version(), dataset()->output_dtypes(),
       dataset()->output_shapes(), start_index_, &dataset_of_snapshot_files));
 
   Tensor input_dataset_tensor(DT_VARIANT, TensorShape({}));
@@ -698,18 +574,18 @@ SnapshotDatasetV2Op::Dataset::Iterator::Writer::Writer(const Params& params)
 
 SnapshotDatasetV2Op::Dataset::Iterator::Writer::~Writer() {
   mutex_lock l(mu_);
-  StopWriterThreads(true);
+  SignalEOF(true);
 }
 
-void SnapshotDatasetV2Op::Dataset::Iterator::Writer::StopWriterThreads(
-    bool mark_closed) TF_EXCLUSIVE_LOCKS_REQUIRED(mu_) {
+void SnapshotDatasetV2Op::Dataset::Iterator::Writer::SignalEOF(bool mark_closed)
+    TF_EXCLUSIVE_LOCKS_REQUIRED(mu_) {
   if (!writers_closed_) {
     // Push the end of sequence signal to each of the threads to close files.
-    for (auto& writer_thread : writer_threads_) {
-      writer_thread.second->StopThread();
+    for (auto& writer : writers_) {
+      writer.second->SignalEOF();
     }
 
-    writer_threads_.clear();
+    writers_.clear();
     writers_closed_ = mark_closed;
   }
 }
@@ -722,15 +598,15 @@ Status SnapshotDatasetV2Op::Dataset::Iterator::Writer::WriteMetadataFile(
   metadata.set_creation_timestamp(EnvTime::NowMicros());
   metadata.set_graph_hash(absl::StrFormat("%d", dataset()->hash_));
   metadata.set_run_id(absl::StrFormat("%d", run_id_));
-  metadata.set_version(kSnapshotFileFormatVersion);
+  metadata.set_version(kFileFormatVersion);
   for (const auto& output_dtype : dataset()->output_dtypes()) {
     metadata.add_dtype(output_dtype);
   }
   metadata.set_finalized(finalized);
-  tstring hash_directory = HashDirectory(dataset()->path_, dataset()->hash_);
+  tstring hash_directory =
+      snapshot_util::HashDirectory(dataset()->path_, dataset()->hash_);
 
-  TF_RETURN_IF_ERROR(env->RecursivelyCreateDir(hash_directory));
-  return snapshot_util::WriteMetadataFile(hash_directory, &metadata);
+  return snapshot_util::WriteMetadataFile(env, hash_directory, &metadata);
 }
 
 Status SnapshotDatasetV2Op::Dataset::Iterator::Writer::Initialize(
@@ -744,12 +620,13 @@ Status SnapshotDatasetV2Op::Dataset::Iterator::Writer::Initialize(
 }
 
 Status SnapshotDatasetV2Op::Dataset::Iterator::Writer::GetShardIndex(
-    std::vector<Tensor>* tensors, int64* shard_index) {
+    IteratorContext* ctx, const std::vector<Tensor>& tensors,
+    int64* shard_index) {
   std::vector<Tensor> output_tensors;
 
   // Run the shard function
-  TF_RETURN_IF_ERROR(
-      instantiated_shard_func_->RunInstantiated(*tensors, &output_tensors));
+  TF_RETURN_IF_ERROR(instantiated_shard_func_->RunWithBorrowedArgs(
+      ctx, tensors, &output_tensors));
 
   if (output_tensors.size() != 1 || output_tensors[0].dtype() != DT_INT64 ||
       output_tensors[0].NumElements() != 1) {
@@ -765,7 +642,7 @@ Status SnapshotDatasetV2Op::Dataset::Iterator::Writer::GetNextInternal(
     IteratorContext* ctx, std::vector<Tensor>* out_tensors,
     bool* end_of_sequence) {
   *end_of_sequence = false;
-  WriterThread* current_writer_thread;
+  snapshot_util::AsyncWriter* current_writer;
 
   {
     std::vector<Tensor> output_tensors;
@@ -779,8 +656,9 @@ Status SnapshotDatasetV2Op::Dataset::Iterator::Writer::GetNextInternal(
       run_id_ = random::New64();
 
       // Creates the run directory.
-      run_dir_ = RunDirectory(HashDirectory(dataset()->path_, dataset()->hash_),
-                              run_id_);
+      run_dir_ = snapshot_util::RunDirectory(
+          snapshot_util::HashDirectory(dataset()->path_, dataset()->hash_),
+          run_id_);
       TF_RETURN_IF_ERROR(ctx->env()->RecursivelyCreateDir(run_dir_));
       TF_RETURN_IF_ERROR(WriteMetadataFile(ctx->env(), /*finalized=*/false));
     }
@@ -795,27 +673,33 @@ Status SnapshotDatasetV2Op::Dataset::Iterator::Writer::GetNextInternal(
 
     // Finalize metadata file when we are at the end of the iterator.
     if (*end_of_sequence) {
-      StopWriterThreads(/*mark_closed=*/true);
+      SignalEOF(/*mark_closed=*/true);
       TF_RETURN_IF_ERROR(writer_status_);
       return WriteMetadataFile(ctx->env(), /*finalized=*/true);
     }
 
     int64 shard_index = 0;
-    TF_RETURN_IF_ERROR(GetShardIndex(out_tensors, &shard_index));
+    TF_RETURN_IF_ERROR(GetShardIndex(ctx, *out_tensors, &shard_index));
 
     // If the index does not exist, we will start a new thread.
-    if (writer_threads_.count(shard_index) == 0) {
-      const tstring snapshot_shard_directory =
-          SnapshotShardDirectory(run_dir_, shard_index);
-      auto thread_data = std::make_unique<WriterThread>(
-          this, ctx->env(), shard_index, snapshot_shard_directory,
-          current_checkpoint_id_);
-      writer_threads_.insert({shard_index, std::move(thread_data)});
+    if (writers_.count(shard_index) == 0) {
+      auto snapshot_shard_directory =
+          snapshot_util::ShardDirectory(run_dir_, shard_index);
+      auto writer = std::make_unique<snapshot_util::AsyncWriter>(
+          ctx->env(), shard_index, snapshot_shard_directory,
+          current_checkpoint_id_, dataset()->compression_, kFileFormatVersion,
+          dataset()->output_dtypes(), [this](Status s) {
+            if (!s.ok()) {
+              mutex_lock l(mu_);
+              writer_status_ = s;
+            }
+          });
+      writers_.insert({shard_index, std::move(writer)});
     }
-    current_writer_thread = writer_threads_[shard_index].get();
+    current_writer = writers_[shard_index].get();
   }
 
-  current_writer_thread->EnqueueTensors(*out_tensors);
+  current_writer->Write(*out_tensors);
   return Status::OK();
 }
 
@@ -827,11 +711,9 @@ Status SnapshotDatasetV2Op::Dataset::Iterator::Writer::SaveInternal(
   TF_RETURN_IF_ERROR(
       writer->WriteScalar(full_name(kCurrentCheckpointId),
                           static_cast<int64>(current_checkpoint_id_)));
-
-  StopWriterThreads(/*mark_closed=*/false);
-  writer_threads_.clear();
+  SignalEOF(/*mark_closed=*/false);
+  writers_.clear();
   current_checkpoint_id_++;
-
   return SaveInput(ctx, writer, input_impl_);
 }
 
@@ -846,8 +728,9 @@ Status SnapshotDatasetV2Op::Dataset::Iterator::Writer::RestoreInternal(
                                         &current_checkpoint_id));
 
   run_id_ = static_cast<uint64>(run_id_signed);
-  run_dir_ =
-      RunDirectory(HashDirectory(dataset()->path_, dataset()->hash_), run_id_);
+  run_dir_ = snapshot_util::RunDirectory(
+      snapshot_util::HashDirectory(dataset()->path_, dataset()->hash_),
+      run_id_);
   current_checkpoint_id_ = static_cast<uint64>(current_checkpoint_id);
 
   return RestoreInput(ctx, reader, input_impl_);
@@ -1063,7 +946,7 @@ class SnapshotDatasetOp : public UnaryDatasetOpKernel {
     OP_REQUIRES_OK(ctx, ComputeDatasetHash(graph_def, path, &hash));
 
     Status dump_status =
-        snapshot_util::DumpDatasetGraph(path, hash, &graph_def);
+        snapshot_util::DumpDatasetGraph(ctx->env(), path, hash, &graph_def);
     if (!dump_status.ok()) {
       LOG(WARNING) << "Unable to write graphdef to disk, error: "
                    << dump_status.ToString();
@@ -1251,7 +1134,7 @@ class SnapshotDatasetOp : public UnaryDatasetOpKernel {
           experimental::SnapshotMetadataRecord metadata;
           bool file_exists;
           TF_RETURN_IF_ERROR(snapshot_util::ReadMetadataFile(
-              hash_dir_, &metadata, &file_exists));
+              ctx->env(), hash_dir_, &metadata, &file_exists));
           TF_RETURN_IF_ERROR(snapshot_util::DetermineOpState(
               dataset()->mode_, file_exists, &metadata,
               dataset()->pending_snapshot_expiry_seconds_, &state_));
@@ -1291,8 +1174,8 @@ class SnapshotDatasetOp : public UnaryDatasetOpKernel {
         }
         experimental::SnapshotMetadataRecord metadata;
         bool file_exists;
-        TF_RETURN_IF_ERROR(snapshot_util::ReadMetadataFile(hash_dir_, &metadata,
-                                                           &file_exists));
+        TF_RETURN_IF_ERROR(snapshot_util::ReadMetadataFile(
+            ctx->env(), hash_dir_, &metadata, &file_exists));
         TF_RETURN_IF_ERROR(InitializeIterator(ctx, metadata));
         VLOG(2) << "Restoring Snapshot iterator: " << state_;
         return RestoreInput(ctx, reader, iterator_);
@@ -1352,6 +1235,7 @@ class SnapshotDatasetOp : public UnaryDatasetOpKernel {
                     dataset(), absl::StrCat(prefix(), "PassthroughImpl")});
             break;
         }
+        TF_RETURN_IF_ERROR(iterator_->InitializeBase(ctx, this));
         return iterator_->Initialize(ctx);
       }
 
@@ -1606,7 +1490,7 @@ class SnapshotDatasetOp : public UnaryDatasetOpKernel {
         Status ReadFile(Env* env, const string& filename) {
           std::unique_ptr<snapshot_util::Reader> reader;
           TF_RETURN_IF_ERROR(snapshot_util::Reader::Create(
-              Env::Default(), filename, dataset()->compression_, version_,
+              env, filename, dataset()->compression_, version_,
               dataset()->output_dtypes(), &reader));
           while (true) {
             // Wait for a slot in the buffer.
@@ -1819,8 +1703,7 @@ class SnapshotDatasetOp : public UnaryDatasetOpKernel {
               // If we're restoring then the directory already exists and we
               // don't want to overwrite the snapshot metadata file.
               if (!is_restored_) {
-                TF_RETURN_IF_ERROR(
-                    Env::Default()->RecursivelyCreateDir(run_dir_));
+                TF_RETURN_IF_ERROR(ctx->env()->RecursivelyCreateDir(run_dir_));
                 experimental::SnapshotMetadataRecord metadata;
                 metadata.set_creation_timestamp(EnvTime::NowMicros());
                 metadata.set_graph_hash(dataset()->graph_hash_);
@@ -1830,8 +1713,8 @@ class SnapshotDatasetOp : public UnaryDatasetOpKernel {
                   metadata.add_dtype(output_dtype);
                 }
                 metadata.set_finalized(false);
-                TF_RETURN_IF_ERROR(
-                    snapshot_util::WriteMetadataFile(hash_dir_, &metadata));
+                TF_RETURN_IF_ERROR(snapshot_util::WriteMetadataFile(
+                    ctx->env(), hash_dir_, &metadata));
               }
               for (int i = 0; i < dataset()->num_writer_threads_; ++i) {
                 ++num_active_threads_;
@@ -2075,11 +1958,6 @@ class SnapshotDatasetOp : public UnaryDatasetOpKernel {
         }
 
        private:
-        struct BufferElement {
-          std::vector<Tensor> value;
-          bool end_of_sequence;
-        };
-
         string GetSnapshotFilename() {
           mutex_lock l(mu_);
           string snapshot_data_filename = io::JoinPath(
@@ -2091,7 +1969,7 @@ class SnapshotDatasetOp : public UnaryDatasetOpKernel {
         }
 
         Status FillBuffer(IteratorContext* ctx) TF_LOCKS_EXCLUDED(mu_) {
-          BufferElement elem;
+          snapshot_util::ElementOrEOF elem;
           TF_RETURN_IF_ERROR(
               input_impl_->GetNext(ctx, &elem.value, &elem.end_of_sequence));
 
@@ -2125,16 +2003,16 @@ class SnapshotDatasetOp : public UnaryDatasetOpKernel {
                 "maximum size: ", dataset()->writer_buffer_size_);
           }
 
-          BufferElement elem_copy = next_elem_;
+          snapshot_util::ElementOrEOF elem_copy = next_elem_;
           buffer_.push_back(elem_copy);
           cond_var_.notify_all();
           return Status::OK();
         }
 
-        Status ProcessOneElement(int64* bytes_written,
+        Status ProcessOneElement(Env* env, int64* bytes_written,
                                  string* snapshot_data_filename,
                                  std::unique_ptr<snapshot_util::Writer>* writer,
-                                 bool* end_of_processing, Env* env) {
+                                 bool* end_of_processing) {
           profiler::TraceMe activity(
               [&]() {
                 return absl::StrCat(prefix(), kSeparator, kProcessOneElement);
@@ -2144,7 +2022,7 @@ class SnapshotDatasetOp : public UnaryDatasetOpKernel {
           *end_of_processing = false;
           bool produced_elem = false;
           bool snapshot_failed = false;
-          BufferElement elem;
+          snapshot_util::ElementOrEOF elem;
           {
             mutex_lock l(mu_);
             // Wait for buffer to not be empty.
@@ -2181,7 +2059,7 @@ class SnapshotDatasetOp : public UnaryDatasetOpKernel {
 
             bool should_close;
             TF_RETURN_IF_ERROR(
-                ShouldCloseWriter(*snapshot_data_filename, *bytes_written,
+                ShouldCloseWriter(env, *snapshot_data_filename, *bytes_written,
                                   (*writer).get(), &should_close));
             if (should_close) {
               // If we exceed the shard size, we get a new file and reset.
@@ -2204,12 +2082,12 @@ class SnapshotDatasetOp : public UnaryDatasetOpKernel {
               experimental::SnapshotMetadataRecord metadata;
               bool file_exists;
               TF_RETURN_IF_ERROR(snapshot_util::ReadMetadataFile(
-                  hash_dir_, &metadata, &file_exists));
+                  env, hash_dir_, &metadata, &file_exists));
 
               if (metadata.run_id() == run_id_) {
                 metadata.set_finalized(true);
-                TF_RETURN_IF_ERROR(
-                    snapshot_util::WriteMetadataFile(hash_dir_, &metadata));
+                TF_RETURN_IF_ERROR(snapshot_util::WriteMetadataFile(
+                    env, hash_dir_, &metadata));
               } else {
                 // TODO(frankchn): We lost the race, remove all snapshots.
               }
@@ -2246,8 +2124,8 @@ class SnapshotDatasetOp : public UnaryDatasetOpKernel {
           bool end_of_processing = false;
           while (!end_of_processing) {
             Status s =
-                ProcessOneElement(&bytes_written, &snapshot_data_filename,
-                                  &writer, &end_of_processing, env);
+                ProcessOneElement(env, &bytes_written, &snapshot_data_filename,
+                                  &writer, &end_of_processing);
             if (!s.ok()) {
               LOG(INFO) << "Error while writing snapshot data to disk: "
                         << s.ToString();
@@ -2261,7 +2139,8 @@ class SnapshotDatasetOp : public UnaryDatasetOpKernel {
           }
         }
 
-        Status ShouldCloseWriter(const string& filename, uint64 bytes_written,
+        Status ShouldCloseWriter(Env* env, const string& filename,
+                                 uint64 bytes_written,
                                  snapshot_util::Writer* writer,
                                  bool* should_close) {
           // If the compression ratio has been estimated, use it to decide
@@ -2286,7 +2165,7 @@ class SnapshotDatasetOp : public UnaryDatasetOpKernel {
           // Make sure that all bytes are written out.
           TF_RETURN_IF_ERROR(writer->Sync());
           uint64 file_size;
-          TF_RETURN_IF_ERROR(Env::Default()->GetFileSize(filename, &file_size));
+          TF_RETURN_IF_ERROR(env->GetFileSize(filename, &file_size));
           mutex_lock l(mu_);
           compression_ratio_ = static_cast<double>(bytes_written) /
                                static_cast<double>(file_size);
@@ -2306,7 +2185,7 @@ class SnapshotDatasetOp : public UnaryDatasetOpKernel {
         // 5. By the background threads when they finish.
         condition_variable cond_var_;
 
-        BufferElement next_elem_ TF_GUARDED_BY(mu_);
+        snapshot_util::ElementOrEOF next_elem_ TF_GUARDED_BY(mu_);
         std::unique_ptr<IteratorBase> input_impl_;
 
         const string hash_dir_;
@@ -2319,7 +2198,7 @@ class SnapshotDatasetOp : public UnaryDatasetOpKernel {
         int64 time_spent_micros_ TF_GUARDED_BY(mu_) = 0;
         int64 bytes_produced_ TF_GUARDED_BY(mu_) = 0;
 
-        std::deque<BufferElement> buffer_ TF_GUARDED_BY(mu_);
+        std::deque<snapshot_util::ElementOrEOF> buffer_ TF_GUARDED_BY(mu_);
         bool snapshot_failed_ TF_GUARDED_BY(mu_) = false;
         bool cancelled_ TF_GUARDED_BY(mu_) = false;
         bool first_call_ TF_GUARDED_BY(mu_) = true;
diff --git a/tensorflow/core/kernels/data/experimental/snapshot_dataset_op.h b/tensorflow/core/kernels/data/experimental/snapshot_dataset_op.h
index daab6d20e9b..c9182383f2d 100644
--- a/tensorflow/core/kernels/data/experimental/snapshot_dataset_op.h
+++ b/tensorflow/core/kernels/data/experimental/snapshot_dataset_op.h
@@ -39,8 +39,6 @@ namespace tensorflow {
 namespace data {
 namespace experimental {
 
-const int64 kSnapshotFileFormatVersion = 1;
-
 class SnapshotDatasetV2Op : public UnaryDatasetOpKernel {
  public:
   static constexpr const char* const kDatasetType = "Snapshot";
@@ -64,6 +62,8 @@ class SnapshotDatasetV2Op : public UnaryDatasetOpKernel {
                    DatasetBase** output) override;
 
  private:
+  static constexpr const int kFileFormatVersion = 2;
+
   class Dataset;
 
   const int graph_def_version_;
diff --git a/tensorflow/core/kernels/data/experimental/snapshot_util.cc b/tensorflow/core/kernels/data/experimental/snapshot_util.cc
index 97a86977e82..7cecb26e20e 100644
--- a/tensorflow/core/kernels/data/experimental/snapshot_util.cc
+++ b/tensorflow/core/kernels/data/experimental/snapshot_util.cc
@@ -49,10 +49,27 @@ namespace snapshot_util {
 /* static */ constexpr const int64
     CustomReader::kSnappyReaderOutputBufferSizeBytes;
 
-std::string GetCurrentCheckpointFile(const std::string& shard_directory,
-                                     const uint64 current_checkpoint_id) {
+std::string HashDirectory(const std::string& path, uint64 hash) {
+  return io::JoinPath(path, absl::StrFormat("%d", hash));
+}
+
+std::string RunDirectory(const std::string& hash_directory, uint64 run_id) {
+  return RunDirectory(hash_directory, absl::StrFormat("%d", run_id));
+}
+
+std::string RunDirectory(const std::string& hash_directory,
+                         const std::string& run_id) {
+  return io::JoinPath(hash_directory, run_id);
+}
+
+std::string ShardDirectory(const std::string& run_directory, int64 shard_id) {
+  return io::JoinPath(run_directory, absl::StrFormat("%08d%s", shard_id,
+                                                     kShardDirectorySuffix));
+}
+std::string GetCheckpointFileName(const std::string& shard_directory,
+                                  uint64 checkpoint_id) {
   return io::JoinPath(shard_directory,
-                      absl::StrFormat("%08d.snapshot", current_checkpoint_id));
+                      absl::StrFormat("%08d.snapshot", checkpoint_id));
 }
 
 Status Writer::Create(Env* env, const std::string& filename,
@@ -357,13 +374,6 @@ class Reader::Dataset : public DatasetBase {
   }
 
  private:
-  const std::string shard_dir_;
-  const std::string compression_;
-  const int64 version_;
-  const DataTypeVector dtypes_;
-  const std::vector<PartialTensorShape> shapes_;
-  const int64 start_index_;
-
   class Iterator : public DatasetIterator<Dataset> {
    public:
     explicit Iterator(const Params& params)
@@ -417,16 +427,9 @@ class Reader::Dataset : public DatasetBase {
     }
 
    private:
-    std::unique_ptr<Reader> reader_;
-
-    // Stores the id current checkpoint file that we are in the process of
-    // reading (e.g. if the file is currently 00000001.snapshot, then this will
-    // be 1).
-    uint64 current_checkpoint_id_;
-
     std::string GetCurrentFilename() {
-      return GetCurrentCheckpointFile(dataset()->shard_dir_,
-                                      current_checkpoint_id_);
+      return GetCheckpointFileName(dataset()->shard_dir_,
+                                   current_checkpoint_id_);
     }
 
     Status AdvanceToNextFile(Env* env) {
@@ -435,7 +438,21 @@ class Reader::Dataset : public DatasetBase {
       return Reader::Create(env, GetCurrentFilename(), dataset()->compression_,
                             dataset()->version_, dataset()->dtypes_, &reader_);
     }
+
+    std::unique_ptr<Reader> reader_;
+
+    // Stores the id current checkpoint file that we are in the process of
+    // reading (e.g. if the file is currently 00000001.snapshot, then this will
+    // be 1).
+    uint64 current_checkpoint_id_;
   };
+
+  const std::string shard_dir_;
+  const std::string compression_;
+  const int64 version_;
+  const DataTypeVector dtypes_;
+  const std::vector<PartialTensorShape> shapes_;
+  const int64 start_index_;
 };
 
 class Reader::NestedDataset : public DatasetBase {
@@ -571,7 +588,7 @@ TFRecordReader::TFRecordReader(const std::string& filename,
       dtypes_(dtypes) {}
 
 Status TFRecordReader::Initialize(Env* env) {
-  TF_RETURN_IF_ERROR(Env::Default()->NewRandomAccessFile(filename_, &file_));
+  TF_RETURN_IF_ERROR(env->NewRandomAccessFile(filename_, &file_));
 
   record_reader_ = absl::make_unique<io::RecordReader>(
       file_.get(), io::RecordReaderOptions::CreateRecordReaderOptions(
@@ -607,7 +624,7 @@ CustomReader::CustomReader(const std::string& filename,
       dtypes_(dtypes) {}
 
 Status CustomReader::Initialize(Env* env) {
-  TF_RETURN_IF_ERROR(Env::Default()->NewRandomAccessFile(filename_, &file_));
+  TF_RETURN_IF_ERROR(env->NewRandomAccessFile(filename_, &file_));
   input_stream_ = std::make_unique<io::RandomAccessInputStream>(file_.get());
 
 #if defined(IS_SLIM_BUILD)
@@ -807,31 +824,31 @@ Status CustomReader::ReadRecord(absl::Cord* record) {
 }
 #endif
 
-Status WriteMetadataFile(const string& hash_dir,
+Status WriteMetadataFile(Env* env, const string& dir,
                          const experimental::SnapshotMetadataRecord* metadata) {
-  string metadata_filename = io::JoinPath(hash_dir, kMetadataFilename);
-  TF_RETURN_IF_ERROR(Env::Default()->RecursivelyCreateDir(hash_dir));
+  string metadata_filename = io::JoinPath(dir, kMetadataFilename);
+  TF_RETURN_IF_ERROR(env->RecursivelyCreateDir(dir));
   std::string tmp_filename =
       absl::StrCat(metadata_filename, "-tmp-", random::New64());
-  TF_RETURN_IF_ERROR(WriteBinaryProto(Env::Default(), tmp_filename, *metadata));
-  return Env::Default()->RenameFile(tmp_filename, metadata_filename);
+  TF_RETURN_IF_ERROR(WriteBinaryProto(env, tmp_filename, *metadata));
+  return env->RenameFile(tmp_filename, metadata_filename);
 }
 
-Status ReadMetadataFile(const string& hash_dir,
+Status ReadMetadataFile(Env* env, const string& dir,
                         experimental::SnapshotMetadataRecord* metadata,
                         bool* file_exists) {
-  string metadata_filename = io::JoinPath(hash_dir, kMetadataFilename);
-  Status s = Env::Default()->FileExists(metadata_filename);
+  string metadata_filename = io::JoinPath(dir, kMetadataFilename);
+  Status s = env->FileExists(metadata_filename);
   *file_exists = s.ok();
 
   if (*file_exists) {
-    return ReadBinaryProto(Env::Default(), metadata_filename, metadata);
+    return ReadBinaryProto(env, metadata_filename, metadata);
   } else {
     return Status::OK();
   }
 }
 
-Status DumpDatasetGraph(const std::string& path, uint64 hash,
+Status DumpDatasetGraph(Env* env, const std::string& path, uint64 hash,
                         const GraphDef* graph) {
   std::string hash_hex =
       strings::StrCat(strings::Hex(hash, strings::kZeroPad16));
@@ -839,8 +856,8 @@ Status DumpDatasetGraph(const std::string& path, uint64 hash,
       io::JoinPath(path, absl::StrCat(hash_hex, "-graph.pbtxt"));
 
   LOG(INFO) << "Graph hash is " << hash_hex << ", writing to " << graph_file;
-  TF_RETURN_IF_ERROR(Env::Default()->RecursivelyCreateDir(path));
-  return WriteTextProto(Env::Default(), graph_file, *graph);
+  TF_RETURN_IF_ERROR(env->RecursivelyCreateDir(path));
+  return WriteTextProto(env, graph_file, *graph);
 }
 
 Status DetermineOpState(const std::string& mode_string, bool file_exists,
@@ -893,6 +910,68 @@ Status DetermineOpState(const std::string& mode_string, bool file_exists,
   }
 }
 
+AsyncWriter::AsyncWriter(Env* env, int64 file_index,
+                         const std::string& shard_directory,
+                         uint64 checkpoint_id, const std::string& compression,
+                         int64 version, const DataTypeVector& output_types,
+                         std::function<void(Status)> done) {
+  thread_ = absl::WrapUnique(env->StartThread(
+      ThreadOptions(), absl::StrCat("writer_thread_", file_index),
+      [this, env, shard_directory, checkpoint_id, compression, version,
+       &output_types, done = std::move(done)] {
+        done(WriterThread(env, shard_directory, checkpoint_id, compression,
+                          version, output_types));
+      }));
+}
+
+void AsyncWriter::Write(const std::vector<Tensor>& tensors) {
+  mutex_lock l(mu_);
+  ElementOrEOF element;
+  element.value = tensors;
+  deque_.push_back(std::move(element));
+}
+
+void AsyncWriter::SignalEOF() {
+  mutex_lock l(mu_);
+  ElementOrEOF be;
+  be.end_of_sequence = true;
+  deque_.push_back(std::move(be));
+}
+
+void AsyncWriter::Consume(ElementOrEOF* be) {
+  mutex_lock l(mu_);
+  mu_.Await(tensorflow::Condition(this, &AsyncWriter::ElementAvailable));
+  *be = deque_.front();
+  deque_.pop_front();
+}
+
+bool AsyncWriter::ElementAvailable() { return !deque_.empty(); }
+
+Status AsyncWriter::WriterThread(Env* env, const std::string& shard_directory,
+                                 uint64 checkpoint_id,
+                                 const std::string& compression, int64 version,
+                                 DataTypeVector output_types) {
+  std::unique_ptr<snapshot_util::Writer> writer;
+  TF_RETURN_IF_ERROR(env->RecursivelyCreateDir(shard_directory));
+
+  TF_RETURN_IF_ERROR(snapshot_util::Writer::Create(
+      env, GetCheckpointFileName(shard_directory, checkpoint_id), compression,
+      version, std::move(output_types), &writer));
+
+  while (true) {
+    ElementOrEOF be;
+    Consume(&be);
+
+    if (be.end_of_sequence) {
+      TF_RETURN_IF_ERROR(writer->Close());
+      break;
+    }
+
+    TF_RETURN_IF_ERROR(writer->WriteTensors(be.value));
+  }
+  return Status::OK();
+}
+
 }  // namespace snapshot_util
 }  // namespace data
 }  // namespace tensorflow
diff --git a/tensorflow/core/kernels/data/experimental/snapshot_util.h b/tensorflow/core/kernels/data/experimental/snapshot_util.h
index 71dc242a1ed..5b228468861 100644
--- a/tensorflow/core/kernels/data/experimental/snapshot_util.h
+++ b/tensorflow/core/kernels/data/experimental/snapshot_util.h
@@ -25,6 +25,7 @@ limitations under the License.
 #include "tensorflow/core/lib/io/record_writer.h"
 #include "tensorflow/core/platform/env.h"
 #include "tensorflow/core/platform/file_system.h"
+#include "tensorflow/core/platform/path.h"
 #include "tensorflow/core/platform/status.h"
 
 namespace tensorflow {
@@ -48,11 +49,25 @@ constexpr char kModeAuto[] = "auto";
 constexpr char kModeWrite[] = "write";
 constexpr char kModeRead[] = "read";
 constexpr char kModePassthrough[] = "passthrough";
+constexpr char kShardDirectorySuffix[] = ".shard";
 
 enum Mode { READER = 0, WRITER = 1, PASSTHROUGH = 2 };
 
-std::string GetCurrentCheckpointFile(const std::string& shard_directory,
-                                     const uint64 current_checkpoint_id);
+// Returns the name of the "hash" directory for the given base path and hash ID.
+std::string HashDirectory(const std::string& path, uint64 hash);
+
+// Returns the name of the "run" directory for the given base path and run ID.
+std::string RunDirectory(const std::string& hash_directory, uint64 run_id);
+std::string RunDirectory(const std::string& hash_directory,
+                         const std::string& run_id);
+
+// Returns the name of the "shard" directory for the given base path and shard
+// ID.
+std::string ShardDirectory(const std::string& run_directory, int64 shard_id);
+
+// Returns the checkpoint file name for the given directory and checkpoint ID.
+std::string GetCheckpointFileName(const std::string& shard_directory,
+                                  const uint64 checkpoint_id);
 
 // This is a interface class that exposes snapshot writing functionality.
 class Writer {
@@ -82,8 +97,8 @@ class Writer {
 // Writes snapshots with the standard TFRecord file format.
 class TFRecordWriter : public Writer {
  public:
-  explicit TFRecordWriter(const std::string& filename,
-                          const std::string& compression_type);
+  TFRecordWriter(const std::string& filename,
+                 const std::string& compression_type);
 
   Status WriteTensors(const std::vector<Tensor>& tensors) override;
 
@@ -114,9 +129,8 @@ class CustomWriter : public Writer {
   static constexpr const char* const kWriteCord = "WriteCord";
   static constexpr const char* const kSeparator = "::";
 
-  explicit CustomWriter(const std::string& filename,
-                        const std::string& compression_type,
-                        const DataTypeVector& dtypes);
+  CustomWriter(const std::string& filename, const std::string& compression_type,
+               const DataTypeVector& dtypes);
 
   Status WriteTensors(const std::vector<Tensor>& tensors) override;
 
@@ -192,9 +206,8 @@ class Reader {
 // Reads snapshots previously written with `TFRecordWriter`.
 class TFRecordReader : public Reader {
  public:
-  explicit TFRecordReader(const std::string& filename,
-                          const string& compression_type,
-                          const DataTypeVector& dtypes);
+  TFRecordReader(const std::string& filename, const string& compression_type,
+                 const DataTypeVector& dtypes);
 
   Status ReadTensors(std::vector<Tensor>* read_tensors) override;
 
@@ -231,9 +244,8 @@ class CustomReader : public Reader {
   static constexpr const char* const kReadCord = "ReadCord";
   static constexpr const char* const kSeparator = "::";
 
-  explicit CustomReader(const std::string& filename,
-                        const string& compression_type, const int version,
-                        const DataTypeVector& dtypes);
+  CustomReader(const std::string& filename, const string& compression_type,
+               const int version, const DataTypeVector& dtypes);
 
   Status ReadTensors(std::vector<Tensor>* read_tensors) override;
 
@@ -268,14 +280,17 @@ class CustomReader : public Reader {
   std::vector<bool> simple_tensor_mask_;  // true for simple, false for complex.
 };
 
-Status WriteMetadataFile(const string& hash_dir,
+// Writes snapshot metadata to the given directory.
+Status WriteMetadataFile(Env* env, const string& dir,
                          const experimental::SnapshotMetadataRecord* metadata);
 
-Status ReadMetadataFile(const string& hash_dir,
+// Reads snapshot metadata from the given directory.
+Status ReadMetadataFile(Env* env, const string& dir,
                         experimental::SnapshotMetadataRecord* metadata,
                         bool* file_exists);
 
-Status DumpDatasetGraph(const std::string& path, uint64 hash,
+// Writes a dataset graph to the given directory.
+Status DumpDatasetGraph(Env* env, const std::string& path, uint64 hash,
                         const GraphDef* graph);
 
 Status DetermineOpState(const std::string& mode_string, bool file_exists,
@@ -283,6 +298,59 @@ Status DetermineOpState(const std::string& mode_string, bool file_exists,
                         const uint64 pending_snapshot_expiry_seconds,
                         Mode* mode);
 
+// Represents a dataset element or EOF.
+struct ElementOrEOF {
+  std::vector<Tensor> value;
+  bool end_of_sequence = false;
+};
+
+// AsyncWriter provides API for asynchronously writing dataset elements
+// (each represented as a vector of tensors) to a file.
+//
+// The expected use of this API is:
+//
+// std::unique_ptr<AsyncWriter> writer = absl_make_unique<AsyncWriter>(...);
+//
+// while (data_available()) {
+//   std::vector<Tensor> data = read_data()
+//   writer->Write(data);
+// }
+// writer->SignalEOF();
+// writer = nullptr;  // This will block until writes are flushed.
+class AsyncWriter {
+ public:
+  explicit AsyncWriter(Env* env, int64 file_index,
+                       const std::string& shard_directory, uint64 checkpoint_id,
+                       const std::string& compression, int64 version,
+                       const DataTypeVector& output_types,
+                       std::function<void(Status)> done);
+
+  // Writes the given tensors. The method is non-blocking and returns without
+  // waiting for the element to be written.
+  void Write(const std::vector<Tensor>& tensors) TF_LOCKS_EXCLUDED(mu_);
+
+  // Signals the end of input. The method is non-blocking and returns without
+  // waiting for the writer to be closed.
+  void SignalEOF() TF_LOCKS_EXCLUDED(mu_);
+
+ private:
+  void Consume(ElementOrEOF* be) TF_LOCKS_EXCLUDED(mu_);
+  bool ElementAvailable() TF_EXCLUSIVE_LOCKS_REQUIRED(mu_);
+  Status WriterThread(Env* env, const std::string& shard_directory,
+                      uint64 checkpoint_id, const std::string& compression,
+                      int64 version, DataTypeVector output_types);
+
+  mutex mu_;
+  std::deque<ElementOrEOF> deque_ TF_GUARDED_BY(mu_);
+
+  // This has to be last. During destruction, we need to make sure that the
+  // Thread object is destroyed first as its destructor blocks on thread
+  // completion. If there are other member variables after this, they may get
+  // destroyed first before the thread finishes, potentially causing the
+  // thread to access invalid memory.
+  std::unique_ptr<Thread> thread_;
+};
+
 }  // namespace snapshot_util
 }  // namespace data
 }  // namespace tensorflow
diff --git a/tensorflow/core/kernels/data/iterator_ops.cc b/tensorflow/core/kernels/data/iterator_ops.cc
index 9fb3c5fb46e..8dd7f4c364b 100644
--- a/tensorflow/core/kernels/data/iterator_ops.cc
+++ b/tensorflow/core/kernels/data/iterator_ops.cc
@@ -548,7 +548,10 @@ namespace {
 class ToSingleElementOp : public HybridAsyncOpKernel {
  public:
   explicit ToSingleElementOp(OpKernelConstruction* ctx)
-      : HybridAsyncOpKernel(ctx, "tf_data_to_single_element") {}
+      : HybridAsyncOpKernel(ctx, "tf_data_to_single_element") {
+    OP_REQUIRES_OK(ctx, ctx->GetAttr("output_types", &output_types_));
+    OP_REQUIRES_OK(ctx, ctx->GetAttr("output_shapes", &output_shapes_));
+  }
 
  protected:
   Status DoCompute(OpKernelContext* ctx) override {
@@ -580,8 +583,9 @@ class ToSingleElementOp : public HybridAsyncOpKernel {
     if (end_of_sequence) {
       return errors::InvalidArgument("Dataset was empty.");
     }
+    TF_RETURN_IF_ERROR(VerifyTypesMatch(output_types_, components));
+    TF_RETURN_IF_ERROR(VerifyShapesCompatible(output_shapes_, components));
     for (int i = 0; i < components.size(); ++i) {
-      // TODO(mrry): Check that the shapes match the shape attrs.
       ctx->set_output(i, components[i]);
     }
 
@@ -593,6 +597,10 @@ class ToSingleElementOp : public HybridAsyncOpKernel {
     }
     return Status::OK();
   }
+
+ private:
+  DataTypeVector output_types_;
+  std::vector<PartialTensorShape> output_shapes_;
 };
 
 class ReduceDatasetOp : public HybridAsyncOpKernel {
@@ -674,33 +682,9 @@ class ReduceDatasetOp : public HybridAsyncOpKernel {
       std::swap(reduce_func_output, state);
     }
 
-    if (state.size() != output_types_.size()) {
-      return errors::InvalidArgument(
-          "The number of result elements does not match "
-          "the size of output types: ",
-          state.size(), " vs. ", output_types_.size());
-    }
-    if (state.size() != output_shapes_.size()) {
-      return errors::InvalidArgument(
-          "The number of result elements does not match "
-          "the size of output shapes: ",
-          state.size(), " vs. ", output_shapes_.size());
-    }
+    TF_RETURN_IF_ERROR(VerifyTypesMatch(output_types_, state));
+    TF_RETURN_IF_ERROR(VerifyShapesCompatible(output_shapes_, state));
     for (size_t i = 0; i < state.size(); ++i) {
-      if (state[i].dtype() != output_types_[i]) {
-        return errors::InvalidArgument(
-            "The result does not match the expected type for "
-            "component ",
-            i, ". Expected: ", DataTypeString(output_types_[i]),
-            ". Actual: ", DataTypeString(state[i].dtype()), ".");
-      }
-      if (!output_shapes_[i].IsCompatibleWith(state[i].shape())) {
-        return errors::InvalidArgument(
-            "The result does not match the expected shape for "
-            "component ",
-            i, ". Expected: ", output_shapes_[i].DebugString(),
-            ". Actual: ", state[i].shape().DebugString(), ".");
-      }
       ctx->set_output(i, state[i]);
     }
     return Status::OK();
@@ -917,8 +901,9 @@ Status IteratorGetNextOp::DoCompute(OpKernelContext* ctx) {
   if (end_of_sequence) {
     return errors::OutOfRange("End of sequence");
   }
+  TF_RETURN_IF_ERROR(VerifyTypesMatch(output_types_, components));
+  TF_RETURN_IF_ERROR(VerifyShapesCompatible(output_shapes_, components));
   for (int i = 0; i < components.size(); ++i) {
-    // TODO(mrry): Check that the shapes match the shape attrs.
     ctx->set_output(i, components[i]);
   }
   return Status::OK();
diff --git a/tensorflow/core/kernels/data/iterator_ops.h b/tensorflow/core/kernels/data/iterator_ops.h
index 86db80ed75c..938b218bcb7 100644
--- a/tensorflow/core/kernels/data/iterator_ops.h
+++ b/tensorflow/core/kernels/data/iterator_ops.h
@@ -216,12 +216,19 @@ class MakeIteratorOp : public HybridAsyncOpKernel {
 class IteratorGetNextOp : public HybridAsyncOpKernel {
  public:
   explicit IteratorGetNextOp(OpKernelConstruction* ctx)
-      : HybridAsyncOpKernel(ctx, "tf_data_iterator_get_next") {}
+      : HybridAsyncOpKernel(ctx, "tf_data_iterator_get_next") {
+    OP_REQUIRES_OK(ctx, ctx->GetAttr("output_types", &output_types_));
+    OP_REQUIRES_OK(ctx, ctx->GetAttr("output_shapes", &output_shapes_));
+  }
 
   AsyncOpKernel* AsAsync() override;
 
  protected:
   Status DoCompute(OpKernelContext* ctx) override;
+
+ private:
+  DataTypeVector output_types_;
+  std::vector<PartialTensorShape> output_shapes_;
 };
 
 class DeleteIteratorOp : public HybridAsyncOpKernel {
diff --git a/tensorflow/core/kernels/data/optimize_dataset_op.cc b/tensorflow/core/kernels/data/optimize_dataset_op.cc
index d5d4e8c8f14..c976a8f7b08 100644
--- a/tensorflow/core/kernels/data/optimize_dataset_op.cc
+++ b/tensorflow/core/kernels/data/optimize_dataset_op.cc
@@ -60,6 +60,10 @@ void OptimizeDatasetOp::MakeDataset(OpKernelContext* ctx, DatasetBase* input,
   if (errors::IsDeadlineExceeded(s)) {
     // Ignore DeadlineExceeded as it implies that the attempted rewrite took too
     // long which should not prevent further computation.
+    LOG(WARNING) << s.ToString();
+
+    *output = input;
+    input->Ref();
     return;
   }
   OP_REQUIRES_OK(ctx, s);
diff --git a/tensorflow/core/kernels/data/shuffle_dataset_op.cc b/tensorflow/core/kernels/data/shuffle_dataset_op.cc
index 3e549246a95..7ac27ead6c2 100644
--- a/tensorflow/core/kernels/data/shuffle_dataset_op.cc
+++ b/tensorflow/core/kernels/data/shuffle_dataset_op.cc
@@ -72,7 +72,7 @@ constexpr char kEpochNumRandomSamples[] = "epoch_num_random_samples";
 constexpr char kShuffleDatasetV1[] = "ShuffleDataset";
 constexpr char kShuffleDatasetV2[] = "ShuffleDatasetV2";
 constexpr char kShuffleDatasetV3[] = "ShuffleDatasetV3";
-constexpr char kShuffleAndRepeatDatasetV1[] = "ShuffleAndRepeatDatasetV1";
+constexpr char kShuffleAndRepeatDatasetV1[] = "ShuffleAndRepeatDataset";
 constexpr char kShuffleAndRepeatDatasetV2[] = "ShuffleAndRepeatDatasetV2";
 
 ShuffleDatasetOpBase::ShuffleDatasetOpBase(OpKernelConstruction* ctx)
diff --git a/tensorflow/core/kernels/debug_ops.h b/tensorflow/core/kernels/debug_ops.h
index 42364e416ea..3fef822244d 100644
--- a/tensorflow/core/kernels/debug_ops.h
+++ b/tensorflow/core/kernels/debug_ops.h
@@ -428,13 +428,21 @@ class DebugIdentityV2Op : public OpKernel {
     OP_REQUIRES_OK(context, context->GetAttr("output_slot", &output_slot_));
     OP_REQUIRES_OK(context,
                    context->GetAttr("tensor_debug_mode", &tensor_debug_mode_));
+    if (context->HasAttr("circular_buffer_size")) {
+      OP_REQUIRES_OK(context, context->GetAttr("circular_buffer_size",
+                                               &circular_buffer_size_));
+    } else {
+      circular_buffer_size_ =
+          tfdbg::DebugEventsWriter::kDefaultCyclicBufferSize;
+    }
   }
 
   void Compute(OpKernelContext* context) override {
     const Tensor& tensor = context->input(0);
     for (const string& dump_root : dump_roots_) {
       tfdbg::DebugEventsWriter* debug_events_writer =
-          tfdbg::DebugEventsWriter::GetDebugEventsWriter(dump_root);
+          tfdbg::DebugEventsWriter::GetDebugEventsWriter(dump_root,
+                                                         circular_buffer_size_);
       OP_REQUIRES_OK(context, debug_events_writer->WriteGraphExecutionTrace(
                                   tfdbg_context_id_, device_name_, op_name_,
                                   output_slot_, tensor_debug_mode_, tensor));
@@ -449,6 +457,7 @@ class DebugIdentityV2Op : public OpKernel {
   string op_name_;
   int32 output_slot_;
   int32 tensor_debug_mode_;
+  int64 circular_buffer_size_;
 };
 
 typedef Eigen::ThreadPoolDevice CPUDevice;
diff --git a/tensorflow/core/kernels/diag_op.cc b/tensorflow/core/kernels/diag_op.cc
index 811d48af091..f34fc6c6be7 100644
--- a/tensorflow/core/kernels/diag_op.cc
+++ b/tensorflow/core/kernels/diag_op.cc
@@ -176,8 +176,7 @@ TF_CALL_double(REGISTER_DIAGOP);
 TF_CALL_float(REGISTER_DIAGOP);
 TF_CALL_int32(REGISTER_DIAGOP);
 TF_CALL_int64(REGISTER_DIAGOP);
-TF_CALL_complex64(REGISTER_DIAGOP);
-TF_CALL_complex128(REGISTER_DIAGOP);
+TF_CALL_COMPLEX_TYPES(REGISTER_DIAGOP);
 TF_CALL_half(REGISTER_DIAGOP);
 #undef REGISTER_DIAGOP
 
@@ -190,8 +189,7 @@ TF_CALL_double(REGISTER_DIAGPARTOP);
 TF_CALL_float(REGISTER_DIAGPARTOP);
 TF_CALL_int32(REGISTER_DIAGPARTOP);
 TF_CALL_int64(REGISTER_DIAGPARTOP);
-TF_CALL_complex64(REGISTER_DIAGPARTOP);
-TF_CALL_complex128(REGISTER_DIAGPARTOP);
+TF_CALL_COMPLEX_TYPES(REGISTER_DIAGPARTOP);
 TF_CALL_half(REGISTER_DIAGPARTOP);
 #undef REGISTER_DIAGPARTOP
 
@@ -217,8 +215,7 @@ TF_CALL_double(REGISTER_DIAGOP_GPU);
 TF_CALL_float(REGISTER_DIAGOP_GPU);
 TF_CALL_int32(REGISTER_DIAGOP_GPU);
 TF_CALL_int64(REGISTER_DIAGOP_GPU);
-TF_CALL_complex64(REGISTER_DIAGOP_GPU);
-TF_CALL_complex128(REGISTER_DIAGOP_GPU);
+TF_CALL_COMPLEX_TYPES(REGISTER_DIAGOP_GPU);
 TF_CALL_half(REGISTER_DIAGOP_GPU);
 #undef REGISTER_DIAGOP_GPU
 
@@ -242,8 +239,7 @@ TF_CALL_double(REGISTER_DIAGPARTOP_GPU);
 TF_CALL_float(REGISTER_DIAGPARTOP_GPU);
 TF_CALL_int32(REGISTER_DIAGPARTOP_GPU);
 TF_CALL_int64(REGISTER_DIAGPARTOP_GPU);
-TF_CALL_complex64(REGISTER_DIAGPARTOP_GPU);
-TF_CALL_complex128(REGISTER_DIAGPARTOP_GPU);
+TF_CALL_COMPLEX_TYPES(REGISTER_DIAGPARTOP_GPU);
 TF_CALL_half(REGISTER_DIAGPARTOP_GPU);
 #undef REGISTER_DIAGPARTOP_GPU
 
diff --git a/tensorflow/core/kernels/dynamic_partition_op_gpu.cu.cc b/tensorflow/core/kernels/dynamic_partition_op_gpu.cu.cc
index 98c2fb57833..7b64d9e8484 100644
--- a/tensorflow/core/kernels/dynamic_partition_op_gpu.cu.cc
+++ b/tensorflow/core/kernels/dynamic_partition_op_gpu.cu.cc
@@ -467,8 +467,7 @@ class DynamicPartitionOpGPU : public AsyncOpKernel {
       DynamicPartitionOpGPU<T>)
 
 TF_CALL_GPU_NUMBER_TYPES(REGISTER_DYNAMIC_PARTITION_GPU);
-TF_CALL_complex64(REGISTER_DYNAMIC_PARTITION_GPU);
-TF_CALL_complex128(REGISTER_DYNAMIC_PARTITION_GPU);
+TF_CALL_COMPLEX_TYPES(REGISTER_DYNAMIC_PARTITION_GPU);
 #undef REGISTER_DYNAMIC_PARTITION_GPU
 
 }  // namespace tensorflow
diff --git a/tensorflow/core/kernels/dynamic_stitch_op.cc b/tensorflow/core/kernels/dynamic_stitch_op.cc
index 86f9c3e4621..5f6b0357f95 100644
--- a/tensorflow/core/kernels/dynamic_stitch_op.cc
+++ b/tensorflow/core/kernels/dynamic_stitch_op.cc
@@ -147,11 +147,11 @@ void DynamicStitchGPUImpl(const Eigen::GpuDevice& gpu_device,
       const int32 first_dim_size,                                 \
       const GpuDeviceArrayStruct<int32>& input_indices,           \
       const GpuDeviceArrayStruct<const T*>& input_ptrs, T* output);
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
-TF_CALL_complex64(REGISTER_GPU);
-TF_CALL_complex128(REGISTER_GPU);
-TF_CALL_int64(REGISTER_GPU);
+
 TF_CALL_int32(REGISTER_GPU);
+TF_CALL_int64(REGISTER_GPU);
+TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
+TF_CALL_COMPLEX_TYPES(REGISTER_GPU);
 #undef REGISTER_GPU
 
 template <class T>
@@ -357,11 +357,10 @@ TF_CALL_QUANTIZED_TYPES(REGISTER_DYNAMIC_STITCH);
                               .HostMemory("merged"),     \
                           ParallelDynamicStitchOpCPU<type>)
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_DYNAMIC_STITCH_GPU);
-TF_CALL_complex64(REGISTER_DYNAMIC_STITCH_GPU);
-TF_CALL_complex128(REGISTER_DYNAMIC_STITCH_GPU);
-TF_CALL_int64(REGISTER_DYNAMIC_STITCH_GPU);
 TF_CALL_int32(REGISTER_DYNAMIC_STITCH_GPU);
+TF_CALL_int64(REGISTER_DYNAMIC_STITCH_GPU);
+TF_CALL_GPU_NUMBER_TYPES(REGISTER_DYNAMIC_STITCH_GPU);
+TF_CALL_COMPLEX_TYPES(REGISTER_DYNAMIC_STITCH_GPU);
 #undef REGISTER_DYNAMIC_STITCH_GPU
 
 #endif  // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
diff --git a/tensorflow/core/kernels/dynamic_stitch_op_gpu.cu.cc b/tensorflow/core/kernels/dynamic_stitch_op_gpu.cu.cc
index 22fed448c1f..c0a3df38b5d 100644
--- a/tensorflow/core/kernels/dynamic_stitch_op_gpu.cu.cc
+++ b/tensorflow/core/kernels/dynamic_stitch_op_gpu.cu.cc
@@ -70,11 +70,10 @@ void DynamicStitchGPUImpl(const Eigen::GpuDevice& gpu_device,
       const GpuDeviceArrayStruct<int32>& input_indices,           \
       const GpuDeviceArrayStruct<const T*>& input_ptrs, T* output);
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
-TF_CALL_complex64(REGISTER_GPU);
-TF_CALL_complex128(REGISTER_GPU);
+TF_CALL_int32(REGISTER_GPU);
 TF_CALL_int64(REGISTER_GPU);
-TF_CALL_int32(REGISTER_GPU)
+TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
+TF_CALL_COMPLEX_TYPES(REGISTER_GPU);
 
 #undef REGISTER_GPU
 
diff --git a/tensorflow/core/kernels/einsum_op_gpu.cu.cc b/tensorflow/core/kernels/einsum_op_gpu.cu.cc
index 36a97691297..2935b7fd02a 100644
--- a/tensorflow/core/kernels/einsum_op_gpu.cu.cc
+++ b/tensorflow/core/kernels/einsum_op_gpu.cu.cc
@@ -33,11 +33,8 @@ namespace tensorflow {
   DECLARE_GPU_SPECS_NDIM(T, 5); \
   DECLARE_GPU_SPECS_NDIM(T, 6);
 
-TF_CALL_half(DECLARE_GPU_SPECS);
-TF_CALL_float(DECLARE_GPU_SPECS);
-TF_CALL_double(DECLARE_GPU_SPECS);
-TF_CALL_complex64(DECLARE_GPU_SPECS);
-TF_CALL_complex128(DECLARE_GPU_SPECS);
+TF_CALL_GPU_NUMBER_TYPES(DECLARE_GPU_SPECS);
+TF_CALL_COMPLEX_TYPES(DECLARE_GPU_SPECS);
 
 #undef DECLARE_GPU_SPECS_NDIM
 #undef DECLARE_GPU_SPECS
diff --git a/tensorflow/core/kernels/gather_functor.cc b/tensorflow/core/kernels/gather_functor.cc
index e4f2182be3c..a0293951660 100644
--- a/tensorflow/core/kernels/gather_functor.cc
+++ b/tensorflow/core/kernels/gather_functor.cc
@@ -39,8 +39,7 @@ namespace functor {
 
 TF_CALL_int64(DECLARE_GPU_SPECS);
 TF_CALL_GPU_NUMBER_TYPES(DECLARE_GPU_SPECS);
-TF_CALL_complex64(DECLARE_GPU_SPECS);
-TF_CALL_complex128(DECLARE_GPU_SPECS);
+TF_CALL_COMPLEX_TYPES(DECLARE_GPU_SPECS);
 
 #undef DECLARE_GPU_SPECS
 #undef DECLARE_GPU_SPECS_INDEX
diff --git a/tensorflow/core/kernels/gather_functor_batched.cc b/tensorflow/core/kernels/gather_functor_batched.cc
index 0960b3a2472..d1ef076260b 100644
--- a/tensorflow/core/kernels/gather_functor_batched.cc
+++ b/tensorflow/core/kernels/gather_functor_batched.cc
@@ -39,8 +39,7 @@ namespace functor {
 
 TF_CALL_int64(DECLARE_GPU_SPECS);
 TF_CALL_GPU_NUMBER_TYPES(DECLARE_GPU_SPECS);
-TF_CALL_complex64(DECLARE_GPU_SPECS);
-TF_CALL_complex128(DECLARE_GPU_SPECS);
+TF_CALL_COMPLEX_TYPES(DECLARE_GPU_SPECS);
 
 #undef DECLARE_GPU_SPECS
 #undef DECLARE_GPU_SPECS_INDEX
diff --git a/tensorflow/core/kernels/gather_functor_batched_gpu.cu.cc b/tensorflow/core/kernels/gather_functor_batched_gpu.cu.cc
index f118d8dc72b..40b9894776d 100644
--- a/tensorflow/core/kernels/gather_functor_batched_gpu.cu.cc
+++ b/tensorflow/core/kernels/gather_functor_batched_gpu.cu.cc
@@ -31,12 +31,9 @@ typedef Eigen::GpuDevice GPUDevice;
   DEFINE_GPU_SPECS_INDEX(T, int32); \
   DEFINE_GPU_SPECS_INDEX(T, int64);
 
-TF_CALL_bool(DEFINE_GPU_SPECS);
 TF_CALL_int32(DEFINE_GPU_SPECS);
 TF_CALL_int64(DEFINE_GPU_SPECS);
-TF_CALL_GPU_NUMBER_TYPES(DEFINE_GPU_SPECS);
-TF_CALL_complex64(DEFINE_GPU_SPECS);
-TF_CALL_complex128(DEFINE_GPU_SPECS);
+TF_CALL_GPU_ALL_TYPES(DEFINE_GPU_SPECS);
 
 #undef DEFINE_GPU_SPECS
 #undef DEFINE_GPU_SPECS_INDEX
diff --git a/tensorflow/core/kernels/gather_functor_gpu.cu.cc b/tensorflow/core/kernels/gather_functor_gpu.cu.cc
index c548abb8bde..39402ebacec 100644
--- a/tensorflow/core/kernels/gather_functor_gpu.cu.cc
+++ b/tensorflow/core/kernels/gather_functor_gpu.cu.cc
@@ -31,12 +31,9 @@ typedef Eigen::GpuDevice GPUDevice;
   DEFINE_GPU_SPECS_INDEX(T, int32); \
   DEFINE_GPU_SPECS_INDEX(T, int64);
 
-TF_CALL_bool(DEFINE_GPU_SPECS);
 TF_CALL_int32(DEFINE_GPU_SPECS);
 TF_CALL_int64(DEFINE_GPU_SPECS);
-TF_CALL_GPU_NUMBER_TYPES(DEFINE_GPU_SPECS);
-TF_CALL_complex64(DEFINE_GPU_SPECS);
-TF_CALL_complex128(DEFINE_GPU_SPECS);
+TF_CALL_GPU_ALL_TYPES(DEFINE_GPU_SPECS);
 
 #undef DEFINE_GPU_SPECS
 #undef DEFINE_GPU_SPECS_INDEX
diff --git a/tensorflow/core/kernels/gather_nd_op.cc b/tensorflow/core/kernels/gather_nd_op.cc
index 517f78ff232..ad759489dc6 100644
--- a/tensorflow/core/kernels/gather_nd_op.cc
+++ b/tensorflow/core/kernels/gather_nd_op.cc
@@ -105,8 +105,7 @@ namespace functor {
 TF_CALL_int32(DECLARE_GPU_SPECS);
 TF_CALL_int64(DECLARE_GPU_SPECS);
 TF_CALL_GPU_NUMBER_TYPES(DECLARE_GPU_SPECS);
-TF_CALL_complex64(DECLARE_GPU_SPECS);
-TF_CALL_complex128(DECLARE_GPU_SPECS);
+TF_CALL_COMPLEX_TYPES(DECLARE_GPU_SPECS);
 
 #undef DECLARE_GPU_SPECS
 #undef DECLARE_GPU_SPECS_INDEX
@@ -118,8 +117,7 @@ TF_CALL_complex128(DECLARE_GPU_SPECS);
 TF_CALL_int32(REGISTER_GATHER_ND_GPU);
 TF_CALL_int64(REGISTER_GATHER_ND_GPU);
 TF_CALL_GPU_NUMBER_TYPES(REGISTER_GATHER_ND_GPU);
-TF_CALL_complex64(REGISTER_GATHER_ND_GPU);
-TF_CALL_complex128(REGISTER_GATHER_ND_GPU);
+TF_CALL_COMPLEX_TYPES(REGISTER_GATHER_ND_GPU);
 
 #undef REGISTER_GATHER_ND_GPU
 
diff --git a/tensorflow/core/kernels/gather_nd_op_gpu.cu.cc b/tensorflow/core/kernels/gather_nd_op_gpu.cu.cc
index 95ead6b0fd3..216ca2de114 100644
--- a/tensorflow/core/kernels/gather_nd_op_gpu.cu.cc
+++ b/tensorflow/core/kernels/gather_nd_op_gpu.cu.cc
@@ -121,8 +121,7 @@ struct GatherNdSlice<GPUDevice, T, Index, IXDIM> {
 TF_CALL_int32(DEFINE_GPU_SPECS);
 TF_CALL_int64(DEFINE_GPU_SPECS);
 TF_CALL_GPU_NUMBER_TYPES(DEFINE_GPU_SPECS);
-TF_CALL_complex64(DEFINE_GPU_SPECS);
-TF_CALL_complex128(DEFINE_GPU_SPECS);
+TF_CALL_COMPLEX_TYPES(DEFINE_GPU_SPECS);
 
 #undef DEFINE_GPU_SPECS
 #undef DEFINE_GPU_SPECS_INDEX
diff --git a/tensorflow/core/kernels/gather_op.cc b/tensorflow/core/kernels/gather_op.cc
index 5e6bd1de9d6..6d493a5f2ea 100644
--- a/tensorflow/core/kernels/gather_op.cc
+++ b/tensorflow/core/kernels/gather_op.cc
@@ -221,12 +221,9 @@ TF_CALL_uint64(REGISTER_GATHER_CPU);
 // Registration of the GPU implementations.
 #define REGISTER_GATHER_GPU(type) REGISTER_GATHER_ALL_INDICES(GPU, type)
 
-TF_CALL_bool(REGISTER_GATHER_GPU);
 TF_CALL_int32(REGISTER_GATHER_GPU);
 TF_CALL_int64(REGISTER_GATHER_GPU);
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_GATHER_GPU);
-TF_CALL_complex64(REGISTER_GATHER_GPU);
-TF_CALL_complex128(REGISTER_GATHER_GPU);
+TF_CALL_GPU_ALL_TYPES(REGISTER_GATHER_GPU);
 
 #undef REGISTER_GATHER_GPU
 
diff --git a/tensorflow/core/kernels/list_kernels.cu.cc b/tensorflow/core/kernels/list_kernels.cu.cc
index 1a1447ef1e6..b95a065edb8 100644
--- a/tensorflow/core/kernels/list_kernels.cu.cc
+++ b/tensorflow/core/kernels/list_kernels.cu.cc
@@ -105,13 +105,10 @@ typedef Eigen::GpuDevice GPUDevice;
                               .HostMemory("lengths"),                      \
                           TensorListSplit<GPUDevice, T>)
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_TENSOR_LIST_OPS_GPU);
-REGISTER_TENSOR_LIST_OPS_GPU(bfloat16);
-TF_CALL_complex64(REGISTER_TENSOR_LIST_OPS_GPU);
-TF_CALL_complex128(REGISTER_TENSOR_LIST_OPS_GPU);
 TF_CALL_int32(REGISTER_TENSOR_LIST_OPS_GPU);
 TF_CALL_int64(REGISTER_TENSOR_LIST_OPS_GPU);
-REGISTER_TENSOR_LIST_OPS_GPU(bool);
+TF_CALL_bfloat16(REGISTER_TENSOR_LIST_OPS_GPU);
+TF_CALL_GPU_ALL_TYPES(REGISTER_TENSOR_LIST_OPS_GPU);
 
 #undef REGISTER_TENSOR_LIST_OPS_GPU
 
diff --git a/tensorflow/core/kernels/matmul_op.cc b/tensorflow/core/kernels/matmul_op.cc
index 148540a3d82..2e3c120248f 100644
--- a/tensorflow/core/kernels/matmul_op.cc
+++ b/tensorflow/core/kernels/matmul_op.cc
@@ -581,21 +581,14 @@ struct MatMulFunctor<SYCLDevice, T> {
                               .Label("cublas"),                    \
                           MatMulOp<GPUDevice, T, true /* cublas */>)
 
-TF_CALL_float(REGISTER_CPU);
-TF_CALL_double(REGISTER_CPU);
-TF_CALL_half(REGISTER_CPU);
-TF_CALL_bfloat16(REGISTER_CPU);
 TF_CALL_int32(REGISTER_CPU);
 TF_CALL_int64(REGISTER_CPU);
-TF_CALL_complex64(REGISTER_CPU);
-TF_CALL_complex128(REGISTER_CPU);
+TF_CALL_FLOAT_TYPES(REGISTER_CPU);
+TF_CALL_COMPLEX_TYPES(REGISTER_CPU);
 
 #if GOOGLE_CUDA || TENSORFLOW_USE_ROCM
-TF_CALL_float(REGISTER_GPU);
-TF_CALL_double(REGISTER_GPU);
-TF_CALL_complex64(REGISTER_GPU);
-TF_CALL_complex128(REGISTER_GPU);
-TF_CALL_half(REGISTER_GPU);
+TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
+TF_CALL_COMPLEX_TYPES(REGISTER_GPU);
 #endif  // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
 
 #ifdef TENSORFLOW_USE_SYCL
diff --git a/tensorflow/core/kernels/matrix_band_part_op.cc b/tensorflow/core/kernels/matrix_band_part_op.cc
index 5254ade87b9..4dcce5a8f58 100644
--- a/tensorflow/core/kernels/matrix_band_part_op.cc
+++ b/tensorflow/core/kernels/matrix_band_part_op.cc
@@ -210,10 +210,7 @@ namespace functor {
   };                                                                   \
   extern template struct MatrixBandPartFunctor<GPUDevice, T>;
 
-TF_CALL_GPU_NUMBER_TYPES(DECLARE_GPU_SPEC);
-TF_CALL_bool(DECLARE_GPU_SPEC);
-TF_CALL_complex64(DECLARE_GPU_SPEC);
-TF_CALL_complex128(DECLARE_GPU_SPEC);
+TF_CALL_GPU_ALL_TYPES(DECLARE_GPU_SPEC);
 #undef DECLARE_GPU_SPEC
 }  // namespace functor
 
@@ -225,10 +222,7 @@ TF_CALL_complex128(DECLARE_GPU_SPEC);
                               .HostMemory("num_lower")   \
                               .HostMemory("num_upper"),  \
                           MatrixBandPartOp<GPUDevice, type>);
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_MATRIX_BAND_PART_GPU);
-TF_CALL_bool(REGISTER_MATRIX_BAND_PART_GPU);
-TF_CALL_complex64(REGISTER_MATRIX_BAND_PART_GPU);
-TF_CALL_complex128(REGISTER_MATRIX_BAND_PART_GPU);
+TF_CALL_GPU_ALL_TYPES(REGISTER_MATRIX_BAND_PART_GPU);
 #undef REGISTER_MATRIX_BAND_PART_GPU
 
 // Registration of the deprecated kernel.
diff --git a/tensorflow/core/kernels/matrix_band_part_op_gpu.cu.cc b/tensorflow/core/kernels/matrix_band_part_op_gpu.cu.cc
index 4a94c51e878..9eb3e4f72a2 100644
--- a/tensorflow/core/kernels/matrix_band_part_op_gpu.cu.cc
+++ b/tensorflow/core/kernels/matrix_band_part_op_gpu.cu.cc
@@ -68,10 +68,7 @@ struct MatrixBandPartFunctor<GPUDevice, Scalar> {
 
 #define DEFINE_GPU_SPEC(T) template struct MatrixBandPartFunctor<GPUDevice, T>;
 
-TF_CALL_GPU_NUMBER_TYPES(DEFINE_GPU_SPEC);
-TF_CALL_bool(DEFINE_GPU_SPEC);
-TF_CALL_complex64(DEFINE_GPU_SPEC);
-TF_CALL_complex128(DEFINE_GPU_SPEC);
+TF_CALL_GPU_ALL_TYPES(DEFINE_GPU_SPEC);
 
 #undef DEFINE_GPU_SPEC
 }  // namespace functor
diff --git a/tensorflow/core/kernels/matrix_diag_op.cc b/tensorflow/core/kernels/matrix_diag_op.cc
index 9796fd25e39..05d7e4e6f86 100644
--- a/tensorflow/core/kernels/matrix_diag_op.cc
+++ b/tensorflow/core/kernels/matrix_diag_op.cc
@@ -469,10 +469,7 @@ namespace functor {
       const bool left_align_subdiagonal);                                      \
   extern template struct MatrixDiagPart<GPUDevice, T>;
 
-TF_CALL_GPU_NUMBER_TYPES(DECLARE_GPU_SPEC);
-TF_CALL_bool(DECLARE_GPU_SPEC);
-TF_CALL_complex64(DECLARE_GPU_SPEC);
-TF_CALL_complex128(DECLARE_GPU_SPEC);
+TF_CALL_GPU_ALL_TYPES(DECLARE_GPU_SPEC);
 
 }  // namespace functor
 
@@ -513,10 +510,7 @@ TF_CALL_complex128(DECLARE_GPU_SPEC);
                               .HostMemory("padding_value"),                \
                           MatrixDiagPartOp<GPUDevice, type>);
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_MATRIX_DIAG_GPU);
-TF_CALL_bool(REGISTER_MATRIX_DIAG_GPU);
-TF_CALL_complex64(REGISTER_MATRIX_DIAG_GPU);
-TF_CALL_complex128(REGISTER_MATRIX_DIAG_GPU);
+TF_CALL_GPU_ALL_TYPES(REGISTER_MATRIX_DIAG_GPU);
 #undef REGISTER_MATRIX_DIAG_GPU
 
 // Registration of the deprecated kernel.
diff --git a/tensorflow/core/kernels/matrix_diag_op_gpu.cu.cc b/tensorflow/core/kernels/matrix_diag_op_gpu.cu.cc
index 53cd2d2dc46..76271798d5f 100644
--- a/tensorflow/core/kernels/matrix_diag_op_gpu.cu.cc
+++ b/tensorflow/core/kernels/matrix_diag_op_gpu.cu.cc
@@ -163,10 +163,7 @@ struct MatrixDiagPart<GPUDevice, T> {
   template struct MatrixDiag<GPUDevice, T>; \
   template struct MatrixDiagPart<GPUDevice, T>;
 
-TF_CALL_GPU_NUMBER_TYPES(DEFINE_GPU_SPEC);
-TF_CALL_bool(DEFINE_GPU_SPEC);
-TF_CALL_complex64(DEFINE_GPU_SPEC);
-TF_CALL_complex128(DEFINE_GPU_SPEC);
+TF_CALL_GPU_ALL_TYPES(DEFINE_GPU_SPEC);
 
 }  // namespace functor
 }  // namespace tensorflow
diff --git a/tensorflow/core/kernels/matrix_set_diag_op.cc b/tensorflow/core/kernels/matrix_set_diag_op.cc
index 2701ff788f7..bf98fd0d47d 100644
--- a/tensorflow/core/kernels/matrix_set_diag_op.cc
+++ b/tensorflow/core/kernels/matrix_set_diag_op.cc
@@ -272,10 +272,7 @@ namespace functor {
       const bool left_align_superdiagonal, const bool left_align_subdiagonal); \
   extern template struct MatrixSetDiag<GPUDevice, T>;
 
-TF_CALL_GPU_NUMBER_TYPES(DECLARE_GPU_SPEC);
-TF_CALL_bool(DECLARE_GPU_SPEC);
-TF_CALL_complex64(DECLARE_GPU_SPEC);
-TF_CALL_complex128(DECLARE_GPU_SPEC);
+TF_CALL_GPU_ALL_TYPES(DECLARE_GPU_SPEC);
 
 }  // namespace functor
 
@@ -295,10 +292,7 @@ TF_CALL_complex128(DECLARE_GPU_SPEC);
                               .HostMemory("k"),                           \
                           MatrixSetDiagOp<GPUDevice, type>);
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_MATRIX_SET_DIAG_GPU);
-TF_CALL_bool(REGISTER_MATRIX_SET_DIAG_GPU);
-TF_CALL_complex64(REGISTER_MATRIX_SET_DIAG_GPU);
-TF_CALL_complex128(REGISTER_MATRIX_SET_DIAG_GPU);
+TF_CALL_GPU_ALL_TYPES(REGISTER_MATRIX_SET_DIAG_GPU);
 #undef REGISTER_MATRIX_SET_DIAG_GPU
 
 // Registration of the deprecated kernel.
diff --git a/tensorflow/core/kernels/matrix_set_diag_op_gpu.cu.cc b/tensorflow/core/kernels/matrix_set_diag_op_gpu.cu.cc
index 4f742b90bff..4e32f8a52e8 100644
--- a/tensorflow/core/kernels/matrix_set_diag_op_gpu.cu.cc
+++ b/tensorflow/core/kernels/matrix_set_diag_op_gpu.cu.cc
@@ -136,10 +136,7 @@ struct MatrixSetDiag<GPUDevice, Scalar> {
 
 #define DEFINE_GPU_SPEC(T) template struct MatrixSetDiag<GPUDevice, T>;
 
-TF_CALL_GPU_NUMBER_TYPES(DEFINE_GPU_SPEC);
-TF_CALL_bool(DEFINE_GPU_SPEC);
-TF_CALL_complex64(DEFINE_GPU_SPEC);
-TF_CALL_complex128(DEFINE_GPU_SPEC);
+TF_CALL_GPU_ALL_TYPES(DEFINE_GPU_SPEC);
 
 }  // namespace functor
 }  // namespace tensorflow
diff --git a/tensorflow/core/kernels/matrix_triangular_solve_op_complex.cc b/tensorflow/core/kernels/matrix_triangular_solve_op_complex.cc
index 47f958ff6a9..ae3702078a0 100644
--- a/tensorflow/core/kernels/matrix_triangular_solve_op_complex.cc
+++ b/tensorflow/core/kernels/matrix_triangular_solve_op_complex.cc
@@ -13,16 +13,15 @@ See the License for the specific language governing permissions and
 limitations under the License.
 ==============================================================================*/
 
+#include "tensorflow/core/framework/register_types.h"
 #include "tensorflow/core/kernels/matrix_triangular_solve_op_impl.h"
 
 namespace tensorflow {
 
-TF_CALL_complex64(REGISTER_BATCH_MATRIX_TRIANGULAR_SOLVE_CPU);
-TF_CALL_complex128(REGISTER_BATCH_MATRIX_TRIANGULAR_SOLVE_CPU);
+TF_CALL_COMPLEX_TYPES(REGISTER_BATCH_MATRIX_TRIANGULAR_SOLVE_CPU);
 
 #if GOOGLE_CUDA
-TF_CALL_complex64(REGISTER_BATCH_MATRIX_TRIANGULAR_SOLVE_GPU);
-TF_CALL_complex128(REGISTER_BATCH_MATRIX_TRIANGULAR_SOLVE_GPU);
+TF_CALL_COMPLEX_TYPES(REGISTER_BATCH_MATRIX_TRIANGULAR_SOLVE_GPU);
 #endif  // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
 
 }  // namespace tensorflow
diff --git a/tensorflow/core/kernels/mkl_conv_ops.cc b/tensorflow/core/kernels/mkl_conv_ops.cc
index 59de3229211..7d0510d03ac 100644
--- a/tensorflow/core/kernels/mkl_conv_ops.cc
+++ b/tensorflow/core/kernels/mkl_conv_ops.cc
@@ -2309,11 +2309,20 @@ REGISTER_KERNEL_BUILDER(
         .TypeConstraint<quint8>("out_type"),
     NoOp);
 
-REGISTER_KERNEL_BUILDER(Name("_FusedDepthwiseConv2dNative")
+REGISTER_KERNEL_BUILDER(Name("DepthwiseConv2dNative")
                             .Device(DEVICE_CPU)
-                            .TypeConstraint<float>("T"),
+                            .TypeConstraint<bfloat16>("T"),
                         NoOp);
 
+#define REGISTER_NO_OP_CPU_2D_DEPTHWISE(T)                    \
+  REGISTER_KERNEL_BUILDER(Name("_FusedDepthwiseConv2dNative") \
+                              .Device(DEVICE_CPU)             \
+                              .TypeConstraint<T>("T"),        \
+                          NoOp);
+
+TF_CALL_float(REGISTER_NO_OP_CPU_2D_DEPTHWISE);
+TF_CALL_bfloat16(REGISTER_NO_OP_CPU_2D_DEPTHWISE);
+
 // Register templatized MKL kernels for non-fused and fused-versions of
 // QuantizedDepthwiseConv2D.
 REGISTER_KERNEL_BUILDER(Name("_MklQuantizedDepthwiseConv2D")
@@ -2367,14 +2376,6 @@ REGISTER_KERNEL_BUILDER(
     MklQuantizedConv2DReluOp<CPUDevice, quint8, qint32, quint8, quint8, true,
                              true>);
 
-REGISTER_KERNEL_BUILDER(
-    Name("_MklFusedDepthwiseConv2dNative")
-        .Device(DEVICE_CPU)
-        .TypeConstraint<float>("T")
-        .Label(mkl_op_registry::kMklLayoutDependentOpLabel),
-    MklFusedDepthwiseConvOp<CPUDevice, float, float, float, float, float, int32,
-                            false, true, true>);
-
 // Register 2D operations
 #define REGISTER_MKL_CPU_2D(T)                                                 \
   REGISTER_KERNEL_BUILDER(                                                     \
@@ -2426,13 +2427,20 @@ REGISTER_KERNEL_BUILDER(
 TF_CALL_float(REGISTER_MKL_CPU_2D);
 TF_CALL_bfloat16(REGISTER_MKL_CPU_2D);
 
-#define REGISTER_MKL_CPU_2D_DEPTHWISE(T)                       \
-  REGISTER_KERNEL_BUILDER(                                     \
-      Name("_MklDepthwiseConv2dNative")                        \
-          .Device(DEVICE_CPU)                                  \
-          .TypeConstraint<T>("T")                              \
-          .Label(mkl_op_registry::kMklLayoutDependentOpLabel), \
-      MklConvOp<CPUDevice, T, T, T, T, T, int32, false, false, true, false>);
+#define REGISTER_MKL_CPU_2D_DEPTHWISE(T)                                      \
+  REGISTER_KERNEL_BUILDER(                                                    \
+      Name("_MklDepthwiseConv2dNative")                                       \
+          .Device(DEVICE_CPU)                                                 \
+          .TypeConstraint<T>("T")                                             \
+          .Label(mkl_op_registry::kMklLayoutDependentOpLabel),                \
+      MklConvOp<CPUDevice, T, T, T, T, T, int32, false, false, true, false>); \
+  REGISTER_KERNEL_BUILDER(                                                    \
+      Name("_MklFusedDepthwiseConv2dNative")                                  \
+          .Device(DEVICE_CPU)                                                 \
+          .TypeConstraint<T>("T")                                             \
+          .Label(mkl_op_registry::kMklLayoutDependentOpLabel),                \
+      MklFusedDepthwiseConvOp<CPUDevice, T, T, T, T, T, int32, false, true,   \
+                              true>);
 
 TF_CALL_float(REGISTER_MKL_CPU_2D_DEPTHWISE);
 TF_CALL_bfloat16(REGISTER_MKL_CPU_2D_DEPTHWISE);
diff --git a/tensorflow/core/kernels/one_hot_op.cc b/tensorflow/core/kernels/one_hot_op.cc
index 0548e389b7a..e6cb84bab74 100644
--- a/tensorflow/core/kernels/one_hot_op.cc
+++ b/tensorflow/core/kernels/one_hot_op.cc
@@ -160,10 +160,9 @@ namespace functor {
   DECLARE_GPU_SPEC_INDEX(T, int32); \
   DECLARE_GPU_SPEC_INDEX(T, int64);
 
-TF_CALL_GPU_NUMBER_TYPES(DECLARE_GPU_SPEC);
-TF_CALL_bool(DECLARE_GPU_SPEC);
 TF_CALL_int32(DECLARE_GPU_SPEC);
 TF_CALL_int64(DECLARE_GPU_SPEC);
+TF_CALL_GPU_ALL_TYPES(DECLARE_GPU_SPEC);
 
 #undef DECLARE_GPU_SPEC_INDEX
 #undef DECLARE_GPU_SPEC
@@ -184,10 +183,9 @@ TF_CALL_int64(DECLARE_GPU_SPEC);
   REGISTER_ONE_HOT_GPU_INDEX(type, int32); \
   REGISTER_ONE_HOT_GPU_INDEX(type, int64);
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_ONE_HOT_GPU);
-TF_CALL_bool(REGISTER_ONE_HOT_GPU);
 TF_CALL_int32(REGISTER_ONE_HOT_GPU);
 TF_CALL_int64(REGISTER_ONE_HOT_GPU);
+TF_CALL_GPU_ALL_TYPES(REGISTER_ONE_HOT_GPU);
 
 #undef REGISTER_ONE_HOT_GPU_INDEX
 #undef REGISTER_ONE_HOT_GPU
diff --git a/tensorflow/core/kernels/one_hot_op_gpu.cu.cc b/tensorflow/core/kernels/one_hot_op_gpu.cu.cc
index 83ba272433f..47af41477c7 100644
--- a/tensorflow/core/kernels/one_hot_op_gpu.cu.cc
+++ b/tensorflow/core/kernels/one_hot_op_gpu.cu.cc
@@ -37,10 +37,9 @@ typedef Eigen::GpuDevice GPUDevice;
   DEFINE_GPU_SPEC_INDEX(T, int32); \
   DEFINE_GPU_SPEC_INDEX(T, int64)
 
-TF_CALL_GPU_NUMBER_TYPES(DEFINE_GPU_SPEC);
-TF_CALL_bool(DEFINE_GPU_SPEC);
 TF_CALL_int32(DEFINE_GPU_SPEC);
 TF_CALL_int64(DEFINE_GPU_SPEC);
+TF_CALL_GPU_ALL_TYPES(DEFINE_GPU_SPEC);
 
 #undef DEFINE_GPU_SPEC_INDEX
 #undef DEFINE_GPU_SPEC
diff --git a/tensorflow/core/kernels/pack_op.cc b/tensorflow/core/kernels/pack_op.cc
index cf2b6bb1100..04b5c72b3cf 100644
--- a/tensorflow/core/kernels/pack_op.cc
+++ b/tensorflow/core/kernels/pack_op.cc
@@ -152,13 +152,10 @@ REGISTER_PACK(tstring);
       Name("Pack").Device(DEVICE_GPU).TypeConstraint<type>("T"), \
       PackOp<GPUDevice, type>)
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
 TF_CALL_bfloat16(REGISTER_GPU);
 TF_CALL_int64(REGISTER_GPU);
 TF_CALL_int16(REGISTER_GPU);
-TF_CALL_bool(REGISTER_GPU);
-TF_CALL_complex64(REGISTER_GPU);
-TF_CALL_complex128(REGISTER_GPU);
+TF_CALL_GPU_ALL_TYPES(REGISTER_GPU);
 #undef REGISTER_GPU
 
 // A special GPU kernel for int32.
diff --git a/tensorflow/core/kernels/reduction_ops_euclidean.cc b/tensorflow/core/kernels/reduction_ops_euclidean.cc
index cf719e76cd8..9bc11e29069 100644
--- a/tensorflow/core/kernels/reduction_ops_euclidean.cc
+++ b/tensorflow/core/kernels/reduction_ops_euclidean.cc
@@ -52,8 +52,7 @@ TF_CALL_NUMBER_TYPES(REGISTER_CPU_KERNELS);
                                       functor::EuclideanNormReducer<type>>);
 TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU_KERNELS);
 #if GOOGLE_CUDA
-TF_CALL_complex64(REGISTER_GPU_KERNELS);
-TF_CALL_complex128(REGISTER_GPU_KERNELS);
+TF_CALL_COMPLEX_TYPES(REGISTER_GPU_KERNELS);
 #endif
 #undef REGISTER_GPU_KERNELS
 
diff --git a/tensorflow/core/kernels/reduction_ops_mean.cc b/tensorflow/core/kernels/reduction_ops_mean.cc
index d314f1953dc..e96d6f829ac 100644
--- a/tensorflow/core/kernels/reduction_ops_mean.cc
+++ b/tensorflow/core/kernels/reduction_ops_mean.cc
@@ -52,8 +52,7 @@ TF_CALL_NUMBER_TYPES(REGISTER_CPU_KERNELS);
       ReductionOp<GPUDevice, type, int64, functor::MeanReducer<type>>);
 TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU_KERNELS);
 #if GOOGLE_CUDA
-TF_CALL_complex64(REGISTER_GPU_KERNELS);
-TF_CALL_complex128(REGISTER_GPU_KERNELS);
+TF_CALL_COMPLEX_TYPES(REGISTER_GPU_KERNELS);
 #endif
 #undef REGISTER_GPU_KERNELS
 
diff --git a/tensorflow/core/kernels/reduction_ops_prod.cc b/tensorflow/core/kernels/reduction_ops_prod.cc
index 0642bad9218..33742e97146 100644
--- a/tensorflow/core/kernels/reduction_ops_prod.cc
+++ b/tensorflow/core/kernels/reduction_ops_prod.cc
@@ -50,11 +50,10 @@ TF_CALL_NUMBER_TYPES(REGISTER_CPU_KERNELS);
                               .HostMemory("reduction_indices"),             \
                           ReductionOp<GPUDevice, type, int64,               \
                                       Eigen::internal::ProdReducer<type>>);
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU_KERNELS);
 TF_CALL_int32(REGISTER_GPU_KERNELS);
+TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU_KERNELS);
 #if GOOGLE_CUDA
-TF_CALL_complex64(REGISTER_GPU_KERNELS);
-TF_CALL_complex128(REGISTER_GPU_KERNELS);
+TF_CALL_COMPLEX_TYPES(REGISTER_GPU_KERNELS);
 #endif
 #undef REGISTER_GPU_KERNELS
 
diff --git a/tensorflow/core/kernels/reduction_ops_sum.cc b/tensorflow/core/kernels/reduction_ops_sum.cc
index d79684df290..b5f7a5d7089 100644
--- a/tensorflow/core/kernels/reduction_ops_sum.cc
+++ b/tensorflow/core/kernels/reduction_ops_sum.cc
@@ -50,11 +50,10 @@ TF_CALL_NUMBER_TYPES(REGISTER_CPU_KERNELS);
           .TypeConstraint<int64>("Tidx")                                       \
           .HostMemory("reduction_indices"),                                    \
       ReductionOp<GPUDevice, type, int64, Eigen::internal::SumReducer<type>>);
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU_KERNELS);
 TF_CALL_int64(REGISTER_GPU_KERNELS);
+TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU_KERNELS);
 #if GOOGLE_CUDA
-TF_CALL_complex64(REGISTER_GPU_KERNELS);
-TF_CALL_complex128(REGISTER_GPU_KERNELS);
+TF_CALL_COMPLEX_TYPES(REGISTER_GPU_KERNELS);
 #endif
 #undef REGISTER_GPU_KERNELS
 
diff --git a/tensorflow/core/kernels/resize_bilinear_op.cc b/tensorflow/core/kernels/resize_bilinear_op.cc
index 57c39d1ce53..a0673fea73d 100644
--- a/tensorflow/core/kernels/resize_bilinear_op.cc
+++ b/tensorflow/core/kernels/resize_bilinear_op.cc
@@ -18,6 +18,10 @@ limitations under the License.
 
 #include "tensorflow/core/kernels/resize_bilinear_op.h"
 
+#ifdef __SSE4_1__
+#include <xmmintrin.h>
+#endif
+
 #include <memory>
 #include "third_party/eigen3/unsupported/Eigen/CXX11/Tensor"
 #include "tensorflow/core/framework/op_kernel.h"
@@ -107,6 +111,107 @@ inline float compute_lerp(const float top_left, const float top_right,
   return top + (bottom - top) * y_lerp;
 }
 
+#ifdef __SSE4_1__
+/* Vector version of the above */
+inline __m128 compute_lerp_v(const __m128 top_left, const __m128 top_right,
+                             const __m128 bottom_left,
+                             const __m128 bottom_right, const __m128 x_lerp,
+                             const __m128 y_lerp) {
+  const __m128 top =
+      _mm_add_ps(top_left, _mm_mul_ps(_mm_sub_ps(top_right, top_left), x_lerp));
+  const __m128 bottom = _mm_add_ps(
+      bottom_left, _mm_mul_ps(_mm_sub_ps(bottom_right, bottom_left), x_lerp));
+  return _mm_add_ps(top, _mm_mul_ps(_mm_sub_ps(bottom, top), y_lerp));
+}
+#endif
+
+template <typename T>
+void ResizeLine3Channels(const T* const ys_input_lower_ptr,
+                         const T* const ys_input_upper_ptr,
+                         const CachedInterpolation* const xs,
+                         const float ys_lerp, const int64 out_width,
+                         float* out_y) {
+  for (int64 x = 0; x < out_width; ++x) {
+    const int64 xs_lower = xs[x].lower;
+    const int64 xs_upper = xs[x].upper;
+    const float xs_lerp = xs[x].lerp;
+
+    // Read channel 0.
+    const float top_left0(ys_input_lower_ptr[xs_lower + 0]);
+    const float top_right0(ys_input_lower_ptr[xs_upper + 0]);
+    const float bottom_left0(ys_input_upper_ptr[xs_lower + 0]);
+    const float bottom_right0(ys_input_upper_ptr[xs_upper + 0]);
+
+    // Read channel 1.
+    const float top_left1(ys_input_lower_ptr[xs_lower + 1]);
+    const float top_right1(ys_input_lower_ptr[xs_upper + 1]);
+    const float bottom_left1(ys_input_upper_ptr[xs_lower + 1]);
+    const float bottom_right1(ys_input_upper_ptr[xs_upper + 1]);
+
+    // Read channel 2.
+    const float top_left2(ys_input_lower_ptr[xs_lower + 2]);
+    const float top_right2(ys_input_lower_ptr[xs_upper + 2]);
+    const float bottom_left2(ys_input_upper_ptr[xs_lower + 2]);
+    const float bottom_right2(ys_input_upper_ptr[xs_upper + 2]);
+
+    // Compute output.
+    out_y[x * 3 + 0] = compute_lerp(top_left0, top_right0, bottom_left0,
+                                    bottom_right0, xs_lerp, ys_lerp);
+    out_y[x * 3 + 1] = compute_lerp(top_left1, top_right1, bottom_left1,
+                                    bottom_right1, xs_lerp, ys_lerp);
+    out_y[x * 3 + 2] = compute_lerp(top_left2, top_right2, bottom_left2,
+                                    bottom_right2, xs_lerp, ys_lerp);
+  }
+}
+
+#ifdef __SSE4_1__
+
+// Load 3 floats from the given buffer, which must be of size at least 4.
+template <typename T>
+inline __m128 load_3xfloat_v(T* values) {
+  return _mm_set_ps(0.0f, static_cast<float>(values[2]),
+                    static_cast<float>(values[1]),
+                    static_cast<float>(values[0]));
+}
+
+// Specialize cases that can be done more efficiently.
+template <>
+inline __m128 load_3xfloat_v(float* values) {
+  return _mm_loadu_ps(values);
+}
+
+template <typename T>
+void ResizeLine3ChannelsVector(const T* const ys_input_lower_ptr,
+                               const T* const ys_input_upper_ptr,
+                               const CachedInterpolation* const xs,
+                               const float ys_lerp, const int64 out_width,
+                               float* out_y) {
+  const __m128 ys_lerp_v = _mm_set1_ps(ys_lerp);
+  // All pixels but the last one can overflow, vectorize the inside of the
+  // row.
+  int64 x = 0;
+  for (x = 0; x < out_width - 1; ++x) {
+    const int64 xs_lower = xs[x].lower;
+    const int64 xs_upper = xs[x].upper;
+    const __m128 xs_lerp_v = _mm_set1_ps(xs[x].lerp);
+
+    const __m128 top_left_v = load_3xfloat_v(ys_input_lower_ptr + xs_lower);
+    const __m128 top_right_v = load_3xfloat_v(ys_input_lower_ptr + xs_upper);
+    const __m128 bottom_left_v = load_3xfloat_v(ys_input_upper_ptr + xs_lower);
+    const __m128 bottom_right_v = load_3xfloat_v(ys_input_upper_ptr + xs_upper);
+
+    _mm_storeu_ps(out_y + x * 3,
+                  compute_lerp_v(top_left_v, top_right_v, bottom_left_v,
+                                 bottom_right_v, xs_lerp_v, ys_lerp_v));
+  }
+  // The last pixel of each row must be done in a non-vectorized way
+  // because we cannot overflow.
+  ResizeLine3Channels(ys_input_lower_ptr, ys_input_upper_ptr,
+                      xs + out_width - 1, ys_lerp, 1,
+                      out_y + (out_width - 1) * 3);
+}
+#endif
+
 template <typename T>
 void resize_image(
     typename TTypes<T, 4>::ConstTensor images, const int batch_size,
@@ -136,41 +241,13 @@ void resize_image(typename TTypes<T, 4>::ConstTensor images,
       for (int64 y = 0; y < out_height; ++y) {
         const T* ys_input_lower_ptr = input_b_ptr + ys[y].lower * in_row_size;
         const T* ys_input_upper_ptr = input_b_ptr + ys[y].upper * in_row_size;
-        const float ys_lerp = ys[y].lerp;
-        for (int64 x = 0; x < out_width; ++x) {
-          const int64 xs_lower = xs[x].lower;
-          const int64 xs_upper = xs[x].upper;
-          const float xs_lerp = xs[x].lerp;
-
-          // Read channel 0.
-          const float top_left0(ys_input_lower_ptr[xs_lower + 0]);
-          const float top_right0(ys_input_lower_ptr[xs_upper + 0]);
-          const float bottom_left0(ys_input_upper_ptr[xs_lower + 0]);
-          const float bottom_right0(ys_input_upper_ptr[xs_upper + 0]);
-
-          // Read channel 1.
-          const float top_left1(ys_input_lower_ptr[xs_lower + 1]);
-          const float top_right1(ys_input_lower_ptr[xs_upper + 1]);
-          const float bottom_left1(ys_input_upper_ptr[xs_lower + 1]);
-          const float bottom_right1(ys_input_upper_ptr[xs_upper + 1]);
-
-          // Read channel 2.
-          const float top_left2(ys_input_lower_ptr[xs_lower + 2]);
-          const float top_right2(ys_input_lower_ptr[xs_upper + 2]);
-          const float bottom_left2(ys_input_upper_ptr[xs_lower + 2]);
-          const float bottom_right2(ys_input_upper_ptr[xs_upper + 2]);
-
-          // Compute output.
-          output_y_ptr[x * channels + 0] =
-              compute_lerp(top_left0, top_right0, bottom_left0, bottom_right0,
-                           xs_lerp, ys_lerp);
-          output_y_ptr[x * channels + 1] =
-              compute_lerp(top_left1, top_right1, bottom_left1, bottom_right1,
-                           xs_lerp, ys_lerp);
-          output_y_ptr[x * channels + 2] =
-              compute_lerp(top_left2, top_right2, bottom_left2, bottom_right2,
-                           xs_lerp, ys_lerp);
-        }
+#ifdef __SSE4_1__
+        ResizeLine3ChannelsVector(ys_input_lower_ptr, ys_input_upper_ptr, xs,
+                                  ys[y].lerp, out_width, output_y_ptr);
+#else
+        ResizeLine3Channels(ys_input_lower_ptr, ys_input_upper_ptr, xs,
+                            ys[y].lerp, out_width, output_y_ptr);
+#endif
         output_y_ptr += out_row_size;
       }
       input_b_ptr += in_batch_num_values;
@@ -338,6 +415,7 @@ struct ResizeBilinearGrad<CPUDevice, T> {
               static_cast<Eigen::Index>(ceilf(in_x)), original_width - 1);
           const float x_lerp = in_x - floorf(in_x);
           const float inverse_x_lerp = (1.0f - x_lerp);
+          // TODO(b/158287314): Look into vectorizing this.
           for (Eigen::Index c = 0; c < channels; ++c) {
             output_grad(b, top_y_index, left_x_index, c) +=
                 T(input_grad(b, y, x, c) * inverse_y_lerp * inverse_x_lerp);
diff --git a/tensorflow/core/kernels/resize_bilinear_op_test.cc b/tensorflow/core/kernels/resize_bilinear_op_test.cc
index 4873b49612d..df00ca281e7 100644
--- a/tensorflow/core/kernels/resize_bilinear_op_test.cc
+++ b/tensorflow/core/kernels/resize_bilinear_op_test.cc
@@ -28,6 +28,7 @@ limitations under the License.
 #include "tensorflow/core/lib/random/random.h"
 #include "tensorflow/core/lib/strings/str_util.h"
 #include "tensorflow/core/platform/test.h"
+#include "tensorflow/core/platform/test_benchmark.h"
 #include "tensorflow/core/public/session_options.h"
 
 namespace tensorflow {
@@ -543,4 +544,39 @@ INSTANTIATE_TEST_SUITE_P(ResizeBilinearOpAlignCornersTestGpu,
                          ResizeBilinearOpAlignCornersTest,
                          ::testing::Values(TestDevice::GPU));
 #endif  // GOOGLE_CUDA
+
+class ResizeBM : public ResizeBilinearOpTest {
+ public:
+  void TestBody() override {}
+  void SetUpBenchmark(int input_width, int input_height, int num_channels,
+                      int output_width, int output_height) {
+    TF_EXPECT_OK(NodeDefBuilder("resize_bilinear_op", "ResizeBilinear")
+                     .Input(FakeInput(DT_FLOAT))
+                     .Input(FakeInput(DT_INT32))
+                     .Attr("align_corners", align_corners_)
+                     .Attr("half_pixel_centers", half_pixel_centers_)
+                     .Finalize(node_def()));
+    TF_EXPECT_OK(InitOp());
+    const TensorShape shape(
+        {/*batch_size*/ 1, input_width, input_height, num_channels});
+    SetRandomImageInput(shape);
+    AddInputFromArray<int32>(TensorShape({2}), {output_width, output_height});
+  }
+
+  using ResizeBilinearOpTest::RunOpKernel;
+};
+
+#ifdef PLATFORM_GOOGLE
+
+void BM_Resize(benchmark::State& state) {
+  ResizeBM bench;
+  bench.SetUpBenchmark(640, 480, 3, 1024, 768);
+  for (const auto _ : state) {
+    CHECK(bench.RunOpKernel().ok());
+  }
+}
+BENCHMARK(BM_Resize);
+
+#endif
+
 }  // namespace tensorflow
diff --git a/tensorflow/core/kernels/reverse_op.cc b/tensorflow/core/kernels/reverse_op.cc
index 98bf8bf8e91..7ec6923492f 100644
--- a/tensorflow/core/kernels/reverse_op.cc
+++ b/tensorflow/core/kernels/reverse_op.cc
@@ -342,12 +342,7 @@ namespace functor {
 
 TF_CALL_uint8(DECLARE_GPU_SPEC);
 TF_CALL_int8(DECLARE_GPU_SPEC);
-TF_CALL_bool(DECLARE_GPU_SPEC);
-TF_CALL_half(DECLARE_GPU_SPEC);
-TF_CALL_float(DECLARE_GPU_SPEC);
-TF_CALL_double(DECLARE_GPU_SPEC);
-TF_CALL_complex64(DECLARE_GPU_SPEC);
-TF_CALL_complex128(DECLARE_GPU_SPEC);
+TF_CALL_GPU_ALL_TYPES(DECLARE_GPU_SPEC);
 #undef DECLARE_GPU_SPEC
 #undef DECLARE_GPU_SPEC_DIM
 }  // namespace functor
@@ -373,12 +368,7 @@ TF_CALL_complex128(DECLARE_GPU_SPEC);
                           ReverseV2Op<GPUDevice, T, int64>)
 TF_CALL_uint8(REGISTER_GPU_KERNELS);
 TF_CALL_int8(REGISTER_GPU_KERNELS);
-TF_CALL_bool(REGISTER_GPU_KERNELS);
-TF_CALL_half(REGISTER_GPU_KERNELS);
-TF_CALL_float(REGISTER_GPU_KERNELS);
-TF_CALL_double(REGISTER_GPU_KERNELS);
-TF_CALL_complex64(REGISTER_GPU_KERNELS);
-TF_CALL_complex128(REGISTER_GPU_KERNELS);
+TF_CALL_GPU_ALL_TYPES(REGISTER_GPU_KERNELS);
 #undef REGISTER_GPU_KERNEL
 
 // A special GPU kernel for int32.
diff --git a/tensorflow/core/kernels/reverse_op_gpu.cu.cc b/tensorflow/core/kernels/reverse_op_gpu.cu.cc
index 2917a0d5f11..28c50bc66df 100644
--- a/tensorflow/core/kernels/reverse_op_gpu.cu.cc
+++ b/tensorflow/core/kernels/reverse_op_gpu.cu.cc
@@ -40,12 +40,7 @@ typedef Eigen::GpuDevice GPUDevice;
 
 TF_CALL_uint8(DEFINE_REVERSE_ALL_DIMS);
 TF_CALL_int8(DEFINE_REVERSE_ALL_DIMS);
-TF_CALL_bool(DEFINE_REVERSE_ALL_DIMS);
-TF_CALL_half(DEFINE_REVERSE_ALL_DIMS);
-TF_CALL_float(DEFINE_REVERSE_ALL_DIMS);
-TF_CALL_double(DEFINE_REVERSE_ALL_DIMS);
-TF_CALL_complex64(DEFINE_REVERSE_ALL_DIMS);
-TF_CALL_complex128(DEFINE_REVERSE_ALL_DIMS);
+TF_CALL_GPU_ALL_TYPES(DEFINE_REVERSE_ALL_DIMS);
 #undef DEFINE_REVERSE
 #undef DEFINE_REVERSE_ALL_DIMS
 
diff --git a/tensorflow/core/kernels/roll_op.cc b/tensorflow/core/kernels/roll_op.cc
index 8d6147801e7..2a141864e18 100644
--- a/tensorflow/core/kernels/roll_op.cc
+++ b/tensorflow/core/kernels/roll_op.cc
@@ -397,10 +397,9 @@ TF_CALL_ALL_TYPES(REGISTER_CPU);
 
 TF_CALL_int32(REGISTER_KERNEL);
 TF_CALL_int64(REGISTER_KERNEL);
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_KERNEL);
-TF_CALL_complex64(REGISTER_KERNEL);
-TF_CALL_complex128(REGISTER_KERNEL);
 TF_CALL_uint32(REGISTER_KERNEL);
+TF_CALL_GPU_NUMBER_TYPES(REGISTER_KERNEL);
+TF_CALL_COMPLEX_TYPES(REGISTER_KERNEL);
 
 #undef REGISTER_KERNEL
 #endif  // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
diff --git a/tensorflow/core/kernels/roll_op_gpu.cu.cc b/tensorflow/core/kernels/roll_op_gpu.cu.cc
index 7ba37e2f59c..0e81b576330 100644
--- a/tensorflow/core/kernels/roll_op_gpu.cu.cc
+++ b/tensorflow/core/kernels/roll_op_gpu.cu.cc
@@ -93,10 +93,9 @@ struct Roll<GPUDevice, T> {
 
 TF_CALL_int32(DEFINE_GPU_SPECS);
 TF_CALL_int64(DEFINE_GPU_SPECS);
+TF_CALL_uint32(DEFINE_GPU_SPECS);
 TF_CALL_GPU_NUMBER_TYPES(DEFINE_GPU_SPECS);
-TF_CALL_complex64(DEFINE_GPU_SPECS);
-TF_CALL_complex128(DEFINE_GPU_SPECS);
-TF_CALL_uint32(DEFINE_GPU_SPECS)
+TF_CALL_COMPLEX_TYPES(DEFINE_GPU_SPECS);
 
 #undef DEFINE_GPU_SPECS
 }  // namespace functor
diff --git a/tensorflow/core/kernels/scatter_nd_op.cc b/tensorflow/core/kernels/scatter_nd_op.cc
index 96711fce643..c6c93077f01 100644
--- a/tensorflow/core/kernels/scatter_nd_op.cc
+++ b/tensorflow/core/kernels/scatter_nd_op.cc
@@ -502,8 +502,7 @@ TF_CALL_int64(REGISTER_SCATTER_ND_ALL_GPU);
 TF_CALL_int64(REGISTER_SCATTER_ND_MIN_MAX_GPU);
 TF_CALL_GPU_NUMBER_TYPES(REGISTER_SCATTER_ND_ALL_GPU);
 TF_CALL_GPU_NUMBER_TYPES(REGISTER_SCATTER_ND_MIN_MAX_GPU);
-TF_CALL_complex64(REGISTER_SCATTER_ND_ALL_GPU);
-TF_CALL_complex128(REGISTER_SCATTER_ND_ALL_GPU);
+TF_CALL_COMPLEX_TYPES(REGISTER_SCATTER_ND_ALL_GPU);
 
 #undef REGISTER_SCATTER_ND_ALL_GPU
 
@@ -563,8 +562,7 @@ TF_CALL_int64(REGISTER_SCATTER_ND_TENSOR_GPU);
 TF_CALL_int64(REGISTER_SCATTER_ND_TENSOR_GPU_MIN_MAX);
 TF_CALL_GPU_NUMBER_TYPES_NO_HALF(REGISTER_SCATTER_ND_TENSOR_GPU);
 TF_CALL_GPU_NUMBER_TYPES_NO_HALF(REGISTER_SCATTER_ND_TENSOR_GPU_MIN_MAX);
-TF_CALL_complex64(REGISTER_SCATTER_ND_TENSOR_GPU);
-TF_CALL_complex128(REGISTER_SCATTER_ND_TENSOR_GPU);
+TF_CALL_COMPLEX_TYPES(REGISTER_SCATTER_ND_TENSOR_GPU);
 
 #undef REGISTER_SCATTER_ND_ADD
 #undef REGISTER_SCATTER_ND_ADD_SUB
@@ -862,8 +860,7 @@ TF_CALL_int32(DECLARE_GPU_SPECS);
 TF_CALL_int32(DECLARE_GPU_SPECS_MIN_MAX);
 TF_CALL_GPU_NUMBER_TYPES(DECLARE_GPU_SPECS);
 TF_CALL_GPU_NUMBER_TYPES(DECLARE_GPU_SPECS_MIN_MAX);
-TF_CALL_complex64(DECLARE_GPU_SPECS);
-TF_CALL_complex128(DECLARE_GPU_SPECS);
+TF_CALL_COMPLEX_TYPES(DECLARE_GPU_SPECS);
 
 #undef DECLARE_GPU_SPECS_MIN_MAX
 #undef DECLARE_GPU_SPECS
diff --git a/tensorflow/core/kernels/scatter_nd_op_gpu.cu.cc b/tensorflow/core/kernels/scatter_nd_op_gpu.cu.cc
index 625e03b4352..64b69af423f 100644
--- a/tensorflow/core/kernels/scatter_nd_op_gpu.cu.cc
+++ b/tensorflow/core/kernels/scatter_nd_op_gpu.cu.cc
@@ -58,14 +58,14 @@ struct LeftUpdate<T, scatter_nd_op::UpdateOp::SUB> {
 template <typename T>
 struct LeftUpdate<T, scatter_nd_op::UpdateOp::MAX> {
   EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC void operator()(T* out, const T& val) {
-    CudaAtomicMax(out, val);
+    GpuAtomicMax(out, val);
   }
 };
 
 template <typename T>
 struct LeftUpdate<T, scatter_nd_op::UpdateOp::MIN> {
   EIGEN_STRONG_INLINE EIGEN_DEVICE_FUNC void operator()(T* out, const T& val) {
-    CudaAtomicMin(out, val);
+    GpuAtomicMin(out, val);
   }
 };
 
@@ -200,8 +200,7 @@ TF_CALL_int64(DECLARE_GPU_SPECS);
 TF_CALL_int64(DECLARE_GPU_SPECS_MINMAX);
 TF_CALL_GPU_NUMBER_TYPES(DECLARE_GPU_SPECS);
 TF_CALL_GPU_NUMBER_TYPES(DECLARE_GPU_SPECS_MINMAX);
-TF_CALL_complex64(DECLARE_GPU_SPECS);
-TF_CALL_complex128(DECLARE_GPU_SPECS);
+TF_CALL_COMPLEX_TYPES(DECLARE_GPU_SPECS);
 
 #undef DECLARE_GPU_SPECS
 #undef DECLARE_GPU_SPECS_MINMAX
diff --git a/tensorflow/core/kernels/segment_reduction_ops_gpu.cu.cc b/tensorflow/core/kernels/segment_reduction_ops_gpu.cu.cc
index bc28e64c4d7..418af1d6b6d 100644
--- a/tensorflow/core/kernels/segment_reduction_ops_gpu.cu.cc
+++ b/tensorflow/core/kernels/segment_reduction_ops_gpu.cu.cc
@@ -244,8 +244,7 @@ TF_CALL_int32(DEFINE_SUM_GPU_SPECS);
 
 // TODO(rocm): support atomicAdd for complex numbers on ROCm
 #if GOOGLE_CUDA
-TF_CALL_complex64(DEFINE_SUM_GPU_SPECS);
-TF_CALL_complex128(DEFINE_SUM_GPU_SPECS);
+TF_CALL_COMPLEX_TYPES(DEFINE_SUM_GPU_SPECS);
 #endif
 
 #undef DEFINE_SORTED_GPU_SPECS_INDEX
diff --git a/tensorflow/core/kernels/segment_reduction_ops_impl_3.cc b/tensorflow/core/kernels/segment_reduction_ops_impl_3.cc
index d0d46938629..eef5a532b29 100644
--- a/tensorflow/core/kernels/segment_reduction_ops_impl_3.cc
+++ b/tensorflow/core/kernels/segment_reduction_ops_impl_3.cc
@@ -113,8 +113,7 @@ TF_CALL_GPU_NUMBER_TYPES(REGISTER_SUM_GPU_UNSORTED_KERNELS_ALL);
 TF_CALL_int32(REGISTER_SUM_GPU_UNSORTED_KERNELS_ALL);
 // TODO(rocm): support atomicAdd for complex numbers on ROCm
 #if GOOGLE_CUDA
-TF_CALL_complex64(REGISTER_SUM_GPU_UNSORTED_KERNELS_ALL);
-TF_CALL_complex128(REGISTER_SUM_GPU_UNSORTED_KERNELS_ALL);
+TF_CALL_COMPLEX_TYPES(REGISTER_SUM_GPU_UNSORTED_KERNELS_ALL);
 #endif
 
 #undef REGISTER_GPU_KERNEL_UNSORTEDSEGMENT
diff --git a/tensorflow/core/kernels/segment_reduction_ops_impl_4.cc b/tensorflow/core/kernels/segment_reduction_ops_impl_4.cc
index 92caeb3c544..cad6f8a5e08 100644
--- a/tensorflow/core/kernels/segment_reduction_ops_impl_4.cc
+++ b/tensorflow/core/kernels/segment_reduction_ops_impl_4.cc
@@ -113,8 +113,7 @@ TF_CALL_GPU_NUMBER_TYPES(REGISTER_SUM_GPU_UNSORTED_KERNELS_ALL);
 TF_CALL_int32(REGISTER_SUM_GPU_UNSORTED_KERNELS_ALL);
 // TODO(rocm): support atomicAdd for complex numbers on ROCm
 #if GOOGLE_CUDA
-TF_CALL_complex64(REGISTER_SUM_GPU_UNSORTED_KERNELS_ALL);
-TF_CALL_complex128(REGISTER_SUM_GPU_UNSORTED_KERNELS_ALL);
+TF_CALL_COMPLEX_TYPES(REGISTER_SUM_GPU_UNSORTED_KERNELS_ALL);
 #endif
 
 #undef REGISTER_GPU_KERNEL_UNSORTEDSEGMENT
diff --git a/tensorflow/core/kernels/slice_op.cc b/tensorflow/core/kernels/slice_op.cc
index 62c63263d48..6d7cd6f2a3d 100644
--- a/tensorflow/core/kernels/slice_op.cc
+++ b/tensorflow/core/kernels/slice_op.cc
@@ -300,14 +300,11 @@ namespace functor {
   DECLARE_GPU_SPEC(T, 7); \
   DECLARE_GPU_SPEC(T, 8);
 
-TF_CALL_GPU_NUMBER_TYPES(DECLARE_FOR_N);
-TF_CALL_complex64(DECLARE_FOR_N);
-TF_CALL_complex128(DECLARE_FOR_N);
 TF_CALL_bfloat16(DECLARE_FOR_N);
-TF_CALL_bool(DECLARE_FOR_N);
 TF_CALL_int8(DECLARE_FOR_N);
+TF_CALL_int32(DECLARE_FOR_N);
 TF_CALL_int64(DECLARE_FOR_N);
-DECLARE_FOR_N(int32);
+TF_CALL_GPU_ALL_TYPES(DECLARE_FOR_N);
 
 #undef DECLARE_FOR_N
 #undef DECLARE_GPU_SPEC
@@ -321,13 +318,10 @@ DECLARE_FOR_N(int32);
                               .HostMemory("size"),       \
                           SliceOp<GPUDevice, type>)
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
-TF_CALL_complex64(REGISTER_GPU);
-TF_CALL_complex128(REGISTER_GPU);
 TF_CALL_bfloat16(REGISTER_GPU);
-TF_CALL_bool(REGISTER_GPU);
 TF_CALL_int8(REGISTER_GPU);
 TF_CALL_int64(REGISTER_GPU);
+TF_CALL_GPU_ALL_TYPES(REGISTER_GPU);
 
 // A special GPU kernel for int32.
 // TODO(b/25387198): Also enable int32 in device memory. This kernel
diff --git a/tensorflow/core/kernels/slice_op_gpu.cu.cc b/tensorflow/core/kernels/slice_op_gpu.cu.cc
index 5a9d2ff950a..c20d01751d9 100644
--- a/tensorflow/core/kernels/slice_op_gpu.cu.cc
+++ b/tensorflow/core/kernels/slice_op_gpu.cu.cc
@@ -37,14 +37,11 @@ typedef Eigen::GpuDevice GPUDevice;
   template struct functor::Slice<GPUDevice, T, 7>; \
   template struct functor::Slice<GPUDevice, T, 8>;
 
-TF_CALL_GPU_NUMBER_TYPES(DEFINE_GPU_KERNELS);
-TF_CALL_complex64(DEFINE_GPU_KERNELS);
-TF_CALL_complex128(DEFINE_GPU_KERNELS);
 TF_CALL_bfloat16(DEFINE_GPU_KERNELS);
-TF_CALL_bool(DEFINE_GPU_KERNELS);
 TF_CALL_int8(DEFINE_GPU_KERNELS);
-DEFINE_GPU_KERNELS(int32);
-DEFINE_GPU_KERNELS(int64);
+TF_CALL_int32(DEFINE_GPU_KERNELS);
+TF_CALL_int64(DEFINE_GPU_KERNELS);
+TF_CALL_GPU_ALL_TYPES(DEFINE_GPU_KERNELS);
 
 #undef DEFINE_GPU_KERNELS
 
diff --git a/tensorflow/core/kernels/softmax_op.cc b/tensorflow/core/kernels/softmax_op.cc
index 9bdfca32d88..7d09b39ad4b 100644
--- a/tensorflow/core/kernels/softmax_op.cc
+++ b/tensorflow/core/kernels/softmax_op.cc
@@ -82,19 +82,16 @@ class SoftmaxOp : public OpKernel {
   REGISTER_KERNEL_BUILDER(                                       \
       Name("Softmax").Device(DEVICE_CPU).TypeConstraint<T>("T"), \
       SoftmaxOp<CPUDevice, T>);
-TF_CALL_bfloat16(REGISTER_CPU);
-TF_CALL_half(REGISTER_CPU);
-TF_CALL_float(REGISTER_CPU);
-TF_CALL_double(REGISTER_CPU);
+TF_CALL_FLOAT_TYPES(REGISTER_CPU);
 
 #undef REGISTER_CPU
 #define REGISTER_CPU(T)                                             \
   REGISTER_KERNEL_BUILDER(                                          \
       Name("LogSoftmax").Device(DEVICE_CPU).TypeConstraint<T>("T"), \
       SoftmaxOp<CPUDevice, T>);
-TF_CALL_half(REGISTER_CPU);
-TF_CALL_float(REGISTER_CPU);
-TF_CALL_double(REGISTER_CPU);
+TF_CALL_FLOAT_TYPES(REGISTER_CPU);
+
+#undef REGISTER_CPU
 
 #ifdef TENSORFLOW_USE_SYCL
 REGISTER_KERNEL_BUILDER(
diff --git a/tensorflow/core/kernels/softmax_op_gpu.cu.cc b/tensorflow/core/kernels/softmax_op_gpu.cu.cc
index 0c09fd2852b..3cf357713e9 100644
--- a/tensorflow/core/kernels/softmax_op_gpu.cu.cc
+++ b/tensorflow/core/kernels/softmax_op_gpu.cu.cc
@@ -281,21 +281,20 @@ class SoftmaxOpGPU : public OpKernel {
   bool log_;
 };
 
-REGISTER_KERNEL_BUILDER(
-    Name("Softmax").Device(DEVICE_GPU).TypeConstraint<Eigen::half>("T"),
-    SoftmaxOpGPU<Eigen::half>);
-REGISTER_KERNEL_BUILDER(
-    Name("Softmax").Device(DEVICE_GPU).TypeConstraint<float>("T"),
-    SoftmaxOpGPU<float>);
-REGISTER_KERNEL_BUILDER(
-    Name("Softmax").Device(DEVICE_GPU).TypeConstraint<double>("T"),
-    SoftmaxOpGPU<double>);
-REGISTER_KERNEL_BUILDER(
-    Name("LogSoftmax").Device(DEVICE_GPU).TypeConstraint<Eigen::half>("T"),
-    SoftmaxOpGPU<Eigen::half>);
-REGISTER_KERNEL_BUILDER(
-    Name("LogSoftmax").Device(DEVICE_GPU).TypeConstraint<float>("T"),
-    SoftmaxOpGPU<float>);
+#define REGISTER_GPU(T)                                          \
+  REGISTER_KERNEL_BUILDER(                                       \
+      Name("Softmax").Device(DEVICE_GPU).TypeConstraint<T>("T"), \
+      SoftmaxOpGPU<T>);
+TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
+
+#undef REGISTER_GPU
+#define REGISTER_GPU(T)                                             \
+  REGISTER_KERNEL_BUILDER(                                          \
+      Name("LogSoftmax").Device(DEVICE_GPU).TypeConstraint<T>("T"), \
+      SoftmaxOpGPU<T>);
+TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
+
+#undef REGISTER_GPU
 
 }  // end namespace tensorflow
 
diff --git a/tensorflow/core/kernels/sparse_add_op.cc b/tensorflow/core/kernels/sparse_add_op.cc
index d16317af671..0cf40a709a3 100644
--- a/tensorflow/core/kernels/sparse_add_op.cc
+++ b/tensorflow/core/kernels/sparse_add_op.cc
@@ -159,7 +159,9 @@ class SparseAddOp : public OpKernel {
       out_indices_mat.chip<0>(i) =
           from_a ? a_indices_mat.chip<0>(idx) : b_indices_mat.chip<0>(idx);
     }
-    std::copy_n(out_values.begin(), sum_nnz, &out_values_flat(0));
+    if (sum_nnz > 0) {
+      std::copy_n(out_values.begin(), sum_nnz, &out_values_flat(0));
+    }
     ctx->set_output(2, *a_shape);
   }
 };
diff --git a/tensorflow/core/kernels/split_lib_gpu.cu.cc b/tensorflow/core/kernels/split_lib_gpu.cu.cc
index b094a5320f7..b4379a01ce1 100644
--- a/tensorflow/core/kernels/split_lib_gpu.cu.cc
+++ b/tensorflow/core/kernels/split_lib_gpu.cu.cc
@@ -51,20 +51,16 @@ void SplitCustom<Device, T>::operator()(
   template struct Split<Eigen::GpuDevice, T, 2>; \
   template struct Split<Eigen::GpuDevice, T, 3>;
 
-TF_CALL_GPU_NUMBER_TYPES(DEFINE_GPU_KERNELS);
-TF_CALL_complex64(DEFINE_GPU_KERNELS);
-TF_CALL_complex128(DEFINE_GPU_KERNELS);
 TF_CALL_int64(DEFINE_GPU_KERNELS);
 TF_CALL_bfloat16(DEFINE_GPU_KERNELS);
 TF_CALL_uint8(DEFINE_GPU_KERNELS);
-TF_CALL_bool(DEFINE_GPU_KERNELS);
+TF_CALL_GPU_ALL_TYPES(DEFINE_GPU_KERNELS);
 
 #undef DEFINE_GPU_KERNELS
 #define DEFINE_GPU_KERNELS(T) template struct SplitCustom<Eigen::GpuDevice, T>;
 
 TF_CALL_GPU_NUMBER_TYPES(DEFINE_GPU_KERNELS);
-TF_CALL_complex64(DEFINE_GPU_KERNELS);
-TF_CALL_complex128(DEFINE_GPU_KERNELS);
+TF_CALL_COMPLEX_TYPES(DEFINE_GPU_KERNELS);
 TF_CALL_bfloat16(DEFINE_GPU_KERNELS);
 
 #undef DEFINE_GPU_KERNELS
@@ -248,8 +244,7 @@ void SplitVOpGPULaunch<T, IntType>::Run(
 #define REGISTER_GPU_KERNEL(T) template struct SplitOpGPULaunch<T>;
 
 TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU_KERNEL);
-TF_CALL_complex64(REGISTER_GPU_KERNEL);
-TF_CALL_complex128(REGISTER_GPU_KERNEL);
+TF_CALL_COMPLEX_TYPES(REGISTER_GPU_KERNEL);
 TF_CALL_bfloat16(REGISTER_GPU_KERNEL);
 #undef REGISTER_GPU_KERNEL
 #define REGISTER_GPU_KERNEL(T)                 \
@@ -257,8 +252,7 @@ TF_CALL_bfloat16(REGISTER_GPU_KERNEL);
   template struct SplitVOpGPULaunch<T, int64>;
 
 TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU_KERNEL);
-TF_CALL_complex64(REGISTER_GPU_KERNEL);
-TF_CALL_complex128(REGISTER_GPU_KERNEL);
+TF_CALL_COMPLEX_TYPES(REGISTER_GPU_KERNEL);
 TF_CALL_bfloat16(REGISTER_GPU_KERNEL);
 #undef REGISTER_GPU_KERNEL
 
diff --git a/tensorflow/core/kernels/split_lib_gpu.h b/tensorflow/core/kernels/split_lib_gpu.h
index 20feb7df143..f2c343ae5fd 100644
--- a/tensorflow/core/kernels/split_lib_gpu.h
+++ b/tensorflow/core/kernels/split_lib_gpu.h
@@ -50,12 +50,9 @@ struct SplitVOpGPULaunch {
   extern template struct SplitVOpGPULaunch<T, int32>; \
   extern template struct SplitVOpGPULaunch<T, int64>;
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU_KERNEL);
-TF_CALL_complex64(REGISTER_GPU_KERNEL);
-TF_CALL_complex128(REGISTER_GPU_KERNEL);
 TF_CALL_bfloat16(REGISTER_GPU_KERNEL);
 TF_CALL_uint8(REGISTER_GPU_KERNEL);
-TF_CALL_bool(REGISTER_GPU_KERNEL);
+TF_CALL_GPU_ALL_TYPES(REGISTER_GPU_KERNEL);
 #undef REGISTER_GPU_KERNEL
 
 }  // namespace tensorflow
diff --git a/tensorflow/core/kernels/split_op.cc b/tensorflow/core/kernels/split_op.cc
index f8bf5bce197..f09740c6198 100644
--- a/tensorflow/core/kernels/split_op.cc
+++ b/tensorflow/core/kernels/split_op.cc
@@ -417,10 +417,9 @@ REGISTER_SPLIT(uint64);
                               .HostMemory("split_dim"),  \
                           SplitOpGPU<type>)
 
+TF_CALL_bfloat16(REGISTER_GPU);
 TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
-TF_CALL_complex64(REGISTER_GPU);
-TF_CALL_complex128(REGISTER_GPU);
-REGISTER_GPU(bfloat16);
+TF_CALL_COMPLEX_TYPES(REGISTER_GPU);
 #undef REGISTER_GPU
 
 #endif  // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
diff --git a/tensorflow/core/kernels/split_v_op.cc b/tensorflow/core/kernels/split_v_op.cc
index 1eaeda927f8..4569e11dd13 100644
--- a/tensorflow/core/kernels/split_v_op.cc
+++ b/tensorflow/core/kernels/split_v_op.cc
@@ -471,10 +471,9 @@ TF_CALL_ALL_TYPES(REGISTER_SPLIT_LEN);
   REGISTER_GPU(type, int32);   \
   REGISTER_GPU(type, int64);
 
+TF_CALL_bfloat16(REGISTER_GPU_LEN);
 TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU_LEN);
-TF_CALL_complex64(REGISTER_GPU_LEN);
-TF_CALL_complex128(REGISTER_GPU_LEN);
-REGISTER_GPU_LEN(bfloat16);
+TF_CALL_COMPLEX_TYPES(REGISTER_GPU_LEN);
 #undef REGISTER_GPU_LEN
 #undef REGISTER_GPU
 
diff --git a/tensorflow/core/kernels/strided_slice_op.cc b/tensorflow/core/kernels/strided_slice_op.cc
index dd23c251897..ccc1984bb98 100644
--- a/tensorflow/core/kernels/strided_slice_op.cc
+++ b/tensorflow/core/kernels/strided_slice_op.cc
@@ -486,12 +486,9 @@ TF_CALL_uint64(REGISTER_STRIDED_SLICE);
                               .HostMemory("strides"),                   \
                           StridedSliceAssignOp<GPUDevice, type, true>)
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
-TF_CALL_bool(REGISTER_GPU);
 TF_CALL_int8(REGISTER_GPU);
-TF_CALL_complex64(REGISTER_GPU);
-TF_CALL_complex128(REGISTER_GPU);
 TF_CALL_int64(REGISTER_GPU);
+TF_CALL_GPU_ALL_TYPES(REGISTER_GPU);
 
 // A special GPU kernel for int32.
 // TODO(b/25387198): Also enable int32 in device memory. This kernel
diff --git a/tensorflow/core/kernels/strided_slice_op_gpu_complex.cu.cc b/tensorflow/core/kernels/strided_slice_op_gpu_complex.cu.cc
index a930b8a3fac..33d94f4fc71 100644
--- a/tensorflow/core/kernels/strided_slice_op_gpu_complex.cu.cc
+++ b/tensorflow/core/kernels/strided_slice_op_gpu_complex.cu.cc
@@ -21,8 +21,7 @@ limitations under the License.
 #include "tensorflow/core/kernels/strided_slice_op_gpu_impl.h"
 
 namespace tensorflow {
-TF_CALL_complex64(DEFINE_GPU_KERNELS);
-TF_CALL_complex128(DEFINE_GPU_KERNELS);
+TF_CALL_COMPLEX_TYPES(DEFINE_GPU_KERNELS);
 }  // end namespace tensorflow
 
 #endif  // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
diff --git a/tensorflow/core/kernels/strided_slice_op_impl.h b/tensorflow/core/kernels/strided_slice_op_impl.h
index bf69a19abc5..1ae959b7b3f 100644
--- a/tensorflow/core/kernels/strided_slice_op_impl.h
+++ b/tensorflow/core/kernels/strided_slice_op_impl.h
@@ -278,16 +278,12 @@ class HandleStridedSliceAssignCase<Device, T, 0> {
 
 #if GOOGLE_CUDA || TENSORFLOW_USE_ROCM
 TF_CALL_GPU_PROXY_TYPES(PREVENT_FOR_N_GPU);
-TF_CALL_complex64(PREVENT_FOR_N_GPU);
-TF_CALL_complex128(PREVENT_FOR_N_GPU);
+TF_CALL_COMPLEX_TYPES(PREVENT_FOR_N_GPU);
 
-TF_CALL_GPU_NUMBER_TYPES(DECLARE_FOR_N_GPU);
-TF_CALL_complex64(DECLARE_FOR_N_GPU);
-TF_CALL_complex128(DECLARE_FOR_N_GPU);
-TF_CALL_bool(DECLARE_FOR_N_GPU);
 TF_CALL_int8(DECLARE_FOR_N_GPU);
-DECLARE_FOR_N_GPU(int32);
-DECLARE_FOR_N_GPU(int64);
+TF_CALL_int32(DECLARE_FOR_N_GPU);
+TF_CALL_int64(DECLARE_FOR_N_GPU);
+TF_CALL_GPU_ALL_TYPES(DECLARE_FOR_N_GPU);
 #endif  // END GOOGLE_CUDA || TENSORFLOW_USE_ROCM
 
 TF_CALL_ALL_TYPES(DECLARE_FOR_N_CPU);
diff --git a/tensorflow/core/kernels/tensor_array.cc b/tensorflow/core/kernels/tensor_array.cc
index 69efc016a1f..8613ed8c80d 100644
--- a/tensorflow/core/kernels/tensor_array.cc
+++ b/tensorflow/core/kernels/tensor_array.cc
@@ -46,8 +46,7 @@ TF_CALL_NUMBER_TYPES(TENSOR_ARRAY_WRITE_OR_ADD_CPU)
 
 #define TENSOR_ARRAY_WRITE_OR_ADD_GPU(T) TENSOR_ARRAY_WRITE_OR_ADD(GPUDevice, T)
 TF_CALL_GPU_NUMBER_TYPES(TENSOR_ARRAY_WRITE_OR_ADD_GPU);
-TF_CALL_complex64(TENSOR_ARRAY_WRITE_OR_ADD_GPU);
-TF_CALL_complex128(TENSOR_ARRAY_WRITE_OR_ADD_GPU);
+TF_CALL_COMPLEX_TYPES(TENSOR_ARRAY_WRITE_OR_ADD_GPU);
 #undef TENSOR_ARRAY_WRITE_OR_ADD_GPU
 
 #endif  // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
@@ -71,8 +70,7 @@ TF_CALL_bool(TENSOR_ARRAY_SET_ZERO_CPU);
 
 #define TENSOR_ARRAY_SET_ZERO_GPU(T) TENSOR_ARRAY_SET_ZERO(GPUDevice, T)
 TF_CALL_GPU_NUMBER_TYPES(TENSOR_ARRAY_SET_ZERO_GPU);
-TF_CALL_complex64(TENSOR_ARRAY_SET_ZERO_GPU);
-TF_CALL_complex128(TENSOR_ARRAY_SET_ZERO_GPU);
+TF_CALL_COMPLEX_TYPES(TENSOR_ARRAY_SET_ZERO_GPU);
 #undef TENSOR_ARRAY_SET_ZERO_GPU
 
 #endif  // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
diff --git a/tensorflow/core/kernels/tensor_array.h b/tensorflow/core/kernels/tensor_array.h
index 405eba6c542..9bed5c533cf 100644
--- a/tensorflow/core/kernels/tensor_array.h
+++ b/tensorflow/core/kernels/tensor_array.h
@@ -61,8 +61,7 @@ TF_CALL_NUMBER_TYPES(TENSOR_ARRAY_WRITE_OR_ADD_CPU)
 
 #define TENSOR_ARRAY_WRITE_OR_ADD_GPU(T) TENSOR_ARRAY_WRITE_OR_ADD(GPUDevice, T)
 TF_CALL_GPU_NUMBER_TYPES(TENSOR_ARRAY_WRITE_OR_ADD_GPU);
-TF_CALL_complex64(TENSOR_ARRAY_WRITE_OR_ADD_GPU);
-TF_CALL_complex128(TENSOR_ARRAY_WRITE_OR_ADD_GPU);
+TF_CALL_COMPLEX_TYPES(TENSOR_ARRAY_WRITE_OR_ADD_GPU);
 #undef TENSOR_ARRAY_WRITE_OR_ADD_GPU
 
 #endif  // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
@@ -89,8 +88,7 @@ TF_CALL_bool(TENSOR_ARRAY_SET_ZERO_CPU);
 
 #define TENSOR_ARRAY_SET_ZERO_GPU(T) TENSOR_ARRAY_SET_ZERO(GPUDevice, T)
 TF_CALL_GPU_NUMBER_TYPES(TENSOR_ARRAY_SET_ZERO_GPU);
-TF_CALL_complex64(TENSOR_ARRAY_SET_ZERO_GPU);
-TF_CALL_complex128(TENSOR_ARRAY_SET_ZERO_GPU);
+TF_CALL_COMPLEX_TYPES(TENSOR_ARRAY_SET_ZERO_GPU);
 #undef TENSOR_ARRAY_SET_ZERO_GPU
 
 #endif  // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
diff --git a/tensorflow/core/kernels/tensor_array_ops.cc b/tensorflow/core/kernels/tensor_array_ops.cc
index ea8e04a33f4..048048cea9e 100644
--- a/tensorflow/core/kernels/tensor_array_ops.cc
+++ b/tensorflow/core/kernels/tensor_array_ops.cc
@@ -256,11 +256,10 @@ REGISTER_KERNEL_BUILDER(Name("TensorArrayV3").Device(DEVICE_CPU),
                               .HostMemory("handle"),         \
                           TensorArrayOp);
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
-TF_CALL_complex64(REGISTER_GPU);
-TF_CALL_complex128(REGISTER_GPU);
 TF_CALL_int64(REGISTER_GPU);
-REGISTER_GPU(bfloat16);
+TF_CALL_bfloat16(REGISTER_GPU);
+TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
+TF_CALL_COMPLEX_TYPES(REGISTER_GPU);
 #undef REGISTER_GPU
 
 #endif  // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
@@ -483,10 +482,9 @@ TF_CALL_ALL_TYPES(REGISTER_WRITE);
                               .HostMemory("index"),             \
                           TensorArrayWriteOp<GPUDevice, type>);
 
+TF_CALL_bfloat16(REGISTER_GPU);
 TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
-TF_CALL_complex64(REGISTER_GPU);
-TF_CALL_complex128(REGISTER_GPU);
-REGISTER_GPU(bfloat16);
+TF_CALL_COMPLEX_TYPES(REGISTER_GPU);
 #undef REGISTER_GPU
 
 #endif  // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
@@ -572,11 +570,10 @@ TF_CALL_ALL_TYPES(REGISTER_READ)
                               .HostMemory("index"),            \
                           TensorArrayReadOp<GPUDevice, type>);
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
-TF_CALL_complex64(REGISTER_GPU);
-TF_CALL_complex128(REGISTER_GPU);
 TF_CALL_int64(REGISTER_GPU);
-REGISTER_GPU(bfloat16);
+TF_CALL_bfloat16(REGISTER_GPU);
+TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
+TF_CALL_COMPLEX_TYPES(REGISTER_GPU);
 #undef REGISTER_GPU
 
 #endif  // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
@@ -774,10 +771,9 @@ REGISTER_GATHER_AND_PACK(qint32);
           .HostMemory("handle"),                                            \
       TensorArrayPackOrGatherOp<GPUDevice, type, false /* LEGACY_PACK */>);
 
+TF_CALL_bfloat16(REGISTER_GPU);
 TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
-TF_CALL_complex64(REGISTER_GPU);
-TF_CALL_complex128(REGISTER_GPU);
-REGISTER_GPU(bfloat16);
+TF_CALL_COMPLEX_TYPES(REGISTER_GPU);
 #undef REGISTER_GPU
 
 // A special GPU kernel for int32.
@@ -995,10 +991,9 @@ REGISTER_CONCAT(qint32);
                               .HostMemory("handle"),             \
                           TensorArrayConcatOp<GPUDevice, type>)
 
+TF_CALL_bfloat16(REGISTER_GPU);
 TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
-TF_CALL_complex64(REGISTER_GPU);
-TF_CALL_complex128(REGISTER_GPU);
-REGISTER_GPU(bfloat16);
+TF_CALL_COMPLEX_TYPES(REGISTER_GPU);
 #undef REGISTER_GPU
 
 // A special GPU kernel for int32.
@@ -1215,10 +1210,9 @@ TF_CALL_ALL_TYPES(REGISTER_SCATTER_AND_UNPACK);
       TensorArrayUnpackOrScatterOp<GPUDevice, type,             \
                                    false /* LEGACY_UNPACK */>);
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
-TF_CALL_complex64(REGISTER_GPU);
-TF_CALL_complex128(REGISTER_GPU);
 TF_CALL_int64(REGISTER_GPU);
+TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
+TF_CALL_COMPLEX_TYPES(REGISTER_GPU);
 #undef REGISTER_GPU
 
 #endif  // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
@@ -1387,8 +1381,7 @@ TF_CALL_ALL_TYPES(REGISTER_SPLIT);
                           TensorArraySplitOp<GPUDevice, type>);
 
 TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
-TF_CALL_complex64(REGISTER_GPU);
-TF_CALL_complex128(REGISTER_GPU);
+TF_CALL_COMPLEX_TYPES(REGISTER_GPU);
 #undef REGISTER_GPU
 
 #endif  // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
diff --git a/tensorflow/core/kernels/training_ops.cc b/tensorflow/core/kernels/training_ops.cc
index 7e2a8b363c5..b3714e4d27e 100644
--- a/tensorflow/core/kernels/training_ops.cc
+++ b/tensorflow/core/kernels/training_ops.cc
@@ -727,12 +727,8 @@ class ApplyGradientDescentOp<SYCLDevice, T> : public OpKernel {
                           ApplyGradientDescentOp<D##Device, T>);
 #define REGISTER_CPU_KERNELS(T) REGISTER_KERNELS(CPU, T);
 
-TF_CALL_half(REGISTER_CPU_KERNELS);
-TF_CALL_bfloat16(REGISTER_CPU_KERNELS);
-TF_CALL_float(REGISTER_CPU_KERNELS);
-TF_CALL_double(REGISTER_CPU_KERNELS);
-TF_CALL_complex64(REGISTER_CPU_KERNELS);
-TF_CALL_complex128(REGISTER_CPU_KERNELS);
+TF_CALL_FLOAT_TYPES(REGISTER_CPU_KERNELS);
+TF_CALL_COMPLEX_TYPES(REGISTER_CPU_KERNELS);
 
 #if GOOGLE_CUDA || TENSORFLOW_USE_ROCM
 // Forward declarations of the functor specializations for GPU.
@@ -898,12 +894,8 @@ class ApplyAdadeltaOp : public OpKernel {
                           ApplyAdadeltaOp<D##Device, T>);
 #define REGISTER_CPU_KERNELS(T) REGISTER_KERNELS(CPU, T);
 
-TF_CALL_half(REGISTER_CPU_KERNELS);
-TF_CALL_bfloat16(REGISTER_CPU_KERNELS);
-TF_CALL_float(REGISTER_CPU_KERNELS);
-TF_CALL_double(REGISTER_CPU_KERNELS);
-TF_CALL_complex64(REGISTER_CPU_KERNELS);
-TF_CALL_complex128(REGISTER_CPU_KERNELS);
+TF_CALL_FLOAT_TYPES(REGISTER_CPU_KERNELS);
+TF_CALL_COMPLEX_TYPES(REGISTER_CPU_KERNELS);
 
 #if GOOGLE_CUDA || TENSORFLOW_USE_ROCM
 // Forward declarations of the functor specializations for GPU.
@@ -1089,12 +1081,8 @@ class SparseApplyAdadeltaOp : public OpKernel {
   REGISTER_KERNELS(T, int32);   \
   REGISTER_KERNELS(T, int64);
 
-TF_CALL_half(REGISTER_CPU_KERNELS);
-TF_CALL_bfloat16(REGISTER_CPU_KERNELS);
-TF_CALL_float(REGISTER_CPU_KERNELS);
-TF_CALL_double(REGISTER_CPU_KERNELS);
-TF_CALL_complex64(REGISTER_CPU_KERNELS);
-TF_CALL_complex128(REGISTER_CPU_KERNELS);
+TF_CALL_FLOAT_TYPES(REGISTER_CPU_KERNELS);
+TF_CALL_COMPLEX_TYPES(REGISTER_CPU_KERNELS);
 
 #undef REGISTER_CPU_KERNELS
 #undef REGISTER_KERNELS
@@ -1383,12 +1371,8 @@ class ApplyAdagradOp : public OpKernel {
                           ApplyAdagradOp<D##Device, T>);
 #define REGISTER_CPU_KERNELS(T) REGISTER_KERNELS(CPU, T);
 
-TF_CALL_half(REGISTER_CPU_KERNELS);
-TF_CALL_bfloat16(REGISTER_CPU_KERNELS);
-TF_CALL_float(REGISTER_CPU_KERNELS);
-TF_CALL_double(REGISTER_CPU_KERNELS);
-TF_CALL_complex64(REGISTER_CPU_KERNELS);
-TF_CALL_complex128(REGISTER_CPU_KERNELS);
+TF_CALL_FLOAT_TYPES(REGISTER_CPU_KERNELS);
+TF_CALL_COMPLEX_TYPES(REGISTER_CPU_KERNELS);
 
 #if GOOGLE_CUDA || TENSORFLOW_USE_ROCM
 // Forward declarations of the functor specializations for GPU.
@@ -1492,12 +1476,8 @@ class ApplyAdagradV2Op : public OpKernel {
                           ApplyAdagradV2Op<D##Device, T>);
 #define REGISTER_CPU_KERNELS(T) REGISTER_KERNELS(CPU, T);
 
-TF_CALL_half(REGISTER_CPU_KERNELS);
-TF_CALL_bfloat16(REGISTER_CPU_KERNELS);
-TF_CALL_float(REGISTER_CPU_KERNELS);
-TF_CALL_double(REGISTER_CPU_KERNELS);
-TF_CALL_complex64(REGISTER_CPU_KERNELS);
-TF_CALL_complex128(REGISTER_CPU_KERNELS);
+TF_CALL_FLOAT_TYPES(REGISTER_CPU_KERNELS);
+TF_CALL_COMPLEX_TYPES(REGISTER_CPU_KERNELS);
 
 #if GOOGLE_CUDA || TENSORFLOW_USE_ROCM
 // Forward declarations of the functor specializations for GPU.
@@ -1801,12 +1781,8 @@ class SparseApplyAdagradOp : public OpKernel {
   REGISTER_KERNELS(T, int32);   \
   REGISTER_KERNELS(T, int64);
 
-TF_CALL_half(REGISTER_CPU_KERNELS);
-TF_CALL_bfloat16(REGISTER_CPU_KERNELS);
-TF_CALL_float(REGISTER_CPU_KERNELS);
-TF_CALL_double(REGISTER_CPU_KERNELS);
-TF_CALL_complex64(REGISTER_CPU_KERNELS);
-TF_CALL_complex128(REGISTER_CPU_KERNELS);
+TF_CALL_FLOAT_TYPES(REGISTER_CPU_KERNELS);
+TF_CALL_COMPLEX_TYPES(REGISTER_CPU_KERNELS);
 
 #undef REGISTER_CPU_KERNELS
 #undef REGISTER_KERNELS
@@ -1976,12 +1952,8 @@ class SparseApplyAdagradV2Op : public OpKernel {
   REGISTER_KERNELS(T, int32);   \
   REGISTER_KERNELS(T, int64);
 
-TF_CALL_half(REGISTER_CPU_KERNELS);
-TF_CALL_bfloat16(REGISTER_CPU_KERNELS);
-TF_CALL_float(REGISTER_CPU_KERNELS);
-TF_CALL_double(REGISTER_CPU_KERNELS);
-TF_CALL_complex64(REGISTER_CPU_KERNELS);
-TF_CALL_complex128(REGISTER_CPU_KERNELS);
+TF_CALL_FLOAT_TYPES(REGISTER_CPU_KERNELS);
+TF_CALL_COMPLEX_TYPES(REGISTER_CPU_KERNELS);
 
 #undef REGISTER_CPU_KERNELS
 #undef REGISTER_KERNELS
@@ -3054,12 +3026,8 @@ class ApplyMomentumOp : public OpKernel {
                           ApplyMomentumOp<D##Device, T>);
 #define REGISTER_CPU_KERNELS(T) REGISTER_KERNELS(CPU, T);
 
-TF_CALL_half(REGISTER_CPU_KERNELS);
-TF_CALL_bfloat16(REGISTER_CPU_KERNELS);
-TF_CALL_float(REGISTER_CPU_KERNELS);
-TF_CALL_double(REGISTER_CPU_KERNELS);
-TF_CALL_complex64(REGISTER_CPU_KERNELS);
-TF_CALL_complex128(REGISTER_CPU_KERNELS);
+TF_CALL_FLOAT_TYPES(REGISTER_CPU_KERNELS);
+TF_CALL_COMPLEX_TYPES(REGISTER_CPU_KERNELS);
 
 #if GOOGLE_CUDA || TENSORFLOW_USE_ROCM
 // Forward declarations of the functor specializations for GPU.
@@ -3209,12 +3177,8 @@ class SparseApplyMomentumOp : public OpKernel {
   REGISTER_KERNELS(T, int32);   \
   REGISTER_KERNELS(T, int64);
 
-TF_CALL_half(REGISTER_CPU_KERNELS);
-TF_CALL_bfloat16(REGISTER_CPU_KERNELS);
-TF_CALL_float(REGISTER_CPU_KERNELS);
-TF_CALL_double(REGISTER_CPU_KERNELS);
-TF_CALL_complex64(REGISTER_CPU_KERNELS);
-TF_CALL_complex128(REGISTER_CPU_KERNELS);
+TF_CALL_FLOAT_TYPES(REGISTER_CPU_KERNELS);
+TF_CALL_COMPLEX_TYPES(REGISTER_CPU_KERNELS);
 
 #undef REGISTER_CPU_KERNELS
 #undef REGISTER_KERNELS
@@ -3288,12 +3252,8 @@ class ApplyKerasMomentumOp : public OpKernel {
                           ApplyKerasMomentumOp<D##Device, T>);
 #define REGISTER_CPU_KERNELS(T) REGISTER_KERNELS(CPU, T);
 
-TF_CALL_half(REGISTER_CPU_KERNELS);
-TF_CALL_bfloat16(REGISTER_CPU_KERNELS);
-TF_CALL_float(REGISTER_CPU_KERNELS);
-TF_CALL_double(REGISTER_CPU_KERNELS);
-TF_CALL_complex64(REGISTER_CPU_KERNELS);
-TF_CALL_complex128(REGISTER_CPU_KERNELS);
+TF_CALL_FLOAT_TYPES(REGISTER_CPU_KERNELS);
+TF_CALL_COMPLEX_TYPES(REGISTER_CPU_KERNELS);
 
 #if GOOGLE_CUDA || TENSORFLOW_USE_ROCM
 // Forward declarations of the functor specializations for GPU.
@@ -3423,12 +3383,9 @@ class SparseApplyKerasMomentumOp : public OpKernel {
   REGISTER_KERNELS(T, CPU, int32); \
   REGISTER_KERNELS(T, CPU, int64);
 
-TF_CALL_half(REGISTER_CPU_KERNELS);
-TF_CALL_bfloat16(REGISTER_CPU_KERNELS);
-TF_CALL_float(REGISTER_CPU_KERNELS);
-TF_CALL_double(REGISTER_CPU_KERNELS);
-TF_CALL_complex64(REGISTER_CPU_KERNELS);
-TF_CALL_complex128(REGISTER_CPU_KERNELS);
+TF_CALL_FLOAT_TYPES(REGISTER_CPU_KERNELS);
+TF_CALL_COMPLEX_TYPES(REGISTER_CPU_KERNELS);
+
 #undef REGISTER_CPU_KERNELS
 
 #if GOOGLE_CUDA || TENSORFLOW_USE_ROCM
@@ -3699,12 +3656,8 @@ class ApplyAdamOp<SYCLDevice, T> : public OpKernel {
                           ApplyAdamOp<D##Device, T>);
 #define REGISTER_CPU_KERNELS(T) REGISTER_KERNELS(CPU, T);
 
-TF_CALL_half(REGISTER_CPU_KERNELS);
-TF_CALL_bfloat16(REGISTER_CPU_KERNELS);
-TF_CALL_float(REGISTER_CPU_KERNELS);
-TF_CALL_double(REGISTER_CPU_KERNELS);
-TF_CALL_complex64(REGISTER_CPU_KERNELS);
-TF_CALL_complex128(REGISTER_CPU_KERNELS);
+TF_CALL_FLOAT_TYPES(REGISTER_CPU_KERNELS);
+TF_CALL_COMPLEX_TYPES(REGISTER_CPU_KERNELS);
 
 #ifdef TENSORFLOW_USE_SYCL
 #define REGISTER_SYCL_KERNELS(T) REGISTER_KERNELS(SYCL, T);
@@ -4226,12 +4179,8 @@ class ApplyCenteredRMSPropOp : public OpKernel {
                           ApplyCenteredRMSPropOp<D##Device, T>);
 #define REGISTER_CPU_KERNELS(T) REGISTER_KERNELS(CPU, T);
 
-TF_CALL_half(REGISTER_CPU_KERNELS);
-TF_CALL_bfloat16(REGISTER_CPU_KERNELS);
-TF_CALL_float(REGISTER_CPU_KERNELS);
-TF_CALL_double(REGISTER_CPU_KERNELS);
-TF_CALL_complex64(REGISTER_CPU_KERNELS);
-TF_CALL_complex128(REGISTER_CPU_KERNELS);
+TF_CALL_FLOAT_TYPES(REGISTER_CPU_KERNELS);
+TF_CALL_COMPLEX_TYPES(REGISTER_CPU_KERNELS);
 
 #if GOOGLE_CUDA || TENSORFLOW_USE_ROCM
 // Forward declarations of the functor specializations for GPU.
diff --git a/tensorflow/core/kernels/unpack_op.cc b/tensorflow/core/kernels/unpack_op.cc
index ce02aa17225..7ac02e8b4d4 100644
--- a/tensorflow/core/kernels/unpack_op.cc
+++ b/tensorflow/core/kernels/unpack_op.cc
@@ -144,12 +144,9 @@ TF_CALL_ALL_TYPES(REGISTER_UNPACK);
       Name("Unpack").Device(DEVICE_GPU).TypeConstraint<type>("T"), \
       UnpackOp<GPUDevice, type>)
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU);
 TF_CALL_bfloat16(REGISTER_GPU);
 TF_CALL_uint8(REGISTER_GPU);
-TF_CALL_bool(REGISTER_GPU);
-TF_CALL_complex64(REGISTER_GPU);
-TF_CALL_complex128(REGISTER_GPU);
+TF_CALL_GPU_ALL_TYPES(REGISTER_GPU);
 #undef REGISTER_GPU
 
 // A special GPU kernel for int32.
diff --git a/tensorflow/core/kernels/variable_ops.cc b/tensorflow/core/kernels/variable_ops.cc
index d3aadf04144..6f5e0b94eca 100644
--- a/tensorflow/core/kernels/variable_ops.cc
+++ b/tensorflow/core/kernels/variable_ops.cc
@@ -16,13 +16,31 @@ limitations under the License.
 #define EIGEN_USE_THREADS
 #include "tensorflow/core/kernels/variable_ops.h"
 
+#include "tensorflow/core/framework/control_flow.h"
 #include "tensorflow/core/framework/op_kernel.h"
 #include "tensorflow/core/framework/register_types.h"
 #include "tensorflow/core/lib/core/errors.h"
+#include "tensorflow/core/platform/strcat.h"
 #include "tensorflow/core/platform/types.h"
 
 namespace tensorflow {
 
+namespace {
+
+// Makes a unique name for a temporary variable inside a while loop body,
+// because loop can be executed in multiple iterations in parallel.
+string TemporaryVariableName(const string& var_name,
+                             const FrameAndIter& control_frame) {
+  if (control_frame.frame_id != kIllegalFrameId &&
+      control_frame.iter_id != kIllegalIterId) {
+    return strings::StrCat(var_name, "/frame:", control_frame.frame_id,
+                           "/iter:", control_frame.iter_id);
+  }
+  return var_name;
+}
+
+}  // namespace
+
 // Resource stored by variables in the resource manager
 // (legacy, ref-style version).
 class LegacyVar : public ResourceBase {
@@ -89,15 +107,16 @@ class TemporaryVariableOp : public OpKernel {
     Status s;
     ResourceMgr* rm = context->resource_manager();
     OP_REQUIRES(context, rm, errors::Internal("No per-step resource manager."));
+    auto unique_name = TemporaryVariableName(var_name_, context->frame_iter());
     auto* tmp_var = new TmpVar;
     OP_REQUIRES(context, tmp_var,
                 errors::ResourceExhausted("Could not allocate TmpVar."));
-    tmp_var->name = var_name_;
+    tmp_var->name = unique_name;
     s = context->allocate_temp(dtype_, shape_, &tmp_var->val);
     if (!s.ok()) tmp_var->Unref();
     OP_REQUIRES_OK(context, s);
     OP_REQUIRES_OK(context,
-                   context->step_container()->Create(rm, var_name_, tmp_var));
+                   context->step_container()->Create(rm, unique_name, tmp_var));
     context->set_output_ref(0, &tmp_var->mu, &tmp_var->val);
     if (context->track_allocations()) {
       context->record_persistent_memory_allocation(
@@ -141,9 +160,10 @@ class DestroyTemporaryVariableOp : public OpKernel {
     context->set_output(0, tmpvar);
     ResourceMgr* rm = context->resource_manager();
     OP_REQUIRES(context, rm, errors::Internal("No per-step resource manager."));
+    auto unique_name = TemporaryVariableName(var_name_, context->frame_iter());
     OP_REQUIRES_OK(
         context, context->step_container()->Delete<TemporaryVariableOp::TmpVar>(
-                     rm, var_name_));
+                     rm, unique_name));
     if (context->track_allocations()) {
       context->record_persistent_memory_allocation(
           -static_cast<int64>(tmpvar.AllocatedBytes()));
@@ -230,11 +250,10 @@ TF_CALL_GPU_NUMBER_TYPES_NO_HALF(REGISTER_SYCL_KERNEL);
                               .HostMemory("is_initialized"),               \
                           IsVariableInitializedOp);
 
-TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU_KERNELS);
-TF_CALL_complex64(REGISTER_GPU_KERNELS);
-TF_CALL_complex128(REGISTER_GPU_KERNELS);
 TF_CALL_int64(REGISTER_GPU_KERNELS);
 TF_CALL_uint32(REGISTER_GPU_KERNELS);
+TF_CALL_GPU_NUMBER_TYPES(REGISTER_GPU_KERNELS);
+TF_CALL_COMPLEX_TYPES(REGISTER_GPU_KERNELS);
 #undef REGISTER_GPU_KERNELS
 #endif  // GOOGLE_CUDA || TENSORFLOW_USE_ROCM
 
diff --git a/tensorflow/core/lib/io/random_inputstream.cc b/tensorflow/core/lib/io/random_inputstream.cc
index 10f734a5bae..bd0054ce753 100644
--- a/tensorflow/core/lib/io/random_inputstream.cc
+++ b/tensorflow/core/lib/io/random_inputstream.cc
@@ -92,7 +92,7 @@ Status RandomAccessInputStream::SkipNBytes(int64 bytes_to_skip) {
     } else {
       return s;
     }
-    if (data.size() < bytes_to_read) {
+    if (data.size() < static_cast<size_t>(bytes_to_read)) {
       return errors::OutOfRange("reached end of file");
     }
     bytes_to_skip -= bytes_to_read;
diff --git a/tensorflow/core/lib/io/recordio_test.cc b/tensorflow/core/lib/io/recordio_test.cc
index f21f88dfa22..ee02fcbbb9e 100644
--- a/tensorflow/core/lib/io/recordio_test.cc
+++ b/tensorflow/core/lib/io/recordio_test.cc
@@ -64,7 +64,7 @@ class StringDest : public WritableFile {
   }
 #if defined(PLATFORM_GOOGLE)
   Status Append(const absl::Cord& data) override {
-    contents_->append(data.ToString());
+    contents_->append(std::string(data));
     return Status::OK();
   }
 #endif
diff --git a/tensorflow/core/lib/io/snappy/snappy_inputbuffer.cc b/tensorflow/core/lib/io/snappy/snappy_inputbuffer.cc
index a331d4173cf..53939f2d8a3 100644
--- a/tensorflow/core/lib/io/snappy/snappy_inputbuffer.cc
+++ b/tensorflow/core/lib/io/snappy/snappy_inputbuffer.cc
@@ -134,7 +134,7 @@ Status SnappyInputBuffer::ReadCompressedBlockLength(uint32* length) {
     }
     size_t readable = std::min(bytes_to_read, avail_in_);
 
-    for (int i = 0; i < readable; i++) {
+    for (size_t i = 0; i < readable; i++) {
       // The "unsigned char" type cast is intentional to avoid implicit type
       // casting of the signed char to unsigned int during bitwise OR which
       // causes weird overflow errors.
diff --git a/tensorflow/core/lib/io/snappy/snappy_outputbuffer.cc b/tensorflow/core/lib/io/snappy/snappy_outputbuffer.cc
index 563503a1319..fe3a53c6c25 100644
--- a/tensorflow/core/lib/io/snappy/snappy_outputbuffer.cc
+++ b/tensorflow/core/lib/io/snappy/snappy_outputbuffer.cc
@@ -76,7 +76,7 @@ Status SnappyOutputBuffer::Write(StringPiece data) {
 
   // If there is sufficient free space in input_buffer_ to fit data we
   // add it there and return.
-  if (bytes_to_write <= AvailableInputSpace()) {
+  if (static_cast<int32>(bytes_to_write) <= AvailableInputSpace()) {
     AddToInputBuffer(data);
     return Status::OK();
   }
@@ -87,7 +87,7 @@ Status SnappyOutputBuffer::Write(StringPiece data) {
   TF_RETURN_IF_ERROR(DeflateBuffered());
 
   // input_buffer_ should be empty at this point.
-  if (bytes_to_write <= AvailableInputSpace()) {
+  if (static_cast<int32>(bytes_to_write) <= AvailableInputSpace()) {
     AddToInputBuffer(data);
     return Status::OK();
   }
@@ -144,7 +144,7 @@ void SnappyOutputBuffer::AddToInputBuffer(StringPiece data) {
   const int32 free_tail_bytes =
       input_buffer_capacity_ - (read_bytes + unread_bytes);
 
-  if (bytes_to_write > free_tail_bytes) {
+  if (static_cast<int32>(bytes_to_write) > free_tail_bytes) {
     memmove(input_buffer_.get(), next_in_, avail_in_);
     next_in_ = input_buffer_.get();
   }
diff --git a/tensorflow/core/lib/io/zlib_outputbuffer.cc b/tensorflow/core/lib/io/zlib_outputbuffer.cc
index 5840ca60242..d475d0eaa5c 100644
--- a/tensorflow/core/lib/io/zlib_outputbuffer.cc
+++ b/tensorflow/core/lib/io/zlib_outputbuffer.cc
@@ -98,7 +98,7 @@ void ZlibOutputBuffer::AddToInputBuffer(StringPiece data) {
   int32 unread_bytes = z_stream_->avail_in;
   int32 free_tail_bytes = input_buffer_capacity_ - (read_bytes + unread_bytes);
 
-  if (bytes_to_write > free_tail_bytes) {
+  if (static_cast<int32>(bytes_to_write) > free_tail_bytes) {
     memmove(z_stream_input_.get(), z_stream_->next_in, z_stream_->avail_in);
     z_stream_->next_in = z_stream_input_.get();
   }
@@ -154,7 +154,7 @@ Status ZlibOutputBuffer::Append(StringPiece data) {
 
   size_t bytes_to_write = data.size();
 
-  if (bytes_to_write <= AvailableInputSpace()) {
+  if (static_cast<int32>(bytes_to_write) <= AvailableInputSpace()) {
     AddToInputBuffer(data);
     return Status::OK();
   }
@@ -162,7 +162,7 @@ Status ZlibOutputBuffer::Append(StringPiece data) {
   TF_RETURN_IF_ERROR(DeflateBuffered(zlib_options_.flush_mode));
 
   // At this point input stream should be empty.
-  if (bytes_to_write <= AvailableInputSpace()) {
+  if (static_cast<int32>(bytes_to_write) <= AvailableInputSpace()) {
     AddToInputBuffer(data);
     return Status::OK();
   }
diff --git a/tensorflow/core/lib/strings/proto_serialization.cc b/tensorflow/core/lib/strings/proto_serialization.cc
index 5ffc845d098..522f35c78bd 100644
--- a/tensorflow/core/lib/strings/proto_serialization.cc
+++ b/tensorflow/core/lib/strings/proto_serialization.cc
@@ -72,7 +72,8 @@ bool SerializeToBufferDeterministic(const protobuf::MessageLite& msg,
   protobuf::io::CodedOutputStream output_stream(&array_stream);
   output_stream.SetSerializationDeterministic(true);
   msg.SerializeWithCachedSizes(&output_stream);
-  return !output_stream.HadError() && size == output_stream.ByteCount();
+  return !output_stream.HadError() &&
+         size == static_cast<size_t>(output_stream.ByteCount());
 }
 
 bool AreSerializedProtosEqual(const protobuf::MessageLite& x,
diff --git a/tensorflow/core/ops/compat/ops_history_v2/DebugIdentityV2.pbtxt b/tensorflow/core/ops/compat/ops_history_v2/DebugIdentityV2.pbtxt
index 81172e5ca4b..9502555a20e 100644
--- a/tensorflow/core/ops/compat/ops_history_v2/DebugIdentityV2.pbtxt
+++ b/tensorflow/core/ops/compat/ops_history_v2/DebugIdentityV2.pbtxt
@@ -50,3 +50,62 @@ op {
   }
   is_stateful: true
 }
+op {
+  name: "DebugIdentityV2"
+  input_arg {
+    name: "input"
+    type_attr: "T"
+  }
+  output_arg {
+    name: "output"
+    type_attr: "T"
+  }
+  attr {
+    name: "T"
+    type: "type"
+  }
+  attr {
+    name: "tfdbg_context_id"
+    type: "string"
+    default_value {
+      s: ""
+    }
+  }
+  attr {
+    name: "op_name"
+    type: "string"
+    default_value {
+      s: ""
+    }
+  }
+  attr {
+    name: "output_slot"
+    type: "int"
+    default_value {
+      i: -1
+    }
+  }
+  attr {
+    name: "tensor_debug_mode"
+    type: "int"
+    default_value {
+      i: -1
+    }
+  }
+  attr {
+    name: "debug_urls"
+    type: "list(string)"
+    default_value {
+      list {
+      }
+    }
+  }
+  attr {
+    name: "circular_buffer_size"
+    type: "int"
+    default_value {
+      i: 1000
+    }
+  }
+  is_stateful: true
+}
diff --git a/tensorflow/core/ops/dataset_ops.cc b/tensorflow/core/ops/dataset_ops.cc
index 6a633fb679d..4f750cc938d 100644
--- a/tensorflow/core/ops/dataset_ops.cc
+++ b/tensorflow/core/ops/dataset_ops.cc
@@ -415,6 +415,11 @@ REGISTER_OP("AnonymousSeedGenerator")
       return Status::OK();
     });
 
+REGISTER_OP("DatasetCardinality")
+    .Input("input_dataset: variant")
+    .Output("cardinality: int64")
+    .SetShapeFn(shape_inference::ScalarShape);
+
 REGISTER_OP("DeleteSeedGenerator")
     .Input("handle: resource")
     .Input("deleter: variant")
diff --git a/tensorflow/core/ops/debug_ops.cc b/tensorflow/core/ops/debug_ops.cc
index 7977974fefc..0ecc58a6a8f 100644
--- a/tensorflow/core/ops/debug_ops.cc
+++ b/tensorflow/core/ops/debug_ops.cc
@@ -90,6 +90,7 @@ REGISTER_OP("DebugIdentityV2")
     .Attr("output_slot: int = -1")
     .Attr("tensor_debug_mode: int = -1")
     .Attr("debug_urls: list(string) = []")
+    .Attr("circular_buffer_size: int = 1000")
     .SetIsStateful()
     .SetShapeFn(shape_inference::UnchangedShape);
 
diff --git a/tensorflow/core/ops/experimental_dataset_ops.cc b/tensorflow/core/ops/experimental_dataset_ops.cc
index 18b35d3a172..ac36540e29a 100644
--- a/tensorflow/core/ops/experimental_dataset_ops.cc
+++ b/tensorflow/core/ops/experimental_dataset_ops.cc
@@ -229,11 +229,6 @@ REGISTER_OP("ExperimentalCSVDataset")
       return shape_inference::ScalarShape(c);
     });
 
-REGISTER_OP("DatasetCardinality")
-    .Input("input_dataset: variant")
-    .Output("cardinality: int64")
-    .SetShapeFn(shape_inference::ScalarShape);
-
 REGISTER_OP("ExperimentalDatasetCardinality")
     .Input("input_dataset: variant")
     .Output("cardinality: int64")
diff --git a/tensorflow/core/ops/ops.pbtxt b/tensorflow/core/ops/ops.pbtxt
index 2a681a79240..55db2a19843 100644
--- a/tensorflow/core/ops/ops.pbtxt
+++ b/tensorflow/core/ops/ops.pbtxt
@@ -10853,6 +10853,13 @@ op {
       }
     }
   }
+  attr {
+    name: "circular_buffer_size"
+    type: "int"
+    default_value {
+      i: 1000
+    }
+  }
   is_stateful: true
 }
 op {
diff --git a/tensorflow/core/platform/build_config_root.bzl b/tensorflow/core/platform/build_config_root.bzl
index 5514ccd3fbf..c5626ca8d8c 100644
--- a/tensorflow/core/platform/build_config_root.bzl
+++ b/tensorflow/core/platform/build_config_root.bzl
@@ -9,6 +9,7 @@ load(
     _tf_additional_grpc_deps_py = "tf_additional_grpc_deps_py",
     _tf_additional_license_deps = "tf_additional_license_deps",
     _tf_additional_plugin_deps = "tf_additional_plugin_deps",
+    _tf_additional_profiler_deps = "tf_additional_profiler_deps",
     _tf_additional_xla_deps_py = "tf_additional_xla_deps_py",
     _tf_cuda_tests_tags = "tf_cuda_tests_tags",
     _tf_exec_properties = "tf_exec_properties",
@@ -23,6 +24,7 @@ register_extension_info = _register_extension_info
 tf_additional_grpc_deps_py = _tf_additional_grpc_deps_py
 tf_additional_license_deps = _tf_additional_license_deps
 tf_additional_plugin_deps = _tf_additional_plugin_deps
+tf_additional_profiler_deps = _tf_additional_profiler_deps
 tf_additional_xla_deps_py = _tf_additional_xla_deps_py
 tf_cuda_tests_tags = _tf_cuda_tests_tags
 tf_exec_properties = _tf_exec_properties
diff --git a/tensorflow/core/platform/default/build_config_root.bzl b/tensorflow/core/platform/default/build_config_root.bzl
index 8b3119cec0d..3afe1de58df 100644
--- a/tensorflow/core/platform/default/build_config_root.bzl
+++ b/tensorflow/core/platform/default/build_config_root.bzl
@@ -35,6 +35,9 @@ def tf_additional_plugin_deps():
         "//conditions:default": [],
     })
 
+def tf_additional_profiler_deps():
+    return []
+
 def tf_additional_xla_deps_py():
     return []
 
diff --git a/tensorflow/core/platform/env.cc b/tensorflow/core/platform/env.cc
index b29cad05459..05d95ba0425 100644
--- a/tensorflow/core/platform/env.cc
+++ b/tensorflow/core/platform/env.cc
@@ -214,7 +214,7 @@ bool Env::FilesExist(const std::vector<string>& files,
     }
     if (fs_status) {
       result &= fs_result;
-      for (int i = 0; i < itr.second.size(); ++i) {
+      for (size_t i = 0; i < itr.second.size(); ++i) {
         per_file_status[itr.second[i]] = fs_status->at(i);
       }
     } else if (!fs_result) {
diff --git a/tensorflow/core/platform/file_system.cc b/tensorflow/core/platform/file_system.cc
index 9e96ceedbdc..c9657e2339f 100644
--- a/tensorflow/core/platform/file_system.cc
+++ b/tensorflow/core/platform/file_system.cc
@@ -308,7 +308,7 @@ StringPiece FileSystem::Basename(StringPiece path) const {
 StringPiece FileSystem::Extension(StringPiece path) const {
   StringPiece basename = this->Basename(path);
 
-  int pos = basename.rfind('.');
+  size_t pos = basename.rfind('.');
   if (pos == StringPiece::npos) {
     return StringPiece(path.data() + path.size(), 0);
   } else {
diff --git a/tensorflow/core/platform/file_system_helper.cc b/tensorflow/core/platform/file_system_helper.cc
index 64b175c4d17..909752389e1 100644
--- a/tensorflow/core/platform/file_system_helper.cc
+++ b/tensorflow/core/platform/file_system_helper.cc
@@ -103,7 +103,7 @@ Status GetMatchingPaths(FileSystem* fs, Env* env, const string& pattern,
                 children_dir_status[i] = fs->IsDirectory(child_path);
               }
             });
-    for (int i = 0; i < children.size(); ++i) {
+    for (size_t i = 0; i < children.size(); ++i) {
       const string child_path = io::JoinPath(current_dir, children[i]);
       // If the IsDirectory call was cancelled we bail.
       if (children_dir_status[i].code() == tensorflow::error::CANCELLED) {
diff --git a/tensorflow/core/platform/protobuf.h b/tensorflow/core/platform/protobuf.h
index 6b4db77ea3f..371912cc2b7 100644
--- a/tensorflow/core/platform/protobuf.h
+++ b/tensorflow/core/platform/protobuf.h
@@ -78,7 +78,7 @@ inline void SetProtobufStringSwapAllowed(std::string* src, std::string* dest) {
 // in core/platform/protobuf.h, so the generation code doesn't need to determine
 // if the type is Cord or string at generation time.
 inline std::string ProtobufStringToString(const absl::Cord& s) {
-  return s.ToString();
+  return std::string(s);
 }
 inline void SetProtobufStringSwapAllowed(std::string* src, absl::Cord* dest) {
   dest->CopyFrom(*src);
diff --git a/tensorflow/core/platform/status.cc b/tensorflow/core/platform/status.cc
index 756b8314148..c85527f27ad 100644
--- a/tensorflow/core/platform/status.cc
+++ b/tensorflow/core/platform/status.cc
@@ -74,7 +74,9 @@ class StatusLogSink : public TFLogSink {
 
     mutex_lock lock(mu_);
     messages_.emplace_back(entry.ToString());
-    if (messages_.size() > num_messages_) messages_.pop_front();
+    if (messages_.size() > static_cast<size_t>(num_messages_)) {
+      messages_.pop_front();
+    }
   }
 
  private:
diff --git a/tensorflow/core/profiler/internal/parse_annotation.cc b/tensorflow/core/profiler/internal/parse_annotation.cc
index 32c26befa3d..a4cdc09739d 100644
--- a/tensorflow/core/profiler/internal/parse_annotation.cc
+++ b/tensorflow/core/profiler/internal/parse_annotation.cc
@@ -50,7 +50,7 @@ std::vector<absl::string_view> SplitNameAndMetadata(
 std::vector<absl::string_view> SplitPairs(absl::string_view metadata) {
   std::vector<absl::string_view> key_value_pairs;
   std::stack<char> quotes;
-  int start = 0, end = 0;
+  size_t start = 0, end = 0;
   for (; end < metadata.size(); ++end) {
     char ch = metadata[end];
     switch (ch) {
diff --git a/tensorflow/core/profiler/utils/group_events.cc b/tensorflow/core/profiler/utils/group_events.cc
index 001fb78b073..8b4d68a0668 100644
--- a/tensorflow/core/profiler/utils/group_events.cc
+++ b/tensorflow/core/profiler/utils/group_events.cc
@@ -18,6 +18,7 @@ limitations under the License.
 #include <algorithm>
 #include <functional>
 #include <iterator>
+#include <map>
 #include <memory>
 #include <string>
 #include <utility>
@@ -132,10 +133,6 @@ void ConnectContextGroups(const ContextGroupMap& context_groups) {
   }
 }
 
-using VirtualEventNodeMap =
-    absl::flat_hash_map<int64 /*step_id*/,
-                        absl::flat_hash_map<int64 /*iter_num*/, EventNode*>>;
-
 std::unique_ptr<XEvent> CreateVirtualEvent(const XStat& step_id_stat,
                                            const XStat& iter_num_stat) {
   auto virtual_event = absl::make_unique<XEvent>();
@@ -144,13 +141,6 @@ std::unique_ptr<XEvent> CreateVirtualEvent(const XStat& step_id_stat,
   return virtual_event;
 }
 
-bool NeedsVirtualEventsForHostTrainingLoop(
-    const std::vector<int64 /*EventType*/>& root_event_types) {
-  return std::find(root_event_types.begin(), root_event_types.end(),
-                   HostEventType::kHostTrainingLoopIteration) !=
-         root_event_types.end();
-}
-
 bool NeedsVirtualEventsForAsyncExecutor(
     const std::vector<int64 /*EventType*/>& root_event_types) {
   return std::find(root_event_types.begin(), root_event_types.end(),
@@ -176,6 +166,16 @@ void ProcessRootEvent(int64 group_id, EventNode* root_event,
   event_group_name_map->emplace(group_id, std::move(group_name));
 }
 
+bool IsTfDataEvent(const EventNode& event_node) {
+  return event_node.FindParent(HostEventType::kTfDataCapturedFunctionRun) ||
+         event_node.FindParent(
+             HostEventType::kTfDataCapturedFunctionRunAsync) ||
+         event_node.FindParent(
+             HostEventType::kTfDataCapturedFunctionRunInstantiated) ||
+         event_node.FindParent(
+             HostEventType::kTfDataCapturedFunctionRunWithBorrowedArgs);
+}
+
 }  // namespace
 
 EventNode::EventNode(const XPlaneVisitor* plane, XLine* raw_line,
@@ -282,7 +282,7 @@ bool EventNode::IsEager() {
          FindParent(HostEventType::kEagerKernelExecute) != nullptr;
 }
 
-EventNode* EventNode::FindParent(int64 event_type) {
+EventNode* EventNode::FindParent(int64 event_type) const {
   if (parent_) {
     if (parent_->GetEventVisitor().Type() == event_type) {
       return parent_;
@@ -292,6 +292,11 @@ EventNode* EventNode::FindParent(int64 event_type) {
   return nullptr;
 }
 
+bool EventNode::StartsBefore(const EventNode& other) const {
+  return GetEventVisitor().TimestampPs() <=
+         other.GetEventVisitor().TimestampPs();
+}
+
 void EventForest::ConnectIntraThread(const XPlaneVisitor& visitor,
                                      XPlane* plane,
                                      ContextGroupMap* context_groups) {
@@ -372,16 +377,18 @@ void EventForest::ConnectInterThread(
 
 void EventForest::CreateEventGroup(
     const std::vector<int64 /*EventType*/>& root_event_types) {
-  int64 next_group_id = 0;
+  for (EventNode* root_event : tf_loop_root_events_) {
+    ProcessRootEvent(next_group_id_++, root_event, &event_group_name_map_);
+  }
   for (EventNode* root_event : root_events_) {
-    ProcessRootEvent(next_group_id++, root_event, &event_group_name_map_);
+    ProcessRootEvent(next_group_id_++, root_event, &event_group_name_map_);
   }
   for (int64 root_event_type : root_event_types) {
     if (auto root_events = gtl::FindOrNull(event_node_map_, root_event_type)) {
       for (const auto& root_event : *root_events) {
         // Skip if it already belongs to a group.
         if (root_event->GetGroupId()) continue;
-        ProcessRootEvent(next_group_id++, root_event.get(),
+        ProcessRootEvent(next_group_id_++, root_event.get(),
                          &event_group_name_map_);
       }
       // Only use the first root event type found.
@@ -408,33 +415,69 @@ void EventForest::MarkEagerlyExecutedCpuTfOps() {
   }
 }
 
-void EventForest::CreateVirtualEventsForHostTrainingLoop() {
-  VirtualEventNodeMap virtual_event_node_map;
-  auto executor_event_node_list =
+void EventForest::ProcessTensorFlowLoop() {
+  struct TensorFlowLoopIteration {
+    EventNode* first_event = nullptr;
+    std::vector<EventNode*> events;
+  };
+  using TensorFlowLoop = std::map<int64 /*iter_num*/, TensorFlowLoopIteration>;
+  absl::flat_hash_map<int64 /*step_id*/, TensorFlowLoop> tf_loops;
+
+  // Sort the TF executor events by TF function/session (step_id) and iter_num.
+  auto executor_event_list =
       gtl::FindOrNull(event_node_map_, HostEventType::kExecutorStateProcess);
-  if (!executor_event_node_list) return;
-  for (auto& executor_event_node : *executor_event_node_list) {
+  if (!executor_event_list) return;
+  for (auto& executor_event : *executor_event_list) {
+    if (IsTfDataEvent(*executor_event)) continue;
     const XStat* step_id_stat =
-        executor_event_node->GetContextStat(StatType::kStepId);
+        executor_event->GetContextStat(StatType::kStepId);
     const XStat* iter_num_stat =
-        executor_event_node->GetContextStat(StatType::kIterNum);
+        executor_event->GetContextStat(StatType::kIterNum);
     if (!step_id_stat || !iter_num_stat) continue;
     int64 step_id = step_id_stat->int64_value();
-    int64 iter_num = iter_num_stat->int64_value();
-    // Process the event with nonzero iter_num only to filter out the events
-    // related to tf.data.
-    // TODO(jihochoi): Filter out tf.data events more reliably.
-    if (!iter_num) continue;
-    EventNode*& virtual_event_node = virtual_event_node_map[step_id][iter_num];
-    if (!virtual_event_node) {
-      auto new_virtual_event_node =
-          absl::make_unique<EventNode>(*executor_event_node);
-      virtual_event_node = new_virtual_event_node.get();
-      // event_node_map_ keeps new_virtual_event_node alive.
-      event_node_map_[HostEventType::kHostTrainingLoopIteration].push_back(
-          std::move(new_virtual_event_node));
+    TensorFlowLoop& tf_loop = tf_loops[step_id];
+    TensorFlowLoopIteration& iteration = tf_loop[iter_num_stat->int64_value()];
+    if (!iteration.first_event ||
+        executor_event->StartsBefore(*iteration.first_event)) {
+      iteration.first_event = executor_event.get();
+    }
+    iteration.events.push_back(executor_event.get());
+  }
+
+  // Sort the TF loops by start time.
+  std::map<int64 /*start_time*/, int64 /*step_id*/> sorted_tf_loops;
+  for (const auto& step_id_and_tf_loop : tf_loops) {
+    auto& iterations = step_id_and_tf_loop.second;
+    // Filter out TF function/session without loops.
+    if (iterations.size() == 1 && iterations.count(0)) continue;
+    int64 start_time = iterations.cbegin()
+                           ->second.first_event->GetEventVisitor()
+                           .TimestampPs();
+    DCHECK_EQ(sorted_tf_loops.count(start_time), 0);
+    sorted_tf_loops[start_time] = step_id_and_tf_loop.first;
+  }
+
+  // Register the first event of each iteration as a root event. Also, add the
+  // other events of the iteration as child to the root event.
+  bool next_group_id_updated = false;
+  for (const auto& start_time_and_step_id : sorted_tf_loops) {
+    TensorFlowLoop& tf_loop = tf_loops[start_time_and_step_id.second];
+    for (auto& iter_num_and_iteration : tf_loop) {
+      if (!next_group_id_updated) {
+        // Set next_group_id_ to the first iter_num of the first TF loop. This
+        // is necessary later when selecting the intersection of the steps from
+        // multiple hosts.
+        next_group_id_ = iter_num_and_iteration.first;
+        next_group_id_updated = true;
+      }
+      TensorFlowLoopIteration& iteration = iter_num_and_iteration.second;
+      EventNode* root_event = iteration.first_event;
+      tf_loop_root_events_.push_back(root_event);
+      for (EventNode* event : iteration.events) {
+        if (event == root_event) continue;
+        root_event->AddChild(event);
+      }
     }
-    virtual_event_node->AddChild(executor_event_node.get());
   }
 }
 
@@ -473,9 +516,7 @@ EventForest::EventForest(
   }
   ConnectInterThread(connect_info_list);
   ConnectContextGroups(context_groups);
-  if (NeedsVirtualEventsForHostTrainingLoop(root_event_types)) {
-    CreateVirtualEventsForHostTrainingLoop();
-  }
+  ProcessTensorFlowLoop();
   if (NeedsVirtualEventsForAsyncExecutor(root_event_types)) {
     CreateVirtualEventsForAsyncExecutor();
   }
@@ -538,8 +579,7 @@ void GroupTfEvents(XSpace* space, EventGroupNameMap* event_group_name_map) {
       CreateInterThreadConnectInfoList();
   const std::vector<int64 /*EventType*/> root_event_types(
       {HostEventType::kTraceContext, HostEventType::kFunctionRun,
-       HostEventType::kSessionRun, HostEventType::kRunGraph,
-       HostEventType::kHostTrainingLoopIteration});
+       HostEventType::kSessionRun, HostEventType::kRunGraph});
   EventForest event_forest(connect_info_list, root_event_types,
                            CreateTfXPlaneVisitor, space);
   if (event_group_name_map) {
diff --git a/tensorflow/core/profiler/utils/group_events.h b/tensorflow/core/profiler/utils/group_events.h
index 544ae39c1ed..2d10480a64f 100644
--- a/tensorflow/core/profiler/utils/group_events.h
+++ b/tensorflow/core/profiler/utils/group_events.h
@@ -90,7 +90,7 @@ class EventNode {
   bool IsNestedIn(EventNode* parent);
 
   // Returns the closest parent of the given event type.
-  EventNode* FindParent(int64 event_type);
+  EventNode* FindParent(int64 event_type) const;
 
   absl::optional<ContextInfo> GetProducerContext() const {
     return producer_context_;
@@ -104,6 +104,8 @@ class EventNode {
 
   bool IsAsync() const { return is_async_; }
 
+  bool StartsBefore(const EventNode& other) const;
+
  private:
   const XPlaneVisitor* plane_;
   XEventVisitor visitor_;
@@ -175,13 +177,11 @@ class EventForest {
   // Sets the is_eager stat to true for the eagerly executed CPU TF op events.
   void MarkEagerlyExecutedCpuTfOps();
 
-  // Create virtual events of HostEventType::kHostTrainingLoopIteration. A
-  // virtual event is created for each iteration of the host training loop and
-  // connected to the HostEventType::kExecutorStateProcess events of the
-  // iteration.
-  void CreateVirtualEventsForHostTrainingLoop();
+  // Processes the TF loops and registers the first TF executor event of each
+  // iteraton to `tf_loop_root_events_`.
+  void ProcessTensorFlowLoop();
 
-  // Create virutal events of HostEventType::kAsyncExecutorTraceContext. A
+  // Creates virtual events of HostEventType::kAsyncExecutorTraceContext. A
   // virtual event is created for every FunctionRun and the following eager ops
   // (e.g., for Keras callback).
   void CreateVirtualEventsForAsyncExecutor();
@@ -190,6 +190,8 @@ class EventForest {
   std::vector<XPlaneVisitor> visitors_;
   EventGroupNameMap event_group_name_map_;
   EventList root_events_;
+  EventList tf_loop_root_events_;
+  int64 next_group_id_ = 0;
 };
 
 std::vector<InterThreadConnectInfo> CreateInterThreadConnectInfoList();
diff --git a/tensorflow/core/profiler/utils/group_events_test.cc b/tensorflow/core/profiler/utils/group_events_test.cc
index 8545bc94e54..ea378b7cb70 100644
--- a/tensorflow/core/profiler/utils/group_events_test.cc
+++ b/tensorflow/core/profiler/utils/group_events_test.cc
@@ -68,13 +68,16 @@ TEST(GroupEventsTest, GroupGpuTraceTest) {
   EXPECT_EQ(event_group_name_map[0], "123");
 }
 
-TEST(GroupEventsTest, GroupHostTrainingLoopTest) {
+TEST(GroupEventsTest, GroupTensorFlowLoopTest) {
   XSpace space;
   XPlaneBuilder host_plane_builder(space.add_planes());
   host_plane_builder.SetName(kHostThreads);
   host_plane_builder.ReserveLines(1);
 
   auto tf_executor_thread = host_plane_builder.GetOrCreateLine(0);
+  CreateXEvent(&host_plane_builder, &tf_executor_thread,
+               HostEventType::kExecutorStateProcess, 5, 10,
+               {{StatType::kStepId, 0}, {StatType::kIterNum, 10}});
   CreateXEvent(&host_plane_builder, &tf_executor_thread,
                HostEventType::kExecutorStateProcess, 20, 80,
                {{StatType::kStepId, 0}, {StatType::kIterNum, 10}});
@@ -96,8 +99,45 @@ TEST(GroupEventsTest, GroupHostTrainingLoopTest) {
   EXPECT_EQ(device_plane_visitor.GetStatType(
                 device_plane->lines(0).events(0).stats(1)),
             StatType::kGroupId);
+  EXPECT_EQ(device_plane->lines(0).events(0).stats(1).int64_value(), 10);
   EXPECT_EQ(event_group_name_map.size(), 1);
-  EXPECT_EQ(event_group_name_map[0], "10");
+  EXPECT_EQ(event_group_name_map[10], "10");
+}
+
+// When there are multiple TF loops, group_id is assigned in the order of TF
+// loops' start times and iter_num. In this test case, the profile captures the
+// last two iterations (iter_num=10,11) of the first TF loop (step_id=0) and the
+// first two iterations (iter_num=0,1) of the second TF loop (step_id=1).
+// group_id is initialized to the first TF loop's first iter_num (10) and then
+// monotonically increased.
+TEST(GroupEventsTest, GroupMultipleTensorFlowLoopsTest) {
+  XSpace space;
+  XPlaneBuilder host_plane_builder(space.add_planes());
+  host_plane_builder.SetName(kHostThreads);
+  host_plane_builder.ReserveLines(2);
+
+  auto first_tf_executor_thread = host_plane_builder.GetOrCreateLine(0);
+  CreateXEvent(&host_plane_builder, &first_tf_executor_thread,
+               HostEventType::kExecutorStateProcess, 220, 80,
+               {{StatType::kStepId, 1}, {StatType::kIterNum, 0}});
+  CreateXEvent(&host_plane_builder, &first_tf_executor_thread,
+               HostEventType::kExecutorStateProcess, 320, 80,
+               {{StatType::kStepId, 1}, {StatType::kIterNum, 1}});
+  auto second_tf_executor_thread = host_plane_builder.GetOrCreateLine(1);
+  CreateXEvent(&host_plane_builder, &second_tf_executor_thread,
+               HostEventType::kExecutorStateProcess, 20, 80,
+               {{StatType::kStepId, 0}, {StatType::kIterNum, 10}});
+  CreateXEvent(&host_plane_builder, &second_tf_executor_thread,
+               HostEventType::kExecutorStateProcess, 120, 80,
+               {{StatType::kStepId, 0}, {StatType::kIterNum, 11}});
+
+  EventGroupNameMap event_group_name_map;
+  GroupTfEvents(&space, &event_group_name_map);
+  EXPECT_EQ(event_group_name_map.size(), 4);
+  EXPECT_TRUE(event_group_name_map.count(10));
+  EXPECT_TRUE(event_group_name_map.count(11));
+  EXPECT_TRUE(event_group_name_map.count(12));
+  EXPECT_TRUE(event_group_name_map.count(13));
 }
 
 TEST(GroupEventsTest, GroupFunctionalOp) {
diff --git a/tensorflow/core/profiler/utils/xplane_schema.cc b/tensorflow/core/profiler/utils/xplane_schema.cc
index 8a4538197f3..81bd4f9959d 100644
--- a/tensorflow/core/profiler/utils/xplane_schema.cc
+++ b/tensorflow/core/profiler/utils/xplane_schema.cc
@@ -108,7 +108,6 @@ const HostEventTypeMap& GetHostEventTypeMap() {
       {"IteratorGetNextOp::DoCompute", kIteratorGetNextOp},
       {"IteratorGetNextAsOptionalOp::DoCompute", kIteratorGetNextAsOptionalOp},
       // Virtual events for grouping.
-      {"HostTrainingLoopIteration", kHostTrainingLoopIteration},
       {"AsyncExecutorTraceContext", kAsyncExecutorTraceContext},
       // GPU related.
       {"KernelLaunch", kKernelLaunch},
diff --git a/tensorflow/core/profiler/utils/xplane_schema.h b/tensorflow/core/profiler/utils/xplane_schema.h
index d0944bf88e1..6c0e7507696 100644
--- a/tensorflow/core/profiler/utils/xplane_schema.h
+++ b/tensorflow/core/profiler/utils/xplane_schema.h
@@ -101,7 +101,6 @@ enum HostEventType {
   kIteratorGetNextOp,
   kIteratorGetNextAsOptionalOp,
   // Virtual events for grouping.
-  kHostTrainingLoopIteration,
   kAsyncExecutorTraceContext,
   // GPU related.
   kKernelLaunch,
diff --git a/tensorflow/core/public/version.h b/tensorflow/core/public/version.h
index c56fac42c1d..46cbe0c4531 100644
--- a/tensorflow/core/public/version.h
+++ b/tensorflow/core/public/version.h
@@ -108,7 +108,7 @@ limitations under the License.
 
 #define TF_GRAPH_DEF_VERSION_MIN_PRODUCER 0
 #define TF_GRAPH_DEF_VERSION_MIN_CONSUMER 0
-#define TF_GRAPH_DEF_VERSION 420  // Updated: 2020/6/2
+#define TF_GRAPH_DEF_VERSION 425  // Updated: 2020/6/7
 
 // Checkpoint compatibility versions (the versions field in SavedSliceMeta).
 //
diff --git a/tensorflow/core/tpu/BUILD b/tensorflow/core/tpu/BUILD
index 5d1b7e1101f..d4bcdfd52c5 100644
--- a/tensorflow/core/tpu/BUILD
+++ b/tensorflow/core/tpu/BUILD
@@ -1,5 +1,10 @@
 # Description: Utilities for TPU Operations
 
+load(
+    "//tensorflow:tensorflow.bzl",
+    "if_windows",
+)
+
 package(
     default_visibility = [
         "//tensorflow/core/tpu:__subpackages__",
@@ -8,6 +13,13 @@ package(
     licenses = ["notice"],  # Apache 2.0
 )
 
+cc_library(
+    name = "libtftpu_header",
+    hdrs = ["libtftpu.h"],
+    visibility = ["//visibility:public"],
+    deps = [],
+)
+
 cc_library(
     name = "tpu_embedding_optimization_parameters_utils",
     srcs = ["tpu_embedding_optimization_parameters_utils.cc"],
@@ -88,14 +100,23 @@ cc_library(
     name = "tpu_config_c_api",
     hdrs = ["tpu_config_c_api.h"],
     deps = [
+        ":libtftpu_header",
         "//tensorflow/c:tf_status",
     ],
 )
 
 cc_library(
     name = "tpu_library_loader",
-    srcs = ["tpu_library_loader.cc"],
+    srcs = if_windows(
+        ["tpu_library_loader_windows.cc"],
+        otherwise = ["tpu_library_loader.cc"],
+    ),
     hdrs = ["tpu_library_loader.h"],
-    visibility = ["//tensorflow:__subpackages__"],
-    deps = ["//tensorflow/core/platform:status"],
+    visibility = ["//visibility:public"],
+    deps = [
+        ":libtftpu_header",
+        ":tpu_config_c_api",
+        "//tensorflow/core/platform:errors",
+        "//tensorflow/core/platform:status",
+    ],
 )
diff --git a/tensorflow/core/tpu/libtftpu.h b/tensorflow/core/tpu/libtftpu.h
new file mode 100644
index 00000000000..a4405df8205
--- /dev/null
+++ b/tensorflow/core/tpu/libtftpu.h
@@ -0,0 +1,52 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#ifndef TENSORFLOW_CORE_TPU_LIBTFTPU_H_
+#define TENSORFLOW_CORE_TPU_LIBTFTPU_H_
+
+// Unfortunately we have to add an Fn suffix because we cannot have the same
+// name for both a function and a element within a struct in the global
+// namespace in gcc. This restriction doesn't exist in clang.
+#define TFTPU_ADD_FN_IN_STRUCT(FnName) decltype(FnName)* FnName##Fn;
+
+#ifdef SWIG
+#define TFTPU_CAPI_EXPORT
+#else
+#if defined(_WIN32)
+#ifdef TF_COMPILE_LIBRARY
+#define TFTPU_CAPI_EXPORT __declspec(dllexport)
+#else
+#define TFTPU_CAPI_EXPORT __declspec(dllimport)
+#endif  // TF_COMPILE_LIBRARY
+#else
+#define TFTPU_CAPI_EXPORT __attribute__((visibility("default")))
+#endif  // _WIN32
+#endif  // SWIG
+
+#ifdef __cplusplus
+extern "C" {
+#endif
+
+TFTPU_CAPI_EXPORT void TfTpu_Initialize();
+
+#ifdef __cplusplus
+}
+#endif
+
+struct TfTpu_BaseFn {
+  TFTPU_ADD_FN_IN_STRUCT(TfTpu_Initialize);
+};
+
+#endif  // TENSORFLOW_CORE_TPU_LIBTFTPU_H_
diff --git a/tensorflow/core/tpu/tpu_config_c_api.h b/tensorflow/core/tpu/tpu_config_c_api.h
index b7caf0648b1..9c0ed203d4b 100644
--- a/tensorflow/core/tpu/tpu_config_c_api.h
+++ b/tensorflow/core/tpu/tpu_config_c_api.h
@@ -20,40 +20,57 @@ limitations under the License.
 #include <cstdint>
 
 #include "tensorflow/c/tf_status.h"
+#include "tensorflow/core/tpu/libtftpu.h"
 
 typedef struct TpuSerializedProto TpuSerializedProto;
 
+namespace tensorflow {
+namespace tpu {
+class TpuMeshStateInterface;
+}  // namespace tpu
+}  // namespace tensorflow
+
 extern "C" {
 
-bool TPUHostInitialized();
+TFTPU_CAPI_EXPORT void ConfigureDistributedTpuOp_DoWork(
+    const size_t num_cores_per_host_size, const int32_t* num_cores_per_host,
+    size_t* host_config_output_size, char** host_config_output,
+    TF_Status* status);
 
-void ConfigureDistributedTpuOp_DoWork(const size_t num_cores_per_host_size,
-                                      const int32_t* num_cores_per_host,
-                                      size_t* host_config_output_size,
-                                      char** host_config_output,
-                                      TF_Status* status);
-
-void WaitForDistributedTpuOp_DoWork(
+TFTPU_CAPI_EXPORT void WaitForDistributedTpuOp_DoWork(
     const size_t num_hosts, const size_t num_cores_per_host,
     const int32_t** host_ordinal_to_global_core_id_map,
+    tensorflow::tpu::TpuMeshStateInterface* tpu_mesh_state_interface,
     size_t* tpu_topology_output_size, char** tpu_topology_output,
     TF_Status* status);
 
-void ShutdownDistributedTpuOp_DoWork(TF_Status* status);
+TFTPU_CAPI_EXPORT void ShutdownDistributedTpuOp_DoWork(TF_Status* status);
 
-void InitializeHostForDistributedTpuOp_DoWork(
+TFTPU_CAPI_EXPORT void InitializeHostForDistributedTpuOp_DoWork(
     const size_t tpu_host_config_size, const char* tpu_host_config,
     const bool enable_whole_mesh_compilations, size_t* core_id_output_size,
     int32_t** core_id_output, TF_Status* status);
 
-void SetGlobalTPUArrayOp_DoWork(const size_t tpu_topology_size,
-                                const char* tpu_topology, TF_Status* status);
+TFTPU_CAPI_EXPORT void SetGlobalTPUArrayOp_DoWork(
+    const size_t tpu_topology_size, const char* tpu_topology,
+    TF_Status* status);
 
-void DisconnectDistributedTpuChipsOp_DoWork(int32_t* number_of_chips_output,
-                                            TF_Status* status);
+TFTPU_CAPI_EXPORT void DisconnectDistributedTpuChipsOp_DoWork(
+    int32_t* number_of_chips_output, TF_Status* status);
 
-void TpuConfigurationApi_FreeCharArray(char* output);
-void TpuConfigurationApi_FreeInt32Array(int32_t* output);
+TFTPU_CAPI_EXPORT void TpuConfigurationApi_FreeCharArray(char* output);
+TFTPU_CAPI_EXPORT void TpuConfigurationApi_FreeInt32Array(int32_t* output);
 }
 
+struct TfTpu_ConfigApiFn {
+  TFTPU_ADD_FN_IN_STRUCT(ConfigureDistributedTpuOp_DoWork);
+  TFTPU_ADD_FN_IN_STRUCT(WaitForDistributedTpuOp_DoWork);
+  TFTPU_ADD_FN_IN_STRUCT(ShutdownDistributedTpuOp_DoWork);
+  TFTPU_ADD_FN_IN_STRUCT(InitializeHostForDistributedTpuOp_DoWork);
+  TFTPU_ADD_FN_IN_STRUCT(SetGlobalTPUArrayOp_DoWork);
+  TFTPU_ADD_FN_IN_STRUCT(DisconnectDistributedTpuChipsOp_DoWork);
+  TFTPU_ADD_FN_IN_STRUCT(TpuConfigurationApi_FreeCharArray);
+  TFTPU_ADD_FN_IN_STRUCT(TpuConfigurationApi_FreeInt32Array);
+};
+
 #endif  // TENSORFLOW_CORE_TPU_TPU_CONFIG_C_API_H_
diff --git a/tensorflow/core/tpu/tpu_library_loader.cc b/tensorflow/core/tpu/tpu_library_loader.cc
index bfd9fe29efe..3bc835c9c7f 100644
--- a/tensorflow/core/tpu/tpu_library_loader.cc
+++ b/tensorflow/core/tpu/tpu_library_loader.cc
@@ -15,14 +15,62 @@ limitations under the License.
 
 #include "tensorflow/core/tpu/tpu_library_loader.h"
 
+#include <dlfcn.h>
+
+#define TFTPU_SET_FN(Struct, FnName) \
+  Struct->FnName##Fn =               \
+      reinterpret_cast<decltype(FnName)*>(dlsym(library_handle, #FnName));
+
+#include "tensorflow/core/platform/errors.h"
 #include "tensorflow/core/platform/status.h"
 
+// Reminder: Update tpu_library_loader_windows.cc if you are adding new publicly
+// visible methods.
+
 namespace tensorflow {
 namespace tpu {
 
-Status InitializeTPULibrary(void* library) {
-  // TODO(frankchn): dlsym the loaded library and populate a struct with the
-  // relevant C APIs necessary for TPUs.
+Status SetTpuInitializeStructFns(void* library_handle) {
+  auto* base_fn = InitializeApiFn();
+
+  TFTPU_SET_FN(base_fn, TfTpu_Initialize);
+
+  return Status::OK();
+}
+
+Status SetTpuConfigStructFns(void* library_handle) {
+  auto* config_fn = ConfigApiFn();
+
+  TFTPU_SET_FN(config_fn, ConfigureDistributedTpuOp_DoWork);
+  TFTPU_SET_FN(config_fn, WaitForDistributedTpuOp_DoWork);
+  TFTPU_SET_FN(config_fn, ShutdownDistributedTpuOp_DoWork);
+  TFTPU_SET_FN(config_fn, InitializeHostForDistributedTpuOp_DoWork);
+  TFTPU_SET_FN(config_fn, SetGlobalTPUArrayOp_DoWork);
+  TFTPU_SET_FN(config_fn, DisconnectDistributedTpuChipsOp_DoWork);
+  TFTPU_SET_FN(config_fn, TpuConfigurationApi_FreeCharArray);
+  TFTPU_SET_FN(config_fn, TpuConfigurationApi_FreeInt32Array);
+
+  return Status::OK();
+}
+
+TfTpu_BaseFn* InitializeApiFn() {
+  static TfTpu_BaseFn base_fn;
+  return &base_fn;
+}
+
+TfTpu_ConfigApiFn* ConfigApiFn() {
+  static TfTpu_ConfigApiFn config_api_fn;
+  return &config_api_fn;
+}
+
+Status InitializeTpuLibrary(void* library_handle) {
+  if (library_handle == nullptr) {
+    library_handle = dlopen(nullptr, RTLD_LAZY);
+  }
+
+  TF_RETURN_IF_ERROR(SetTpuInitializeStructFns(library_handle));
+  TF_RETURN_IF_ERROR(SetTpuConfigStructFns(library_handle));
+
   return Status::OK();
 }
 
diff --git a/tensorflow/core/tpu/tpu_library_loader.h b/tensorflow/core/tpu/tpu_library_loader.h
index 35a7dd7c9be..a51948cf719 100644
--- a/tensorflow/core/tpu/tpu_library_loader.h
+++ b/tensorflow/core/tpu/tpu_library_loader.h
@@ -17,13 +17,21 @@ limitations under the License.
 #define TENSORFLOW_CORE_TPU_TPU_LIBRARY_LOADER_H_
 
 #include "tensorflow/core/platform/status.h"
+#include "tensorflow/core/tpu/libtftpu.h"
+#include "tensorflow/core/tpu/tpu_config_c_api.h"
 
+// LINT.IfChange
 namespace tensorflow {
 namespace tpu {
 
-Status InitializeTPULibrary(void* library);
+Status InitializeTpuLibrary(void* library_handle);
+
+TfTpu_BaseFn* InitializeApiFn();
+
+TfTpu_ConfigApiFn* ConfigApiFn();
 
 }  // namespace tpu
 }  // namespace tensorflow
+// LINT.ThenChange(//tensorflow/core/tpu/tpu_library_loader_windows.cc)
 
 #endif  // TENSORFLOW_CORE_TPU_TPU_LIBRARY_LOADER_H_
diff --git a/tensorflow/core/tpu/tpu_library_loader_windows.cc b/tensorflow/core/tpu/tpu_library_loader_windows.cc
new file mode 100644
index 00000000000..e7c25df415e
--- /dev/null
+++ b/tensorflow/core/tpu/tpu_library_loader_windows.cc
@@ -0,0 +1,36 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include "tensorflow/core/platform/errors.h"
+#include "tensorflow/core/platform/status.h"
+#include "tensorflow/core/tpu/tpu_library_loader.h"
+
+// Reminder: Update tpu_library_loader.cc if you are adding new publicly
+// visible methods.
+
+namespace tensorflow {
+namespace tpu {
+
+TfTpu_BaseFn* InitializeApiFn() { return nullptr; }
+
+TfTpu_ConfigApiFn* ConfigApiFn() { return nullptr; }
+
+Status InitializeTpuLibrary(void* library_handle) {
+  return errors::Unimplemented(
+      "Loading TPU library is not supported on Windows.");
+}
+
+}  // namespace tpu
+}  // namespace tensorflow
diff --git a/tensorflow/core/util/gpu_device_functions.h b/tensorflow/core/util/gpu_device_functions.h
index b4de2ffa8e9..de55b5c33c4 100644
--- a/tensorflow/core/util/gpu_device_functions.h
+++ b/tensorflow/core/util/gpu_device_functions.h
@@ -658,6 +658,16 @@ __device__ Eigen::half GpuAtomicCasHelper(Eigen::half* ptr, F accumulate) {
   }
 }
 
+template <typename F>
+__device__ long long GpuAtomicCasHelper(long long* ptr, F accumulate) {
+  return static_cast<long long>(
+      GpuAtomicCasHelper(reinterpret_cast<unsigned long long*>(ptr),
+                         [accumulate](unsigned long long a) {
+                           return static_cast<unsigned long long>(
+                               accumulate(static_cast<long long>(a)));
+                         }));
+}
+
 template <typename From, typename To>
 using ToTypeIfConvertible =
     typename std::enable_if<std::is_convertible<From, To>::value, To>::type;
@@ -779,6 +789,11 @@ __device__ inline double GpuAtomicMax(double* ptr, double value) {
       ptr, [value](double a) { return fmax(a, value); });
 }
 
+__device__ inline long long GpuAtomicMax(long long* ptr, long long value) {
+  return detail::GpuAtomicCasHelper(
+      ptr, [value](long long a) { return max(a, value); });
+}
+
 #else
 
 __device__ inline float GpuAtomicMax(float* ptr, float value) {
@@ -839,6 +854,11 @@ __device__ inline double GpuAtomicMin(double* ptr, double value) {
       ptr, [value](double a) { return fmin(a, value); });
 }
 
+__device__ inline long long GpuAtomicMin(long long* ptr, long long value) {
+  return detail::GpuAtomicCasHelper(
+      ptr, [value](long long a) { return min(a, value); });
+}
+
 #else
 
 __device__ inline float GpuAtomicMin(float* ptr, float value) {
diff --git a/tensorflow/go/op/wrappers.go b/tensorflow/go/op/wrappers.go
index 9f3a15af957..f118e2bd494 100644
--- a/tensorflow/go/op/wrappers.go
+++ b/tensorflow/go/op/wrappers.go
@@ -9487,6 +9487,14 @@ func DebugIdentityV2DebugUrls(value []string) DebugIdentityV2Attr {
 	}
 }
 
+// DebugIdentityV2CircularBufferSize sets the optional circular_buffer_size attribute to value.
+// If not specified, defaults to 1000
+func DebugIdentityV2CircularBufferSize(value int64) DebugIdentityV2Attr {
+	return func(m optionalAttr) {
+		m["circular_buffer_size"] = value
+	}
+}
+
 // Debug Identity V2 Op.
 //
 // Provides an identity mapping from input to output, while writing the content of
@@ -26646,19 +26654,23 @@ func FakeQuantWithMinMaxArgsNarrowRange(value bool) FakeQuantWithMinMaxArgsAttr
 
 // Fake-quantize the 'inputs' tensor, type float to 'outputs' tensor of same type.
 //
-// Attributes `[min; max]` define the clamping range for the `inputs` data.
-// `inputs` values are quantized into the quantization range (`[0; 2^num_bits - 1]`
-// when `narrow_range` is false and `[1; 2^num_bits - 1]` when it is true) and
-// then de-quantized and output as floats in `[min; max]` interval.
-// `num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
+// Attributes
+//
+// *   `[min; max]` define the clamping range for the `inputs` data.
+// *   `inputs` values are quantized into the quantization range (
+// `[0; 2^num_bits - 1]` when `narrow_range` is false and `[1; 2^num_bits - 1]`
+// when it is true) and then de-quantized and output as floats in `[min; max]`
+// interval.
+// *   `num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
 //
 // Before quantization, `min` and `max` values are adjusted with the following
 // logic.
 // It is suggested to have `min <= 0 <= max`. If `0` is not in the range of values,
 // the behavior can be unexpected:
-// If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
-// If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
-// If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
+//
+// *   If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
+// *   If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
+// *   If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
 // `min_adj = scale * round(min / scale)` and `max_adj = max + min_adj - min`.
 //
 // Quantization is called fake since the output is still in floating point.
@@ -37416,24 +37428,29 @@ func FakeQuantWithMinMaxVarsPerChannelNarrowRange(value bool) FakeQuantWithMinMa
 	}
 }
 
-// Fake-quantize the 'inputs' tensor of type float and one of the shapes: `[d]`,
+// Fake-quantize the 'inputs' tensor of type float via per-channel floats
 //
-// `[b, d]` `[b, h, w, d]` via per-channel floats `min` and `max` of shape `[d]`
-// to 'outputs' tensor of same shape as `inputs`.
+// Fake-quantize the `inputs` tensor of type float per-channel and one of the
+// shapes: `[d]`, `[b, d]` `[b, h, w, d]` via per-channel floats `min` and `max`
+// of shape `[d]` to `outputs` tensor of same shape as `inputs`.
 //
-// `[min; max]` define the clamping range for the `inputs` data.
-// `inputs` values are quantized into the quantization range (`[0; 2^num_bits - 1]`
-// when `narrow_range` is false and `[1; 2^num_bits - 1]` when it is true) and
-// then de-quantized and output as floats in `[min; max]` interval.
-// `num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
+// Attributes
+//
+// *   `[min; max]` define the clamping range for the `inputs` data.
+// *   `inputs` values are quantized into the quantization range (
+// `[0; 2^num_bits - 1]` when `narrow_range` is false and `[1; 2^num_bits - 1]`
+// when it is true) and then de-quantized and output as floats in `[min; max]`
+// interval.
+// *   `num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
 //
 // Before quantization, `min` and `max` values are adjusted with the following
 // logic.
 // It is suggested to have `min <= 0 <= max`. If `0` is not in the range of values,
 // the behavior can be unexpected:
-// If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
-// If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
-// If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
+//
+// *   If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
+// *   If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
+// *   If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
 // `min_adj = scale * round(min / scale)` and `max_adj = max + min_adj - min`.
 //
 // This operation has a gradient and thus allows for training `min` and `max`
@@ -45265,23 +45282,28 @@ func FakeQuantWithMinMaxVarsNarrowRange(value bool) FakeQuantWithMinMaxVarsAttr
 	}
 }
 
-// Fake-quantize the 'inputs' tensor of type float via global float scalars `min`
+// Fake-quantize the 'inputs' tensor of type float via global float scalars
 //
-// and `max` to 'outputs' tensor of same shape as `inputs`.
+// Fake-quantize the `inputs` tensor of type float via global float scalars
+// `min` and `max` to `outputs` tensor of same shape as `inputs`.
 //
-// `[min; max]` define the clamping range for the `inputs` data.
-// `inputs` values are quantized into the quantization range (`[0; 2^num_bits - 1]`
-// when `narrow_range` is false and `[1; 2^num_bits - 1]` when it is true) and
-// then de-quantized and output as floats in `[min; max]` interval.
-// `num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
+// Attributes
+//
+// *   `[min; max]` define the clamping range for the `inputs` data.
+// *   `inputs` values are quantized into the quantization range (
+// `[0; 2^num_bits - 1]` when `narrow_range` is false and `[1; 2^num_bits - 1]`
+// when it is true) and then de-quantized and output as floats in `[min; max]`
+// interval.
+// *   `num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
 //
 // Before quantization, `min` and `max` values are adjusted with the following
 // logic.
 // It is suggested to have `min <= 0 <= max`. If `0` is not in the range of values,
 // the behavior can be unexpected:
-// If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
-// If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
-// If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
+//
+// *   If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
+// *   If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
+// *   If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
 // `min_adj = scale * round(min / scale)` and `max_adj = max + min_adj - min`.
 //
 // This operation has a gradient and thus allows for training `min` and `max`
diff --git a/tensorflow/lite/BUILD b/tensorflow/lite/BUILD
index 810f3ab1a2b..a888cbd9407 100644
--- a/tensorflow/lite/BUILD
+++ b/tensorflow/lite/BUILD
@@ -93,6 +93,8 @@ cc_library(
 alias(
     name = "schema_fbs_version",
     actual = ":version",
+    # avoid_dep tells build_cleaner to not use schema_fbs_version.
+    tags = ["avoid_dep"],
 )
 
 cc_library(
diff --git a/tensorflow/lite/c/common.h b/tensorflow/lite/c/common.h
index 31d8d53d563..ab769fec249 100644
--- a/tensorflow/lite/c/common.h
+++ b/tensorflow/lite/c/common.h
@@ -58,7 +58,7 @@ typedef enum TfLiteExternalContextType {
   kTfLiteEigenContext = 0,       // include eigen_support.h to use.
   kTfLiteGemmLowpContext = 1,    // include gemm_support.h to use.
   kTfLiteEdgeTpuContext = 2,     // Placeholder for Edge TPU support.
-  kTfLiteCpuBackendContext = 3,  // include cpu_backend_support.h to use.
+  kTfLiteCpuBackendContext = 3,  // include cpu_backend_context.h to use.
   kTfLiteMaxExternalContexts = 4
 } TfLiteExternalContextType;
 
diff --git a/tensorflow/lite/core/api/BUILD b/tensorflow/lite/core/api/BUILD
index 419a3b2486d..97a3d3f78de 100644
--- a/tensorflow/lite/core/api/BUILD
+++ b/tensorflow/lite/core/api/BUILD
@@ -24,9 +24,12 @@ cc_library(
     build_for_embedded = True,
     copts = tflite_copts() + micro_copts(),
     deps = [
-        "//tensorflow/lite/c:common",
-        "//tensorflow/lite/schema:schema_fbs",
         "@flatbuffers//:runtime_cc",
+        "//tensorflow/lite/c:common",
+        # TODO(b/158301698): consider moving internal:compatibility to a more
+        # central location.
+        "//tensorflow/lite/kernels/internal:compatibility",
+        "//tensorflow/lite/schema:schema_fbs",
     ],
 )
 
diff --git a/tensorflow/lite/core/api/flatbuffer_conversions.cc b/tensorflow/lite/core/api/flatbuffer_conversions.cc
index c52fc9f690b..5f39732b970 100644
--- a/tensorflow/lite/core/api/flatbuffer_conversions.cc
+++ b/tensorflow/lite/core/api/flatbuffer_conversions.cc
@@ -23,6 +23,7 @@ limitations under the License.
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
 #include "tensorflow/lite/core/api/error_reporter.h"
+#include "tensorflow/lite/kernels/internal/compatibility.h"
 #include "tensorflow/lite/schema/schema_generated.h"
 
 namespace tflite {
@@ -89,6 +90,25 @@ TfLiteStatus FlatBufferIntVectorToArray(
   return kTfLiteOk;
 }
 
+// Converts the flatbuffer activation to what is used at runtime.
+TfLiteFusedActivation ConvertActivation(ActivationFunctionType activation) {
+  switch (activation) {
+    case ActivationFunctionType_NONE:
+      return kTfLiteActNone;
+    case ActivationFunctionType_RELU:
+      return kTfLiteActRelu;
+    case ActivationFunctionType_RELU_N1_TO_1:
+      return kTfLiteActRelu1;
+    case ActivationFunctionType_RELU6:
+      return kTfLiteActRelu6;
+    case ActivationFunctionType_TANH:
+      return kTfLiteActTanh;
+    case ActivationFunctionType_SIGN_BIT:
+      return kTfLiteActSignBit;
+  }
+  return kTfLiteActNone;
+}
+
 }  // namespace
 
 TfLiteStatus ConvertTensorType(TensorType tensor_type, TfLiteType* type,
@@ -135,12 +155,131 @@ TfLiteStatus ConvertTensorType(TensorType tensor_type, TfLiteType* type,
   }
 }
 
-// Parse the appropriate data out of the op.
-//
-// This handles builtin data explicitly as there are flatbuffer schemas.
-// If it returns kTfLiteOk, it passes the data out with `builtin_data`, which
-// need to be released by calling `free`.`
-// If it returns kTfLiteError, `builtin_data` will be `nullptr`.
+// We have this parse function instead of directly returning kTfLiteOk from the
+// switch-case in ParseOpData because this function is used as part of the
+// selective registration for the OpResolver implementation in micro.
+TfLiteStatus ParseDequantize(const Operator*, BuiltinOperator, ErrorReporter*,
+                             BuiltinDataAllocator*, void**) {
+  return kTfLiteOk;
+}
+
+TfLiteStatus ParseFullyConnected(const Operator* op, BuiltinOperator,
+                                 ErrorReporter* error_reporter,
+                                 BuiltinDataAllocator* allocator,
+                                 void** builtin_data) {
+  TFLITE_DCHECK(op != nullptr);
+  TFLITE_DCHECK(error_reporter != nullptr);
+  TFLITE_DCHECK(allocator != nullptr);
+  TFLITE_DCHECK(builtin_data != nullptr);
+
+  SafeBuiltinDataAllocator safe_allocator(allocator);
+
+  std::unique_ptr<TfLiteFullyConnectedParams,
+                  SafeBuiltinDataAllocator::BuiltinDataDeleter>
+      params = safe_allocator.Allocate<TfLiteFullyConnectedParams>();
+  TF_LITE_ENSURE(error_reporter, params != nullptr);
+
+  const FullyConnectedOptions* schema_params =
+      op->builtin_options_as_FullyConnectedOptions();
+
+  if (schema_params != nullptr) {
+    params->activation =
+        ConvertActivation(schema_params->fused_activation_function());
+    params->keep_num_dims = schema_params->keep_num_dims();
+    params->asymmetric_quantize_inputs =
+        schema_params->asymmetric_quantize_inputs();
+
+    switch (schema_params->weights_format()) {
+      case FullyConnectedOptionsWeightsFormat_DEFAULT:
+        params->weights_format = kTfLiteFullyConnectedWeightsFormatDefault;
+        break;
+      case FullyConnectedOptionsWeightsFormat_SHUFFLED4x16INT8:
+        params->weights_format =
+            kTfLiteFullyConnectedWeightsFormatShuffled4x16Int8;
+        break;
+      default:
+        TF_LITE_REPORT_ERROR(error_reporter,
+                             "Unhandled fully-connected weights format.");
+        return kTfLiteError;
+    }
+  } else {
+    // TODO(b/157480169): We should either return kTfLiteError or fill in some
+    // reasonable defaults in the params struct. We are not doing so until we
+    // better undertand the ramifications of changing the legacy behavior.
+  }
+
+  *builtin_data = params.release();
+  return kTfLiteOk;
+}
+
+// We have this parse function instead of directly returning kTfLiteOk from the
+// switch-case in ParseOpData because this function is used as part of the
+// selective registration for the OpResolver implementation in micro.
+TfLiteStatus ParseQuantize(const Operator*, BuiltinOperator, ErrorReporter*,
+                           BuiltinDataAllocator*, void**) {
+  return kTfLiteOk;
+}
+
+TfLiteStatus ParseSoftmax(const Operator* op, BuiltinOperator,
+                          ErrorReporter* error_reporter,
+                          BuiltinDataAllocator* allocator,
+                          void** builtin_data) {
+  TFLITE_DCHECK(op != nullptr);
+  TFLITE_DCHECK(error_reporter != nullptr);
+  TFLITE_DCHECK(allocator != nullptr);
+  TFLITE_DCHECK(builtin_data != nullptr);
+
+  SafeBuiltinDataAllocator safe_allocator(allocator);
+  std::unique_ptr<TfLiteSoftmaxParams,
+                  SafeBuiltinDataAllocator::BuiltinDataDeleter>
+      params = safe_allocator.Allocate<TfLiteSoftmaxParams>();
+  TF_LITE_ENSURE(error_reporter, params != nullptr);
+
+  const SoftmaxOptions* schema_params = op->builtin_options_as_SoftmaxOptions();
+
+  if (schema_params != nullptr) {
+    params->beta = schema_params->beta();
+  } else {
+    // TODO(b/157480169): We should either return kTfLiteError or fill in some
+    // reasonable defaults in the params struct. We are not doing so until we
+    // better undertand the ramifications of changing the legacy behavior.
+  }
+
+  *builtin_data = params.release();
+  return kTfLiteOk;
+}
+
+TfLiteStatus ParseSvdf(const Operator* op, BuiltinOperator,
+                       ErrorReporter* error_reporter,
+                       BuiltinDataAllocator* allocator, void** builtin_data) {
+  TFLITE_DCHECK(op != nullptr);
+  TFLITE_DCHECK(error_reporter != nullptr);
+  TFLITE_DCHECK(allocator != nullptr);
+  TFLITE_DCHECK(builtin_data != nullptr);
+
+  SafeBuiltinDataAllocator safe_allocator(allocator);
+  std::unique_ptr<TfLiteSVDFParams,
+                  SafeBuiltinDataAllocator::BuiltinDataDeleter>
+      params = safe_allocator.Allocate<TfLiteSVDFParams>();
+  TF_LITE_ENSURE(error_reporter, params != nullptr);
+
+  const SVDFOptions* schema_params = op->builtin_options_as_SVDFOptions();
+  if (schema_params != nullptr) {
+    params->rank = schema_params->rank();
+    params->activation =
+        ConvertActivation(schema_params->fused_activation_function());
+    params->asymmetric_quantize_inputs =
+        schema_params->asymmetric_quantize_inputs();
+  } else {
+    // TODO(b/157480169): We should either return kTfLiteError or fill in some
+    // reasonable defaults in the params struct. We are not doing so until we
+    // better undertand the ramifications of changing the legacy behavior.
+  }
+
+  *builtin_data = params.release();
+  return kTfLiteOk;
+}
+
 TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
                          ErrorReporter* error_reporter,
                          BuiltinDataAllocator* allocator, void** builtin_data) {
@@ -153,23 +292,6 @@ TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
     }
     return kTfLitePaddingUnknown;
   };
-  auto parse_activation = [](ActivationFunctionType activation) {
-    switch (activation) {
-      case ActivationFunctionType_NONE:
-        return kTfLiteActNone;
-      case ActivationFunctionType_RELU:
-        return kTfLiteActRelu;
-      case ActivationFunctionType_RELU_N1_TO_1:
-        return kTfLiteActRelu1;
-      case ActivationFunctionType_RELU6:
-        return kTfLiteActRelu6;
-      case ActivationFunctionType_TANH:
-        return kTfLiteActTanh;
-      case ActivationFunctionType_SIGN_BIT:
-        return kTfLiteActSignBit;
-    }
-    return kTfLiteActNone;
-  };
   auto parseLSHProjectionType = [](LSHProjectionType type) {
     switch (type) {
       case LSHProjectionType_SPARSE:
@@ -195,6 +317,29 @@ TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
   SafeBuiltinDataAllocator safe_allocator(allocator);
   *builtin_data = nullptr;
   switch (op_type) {
+    case BuiltinOperator_DEQUANTIZE: {
+      return ParseDequantize(op, op_type, error_reporter, allocator,
+                             builtin_data);
+    }
+
+    case BuiltinOperator_FULLY_CONNECTED: {
+      return ParseFullyConnected(op, op_type, error_reporter, allocator,
+                                 builtin_data);
+    }
+
+    case BuiltinOperator_QUANTIZE: {
+      return ParseQuantize(op, op_type, error_reporter, allocator,
+                           builtin_data);
+    }
+
+    case BuiltinOperator_SOFTMAX: {
+      return ParseSoftmax(op, op_type, error_reporter, allocator, builtin_data);
+    }
+
+    case BuiltinOperator_SVDF: {
+      return ParseSvdf(op, op_type, error_reporter, allocator, builtin_data);
+    }
+
     case BuiltinOperator_CONV_2D: {
       auto params = safe_allocator.Allocate<TfLiteConvParams>();
       TF_LITE_ENSURE(error_reporter, params != nullptr);
@@ -203,7 +348,7 @@ TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
         params->stride_width = conv_params->stride_w();
         params->stride_height = conv_params->stride_h();
         params->activation =
-            parse_activation(conv_params->fused_activation_function());
+            ConvertActivation(conv_params->fused_activation_function());
 
         params->dilation_width_factor = conv_params->dilation_w_factor();
         params->dilation_height_factor = conv_params->dilation_h_factor();
@@ -247,7 +392,7 @@ TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
         params->filter_width = pool_params->filter_width();
         params->filter_height = pool_params->filter_height();
         params->activation =
-            parse_activation(pool_params->fused_activation_function());
+            ConvertActivation(pool_params->fused_activation_function());
       }
       *builtin_data = params.release();
       return kTfLiteOk;
@@ -262,7 +407,7 @@ TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
         params->stride_height = conv_params->stride_h();
         params->depth_multiplier = conv_params->depth_multiplier();
         params->activation =
-            parse_activation(conv_params->fused_activation_function());
+            ConvertActivation(conv_params->fused_activation_function());
 
         params->dilation_width_factor = conv_params->dilation_w_factor();
         params->dilation_height_factor = conv_params->dilation_h_factor();
@@ -270,26 +415,13 @@ TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
       *builtin_data = params.release();
       return kTfLiteOk;
     }
-    case BuiltinOperator_SVDF: {
-      auto params = safe_allocator.Allocate<TfLiteSVDFParams>();
-      TF_LITE_ENSURE(error_reporter, params != nullptr);
-      if (const auto* svdf_params = op->builtin_options_as_SVDFOptions()) {
-        params->rank = svdf_params->rank();
-        params->activation =
-            parse_activation(svdf_params->fused_activation_function());
-        params->asymmetric_quantize_inputs =
-            svdf_params->asymmetric_quantize_inputs();
-      }
-      *builtin_data = params.release();
-      return kTfLiteOk;
-    }
     case BuiltinOperator_UNIDIRECTIONAL_SEQUENCE_RNN: {
       auto params = safe_allocator.Allocate<TfLiteSequenceRNNParams>();
       TF_LITE_ENSURE(error_reporter, params != nullptr);
       if (const auto* sequence_rnn_params =
               op->builtin_options_as_SequenceRNNOptions()) {
         params->activation =
-            parse_activation(sequence_rnn_params->fused_activation_function());
+            ConvertActivation(sequence_rnn_params->fused_activation_function());
         params->time_major = sequence_rnn_params->time_major();
         params->asymmetric_quantize_inputs =
             sequence_rnn_params->asymmetric_quantize_inputs();
@@ -303,7 +435,7 @@ TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
       TF_LITE_ENSURE(error_reporter, params != nullptr);
       if (const auto* bidi_sequence_rnn_params =
               op->builtin_options_as_BidirectionalSequenceRNNOptions()) {
-        params->activation = parse_activation(
+        params->activation = ConvertActivation(
             bidi_sequence_rnn_params->fused_activation_function());
         params->time_major = bidi_sequence_rnn_params->time_major();
         params->merge_outputs = bidi_sequence_rnn_params->merge_outputs();
@@ -318,7 +450,7 @@ TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
       TF_LITE_ENSURE(error_reporter, params != nullptr);
       if (const auto* rnn_params = op->builtin_options_as_RNNOptions()) {
         params->activation =
-            parse_activation(rnn_params->fused_activation_function());
+            ConvertActivation(rnn_params->fused_activation_function());
         params->asymmetric_quantize_inputs =
             rnn_params->asymmetric_quantize_inputs();
       }
@@ -336,53 +468,17 @@ TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
       *builtin_data = params.release();
       return kTfLiteOk;
     }
-    case BuiltinOperator_FULLY_CONNECTED: {
-      auto params = safe_allocator.Allocate<TfLiteFullyConnectedParams>();
-      TF_LITE_ENSURE(error_reporter, params != nullptr);
-      if (const auto* fully_connected_params =
-              op->builtin_options_as_FullyConnectedOptions()) {
-        params->activation = parse_activation(
-            fully_connected_params->fused_activation_function());
-        params->keep_num_dims = fully_connected_params->keep_num_dims();
-        params->asymmetric_quantize_inputs =
-            fully_connected_params->asymmetric_quantize_inputs();
-        switch (fully_connected_params->weights_format()) {
-          case FullyConnectedOptionsWeightsFormat_DEFAULT:
-            params->weights_format = kTfLiteFullyConnectedWeightsFormatDefault;
-            break;
-          case FullyConnectedOptionsWeightsFormat_SHUFFLED4x16INT8:
-            params->weights_format =
-                kTfLiteFullyConnectedWeightsFormatShuffled4x16Int8;
-            break;
-          default:
-            TF_LITE_REPORT_ERROR(error_reporter,
-                                 "Unhandled fully-connected weights format.");
-            return kTfLiteError;
-        }
-      }
-      *builtin_data = params.release();
-      return kTfLiteOk;
-    }
+
     case BuiltinOperator_HASHTABLE_LOOKUP:
       // no-op.
       return kTfLiteOk;
-    case BuiltinOperator_SOFTMAX: {
-      auto params = safe_allocator.Allocate<TfLiteSoftmaxParams>();
-      TF_LITE_ENSURE(error_reporter, params != nullptr);
-      if (const auto* softmax_params =
-              op->builtin_options_as_SoftmaxOptions()) {
-        params->beta = softmax_params->beta();
-      }
-      *builtin_data = params.release();
-      return kTfLiteOk;
-    }
     case BuiltinOperator_CONCATENATION: {
       auto params = safe_allocator.Allocate<TfLiteConcatenationParams>();
       TF_LITE_ENSURE(error_reporter, params != nullptr);
       if (const auto* concatenation_params =
               op->builtin_options_as_ConcatenationOptions()) {
-        params->activation =
-            parse_activation(concatenation_params->fused_activation_function());
+        params->activation = ConvertActivation(
+            concatenation_params->fused_activation_function());
         params->axis = concatenation_params->axis();
       }
       *builtin_data = params.release();
@@ -393,7 +489,7 @@ TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
       TF_LITE_ENSURE(error_reporter, params != nullptr);
       if (const auto* schema_params = op->builtin_options_as_MulOptions()) {
         params->activation =
-            parse_activation(schema_params->fused_activation_function());
+            ConvertActivation(schema_params->fused_activation_function());
       }
       *builtin_data = params.release();
       return kTfLiteOk;
@@ -403,7 +499,7 @@ TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
       TF_LITE_ENSURE(error_reporter, params != nullptr);
       if (const auto* schema_params = op->builtin_options_as_AddOptions()) {
         params->activation =
-            parse_activation(schema_params->fused_activation_function());
+            ConvertActivation(schema_params->fused_activation_function());
       }
       *builtin_data = params.release();
       return kTfLiteOk;
@@ -413,7 +509,7 @@ TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
       TF_LITE_ENSURE(error_reporter, params != nullptr);
       if (const auto* schema_params = op->builtin_options_as_DivOptions()) {
         params->activation =
-            parse_activation(schema_params->fused_activation_function());
+            ConvertActivation(schema_params->fused_activation_function());
       }
       *builtin_data = params.release();
       return kTfLiteOk;
@@ -423,7 +519,7 @@ TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
       TF_LITE_ENSURE(error_reporter, params != nullptr);
       if (const auto* schema_params = op->builtin_options_as_SubOptions()) {
         params->activation =
-            parse_activation(schema_params->fused_activation_function());
+            ConvertActivation(schema_params->fused_activation_function());
       }
       *builtin_data = params.release();
       return kTfLiteOk;
@@ -433,7 +529,7 @@ TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
       TF_LITE_ENSURE(error_reporter, params != nullptr);
       if (const auto* schema_params = op->builtin_options_as_L2NormOptions()) {
         params->activation =
-            parse_activation(schema_params->fused_activation_function());
+            ConvertActivation(schema_params->fused_activation_function());
       }
       *builtin_data = params.release();
       return kTfLiteOk;
@@ -456,7 +552,7 @@ TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
       TF_LITE_ENSURE(error_reporter, params != nullptr);
       if (const auto* lstm_params = op->builtin_options_as_LSTMOptions()) {
         params->activation =
-            parse_activation(lstm_params->fused_activation_function());
+            ConvertActivation(lstm_params->fused_activation_function());
         params->cell_clip = lstm_params->cell_clip();
         params->proj_clip = lstm_params->proj_clip();
         switch (lstm_params->kernel_type()) {
@@ -489,7 +585,7 @@ TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
       if (const auto* seq_lstm_params =
               op->builtin_options_as_UnidirectionalSequenceLSTMOptions()) {
         params->activation =
-            parse_activation(seq_lstm_params->fused_activation_function());
+            ConvertActivation(seq_lstm_params->fused_activation_function());
         params->cell_clip = seq_lstm_params->cell_clip();
         params->proj_clip = seq_lstm_params->proj_clip();
         params->time_major = seq_lstm_params->time_major();
@@ -506,7 +602,7 @@ TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
       if (const auto* bidi_lstm_params =
               op->builtin_options_as_BidirectionalSequenceLSTMOptions()) {
         params->activation =
-            parse_activation(bidi_lstm_params->fused_activation_function());
+            ConvertActivation(bidi_lstm_params->fused_activation_function());
         params->cell_clip = bidi_lstm_params->cell_clip();
         params->proj_clip = bidi_lstm_params->proj_clip();
         params->merge_outputs = bidi_lstm_params->merge_outputs();
@@ -857,7 +953,6 @@ TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
     case BuiltinOperator_CONCAT_EMBEDDINGS:
     case BuiltinOperator_COS:
     case BuiltinOperator_CUSTOM:
-    case BuiltinOperator_DEQUANTIZE:
     case BuiltinOperator_ELU:
     case BuiltinOperator_EMBEDDING_LOOKUP:
     case BuiltinOperator_EQUAL:
@@ -913,7 +1008,6 @@ TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
     case BuiltinOperator_GATHER_ND:
     case BuiltinOperator_WHERE:
     case BuiltinOperator_RANK:
-    case BuiltinOperator_QUANTIZE:
     case BuiltinOperator_NON_MAX_SUPPRESSION_V4:
     case BuiltinOperator_NON_MAX_SUPPRESSION_V5:
     case BuiltinOperator_SCATTER_ND:
diff --git a/tensorflow/lite/core/api/flatbuffer_conversions.h b/tensorflow/lite/core/api/flatbuffer_conversions.h
index 2feddfaa8e6..45f2c9df3b7 100644
--- a/tensorflow/lite/core/api/flatbuffer_conversions.h
+++ b/tensorflow/lite/core/api/flatbuffer_conversions.h
@@ -69,6 +69,35 @@ TfLiteStatus ParseOpData(const Operator* op, BuiltinOperator op_type,
 TfLiteStatus ConvertTensorType(TensorType tensor_type, TfLiteType* type,
                                ErrorReporter* error_reporter);
 
+// TODO(b/149408647): The (unnecessary) op_type parameter in the functions below
+// is to keep the same signature as ParseOpData. This allows for a gradual
+// transfer to selective registration of the parse function, but should be
+// removed once we are no longer using ParseOpData for the OpResolver
+// implementation in micro.
+
+TfLiteStatus ParseDequantize(const Operator* op, BuiltinOperator op_type,
+                             ErrorReporter* error_reporter,
+                             BuiltinDataAllocator* allocator,
+                             void** builtin_data);
+
+TfLiteStatus ParseFullyConnected(const Operator* op, BuiltinOperator op_type,
+                                 ErrorReporter* error_reporter,
+                                 BuiltinDataAllocator* allocator,
+                                 void** builtin_data);
+
+TfLiteStatus ParseQuantize(const Operator* op, BuiltinOperator op_type,
+                           ErrorReporter* error_reporter,
+                           BuiltinDataAllocator* allocator,
+                           void** builtin_data);
+
+TfLiteStatus ParseSoftmax(const Operator* op, BuiltinOperator op_type,
+                          ErrorReporter* error_reporter,
+                          BuiltinDataAllocator* allocator, void** builtin_data);
+
+TfLiteStatus ParseSvdf(const Operator* op, BuiltinOperator op_type,
+                       ErrorReporter* error_reporter,
+                       BuiltinDataAllocator* allocator, void** builtin_data);
+
 }  // namespace tflite
 
 #endif  // TENSORFLOW_LITE_CORE_API_FLATBUFFER_CONVERSIONS_H_
diff --git a/tensorflow/lite/core/subgraph.cc b/tensorflow/lite/core/subgraph.cc
index 81710df128b..0f11af51488 100644
--- a/tensorflow/lite/core/subgraph.cc
+++ b/tensorflow/lite/core/subgraph.cc
@@ -1323,6 +1323,20 @@ TfLiteStatus Subgraph::RemoveAllDelegates() {
 
 bool Subgraph::HasDelegates() { return !delegates_applied_.empty(); }
 
+void Subgraph::EnsureTensorsVectorCapacity() {
+  const size_t required_capacity = tensors_.size() + kTensorsCapacityHeadroom;
+  if (required_capacity > tensors_.capacity()) {
+    // Whenever it's required to increase the vector capacity, make it at
+    // least twice bigger. The behavior is consistent with the default
+    // behavior of GCC STL's `std::vector::resize()`. This avoids frequently
+    // allocating and copying the underlying buffer.
+    size_t reserved_capacity =
+        std::max(required_capacity, tensors_.capacity() * 2);
+    tensors_.reserve(reserved_capacity);
+    context_.tensors = tensors_.data();
+  }
+}
+
 TfLiteStatus Subgraph::EnsureMemoryAllocations() {
   if (memory_planner_) {
     state_ = kStateUninvokable;
diff --git a/tensorflow/lite/core/subgraph.h b/tensorflow/lite/core/subgraph.h
index d9ccff35105..bee13c9073e 100644
--- a/tensorflow/lite/core/subgraph.h
+++ b/tensorflow/lite/core/subgraph.h
@@ -567,19 +567,7 @@ class Subgraph {
   // capacity. Calling this function may invalidate existing pointers to
   // tensors. After calling this function, adding `kTensorsCapacityHeadroom`
   // more tensors won't invalidate the pointer to existing tensors.
-  void EnsureTensorsVectorCapacity() {
-    const size_t required_capacity = tensors_.size() + kTensorsCapacityHeadroom;
-    if (required_capacity > tensors_.capacity()) {
-      // Whenever it's required to increase the vector capacity, make it at
-      // least twice bigger. The behavior is consistent with the default
-      // behavior of GCC STL's `std::vector::resize()`. This avoids frequently
-      // allocating and copying the underlying buffer.
-      size_t reserved_capacity =
-          std::max(required_capacity, tensors_.capacity() * 2);
-      tensors_.reserve(reserved_capacity);
-      context_.tensors = tensors_.data();
-    }
-  }
+  void EnsureTensorsVectorCapacity();
 
   // Ensures the memory required is planned and allocated.
   TfLiteStatus EnsureMemoryAllocations();
diff --git a/tensorflow/lite/delegates/flex/BUILD b/tensorflow/lite/delegates/flex/BUILD
index 7981f4b5c27..692aa212482 100644
--- a/tensorflow/lite/delegates/flex/BUILD
+++ b/tensorflow/lite/delegates/flex/BUILD
@@ -1,4 +1,6 @@
 load("//tensorflow:tensorflow.bzl", "tf_cc_test", "tf_opts_nortti_if_lite_protos")
+load("//tensorflow/java:build_defs.bzl", "JAVACOPTS")
+load("//tensorflow/lite/delegates/flex:build_def.bzl", "tflite_flex_cc_library", "tflite_flex_jni_library")
 
 #
 # This is a TF Lite delegate that is powered by TensorFlow's Eager.
@@ -6,11 +8,16 @@ load("//tensorflow:tensorflow.bzl", "tf_cc_test", "tf_opts_nortti_if_lite_protos
 package(
     default_visibility = [
         "//tensorflow/compiler/mlir/lite:__subpackages__",
+        "//tensorflow/lite/android:__subpackages__",
         "//tensorflow/lite/toco/tflite:__subpackages__",
     ],
     licenses = ["notice"],  # Apache 2.0
 )
 
+exports_files([
+    "delegate.h",
+])
+
 cc_library(
     name = "buffer_map",
     srcs = ["buffer_map.cc"],
@@ -50,31 +57,18 @@ tf_cc_test(
     ],
 )
 
-# Delegate implementation that pulls in the standard set of TensorFlow ops and
-# kernels.
-cc_library(
+# Define the standard flex delegate library, that pulls in the standard set
+# of TensorFlow ops and kernels, using tflite_flex_cc_library with no
+# portable_tensorflow_lib parameter. Custom flex delegate can be defined with
+# tflite_flex_cc_library if the parameter portable_tensorflow_lib
+# is provided. Ex:
+# tflite_flex_cc_library(
+#   name = "sample",
+#   portable_tensorflow_lib = custom_portable_tensorflow_lib,
+# )
+tflite_flex_cc_library(
     name = "delegate",
-    hdrs = [
-        "delegate.h",
-    ],
     visibility = ["//visibility:public"],
-    deps = [
-        ":delegate_data",
-        ":delegate_only_runtime",
-        "//tensorflow/lite/delegates/utils:simple_delegate",
-    ] + select({
-        "//tensorflow:android": [
-            "//tensorflow/core:portable_tensorflow_lib",
-        ],
-        "//tensorflow:ios": [
-            "//tensorflow/core:portable_tensorflow_lib",
-        ],
-        "//conditions:default": [
-            "//tensorflow/core:tensorflow",
-            "//tensorflow/lite/c:common",
-        ],
-    }),
-    alwayslink = 1,
 )
 
 # Delegate implementation that does *not* pull in the standard set of TensorFlow
@@ -245,3 +239,54 @@ cc_library(
         "whitelisted_flex_ops.h",
     ],
 )
+
+# Following targets are using for testing selective-built flex delegate
+# in Java. Please don't use them for other purposes.
+tflite_flex_jni_library(
+    name = "test",
+    models = [
+        "//tensorflow/lite:testdata/multi_add_flex.bin",
+    ],
+    visibility = ["//tensorflow/lite/android:__subpackages__"],
+)
+
+java_library(
+    name = "test_tensorflowlitelib_flex",
+    testonly = 1,
+    srcs = ["//tensorflow/lite/delegates/flex/java/src/main/java/org/tensorflow/lite/flex:flex_delegate"],
+    javacopts = JAVACOPTS,
+    visibility = ["//visibility:private"],
+    deps = [
+        ":libtensorflowlite_flex_jni.so",
+        "//tensorflow/lite/java:tensorflowlitelib",
+        "@org_checkerframework_qual",
+    ],
+)
+
+java_test(
+    name = "SelectiveBuiltInterpreterFlexTest",
+    size = "small",
+    srcs = [
+        "//tensorflow/lite/java:portable_flex_tests",
+        "//tensorflow/lite/java:portable_test_utils",
+    ],
+    data = [
+        "//tensorflow/lite:testdata/multi_add_flex.bin",
+    ],
+    javacopts = JAVACOPTS,
+    tags = [
+        "no_cuda_on_cpu_tap",  # CUDA + flex is not officially supported.
+        "no_gpu",  # GPU + flex is not officially supported.
+        "no_oss",  # Currently requires --config=monolithic, b/118895218.
+        # TODO(b/121204962): Re-enable test after fixing memory leaks.
+        "noasan",
+    ],
+    test_class = "org.tensorflow.lite.InterpreterFlexTest",
+    visibility = ["//visibility:private"],
+    deps = [
+        ":test_tensorflowlitelib_flex",
+        "//tensorflow/lite/java:tensorflowlitelib",
+        "@com_google_truth",
+        "@junit",
+    ],
+)
diff --git a/tensorflow/lite/delegates/flex/build_def.bzl b/tensorflow/lite/delegates/flex/build_def.bzl
new file mode 100644
index 00000000000..400ea57f290
--- /dev/null
+++ b/tensorflow/lite/delegates/flex/build_def.bzl
@@ -0,0 +1,230 @@
+"""Generate custom library flex delegate."""
+
+load(
+    "//tensorflow:tensorflow.bzl",
+    "if_android",
+    "if_ios",
+    "if_mobile",
+    "tf_copts",
+    "tf_defines_nortti_if_lite_protos",
+    "tf_features_nomodules_if_mobile",
+    "tf_opts_nortti_if_lite_protos",
+    "tf_portable_full_lite_protos",
+)
+load(
+    "//tensorflow/lite:build_def.bzl",
+    "tflite_copts",
+    "tflite_jni_binary",
+    "tflite_jni_linkopts",
+)
+
+def generate_flex_kernel_header(
+        name,
+        models):
+    """A rule to generate a header file listing only used operators.
+
+    Args:
+      name: Name of the generated library.
+      models: TFLite models to interpret.
+
+    Returns:
+      A struct with 'header' and 'include_path' fields that
+      contain the generated header and the required include entry.
+    """
+    include_path = "%s_tf_generated_kernel_header" % name
+    header = include_path + "/ops_to_register.h"
+
+    if type(models) != type([]):
+        models = [models]
+
+    # List all flex ops from models.
+    model_file_args = " --graphs=%s" % ",".join(
+        ["$(location %s)" % f for f in models],
+    )
+    list_ops_output = include_path + "/list_flex_ops"
+    list_ops_tool = "//tensorflow/lite/tools:list_flex_ops_main"
+    native.genrule(
+        name = "%s_custom_list_flex_ops" % name,
+        srcs = models,
+        outs = [list_ops_output],
+        tools = [list_ops_tool],
+        message = "Listing flex ops from %s..." % ",".join(models),
+        cmd = ("$(location " + list_ops_tool + ")" +
+               model_file_args + " > \"$@\""),
+    )
+
+    # Generate the kernel registration header file from list of flex ops.
+    tool = "//tensorflow/python/tools:print_selective_registration_header"
+    native.genrule(
+        name = "%s_custom_kernel_registration" % name,
+        srcs = [list_ops_output],
+        outs = [header],
+        tools = [tool],
+        message = "Processing %s..." % list_ops_output,
+        cmd = ("$(location " + tool + ")" +
+               " --default_ops=\"\"" +
+               " --proto_fileformat=ops_list" +
+               " --graphs=" + "$(location " + list_ops_output + ") > \"$@\""),
+    )
+    return struct(include_path = include_path, header = header)
+
+def tflite_flex_cc_library(
+        name,
+        portable_tensorflow_lib = "//tensorflow/core:portable_tensorflow_lib",
+        visibility = ["//visibility:public"]):
+    """A rule to generate a flex delegate with custom android and ios tensorflow libs.
+
+    These libs should be a custom version of android_tensorflow_lib and ios_tensorflow_lib
+    and contain ops registrations and kernels. If not defined, the default libs will be used.
+
+    Args:
+      name: Name of the generated rule.
+      portable_tensorflow_lib: the tensorflow_lib to be added in deps for android and ios,
+          can be a full or trimmed version.
+      visibility: visibility of the generated rule.
+    """
+    native.cc_library(
+        name = name,
+        hdrs = [
+            "//tensorflow/lite/delegates/flex:delegate.h",
+        ],
+        visibility = visibility,
+        deps = [
+            "//tensorflow/lite/delegates/flex:delegate_data",
+            "//tensorflow/lite/delegates/flex:delegate_only_runtime",
+            "//tensorflow/lite/delegates/utils:simple_delegate",
+        ] + select({
+            "//tensorflow:android": [
+                portable_tensorflow_lib,
+            ],
+            "//tensorflow:ios": [
+                portable_tensorflow_lib,
+            ],
+            "//conditions:default": [
+                "//tensorflow/core:tensorflow",
+                "//tensorflow/lite/c:common",
+            ],
+        }),
+        alwayslink = 1,
+    )
+
+def tflite_flex_jni_library(
+        name,
+        models,
+        visibility = ["//visibility:private"]):
+    """A rule to generate a jni library listing only used operators.
+
+    The libtensorflowlite_flex_jni.so name is fixed due to a limitation in JNI
+    Java wrapper, so please make sure there is no naming conflicts.
+
+    Args:
+      name: Name of the generated library.
+      models: TFLite models to interpret.
+      visibility: visibility of the generated rules.
+
+    Returns:
+      Generate a jni library support flex ops.
+    """
+    portable_tensorflow_lib = "//tensorflow/core:portable_tensorflow_lib"
+    if models:
+        CUSTOM_KERNEL_HEADER = generate_flex_kernel_header(
+            name = "%s_custom_tf_op_headers" % name,
+            models = models,
+        )
+
+        # Define a custom_tensorflow_lib with selective registration.
+        # The library will only contain ops exist in provided models.
+        native.cc_library(
+            name = "%s_custom_tensorflow_lib" % name,
+            srcs = if_mobile([
+                "//tensorflow/core:portable_op_registrations_and_gradients",
+                "//tensorflow/core/kernels:android_all_ops",
+            ]) + [CUSTOM_KERNEL_HEADER.header],
+            copts = tf_copts(android_optimization_level_override = None) + tf_opts_nortti_if_lite_protos() + if_ios(["-Os"]),
+            defines = [
+                "SELECTIVE_REGISTRATION",
+                "SUPPORT_SELECTIVE_REGISTRATION",
+            ] + tf_portable_full_lite_protos(
+                full = [],
+                lite = ["TENSORFLOW_LITE_PROTOS"],
+            ) + tf_defines_nortti_if_lite_protos(),
+            features = tf_features_nomodules_if_mobile(),
+            linkopts = if_android(["-lz"]) + if_ios(["-lz"]),
+            includes = [
+                CUSTOM_KERNEL_HEADER.include_path,
+            ],
+            textual_hdrs = [
+                "//tensorflow/core/kernels:android_all_ops_textual_hdrs",
+            ],
+            visibility = visibility,
+            deps = [
+                "@com_google_absl//absl/strings:str_format",
+                "//third_party/fft2d:fft2d_headers",
+                "//third_party/eigen3",
+                "@com_google_absl//absl/types:optional",
+                "@gemmlowp",
+                "//tensorflow/core:protos_all_cc",
+                "//tensorflow/core:portable_tensorflow_lib_lite",
+            ],
+            alwayslink = 1,
+        )
+        portable_tensorflow_lib = ":%s_custom_tensorflow_lib" % name
+
+    # Define a custom_init_tensorflow that depends on the custom_tensorflow_lib.
+    # This will avoid the symbols re-definition errors.
+    native.cc_library(
+        name = "%s_custom_init_tensorflow" % name,
+        srcs = [
+            "//tensorflow/lite/testing:init_tensorflow.cc",
+        ],
+        hdrs = [
+            "//tensorflow/lite/testing:init_tensorflow.h",
+        ],
+        visibility = visibility,
+        deps = select({
+            "//conditions:default": [
+                "//tensorflow/core:lib",
+            ],
+            "//tensorflow:android": [
+                portable_tensorflow_lib,
+            ],
+            "//tensorflow:ios": [
+                portable_tensorflow_lib,
+            ],
+        }),
+    )
+
+    # Define a custom_flex_delegate that depends on custom_tensorflow_lib.
+    # This will reduce the binary size comparing to the original flex delegate.
+    tflite_flex_cc_library(
+        name = "%s_custom_flex_delegate" % name,
+        portable_tensorflow_lib = portable_tensorflow_lib,
+        visibility = visibility,
+    )
+
+    # Define a custom_flex_native that depends on custom_flex_delegate and custom_init_tensorflow.
+    native.cc_library(
+        name = "%s_custom_flex_native" % name,
+        srcs = [
+            "//tensorflow/lite/delegates/flex/java/src/main/native:flex_delegate_jni.cc",
+        ],
+        copts = tflite_copts(),
+        visibility = visibility,
+        deps = [
+            ":%s_custom_flex_delegate" % name,
+            "%s_custom_init_tensorflow" % name,
+            "//tensorflow/lite/java/jni",
+            "//tensorflow/lite/delegates/utils:simple_delegate",
+        ],
+        alwayslink = 1,
+    )
+
+    # Build the jni binary based on the custom_flex_native.
+    # The library name is fixed as libtensorflowlite_flex_jni.so in FlexDelegate.java.
+    tflite_jni_binary(
+        name = "libtensorflowlite_flex_jni.so",
+        linkopts = tflite_jni_linkopts(),
+        deps = [
+            ":%s_custom_flex_native" % name,
+        ],
+    )
diff --git a/tensorflow/lite/delegates/flex/java/src/main/native/BUILD b/tensorflow/lite/delegates/flex/java/src/main/native/BUILD
index 4732b963ed2..b9f8cfc1206 100644
--- a/tensorflow/lite/delegates/flex/java/src/main/native/BUILD
+++ b/tensorflow/lite/delegates/flex/java/src/main/native/BUILD
@@ -2,12 +2,17 @@
 # Java Native Interface (JNI) library intended for implementing the
 # TensorFlow Lite Flex delegate for using TensorFlow ops with TensorFlow Lite.
 
-package(default_visibility = ["//visibility:public"])
-
 load("//tensorflow/lite:build_def.bzl", "tflite_copts")
 
+package(default_visibility = ["//visibility:public"])
+
 licenses(["notice"])  # Apache 2.0
 
+exports_files(
+    srcs = ["flex_delegate_jni.cc"],
+    visibility = ["//visibility:public"],
+)
+
 cc_library(
     name = "native",
     srcs = ["flex_delegate_jni.cc"],
diff --git a/tensorflow/lite/delegates/flex/whitelisted_flex_ops.cc b/tensorflow/lite/delegates/flex/whitelisted_flex_ops.cc
index d9150698298..e173b47ec32 100644
--- a/tensorflow/lite/delegates/flex/whitelisted_flex_ops.cc
+++ b/tensorflow/lite/delegates/flex/whitelisted_flex_ops.cc
@@ -163,6 +163,7 @@ bool IsWhitelistedFlexOp(const std::string& tensorflow_op_name) {
           "IRFFT",
           "IRFFT2D",
           "IRFFT3D",
+          "Imag",
           "ImmutableConst",
           "InTopK",
           "InTopKV2",
@@ -283,6 +284,7 @@ bool IsWhitelistedFlexOp(const std::string& tensorflow_op_name) {
           "RandomUniformInt",
           "Range",
           "Rank",
+          "Real",
           "RealDiv",
           "Reciprocal",
           "ReciprocalGrad",
@@ -423,6 +425,7 @@ bool IsWhitelistedFlexOp(const std::string& tensorflow_op_name) {
           "StridedSliceAssign",
           "StridedSliceGrad",
           "StringJoin",
+          "StringToHashBucketFast",
           "Sub",
           "Sum",
           "Switch",
diff --git a/tensorflow/lite/delegates/gpu/BUILD b/tensorflow/lite/delegates/gpu/BUILD
index 181d96b9cdf..54fc124cde6 100644
--- a/tensorflow/lite/delegates/gpu/BUILD
+++ b/tensorflow/lite/delegates/gpu/BUILD
@@ -90,6 +90,7 @@ objc_library(
         "//tensorflow/lite/delegates/gpu/common:model",
         "//tensorflow/lite/delegates/gpu/common:model_builder",
         "//tensorflow/lite/delegates/gpu/common:model_transformer",
+        "//tensorflow/lite/delegates/gpu/common:quantization_util",
         "//tensorflow/lite/delegates/gpu/common:shape",
         "//tensorflow/lite/delegates/gpu/common:status",
         "//tensorflow/lite/delegates/gpu/common:tensor",
diff --git a/tensorflow/lite/delegates/gpu/cl/arguments.cc b/tensorflow/lite/delegates/gpu/cl/arguments.cc
index d5c0f73469b..ef4683a9bee 100644
--- a/tensorflow/lite/delegates/gpu/cl/arguments.cc
+++ b/tensorflow/lite/delegates/gpu/cl/arguments.cc
@@ -45,6 +45,7 @@ size_t FindEnclosingBracket(const std::string& text, size_t first_pos,
       {'(', ')'},
       {'{', '}'},
       {'[', ']'},
+      {'<', '>'},
   };
   char b_open = bracket;
   auto it = brackets.find(b_open);
@@ -70,6 +71,28 @@ size_t FindEnclosingBracket(const std::string& text, size_t first_pos,
   }
 }
 
+absl::Status ParseArgsInsideBrackets(const std::string& text,
+                                     size_t open_bracket_pos,
+                                     size_t* close_bracket_pos,
+                                     std::vector<std::string>* args) {
+  *close_bracket_pos =
+      FindEnclosingBracket(text, open_bracket_pos + 1, text[open_bracket_pos]);
+  if (*close_bracket_pos == -1) {
+    return absl::NotFoundError("Not found enclosing bracket");
+  }
+  std::string str_args = text.substr(open_bracket_pos + 1,
+                                     *close_bracket_pos - open_bracket_pos - 2);
+  std::vector<absl::string_view> words = absl::StrSplit(str_args, ',');
+  args->reserve(words.size());
+  for (const auto& word : words) {
+    absl::string_view arg = absl::StripAsciiWhitespace(word);
+    if (!arg.empty()) {
+      args->push_back(std::string(arg));
+    }
+  }
+  return absl::OkStatus();
+}
+
 void ReplaceAllWords(const std::string& old_word, const std::string& new_word,
                      std::string* str) {
   size_t position = str->find(old_word);
@@ -107,6 +130,9 @@ std::string GetImageModifier(AccessType access) {
 
 }  // namespace
 
+// Static
+constexpr char Arguments::kArgsPrefix[];
+
 Arguments::Arguments(Arguments&& args)
     : int_values_(std::move(args.int_values_)),
       shared_int4s_data_(std::move(args.shared_int4s_data_)),
@@ -534,10 +560,10 @@ void Arguments::ResolveObjectNames(const std::string& object_name,
   }
 }
 
-absl::Status Arguments::ResolveSelector(const std::string& object_name,
-                                        const std::string& selector,
-                                        const std::vector<std::string>& args,
-                                        std::string* result) {
+absl::Status Arguments::ResolveSelector(
+    const std::string& object_name, const std::string& selector,
+    const std::vector<std::string>& args,
+    const std::vector<std::string>& template_args, std::string* result) {
   const GPUObjectDescriptor* desc_ptr;
   AccessType access_type;
   if (auto it = object_refs_.find(object_name); it != object_refs_.end()) {
@@ -550,7 +576,8 @@ absl::Status Arguments::ResolveSelector(const std::string& object_name,
     return absl::NotFoundError(
         absl::StrCat("No object with name - ", object_name));
   }
-  RETURN_IF_ERROR(desc_ptr->PerformSelector(selector, args, result));
+  RETURN_IF_ERROR(
+      desc_ptr->PerformSelector(selector, args, template_args, result));
   auto names = desc_ptr->GetGPUResources(access_type).GetNames();
   ResolveObjectNames(object_name, names, result);
   return absl::OkStatus();
@@ -570,32 +597,26 @@ absl::Status Arguments::ResolveSelectorsPass(std::string* code) {
       std::string selector_name = GetNextWord(*code, next_position);
       next_position += selector_name.size();
       next = (*code)[next_position];
+      std::vector<std::string> template_args;
+      if (next == '<') {
+        size_t close_bracket_pos;
+        RETURN_IF_ERROR(ParseArgsInsideBrackets(
+            *code, next_position, &close_bracket_pos, &template_args));
+        next_position = close_bracket_pos;
+        next = (*code)[next_position];
+      }
       if (next != '(') {
         return absl::NotFoundError(
             absl::StrCat("Expected ( after function ", selector_name, " call"));
       }
-      next_position += 1;
-      size_t bracket_pos = FindEnclosingBracket(*code, next_position, '(');
-      if (bracket_pos == -1) {
-        return absl::NotFoundError(
-            absl::StrCat("Not found enclosing bracket for function ",
-                         selector_name, " call"));
-      }
-      std::string str_args =
-          code->substr(next_position, bracket_pos - next_position - 1);
-      std::vector<absl::string_view> words = absl::StrSplit(str_args, ',');
       std::vector<std::string> args;
-      args.reserve(words.size());
-      for (const auto& word : words) {
-        absl::string_view arg = absl::StripAsciiWhitespace(word);
-        if (!arg.empty()) {
-          args.push_back(std::string(arg));
-        }
-      }
+      size_t close_bracket_pos;
+      RETURN_IF_ERROR(ParseArgsInsideBrackets(*code, next_position,
+                                              &close_bracket_pos, &args));
       std::string patch;
-      RETURN_IF_ERROR(
-          ResolveSelector(object_name, selector_name, args, &patch));
-      code->replace(arg_pos, bracket_pos - arg_pos, patch);
+      RETURN_IF_ERROR(ResolveSelector(object_name, selector_name, args,
+                                      template_args, &patch));
+      code->replace(arg_pos, close_bracket_pos - arg_pos, patch);
       position = arg_pos + patch.size();
     } else {
       position = arg_pos + strlen(kArgsPrefix);
diff --git a/tensorflow/lite/delegates/gpu/cl/arguments.h b/tensorflow/lite/delegates/gpu/cl/arguments.h
index 17ef4353de8..453ffcb56b2 100644
--- a/tensorflow/lite/delegates/gpu/cl/arguments.h
+++ b/tensorflow/lite/delegates/gpu/cl/arguments.h
@@ -88,6 +88,7 @@ class Arguments {
   absl::Status ResolveSelector(const std::string& object_name,
                                const std::string& selector,
                                const std::vector<std::string>& args,
+                               const std::vector<std::string>& template_args,
                                std::string* result);
 
   void ResolveObjectNames(const std::string& object_name,
diff --git a/tensorflow/lite/delegates/gpu/cl/egl_sync.cc b/tensorflow/lite/delegates/gpu/cl/egl_sync.cc
index f50bc75b8be..2cc0d6303dc 100644
--- a/tensorflow/lite/delegates/gpu/cl/egl_sync.cc
+++ b/tensorflow/lite/delegates/gpu/cl/egl_sync.cc
@@ -21,13 +21,40 @@ namespace tflite {
 namespace gpu {
 namespace cl {
 
+namespace {
+
+bool HasExtension(EGLDisplay display, const char* extension) {
+  const char* extensions = eglQueryString(display, EGL_EXTENSIONS);
+  return extensions && std::strstr(extensions, extension);
+}
+
+absl::Status IsEglFenceSyncSupported(EGLDisplay display) {
+  static bool supported = HasExtension(display, "EGL_KHR_fence_sync");
+  if (supported) {
+    return absl::OkStatus();
+  }
+  return absl::InternalError("Not supported: EGL_KHR_fence_sync");
+}
+
+absl::Status IsEglWaitSyncSupported(EGLDisplay display) {
+  static bool supported = HasExtension(display, "EGL_KHR_wait_sync");
+  if (supported) {
+    return absl::OkStatus();
+  }
+  return absl::InternalError("Not supported: EGL_KHR_wait_sync");
+}
+
+}  // anonymous namespace
+
 absl::Status EglSync::NewFence(EGLDisplay display, EglSync* sync) {
+  RETURN_IF_ERROR(IsEglFenceSyncSupported(display));
   static auto* egl_create_sync_khr =
       reinterpret_cast<decltype(&eglCreateSyncKHR)>(
           eglGetProcAddress("eglCreateSyncKHR"));
   if (egl_create_sync_khr == nullptr) {
     // Needs extension: EGL_KHR_fence_sync (EGL) / GL_OES_EGL_sync (OpenGL ES).
-    return absl::InternalError("Not supported: eglCreateSyncKHR.");
+    return absl::InternalError(
+        "Not supported / bad EGL implementation: eglCreateSyncKHR.");
   }
   EGLSyncKHR egl_sync;
   RETURN_IF_ERROR(TFLITE_GPU_CALL_EGL(*egl_create_sync_khr, &egl_sync, display,
@@ -54,7 +81,7 @@ void EglSync::Invalidate() {
         reinterpret_cast<decltype(&eglDestroySyncKHR)>(
             eglGetProcAddress("eglDestroySyncKHR"));
     // Needs extension: EGL_KHR_fence_sync (EGL) / GL_OES_EGL_sync (OpenGL ES).
-    if (egl_destroy_sync_khr) {
+    if (IsEglFenceSyncSupported(display_).ok() && egl_destroy_sync_khr) {
       // Note: we're doing nothing when the function pointer is nullptr, or the
       // call returns EGL_FALSE.
       (*egl_destroy_sync_khr)(display_, sync_);
@@ -64,6 +91,7 @@ void EglSync::Invalidate() {
 }
 
 absl::Status EglSync::ServerWait() {
+  RETURN_IF_ERROR(IsEglWaitSyncSupported(display_));
   static auto* egl_wait_sync_khr = reinterpret_cast<decltype(&eglWaitSyncKHR)>(
       eglGetProcAddress("eglWaitSyncKHR"));
   if (egl_wait_sync_khr == nullptr) {
@@ -78,6 +106,7 @@ absl::Status EglSync::ServerWait() {
 }
 
 absl::Status EglSync::ClientWait() {
+  RETURN_IF_ERROR(IsEglFenceSyncSupported(display_));
   static auto* egl_client_wait_sync_khr =
       reinterpret_cast<decltype(&eglClientWaitSyncKHR)>(
           eglGetProcAddress("eglClientWaitSyncKHR"));
diff --git a/tensorflow/lite/delegates/gpu/cl/gpu_object.h b/tensorflow/lite/delegates/gpu/cl/gpu_object.h
index b936c1b01ee..fec8999e2bc 100644
--- a/tensorflow/lite/delegates/gpu/cl/gpu_object.h
+++ b/tensorflow/lite/delegates/gpu/cl/gpu_object.h
@@ -123,9 +123,10 @@ class GPUObjectDescriptor {
     return "";
   }
 
-  virtual absl::Status PerformSelector(const std::string& selector,
-                                       const std::vector<std::string>& args,
-                                       std::string* result) const {
+  virtual absl::Status PerformSelector(
+      const std::string& selector, const std::vector<std::string>& args,
+      const std::vector<std::string>& template_args,
+      std::string* result) const {
     *result = "";
     return absl::OkStatus();
   }
diff --git a/tensorflow/lite/delegates/gpu/cl/kernels/BUILD b/tensorflow/lite/delegates/gpu/cl/kernels/BUILD
index 8b54078dbf8..5c0099fa0a9 100644
--- a/tensorflow/lite/delegates/gpu/cl/kernels/BUILD
+++ b/tensorflow/lite/delegates/gpu/cl/kernels/BUILD
@@ -1123,11 +1123,13 @@ cc_library(
         ":gpu_operation",
         ":util",
         ":work_group_picking",
+        "//tensorflow/lite/delegates/gpu/cl:arguments",
         "//tensorflow/lite/delegates/gpu/cl:cl_kernel",
         "//tensorflow/lite/delegates/gpu/cl:precision",
         "//tensorflow/lite/delegates/gpu/cl:tensor",
         "//tensorflow/lite/delegates/gpu/common:status",
         "//tensorflow/lite/delegates/gpu/common:types",
+        "@com_google_absl//absl/strings",
     ],
 )
 
diff --git a/tensorflow/lite/delegates/gpu/cl/kernels/quantize_and_dequantize.h b/tensorflow/lite/delegates/gpu/cl/kernels/quantize_and_dequantize.h
index 76436ea5dd0..47009d4c0f5 100644
--- a/tensorflow/lite/delegates/gpu/cl/kernels/quantize_and_dequantize.h
+++ b/tensorflow/lite/delegates/gpu/cl/kernels/quantize_and_dequantize.h
@@ -37,11 +37,10 @@ namespace cl {
 // on the GPU, which cannot represent int8 tensors.
 //
 // Implemented as:
-// qvalue = round((min(qmax, max(qmin, src_val)) - qmin) * (1/qscale) + 0.5)
+// qvalue = round((min(qmax, max(qmin, src_val)) - qmin) * (1/qscale))
 // dq_value = qvalue * qscale + qmin
 // Here, qmin, qmax & qscale refer to the quantization values as implemented in
-// TensorFlow Lite's 'FakeQuant' kernel. round(x + 0.5) ensures we round away
-// from zero.
+// TensorFlow Lite's 'FakeQuant' kernel.
 //
 // NOTE: We do not need to nudge min/max values in this op, since they would
 // already be adjusted while generating the quantized model.
diff --git a/tensorflow/lite/delegates/gpu/cl/kernels/softmax.cc b/tensorflow/lite/delegates/gpu/cl/kernels/softmax.cc
index 0f9fcb03097..44c0c883500 100644
--- a/tensorflow/lite/delegates/gpu/cl/kernels/softmax.cc
+++ b/tensorflow/lite/delegates/gpu/cl/kernels/softmax.cc
@@ -17,6 +17,7 @@ limitations under the License.
 
 #include <string>
 
+#include "absl/strings/substitute.h"
 #include "tensorflow/lite/delegates/gpu/cl/kernels/util.h"
 #include "tensorflow/lite/delegates/gpu/cl/kernels/work_group_picking.h"
 #include "tensorflow/lite/delegates/gpu/common/status.h"
@@ -28,37 +29,45 @@ namespace {
 
 std::string GetSoftmaxKernelCode(
     const OperationDef& op_def,
-    const std::vector<ElementwiseOperation*>& linked_operations) {
-  TensorCodeGenerator src_tensor("src_data",
-                                 WHSPoint{"size.x", "size.y", "size.z"},
-                                 op_def.src_tensors[0]);
-  TensorCodeGenerator dst_tensor("dst_data",
-                                 WHSPoint{"size.x", "size.y", "size.z"},
-                                 op_def.dst_tensors[0]);
+    const std::vector<ElementwiseOperation*>& linked_operations,
+    Arguments* args) {
+  auto src_desc = absl::make_unique<TensorDescriptor>(op_def.src_tensors[0]);
+  if (op_def.IsBatchSupported()) {
+    src_desc->SetStateVar("BatchedWidth", "true");
+  }
+  args->AddObjectRef("src_tensor", AccessType::READ, std::move(src_desc));
+  auto dst_desc = absl::make_unique<TensorDescriptor>(op_def.dst_tensors[0]);
+  if (op_def.IsBatchSupported()) {
+    dst_desc->SetStateVar("BatchedWidth", "true");
+  }
+  args->AddObjectRef("dst_tensor", AccessType::WRITE, std::move(dst_desc));
 
   std::string c = GetCommonDefines(op_def.precision);
+  std::string linked_args = GetArgsDeclaration(linked_operations);
+  if (linked_args[0] == ',') {
+    linked_args[0] = ' ';
+  }
   c += "__kernel void main_function(\n";
-  c += src_tensor.GetDeclaration(AccessType::READ);
-  c += GetArgsDeclaration(linked_operations);
-  c += dst_tensor.GetDeclaration(AccessType::WRITE) + ",\n";
-  c += "    int4 size,\n";
-  c += "    float4 mask\n";
-  c += ") {\n";
+  c += linked_args;
+  c += "$0) {\n";
   c += "  int X = get_global_id(0);\n";
   c += "  int Y = get_global_id(1);\n";
-  c += "  if (X >= size.x || Y >= size.y) return; \n";
+  c += "  if (X >= args.dst_tensor.Width() || Y >= args.dst_tensor.Height()) "
+       "return; \n";
   c += "  float sum = 0.0f;\n";
-  c += "  for (int d = 0; d < size.z; ++d) {\n";
-  c += "    float4 mask_temp = d == size.z - 1 ? mask : (float4)(1.0f);\n";
-  c += "    float4 t = " + src_tensor.ReadAsFloatWHS("X", "Y", "d") + ";\n";
-  c += "    sum += dot(mask_temp, exp(t));\n";
+  c += "  for (int d = 0; d < args.dst_tensor.Slices(); ++d) {\n";
+  c += "    float4 t = args.src_tensor.Read<float>(X, Y, d);\n";
+  c += "    sum += exp(t.x);\n";
+  c += "    if (d * 4 + 1 < args.dst_tensor.Channels()) sum += exp(t.y);\n";
+  c += "    if (d * 4 + 2 < args.dst_tensor.Channels()) sum += exp(t.z);\n";
+  c += "    if (d * 4 + 3 < args.dst_tensor.Channels()) sum += exp(t.w);\n";
   c += "  }\n";
-  c += "  for (int d = 0; d < size.z; ++d) {\n";
-  c += "    float4 t = " + src_tensor.ReadAsFloatWHS("X", "Y", "d") + ";\n";
+  c += "  for (int d = 0; d < args.dst_tensor.Slices(); ++d) {\n";
+  c += "    float4 t = args.src_tensor.Read<float>(X, Y, d);\n";
   c += "    t = exp(t) / sum;\n";
   c += "    FLT4 result = TO_FLT4(t);\n";
   c += PostProcess(linked_operations, {"result", "X", "Y", "d"});
-  c += "    " + dst_tensor.WriteWHS("result", "X", "Y", "d");
+  c += "    args.dst_tensor.Write(result, X, Y, d);\n";
   c += "  }\n";
   c += "}\n";
   return c;
@@ -67,11 +76,13 @@ std::string GetSoftmaxKernelCode(
 
 Softmax::Softmax(Softmax&& kernel)
     : GPUOperation(std::move(kernel)),
+      args_(std::move(kernel.args_)),
       kernel_(std::move(kernel.kernel_)),
       work_group_size_(kernel.work_group_size_) {}
 
 Softmax& Softmax::operator=(Softmax&& kernel) {
   if (this != &kernel) {
+    args_ = std::move(kernel.args_);
     kernel_ = std::move(kernel.kernel_);
     std::swap(work_group_size_, kernel.work_group_size_);
     GPUOperation::operator=(std::move(kernel));
@@ -80,20 +91,21 @@ Softmax& Softmax::operator=(Softmax&& kernel) {
 }
 
 absl::Status Softmax::Compile(const CreationContext& creation_context) {
-  const auto code = GetSoftmaxKernelCode(definition_, linked_operations_);
+  std::string code =
+      GetSoftmaxKernelCode(definition_, linked_operations_, &args_);
+  RETURN_IF_ERROR(args_.TransformToCLCode(&code));
+  code = absl::Substitute(code, args_.GetListOfArgs());
   return creation_context.cache->GetOrCreateCLKernel(
       code, "main_function", *creation_context.context,
       *creation_context.device, &kernel_);
 }
 
 absl::Status Softmax::BindArguments() {
+  RETURN_IF_ERROR(args_.SetObjectRef("src_tensor", src_[0]));
+  RETURN_IF_ERROR(args_.SetObjectRef("dst_tensor", dst_[0]));
   kernel_.ResetBindingCounter();
-  RETURN_IF_ERROR(kernel_.SetMemoryAuto(src_[0]->GetMemoryPtr()));
   RETURN_IF_ERROR(BindArgs(&kernel_, linked_operations_));
-  RETURN_IF_ERROR(kernel_.SetMemoryAuto(dst_[0]->GetMemoryPtrForWriting()));
-  RETURN_IF_ERROR(kernel_.SetBytesAuto(src_[0]->GetWBatchedHSB()));
-  RETURN_IF_ERROR(
-      kernel_.SetBytesAuto(GetMaskForLastPlane(src_[0]->Channels())));
+  RETURN_IF_ERROR(args_.Bind(kernel_.kernel(), kernel_.GetBindingCounter()));
   return absl::OkStatus();
 }
 
diff --git a/tensorflow/lite/delegates/gpu/cl/kernels/softmax.h b/tensorflow/lite/delegates/gpu/cl/kernels/softmax.h
index 703a40a4e89..f9598542b11 100644
--- a/tensorflow/lite/delegates/gpu/cl/kernels/softmax.h
+++ b/tensorflow/lite/delegates/gpu/cl/kernels/softmax.h
@@ -16,6 +16,7 @@ limitations under the License.
 #ifndef TENSORFLOW_LITE_DELEGATES_GPU_CL_KERNELS_SOFTMAX_H_
 #define TENSORFLOW_LITE_DELEGATES_GPU_CL_KERNELS_SOFTMAX_H_
 
+#include "tensorflow/lite/delegates/gpu/cl/arguments.h"
 #include "tensorflow/lite/delegates/gpu/cl/cl_kernel.h"
 #include "tensorflow/lite/delegates/gpu/cl/kernels/gpu_operation.h"
 #include "tensorflow/lite/delegates/gpu/cl/precision.h"
@@ -46,6 +47,7 @@ class Softmax : public GPUOperation {
  private:
   absl::Status BindArguments();
   int3 GetGridSize() const;
+  Arguments args_;
   CLKernel kernel_;
   int3 work_group_size_ = int3(8, 4, 1);
 };
diff --git a/tensorflow/lite/delegates/gpu/cl/kernels/winograd.cc b/tensorflow/lite/delegates/gpu/cl/kernels/winograd.cc
index eeb95ebaff7..66687c40c6a 100644
--- a/tensorflow/lite/delegates/gpu/cl/kernels/winograd.cc
+++ b/tensorflow/lite/delegates/gpu/cl/kernels/winograd.cc
@@ -38,16 +38,6 @@ std::string GetWinograd4x4To36Code(
     const OperationDef& op_def,
     const std::vector<ElementwiseOperation*>& linked_operations,
     Arguments* args) {
-  TensorCodeGenerator src_tensor(
-      "src_data",
-      WHSBPoint{"src_size.x", "src_size.y", "src_size.z", "src_size.w"},
-      op_def.src_tensors[0]);
-  TensorCodeGenerator dst_tensor(
-      "dst_data",
-      WHSBPoint{"dst_size.x", "dst_size.y", "dst_size.z", "dst_size.w"},
-      op_def.dst_tensors[0]);
-
-  const std::string batch_id = op_def.IsBatchSupported() ? "batch_id" : "";
   std::string c = GetCommonDefines(op_def.precision);
 
   const auto src_tensor_type = op_def.src_tensors[0].storage_type;
@@ -80,23 +70,30 @@ std::string GetWinograd4x4To36Code(
   }
   c += "};\n";
 
+  std::string cl_type = accum_type == DataType::FLOAT16 ? "half" : "float";
+  auto src_desc = absl::make_unique<TensorDescriptor>(op_def.src_tensors[0]);
+  src_desc->SetStateVar("ACCUM_FLT", cl_type);
+  args->AddObjectRef("src_tensor", AccessType::READ, std::move(src_desc));
+  args->AddObjectRef(
+      "dst_tensor", AccessType::WRITE,
+      absl::make_unique<TensorDescriptor>(op_def.dst_tensors[0]));
   args->AddInt("padding_x");
   args->AddInt("padding_y");
   args->AddInt("tiles_total");
   args->AddInt("tiles_x");
 
+  std::string linked_args = GetArgsDeclaration(linked_operations);
+  if (linked_args[0] == ',') {
+    linked_args[0] = ' ';
+  }
   c += "__kernel void main_function(\n";
-  c += src_tensor.GetDeclaration(AccessType::READ);
-  c += GetArgsDeclaration(linked_operations);
-  c += dst_tensor.GetDeclaration(AccessType::WRITE) + ",\n";
-  c += "    int4 src_size,                              \n";
-  c += "    int4 dst_size,\n  ";
+  c += linked_args;
   c += "$0) {\n";
   c += "  int DST_X = get_global_id(0);\n";
   c += "  int DST_Y = get_global_id(1);\n";
   c += "  int DST_Z = get_global_id(2);\n";
-  c += "  if (DST_X >= args.tiles_total || DST_Y >= 6 || DST_Z >= dst_size.z) "
-       "{\n";
+  c += "  if (DST_X >= args.tiles_total || DST_Y >= 6 || DST_Z >= "
+       "args.dst_tensor.Slices()) {\n";
   c += "    return; \n";
   c += "  }\n";
   c += "  int tile_x = (DST_X % args.tiles_x) * 4;\n";
@@ -114,19 +111,16 @@ std::string GetWinograd4x4To36Code(
   c += "  bt_ar[5] = t1.y;\n";
   auto read_src = [&](const std::string& src, const std::string& xs) {
     if (is_image_buffer) {
-      c += "    ACCUM_FLT4 " + src + " = " +
-           src_tensor.ReadAsType(accum_type, "src_a_" + xs + " + offset") +
-           ";\n";
+      c += "    ACCUM_FLT4 " + src +
+           " = args.src_tensor.Read<ACCUM_FLT>(src_a_" + xs + " + offset);\n";
     } else if (is_buffer) {
-      c += "    ACCUM_FLT4 " + src + " = " +
-           src_tensor.ReadAsType(accum_type, "src_a_" + xs + " + offset") +
-           " * m" + xs + "_x;\n";
+      c += "    ACCUM_FLT4 " + src +
+           " = args.src_tensor.Read<ACCUM_FLT>(src_a_" + xs + " + offset) * m" +
+           xs + "_x;\n";
     } else {
-      c += "    ACCUM_FLT4 " + src + " = " +
-           src_tensor.ReadAsTypeWHSB(accum_type,
-                                     "tile_x + args.padding_x + " + xs, "yc",
-                                     "DST_Z", batch_id) +
-           ";\n";
+      c += "    ACCUM_FLT4 " + src +
+           " = args.src_tensor.Read<ACCUM_FLT>(tile_x + args.padding_x + " +
+           xs + ", yc, DST_Z);\n";
     }
   };
   if (is_buffer || is_image_buffer) {
@@ -134,14 +128,17 @@ std::string GetWinograd4x4To36Code(
       const std::string xs = std::to_string(x);
       c += "  int xc" + xs + " = tile_x + args.padding_x + " + xs + ";\n";
       c += "  ACCUM_FLT m" + xs + "_x = (ACCUM_FLT)(xc" + xs + " >= 0 && xc" +
-           xs + " < src_size.x);\n";
+           xs + " < args.src_tensor.Width());\n";
       c += "  bool inx" + xs + " = (xc" + xs + " >= 0 && xc" + xs +
-           " < src_size.x);\n";
-      c += "  xc" + xs + " = clamp(xc" + xs + ", 0, src_size.x - 1);\n";
-      c += "  " + src_tensor.GetAddressWHSB("src_a_" + xs, "xc" + xs, "0",
-                                            "DST_Z", batch_id);
+           " < args.src_tensor.Width());\n";
+      c += "  xc" + xs + " = clamp(xc" + xs +
+           ", 0, args.src_tensor.Width() - 1);\n";
+      c += "  args.src_tensor.GetAddress(src_a_" + xs + ", xc" + xs +
+           ", 0, DST_Z);\n";
       if (is_image_buffer) {
-        c += "  src_a_" + xs + " = select(-src_size.x * src_size.y, src_a_" +
+        c += "  src_a_" + xs +
+             " = select(-args.src_tensor.Width() * args.src_tensor.Height(), "
+             "src_a_" +
              xs + ", inx" + xs + ");\n";
       }
     }
@@ -149,8 +146,8 @@ std::string GetWinograd4x4To36Code(
   c += "  {\n";
   c += "    int yc = tile_y + args.padding_y;\n";
   if (is_buffer || is_image_buffer) {
-    c += "    bool iny = (yc >= 0 && yc < src_size.y);\n";
-    c += "    int offset = select(0, yc * src_size.x, iny);\n";
+    c += "    bool iny = (yc >= 0 && yc < args.src_tensor.Height());\n";
+    c += "    int offset = select(0, yc * args.src_tensor.Width(), iny);\n";
     c += "    ACCUM_FLT bt = bt_ar[0] * (ACCUM_FLT)(iny);\n";
   } else {
     c += "    ACCUM_FLT bt = bt_ar[0];\n";
@@ -167,8 +164,8 @@ std::string GetWinograd4x4To36Code(
     c += "  {\n";
     c += "    int yc = tile_y + args.padding_y + (" + ys + ");\n";
     if (is_buffer || is_image_buffer) {
-      c += "    bool iny = (yc >= 0 && yc < src_size.y);\n";
-      c += "    int offset = select(0, yc * src_size.x, iny);\n";
+      c += "    bool iny = (yc >= 0 && yc < args.src_tensor.Height());\n";
+      c += "    int offset = select(0, yc * args.src_tensor.Width(), iny);\n";
       c += "    ACCUM_FLT bt = bt_ar[" + ys + "] * (ACCUM_FLT)(iny);\n";
     } else {
       c += "    ACCUM_FLT bt = bt_ar[" + ys + "];\n";
@@ -185,14 +182,14 @@ std::string GetWinograd4x4To36Code(
   c += "  {\n";
   c += "    FLT4 r0 = TO_FLT4(I0 + Bt[2] * I2 + Bt[4] * I4);\n";
   c += PostProcess(linked_operations, context);
-  c += "    " + dst_tensor.WriteWHSB("r0", "DST_X", "DST_Y", "DST_Z", batch_id);
+  c += "    args.dst_tensor.Write(r0, DST_X, DST_Y, DST_Z);\n";
   c += "    DST_Y++;\n";
   c += "  }\n";
   c += "  {\n";
   c += "    FLT4 r0 = TO_FLT4(Bt[7] * I1 + Bt[8] * I2 + Bt[9] * I3 + Bt[10] * "
        "I4);\n";
   c += PostProcess(linked_operations, context);
-  c += "    " + dst_tensor.WriteWHSB("r0", "DST_X", "DST_Y", "DST_Z", batch_id);
+  c += "    args.dst_tensor.Write(r0, DST_X, DST_Y, DST_Z);\n";
   c += "    DST_Y++;\n";
   c += "  }\n";
   c += "  {\n";
@@ -200,7 +197,7 @@ std::string GetWinograd4x4To36Code(
        "* "
        "I4);\n";
   c += PostProcess(linked_operations, context);
-  c += "    " + dst_tensor.WriteWHSB("r0", "DST_X", "DST_Y", "DST_Z", batch_id);
+  c += "    args.dst_tensor.Write(r0, DST_X, DST_Y, DST_Z);\n";
   c += "    DST_Y++;\n";
   c += "  }\n";
   c += "  {\n";
@@ -208,7 +205,7 @@ std::string GetWinograd4x4To36Code(
        "* "
        "I4);\n";
   c += PostProcess(linked_operations, context);
-  c += "    " + dst_tensor.WriteWHSB("r0", "DST_X", "DST_Y", "DST_Z", batch_id);
+  c += "    args.dst_tensor.Write(r0, DST_X, DST_Y, DST_Z);\n";
   c += "    DST_Y++;\n";
   c += "  }\n";
   c += "  {\n";
@@ -216,13 +213,13 @@ std::string GetWinograd4x4To36Code(
        "* "
        "I4);\n";
   c += PostProcess(linked_operations, context);
-  c += "    " + dst_tensor.WriteWHSB("r0", "DST_X", "DST_Y", "DST_Z", batch_id);
+  c += "    args.dst_tensor.Write(r0, DST_X, DST_Y, DST_Z);\n";
   c += "    DST_Y++;\n";
   c += "  }\n";
   c += "  {\n";
   c += "    FLT4 r0 = TO_FLT4(Bt[31] * I1 + Bt[33] * I3 + I5);\n";
   c += PostProcess(linked_operations, context);
-  c += "    " + dst_tensor.WriteWHSB("r0", "DST_X", "DST_Y", "DST_Z", batch_id);
+  c += "    args.dst_tensor.Write(r0, DST_X, DST_Y, DST_Z);\n";
   c += "    DST_Y++;\n";
   c += "  }\n";
   c += "}\n";
@@ -443,16 +440,14 @@ absl::Status Winograd4x4To36::BindArguments() {
   const int tiles_y = DivideRoundUp(
       src_[0]->Height() + padding_.prepended.h + padding_.appended.h - 2, 4);
   const int tiles_total = tiles_x * tiles_y;
+  RETURN_IF_ERROR(args_.SetObjectRef("src_tensor", src_[0]));
+  RETURN_IF_ERROR(args_.SetObjectRef("dst_tensor", dst_[0]));
   RETURN_IF_ERROR(args_.SetInt("padding_x", -padding_.prepended.w));
   RETURN_IF_ERROR(args_.SetInt("padding_y", -padding_.prepended.h));
   RETURN_IF_ERROR(args_.SetInt("tiles_total", tiles_total));
   RETURN_IF_ERROR(args_.SetInt("tiles_x", tiles_x));
   kernel_.ResetBindingCounter();
-  RETURN_IF_ERROR(kernel_.SetMemoryAuto(src_[0]->GetMemoryPtr()));
   RETURN_IF_ERROR(BindArgs(&kernel_, linked_operations_));
-  RETURN_IF_ERROR(kernel_.SetMemoryAuto(dst_[0]->GetMemoryPtrForWriting()));
-  RETURN_IF_ERROR(kernel_.SetBytesAuto(src_[0]->GetWHSB()));
-  RETURN_IF_ERROR(kernel_.SetBytesAuto(dst_[0]->GetWHSB()));
   RETURN_IF_ERROR(args_.Bind(kernel_.kernel(), kernel_.GetBindingCounter()));
   return absl::OkStatus();
 }
diff --git a/tensorflow/lite/delegates/gpu/cl/linear_storage.cc b/tensorflow/lite/delegates/gpu/cl/linear_storage.cc
index 7edf83f57ff..84d91b9136e 100644
--- a/tensorflow/lite/delegates/gpu/cl/linear_storage.cc
+++ b/tensorflow/lite/delegates/gpu/cl/linear_storage.cc
@@ -44,7 +44,7 @@ GPUResources TensorLinearDescriptor::GetGPUResources(
 
 absl::Status TensorLinearDescriptor::PerformSelector(
     const std::string& selector, const std::vector<std::string>& args,
-    std::string* result) const {
+    const std::vector<std::string>& template_args, std::string* result) const {
   if (selector == "Length") {
     *result = "length";
     return absl::OkStatus();
diff --git a/tensorflow/lite/delegates/gpu/cl/linear_storage.h b/tensorflow/lite/delegates/gpu/cl/linear_storage.h
index 83c41e2c833..474c5652db2 100644
--- a/tensorflow/lite/delegates/gpu/cl/linear_storage.h
+++ b/tensorflow/lite/delegates/gpu/cl/linear_storage.h
@@ -57,6 +57,7 @@ struct TensorLinearDescriptor : public GPUObjectDescriptor {
 
   absl::Status PerformSelector(const std::string& selector,
                                const std::vector<std::string>& args,
+                               const std::vector<std::string>& template_args,
                                std::string* result) const override;
 
   GPUResources GetGPUResources(AccessType access_type) const override;
diff --git a/tensorflow/lite/delegates/gpu/cl/selectors/operation_selector.cc b/tensorflow/lite/delegates/gpu/cl/selectors/operation_selector.cc
index 3b7ae2f297e..1863cedb793 100644
--- a/tensorflow/lite/delegates/gpu/cl/selectors/operation_selector.cc
+++ b/tensorflow/lite/delegates/gpu/cl/selectors/operation_selector.cc
@@ -138,54 +138,55 @@ absl::Status GPUOperationFromNode(const CreationContext& creation_context,
   auto op_type = OperationTypeFromString(node.operation.type);
   switch (op_type) {
     case OperationType::ADD: {
-      auto attr = absl::any_cast<AddAttributes>(node.operation.attributes);
-      const float* scalar = absl::get_if<float>(&attr.param);
-      const auto* linear_tensor =
-          absl::get_if<tflite::gpu::Tensor<Linear, DataType::FLOAT32>>(
-              &attr.param);
-      const auto* hwc_tensor =
-          absl::get_if<tflite::gpu::Tensor<HWC, DataType::FLOAT32>>(
-              &attr.param);
-      if (scalar) {
-        ElementwiseOneRuntimeOneScalar operation =
-            CreateElementwiseOneRuntimeOneScalar(creation_context, op_def,
-                                                 op_type, *scalar);
-        *gpu_op = absl::make_unique<ElementwiseOneRuntimeOneScalar>(
-            std::move(operation));
-        return absl::OkStatus();
-      } else if (linear_tensor) {
-        ElementwiseTwoInput operation;
-        RETURN_IF_ERROR(CreateElementwiseTwoInput(
-            creation_context, op_def, op_type, *linear_tensor, &operation));
+      if (inputs.size() == 2 &&
+          (inputs[0]->tensor.shape.c == inputs[1]->tensor.shape.c ||
+           inputs[1]->tensor.shape.c == 1)) {
+        ElementwiseTwoInput operation =
+            CreateElementwiseTwoInput(op_def, op_type, inputs[1]->tensor.shape);
         *gpu_op = absl::make_unique<ElementwiseTwoInput>(std::move(operation));
         return absl::OkStatus();
-      } else if (hwc_tensor) {
-        ElementwiseTwoInput operation;
-        RETURN_IF_ERROR(CreateElementwiseTwoInput(
-            creation_context, op_def, op_type, *hwc_tensor, &operation));
-        *gpu_op = absl::make_unique<ElementwiseTwoInput>(std::move(operation));
+      } else if (inputs.size() >= 2) {
+        auto output = outputs[0];
+        std::vector<int> channels(inputs.size());
+        for (int i = 0; i < inputs.size(); ++i) {
+          channels[i] = inputs[i]->tensor.shape.c;
+        }
+        SelectAdd(op_def, channels, output->tensor.shape.c, gpu_op);
         return absl::OkStatus();
-      } else {
-        if (inputs.size() == 2 &&
-            (inputs[0]->tensor.shape.c == inputs[1]->tensor.shape.c ||
-             inputs[1]->tensor.shape.c == 1)) {
-          ElementwiseTwoInput operation = CreateElementwiseTwoInput(
-              op_def, op_type, inputs[1]->tensor.shape);
+      } else if (inputs.size() == 1 && node.operation.attributes.has_value()) {
+        auto attr = absl::any_cast<AddAttributes>(node.operation.attributes);
+        const float* scalar = absl::get_if<float>(&attr.param);
+        const auto* linear_tensor =
+            absl::get_if<tflite::gpu::Tensor<Linear, DataType::FLOAT32>>(
+                &attr.param);
+        const auto* hwc_tensor =
+            absl::get_if<tflite::gpu::Tensor<HWC, DataType::FLOAT32>>(
+                &attr.param);
+        if (scalar) {
+          ElementwiseOneRuntimeOneScalar operation =
+              CreateElementwiseOneRuntimeOneScalar(creation_context, op_def,
+                                                   op_type, *scalar);
+          *gpu_op = absl::make_unique<ElementwiseOneRuntimeOneScalar>(
+              std::move(operation));
+          return absl::OkStatus();
+        } else if (linear_tensor) {
+          ElementwiseTwoInput operation;
+          RETURN_IF_ERROR(CreateElementwiseTwoInput(
+              creation_context, op_def, op_type, *linear_tensor, &operation));
           *gpu_op =
               absl::make_unique<ElementwiseTwoInput>(std::move(operation));
           return absl::OkStatus();
-        } else if (inputs.size() >= 2) {
-          auto output = outputs[0];
-          std::vector<int> channels(inputs.size());
-          for (int i = 0; i < inputs.size(); ++i) {
-            channels[i] = inputs[i]->tensor.shape.c;
-          }
-          SelectAdd(op_def, channels, output->tensor.shape.c, gpu_op);
+        } else if (hwc_tensor) {
+          ElementwiseTwoInput operation;
+          RETURN_IF_ERROR(CreateElementwiseTwoInput(
+              creation_context, op_def, op_type, *hwc_tensor, &operation));
+          *gpu_op =
+              absl::make_unique<ElementwiseTwoInput>(std::move(operation));
           return absl::OkStatus();
         }
-        return absl::UnimplementedError(absl::StrCat(
-            "No support of ", node.operation.type, " with this parameters"));
       }
+      return absl::UnimplementedError(absl::StrCat(
+          "No support of ", node.operation.type, " with this parameters"));
     }
     case OperationType::CONCAT: {
       auto attr = absl::any_cast<ConcatAttributes>(node.operation.attributes);
@@ -280,45 +281,46 @@ absl::Status GPUOperationFromNode(const CreationContext& creation_context,
       return SelectMean(attr, op_def, gpu_op);
     }
     case OperationType::MUL: {
-      auto attr = absl::any_cast<MultiplyAttributes>(node.operation.attributes);
-      const float* scalar = absl::get_if<float>(&attr.param);
-      const auto* linear_tensor =
-          absl::get_if<tflite::gpu::Tensor<Linear, DataType::FLOAT32>>(
-              &attr.param);
-      const auto* hwc_tensor =
-          absl::get_if<tflite::gpu::Tensor<HWC, DataType::FLOAT32>>(
-              &attr.param);
-      if (scalar) {
-        ElementwiseOneRuntimeOneScalar operation =
-            CreateElementwiseOneRuntimeOneScalar(creation_context, op_def,
-                                                 op_type, *scalar);
-        *gpu_op = absl::make_unique<ElementwiseOneRuntimeOneScalar>(
-            std::move(operation));
-        return absl::OkStatus();
-      } else if (linear_tensor) {
-        ElementwiseTwoInput operation;
-        RETURN_IF_ERROR(CreateElementwiseTwoInput(
-            creation_context, op_def, op_type, *linear_tensor, &operation));
+      if (inputs.size() == 2) {
+        ElementwiseTwoInput operation =
+            CreateElementwiseTwoInput(op_def, op_type, inputs[1]->tensor.shape);
         *gpu_op = absl::make_unique<ElementwiseTwoInput>(std::move(operation));
         return absl::OkStatus();
-      } else if (hwc_tensor) {
-        ElementwiseTwoInput operation;
-        RETURN_IF_ERROR(CreateElementwiseTwoInput(
-            creation_context, op_def, op_type, *hwc_tensor, &operation));
-        *gpu_op = absl::make_unique<ElementwiseTwoInput>(std::move(operation));
-        return absl::OkStatus();
-      } else {
-        if (inputs.size() == 2) {
-          ElementwiseTwoInput operation = CreateElementwiseTwoInput(
-              op_def, op_type, inputs[1]->tensor.shape);
+      } else if (inputs.size() == 1 && node.operation.attributes.has_value()) {
+        auto attr =
+            absl::any_cast<MultiplyAttributes>(node.operation.attributes);
+        const float* scalar = absl::get_if<float>(&attr.param);
+        const auto* linear_tensor =
+            absl::get_if<tflite::gpu::Tensor<Linear, DataType::FLOAT32>>(
+                &attr.param);
+        const auto* hwc_tensor =
+            absl::get_if<tflite::gpu::Tensor<HWC, DataType::FLOAT32>>(
+                &attr.param);
+        if (scalar) {
+          ElementwiseOneRuntimeOneScalar operation =
+              CreateElementwiseOneRuntimeOneScalar(creation_context, op_def,
+                                                   op_type, *scalar);
+          *gpu_op = absl::make_unique<ElementwiseOneRuntimeOneScalar>(
+              std::move(operation));
+          return absl::OkStatus();
+        } else if (linear_tensor) {
+          ElementwiseTwoInput operation;
+          RETURN_IF_ERROR(CreateElementwiseTwoInput(
+              creation_context, op_def, op_type, *linear_tensor, &operation));
+          *gpu_op =
+              absl::make_unique<ElementwiseTwoInput>(std::move(operation));
+          return absl::OkStatus();
+        } else if (hwc_tensor) {
+          ElementwiseTwoInput operation;
+          RETURN_IF_ERROR(CreateElementwiseTwoInput(
+              creation_context, op_def, op_type, *hwc_tensor, &operation));
           *gpu_op =
               absl::make_unique<ElementwiseTwoInput>(std::move(operation));
           return absl::OkStatus();
-        } else {
-          return absl::UnimplementedError(absl::StrCat(
-              "No support of ", node.operation.type, " with this parameters"));
         }
       }
+      return absl::UnimplementedError(absl::StrCat(
+          "No support of ", node.operation.type, " with this parameters"));
     }
     case OperationType::PAD: {
       auto attr = absl::any_cast<PadAttributes>(node.operation.attributes);
@@ -399,46 +401,46 @@ absl::Status GPUOperationFromNode(const CreationContext& creation_context,
     case OperationType::POW:
     case OperationType::SQUARED_DIFF:
     case OperationType::SUB: {
-      auto attr =
-          absl::any_cast<ElementwiseAttributes>(node.operation.attributes);
-      const float* scalar = absl::get_if<float>(&attr.param);
-      const auto* linear_tensor =
-          absl::get_if<tflite::gpu::Tensor<Linear, DataType::FLOAT32>>(
-              &attr.param);
-      const auto* hwc_tensor =
-          absl::get_if<tflite::gpu::Tensor<HWC, DataType::FLOAT32>>(
-              &attr.param);
-      if (scalar) {
-        ElementwiseOneRuntimeOneScalar operation =
-            CreateElementwiseOneRuntimeOneScalar(creation_context, op_def,
-                                                 op_type, *scalar);
-        *gpu_op = absl::make_unique<ElementwiseOneRuntimeOneScalar>(
-            std::move(operation));
-        return absl::OkStatus();
-      } else if (linear_tensor) {
-        ElementwiseTwoInput operation;
-        RETURN_IF_ERROR(CreateElementwiseTwoInput(
-            creation_context, op_def, op_type, *linear_tensor, &operation));
+      if (inputs.size() == 2) {
+        ElementwiseTwoInput operation =
+            CreateElementwiseTwoInput(op_def, op_type, inputs[1]->tensor.shape);
         *gpu_op = absl::make_unique<ElementwiseTwoInput>(std::move(operation));
         return absl::OkStatus();
-      } else if (hwc_tensor) {
-        ElementwiseTwoInput operation;
-        RETURN_IF_ERROR(CreateElementwiseTwoInput(
-            creation_context, op_def, op_type, *hwc_tensor, &operation));
-        *gpu_op = absl::make_unique<ElementwiseTwoInput>(std::move(operation));
-        return absl::OkStatus();
-      } else {
-        if (inputs.size() == 2) {
-          ElementwiseTwoInput operation = CreateElementwiseTwoInput(
-              op_def, op_type, inputs[1]->tensor.shape);
+      } else if (inputs.size() == 1 && node.operation.attributes.has_value()) {
+        auto attr =
+            absl::any_cast<ElementwiseAttributes>(node.operation.attributes);
+        const float* scalar = absl::get_if<float>(&attr.param);
+        const auto* linear_tensor =
+            absl::get_if<tflite::gpu::Tensor<Linear, DataType::FLOAT32>>(
+                &attr.param);
+        const auto* hwc_tensor =
+            absl::get_if<tflite::gpu::Tensor<HWC, DataType::FLOAT32>>(
+                &attr.param);
+        if (scalar) {
+          ElementwiseOneRuntimeOneScalar operation =
+              CreateElementwiseOneRuntimeOneScalar(creation_context, op_def,
+                                                   op_type, *scalar);
+          *gpu_op = absl::make_unique<ElementwiseOneRuntimeOneScalar>(
+              std::move(operation));
+          return absl::OkStatus();
+        } else if (linear_tensor) {
+          ElementwiseTwoInput operation;
+          RETURN_IF_ERROR(CreateElementwiseTwoInput(
+              creation_context, op_def, op_type, *linear_tensor, &operation));
+          *gpu_op =
+              absl::make_unique<ElementwiseTwoInput>(std::move(operation));
+          return absl::OkStatus();
+        } else if (hwc_tensor) {
+          ElementwiseTwoInput operation;
+          RETURN_IF_ERROR(CreateElementwiseTwoInput(
+              creation_context, op_def, op_type, *hwc_tensor, &operation));
           *gpu_op =
               absl::make_unique<ElementwiseTwoInput>(std::move(operation));
           return absl::OkStatus();
-        } else {
-          return absl::UnimplementedError(absl::StrCat(
-              "No support of ", node.operation.type, " with this parameters"));
         }
       }
+      return absl::UnimplementedError(absl::StrCat(
+          "No support of ", node.operation.type, " with this parameters"));
     }
     default:
       return SelectDefault(creation_context, op_def, hints, inputs, outputs,
diff --git a/tensorflow/lite/delegates/gpu/cl/tensor.cc b/tensorflow/lite/delegates/gpu/cl/tensor.cc
index 053514056f0..f35d5554b1f 100644
--- a/tensorflow/lite/delegates/gpu/cl/tensor.cc
+++ b/tensorflow/lite/delegates/gpu/cl/tensor.cc
@@ -148,6 +148,7 @@ GPUResourcesWithValue Tensor::GetGPUResources(AccessType access_type) const {
   GPUResourcesWithValue resources;
   if (descriptor_.HasAxis(Axis::WIDTH)) {
     resources.ints.push_back({"width", Width()});
+    resources.ints.push_back({"width_batched", Width() * Batch()});
   }
   if (descriptor_.HasAxis(Axis::HEIGHT)) {
     resources.ints.push_back({"height", Height()});
diff --git a/tensorflow/lite/delegates/gpu/cl/tensor_type.cc b/tensorflow/lite/delegates/gpu/cl/tensor_type.cc
index 11e1ca2ca07..c685cba8f45 100644
--- a/tensorflow/lite/delegates/gpu/cl/tensor_type.cc
+++ b/tensorflow/lite/delegates/gpu/cl/tensor_type.cc
@@ -23,110 +23,6 @@ namespace tflite {
 namespace gpu {
 namespace cl {
 namespace {
-std::string GetGlobalAddressNoDeclarationWHS(const std::string& x,
-                                             const std::string& y,
-                                             const std::string& s,
-                                             TensorStorageType storage_type) {
-  switch (storage_type) {
-    case TensorStorageType::BUFFER:
-    case TensorStorageType::IMAGE_BUFFER:
-      return absl::Substitute("((($2) * height + ($1)) * width + ($0))", x, y,
-                              s);
-    case TensorStorageType::TEXTURE_2D:
-      return absl::Substitute("(int2)(($0), ($1) * slices + ($2))", x, y, s);
-    case TensorStorageType::SINGLE_TEXTURE_2D:
-      return absl::StrCat("(int2)(", x, ", ", y, ")");
-    case TensorStorageType::TEXTURE_ARRAY:
-    case TensorStorageType::TEXTURE_3D:
-      return absl::StrCat("(int4)(", x, ", ", y, ", ", s, ", 0)");
-    case TensorStorageType::UNKNOWN:
-      return "error";
-  }
-}
-
-std::string GetGlobalAddressNoDeclarationWHSB(const std::string& x,
-                                              const std::string& y,
-                                              const std::string& s,
-                                              const std::string& b,
-                                              TensorStorageType storage_type) {
-  switch (storage_type) {
-    case TensorStorageType::BUFFER:
-    case TensorStorageType::IMAGE_BUFFER:
-      return absl::Substitute(
-          "(((($3) * height + $2) * width + ($1)) * batch + ($0))", b, x, y, s);
-    case TensorStorageType::TEXTURE_2D:
-      return absl::Substitute(
-          "(int2)(($0) * batch + ($1), ($2) * slices + ($3))", x, b, y, s);
-    case TensorStorageType::SINGLE_TEXTURE_2D:
-      return absl::Substitute("(int2)(($0) * batch + ($1), ($2))", x, b, y);
-    case TensorStorageType::TEXTURE_ARRAY:
-    case TensorStorageType::TEXTURE_3D:
-      return absl::Substitute("(int4)(($0) * batch + ($1), ($2), ($3), 0)", x,
-                              b, y, s);
-    case TensorStorageType::UNKNOWN:
-      return "error";
-    default:
-      return "error";
-  }
-}
-
-std::string GetGlobalAddressNoDeclarationWHDS(const std::string& x,
-                                              const std::string& y,
-                                              const std::string& z,
-                                              const std::string& s,
-                                              TensorStorageType storage_type) {
-  switch (storage_type) {
-    case TensorStorageType::BUFFER:
-    case TensorStorageType::IMAGE_BUFFER:
-      return absl::Substitute(
-          "(((($3) * slices + ($2)) * height + ($1)) * width + ($0))", x, y, s,
-          z);
-    case TensorStorageType::TEXTURE_2D:
-      return absl::Substitute(
-          "(int2)(($0) * depth + ($1), ($2) * slices + ($3))", x, z, y, s);
-    case TensorStorageType::SINGLE_TEXTURE_2D:
-      return absl::Substitute("(int2)(($0) * depth + ($1), ($2))", x, z, y);
-    case TensorStorageType::TEXTURE_ARRAY:
-    case TensorStorageType::TEXTURE_3D:
-      return absl::Substitute("(int4)(($0), ($1), ($2) * slices + ($3), 0)", x,
-                              y, z, s);
-    case TensorStorageType::UNKNOWN:
-      return "error";
-  }
-}
-
-std::string GetGlobalAddressNoDeclarationWHDSB(const std::string& x,
-                                               const std::string& y,
-                                               const std::string& z,
-                                               const std::string& s,
-                                               const std::string& b,
-                                               TensorStorageType storage_type) {
-  switch (storage_type) {
-    case TensorStorageType::BUFFER:
-    case TensorStorageType::IMAGE_BUFFER:
-      return absl::Substitute(
-          "((((($4) * slices + ($3)) * height + $2) * width + ($1)) * batch + "
-          "($0))",
-          b, x, y, s, z);
-    case TensorStorageType::TEXTURE_2D:
-      return absl::Substitute(
-          "(int2)((($0) * batch + ($1)) * depth + ($2), ($3) * slices + ($4))",
-          x, b, z, y, s);
-    case TensorStorageType::SINGLE_TEXTURE_2D:
-      return absl::Substitute(
-          "(int2)((($0) * batch + ($1)) * depth + ($2), ($3))", x, b, z, y);
-    case TensorStorageType::TEXTURE_ARRAY:
-    case TensorStorageType::TEXTURE_3D:
-      return absl::Substitute(
-          "(int4)(($0) * batch + ($1), ($2), ($3) * slices + ($4), 0)", x, b, y,
-          z, s);
-    case TensorStorageType::UNKNOWN:
-      return "error";
-    default:
-      return "error";
-  }
-}
-
 std::string GetReadImageFromDataType(DataType data_type) {
   if (data_type == DataType::FLOAT32) {
     return "read_imagef";
@@ -172,6 +68,7 @@ GPUResources TensorDescriptor::GetGPUResources(AccessType access_type) const {
   GPUResources resources;
   if (HasAxis(Axis::WIDTH)) {
     resources.ints.push_back("width");
+    resources.ints.push_back("width_batched");
   }
   if (HasAxis(Axis::HEIGHT)) {
     resources.ints.push_back("height");
@@ -227,9 +124,13 @@ GPUResources TensorDescriptor::GetGPUResources(AccessType access_type) const {
 
 absl::Status TensorDescriptor::PerformSelector(
     const std::string& selector, const std::vector<std::string>& args,
-    std::string* result) const {
+    const std::vector<std::string>& template_args, std::string* result) const {
   if (selector == "Width") {
-    *result = "width";
+    if (IsBatchedWidth()) {
+      *result = "width_batched";
+    } else {
+      *result = "width";
+    }
     return absl::OkStatus();
   } else if (selector == "Height") {
     *result = "height";
@@ -255,9 +156,11 @@ absl::Status TensorDescriptor::PerformSelector(
     *result = "";
     return absl::OkStatus();
   } else if (selector == "Read") {
-    return PerformReadSelector(args, result);
+    return PerformReadSelector(args, template_args, result);
   } else if (selector == "Write") {
     return PerformWriteSelector(args, result);
+  } else if (selector == "GetAddress") {
+    return PerformGetAddressSelector(args, result);
   } else {
     return absl::NotFoundError(absl::StrCat(
         "TensorDescriptor don't have selector with name - ", selector));
@@ -265,7 +168,29 @@ absl::Status TensorDescriptor::PerformSelector(
 }
 
 absl::Status TensorDescriptor::PerformReadSelector(
-    const std::vector<std::string>& args, std::string* result) const {
+    const std::vector<std::string>& args,
+    const std::vector<std::string>& template_args, std::string* result) const {
+  DataType read_as_type = data_type;
+  if (!template_args.empty()) {
+    if (template_args.size() != 1) {
+      return absl::NotFoundError(
+          "Unrecognized Read selector template arguments.");
+    } else {
+      RETURN_IF_ERROR(
+          GetDataTypeFromTemplateArgs(template_args[0], &read_as_type));
+    }
+  }
+  if (args.size() == 1) {  // function overload for 1D linear types.
+    if (storage_type == TensorStorageType::BUFFER ||
+        storage_type == TensorStorageType::IMAGE_BUFFER) {
+      *result = Read(read_as_type, args[0]);
+      return absl::OkStatus();
+    } else {
+      return absl::InvalidArgumentError(
+          "Read selector with single argument can be used only with linear "
+          "storage types(BUFFER or IMAGE_BUFFER)");
+    }
+  }
   std::string xc;
   std::string yc;
   std::string zc;
@@ -276,24 +201,9 @@ absl::Status TensorDescriptor::PerformReadSelector(
     return absl::NotFoundError("Unrecognized Read selector");
   }
 
-  if (layout == Layout::HWC) {
-    *result = Read(GetGlobalAddressNoDeclarationWHS(xc, yc, sc, storage_type));
-    return absl::OkStatus();
-  } else if (layout == Layout::BHWC) {
-    *result =
-        Read(GetGlobalAddressNoDeclarationWHSB(xc, yc, sc, bc, storage_type));
-    return absl::OkStatus();
-  } else if (layout == Layout::HWDC) {
-    *result =
-        Read(GetGlobalAddressNoDeclarationWHDS(xc, yc, zc, sc, storage_type));
-    return absl::OkStatus();
-  } else if (layout == Layout::BHWDC) {
-    *result = Read(
-        GetGlobalAddressNoDeclarationWHDSB(xc, yc, zc, sc, bc, storage_type));
-    return absl::OkStatus();
-  } else {
-    return absl::NotFoundError("Unsupported layout");
-  }
+  *result =
+      Read(read_as_type, GetGlobalAddressNoDeclaration(xc, yc, zc, sc, bc));
+  return absl::OkStatus();
 }
 
 absl::Status TensorDescriptor::PerformWriteSelector(
@@ -307,29 +217,14 @@ absl::Status TensorDescriptor::PerformWriteSelector(
   if (args.size() < 2 || !parsed) {
     return absl::NotFoundError("Unrecognized Write selector");
   }
-
-  if (layout == Layout::HWC) {
-    *result = Write(args[0],
-                    GetGlobalAddressNoDeclarationWHS(xc, yc, sc, storage_type));
-    return absl::OkStatus();
-  } else if (layout == Layout::BHWC) {
-    *result = Write(args[0], GetGlobalAddressNoDeclarationWHSB(xc, yc, sc, bc,
-                                                               storage_type));
-    return absl::OkStatus();
-  } else if (layout == Layout::HWDC) {
-    *result = Write(args[0], GetGlobalAddressNoDeclarationWHDS(xc, yc, zc, sc,
-                                                               storage_type));
-    return absl::OkStatus();
-  } else if (layout == Layout::BHWDC) {
-    *result = Write(args[0], GetGlobalAddressNoDeclarationWHDSB(
-                                 xc, yc, zc, sc, bc, storage_type));
-    return absl::OkStatus();
-  } else {
-    return absl::NotFoundError("Unsupported layout");
-  }
+  *result = Write(args[0], GetGlobalAddressNoDeclaration(xc, yc, zc, sc, bc));
+  return absl::OkStatus();
 }
 
-std::string TensorDescriptor::Read(const std::string& global_address) const {
+std::string TensorDescriptor::Read(DataType read_as_type,
+                                   const std::string& global_address) const {
+  const std::string read_as =
+      read_as_type == DataType::FLOAT16 ? "read_imageh" : "read_imagef";
   std::string image_type;
   if (storage_type == TensorStorageType::TEXTURE_2D ||
       storage_type == TensorStorageType::SINGLE_TEXTURE_2D) {
@@ -341,16 +236,22 @@ std::string TensorDescriptor::Read(const std::string& global_address) const {
   }
   switch (storage_type) {
     case TensorStorageType::BUFFER:
-      return absl::StrCat("buffer[", global_address, "]");
+      if (read_as_type == data_type) {
+        return absl::StrCat("buffer[", global_address, "]");
+      } else {
+        const std::string conversion = read_as_type == DataType::FLOAT16
+                                           ? "convert_half4"
+                                           : "convert_float4";
+        return absl::StrCat(conversion, "(buffer[", global_address, "])");
+      }
     case TensorStorageType::TEXTURE_2D:
     case TensorStorageType::TEXTURE_3D:
     case TensorStorageType::SINGLE_TEXTURE_2D:
     case TensorStorageType::TEXTURE_ARRAY:
-      return absl::StrCat(GetReadImageFromDataType(data_type), "(", image_type,
-                          ", smp_none, ", global_address, ")");
+      return absl::StrCat(read_as, "(", image_type, ", smp_none, ",
+                          global_address, ")");
     case TensorStorageType::IMAGE_BUFFER:
-      return absl::StrCat(GetReadImageFromDataType(data_type),
-                          "(image_buffer, ", global_address, ")");
+      return absl::StrCat(read_as, "(image_buffer, ", global_address, ")");
     case TensorStorageType::UNKNOWN:
       return "";
   }
@@ -382,6 +283,185 @@ std::string TensorDescriptor::Write(const std::string& var_name,
   }
 }
 
+absl::Status TensorDescriptor::PerformGetAddressSelector(
+    const std::vector<std::string>& args, std::string* result) const {
+  std::string xc;
+  std::string yc;
+  std::string zc;
+  std::string sc;
+  std::string bc;
+  bool parsed = ParseCoordsFromArgs(args, 1, &xc, &yc, &zc, &sc, &bc);
+  if (args.size() < 3 || !parsed) {
+    return absl::NotFoundError("Unrecognized GetAddress selector");
+  }
+
+  *result = DeclareAddress(args[0],
+                           GetGlobalAddressNoDeclaration(xc, yc, zc, sc, bc));
+  return absl::OkStatus();
+}
+
+std::string TensorDescriptor::DeclareAddress(const std::string& var_name,
+                                             const std::string& address) const {
+  return absl::StrCat(StorageTypeToAddressType(), " ", var_name, " = ", address,
+                      ";");
+}
+
+std::string TensorDescriptor::StorageTypeToAddressType() const {
+  switch (storage_type) {
+    case TensorStorageType::BUFFER:
+    case TensorStorageType::IMAGE_BUFFER:
+      return "int";
+    case TensorStorageType::TEXTURE_2D:
+    case TensorStorageType::SINGLE_TEXTURE_2D:
+      return "int2";
+    case TensorStorageType::TEXTURE_ARRAY:
+    case TensorStorageType::TEXTURE_3D:
+      return "int4";
+    case TensorStorageType::UNKNOWN:
+      return "";
+  }
+}
+
+std::string TensorDescriptor::GetGlobalAddressNoDeclarationWHS(
+    const std::string& x, const std::string& y, const std::string& s) const {
+  switch (storage_type) {
+    case TensorStorageType::BUFFER:
+    case TensorStorageType::IMAGE_BUFFER: {
+      const std::string width = IsBatchedWidth() ? "width_batched" : "width";
+      return absl::Substitute("((($2) * height + ($1)) * $3 + ($0))", x, y, s,
+                              width);
+    }
+    case TensorStorageType::TEXTURE_2D:
+      return absl::Substitute("(int2)(($0), ($1) * slices + ($2))", x, y, s);
+    case TensorStorageType::SINGLE_TEXTURE_2D:
+      return absl::StrCat("(int2)(", x, ", ", y, ")");
+    case TensorStorageType::TEXTURE_ARRAY:
+    case TensorStorageType::TEXTURE_3D:
+      return absl::StrCat("(int4)(", x, ", ", y, ", ", s, ", 0)");
+    case TensorStorageType::UNKNOWN:
+      return "error";
+  }
+}
+
+std::string TensorDescriptor::GetGlobalAddressNoDeclarationWHSB(
+    const std::string& x, const std::string& y, const std::string& s,
+    const std::string& b) const {
+  switch (storage_type) {
+    case TensorStorageType::BUFFER:
+    case TensorStorageType::IMAGE_BUFFER:
+      return absl::Substitute(
+          "(((($3) * height + $2) * width + ($1)) * batch + ($0))", b, x, y, s);
+    case TensorStorageType::TEXTURE_2D:
+      return absl::Substitute(
+          "(int2)(($0) * batch + ($1), ($2) * slices + ($3))", x, b, y, s);
+    case TensorStorageType::SINGLE_TEXTURE_2D:
+      return absl::Substitute("(int2)(($0) * batch + ($1), ($2))", x, b, y);
+    case TensorStorageType::TEXTURE_ARRAY:
+    case TensorStorageType::TEXTURE_3D:
+      return absl::Substitute("(int4)(($0) * batch + ($1), ($2), ($3), 0)", x,
+                              b, y, s);
+    case TensorStorageType::UNKNOWN:
+      return "error";
+    default:
+      return "error";
+  }
+}
+
+std::string TensorDescriptor::GetGlobalAddressNoDeclarationWHDS(
+    const std::string& x, const std::string& y, const std::string& z,
+    const std::string& s) const {
+  switch (storage_type) {
+    case TensorStorageType::BUFFER:
+    case TensorStorageType::IMAGE_BUFFER: {
+      const std::string width = IsBatchedWidth() ? "width_batched" : "width";
+      return absl::Substitute(
+          "(((($3) * slices + ($2)) * height + ($1)) * $4 + ($0))", x, y, s, z,
+          width);
+    }
+    case TensorStorageType::TEXTURE_2D:
+      return absl::Substitute(
+          "(int2)(($0) * depth + ($1), ($2) * slices + ($3))", x, z, y, s);
+    case TensorStorageType::SINGLE_TEXTURE_2D:
+      return absl::Substitute("(int2)(($0) * depth + ($1), ($2))", x, z, y);
+    case TensorStorageType::TEXTURE_ARRAY:
+    case TensorStorageType::TEXTURE_3D:
+      return absl::Substitute("(int4)(($0), ($1), ($2) * slices + ($3), 0)", x,
+                              y, z, s);
+    case TensorStorageType::UNKNOWN:
+      return "error";
+  }
+}
+
+std::string TensorDescriptor::GetGlobalAddressNoDeclarationWHDSB(
+    const std::string& x, const std::string& y, const std::string& z,
+    const std::string& s, const std::string& b) const {
+  switch (storage_type) {
+    case TensorStorageType::BUFFER:
+    case TensorStorageType::IMAGE_BUFFER:
+      return absl::Substitute(
+          "((((($4) * slices + ($3)) * height + $2) * width + ($1)) * batch + "
+          "($0))",
+          b, x, y, s, z);
+    case TensorStorageType::TEXTURE_2D:
+      return absl::Substitute(
+          "(int2)((($0) * batch + ($1)) * depth + ($2), ($3) * slices + ($4))",
+          x, b, z, y, s);
+    case TensorStorageType::SINGLE_TEXTURE_2D:
+      return absl::Substitute(
+          "(int2)((($0) * batch + ($1)) * depth + ($2), ($3))", x, b, z, y);
+    case TensorStorageType::TEXTURE_ARRAY:
+    case TensorStorageType::TEXTURE_3D:
+      return absl::Substitute(
+          "(int4)(($0) * batch + ($1), ($2), ($3) * slices + ($4), 0)", x, b, y,
+          z, s);
+    case TensorStorageType::UNKNOWN:
+      return "error";
+    default:
+      return "error";
+  }
+}
+
+std::string TensorDescriptor::GetGlobalAddressNoDeclaration(
+    const std::string& xc, const std::string& yc, const std::string& zc,
+    const std::string& sc, const std::string& bc) const {
+  if (layout == Layout::HWC || (IsBatchedWidth() && layout == Layout::BHWC)) {
+    return GetGlobalAddressNoDeclarationWHS(xc, yc, sc);
+  } else if (layout == Layout::BHWC) {
+    return GetGlobalAddressNoDeclarationWHSB(xc, yc, sc, bc);
+  } else if (layout == Layout::HWDC ||
+             (IsBatchedWidth() && layout == Layout::BHWDC)) {
+    return GetGlobalAddressNoDeclarationWHDS(xc, yc, zc, sc);
+  } else if (layout == Layout::BHWDC) {
+    return GetGlobalAddressNoDeclarationWHDSB(xc, yc, zc, sc, bc);
+  } else {
+    return "Unsupported layout";
+  }
+}
+
+absl::Status TensorDescriptor::GetDataTypeFromTemplateArgs(
+    const std::string& template_arg, DataType* result) const {
+  std::string read_type = template_arg;
+  if (read_type == "FLT" || read_type == "ACCUM_FLT") {
+    auto it = state_vars_.find(read_type);
+    if (it == state_vars_.end()) {
+      return absl::UnavailableError(absl::StrCat(
+          "Read selector template argument ", read_type, " uninitialized."));
+    } else {
+      read_type = it->second;
+    }
+  }
+
+  if (read_type == "half") {
+    *result = DataType::FLOAT16;
+  } else if (read_type == "float") {
+    *result = DataType::FLOAT32;
+  } else {
+    return absl::NotFoundError(absl::StrCat(
+        "Unrecognized Read selector template argument - ", read_type));
+  }
+  return absl::OkStatus();
+}
+
 bool TensorDescriptor::HasAxis(Axis axis) const {
   if (axis == Axis::WIDTH || axis == Axis::HEIGHT || axis == Axis::CHANNELS) {
     return true;
@@ -426,7 +506,7 @@ bool TensorDescriptor::ParseCoordsFromArgs(const std::vector<std::string>& args,
       *sc = args[offset++];
     }
   }
-  if (HasAxis(Axis::BATCH)) {
+  if (HasAxis(Axis::BATCH) && !IsBatchedWidth()) {
     if (offset >= args.size()) {
       auto it = state_vars_.find("batch_id");
       if (it == state_vars_.end()) {
@@ -441,6 +521,11 @@ bool TensorDescriptor::ParseCoordsFromArgs(const std::vector<std::string>& args,
   return true;
 }
 
+bool TensorDescriptor::IsBatchedWidth() const {
+  auto it = state_vars_.find("BatchedWidth");
+  return it != state_vars_.end() && it->second == "true";
+}
+
 }  // namespace cl
 }  // namespace gpu
 }  // namespace tflite
diff --git a/tensorflow/lite/delegates/gpu/cl/tensor_type.h b/tensorflow/lite/delegates/gpu/cl/tensor_type.h
index 7d5ff888a85..30e9e4fa3fb 100644
--- a/tensorflow/lite/delegates/gpu/cl/tensor_type.h
+++ b/tensorflow/lite/delegates/gpu/cl/tensor_type.h
@@ -65,6 +65,7 @@ struct TensorDescriptor : public GPUObjectDescriptor {
 
   absl::Status PerformSelector(const std::string& selector,
                                const std::vector<std::string>& args,
+                               const std::vector<std::string>& template_args,
                                std::string* result) const override;
 
   GPUResources GetGPUResources(AccessType access_type) const override;
@@ -79,19 +80,56 @@ struct TensorDescriptor : public GPUObjectDescriptor {
       Layout::UNKNOWN;  // Supported layouts is HWC, BHWC, HWDC, BHWDC
 
  private:
-  absl::Status PerformReadSelector(const std::vector<std::string>& args,
-                                   std::string* result) const;
+  absl::Status PerformReadSelector(
+      const std::vector<std::string>& args,
+      const std::vector<std::string>& template_args, std::string* result) const;
+
+  absl::Status PerformGetAddressSelector(const std::vector<std::string>& args,
+                                         std::string* result) const;
+
+  std::string DeclareAddress(const std::string& var_name,
+                             const std::string& address) const;
+
+  std::string StorageTypeToAddressType() const;
 
   absl::Status PerformWriteSelector(const std::vector<std::string>& args,
                                     std::string* result) const;
 
-  std::string Read(const std::string& global_address) const;
+  std::string Read(DataType read_as_type,
+                   const std::string& global_address) const;
   std::string Write(const std::string& var_name,
                     const std::string& global_address) const;
 
+  absl::Status GetDataTypeFromTemplateArgs(const std::string& template_arg,
+                                           DataType* result) const;
+
+  std::string GetGlobalAddressNoDeclarationWHS(const std::string& x,
+                                               const std::string& y,
+                                               const std::string& s) const;
+  std::string GetGlobalAddressNoDeclarationWHSB(const std::string& x,
+                                                const std::string& y,
+                                                const std::string& s,
+                                                const std::string& b) const;
+  std::string GetGlobalAddressNoDeclarationWHDS(const std::string& x,
+                                                const std::string& y,
+                                                const std::string& z,
+                                                const std::string& s) const;
+  std::string GetGlobalAddressNoDeclarationWHDSB(const std::string& x,
+                                                 const std::string& y,
+                                                 const std::string& z,
+                                                 const std::string& s,
+                                                 const std::string& b) const;
+  std::string GetGlobalAddressNoDeclaration(const std::string& xc,
+                                            const std::string& yc,
+                                            const std::string& zc,
+                                            const std::string& sc,
+                                            const std::string& bc) const;
+
   bool ParseCoordsFromArgs(const std::vector<std::string>& args, int offset,
                            std::string* xc, std::string* yc, std::string* zc,
                            std::string* sc, std::string* bc) const;
+
+  bool IsBatchedWidth() const;
 };
 
 std::string ToString(TensorStorageType type);
diff --git a/tensorflow/lite/delegates/gpu/common/model_builder.cc b/tensorflow/lite/delegates/gpu/common/model_builder.cc
index 9cb738cf935..dc671a47691 100644
--- a/tensorflow/lite/delegates/gpu/common/model_builder.cc
+++ b/tensorflow/lite/delegates/gpu/common/model_builder.cc
@@ -345,9 +345,15 @@ absl::Status ParseInputsWithConstTensor(Node* node, ObjectReader* reader,
       RETURN_IF_ERROR(reader->ReadTensor(constant_tensor, &tensor));
       *tensor_or_scalar = tensor.data[0];
     } else {
-      Tensor<Linear, DataType::FLOAT32> tensor;
-      RETURN_IF_ERROR(reader->ReadTensor(constant_tensor, &tensor));
-      *tensor_or_scalar = std::move(tensor);
+      if (CheckIfLinearConvertible(constant_dims).ok()) {
+        Tensor<Linear, DataType::FLOAT32> tensor;
+        RETURN_IF_ERROR(reader->ReadTensor(constant_tensor, &tensor));
+        *tensor_or_scalar = std::move(tensor);
+      } else {
+        Tensor<HWC, DataType::FLOAT32> tensor;
+        RETURN_IF_ERROR(reader->ReadTensor(constant_tensor, &tensor));
+        *tensor_or_scalar = std::move(tensor);
+      }
     }
   }
   return absl::OkStatus();
@@ -363,12 +369,6 @@ class AddOperationParser : public TFLiteOperationParser {
       return absl::UnimplementedError("ADD requires two input tensors.");
     }
     // TODO(eignasheva): Add shapes check.
-    for (int i = 0; i < 2; i++) {
-      auto input = tflite::GetInput(context, tflite_node, i);
-      if (IsConstantTensor(input) && input->dims->size > 0) {
-        RETURN_IF_ERROR(CheckIfLinearConvertible(input->dims));
-      }
-    }
 
     const TfLiteAddParams* tf_options;
     return RetrieveBuiltinData(tflite_node, &tf_options);
@@ -1048,29 +1048,45 @@ class LSTMOperationParser : public TFLiteOperationParser {
                            const TfLiteNode* tflite_node,
                            const TfLiteRegistration* registration) final {
     RETURN_IF_ERROR(CheckMaxSupportedOpVersion(registration, 2));
-    // TODO(eignasheva): Fix bad check.
-    // RETURN_IF_ERROR(CheckInputsOutputs(context, tflite_node,
-    // /*runtime_inputs=*/5,
-    //                                    /*outputs=*/4));
     const TfLiteLSTMParams* tf_options;
     RETURN_IF_ERROR(RetrieveBuiltinData(tflite_node, &tf_options));
-    RETURN_IF_ERROR(CheckParameters(tf_options));
-    return absl::OkStatus();
+    switch (tf_options->kernel_type) {
+      case kTfLiteLSTMFullKernel:
+        // TODO(b/157166356): Add check for input/output tensor counts.
+        return CheckFullParameters(tf_options);
+      case kTfLiteLSTMBasicKernel:
+        RETURN_IF_ERROR(
+            CheckInputsConstsOutputs(context, tflite_node, /*runtime_inputs=*/3,
+                                     /*const_inputs=*/2, /*outputs=*/4));
+        return CheckBasicParameters(tf_options);
+    }
   }
 
   absl::Status Parse(const TfLiteNode* tflite_node,
                      const TfLiteRegistration* registration,
                      GraphFloat32* graph, ObjectReader* reader) final {
+    const TfLiteLSTMParams* tf_options;
+    RETURN_IF_ERROR(RetrieveBuiltinData(tflite_node, &tf_options));
+    switch (tf_options->kernel_type) {
+      case kTfLiteLSTMFullKernel:
+        return ParseFull(tflite_node, registration, graph, reader, tf_options);
+      case kTfLiteLSTMBasicKernel:
+        return ParseBasic(tflite_node, registration, graph, reader, tf_options);
+    }
+  }
+
+ private:
+  absl::Status ParseBasic(const TfLiteNode* tflite_node,
+                          const TfLiteRegistration* registration,
+                          GraphFloat32* graph, ObjectReader* reader,
+                          const TfLiteLSTMParams* tf_options) {
     if (tflite_node->inputs->size != 5) {
       return absl::InvalidArgumentError("LSTM should have 5 input tensors");
     }
     if (tflite_node->outputs->size != 4) {
       return absl::InvalidArgumentError("LSTM should have 4 output tensors");
     }
-
-    const TfLiteLSTMParams* tf_options;
-    RETURN_IF_ERROR(RetrieveBuiltinData(tflite_node, &tf_options));
-    RETURN_IF_ERROR(CheckParameters(tf_options));
+    RETURN_IF_ERROR(CheckBasicParameters(tf_options));
 
     Node* concat_node = graph->NewNode();
     concat_node->operation.type = ToString(OperationType::CONCAT);
@@ -1114,13 +1130,7 @@ class LSTMOperationParser : public TFLiteOperationParser {
     return absl::OkStatus();
   }
 
- private:
-  absl::Status CheckParameters(const TfLiteLSTMParams* tf_options) {
-    if (tf_options->kernel_type !=
-        TfLiteLSTMKernelType::kTfLiteLSTMBasicKernel) {
-      return absl::UnimplementedError(
-          "Only kTfLiteLSTMBasicKernel is supported.");
-    }
+  absl::Status CheckBasicParameters(const TfLiteLSTMParams* tf_options) {
     if (tf_options->activation != kTfLiteActTanh) {
       return absl::UnimplementedError("Only TANH activation is supported.");
     }
@@ -1132,6 +1142,19 @@ class LSTMOperationParser : public TFLiteOperationParser {
     }
     return absl::OkStatus();
   }
+
+  absl::Status ParseFull(const TfLiteNode* tflite_node,
+                         const TfLiteRegistration* registration,
+                         GraphFloat32* graph, ObjectReader* reader,
+                         const TfLiteLSTMParams* tf_options) {
+    return absl::UnimplementedError(
+        "Full LSTM support is not yet implemented.");
+  }
+
+  absl::Status CheckFullParameters(const TfLiteLSTMParams* tf_options) {
+    return absl::UnimplementedError(
+        "Full LSTM support is not yet implemented.");
+  }
 };
 
 class MulOperationParser : public TFLiteOperationParser {
@@ -1143,6 +1166,30 @@ class MulOperationParser : public TFLiteOperationParser {
     if (tflite_node->inputs->size != 2) {
       return absl::UnimplementedError("MUL requires two input tensors.");
     }
+    auto input0 = tflite::GetInput(context, tflite_node, 0);
+    auto input1 = tflite::GetInput(context, tflite_node, 1);
+    if (input0->dims->size == input1->dims->size) {
+      // this code checks that at least one input of Mul not smaller in all
+      // dimensions. Sometimes Mul used for matrix-vector multiplication that we
+      // currently don't support. For example input0 HWC(1, 256, 1), input1
+      // HWC(1, 1, 256) -> output HWC (1, 256, 256). In this case it can be
+      // replaced with Convolution operation.
+      bool first_has_smaller_dim = false;
+      bool second_has_smaller_dim = false;
+      for (int i = 0; i < input0->dims->size; ++i) {
+        if (input0->dims->data[i] < input1->dims->data[i]) {
+          first_has_smaller_dim = true;
+        }
+        if (input1->dims->data[i] < input0->dims->data[i]) {
+          second_has_smaller_dim = true;
+        }
+      }
+      if (first_has_smaller_dim && second_has_smaller_dim) {
+        return absl::UnimplementedError(
+            "MUL requires one tensor that not less than second in all "
+            "dimensions.");
+      }
+    }
     const TfLiteMulParams* tf_options;
     RETURN_IF_ERROR(RetrieveBuiltinData(tflite_node, &tf_options));
     return IsActivationSupported(tf_options->activation);
diff --git a/tensorflow/lite/delegates/gpu/common/model_builder_test.cc b/tensorflow/lite/delegates/gpu/common/model_builder_test.cc
index f0525e5e2c9..c5ee71b3f3f 100644
--- a/tensorflow/lite/delegates/gpu/common/model_builder_test.cc
+++ b/tensorflow/lite/delegates/gpu/common/model_builder_test.cc
@@ -250,7 +250,7 @@ class InterpreterFp16 : public DelegatedInterpreter {
 InterpreterFp16* interpreter_fp16_add_op =
     new InterpreterFp16(kTfLiteBuiltinAdd);
 
-TEST(ModelBuilderTest, GetOpsToReplacePrunesFp16DequantizeNodes) {
+TEST(ModelBuilderTest, GetOpsToReplaceAcceptsFp16DequantizeNodes) {
   // Before pruning, the graph has three nodes:
   //
   //   t0 (FP16) -> DequantNode -> t1 (FP32) -> Add -> t4
@@ -283,14 +283,15 @@ TEST(ModelBuilderTest, GetOpsToReplacePrunesFp16DequantizeNodes) {
   context->PreviewDelegatePartitioning =
       [](struct TfLiteContext* context, const TfLiteIntArray* nodes_to_replace,
          TfLiteDelegateParams** partition_params_array, int* num_partitions) {
+        // The partitioner should accept only the Add op initially.
+        EXPECT_EQ(nodes_to_replace->size, 1);
+        // Single partition output.
         auto params = interpreter_fp16_add_op->add_delegate_params();
-        params->nodes_to_replace = TfLiteIntArrayCreate(3);
-        params->nodes_to_replace->data[0] = 0;
-        params->nodes_to_replace->data[1] = 1;
-        params->nodes_to_replace->data[2] = 2;
+        params->nodes_to_replace = TfLiteIntArrayCreate(1);
+        params->nodes_to_replace->data[0] = 2;
         params->input_tensors = TfLiteIntArrayCreate(2);
-        params->input_tensors->data[0] = 0;
-        params->input_tensors->data[1] = 2;
+        params->input_tensors->data[0] = 1;
+        params->input_tensors->data[1] = 3;
         params->output_tensors = TfLiteIntArrayCreate(1);
         params->output_tensors->data[0] = 4;
 
@@ -301,11 +302,13 @@ TEST(ModelBuilderTest, GetOpsToReplacePrunesFp16DequantizeNodes) {
 
   TfLiteIntArray* ops_to_replace = GetOpsToReplace(context);
 
-  // Replace all nodes.
+  // The Dequant nodes are added to ops_to_replace as a post-processing step by
+  // the FP16GraphPartitioner. ADD is delegated with its inputs pointing to the
+  // FP16 inputs.
   EXPECT_EQ(ops_to_replace->size, 3);
   TfLiteNode* node = nullptr;
   TfLiteRegistration* registration = nullptr;
-  context->GetNodeAndRegistration(context, ops_to_replace->data[2], &node,
+  context->GetNodeAndRegistration(context, ops_to_replace->data[0], &node,
                                   &registration);
   EXPECT_EQ(context->tensors[node->inputs->data[0]].type,
             TfLiteType::kTfLiteFloat16);
@@ -317,14 +320,14 @@ TEST(ModelBuilderTest, GetOpsToReplacePrunesFp16DequantizeNodes) {
 InterpreterFp16* interpreter_fp16_gt_op =
     new InterpreterFp16(kTfLiteBuiltinGreater);
 
-TEST(ModelBuilderTest, GetOpsToReplaceKeepsFp16DequantizeNodes) {
+TEST(ModelBuilderTest, GetOpsToReplaceRejectsFp16DequantizeNodes) {
   // Before pruning, the graph has three nodes:
   //
   //   t0 (FP16) -> DequantNode -> t1 (FP32) -> Greater Op -> t4
   //   t2 (FP16) -> DequantNode -> t3 (FP32) --/
   //
-  // Because there is no GPU equivalent for the Greater op, we don't prune
-  // the Dequantize nodes.
+  // Because there is no GPU equivalent for the Greater op, we don't choose any
+  // nodes.
 
   TfLiteContext* context = interpreter_fp16_gt_op->context();
   // These functions are meant to be called inside delegates. Swap out
@@ -346,26 +349,10 @@ TEST(ModelBuilderTest, GetOpsToReplaceKeepsFp16DequantizeNodes) {
   context->PreviewDelegatePartitioning =
       [](struct TfLiteContext* context, const TfLiteIntArray* nodes_to_replace,
          TfLiteDelegateParams** partition_params_array, int* num_partitions) {
-        auto params = interpreter_fp16_gt_op->add_delegate_params();
-        // First partition for DequantNode (t0->t1)
-        params->nodes_to_replace = TfLiteIntArrayCreate(1);
-        params->nodes_to_replace->data[0] = 0;
-        params->input_tensors = TfLiteIntArrayCreate(1);
-        params->input_tensors->data[0] = 0;
-        params->output_tensors = TfLiteIntArrayCreate(1);
-        params->output_tensors->data[0] = 1;
-
-        // Second partition for DequantNode (t2->t3)
-        params = interpreter_fp16_gt_op->add_delegate_params();
-        params->nodes_to_replace = TfLiteIntArrayCreate(1);
-        params->nodes_to_replace->data[0] = 0;
-        params->input_tensors = TfLiteIntArrayCreate(1);
-        params->input_tensors->data[0] = 0;
-        params->output_tensors = TfLiteIntArrayCreate(1);
-        params->output_tensors->data[0] = 1;
-
-        *partition_params_array = interpreter_fp16_gt_op->delegate_params();
-        *num_partitions = interpreter_fp16_gt_op->num_delegate_params();
+        // No selected nodes.
+        EXPECT_EQ(nodes_to_replace->size, 0);
+        *partition_params_array = nullptr;
+        *num_partitions = 0;
         return kTfLiteOk;
       };
 
@@ -685,7 +672,7 @@ TEST(ModelBuilderTest, GetOpsToReplaceMultiplePartitions) {
 
 class InterpreterMultiNode : public DelegatedInterpreter {
  public:
-  explicit InterpreterMultiNode(bool add_op_first = true)
+  explicit InterpreterMultiNode(bool both_ops_supported = true)
       : DelegatedInterpreter(5) {
     void* builtin_data = malloc(sizeof(int));
     EXPECT_EQ(interpreter_.AddTensors(8), kTfLiteOk);
@@ -707,8 +694,8 @@ class InterpreterMultiNode : public DelegatedInterpreter {
                 kTfLiteOk);
     }
 
-    if (add_op_first) {
-      // Add the ADD op node that GPU delegate supports.
+    if (both_ops_supported) {
+      // Add 2 ADD ops.
       const TfLiteRegistration reg_add0 = {
           [](TfLiteContext* context, const char* buffer, size_t length) {
             return reinterpret_cast<void*>(new int(1));
@@ -727,8 +714,7 @@ class InterpreterMultiNode : public DelegatedInterpreter {
                     /*registration=*/&reg_add0),
                 kTfLiteOk);
 
-      // Add the GREATER op node that GPU delegate doesn't support.
-      const TfLiteRegistration reg_greater = {
+      const TfLiteRegistration reg_add1 = {
           [](TfLiteContext* context, const char* buffer, size_t length) {
             return reinterpret_cast<void*>(new int(1));
           },
@@ -738,12 +724,12 @@ class InterpreterMultiNode : public DelegatedInterpreter {
           nullptr,
           nullptr,
           nullptr,
-          kTfLiteBuiltinGreater};
+          kTfLiteBuiltinAdd};
       EXPECT_EQ(interpreter_.AddNodeWithParameters(
                     /*inputs=*/{3, 4}, /*outputs=*/{6}, /*init_data=*/nullptr,
                     /*init_data_size=*/0,
                     /*builtin_data=*/builtin_data,
-                    /*registration=*/&reg_greater),
+                    /*registration=*/&reg_add1),
                 kTfLiteOk);
     } else {
       // Add the GREATER op node that GPU delegate doesn't support.
@@ -828,19 +814,19 @@ class InterpreterMultiNode : public DelegatedInterpreter {
   }
 };
 
-InterpreterMultiNode* interpreter_mn = new InterpreterMultiNode();
+InterpreterMultiNode* interpreter_mn =
+    new InterpreterMultiNode(/*both_ops_supported*/ false);
 
-TEST(ModelBuilderTest, GetOpsToReplaceSelectsCorrectDequantsAddFirst) {
+TEST(ModelBuilderTest, GetOpsToReplaceSelectsCorrectFp16Nodes_SinglePartition) {
   // A graph with three Dequant nodes feeding two ops, 'Add' and 'Greater'.
   // 'Add' can be replaced by the GPU delegate, but 'Greater' can not.
-  //   t0 (FP16) --> Dequant(0) --> t3 (FP32) --> Greater(4) -> t6
+  //   t0 (FP16) --> Dequant(0) --> t3 (FP32) --> Greater(3) -> t6
   //   t1 (FP16) --> Dequant(1) --> t4 (FP32) --/
   //                                          --\
-  //   t3 (FP16) --> Dequant(2) --> t5 (FP32) --> Add(3) -> t7
+  //   t3 (FP16) --> Dequant(2) --> t5 (FP32) --> Add(4) -> t7
   //
-  //  OpsToReplace should replace the 'Add' op and the Dequant outputing
-  //  t5, but leave the other Dequant nodes because 'Greater' must run
-  //  on the CPU.
+  //  OpsToReplace should accept 'Add' & the Dequant nodes that only output to
+  //  it (in this case, Dequant(2)).
   TfLiteContext* context = interpreter_mn->context();
 
   // These functions are meant to be called inside delegates. Swap out
@@ -861,33 +847,16 @@ TEST(ModelBuilderTest, GetOpsToReplaceSelectsCorrectDequantsAddFirst) {
   context->PreviewDelegatePartitioning =
       [](struct TfLiteContext* context, const TfLiteIntArray* nodes_to_replace,
          TfLiteDelegateParams** partition_params_array, int* num_partitions) {
+        // The FP16GraphPartitioner should only mark the ADD op as accepted.
+        EXPECT_EQ(nodes_to_replace->size, 1);
+        EXPECT_EQ(nodes_to_replace->data[0], 4);
+        // Single partition.
         auto params = interpreter_mn->add_delegate_params();
-        // First partition for DequantNode(0)
         params->nodes_to_replace = TfLiteIntArrayCreate(1);
-        params->nodes_to_replace->data[0] = 0;
-        params->input_tensors = TfLiteIntArrayCreate(1);
-        params->input_tensors->data[0] = 0;
-        params->output_tensors = TfLiteIntArrayCreate(1);
-        params->output_tensors->data[0] = 3;
-
-        // Second partition for DequantNode(1)
-        params = interpreter_mn->add_delegate_params();
-        params->nodes_to_replace = TfLiteIntArrayCreate(1);
-        params->nodes_to_replace->data[0] = 1;
-        params->input_tensors = TfLiteIntArrayCreate(1);
-        params->input_tensors->data[0] = 1;
-        params->output_tensors = TfLiteIntArrayCreate(1);
-        params->output_tensors->data[0] = 4;
-
-        // Third partition for DequantNode(1), DequantNode(2), ADD(3)
-        params = interpreter_mn->add_delegate_params();
-        params->nodes_to_replace = TfLiteIntArrayCreate(3);
-        params->nodes_to_replace->data[0] = 1;
-        params->nodes_to_replace->data[1] = 2;
-        params->nodes_to_replace->data[2] = 3;
+        params->nodes_to_replace->data[0] = 4;
         params->input_tensors = TfLiteIntArrayCreate(2);
         params->input_tensors->data[0] = 1;
-        params->input_tensors->data[0] = 3;
+        params->input_tensors->data[1] = 3;
         params->output_tensors = TfLiteIntArrayCreate(1);
         params->output_tensors->data[0] = 7;
 
@@ -898,16 +867,16 @@ TEST(ModelBuilderTest, GetOpsToReplaceSelectsCorrectDequantsAddFirst) {
 
   TfLiteIntArray* ops_to_replace = GetOpsToReplace(context);
 
+  // Post-PreviewDelegatePartitioning, the partitioner will add Dequant(2) to
+  // ops_to_replace, since it only outputs to a delegated node.
   EXPECT_EQ(ops_to_replace->size, 2);
-  // Op at index 2 is the Dequant op (t3 -> t5).
-  EXPECT_EQ(ops_to_replace->data[0], 2);
-  // Op at index 3 is the Add op.
-  EXPECT_EQ(ops_to_replace->data[1], 3);
-
+  // Op at index 4 is the Add op.
+  EXPECT_EQ(ops_to_replace->data[0], 4);
+  EXPECT_EQ(ops_to_replace->data[1], 2);
+  // Verify that Add op has fp16 inputs.
   TfLiteNode* node = nullptr;
   TfLiteRegistration* registration = nullptr;
-  // Verify that Add op has fp16 inputs.
-  context->GetNodeAndRegistration(context, ops_to_replace->data[1], &node,
+  context->GetNodeAndRegistration(context, ops_to_replace->data[0], &node,
                                   &registration);
   EXPECT_EQ(context->tensors[node->inputs->data[0]].type,
             TfLiteType::kTfLiteFloat16);
@@ -917,21 +886,18 @@ TEST(ModelBuilderTest, GetOpsToReplaceSelectsCorrectDequantsAddFirst) {
 }
 
 InterpreterMultiNode* interpreter_mn2 =
-    new InterpreterMultiNode(/*add_op_first=*/false);
-TEST(ModelBuilderTest, GetOpsToReplaceSelectsCorrectDequantsGreaterFirst) {
-  // A graph with three Dequant nodes feeding two ops, 'Add' and 'Greater'.
-  // 'Add' can be replaced by the GPU delegate, but 'Greater' can not.
-  //   t0 (FP16) --> Dequant(0) --> t3 (FP32) --> Greater(3) -> t6
+    new InterpreterMultiNode(/*both_ops_supported*/ true);
+TEST(ModelBuilderTest,
+     GetOpsToReplaceSelectsCorrectFp16Nodes_MultiplePartitions) {
+  // A graph with three Dequant nodes feeding two Add ops.
+  //   t0 (FP16) --> Dequant(0) --> t3 (FP32) --> Add(3) -> t6
   //   t1 (FP16) --> Dequant(1) --> t4 (FP32) --/
   //                                          --\
   //   t3 (FP16) --> Dequant(2) --> t5 (FP32) --> Add(4) -> t7
   //
-  // Note: the graph dependency is exactly same w/ that in
-  // GetOpsToReplaceSelectsCorrectDequantsAddFirst, but the unsupported
-  // 'Greater' op appears first in the execution plan. Despite this,
-  // OpsToReplace should still replace the 'Add' op and the Dequant outputing
-  //  t5, but leave the other Dequant nodes because 'Greater' must run
-  //  on the CPU.
+  // In this test case, we purposely partition Add(3) & Add(4) into different
+  // partitions, to check if Dequant nodes that output *only* to the first
+  // partition nodes are accepted.
 
   TfLiteContext* context = interpreter_mn2->context();
 
@@ -954,33 +920,29 @@ TEST(ModelBuilderTest, GetOpsToReplaceSelectsCorrectDequantsGreaterFirst) {
   context->PreviewDelegatePartitioning =
       [](struct TfLiteContext* context, const TfLiteIntArray* nodes_to_replace,
          TfLiteDelegateParams** partition_params_array, int* num_partitions) {
+        // The FP16GraphPartitioner should only mark both ADD ops as accepted.
+        EXPECT_EQ(nodes_to_replace->size, 2);
+        EXPECT_EQ(nodes_to_replace->data[0], 3);
+        EXPECT_EQ(nodes_to_replace->data[1], 4);
+        // Technically, both ADD ops should end up in the same partition.
+        // But we put them in different partitions to test post-processing with
+        // DEQUANTIZE nodes.
+        // First partition with Add(3).
         auto params = interpreter_mn2->add_delegate_params();
-        // First partition for DequantNode(0)
         params->nodes_to_replace = TfLiteIntArrayCreate(1);
-        params->nodes_to_replace->data[0] = 0;
-        params->input_tensors = TfLiteIntArrayCreate(1);
-        params->input_tensors->data[0] = 0;
-        params->output_tensors = TfLiteIntArrayCreate(1);
-        params->output_tensors->data[0] = 3;
-
-        // Second partition for DequantNode(1)
-        params = interpreter_mn2->add_delegate_params();
-        params->nodes_to_replace = TfLiteIntArrayCreate(1);
-        params->nodes_to_replace->data[0] = 1;
-        params->input_tensors = TfLiteIntArrayCreate(1);
-        params->input_tensors->data[0] = 1;
-        params->output_tensors = TfLiteIntArrayCreate(1);
-        params->output_tensors->data[0] = 4;
-
-        // Third partition for DequantNode(1), DequantNode(2), ADD(4)
-        params = interpreter_mn2->add_delegate_params();
-        params->nodes_to_replace = TfLiteIntArrayCreate(3);
-        params->nodes_to_replace->data[0] = 1;
-        params->nodes_to_replace->data[1] = 2;
-        params->nodes_to_replace->data[2] = 4;
+        params->nodes_to_replace->data[0] = 3;
         params->input_tensors = TfLiteIntArrayCreate(2);
-        params->input_tensors->data[0] = 1;
         params->input_tensors->data[0] = 3;
+        params->input_tensors->data[1] = 4;
+        params->output_tensors = TfLiteIntArrayCreate(1);
+        params->output_tensors->data[0] = 6;
+        // Second partition with Add(4).
+        params = interpreter_mn2->add_delegate_params();
+        params->nodes_to_replace = TfLiteIntArrayCreate(1);
+        params->nodes_to_replace->data[0] = 4;
+        params->input_tensors = TfLiteIntArrayCreate(2);
+        params->input_tensors->data[0] = 4;
+        params->input_tensors->data[1] = 5;
         params->output_tensors = TfLiteIntArrayCreate(1);
         params->output_tensors->data[0] = 7;
 
@@ -989,23 +951,32 @@ TEST(ModelBuilderTest, GetOpsToReplaceSelectsCorrectDequantsGreaterFirst) {
         return kTfLiteOk;
       };
 
-  TfLiteIntArray* ops_to_replace = GetOpsToReplace(context);
+  TfLiteIntArray* ops_to_replace = GetOpsToReplace(
+      context, /*allow_quant_ops*/ false, /*max_delegated_partitions*/ 2);
 
-  EXPECT_EQ(ops_to_replace->size, 2);
-  // Op at index 2 is the Dequant op (t3 -> t5).
-  EXPECT_EQ(ops_to_replace->data[0], 2);
-  // Op at index 4 is the Add op.
-  EXPECT_EQ(ops_to_replace->data[1], 4);
+  // Three ops should be selected:
+  // Add(3), Dequant(x), Add(4)
+  // Since both partitions are of size 1, either could end up as the 'first'
+  // partition with one Dequant node added for it.
+  EXPECT_EQ(ops_to_replace->size, 3);
 
   TfLiteNode* node = nullptr;
   TfLiteRegistration* registration = nullptr;
-  // Verify that Add op has fp16 inputs.
-  context->GetNodeAndRegistration(context, ops_to_replace->data[1], &node,
+  // Verify that both Add ops have fp16 inputs.
+  context->GetNodeAndRegistration(context, ops_to_replace->data[0], &node,
                                   &registration);
   EXPECT_EQ(context->tensors[node->inputs->data[0]].type,
             TfLiteType::kTfLiteFloat16);
   EXPECT_EQ(context->tensors[node->inputs->data[1]].type,
             TfLiteType::kTfLiteFloat16);
+  context->GetNodeAndRegistration(context, ops_to_replace->data[2], &node,
+                                  &registration);
+  EXPECT_EQ(context->tensors[node->inputs->data[0]].type,
+            TfLiteType::kTfLiteFloat16);
+  EXPECT_EQ(context->tensors[node->inputs->data[1]].type,
+            TfLiteType::kTfLiteFloat16);
+  // Verify that the op at index 1 is a Dequant outputing to a single Add.
+  EXPECT_TRUE(ops_to_replace->data[1] == 0 || ops_to_replace->data[1] == 2);
   TfLiteIntArrayFree(ops_to_replace);
 }
 
diff --git a/tensorflow/lite/delegates/gpu/gl/kernels/quantize_and_dequantize.h b/tensorflow/lite/delegates/gpu/gl/kernels/quantize_and_dequantize.h
index 1fa6ad918c4..46c40385766 100644
--- a/tensorflow/lite/delegates/gpu/gl/kernels/quantize_and_dequantize.h
+++ b/tensorflow/lite/delegates/gpu/gl/kernels/quantize_and_dequantize.h
@@ -30,11 +30,10 @@ namespace gl {
 // on the GPU, which cannot represent int8 tensors.
 //
 // Implemented as:
-// qvalue = round((min(qmax, max(qmin, src_val)) - qmin) * (1/qscale) + 0.5)
+// qvalue = round((min(qmax, max(qmin, src_val)) - qmin) * (1/qscale))
 // dq_value = qvalue * qscale + qmin
 // Here, qmin, qmax & qscale refer to the quantization values as implemented in
-// TensorFlow Lite's 'FakeQuant' kernel. round(x + 0.5) ensures we round away
-// from zero.
+// TensorFlow Lite's 'FakeQuant' kernel.
 //
 // NOTE: We do not need to nudge min/max values in this op, since they would
 // already be adjusted while generating the quantized model.
diff --git a/tensorflow/lite/delegates/gpu/java/src/main/java/org/tensorflow/lite/gpu/BUILD b/tensorflow/lite/delegates/gpu/java/src/main/java/org/tensorflow/lite/gpu/BUILD
index ab2ad036d66..fbd7f09ce65 100644
--- a/tensorflow/lite/delegates/gpu/java/src/main/java/org/tensorflow/lite/gpu/BUILD
+++ b/tensorflow/lite/delegates/gpu/java/src/main/java/org/tensorflow/lite/gpu/BUILD
@@ -4,6 +4,9 @@ package(
 
 filegroup(
     name = "gpu_delegate",
-    srcs = ["GpuDelegate.java"],
+    srcs = [
+        "GpuDelegate.java",
+        "Whitelist.java",
+    ],
     visibility = ["//visibility:public"],
 )
diff --git a/tensorflow/lite/delegates/gpu/java/src/main/java/org/tensorflow/lite/gpu/Whitelist.java b/tensorflow/lite/delegates/gpu/java/src/main/java/org/tensorflow/lite/gpu/Whitelist.java
new file mode 100644
index 00000000000..c0b3bf2ca37
--- /dev/null
+++ b/tensorflow/lite/delegates/gpu/java/src/main/java/org/tensorflow/lite/gpu/Whitelist.java
@@ -0,0 +1,93 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+package org.tensorflow.lite.gpu;
+
+import java.io.Closeable;
+
+/**
+ * GPU Delegate Whitelisting data.
+ *
+ * <p>The GPU delegate is not supported on all Android devices, due to differences in available
+ * OpenGL versions, driver features, and device resources. This class provides information on
+ * whether the GPU delegate is suitable for the current device.
+ *
+ * <p>This API is experimental and subject to change.
+ *
+ * <p><b>WARNING:</b> the whitelist is constructed from testing done on a limited set of models. You
+ * should plan to verify that your own model(s) work.
+ *
+ * <p>Example usage:
+ *
+ * <pre>{@code
+ * Interpreter.Options options = new Interpreter.Options();
+ * try (Whitelist whitelist = new Whitelist()) {
+ *   if (whitelist.isDelegateSupportedOnThisDevice()) {
+ *     GpuDelegate.Options delegateOptions = whitelist.getBestOptionsForThisDevice();
+ *     gpuDelegate = new GpuDelegate(delegateOptions):
+ *     options.addDelegate(gpuDelegate);
+ *   }
+ * }
+ * Interpreter interpreter = new Interpreter(modelBuffer, options);
+ * }</pre>
+ */
+public class Whitelist implements Closeable {
+
+  private static final long INVALID_WHITELIST_HANDLE = 0;
+  private static final String TFLITE_GPU_LIB = "tensorflowlite_gpu_jni";
+
+  private long whitelistHandle = INVALID_WHITELIST_HANDLE;
+
+  /** Whether the GPU delegate is supported on this device. */
+  public boolean isDelegateSupportedOnThisDevice() {
+    if (whitelistHandle == INVALID_WHITELIST_HANDLE) {
+      throw new IllegalStateException("Trying to query a closed whitelist.");
+    }
+    return nativeIsDelegateSupportedOnThisDevice(whitelistHandle);
+  }
+
+  /** What options should be used for the GPU delegate. */
+  public GpuDelegate.Options getBestOptionsForThisDevice() {
+    // For forward compatibility, when the whitelist contains more information.
+    return new GpuDelegate.Options();
+  }
+
+  public Whitelist() {
+    whitelistHandle = createWhitelist();
+  }
+
+  /**
+   * Frees TFLite resources in C runtime.
+   *
+   * <p>User is expected to call this method explicitly.
+   */
+  @Override
+  public void close() {
+    if (whitelistHandle != INVALID_WHITELIST_HANDLE) {
+      deleteWhitelist(whitelistHandle);
+      whitelistHandle = INVALID_WHITELIST_HANDLE;
+    }
+  }
+
+  static {
+    System.loadLibrary(TFLITE_GPU_LIB);
+  }
+
+  private static native long createWhitelist();
+
+  private static native void deleteWhitelist(long handle);
+
+  private static native boolean nativeIsDelegateSupportedOnThisDevice(long handle);
+}
diff --git a/tensorflow/lite/delegates/gpu/java/src/main/native/BUILD b/tensorflow/lite/delegates/gpu/java/src/main/native/BUILD
index 774fd417758..57d6e013a4a 100644
--- a/tensorflow/lite/delegates/gpu/java/src/main/native/BUILD
+++ b/tensorflow/lite/delegates/gpu/java/src/main/native/BUILD
@@ -26,7 +26,13 @@ cc_library(
     ],
     deps = [
         "//tensorflow/lite/delegates/gpu:delegate",
+        "//tensorflow/lite/delegates/gpu/common:gpu_info",
+        "//tensorflow/lite/delegates/gpu/gl:egl_environment",
+        "//tensorflow/lite/delegates/gpu/gl:request_gpu_info",
+        "//tensorflow/lite/experimental/acceleration/whitelist:android_info",
+        "//tensorflow/lite/experimental/acceleration/whitelist:gpu_whitelist",
         "//tensorflow/lite/java/jni",
+        "@com_google_absl//absl/status",
     ],
     alwayslink = 1,
 )
diff --git a/tensorflow/lite/delegates/gpu/java/src/main/native/gpu_delegate_jni.cc b/tensorflow/lite/delegates/gpu/java/src/main/native/gpu_delegate_jni.cc
index 900cc0e0d75..017ffcfcd32 100644
--- a/tensorflow/lite/delegates/gpu/java/src/main/native/gpu_delegate_jni.cc
+++ b/tensorflow/lite/delegates/gpu/java/src/main/native/gpu_delegate_jni.cc
@@ -15,7 +15,13 @@ limitations under the License.
 
 #include <jni.h>
 
+#include "absl/status/status.h"
+#include "tensorflow/lite/delegates/gpu/common/gpu_info.h"
 #include "tensorflow/lite/delegates/gpu/delegate.h"
+#include "tensorflow/lite/delegates/gpu/gl/egl_environment.h"
+#include "tensorflow/lite/delegates/gpu/gl/request_gpu_info.h"
+#include "tensorflow/lite/experimental/acceleration/whitelist/android_info.h"
+#include "tensorflow/lite/experimental/acceleration/whitelist/gpu_whitelist.h"
 
 #ifdef __cplusplus
 extern "C" {
@@ -44,6 +50,66 @@ JNIEXPORT void JNICALL Java_org_tensorflow_lite_gpu_GpuDelegate_deleteDelegate(
   TfLiteGpuDelegateV2Delete(reinterpret_cast<TfLiteDelegate*>(delegate));
 }
 
+namespace {
+class WhitelistHelper {
+ public:
+  absl::Status ReadInfo() {
+    auto status = tflite::acceleration::RequestAndroidInfo(&android_info_);
+    if (!status.ok()) return status;
+
+    if (android_info_.android_sdk_version < "21") {
+      // Weakly linked symbols may not be available on pre-21, and the GPU is
+      // not supported anyway so return early.
+      return absl::OkStatus();
+    }
+
+    std::unique_ptr<tflite::gpu::gl::EglEnvironment> env;
+    status = tflite::gpu::gl::EglEnvironment::NewEglEnvironment(&env);
+    if (!status.ok()) return status;
+
+    status = tflite::gpu::gl::RequestGpuInfo(&gpu_info_);
+    if (!status.ok()) return status;
+
+    return absl::OkStatus();
+  }
+
+  bool IsDelegateSupportedOnThisDevice() {
+    return whitelist_.Includes(android_info_, gpu_info_);
+  }
+
+ private:
+  tflite::acceleration::AndroidInfo android_info_;
+  tflite::gpu::GpuInfo gpu_info_;
+  tflite::acceleration::GPUWhitelist whitelist_;
+};
+}  // namespace
+
+JNIEXPORT jlong JNICALL Java_org_tensorflow_lite_gpu_Whitelist_createWhitelist(
+    JNIEnv* env, jclass clazz) {
+  WhitelistHelper* whitelist = new WhitelistHelper;
+  auto status = whitelist->ReadInfo();
+  // Errors in ReadInfo should almost always be failures to construct the OpenGL
+  // environment. Treating that as "GPU unsupported" is reasonable, and we can
+  // swallow the error.
+  status.IgnoreError();
+  return reinterpret_cast<jlong>(whitelist);
+}
+
+JNIEXPORT jboolean JNICALL
+Java_org_tensorflow_lite_gpu_Whitelist_nativeIsDelegateSupportedOnThisDevice(
+    JNIEnv* env, jclass clazz, jlong whitelist_handle) {
+  WhitelistHelper* whitelist =
+      reinterpret_cast<WhitelistHelper*>(whitelist_handle);
+  return whitelist->IsDelegateSupportedOnThisDevice() ? JNI_TRUE : JNI_FALSE;
+}
+
+JNIEXPORT void JNICALL Java_org_tensorflow_lite_gpu_Whitelist_deleteWhitelist(
+    JNIEnv* env, jclass clazz, jlong whitelist_handle) {
+  WhitelistHelper* whitelist =
+      reinterpret_cast<WhitelistHelper*>(whitelist_handle);
+  delete whitelist;
+}
+
 #ifdef __cplusplus
 }  // extern "C"
 #endif  // __cplusplus
diff --git a/tensorflow/lite/delegates/gpu/metal/api.cc b/tensorflow/lite/delegates/gpu/metal/api.cc
index 30c12dfe7e4..6a54e4e26bd 100644
--- a/tensorflow/lite/delegates/gpu/metal/api.cc
+++ b/tensorflow/lite/delegates/gpu/metal/api.cc
@@ -35,10 +35,10 @@ limitations under the License.
 #include "tensorflow/lite/delegates/gpu/metal/kernels/fully_connected.h"
 #include "tensorflow/lite/delegates/gpu/metal/kernels/max_unpooling.h"
 #include "tensorflow/lite/delegates/gpu/metal/kernels/mean.h"
-#include "tensorflow/lite/delegates/gpu/metal/kernels/mul.h"
 #include "tensorflow/lite/delegates/gpu/metal/kernels/padding.h"
 #include "tensorflow/lite/delegates/gpu/metal/kernels/pooling.h"
 #include "tensorflow/lite/delegates/gpu/metal/kernels/prelu.h"
+#include "tensorflow/lite/delegates/gpu/metal/kernels/quantize_and_dequantize.h"
 #include "tensorflow/lite/delegates/gpu/metal/kernels/relu.h"
 #include "tensorflow/lite/delegates/gpu/metal/kernels/reshape.h"
 #include "tensorflow/lite/delegates/gpu/metal/kernels/resize.h"
@@ -54,22 +54,6 @@ namespace gpu {
 namespace metal {
 namespace {
 
-bool IsWidthBroadcastedForSecondInput(const std::vector<Value*>& inputs) {
-  return inputs.size() == 2 &&
-         inputs[0]->tensor.shape.w != inputs[1]->tensor.shape.w &&
-         inputs[1]->tensor.shape.w == 1;
-}
-bool IsHeightBroadcastedForSecondInput(const std::vector<Value*>& inputs) {
-  return inputs.size() == 2 &&
-         inputs[0]->tensor.shape.h != inputs[1]->tensor.shape.h &&
-         inputs[1]->tensor.shape.h == 1;
-}
-bool IsChannelsBroadcastedForSecondInput(const std::vector<Value*>& inputs) {
-  return inputs.size() == 2 &&
-         inputs[0]->tensor.shape.c != inputs[1]->tensor.shape.c &&
-         inputs[1]->tensor.shape.c == 1;
-}
-
 std::vector<ComputeTaskDescriptorPtr> SelectDepthWiseConv(
     int id, ValueId input_id, ValueId output_id,
     const DepthwiseConvolution2DAttributes& attr,
@@ -96,6 +80,12 @@ std::vector<ComputeTaskDescriptorPtr> SelectConvolutionTransposed(
   }
 }
 
+std::vector<ComputeTaskDescriptorPtr> SelectQuantizeAndDequantize(
+    int id, ValueId input_id, ValueId output_id,
+    const QuantizeAndDequantizeAttributes& attr) {
+  return QuantizeAndDequantize(id, input_id, output_id, attr);
+}
+
 std::vector<ComputeTaskDescriptorPtr> SelectPReLU(
     const GraphFloat32& graph, int id, ValueId input_id, ValueId output_id,
     const PReLUAttributes& attr, const metal::RuntimeOptions& options) {
@@ -198,26 +188,22 @@ absl::Status RegisterPrimaryOps(const GraphFloat32& graph, const Node* node,
   auto op_type = OperationTypeFromString(node->operation.type);
   switch (op_type) {
     case OperationType::ADD: {
-      const AddAttributes& attr =
-          absl::any_cast<AddAttributes>(node->operation.attributes);
-      const auto* hwc_tensor =
-          absl::get_if<tflite::gpu::Tensor<HWC, DataType::FLOAT32>>(
-              &attr.param);
-      if (hwc_tensor) {
-        return absl::UnimplementedError(
-            "Unsupported op: " + node->operation.type +
-            ", no support of HWC constant tensor");
-      }
-      const auto srcs = graph.FindInputs(node_id);
-      ElementwiseBroadcastSettings broadcast;
-      broadcast.width = IsWidthBroadcastedForSecondInput(srcs);
-      broadcast.height = IsHeightBroadcastedForSecondInput(srcs);
-      broadcast.channels = IsChannelsBroadcastedForSecondInput(srcs);
-      if (broadcast.width || broadcast.height || broadcast.channels) {
-        *tasks = ElementwiseWithTwoInputs(node_id, inputs, outputs[0], op_type,
-                                          broadcast);
-      } else {
-        *tasks = Add(node_id, inputs, outputs[0], attr, options);
+      if (inputs.size() == 1) {
+        if (node->operation.attributes.has_value()) {
+          auto attr = absl::any_cast<AddAttributes>(node->operation.attributes);
+          *tasks = ElementwiseWithOneInputAndConstantArguent(
+              node_id, inputs[0], outputs[0], options, op_type, attr.param);
+        } else {
+          return absl::UnimplementedError(
+              "Missing attributes for single input op: " +
+              node->operation.type);
+        }
+      } else if (inputs.size() == 2) {
+        const auto srcs = graph.FindInputs(node_id);
+        *tasks = ElementwiseWithTwoInputs(node_id, inputs, outputs[0],
+                                          srcs[1]->tensor.shape, op_type);
+      } else {  // more than 2 inputs
+        *tasks = Add(node_id, inputs, outputs[0], options);
       }
       break;
     }
@@ -302,31 +288,21 @@ absl::Status RegisterPrimaryOps(const GraphFloat32& graph, const Node* node,
                     absl::any_cast<MeanAttributes>(node->operation.attributes));
       break;
     case OperationType::MUL:
-      if (node->operation.attributes.has_value()) {
-        const MultiplyAttributes& attr =
-            absl::any_cast<MultiplyAttributes>(node->operation.attributes);
-        const auto* hwc_tensor =
-            absl::get_if<tflite::gpu::Tensor<HWC, DataType::FLOAT32>>(
-                &attr.param);
-        if (hwc_tensor) {
-          return absl::UnimplementedError(
-              "Unsupported op: " + node->operation.type +
-              ", no support of HWC constant tensor");
-        }
-        *tasks = Multiply(node_id, inputs[0], outputs[0], attr, options);
-      } else {
-        if (inputs.size() == 2) {
-          const auto srcs = graph.FindInputs(node_id);
-          ElementwiseBroadcastSettings broadcast;
-          broadcast.width = IsWidthBroadcastedForSecondInput(srcs);
-          broadcast.height = IsHeightBroadcastedForSecondInput(srcs);
-          broadcast.channels = IsChannelsBroadcastedForSecondInput(srcs);
-          *tasks = ElementwiseWithTwoInputs(node_id, inputs, outputs[0],
-                                            op_type, broadcast);
+      if (inputs.size() == 1) {
+        if (node->operation.attributes.has_value()) {
+          auto attr =
+              absl::any_cast<MultiplyAttributes>(node->operation.attributes);
+          *tasks = ElementwiseWithOneInputAndConstantArguent(
+              node_id, inputs[0], outputs[0], options, op_type, attr.param);
         } else {
           return absl::UnimplementedError(
-              "No support of multiply with more than 2 inputs");
+              "Missing attributes for single input op: " +
+              node->operation.type);
         }
+      } else if (inputs.size() == 2) {
+        const auto srcs = graph.FindInputs(node_id);
+        *tasks = ElementwiseWithTwoInputs(node_id, inputs, outputs[0],
+                                          srcs[1]->tensor.shape, op_type);
       }
       break;
     case OperationType::PAD: {
@@ -351,6 +327,12 @@ absl::Status RegisterPrimaryOps(const GraphFloat32& graph, const Node* node,
       *tasks = ReLU(node_id, inputs[0], outputs[0],
                     absl::any_cast<ReLUAttributes>(node->operation.attributes));
       break;
+    case OperationType::QUANTIZE_AND_DEQUANTIZE:
+      *tasks = SelectQuantizeAndDequantize(
+          node_id, inputs[0], outputs[0],
+          absl::any_cast<QuantizeAndDequantizeAttributes>(
+              node->operation.attributes));
+      break;
     case OperationType::RESHAPE:
       *tasks = SelectReshape(
           graph, node_id, inputs[0], outputs[0],
@@ -400,34 +382,27 @@ absl::Status RegisterPrimaryOps(const GraphFloat32& graph, const Node* node,
     case OperationType::POW:
     case OperationType::SQUARED_DIFF:
     case OperationType::SUB: {
-      const ElementwiseAttributes* attr =
-          absl::any_cast<ElementwiseAttributes>(&node->operation.attributes);
-      if (attr) {
-        const auto* hwc_tensor =
-            absl::get_if<tflite::gpu::Tensor<HWC, DataType::FLOAT32>>(
-                &attr->param);
-        if (hwc_tensor) {
+      if (inputs.size() == 1) {
+        if (node->operation.attributes.has_value()) {
+          auto attr =
+              absl::any_cast<ElementwiseAttributes>(node->operation.attributes);
+          *tasks = ElementwiseWithOneInputAndConstantArguent(
+              node_id, inputs[0], outputs[0], options, op_type, attr.param);
+        } else {
           return absl::UnimplementedError(
-              "Unsupported op: " + node->operation.type +
-              ", no support of HWC constant tensor");
+              "Missing attributes for single input op: " +
+              node->operation.type);
         }
-        *tasks = ElementwiseWithOneInputAndConstantArguent(
-            node_id, inputs[0], outputs[0], options, op_type, *attr);
-      } else {
+      } else if (inputs.size() == 2) {
         const auto srcs = graph.FindInputs(node_id);
-        ElementwiseBroadcastSettings broadcast;
-        broadcast.width = IsWidthBroadcastedForSecondInput(srcs);
-        broadcast.height = IsHeightBroadcastedForSecondInput(srcs);
-        broadcast.channels = IsChannelsBroadcastedForSecondInput(srcs);
-        *tasks = ElementwiseWithTwoInputs(node_id, inputs, outputs[0], op_type,
-                                          broadcast);
+        *tasks = ElementwiseWithTwoInputs(node_id, inputs, outputs[0],
+                                          srcs[1]->tensor.shape, op_type);
       }
     } break;
     case OperationType::BATCH_NORMALIZATION:
     case OperationType::BATCH_TO_SPACE:
     case OperationType::CONST:
     case OperationType::LSTM:
-    case OperationType::QUANTIZE_AND_DEQUANTIZE:
     case OperationType::SPACE_TO_BATCH:
     case OperationType::TRANSPOSE:
     case OperationType::UNKNOWN:
diff --git a/tensorflow/lite/delegates/gpu/metal/kernels/BUILD b/tensorflow/lite/delegates/gpu/metal/kernels/BUILD
index 657e9b53a59..6385b87c403 100644
--- a/tensorflow/lite/delegates/gpu/metal/kernels/BUILD
+++ b/tensorflow/lite/delegates/gpu/metal/kernels/BUILD
@@ -27,10 +27,10 @@ cc_library(
         ":fully_connected",
         ":max_unpooling",
         ":mean",
-        ":mul",
         ":padding",
         ":pooling",
         ":prelu",
+        ":quantize_and_dequantize",
         ":relu",
         ":reshape",
         ":resize",
@@ -227,6 +227,7 @@ cc_library(
     srcs = ["elementwise.cc"],
     hdrs = ["elementwise.h"],
     deps = [
+        "//tensorflow/lite/delegates/gpu/common:convert",
         "//tensorflow/lite/delegates/gpu/common:model",
         "//tensorflow/lite/delegates/gpu/common:operations",
         "//tensorflow/lite/delegates/gpu/common:shape",
@@ -380,46 +381,6 @@ ios_unit_test(
     deps = [":mean_test_lib"],
 )
 
-cc_library(
-    name = "mul",
-    srcs = ["mul.cc"],
-    hdrs = ["mul.h"],
-    deps = [
-        "//tensorflow/lite/delegates/gpu/common:model",
-        "//tensorflow/lite/delegates/gpu/common:operations",
-        "//tensorflow/lite/delegates/gpu/common:shape",
-        "//tensorflow/lite/delegates/gpu/common:types",
-        "//tensorflow/lite/delegates/gpu/common:util",
-        "//tensorflow/lite/delegates/gpu/metal:compute_task_descriptor",
-        "//tensorflow/lite/delegates/gpu/metal:runtime_options",
-        "@com_google_absl//absl/strings",
-        "@com_google_absl//absl/types:variant",
-    ],
-)
-
-objc_library(
-    name = "mul_test_lib",
-    testonly = 1,
-    srcs = ["mul_test.mm"],
-    sdk_frameworks = ["XCTest"],
-    deps = [
-        ":mul",
-        ":test_util",
-    ],
-)
-
-ios_unit_test(
-    name = "mul_test",
-    testonly = 1,
-    minimum_os_version = "9.0",
-    runner = tflite_ios_lab_runner("IOS_LATEST"),
-    tags = tf_gpu_tests_tags() + [
-        "notap",
-        "tflite_not_portable_android",
-    ],
-    deps = [":mul_test_lib"],
-)
-
 cc_library(
     name = "padding",
     srcs = ["padding.cc"],
@@ -539,6 +500,53 @@ ios_unit_test(
     deps = [":prelu_test_lib"],
 )
 
+cc_library(
+    name = "quantize_and_dequantize",
+    srcs = ["quantize_and_dequantize.cc"],
+    hdrs = ["quantize_and_dequantize.h"],
+    deps = [
+        "//tensorflow/lite/delegates/gpu/common:model",
+        "//tensorflow/lite/delegates/gpu/common:operations",
+        "//tensorflow/lite/delegates/gpu/common:shape",
+        "//tensorflow/lite/delegates/gpu/common:types",
+        "//tensorflow/lite/delegates/gpu/common:util",
+        "//tensorflow/lite/delegates/gpu/metal:compute_task_descriptor",
+        "//tensorflow/lite/delegates/gpu/metal:runtime_options",
+        "@com_google_absl//absl/strings",
+    ],
+)
+
+objc_library(
+    name = "quantize_and_dequantize_test_lib",
+    testonly = 1,
+    srcs = ["quantize_and_dequantize_test.mm"],
+    sdk_frameworks = ["XCTest"],
+    deps = [
+        ":quantize_and_dequantize",
+        ":test_util",
+        "//tensorflow/lite/delegates/gpu/common:operations",
+        "//tensorflow/lite/delegates/gpu/common:shape",
+        "//tensorflow/lite/delegates/gpu/common:status",
+        "//tensorflow/lite/delegates/gpu/common:tensor",
+        "//tensorflow/lite/delegates/gpu/common:util",
+        "//tensorflow/lite/delegates/gpu/metal:compute_task_descriptor",
+        "//tensorflow/lite/delegates/gpu/metal:runtime_options",
+        "//tensorflow/lite/kernels/internal:quantization_util",
+    ],
+)
+
+ios_unit_test(
+    name = "quantize_and_dequantize_test",
+    testonly = 1,
+    minimum_os_version = "11.0",
+    runner = tflite_ios_lab_runner("IOS_LATEST"),
+    tags = tf_gpu_tests_tags() + [
+        "notap",
+        "tflite_not_portable_android",
+    ],
+    deps = [":quantize_and_dequantize_test_lib"],
+)
+
 cc_library(
     name = "relu",
     srcs = ["relu.cc"],
@@ -905,7 +913,6 @@ objc_library(
         "elementwise_test.mm",
         "fully_connected_test.mm",
         "max_unpooling_test.mm",
-        "mul_test.mm",
         "padding_test.mm",
         "pooling_test.mm",
         "prelu_test.mm",
diff --git a/tensorflow/lite/delegates/gpu/metal/kernels/add.cc b/tensorflow/lite/delegates/gpu/metal/kernels/add.cc
index b4a8e781c72..79c2c531cdb 100644
--- a/tensorflow/lite/delegates/gpu/metal/kernels/add.cc
+++ b/tensorflow/lite/delegates/gpu/metal/kernels/add.cc
@@ -52,39 +52,9 @@ std::string GetAddTableCodeFused(int src_count) {
 std::vector<ComputeTaskDescriptorPtr> Add(int id,
                                           const std::vector<ValueId> input_ids,
                                           ValueId output_id,
-                                          const AddAttributes& attr,
                                           const RuntimeOptions& options) {
   auto desc = std::make_shared<ComputeTaskDescriptor>();
   desc->id = id;
-
-  // Add scalar
-  const float* add_value = absl::get_if<float>(&attr.param);
-  if (add_value) {
-    desc->is_linkable = true;
-    desc->shader_source =
-        R"(FLT4 linkable$0(FLT4 value, int linear_index, uint3 gid) {
-      return value + )" +
-        std::to_string(*add_value) + ";}";
-    desc->input_buffers = {{input_ids[0]}};
-    desc->output_buffer = {output_id};
-    return {desc};
-  }
-  // Add vector
-  auto broadcast = absl::get_if<Tensor<Linear, DataType::FLOAT32>>(&attr.param);
-  if (broadcast) {
-    desc->is_linkable = true;
-    desc->shader_source =
-        R"(FLT4 linkable$0(FLT4 value, int linear_index, uint3 gid,
-      device FLT4* const broadcast) { return value + broadcast[gid.z]; })";
-    desc->input_buffers = {{input_ids[0]}};
-    desc->output_buffer = {output_id};
-    desc->immutable_buffers = {
-        {"device FLT4* const",
-         GetByteBufferConverted(broadcast->data, options.storage_precision)},
-    };
-    return {desc};
-  }
-
   desc->is_linkable = true;
   desc->is_associative_op = true;
   desc->shader_source = GetAddTableCodeFused(input_ids.size() - 1);
diff --git a/tensorflow/lite/delegates/gpu/metal/kernels/add.h b/tensorflow/lite/delegates/gpu/metal/kernels/add.h
index 5a8b73e0ae6..003dde4426a 100644
--- a/tensorflow/lite/delegates/gpu/metal/kernels/add.h
+++ b/tensorflow/lite/delegates/gpu/metal/kernels/add.h
@@ -27,11 +27,9 @@ namespace tflite {
 namespace gpu {
 namespace metal {
 
-// Add with broadcast.
 std::vector<ComputeTaskDescriptorPtr> Add(int id,
                                           const std::vector<ValueId> input_ids,
                                           ValueId output_id,
-                                          const AddAttributes& attr,
                                           const RuntimeOptions& options);
 
 }  // namespace metal
diff --git a/tensorflow/lite/delegates/gpu/metal/kernels/elementwise.cc b/tensorflow/lite/delegates/gpu/metal/kernels/elementwise.cc
index 9d9e054f40a..963bc1cd5ab 100644
--- a/tensorflow/lite/delegates/gpu/metal/kernels/elementwise.cc
+++ b/tensorflow/lite/delegates/gpu/metal/kernels/elementwise.cc
@@ -20,6 +20,7 @@ limitations under the License.
 #include <vector>
 
 #include "absl/strings/substitute.h"
+#include "tensorflow/lite/delegates/gpu/common/convert.h"
 #include "tensorflow/lite/delegates/gpu/common/operations.h"
 #include "tensorflow/lite/delegates/gpu/common/util.h"
 #include "tensorflow/lite/delegates/gpu/metal/compute_task_descriptor.h"
@@ -77,20 +78,20 @@ std::string TwoInputFunctor(OperationType op_type, const std::string& value0,
 
 std::vector<ComputeTaskDescriptorPtr> ElementwiseWithTwoInputs(
     int id, std::vector<ValueId> input_ids, ValueId output_id,
-    OperationType op_type, const ElementwiseBroadcastSettings& settings) {
+    const BHWC& second_shape, OperationType op_type) {
   auto desc = std::make_shared<ComputeTaskDescriptor>();
   desc->id = id;
   desc->is_linkable = true;
-  const std::string x_coord = settings.width ? "0" : "int(gid.x)";
-  const std::string y_coord = settings.height ? "0" : "int(gid.y)";
-  const std::string s_coord = settings.channels ? "0" : "int(gid.z)";
+  const std::string x_coord = second_shape.w == 1 ? "0" : "int(gid.x)";
+  const std::string y_coord = second_shape.h == 1 ? "0" : "int(gid.y)";
+  const std::string s_coord = second_shape.c == 1 ? "0" : "int(gid.z)";
   std::string code =
       "FLT4 linkable$0(FLT4 value, int linear_index, uint3 gid, device FLT4* "
       "const second_tensor, int2 second_size) {\n";
   code += "  int second_index = (" + s_coord + " * second_size.y + " + y_coord +
           ") * second_size.x + " + x_coord + ";\n";
   code += "  FLT4 src_1 = second_tensor[second_index];\n";
-  if (settings.channels) {
+  if (second_shape.c == 1) {
     code += "  src_1.y = src_1.x;\n";
     code += "  src_1.z = src_1.x;\n";
     code += "  src_1.w = src_1.x;\n";
@@ -138,13 +139,13 @@ std::vector<ComputeTaskDescriptorPtr> ElementwiseWithOneInput(
 
 std::vector<ComputeTaskDescriptorPtr> ElementwiseWithOneInputAndConstantArguent(
     int id, ValueId input_id, ValueId output_id, const RuntimeOptions& options,
-    OperationType op_type, const ElementwiseAttributes& attr) {
+    OperationType op_type, const TensorOrScalar& attr) {
   auto desc = std::make_shared<ComputeTaskDescriptor>();
   desc->id = id;
   desc->is_linkable = true;
-  auto scalar = absl::get_if<float>(&attr.param);
-  auto linear_buf =
-      absl::get_if<Tensor<Linear, DataType::FLOAT32>>(&attr.param);
+  auto scalar = absl::get_if<float>(&attr);
+  auto linear_buf = absl::get_if<Tensor<Linear, DataType::FLOAT32>>(&attr);
+  auto hwc_buf = absl::get_if<Tensor<HWC, DataType::FLOAT32>>(&attr);
   std::string param_desc;
   if (scalar) {
     param_desc += ", float scalar_val";
@@ -152,6 +153,9 @@ std::vector<ComputeTaskDescriptorPtr> ElementwiseWithOneInputAndConstantArguent(
   if (linear_buf) {
     param_desc += ", device FLT4* const linear_buf";
   }
+  if (hwc_buf) {
+    param_desc += ", device FLT4* const hwc_buf, int2 hwc_size";
+  }
   desc->shader_source =
       "FLT4 linkable$0(FLT4 value, int linear_index, uint3 gid" + param_desc +
       ") {\n";
@@ -159,6 +163,18 @@ std::vector<ComputeTaskDescriptorPtr> ElementwiseWithOneInputAndConstantArguent(
     desc->shader_source += "     FLT4 second_arg = FLT4(scalar_val);\n";
   } else if (linear_buf) {
     desc->shader_source += "     FLT4 second_arg = linear_buf[gid.z];\n";
+  } else if (hwc_buf) {
+    const std::string x_coord = hwc_buf->shape.w == 1 ? "0" : "int(gid.x)";
+    const std::string y_coord = hwc_buf->shape.h == 1 ? "0" : "int(gid.y)";
+    const std::string s_coord = hwc_buf->shape.c == 1 ? "0" : "int(gid.z)";
+    std::string index = "(" + s_coord + " * hwc_size.y + " + y_coord +
+                        ") * hwc_size.x + " + x_coord;
+    desc->shader_source += "  FLT4 second_arg = hwc_buf[" + index + "];\n";
+    if (hwc_buf->shape.c == 1) {
+      desc->shader_source += "  second_arg.y = second_arg.x;\n";
+      desc->shader_source += "  second_arg.z = second_arg.x;\n";
+      desc->shader_source += "  second_arg.w = second_arg.x;\n";
+    }
   }
   desc->shader_source +=
       "    return " + TwoInputFunctor(op_type, "value", "second_arg") + ";\n";
@@ -180,6 +196,20 @@ std::vector<ComputeTaskDescriptorPtr> ElementwiseWithOneInputAndConstantArguent(
         {"device FLT4* const",
          GetByteBufferConverted(linear_buf->data, options.storage_precision)},
     };
+  } else if (hwc_buf) {
+    std::vector<uint8_t> size_bits =
+        GetByteBuffer(std::vector<int>{hwc_buf->shape.w, hwc_buf->shape.h});
+    desc->uniform_buffers = {
+        {"constant int2&",
+         [size_bits](const std::map<ValueId, BHWC>& buffers) {
+           return size_bits;
+         }},
+    };
+    desc->immutable_buffers = {
+        {"device FLT4* const",
+         GetByteBufferConverted(ConvertToPHWC4(*hwc_buf),
+                                options.storage_precision)},
+    };
   }
   return {desc};
 }
diff --git a/tensorflow/lite/delegates/gpu/metal/kernels/elementwise.h b/tensorflow/lite/delegates/gpu/metal/kernels/elementwise.h
index 2520c2f2df4..dea466b3b6e 100644
--- a/tensorflow/lite/delegates/gpu/metal/kernels/elementwise.h
+++ b/tensorflow/lite/delegates/gpu/metal/kernels/elementwise.h
@@ -18,6 +18,7 @@ limitations under the License.
 
 #include <vector>
 
+#include "tensorflow/lite/delegates/gpu/common/model.h"
 #include "tensorflow/lite/delegates/gpu/common/operations.h"
 #include "tensorflow/lite/delegates/gpu/metal/compute_task_descriptor.h"
 
@@ -25,25 +26,19 @@ namespace tflite {
 namespace gpu {
 namespace metal {
 
-struct ElementwiseBroadcastSettings {
-  bool width = false;
-  bool height = false;
-  bool channels = false;
-};
-
-// Two inputs are two runtime tensors
-std::vector<ComputeTaskDescriptorPtr> ElementwiseWithTwoInputs(
-    int id, std::vector<ValueId> input_ids, ValueId output_id,
-    OperationType op_type, const ElementwiseBroadcastSettings& settings);
-
 // One input is one runtime tensor
 std::vector<ComputeTaskDescriptorPtr> ElementwiseWithOneInput(
     int id, ValueId input_id, ValueId output_id, OperationType op_type);
 
+// Two inputs are two runtime tensors
+std::vector<ComputeTaskDescriptorPtr> ElementwiseWithTwoInputs(
+    int id, std::vector<ValueId> input_ids, ValueId output_id,
+    const BHWC& second_shape, OperationType op_type);
+
 // First input is one runtime tensor and second input is constant argument
 std::vector<ComputeTaskDescriptorPtr> ElementwiseWithOneInputAndConstantArguent(
     int id, ValueId input_id, ValueId output_id, const RuntimeOptions& options,
-    OperationType op_type, const ElementwiseAttributes& attr);
+    OperationType op_type, const TensorOrScalar& attr);
 
 }  // namespace metal
 }  // namespace gpu
diff --git a/tensorflow/lite/delegates/gpu/metal/kernels/elementwise_test.mm b/tensorflow/lite/delegates/gpu/metal/kernels/elementwise_test.mm
index 6b30bc5c703..d3327e9ec2c 100644
--- a/tensorflow/lite/delegates/gpu/metal/kernels/elementwise_test.mm
+++ b/tensorflow/lite/delegates/gpu/metal/kernels/elementwise_test.mm
@@ -30,6 +30,7 @@ limitations under the License.
 #include "tensorflow/lite/delegates/gpu/metal/runtime_options.h"
 
 using ::tflite::gpu::DataType;
+using ::tflite::gpu::HWC;
 using ::tflite::gpu::BHWC;
 using ::tflite::gpu::OperationType;
 using ::tflite::gpu::TensorRef;
@@ -163,6 +164,42 @@ TensorRef<BHWC> GetTensorRef(int ref, const BHWC& shape) {
   XCTAssertTrue(status.ok(), @"%s", std::string(status.message()).c_str());
 }
 
+- (void)testMaximumWithConstantHWCTensor {
+  OperationType op_type = OperationType::MAXIMUM;
+  const BHWC shape(1, 2, 1, 2);
+  tflite::gpu::ElementwiseAttributes attr;
+  tflite::gpu::Tensor<HWC, DataType::FLOAT32> hwc_tensor;
+  hwc_tensor.shape = HWC(2, 1, 2);
+  hwc_tensor.data = {0.5f, 2.0f, 0.7f, 4.7f};
+  attr.param = hwc_tensor;
+  SingleOpModel model({/*type=*/ToString(op_type), /*attributes=*/attr},
+                      /*inputs=*/{GetTensorRef(0, shape)},
+                      /*outputs=*/{GetTensorRef(1, shape)});
+  XCTAssertTrue(model.PopulateTensor(0, {1.0f, -6.2f, -2.0f, 3.0f}));
+  auto status = model.Invoke();
+  XCTAssertTrue(status.ok(), @"%s", std::string(status.message()).c_str());
+  status = CompareVectors({1.0f, 2.0f, 0.7f, 4.7f}, model.GetOutput(0), 1e-6f);
+  XCTAssertTrue(status.ok(), @"%s", std::string(status.message()).c_str());
+}
+
+- (void)testMaximumWithConstantHWCTensorBroadcastChannels {
+  OperationType op_type = OperationType::MAXIMUM;
+  const BHWC shape(1, 2, 1, 2);
+  tflite::gpu::ElementwiseAttributes attr;
+  tflite::gpu::Tensor<HWC, DataType::FLOAT32> hwc_tensor;
+  hwc_tensor.shape = HWC(2, 1, 1);
+  hwc_tensor.data = {0.5f, 2.0f};
+  attr.param = hwc_tensor;
+  SingleOpModel model({/*type=*/ToString(op_type), /*attributes=*/attr},
+                      /*inputs=*/{GetTensorRef(0, shape)},
+                      /*outputs=*/{GetTensorRef(1, shape)});
+  XCTAssertTrue(model.PopulateTensor(0, {1.0f, -6.2f, -2.0f, 3.0f}));
+  auto status = model.Invoke();
+  XCTAssertTrue(status.ok(), @"%s", std::string(status.message()).c_str());
+  status = CompareVectors({1.0f, 0.5f, 2.0f, 3.0f}, model.GetOutput(0), 1e-6f);
+  XCTAssertTrue(status.ok(), @"%s", std::string(status.message()).c_str());
+}
+
 - (void)testMinimum {
   OperationType op_type = OperationType::MINIMUM;
   const BHWC shape(1, 2, 2, 1);
diff --git a/tensorflow/lite/delegates/gpu/metal/kernels/mul.cc b/tensorflow/lite/delegates/gpu/metal/kernels/mul.cc
deleted file mode 100644
index e90ab6b4f12..00000000000
--- a/tensorflow/lite/delegates/gpu/metal/kernels/mul.cc
+++ /dev/null
@@ -1,83 +0,0 @@
-/* Copyright 2019 The TensorFlow Authors. All Rights Reserved.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-==============================================================================*/
-
-#include "tensorflow/lite/delegates/gpu/metal/kernels/mul.h"
-
-#include <cstdint>
-#include <map>
-#include <memory>
-#include <string>
-#include <vector>
-
-#include "absl/strings/substitute.h"
-#include "absl/types/variant.h"
-#include "tensorflow/lite/delegates/gpu/common/data_type.h"
-#include "tensorflow/lite/delegates/gpu/common/model.h"
-#include "tensorflow/lite/delegates/gpu/common/operations.h"
-#include "tensorflow/lite/delegates/gpu/common/shape.h"
-#include "tensorflow/lite/delegates/gpu/common/tensor.h"
-#include "tensorflow/lite/delegates/gpu/common/util.h"
-#include "tensorflow/lite/delegates/gpu/metal/compute_task_descriptor.h"
-#include "tensorflow/lite/delegates/gpu/metal/runtime_options.h"
-
-namespace tflite {
-namespace gpu {
-namespace metal {
-std::vector<ComputeTaskDescriptorPtr> Multiply(int id, ValueId input_id,
-                                               ValueId output_id,
-                                               const MultiplyAttributes& attr,
-                                               const RuntimeOptions& options) {
-  auto desc = std::make_shared<ComputeTaskDescriptor>();
-  desc->id = id;
-  desc->is_linkable = true;
-  auto multiplier = absl::get_if<float>(&attr.param);
-  auto mul_buffer =
-      absl::get_if<Tensor<Linear, DataType::FLOAT32>>(&attr.param);
-  const bool scalar = multiplier != nullptr;
-  const std::string param_desc =
-      scalar ? "float multiplier" : "device FLT4* const mul_buf";
-  std::string code =
-      "FLT4 linkable$0(FLT4 value, int linear_index, uint3 gid, ";
-  code += param_desc + ") {\n";
-  if (scalar) {
-    code += "return value * multiplier;\n";
-  } else {
-    code += "return value * mul_buf[gid.z];\n";
-  }
-  code += "}\n";
-  desc->shader_source = code;
-  desc->input_buffers = {{input_id}};
-  desc->output_buffer = {output_id};
-  if (scalar) {
-    std::vector<uint8_t> multiplier_bits =
-        GetByteBuffer(std::vector<float>{*multiplier});
-    desc->uniform_buffers = {
-        {"constant float&",
-         [multiplier_bits](const std::map<ValueId, BHWC>& buffers) {
-           return multiplier_bits;
-         }},
-    };
-  } else {
-    desc->immutable_buffers = {
-        {"device FLT4* const",
-         GetByteBufferConverted(mul_buffer->data, options.storage_precision)},
-    };
-  }
-  return {desc};
-}
-
-}  // namespace metal
-}  // namespace gpu
-}  // namespace tflite
diff --git a/tensorflow/lite/delegates/gpu/metal/kernels/mul.h b/tensorflow/lite/delegates/gpu/metal/kernels/mul.h
deleted file mode 100644
index b5ff37cf560..00000000000
--- a/tensorflow/lite/delegates/gpu/metal/kernels/mul.h
+++ /dev/null
@@ -1,37 +0,0 @@
-/* Copyright 2019 The TensorFlow Authors. All Rights Reserved.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-==============================================================================*/
-
-#ifndef TENSORFLOW_LITE_DELEGATES_GPU_METAL_KERNELS_MUL_H_
-#define TENSORFLOW_LITE_DELEGATES_GPU_METAL_KERNELS_MUL_H_
-
-#include "tensorflow/lite/delegates/gpu/common/model.h"
-#include "tensorflow/lite/delegates/gpu/common/operations.h"
-#include "tensorflow/lite/delegates/gpu/metal/compute_task_descriptor.h"
-#include "tensorflow/lite/delegates/gpu/metal/runtime_options.h"
-
-namespace tflite {
-namespace gpu {
-namespace metal {
-
-// Multiply operation, supports scalar and vector broadcast.
-std::vector<ComputeTaskDescriptorPtr> Multiply(int id, ValueId input_id,
-                                               ValueId output_id,
-                                               const MultiplyAttributes& attr,
-                                               const RuntimeOptions& options);
-}  // namespace metal
-}  // namespace gpu
-}  // namespace tflite
-
-#endif  // TENSORFLOW_LITE_DELEGATES_GPU_METAL_KERNELS_MUL_H_
diff --git a/tensorflow/lite/delegates/gpu/metal/kernels/mul_test.mm b/tensorflow/lite/delegates/gpu/metal/kernels/mul_test.mm
deleted file mode 100644
index d881950c831..00000000000
--- a/tensorflow/lite/delegates/gpu/metal/kernels/mul_test.mm
+++ /dev/null
@@ -1,98 +0,0 @@
-/* Copyright 2019 The TensorFlow Authors. All Rights Reserved.
-
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-
-    http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-==============================================================================*/
-
-#include "tensorflow/lite/delegates/gpu/metal/kernels/add.h"
-
-#import <XCTest/XCTest.h>
-
-#include <string>
-#include <vector>
-
-#include "tensorflow/lite/delegates/gpu/common/operations.h"
-#include "tensorflow/lite/delegates/gpu/common/shape.h"
-#include "tensorflow/lite/delegates/gpu/common/status.h"
-#include "tensorflow/lite/delegates/gpu/common/tensor.h"
-#include "tensorflow/lite/delegates/gpu/common/util.h"
-#include "tensorflow/lite/delegates/gpu/metal/compute_task_descriptor.h"
-#include "tensorflow/lite/delegates/gpu/metal/kernels/test_util.h"
-#include "tensorflow/lite/delegates/gpu/metal/runtime_options.h"
-
-using ::tflite::gpu::DataType;
-using ::tflite::gpu::BHWC;
-using ::tflite::gpu::Linear;
-using ::tflite::gpu::MultiplyAttributes;
-using ::tflite::gpu::OperationType;
-using ::tflite::gpu::Tensor;
-using ::tflite::gpu::TensorRef;
-using ::tflite::gpu::metal::CompareVectors;
-using ::tflite::gpu::metal::SingleOpModel;
-
-@interface MulTest : XCTestCase
-@end
-
-@implementation MulTest
-- (void)setUp {
-  [super setUp];
-}
-
-- (void)testMulScalar {
-  TensorRef<BHWC> input;
-  input.type = DataType::FLOAT32;
-  input.ref = 0;
-  input.shape = BHWC(1, 2, 2, 1);
-
-  TensorRef<BHWC> output;
-  output.type = DataType::FLOAT32;
-  output.ref = 1;
-  output.shape = BHWC(1, 2, 2, 1);
-
-  MultiplyAttributes attr;
-  attr.param = 2;
-
-  SingleOpModel model({ToString(OperationType::MUL), attr}, {input}, {output});
-  XCTAssertTrue(model.PopulateTensor(0, {1, 2, 3, 4}));
-  auto status = model.Invoke();
-  XCTAssertTrue(status.ok(), @"%s", std::string(status.message()).c_str());
-  status = CompareVectors({2, 4, 6, 8}, model.GetOutput(0), 1e-6f);
-  XCTAssertTrue(status.ok(), @"%s", std::string(status.message()).c_str());
-}
-
-- (void)testMulLinear {
-  TensorRef<BHWC> input;
-  input.type = DataType::FLOAT32;
-  input.ref = 0;
-  input.shape = BHWC(1, 1, 2, 2);
-
-  TensorRef<BHWC> output;
-  output.type = DataType::FLOAT32;
-  output.ref = 1;
-  output.shape = BHWC(1, 1, 2, 2);
-
-  MultiplyAttributes attr;
-  Tensor<Linear, DataType::FLOAT32> tensor;
-  tensor.shape.v = 2;
-  tensor.id = 1;
-  tensor.data = {2, 3};
-  attr.param = std::move(tensor);
-
-  SingleOpModel model({ToString(OperationType::MUL), attr}, {input}, {output});
-  XCTAssertTrue(model.PopulateTensor(0, {1, 2, 3, 4}));
-  auto status = model.Invoke();
-  XCTAssertTrue(status.ok(), @"%s", std::string(status.message()).c_str());
-  status = CompareVectors({2, 6, 6, 12}, model.GetOutput(0), 1e-6f);
-  XCTAssertTrue(status.ok(), @"%s", std::string(status.message()).c_str());
-}
-
-@end
diff --git a/tensorflow/lite/delegates/gpu/metal/kernels/quantize_and_dequantize.cc b/tensorflow/lite/delegates/gpu/metal/kernels/quantize_and_dequantize.cc
new file mode 100644
index 00000000000..e6bcbe24d2d
--- /dev/null
+++ b/tensorflow/lite/delegates/gpu/metal/kernels/quantize_and_dequantize.cc
@@ -0,0 +1,54 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include "tensorflow/lite/delegates/gpu/metal/kernels/quantize_and_dequantize.h"
+
+#include "tensorflow/lite/delegates/gpu/common/model.h"
+#include "tensorflow/lite/delegates/gpu/common/operations.h"
+#include "tensorflow/lite/delegates/gpu/common/shape.h"
+#include "tensorflow/lite/delegates/gpu/metal/compute_task_descriptor.h"
+
+namespace tflite {
+namespace gpu {
+namespace metal {
+std::vector<ComputeTaskDescriptorPtr> QuantizeAndDequantize(
+    int id, ValueId input_id, ValueId output_id,
+    const QuantizeAndDequantizeAttributes& attr) {
+  auto desc = std::make_shared<ComputeTaskDescriptor>();
+  desc->id = id;
+  desc->is_linkable = true;
+  desc->shader_source = R"(
+    FLT4 linkable$0(FLT4 value, int linear_index, uint3 gid, float3 params) {
+      value = clamp(value, FLT4(params.x), FLT4(params.y));
+      value = (value - FLT4(params.x)) / FLT4(params.z);
+      return round(value) * FLT4(params.z) + FLT4(params.x);
+    }
+  )";
+
+  desc->input_buffers = {{input_id}};
+  desc->output_buffer = {output_id};
+  desc->uniform_buffers = {
+      {"constant float3&",
+       [attr](const std::map<ValueId, BHWC>& buffers) {
+         return GetByteBuffer(
+             std::vector<float>{attr.min, attr.max, attr.scale});
+       }},
+  };
+  return {desc};
+}
+
+}  // namespace metal
+}  // namespace gpu
+}  // namespace tflite
diff --git a/tensorflow/lite/delegates/gpu/metal/kernels/quantize_and_dequantize.h b/tensorflow/lite/delegates/gpu/metal/kernels/quantize_and_dequantize.h
new file mode 100644
index 00000000000..d43f659e300
--- /dev/null
+++ b/tensorflow/lite/delegates/gpu/metal/kernels/quantize_and_dequantize.h
@@ -0,0 +1,50 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#ifndef TENSORFLOW_LITE_DELEGATES_GPU_METAL_KERNELS_QUANTIZE_AND_DEQUANTIZE_H_
+#define TENSORFLOW_LITE_DELEGATES_GPU_METAL_KERNELS_QUANTIZE_AND_DEQUANTIZE_H_
+
+#include <vector>
+
+#include "tensorflow/lite/delegates/gpu/common/model.h"
+#include "tensorflow/lite/delegates/gpu/common/operations.h"
+#include "tensorflow/lite/delegates/gpu/metal/compute_task_descriptor.h"
+#include "tensorflow/lite/delegates/gpu/metal/runtime_options.h"
+
+namespace tflite {
+namespace gpu {
+namespace metal {
+
+// Performs the operation: {Quantize, Dequantize} on floating-point data.
+// We need this operation to emulate the error introduced by quantization
+// on the GPU, which cannot represent int8 tensors.
+//
+// Implemented as:
+// qvalue = round((min(qmax, max(qmin, src_val)) - qmin) * (1/qscale))
+// dq_value = qvalue * qscale + qmin
+// Here, qmin, qmax & qscale refer to the quantization values as implemented in
+// TensorFlow Lite's 'FakeQuant' kernel.
+//
+// NOTE: We do not need to nudge min/max values in this op, since they would
+// already be adjusted while generating the quantized model.
+std::vector<ComputeTaskDescriptorPtr> QuantizeAndDequantize(
+    int id, ValueId input_id, ValueId output_id,
+    const QuantizeAndDequantizeAttributes& attr);
+
+}  // namespace metal
+}  // namespace gpu
+}  // namespace tflite
+
+#endif  // TENSORFLOW_LITE_DELEGATES_GPU_METAL_KERNELS_QUANTIZE_AND_DEQUANTIZE_H_
diff --git a/tensorflow/lite/delegates/gpu/metal/kernels/quantize_and_dequantize_test.mm b/tensorflow/lite/delegates/gpu/metal/kernels/quantize_and_dequantize_test.mm
new file mode 100644
index 00000000000..7a16f1d25b8
--- /dev/null
+++ b/tensorflow/lite/delegates/gpu/metal/kernels/quantize_and_dequantize_test.mm
@@ -0,0 +1,167 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#import <XCTest/XCTest.h>
+
+#include <string>
+#include <vector>
+
+#include "tensorflow/lite/delegates/gpu/common/operations.h"
+#include "tensorflow/lite/delegates/gpu/common/shape.h"
+#include "tensorflow/lite/delegates/gpu/common/status.h"
+#include "tensorflow/lite/delegates/gpu/common/tensor.h"
+#include "tensorflow/lite/delegates/gpu/common/util.h"
+#include "tensorflow/lite/delegates/gpu/metal/compute_task_descriptor.h"
+#include "tensorflow/lite/delegates/gpu/metal/kernels/test_util.h"
+#include "tensorflow/lite/delegates/gpu/metal/runtime_options.h"
+#include "tensorflow/lite/kernels/internal/quantization_util.h"
+
+using ::tflite::NudgeQuantizationRange;
+using ::tflite::gpu::DataType;
+using ::tflite::gpu::BHWC;
+using ::tflite::gpu::OperationType;
+using ::tflite::gpu::QuantizeAndDequantizeAttributes;
+using ::tflite::gpu::TensorRef;
+using ::tflite::gpu::metal::CompareVectors;
+using ::tflite::gpu::metal::SingleOpModel;
+
+// TODO: Add per-op test if possible.
+@interface QuantizeAndDequantizeTest : XCTestCase
+@end
+
+@implementation QuantizeAndDequantizeTest
+- (void)setUp {
+  [super setUp];
+}
+
+- (void)testDim2Bits8 {
+  TensorRef<BHWC> input;
+  input.type = DataType::FLOAT32;
+  input.ref = 0;
+  input.shape = BHWC(1, 3, 2, 1);
+
+  // Unlike TFLite's FakeQuant kernel, we assume that the incoming values are
+  // pre-nudged, since this should be done during model conversion.
+  const int num_bits = 8;
+  const int quant_min = 0;
+  const int quant_max = (1 << num_bits) - 1;
+  QuantizeAndDequantizeAttributes attr;
+  NudgeQuantizationRange(/**original_min**/ 0.0, /**original_max**/ 1.0, quant_min, quant_max,
+                         &attr.min, &attr.max, &attr.scale);
+
+  TensorRef<BHWC> output;
+  output.type = DataType::FLOAT32;
+  output.ref = 1;
+  output.shape = BHWC(1, 3, 2, 1);
+
+  SingleOpModel model({ToString(OperationType::QUANTIZE_AND_DEQUANTIZE), attr}, {input}, {output});
+  XCTAssertTrue(model.PopulateTensor(0, {0.0, 1.0, 0.25, 0.50, 0.4444444, 0.00001}));
+  auto status = model.Invoke();
+  XCTAssertTrue(status.ok(), @"%s", std::string(status.message()).c_str());
+  std::vector<float> expected_output = {0.0f, 1.0f, 0.25098f, 0.498039f, 0.443137f, 0.0f};
+  status =
+      CompareVectors({0.0f, 1.0f, 0.25098f, 0.498039f, 0.443137f, 0.0f}, model.GetOutput(0), 1e-6f);
+  XCTAssertTrue(status.ok(), @"%s", std::string(status.message()).c_str());
+}
+
+- (void)testDim3Bits8_NegativeRange {
+  TensorRef<BHWC> input;
+  input.type = DataType::FLOAT32;
+  input.ref = 0;
+  input.shape = BHWC(1, 3, 1, 2);
+
+  // Unlike TFLite's FakeQuant kernel, we assume that the incoming values are
+  // pre-nudged, since this should be done during model conversion.
+  const int num_bits = 8;
+  const int quant_min = 0;
+  const int quant_max = (1 << num_bits) - 1;
+  QuantizeAndDequantizeAttributes attr;
+  NudgeQuantizationRange(/**original_min**/ -0.9, /**original_max**/ 0.9, quant_min, quant_max,
+                         &attr.min, &attr.max, &attr.scale);
+
+  TensorRef<BHWC> output;
+  output.type = DataType::FLOAT32;
+  output.ref = 1;
+  output.shape = BHWC(1, 3, 1, 2);
+
+  SingleOpModel model({ToString(OperationType::QUANTIZE_AND_DEQUANTIZE), attr}, {input}, {output});
+  XCTAssertTrue(model.PopulateTensor(0, {0.0, -0.9, 0.25, 0.50, 0.4444444, -0.00001}));
+  auto status = model.Invoke();
+  XCTAssertTrue(status.ok(), @"%s", std::string(status.message()).c_str());
+  status = CompareVectors({0.0f, -0.896471f, 0.247059f, 0.501176f, 0.444706f, 0.0f},
+                          model.GetOutput(0), 1e-6f);
+  XCTAssertTrue(status.ok(), @"%s", std::string(status.message()).c_str());
+}
+
+- (void)testDim3Bits16 {
+  TensorRef<BHWC> input;
+  input.type = DataType::FLOAT32;
+  input.ref = 0;
+  input.shape = BHWC(1, 3, 1, 2);
+
+  // Unlike TFLite's FakeQuant kernel, we assume that the incoming values are
+  // pre-nudged, since this should be done during model conversion.
+  const int num_bits = 16;
+  const int quant_min = 0;
+  const int quant_max = (1 << num_bits) - 1;
+  QuantizeAndDequantizeAttributes attr;
+  NudgeQuantizationRange(/**original_min**/ 0.0, /**original_max**/ 1.0, quant_min, quant_max,
+                         &attr.min, &attr.max, &attr.scale);
+
+  TensorRef<BHWC> output;
+  output.type = DataType::FLOAT32;
+  output.ref = 1;
+  output.shape = BHWC(1, 3, 1, 2);
+
+  SingleOpModel model({ToString(OperationType::QUANTIZE_AND_DEQUANTIZE), attr}, {input}, {output});
+  XCTAssertTrue(model.PopulateTensor(0, {0.0, 1.0, 0.25, 0.50, 0.4444444, 0.00001}));
+  auto status = model.Invoke();
+  XCTAssertTrue(status.ok(), @"%s", std::string(status.message()).c_str());
+  status = CompareVectors({0.0f, 1.0f, 0.250004f, 0.500008f, 0.44445f, 1.5259e-05f},
+                          model.GetOutput(0), 1e-6f);
+  XCTAssertTrue(status.ok(), @"%s", std::string(status.message()).c_str());
+}
+
+- (void)testDim2Bits16_NegativeRange {
+  TensorRef<BHWC> input;
+  input.type = DataType::FLOAT32;
+  input.ref = 0;
+  input.shape = BHWC(1, 3, 2, 1);
+
+  // Unlike TFLite's FakeQuant kernel, we assume that the incoming values are
+  // pre-nudged, since this should be done during model conversion.
+  const int num_bits = 16;
+  const int quant_min = 0;
+  const int quant_max = (1 << num_bits) - 1;
+  QuantizeAndDequantizeAttributes attr;
+  NudgeQuantizationRange(/**original_min**/ -0.9, /**original_max**/ 0.9, quant_min, quant_max,
+                         &attr.min, &attr.max, &attr.scale);
+
+  TensorRef<BHWC> output;
+  output.type = DataType::FLOAT32;
+  output.ref = 1;
+  output.shape = BHWC(1, 3, 2, 1);
+
+  SingleOpModel model({ToString(OperationType::QUANTIZE_AND_DEQUANTIZE), attr}, {input}, {output});
+  XCTAssertTrue(model.PopulateTensor(0, {0.0, -0.9, 0.25, 0.50, 0.4444444, -0.00001}));
+  auto status = model.Invoke();
+  XCTAssertTrue(status.ok(), @"%s", std::string(status.message()).c_str());
+  status =
+      CompareVectors({0.0f, -0.900014f, 0.249998f, 0.499995f, 0.444431f, 0.0f}, model.GetOutput(0),
+                     1e-6f);
+  XCTAssertTrue(status.ok(), @"%s", std::string(status.message()).c_str());
+}
+
+@end
diff --git a/tensorflow/lite/delegates/gpu/metal_delegate.h b/tensorflow/lite/delegates/gpu/metal_delegate.h
index 032c92c486d..1cb660c42d0 100644
--- a/tensorflow/lite/delegates/gpu/metal_delegate.h
+++ b/tensorflow/lite/delegates/gpu/metal_delegate.h
@@ -58,6 +58,8 @@ typedef struct {
   // Allows to quantify tensors, downcast values, process in float16 etc.
   bool allow_precision_loss;
   TFLGpuDelegateWaitType wait_type;
+  // Allows execution of integer quantized models
+  bool enable_quantization;
 } TFLGpuDelegateOptions;
 
 // Creates a new delegate instance that need to be destroyed with
diff --git a/tensorflow/lite/delegates/gpu/metal_delegate.mm b/tensorflow/lite/delegates/gpu/metal_delegate.mm
index 8b8bfe147c3..01fa9dd7679 100644
--- a/tensorflow/lite/delegates/gpu/metal_delegate.mm
+++ b/tensorflow/lite/delegates/gpu/metal_delegate.mm
@@ -29,10 +29,12 @@ limitations under the License.
 #include "absl/types/span.h"
 #include "tensorflow/lite/builtin_ops.h"
 #include "tensorflow/lite/c/common.h"
+#include "tensorflow/lite/context_util.h"
 #include "tensorflow/lite/delegates/gpu/common/convert.h"
 #include "tensorflow/lite/delegates/gpu/common/model.h"
 #include "tensorflow/lite/delegates/gpu/common/model_builder.h"
 #include "tensorflow/lite/delegates/gpu/common/model_transformer.h"
+#include "tensorflow/lite/delegates/gpu/common/quantization_util.h"
 #include "tensorflow/lite/delegates/gpu/common/shape.h"
 #include "tensorflow/lite/delegates/gpu/common/status.h"
 #include "tensorflow/lite/delegates/gpu/common/transformations/general_transformations.h"
@@ -40,10 +42,11 @@ limitations under the License.
 #include "tensorflow/lite/delegates/gpu/metal/api.h"
 #include "tensorflow/lite/delegates/gpu/metal/buffer_convert.h"
 #include "tensorflow/lite/delegates/gpu/metal/common.h"
-#include "tensorflow/lite/delegates/gpu/metal/environment.h"
 #include "tensorflow/lite/delegates/gpu/metal/compiled_model.h"
+#include "tensorflow/lite/delegates/gpu/metal/environment.h"
 #include "tensorflow/lite/delegates/gpu/metal/inference_context.h"
 #include "tensorflow/lite/delegates/gpu/metal/runtime_options.h"
+#include "tensorflow/lite/kernels/kernel_util.h"
 #include "tensorflow/lite/minimal_logging.h"
 
 namespace tflite {
@@ -176,6 +179,7 @@ class Delegate {
     } else {
       // Default options.
       options_.allow_precision_loss = false;
+      options_.enable_quantization = false;
       options_.wait_type = TFLGpuDelegateWaitType::TFLGpuDelegateWaitTypePassive;
     }
     metal_device_ = MTLCreateSystemDefaultDevice();
@@ -227,16 +231,38 @@ class Delegate {
     external_command_encoder_ = encoder;
   }
 
+  // This directs the runtime to allocate memory for input/output temporary
+  // tensors that require dequantization/quantization.
+  absl::Status GetRequiredTemporaries(TfLiteContext* context, TfLiteNode* node,
+                                      TfLiteIntArray** temporaries_array_ptr) {
+    if (quant_conversion_map_.empty()) return absl::OkStatus();
+
+    std::vector<int> temporary_tensor_ids;
+    for (auto index : input_tensor_ids_) {
+      if (quant_conversion_map_.find(index) != quant_conversion_map_.end()) {
+        temporary_tensor_ids.push_back(index);
+      }
+    }
+    for (auto index : output_tensor_ids_) {
+      if (quant_conversion_map_.find(index) != quant_conversion_map_.end()) {
+        temporary_tensor_ids.push_back(index);
+      }
+    }
+    *temporaries_array_ptr = TfLiteIntArrayCreate(temporary_tensor_ids.size());
+    for (int i = 0; i < temporary_tensor_ids.size(); ++i) {
+      (*temporaries_array_ptr)->data[i] = temporary_tensor_ids[i];
+    }
+    return absl::OkStatus();
+  }
+
   absl::Status Prepare(TfLiteContext* context, const TfLiteDelegateParams* delegate_params) {
     // Extract TFLite delegate execution plan from the context and convert it into GraphFloat32.
     GraphFloat32 graph;
-    RETURN_IF_ERROR(BuildModel(context, delegate_params, &graph));
-
-    // Apply general transformations on the graph.
-    NullTransformationReporter reporter;
-    ModelTransformer transformer(&graph, &reporter);
-    if (!ApplyGeneralTransformations(&transformer)) {
-      return absl::InternalError("Graph general transformations failed");
+    quant_conversion_map_.clear();
+    if (options_.enable_quantization) {
+      RETURN_IF_ERROR(BuildFinalModel(context, delegate_params, &graph, &quant_conversion_map_));
+    } else {
+      RETURN_IF_ERROR(BuildFinalModel(context, delegate_params, &graph));
     }
 
     // TODO(impjdi): Remove code duplication.
@@ -260,17 +286,23 @@ class Delegate {
     //
     // Note that graph.inputs() cannot be used directly, as the notion of graph input has a
     // different meaning in public API and GPU-internal API.
-    inputs_.reserve(delegate_params->input_tensors->size);
-    for (int i = 0; i < delegate_params->input_tensors->size; ++i) {
-      const int tensor_index = delegate_params->input_tensors->data[i];
-      auto* tensor = context->tensors + tensor_index;
-      if (tensor->allocation_type == TfLiteAllocationType::kTfLiteMmapRo) continue;
+    for (int tensor_index : TfLiteIntArrayView(delegate_params->input_tensors)) {
+      auto* tensor = &context->tensors[tensor_index];
+      if (IsConstantTensor(tensor)) continue;
+      // For quantized models, actual inputs of GPU graph are float tensors, so the 8-bit inputs
+      // to the delegate kernel need to be dequantized berfore feeding to the GPU graph.
+      if (options_.enable_quantization &&
+          quant_conversion_map_.find(tensor_index) != quant_conversion_map_.end()) {
+        tensor_index = quant_conversion_map_[tensor_index];
+        tensor = &context->tensors[tensor_index];
+      }
       const auto* input = find_value(tensor_index);
       if (!input || tensor->type != TfLiteType::kTfLiteFloat32) {
         return absl::NotFoundError("Input tensor is not found in the graph.");
       }
 
       inputs_.push_back(input->id);
+      input_tensor_ids_.push_back(tensor_index);
       tensor->buffer_handle = input->id;
       tensor->delegate = &delegate_;
     }
@@ -279,16 +311,23 @@ class Delegate {
     //
     // Note that graph.outputs() cannot be used directly, as the notion of graph output has a
     // different meaning in public API and GPU-internal API.
-    outputs_.reserve(delegate_params->output_tensors->size);
-    for (int i = 0; i < delegate_params->output_tensors->size; ++i) {
-      const int tensor_index = delegate_params->output_tensors->data[i];
-      auto* tensor = context->tensors + tensor_index;
+    for (int tensor_index : TfLiteIntArrayView(delegate_params->output_tensors)) {
+      auto* tensor = &context->tensors[tensor_index];
+      if (IsConstantTensor(tensor)) continue;
+      // For quantized models, actual outputs of GPU graph are float tensors, so they should be
+      // quantized to be the 8-bit outputs of delegate.
+      if (options_.enable_quantization &&
+          quant_conversion_map_.find(tensor_index) != quant_conversion_map_.end()) {
+        tensor_index = quant_conversion_map_[tensor_index];
+        tensor = &context->tensors[tensor_index];
+      }
       const auto* output = find_value(tensor_index);
       if (!output || tensor->type != TfLiteType::kTfLiteFloat32) {
         return absl::NotFoundError("Output tensor is not found in the graph.");
       }
 
       outputs_.push_back(output->id);
+      output_tensor_ids_.push_back(tensor_index);
       tensor->buffer_handle = output->id;
       tensor->delegate = &delegate_;
     }
@@ -422,12 +461,17 @@ class Delegate {
       encoder = [command_buffer computeCommandEncoder];
     }
 
+    const bool is_quantized_model = !quant_conversion_map_.empty();
+    if (is_quantized_model) {
+      RETURN_IF_ERROR(DequantizeInputs(context, input_tensor_ids_, quant_conversion_map_));
+    }
+
     // CPU HWC input data conversion to PHWC4 and fill the GPU buffer
     for (const auto& input : graph_inputs_) {
       if (input.set_externally) continue;
       // A user provides data on CPU memory for this buffer - need to copy to MTLBuffer
 
-      TfLiteTensor* tensor = context->tensors + input.tensor_id;
+      TfLiteTensor* tensor = &context->tensors[input.tensor_id];
       void* gpu_ptr = [input_output_buffers_[input.id] contents];
       std::memcpy(gpu_ptr, tensor->data.f, input.shape.DimensionsProduct() * sizeof(float));
       if (input_output_buffers_[input.id] == bphwc4_buffers_[input.id]) continue;
@@ -529,9 +573,14 @@ class Delegate {
       const void* gpu_ptr = [input_output_buffers_[output.id] contents];
       std::memcpy(tensor->data.f, gpu_ptr, output.shape.DimensionsProduct() * sizeof(float));
     }
+    if (is_quantized_model) {
+      RETURN_IF_ERROR(QuantizeOutputs(context, output_tensor_ids_, quant_conversion_map_));
+    }
     return absl::OkStatus();
   }
 
+  const TFLGpuDelegateOptions options() const { return options_; }
+
   TfLiteDelegate* tflite_delegate() { return &delegate_; }
 
  private:
@@ -551,6 +600,12 @@ class Delegate {
   std::vector<ValueRef> tensors_;  // indexed by ValueId
   std::vector<ValueId> inputs_;
   std::vector<ValueId> outputs_;
+  std::vector<int64_t> input_tensor_ids_;
+  std::vector<int64_t> output_tensor_ids_;
+  // Whenever quantized inference is enabled, this maps the tensor index of each
+  // originally quantized (8-bit) tensor to its float version added in
+  // model_builder - and vice versa.
+  std::unordered_map<int, int> quant_conversion_map_;
 
   TFLInferenceContext* inference_context_;
   // input and output buffers are passed into Metal inference engine
@@ -595,22 +650,34 @@ TfLiteStatus DelegatePrepare(TfLiteContext* context, TfLiteDelegate* delegate) {
         // forbids that.
         const auto status = metal_delegate->Prepare(context, params);
         if (status.ok()) return metal_delegate;
-        context->ReportError(context, "TfLiteGpuDelegate Prepare: %s",
-                             std::string(status.message()).c_str());
+        TF_LITE_KERNEL_LOG(context, "TfLiteMetalDelegate Prepare: %s",
+                           std::string(status.message()).c_str());
         return nullptr;
       },
       // .free
       [](TfLiteContext*, void* buffer) -> void {},
       // .prepare
       [](TfLiteContext* context, TfLiteNode* node) -> TfLiteStatus {
+        if (!node->user_data) {
+          return kTfLiteError;
+        }
+
+        auto* gpu_delegate_kernel = GetMetalDelegate(node);
+        const auto status =
+            gpu_delegate_kernel->GetRequiredTemporaries(context, node, &node->temporaries);
+        if (!status.ok()) {
+          TF_LITE_KERNEL_LOG(context, "TfLiteMetalDelegate Prepare: %s",
+                             std::string(status.message()).c_str());
+          return kTfLiteError;
+        }
         return node->user_data ? kTfLiteOk : kTfLiteError;
       },
       // .invoke
       [](TfLiteContext* context, TfLiteNode* node) -> TfLiteStatus {
         const auto status = GetMetalDelegate(node)->Invoke(context);
         if (status.ok()) return kTfLiteOk;
-        context->ReportError(context, "TfLiteMetalDelegate Invoke: %s",
-                             std::string(status.message()).c_str());
+        TF_LITE_KERNEL_LOG(context, "TfLiteMetalDelegate Invoke: %s",
+                           std::string(status.message()).c_str());
         return kTfLiteError;
       },
       nullptr,                // .profiling_string
@@ -618,7 +685,8 @@ TfLiteStatus DelegatePrepare(TfLiteContext* context, TfLiteDelegate* delegate) {
       "TfLiteMetalDelegate",  // .custom_name
       1,                      // .version
   };
-  TfLiteIntArray* ops_to_replace = GetOpsToReplace(context);
+  TfLiteIntArray* ops_to_replace =
+      GetOpsToReplace(context, GetMetalDelegate(delegate)->options().enable_quantization);
   const auto status = context->ReplaceNodeSubsetsWithDelegateKernels(context, kRegistration,
                                                                      ops_to_replace, delegate);
   TfLiteIntArrayFree(ops_to_replace);
diff --git a/tensorflow/lite/delegates/nnapi/BUILD b/tensorflow/lite/delegates/nnapi/BUILD
index 5df7b8cf427..ec9f6907f21 100644
--- a/tensorflow/lite/delegates/nnapi/BUILD
+++ b/tensorflow/lite/delegates/nnapi/BUILD
@@ -37,9 +37,6 @@ cc_library(
         "//tensorflow/lite/nnapi:nnapi_implementation",
         "//tensorflow/lite/nnapi:nnapi_lib",
         "//tensorflow/lite/nnapi:nnapi_util",
-        "@com_google_absl//absl/memory",
-        "@com_google_absl//absl/strings",
-        "@com_google_absl//absl/types:optional",
     ],
 )
 
diff --git a/tensorflow/lite/delegates/nnapi/acceleration_test_list.cc b/tensorflow/lite/delegates/nnapi/acceleration_test_list.cc
index 7800e984c7f..b20628016f0 100644
--- a/tensorflow/lite/delegates/nnapi/acceleration_test_list.cc
+++ b/tensorflow/lite/delegates/nnapi/acceleration_test_list.cc
@@ -381,6 +381,7 @@ TransposeTest/.+
 
 # transpose_conv_test
 -TransposeConvOpTest/TransposeConvOpTest.SimpleTestQuantizedPerChannelSingleChannel/0
+-TransposeConvOpTest/TransposeConvOpTest.SimpleTestQuantizedPerChannel16x8/0
 -TransposeConvOpTest/TransposeConvOpTest.TestQuantizedPerChannelMultiChannel/0
 # Const tensor only
 TransposeConvOpTest/TransposeConvOpTest/.+/0,29
diff --git a/tensorflow/lite/delegates/nnapi/nnapi_delegate.cc b/tensorflow/lite/delegates/nnapi/nnapi_delegate.cc
index 3eac83c781c..2ca4cf35ba4 100644
--- a/tensorflow/lite/delegates/nnapi/nnapi_delegate.cc
+++ b/tensorflow/lite/delegates/nnapi/nnapi_delegate.cc
@@ -43,8 +43,6 @@ limitations under the License.
 #include <unistd.h>
 #endif
 
-#include "absl/memory/memory.h"
-#include "absl/types/optional.h"
 #include "tensorflow/lite/allocation.h"
 #include "tensorflow/lite/builtin_op_data.h"
 #include "tensorflow/lite/builtin_ops.h"
@@ -3361,7 +3359,7 @@ TfLiteStatus NNAPIDelegateKernel::GetOperationsSupportedByTargetNnApiDevices(
   const auto nnapi_model_size = nnapi_to_tflite_op_mapping_.size();
 
   // Determine the list of operations the device actually supports
-  auto nnapi_ops_support_flags = absl::make_unique<bool[]>(nnapi_model_size);
+  std::unique_ptr<bool[]> nnapi_ops_support_flags(new bool[nnapi_model_size]);
 
   RETURN_TFLITE_ERROR_IF_NN_ERROR(
       context,
@@ -4152,17 +4150,16 @@ void StatefulNnApiDelegate::Data::CacheDelegateKernel(
   delegate_state_cache.emplace(cache_key, delegate_state);
 }
 
-absl::optional<NNAPIDelegateKernel*>
-StatefulNnApiDelegate::Data::GetCachedDelegateKernel(
+NNAPIDelegateKernel* StatefulNnApiDelegate::Data::MaybeGetCachedDelegateKernel(
     const TfLiteDelegateParams* delegate_params) {
   const int cache_key = delegate_params->nodes_to_replace->data[0];
   const auto cached_state = delegate_state_cache.find(cache_key);
   if (cached_state != std::end(delegate_state_cache)) {
-    auto result = absl::optional<NNAPIDelegateKernel*>(cached_state->second);
+    auto result = cached_state->second;
     delegate_state_cache.erase(cached_state);
     return result;
   } else {
-    return absl::nullopt;
+    return nullptr;
   }
 }
 
@@ -4302,7 +4299,8 @@ TfLiteStatus StatefulNnApiDelegate::GetNodesSupportedByAccelerator(
   delegate_data->delegate_state_cache.clear();
   for (int idx = 0; idx < *num_partitions; idx++) {
     const auto& partition_params = (*params_array)[idx];
-    auto kernel_state = absl::make_unique<NNAPIDelegateKernel>(nnapi);
+    std::unique_ptr<NNAPIDelegateKernel> kernel_state(
+        new NNAPIDelegateKernel(nnapi));
     TfLiteDelegateParams params_with_delegate = partition_params;
     params_with_delegate.delegate = delegate;
     TF_LITE_ENSURE_STATUS(
@@ -4471,13 +4469,9 @@ TfLiteStatus StatefulNnApiDelegate::DoPrepare(TfLiteContext* context,
         auto* delegate_data = static_cast<Data*>(params->delegate->data_);
         int* nnapi_errno = &(delegate_data->nnapi_errno);
 
-        auto delegate_state_maybe =
-            delegate_data->GetCachedDelegateKernel(params);
-
-        NNAPIDelegateKernel* kernel_state;
-        if (delegate_state_maybe.has_value()) {
-          kernel_state = *delegate_state_maybe;
-        } else {
+        NNAPIDelegateKernel* kernel_state =
+            delegate_data->MaybeGetCachedDelegateKernel(params);
+        if (!kernel_state) {
           kernel_state = new NNAPIDelegateKernel(delegate_data->nnapi);
           kernel_state->Init(context, params, nnapi_errno);
         }
diff --git a/tensorflow/lite/delegates/nnapi/nnapi_delegate.h b/tensorflow/lite/delegates/nnapi/nnapi_delegate.h
index 7ef02bc5107..27add64563d 100644
--- a/tensorflow/lite/delegates/nnapi/nnapi_delegate.h
+++ b/tensorflow/lite/delegates/nnapi/nnapi_delegate.h
@@ -20,7 +20,6 @@ limitations under the License.
 #include <unordered_map>
 #include <vector>
 
-#include "absl/types/optional.h"
 #include "tensorflow/lite/c/common.h"
 #include "tensorflow/lite/nnapi/NeuralNetworksTypes.h"
 #include "tensorflow/lite/nnapi/nnapi_implementation.h"
@@ -234,7 +233,7 @@ class StatefulNnApiDelegate : public TfLiteDelegate {
                              NNAPIDelegateKernel* delegate_state);
     // Returns a cached NNAPIDelegateKernel if available and removes it
     // from the cache transferring the ownership to the caller.
-    absl::optional<NNAPIDelegateKernel*> GetCachedDelegateKernel(
+    NNAPIDelegateKernel* MaybeGetCachedDelegateKernel(
         const TfLiteDelegateParams* delegate_params);
   };
 
diff --git a/tensorflow/lite/delegates/nnapi/nnapi_delegate_device_selection_test.cc b/tensorflow/lite/delegates/nnapi/nnapi_delegate_device_selection_test.cc
index 9a0d13af87a..2485c542296 100644
--- a/tensorflow/lite/delegates/nnapi/nnapi_delegate_device_selection_test.cc
+++ b/tensorflow/lite/delegates/nnapi/nnapi_delegate_device_selection_test.cc
@@ -39,45 +39,24 @@ limitations under the License.
 namespace tflite {
 namespace {
 
-class SingleOpModelWithNNAPI : public SingleOpModel {
- public:
-  SingleOpModelWithNNAPI() = default;
-  void Init(const NnApi* nnapi,
-            tflite::StatefulNnApiDelegate::Options options) {
-    stateful_delegate_.reset(new StatefulNnApiDelegate(nnapi, options));
-    auto* delegate = stateful_delegate_.get();
-    this->SetApplyDelegate([delegate, this](Interpreter* interpreter) {
-      compilation_status_ = interpreter->ModifyGraphWithDelegate(delegate);
-    });
-  }
-
-  StatefulNnApiDelegate* GetDelegate() { return stateful_delegate_.get(); }
-
-  void SetBufferHandle(int index, TfLiteBufferHandle handle) {
-    interpreter_->SetBufferHandle(index, handle, stateful_delegate_.get());
-  }
-  TfLiteStatus GetCompilationStatus() { return compilation_status_; }
-
- private:
-  std::unique_ptr<StatefulNnApiDelegate> stateful_delegate_;
-  TfLiteStatus compilation_status_;
-};
-
-class FloatAddOpModel : public SingleOpModelWithNNAPI {
+class FloatAddOpModel : public SingleOpModel {
  public:
   FloatAddOpModel() = default;
   void Init(const NnApi* nnapi, tflite::StatefulNnApiDelegate::Options options,
             const TensorData& input1, const TensorData& input2,
             const TensorData& output, ActivationFunctionType activation_type,
             bool allow_fp32_relax_to_fp16 = false) {
-    SingleOpModelWithNNAPI::Init(nnapi, options);
+    stateful_delegate_.reset(new StatefulNnApiDelegate(nnapi, options));
+    SetDelegate(stateful_delegate_.get());
+
     input1_ = AddInput(input1);
     input2_ = AddInput(input2);
     output_ = AddOutput(output);
     SetBuiltinOp(BuiltinOperator_ADD, BuiltinOptions_AddOptions,
                  CreateAddOptions(builder_, activation_type).Union());
-    BuildInterpreter({GetShape(input1_), GetShape(input2_)},
-                     allow_fp32_relax_to_fp16);
+    BuildInterpreter({GetShape(input1_), GetShape(input2_)}, /*num_threads=*/-1,
+                     allow_fp32_relax_to_fp16, /*apply_delegate=*/false);
+    compilation_status_ = ApplyDelegate();
   }
 
   int input1() { return input1_; }
@@ -85,12 +64,16 @@ class FloatAddOpModel : public SingleOpModelWithNNAPI {
 
   std::vector<float> GetOutput() { return ExtractVector<float>(output_); }
 
+  TfLiteStatus GetCompilationStatus() { return compilation_status_; }
+
  protected:
   int input1_;
   int input2_;
   int output_;
 
  private:
+  std::unique_ptr<StatefulNnApiDelegate> stateful_delegate_;
+  TfLiteStatus compilation_status_;
 };
 
 struct NnApiDeviceSelectionTest
@@ -281,10 +264,7 @@ class ArgMaxOpModel : public SingleOpModel, public AcceleratedModel {
 
   void Init(std::initializer_list<int> input_shape, TensorType input_type,
             int axis_value, TensorType output_type) {
-    auto* delegate = GetDelegate();
-    this->SetApplyDelegate([delegate](Interpreter* interpreter) {
-      interpreter->ModifyGraphWithDelegate(delegate);
-    });
+    SetDelegate(GetDelegate());
     input_ = AddInput(input_type);
     axis_ = AddConstInput(TensorType_INT32, {axis_value}, {1});
     output_ = AddOutput(output_type);
@@ -395,10 +375,7 @@ class AddSubOpsAcceleratedModel : public MultiOpModel, public AcceleratedModel {
                             const std::string& accelerator_name,
                             bool allow_fp32_relax_to_fp16 = false)
       : MultiOpModel(), AcceleratedModel(nnapi, accelerator_name) {
-    auto* delegate = GetDelegate();
-    this->SetApplyDelegate([delegate](Interpreter* interpreter) {
-      interpreter->ModifyGraphWithDelegate(delegate);
-    });
+    SetDelegate(GetDelegate());
     Init(input1, input2, input3, output, activation_type,
          allow_fp32_relax_to_fp16);
   }
@@ -433,7 +410,8 @@ class AddSubOpsAcceleratedModel : public MultiOpModel, public AcceleratedModel {
                  CreateSubOptions(builder_, activation_type).Union(),
                  {add_output, input3_}, {output_});
     BuildInterpreter({GetShape(input1_), GetShape(input2_), GetShape(input3_)},
-                     allow_fp32_relax_to_fp16);
+                     /*num_threads=*/-1, allow_fp32_relax_to_fp16,
+                     /*apply_delegate=*/true);
   }
 };
 
@@ -584,10 +562,7 @@ class HardSwishAddOpsAcceleratedModel : public MultiOpModel,
                                   const std::string& accelerator_name,
                                   bool allow_fp32_relax_to_fp16 = false)
       : MultiOpModel(), AcceleratedModel(nnapi, accelerator_name) {
-    auto* delegate = GetDelegate();
-    this->SetApplyDelegate([delegate](Interpreter* interpreter) {
-      interpreter->ModifyGraphWithDelegate(delegate);
-    });
+    SetDelegate(GetDelegate());
     Init(input1, input2, output, activation_type, allow_fp32_relax_to_fp16);
   }
 
@@ -616,8 +591,8 @@ class HardSwishAddOpsAcceleratedModel : public MultiOpModel,
     AddBuiltinOp(BuiltinOperator_ADD, BuiltinOptions_AddOptions,
                  CreateAddOptions(builder_, activation_type).Union(),
                  {input1_, hard_swish_output}, {output_});
-    BuildInterpreter({GetShape(input1_), GetShape(input2_)},
-                     allow_fp32_relax_to_fp16);
+    BuildInterpreter({GetShape(input1_), GetShape(input2_)}, /*num_threads=*/-1,
+                     allow_fp32_relax_to_fp16, /*apply_delegate=*/true);
   }
 };
 
@@ -723,10 +698,7 @@ class QuantizedWeightsConvolutionOpModel : public SingleOpModel,
       int dilation_width_factor = 1, int dilation_height_factor = 1,
       int num_threads = -1, std::initializer_list<uint8_t> filter_data = {})
       : SingleOpModel(), AcceleratedModel(nnapi, accelerator_name) {
-    auto* delegate = GetDelegate();
-    this->SetApplyDelegate([delegate](Interpreter* interpreter) {
-      interpreter->ModifyGraphWithDelegate(delegate);
-    });
+    SetDelegate(GetDelegate());
 
     input_ = AddInput(input);
 
@@ -749,7 +721,8 @@ class QuantizedWeightsConvolutionOpModel : public SingleOpModel,
                      .Union());
 
     BuildInterpreter({GetShape(input_), GetShape(filter_), GetShape(bias_)},
-                     num_threads);
+                     num_threads, /*allow_fp32_relax_to_fp16=*/false,
+                     /*apply_delegate=*/true);
   }
 
   void SetInput(std::initializer_list<float> data) {
@@ -859,10 +832,7 @@ class LongIdentityModel : public MultiOpModel, public AcceleratedModel {
  private:
   void Init(const std::vector<int>& input_shape, int graph_size,
             const std::unordered_set<int>& custom_nodes_indexes) {
-    auto* delegate = GetDelegate();
-    this->SetApplyDelegate([delegate](Interpreter* interpreter) {
-      interpreter->ModifyGraphWithDelegate(delegate);
-    });
+    SetDelegate(GetDelegate());
 
     const TensorData tensor_data{TensorType_FLOAT32, input_shape};
 
diff --git a/tensorflow/lite/delegates/nnapi/nnapi_delegate_disabled.cc b/tensorflow/lite/delegates/nnapi/nnapi_delegate_disabled.cc
index 2bc7ae58449..325c6233a8f 100644
--- a/tensorflow/lite/delegates/nnapi/nnapi_delegate_disabled.cc
+++ b/tensorflow/lite/delegates/nnapi/nnapi_delegate_disabled.cc
@@ -55,10 +55,9 @@ void StatefulNnApiDelegate::Data::CacheDelegateKernel(
     const TfLiteDelegateParams* delegate_params,
     NNAPIDelegateKernel* delegate_state) {}
 
-absl::optional<NNAPIDelegateKernel*>
-StatefulNnApiDelegate::Data::GetCachedDelegateKernel(
+NNAPIDelegateKernel* StatefulNnApiDelegate::Data::MaybeGetCachedDelegateKernel(
     const TfLiteDelegateParams* delegate_params) {
-  return absl::nullopt;
+  return nullptr;
 }
 
 }  // namespace tflite
diff --git a/tensorflow/lite/delegates/nnapi/nnapi_delegate_errno_test.cc b/tensorflow/lite/delegates/nnapi/nnapi_delegate_errno_test.cc
index b2c956eb309..f347799b4b8 100644
--- a/tensorflow/lite/delegates/nnapi/nnapi_delegate_errno_test.cc
+++ b/tensorflow/lite/delegates/nnapi/nnapi_delegate_errno_test.cc
@@ -31,10 +31,7 @@ class SingleOpModelWithNNAPI : public SingleOpModel {
  public:
   explicit SingleOpModelWithNNAPI(const NnApi* nnapi) {
     stateful_delegate_.reset(new StatefulNnApiDelegate(nnapi));
-    auto* delegate = stateful_delegate_.get();
-    this->SetApplyDelegate([delegate](Interpreter* interpreter) {
-      interpreter->ModifyGraphWithDelegate(delegate);
-    });
+    this->SetDelegate(stateful_delegate_.get());
   }
 
   StatefulNnApiDelegate* GetDelegate() { return stateful_delegate_.get(); }
@@ -77,8 +74,8 @@ class FloatAddOpModel : public SingleOpModelWithNNAPI {
     output_ = AddOutput(output);
     SetBuiltinOp(BuiltinOperator_ADD, BuiltinOptions_AddOptions,
                  CreateAddOptions(builder_, activation_type).Union());
-    BuildInterpreter({GetShape(input1_), GetShape(input2_)},
-                     allow_fp32_relax_to_fp16);
+    BuildInterpreter({GetShape(input1_), GetShape(input2_)}, /*num_threads=*/-1,
+                     allow_fp32_relax_to_fp16, /*apply_delegate=*/true);
   }
 };
 
diff --git a/tensorflow/lite/delegates/nnapi/nnapi_delegate_test.cc b/tensorflow/lite/delegates/nnapi/nnapi_delegate_test.cc
index acfa0c77d30..4caf5448b99 100644
--- a/tensorflow/lite/delegates/nnapi/nnapi_delegate_test.cc
+++ b/tensorflow/lite/delegates/nnapi/nnapi_delegate_test.cc
@@ -46,19 +46,12 @@ MATCHER(QuantizedNear, "") {
 
 class SingleOpModelWithNNAPI : public SingleOpModel {
  public:
-  SingleOpModelWithNNAPI() {
-    this->SetApplyDelegate([](Interpreter* interpreter) {
-      interpreter->ModifyGraphWithDelegate(NnApiDelegate());
-    });
-  }
+  SingleOpModelWithNNAPI() { SetDelegate(NnApiDelegate()); }
 
   explicit SingleOpModelWithNNAPI(
       const StatefulNnApiDelegate::Options& options) {
     stateful_delegate_.reset(new StatefulNnApiDelegate(options));
-    auto* delegate = stateful_delegate_.get();
-    this->SetApplyDelegate([delegate](Interpreter* interpreter) {
-      interpreter->ModifyGraphWithDelegate(delegate);
-    });
+    SetDelegate(stateful_delegate_.get());
   }
 
   TfLiteStatus ResizeInputTensor(int tensor_index,
@@ -154,8 +147,8 @@ class FloatAddOpModel : public SingleOpModelWithNNAPI {
     output_ = AddOutput(output);
     SetBuiltinOp(BuiltinOperator_ADD, BuiltinOptions_AddOptions,
                  CreateAddOptions(builder_, activation_type).Union());
-    BuildInterpreter({GetShape(input1_), GetShape(input2_)},
-                     allow_fp32_relax_to_fp16);
+    BuildInterpreter({GetShape(input1_), GetShape(input2_)}, /*num_threads=*/-1,
+                     allow_fp32_relax_to_fp16, /*apply_delegate=*/true);
   }
 };
 
diff --git a/tensorflow/lite/delegates/utils.cc b/tensorflow/lite/delegates/utils.cc
index 135b4d531f9..873cadc180f 100644
--- a/tensorflow/lite/delegates/utils.cc
+++ b/tensorflow/lite/delegates/utils.cc
@@ -150,74 +150,112 @@ TfLiteStatus GraphPartitionHelper::PrepareSupportedNodes(
   return kTfLiteOk;
 }
 
-TfLiteStatus FP16GraphPartitionHelper::Partition(
-    std::set<std::string>* unsupported_nodes_info) {
-  const auto status = GraphPartitionHelper::Partition(unsupported_nodes_info);
-  // Clean up those partitions that have a single dequant op. NoteThose
-  // removed dequant ops have to be reserved in the graph and should not be
-  // delegated.
-  RemoveSingleDequantNodePartitions();
-  return status;
-}
-
 std::vector<int>
 FP16GraphPartitionHelper::GetNodesOfFirstNLargestPartitionsImpl(
     int n, int min_nodes_per_partition) {
-  std::vector<int> ops_to_replace =
-      GraphPartitionHelper::GetNodesOfFirstNLargestPartitionsImpl(
-          n, min_nodes_per_partition);
-  RemapInputTensors(ops_to_replace);
-  RemoveReservedDequantsFromNodes(&ops_to_replace);
+  auto first_n_partitions =
+      GetFirstNLargestPartitions(n, min_nodes_per_partition);
+  std::vector<int> ops_to_replace;
+  if (first_n_partitions.empty()) return ops_to_replace;
+
+  // Handle the first delegated partition specially.
+  // All fp16 DEQUANTIZE nodes whose consumers exist only in this partition can
+  // be added to the ops to delegate. Others have to be preserved in the graph,
+  // since the partitioning algorithm will put such nodes greedily in the first
+  // partition.
+  const auto* first_partition = first_n_partitions[0];
+  std::unordered_map<int, int> delegated_dequant_consumers;
+  for (int i = 0; i < first_partition->nodes_to_replace->size; ++i) {
+    const int node_id = first_partition->nodes_to_replace->data[i];
+    ops_to_replace.push_back(node_id);
+    TfLiteNode* node;
+    TfLiteRegistration* registration;
+    const auto status = context_->GetNodeAndRegistration(context_, node_id,
+                                                         &node, &registration);
+    if (status != kTfLiteOk) {
+      TF_LITE_KERNEL_LOG(context_,
+                         "Couldn't get node and registration info for op: %d\n",
+                         node_id);
+      ops_to_replace.clear();
+      return ops_to_replace;
+    }
+    // See if any input to the op is a (converted) fp16 value. If yes, increment
+    // its value in delegated_dequant_consumers.
+    for (int j = 0; j < node->inputs->size; ++j) {
+      const int input_tid = node->inputs->data[j];
+      if (dequant_consumers_.find(input_tid) != dequant_consumers_.end()) {
+        delegated_dequant_consumers[input_tid] += 1;
+      }
+    }
+  }
+  // Check all dequant nodes that have some consumers in the first partition.
+  // If the number of delegated consumers is same as total number of consumers,
+  // add the corresponding DEQUANTIZE op to the delegated nodes.
+  for (auto tensor_and_consumers : delegated_dequant_consumers) {
+    if (dequant_consumers_[tensor_and_consumers.first] ==
+        tensor_and_consumers.second) {
+      ops_to_replace.emplace_back(dequant_nodes_[tensor_and_consumers.first]);
+    }
+  }
+
+  // For all other partitions after the first one, insert all nodes into
+  // ops_to_replace.
+  for (int i = 1; i < first_n_partitions.size(); ++i) {
+    auto nodes = first_n_partitions[i]->nodes_to_replace;
+    ops_to_replace.insert(ops_to_replace.end(), nodes->data,
+                          nodes->data + nodes->size);
+  }
+
+  // Modify the inputs of relevant ops that support fp16 constants.
+  // TODO(b/156707497): Ensure that these inputs are remapped during the
+  // delegate's 'free', so that CPU fallback works for fp16 models.
+  RemapFp16InputTensors(ops_to_replace);
   return ops_to_replace;
 }
 
 bool FP16GraphPartitionHelper::IsNodeSupported(
     TfLiteContext* context, TfLiteNode* node, TfLiteRegistration* registration,
     int node_id, std::string* unsupported_details) {
-  // If we need to handle dequant nodes, we have to remap input tensors of
-  // this node if some of them come from a dequant node before testing if
-  // the node is supported.
-  std::vector<int> orig_inputs;
-  if (RecordAndRemapInputTensors(registration->builtin_code, node_id, node,
-                                 &orig_inputs)) {
-    // We have a dequant op here. Note that we retrun an Ok status because a
-    // dequant node is first added as supported. Later, this dequant node
-    // will be removed if it has to be preserved in the graph which happens
-    // when its immediate downstream nodes cannot be supported.
-    return true;
-  }
-  const auto status = GraphPartitionHelper::IsNodeSupported(
-      context, node, registration, node_id, unsupported_details);
-  RestoreToOrigInputTensors(node, orig_inputs);
-  return status;
-}
-
-bool FP16GraphPartitionHelper::RecordAndRemapInputTensors(
-    int32_t op_code, int node_id, TfLiteNode* node,
-    std::vector<int>* orig_inputs) {
-  orig_inputs->clear();
-  // Record the dequant node.
-  if (op_code == kTfLiteBuiltinDequantize &&
+  if (registration->builtin_code == kTfLiteBuiltinDequantize &&
       context_->tensors[node->inputs->data[0]].type ==
           TfLiteType::kTfLiteFloat16) {
-    dequant_nodes_[node->outputs->data[0]] = node->inputs->data[0];
-    return true;
+    // Update mappings if this node is a fp16 DEQUANTIZE node.
+    dequant_map_[node->outputs->data[0]] = node->inputs->data[0];
+    dequant_nodes_[node->outputs->data[0]] = node_id;
+    // We do not accept these ops right now.
+    // This is done to support use-cases where a DEQUANTIZE output might be
+    // consumed by a CPU op.
+    return false;
   }
-  // For a dequantize op, there's no need to remap its input tensors.
-  if (dequant_nodes_.empty()) return false;
-  RemapInputTensors(node, orig_inputs);
-  return false;
+
+  // To check if a (possibly) FP16 node is supported, we temporarily point the
+  // node's inputs to the original fp16 tensors. This 'mutated' node is then
+  // passed to the base IsNodeSupported function for checking. After the check,
+  // we remap the original node inputs, so that the TFLite graph remains the
+  // same.
+  std::vector<int> orig_inputs;
+  if (!dequant_nodes_.empty()) {
+    RemapFp16InputTensors(node, &orig_inputs);
+  }
+
+  const auto is_supported = GraphPartitionHelper::IsNodeSupported(
+      context, node, registration, node_id, unsupported_details);
+
+  if (!orig_inputs.empty() && node->inputs->size == orig_inputs.size()) {
+    // Remapping happened. Restore original inputs.
+    for (int j = 0; j < node->inputs->size; ++j) {
+      node->inputs->data[j] = orig_inputs[j];
+      if (dequant_nodes_.find(orig_inputs[j]) != dequant_nodes_.end()) {
+        // If its a fp16 tensor, increment number of consumers of the
+        // corresponding DEQUANTIZE.
+        dequant_consumers_[orig_inputs[j]] += 1;
+      }
+    }
+  }
+  return is_supported;
 }
 
-void FP16GraphPartitionHelper::RestoreToOrigInputTensors(
-    TfLiteNode* node, const std::vector<int>& orig_inputs) {
-  if (node->inputs->size != orig_inputs.size()) return;
-  for (int j = 0; j < node->inputs->size; ++j) {
-    node->inputs->data[j] = orig_inputs[j];
-  }
-}
-
-void FP16GraphPartitionHelper::RemapInputTensors(
+void FP16GraphPartitionHelper::RemapFp16InputTensors(
     const std::vector<int>& nodes) const {
   for (int node_id : nodes) {
     TfLiteNode* node;
@@ -229,56 +267,11 @@ void FP16GraphPartitionHelper::RemapInputTensors(
                          "Couldn't get node and registration info for op: %d\n",
                          node_id);
     }
-    RemapInputTensors(node, nullptr /* orig_inputs*/);
+    RemapFp16InputTensors(node, nullptr /* orig_inputs*/);
   }
 }
 
-void FP16GraphPartitionHelper::RemoveSingleDequantNodePartitions() {
-  auto it = partitions_.begin();
-  while (it != partitions_.end()) {
-    auto p = *it;
-    if (p->nodes_to_replace->size != 1) {
-      ++it;
-      continue;
-    }
-    int node_id = p->nodes_to_replace->data[0];
-    TfLiteNode* node = nullptr;
-    TfLiteRegistration* registration = nullptr;
-
-    TfLiteStatus status = context_->GetNodeAndRegistration(
-        context_, node_id, &node, &registration);
-    if (status != kTfLiteOk) {
-      TF_LITE_KERNEL_LOG(context_,
-                         "Couldn't get node and registration info for op: %d\n",
-                         node_id);
-    }
-    if (registration->builtin_code != kTfLiteBuiltinDequantize ||
-        context_->tensors[node->inputs->data[0]].type !=
-            TfLiteType::kTfLiteFloat16) {
-      ++it;
-      continue;
-    }
-    // Note such dequant nodes have to be preserved in the graph as dequant
-    // ops are not actually supported in the GPU delegate.
-    dequant_nodes_to_save_.insert(node_id);
-    it = partitions_.erase(it);
-  }
-}
-
-void FP16GraphPartitionHelper::RemoveReservedDequantsFromNodes(
-    std::vector<int>* nodes) {
-  if (dequant_nodes_to_save_.empty()) return;
-  auto it = nodes->begin();
-  while (it != nodes->end()) {
-    if (dequant_nodes_to_save_.find(*it) == dequant_nodes_to_save_.end()) {
-      ++it;
-      continue;
-    }
-    it = nodes->erase(it);
-  }
-}
-
-void FP16GraphPartitionHelper::RemapInputTensors(
+void FP16GraphPartitionHelper::RemapFp16InputTensors(
     TfLiteNode* node, std::vector<int>* orig_inputs) const {
   TfLiteIntArray* inputs = node->inputs;
   auto inputs_view = TfLiteIntArrayView(inputs);
@@ -296,8 +289,8 @@ void FP16GraphPartitionHelper::RemapInputTensors(
   bool is_remapped = false;
   for (int j = 0; j < inputs->size; ++j) {
     const int input_tid = inputs->data[j];
-    const auto it = dequant_nodes_.find(input_tid);
-    if (it != dequant_nodes_.end()) {
+    const auto it = dequant_map_.find(input_tid);
+    if (it != dequant_map_.end()) {
       inputs->data[j] = it->second;
       is_remapped = true;
     }
diff --git a/tensorflow/lite/delegates/utils.h b/tensorflow/lite/delegates/utils.h
index 6b498b908f9..12684fcb84a 100644
--- a/tensorflow/lite/delegates/utils.h
+++ b/tensorflow/lite/delegates/utils.h
@@ -127,19 +127,23 @@ class GraphPartitionHelper {
   TfLiteIntArray* supported_nodes_ = nullptr;  // owns the memory
 };
 
-// While partitioning the graph, this claims DEQUANTIZE nodes (FP16->FP32) in
-// addition to supported nodes for the delegate, when the DEQUANTIZE node's
-// output is an input to the kernel that supports FP16 input.
+// Specialized partitioner for graphs that possibly contain fp16 tensors.
+//
+// From nodes that accept fp16 inputs, this delegates the following:
+// 1. All nodes (except DEQUANTIZE) that are supported with fp16 inputs by the
+// delegate (in the TFLite graph, these nodes take in dequantized FP32
+// outputs).
+// 2. All fp16 DEQUANTIZE nodes that have *all* their consumers in the *first*
+// delegated partition. This is because TFLite's partitioning algorithm
+// greedily puts all such nodes in the first partition.
 class FP16GraphPartitionHelper : public GraphPartitionHelper {
  public:
   FP16GraphPartitionHelper(TfLiteContext* context,
                            IsNodeSupportedFn is_node_supported_fn)
       : GraphPartitionHelper(context, std::move(is_node_supported_fn)) {}
 
-  TfLiteStatus Partition(
-      std::set<std::string>* unsupported_nodes_info) override;
-
  protected:
+  // Specialized function to handle fp16 nodes.
   bool IsNodeSupported(TfLiteContext* context, TfLiteNode* node,
                        TfLiteRegistration* registration, int node_id,
                        std::string* unsupported_details) override;
@@ -149,39 +153,25 @@ class FP16GraphPartitionHelper : public GraphPartitionHelper {
       int n, int min_nodes_per_partition) override;
 
  private:
-  // Record 'node' if it is a dequant op (i.e. a fp16 one here) and return true.
-  // When it's not a dequant op, remap its inputs to the inputs of the preceding
-  // dequant if there's a one and returns false. 'orig_inputs' records original
-  // input tensor ids of this node if any input is remapped.
-  bool RecordAndRemapInputTensors(int32_t op_code, int node_id,
-                                  TfLiteNode* node,
-                                  std::vector<int>* orig_inputs);
+  // This remaps fp32 inputs of the given node to their corresponding fp16
+  // version, if applicable. Can be summarized as:
+  // fp16 -> DEQUANTIZE -> fp32 -> OP -> output
+  // becomes
+  // fp16 -> OP -> output
+  void RemapFp16InputTensors(TfLiteNode* node,
+                             std::vector<int>* orig_inputs) const;
 
-  // Restore inputs of 'node' to 'orig_inputs' only if two sizes match.
-  void RestoreToOrigInputTensors(TfLiteNode* node,
-                                 const std::vector<int>& orig_inputs);
+  // Performs the above remapping for all nodes in the given list, without
+  // tracking the original inputs.
+  void RemapFp16InputTensors(const std::vector<int>& nodes) const;
 
-  // Remap input tensors of every node in 'nodes' (i.e. node indices) if some of
-  // them are from dequant ops.
-  void RemapInputTensors(const std::vector<int>& nodes) const;
-
-  void RemoveSingleDequantNodePartitions();
-
-  void RemoveReservedDequantsFromNodes(std::vector<int>* nodes);
-
-  // Remap input tensors of a single 'node' if some of come from a dequant op.
-  // If 'orig_inputs' isn't nullptr, it records original input tensor ids of
-  // this node if any input is remapped.
-  void RemapInputTensors(TfLiteNode* node, std::vector<int>* orig_inputs) const;
-
-  // A map recording dequantize nodes's input/output tensors of this selected
-  // graph. The key is the output tensor id, and the value is the input tensor
-  // id.
+  // ('dequantize' here refers to fp16 DEQUANTIZE)
+  // Mapping of dequantize nodes' output tensor-id to its node id.
   std::unordered_map<int, int> dequant_nodes_;
-
-  // A set of dequant nodes as in node indices that have to be preserved in the
-  // graph.
-  std::set<int> dequant_nodes_to_save_;
+  // Mapping of DEQUANTIZE node's output (fp32) to its input (fp16).
+  std::unordered_map<int, int> dequant_map_;
+  // mapping of DEQUANTIZE output tensor-id to its number of consumers.
+  std::unordered_map<int, int> dequant_consumers_;
 };
 
 }  // namespace delegates
diff --git a/tensorflow/lite/delegates/xnnpack/BUILD b/tensorflow/lite/delegates/xnnpack/BUILD
index 880145e51eb..455da14e8fe 100644
--- a/tensorflow/lite/delegates/xnnpack/BUILD
+++ b/tensorflow/lite/delegates/xnnpack/BUILD
@@ -254,6 +254,21 @@ cc_test(
     ],
 )
 
+cc_test(
+    name = "div_test",
+    srcs = ["div_test.cc"],
+    linkopts = select({
+        "//tensorflow:emscripten": EMSCRIPTEN_LINKOPTS,
+        "//conditions:default": [],
+    }),
+    deps = [
+        ":binary_elementwise_tester",
+        ":test_main",
+        ":xnnpack_delegate_test_mode",
+        "@com_google_googletest//:gtest",
+    ],
+)
+
 cc_test(
     name = "fully_connected_test",
     srcs = ["fully_connected_test.cc"],
@@ -314,6 +329,36 @@ cc_test(
     ],
 )
 
+cc_test(
+    name = "maximum_test",
+    srcs = ["maximum_test.cc"],
+    linkopts = select({
+        "//tensorflow:emscripten": EMSCRIPTEN_LINKOPTS,
+        "//conditions:default": [],
+    }),
+    deps = [
+        ":binary_elementwise_tester",
+        ":test_main",
+        ":xnnpack_delegate_test_mode",
+        "@com_google_googletest//:gtest",
+    ],
+)
+
+cc_test(
+    name = "minimum_test",
+    srcs = ["minimum_test.cc"],
+    linkopts = select({
+        "//tensorflow:emscripten": EMSCRIPTEN_LINKOPTS,
+        "//conditions:default": [],
+    }),
+    deps = [
+        ":binary_elementwise_tester",
+        ":test_main",
+        ":xnnpack_delegate_test_mode",
+        "@com_google_googletest//:gtest",
+    ],
+)
+
 cc_test(
     name = "mul_test",
     srcs = ["mul_test.cc"],
@@ -404,4 +449,34 @@ cc_test(
     ],
 )
 
+cc_test(
+    name = "squared_difference_test",
+    srcs = ["squared_difference_test.cc"],
+    linkopts = select({
+        "//tensorflow:emscripten": EMSCRIPTEN_LINKOPTS,
+        "//conditions:default": [],
+    }),
+    deps = [
+        ":binary_elementwise_tester",
+        ":test_main",
+        ":xnnpack_delegate_test_mode",
+        "@com_google_googletest//:gtest",
+    ],
+)
+
+cc_test(
+    name = "sub_test",
+    srcs = ["sub_test.cc"],
+    linkopts = select({
+        "//tensorflow:emscripten": EMSCRIPTEN_LINKOPTS,
+        "//conditions:default": [],
+    }),
+    deps = [
+        ":binary_elementwise_tester",
+        ":test_main",
+        ":xnnpack_delegate_test_mode",
+        "@com_google_googletest//:gtest",
+    ],
+)
+
 tflite_portable_test_suite_combined(combine_conditions = {"deps": [":test_main"]})
diff --git a/tensorflow/lite/delegates/xnnpack/README.md b/tensorflow/lite/delegates/xnnpack/README.md
index 98a08a4f647..c50ea60a88e 100644
--- a/tensorflow/lite/delegates/xnnpack/README.md
+++ b/tensorflow/lite/delegates/xnnpack/README.md
@@ -116,6 +116,12 @@ Below is the list of current operators and limitations:
 * Fused `NONE`, `RELU`, `RELU_N1_TO_1`, and `RELU6` activations are supported,
   but fused `TANH` and `SIGN_BIT` activations are not.
 
+### `DIV`
+
+* Inputs and outputs must be in 32-bit floating-point format.
+* Fused `NONE`, `RELU`, `RELU_N1_TO_1`, and `RELU6` activations are supported,
+  but fused `TANH` and `SIGN_BIT` activations are not.
+
 ### `FULLY_CONNECTED`
 
 * Inputs and outputs must be in 32-bit floating-point format.
@@ -139,6 +145,14 @@ Below is the list of current operators and limitations:
 * Fused `NONE`, `RELU`, `RELU_N1_TO_1`, and `RELU6` activations are supported,
   but fused `TANH` and `SIGN_BIT` activations are not.
 
+### `MAXIMUM`
+
+* Inputs and outputs must be in 32-bit floating-point format.
+
+### `MINIMUM`
+
+* Inputs and outputs must be in 32-bit floating-point format.
+
 ### `MUL`
 
 * Inputs and outputs must be in 32-bit floating-point format.
@@ -176,6 +190,16 @@ Below is the list of current operators and limitations:
 * Inputs and outputs must be in 32-bit floating-point format.
 * Only `beta = 1.0` is supported.
 
+### `SQUARED_DIFFERENCE`
+
+* Inputs and outputs must be in 32-bit floating-point format.
+
+### `SUB`
+
+* Inputs and outputs must be in 32-bit floating-point format.
+* Fused `NONE`, `RELU`, `RELU_N1_TO_1`, and `RELU6` activations are supported,
+  but fused `TANH` and `SIGN_BIT` activations are not.
+
 ### Other limitations
 
 * Dynamically allocated (with `kTfLiteDynamic` allocation type) inputs and
diff --git a/tensorflow/lite/delegates/xnnpack/div_test.cc b/tensorflow/lite/delegates/xnnpack/div_test.cc
new file mode 100644
index 00000000000..2328a3237da
--- /dev/null
+++ b/tensorflow/lite/delegates/xnnpack/div_test.cc
@@ -0,0 +1,840 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include <cstdint>
+#include <functional>
+#include <memory>
+#include <random>
+
+#include <gtest/gtest.h>
+#include "tensorflow/lite/delegates/xnnpack/binary_elementwise_tester.h"
+#include "tensorflow/lite/delegates/xnnpack/xnnpack_delegate.h"
+
+namespace tflite {
+namespace xnnpack {
+
+TEST(Div, 4DBy4D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 4DBy4DBroadcastChannels) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, 1, 1, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, 1, 1, channels})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 4DBy4DBroadcastWidth) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, 1, width, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, 1, width, 1})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 4DBy4DBroadcastHeight) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, height, 1, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, height, 1, 1})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 4DBy4DBroadcastBatch) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, 1, 1, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, 1, 1, 1})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 4DBy4DBroadcastHeightWidthChannels) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, height, width, channels})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 4DBy3D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({height, width, channels})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 4DBy2D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({width, channels})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 4DBy1D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({channels})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 4DBy0D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 2DBy2D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({batch, channels})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 2DBy1D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({channels})
+      .Input2Shape({batch, channels})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({channels})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 2DBy0D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({})
+      .Input2Shape({batch, channels})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 4DByStatic4D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 4DByStatic4DBroadcastChannels) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, 1, 1, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, 1, 1, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 4DByStatic4DBroadcastWidth) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, 1, width, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, 1, width, 1})
+      .Input2Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 4DByStatic4DBroadcastHeight) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, height, 1, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, height, 1, 1})
+      .Input2Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 4DByStatic4DBroadcastBatch) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, 1, 1, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, 1, 1, 1})
+      .Input2Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 4DByStatic4DBroadcastHeightWidthChannels) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, height, width, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 4DByStatic3D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({height, width, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 4DByStatic2D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({width, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 4DByStatic1D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 4DByStatic0D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({})
+      .Input2Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 2DByStatic2D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({batch, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({batch, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 2DByStatic1D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({channels})
+      .Input2Shape({batch, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, 2DByStatic0D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({})
+      .Input2Shape({batch, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({})
+      .Input2Static(true)
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, FP16Weights) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .FP16Weights()
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input2Static(true)
+      .FP16Weights()
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, ReluActivation) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .ReluActivation()
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, Relu6Activation) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Relu6Activation()
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, ReluMinus1To1Activation) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .ReluMinus1To1Activation()
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, DISABLED_TanhActivation) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .TanhActivation()
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, DISABLED_SignBitActivation) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .SignBitActivation()
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+TEST(Div, MultiThreading) {
+  TfLiteXNNPackDelegateOptions delegate_options =
+      TfLiteXNNPackDelegateOptionsDefault();
+  delegate_options.num_threads = 2;
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(&delegate_options),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_DIV, xnnpack_delegate.get());
+}
+
+}  // namespace xnnpack
+}  // namespace tflite
diff --git a/tensorflow/lite/delegates/xnnpack/maximum_test.cc b/tensorflow/lite/delegates/xnnpack/maximum_test.cc
new file mode 100644
index 00000000000..0bdb770bbe9
--- /dev/null
+++ b/tensorflow/lite/delegates/xnnpack/maximum_test.cc
@@ -0,0 +1,735 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include <cstdint>
+#include <functional>
+#include <memory>
+#include <random>
+
+#include <gtest/gtest.h>
+#include "tensorflow/lite/delegates/xnnpack/binary_elementwise_tester.h"
+#include "tensorflow/lite/delegates/xnnpack/xnnpack_delegate.h"
+
+namespace tflite {
+namespace xnnpack {
+
+TEST(Maximum, 4DBy4D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 4DBy4DBroadcastChannels) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, 1, 1, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, 1, 1, channels})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 4DBy4DBroadcastWidth) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, 1, width, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, 1, width, 1})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 4DBy4DBroadcastHeight) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, height, 1, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, height, 1, 1})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 4DBy4DBroadcastBatch) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, 1, 1, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, 1, 1, 1})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 4DBy4DBroadcastHeightWidthChannels) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, height, width, channels})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 4DBy3D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({height, width, channels})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 4DBy2D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({width, channels})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 4DBy1D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({channels})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 4DBy0D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 2DBy2D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({batch, channels})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 2DBy1D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({channels})
+      .Input2Shape({batch, channels})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({channels})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 2DBy0D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({})
+      .Input2Shape({batch, channels})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 4DByStatic4D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 4DByStatic4DBroadcastChannels) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, 1, 1, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, 1, 1, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 4DByStatic4DBroadcastWidth) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, 1, width, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, 1, width, 1})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 4DByStatic4DBroadcastHeight) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, height, 1, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, height, 1, 1})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 4DByStatic4DBroadcastBatch) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, 1, 1, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, 1, 1, 1})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 4DByStatic4DBroadcastHeightWidthChannels) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, height, width, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 4DByStatic3D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({height, width, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 4DByStatic2D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({width, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 4DByStatic1D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 4DByStatic0D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 2DByStatic2D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({batch, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({batch, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 2DByStatic1D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({channels})
+      .Input2Shape({batch, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, 2DByStatic0D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({})
+      .Input2Shape({batch, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, FP16Weights) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .FP16Weights()
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input2Static(true)
+      .FP16Weights()
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+TEST(Maximum, MultiThreading) {
+  TfLiteXNNPackDelegateOptions delegate_options =
+      TfLiteXNNPackDelegateOptionsDefault();
+  delegate_options.num_threads = 2;
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(&delegate_options),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MAXIMUM, xnnpack_delegate.get());
+}
+
+}  // namespace xnnpack
+}  // namespace tflite
diff --git a/tensorflow/lite/delegates/xnnpack/minimum_test.cc b/tensorflow/lite/delegates/xnnpack/minimum_test.cc
new file mode 100644
index 00000000000..6bfbe9511b2
--- /dev/null
+++ b/tensorflow/lite/delegates/xnnpack/minimum_test.cc
@@ -0,0 +1,735 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include <cstdint>
+#include <functional>
+#include <memory>
+#include <random>
+
+#include <gtest/gtest.h>
+#include "tensorflow/lite/delegates/xnnpack/binary_elementwise_tester.h"
+#include "tensorflow/lite/delegates/xnnpack/xnnpack_delegate.h"
+
+namespace tflite {
+namespace xnnpack {
+
+TEST(Minimum, 4DBy4D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 4DBy4DBroadcastChannels) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, 1, 1, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, 1, 1, channels})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 4DBy4DBroadcastWidth) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, 1, width, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, 1, width, 1})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 4DBy4DBroadcastHeight) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, height, 1, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, height, 1, 1})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 4DBy4DBroadcastBatch) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, 1, 1, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, 1, 1, 1})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 4DBy4DBroadcastHeightWidthChannels) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, height, width, channels})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 4DBy3D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({height, width, channels})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 4DBy2D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({width, channels})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 4DBy1D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({channels})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 4DBy0D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 2DBy2D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({batch, channels})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 2DBy1D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({channels})
+      .Input2Shape({batch, channels})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({channels})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 2DBy0D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({})
+      .Input2Shape({batch, channels})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 4DByStatic4D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 4DByStatic4DBroadcastChannels) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, 1, 1, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, 1, 1, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 4DByStatic4DBroadcastWidth) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, 1, width, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, 1, width, 1})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 4DByStatic4DBroadcastHeight) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, height, 1, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, height, 1, 1})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 4DByStatic4DBroadcastBatch) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, 1, 1, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, 1, 1, 1})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 4DByStatic4DBroadcastHeightWidthChannels) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, height, width, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 4DByStatic3D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({height, width, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 4DByStatic2D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({width, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 4DByStatic1D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 4DByStatic0D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 2DByStatic2D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({batch, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({batch, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 2DByStatic1D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({channels})
+      .Input2Shape({batch, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, 2DByStatic0D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({})
+      .Input2Shape({batch, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({})
+      .Input2Static(true)
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, FP16Weights) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .FP16Weights()
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input2Static(true)
+      .FP16Weights()
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+TEST(Minimum, MultiThreading) {
+  TfLiteXNNPackDelegateOptions delegate_options =
+      TfLiteXNNPackDelegateOptionsDefault();
+  delegate_options.num_threads = 2;
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(&delegate_options),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_MINIMUM, xnnpack_delegate.get());
+}
+
+}  // namespace xnnpack
+}  // namespace tflite
diff --git a/tensorflow/lite/delegates/xnnpack/squared_difference_test.cc b/tensorflow/lite/delegates/xnnpack/squared_difference_test.cc
new file mode 100644
index 00000000000..f610dc790df
--- /dev/null
+++ b/tensorflow/lite/delegates/xnnpack/squared_difference_test.cc
@@ -0,0 +1,735 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include <cstdint>
+#include <functional>
+#include <memory>
+#include <random>
+
+#include <gtest/gtest.h>
+#include "tensorflow/lite/delegates/xnnpack/binary_elementwise_tester.h"
+#include "tensorflow/lite/delegates/xnnpack/xnnpack_delegate.h"
+
+namespace tflite {
+namespace xnnpack {
+
+TEST(SquaredDifference, 4DBy4D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 4DBy4DBroadcastChannels) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, 1, 1, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, 1, 1, channels})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 4DBy4DBroadcastWidth) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, 1, width, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, 1, width, 1})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 4DBy4DBroadcastHeight) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, height, 1, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, height, 1, 1})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 4DBy4DBroadcastBatch) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, 1, 1, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, 1, 1, 1})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 4DBy4DBroadcastHeightWidthChannels) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, height, width, channels})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 4DBy3D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({height, width, channels})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 4DBy2D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({width, channels})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 4DBy1D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({channels})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 4DBy0D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 2DBy2D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({batch, channels})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 2DBy1D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({channels})
+      .Input2Shape({batch, channels})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({channels})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 2DBy0D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({})
+      .Input2Shape({batch, channels})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 4DByStatic4D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 4DByStatic4DBroadcastChannels) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, 1, 1, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, 1, 1, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 4DByStatic4DBroadcastWidth) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, 1, width, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, 1, width, 1})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 4DByStatic4DBroadcastHeight) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, height, 1, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, height, 1, 1})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 4DByStatic4DBroadcastBatch) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, 1, 1, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, 1, 1, 1})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 4DByStatic4DBroadcastHeightWidthChannels) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, height, width, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 4DByStatic3D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({height, width, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 4DByStatic2D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({width, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 4DByStatic1D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 4DByStatic0D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 2DByStatic2D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({batch, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({batch, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 2DByStatic1D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({channels})
+      .Input2Shape({batch, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, 2DByStatic0D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({})
+      .Input2Shape({batch, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, FP16Weights) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .FP16Weights()
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input2Static(true)
+      .FP16Weights()
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+TEST(SquaredDifference, MultiThreading) {
+  TfLiteXNNPackDelegateOptions delegate_options =
+      TfLiteXNNPackDelegateOptionsDefault();
+  delegate_options.num_threads = 2;
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(&delegate_options),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SQUARED_DIFFERENCE, xnnpack_delegate.get());
+}
+
+}  // namespace xnnpack
+}  // namespace tflite
diff --git a/tensorflow/lite/delegates/xnnpack/sub_test.cc b/tensorflow/lite/delegates/xnnpack/sub_test.cc
new file mode 100644
index 00000000000..8f3a50d3625
--- /dev/null
+++ b/tensorflow/lite/delegates/xnnpack/sub_test.cc
@@ -0,0 +1,840 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include <cstdint>
+#include <functional>
+#include <memory>
+#include <random>
+
+#include <gtest/gtest.h>
+#include "tensorflow/lite/delegates/xnnpack/binary_elementwise_tester.h"
+#include "tensorflow/lite/delegates/xnnpack/xnnpack_delegate.h"
+
+namespace tflite {
+namespace xnnpack {
+
+TEST(Sub, 4DBy4D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 4DBy4DBroadcastChannels) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, 1, 1, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, 1, 1, channels})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 4DBy4DBroadcastWidth) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, 1, width, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, 1, width, 1})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 4DBy4DBroadcastHeight) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, height, 1, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, height, 1, 1})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 4DBy4DBroadcastBatch) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, 1, 1, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, 1, 1, 1})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 4DBy4DBroadcastHeightWidthChannels) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, height, width, channels})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 4DBy3D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({height, width, channels})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 4DBy2D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({width, channels})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 4DBy1D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({channels})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 4DBy0D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 2DBy2D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({batch, channels})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 2DBy1D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({channels})
+      .Input2Shape({batch, channels})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({channels})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 2DBy0D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({})
+      .Input2Shape({batch, channels})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 4DByStatic4D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 4DByStatic4DBroadcastChannels) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, 1, 1, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, 1, 1, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 4DByStatic4DBroadcastWidth) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, 1, width, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, 1, width, 1})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 4DByStatic4DBroadcastHeight) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, height, 1, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, height, 1, 1})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 4DByStatic4DBroadcastBatch) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, 1, 1, 1})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, 1, 1, 1})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 4DByStatic4DBroadcastHeightWidthChannels) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({1, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({1, height, width, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 4DByStatic3D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({height, width, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 4DByStatic2D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({width, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 4DByStatic1D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 4DByStatic0D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 2DByStatic2D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({batch, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({batch, channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 2DByStatic1D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({channels})
+      .Input2Shape({batch, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({channels})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, 2DByStatic0D) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({})
+      .Input2Shape({batch, channels})
+      .Input1Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, channels})
+      .Input2Shape({})
+      .Input2Static(true)
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, FP16Weights) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input1Static(true)
+      .FP16Weights()
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Input2Static(true)
+      .FP16Weights()
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, ReluActivation) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .ReluActivation()
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, Relu6Activation) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Relu6Activation()
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, ReluMinus1To1Activation) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .ReluMinus1To1Activation()
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, DISABLED_TanhActivation) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .TanhActivation()
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, DISABLED_SignBitActivation) {
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(nullptr),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .SignBitActivation()
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+TEST(Sub, MultiThreading) {
+  TfLiteXNNPackDelegateOptions delegate_options =
+      TfLiteXNNPackDelegateOptionsDefault();
+  delegate_options.num_threads = 2;
+  std::unique_ptr<TfLiteDelegate, decltype(&TfLiteXNNPackDelegateDelete)>
+      xnnpack_delegate(TfLiteXNNPackDelegateCreate(&delegate_options),
+                       TfLiteXNNPackDelegateDelete);
+
+  std::random_device random_device;
+  auto rng = std::mt19937(random_device());
+  auto shape_rng =
+      std::bind(std::uniform_int_distribution<int32_t>(2, 5), std::ref(rng));
+  const auto batch = shape_rng();
+  const auto height = shape_rng();
+  const auto width = shape_rng();
+  const auto channels = shape_rng();
+
+  BinaryElementwiseTester()
+      .Input1Shape({batch, height, width, channels})
+      .Input2Shape({batch, height, width, channels})
+      .Test(BuiltinOperator_SUB, xnnpack_delegate.get());
+}
+
+}  // namespace xnnpack
+}  // namespace tflite
diff --git a/tensorflow/lite/delegates/xnnpack/xnnpack_delegate.cc b/tensorflow/lite/delegates/xnnpack/xnnpack_delegate.cc
index 9cbdea60706..fe39ac24d52 100644
--- a/tensorflow/lite/delegates/xnnpack/xnnpack_delegate.cc
+++ b/tensorflow/lite/delegates/xnnpack/xnnpack_delegate.cc
@@ -797,6 +797,13 @@ class Subgraph {
                                         node, context->tensors, dwconv_params,
                                         quasi_static_tensors, xnnpack_tensors);
       }
+      case kTfLiteBuiltinDiv: {
+        const TfLiteDivParams* div_params =
+            static_cast<const TfLiteDivParams*>(node->builtin_data);
+
+        return VisitDivNode(subgraph, logging_context, node_index, node,
+                            context->tensors, div_params, xnnpack_tensors);
+      }
       case kTfLiteBuiltinFullyConnected: {
         const TfLiteFullyConnectedParams* fc_params =
             static_cast<const TfLiteFullyConnectedParams*>(node->builtin_data);
@@ -819,6 +826,12 @@ class Subgraph {
                                   context->tensors, pool_params,
                                   xnnpack_tensors);
       }
+      case kTfLiteBuiltinMaximum:
+        return VisitMaximumNode(subgraph, logging_context, node_index, node,
+                                context->tensors, xnnpack_tensors);
+      case kTfLiteBuiltinMinimum:
+        return VisitMinimumNode(subgraph, logging_context, node_index, node,
+                                context->tensors, xnnpack_tensors);
       case kTfLiteBuiltinMul: {
         const TfLiteMulParams* mul_params =
             static_cast<const TfLiteMulParams*>(node->builtin_data);
@@ -851,6 +864,17 @@ class Subgraph {
                                 context->tensors, softmax_params,
                                 xnnpack_tensors);
       }
+      case kTfLiteBuiltinSquaredDifference:
+        return VisitSquaredDifferenceNode(subgraph, logging_context, node_index,
+                                          node, context->tensors,
+                                          xnnpack_tensors);
+      case kTfLiteBuiltinSub: {
+        const TfLiteSubParams* sub_params =
+            static_cast<const TfLiteSubParams*>(node->builtin_data);
+
+        return VisitSubNode(subgraph, logging_context, node_index, node,
+                            context->tensors, sub_params, xnnpack_tensors);
+      }
       case kTfLiteBuiltinCustom: {
         if (strcmp(registration->custom_name, "Convolution2DTransposeBias") ==
             0) {
@@ -1179,6 +1203,56 @@ class Subgraph {
     return kTfLiteOk;
   }
 
+  static TfLiteStatus VisitDivNode(
+      xnn_subgraph_t subgraph, TfLiteContext* logging_context, int node_index,
+      TfLiteNode* node, const TfLiteTensor* tensors,
+      const TfLiteDivParams* div_params,
+      const std::vector<uint32_t>& xnnpack_tensors) {
+    TF_LITE_ENSURE_STATUS(
+        CheckNumInputsAndOutputs(logging_context, node, 2, 1, node_index));
+
+    const TfLiteTensor& input1_tensor = tensors[node->inputs->data[0]];
+    TF_LITE_ENSURE_STATUS(CheckTensorFloatType(
+        logging_context, input1_tensor, node->inputs->data[0], node_index));
+    TF_LITE_ENSURE_STATUS(CheckTensorNonDynamicAllocation(
+        logging_context, input1_tensor, node->inputs->data[0], node_index));
+
+    const TfLiteTensor& input2_tensor = tensors[node->inputs->data[1]];
+    TF_LITE_ENSURE_STATUS(CheckTensorFloatType(
+        logging_context, input2_tensor, node->inputs->data[1], node_index));
+    TF_LITE_ENSURE_STATUS(CheckTensorNonDynamicAllocation(
+        logging_context, input2_tensor, node->inputs->data[1], node_index));
+
+    const TfLiteTensor& output_tensor = tensors[node->outputs->data[0]];
+    TF_LITE_ENSURE_STATUS(CheckTensorFloatType(
+        logging_context, output_tensor, node->outputs->data[0], node_index));
+    TF_LITE_ENSURE_STATUS(CheckTensorNonDynamicAllocation(
+        logging_context, output_tensor, node->outputs->data[0], node_index));
+
+    float output_min = -std::numeric_limits<float>::infinity();
+    float output_max = +std::numeric_limits<float>::infinity();
+    if (div_params != nullptr) {
+      TF_LITE_ENSURE_STATUS(ConvertActivationToOutputRange(
+          logging_context, node_index, div_params->activation, &output_min,
+          &output_max));
+    }
+
+    if (subgraph != nullptr) {
+      const xnn_status status = xnn_define_divide(
+          subgraph, output_min, output_max,
+          /*input1_id=*/xnnpack_tensors[node->inputs->data[0]],
+          /*input2_id=*/xnnpack_tensors[node->inputs->data[1]],
+          /*output_id=*/xnnpack_tensors[node->outputs->data[0]], /*flags=*/0);
+      if (status != xnn_status_success) {
+        TF_LITE_KERNEL_LOG(logging_context, "failed to delegate DIV node #%d",
+                           node_index);
+        return kTfLiteError;
+      }
+    }
+
+    return kTfLiteOk;
+  }
+
   static TfLiteStatus VisitFullyConnectedNode(
       xnn_subgraph_t subgraph, TfLiteContext* logging_context, int node_index,
       TfLiteNode* node, const TfLiteTensor* tensors,
@@ -1448,6 +1522,46 @@ class Subgraph {
     return kTfLiteOk;
   }
 
+  static TfLiteStatus VisitMaximumNode(
+      xnn_subgraph_t subgraph, TfLiteContext* logging_context, int node_index,
+      TfLiteNode* node, const TfLiteTensor* tensors,
+      const std::vector<uint32_t>& xnnpack_tensors) {
+    TF_LITE_ENSURE_STATUS(
+        CheckNumInputsAndOutputs(logging_context, node, 2, 1, node_index));
+
+    const TfLiteTensor& input1_tensor = tensors[node->inputs->data[0]];
+    TF_LITE_ENSURE_STATUS(CheckTensorFloatType(
+        logging_context, input1_tensor, node->inputs->data[0], node_index));
+    TF_LITE_ENSURE_STATUS(CheckTensorNonDynamicAllocation(
+        logging_context, input1_tensor, node->inputs->data[0], node_index));
+
+    const TfLiteTensor& input2_tensor = tensors[node->inputs->data[1]];
+    TF_LITE_ENSURE_STATUS(CheckTensorFloatType(
+        logging_context, input2_tensor, node->inputs->data[1], node_index));
+    TF_LITE_ENSURE_STATUS(CheckTensorNonDynamicAllocation(
+        logging_context, input2_tensor, node->inputs->data[1], node_index));
+
+    const TfLiteTensor& output_tensor = tensors[node->outputs->data[0]];
+    TF_LITE_ENSURE_STATUS(CheckTensorFloatType(
+        logging_context, output_tensor, node->outputs->data[0], node_index));
+    TF_LITE_ENSURE_STATUS(CheckTensorNonDynamicAllocation(
+        logging_context, output_tensor, node->outputs->data[0], node_index));
+
+    if (subgraph != nullptr) {
+      const xnn_status status = xnn_define_maximum2(
+          subgraph, /*input1_id=*/xnnpack_tensors[node->inputs->data[0]],
+          /*input2_id=*/xnnpack_tensors[node->inputs->data[1]],
+          /*output_id=*/xnnpack_tensors[node->outputs->data[0]], /*flags=*/0);
+      if (status != xnn_status_success) {
+        TF_LITE_KERNEL_LOG(logging_context,
+                           "failed to delegate MAXIMUM node #%d", node_index);
+        return kTfLiteError;
+      }
+    }
+
+    return kTfLiteOk;
+  }
+
   static TfLiteStatus VisitMediaPipeDeconvolutionNode(
       xnn_subgraph_t subgraph, TfLiteContext* logging_context, int node_index,
       TfLiteNode* node, const TfLiteTensor* tensors,
@@ -1672,6 +1786,46 @@ class Subgraph {
     return kTfLiteOk;
   }
 
+  static TfLiteStatus VisitMinimumNode(
+      xnn_subgraph_t subgraph, TfLiteContext* logging_context, int node_index,
+      TfLiteNode* node, const TfLiteTensor* tensors,
+      const std::vector<uint32_t>& xnnpack_tensors) {
+    TF_LITE_ENSURE_STATUS(
+        CheckNumInputsAndOutputs(logging_context, node, 2, 1, node_index));
+
+    const TfLiteTensor& input1_tensor = tensors[node->inputs->data[0]];
+    TF_LITE_ENSURE_STATUS(CheckTensorFloatType(
+        logging_context, input1_tensor, node->inputs->data[0], node_index));
+    TF_LITE_ENSURE_STATUS(CheckTensorNonDynamicAllocation(
+        logging_context, input1_tensor, node->inputs->data[0], node_index));
+
+    const TfLiteTensor& input2_tensor = tensors[node->inputs->data[1]];
+    TF_LITE_ENSURE_STATUS(CheckTensorFloatType(
+        logging_context, input2_tensor, node->inputs->data[1], node_index));
+    TF_LITE_ENSURE_STATUS(CheckTensorNonDynamicAllocation(
+        logging_context, input2_tensor, node->inputs->data[1], node_index));
+
+    const TfLiteTensor& output_tensor = tensors[node->outputs->data[0]];
+    TF_LITE_ENSURE_STATUS(CheckTensorFloatType(
+        logging_context, output_tensor, node->outputs->data[0], node_index));
+    TF_LITE_ENSURE_STATUS(CheckTensorNonDynamicAllocation(
+        logging_context, output_tensor, node->outputs->data[0], node_index));
+
+    if (subgraph != nullptr) {
+      const xnn_status status = xnn_define_minimum2(
+          subgraph, /*input1_id=*/xnnpack_tensors[node->inputs->data[0]],
+          /*input2_id=*/xnnpack_tensors[node->inputs->data[1]],
+          /*output_id=*/xnnpack_tensors[node->outputs->data[0]], /*flags=*/0);
+      if (status != xnn_status_success) {
+        TF_LITE_KERNEL_LOG(logging_context,
+                           "failed to delegate MINIMUM node #%d", node_index);
+        return kTfLiteError;
+      }
+    }
+
+    return kTfLiteOk;
+  }
+
   static TfLiteStatus VisitMulNode(
       xnn_subgraph_t subgraph, TfLiteContext* logging_context, int node_index,
       TfLiteNode* node, const TfLiteTensor* tensors,
@@ -1928,6 +2082,97 @@ class Subgraph {
     return kTfLiteOk;
   }
 
+  static TfLiteStatus VisitSquaredDifferenceNode(
+      xnn_subgraph_t subgraph, TfLiteContext* logging_context, int node_index,
+      TfLiteNode* node, const TfLiteTensor* tensors,
+      const std::vector<uint32_t>& xnnpack_tensors) {
+    TF_LITE_ENSURE_STATUS(
+        CheckNumInputsAndOutputs(logging_context, node, 2, 1, node_index));
+
+    const TfLiteTensor& input1_tensor = tensors[node->inputs->data[0]];
+    TF_LITE_ENSURE_STATUS(CheckTensorFloatType(
+        logging_context, input1_tensor, node->inputs->data[0], node_index));
+    TF_LITE_ENSURE_STATUS(CheckTensorNonDynamicAllocation(
+        logging_context, input1_tensor, node->inputs->data[0], node_index));
+
+    const TfLiteTensor& input2_tensor = tensors[node->inputs->data[1]];
+    TF_LITE_ENSURE_STATUS(CheckTensorFloatType(
+        logging_context, input2_tensor, node->inputs->data[1], node_index));
+    TF_LITE_ENSURE_STATUS(CheckTensorNonDynamicAllocation(
+        logging_context, input2_tensor, node->inputs->data[1], node_index));
+
+    const TfLiteTensor& output_tensor = tensors[node->outputs->data[0]];
+    TF_LITE_ENSURE_STATUS(CheckTensorFloatType(
+        logging_context, output_tensor, node->outputs->data[0], node_index));
+    TF_LITE_ENSURE_STATUS(CheckTensorNonDynamicAllocation(
+        logging_context, output_tensor, node->outputs->data[0], node_index));
+
+    if (subgraph != nullptr) {
+      const xnn_status status = xnn_define_squared_difference(
+          subgraph, /*input1_id=*/xnnpack_tensors[node->inputs->data[0]],
+          /*input2_id=*/xnnpack_tensors[node->inputs->data[1]],
+          /*output_id=*/xnnpack_tensors[node->outputs->data[0]], /*flags=*/0);
+      if (status != xnn_status_success) {
+        TF_LITE_KERNEL_LOG(logging_context,
+                           "failed to delegate SQUARED_DIFFERENCE node #%d",
+                           node_index);
+        return kTfLiteError;
+      }
+    }
+
+    return kTfLiteOk;
+  }
+
+  static TfLiteStatus VisitSubNode(
+      xnn_subgraph_t subgraph, TfLiteContext* logging_context, int node_index,
+      TfLiteNode* node, const TfLiteTensor* tensors,
+      const TfLiteSubParams* sub_params,
+      const std::vector<uint32_t>& xnnpack_tensors) {
+    TF_LITE_ENSURE_STATUS(
+        CheckNumInputsAndOutputs(logging_context, node, 2, 1, node_index));
+
+    const TfLiteTensor& input1_tensor = tensors[node->inputs->data[0]];
+    TF_LITE_ENSURE_STATUS(CheckTensorFloatType(
+        logging_context, input1_tensor, node->inputs->data[0], node_index));
+    TF_LITE_ENSURE_STATUS(CheckTensorNonDynamicAllocation(
+        logging_context, input1_tensor, node->inputs->data[0], node_index));
+
+    const TfLiteTensor& input2_tensor = tensors[node->inputs->data[1]];
+    TF_LITE_ENSURE_STATUS(CheckTensorFloatType(
+        logging_context, input2_tensor, node->inputs->data[1], node_index));
+    TF_LITE_ENSURE_STATUS(CheckTensorNonDynamicAllocation(
+        logging_context, input2_tensor, node->inputs->data[1], node_index));
+
+    const TfLiteTensor& output_tensor = tensors[node->outputs->data[0]];
+    TF_LITE_ENSURE_STATUS(CheckTensorFloatType(
+        logging_context, output_tensor, node->outputs->data[0], node_index));
+    TF_LITE_ENSURE_STATUS(CheckTensorNonDynamicAllocation(
+        logging_context, output_tensor, node->outputs->data[0], node_index));
+
+    float output_min = -std::numeric_limits<float>::infinity();
+    float output_max = +std::numeric_limits<float>::infinity();
+    if (sub_params != nullptr) {
+      TF_LITE_ENSURE_STATUS(ConvertActivationToOutputRange(
+          logging_context, node_index, sub_params->activation, &output_min,
+          &output_max));
+    }
+
+    if (subgraph != nullptr) {
+      const xnn_status status = xnn_define_subtract(
+          subgraph, output_min, output_max,
+          /*input1_id=*/xnnpack_tensors[node->inputs->data[0]],
+          /*input2_id=*/xnnpack_tensors[node->inputs->data[1]],
+          /*output_id=*/xnnpack_tensors[node->outputs->data[0]], /*flags=*/0);
+      if (status != xnn_status_success) {
+        TF_LITE_KERNEL_LOG(logging_context, "failed to delegate SUB node #%d",
+                           node_index);
+        return kTfLiteError;
+      }
+    }
+
+    return kTfLiteOk;
+  }
+
  private:
   Subgraph(xnn_runtime_t runtime, std::unordered_set<int>&& externals)
       : runtime_(runtime, &xnn_delete_runtime), externals_(externals) {}
diff --git a/tensorflow/lite/examples/python/README.md b/tensorflow/lite/examples/python/README.md
index e59715488cc..82b7ad690fc 100644
--- a/tensorflow/lite/examples/python/README.md
+++ b/tensorflow/lite/examples/python/README.md
@@ -18,7 +18,7 @@ make sure you [have TensorFlow installed](https://www.tensorflow.org/install).
 You can use any compatible model, but the following MobileNet v1 model offers
 a good demonstration of a model trained to recognize 1,000 different objects.
 
-```
+```sh
 # Get photo
 curl https://raw.githubusercontent.com/tensorflow/tensorflow/master/tensorflow/lite/examples/label_image/testdata/grace_hopper.bmp > /tmp/grace_hopper.bmp
 # Get model
@@ -33,7 +33,7 @@ mv /tmp/mobilenet_v1_1.0_224/labels.txt /tmp/
 
 Note: Instead use `python` if you're using Python 2.x.
 
-```
+```sh
 python3 label_image.py \
   --model_file /tmp/mobilenet_v1_1.0_224.tflite \
   --label_file /tmp/labels.txt \
diff --git a/tensorflow/lite/experimental/acceleration/README.md b/tensorflow/lite/experimental/acceleration/README.md
new file mode 100644
index 00000000000..c3209fe99e9
--- /dev/null
+++ b/tensorflow/lite/experimental/acceleration/README.md
@@ -0,0 +1,15 @@
+# Accelerator whitelisting
+
+Experimental library and tools for determining whether an accelerator engine
+works well on a given device, and for a given model.
+
+## Platform-agnostic, Android-first
+
+Android-focused, since the much smaller set of configurations on iOS means there
+is much less need for whitelisting on iOS.
+
+## Not just for TfLite
+
+This code lives in the TfLite codebase, since TfLite is the first open-source
+customer. It is however meant to support other users (direct use of NNAPI,
+mediapipe).
diff --git a/tensorflow/lite/experimental/acceleration/whitelist/BUILD b/tensorflow/lite/experimental/acceleration/whitelist/BUILD
new file mode 100644
index 00000000000..97959010f16
--- /dev/null
+++ b/tensorflow/lite/experimental/acceleration/whitelist/BUILD
@@ -0,0 +1,163 @@
+# Copyright 2019 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+load("@flatbuffers//:build_defs.bzl", "flatbuffer_cc_library")
+load("//tensorflow/lite:special_rules.bzl", "tflite_portable_test_suite")
+
+package(
+    default_visibility = [
+        "//visibility:public",
+    ],
+    licenses = ["notice"],  # Apache 2.0
+)
+
+flatbuffer_cc_library(
+    name = "database_fbs",
+    srcs = ["database.fbs"],
+)
+
+cc_library(
+    name = "devicedb",
+    srcs = [
+        "devicedb.cc",
+    ],
+    hdrs = [
+        "devicedb.h",
+        "variables.h",
+    ],
+    deps = [
+        ":database_fbs",
+    ],
+)
+
+cc_binary(
+    name = "json_to_fb",
+    srcs = ["json_to_fb.cc"],
+    deps = [
+        "//tensorflow/lite/tools:command_line_flags",
+        "@flatbuffers",
+    ],
+)
+
+genrule(
+    name = "devicedb-sample_bin",
+    srcs = [
+        "database.fbs",
+        "devicedb-sample.json",
+    ],
+    outs = ["devicedb-sample.bin"],
+    cmd = """
+    $(location :json_to_fb) \
+        --fbs=$(location :database.fbs) \
+        --json_input=$(location :devicedb-sample.json) \
+        --fb_output=$(@)
+    """,
+    tools = [":json_to_fb"],
+)
+
+py_binary(
+    name = "convert_binary_to_cc_source",
+    srcs = ["convert_binary_to_cc_source.py"],
+    python_version = "PY3",
+    srcs_version = "PY2AND3",
+    visibility = ["//visibility:public"],
+)
+
+genrule(
+    name = "devicedb-sample_cc",
+    srcs = ["devicedb-sample.bin"],
+    outs = [
+        "devicedb-sample.cc",
+        "devicedb-sample.h",
+    ],
+    # convert_file_to_c_source for some reason doesn't define the global with
+    # 'extern', which is needed for global const variables in C++.
+    cmd = """
+    $(location :convert_binary_to_cc_source) \
+        --input_binary_file $(location :devicedb-sample.bin) \
+        --output_header_file $(location :devicedb-sample.h) \
+        --output_source_file $(location :devicedb-sample.cc) \
+        --array_variable_name g_tflite_acceleration_devicedb_sample_binary
+    """,
+    tools = [":convert_binary_to_cc_source"],
+)
+
+cc_test(
+    name = "devicedb_test",
+    srcs = [
+        "devicedb-sample.cc",
+        "devicedb-sample.h",
+        "devicedb_test.cc",
+    ],
+    deps = [
+        ":database_fbs",
+        ":devicedb",
+        "//tensorflow/lite/testing:util",
+        "@com_google_googletest//:gtest",
+        "@flatbuffers",
+    ],
+)
+
+genrule(
+    name = "gpu_whitelist_binary",
+    srcs = ["gpu_whitelist.bin"],
+    outs = [
+        "gpu_whitelist_binary.h",
+        "gpu_whitelist_binary.cc",
+    ],
+    # convert_file_to_c_source for some reason doesn't define the global with
+    # 'extern', which is needed for global const variables in C++.
+    cmd = """
+    $(location :convert_binary_to_cc_source) \
+        --input_binary_file $(location :gpu_whitelist.bin) \
+        --output_header_file $(location :gpu_whitelist_binary.h) \
+        --output_source_file $(location :gpu_whitelist_binary.cc) \
+        --array_variable_name g_tflite_acceleration_gpu_whitelist_binary
+    """,
+    tools = [":convert_binary_to_cc_source"],
+)
+
+cc_library(
+    name = "android_info",
+    srcs = ["android_info.cc"],
+    hdrs = ["android_info.h"],
+    deps = [
+        "@com_google_absl//absl/status",
+    ],
+)
+
+cc_library(
+    name = "gpu_whitelist",
+    srcs = [
+        "gpu_whitelist.cc",
+        "gpu_whitelist_binary.cc",
+        "gpu_whitelist_binary.h",
+    ],
+    hdrs = [
+        "gpu_whitelist.h",
+    ],
+    deps = [
+        ":android_info",
+        ":database_fbs",
+        ":devicedb",
+        "//tensorflow/lite/delegates/gpu:delegate",
+        "//tensorflow/lite/delegates/gpu/common:gpu_info",
+        "@com_google_absl//absl/status",
+        "@com_google_absl//absl/strings",
+        "@flatbuffers",
+    ],
+)
+
+tflite_portable_test_suite()
diff --git a/tensorflow/lite/experimental/acceleration/whitelist/README.md b/tensorflow/lite/experimental/acceleration/whitelist/README.md
new file mode 100644
index 00000000000..24ee794aef6
--- /dev/null
+++ b/tensorflow/lite/experimental/acceleration/whitelist/README.md
@@ -0,0 +1,13 @@
+# GPU delegate whitelist
+
+This package provides data and code for deciding if the GPU delegate is
+supported on a specific Android device.
+
+## Customizing the GPU whitelist
+
+-   Convert from checked-in flatbuffer to json by running `flatc -t --raw-binary
+    --strict-json database.fbs -- gpu_whitelist.bin`
+-   Edit the json
+-   Convert from json to flatbuffer `flatc -b database.fbs --
+    gpu_whitelist.json`
+-   Rebuild ../../../java:tensorflow-lite-gpu
diff --git a/tensorflow/lite/experimental/acceleration/whitelist/android_info.cc b/tensorflow/lite/experimental/acceleration/whitelist/android_info.cc
new file mode 100644
index 00000000000..4618ac90807
--- /dev/null
+++ b/tensorflow/lite/experimental/acceleration/whitelist/android_info.cc
@@ -0,0 +1,52 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+#include "tensorflow/lite/experimental/acceleration/whitelist/android_info.h"
+
+#include <string>
+
+#include "absl/status/status.h"
+
+#ifdef __ANDROID__
+#include <sys/system_properties.h>
+#endif  // __ANDROID__
+
+namespace {
+std::string GetPropertyValue(const std::string& property) {
+#ifdef __ANDROID__
+  char value[PROP_VALUE_MAX];
+  __system_property_get(property.c_str(), value);
+  return std::string(value);
+#else   // !__ANDROID__
+  return std::string();
+#endif  // __ANDROID__
+}
+}  // namespace
+
+namespace tflite {
+namespace acceleration {
+
+absl::Status RequestAndroidInfo(AndroidInfo* info_out) {
+  if (!info_out) {
+    return absl::InvalidArgumentError("info_out may not be null");
+  }
+  info_out->android_sdk_version = GetPropertyValue("ro.build.version.sdk");
+  info_out->device = GetPropertyValue("ro.product.device");
+  info_out->model = GetPropertyValue("ro.product.model");
+  info_out->manufacturer = GetPropertyValue("ro.product.manufacturer");
+  return absl::OkStatus();
+}
+
+}  // namespace acceleration
+}  // namespace tflite
diff --git a/tensorflow/lite/experimental/acceleration/whitelist/android_info.h b/tensorflow/lite/experimental/acceleration/whitelist/android_info.h
new file mode 100644
index 00000000000..81b3ee7479c
--- /dev/null
+++ b/tensorflow/lite/experimental/acceleration/whitelist/android_info.h
@@ -0,0 +1,43 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+#ifndef TENSORFLOW_LITE_EXPERIMENTAL_ACCELERATION_WHITELIST_ANDROID_INFO_H_
+#define TENSORFLOW_LITE_EXPERIMENTAL_ACCELERATION_WHITELIST_ANDROID_INFO_H_
+
+#include <string>
+
+#include "absl/status/status.h"
+
+namespace tflite {
+namespace acceleration {
+
+// Information about and Android device, used for determining whitelisting
+// status.
+struct AndroidInfo {
+  // Property ro.build.version.sdk
+  std::string android_sdk_version;
+  // Property ro.product.model
+  std::string model;
+  // Property ro.product.device
+  std::string device;
+  // Property ro.product.manufacturer
+  std::string manufacturer;
+};
+
+absl::Status RequestAndroidInfo(AndroidInfo* info_out);
+
+}  // namespace acceleration
+}  // namespace tflite
+
+#endif  // TENSORFLOW_LITE_EXPERIMENTAL_ACCELERATION_WHITELIST_ANDROID_INFO_H_
diff --git a/tensorflow/lite/experimental/acceleration/whitelist/convert_binary_to_cc_source.py b/tensorflow/lite/experimental/acceleration/whitelist/convert_binary_to_cc_source.py
new file mode 100644
index 00000000000..092e206c37e
--- /dev/null
+++ b/tensorflow/lite/experimental/acceleration/whitelist/convert_binary_to_cc_source.py
@@ -0,0 +1,195 @@
+# Lint as: python2, python3
+# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Simple script to convert binary file to C++ source code for embedding."""
+
+# This is a version of //tensorflow/lite/python/convert_file_to_c_source.py
+# with minimal dependencies to reduce build times. See b/158254039.
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import argparse
+import datetime
+import sys
+
+
+# Cribbed from //tensorflow/lite/python/util.py
+# Changed:
+# - Alignment from 4 to 16 for generality (16 can be required for SIMD)
+# - Added 'extern' to source for building on C++ target platforms
+# - Changed comments to refer to this script, and C++ rather than C
+def _convert_bytes_to_cc_source(data,
+                                array_name,
+                                max_line_width=80,
+                                include_guard=None,
+                                include_path=None,
+                                use_tensorflow_license=False):
+  """Returns strings representing a C++ constant array containing `data`.
+
+  Args:
+    data: Byte array that will be converted into a C++ constant.
+    array_name: String to use as the variable name for the constant array.
+    max_line_width: The longest line length, for formatting purposes.
+    include_guard: Name to use for the include guard macro definition.
+    include_path: Optional path to include in the source file.
+    use_tensorflow_license: Whether to include the standard TensorFlow Apache2
+      license in the generated files.
+
+  Returns:
+    Text that can be compiled as a C++ source file to link in the data as a
+    literal array of values.
+    Text that can be used as a C++ header file to reference the literal array.
+  """
+
+  starting_pad = "   "
+  array_lines = []
+  array_line = starting_pad
+  for value in bytearray(data):
+    if (len(array_line) + 4) > max_line_width:
+      array_lines.append(array_line + "\n")
+      array_line = starting_pad
+    array_line += " 0x%02x," % value
+  if len(array_line) > len(starting_pad):
+    array_lines.append(array_line + "\n")
+  array_values = "".join(array_lines)
+
+  if include_guard is None:
+    include_guard = "TENSORFLOW_LITE_UTIL_" + array_name.upper() + "_DATA_H_"
+
+  if include_path is not None:
+    include_line = "#include \"{include_path}\"\n".format(
+        include_path=include_path)
+  else:
+    include_line = ""
+
+  if use_tensorflow_license:
+    license_text = """
+/* Copyright {year} The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+""".format(year=datetime.date.today().year)
+  else:
+    license_text = ""
+
+  source_template = """{license_text}
+// This is a binary file that has been converted into a C++ data array using the
+// //tensorflow/lite/experimental/acceleration/whitelist/convert_binary_to_cc_source.py
+// script. This form is useful for compiling into a binary to simplify
+// deployment on mobile devices
+
+{include_line}
+// We need to keep the data array aligned on some architectures.
+#ifdef __has_attribute
+#define HAVE_ATTRIBUTE(x) __has_attribute(x)
+#else
+#define HAVE_ATTRIBUTE(x) 0
+#endif
+#if HAVE_ATTRIBUTE(aligned) || (defined(__GNUC__) && !defined(__clang__))
+#define DATA_ALIGN_ATTRIBUTE __attribute__((aligned(16)))
+#else
+#define DATA_ALIGN_ATTRIBUTE
+#endif
+
+extern const unsigned char {array_name}[] DATA_ALIGN_ATTRIBUTE = {{
+{array_values}}};
+extern const int {array_name}_len = {array_length};
+"""
+
+  source_text = source_template.format(
+      array_name=array_name,
+      array_length=len(data),
+      array_values=array_values,
+      license_text=license_text,
+      include_line=include_line)
+
+  header_template = """
+{license_text}
+
+// This is a binary file that has been converted into a C++ data array using the
+// //tensorflow/lite/experimental/acceleration/whitelist/convert_binary_to_cc_source.py
+// script. This form is useful for compiling into a binary to simplify
+// deployment on mobile devices
+
+#ifndef {include_guard}
+#define {include_guard}
+
+extern const unsigned char {array_name}[];
+extern const int {array_name}_len;
+
+#endif  // {include_guard}
+"""
+
+  header_text = header_template.format(
+      array_name=array_name,
+      include_guard=include_guard,
+      license_text=license_text)
+
+  return source_text, header_text
+
+
+def main():
+  parser = argparse.ArgumentParser(
+      description=("Binary to C++ source converter"))
+  parser.add_argument(
+      "--input_binary_file",
+      type=str,
+      help="Full filepath of input binary.",
+      required=True)
+  parser.add_argument(
+      "--output_header_file",
+      type=str,
+      help="Full filepath of output header.",
+      required=True)
+  parser.add_argument(
+      "--array_variable_name",
+      type=str,
+      help="Full filepath of output source.",
+      required=True)
+  parser.add_argument(
+      "--output_source_file",
+      type=str,
+      help="Name of global variable that will contain the binary data.",
+      required=True)
+  flags, _ = parser.parse_known_args(args=sys.argv[1:])
+  with open(flags.input_binary_file, "rb") as input_handle:
+    input_data = input_handle.read()
+
+  source, header = _convert_bytes_to_cc_source(
+      data=input_data,
+      array_name=flags.array_variable_name,
+      use_tensorflow_license=True)
+
+  with open(flags.output_source_file, "w") as source_handle:
+    source_handle.write(source)
+
+  with open(flags.output_header_file, "w") as header_handle:
+    header_handle.write(header)
+
+
+if __name__ == "__main__":
+  main()
diff --git a/tensorflow/lite/experimental/acceleration/whitelist/database.fbs b/tensorflow/lite/experimental/acceleration/whitelist/database.fbs
new file mode 100644
index 00000000000..6340fcfcf3a
--- /dev/null
+++ b/tensorflow/lite/experimental/acceleration/whitelist/database.fbs
@@ -0,0 +1,58 @@
+// Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+namespace tflite.acceleration;
+
+enum Comparison : byte {
+  EQUAL = 0,
+  MINIMUM = 1,
+}
+
+// Mapping from available device features to whitelisting decisions. Basic usage is to:
+// 1) Map easily available device data (like Android version,
+// Manufacturer, Device) to things like SoC vendor, SoC model.
+// 2) Map complete device data to delegate-specific features and support status
+// 3) Map delegate-specific features to delegate configuration.
+//
+// The structure describes a decision tree, with multiple matching branches.
+// The branches are applied depth-first.
+table DeviceDatabase {
+  root:[tflite.acceleration.DeviceDecisionTreeNode];
+}
+
+table DeviceDecisionTreeNode {
+  // The variables are strings, as we have multiple clients that want to
+  // introduce their own fields. Known variables are listed in variables.h.
+  variable:string (shared);
+  comparison:tflite.acceleration.Comparison;
+  items:[tflite.acceleration.DeviceDecisionTreeEdge];
+}
+
+table DeviceDecisionTreeEdge {
+  // Under which variable value does this item match.
+  value:string (key, shared);
+  // Which child branches should also be consulted and used to override this
+  // node.
+  children:[tflite.acceleration.DeviceDecisionTreeNode];
+  // What information can be derived about this device.
+  derived_properties:[tflite.acceleration.DerivedProperty];
+}
+
+// Derived variable value to combine with detected variables.
+table DerivedProperty {
+  variable:string (shared);
+  value:string (shared);
+}
+
+root_type DeviceDatabase;
diff --git a/tensorflow/lite/experimental/acceleration/whitelist/devicedb-sample.json b/tensorflow/lite/experimental/acceleration/whitelist/devicedb-sample.json
new file mode 100644
index 00000000000..187989673d1
--- /dev/null
+++ b/tensorflow/lite/experimental/acceleration/whitelist/devicedb-sample.json
@@ -0,0 +1,169 @@
+{
+  "root": [
+    {
+      "variable": "tflite.device_model",
+      "items": [
+        {
+          "value": "m712c",
+          "derived_properties": [
+            {
+              "variable": "tflite.soc_model",
+              "value": "exynos_7872"
+            }
+          ]
+        },
+        {
+          "value": "sc_02l",
+          "derived_properties": [
+            {
+              "variable": "tflite.soc_model",
+              "value": "exynos_7885"
+            }
+          ]
+        }
+      ]
+    },
+    {
+      "variable": "tflite.opengl_es_version",
+      "items": [
+        {
+          "value": "3.1",
+          "children": [
+            {
+              "variable": "tflite.soc_model",
+              "items": [
+                {
+                  "value": "exynos_7872",
+                  "children": [
+                    {
+                      "variable": "tflite.android_sdk_version",
+                      "items": [
+                        {
+                          "value": "24",
+                          "derived_properties": [
+                            {
+                              "variable": "tflite.gpu.status",
+                              "value": "WHITELISTED"
+                            }
+                          ]
+                        }
+                      ],
+                      "comparison": "MINIMUM"
+                    }
+                  ]
+                },
+                {
+                  "value": "exynos_7883",
+                  "children": [
+                    {
+                      "variable": "tflite.android_sdk_version",
+                      "items": [
+                        {
+                          "value": "28",
+                          "derived_properties": [
+                            {
+                              "variable": "tflite.gpu.status",
+                              "value": "WHITELISTED"
+                            }
+                          ]
+                        }
+                      ],
+                      "comparison": "MINIMUM"
+                    }
+                  ]
+                }
+              ]
+            }
+
+          ]
+        }
+      ]
+    },
+    {
+      "variable": "tflite.android_sdk_version",
+      "items": [
+        {
+          "value": "21",
+          "children": [
+            {
+              "variable": "tflite.device_model",
+              "items": [
+                {
+                  "value": "huawei_gra_l09",
+                  "children": [
+                    {
+                      "variable": "tflite.device_name",
+                      "items": [
+                        {
+                          "value": "hwgra",
+                          "derived_properties": [
+                            {
+                              "variable": "tflite.gpu.status",
+                              "value": "WHITELISTED"
+                            },
+                            {
+                              "variable": "tflite.gpu.opencl_status",
+                              "value": "WHITELISTED"
+                            }
+                          ]
+                        }
+                      ]
+                    }
+                  ]
+                }
+              ]
+            }
+          ]
+        },
+        {
+          "value": "24",
+          "children": [
+            {
+              "variable": "tflite.device_model",
+              "items": [
+                  {
+                  "value": "sm_j810f",
+                  "children": [
+                    {
+                      "variable": "tflite.device_name",
+                      "items": [
+                        {
+                          "value": "j8y18lte",
+                          "derived_properties": [
+                            {
+                              "variable": "tflite.gpu.status",
+                              "value": "BLACKLISTED"
+                            }
+                          ]
+                        }
+                      ]
+                    }
+                  ]
+                },
+                {
+                  "value": "sm_j810m",
+                  "children": [
+                    {
+                      "variable": "tflite.device_name",
+                      "items": [
+                        {
+                          "value": "j8y18lte",
+                          "derived_properties": [
+                            {
+                              "variable": "tflite.gpu.opencl_status",
+                              "value": "WHITELISTED"
+                            }
+                          ]
+                        }
+                      ]
+                    }
+                  ]
+                }
+              ]
+            }
+          ]
+        }
+      ]
+    }
+  ]
+}
diff --git a/tensorflow/lite/experimental/acceleration/whitelist/devicedb.cc b/tensorflow/lite/experimental/acceleration/whitelist/devicedb.cc
new file mode 100644
index 00000000000..978495a3234
--- /dev/null
+++ b/tensorflow/lite/experimental/acceleration/whitelist/devicedb.cc
@@ -0,0 +1,91 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+#include "tensorflow/lite/experimental/acceleration/whitelist/devicedb.h"
+
+#include <map>
+#include <string>
+#include <vector>
+
+#include "tensorflow/lite/experimental/acceleration/whitelist/database_generated.h"
+
+namespace tflite {
+namespace acceleration {
+namespace {
+
+std::vector<const DeviceDecisionTreeEdge*> Find(
+    const DeviceDecisionTreeNode* root, const std::string& value) {
+  std::vector<const DeviceDecisionTreeEdge*> found;
+  if (root->comparison() == Comparison_EQUAL) {
+    // Exact match.
+    const DeviceDecisionTreeEdge* possible =
+        root->items()->LookupByKey(value.c_str());
+    if (possible) {
+      found.push_back(possible);
+    }
+  } else {
+    // Minimum: value should be at least item's value.
+    for (const DeviceDecisionTreeEdge* item : *(root->items())) {
+      if (value >= item->value()->str()) {
+        found.push_back(item);
+      }
+    }
+  }
+  return found;
+}
+
+void UpdateVariablesFromDeviceDecisionTreeEdges(
+    std::map<std::string, std::string>* variable_values,
+    const DeviceDecisionTreeEdge& item) {
+  if (item.derived_properties()) {
+    for (const DerivedProperty* p : *(item.derived_properties())) {
+      (*variable_values)[p->variable()->str()] = p->value()->str();
+    }
+  }
+}
+
+void Follow(const DeviceDecisionTreeNode* root,
+            std::map<std::string, std::string>* variable_values) {
+  if (!root->variable()) {
+    return;
+  }
+  auto possible_value = variable_values->find(root->variable()->str());
+  if (possible_value == variable_values->end()) {
+    return;
+  }
+  std::vector<const DeviceDecisionTreeEdge*> edges =
+      Find(root, possible_value->second);
+  for (const DeviceDecisionTreeEdge* edge : edges) {
+    UpdateVariablesFromDeviceDecisionTreeEdges(variable_values, *edge);
+    if (edge->children()) {
+      for (const DeviceDecisionTreeNode* root : *(edge->children())) {
+        Follow(root, variable_values);
+      }
+    }
+  }
+}
+
+}  // namespace
+
+void UpdateVariablesFromDatabase(
+    std::map<std::string, std::string>* variable_values,
+    const DeviceDatabase& database) {
+  if (!database.root()) return;
+  for (const DeviceDecisionTreeNode* root : *(database.root())) {
+    Follow(root, variable_values);
+  }
+}
+
+}  // namespace acceleration
+}  // namespace tflite
diff --git a/tensorflow/lite/experimental/acceleration/whitelist/devicedb.h b/tensorflow/lite/experimental/acceleration/whitelist/devicedb.h
new file mode 100644
index 00000000000..74a0d78e44e
--- /dev/null
+++ b/tensorflow/lite/experimental/acceleration/whitelist/devicedb.h
@@ -0,0 +1,38 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+#ifndef TENSORFLOW_LITE_EXPERIMENTAL_ACCELERATION_WHITELIST_DECISION_TREE_H_
+#define TENSORFLOW_LITE_EXPERIMENTAL_ACCELERATION_WHITELIST_DECISION_TREE_H_
+
+#include <map>
+#include <string>
+
+#include "tensorflow/lite/experimental/acceleration/whitelist/database_generated.h"
+
+namespace tflite {
+namespace acceleration {
+
+// Use the variables in `variable_values` to evaluate the decision tree in
+// `database` and update the `variable_values` based on derived properties in
+// the decision tree.
+//
+// See database.fbs for a description of the decision tree.
+void UpdateVariablesFromDatabase(
+    std::map<std::string, std::string>* variable_values,
+    const DeviceDatabase& database);
+
+}  // namespace acceleration
+}  // namespace tflite
+
+#endif  // TENSORFLOW_LITE_EXPERIMENTAL_ACCELERATION_WHITELIST_DECISION_TREE_H_
diff --git a/tensorflow/lite/experimental/acceleration/whitelist/devicedb_test.cc b/tensorflow/lite/experimental/acceleration/whitelist/devicedb_test.cc
new file mode 100644
index 00000000000..ae020dd7ba2
--- /dev/null
+++ b/tensorflow/lite/experimental/acceleration/whitelist/devicedb_test.cc
@@ -0,0 +1,142 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+#include "tensorflow/lite/experimental/acceleration/whitelist/devicedb.h"
+
+#include <memory>
+#include <string>
+
+#include <gtest/gtest.h>
+#include "flatbuffers/flatbuffers.h"  // from @flatbuffers
+#include "tensorflow/lite/experimental/acceleration/whitelist/database_generated.h"
+#include "tensorflow/lite/experimental/acceleration/whitelist/devicedb-sample.h"
+#include "tensorflow/lite/experimental/acceleration/whitelist/variables.h"
+#include "tensorflow/lite/testing/util.h"
+
+namespace tflite {
+namespace acceleration {
+namespace {
+
+class DeviceDbTest : public ::testing::Test {
+ protected:
+  void LoadSample() {
+    device_db_ = flatbuffers::GetRoot<DeviceDatabase>(
+        g_tflite_acceleration_devicedb_sample_binary);
+  }
+
+  const DeviceDatabase* device_db_ = nullptr;
+};
+
+TEST_F(DeviceDbTest, Load) {
+  LoadSample();
+  ASSERT_TRUE(device_db_);
+  ASSERT_TRUE(device_db_->root());
+  EXPECT_EQ(device_db_->root()->size(), 3);
+}
+
+TEST_F(DeviceDbTest, SocLookup) {
+  LoadSample();
+  ASSERT_TRUE(device_db_);
+  std::map<std::string, std::string> variables;
+
+  // Find first device mapping.
+  variables[kDeviceModel] = "m712c";
+  UpdateVariablesFromDatabase(&variables, *device_db_);
+  EXPECT_EQ(variables[kSoCModel], "exynos_7872");
+
+  // Find second device mapping.
+  variables.clear();
+  variables[kDeviceModel] = "sc_02l";
+  UpdateVariablesFromDatabase(&variables, *device_db_);
+  EXPECT_EQ(variables[kSoCModel], "exynos_7885");
+
+  // Make sure no results are returned without a match.
+  variables.clear();
+  variables[kDeviceModel] = "nosuch";
+  UpdateVariablesFromDatabase(&variables, *device_db_);
+  EXPECT_EQ(variables.find(kSoCModel), variables.end());
+}
+
+TEST_F(DeviceDbTest, StatusLookupWithSoC) {
+  LoadSample();
+  ASSERT_TRUE(device_db_);
+  std::map<std::string, std::string> variables;
+
+  // Find exact match.
+  variables[kOpenGLESVersion] = "3.1";
+  variables[kSoCModel] = "exynos_7872";
+  variables[kAndroidSdkVersion] = "24";
+  UpdateVariablesFromDatabase(&variables, *device_db_);
+  EXPECT_EQ(variables[gpu::kStatus], gpu::kStatusWhitelisted);
+
+  // Ensure no results without a match.
+  variables[kOpenGLESVersion] = "3.0";
+  variables.erase(variables.find(gpu::kStatus));
+  UpdateVariablesFromDatabase(&variables, *device_db_);
+  EXPECT_EQ(variables.find(gpu::kStatus), variables.end());
+
+  // Find no results with too low an android version.
+  variables.clear();
+  variables[kOpenGLESVersion] = "3.1";
+  variables[kSoCModel] = "exynos_7883";
+  variables[kAndroidSdkVersion] = "24";
+  UpdateVariablesFromDatabase(&variables, *device_db_);
+  EXPECT_EQ(variables.find(gpu::kStatus), variables.end());
+  // Find a match with android version above minimum.
+  variables[kAndroidSdkVersion] = "29";
+  UpdateVariablesFromDatabase(&variables, *device_db_);
+  EXPECT_EQ(variables[gpu::kStatus], gpu::kStatusWhitelisted);
+}
+
+TEST_F(DeviceDbTest, StatusLookupWithDevice) {
+  LoadSample();
+  ASSERT_TRUE(device_db_);
+  std::map<std::string, std::string> variables;
+  // Find blacklisted device (same model, different device).
+  variables[kAndroidSdkVersion] = "24";
+  variables[kDeviceModel] = "sm_j810f";
+  variables[kDeviceName] = "j8y18lte";
+  UpdateVariablesFromDatabase(&variables, *device_db_);
+  EXPECT_EQ(variables[gpu::kStatus], gpu::kStatusBlacklisted);
+
+  // Find whitelisted device (same model, different device).
+  variables.clear();
+  variables[kAndroidSdkVersion] = "24";
+  variables[kDeviceModel] = "sm_j810m";
+  variables[kDeviceName] = "j8y18lte";
+  UpdateVariablesFromDatabase(&variables, *device_db_);
+  EXPECT_EQ(variables[gpu::kOpenCLStatus], gpu::kStatusWhitelisted);
+}
+
+TEST_F(DeviceDbTest, StatusLookupBasedOnDerivedProperties) {
+  LoadSample();
+  ASSERT_TRUE(device_db_);
+  std::map<std::string, std::string> variables;
+  // Find status based on SoC derived from model.
+  variables[kOpenGLESVersion] = "3.1";
+  variables[kAndroidSdkVersion] = "24";
+  variables[kDeviceModel] = "m712c";
+  UpdateVariablesFromDatabase(&variables, *device_db_);
+  EXPECT_EQ(variables[gpu::kStatus], gpu::kStatusWhitelisted);
+}
+
+}  // namespace
+}  // namespace acceleration
+}  // namespace tflite
+
+int main(int argc, char** argv) {
+  ::tflite::LogToStderr();
+  ::testing::InitGoogleTest(&argc, argv);
+  return RUN_ALL_TESTS();
+}
diff --git a/tensorflow/lite/experimental/acceleration/whitelist/gpu_whitelist.bin b/tensorflow/lite/experimental/acceleration/whitelist/gpu_whitelist.bin
new file mode 100644
index 00000000000..5d42a3ec242
Binary files /dev/null and b/tensorflow/lite/experimental/acceleration/whitelist/gpu_whitelist.bin differ
diff --git a/tensorflow/lite/experimental/acceleration/whitelist/gpu_whitelist.cc b/tensorflow/lite/experimental/acceleration/whitelist/gpu_whitelist.cc
new file mode 100644
index 00000000000..efc6de91e99
--- /dev/null
+++ b/tensorflow/lite/experimental/acceleration/whitelist/gpu_whitelist.cc
@@ -0,0 +1,99 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+#include "tensorflow/lite/experimental/acceleration/whitelist/gpu_whitelist.h"
+
+#include <cctype>
+#include <map>
+#include <string>
+
+#include "absl/strings/string_view.h"
+#include "flatbuffers/flatbuffers.h"  // from @flatbuffers
+#include "tensorflow/lite/experimental/acceleration/whitelist/database_generated.h"
+#include "tensorflow/lite/experimental/acceleration/whitelist/devicedb.h"
+#include "tensorflow/lite/experimental/acceleration/whitelist/gpu_whitelist_binary.h"
+#include "tensorflow/lite/experimental/acceleration/whitelist/variables.h"
+
+namespace tflite {
+namespace acceleration {
+namespace {
+
+std::string CanonicalizeValue(absl::string_view input) {
+  // This assumes ASCII, which holds for all values we have in the whitelist.
+  std::string output(input);
+  for (int i = 0; i < output.size(); i++) {
+    char c = output[i];
+    if (c == ' ' || c == '-') {
+      output[i] = '_';
+    } else if (isalpha(c)) {
+      output[i] = tolower(c);
+    }
+  }
+  return output;
+}
+
+void CanonicalizeValues(std::map<std::string, std::string>* variable_values) {
+  for (auto& i : *variable_values) {
+    i.second = CanonicalizeValue(i.second);
+  }
+}
+
+}  // namespace
+
+GPUWhitelist::GPUWhitelist()
+    : GPUWhitelist(g_tflite_acceleration_gpu_whitelist_binary) {}
+
+GPUWhitelist::GPUWhitelist(const unsigned char* whitelist_flatbuffer) {
+  if (!whitelist_flatbuffer) return;
+  database_ = flatbuffers::GetRoot<DeviceDatabase>(whitelist_flatbuffer);
+}
+
+std::map<std::string, std::string> GPUWhitelist::CalculateVariables(
+    const AndroidInfo& android_info,
+    const ::tflite::gpu::GpuInfo& gpu_info) const {
+  std::map<std::string, std::string> variables;
+
+  variables[kAndroidSdkVersion] = android_info.android_sdk_version;
+  variables[kDeviceModel] = android_info.model;
+  variables[kDeviceName] = android_info.device;
+  variables[kManufacturer] = android_info.manufacturer;
+  variables[kGPUModel] = gpu_info.renderer_name;
+  char buffer[128];
+  int len = snprintf(buffer, 128 - 1, "%d.%d", gpu_info.major_version,
+                     gpu_info.minor_version);
+  buffer[len] = '\0';
+  variables[kOpenGLESVersion] = std::string(buffer);
+  CanonicalizeValues(&variables);
+  if (!database_) return variables;
+  UpdateVariablesFromDatabase(&variables, *database_);
+  return variables;
+}
+
+bool GPUWhitelist::Includes(const AndroidInfo& android_info,
+                            const ::tflite::gpu::GpuInfo& gpu_info) const {
+  auto variables = CalculateVariables(android_info, gpu_info);
+  return variables[gpu::kStatus] == std::string(gpu::kStatusWhitelisted);
+}
+
+TfLiteGpuDelegateOptionsV2 GPUWhitelist::GetBestOptionsFor(
+    const AndroidInfo& /* android_info */,
+    const ::tflite::gpu::GpuInfo& /* gpu_info */) const {
+  // This method is for forwards-compatibility: the whitelist may later include
+  // information about which backend to choose (OpenGL/OpenCL/Vulkan) or other
+  // options.
+  return TfLiteGpuDelegateOptionsV2Default();
+}
+
+}  // namespace acceleration
+}  // namespace tflite
diff --git a/tensorflow/lite/experimental/acceleration/whitelist/gpu_whitelist.h b/tensorflow/lite/experimental/acceleration/whitelist/gpu_whitelist.h
new file mode 100644
index 00000000000..a28e0d1e5d2
--- /dev/null
+++ b/tensorflow/lite/experimental/acceleration/whitelist/gpu_whitelist.h
@@ -0,0 +1,85 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+#ifndef TENSORFLOW_LITE_EXPERIMENTAL_ACCELERATION_WHITELIST_GPU_WHITELIST_H_
+#define TENSORFLOW_LITE_EXPERIMENTAL_ACCELERATION_WHITELIST_GPU_WHITELIST_H_
+
+#include <map>
+#include <string>
+
+#include "tensorflow/lite/delegates/gpu/common/gpu_info.h"
+#include "tensorflow/lite/delegates/gpu/delegate.h"
+#include "tensorflow/lite/experimental/acceleration/whitelist/android_info.h"
+#include "tensorflow/lite/experimental/acceleration/whitelist/devicedb.h"
+
+namespace tflite {
+namespace acceleration {
+
+// This class provides information on GPU delegate support.
+//
+// The GPU delegate is supported on a subset of Android devices, depending on
+// Android version, OpenGL ES version, GPU chipset etc. The support is based on
+// measure stability, correctness and peformance. For more detail see README.md.
+//
+// Example usage:
+//   tflite::Interpreter* interpreter = ... ;
+//   tflite::acceleration::AndroidInfo android_info;
+//   tflite::gpu::GpuInfo gpu_info;
+//   EXPECT_OK(tflite::acceleration::RequestAndroidInfo(&android_info));
+//   EXPECT_OK(tflite::gpu::gl::EglEnvironment::NewEglEnvironment(&env));
+//   EXPECT_OK(tflite::gpu::gl::RequestGpuInfo(&tflite_gpu_info));
+//   tflite::acceleration::GPUWhitelist whitelist;
+//   TfLiteDelegate* gpu_delegate = nullptr;
+//   TfLiteGpuDelegateOptions gpu_options;
+//   if (whitelist.Includes(android_info, gpu_info)) {
+//     gpu_options = whitelist.BestOptionsFor(android_info, gpu_info);
+//     gpu_delegate = TfLiteGpuDelegateCreate(&gpu_options);
+//     EXPECT_EQ(interpreter->ModifyGraphWithDelegate(gpu_delegate), TfLiteOk);
+//   } else {
+//     // Fallback path.
+//   }
+class GPUWhitelist {
+ public:
+  // Construct whitelist from bundled data.
+  GPUWhitelist();
+  // Returns true if the provided device specs are whitelisted by the database.
+  bool Includes(const AndroidInfo& android_info,
+                const ::tflite::gpu::GpuInfo& gpu_info) const;
+
+  // Returns the best TfLiteGpuDelegateOptionsV2 for the provided device specs
+  // based on the database. The output can be modified as desired before passing
+  // to delegate creation.
+  TfLiteGpuDelegateOptionsV2 GetBestOptionsFor(
+      const AndroidInfo& android_info,
+      const ::tflite::gpu::GpuInfo& gpu_info) const;
+
+  // Convert android_info and gpu_info into a set of variables used for querying
+  // the whitelist, and update variables from whitelist data. See variables.h
+  // and devicedb.h for more information.
+  std::map<std::string, std::string> CalculateVariables(
+      const AndroidInfo& android_info,
+      const ::tflite::gpu::GpuInfo& gpu_info) const;
+
+  GPUWhitelist(const GPUWhitelist&) = delete;
+  GPUWhitelist& operator=(const GPUWhitelist&) = delete;
+
+ protected:
+  explicit GPUWhitelist(const unsigned char* whitelist_flatbuffer);
+  const DeviceDatabase* database_;
+};
+
+}  // namespace acceleration
+}  // namespace tflite
+
+#endif  // TENSORFLOW_LITE_EXPERIMENTAL_ACCELERATION_WHITELIST_GPU_WHITELIST_H_
diff --git a/tensorflow/lite/experimental/acceleration/whitelist/json_to_fb.cc b/tensorflow/lite/experimental/acceleration/whitelist/json_to_fb.cc
new file mode 100644
index 00000000000..11638895ae8
--- /dev/null
+++ b/tensorflow/lite/experimental/acceleration/whitelist/json_to_fb.cc
@@ -0,0 +1,92 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+// Simple program to convert from JSON to binary flatbuffers for given schema.
+//
+// Used for creating the binary version of a whitelist.
+//
+// The flatc command line is not available in all build environments.
+#include <fstream>
+#include <iostream>
+#include <string>
+#include <vector>
+
+#include "flatbuffers/flatbuffers.h"  // from @flatbuffers
+#include "flatbuffers/idl.h"  // from @flatbuffers
+#include "flatbuffers/reflection.h"  // from @flatbuffers
+#include "flatbuffers/reflection_generated.h"  // from @flatbuffers
+#include "flatbuffers/util.h"  // from @flatbuffers
+#include "tensorflow/lite/tools/command_line_flags.h"
+
+int main(int argc, char** argv) {
+  std::string json_path, fbs_path, fb_path;
+  std::vector<tflite::Flag> flags = {
+      tflite::Flag::CreateFlag("json_input", &json_path,
+                               "Path to input json file."),
+      tflite::Flag::CreateFlag("fbs", &fbs_path,
+                               "Path to flatbuffer schema to use."),
+      tflite::Flag::CreateFlag("fb_output", &fb_path,
+                               "Path to a output binary flatbuffer."),
+  };
+  const bool parse_result =
+      tflite::Flags::Parse(&argc, const_cast<const char**>(argv), flags);
+  if (!parse_result || json_path.empty() || fbs_path.empty() ||
+      fb_path.empty()) {
+    std::cerr << tflite::Flags::Usage(argv[0], flags);
+    return 1;
+  }
+  std::string json_contents;
+  if (!flatbuffers::LoadFile(json_path.c_str(), false, &json_contents)) {
+    std::cerr << "Unable to load file " << json_path << std::endl;
+    return 2;
+  }
+  std::string fbs_contents;
+  if (!flatbuffers::LoadFile(fbs_path.c_str(), false, &fbs_contents)) {
+    std::cerr << "Unable to load file " << fbs_path << std::endl;
+    return 3;
+  }
+  const char* include_directories[] = {nullptr};
+  flatbuffers::Parser schema_parser;
+  if (!schema_parser.Parse(fbs_contents.c_str(), include_directories)) {
+    std::cerr << "Unable to parse schema " << schema_parser.error_ << std::endl;
+    return 4;
+  }
+  schema_parser.Serialize();
+  auto schema =
+      reflection::GetSchema(schema_parser.builder_.GetBufferPointer());
+  auto root_table = schema->root_table();
+  flatbuffers::Parser parser;
+  parser.Deserialize(schema_parser.builder_.GetBufferPointer(),
+                     schema_parser.builder_.GetSize());
+
+  if (!parser.Parse(json_contents.c_str(), include_directories,
+                    json_path.c_str())) {
+    std::cerr << "Unable to parse json " << parser.error_ << std::endl;
+    return 5;
+  }
+
+  // Use CopyTable() to deduplicate the strings.
+  const uint8_t* buffer = parser.builder_.GetBufferPointer();
+  flatbuffers::FlatBufferBuilder fbb;
+  auto root_offset = flatbuffers::CopyTable(
+      fbb, *schema, *root_table, *flatbuffers::GetAnyRoot(buffer), true);
+  fbb.Finish(root_offset);
+  std::string binary(reinterpret_cast<const char*>(fbb.GetBufferPointer()),
+                     fbb.GetSize());
+  std::ofstream output;
+  output.open(fb_path);
+  output << binary;
+  output.close();
+  return 0;
+}
diff --git a/tensorflow/lite/experimental/acceleration/whitelist/variables.h b/tensorflow/lite/experimental/acceleration/whitelist/variables.h
new file mode 100644
index 00000000000..178343e5c9c
--- /dev/null
+++ b/tensorflow/lite/experimental/acceleration/whitelist/variables.h
@@ -0,0 +1,87 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+#ifndef TENSORFLOW_LITE_EXPERIMENTAL_ACCELERATION_WHITELIST_VARIABLES_H_
+#define TENSORFLOW_LITE_EXPERIMENTAL_ACCELERATION_WHITELIST_VARIABLES_H_
+
+// This file lists generally useful whitelisting properties.
+// Properties starting with "tflite." are reserved.
+// Users of the whitelisting library can use arbitrary other property names.
+
+namespace tflite {
+namespace acceleration {
+// System properties, not specific to any single delegate.
+
+// Android properties.
+//
+// Android SDK version number. Android system property ro.build.version.sdk.
+// E.g., "28".
+constexpr char kAndroidSdkVersion[] = "tflite.android_sdk_version";
+// SoC model. Looked up from database or possibly returned from Android system
+// property ro.board.platform, normalized. E.g., "sdm450".
+constexpr char kSoCModel[] = "tflite.soc_model";
+// SoC vendor. Looked up from database. E.g., "qualcomm".
+constexpr char kSoCVendor[] = "tflite.soc_vendor";
+// Device manufacturer. Android API android.os.Build.MANUFACTURER, normalized.
+// E.g., "google".
+constexpr char kManufacturer[] = "tflite.manufacturer";
+// Device model. Android API android.os.Build.MODEL, normalized.
+// E.g., "pixel_2".
+constexpr char kDeviceModel[] = "tflite.device_model";
+// Device name. Android API android.os.Build.DEVICE, normalized.
+// E.g., "walleye".
+constexpr char kDeviceName[] = "tflite.device_name";
+
+// GPU-related properties.
+//
+// OpenGL ES version. E.g., 3.2.
+constexpr char kOpenGLESVersion[] = "tflite.opengl_es_version";
+// GPU model, result of querying GL_RENDERER, normalized. E.g.,
+// "adreno_(tm)_505".
+constexpr char kGPUModel[] = "tflite.gpu_model";
+// GPU vendor, normalized. E.g., "adreno_(tm)_505".
+constexpr char kGPUVendor[] = "tflite.gpu_vendor";
+// OpenGL driver version, result of querying GL_VERSION. E.g.,
+// "opengl_es_3.2_v@328.0_(git@6fb5a5b,_ife855c4895)_(date:08/21/18)"
+constexpr char kOpenGLDriverVersion[] = "tflite.opengl_driver_version";
+
+// NNAPI-related properties.
+//
+// NNAPI accelerator name, returned by ANeuralNetworksDevice_getName. E.g.,
+// "qti-dsp".
+constexpr char kNNAPIAccelerator[] = "tflite.nnapi_accelerator";
+// NNAPI accelerator feature level, returned by
+// ANeuralNetworksDevice_getFeatureLevel. E.g., 29. Actual variables are named
+// "tflite.nnapi_feature_level.<accelerator name>", e.g.,
+// "tflite.nnapi_feature_level.qti-dsp".
+constexpr char kNNAPIFeatureLevelPrefix[] = "tflite.nnapi_feature_level";
+
+namespace gpu {
+// GPU-delegate derived properties.
+
+// Whether the GPU delegate works in general.
+// ("UNSET", "UNKNOWN", "WHITELISTED", "BLACKLISTED").
+constexpr char kStatus[] = "tflite.gpu.status";
+
+// Whether OpenCL should be allowed. Possible values are the SupportStatus enums
+// ("UNSET", "UNKNOWN", "WHITELISTED", "BLACKLISTED").
+constexpr char kOpenCLStatus[] = "tflite.gpu.opencl_status";
+constexpr char kStatusWhitelisted[] = "WHITELISTED";
+constexpr char kStatusBlacklisted[] = "BLACKLISTED";
+}  // namespace gpu
+
+}  // namespace acceleration
+}  // namespace tflite
+
+#endif  // TENSORFLOW_LITE_EXPERIMENTAL_ACCELERATION_WHITELIST_VARIABLES_H_
diff --git a/tensorflow/lite/experimental/delegates/hexagon/README.md b/tensorflow/lite/experimental/delegates/hexagon/README.md
index 106ddce038b..7af76059c58 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/README.md
+++ b/tensorflow/lite/experimental/delegates/hexagon/README.md
@@ -58,7 +58,7 @@ Hexagon only supports ops that have inputs/outputs of <= 4 dimensions.
 The following operations have been implemented, with a few constraints that
 are verified in `IsNodeSupportedByHexagon`:
 
-* Add
+* Add (Support relu activations)
 * ArgMax
 * ArgMin
 * AveragePool2D:
@@ -84,7 +84,7 @@ are verified in `IsNodeSupportedByHexagon`:
 * Mean
 * Minimum
 * MirrorPad
-* Mul (without any activation) (b/129276536)
+* Mul (Support relu activations)
 * Neg
 * Pack
 * Pad: Only supports 0 padding (b/139277813)
@@ -100,7 +100,8 @@ are verified in `IsNodeSupportedByHexagon`:
 * SoftMax
 * SpaceToDepth
 * Split
-* Sub
+* Strided Slice
+* Sub (Support relu activations)
 * Tanh
 * Transpose
 * TransposeConv2D:
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/BUILD b/tensorflow/lite/experimental/delegates/hexagon/builders/BUILD
index feadd096c54..e3dd9a01c33 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/BUILD
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/BUILD
@@ -35,6 +35,7 @@ cc_library(
         "softmax_builder.cc",
         "space_to_depth_builder.cc",
         "split_builder.cc",
+        "strided_slice_builder.cc",
         "transpose_builder.cc",
         "transpose_conv_2d_builder.cc",
     ],
@@ -65,6 +66,7 @@ cc_library(
         "softmax_builder.h",
         "space_to_depth_builder.h",
         "split_builder.h",
+        "strided_slice_builder.h",
         "transpose_builder.h",
         "transpose_conv_2d_builder.h",
     ],
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/arg_min_max_builder.cc b/tensorflow/lite/experimental/delegates/hexagon/builders/arg_min_max_builder.cc
index eda69205359..61b609c2bb7 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/arg_min_max_builder.cc
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/arg_min_max_builder.cc
@@ -50,7 +50,7 @@ TfLiteStatus ArgMinMaxOpBuilder::PopulateSubGraph(const TfLiteIntArray* inputs,
     axis_value += input_tensor.dims->size;
   }
   auto* input_axis_const = graph_builder_->AddConstNodeWithData(
-      kScalarShape, reinterpret_cast<char*>(&axis_value), sizeof(int32_t));
+      kScalarShape, reinterpret_cast<char*>(&axis_value), sizeof(int));
   AddInput(TensorID(input_axis_const->GetID(), 0));
 
   // Compute Min/Max
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/arithmetic_builder.cc b/tensorflow/lite/experimental/delegates/hexagon/builders/arithmetic_builder.cc
index 22450bffbf5..5211f9f266a 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/arithmetic_builder.cc
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/arithmetic_builder.cc
@@ -18,6 +18,7 @@ limitations under the License.
 
 #include <limits>
 
+#include "hexagon/hexagon_nn_ops.h"
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/experimental/delegates/hexagon/hexagon_nn/hexagon_nn.h"
 #include "tensorflow/lite/kernels/kernel_util.h"
@@ -75,11 +76,11 @@ TfLiteStatus ArithmeticOpBuilder::PopulateSubGraph(
   }
 
   if (op_node_.op_type == OP_QuantizedMul_8x8to32) {
-    const auto& math_out = AddOutput(sizeof(int32_t), 4,
+    const auto& math_out = AddOutput(sizeof(int), 4,
                                      {output_batch_size, output_height_size,
                                       output_width_size, output_depth_size});
-    const auto& math_out_min = AddOutput(sizeof(float), 4, {1, 1, 1, 1});
-    const auto& math_out_max = AddOutput(sizeof(float), 4, {1, 1, 1, 1});
+    const auto& math_out_min = AddOutput(sizeof(float), 4, kScalarShape);
+    const auto& math_out_max = AddOutput(sizeof(float), 4, kScalarShape);
 
     auto* requantize_op = graph_builder_->AddNode(GetTFLiteNodeID());
     requantize_op->SetOpType(OP_Requantize_32to8);
@@ -92,14 +93,28 @@ TfLiteStatus ArithmeticOpBuilder::PopulateSubGraph(
         requantize_op->AddOutput(sizeof(uint8_t), 4,
                                  {output_batch_size, output_height_size,
                                   output_width_size, output_depth_size});
-    requantize_op->AddOutput(sizeof(float), 4, {1, 1, 1, 1});
-    requantize_op->AddOutput(sizeof(float), 4, {1, 1, 1, 1});
+    requantize_op->AddOutput(sizeof(float), 4, kScalarShape);
+    requantize_op->AddOutput(sizeof(float), 4, kScalarShape);
   } else {
-    node_output_ = AddOutput(sizeof(uint8_t), 4,
-                             {output_batch_size, output_height_size,
-                              output_width_size, output_depth_size});
-    AddOutput(sizeof(float), 4, {1, 1, 1, 1});
-    AddOutput(sizeof(float), 4, {1, 1, 1, 1});
+    auto result_out = AddOutput(sizeof(uint8_t), 4,
+                                {output_batch_size, output_height_size,
+                                 output_width_size, output_depth_size});
+    auto result_min = AddOutput(sizeof(float), 4, kScalarShape);
+    auto result_max = AddOutput(sizeof(float), 4, kScalarShape);
+
+    auto* requantize_op = graph_builder_->AddNode(GetTFLiteNodeID());
+    requantize_op->SetOpType(OP_Requantize_8to8);
+    requantize_op->AddInput(result_out);
+    requantize_op->AddInput(result_min);
+    requantize_op->AddInput(result_max);
+    requantize_op->AddInput(TensorID(output_min_const->GetID(), 0));
+    requantize_op->AddInput(TensorID(output_max_const->GetID(), 0));
+    node_output_ =
+        requantize_op->AddOutput(sizeof(uint8_t), 4,
+                                 {output_batch_size, output_height_size,
+                                  output_width_size, output_depth_size});
+    requantize_op->AddOutput(sizeof(float), 4, kScalarShape);
+    requantize_op->AddOutput(sizeof(float), 4, kScalarShape);
   }
 
   return kTfLiteOk;
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/batch_seq_builder.cc b/tensorflow/lite/experimental/delegates/hexagon/builders/batch_seq_builder.cc
index 17c40afe83a..ecc45beeeac 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/batch_seq_builder.cc
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/batch_seq_builder.cc
@@ -29,10 +29,10 @@ TfLiteStatus BatchSeqBuilder::PopulateSubGraph(const TfLiteIntArray* inputs,
   // Options is currently 0 or 1, 0 is default and batches
   // will run in increasing order, this behavior can be disabled by setting 1.
   // Refer to Hexagon NN docs for more details.
-  int32_t config[] = {max_size_for_batch_, 1, 0};
+  int config[] = {max_size_for_batch_, 1, 0};
 
   auto* input_config = graph_builder_->AddConstNodeWithData(
-      config_shape, reinterpret_cast<char*>(&config), sizeof(int32_t) * 3);
+      config_shape, reinterpret_cast<char*>(&config), sizeof(int) * 3);
   AddInput(TensorID(input_config->GetID(), 0));
 
   // Add Input batch details.
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/conv_2d_helpers.cc b/tensorflow/lite/experimental/delegates/hexagon/builders/conv_2d_helpers.cc
index 6cb5ddaa86f..f897d56bf1d 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/conv_2d_helpers.cc
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/conv_2d_helpers.cc
@@ -199,7 +199,7 @@ TfLiteStatus Conv2dOpBuilder::ProcessPerChannelQuantizedBias(
   const float input_scale = input_quant_params->scale->data[0];
   // Now dequantize bias values to float first, to adjust for the
   // normalization of channel scales.
-  int32_t* bias_data = bias_tensor.data.i32;
+  auto* bias_data = bias_tensor.data.i32;
   const int bias_size = NumElements(&bias_tensor);
   if (bias_size != num_scale_values_) {
     TF_LITE_KERNEL_LOG(
@@ -221,10 +221,10 @@ TfLiteStatus Conv2dOpBuilder::ProcessPerChannelQuantizedBias(
   *bias_max = *bias_max * 8;
   *bias_min = -1 * *bias_max;
   // Now requantize the bias values to the new min/max values.
-  std::vector<int32_t> preprocessed_bias_data;
+  std::vector<int> preprocessed_bias_data;
   preprocessed_bias_data.reserve(num_scale_values_);
   for (int i = 0; i < bias_size; ++i) {
-    preprocessed_bias_data.push_back(static_cast<int32_t>(
+    preprocessed_bias_data.push_back(static_cast<int>(
         std::round(std::pow(2, 31) * (dequantized_bias[i] / *bias_max))));
   }
   // Add nodes for bias.
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/matmul_builder.cc b/tensorflow/lite/experimental/delegates/hexagon/builders/matmul_builder.cc
index 8f0ce269016..f0253165c86 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/matmul_builder.cc
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/matmul_builder.cc
@@ -76,7 +76,7 @@ TfLiteStatus AddFullyConnectedHelper(const TfLiteIntArray* inputs,
   GetDims(&output_batch_size, &output_height_size, &output_width_size,
           &output_depth_size, context->tensors[outputs->data[0]].dims);
   const auto& matmul_out =
-      matmul_op->AddOutput(sizeof(int32_t), 4,
+      matmul_op->AddOutput(sizeof(int), 4,
                            {output_batch_size, output_height_size,
                             output_width_size, output_depth_size});
   const auto& matmul_out_min =
@@ -108,7 +108,7 @@ TfLiteStatus AddFullyConnectedHelper(const TfLiteIntArray* inputs,
     bias_add_op->AddInput(OpBuilder::TensorID(bias_min_const->GetID(), 0));
     bias_add_op->AddInput(OpBuilder::TensorID(bias_max_const->GetID(), 0));
     matmul_and_bias_out =
-        bias_add_op->AddOutput(sizeof(int32_t), 4,
+        bias_add_op->AddOutput(sizeof(int), 4,
                                {output_batch_size, output_height_size,
                                 output_width_size, output_depth_size});
     matmul_and_bias_out_min =
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/matmul_builder.h b/tensorflow/lite/experimental/delegates/hexagon/builders/matmul_builder.h
index 89f3c1273d7..25d2f20287c 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/matmul_builder.h
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/matmul_builder.h
@@ -40,8 +40,7 @@ class MatMulWithConstWeightsOpBuilder : public OpBuilder {
   TensorID node_output_;
   std::vector<int> weights_shape_, bias_shape_;
   std::vector<float> transposed_weights_;
-  float data_min_, data_max_, weights_min_, weights_max_, bias_min_, bias_max_,
-      output_min_, output_max_;
+  float weights_min_, weights_max_;
 };
 
 // Builder for FullyConnected op in Hexagon with non const weights.
@@ -68,8 +67,7 @@ class MatMulOpBuilder : public OpBuilder {
   TensorID node_output_;
   std::vector<int> weights_shape_, bias_shape_;
   std::vector<float> transposed_weights_;
-  float data_min_, data_max_, weights_min_, weights_max_, bias_min_, bias_max_,
-      output_min_, output_max_;
+  float weights_min_, weights_max_;
 };
 
 }  // namespace hexagon
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/op_builder.cc b/tensorflow/lite/experimental/delegates/hexagon/builders/op_builder.cc
index 0d68125392d..b91c825199c 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/op_builder.cc
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/op_builder.cc
@@ -108,6 +108,8 @@ OpBuilder* GraphBuilder::CreateOpBuilderFromTfLiteOp(int op_type,
       return CreateSliceOpBuilder(this, OP_QuantizedSlice_8);
     case kTfLiteBuiltinPack:
       return CreatePackBuilder(this, OP_QuantizedPack_8);
+    case kTfLiteBuiltinStridedSlice:
+      return CreateStridedSliceBuilder(this, OP_QuantizedStridedSlice_8);
     default:
       context_->ReportError(context_, "Op not supported: %d", op_type);
       return nullptr;
@@ -277,6 +279,9 @@ const OpNode* OpBuilder::Build() {
   return &op_node_;
 }
 
+// Static
+constexpr int OpBuilder::kScalarShape[];
+
 OpBuilder* GraphBuilder::AddNode(int tflite_node_index) {
   OpBuilder* op = new OpBuilder(this, OP_Nop);
   builders_.emplace_back(op);
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/op_builder.h b/tensorflow/lite/experimental/delegates/hexagon/builders/op_builder.h
index cf213e9b08e..e130e6d2efb 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/op_builder.h
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/op_builder.h
@@ -32,6 +32,7 @@ namespace tflite {
 namespace delegates {
 namespace hexagon {
 
+// Wrapper that holds all data representing a single node in the Hexagon graph.
 struct OpNode {
   std::vector<hexagon_nn_input> inputs;
   std::vector<hexagon_nn_output> outputs;
@@ -48,6 +49,15 @@ struct OpNode {
 
 class GraphBuilder;
 
+// Represents a single Op in the TFLite graph.
+// For each op in TFLite there should be an OpBuidler, this builder is
+// responsible for constructing equivalent node(s) in the hexagon graph. A
+// single builder can create one or more ops in the hexagon graph. When adding
+// new op* users should inherit from this class and implement
+// - PopulateSubgraph: which given inputs/outputs should construct the
+// equivalent hexagon nodes.
+// - RegisterOutputs: Which should have logic that maps final outputs from a
+// given node to the equivalent in Hexagon graph.
 class OpBuilder {
  public:
   // Const representing the shape of a scalar value.
@@ -65,30 +75,59 @@ class OpBuilder {
 
   virtual ~OpBuilder() {}
 
-  // TODO(karimnosseir): Do we need to have builder pattern, or they are few not
-  // worth it ?
+  // Sets the op type in the hexagon graph.
   void SetOpType(int op_type) { op_node_.op_type = op_type; }
 
+  // Sets the node id in the hexagon graph.
   void SetNodeId(int node_id) { op_node_.node_id = node_id; }
 
+  // Sets the TfLite node index in the TfLite graph.
   void SetTFLiteNodeId(int node_index) {
     op_node_.tflite_node_index = node_index;
   }
 
+  // Marks this node as Const node.
   void SetConstNode() { op_node_.op_type = OP_Const; }
 
+  // Sets the padding type of the current node.
   void SetPaddingType(hexagon_nn_padding_type padding_type) {
     op_node_.padding_type = padding_type;
   }
 
+  // Sets the builtin_data of TFLite node that this Builder is responsible for.
   void SetBuiltinData(void* builtin_data) { builtin_data_ = builtin_data; }
 
+  // Returns true if the current op is a const Op.
   bool IsConstNode() const { return op_node_.op_type == OP_Const; }
 
-  void print() {}
+  // Subclasses should override it and have logic which handles initializing
+  // hexagon node(s) for the current op, given 'inputs' 'outputs'
+  virtual TfLiteStatus PopulateSubGraph(const TfLiteIntArray* inputs,
+                                        const TfLiteIntArray* outputs,
+                                        TfLiteContext* context) {
+    return kTfLiteOk;
+  }
 
+  // Subclasses should override it and register the final output(s) from the
+  // node to the equivalent in hexagon graph.
+  virtual TfLiteStatus RegisterOutputs(const TfLiteIntArray* outputs,
+                                       TfLiteContext* context) {
+    return kTfLiteOk;
+  }
+
+  // Constructs OpNode which represents a node in the Hexagon graph.
   const OpNode* Build();
 
+  // Returns the Node index in TFLite graph.
+  int GetTFLiteNodeID() const { return op_node_.tflite_node_index; }
+
+  // Returns the Op type of the current Op (in Hexagon graph)
+  int GetOpType() const { return op_node_.op_type; }
+
+  // Returns the node id in the hexagon graph.
+  int GetID() const { return op_node_.node_id; }
+
+  // Adds tensor identified by 'tensor_id' as input to the current Op.
   void AddInput(const TensorID& tensor_id) { input_ids_.push_back(tensor_id); }
 
   // Adds Output to the current node, the output has shape defined in 'dims'.
@@ -106,25 +145,13 @@ class OpBuilder {
   // Same as above but accepts pointer instead of std::vector.
   TensorID AddOutput(int elementsize, int rank, const int* max_sizes_vect);
 
-  int GetID() const { return op_node_.node_id; }
-
-  int GetTFLiteNodeID() const { return op_node_.tflite_node_index; }
-
-  int GetOpType() const { return op_node_.op_type; }
-
+  // Sets the node that corresponds to this builder in TFLite graph.
   void SetTfLiteNode(const TfLiteNode* node) { tflite_node_ = node; }
 
-  virtual TfLiteStatus PopulateSubGraph(const TfLiteIntArray* inputs,
-                                        const TfLiteIntArray* outputs,
-                                        TfLiteContext* context) {
-    return kTfLiteOk;
-  }
-
-  virtual TfLiteStatus RegisterOutputs(const TfLiteIntArray* outputs,
-                                       TfLiteContext* context) {
-    return kTfLiteOk;
-  }
-
+  // Static
+  // Computes the min/max values of 'tensor' and sets the values in
+  // the out params 'min' and 'max'.
+  // Returns kTfLiteOk on success.
   static TfLiteStatus ComputeMinAndMaxQuantValues(const TfLiteTensor& tensor,
                                                   float* min, float* max) {
     if (tensor.type == kTfLiteUInt8) {
@@ -137,8 +164,8 @@ class OpBuilder {
                                          std::numeric_limits<int8_t>::max());
     } else if (tensor.type == kTfLiteInt32) {
       return ComputeMinAndMaxQuantValues(tensor, min, max,
-                                         std::numeric_limits<int32_t>::min(),
-                                         std::numeric_limits<int32_t>::max());
+                                         std::numeric_limits<int>::min(),
+                                         std::numeric_limits<int>::max());
     }
     return kTfLiteError;
   }
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/op_factory.h b/tensorflow/lite/experimental/delegates/hexagon/builders/op_factory.h
index 33b56e91f0a..95879013d53 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/op_factory.h
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/op_factory.h
@@ -59,6 +59,7 @@ OpBuilder* CreateMinMaxBuilder(GraphBuilder* graph_builder, int op_type);
 OpBuilder* CreateSliceOpBuilder(GraphBuilder* graph_builder, int op_type);
 OpBuilder* CreatePackBuilder(GraphBuilder* graph_builder, int op_type);
 OpBuilder* CreateMatMulOpBuilder(GraphBuilder* graph_builder, int op_type);
+OpBuilder* CreateStridedSliceBuilder(GraphBuilder* graph_builder, int op_type);
 
 }  // namespace hexagon
 }  // namespace delegates
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/reduce_builder.h b/tensorflow/lite/experimental/delegates/hexagon/builders/reduce_builder.h
index 597ad1ce2d4..4f3f3a43ddc 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/reduce_builder.h
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/reduce_builder.h
@@ -38,7 +38,7 @@ class ReduceOpBuilder : public OpBuilder {
 
  private:
   TensorID node_output_;
-  float input_min_, input_max_, output_min_, output_max_;
+  float input_min_, input_max_;
 };
 
 }  // namespace hexagon
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/slice_builder.cc b/tensorflow/lite/experimental/delegates/hexagon/builders/slice_builder.cc
index 3520ade3cd0..c58754cdf56 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/slice_builder.cc
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/slice_builder.cc
@@ -43,23 +43,23 @@ TfLiteStatus SliceOpBuilder::PopulateSubGraph(const TfLiteIntArray* inputs,
   // Start / Size
   const auto& begin_tensor = context->tensors[inputs->data[1]];
   const auto& size_tensor = context->tensors[inputs->data[2]];
-  std::vector<int32_t> begins, sizes;
+  std::vector<int> begins, sizes;
   if (begin_tensor.type == kTfLiteInt32) {
-    GetBeginAndSizeVectors<int32_t>(input_tensor.dims->size, &begin_tensor,
-                                    &size_tensor, &begins, &sizes);
+    GetBeginAndSizeVectors<int>(input_tensor.dims->size, &begin_tensor,
+                                &size_tensor, &begins, &sizes);
   } else if (begin_tensor.type == kTfLiteInt64) {
     GetBeginAndSizeVectors<int64_t>(input_tensor.dims->size, &begin_tensor,
                                     &size_tensor, &begins, &sizes);
   } else {
     return kTfLiteError;
   }
-  const int32_t begins_shape[] = {1, 1, 1, static_cast<int32_t>(begins.size())};
+  const int begins_shape[] = {1, 1, 1, static_cast<int>(begins.size())};
   auto begins_node = graph_builder_->AddConstNodeWithData(
       begins_shape, reinterpret_cast<char*>(begins.data()),
-      sizeof(int32_t) * begins.size());
+      sizeof(int) * begins.size());
   auto sizes_node = graph_builder_->AddConstNodeWithData(
       begins_shape, reinterpret_cast<char*>(sizes.data()),
-      sizeof(int32_t) * begins.size());
+      sizeof(int) * begins.size());
   AddInput(TensorID(begins_node->GetID(), 0));
   AddInput(TensorID(sizes_node->GetID(), 0));
 
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/softmax_builder.h b/tensorflow/lite/experimental/delegates/hexagon/builders/softmax_builder.h
index c48a621d2a6..fe4333f2b31 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/softmax_builder.h
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/softmax_builder.h
@@ -39,7 +39,7 @@ class SoftmaxOpBuilder : public OpBuilder {
  private:
   TensorID node_output_;
   float beta_value_ = 1.0f;
-  float input_min_, input_max_, output_min_, output_max_;
+  float input_min_, input_max_;
 };
 
 }  // namespace hexagon
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/space_to_depth_builder.h b/tensorflow/lite/experimental/delegates/hexagon/builders/space_to_depth_builder.h
index d4691b6b406..2e07a2b1ec6 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/space_to_depth_builder.h
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/space_to_depth_builder.h
@@ -40,7 +40,7 @@ class SpaceToDepthOpBuilder : public OpBuilder {
 
  private:
   TensorID node_output_;
-  float input_min_, input_max_, output_min_, output_max_;
+  float input_min_, input_max_;
   int block_size_;
 };
 
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/split_builder.cc b/tensorflow/lite/experimental/delegates/hexagon/builders/split_builder.cc
index 00308352fec..4a809f3f1c5 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/split_builder.cc
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/split_builder.cc
@@ -48,7 +48,7 @@ TfLiteStatus SplitOpBuilder::PopulateSubGraph(const TfLiteIntArray* inputs,
     axis_value += input_tensor.dims->size;
   }
   auto* input_axis_const = graph_builder_->AddConstNodeWithData(
-      kScalarShape, reinterpret_cast<char*>(&axis_value), sizeof(int32_t));
+      kScalarShape, reinterpret_cast<char*>(&axis_value), sizeof(int));
   AddInput(TensorID(input_axis_const->GetID(), 0));
 
   // Input data tensor & min/max.
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/strided_slice_builder.cc b/tensorflow/lite/experimental/delegates/hexagon/builders/strided_slice_builder.cc
new file mode 100644
index 00000000000..2859439905d
--- /dev/null
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/strided_slice_builder.cc
@@ -0,0 +1,106 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+#include "tensorflow/lite/experimental/delegates/hexagon/builders/strided_slice_builder.h"
+
+#include <vector>
+
+#include "tensorflow/lite/c/common.h"
+#include "tensorflow/lite/kernels/internal/tensor.h"
+
+namespace tflite {
+namespace delegates {
+namespace hexagon {
+namespace {}  // namespace
+
+TfLiteStatus StridedSliceOpBuilder::PopulateSubGraph(
+    const TfLiteIntArray* inputs, const TfLiteIntArray* outputs,
+    TfLiteContext* context) {
+  // Input data tensor.
+  const auto& input_tensor = context->tensors[inputs->data[0]];
+  AddInput(graph_builder_->GetHexagonTensorId(inputs->data[0]));
+  // Begin/End/Step.
+  const auto& begin_tensor = context->tensors[inputs->data[1]];
+  const auto& end_tensor = context->tensors[inputs->data[2]];
+  const auto& step_tensor = context->tensors[inputs->data[3]];
+  auto begins_node =
+      graph_builder_->AddConstNodeWithData(inputs->data[1], begin_tensor);
+  auto ends_node =
+      graph_builder_->AddConstNodeWithData(inputs->data[2], end_tensor);
+  auto steps_node =
+      graph_builder_->AddConstNodeWithData(inputs->data[3], step_tensor);
+  AddInput(TensorID(begins_node->GetID(), 0));
+  AddInput(TensorID(ends_node->GetID(), 0));
+  AddInput(TensorID(steps_node->GetID(), 0));
+  // Begin/End/Shrink-Axis masks.
+  // Hexagon's op always expects bits at 0, 1, 2 & 3 to correspond to BHWD.
+  // So we have to left-shift the mask by (4 - begins.size()).
+  const TfLiteStridedSliceParams* params =
+      reinterpret_cast<const TfLiteStridedSliceParams*>(builtin_data_);
+  int begin_mask = params->begin_mask;
+  int end_mask = params->end_mask;
+  int shrink_axis_mask = params->shrink_axis_mask;
+  int original_mask_size = input_tensor.dims->size;
+  begin_mask = begin_mask << (4 - original_mask_size);
+  end_mask = end_mask << (4 - original_mask_size);
+  shrink_axis_mask = shrink_axis_mask << (4 - original_mask_size);
+  auto* begin_mask_const = graph_builder_->AddConstNodeWithData(
+      kScalarShape, reinterpret_cast<char*>(&begin_mask), sizeof(begin_mask));
+  AddInput(TensorID(begin_mask_const->GetID(), 0));
+  auto* end_mask_const = graph_builder_->AddConstNodeWithData(
+      kScalarShape, reinterpret_cast<char*>(&end_mask), sizeof(end_mask));
+  AddInput(TensorID(end_mask_const->GetID(), 0));
+  auto* shrink_axis_mask_const = graph_builder_->AddConstNodeWithData(
+      kScalarShape, reinterpret_cast<char*>(&shrink_axis_mask),
+      sizeof(shrink_axis_mask));
+  AddInput(TensorID(shrink_axis_mask_const->GetID(), 0));
+
+  // Input min/max
+  TF_LITE_ENSURE_STATUS(
+      ComputeMinAndMaxQuantValues(input_tensor, &input_min_, &input_max_));
+  auto* input_min_const = graph_builder_->AddConstNodeWithData(
+      kScalarShape, reinterpret_cast<char*>(&input_min_), sizeof(input_min_));
+  auto* input_max_const = graph_builder_->AddConstNodeWithData(
+      kScalarShape, reinterpret_cast<char*>(&input_max_), sizeof(input_max_));
+  AddInput(TensorID(input_min_const->GetID(), 0));
+  AddInput(TensorID(input_max_const->GetID(), 0));
+
+  // Slice outputs.
+  int output_batch_size, output_height_size, output_width_size,
+      output_depth_size;
+  GetDims(&output_batch_size, &output_height_size, &output_width_size,
+          &output_depth_size, context->tensors[outputs->data[0]].dims);
+  node_output_ = AddOutput(sizeof(uint8_t), 4,
+                           {output_batch_size, output_height_size,
+                            output_width_size, output_depth_size});
+  AddOutput(sizeof(float), 4, kScalarShape);
+  AddOutput(sizeof(float), 4, kScalarShape);
+
+  return kTfLiteOk;
+}
+
+TfLiteStatus StridedSliceOpBuilder::RegisterOutputs(
+    const TfLiteIntArray* outputs, TfLiteContext* context) {
+  // Should be only 1 output.
+  graph_builder_->AddTensorWithID(outputs->data[0], node_output_.first,
+                                  node_output_.second);
+  return kTfLiteOk;
+}
+
+OpBuilder* CreateStridedSliceBuilder(GraphBuilder* graph_builder, int op_type) {
+  return new StridedSliceOpBuilder(graph_builder, op_type);
+}
+}  // namespace hexagon
+}  // namespace delegates
+}  // namespace tflite
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/strided_slice_builder.h b/tensorflow/lite/experimental/delegates/hexagon/builders/strided_slice_builder.h
new file mode 100644
index 00000000000..eb91e696fdb
--- /dev/null
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/strided_slice_builder.h
@@ -0,0 +1,45 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+#ifndef TENSORFLOW_LITE_EXPERIMENTAL_DELEGATES_HEXAGON_BUILDERS_STRIDED_SLICE_BUILDER_H_
+#define TENSORFLOW_LITE_EXPERIMENTAL_DELEGATES_HEXAGON_BUILDERS_STRIDED_SLICE_BUILDER_H_
+
+#include "tensorflow/lite/experimental/delegates/hexagon/builders/op_builder.h"
+
+namespace tflite {
+namespace delegates {
+namespace hexagon {
+
+class StridedSliceOpBuilder : public OpBuilder {
+ public:
+  explicit StridedSliceOpBuilder(GraphBuilder* graph_builder, int op_type)
+      : OpBuilder(graph_builder, op_type) {}
+
+  TfLiteStatus PopulateSubGraph(const TfLiteIntArray* inputs,
+                                const TfLiteIntArray* outputs,
+                                TfLiteContext* context) override;
+
+  TfLiteStatus RegisterOutputs(const TfLiteIntArray* outputs,
+                               TfLiteContext* context) override;
+
+ private:
+  TensorID node_output_;
+  float input_min_, input_max_;
+};
+
+}  // namespace hexagon
+}  // namespace delegates
+}  // namespace tflite
+
+#endif  // TENSORFLOW_LITE_EXPERIMENTAL_DELEGATES_HEXAGON_BUILDERS_STRIDED_SLICE_BUILDER_H_
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/tests/BUILD b/tensorflow/lite/experimental/delegates/hexagon/builders/tests/BUILD
index bfa6618447e..f7e6d3a220c 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/tests/BUILD
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/tests/BUILD
@@ -49,6 +49,7 @@ hexagon_op_tests(
         "softmax_test.cc",
         "space_to_depth_test.cc",
         "split_test.cc",
+        "strided_slice_test.cc",
         "transpose_conv_test.cc",
         "transpose_test.cc",
     ],
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/tests/arg_min_max_test.cc b/tensorflow/lite/experimental/delegates/hexagon/builders/tests/arg_min_max_test.cc
index c4900ac9453..7a878c98876 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/tests/arg_min_max_test.cc
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/tests/arg_min_max_test.cc
@@ -28,8 +28,8 @@ class ArgBaseOpModel : public SingleOpModelWithHexagon {
 
   int input() const { return input_; }
 
-  std::vector<int32_t> GetInt32Output() const {
-    return ExtractVector<int32_t>(output_);
+  std::vector<int> GetInt32Output() const {
+    return ExtractVector<int>(output_);
   }
   std::vector<int> GetOutputShape() { return GetTensorShape(output_); }
 
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/tests/arithmetic_test.cc b/tensorflow/lite/experimental/delegates/hexagon/builders/tests/arithmetic_test.cc
index 6f9b48054be..9bd19eb137e 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/tests/arithmetic_test.cc
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/tests/arithmetic_test.cc
@@ -75,25 +75,29 @@ class ArithmeticOpBaseModel : public SingleOpModelWithHexagon {
 class AddOpModel : public ArithmeticOpBaseModel {
  public:
   AddOpModel(const TensorData& input1, const TensorData& input2,
-             const TensorData& output)
-      : ArithmeticOpBaseModel(input1, input2, output) {}
+             const TensorData& output, ActivationFunctionType activation_func)
+      : ArithmeticOpBaseModel(input1, input2, output),
+        activation_func_(activation_func) {}
   AddOpModel(const TensorData& input1, const TensorData& input2,
              const TensorData& output,
              const std::initializer_list<uint8_t>& input1_data,
-             const std::initializer_list<uint8_t>& input2_data)
-      : ArithmeticOpBaseModel(input1, input2, output, input1_data,
-                              input2_data) {}
+             const std::initializer_list<uint8_t>& input2_data,
+             ActivationFunctionType activation_func)
+      : ArithmeticOpBaseModel(input1, input2, output, input1_data, input2_data),
+        activation_func_(activation_func) {}
 
   void InitInterpreter() {
-    SetBuiltinOp(
-        BuiltinOperator_ADD, BuiltinOptions_AddOptions,
-        CreateAddOptions(builder_, ActivationFunctionType_NONE).Union());
+    SetBuiltinOp(BuiltinOperator_ADD, BuiltinOptions_AddOptions,
+                 CreateAddOptions(builder_, activation_func_).Union());
     ArithmeticOpBaseModel::InitInterpreter();
   }
+
+ private:
+  ActivationFunctionType activation_func_;
 };
 
 template <TensorType tensor_type, typename integer_dtype>
-void QuantizedTestsNoActivation() {
+void QuantizedTestsNoActivation(ActivationFunctionType activation_func) {
   const float kQuantizedTolerance = 2.0 / 255.0;
   std::vector<std::vector<float>> inputs1 = {
       {0.1, 0.2, 0.3, 0.4}, {-0.8, 0.2, 0.4, 0.7}, {-0.8, 0.2, 0.7, 0.3}};
@@ -102,7 +106,7 @@ void QuantizedTestsNoActivation() {
   for (size_t i = 0; i < 1; ++i) {
     AddOpModel m({tensor_type, {1, 2, 2, 1}, -1.0, 1.0},
                  {tensor_type, {1, 2, 2, 1}, -1.0, 1.0},
-                 {tensor_type, {1, 2, 2, 1}, -1.0, 1.0});
+                 {tensor_type, {1, 2, 2, 1}, -1.0, 1.0}, activation_func);
     m.InitInterpreter();
     m.SetInput1<integer_dtype>(inputs1[i]);
     m.SetInput2<integer_dtype>(inputs2[i]);
@@ -116,20 +120,23 @@ void QuantizedTestsNoActivation() {
   }
 }
 
-TEST(QuantizedAddOpModel, QuantizedTestsNoActivationUInt8) {
-  QuantizedTestsNoActivation<TensorType_UINT8, uint8_t>();
+class QuantizedAddOpModel
+    : public testing::TestWithParam<ActivationFunctionType> {};
+
+TEST_P(QuantizedAddOpModel, QuantizedTestsNoActivationUInt8) {
+  QuantizedTestsNoActivation<TensorType_UINT8, uint8_t>(GetParam());
 }
 
-TEST(QuantizedAddOpModel, QuantizedTestsNoActivationInt8) {
-  QuantizedTestsNoActivation<TensorType_INT8, int8_t>();
+TEST_P(QuantizedAddOpModel, QuantizedTestsNoActivationInt8) {
+  QuantizedTestsNoActivation<TensorType_INT8, int8_t>(GetParam());
 }
 
-TEST(QuantizedAddOpModel, QuantizedTestsNoActivationUInt8_ConstInput_1) {
+TEST(QuantizedAddOpModelNoActivation, TestUInt8_ConstInput_1) {
   const float kQuantizedTolerance = 2.0 / 255.0;
   AddOpModel m({TensorType_UINT8, {1, 2, 2, 1}, -1.0, 1.0},
                {TensorType_UINT8, {1, 2, 2, 1}, -1.0, 1.0},
                {TensorType_UINT8, {1, 2, 2, 1}, -1.0, 1.0},
-               {110, 142, 156, 171}, {});
+               {110, 142, 156, 171}, {}, ActivationFunctionType_NONE);
   m.InitInterpreter();
   m.SetInput1<uint8_t>({0.1, 0.2, 0.3, 0.4});
   m.Invoke();
@@ -140,12 +147,12 @@ TEST(QuantizedAddOpModel, QuantizedTestsNoActivationUInt8_ConstInput_1) {
       ElementsAreArray(ArrayFloatNear(reference_output, kQuantizedTolerance)));
 }
 
-TEST(QuantizedAddOpModel, QuantizedTestsNoActivationUInt8_ConstInput_2) {
+TEST(QuantizedAddOpModelNoActivation, TestUInt8_ConstInput_2) {
   const float kQuantizedTolerance = 2.0 / 255.0;
   AddOpModel m({TensorType_UINT8, {1, 2, 2, 1}, -1.0, 1.0},
                {TensorType_UINT8, {1, 2, 2, 1}, -1.0, 1.0},
                {TensorType_UINT8, {1, 2, 2, 1}, -1.0, 1.0}, {},
-               {110, 142, 156, 171});
+               {110, 142, 156, 171}, ActivationFunctionType_NONE);
   m.InitInterpreter();
   m.SetInput2<uint8_t>({0.1, 0.2, 0.3, 0.4});
   m.Invoke();
@@ -156,12 +163,12 @@ TEST(QuantizedAddOpModel, QuantizedTestsNoActivationUInt8_ConstInput_2) {
       ElementsAreArray(ArrayFloatNear(reference_output, kQuantizedTolerance)));
 }
 
-TEST(QuantizedAddOpModel, QuantizedTestsNoActivationInt8_ConstInput) {
+TEST(QuantizedAddOpModelNoActivation, TestInt8_ConstInput) {
   const float kQuantizedTolerance = 2.0 / 255.0;
   AddOpModel m({TensorType_INT8, {1, 2, 2, 1}, -1.0, 1.0},
                {TensorType_INT8, {1, 2, 2, 1}, -1.0, 1.0},
                {TensorType_INT8, {1, 2, 2, 1}, -1.0, 1.0}, {},
-               {110, 101, 105, 120});
+               {110, 101, 105, 120}, ActivationFunctionType_NONE);
   m.InitInterpreter();
   m.SetInput2<int8_t>({0.1, 0.2, 0.3, 0.4});
   m.Invoke();
@@ -172,4 +179,10 @@ TEST(QuantizedAddOpModel, QuantizedTestsNoActivationInt8_ConstInput) {
       ElementsAreArray(ArrayFloatNear(reference_output, kQuantizedTolerance)));
 }
 
+INSTANTIATE_TEST_SUITE_P(QuantizedAddOpModel, QuantizedAddOpModel,
+                         testing::Values(ActivationFunctionType_NONE,
+                                         ActivationFunctionType_RELU,
+                                         ActivationFunctionType_RELU_N1_TO_1,
+                                         ActivationFunctionType_RELU6));
+
 }  // namespace tflite
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/tests/conv_test.cc b/tensorflow/lite/experimental/delegates/hexagon/builders/tests/conv_test.cc
index f204713304d..dc230f514ca 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/tests/conv_test.cc
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/tests/conv_test.cc
@@ -109,7 +109,7 @@ class QuantizedConvolutionOpModel : public SingleOpModelWithHexagon {
   }
 
   void SetBias(std::initializer_list<float> data) {
-    QuantizeAndPopulate<int32_t>(bias_, data);
+    QuantizeAndPopulate<int>(bias_, data);
   }
 
   template <typename T>
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/tests/matmul_test.cc b/tensorflow/lite/experimental/delegates/hexagon/builders/tests/matmul_test.cc
index 2be8753a84c..3b426b38da7 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/tests/matmul_test.cc
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/tests/matmul_test.cc
@@ -69,7 +69,7 @@ class FullyConnectedOpModel : public SingleOpModelWithHexagon {
   }
 
   void SetBias(const std::vector<float>& data) {
-    QuantizeAndPopulate<int32_t>(bias_, data);
+    QuantizeAndPopulate<int>(bias_, data);
   }
 
   template <typename T>
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/tests/mul_test.cc b/tensorflow/lite/experimental/delegates/hexagon/builders/tests/mul_test.cc
index 1ed2232f31f..c0b2ea78815 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/tests/mul_test.cc
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/tests/mul_test.cc
@@ -22,13 +22,13 @@ using testing::ElementsAreArray;
 class MulOpModel : public SingleOpModelWithHexagon {
  public:
   explicit MulOpModel(const TensorData& input1, const TensorData& input2,
-                      const TensorData& output) {
+                      const TensorData& output,
+                      ActivationFunctionType activation_func) {
     input1_ = AddInput(input1);
     input2_ = AddInput(input2);
     output_ = AddOutput(output);
-    SetBuiltinOp(
-        BuiltinOperator_MUL, BuiltinOptions_MulOptions,
-        CreateMulOptions(builder_, ActivationFunctionType_NONE).Union());
+    SetBuiltinOp(BuiltinOperator_MUL, BuiltinOptions_MulOptions,
+                 CreateMulOptions(builder_, activation_func).Union());
     BuildInterpreter({GetShape(input1_), GetShape(input2_)});
   }
 
@@ -57,11 +57,11 @@ class MulOpModel : public SingleOpModelWithHexagon {
 };
 
 template <TensorType tensor_type, typename integer_dtype>
-void TestMulOutputImpl() {
+void TestMulOutputImpl(ActivationFunctionType activation_func) {
   MulOpModel model(
       /*input1=*/{tensor_type, {2, 3}, -0.44f, 8.0f},
       /*input2=*/{tensor_type, {1, 3}, 0, 0.999f},
-      /*output=*/{tensor_type, {2, 3}, -0.44f, 4.996f});
+      /*output=*/{tensor_type, {2, 3}, -1.0f, 1.0f}, activation_func);
   model.SetInput1<integer_dtype>({1, 2, 3, 4, 5, 6});
   model.SetInput2<integer_dtype>({0.1f, 0.2f, 0.3f});
 
@@ -76,11 +76,11 @@ void TestMulOutputImpl() {
 }
 
 template <TensorType tensor_type, typename integer_dtype>
-void TestLargeInputRangeImpl() {
+void TestLargeInputRangeImpl(ActivationFunctionType activation_func) {
   MulOpModel model(
       /*input1=*/{tensor_type, {1, 2, 2, 3}, -0.44f, 55.7f},
       /*input2=*/{tensor_type, {1, 1, 2, 3}, 0, 0.999f},
-      /*output=*/{tensor_type, {1, 2, 2, 3}, -0.44f, 4.996f});
+      /*output=*/{tensor_type, {1, 2, 2, 3}, -1.0f, 1.0f}, activation_func);
   model.SetInput1<integer_dtype>({1, 2, 3, 4, 5, 6, 20, 30, 40, 50, 52, 55});
   model.SetInput2<integer_dtype>({0.8f, 0.9f, 0.99f, 0.8f, 0.9f, 0.99f});
 
@@ -94,20 +94,28 @@ void TestLargeInputRangeImpl() {
               ElementsAreArray(ArrayFloatNear(reference_out, 0.03)));
 }
 
-TEST(MulOpModel, MulOutput_UInt8) {
-  TestMulOutputImpl<TensorType_UINT8, uint8_t>();
+class MulOpModelTest : public testing::TestWithParam<ActivationFunctionType> {};
+
+TEST_P(MulOpModelTest, MulOutput_UInt8) {
+  TestMulOutputImpl<TensorType_UINT8, uint8_t>(GetParam());
 }
 
-TEST(MulOpModel, MulOutput_Int8) {
-  TestMulOutputImpl<TensorType_INT8, int8_t>();
+TEST_P(MulOpModelTest, MulOutput_Int8) {
+  TestMulOutputImpl<TensorType_INT8, int8_t>(GetParam());
 }
 
-TEST(MulOpModel, LargeInputRange_UInt8) {
-  TestLargeInputRangeImpl<TensorType_UINT8, uint8_t>();
+TEST_P(MulOpModelTest, LargeInputRange_UInt8) {
+  TestLargeInputRangeImpl<TensorType_UINT8, uint8_t>(GetParam());
 }
 
-TEST(MulOpModel, LargeInputRange_Int8) {
-  TestLargeInputRangeImpl<TensorType_INT8, int8_t>();
+TEST_P(MulOpModelTest, LargeInputRange_Int8) {
+  TestLargeInputRangeImpl<TensorType_INT8, int8_t>(GetParam());
 }
 
+INSTANTIATE_TEST_SUITE_P(MulOpModelTest, MulOpModelTest,
+                         testing::Values(ActivationFunctionType_NONE,
+                                         ActivationFunctionType_RELU,
+                                         ActivationFunctionType_RELU_N1_TO_1,
+                                         ActivationFunctionType_RELU6));
+
 }  // namespace tflite
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/tests/slice_test.cc b/tensorflow/lite/experimental/delegates/hexagon/builders/tests/slice_test.cc
index d3bcfb6a6c2..5b14a5d97d8 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/tests/slice_test.cc
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/tests/slice_test.cc
@@ -55,10 +55,10 @@ class SliceOpModel : public SingleOpModelWithHexagon {
 };
 
 TEST(SliceOpTest, Input_1D_Uint8) {
-  SliceOpModel<int32_t> m(/*input=*/{TensorType_UINT8, {4}, -10, 10},
-                          /*output=*/{TensorType_UINT8, {2}, -10, 10},
-                          {TensorType_INT32, {1}}, {TensorType_INT32, {1}}, {1},
-                          {2});
+  SliceOpModel<int> m(/*input=*/{TensorType_UINT8, {4}, -10, 10},
+                      /*output=*/{TensorType_UINT8, {2}, -10, 10},
+                      {TensorType_INT32, {1}}, {TensorType_INT32, {1}}, {1},
+                      {2});
   m.SetInput<uint8_t>({1, 2, 3, 4});
   m.ApplyDelegateAndInvoke();
   EXPECT_THAT(m.GetOutputShape(), ElementsAreArray({2}));
@@ -67,7 +67,7 @@ TEST(SliceOpTest, Input_1D_Uint8) {
 }
 
 TEST(SliceOpTest, Input_2D_Uint8) {
-  SliceOpModel<int32_t> m(
+  SliceOpModel<int> m(
       /*input=*/{TensorType_UINT8, {2, 3}, -10, 10},
       /*output=*/{TensorType_UINT8, {1, 2}, -10, 10}, {TensorType_INT32, {2}},
       {TensorType_INT32, {2}}, {1, 0}, {1, 2});
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/tests/strided_slice_test.cc b/tensorflow/lite/experimental/delegates/hexagon/builders/tests/strided_slice_test.cc
new file mode 100644
index 00000000000..8d62804ba8c
--- /dev/null
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/tests/strided_slice_test.cc
@@ -0,0 +1,235 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+#include <gtest/gtest.h>
+#include "tensorflow/lite/experimental/delegates/hexagon/builders/tests/hexagon_delegate_op_model.h"
+
+namespace tflite {
+using testing::ElementsAreArray;
+
+template <typename input_type>
+class StridedSliceOpModel : public SingleOpModelWithHexagon {
+ public:
+  StridedSliceOpModel(const TensorData& input, const TensorData& output,
+                      const TensorData& begin,
+                      std::initializer_list<int> begin_data,
+                      const TensorData& end,
+                      std::initializer_list<int> end_data,
+                      const TensorData& strides,
+                      std::initializer_list<int> strides_data, int begin_mask,
+                      int end_mask, int shrink_axis_mask) {
+    input_ = AddInput(input);
+    begin_ = AddConstInput(begin, begin_data);
+    end_ = AddConstInput(end, end_data);
+    strides_ = AddConstInput(strides, strides_data);
+    output_ = AddOutput(output);
+    SetBuiltinOp(BuiltinOperator_STRIDED_SLICE,
+                 BuiltinOptions_StridedSliceOptions,
+                 CreateStridedSliceOptions(
+                     builder_, begin_mask, end_mask, /*ellipsis_mask*/ 0,
+                     /*new_axis_mask*/ 0, shrink_axis_mask)
+                     .Union());
+    BuildInterpreter({GetShape(input_), GetShape(begin_), GetShape(end_),
+                      GetShape(strides_)});
+  }
+
+  void SetInput(std::initializer_list<input_type> data) {
+    PopulateTensor<input_type>(input_, data);
+  }
+
+  std::vector<input_type> GetOutput() {
+    return ExtractVector<input_type>(output_);
+  }
+  std::vector<int> GetOutputShape() { return GetTensorShape(output_); }
+
+ private:
+  int input_;
+  int begin_;
+  int end_;
+  int strides_;
+  int output_;
+};
+
+TEST(StridedSliceOpModel, In1D_UInt8) {
+  StridedSliceOpModel<uint8_t> m(
+      /*input=*/{TensorType_UINT8, {4}, -10, 10},
+      /*output=*/{TensorType_UINT8, {2}, -10, 10},
+      /*begin*/ {TensorType_INT32, {1}},
+      /*begin_data*/ {1},
+      /*end*/ {TensorType_INT32, {1}},
+      /*end_data*/ {3},
+      /*strides*/ {TensorType_INT32, {1}},
+      /*strides_data*/ {1},
+      /*begin_mask*/ 0, /*end_mask*/ 0, /*shrink_axis_mask*/ 0);
+  m.SetInput({1, 2, 3, 4});
+  m.ApplyDelegateAndInvoke();
+  EXPECT_THAT(m.GetOutputShape(), ElementsAreArray({2}));
+  EXPECT_THAT(m.GetOutput(), ElementsAreArray({2, 3}));
+}
+
+TEST(StridedSliceOpModel, In1D_NegativeBegin_Int8) {
+  StridedSliceOpModel<int8_t> m(
+      /*input=*/{TensorType_INT8, {4}, -10, 10},
+      /*output=*/{TensorType_INT8, {2}, -10, 10},
+      /*begin*/ {TensorType_INT32, {1}},
+      /*begin_data*/ {-3},
+      /*end*/ {TensorType_INT32, {1}},
+      /*end_data*/ {3},
+      /*strides*/ {TensorType_INT32, {1}},
+      /*strides_data*/ {1},
+      /*begin_mask*/ 0, /*end_mask*/ 0, /*shrink_axis_mask*/ 0);
+  m.SetInput({1, 2, 3, 4});
+  m.ApplyDelegateAndInvoke();
+  EXPECT_THAT(m.GetOutputShape(), ElementsAreArray({2}));
+  EXPECT_THAT(m.GetOutput(), ElementsAreArray({2, 3}));
+}
+
+TEST(StridedSliceOpModel, In1D_NegativeEnd_Int8) {
+  StridedSliceOpModel<int8_t> m(
+      /*input=*/{TensorType_INT8, {4}, -10, 10},
+      /*output=*/{TensorType_INT8, {1}, -10, 10},
+      /*begin*/ {TensorType_INT32, {1}},
+      /*begin_data*/ {1},
+      /*end*/ {TensorType_INT32, {1}},
+      /*end_data*/ {-2},
+      /*strides*/ {TensorType_INT32, {1}},
+      /*strides_data*/ {1},
+      /*begin_mask*/ 0, /*end_mask*/ 0, /*shrink_axis_mask*/ 0);
+  m.SetInput({1, 2, 3, 4});
+  m.ApplyDelegateAndInvoke();
+  EXPECT_THAT(m.GetOutputShape(), ElementsAreArray({1}));
+  EXPECT_THAT(m.GetOutput(), ElementsAreArray({2}));
+}
+
+TEST(StridedSliceOpModel, In2D_MultipleStrides_UInt8) {
+  StridedSliceOpModel<uint8_t> m(
+      /*input=*/{TensorType_UINT8, {2, 3}, -10, 10},
+      /*output=*/{TensorType_UINT8, {1, 3}, -10, 10},
+      /*begin*/ {TensorType_INT32, {2}},
+      /*begin_data*/ {1, -1},
+      /*end*/ {TensorType_INT32, {2}},
+      /*end_data*/ {2, -4},
+      /*strides*/ {TensorType_INT32, {2}},
+      /*strides_data*/ {2, -1},
+      /*begin_mask*/ 0, /*end_mask*/ 0, /*shrink_axis_mask*/ 0);
+  m.SetInput({1, 2, 3, 4, 5, 6});
+  m.ApplyDelegateAndInvoke();
+  EXPECT_THAT(m.GetOutputShape(), ElementsAreArray({1, 3}));
+  EXPECT_THAT(m.GetOutput(), ElementsAreArray({6, 5, 4}));
+}
+
+TEST(StridedSliceOpModel, In2D_EndMask_Int8) {
+  StridedSliceOpModel<int8_t> m(
+      /*input=*/{TensorType_INT8, {2, 3}, -127, 128},
+      /*output=*/{TensorType_INT8, {1, 3}, -127, 128},
+      /*begin*/ {TensorType_INT32, {2}},
+      /*begin_data*/ {1, 0},
+      /*end*/ {TensorType_INT32, {2}},
+      /*end_data*/ {2, 2},
+      /*strides*/ {TensorType_INT32, {2}},
+      /*strides_data*/ {1, 1},
+      /*begin_mask*/ 0, /*end_mask*/ 2, /*shrink_axis_mask*/ 0);
+  m.SetInput({1, 2, 3, 4, 5, 6});
+  m.ApplyDelegateAndInvoke();
+  EXPECT_THAT(m.GetOutputShape(), ElementsAreArray({1, 3}));
+  EXPECT_THAT(m.GetOutput(), ElementsAreArray({4, 5, 6}));
+}
+
+TEST(StridedSliceOpModel, In2D_NegStrideBeginMask_UInt8) {
+  StridedSliceOpModel<uint8_t> m(
+      /*input=*/{TensorType_UINT8, {2, 3}, -10, 10},
+      /*output=*/{TensorType_UINT8, {1, 3}, -10, 10},
+      /*begin*/ {TensorType_INT32, {2}},
+      /*begin_data*/ {1, -2},
+      /*end*/ {TensorType_INT32, {2}},
+      /*end_data*/ {2, -4},
+      /*strides*/ {TensorType_INT32, {2}},
+      /*strides_data*/ {1, -1},
+      /*begin_mask*/ 2, /*end_mask*/ 0, /*shrink_axis_mask*/ 0);
+  m.SetInput({1, 2, 3, 4, 5, 6});
+  m.ApplyDelegateAndInvoke();
+  EXPECT_THAT(m.GetOutputShape(), ElementsAreArray({1, 3}));
+  EXPECT_THAT(m.GetOutput(), ElementsAreArray({6, 5, 4}));
+}
+
+TEST(StridedSliceOpModel, In2D_ShrinkAxis2_BeginEndAxis1_NegativeSlice_Int8) {
+  StridedSliceOpModel<int8_t> m(
+      /*input=*/{TensorType_INT8, {4, 1}, -10, 10},
+      /*output=*/{TensorType_INT8, {4}, -10, 10},
+      /*begin*/ {TensorType_INT32, {2}},
+      /*begin_data*/ {0, -1},
+      /*end*/ {TensorType_INT32, {2}},
+      /*end_data*/ {0, 0},
+      /*strides*/ {TensorType_INT32, {2}},
+      /*strides_data*/ {1, 1},
+      /*begin_mask*/ 1, /*end_mask*/ 1, /*shrink_axis_mask*/ 2);
+  m.SetInput({0, 1, 2, 3});
+  m.ApplyDelegateAndInvoke();
+  EXPECT_THAT(m.GetOutputShape(), ElementsAreArray({4}));
+  EXPECT_THAT(m.GetOutput(), ElementsAreArray({0, 1, 2, 3}));
+}
+
+TEST(StridedSliceOpModel, In2D_ShrinkAxisMask3_Int8) {
+  StridedSliceOpModel<int8_t> m(
+      /*input=*/{TensorType_INT8, {2, 3}, -10, 10},
+      /*output=*/{TensorType_INT8, {}, -10, 10},
+      /*begin*/ {TensorType_INT32, {2}},
+      /*begin_data*/ {0, 0},
+      /*end*/ {TensorType_INT32, {2}},
+      /*end_data*/ {1, 1},
+      /*strides*/ {TensorType_INT32, {2}},
+      /*strides_data*/ {1, 1},
+      /*begin_mask*/ 0, /*end_mask*/ 0, /*shrink_axis_mask*/ 3);
+  m.SetInput({1, 2, 3, 4, 5, 6});
+  m.ApplyDelegateAndInvoke();
+  EXPECT_TRUE(m.GetOutputShape().empty());
+  EXPECT_THAT(m.GetOutput(), ElementsAreArray({1}));
+}
+
+TEST(StridedSliceOpModel, In3D_Identity_UInt8) {
+  StridedSliceOpModel<uint8_t> m(
+      /*input=*/{TensorType_UINT8, {2, 3, 2}, -15, 15},
+      /*output=*/{TensorType_UINT8, {2, 3, 2}, -15, 15},
+      /*begin*/ {TensorType_INT32, {3}},
+      /*begin_data*/ {0, 0, 0},
+      /*end*/ {TensorType_INT32, {3}},
+      /*end_data*/ {2, 3, 2},
+      /*strides*/ {TensorType_INT32, {3}},
+      /*strides_data*/ {1, 1, 1},
+      /*begin_mask*/ 0, /*end_mask*/ 0, /*shrink_axis_mask*/ 0);
+  m.SetInput({1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12});
+  m.ApplyDelegateAndInvoke();
+  EXPECT_THAT(m.GetOutputShape(), ElementsAreArray({2, 3, 2}));
+  EXPECT_THAT(m.GetOutput(),
+              ElementsAreArray({1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12}));
+}
+
+TEST(StridedSliceOpModel, In3D_IdentityShrinkAxis4_Int8) {
+  StridedSliceOpModel<int8_t> m(
+      /*input=*/{TensorType_INT8, {2, 3, 2}, -15, 15},
+      /*output=*/{TensorType_INT8, {2, 3, 2}, -15, 15},
+      /*begin*/ {TensorType_INT32, {3}},
+      /*begin_data*/ {0, 0, 0},
+      /*end*/ {TensorType_INT32, {3}},
+      /*end_data*/ {2, 3, 1},
+      /*strides*/ {TensorType_INT32, {3}},
+      /*strides_data*/ {1, 1, 1},
+      /*begin_mask*/ 0, /*end_mask*/ 0, /*shrink_axis_mask*/ 4);
+  m.SetInput({1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12});
+  m.ApplyDelegateAndInvoke();
+  EXPECT_THAT(m.GetOutputShape(), ElementsAreArray({2, 3}));
+  EXPECT_THAT(m.GetOutput(), ElementsAreArray({1, 3, 5, 7, 9, 11}));
+}
+
+}  // namespace tflite
diff --git a/tensorflow/lite/experimental/delegates/hexagon/builders/tests/transpose_test.cc b/tensorflow/lite/experimental/delegates/hexagon/builders/tests/transpose_test.cc
index bb2494d14e6..3d790253a3b 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/builders/tests/transpose_test.cc
+++ b/tensorflow/lite/experimental/delegates/hexagon/builders/tests/transpose_test.cc
@@ -35,7 +35,7 @@ class TransposeOpModel : public SingleOpModelWithHexagon {
                  CreateTransposeOptions(builder_).Union());
     BuildInterpreter({GetShape(input_)});
     if (!const_perm) {
-      PopulateTensor<int32_t>(perm_, perm);
+      PopulateTensor<int>(perm_, perm);
     }
   }
 
diff --git a/tensorflow/lite/experimental/delegates/hexagon/hexagon_delegate_kernel.cc b/tensorflow/lite/experimental/delegates/hexagon/hexagon_delegate_kernel.cc
index c4a73dab6e6..e051e48ab94 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/hexagon_delegate_kernel.cc
+++ b/tensorflow/lite/experimental/delegates/hexagon/hexagon_delegate_kernel.cc
@@ -28,12 +28,6 @@ limitations under the License.
 namespace tflite {
 
 namespace {
-
-// Used to convert int8 <-> uint8.
-constexpr int kSameScaleEffectiveMultiplier = 1 << 30;
-constexpr int kSameScaleEffectiveShift = 1;
-constexpr int kInt8Uint8ZeroPointDiff = 128;
-
 // Returns uint64 representing total cycles in 'perf_info' by
 // combining lo and hi counters.
 inline uint64_t GetCycles(const hexagon_nn_perfinfo& perf_info) {
diff --git a/tensorflow/lite/experimental/delegates/hexagon/utils.cc b/tensorflow/lite/experimental/delegates/hexagon/utils.cc
index 43096081615..fa3724715fb 100644
--- a/tensorflow/lite/experimental/delegates/hexagon/utils.cc
+++ b/tensorflow/lite/experimental/delegates/hexagon/utils.cc
@@ -97,6 +97,7 @@ bool CheckOpVersion(const TfLiteRegistration* registration) {
     case kTfLiteBuiltinSoftmax:
     case kTfLiteBuiltinSpaceToDepth:
     case kTfLiteBuiltinSplit:
+    case kTfLiteBuiltinStridedSlice:
     case kTfLiteBuiltinSub:
     case kTfLiteBuiltinTanh:
     case kTfLiteBuiltinTranspose:
@@ -142,7 +143,7 @@ bool IsNodeSupportedByHexagon(const TfLiteRegistration* registration,
         return false;
       const TfLiteAddParams* add_params =
           reinterpret_cast<const TfLiteAddParams*>(node->builtin_data);
-      return add_params->activation == kTfLiteActNone;
+      return IsActivationReluOrNone(add_params->activation);
     }
     case kTfLiteBuiltinMul: {
       if (!InputsWithCorrectTypes(
@@ -152,7 +153,7 @@ bool IsNodeSupportedByHexagon(const TfLiteRegistration* registration,
       const TfLiteMulParams* mul_params =
           reinterpret_cast<const TfLiteMulParams*>(node->builtin_data);
       // TODO(b/129276536): Add support for activation on Mul node.
-      return mul_params->activation == kTfLiteActNone;
+      return IsActivationReluOrNone(mul_params->activation);
     }
     case kTfLiteBuiltinSub: {
       if (!InputsWithCorrectTypes(
@@ -161,7 +162,7 @@ bool IsNodeSupportedByHexagon(const TfLiteRegistration* registration,
         return false;
       const TfLiteSubParams* sub_params =
           reinterpret_cast<const TfLiteSubParams*>(node->builtin_data);
-      return sub_params->activation == kTfLiteActNone;
+      return IsActivationReluOrNone(sub_params->activation);
     }
     case kTfLiteBuiltinSum:
       // TODO(b/139277813): Enable these when they pass unit tests. These seem
@@ -406,6 +407,24 @@ bool IsNodeSupportedByHexagon(const TfLiteRegistration* registration,
       }
       return true;
     }
+    case kTfLiteBuiltinStridedSlice: {
+      if (!InputsWithCorrectTypes(node, context,
+                                  {{kTfLiteUInt8, kTfLiteInt8},
+                                   {kTfLiteInt32},
+                                   {kTfLiteInt32},
+                                   {kTfLiteInt32}}))
+        return false;
+      const auto& begins_tensor = context->tensors[node->inputs->data[1]];
+      const auto& ends_tensor = context->tensors[node->inputs->data[2]];
+      const auto& step_tensor = context->tensors[node->inputs->data[3]];
+      if (!IsConstantTensor(&begins_tensor) ||
+          !IsConstantTensor(&ends_tensor) || !IsConstantTensor(&step_tensor))
+        return false;
+      const TfLiteStridedSliceParams* params =
+          reinterpret_cast<const TfLiteStridedSliceParams*>(node->builtin_data);
+      // Hexagon doesn't support ellipsis/new-axis masks.
+      return (params->ellipsis_mask == 0 && params->new_axis_mask == 0);
+    }
     default:
       return false;
   }
diff --git a/tensorflow/lite/experimental/ios/BUILD.apple b/tensorflow/lite/experimental/ios/BUILD.apple
index ddbfc0dec5b..aa41b9e2d62 100644
--- a/tensorflow/lite/experimental/ios/BUILD.apple
+++ b/tensorflow/lite/experimental/ios/BUILD.apple
@@ -11,16 +11,13 @@ package(
     licenses = ["notice"],  # Apache 2.0
 )
 
-TFL_FRAMEWORK_HDRS = [
-    "//tensorflow/lite/delegates/gpu:metal_delegate.h",
-    "//tensorflow/lite/c:c_api.h",
-    "//tensorflow/lite/c:common.h",
-]
-
 # bazel build -c opt --config=ios_fat //tensorflow/lite/experimental/ios:TensorFlowLiteC_framework
 ios_static_framework(
     name = "TensorFlowLiteC_framework",
-    hdrs = TFL_FRAMEWORK_HDRS,
+    hdrs = [
+        "//tensorflow/lite/c:c_api.h",
+        "//tensorflow/lite/c:common.h",
+    ],
     bundle_name = "TensorFlowLiteC",
     minimum_os_version = TFL_MINIMUM_OS_VERSION,
     deps = [
diff --git a/tensorflow/lite/experimental/support/metadata/cc/BUILD b/tensorflow/lite/experimental/support/metadata/cc/BUILD
index 2b288abe368..832e2edb56d 100644
--- a/tensorflow/lite/experimental/support/metadata/cc/BUILD
+++ b/tensorflow/lite/experimental/support/metadata/cc/BUILD
@@ -10,7 +10,9 @@ cc_library(
     deps = [
         "//tensorflow/lite/c:common",
         "//tensorflow/lite/experimental/support/metadata:metadata_schema_cc",
+        "//tensorflow/lite/kernels/internal:compatibility",
         "//tensorflow/lite/tools:logging",
+        "@com_google_absl//absl/strings",
         "@flatbuffers",
     ],
 )
diff --git a/tensorflow/lite/experimental/support/metadata/cc/metadata_version.cc b/tensorflow/lite/experimental/support/metadata/cc/metadata_version.cc
index 4f43c1431a7..971465f7747 100644
--- a/tensorflow/lite/experimental/support/metadata/cc/metadata_version.cc
+++ b/tensorflow/lite/experimental/support/metadata/cc/metadata_version.cc
@@ -14,16 +14,171 @@ limitations under the License.
 ==============================================================================*/
 #include "tensorflow/lite/experimental/support/metadata/cc/metadata_version.h"
 
+#include <stddef.h>
+#include <stdint.h>
+
+#include <array>
+#include <ostream>
+#include <string>
+#include <vector>
+
+#include "absl/strings/str_join.h"
+#include "absl/strings/str_split.h"
 #include "flatbuffers/flatbuffers.h"  // from @flatbuffers
+#include "tensorflow/lite/c/common.h"
 #include "tensorflow/lite/experimental/support/metadata/metadata_schema_generated.h"
+#include "tensorflow/lite/kernels/internal/compatibility.h"
 #include "tensorflow/lite/tools/logging.h"
 
 namespace tflite {
 namespace metadata {
+namespace {
+
+// Members that are added to the metadata schema after the initial version
+// of 1.0.0.
+enum class SchemaMembers {
+  kAssociatedFileTypeVocabulary = 0,
+};
+
+// Helper class to compare semantic versions in terms of three integers, major,
+// minor, and patch.
+class Version {
+ public:
+  explicit Version(int major, int minor = 0, int patch = 0)
+      : version_({major, minor, patch}) {}
+
+  explicit Version(const std::string& version) {
+    const std::vector<std::string> vec = absl::StrSplit(version, '.');
+    // The version string should always be less than four numbers.
+    TFLITE_DCHECK(vec.size() <= kElementNumber && !vec.empty());
+    version_[0] = std::stoi(vec[0]);
+    version_[1] = vec.size() > 1 ? std::stoi(vec[1]) : 0;
+    version_[2] = vec.size() > 2 ? std::stoi(vec[2]) : 0;
+  }
+
+  // Compares two semantic version numbers.
+  //
+  // Example results when comparing two versions strings:
+  //   "1.9" precedes "1.14";
+  //   "1.14" precedes "1.14.1";
+  //   "1.14" and "1.14.0" are equal.
+  //
+  // Returns the value 0 if the two versions are equal; a value less than 0 if
+  // *this precedes v; a value greater than 0 if v precedes *this.
+  int Compare(const Version& v) {
+    for (int i = 0; i < kElementNumber; ++i) {
+      if (version_[i] != v.version_[i]) {
+        return version_[i] < v.version_[i] ? -1 : 1;
+      }
+    }
+    return 0;
+  }
+
+  // Converts version_ into a version string.
+  std::string ToString() { return absl::StrJoin(version_, "."); }
+
+ private:
+  static constexpr int kElementNumber = 3;
+  std::array<int, kElementNumber> version_;
+};
+
+Version GetMemberVersion(SchemaMembers member) {
+  switch (member) {
+    case SchemaMembers::kAssociatedFileTypeVocabulary:
+      return Version(1, 0, 1);
+    default:
+      TFLITE_LOG(FATAL) << "Unsupported schema member: "
+                        << static_cast<int>(member);
+  }
+}
+
+// Updates min_version if it precedes the new_version.
+inline void UpdateMinimumVersion(const Version& new_version,
+                                 Version* min_version) {
+  if (min_version->Compare(new_version) < 0) {
+    *min_version = new_version;
+  }
+}
+
+void UpdateMinimumVersionForAssociatedFile(
+    const tflite::AssociatedFile* associated_file, Version* min_version) {
+  if (associated_file == nullptr) return;
+
+  if (associated_file->type() == AssociatedFileType_VOCABULARY) {
+    UpdateMinimumVersion(
+        GetMemberVersion(SchemaMembers::kAssociatedFileTypeVocabulary),
+        min_version);
+  }
+}
+
+void UpdateMinimumVersionForAssociatedFileArray(
+    const flatbuffers::Vector<flatbuffers::Offset<tflite::AssociatedFile>>*
+        associated_files,
+    Version* min_version) {
+  if (associated_files == nullptr) return;
+
+  for (int i = 0; i < associated_files->size(); ++i) {
+    UpdateMinimumVersionForAssociatedFile(associated_files->Get(i),
+                                          min_version);
+  }
+}
+
+void UpdateMinimumVersionForTensorMetadata(
+    const tflite::TensorMetadata* tensor_metadata, Version* min_version) {
+  if (tensor_metadata == nullptr) return;
+
+  // Checks the associated_files field.
+  UpdateMinimumVersionForAssociatedFileArray(
+      tensor_metadata->associated_files(), min_version);
+}
+
+void UpdateMinimumVersionForTensorMetadataArray(
+    const flatbuffers::Vector<flatbuffers::Offset<tflite::TensorMetadata>>*
+        tensor_metadata_array,
+    Version* min_version) {
+  if (tensor_metadata_array == nullptr) return;
+
+  for (int i = 0; i < tensor_metadata_array->size(); ++i) {
+    UpdateMinimumVersionForTensorMetadata(tensor_metadata_array->Get(i),
+                                          min_version);
+  }
+}
+
+void UpdateMinimumVersionForSubGraphMetadata(
+    const tflite::SubGraphMetadata* subgraph_metadata, Version* min_version) {
+  if (subgraph_metadata == nullptr) return;
+
+  // Checks in the input/output metadata arrays.
+  UpdateMinimumVersionForTensorMetadataArray(
+      subgraph_metadata->input_tensor_metadata(), min_version);
+  UpdateMinimumVersionForTensorMetadataArray(
+      subgraph_metadata->output_tensor_metadata(), min_version);
+
+  // Checks the associated_files field.
+  UpdateMinimumVersionForAssociatedFileArray(
+      subgraph_metadata->associated_files(), min_version);
+}
+
+void UpdateMinimumVersionForModelMetadata(
+    const tflite::ModelMetadata& model_metadata, Version* min_version) {
+  // Checks the subgraph_metadata field.
+  if (model_metadata.subgraph_metadata() != nullptr) {
+    for (int i = 0; i < model_metadata.subgraph_metadata()->size(); ++i) {
+      UpdateMinimumVersionForSubGraphMetadata(
+          model_metadata.subgraph_metadata()->Get(i), min_version);
+    }
+  }
+
+  // Checks the associated_files field.
+  UpdateMinimumVersionForAssociatedFileArray(model_metadata.associated_files(),
+                                             min_version);
+}
+
+}  // namespace
 
 TfLiteStatus GetMinimumMetadataParserVersion(const uint8_t* buffer_data,
                                              size_t buffer_size,
-                                             std::string* min_version) {
+                                             std::string* min_version_str) {
   flatbuffers::Verifier verifier =
       flatbuffers::Verifier(buffer_data, buffer_size);
   if (!tflite::VerifyModelMetadataBuffer(verifier)) {
@@ -31,18 +186,27 @@ TfLiteStatus GetMinimumMetadataParserVersion(const uint8_t* buffer_data,
     return kTfLiteError;
   }
 
-  // Returns the version as the initial default one, "1.0.0", because it is the
-  // first version ever for metadata_schema.fbs.
-  //
-  // Later, when new fields are added to the schema, we'll update the logic of
-  // getting the minimum metadata parser version. To be more specific, we'll
-  // have a table that records the new fields and the versions of the schema
-  // they are added to. And the minimum metadata parser version will be the
-  // largest version number of all fields that has been added to a metadata
-  // flatbuffer.
-  // TODO(b/156539454): replace the hardcoded version with template + genrule.
   static constexpr char kDefaultVersion[] = "1.0.0";
-  *min_version = kDefaultVersion;
+  Version min_version = Version(kDefaultVersion);
+
+  // Checks if any member declared after 1.0.0 (such as those in
+  // SchemaMembers) exists, and updates min_version accordingly. The minimum
+  // metadata parser version will be the largest version number of all fields
+  // that has been added to a metadata flatbuffer
+  const tflite::ModelMetadata* model_metadata = GetModelMetadata(buffer_data);
+
+  // All tables in the metadata schema should have their dedicated
+  // UpdateMinimumVersionFor**() methods, respectively. We'll gradually add
+  // these methods when new fields show up in later schema versions.
+  //
+  // UpdateMinimumVersionFor<Foo>() takes a const pointer of Foo. The pointer
+  // can be a nullptr if Foo is not populated into the corresponding table of
+  // the Flatbuffer object. In this case, UpdateMinimumVersionFor<Foo>() will be
+  // skipped. An exception is UpdateMinimumVersionForModelMetadata(), where
+  // ModelMetadata is the root table, and it won't be null.
+  UpdateMinimumVersionForModelMetadata(*model_metadata, &min_version);
+
+  *min_version_str = min_version.ToString();
   return kTfLiteOk;
 }
 
diff --git a/tensorflow/lite/experimental/support/metadata/cc/metadata_version.h b/tensorflow/lite/experimental/support/metadata/cc/metadata_version.h
index 71e90788af4..c4127118bc7 100644
--- a/tensorflow/lite/experimental/support/metadata/cc/metadata_version.h
+++ b/tensorflow/lite/experimental/support/metadata/cc/metadata_version.h
@@ -15,6 +15,9 @@ limitations under the License.
 #ifndef TENSORFLOW_LITE_EXPERIMENTAL_SUPPORT_METADATA_CC_METADATA_VERSION_H_
 #define TENSORFLOW_LITE_EXPERIMENTAL_SUPPORT_METADATA_CC_METADATA_VERSION_H_
 
+#include <stddef.h>
+#include <stdint.h>
+
 #include <string>
 
 #include "tensorflow/lite/c/common.h"
diff --git a/tensorflow/lite/experimental/support/metadata/cc/test/metadata_version_test.cc b/tensorflow/lite/experimental/support/metadata/cc/test/metadata_version_test.cc
index 00d9c0902c6..02ecfdbd232 100644
--- a/tensorflow/lite/experimental/support/metadata/cc/test/metadata_version_test.cc
+++ b/tensorflow/lite/experimental/support/metadata/cc/test/metadata_version_test.cc
@@ -14,6 +14,8 @@ limitations under the License.
 ==============================================================================*/
 #include "tensorflow/lite/experimental/support/metadata/cc/metadata_version.h"
 
+#include <vector>
+
 #include <gmock/gmock.h>
 #include <gtest/gtest.h>
 #include "flatbuffers/flatbuffers.h"  // from @flatbuffers
@@ -24,6 +26,7 @@ namespace metadata {
 namespace {
 
 using ::testing::MatchesRegex;
+using ::testing::StrEq;
 
 TEST(MetadataVersionTest,
      GetMinimumMetadataParserVersionSucceedsWithValidMetadata) {
@@ -45,7 +48,7 @@ TEST(MetadataVersionTest,
 }
 
 TEST(MetadataVersionTest,
-     GetMinimumMetadataParserVersionSucceedsWithInvalidIdentifier) {
+     GetMinimumMetadataParserVersionFailsWithInvalidIdentifier) {
   // Creates a dummy metadata flatbuffer without identifier.
   flatbuffers::FlatBufferBuilder builder(1024);
   ModelMetadataBuilder metadata_builder(builder);
@@ -60,6 +63,125 @@ TEST(MetadataVersionTest,
   EXPECT_TRUE(min_version.empty());
 }
 
+TEST(MetadataVersionTest,
+     GetMinimumMetadataParserVersionForModelMetadataVocabAssociatedFiles) {
+  // Creates a metadata flatbuffer with the field,
+  // ModelMetadata.associated_fiels, populated with the vocabulary file type.
+  flatbuffers::FlatBufferBuilder builder(1024);
+  AssociatedFileBuilder associated_file_builder(builder);
+  associated_file_builder.add_type(tflite::AssociatedFileType_VOCABULARY);
+  auto associated_files =
+      builder.CreateVector(std::vector<flatbuffers::Offset<AssociatedFile>>{
+          associated_file_builder.Finish()});
+  ModelMetadataBuilder metadata_builder(builder);
+  metadata_builder.add_associated_files(associated_files);
+  FinishModelMetadataBuffer(builder, metadata_builder.Finish());
+
+  // Gets the mimimum metadata parser version.
+  std::string min_version;
+  EXPECT_EQ(GetMinimumMetadataParserVersion(builder.GetBufferPointer(),
+                                            builder.GetSize(), &min_version),
+            kTfLiteOk);
+  // Validates that the version is exactly 1.0.1.
+  EXPECT_THAT(min_version, StrEq("1.0.1"));
+}
+
+TEST(MetadataVersionTest,
+     GetMinimumMetadataParserVersionForSubGraphMetadataVocabAssociatedFiles) {
+  // Creates a metadata flatbuffer with the field,
+  // SubGraphMetadata.associated_fiels, populated with the vocabulary file type.
+  flatbuffers::FlatBufferBuilder builder(1024);
+  AssociatedFileBuilder associated_file_builder(builder);
+  associated_file_builder.add_type(tflite::AssociatedFileType_VOCABULARY);
+  auto associated_files =
+      builder.CreateVector(std::vector<flatbuffers::Offset<AssociatedFile>>{
+          associated_file_builder.Finish()});
+  SubGraphMetadataBuilder subgraph_builder(builder);
+  subgraph_builder.add_associated_files(associated_files);
+  auto subgraphs =
+      builder.CreateVector(std::vector<flatbuffers::Offset<SubGraphMetadata>>{
+          subgraph_builder.Finish()});
+  ModelMetadataBuilder metadata_builder(builder);
+  metadata_builder.add_subgraph_metadata(subgraphs);
+  FinishModelMetadataBuffer(builder, metadata_builder.Finish());
+
+  // Gets the mimimum metadata parser version.
+  std::string min_version;
+  EXPECT_EQ(GetMinimumMetadataParserVersion(builder.GetBufferPointer(),
+                                            builder.GetSize(), &min_version),
+            kTfLiteOk);
+  // Validates that the version is exactly 1.0.1.
+  EXPECT_THAT(min_version, StrEq("1.0.1"));
+}
+
+TEST(MetadataVersionTest,
+     GetMinimumMetadataParserVersionForInputMetadataVocabAssociatedFiles) {
+  // Creates a metadata flatbuffer with the field,
+  // SubGraphMetadata.input_tensor_metadata.associated_fiels, populated with the
+  // vocabulary file type.
+  flatbuffers::FlatBufferBuilder builder(1024);
+  AssociatedFileBuilder associated_file_builder(builder);
+  associated_file_builder.add_type(tflite::AssociatedFileType_VOCABULARY);
+  auto associated_files =
+      builder.CreateVector(std::vector<flatbuffers::Offset<AssociatedFile>>{
+          associated_file_builder.Finish()});
+  TensorMetadataBuilder tensor_builder(builder);
+  tensor_builder.add_associated_files(associated_files);
+  auto tensors =
+      builder.CreateVector(std::vector<flatbuffers::Offset<TensorMetadata>>{
+          tensor_builder.Finish()});
+  SubGraphMetadataBuilder subgraph_builder(builder);
+  subgraph_builder.add_input_tensor_metadata(tensors);
+  auto subgraphs =
+      builder.CreateVector(std::vector<flatbuffers::Offset<SubGraphMetadata>>{
+          subgraph_builder.Finish()});
+  ModelMetadataBuilder metadata_builder(builder);
+  metadata_builder.add_subgraph_metadata(subgraphs);
+  FinishModelMetadataBuffer(builder, metadata_builder.Finish());
+
+  // Gets the mimimum metadata parser version.
+  std::string min_version;
+  EXPECT_EQ(GetMinimumMetadataParserVersion(builder.GetBufferPointer(),
+                                            builder.GetSize(), &min_version),
+            kTfLiteOk);
+  // Validates that the version is exactly 1.0.1.
+  EXPECT_THAT(min_version, StrEq("1.0.1"));
+}
+
+TEST(MetadataVersionTest,
+     GetMinimumMetadataParserVersionForOutputMetadataVocabAssociatedFiles) {
+  // Creates a metadata flatbuffer with the field,
+  // SubGraphMetadata.output_tensor_metadata.associated_fiels, populated with
+  // the vocabulary file type.
+  flatbuffers::FlatBufferBuilder builder(1024);
+  AssociatedFileBuilder associated_file_builder(builder);
+  associated_file_builder.add_type(tflite::AssociatedFileType_VOCABULARY);
+  auto associated_files =
+      builder.CreateVector(std::vector<flatbuffers::Offset<AssociatedFile>>{
+          associated_file_builder.Finish()});
+  TensorMetadataBuilder tensor_builder(builder);
+  tensor_builder.add_associated_files(associated_files);
+  auto tensors =
+      builder.CreateVector(std::vector<flatbuffers::Offset<TensorMetadata>>{
+          tensor_builder.Finish()});
+  SubGraphMetadataBuilder subgraph_builder(builder);
+  subgraph_builder.add_output_tensor_metadata(tensors);
+  auto subgraphs =
+      builder.CreateVector(std::vector<flatbuffers::Offset<SubGraphMetadata>>{
+          subgraph_builder.Finish()});
+  ModelMetadataBuilder metadata_builder(builder);
+  metadata_builder.add_subgraph_metadata(subgraphs);
+  FinishModelMetadataBuffer(builder, metadata_builder.Finish());
+
+  // Gets the mimimum metadata parser version.
+  std::string min_version;
+  EXPECT_EQ(GetMinimumMetadataParserVersion(builder.GetBufferPointer(),
+                                            builder.GetSize(), &min_version),
+            kTfLiteOk);
+  // Validates that the version is exactly 1.0.1.
+  EXPECT_EQ(min_version, "1.0.1");
+}
+
 }  // namespace
 }  // namespace metadata
 }  // namespace tflite
diff --git a/tensorflow/lite/experimental/support/metadata/java/src/java/org/tensorflow/lite/support/metadata/MetadataExtractor.java b/tensorflow/lite/experimental/support/metadata/java/src/java/org/tensorflow/lite/support/metadata/MetadataExtractor.java
index be4d8caf577..9da5b59cf46 100644
--- a/tensorflow/lite/experimental/support/metadata/java/src/java/org/tensorflow/lite/support/metadata/MetadataExtractor.java
+++ b/tensorflow/lite/experimental/support/metadata/java/src/java/org/tensorflow/lite/support/metadata/MetadataExtractor.java
@@ -55,7 +55,7 @@ public class MetadataExtractor {
   // TODO(b/156539454): remove the hardcode versioning number and populate the version through
   // genrule.
   /** The version of the metadata parser that this {@link MetadataExtractor} library depends on. */
-  public static final String METADATA_PARSER_VERSION = "1.0.0";
+  public static final String METADATA_PARSER_VERSION = "1.0.1";
 
   /** The helper class to load metadata from TFLite model FlatBuffer. */
   private final ModelInfo modelInfo;
diff --git a/tensorflow/lite/experimental/support/metadata/metadata_schema.fbs b/tensorflow/lite/experimental/support/metadata/metadata_schema.fbs
index 2883a91bd0c..f542ff7c172 100644
--- a/tensorflow/lite/experimental/support/metadata/metadata_schema.fbs
+++ b/tensorflow/lite/experimental/support/metadata/metadata_schema.fbs
@@ -40,23 +40,28 @@ namespace tflite;
 // New fields and types will have associated comments with the schema version for
 // which they were added.
 //
-// Schema Semantic version: 1.0.0
+// Schema Semantic version: 1.0.1
 
 // This indicates the flatbuffer compatibility. The number will bump up when a
 // break change is applied to the schema, such as removing fields or adding new
 // fields to the middle of a table.
 file_identifier "M001";
+
+// History:
+// 1.0.1 - Added VOCABULARY type to AssociatedFileType.
+
 // File extension of any written files.
 file_extension "tflitemeta";
 
 // LINT.ThenChange(//tensorflow/lite/experimental/\
-//     /supportmetadata/java/src/java/org/tensorflow/lite/support/metadata/\
+//     /support/metadata/java/src/java/org/tensorflow/lite/support/metadata/\
 //     MetadataExtractor.java)
 
 // LINT.IfChange
 enum AssociatedFileType : byte {
   UNKNOWN = 0,
-  // Files such as readme.txt
+
+  // Files such as readme.txt.
   DESCRIPTIONS = 1,
 
   // Contains labels that annotate certain axis of the tensor. For example,
@@ -98,6 +103,11 @@ enum AssociatedFileType : byte {
   //
   // [1]: https://en.cppreference.com/w/c/string/byte/strtof
   TENSOR_AXIS_SCORE_CALIBRATION = 4,
+
+  // Contains a list of unique words (characters separated by "\n" or in lines)
+  // that help to convert natural language words to embedding vectors.
+  // Added in: 1.0.1
+  VOCABULARY = 5,
 }
 
 table AssociatedFile {
diff --git a/tensorflow/lite/experimental/swift/BUILD.apple b/tensorflow/lite/experimental/swift/BUILD.apple
index b5e502b90f0..9ea45854fed 100644
--- a/tensorflow/lite/experimental/swift/BUILD.apple
+++ b/tensorflow/lite/experimental/swift/BUILD.apple
@@ -2,7 +2,7 @@
 
 load("//tensorflow/lite:special_rules.bzl", "ios_visibility_whitelist", "tflite_ios_lab_runner")
 load("//tensorflow/lite/experimental/ios:ios.bzl", "TFL_DEFAULT_TAGS", "TFL_DISABLED_SANITIZER_TAGS", "TFL_MINIMUM_OS_VERSION")
-load("@build_bazel_rules_apple//apple:ios.bzl", "ios_application", "ios_static_framework", "ios_unit_test")
+load("@build_bazel_rules_apple//apple:ios.bzl", "ios_static_framework", "ios_unit_test")
 load("@build_bazel_rules_swift//swift:swift.bzl", "swift_library")
 
 package(
@@ -10,21 +10,72 @@ package(
     licenses = ["notice"],  # Apache 2.0
 )
 
-# TODO(b/153554551): investigate if separate delegate libraries can be made with same module_name
-# If you don't need delegates and want to reduce size of the app, you can exclude Metal/Core ML
-# delegate related dependencies from the rule.
-# For example, if you don't want to use Core ML delegate:
-# 1. add `exclude = ["Sources/CoreMLDelegate.swift"]` to `glob`, so that `srcs` would look like this:
-#    ```
-#    srcs = glob(
-#        ["Sources/*.swift"],
-#        exclude = ["Sources/CoreMLDelegate.swift"],
-#    ),
-# 2. remove "-Wl,-weak_framework,CoreML" from `linkopts`
-# 3. remove "...:coreml_delegate" from `deps`
+config_setting(
+    name = "use_coreml_delegate",
+    values = {"define": "use_coreml_delegate=1"},
+)
 
+config_setting(
+    name = "use_metal_delegate",
+    values = {"define": "use_metal_delegate=1"},
+)
+
+# By default this builds with no delegates.
+# To build with the Metal delegate pass --define=use_metal_delegate=1
+# To build with the CoreML delegate pass --define=use_coreml_delegate=1
 swift_library(
     name = "TensorFlowLite",
+    srcs = glob(
+        [
+            "Sources/*.swift",
+        ],
+        exclude = [
+            "Sources/CoreMLDelegate.swift",
+            "Sources/MetalDelegate.swift",
+        ],
+    ) + select({
+        ":use_coreml_delegate": [
+            "Sources/CoreMLDelegate.swift",
+        ],
+        "//conditions:default": [],
+    }) + select({
+        ":use_metal_delegate": [
+            "Sources/MetalDelegate.swift",
+        ],
+        "//conditions:default": [],
+    }),
+    linkopts = select({
+        ":use_coreml_delegate": [
+            "-Wl,-weak_framework,CoreML",
+        ],
+        "//conditions:default": [],
+    }) + select({
+        ":use_metal_delegate": [
+            "-Wl,-weak_framework,Metal",
+        ],
+        "//conditions:default": [],
+    }),
+    module_name = "TensorFlowLite",
+    tags = TFL_DEFAULT_TAGS,
+    visibility = ios_visibility_whitelist(),
+    deps = [
+        "//tensorflow/lite/experimental/ios:tensorflow_lite_c",
+    ] + select({
+        ":use_coreml_delegate": [
+            "//tensorflow/lite/experimental/delegates/coreml:coreml_delegate",
+        ],
+        "//conditions:default": [],
+    }) + select({
+        ":use_metal_delegate": [
+            "//tensorflow/lite/delegates/gpu:metal_delegate",
+        ],
+        "//conditions:default": [],
+    }),
+)
+
+swift_library(
+    name = "TensorFlowLiteAllDelegates",
+    testonly = 1,
     srcs = glob(["Sources/*.swift"]),
     linkopts = [
         "-Wl,-weak_framework,CoreML",
@@ -32,7 +83,6 @@ swift_library(
     ],
     module_name = "TensorFlowLite",
     tags = TFL_DEFAULT_TAGS,
-    visibility = ios_visibility_whitelist(),
     deps = [
         "//tensorflow/lite/delegates/gpu:metal_delegate",
         "//tensorflow/lite/experimental/delegates/coreml:coreml_delegate",
@@ -71,38 +121,7 @@ swift_library(
     tags = TFL_DEFAULT_TAGS,
     deps = [
         ":Resources",
-        ":TensorFlowLite",
-    ],
-)
-
-ios_application(
-    name = "TestApp",
-    app_icons = glob(["TestApp/TestApp/Assets.xcassets/AppIcon.appiconset/**"]),
-    bundle_id = "com.tensorflow.lite.swift.TestApp",
-    families = [
-        "ipad",
-        "iphone",
-    ],
-    infoplists = ["TestApp/TestApp/Info.plist"],
-    minimum_os_version = TFL_MINIMUM_OS_VERSION,
-    sdk_frameworks = [
-        "CoreGraphics",
-    ],
-    tags = TFL_DEFAULT_TAGS,
-    deps = [
-        ":TestAppLibrary",
-    ],
-)
-
-swift_library(
-    name = "TestAppLibrary",
-    srcs = glob(["TestApp/TestApp/*.swift"]),
-    data = glob(["TestApp/TestApp/Base.lproj/*.storyboard"]),
-    module_name = "TestApp",
-    tags = TFL_DEFAULT_TAGS + ["manual"],
-    deps = [
-        ":Resources",
-        ":TensorFlowLite",
+        ":TensorFlowLiteAllDelegates",
     ],
 )
 
diff --git a/tensorflow/lite/experimental/swift/TensorFlowLite.tulsiproj/Configs/TensorFlowLite.tulsigen b/tensorflow/lite/experimental/swift/TensorFlowLite.tulsiproj/Configs/TensorFlowLite.tulsigen
index d919ada871d..21e59a675bc 100644
--- a/tensorflow/lite/experimental/swift/TensorFlowLite.tulsiproj/Configs/TensorFlowLite.tulsigen
+++ b/tensorflow/lite/experimental/swift/TensorFlowLite.tulsiproj/Configs/TensorFlowLite.tulsigen
@@ -1,57 +1,61 @@
 {
-  "sourceFilters" : [
-    "tensorflow/lite/c",
-    "tensorflow/lite/experimental/swift",
-    "tensorflow/lite/experimental/swift/Sources",
-    "tensorflow/lite/experimental/swift/TestApp/TestApp",
-    "tensorflow/lite/experimental/swift/TestApp/TestApp/Base.lproj",
-    "tensorflow/lite/experimental/swift/Tests",
+  "additionalFilePaths" : [
+    "tensorflow/lite/experimental/swift/BUILD"
   ],
   "buildTargets" : [
-    "//tensorflow/lite/experimental/swift:TensorFlowLite",
-    "//tensorflow/lite/experimental/swift:TestApp",
+    "//tensorflow/lite/experimental/swift:TensorFlowLiteAllDelegates",
     "//tensorflow/lite/experimental/swift:Tests",
+    "//tensorflow/lite/experimental/swift:TestsLibrary"
   ],
-  "projectName" : "TensorFlowLite",
   "optionSet" : {
-    "LaunchActionPreActionScript" : {
-      "p" : "$(inherited)"
-    },
-    "BazelBuildStartupOptionsRelease" : {
+    "BazelBuildOptionsDebug" : {
       "p" : "$(inherited)"
     },
     "BazelBuildOptionsRelease" : {
       "p" : "$(inherited)"
     },
-    "BazelBuildOptionsDebug" : {
-      "p" : "$(inherited)"
-    },
-    "EnvironmentVariables" : {
-      "p" : "$(inherited)"
-    },
-    "BuildActionPreActionScript" : {
-      "p" : "$(inherited)"
-    },
-    "CommandlineArguments" : {
-      "p" : "$(inherited)"
-    },
-    "TestActionPreActionScript" : {
-      "p" : "$(inherited)"
-    },
     "BazelBuildStartupOptionsDebug" : {
       "p" : "$(inherited)"
     },
+    "BazelBuildStartupOptionsRelease" : {
+      "p" : "$(inherited)"
+    },
     "BuildActionPostActionScript" : {
       "p" : "$(inherited)"
     },
-    "TestActionPostActionScript" : {
+    "BuildActionPreActionScript" : {
+      "p" : "$(inherited)"
+    },
+    "CLANG_CXX_LANGUAGE_STANDARD" : {
+      "p" : "c++14"
+    },
+    "CommandlineArguments" : {
+      "p" : "$(inherited)"
+    },
+    "EnvironmentVariables" : {
       "p" : "$(inherited)"
     },
     "LaunchActionPostActionScript" : {
       "p" : "$(inherited)"
+    },
+    "LaunchActionPreActionScript" : {
+      "p" : "$(inherited)"
+    },
+    "ProjectGenerationCompilationMode" : {
+      "p" : "opt"
+    },
+    "TestActionPostActionScript" : {
+      "p" : "$(inherited)"
+    },
+    "TestActionPreActionScript" : {
+      "p" : "$(inherited)"
     }
   },
-  "additionalFilePaths" : [
-    "tensorflow/lite/experimental/swift/BUILD"
+  "projectName" : "TensorFlowLite",
+  "sourceFilters" : [
+    "tensorflow/lite/c",
+    "tensorflow/lite/experimental/swift",
+    "tensorflow/lite/experimental/swift/Sources",
+    "tensorflow/lite/experimental/swift/Tests"
   ]
 }
diff --git a/tensorflow/lite/experimental/swift/TestApp/Podfile b/tensorflow/lite/experimental/swift/TestApp/Podfile
deleted file mode 100644
index 2cd46d56689..00000000000
--- a/tensorflow/lite/experimental/swift/TestApp/Podfile
+++ /dev/null
@@ -1,6 +0,0 @@
-platform :ios, '9.0'
-
-target 'TestApp' do
-  use_frameworks!
-  pod 'TensorFlowLiteSwift'
-end
diff --git a/tensorflow/lite/experimental/swift/TestApp/README.md b/tensorflow/lite/experimental/swift/TestApp/README.md
new file mode 100644
index 00000000000..7cd46132ac6
--- /dev/null
+++ b/tensorflow/lite/experimental/swift/TestApp/README.md
@@ -0,0 +1,7 @@
+# TensorFlow Lite Swift API Test App
+
+The TensorFlow Lite Swift API usage can be found in various iOS example apps
+provided in the following locations.
+
+*   [TensorFlow Lite example apps](https://www.tensorflow.org/lite/examples)
+*   [tensorflow/examples Repository](https://github.com/tensorflow/examples)
diff --git a/tensorflow/lite/experimental/swift/TestApp/TestApp.xcodeproj/project.pbxproj b/tensorflow/lite/experimental/swift/TestApp/TestApp.xcodeproj/project.pbxproj
deleted file mode 100644
index e6723e5a9a2..00000000000
--- a/tensorflow/lite/experimental/swift/TestApp/TestApp.xcodeproj/project.pbxproj
+++ /dev/null
@@ -1,358 +0,0 @@
-// !$*UTF8*$!
-{
-	archiveVersion = 1;
-	classes = {
-	};
-	objectVersion = 50;
-	objects = {
-
-/* Begin PBXBuildFile section */
-		4A1E2BA0227C8B53006C23E2 /* multi_add.bin in Resources */ = {isa = PBXBuildFile; fileRef = 4A1E2B9D227C8B51006C23E2 /* multi_add.bin */; };
-		4A1E2BA1227C8B53006C23E2 /* add_quantized.bin in Resources */ = {isa = PBXBuildFile; fileRef = 4A1E2B9E227C8B52006C23E2 /* add_quantized.bin */; };
-		4A1E2BA2227C8B53006C23E2 /* add.bin in Resources */ = {isa = PBXBuildFile; fileRef = 4A1E2B9F227C8B52006C23E2 /* add.bin */; };
-		4A7304B421500B8400C90B21 /* Data+TensorFlowLite.swift in Sources */ = {isa = PBXBuildFile; fileRef = 4A7304B321500B8300C90B21 /* Data+TensorFlowLite.swift */; };
-		4AA72B732146ED64006C3AEF /* AppDelegate.swift in Sources */ = {isa = PBXBuildFile; fileRef = 4AA72B722146ED64006C3AEF /* AppDelegate.swift */; };
-		4AA72B752146ED64006C3AEF /* ViewController.swift in Sources */ = {isa = PBXBuildFile; fileRef = 4AA72B742146ED64006C3AEF /* ViewController.swift */; };
-		4AA72B782146ED64006C3AEF /* Main.storyboard in Resources */ = {isa = PBXBuildFile; fileRef = 4AA72B762146ED64006C3AEF /* Main.storyboard */; };
-		4AA72B7A2146ED66006C3AEF /* Assets.xcassets in Resources */ = {isa = PBXBuildFile; fileRef = 4AA72B792146ED66006C3AEF /* Assets.xcassets */; };
-		4AA72B7D2146ED66006C3AEF /* LaunchScreen.storyboard in Resources */ = {isa = PBXBuildFile; fileRef = 4AA72B7B2146ED66006C3AEF /* LaunchScreen.storyboard */; };
-		4ADDE0CE2176600E00FF07A2 /* Array+TensorFlowLite.swift in Sources */ = {isa = PBXBuildFile; fileRef = 4ADDE0CD2176600900FF07A2 /* Array+TensorFlowLite.swift */; };
-/* End PBXBuildFile section */
-
-/* Begin PBXFileReference section */
-		4A1E2B9D227C8B51006C23E2 /* multi_add.bin */ = {isa = PBXFileReference; lastKnownFileType = archive.macbinary; name = multi_add.bin; path = ../../../../testdata/multi_add.bin; sourceTree = "<group>"; };
-		4A1E2B9E227C8B52006C23E2 /* add_quantized.bin */ = {isa = PBXFileReference; lastKnownFileType = archive.macbinary; name = add_quantized.bin; path = ../../../../testdata/add_quantized.bin; sourceTree = "<group>"; };
-		4A1E2B9F227C8B52006C23E2 /* add.bin */ = {isa = PBXFileReference; lastKnownFileType = archive.macbinary; name = add.bin; path = ../../../../testdata/add.bin; sourceTree = "<group>"; };
-		4A7304B321500B8300C90B21 /* Data+TensorFlowLite.swift */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.swift; path = "Data+TensorFlowLite.swift"; sourceTree = "<group>"; };
-		4AA72B6F2146ED64006C3AEF /* TestApp.app */ = {isa = PBXFileReference; explicitFileType = wrapper.application; includeInIndex = 0; path = TestApp.app; sourceTree = BUILT_PRODUCTS_DIR; };
-		4AA72B722146ED64006C3AEF /* AppDelegate.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = AppDelegate.swift; sourceTree = "<group>"; };
-		4AA72B742146ED64006C3AEF /* ViewController.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = ViewController.swift; sourceTree = "<group>"; };
-		4AA72B772146ED64006C3AEF /* Base */ = {isa = PBXFileReference; lastKnownFileType = file.storyboard; name = Base; path = Base.lproj/Main.storyboard; sourceTree = "<group>"; };
-		4AA72B792146ED66006C3AEF /* Assets.xcassets */ = {isa = PBXFileReference; lastKnownFileType = folder.assetcatalog; path = Assets.xcassets; sourceTree = "<group>"; };
-		4AA72B7C2146ED66006C3AEF /* Base */ = {isa = PBXFileReference; lastKnownFileType = file.storyboard; name = Base; path = Base.lproj/LaunchScreen.storyboard; sourceTree = "<group>"; };
-		4AA72B7E2146ED66006C3AEF /* Info.plist */ = {isa = PBXFileReference; lastKnownFileType = text.plist.xml; path = Info.plist; sourceTree = "<group>"; };
-		4ADDE0CD2176600900FF07A2 /* Array+TensorFlowLite.swift */ = {isa = PBXFileReference; fileEncoding = 4; lastKnownFileType = sourcecode.swift; path = "Array+TensorFlowLite.swift"; sourceTree = "<group>"; };
-/* End PBXFileReference section */
-
-/* Begin PBXFrameworksBuildPhase section */
-		4AA72B6C2146ED64006C3AEF /* Frameworks */ = {
-			isa = PBXFrameworksBuildPhase;
-			buildActionMask = 2147483647;
-			files = (
-			);
-			runOnlyForDeploymentPostprocessing = 0;
-		};
-/* End PBXFrameworksBuildPhase section */
-
-/* Begin PBXGroup section */
-		4AA72B662146ED64006C3AEF = {
-			isa = PBXGroup;
-			children = (
-				4AA72B712146ED64006C3AEF /* TestApp */,
-				4AA72B702146ED64006C3AEF /* Products */,
-			);
-			sourceTree = "<group>";
-		};
-		4AA72B702146ED64006C3AEF /* Products */ = {
-			isa = PBXGroup;
-			children = (
-				4AA72B6F2146ED64006C3AEF /* TestApp.app */,
-			);
-			name = Products;
-			sourceTree = "<group>";
-		};
-		4AA72B712146ED64006C3AEF /* TestApp */ = {
-			isa = PBXGroup;
-			children = (
-				4AA72B722146ED64006C3AEF /* AppDelegate.swift */,
-				4ADDE0CD2176600900FF07A2 /* Array+TensorFlowLite.swift */,
-				4A7304B321500B8300C90B21 /* Data+TensorFlowLite.swift */,
-				4AA72B742146ED64006C3AEF /* ViewController.swift */,
-				4AA72B762146ED64006C3AEF /* Main.storyboard */,
-				4AA72B792146ED66006C3AEF /* Assets.xcassets */,
-				4AA72B7B2146ED66006C3AEF /* LaunchScreen.storyboard */,
-				4AA72B7E2146ED66006C3AEF /* Info.plist */,
-				4A1E2B9E227C8B52006C23E2 /* add_quantized.bin */,
-				4A1E2B9F227C8B52006C23E2 /* add.bin */,
-				4A1E2B9D227C8B51006C23E2 /* multi_add.bin */,
-			);
-			path = TestApp;
-			sourceTree = "<group>";
-		};
-/* End PBXGroup section */
-
-/* Begin PBXNativeTarget section */
-		4AA72B6E2146ED64006C3AEF /* TestApp */ = {
-			isa = PBXNativeTarget;
-			buildConfigurationList = 4AA72B812146ED66006C3AEF /* Build configuration list for PBXNativeTarget "TestApp" */;
-			buildPhases = (
-				4AA72B6B2146ED64006C3AEF /* Sources */,
-				4AA72B6C2146ED64006C3AEF /* Frameworks */,
-				4AA72B6D2146ED64006C3AEF /* Resources */,
-			);
-			buildRules = (
-			);
-			dependencies = (
-			);
-			name = TestApp;
-			productName = TestApp;
-			productReference = 4AA72B6F2146ED64006C3AEF /* TestApp.app */;
-			productType = "com.apple.product-type.application";
-		};
-/* End PBXNativeTarget section */
-
-/* Begin PBXProject section */
-		4AA72B672146ED64006C3AEF /* Project object */ = {
-			isa = PBXProject;
-			attributes = {
-				LastSwiftUpdateCheck = 0940;
-				LastUpgradeCheck = 0940;
-				ORGANIZATIONNAME = Google;
-				TargetAttributes = {
-					4AA72B6E2146ED64006C3AEF = {
-						CreatedOnToolsVersion = 9.4.1;
-						LastSwiftMigration = 1020;
-					};
-				};
-			};
-			buildConfigurationList = 4AA72B6A2146ED64006C3AEF /* Build configuration list for PBXProject "TestApp" */;
-			compatibilityVersion = "Xcode 9.3";
-			developmentRegion = en;
-			hasScannedForEncodings = 0;
-			knownRegions = (
-				en,
-				Base,
-			);
-			mainGroup = 4AA72B662146ED64006C3AEF;
-			productRefGroup = 4AA72B702146ED64006C3AEF /* Products */;
-			projectDirPath = "";
-			projectRoot = "";
-			targets = (
-				4AA72B6E2146ED64006C3AEF /* TestApp */,
-			);
-		};
-/* End PBXProject section */
-
-/* Begin PBXResourcesBuildPhase section */
-		4AA72B6D2146ED64006C3AEF /* Resources */ = {
-			isa = PBXResourcesBuildPhase;
-			buildActionMask = 2147483647;
-			files = (
-				4A1E2BA1227C8B53006C23E2 /* add_quantized.bin in Resources */,
-				4AA72B7D2146ED66006C3AEF /* LaunchScreen.storyboard in Resources */,
-				4AA72B7A2146ED66006C3AEF /* Assets.xcassets in Resources */,
-				4A1E2BA2227C8B53006C23E2 /* add.bin in Resources */,
-				4AA72B782146ED64006C3AEF /* Main.storyboard in Resources */,
-				4A1E2BA0227C8B53006C23E2 /* multi_add.bin in Resources */,
-			);
-			runOnlyForDeploymentPostprocessing = 0;
-		};
-/* End PBXResourcesBuildPhase section */
-
-/* Begin PBXSourcesBuildPhase section */
-		4AA72B6B2146ED64006C3AEF /* Sources */ = {
-			isa = PBXSourcesBuildPhase;
-			buildActionMask = 2147483647;
-			files = (
-				4AA72B732146ED64006C3AEF /* AppDelegate.swift in Sources */,
-				4ADDE0CE2176600E00FF07A2 /* Array+TensorFlowLite.swift in Sources */,
-				4A7304B421500B8400C90B21 /* Data+TensorFlowLite.swift in Sources */,
-				4AA72B752146ED64006C3AEF /* ViewController.swift in Sources */,
-			);
-			runOnlyForDeploymentPostprocessing = 0;
-		};
-/* End PBXSourcesBuildPhase section */
-
-/* Begin PBXVariantGroup section */
-		4AA72B762146ED64006C3AEF /* Main.storyboard */ = {
-			isa = PBXVariantGroup;
-			children = (
-				4AA72B772146ED64006C3AEF /* Base */,
-			);
-			name = Main.storyboard;
-			sourceTree = "<group>";
-		};
-		4AA72B7B2146ED66006C3AEF /* LaunchScreen.storyboard */ = {
-			isa = PBXVariantGroup;
-			children = (
-				4AA72B7C2146ED66006C3AEF /* Base */,
-			);
-			name = LaunchScreen.storyboard;
-			sourceTree = "<group>";
-		};
-/* End PBXVariantGroup section */
-
-/* Begin XCBuildConfiguration section */
-		4AA72B7F2146ED66006C3AEF /* Debug */ = {
-			isa = XCBuildConfiguration;
-			buildSettings = {
-				ALWAYS_SEARCH_USER_PATHS = NO;
-				CLANG_ANALYZER_NONNULL = YES;
-				CLANG_ANALYZER_NUMBER_OBJECT_CONVERSION = YES_AGGRESSIVE;
-				CLANG_CXX_LANGUAGE_STANDARD = "gnu++14";
-				CLANG_CXX_LIBRARY = "libc++";
-				CLANG_ENABLE_MODULES = YES;
-				CLANG_ENABLE_OBJC_ARC = YES;
-				CLANG_ENABLE_OBJC_WEAK = YES;
-				CLANG_WARN_BLOCK_CAPTURE_AUTORELEASING = YES;
-				CLANG_WARN_BOOL_CONVERSION = YES;
-				CLANG_WARN_COMMA = YES;
-				CLANG_WARN_CONSTANT_CONVERSION = YES;
-				CLANG_WARN_DEPRECATED_OBJC_IMPLEMENTATIONS = YES;
-				CLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR;
-				CLANG_WARN_DOCUMENTATION_COMMENTS = YES;
-				CLANG_WARN_EMPTY_BODY = YES;
-				CLANG_WARN_ENUM_CONVERSION = YES;
-				CLANG_WARN_INFINITE_RECURSION = YES;
-				CLANG_WARN_INT_CONVERSION = YES;
-				CLANG_WARN_NON_LITERAL_NULL_CONVERSION = YES;
-				CLANG_WARN_OBJC_IMPLICIT_RETAIN_SELF = YES;
-				CLANG_WARN_OBJC_LITERAL_CONVERSION = YES;
-				CLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR;
-				CLANG_WARN_RANGE_LOOP_ANALYSIS = YES;
-				CLANG_WARN_STRICT_PROTOTYPES = YES;
-				CLANG_WARN_SUSPICIOUS_MOVE = YES;
-				CLANG_WARN_UNGUARDED_AVAILABILITY = YES_AGGRESSIVE;
-				CLANG_WARN_UNREACHABLE_CODE = YES;
-				CLANG_WARN__DUPLICATE_METHOD_MATCH = YES;
-				CODE_SIGN_IDENTITY = "iPhone Developer";
-				COPY_PHASE_STRIP = NO;
-				DEBUG_INFORMATION_FORMAT = dwarf;
-				ENABLE_STRICT_OBJC_MSGSEND = YES;
-				ENABLE_TESTABILITY = YES;
-				GCC_C_LANGUAGE_STANDARD = gnu11;
-				GCC_DYNAMIC_NO_PIC = NO;
-				GCC_NO_COMMON_BLOCKS = YES;
-				GCC_OPTIMIZATION_LEVEL = 0;
-				GCC_PREPROCESSOR_DEFINITIONS = (
-					"DEBUG=1",
-					"$(inherited)",
-				);
-				GCC_WARN_64_TO_32_BIT_CONVERSION = YES;
-				GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR;
-				GCC_WARN_UNDECLARED_SELECTOR = YES;
-				GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE;
-				GCC_WARN_UNUSED_FUNCTION = YES;
-				GCC_WARN_UNUSED_VARIABLE = YES;
-				IPHONEOS_DEPLOYMENT_TARGET = 11.4;
-				MTL_ENABLE_DEBUG_INFO = YES;
-				ONLY_ACTIVE_ARCH = YES;
-				SDKROOT = iphoneos;
-				SWIFT_ACTIVE_COMPILATION_CONDITIONS = DEBUG;
-				SWIFT_OPTIMIZATION_LEVEL = "-Onone";
-			};
-			name = Debug;
-		};
-		4AA72B802146ED66006C3AEF /* Release */ = {
-			isa = XCBuildConfiguration;
-			buildSettings = {
-				ALWAYS_SEARCH_USER_PATHS = NO;
-				CLANG_ANALYZER_NONNULL = YES;
-				CLANG_ANALYZER_NUMBER_OBJECT_CONVERSION = YES_AGGRESSIVE;
-				CLANG_CXX_LANGUAGE_STANDARD = "gnu++14";
-				CLANG_CXX_LIBRARY = "libc++";
-				CLANG_ENABLE_MODULES = YES;
-				CLANG_ENABLE_OBJC_ARC = YES;
-				CLANG_ENABLE_OBJC_WEAK = YES;
-				CLANG_WARN_BLOCK_CAPTURE_AUTORELEASING = YES;
-				CLANG_WARN_BOOL_CONVERSION = YES;
-				CLANG_WARN_COMMA = YES;
-				CLANG_WARN_CONSTANT_CONVERSION = YES;
-				CLANG_WARN_DEPRECATED_OBJC_IMPLEMENTATIONS = YES;
-				CLANG_WARN_DIRECT_OBJC_ISA_USAGE = YES_ERROR;
-				CLANG_WARN_DOCUMENTATION_COMMENTS = YES;
-				CLANG_WARN_EMPTY_BODY = YES;
-				CLANG_WARN_ENUM_CONVERSION = YES;
-				CLANG_WARN_INFINITE_RECURSION = YES;
-				CLANG_WARN_INT_CONVERSION = YES;
-				CLANG_WARN_NON_LITERAL_NULL_CONVERSION = YES;
-				CLANG_WARN_OBJC_IMPLICIT_RETAIN_SELF = YES;
-				CLANG_WARN_OBJC_LITERAL_CONVERSION = YES;
-				CLANG_WARN_OBJC_ROOT_CLASS = YES_ERROR;
-				CLANG_WARN_RANGE_LOOP_ANALYSIS = YES;
-				CLANG_WARN_STRICT_PROTOTYPES = YES;
-				CLANG_WARN_SUSPICIOUS_MOVE = YES;
-				CLANG_WARN_UNGUARDED_AVAILABILITY = YES_AGGRESSIVE;
-				CLANG_WARN_UNREACHABLE_CODE = YES;
-				CLANG_WARN__DUPLICATE_METHOD_MATCH = YES;
-				CODE_SIGN_IDENTITY = "iPhone Developer";
-				COPY_PHASE_STRIP = NO;
-				DEBUG_INFORMATION_FORMAT = "dwarf-with-dsym";
-				ENABLE_NS_ASSERTIONS = NO;
-				ENABLE_STRICT_OBJC_MSGSEND = YES;
-				GCC_C_LANGUAGE_STANDARD = gnu11;
-				GCC_NO_COMMON_BLOCKS = YES;
-				GCC_WARN_64_TO_32_BIT_CONVERSION = YES;
-				GCC_WARN_ABOUT_RETURN_TYPE = YES_ERROR;
-				GCC_WARN_UNDECLARED_SELECTOR = YES;
-				GCC_WARN_UNINITIALIZED_AUTOS = YES_AGGRESSIVE;
-				GCC_WARN_UNUSED_FUNCTION = YES;
-				GCC_WARN_UNUSED_VARIABLE = YES;
-				IPHONEOS_DEPLOYMENT_TARGET = 11.4;
-				MTL_ENABLE_DEBUG_INFO = NO;
-				SDKROOT = iphoneos;
-				SWIFT_COMPILATION_MODE = wholemodule;
-				SWIFT_OPTIMIZATION_LEVEL = "-O";
-				VALIDATE_PRODUCT = YES;
-			};
-			name = Release;
-		};
-		4AA72B822146ED66006C3AEF /* Debug */ = {
-			isa = XCBuildConfiguration;
-			buildSettings = {
-				ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon;
-				CODE_SIGN_STYLE = Automatic;
-				INFOPLIST_FILE = TestApp/Info.plist;
-				LD_RUNPATH_SEARCH_PATHS = (
-					"$(inherited)",
-					"@executable_path/Frameworks",
-				);
-				PRODUCT_BUNDLE_IDENTIFIER = com.tensorflow.lite.swift.TestApp;
-				PRODUCT_NAME = "$(TARGET_NAME)";
-				SWIFT_VERSION = 5.0;
-				TARGETED_DEVICE_FAMILY = "1,2";
-			};
-			name = Debug;
-		};
-		4AA72B832146ED66006C3AEF /* Release */ = {
-			isa = XCBuildConfiguration;
-			buildSettings = {
-				ASSETCATALOG_COMPILER_APPICON_NAME = AppIcon;
-				CODE_SIGN_STYLE = Automatic;
-				INFOPLIST_FILE = TestApp/Info.plist;
-				LD_RUNPATH_SEARCH_PATHS = (
-					"$(inherited)",
-					"@executable_path/Frameworks",
-				);
-				PRODUCT_BUNDLE_IDENTIFIER = com.tensorflow.lite.swift.TestApp;
-				PRODUCT_NAME = "$(TARGET_NAME)";
-				SWIFT_VERSION = 5.0;
-				TARGETED_DEVICE_FAMILY = "1,2";
-			};
-			name = Release;
-		};
-/* End XCBuildConfiguration section */
-
-/* Begin XCConfigurationList section */
-		4AA72B6A2146ED64006C3AEF /* Build configuration list for PBXProject "TestApp" */ = {
-			isa = XCConfigurationList;
-			buildConfigurations = (
-				4AA72B7F2146ED66006C3AEF /* Debug */,
-				4AA72B802146ED66006C3AEF /* Release */,
-			);
-			defaultConfigurationIsVisible = 0;
-			defaultConfigurationName = Release;
-		};
-		4AA72B812146ED66006C3AEF /* Build configuration list for PBXNativeTarget "TestApp" */ = {
-			isa = XCConfigurationList;
-			buildConfigurations = (
-				4AA72B822146ED66006C3AEF /* Debug */,
-				4AA72B832146ED66006C3AEF /* Release */,
-			);
-			defaultConfigurationIsVisible = 0;
-			defaultConfigurationName = Release;
-		};
-/* End XCConfigurationList section */
-	};
-	rootObject = 4AA72B672146ED64006C3AEF /* Project object */;
-}
diff --git a/tensorflow/lite/experimental/swift/TestApp/TestApp/AppDelegate.swift b/tensorflow/lite/experimental/swift/TestApp/TestApp/AppDelegate.swift
deleted file mode 100644
index cc22043f7bb..00000000000
--- a/tensorflow/lite/experimental/swift/TestApp/TestApp/AppDelegate.swift
+++ /dev/null
@@ -1,20 +0,0 @@
-// Copyright 2019 Google Inc. All rights reserved.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at:
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-import UIKit
-
-@UIApplicationMain
-final class AppDelegate: UIResponder, UIApplicationDelegate {
-  var window: UIWindow?
-}
diff --git a/tensorflow/lite/experimental/swift/TestApp/TestApp/Array+TensorFlowLite.swift b/tensorflow/lite/experimental/swift/TestApp/TestApp/Array+TensorFlowLite.swift
deleted file mode 100644
index e2853e153eb..00000000000
--- a/tensorflow/lite/experimental/swift/TestApp/TestApp/Array+TensorFlowLite.swift
+++ /dev/null
@@ -1,39 +0,0 @@
-// Copyright 2019 Google Inc. All rights reserved.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at:
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-import Foundation
-
-extension Array {
-  /// Creates a new array from the bytes of the given unsafe data.
-  ///
-  /// - Warning: The array's `Element` type must be trivial in that it can be copied bit for bit
-  ///     with no indirection or reference-counting operations; otherwise, copying the raw bytes in
-  ///     the `unsafeData`'s buffer to a new array returns an unsafe copy.
-  /// - Note: Returns `nil` if `unsafeData.count` is not a multiple of
-  ///     `MemoryLayout<Element>.stride`.
-  /// - Parameter unsafeData: The data containing the bytes to turn into an array.
-  init?(unsafeData: Data) {
-    guard unsafeData.count % MemoryLayout<Element>.stride == 0 else { return nil }
-    #if swift(>=5.0)
-    self = unsafeData.withUnsafeBytes { .init($0.bindMemory(to: Element.self)) }
-    #else
-    self = unsafeData.withUnsafeBytes {
-      .init(UnsafeBufferPointer<Element>(
-        start: $0,
-        count: unsafeData.count / MemoryLayout<Element>.stride
-      ))
-    }
-    #endif  // swift(>=5.0)
-  }
-}
diff --git a/tensorflow/lite/experimental/swift/TestApp/TestApp/Assets.xcassets/AppIcon.appiconset/Contents.json b/tensorflow/lite/experimental/swift/TestApp/TestApp/Assets.xcassets/AppIcon.appiconset/Contents.json
deleted file mode 100644
index d8db8d65fd7..00000000000
--- a/tensorflow/lite/experimental/swift/TestApp/TestApp/Assets.xcassets/AppIcon.appiconset/Contents.json
+++ /dev/null
@@ -1,98 +0,0 @@
-{
-  "images" : [
-    {
-      "idiom" : "iphone",
-      "size" : "20x20",
-      "scale" : "2x"
-    },
-    {
-      "idiom" : "iphone",
-      "size" : "20x20",
-      "scale" : "3x"
-    },
-    {
-      "idiom" : "iphone",
-      "size" : "29x29",
-      "scale" : "2x"
-    },
-    {
-      "idiom" : "iphone",
-      "size" : "29x29",
-      "scale" : "3x"
-    },
-    {
-      "idiom" : "iphone",
-      "size" : "40x40",
-      "scale" : "2x"
-    },
-    {
-      "idiom" : "iphone",
-      "size" : "40x40",
-      "scale" : "3x"
-    },
-    {
-      "idiom" : "iphone",
-      "size" : "60x60",
-      "scale" : "2x"
-    },
-    {
-      "idiom" : "iphone",
-      "size" : "60x60",
-      "scale" : "3x"
-    },
-    {
-      "idiom" : "ipad",
-      "size" : "20x20",
-      "scale" : "1x"
-    },
-    {
-      "idiom" : "ipad",
-      "size" : "20x20",
-      "scale" : "2x"
-    },
-    {
-      "idiom" : "ipad",
-      "size" : "29x29",
-      "scale" : "1x"
-    },
-    {
-      "idiom" : "ipad",
-      "size" : "29x29",
-      "scale" : "2x"
-    },
-    {
-      "idiom" : "ipad",
-      "size" : "40x40",
-      "scale" : "1x"
-    },
-    {
-      "idiom" : "ipad",
-      "size" : "40x40",
-      "scale" : "2x"
-    },
-    {
-      "idiom" : "ipad",
-      "size" : "76x76",
-      "scale" : "1x"
-    },
-    {
-      "idiom" : "ipad",
-      "size" : "76x76",
-      "scale" : "2x"
-    },
-    {
-      "idiom" : "ipad",
-      "size" : "83.5x83.5",
-      "scale" : "2x"
-    },
-    {
-      "idiom" : "ios-marketing",
-      "size" : "1024x1024",
-      "scale" : "1x"
-    }
-  ],
-  "info" : {
-    "version" : 1,
-    "author" : "xcode"
-  }
-}
\ No newline at end of file
diff --git a/tensorflow/lite/experimental/swift/TestApp/TestApp/Assets.xcassets/Contents.json b/tensorflow/lite/experimental/swift/TestApp/TestApp/Assets.xcassets/Contents.json
deleted file mode 100644
index da4a164c918..00000000000
--- a/tensorflow/lite/experimental/swift/TestApp/TestApp/Assets.xcassets/Contents.json
+++ /dev/null
@@ -1,6 +0,0 @@
-{
-  "info" : {
-    "version" : 1,
-    "author" : "xcode"
-  }
-}
\ No newline at end of file
diff --git a/tensorflow/lite/experimental/swift/TestApp/TestApp/Base.lproj/LaunchScreen.storyboard b/tensorflow/lite/experimental/swift/TestApp/TestApp/Base.lproj/LaunchScreen.storyboard
deleted file mode 100644
index 83d172a2309..00000000000
--- a/tensorflow/lite/experimental/swift/TestApp/TestApp/Base.lproj/LaunchScreen.storyboard
+++ /dev/null
@@ -1,43 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="14490.70" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" launchScreen="YES" useTraitCollections="YES" colorMatched="YES" initialViewController="01J-lp-oVM">
-    <device id="retina4_7" orientation="portrait">
-        <adaptation id="fullscreen"/>
-    </device>
-    <dependencies>
-        <plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="14490.49"/>
-        <capability name="documents saved in the Xcode 8 format" minToolsVersion="8.0"/>
-    </dependencies>
-    <scenes>
-        <!--View Controller-->
-        <scene sceneID="EHf-IW-A2E">
-            <objects>
-                <viewController id="01J-lp-oVM" sceneMemberID="viewController">
-                    <layoutGuides>
-                        <viewControllerLayoutGuide type="top" id="Llm-lL-Icb"/>
-                        <viewControllerLayoutGuide type="bottom" id="xb3-aO-Qok"/>
-                    </layoutGuides>
-                    <view key="view" contentMode="scaleToFill" id="Ze5-6b-2t3">
-                        <rect key="frame" x="0.0" y="0.0" width="375" height="667"/>
-                        <autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
-                        <subviews>
-                            <label opaque="NO" userInteractionEnabled="NO" contentMode="left" horizontalHuggingPriority="251" verticalHuggingPriority="251" text="TensorFlow Lite" textAlignment="center" lineBreakMode="tailTruncation" baselineAdjustment="alignBaselines" adjustsFontSizeToFit="NO" translatesAutoresizingMaskIntoConstraints="NO" id="3Gq-PV-hia">
-                                <rect key="frame" x="16" y="314" width="343" height="39"/>
-                                <fontDescription key="fontDescription" type="boldSystem" pointSize="32"/>
-                                <color key="textColor" red="1" green="0.43529411764705883" blue="0.0" alpha="1" colorSpace="custom" customColorSpace="sRGB"/>
-                                <nil key="highlightedColor"/>
-                            </label>
-                        </subviews>
-                        <color key="backgroundColor" red="1" green="1" blue="1" alpha="1" colorSpace="custom" customColorSpace="sRGB"/>
-                        <constraints>
-                            <constraint firstItem="3Gq-PV-hia" firstAttribute="leading" secondItem="Ze5-6b-2t3" secondAttribute="leading" constant="16" id="aXL-9T-5Pf"/>
-                            <constraint firstItem="3Gq-PV-hia" firstAttribute="centerY" secondItem="Ze5-6b-2t3" secondAttribute="centerY" id="cDf-Go-1FR"/>
-                            <constraint firstAttribute="trailing" secondItem="3Gq-PV-hia" secondAttribute="trailing" constant="16" id="fB9-BX-A3B"/>
-                        </constraints>
-                    </view>
-                </viewController>
-                <placeholder placeholderIdentifier="IBFirstResponder" id="iYj-Kq-Ea1" userLabel="First Responder" sceneMemberID="firstResponder"/>
-            </objects>
-            <point key="canvasLocation" x="52" y="374.66266866566718"/>
-        </scene>
-    </scenes>
-</document>
diff --git a/tensorflow/lite/experimental/swift/TestApp/TestApp/Base.lproj/Main.storyboard b/tensorflow/lite/experimental/swift/TestApp/TestApp/Base.lproj/Main.storyboard
deleted file mode 100644
index a93a4f0272e..00000000000
--- a/tensorflow/lite/experimental/swift/TestApp/TestApp/Base.lproj/Main.storyboard
+++ /dev/null
@@ -1,96 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<document type="com.apple.InterfaceBuilder3.CocoaTouch.Storyboard.XIB" version="3.0" toolsVersion="14490.70" targetRuntime="iOS.CocoaTouch" propertyAccessControl="none" useAutolayout="YES" useTraitCollections="YES" colorMatched="YES" initialViewController="BYZ-38-t0r">
-    <device id="retina4_7" orientation="portrait">
-        <adaptation id="fullscreen"/>
-    </device>
-    <dependencies>
-        <deployment identifier="iOS"/>
-        <plugIn identifier="com.apple.InterfaceBuilder.IBCocoaTouchPlugin" version="14490.49"/>
-        <capability name="documents saved in the Xcode 8 format" minToolsVersion="8.0"/>
-    </dependencies>
-    <scenes>
-        <!--View Controller-->
-        <scene sceneID="tne-QT-ifu">
-            <objects>
-                <viewController storyboardIdentifier="viewController" useStoryboardIdentifierAsRestorationIdentifier="YES" id="BYZ-38-t0r" customClass="ViewController" customModule="TestApp" customModuleProvider="target" sceneMemberID="viewController">
-                    <layoutGuides>
-                        <viewControllerLayoutGuide type="top" id="y3c-jy-aDJ"/>
-                        <viewControllerLayoutGuide type="bottom" id="wfy-db-euE"/>
-                    </layoutGuides>
-                    <view key="view" contentMode="scaleToFill" id="8bC-Xf-vdC">
-                        <rect key="frame" x="0.0" y="0.0" width="375" height="667"/>
-                        <autoresizingMask key="autoresizingMask" widthSizable="YES" heightSizable="YES"/>
-                        <subviews>
-                            <textView clipsSubviews="YES" multipleTouchEnabled="YES" contentMode="scaleToFill" editable="NO" selectable="NO" translatesAutoresizingMaskIntoConstraints="NO" id="7Mj-sL-hrd">
-                                <rect key="frame" x="0.0" y="367" width="375" height="300"/>
-                                <color key="backgroundColor" red="0.0" green="0.47843137250000001" blue="1" alpha="1" colorSpace="custom" customColorSpace="sRGB"/>
-                                <constraints>
-                                    <constraint firstAttribute="height" constant="300" id="YUb-MC-D5w"/>
-                                </constraints>
-                                <color key="textColor" cocoaTouchSystemColor="tableCellGroupedBackgroundColor"/>
-                                <fontDescription key="fontDescription" type="system" pointSize="14"/>
-                                <textInputTraits key="textInputTraits" autocapitalizationType="sentences"/>
-                            </textView>
-                            <toolbar opaque="NO" clearsContextBeforeDrawing="NO" contentMode="scaleToFill" translatesAutoresizingMaskIntoConstraints="NO" id="Qwg-EP-bd6" userLabel="Bottom Toolbar">
-                                <rect key="frame" x="0.0" y="323" width="375" height="44"/>
-                                <constraints>
-                                    <constraint firstAttribute="height" constant="44" id="jhT-Q0-E9N"/>
-                                </constraints>
-                                <items>
-                                    <barButtonItem style="plain" systemItem="flexibleSpace" id="P3q-uA-YUa"/>
-                                    <barButtonItem title="Invoke Interpreter" id="A4J-Mg-nmd" userLabel="Invoke Button">
-                                        <connections>
-                                            <action selector="invokeInterpreter:" destination="BYZ-38-t0r" id="lZU-x7-PsJ"/>
-                                        </connections>
-                                    </barButtonItem>
-                                    <barButtonItem style="plain" systemItem="flexibleSpace" id="Qad-Pa-ySg"/>
-                                </items>
-                            </toolbar>
-                            <toolbar opaque="NO" clearsContextBeforeDrawing="NO" contentMode="scaleToFill" translatesAutoresizingMaskIntoConstraints="NO" id="Gkb-TR-PCB" userLabel="Top Toolbar">
-                                <rect key="frame" x="0.0" y="28" width="375" height="44"/>
-                                <constraints>
-                                    <constraint firstAttribute="height" constant="44" id="hSD-2q-fUE"/>
-                                </constraints>
-                                <items>
-                                    <barButtonItem style="plain" id="LKw-TX-bbH">
-                                        <segmentedControl key="customView" opaque="NO" contentMode="scaleToFill" contentHorizontalAlignment="left" contentVerticalAlignment="top" segmentControlStyle="bar" selectedSegmentIndex="0" id="rhA-nW-xzT">
-                                            <rect key="frame" x="16" y="7" width="343" height="30"/>
-                                            <autoresizingMask key="autoresizingMask" flexibleMaxX="YES" flexibleMaxY="YES"/>
-                                            <segments>
-                                                <segment title="Add"/>
-                                                <segment title="AddQuantized"/>
-                                                <segment title="MultiAdd"/>
-                                            </segments>
-                                            <connections>
-                                                <action selector="modelChanged:" destination="BYZ-38-t0r" eventType="valueChanged" id="YnG-Ov-B5D"/>
-                                            </connections>
-                                        </segmentedControl>
-                                    </barButtonItem>
-                                </items>
-                            </toolbar>
-                        </subviews>
-                        <color key="backgroundColor" red="1" green="1" blue="1" alpha="1" colorSpace="custom" customColorSpace="sRGB"/>
-                        <constraints>
-                            <constraint firstAttribute="trailing" secondItem="Gkb-TR-PCB" secondAttribute="trailing" id="4Cr-Sf-I7n"/>
-                            <constraint firstItem="7Mj-sL-hrd" firstAttribute="bottom" secondItem="wfy-db-euE" secondAttribute="top" id="6ot-zD-sze"/>
-                            <constraint firstItem="7Mj-sL-hrd" firstAttribute="top" secondItem="Qwg-EP-bd6" secondAttribute="bottom" id="ELA-C6-NiG"/>
-                            <constraint firstAttribute="trailing" secondItem="7Mj-sL-hrd" secondAttribute="trailing" id="HDO-xr-mBl"/>
-                            <constraint firstItem="Gkb-TR-PCB" firstAttribute="leading" secondItem="8bC-Xf-vdC" secondAttribute="leading" id="Kmo-6K-gS4"/>
-                            <constraint firstItem="Qwg-EP-bd6" firstAttribute="leading" secondItem="8bC-Xf-vdC" secondAttribute="leading" id="hGu-lm-fMG"/>
-                            <constraint firstAttribute="trailing" secondItem="Qwg-EP-bd6" secondAttribute="trailing" id="iXR-LK-nTO"/>
-                            <constraint firstItem="7Mj-sL-hrd" firstAttribute="leading" secondItem="8bC-Xf-vdC" secondAttribute="leading" id="nr7-jW-ZYf"/>
-                            <constraint firstItem="Gkb-TR-PCB" firstAttribute="top" secondItem="y3c-jy-aDJ" secondAttribute="bottom" constant="8" id="uCF-VW-rR0"/>
-                        </constraints>
-                    </view>
-                    <connections>
-                        <outlet property="invokeButton" destination="A4J-Mg-nmd" id="UxZ-Ft-E45"/>
-                        <outlet property="modelControl" destination="rhA-nW-xzT" id="KKf-TT-BQ2"/>
-                        <outlet property="resultsTextView" destination="7Mj-sL-hrd" id="T4I-z4-tYA"/>
-                    </connections>
-                </viewController>
-                <placeholder placeholderIdentifier="IBFirstResponder" id="dkx-z0-nzr" sceneMemberID="firstResponder"/>
-            </objects>
-            <point key="canvasLocation" x="125.59999999999999" y="133.5832083958021"/>
-        </scene>
-    </scenes>
-</document>
diff --git a/tensorflow/lite/experimental/swift/TestApp/TestApp/Data+TensorFlowLite.swift b/tensorflow/lite/experimental/swift/TestApp/TestApp/Data+TensorFlowLite.swift
deleted file mode 100644
index 0a845f6b354..00000000000
--- a/tensorflow/lite/experimental/swift/TestApp/TestApp/Data+TensorFlowLite.swift
+++ /dev/null
@@ -1,27 +0,0 @@
-// Copyright 2019 Google Inc. All rights reserved.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at:
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-import Foundation
-
-extension Data {
-  /// Creates a new buffer by copying the buffer pointer of the given array.
-  ///
-  /// - Warning: The given array's element type `T` must be trivial in that it can be copied bit
-  ///     for bit with no indirection or reference-counting operations; otherwise, reinterpreting
-  ///     data from the resulting buffer has undefined behavior.
-  /// - Parameter array: An array with elements of type `T`.
-  init<T>(copyingBufferOf array: [T]) {
-    self = array.withUnsafeBufferPointer(Data.init)
-  }
-}
diff --git a/tensorflow/lite/experimental/swift/TestApp/TestApp/Info.plist b/tensorflow/lite/experimental/swift/TestApp/TestApp/Info.plist
deleted file mode 100644
index 3ca3875f04e..00000000000
--- a/tensorflow/lite/experimental/swift/TestApp/TestApp/Info.plist
+++ /dev/null
@@ -1,46 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
-<plist version="1.0">
-<dict>
-	<key>CFBundleDevelopmentRegion</key>
-	<string>en</string>
-	<key>CFBundleExecutable</key>
-	<string>$(EXECUTABLE_NAME)</string>
-	<key>CFBundleIdentifier</key>
-	<string>$(PRODUCT_BUNDLE_IDENTIFIER)</string>
-	<key>CFBundleInfoDictionaryVersion</key>
-	<string>6.0</string>
-	<key>CFBundleName</key>
-	<string>$(PRODUCT_NAME)</string>
-	<key>CFBundlePackageType</key>
-	<string>APPL</string>
-	<key>CFBundleShortVersionString</key>
-	<string>1.0</string>
-	<key>CFBundleVersion</key>
-	<string>0.0.1</string>
-	<key>LSRequiresIPhoneOS</key>
-	<true/>
-	<key>NSCameraUsageDescription</key>
-	<string>NSCameraUsageDescription</string>
-	<key>NSPhotoLibraryUsageDescription</key>
-	<string>Select a photo to detect objects in.</string>
-	<key>UILaunchStoryboardName</key>
-	<string>LaunchScreen</string>
-	<key>UIMainStoryboardFile</key>
-	<string>Main</string>
-	<key>UIRequiredDeviceCapabilities</key>
-	<array>
-		<string>armv7</string>
-	</array>
-	<key>UISupportedInterfaceOrientations</key>
-	<array>
-		<string>UIInterfaceOrientationPortrait</string>
-		<string>UIInterfaceOrientationPortraitUpsideDown</string>
-	</array>
-	<key>UISupportedInterfaceOrientations~ipad</key>
-	<array>
-		<string>UIInterfaceOrientationPortrait</string>
-		<string>UIInterfaceOrientationPortraitUpsideDown</string>
-	</array>
-</dict>
-</plist>
diff --git a/tensorflow/lite/experimental/swift/TestApp/TestApp/ViewController.swift b/tensorflow/lite/experimental/swift/TestApp/TestApp/ViewController.swift
deleted file mode 100644
index add37475156..00000000000
--- a/tensorflow/lite/experimental/swift/TestApp/TestApp/ViewController.swift
+++ /dev/null
@@ -1,314 +0,0 @@
-// Copyright 2019 Google Inc. All rights reserved.
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at:
-//
-// http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-import TensorFlowLite
-import UIKit
-
-class ViewController: UIViewController {
-
-  // MARK: - Properties
-
-  /// TensorFlow Lite interpreter object for performing inference from a given model.
-  private var interpreter: Interpreter?
-
-  /// Serial dispatch queue for managing `Interpreter` calls.
-  private let interpreterQueue = DispatchQueue(
-    label: Constant.dispatchQueueLabel,
-    qos: .userInitiated
-  )
-
-  /// The currently selected model.
-  private var currentModel: Model {
-    guard let currentModel = Model(rawValue: modelControl.selectedSegmentIndex) else {
-      preconditionFailure("Invalid model for selected segment index.")
-    }
-    return currentModel
-  }
-
-  /// A description of the current model.
-  private var modelDescription: String {
-    guard let interpreter = interpreter else { return "" }
-    let inputCount = interpreter.inputTensorCount
-    let outputCount = interpreter.outputTensorCount
-    let inputTensors = (0..<inputCount).map { index in
-      var tensorInfo = "  Input \(index + 1): "
-      do {
-        let tensor = try interpreter.input(at: index)
-        tensorInfo += "\(tensor)"
-      } catch let error {
-        tensorInfo += "\(error.localizedDescription)"
-      }
-      return tensorInfo
-    }.joined(separator: "\n")
-    let outputTensors = (0..<outputCount).map { index in
-      var tensorInfo = "  Output \(index + 1): "
-      do {
-        let tensor = try interpreter.output(at: index)
-        tensorInfo += "\(tensor)"
-      } catch let error {
-        tensorInfo += "\(error.localizedDescription)"
-      }
-      return tensorInfo
-    }.joined(separator: "\n")
-    return "Model Description:\n" +
-             "  Input Tensor Count = \(inputCount)\n\(inputTensors)\n\n" +
-             "  Output Tensor Count = \(outputCount)\n\(outputTensors)"
-  }
-
-  // MARK: - IBOutlets
-
-  /// A segmented control for changing models. See the `Model` enum for available models.
-  @IBOutlet private var modelControl: UISegmentedControl!
-
-  @IBOutlet private var resultsTextView: UITextView!
-  @IBOutlet private var invokeButton: UIBarButtonItem!
-
-  // MARK: - UIViewController
-
-  override func viewDidLoad() {
-    super.viewDidLoad()
-
-    invokeButton.isEnabled = false
-    updateResultsText("Using TensorFlow Lite runtime version \(TensorFlowLite.Runtime.version).")
-    loadModel()
-  }
-
-  // MARK: - IBActions
-
-  @IBAction func modelChanged(_ sender: Any) {
-    invokeButton.isEnabled = false
-    updateResultsText("Switched to the \(currentModel.description) model.")
-    loadModel()
-  }
-
-  @IBAction func invokeInterpreter(_ sender: Any) {
-    switch currentModel {
-    case .add:
-      invokeAdd()
-    case .addQuantized:
-      invokeAddQuantized()
-    case .multiAdd:
-      invokeMultiAdd()
-    }
-  }
-
-  // MARK: - Private
-
-  private func loadModel() {
-    let fileInfo = currentModel.fileInfo
-    guard let modelPath = Bundle.main.path(forResource: fileInfo.name, ofType: fileInfo.extension)
-    else {
-      updateResultsText("Failed to load the \(currentModel.description) model.")
-      return
-    }
-    setUpInterpreter(withModelPath: modelPath)
-  }
-
-  private func setUpInterpreter(withModelPath modelPath: String) {
-    interpreterQueue.async {
-      do {
-        var options = Interpreter.Options()
-        options.threadCount = 2
-        self.interpreter = try Interpreter(modelPath: modelPath, options: options)
-      } catch let error {
-        self.updateResultsText(
-          "Failed to create the interpreter with error: \(error.localizedDescription)"
-        )
-        return
-      }
-      safeDispatchOnMain { self.invokeButton.isEnabled = true }
-    }
-  }
-
-  private func invokeAdd() {
-    interpreterQueue.async {
-      guard let interpreter = self.interpreter else {
-        self.updateResultsText(Constant.nilInterpreterErrorMessage)
-        return
-      }
-      do {
-        try interpreter.resizeInput(at: 0, to: [2])
-        try interpreter.allocateTensors()
-        let input: [Float32] = [1, 3]
-        let resultsText = self.modelDescription + "\n\n" +
-          "Performing 2 add operations on input \(input.description) equals: "
-        self.updateResultsText(resultsText)
-        let data = Data(copyingBufferOf: input)
-        try interpreter.copy(data, toInputAt: 0)
-        try interpreter.invoke()
-        let outputTensor = try interpreter.output(at: 0)
-        let results: () -> String = {
-          guard let results = [Float32](unsafeData: outputTensor.data) else { return "No results." }
-          return resultsText + results.description
-        }
-        self.updateResultsText(results())
-      } catch let error {
-        self.updateResultsText(
-          "Failed to invoke the interpreter with error: \(error.localizedDescription)"
-        )
-        return
-      }
-    }
-  }
-
-  private func invokeAddQuantized() {
-    interpreterQueue.async {
-      guard let interpreter = self.interpreter else {
-        self.updateResultsText(Constant.nilInterpreterErrorMessage)
-        return
-      }
-      do {
-        try interpreter.resizeInput(at: 0, to: [2])
-        try interpreter.allocateTensors()
-        let input: [UInt8] = [1, 3]
-        let resultsText = self.modelDescription + "\n\n" +
-          "Performing 2 add operations on quantized input \(input.description) equals: "
-        self.updateResultsText(resultsText)
-        let data = Data(input)
-        try interpreter.copy(data, toInputAt: 0)
-        try interpreter.invoke()
-        let outputTensor = try interpreter.output(at: 0)
-        let results: () -> String = {
-          guard let quantizationParameters = outputTensor.quantizationParameters else {
-            return "No results."
-          }
-          let quantizedResults = [UInt8](outputTensor.data)
-          let dequantizedResults = quantizedResults.map {
-            quantizationParameters.scale * Float(Int($0) - quantizationParameters.zeroPoint)
-          }
-          return resultsText + quantizedResults.description +
-                   ", dequantized results: " + dequantizedResults.description
-        }
-        self.updateResultsText(results())
-      } catch let error {
-        self.updateResultsText(
-          "Failed to invoke the interpreter with error: \(error.localizedDescription)"
-        )
-        return
-      }
-    }
-  }
-
-  private func invokeMultiAdd() {
-    interpreterQueue.async {
-      guard let interpreter = self.interpreter else {
-        self.updateResultsText(Constant.nilInterpreterErrorMessage)
-        return
-      }
-      do {
-        let shape = Tensor.Shape(2)
-        try (0..<interpreter.inputTensorCount).forEach { index in
-          try interpreter.resizeInput(at: index, to: shape)
-        }
-        try interpreter.allocateTensors()
-        let inputs = try (0..<interpreter.inputTensorCount).map { index -> [Float32] in
-          let input = [Float32(index + 1), Float32(index + 2)]
-          let data = Data(copyingBufferOf: input)
-          try interpreter.copy(data, toInputAt: index)
-          return input
-        }
-        let resultsText = self.modelDescription + "\n\n" +
-          "Performing 3 add operations on inputs \(inputs.description) equals: "
-        self.updateResultsText(resultsText)
-        try interpreter.invoke()
-        let results = try (0..<interpreter.outputTensorCount).map { index -> [Float32] in
-          let tensor = try interpreter.output(at: index)
-          return [Float32](unsafeData: tensor.data) ?? []
-        }
-        self.updateResultsText(resultsText + results.description)
-      } catch let error {
-        self.updateResultsText(
-          "Failed to invoke the interpreter with error: \(error.localizedDescription)"
-        )
-        return
-      }
-    }
-  }
-
-  private func updateResultsText(_ text: String? = nil) {
-    safeDispatchOnMain { self.resultsTextView.text = text }
-  }
-}
-
-// MARK: - Constants
-
-private enum Constant {
-  static let dispatchQueueLabel = "TensorFlowLiteInterpreterQueue"
-  static let nilInterpreterErrorMessage =
-    "Failed to invoke the interpreter because the interpreter was nil."
-}
-
-/// Models that can be loaded by the TensorFlow Lite `Interpreter`.
-private enum Model: Int, CustomStringConvertible {
-  /// A float model that performs two add operations on one input tensor and returns the result in
-  /// one output tensor.
-  case add = 0
-  /// A quantized model that performs two add operations on one input tensor and returns the result
-  /// in one output tensor.
-  case addQuantized = 1
-  /// A float model that performs three add operations on four input tensors and returns the results
-  /// in 2 output tensors.
-  case multiAdd = 2
-
-  var fileInfo: (name: String, extension: String) {
-    switch self {
-    case .add:
-      return Add.fileInfo
-    case .addQuantized:
-      return AddQuantized.fileInfo
-    case .multiAdd:
-      return MultiAdd.fileInfo
-    }
-  }
-
-  // MARK: - CustomStringConvertible
-
-  var description: String {
-    switch self {
-    case .add:
-      return Add.name
-    case .addQuantized:
-      return AddQuantized.name
-    case .multiAdd:
-      return MultiAdd.name
-    }
-  }
-}
-
-/// Values for the `Add` model.
-private enum Add {
-  static let name = "Add"
-  static let fileInfo = (name: "add", extension: "bin")
-}
-
-/// Values for the `AddQuantized` model.
-private enum AddQuantized {
-  static let name = "AddQuantized"
-  static let fileInfo = (name: "add_quantized", extension: "bin")
-}
-
-/// Values for the `MultiAdd` model.
-private enum MultiAdd {
-  static let name = "MultiAdd"
-  static let fileInfo = (name: "multi_add", extension: "bin")
-}
-
-// MARK: - Fileprivate
-
-/// Safely dispatches the given block on the main queue. If the current thread is `main`, the block
-/// is executed synchronously; otherwise, the block is executed asynchronously on the main thread.
-fileprivate func safeDispatchOnMain(_ block: @escaping () -> Void) {
-  if Thread.isMainThread { block(); return }
-  DispatchQueue.main.async { block() }
-}
diff --git a/tensorflow/lite/external_cpu_backend_context.h b/tensorflow/lite/external_cpu_backend_context.h
index c667057a48c..662734c9cd5 100644
--- a/tensorflow/lite/external_cpu_backend_context.h
+++ b/tensorflow/lite/external_cpu_backend_context.h
@@ -35,6 +35,8 @@ class TfLiteInternalBackendContext {
   // TfLite computation.
   virtual void SetMaxNumThreads(int max_num_threads) = 0;
 
+  // A context may internally cache prepacked versions of constant tensors for
+  // faster computation. This function will clear any caches on the context.
   virtual void ClearCaches() = 0;
 };
 
diff --git a/tensorflow/lite/g3doc/convert/cmdline.md b/tensorflow/lite/g3doc/convert/cmdline.md
index a6594d4a429..64d3e315b97 100644
--- a/tensorflow/lite/g3doc/convert/cmdline.md
+++ b/tensorflow/lite/g3doc/convert/cmdline.md
@@ -21,9 +21,9 @@ custom objects in
 
 ## Usage
 
-The following example shows a SavedModel being converted:
+The following example shows a `SavedModel` being converted:
 
-```bash
+```sh
 tflite_convert \
   --saved_model_dir=/tmp/mobilenet_saved_model \
   --output_file=/tmp/mobilenet.tflite
@@ -39,7 +39,7 @@ The inputs and outputs are specified using the following commonly used flags:
 
 To use all of the available flags, use the following command:
 
-```bash
+```sh
 tflite_convert --help
 ```
 
@@ -57,7 +57,7 @@ To obtain the latest version of the TensorFlow Lite converter CLI, we recommend
 installing the nightly build using
 [pip](https://www.tensorflow.org/install/pip):
 
-```bash
+```sh
 pip install tf-nightly
 ```
 
@@ -65,7 +65,7 @@ Alternatively, you can
 [clone the TensorFlow repository](https://www.tensorflow.org/install/source) and
 use `bazel` to run the command:
 
-```
+```sh
 bazel run //tensorflow/lite/python:tflite_convert -- \
   --saved_model_dir=/tmp/mobilenet_saved_model \
   --output_file=/tmp/mobilenet.tflite
@@ -75,13 +75,13 @@ bazel run //tensorflow/lite/python:tflite_convert -- \
 
 There is a behavior change in how models containing
 [custom ops](https://www.tensorflow.org/lite/guide/ops_custom) (those for which
-users use to set --allow\_custom\_ops before) are handled in the
+users previously set `--allow_custom_ops` before) are handled in the
 [new converter](https://github.com/tensorflow/tensorflow/blob/917ebfe5fc1dfacf8eedcc746b7989bafc9588ef/tensorflow/lite/python/lite.py#L81).
 
 **Built-in TensorFlow op**
 
 If you are converting a model with a built-in TensorFlow op that does not exist
-in TensorFlow Lite, you should set --allow\_custom\_ops argument (same as
+in TensorFlow Lite, you should set `--allow_custom_ops` argument (same as
 before), explained [here](https://www.tensorflow.org/lite/guide/ops_custom).
 
 **Custom op in TensorFlow**
@@ -90,27 +90,27 @@ If you are converting a model with a custom TensorFlow op, it is recommended
 that you write a [TensorFlow kernel](https://www.tensorflow.org/guide/create_op)
 and [TensorFlow Lite kernel](https://www.tensorflow.org/lite/guide/ops_custom).
 This ensures that the model is working end-to-end, from TensorFlow and
-TensorFlow Lite. This also requires setting the --allow\_custom\_ops argument.
+TensorFlow Lite. This also requires setting the `--allow_custom_ops` argument.
 
 **Advanced custom op usage (not recommended)**
 
 If the above is not possible, you can still convert a TensorFlow model
 containing a custom op without a corresponding kernel. You will need to pass the
 [OpDef](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/framework/op_def.proto)
-of the custom op in TensorFlow using --custom\_opdefs flag, as long as you have
+of the custom op in TensorFlow using `--custom_opdefs` flag, as long as you have
 the corresponding OpDef registered in the TensorFlow global op registry. This
 ensures that the TensorFlow model is valid (i.e. loadable by the TensorFlow
 runtime).
 
 If the custom op is not part of the global TensorFlow op registry, then the
-corresponding OpDef needs to be specified via the --custom\_opdefs flag. This is
-a list of an OpDef proto in string that needs to be additionally registered.
-Below is an example of an TFLiteAwesomeCustomOp with 2 inputs, 1 output, and 2
+corresponding OpDef needs to be specified via the `--custom_opdefs` flag. This
+is a list of an OpDef proto in string that needs to be additionally registered.
+Below is an example of a TFLiteAwesomeCustomOp with 2 inputs, 1 output, and 2
 attributes:
 
-```
---custom\_opdefs="name: 'TFLiteAwesomeCustomOp' input\_arg: { name: 'InputA'
-type: DT\_FLOAT } input\_arg: { name: ‘InputB' type: DT\_FLOAT }
-output\_arg: { name: 'Output' type: DT\_FLOAT } attr : { name: 'Attr1' type:
+```sh
+--custom_opdefs="name: 'TFLiteAwesomeCustomOp' input_arg: { name: 'InputA'
+type: DT_FLOAT } input_arg: { name: ‘InputB' type: DT_FLOAT }
+output_arg: { name: 'Output' type: DT_FLOAT } attr : { name: 'Attr1' type:
 'float'} attr : { name: 'Attr2' type: 'list(float)'}"
 ```
diff --git a/tensorflow/lite/g3doc/convert/index.md b/tensorflow/lite/g3doc/convert/index.md
index be4088e84c6..71b5fd71737 100644
--- a/tensorflow/lite/g3doc/convert/index.md
+++ b/tensorflow/lite/g3doc/convert/index.md
@@ -13,8 +13,8 @@ The API for TensorFlow 1.X is available
 
 ## New in TF 2.2
 
-Switching to use a new converter backend by default - in the nightly builds and
-TF 2.2 stable. Why we are switching?
+TensorFlow Lite has switched to use a new converter backend by default - in the
+nightly builds and TF 2.2 stable. Why we did we switch?
 
 *   Enables conversion of new classes of models, including Mask R-CNN, Mobile
     BERT, and many more
@@ -46,9 +46,9 @@ In case you encounter any issues:
     and
     [Command Line Tool](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/g3doc/convert/cmdline.md)
     documentation
-*   Switch to the old converter by setting --experimental_new_converter=false
+*   Switch to the old converter by setting `--experimental_new_converter=false`
     (from the [tflite_convert](https://www.tensorflow.org/lite/convert/cmdline)
-    command line tool) or converter.experimental_new_converter=False (from
+    command line tool) or `converter.experimental_new_converter=False` (from the
     [Python API](https://www.tensorflow.org/api_docs/python/tf/lite/TFLiteConverter))
 
 ## Device deployment
diff --git a/tensorflow/lite/g3doc/convert/metadata.md b/tensorflow/lite/g3doc/convert/metadata.md
index e6a4312d662..29b2c5ce2b3 100644
--- a/tensorflow/lite/g3doc/convert/metadata.md
+++ b/tensorflow/lite/g3doc/convert/metadata.md
@@ -20,7 +20,7 @@ set this up [here](https://www.tensorflow.org/install).
 After setup the Python programming environment, you will need to install
 additional tooling:
 
-```
+```sh
 pip install tflite-support
 ```
 
@@ -53,31 +53,31 @@ Lite metadata:
 
 ### Examples
 
-Note: The export directory specified has to exist before you run the script, it
+Note: The export directory specified has to exist before you run the script; it
 does not get created as part of the process.
 
 You can find examples on how the metadata should be populated for different
 types of models here:
 
-#### Image Classification
+#### Image classification
 
 Download the script
 [here](https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification/metadata/metadata_writer_for_image_classifier.py)
 and run the script like this:
 
-```
+```sh
 python ./metadata_writer_for_image_classifier.py \
     --model_file=./model_without_metadata/mobilenet_v1_0.75_160_quantized.tflite \
     --label_file=./model_without_metadata/labels.txt \
     --export_directory=model_with_metadata
 ```
 
-The rest of this guide will highlight some of the key sections in the Image
-Classification example to illustrate the key elements.
+The rest of this guide will highlight some of the key sections in the image
+classification example to illustrate the key elements.
 
-### Deep dive into the Image Classification example
+### Deep dive into the image classification example
 
-#### Model Information
+#### Model information
 
 Metadata starts by creating a new model info:
 
@@ -103,9 +103,9 @@ model_meta.license = ("Apache License. Version 2.0 "
 
 #### Input / output information
 
-This describe your model's input and output signature and it maybe used by
-automatic code generators to create pre- and post- processing code. To create an
-input or output information about a tensor:
+This section shows you how to describe your model's input and output signature.
+This metadata may be used by automatic code generators to create pre- and post-
+processing code. To create input or output information about a tensor:
 
 ```python
 # Creates input info.
@@ -115,13 +115,13 @@ input_meta = _metadata_fb.TensorMetadataT()
 output_meta = _metadata_fb.TensorMetadataT()
 ```
 
-#### Image Input
+#### Image input
 
 Image is a common input type for machine learning. TensorFlow Lite metadata
 supports information such as colorspace and pre-processing information such as
-normalization. One thing that does not required manual input is the dimension of
-the image as this is already provided by the shape of the input tensor and can
-be automatically inferred.
+normalization. The dimension of the image does not require manual specification
+since it is already provided by the shape of the input tensor and can be
+automatically inferred.
 
 ```python
 input_meta.name = "image"
@@ -153,7 +153,7 @@ input_meta.stats = input_stats
 Label can be mapped to an output tensor via an associated file using
 `TENSOR_AXIS_LABELS`.
 
-```Python
+```python
 # Creates output info.
 output_meta = _metadata_fb.TensorMetadataT()
 output_meta.name = "probability"
@@ -175,7 +175,7 @@ output_meta.associatedFiles = [label_file]
 
 #### Put it all together
 
-The following code pull the model information together with the input and output
+The following code combines the model information with the input and output
 information:
 
 ```python
@@ -192,8 +192,8 @@ b.Finish(
 metadata_buf = b.Output()
 ```
 
-Once the data structure is ready, the writing of the metadata into the tflite
-file is done via the `populate` method:
+Once the data structure is ready, the metadata is written into the TFLite file
+via the `populate` method:
 
 ```python
 populator = _metadata.MetadataPopulator.with_model_file(model_file)
@@ -204,9 +204,9 @@ populator.populate()
 
 #### Verify the metadata
 
-You can read back the metadata in a tflite file using the `MetadataDisplayer`:
+You can read the metadata in a TFLite file using the `MetadataDisplayer`:
 
-```Python
+```python
 displayer = _metadata.MetadataDisplayer.with_model_file(export_model_path)
 export_json_file = os.path.join(FLAGS.export_directory,
                     os.path.splitext(model_basename)[0] + ".json")
diff --git a/tensorflow/lite/g3doc/convert/python_api.md b/tensorflow/lite/g3doc/convert/python_api.md
index 36dda16a77c..ff14c4b92e7 100644
--- a/tensorflow/lite/g3doc/convert/python_api.md
+++ b/tensorflow/lite/g3doc/convert/python_api.md
@@ -192,7 +192,7 @@ specific wrapper code. For more information, please refer to the
 
 The TensorFlow nightly can be installed using the following command:
 
-```
+```sh
 pip install tf-nightly
 ```
 
@@ -208,13 +208,13 @@ either install the nightly build with
 
 There is a behavior change in how models containing
 [custom ops](https://www.tensorflow.org/lite/guide/ops_custom) (those for which
-users use to set allow\_custom\_ops before) are handled in the
+users previously set `allow_custom_ops` before) are handled in the
 [new converter](https://github.com/tensorflow/tensorflow/blob/917ebfe5fc1dfacf8eedcc746b7989bafc9588ef/tensorflow/lite/python/lite.py#L81).
 
 **Built-in TensorFlow op**
 
 If you are converting a model with a built-in TensorFlow op that does not exist
-in TensorFlow Lite, you should set allow\_custom\_ops attribute (same as
+in TensorFlow Lite, you should set the `allow_custom_ops` attribute (same as
 before), explained [here](https://www.tensorflow.org/lite/guide/ops_custom).
 
 **Custom op in TensorFlow**
@@ -223,27 +223,27 @@ If you are converting a model with a custom TensorFlow op, it is recommended
 that you write a [TensorFlow kernel](https://www.tensorflow.org/guide/create_op)
 and [TensorFlow Lite kernel](https://www.tensorflow.org/lite/guide/ops_custom).
 This ensures that the model is working end-to-end, from TensorFlow and
-TensorFlow Lite. This also requires setting the allow\_custom\_ops attribute.
+TensorFlow Lite. This also requires setting the `allow_custom_ops` attribute.
 
 **Advanced custom op usage (not recommended)**
 
 If the above is not possible, you can still convert a TensorFlow model
 containing a custom op without a corresponding kernel. You will need to pass the
 [OpDef](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/framework/op_def.proto)
-of the custom op in TensorFlow using --custom\_opdefs flag, as long as you have
+of the custom op in TensorFlow using `--custom_opdefs` flag, as long as you have
 the corresponding OpDef registered in the TensorFlow global op registry. This
 ensures that the TensorFlow model is valid (i.e. loadable by the TensorFlow
 runtime).
 
 If the custom op is not part of the global TensorFlow op registry, then the
-corresponding OpDef needs to be specified via the --custom\_opdefs flag. This is
-a list of an OpDef proto in string that needs to be additionally registered.
-Below is an example of an TFLiteAwesomeCustomOp with 2 inputs, 1 output, and 2
+corresponding OpDef needs to be specified via the `--custom_opdefs` flag. This
+is a list of an OpDef proto in string that needs to be additionally registered.
+Below is an example of a TFLiteAwesomeCustomOp with 2 inputs, 1 output, and 2
 attributes:
 
-```
-converter.custom\_opdefs="name: 'TFLiteAwesomeCustomOp' input\_arg: { name: 'InputA'
-type: DT\_FLOAT } input\_arg: { name: ‘InputB' type: DT\_FLOAT }
-output\_arg: { name: 'Output' type: DT\_FLOAT } attr : { name: 'Attr1' type:
-'float'} attr : { name: 'Attr2' type: 'list(float)'}"
+```python
+converter.custom_opdefs="""name: 'TFLiteAwesomeCustomOp' input_arg: { name: 'InputA'
+type: DT_FLOAT } input_arg: { name: ‘InputB' type: DT_FLOAT }
+output_arg: { name: 'Output' type: DT_FLOAT } attr : { name: 'Attr1' type:
+'float'} attr : { name: 'Attr2' type: 'list(float)'}"""
 ```
diff --git a/tensorflow/lite/g3doc/convert/quantization.md b/tensorflow/lite/g3doc/convert/quantization.md
index 099921cf6b3..41593fb29f9 100644
--- a/tensorflow/lite/g3doc/convert/quantization.md
+++ b/tensorflow/lite/g3doc/convert/quantization.md
@@ -12,7 +12,7 @@ has latency benefits, but prioritizes size reduction.
 
 During conversion, set the `optimizations` flag to optimize for size:
 
-```
+```python
 converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
 converter.optimizations = [tf.lite.Optimize.DEFAULT]
 tflite_quant_model = converter.convert()
@@ -26,7 +26,7 @@ quantized. To do this, we need to measure the dynamic range of activations and
 inputs with a representative data set. You can simply create an input data
 generator and provide it to our converter.
 
-```
+```python
 import tensorflow as tf
 
 def representative_dataset_gen():
@@ -40,7 +40,7 @@ converter.representative_dataset = representative_dataset_gen
 tflite_quant_model = converter.convert()
 ```
 
-# During training: Quantizing models for integer-only execution.
+# During training: Quantizing models for integer-only execution
 
 Quantizing models for integer-only execution gets a model with even faster
 latency, smaller size, and integer-only accelerators compatible model.
@@ -52,7 +52,7 @@ compatible with 2.0 semantics is in progress.
 
 Convert the graph:
 
-```
+```python
 converter = tf.compat.v1.lite.TFLiteConverter.from_saved_model(saved_model_dir)
 converter.inference_type = tf.lite.constants.QUANTIZED_UINT8
 input_arrays = converter.get_input_arrays()
@@ -75,5 +75,5 @@ the
 `std_dev` is 255 / (float_max - float_min).
 
 For most users, we recommend using post-training quantization. We are working on
-new tools for post-training and during training quantization that we hope will
+new tools for post-training and training-time quantization that we hope will
 simplify generating quantized models.
diff --git a/tensorflow/lite/g3doc/guide/android.md b/tensorflow/lite/g3doc/guide/android.md
index 89a747094ff..b22ea13d722 100644
--- a/tensorflow/lite/g3doc/guide/android.md
+++ b/tensorflow/lite/g3doc/guide/android.md
@@ -191,7 +191,7 @@ build --action_env ANDROID_SDK_API_LEVEL="23"
 build --action_env ANDROID_SDK_HOME="/usr/local/android/android-sdk-linux"
 ```
 
-#### Build and Install
+#### Build and install
 
 Once Bazel is properly configured, you can build the TensorFlow Lite AAR from
 the root checkout directory as follows:
@@ -268,11 +268,13 @@ If you want to use TFLite through C++ libraries, you can build the shared
 libraries:
 
 32bit armeabi-v7a:
-```
+
+```sh
 bazel build -c opt --config=android_arm //tensorflow/lite:libtensorflowlite.so
 ```
 
 64bit arm64-v8a:
-```
+
+```sh
 bazel build -c opt --config=android_arm64 //tensorflow/lite:libtensorflowlite.so
 ```
diff --git a/tensorflow/lite/g3doc/guide/build_rpi.md b/tensorflow/lite/g3doc/guide/build_rpi.md
index dfe3709b024..0f49ed91315 100644
--- a/tensorflow/lite/g3doc/guide/build_rpi.md
+++ b/tensorflow/lite/g3doc/guide/build_rpi.md
@@ -14,23 +14,22 @@ or
 
 ## Cross-compile for Raspberry Pi
 
-Instruction has been tested on Ubuntu 16.04.3 64-bit PC (AMD64) and TensorFlow
-devel docker image
+The following instructions have been tested on Ubuntu 16.04.3 64-bit PC (AMD64)
+and TensorFlow devel docker image
 [tensorflow/tensorflow:nightly-devel](https://hub.docker.com/r/tensorflow/tensorflow/tags/).
 
 To cross compile TensorFlow Lite follow the steps:
 
 1.  Clone official Raspberry Pi cross-compilation toolchain:
 
-    ```bash
+    ```sh
     git clone https://github.com/raspberrypi/tools.git rpi_tools
     ```
 
 2.  Clone TensorFlow repository:
 
-    ```bash
+    ```sh
     git clone https://github.com/tensorflow/tensorflow.git tensorflow_src
-
     ```
 
     **Note:** If you're using the TensorFlow Docker image, the repo is already
@@ -39,7 +38,7 @@ To cross compile TensorFlow Lite follow the steps:
 3.  Run following script at the root of the TensorFlow repository to download
     all the build dependencies:
 
-    ```bash
+    ```sh
     cd tensorflow_src && ./tensorflow/lite/tools/make/download_dependencies.sh
     ```
 
@@ -47,7 +46,7 @@ To cross compile TensorFlow Lite follow the steps:
 
 4.  To build ARMv7 binary for Raspberry Pi 2, 3 and 4 execute:
 
-    ```bash
+    ```sh
     PATH=../rpi_tools/arm-bcm2708/arm-rpi-4.9.3-linux-gnueabihf/bin:$PATH ./tensorflow/lite/tools/make/build_rpi_lib.sh
     ```
 
@@ -56,7 +55,7 @@ To cross compile TensorFlow Lite follow the steps:
 
 5.  To build ARMv6 binary for Raspberry Pi Zero execute:
 
-    ```bash
+    ```sh
     PATH=../rpi_tools/arm-bcm2708/arm-rpi-4.9.3-linux-gnueabihf/bin:$PATH ./tensorflow/lite/tools/make/build_rpi_lib.sh TARGET_ARCH=armv6
     ```
 
@@ -65,28 +64,27 @@ To cross compile TensorFlow Lite follow the steps:
 
 ## Compile natively on Raspberry Pi
 
-Instruction has been tested on Raspberry Pi Zero, Raspbian GNU/Linux 10
-(buster), gcc version 8.3.0 (Raspbian 8.3.0-6+rpi1):
+The following instructions have been tested on Raspberry Pi Zero, Raspbian
+GNU/Linux 10 (buster), gcc version 8.3.0 (Raspbian 8.3.0-6+rpi1):
 
 To natively compile TensorFlow Lite follow the steps:
 
 1.  Log in to your Raspberry Pi and install the toolchain:
 
-    ```bash
+    ```sh
     sudo apt-get install build-essential
     ```
 
 2.  Clone TensorFlow repository:
 
-    ```bash
+    ```sh
     git clone https://github.com/tensorflow/tensorflow.git tensorflow_src
-
     ```
 
 3.  Run following script at the root of the TensorFlow repository to download
     all the build dependencies:
 
-    ```bash
+    ```sh
     cd tensorflow_src && ./tensorflow/lite/tools/make/download_dependencies.sh
     ```
 
@@ -94,7 +92,7 @@ To natively compile TensorFlow Lite follow the steps:
 
 4.  You should then be able to compile TensorFlow Lite with:
 
-    ```bash
+    ```sh
     ./tensorflow/lite/tools/make/build_rpi_lib.sh
     ```
 
diff --git a/tensorflow/lite/g3doc/guide/codegen.md b/tensorflow/lite/g3doc/guide/codegen.md
index 4cf8a677b98..74c404e61fa 100644
--- a/tensorflow/lite/g3doc/guide/codegen.md
+++ b/tensorflow/lite/g3doc/guide/codegen.md
@@ -1,7 +1,7 @@
 # Generate code from TensorFlow Lite metadata
 
 Note: TensorFlow Lite wrapper code generator is in experimental (beta) phase and
-it currently only supports Android.
+currently only supports Android.
 
 For TensorFlow Lite model enhanced with [metadata](../convert/metadata.md),
 developers can use the TensorFlow Lite Android wrapper code generator to create
@@ -19,13 +19,13 @@ to see how the codegen tool parses each field.
 
 You will need to install the following tooling in your terminal:
 
-```
+```sh
 pip install tflite-support
 ```
 
 Once completed, the code generator can be used using the following syntax:
 
-```
+```sh
 tflite_codegen --model=./model_with_metadata/mobilenet_v1_0.75_160_quantized.tflite \
     --package_name=org.tensorflow.lite.classify \
     --model_class_name=MyClassifierModel \
@@ -66,7 +66,7 @@ In the app module that will be consuming the generated library module:
 
 Under the android section, add the following:
 
-```java
+```build
 aaptOptions {
    noCompress "tflite"
 }
@@ -74,14 +74,14 @@ aaptOptions {
 
 Under the dependencies section, add the following:
 
-```java
+```build
 implementation project(":classify_wrapper")
 ```
 
 ### Step 3: Using the model
 
 ```java
-// 1. Initialize the Model
+// 1. Initialize the model
 MyClassifierModel myImageClassifier = null;
 
 try {
@@ -92,14 +92,14 @@ try {
 
 if(null != myImageClassifier) {
 
-    // 2. Setting the input with a Bitmap called inputBitmap
+    // 2. Set the input with a Bitmap called inputBitmap
     MyClassifierModel.Inputs inputs = myImageClassifier.createInputs();
     inputs.loadImage(inputBitmap));
 
-    // 3. Running the model
+    // 3. Run the model
     MyClassifierModel.Outputs outputs = myImageClassifier.run(inputs);
 
-    // 4. Retrieving the result
+    // 4. Retrieve the result
     Map<String, Float> labeledProbability = outputs.getProbability();
 }
 ```
@@ -117,7 +117,7 @@ parameters:
 *   (Optional) **`numThreads`**: Number of threads used to run the model -
     default is one.
 
-For example, to use a NNAPI delegate and up to three threads, you can initiate
+For example, to use a NNAPI delegate and up to three threads, you can initialize
 the model like this:
 
 ```java
@@ -135,7 +135,7 @@ try {
 Under the app module that will uses the library module, insert the following
 lines under the android section:
 
-```java
+```build
 aaptOptions {
    noCompress "tflite"
 }
diff --git a/tensorflow/lite/g3doc/guide/faq.md b/tensorflow/lite/g3doc/guide/faq.md
index c1f9985ada7..197fc2d4a8f 100644
--- a/tensorflow/lite/g3doc/guide/faq.md
+++ b/tensorflow/lite/g3doc/guide/faq.md
@@ -55,7 +55,7 @@ look for the inputs and outputs in the graph. To visualize a `.pb` file, use the
 [`import_pb_to_tensorboard.py`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/python/tools/import_pb_to_tensorboard.py)
 script like below:
 
-```
+```sh
 python import_pb_to_tensorboard.py --model_dir <model path> --log_dir <log dir path>
 ```
 
@@ -68,7 +68,7 @@ script in our repository.
 *   [Clone the TensorFlow repository](https://www.tensorflow.org/install/source)
 *   Run the `visualize.py` script with bazel:
 
-```
+```sh
 bazel run //tensorflow/lite/tools:visualize model.tflite visualized_model.html
 ```
 
@@ -101,8 +101,8 @@ random data to feed to the interpreter.
 
 #### How do I reduce the size of my converted TensorFlow Lite model?
 
-[Post-training quantization](../performance/post_training_quantization.md) can be
-used during conversion to TensorFlow Lite to reduce the size of the model.
+[Post-training quantization](../performance/post_training_quantization.md) can
+be used during conversion to TensorFlow Lite to reduce the size of the model.
 Post-training quantization quantizes weights to 8-bits of precision from
 floating-point and dequantizes them during runtime to perform floating point
 computations. However, note that this could have some accuracy implications.
diff --git a/tensorflow/lite/g3doc/guide/get_started.md b/tensorflow/lite/g3doc/guide/get_started.md
index 61a4be7ae3e..c9543c7f553 100644
--- a/tensorflow/lite/g3doc/guide/get_started.md
+++ b/tensorflow/lite/g3doc/guide/get_started.md
@@ -11,14 +11,13 @@ each step of the developer workflow and provides links to further instructions.
 <a id="1_choose_a_model"></a>
 
 A TensorFlow model is a data structure that contains the logic and knowledge of
-a machine learning network trained to solve a particular problem.
-There are many ways to obtain a TensorFlow model, from using pre-trained models
-to training your own.
+a machine learning network trained to solve a particular problem. There are many
+ways to obtain a TensorFlow model, from using pre-trained models to training
+your own.
 
-To use a model with TensorFlow Lite, you must convert a
-full TensorFlow model into the TensorFlow Lite format—you
-cannot create or train a model using TensorFlow Lite. So you must start with a
-regular TensorFlow model, and then
+To use a model with TensorFlow Lite, you must convert a full TensorFlow model
+into the TensorFlow Lite format—you cannot create or train a model using
+TensorFlow Lite. So you must start with a regular TensorFlow model, and then
 [convert the model](#2_convert_the_model_format).
 
 Note: TensorFlow Lite supports a limited subset of TensorFlow operations, so not
@@ -135,9 +134,9 @@ performance or reduce file size. This is covered in section 4,
 
 ### Ops compatibility
 
-TensorFlow Lite currently supports a [limited subset of TensorFlow
-operations](ops_compatibility.md). The long term goal is for all TensorFlow
-operations to be supported.
+TensorFlow Lite currently supports a
+[limited subset of TensorFlow operations](ops_compatibility.md). The long term
+goal is for all TensorFlow operations to be supported.
 
 If the model you wish to convert contains unsupported operations, you can use
 [TensorFlow Select](ops_select.md) to include operations from TensorFlow. This
@@ -215,11 +214,9 @@ Embedded Linux is an important platform for deploying machine learning. To get
 started using Python to perform inference with your TensorFlow Lite models,
 follow the [Python quickstart](python.md).
 
-To instead install the C++ library, see the
-build instructions for [Raspberry Pi](build_rpi.md) or
-[Arm64-based boards](build_arm64.md) (for boards such as Odroid C2, Pine64, and
-NanoPi).
-
+To instead install the C++ library, see the build instructions for
+[Raspberry Pi](build_rpi.md) or [Arm64-based boards](build_arm64.md) (for boards
+such as Odroid C2, Pine64, and NanoPi).
 
 ### Microcontrollers
 
diff --git a/tensorflow/lite/g3doc/guide/inference.md b/tensorflow/lite/g3doc/guide/inference.md
index ab3265cc3c2..5f3fba98cff 100644
--- a/tensorflow/lite/g3doc/guide/inference.md
+++ b/tensorflow/lite/g3doc/guide/inference.md
@@ -310,7 +310,7 @@ The following example shows how to use the Python interpreter to load a
 import numpy as np
 import tensorflow as tf
 
-# Load TFLite model and allocate tensors.
+# Load the TFLite model and allocate tensors.
 interpreter = tf.lite.Interpreter(model_path="converted_model.tflite")
 interpreter.allocate_tensors()
 
@@ -318,7 +318,7 @@ interpreter.allocate_tensors()
 input_details = interpreter.get_input_details()
 output_details = interpreter.get_output_details()
 
-# Test model on random input data.
+# Test the model on random input data.
 input_shape = input_details[0]['shape']
 input_data = np.array(np.random.random_sample(input_shape), dtype=np.float32)
 interpreter.set_tensor(input_details[0]['index'], input_data)
@@ -331,10 +331,11 @@ output_data = interpreter.get_tensor(output_details[0]['index'])
 print(output_data)
 ```
 
-Alternative to loading the model as a pre-converted `.tflite` file, you can
-combine your code with the [TensorFlow Lite Converter Python API](
-../convert/python_api.md) (`tf.lite.TFLiteConverter`), allowing you to convert
-your TensorFlow model into the TensorFlow Lite format and then run an inference:
+Alternatively to loading the model as a pre-converted `.tflite` file, you can
+combine your code with the
+[TensorFlow Lite Converter Python API](../convert/python_api.md)
+(`tf.lite.TFLiteConverter`), allowing you to convert your TensorFlow model into
+the TensorFlow Lite format and then run an inference:
 
 ```python
 import numpy as np
@@ -350,7 +351,7 @@ with tf.Session() as sess:
   converter = tf.lite.TFLiteConverter.from_session(sess, [img], [out])
   tflite_model = converter.convert()
 
-# Load TFLite model and allocate tensors.
+# Load the TFLite model and allocate tensors.
 interpreter = tf.lite.Interpreter(model_content=tflite_model)
 interpreter.allocate_tensors()
 
@@ -384,22 +385,22 @@ including all the tensors. The latter allows implementations to access their
 inputs and outputs.
 
 When the interpreter loads a model, it calls `init()` once for each node in the
-graph. A given `init()` will be called more than once if the op is used
-multiple times in the graph. For custom ops a configuration buffer will be
-provided, containing a flexbuffer that maps parameter names to their values.
-The buffer is empty for builtin ops because the interpreter has already parsed
-the op parameters. Kernel implementation that require state should initialize
-it here and transfer ownership to the caller.  For each `init()` call, there
-will be a corresponding call to `free()`, allowing implementations to dispose
-of the buffer they might have allocated in `init()`.
+graph. A given `init()` will be called more than once if the op is used multiple
+times in the graph. For custom ops a configuration buffer will be provided,
+containing a flexbuffer that maps parameter names to their values. The buffer is
+empty for builtin ops because the interpreter has already parsed the op
+parameters. Kernel implementations that require state should initialize it here
+and transfer ownership to the caller. For each `init()` call, there will be a
+corresponding call to `free()`, allowing implementations to dispose of the
+buffer they might have allocated in `init()`.
 
-Whenever the input tensors are resized the interpreter will go through the
+Whenever the input tensors are resized, the interpreter will go through the
 graph notifying implementations of the change. This gives them the chance to
 resize their internal buffer, check validity of input shapes and types, and
-recalculate output shapes. This is all done through `prepare()` and
-implementation can access their state using `node->user_data`.
+recalculate output shapes. This is all done through `prepare()`, and
+implementations can access their state using `node->user_data`.
 
-Finally, each time inference runs the interpreter traverses the graph calling
+Finally, each time inference runs, the interpreter traverses the graph calling
 `invoke()`, and here too the state is available as `node->user_data`.
 
 Custom ops can be implemented in exactly the same way as builtin ops, by
diff --git a/tensorflow/lite/g3doc/guide/ios.md b/tensorflow/lite/g3doc/guide/ios.md
index 4e43fee47e4..8f15069201b 100644
--- a/tensorflow/lite/g3doc/guide/ios.md
+++ b/tensorflow/lite/g3doc/guide/ios.md
@@ -68,7 +68,17 @@ pod is used in your app. Alternatively, if you want to depend on the nightly
 builds, you can write:
 
 ```ruby
-pod 'TensorFlowLiteSwift', '0.0.1-nightly'
+pod 'TensorFlowLiteSwift', '~> 0.0.1-nightly'
+```
+
+For nightly version, by default
+[GPU](https://www.tensorflow.org/lite/performance/gpu) and
+[Core ML delegates](https://www.tensorflow.org/lite/performance/coreml_delegate)
+are excluded from the pod to reduce the binary size. You can include them by
+specifying subspec:
+
+```ruby
+pod 'TensorFlowLiteSwift', '~> 0.0.1-nightly', :subspecs => ['CoreML', 'Metal']
 ```
 
 This will allow you to use the latest features added to TensorFlow Lite. Note
diff --git a/tensorflow/lite/g3doc/guide/ops_compatibility.md b/tensorflow/lite/g3doc/guide/ops_compatibility.md
index 17dfedddd65..054b7e0e275 100644
--- a/tensorflow/lite/g3doc/guide/ops_compatibility.md
+++ b/tensorflow/lite/g3doc/guide/ops_compatibility.md
@@ -9,8 +9,8 @@ Since the set of TensorFlow Lite operations is smaller than TensorFlow's, not
 every model is convertible. Even for supported operations, very specific usage
 patterns are sometimes expected, for performance reasons. We expect to expand
 the set of supported operations in future TensorFlow Lite releases. Additional
-ops can be included by [using select TensorFlow ops](ops_select.md), at
-the cost of binary size.
+ops can be included by [using select TensorFlow ops](ops_select.md), at the cost
+of binary size.
 
 The best way to understand how to build a TensorFlow model that can be used with
 TensorFlow Lite is to carefully consider how operations are converted and
@@ -23,7 +23,7 @@ quantized (`uint8`, `int8`) inference, but many ops do not yet for other types
 like `tf.float16` and strings.
 
 Apart from using different version of the operations, the other difference
-between floating-point and quantized models lies in the way they are converted.
+between floating-point and quantized models is the way they are converted.
 Quantized conversion requires dynamic range information for tensors. This
 requires "fake-quantization" during model training, getting range information
 via a calibration data set, or doing "on-the-fly" range estimation. See
@@ -32,16 +32,16 @@ via a calibration data set, or doing "on-the-fly" range estimation. See
 ## Data format and broadcasting
 
 At the moment TensorFlow Lite supports only TensorFlow's "NHWC" format, and
-broadcasting is only support in a limited number of ops (tf.add, tf.mul, tf.sub,
-and tf.div).
+broadcasting is only support in a limited number of ops (`tf.add`, `tf.mul`,
+`tf.sub`, and `tf.div`).
 
 ## Compatible operations
 
 The following TensorFlow operations are usually mapped to their TensorFlow Lite
 counterparts:
 
-*   `tf.batch_to_space_nd` —As long as the input tensor is 4D (1 batch + 2
-    spatial + 1 other) and the crops attribute is not used.
+*   `tf.batch_to_space_nd` —As long as the input tensor is 3D or 4D (1 batch + 1
+    or 2 spatial + 1 other) and the crops attribute is not used.
 *   `tf.exp`
 *   `tf.fake_quant`
 *   `tf.matmul` —As the second argument is constant and transposition is not
@@ -58,28 +58,28 @@ counterparts:
 *   `tf.nn.softmax` —As long as tensors are 2D and axis is the last dimension.
 *   `tf.nn.top_k`
 *   `tf.one_hot`
-*   `tf.pad` —As long as mode and constant_values are not used.
-*   `tf.reduce_mean` —As long as the reduction_indices attribute is not used.
+*   `tf.pad` —As long as `mode` and `constant_values` are not used.
+*   `tf.reduce_mean` —As long as the `reduction_indices` attribute is not used.
 *   `tf.reshape`
 *   `tf.sigmoid`
-*   `tf.space_to_batch_nd` —As long as the input tensor is 4D (1 batch + 2
-    spatial + 1 other).
+*   `tf.space_to_batch_nd` —As long as the input tensor is 3D or 4D (1 batch + 1
+    or 2 spatial + 1 other).
 *   `tf.space_to_depth`
 *   `tf.split` —As long as num is not provided and `num_or_size_split` contains
     number of splits as a 0D tensor.
-*   `tf.squeeze` —As long as axis is not provided.
+*   `tf.squeeze` —As long as `axis` is not provided.
 *   `tf.squared_difference`
-*   `tf.strided_slice` —As long as `ellipsis_mask and new_axis_mask` are not
+*   `tf.strided_slice` —As long as `ellipsis_mask` and `new_axis_mask` are not
     used.
-*   `tf.transpose` —As long as conjugate is not used.
+*   `tf.transpose` —As long as `conjugate` is not used.
 
 ## Straight-forward conversions, constant-folding and fusing
 
 A number of TensorFlow operations can be processed by TensorFlow Lite even
 though they have no direct equivalent. This is the case for operations that can
-be simply removed from the graph (tf.identity), replaced by tensors
-(tf.placeholder), or fused into more complex operations (tf.nn.bias_add). Even
-some supported operations may sometimes be removed through one of these
+be simply removed from the graph (`tf.identity`), replaced by tensors
+(`tf.placeholder`), or fused into more complex operations (`tf.nn.bias_add`).
+Even some supported operations may sometimes be removed through one of these
 processes.
 
 Here is a non-exhaustive list of TensorFlow operations that are usually removed
@@ -115,7 +115,7 @@ from the graph:
 *   `tf.nn.relu`
 *   `tf.nn.relu6`
 
-Note: Many of those operations don't have TensorFlow Lite equivalents and the
+Note: Many of those operations don't have TensorFlow Lite equivalents, and the
 corresponding model will not be convertible if they can't be elided or fused.
 
 ## Unsupported operations
@@ -213,7 +213,7 @@ Options {
 
 ```
 Inputs {
-  0: 4D tensor
+  0: 3D-4D tensor
   1: 1D tensor
   2: 2D tensor
 }
@@ -343,10 +343,10 @@ Outputs {
 **FLOOR**
 
 ```
-inputs {
+Inputs {
   0: tensor
 }
-outputs: {
+Outputs: {
   0: result of computing element-wise floor of the input tensor
 }
 ```
@@ -939,7 +939,7 @@ Options {
 
 ```
 Inputs {
-  0: 4D tensor
+  0: 3D-4D tensor
   1: 1D tensor
   2: 2D tensor
 }
diff --git a/tensorflow/lite/g3doc/guide/ops_custom.md b/tensorflow/lite/g3doc/guide/ops_custom.md
index 749fcb1be6a..e2ccc9c72d8 100644
--- a/tensorflow/lite/g3doc/guide/ops_custom.md
+++ b/tensorflow/lite/g3doc/guide/ops_custom.md
@@ -1,32 +1,31 @@
 # Custom operators
 
 TensorFlow Lite currently supports a subset of TensorFlow operators. It supports
-the use of user-provided implementations (as known as custom implementations) if
+the use of user-provided implementations (known as custom implementations) if
 the model contains an operator that is not supported. Providing custom kernels
-is also a way of evaluating a series of TensorFlow operations as a single fused
-TensorFlow Lite operations.
+is also a way of executing a series of TensorFlow operations as a single fused
+TensorFlow Lite operation.
 
 Using custom operators consists of three steps.
 
-*   Making sure the TensorFlow Graph Def or SavedModel refers to the correctly
+*   Make sure the TensorFlow Graph Def or SavedModel refers to the correctly
     named TensorFlow Lite operator.
 
-*   Registering a custom kernel with TensorFlow Lite so that the runtime knows
-    how to map your operator and parameters in your graph to executable C/C++
-    code.
+*   Register a custom kernel with TensorFlow Lite so that the runtime knows how
+    to map your operator and parameters in your graph to executable C/C++ code.
 
-*   Testing and profiling your operator correctness and performance,
-    respectively. If you wish to test just your custom operator it is best to
-    create a model with just your custom operator and using the
+*   Test and profile your operator correctness and performance, respectively. If
+    you wish to test just your custom operator, it is best to create a model
+    with just your custom operator and using the
     [benchmark_model](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/tools/benchmark/benchmark_model_test.cc)
-    program
+    program.
 
-Below we describe a complete example of defining Sin and some links to existing
-conversion process involving custom operators.
+Below we describe a complete example of defining `Sin` and some links to
+existing conversion process involving custom operators.
 
 ## Making a custom operator for Sin
 
-Let’s walk through this an example of supporting a TensorFlow operator that
+Let’s walk through an example of supporting a TensorFlow operator that
 TensorFlow Lite does not have. Assume we are using the `Sin` operator and that
 we are building a very simple model for a function `y = sin(x + offset)`, where
 `offset` is trainable.
@@ -45,11 +44,11 @@ optimizer = tf.train.GradientDescentOptimizer(0.001)
 train = optimizer.minimize(loss)
 ```
 
-If you convert this model to Tensorflow Lite format using the TensorFlow Lite
+If you convert this model to TensorFlow Lite format using the TensorFlow Lite
 Optimizing Converter with `--allow_custom_ops` argument, and run it with the
 default interpreter, the interpreter will raise the following error messages:
 
-```
+```none
 Didn't find custom op for name 'Sin'
 Registration failed.
 ```
@@ -57,8 +56,7 @@ Registration failed.
 ### Defining the kernel in the TensorFlow Lite runtime
 
 All we need to do to use the op in TensorFlow Lite is define two functions
-(`Prepare` and `Eval`), and construct a `TfLiteRegistration`. This code would
-look something like this:
+(`Prepare` and `Eval`), and construct a `TfLiteRegistration`:
 
 ```cpp
 TfLiteStatus SinPrepare(TfLiteContext* context, TfLiteNode* node) {
@@ -105,44 +103,45 @@ TfLiteRegistration* Register_SIN() {
 }
 ```
 
-When initializing the `OpResolver`, add the custom op into the resolver, this
+When initializing the `OpResolver`, add the custom op into the resolver. This
 will register the operator with Tensorflow Lite so that TensorFlow Lite can use
-the new implementation. Note that the last two arguments in TfLiteRegistration
-correspond to the `SinPrepare` and `SinEval()` functions you defined for the
-custom op. If you used two functions to initialize variables used in the op and
-free up space: `Init()` and `Free()`, then they would be added to the first two
-arguments of TfLiteRegistration; they are set to nullptr in this example.
+the new implementation. Note that the last two arguments in `TfLiteRegistration`
+correspond to the `SinPrepare` and `SinEval` functions you defined for the
+custom op. If you used `SinInit` and `SinFree` functions to initialize variables
+used in the op and to free up space, respectively, then they would be added to
+the first two arguments of `TfLiteRegistration`; those arguments are set to
+`nullptr` in this example.
 
 ```cpp
 tflite::ops::builtin::BuiltinOpResolver builtins;
 builtins.AddCustom("Sin", Register_SIN());
 ```
 
-If you want to make your custom operators in Java, you would currently need to
+If you want to define your custom operators in Java, you would currently need to
 build your own custom JNI layer and compile your own AAR
 [in this jni code](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/java/src/main/native/builtin_ops_jni.cc).
-Similarly, if you wish to make these operators available in Python you can place
-your registrations in the
+Similarly, if you wish to define these operators available in Python you can
+place your registrations in the
 [Python wrapper code](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/python/interpreter_wrapper/interpreter_wrapper.cc).
 
-Note that a similar process as above can be followed for supporting for a set of
+Note that a similar process as above can be followed for supporting a set of
 operations instead of a single operator. Just add as many `AddCustom` operators
 as you need. In addition, `BuiltinOpResolver` also allows you to override
 implementations of builtins by using the `AddBuiltin`.
 
-## Best Practices
+## Best practices
 
 ### Writing TensorFlow Lite kernels best practices
 
-1.  Optimize memory allocations and de-allocations cautiously. It is more
-    efficient to allocate memory in Prepare() instead of Invoke(), and allocate
-    memory before a loop instead of in every iteration. Use temporary tensors
-    data rather than mallocing yourself (see item 2). Use pointers/references
-    instead of copying as much as possible.
+1.  Optimize memory allocations and de-allocations cautiously. Allocating memory
+    in `Prepare` is more efficient than in `Invoke`, and allocating memory
+    before a loop is better than in every iteration. Use temporary tensors data
+    rather than mallocing yourself (see item 2). Use pointers/references instead
+    of copying as much as possible.
 
 2.  If a data structure will persist during the entire operation, we advise
     pre-allocating the memory using temporary tensors. You may need to use
-    OpData struct to reference the tensor indices in other functions. See
+    OpData struct to reference the tensor indices in other functions. See the
     example in the
     [kernel for convolution](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/kernels/conv.cc).
     A sample code snippet is below
@@ -158,23 +157,24 @@ implementations of builtins by using the `AddBuiltin`.
     ```
 
 3.  If it doesn't cost too much wasted memory, prefer using a static fixed size
-    array (or in Resize() pre-allocated std::vector) rather than using a
-    dynamically allocating std::vector every iteration of execution.
+    array (or a pre-allocated `std::vector` in `Resize`) rather than using a
+    dynamically allocated `std::vector` every iteration of execution.
 
 4.  Avoid instantiating standard library container templates that don't already
-    exist, because they affect binary size. For example, if you need a std::map
-    in your operation that doesn't exist in other kernels, using a std::vector
-    with direct indexing mapping could work while keeping the binary size small.
-    See what other kernels use to gain insight (or ask).
+    exist, because they affect binary size. For example, if you need a
+    `std::map` in your operation that doesn't exist in other kernels, using a
+    `std::vector` with direct indexing mapping could work while keeping the
+    binary size small. See what other kernels use to gain insight (or ask).
 
-5.  Check the pointer to the memory returned by malloc. If this pointer is
-    nullptr, no operations should be performed using that pointer. If you
-    malloc() in a function and have an error exit, deallocate memory before you
+5.  Check the pointer to the memory returned by `malloc`. If this pointer is
+    `nullptr`, no operations should be performed using that pointer. If you
+    `malloc` in a function and have an error exit, deallocate memory before you
     exit.
 
-6.  Use TF_LITE_ENSURE(context, condition) to check for a specific condition.
-    Your code must not leave memory hanging when TF_LITE_ENSURE is done, i.e.,
-    these should be done before any resources are allocated that will leak.
+6.  Use `TF_LITE_ENSURE(context, condition)` to check for a specific condition.
+    Your code must not leave memory hanging when `TF_LITE_ENSURE` is used, i.e.,
+    these macros should be used before any resources are allocated that will
+    leak.
 
 ### Conversion best practices
 
@@ -187,10 +187,10 @@ instead of the builtin TensorFlow one.
 #### Converting TensorFlow models to convert graphs
 
 In TensorFlow you can use the `tf.lite.OpHint` class to encapsulate groups of
-operators when you create a TensorFlow graph. This allows you then to extract a
-graph def that has references to those operators. This is currently experimental
-and should only be used by advanced users. There is a full example of how to use
-this in the
+operators when you create a TensorFlow graph. This encapsulation allows you then
+to extract a graph def that has references to those operators. `tf.lite.OpHint`
+is currently experimental and should only be used by advanced users. A full
+example of how to use this class is in the
 [OpHint code](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/python/op_hint.py).
 
 In addition, you can also use a manual graph substitution approach to rewrite
@@ -198,23 +198,23 @@ Tensorflow graphs. There is an example of how this is done in single shot object
 based detection models
 [export script](https://github.com/tensorflow/models/blob/master/research/object_detection/export_tflite_ssd_graph.py).
 
-### TF Graph Attributes
+### TF graph attributes
 
 When `tflite_convert` converts a TensorFlow graph into TFLite format, it makes
-some assumption about custom operations that might not be correct. In this case,
+some assumptions about custom operations. If the assumptions are not correct,
 the generated graph may not execute.
 
-It is possible to add additional information about your custom op output to TF
-graph before it is converted. The following attributes are supported:
+It is possible to add additional information about your custom op output to the
+TF graph before it is converted. The following attributes are supported:
 
 -   **_output_quantized** a boolean attribute, true if the operation outputs are
     quantized
 -   **_output_types** a list of types for output tensors
 -   **_output_shapes** a list of shapes for output tensors
 
-#### Setting the Attributes
+#### Setting the attributes
 
-This is an example how the attributes can be set:
+The following example demonstrates how the attributes can be set:
 
 ```python
 frozen_graph_def = tf.graph_util.convert_variables_to_constants(...)
@@ -231,5 +231,5 @@ tflite_model = tf.lite.toco_convert(
         frozen_graph_def,...)
 ```
 
-**Note:** After the attributes are set, the graph can not be executed by
-Tensorflow, therefore it should be done just before the conversion.
+**Note:** After the attributes are set, the graph cannot be executed by
+TensorFlow. Therefore, the attributes should be set just before the conversion.
diff --git a/tensorflow/lite/g3doc/guide/ops_select.md b/tensorflow/lite/g3doc/guide/ops_select.md
index 8a9109cf54c..2226d86b1c9 100644
--- a/tensorflow/lite/g3doc/guide/ops_select.md
+++ b/tensorflow/lite/g3doc/guide/ops_select.md
@@ -2,9 +2,9 @@
 
 Caution: This feature is experimental.
 
-The TensorFlow Lite builtin op library has grown rapidly, and will continue to
+The TensorFlow Lite builtin op library has grown rapidly and will continue to
 grow, but there remains a long tail of TensorFlow ops that are not yet natively
-supported by TensorFlow Lite . These unsupported ops can be a point of friction
+supported by TensorFlow Lite. These unsupported ops can be a point of friction
 in the TensorFlow Lite model conversion process. To that end, the team has
 recently been working on an experimental mechanism for reducing this friction.
 
@@ -55,7 +55,7 @@ limitations.
 The following example shows how to use this feature in the
 [`TFLiteConverter`](./convert/python_api.md) Python API.
 
-```
+```python
 import tensorflow as tf
 
 converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
@@ -69,7 +69,7 @@ The following example shows how to use this feature in the
 [`tflite_convert`](../convert/cmdline_examples.md) command line tool using the
 command line flag `target_ops`.
 
-```
+```sh
 tflite_convert \
   --output_file=/tmp/foo.tflite \
   --graph_def_file=/tmp/foo.pb \
@@ -81,7 +81,7 @@ tflite_convert \
 When building and running `tflite_convert` directly with `bazel`, please pass
 `--define=tflite_convert_with_select_tf_ops=true` as an additional argument.
 
-```
+```sh
 bazel run --define=tflite_convert_with_select_tf_ops=true tflite_convert -- \
   --output_file=/tmp/foo.tflite \
   --graph_def_file=/tmp/foo.pb \
@@ -157,7 +157,7 @@ Finally, in your app's `build.gradle`, ensure you have the `mavenLocal()`
 dependency and replace the standard TensorFlow Lite dependency with the one that
 has support for select TensorFlow ops:
 
-```
+```build
 allprojects {
     repositories {
         jcenter()
@@ -220,7 +220,7 @@ creating the interpreter at runtime as long as the delegate is linked into the
 client library. It is not necessary to explicitly install the delegate instance
 as is typically required with other delegate types.
 
-### Python pip Package
+### Python pip package
 
 Python support is actively under development.
 
@@ -241,7 +241,7 @@ Build                                | Time (milliseconds)
 Only built-in ops (`TFLITE_BUILTIN`) | 260.7
 Using only TF ops (`SELECT_TF_OPS`)  | 264.5
 
-### Binary Size
+### Binary size
 
 The following table describes the binary size of TensorFlow Lite for each build.
 These targets were built for Android using `--config=android_arm -c opt`.
@@ -251,22 +251,22 @@ Build                 | C++ Binary Size | Android APK Size
 Only built-in ops     | 796 KB          | 561 KB
 Built-in ops + TF ops | 23.0 MB         | 8.0 MB
 
-## Known Limitations
+## Known limitations
 
 The following is a list of some of the known limitations:
 
 *   Control flow ops are not yet supported.
 *   The
     [`post_training_quantization`](https://www.tensorflow.org/performance/post_training_quantization)
-    flag is currently not supported for TensorFlow ops so it will not quantize
+    flag is currently not supported for TensorFlow ops, so it will not quantize
     weights for any TensorFlow ops. In models with both TensorFlow Lite builtin
     ops and TensorFlow ops, the weights for the builtin ops will be quantized.
-*   Ops that require explicit initialization from resources, like HashTableV2,
+*   Ops that require explicit initialization from resources, like `HashTableV2`,
     are not yet supported.
 *   Certain TensorFlow ops may not support the full set of input/output types
     that are typically available on stock TensorFlow.
 
-## Future Plans
+## Future plans
 
 The following is a list of improvements to this pipeline that are in progress:
 
@@ -276,5 +276,5 @@ The following is a list of improvements to this pipeline that are in progress:
 *   *Improved usability* - The conversion process will be simplified to only
     require a single pass through the converter. Additionally, pre-built Android
     AAR and iOS CocoaPod binaries will be provided.
-*   *Improved performance* - There is work being done to ensure TensorFlow Lite
-    with TensorFlow ops has performance parity to TensorFlow Mobile.
+*   *Improved performance* - Work is being done to ensure TensorFlow Lite with
+    TensorFlow ops has performance parity to TensorFlow Mobile.
diff --git a/tensorflow/lite/g3doc/guide/ops_version.md b/tensorflow/lite/g3doc/guide/ops_version.md
index 87e9f6900f4..12e790c7fb2 100644
--- a/tensorflow/lite/g3doc/guide/ops_version.md
+++ b/tensorflow/lite/g3doc/guide/ops_version.md
@@ -13,7 +13,7 @@ existing ops. In addition, it guarantees the following:
     reads a new model that contains a new version of an op which isn't
     supported, it should report the error.
 
-## Example: Adding Dilation into Convolution
+## Example: Adding dilation into convolution
 
 The remainder of this document explains op versioning in TFLite by showing how
 to add dilation parameters to the convolution operation.
@@ -25,7 +25,7 @@ Knowledge of dilation is not required to understand this document. Note that:
 *   Old convolution kernels that don't support dilation are equivalent to
     setting the dilation factors to 1.
 
-### Change FlatBuffer Schema
+### Change FlatBuffer schema
 
 To add new parameters into an op, change the options table in
 `lite/schema/schema.fbs`.
@@ -66,7 +66,7 @@ table Conv2DOptions {
 The file `lite/schema/schema_generated.h` should be re-generated for the new
 schema.
 
-### Change C Structures and Kernel Implementation
+### Change C structures and kernel implementation
 
 In TensorFlow Lite, the kernel implementation is decoupled from FlatBuffer
 definition. The kernels read the parameter from C structures defined in
@@ -103,7 +103,7 @@ typedef struct {
 Please also change the kernel implementation to read the newly added parameters
 from the C structures. The details are omitted here.
 
-### Change the FlatBuffer Reading Code
+### Change the FlatBuffer reading code
 
 The logic to read FlatBuffer and produce C structure is in
 `lite/core/api/flatbuffer_conversions.cc`.
@@ -132,7 +132,7 @@ reads an old model file where dilation factors are missing, it will use 1 as
 the default value, and the new kernel will work consistently with the old
 kernel.
 
-### Change Kernel Registration
+### Change kernel registration
 
 The MutableOpResolver (defined in `lite/op_resolver.h`) provides a few functions
 to register op kernels. The minimum and maximum version are 1 by default:
@@ -192,23 +192,24 @@ int GetVersion(const Operator& op) const override {
 ### Update the operator version map
 
 The last step is to add the new version info into the operator version map. This
-step is required because we need generate the model's minimum required runtime
-version based on this version map.
+step is required because we need to generate the model's minimum required
+runtime version based on this version map.
 
 To do this, you need to add a new map entry in `lite/toco/tflite/op_version.cc`.
 
-In this example, it means you need to add the following into `op_version_map`:
+In this example, you need to add the following entry into `op_version_map`:
+
 ```
 {{OperatorType::kConv, 3}, "kPendingReleaseOpVersion"}
 ```
 (`kPendingReleaseOpVersion` will be replaced with the appropriate release
 version in the next stable release.)
 
-### Delegation Implementation
+### Delegation implementation
 
 TensorFlow Lite provides a delegation API which enables delegating ops to
-hardware backends. In Delegate's `Prepare` function, check if the version
-is supported for every node in Delegation code.
+hardware backends. In the delegate's `Prepare` function, check if the version is
+supported for every node in Delegation code.
 
 ```
 const int kMinVersion = 1;
diff --git a/tensorflow/lite/g3doc/microcontrollers/get_started.md b/tensorflow/lite/g3doc/microcontrollers/get_started.md
index 96fa336c2ef..008d5c65422 100644
--- a/tensorflow/lite/g3doc/microcontrollers/get_started.md
+++ b/tensorflow/lite/g3doc/microcontrollers/get_started.md
@@ -95,14 +95,14 @@ To use the TensorFlow Lite for Microcontrollers library, we must include the
 following header files:
 
 ```C++
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/micro_error_reporter.h"
 #include "tensorflow/lite/micro/micro_interpreter.h"
 #include "tensorflow/lite/schema/schema_generated.h"
 #include "tensorflow/lite/version.h"
 ```
 
--   [`all_ops_resolver.h`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/micro/kernels/all_ops_resolver.h)
+-   [`all_ops_resolver.h`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/micro/all_ops_resolver.h)
     provides the operations used by the interpreter to run the model.
 -   [`micro_error_reporter.h`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/micro/micro_error_reporter.h)
     outputs debug information.
@@ -182,12 +182,12 @@ if (model->version() != TFLITE_SCHEMA_VERSION) {
 ### 6. Instantiate operations resolver
 
 An
-[`AllOpsResolver`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/micro/kernels/all_ops_resolver.h)
+[`AllOpsResolver`](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/micro/all_ops_resolver.h)
 instance is declared. This will be used by the interpreter to access the
 operations that are used by the model:
 
 ```C++
-tflite::ops::micro::AllOpsResolver resolver;
+tflite::AllOpsResolver resolver;
 ```
 
 The `AllOpsResolver` loads all of the operations available in TensorFlow Lite
@@ -372,5 +372,5 @@ applications on GitHub</a>
 To learn how to use the library in your own project, read
 [Understand the C++ library](library.md).
 
-For information about training and convert models for deployment on
+For information about training and converting models for deployment on
 microcontrollers, read [Build and convert models](build_convert.md).
diff --git a/tensorflow/lite/g3doc/models/bert_qa/overview.md b/tensorflow/lite/g3doc/models/bert_qa/overview.md
index 21756a94905..314a8d22b1b 100644
--- a/tensorflow/lite/g3doc/models/bert_qa/overview.md
+++ b/tensorflow/lite/g3doc/models/bert_qa/overview.md
@@ -1,4 +1,4 @@
-# Question and Answer
+# Question and answer
 
 Use a pre-trained model to answer questions based on the content of a given
 passage.
@@ -44,7 +44,7 @@ pre-processing including tokenization and post-processing steps that are
 described in the BERT [paper](https://arxiv.org/abs/1810.04805) and implemented
 in the sample app.
 
-## Performance Benchmarks
+## Performance benchmarks
 
 Performance benchmark numbers are generated with the tool
 [described here](https://www.tensorflow.org/lite/performance/benchmarks).
diff --git a/tensorflow/lite/g3doc/models/image_classification/overview.md b/tensorflow/lite/g3doc/models/image_classification/overview.md
index b0b177e5a70..cff23134bda 100644
--- a/tensorflow/lite/g3doc/models/image_classification/overview.md
+++ b/tensorflow/lite/g3doc/models/image_classification/overview.md
@@ -216,7 +216,7 @@ experiment with different models to find the optimal balance between
 performance, accuracy, and model size. For guidance, see
 <a href="#choose_a_different_model">Choose a different model</a>.
 
-## Performance Benchmarks
+## Performance benchmarks
 
 Performance benchmark numbers are generated with the tool
 [described here](https://www.tensorflow.org/lite/performance/benchmarks).
@@ -260,7 +260,7 @@ Performance benchmark numbers are generated with the tool
 
 ## Choose a different model
 
-There are a large number of image classification models available on our
+A large number of image classification models are available on our
 <a href="../../guide/hosted_models.md">List of hosted models</a>. You should aim
 to choose the optimal model for your application based on performance, accuracy
 and model size. There are trade-offs between each of them.
@@ -302,7 +302,7 @@ Our quantized MobileNet models’ size ranges from 0.5 to 3.4 Mb.
 
 ### Architecture
 
-There are several different architectures of models available on
+Several different model architectures are available on
 <a href="../../guide/hosted_models.md">List of hosted models</a>, indicated by
 the model’s name. For example, you can choose between MobileNet, Inception, and
 others.
diff --git a/tensorflow/lite/g3doc/models/object_detection/overview.md b/tensorflow/lite/g3doc/models/object_detection/overview.md
index 61bdca21927..4b29211f642 100644
--- a/tensorflow/lite/g3doc/models/object_detection/overview.md
+++ b/tensorflow/lite/g3doc/models/object_detection/overview.md
@@ -180,7 +180,7 @@ edges in a similar manner.
 
 Note: Object detection models accept input images of a specific size. This is likely to be different from the size of the raw image captured by your device’s camera, and you will have to write code to crop and scale your raw image to fit the model’s input size (there are examples of this in our <a href="#get_started">example applications</a>).<br /><br />The pixel values output by the model refer to the position in the cropped and scaled image, so you must scale them to fit the raw image in order to interpret them correctly.
 
-## Performance Benchmarks
+## Performance benchmarks
 
 Performance benchmark numbers are generated with the tool
 [described here](https://www.tensorflow.org/lite/performance/benchmarks).
diff --git a/tensorflow/lite/g3doc/models/segmentation/overview.md b/tensorflow/lite/g3doc/models/segmentation/overview.md
index 816ba9244f3..497a9a39eca 100644
--- a/tensorflow/lite/g3doc/models/segmentation/overview.md
+++ b/tensorflow/lite/g3doc/models/segmentation/overview.md
@@ -43,7 +43,7 @@ The current implementation includes the following features:
   <li>DeepLabv3+: We extend DeepLabv3 to include a simple yet effective decoder module to refine the segmentation results especially along object boundaries. Furthermore, in this encoder-decoder structure one can arbitrarily control the resolution of extracted encoder features by atrous convolution to trade-off precision and runtime.</li>
 </ol>
 
-## Performance Benchmarks
+## Performance benchmarks
 
 Performance benchmark numbers are generated with the tool
 [described here](https://www.tensorflow.org/lite/performance/benchmarks).
diff --git a/tensorflow/lite/g3doc/models/text_classification/overview.md b/tensorflow/lite/g3doc/models/text_classification/overview.md
index 446d4625013..a462507b56a 100644
--- a/tensorflow/lite/g3doc/models/text_classification/overview.md
+++ b/tensorflow/lite/g3doc/models/text_classification/overview.md
@@ -1,4 +1,4 @@
-# Text Classification
+# Text classification
 
 Use a pre-trained model to category a paragraph into predefined groups.
 
@@ -44,7 +44,7 @@ Here are the steps to classify a paragraph with the model:
 *   This model was trained on movie reviews dataset so you may experience
     reduced accuracy when classifying text of other domains.
 
-## Performance Benchmarks
+## Performance benchmarks
 
 Performance benchmark numbers are generated with the tool
 [described here](https://www.tensorflow.org/lite/performance/benchmarks).
diff --git a/tensorflow/lite/g3doc/performance/benchmarks.md b/tensorflow/lite/g3doc/performance/benchmarks.md
index 2dc20f2d74c..7b1eb5c9919 100644
--- a/tensorflow/lite/g3doc/performance/benchmarks.md
+++ b/tensorflow/lite/g3doc/performance/benchmarks.md
@@ -19,7 +19,7 @@ and assumed in the `/data/local/tmp` directory.
 
 To run the benchmark:
 
-```
+```sh
 adb shell /data/local/tmp/benchmark_model \
   --num_threads=4 \
   --graph=/data/local/tmp/tflite_models/${GRAPH} \
@@ -27,8 +27,8 @@ adb shell /data/local/tmp/benchmark_model \
   --num_runs=50
 ```
 
-To run with nnapi delegate, please set --use_nnapi=true. To run with gpu
-delegate, please set --use_gpu=true.
+To run with nnapi delegate, please set `--use_nnapi=true`. To run with gpu
+delegate, please set `--use_gpu=true`.
 
 The performance values below are measured on Android 10.
 
diff --git a/tensorflow/lite/g3doc/performance/best_practices.md b/tensorflow/lite/g3doc/performance/best_practices.md
index 7f7ebc465f4..94436865e48 100644
--- a/tensorflow/lite/g3doc/performance/best_practices.md
+++ b/tensorflow/lite/g3doc/performance/best_practices.md
@@ -50,9 +50,8 @@ operator is executed. Check out our
 ## Optimize your model
 
 Model optimization aims to create smaller models that are generally faster and
-more energy efficient, so that they can be deployed on mobile devices. There are
-multiple optimization techniques supported by TensorFlow Lite, such as
-quantization.
+more energy efficient, so that they can be deployed on mobile devices.
+TensorFlow Lite supports multiple optimization techniques, such as quantization.
 
 Check out our [model optimization docs](model_optimization.md) for details.
 
@@ -78,7 +77,7 @@ If your application is not carefully designed, there can be redundant copies
 when feeding the input to and reading the output from the model. Make sure to
 eliminate redundant copies. If you are using higher level APIs, like Java, make
 sure to carefully check the documentation for performance caveats. For example,
-the Java API is a lot faster if ByteBuffers are used as
+the Java API is a lot faster if `ByteBuffers` are used as
 [inputs](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/java/src/main/java/org/tensorflow/lite/Interpreter.java#L175).
 
 ## Profile your application with platform specific tools
diff --git a/tensorflow/lite/g3doc/performance/coreml_delegate.md b/tensorflow/lite/g3doc/performance/coreml_delegate.md
index c3d72b2e01f..36781a18de1 100644
--- a/tensorflow/lite/g3doc/performance/coreml_delegate.md
+++ b/tensorflow/lite/g3doc/performance/coreml_delegate.md
@@ -1,8 +1,8 @@
-# Tensorflow Lite Core ML Delegate
+# Tensorflow Lite Core ML delegate
 
-TensorFlow Lite Core ML Delegate enables running TensorFlow Lite models on
-[Core ML framework](https://developer.apple.com/documentation/coreml),
-which results in faster model inference on iOS devices.
+The TensorFlow Lite Core ML delegate enables running TensorFlow Lite models on
+[Core ML framework](https://developer.apple.com/documentation/coreml), which
+results in faster model inference on iOS devices.
 
 Note: This delegate is in experimental (beta) phase.
 
@@ -25,13 +25,21 @@ The Core ML delegate currently supports float32 models.
 
 The Core ML delegate is already included in nightly release of TensorFlow lite
 CocoaPods. To use Core ML delegate, change your TensorFlow lite pod
-(`TensorflowLiteC` for C++ API, and `TensorFlowLiteSwift` for Swift) version to
-`0.0.1-nightly` in your `Podfile`.
+(`TensorflowLiteC` for C API, and `TensorFlowLiteSwift` for Swift) version to
+`0.0.1-nightly` in your `Podfile`, and include subspec `CoreML`
 
 ```
 target 'YourProjectName'
   # pod 'TensorFlowLiteSwift'
-  pod 'TensorFlowLiteSwift', '~> 0.0.1-nightly'
+  pod 'TensorFlowLiteSwift/CoreML', '~> 0.0.1-nightly'
+```
+
+OR
+
+```
+target 'YourProjectName'
+  # pod 'TensorFlowLiteSwift'
+  pod 'TensorFlowLiteSwift', '~> 0.0.1-nightly', :subspecs => ['CoreML']
 ```
 
 Note: After updating `Podfile`, you should run `pod update` to reflect changes.
diff --git a/tensorflow/lite/g3doc/performance/delegates.md b/tensorflow/lite/g3doc/performance/delegates.md
index 4e6d7d09c73..1d11a8eb391 100644
--- a/tensorflow/lite/g3doc/performance/delegates.md
+++ b/tensorflow/lite/g3doc/performance/delegates.md
@@ -1,6 +1,7 @@
 # TensorFlow Lite delegates
 
 Note: Delegate API is still experimental and is subject to change.
+
 ## What is a TensorFlow Lite delegate?
 
 A TensorFlow Lite delegate is a way to delegate part or all of graph execution
@@ -51,9 +52,9 @@ If a delegate was provided for specific operations, then TensorFlow Lite will
 split the graph into multiple subgraphs where each subgraph will be handled by a
 delegate.
 
-Let's assume that there is a delegate "MyDelegate," which has a faster
-implementation for Conv2D and Mean operations. The resulting main graph will be
-updated to look like below.
+Let's assume that a delegate, `MyDelegate`, has a faster implementation for
+Conv2D and Mean operations. The resulting main graph will be updated to look
+like below.
 
 ![Graph with delegate](../images/performance/tflite_delegate_graph_2.png "Graph with delegate")
 
@@ -74,16 +75,16 @@ _Note that the API used below is experimental and is subject to change._
 Based on the previous section, to add a delegate, we need to do the following:
 
 1.  Define a kernel node that is responsible for evaluating the delegate
-    subgraph
+    subgraph.
 1.  Create an instance of
     [TfLiteDelegate](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/c/common.h#L611),
     which is responsible for registering the kernel node and claiming the nodes
-    that the delegate can execute
+    that the delegate can execute.
 
-To see it in code, let's define a delegate and call it "MyDelegate," which can
+To see it in code, let's define a delegate and call it `MyDelegate`, which can
 execute Conv2D and Mean operations faster.
 
-```
+```c++
 // This is where the execution of the operations or whole graph happens.
 // The class below has an empty implementation just as a guideline
 // on the structure.
@@ -113,9 +114,9 @@ class MyDelegate {
 // the subgraph in the main TfLite graph.
 TfLiteRegistration GetMyDelegateNodeRegistration() {
   // This is the registration for the Delegate Node that gets added to
-  // the TFLite graph instead of the subGraph it replaces.
-  // It is treated as a an OP node. But in our case
-  // Init will initialize the delegate
+  // the TFLite graph instead of the subgraph it replaces.
+  // It is treated as an OP node. But in our case
+  // Init will initialize the delegate.
   // Invoke will run the delegate graph.
   // Prepare for preparing the delegate.
   // Free for any cleaning needed by the delegate.
@@ -232,6 +233,4 @@ if (interpreter->ModifyGraphWithDelegate(my_delegate) !=
 ...
 // Don't forget to delete your delegate
 delete my_delegate;
-
-
 ```
diff --git a/tensorflow/lite/g3doc/performance/gpu.md b/tensorflow/lite/g3doc/performance/gpu.md
index b5abf46f845..0a0826b24b3 100644
--- a/tensorflow/lite/g3doc/performance/gpu.md
+++ b/tensorflow/lite/g3doc/performance/gpu.md
@@ -18,7 +18,7 @@ Another benefit with GPU inference is its power efficiency. GPUs carry out the
 computations in a very efficient and optimized manner, so that they consume less
 power and generate less heat than when the same task is run on CPUs.
 
-## Demo App Tutorials
+## Demo app tutorials
 
 The easiest way to try out the GPU delegate is to follow the below tutorials,
 which go through building our classification demo applications with GPU support.
@@ -35,7 +35,7 @@ Note: This requires OpenCL or OpenGL ES (3.1 or higher).
 
 #### Step 1. Clone the TensorFlow source code and open it in Android Studio
 
-```
+```sh
 git clone https://github.com/tensorflow/tensorflow
 ```
 
@@ -76,6 +76,10 @@ on your phone.
 
 #### Step 2. Modify the Podfile to use the TensorFlow Lite GPU CocoaPod
 
+<section class="zippy">
+
+Until TensorFlow Lite 2.0.0
+
 We have built a binary CocoaPod that includes the GPU delegate. To switch the
 project to use it, modify the
 `tensorflow/tensorflow/lite/examples/ios/camera/Podfile` file to use
@@ -87,7 +91,31 @@ target 'YourProjectName'
   pod 'TensorFlowLiteGpuExperimental'
 ```
 
-#### Step 3. Enable the GPU Delegate
+</section>
+
+From TensorFlow Lite 2.1.0, GPU delegate is inlcuded in the `TensorFlowLiteC`
+pod. You can choose between `TensorFlowLiteC` and `TensorFlowLiteSwift`
+depending on the language.
+
+Note: This behavior will be changed in 2.3.0 and latest nightly releases
+
+For nightly version and upcoming 2.3.0 release, by default GPU delegate is
+excluded from the pod to reduce the binary size. You can include them by
+specifying subspec. For `TensorFlowLiteSwift` pod:
+
+```ruby
+pod 'TensorFlowLiteSwift/Metal', '~> 0.0.1-nightly',
+```
+
+OR
+
+```ruby
+pod 'TensorFlowLiteSwift', '~> 0.0.1-nightly', :subspecs => ['Metal']
+```
+
+You can do similiarly for `TensorFlowLiteC` if you want to use the C API.
+
+#### Step 3. Enable the GPU delegate
 
 To enable the code that will use the GPU delegate, you will need to change
 `TFLITE_USE_GPU_DELEGATE` from 0 to 1 in `CameraExampleViewController.h`.
@@ -100,8 +128,7 @@ To enable the code that will use the GPU delegate, you will need to change
 
 After following the previous step, you should be able to run the app.
 
-
-#### Step 5. Release mode.
+#### Step 5. Release mode
 
 While in Step 4 you ran in debug mode, to get better performance, you should
 change to a release build with the appropriate optimal Metal settings. In
@@ -111,19 +138,18 @@ Scheme...`. Select `Run`. On the `Info` tab, change `Build Configuration`, from
 
 ![setting up release](images/iosdebug.png)
 
-Then
-click the `Options` tab and change `GPU Frame Capture` to `Disabled` and
+Then click the `Options` tab and change `GPU Frame Capture` to `Disabled` and
 `Metal API Validation` to `Disabled`.
 
 ![setting up metal options](images/iosmetal.png)
 
-Lastly make sure Release only builds on 64-bit architecture. Under `Project
-navigator -> tflite_camera_example -> PROJECT -> tflite_camera_example -> Build
-Settings` set `Build Active Architecture Only > Release` to Yes.
+Lastly make sure to select Release-only builds on 64-bit architecture. Under
+`Project navigator -> tflite_camera_example -> PROJECT -> tflite_camera_example
+-> Build Settings` set `Build Active Architecture Only > Release` to Yes.
 
 ![setting up release options](images/iosrelease.png)
 
-## Trying the GPU Delegate on your own model
+## Trying the GPU delegate on your own model
 
 ### Android
 
@@ -197,12 +223,12 @@ To see a full list of supported ops, please see the [advanced documentation](gpu
 ## Non-supported models and ops
 
 If some of the ops are not supported by the GPU delegate, the framework will
-only run a part of the graph on the GPU and the remaining part on the CPU.  Due
+only run a part of the graph on the GPU and the remaining part on the CPU. Due
 to the high cost of CPU/GPU synchronization, a split execution mode like this
-will often result in a performance slower than when the whole network is run on
-the CPU alone.  In this case, the user will get a warning like:
+will often result in slower performance than when the whole network is run on
+the CPU alone. In this case, the user will get a warning like:
 
-```
+```none
 WARNING: op code #42 cannot be handled by this delegate.
 ```
 
@@ -226,6 +252,6 @@ In that sense, if the camera hardware supports image frames in RGBA, feeding
 that 4-channel input is significantly faster as a memory copy (from 3-channel
 RGB to 4-channel RGBX) can be avoided.
 
-For best performance, do not hesitate to retrain your classifier with a mobile-
-optimized network architecture. That is a significant part of optimization for
-on-device inference.
+For best performance, do not hesitate to retrain your classifier with a
+mobile-optimized network architecture. That is a significant part of
+optimization for on-device inference.
diff --git a/tensorflow/lite/g3doc/performance/gpu_advanced.md b/tensorflow/lite/g3doc/performance/gpu_advanced.md
index dce3eb8db6b..c0194627392 100644
--- a/tensorflow/lite/g3doc/performance/gpu_advanced.md
+++ b/tensorflow/lite/g3doc/performance/gpu_advanced.md
@@ -5,7 +5,7 @@ hardware accelerators. This document describes how to use the GPU backend using
 the TensorFlow Lite delegate APIs on Android (requires OpenCL or OpenGL ES 3.1
 and higher) and iOS (requires iOS 8 or later).
 
-## Benefits of GPU Acceleration
+## Benefits of GPU acceleration
 
 ### Speed
 
@@ -24,13 +24,13 @@ GPUs do their computation with 16-bit or 32-bit floating point numbers and
 decreased accuracy made quantization untenable for your models, running your
 neural network on a GPU may eliminate this concern.
 
-### Energy Efficiency
+### Energy efficiency
 
 Another benefit that comes with GPU inference is its power efficiency. A GPU
 carries out computations in a very efficient and optimized way, consuming less
 power and generating less heat than the same task run on a CPU.
 
-## Supported Ops
+## Supported ops
 
 TensorFlow Lite on GPU supports the following ops in 16-bit and 32-bit float
 precision:
@@ -63,12 +63,12 @@ By default, all ops are only supported at version 1. Enabling the
 [experimental quantization support](gpu_advanced.md#running-quantized-models-experimental-android-only)
 allows the appropriate versions; for example, ADD v2.
 
-## Basic Usage
+## Basic usage
 
 ### Android (Java)
 
 Run TensorFlow Lite on GPU with `TfLiteDelegate`. In Java, you can specify the
-GpuDelegate through `Interpreter.Options`.
+`GpuDelegate` through `Interpreter.Options`.
 
 ```java
 // NEW: Prepare GPU delegate.
@@ -167,7 +167,7 @@ then the developer must ensure that `Interpreter::Invoke()` is always called
 from the same thread in which `Interpreter::ModifyGraphWithDelegate()` was
 called.
 
-## Advanced Usage
+## Advanced usage
 
 ### Running quantized models (Experimental, Android only)
 
diff --git a/tensorflow/lite/g3doc/performance/hexagon_delegate.md b/tensorflow/lite/g3doc/performance/hexagon_delegate.md
index b76b4ba9fdf..9d5abe01620 100644
--- a/tensorflow/lite/g3doc/performance/hexagon_delegate.md
+++ b/tensorflow/lite/g3doc/performance/hexagon_delegate.md
@@ -32,9 +32,9 @@ path are also supported, for e.g.,
 [these quantized versions](https://www.tensorflow.org/lite/guide/hosted_models#quantized_models)
 on our Hosted Models page.
 
-## Hexagon Delegate Java API
+## Hexagon delegate Java API
 
-```
+```java
 public class HexagonDelegate implements Delegate, Closeable {
 
   /*
@@ -96,7 +96,7 @@ will need to add the Hexagon shared libs to both 32 and 64-bit lib folders.
 
 #### Step 3. Create a delegate and initialize a TensorFlow Lite Interpreter
 
-```
+```java
 import org.tensorflow.lite.experimental.HexagonDelegate;
 
 // Create the Delegate instance.
@@ -116,9 +116,9 @@ if (hexagonDelegate != null) {
 }
 ```
 
-## Hexagon Delegate C API
+## Hexagon delegate C API
 
-```
+```c
 struct TfLiteHexagonDelegateOptions {
   // This corresponds to the debug level in the Hexagon SDK. 0 (default)
   // means no debug.
@@ -161,7 +161,7 @@ Void TfLiteHexagonInit();
 Void TfLiteHexagonTearDown();
 ```
 
-### Example Usage
+### Example usage
 
 #### Step 1. Edit app/build.gradle to use the nightly Hexagon delegate AAR
 
@@ -213,7 +213,7 @@ will need to add the Hexagon shared libs to both 32 and 64-bit lib folders.
 
 *   Create a delegate, example:
 
-```
+```c
 #include "tensorflow/lite/experimental/delegates/hexagon/hexagon_delegate.h"
 
 // Assuming shared libraries are under "/data/local/tmp/"
diff --git a/tensorflow/lite/g3doc/performance/model_optimization.md b/tensorflow/lite/g3doc/performance/model_optimization.md
index c45aacbb0c8..14ab9c6a0c4 100644
--- a/tensorflow/lite/g3doc/performance/model_optimization.md
+++ b/tensorflow/lite/g3doc/performance/model_optimization.md
@@ -5,8 +5,8 @@ optimizations can be applied to models so that they can be run within these
 constraints. In addition, some optimizations allow the use of specialized
 hardware for accelerated inference.
 
-Tensorflow Lite and the
-[Tensorflow Model Optimization Toolkit](https://www.tensorflow.org/model_optimization)
+TensorFlow Lite and the
+[TensorFlow Model Optimization Toolkit](https://www.tensorflow.org/model_optimization)
 provide tools to minimize the complexity of optimizing inference.
 
 It's recommended that you consider model optimization during your application
@@ -79,9 +79,10 @@ with TensorFlow Lite.
 
 ### Quantization
 
-Quantization works by reducing the precision of the numbers used to represent a
-model's parameters, which by default are 32-bit floating point numbers. This
-results in a smaller model size and faster computation.
+[Quantization](https://www.tensorflow.org/model_optimization/guide/quantization/post_training)
+works by reducing the precision of the numbers used to represent a model's
+parameters, which by default are 32-bit floating point numbers. This results in
+a smaller model size and faster computation.
 
 The following types of quantization are available in TensorFlow Lite:
 
@@ -145,7 +146,7 @@ For cases where the accuracy and latency targets are not met, or hardware
 accelerator support is important,
 [quantization-aware training](https://www.tensorflow.org/model_optimization/guide/quantization/training){:.external}
 is the better option. See additional optimization techniques under the
-[Tensorflow Model Optimization Toolkit](https://www.tensorflow.org/model_optimization).
+[TensorFlow Model Optimization Toolkit](https://www.tensorflow.org/model_optimization).
 
 If you want to further reduce your model size, you can try [pruning](#pruning)
 prior to quantizing your models.
diff --git a/tensorflow/lite/g3doc/performance/nnapi.md b/tensorflow/lite/g3doc/performance/nnapi.md
index 455f28d2fe1..ab3c4b9f88a 100644
--- a/tensorflow/lite/g3doc/performance/nnapi.md
+++ b/tensorflow/lite/g3doc/performance/nnapi.md
@@ -16,9 +16,9 @@ This page describes how to use the NNAPI delegate with the TensorFlow Lite
 Interpreter in Java and Kotlin. For Android C APIs, please refer to
 [Android Native Developer Kit documentation](https://developer.android.com/ndk/guides/neuralnetworks).
 
-## Trying the NNAPI Delegate on your own model
+## Trying the NNAPI delegate on your own model
 
-### Gradle Import
+### Gradle import
 
 The NNAPI delegate is part of the TensorFlow Lite Android interpreter, release
 1.14.0 or higher. You can import it to your project by adding the following to
@@ -69,7 +69,7 @@ if(null != nnApiDelegate) {
 }
 ```
 
-## Best Practices
+## Best practices
 
 ### Test performance before deploying
 
@@ -164,6 +164,6 @@ The following models are known to be compatible with NNAPI:
 NNAPI acceleration is also not supported when the model contains
 dynamically-sized outputs. In this case, you will get a warning like:
 
-```
+```none
 ERROR: Attempting to use a delegate that only supports static-sized tensors with a graph that has dynamic-sized tensors.
 ```
diff --git a/tensorflow/lite/g3doc/performance/post_training_quantization.md b/tensorflow/lite/g3doc/performance/post_training_quantization.md
index a526be75b61..af7d9dbf02d 100644
--- a/tensorflow/lite/g3doc/performance/post_training_quantization.md
+++ b/tensorflow/lite/g3doc/performance/post_training_quantization.md
@@ -2,9 +2,9 @@
 
 Post-training quantization is a conversion technique that can reduce model size
 while also improving CPU and hardware accelerator latency, with little
-degradation in model accuracy. You can perform these techniques using an
-already-trained float TensorFlow model when you convert it to TensorFlow Lite
-format using the [TensorFlow Lite Converter](../convert/).
+degradation in model accuracy. You can quantize an already-trained float
+TensorFlow model when you convert it to TensorFlow Lite format using the
+[TensorFlow Lite Converter](../convert/).
 
 Note: The procedures on this page require TensorFlow 1.15 or higher.
 
@@ -22,8 +22,8 @@ summary table of the choices and the benefits they provide:
 | Float16 quantization | 2x smaller, potential GPU | CPU, GPU         |
 :                      : acceleration              :                  :
 
-This decision tree can help determine which post-training quantization method is
-best for your use case:
+The following decision tree can help determine which post-training quantization
+method is best for your use case:
 
 ![post-training optimization options](images/optimization.jpg)
 
@@ -47,9 +47,9 @@ To further improve latency, "dynamic-range" operators dynamically quantize
 activations based on their range to 8-bits and perform computations with 8-bit
 weights and activations. This optimization provides latencies close to fully
 fixed-point inference. However, the outputs are still stored using floating
-point, so that the speedup with dynamic-range ops is less than a full
-fixed-point computation. Dynamic-range ops are available for the most
-compute-intensive operators in a network:
+point so that the speedup with dynamic-range ops is less than a full fixed-point
+computation. Dynamic-range ops are available for the most compute-intensive
+operators in a network:
 
 *   `tf.keras.layers.Dense`
 *   `tf.keras.layers.Conv2D`
@@ -62,12 +62,12 @@ compute-intensive operators in a network:
 ### Full integer quantization
 
 You can get further latency improvements, reductions in peak memory usage, and
-access to integer only hardware devices or accelerators by making sure all model
-math is integer quantized.
+compatibility with integer only hardware devices or accelerators by making sure
+all model math is integer quantized.
 
-To do this, you need to measure the dynamic range of activations and inputs by
-supplying sample input data to the converter. Refer to the
-`representative_dataset_gen()` function used in the following code.
+For full integer quantization, you need to measure the dynamic range of
+activations and inputs by supplying sample input data to the converter. Refer to
+the `representative_dataset_gen()` function used in the following code.
 
 #### Integer with float fallback (using default float input/output)
 
@@ -87,14 +87,14 @@ converter.representative_dataset = representative_dataset_gen</b>
 tflite_quant_model = converter.convert()
 </pre>
 
-Note: This won't be compatible with integer only devices (such as 8-bit
-microcontrollers) and accelerators (such as the Coral Edge TPU). For convenience
-during inference, the input and output still remain float in order to have the
-same interface as the original float only model.
+Note: This `tflite_quant_model` won't be compatible with integer only devices
+(such as 8-bit microcontrollers) and accelerators (such as the Coral Edge TPU)
+because the input and output still remain float in order to have the same
+interface as the original float only model.
 
 #### Integer only
 
-*This is a common use case for
+*Creating integer only models is a common use case for
 [TensorFlow Lite for Microcontrollers](https://www.tensorflow.org/lite/microcontrollers)
 and [Coral Edge TPUs](https://coral.ai/).*
 
@@ -135,18 +135,18 @@ converter.target_spec.supported_types = [tf.lite.constants.FLOAT16]</b>
 tflite_quant_model = converter.convert()
 </pre>
 
-The advantages of this quantization are as follows:
+The advantages of float16 quantization are as follows:
 
-*   Reduce model size by up to half (since all weights are now half the original
-    size).
-*   Minimal loss in accuracy.
-*   Supports some delegates (e.g. the GPU delegate) can operate directly on
-    float16 data, which results in faster execution than float32 computations.
+*   It reduces model size by up to half (since all weights become half of their
+    original size).
+*   It causes minimal loss in accuracy.
+*   It supports some delegates (e.g. the GPU delegate) which can operate
+    directly on float16 data, resulting in faster execution than float32
+    computations.
 
-The disadvantages of this quantization are as follows:
+The disadvantages of float16 quantization are as follows:
 
-*   Not a good choice for maximum performance (a quantization to fixed point
-    math would be better in that case).
+*   It does not reduce latency as much as a quantization to fixed point math.
 *   By default, a float16 quantized model will "dequantize" the weights values
     to float32 when run on the CPU. (Note that the GPU delegate will not perform
     this dequantization, since it can operate on float16 data.)
diff --git a/tensorflow/lite/g3doc/performance/quantization_spec.md b/tensorflow/lite/g3doc/performance/quantization_spec.md
index e6cc9496f8c..f98f2922fc5 100644
--- a/tensorflow/lite/g3doc/performance/quantization_spec.md
+++ b/tensorflow/lite/g3doc/performance/quantization_spec.md
@@ -27,7 +27,7 @@ values in the range `[-128, 127]`, with a zero-point in range `[-128, 127]`.
 
 There are other exceptions for particular operations that are documented below.
 
-Note: In the past our quantized tooling used per-tensor, asymmetric, `uint8`
+Note: In the past our quantization tooling used per-tensor, asymmetric, `uint8`
 quantization. New tooling, reference kernels, and optimized kernels for 8-bit
 quantization will use this spec.
 
@@ -46,19 +46,19 @@ entire tensor. Per-axis quantization means that there will be one scale and/or
 specifies the dimension of the Tensor's shape that the scales and zero-points
 correspond to. For example, a tensor `t`, with `dims=[4, 3, 2, 1]` with
 quantization params: `scale=[1.0, 2.0, 3.0]`, `zero_point=[1, 2, 3]`,
-`quantization_dimension=1` will be quantized across the second dimension of t:
+`quantization_dimension=1` will be quantized across the second dimension of `t`:
 
     t[:, 0, :, :] will have scale[0]=1.0, zero_point[0]=1
     t[:, 1, :, :] will have scale[1]=2.0, zero_point[1]=2
     t[:, 2, :, :] will have scale[2]=3.0, zero_point[2]=3
 
-Often, the quantized_dimension is the output_channel of the weights of
+Often, the `quantized_dimension` is the `output_channel` of the weights of
 convolutions, but in theory it can be the dimension that corresponds to each
 dot-product in the kernel implementation, allowing more quantization granularity
 without performance implications. This has large improvements to accuracy.
 
 TFLite has per-axis support for a growing number of operations. At the time of
-this document support exists for Conv2d and DepthwiseConv2d.
+this document, support exists for Conv2d and DepthwiseConv2d.
 
 ## Symmetric vs asymmetric
 
@@ -69,7 +69,7 @@ binary bit of precision. Since activations are only multiplied by constant
 weights, the constant zero-point value can be optimized pretty heavily.
 
 Weights are symmetric: forced to have zero-point equal to 0. Weight values are
-multiplied by dynamic input and activation values. This means that there is a
+multiplied by dynamic input and activation values. This means that there is an
 unavoidable runtime cost of multiplying the zero-point of the weight with the
 activation value. By enforcing that zero-point is 0 we can avoid this cost.
 
diff --git a/tensorflow/lite/graph_info.cc b/tensorflow/lite/graph_info.cc
index a419a56a9e6..8968fe6cb21 100644
--- a/tensorflow/lite/graph_info.cc
+++ b/tensorflow/lite/graph_info.cc
@@ -29,6 +29,10 @@ namespace {
 // PartitionGraphIntoIndependentNodeSubsetsImpl partitioner(
 //     info, nodes_to_part, node_subsets);
 // partitioner.Partition();
+//
+// NOTE: Changing the partitioning logic would require a change to
+// FP16GraphPartitionHelper.
+// LINT.IfChange
 class PartitionGraphIntoIndependentNodeSubsetsImpl {
  public:
   PartitionGraphIntoIndependentNodeSubsetsImpl(
@@ -198,6 +202,7 @@ class PartitionGraphIntoIndependentNodeSubsetsImpl {
   // negative values of kEpochNotReady if not assigned.
   std::vector<int> node_epochs_;
 };
+// LINT.ThenChange(//tensorflow/lite/delegates/utils.h)
 
 }  // namespace
 
diff --git a/tensorflow/lite/java/BUILD b/tensorflow/lite/java/BUILD
index 5eb5e8ab023..bfbbe6c3b8c 100644
--- a/tensorflow/lite/java/BUILD
+++ b/tensorflow/lite/java/BUILD
@@ -331,6 +331,7 @@ java_test(
     ],
 )
 
+# portable_tests includes files for running TFLite interpreter tests.
 filegroup(
     name = "portable_tests",
     srcs = [
@@ -342,6 +343,7 @@ filegroup(
     visibility = ["//visibility:public"],
 )
 
+# portable_flex_tests includes files for testing interpreter with Flex delegate.
 filegroup(
     name = "portable_flex_tests",
     srcs = [
@@ -350,11 +352,21 @@ filegroup(
     visibility = ["//visibility:public"],
 )
 
+# portable_test_utils include utilities for loading files and processing images.
+filegroup(
+    name = "portable_test_utils",
+    srcs = [
+        "src/test/java/org/tensorflow/lite/TestUtils.java",
+    ],
+    visibility = ["//visibility:public"],
+)
+
 filegroup(
     name = "portable_gpu_tests",
     srcs = [
         "src/test/java/org/tensorflow/lite/InterpreterTestHelper.java",
         "src/test/java/org/tensorflow/lite/gpu/GpuDelegateTest.java",
+        "src/test/java/org/tensorflow/lite/gpu/WhitelistTest.java",
     ],
     visibility = ["//visibility:public"],
 )
diff --git a/tensorflow/lite/java/src/test/java/org/tensorflow/lite/gpu/WhitelistTest.java b/tensorflow/lite/java/src/test/java/org/tensorflow/lite/gpu/WhitelistTest.java
new file mode 100644
index 00000000000..2c6b2d95f55
--- /dev/null
+++ b/tensorflow/lite/java/src/test/java/org/tensorflow/lite/gpu/WhitelistTest.java
@@ -0,0 +1,34 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+package org.tensorflow.lite.gpu;
+
+import static com.google.common.truth.Truth.assertThat;
+
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.junit.runners.JUnit4;
+
+/** Unit tests for {@link org.tensorflow.lite.gpu.Whitelist}. */
+@RunWith(JUnit4.class)
+public final class WhitelistTest {
+
+  @Test
+  public void testBasic() throws Exception {
+    try (Whitelist whitelist = new Whitelist()) {
+      assertThat(whitelist.isDelegateSupportedOnThisDevice()).isTrue();
+    }
+  }
+}
diff --git a/tensorflow/lite/kernels/BUILD b/tensorflow/lite/kernels/BUILD
index eb62a338a45..aad79ffbc89 100644
--- a/tensorflow/lite/kernels/BUILD
+++ b/tensorflow/lite/kernels/BUILD
@@ -10,24 +10,27 @@ package(
     licenses = ["notice"],  # Apache 2.0
 )
 
-# Enables usage of ruy in TFLite kernels.
+# Enables usage of ruy exclusively as the GEMM backend in TFLite kernels.
+# This will cause TFLite to build with ruy only, providing a smaller binary.
 # WARNING: This build flag is experimental and subject to change.
 config_setting(
-    name = "tflite_with_ruy_explicit_true",
-    define_values = {"tflite_with_ruy": "true"},
+    name = "tflite_with_ruy_only_explicit_true",
+    define_values = {"TFLITE_WITH_RUY_ONLY": "true"},
 )
 
-# Disables usage of ruy in TFLite kernels.
+# Disables usage of ruy as the exclusive GEMM backend in TFLite kernels.
+# TFLite will be built with ruy and other GEMM libraries. Ruy will not be
+# the default GEMM option at runtime.
 # WARNING: This build flag is experimental and subject to change.
 config_setting(
-    name = "tflite_with_ruy_explicit_false",
-    define_values = {"tflite_with_ruy": "false"},
+    name = "tflite_with_ruy_only_explicit_false",
+    define_values = {"TFLITE_WITH_RUY_ONLY": "false"},
 )
 
 ###### Beginning of config_setting's to match aarch64 ######
 #
 # We need to identify the aarch64 instruction set to decide whether to enable
-# tflite_with_ruy by default. This is surprisingly hard to do because select()
+# TFLITE_WITH_RUY_ONLY by default. This is surprisingly hard to do because select()
 # can only consume config_setting's, these config_settings are not centralized,
 # and the "cpu" value which they define are free-form strings and there is no
 # standardization of the strings that we need to match for the aarch64 architecture.
@@ -229,45 +232,45 @@ cc_test(
 )
 
 cc_library(
-    name = "tflite_with_ruy_enabled",
+    name = "tflite_with_ruy_only_enabled",
     build_for_embedded = True,
-    defines = ["TFLITE_WITH_RUY"],
+    defines = ["TFLITE_WITH_RUY_ONLY"],
     visibility = ["//visibility:private"],
 )
 
 cc_library(
-    name = "tflite_with_ruy_and_caching_enabled",
+    name = "tflite_with_ruy_only_and_caching_enabled",
     defines = [
-        "TFLITE_WITH_RUY",
+        "TFLITE_WITH_RUY_ONLY",
         "TFLITE_WITH_RUY_GEMV",
     ],
     visibility = ["//visibility:private"],
 )
 
 cc_library(
-    name = "tflite_with_ruy_default",
+    name = "tflite_with_ruy_only_default",
     build_for_embedded = True,
     select_deps = {
-        ":chromiumos_arm64": [":tflite_with_ruy_enabled"],
-        ":cpu_aarch64": [":tflite_with_ruy_enabled"],
-        ":cpu_arm64": [":tflite_with_ruy_enabled"],
-        ":cpu_arm64e": [":tflite_with_ruy_enabled"],
-        ":cpu_ios_arm64": [":tflite_with_ruy_enabled"],
-        ":cpu_ios_arm64e": [":tflite_with_ruy_enabled"],
-        ":cpu_arm64_v8a": [":tflite_with_ruy_enabled"],
-        "//tensorflow:android_arm": ["tflite_with_ruy_enabled"],
+        ":chromiumos_arm64": [":tflite_with_ruy_only_enabled"],
+        ":cpu_aarch64": [":tflite_with_ruy_only_enabled"],
+        ":cpu_arm64": [":tflite_with_ruy_only_enabled"],
+        ":cpu_arm64e": [":tflite_with_ruy_only_enabled"],
+        ":cpu_ios_arm64": [":tflite_with_ruy_only_enabled"],
+        ":cpu_ios_arm64e": [":tflite_with_ruy_only_enabled"],
+        ":cpu_arm64_v8a": [":tflite_with_ruy_only_enabled"],
+        "//tensorflow:android_arm": ["tflite_with_ruy_only_enabled"],
         "//conditions:default": [],
     },
     visibility = ["//visibility:private"],
 )
 
 cc_library(
-    name = "tflite_with_ruy",
+    name = "tflite_with_ruy_only",
     build_for_embedded = True,
     select_deps = {
-        ":tflite_with_ruy_explicit_true": [":tflite_with_ruy_enabled"],
-        ":tflite_with_ruy_explicit_false": [],
-        "//conditions:default": [":tflite_with_ruy_default"],
+        ":tflite_with_ruy_only_explicit_true": [":tflite_with_ruy_only_enabled"],
+        ":tflite_with_ruy_only_explicit_false": [],
+        "//conditions:default": [":tflite_with_ruy_only_default"],
     },
 )
 
@@ -281,7 +284,7 @@ cc_library(
     ],
     copts = tflite_copts(),
     deps = [
-        ":tflite_with_ruy",
+        ":tflite_with_ruy_only",
         ":op_macros",
         # For now this unconditionally depends on both ruy and gemmlowp.
         # See the comment inside class CpuBackendContext on the
@@ -300,11 +303,11 @@ cc_library(
     copts = tflite_copts(),
     deps = [
         ":cpu_backend_context",
-        ":tflite_with_ruy",
+        ":tflite_with_ruy_only",
         "//tensorflow/lite/kernels/internal:compatibility",
         "//tensorflow/lite/kernels/internal:types",
         # For now this unconditionally depends on both ruy and gemmlowp.
-        # We only need to depend on gemmlowp when tflite_with_ruy
+        # We only need to depend on gemmlowp when tflite_with_ruy_only
         # is false, but putting these dependencies in a select() seems to
         # defeat copybara's rewriting rules.
         "@ruy//ruy:context",
@@ -338,19 +341,19 @@ cc_library(
     ],
     copts = tflite_copts(),
     deps = [
-        ":tflite_with_ruy",
+        ":tflite_with_ruy_only",
         "//tensorflow/lite/kernels/internal:common",
         "//tensorflow/lite/kernels/internal:compatibility",
         "//tensorflow/lite/kernels/internal:types",
         ":cpu_backend_context",
         ":cpu_backend_threadpool",
-        # Depend on ruy regardless of `tflite_with_ruy`. See the comment in
+        # Depend on ruy regardless of `tflite_with_ruy_only`. See the comment in
         # cpu_backend_gemm.h about why ruy is the generic path.
         "@ruy//ruy",
         "@ruy//ruy:matrix",
         "@ruy//ruy:path",
         "@ruy//ruy/profiler:instrumentation",
-        # We only need to depend on gemmlowp and Eigen when tflite_with_ruy
+        # We only need to depend on gemmlowp and Eigen when tflite_with_ruy_only
         # is false, but putting these dependencies in a select() seems to
         # defeat copybara's rewriting rules.
         "@gemmlowp",
@@ -537,8 +540,6 @@ BUILTIN_KERNEL_DEPS = [
     ":lstm_shared",
     ":op_macros",
     ":padding",
-    "@com_google_absl//absl/memory",
-    "@com_google_absl//absl/strings",
     "//third_party/eigen3",
     "@flatbuffers",
     "//tensorflow/lite:framework_lib",
@@ -582,7 +583,7 @@ cc_library(
     ],
     copts = tflite_copts() + tf_opts_nortti_if_android() + EXTRA_EIGEN_COPTS,
     visibility = ["//visibility:private"],
-    deps = BUILTIN_KERNEL_DEPS + ["@farmhash_archive//:farmhash"] + [":tflite_with_ruy_and_caching_enabled"],
+    deps = BUILTIN_KERNEL_DEPS + ["@farmhash_archive//:farmhash"] + [":tflite_with_ruy_only_and_caching_enabled"],
 )
 
 cc_library(
diff --git a/tensorflow/lite/kernels/activations.cc b/tensorflow/lite/kernels/activations.cc
index fc0c461a7c1..749a0d69ef9 100644
--- a/tensorflow/lite/kernels/activations.cc
+++ b/tensorflow/lite/kernels/activations.cc
@@ -60,13 +60,18 @@ struct OpData {
 
 struct SoftmaxOpData {
   struct SoftmaxParams params = {};
-  float table[256]{};
-  const int size_of_lut = 513;
-  int16_t exp_lut[513]{};  // int16 LUT for exp(x), where x uniform distributed
-                           // between [-10.0 , 0.0]
-  int16_t one_over_one_plus_x_lut[513]{};  // int16 LUT for 1 / (1 + x), where
-                                           // x uniform distributed between
-                                           // [0.0 , 1.0]
+  float table[256];
+#ifdef TFLITE_SOFTMAX_USE_UINT16_LUT
+  uint8_t uint8_table1[256];
+  uint8_t uint8_table2[256];
+#endif
+  static constexpr int kInt16LUTArraySize = 513;
+  int16_t exp_lut[kInt16LUTArraySize];  // int16 LUT for exp(x), where x uniform
+                                        // distributed between [-10.0 , 0.0]
+  int16_t one_over_one_plus_x_lut[kInt16LUTArraySize];  // int16 LUT for 1 /
+                                                        // (1 + x), where x
+                                                        // uniform distributed
+                                                        // between [0.0 , 1.0]
 };
 
 struct LogSoftmaxOpData : public OpData {
@@ -134,29 +139,6 @@ void PopulateLookupTable(struct OpData* data, const TfLiteTensor* input,
   }
 }
 
-#if __aarch64__ && __clang__
-namespace {
-// Looks up each element of <indices> in <table>, returns them in a vector.
-inline uint8x16_t aarch64_lookup_vector(const uint8x16x4_t table[4],
-                                        uint8x16_t indices) {
-  // Look up in 1st quarter of the table: top 2 bits of indices == 00
-  uint8x16_t output1 = vqtbl4q_u8(table[0], indices);
-  // Look up in 2nd quarter of the table: top 2 bits of indices == 01
-  uint8x16_t output2 =
-      vqtbl4q_u8(table[1], veorq_u8(indices, vdupq_n_u8(0x40)));
-  // Look up in 3rd quarter of the table: top 2 bits of indices == 10
-  uint8x16_t output3 =
-      vqtbl4q_u8(table[2], veorq_u8(indices, vdupq_n_u8(0x80)));
-  // Look up in 4th quarter of the table: top 2 bits of indices == 11
-  uint8x16_t output4 =
-      vqtbl4q_u8(table[3], veorq_u8(indices, vdupq_n_u8(0xc0)));
-
-  // Combine result of the 4 lookups.
-  return vorrq_u8(vorrq_u8(output1, output2), vorrq_u8(output3, output4));
-}
-}  // namespace
-#endif
-
 // TODO(b/143696793): move this to optimized_ops.
 void EvalUsingLookupTable(struct OpData* data, const TfLiteTensor* input,
                           TfLiteTensor* output) {
@@ -182,7 +164,7 @@ void EvalUsingLookupTable(struct OpData* data, const TfLiteTensor* input,
       size / vectorized_16_loop_step * vectorized_16_loop_step;
   for (; i < vectorized_16_loop_end; i += vectorized_16_loop_step) {
     uint8x16_t input = vld1q_u8(input_data + i);
-    uint8x16_t output = aarch64_lookup_vector(table, input);
+    uint8x16_t output = optimized_ops::aarch64_lookup_vector(table, input);
     vst1q_u8(output_data + i, output);
   }
   // Postamble and non-ARM64 code: simple for loop.
@@ -583,9 +565,26 @@ TfLiteStatus SoftmaxPrepare(TfLiteContext* context, TfLiteNode* node) {
   TF_LITE_ENSURE(context, NumDimensions(input) >= 1);
 
   if (input->type == kTfLiteUInt8 || input->type == kTfLiteInt8) {
-    data->params.table = data->table;
-    optimized_ops::PopulateSoftmaxLookupTable(
-        &data->params, input->params.scale, params->beta);
+    switch (output->type) {
+      case kTfLiteUInt8:
+      case kTfLiteInt8:
+#ifdef TFLITE_SOFTMAX_USE_UINT16_LUT
+        // Only apply when both input & output are uint8/int8 & build with clang
+        // on aarch64.
+        // TODO(b/143709993): Port to ARMv7 and other platforms.
+        data->params.uint8_table1 = data->uint8_table1;
+        data->params.uint8_table2 = data->uint8_table2;
+        optimized_ops::PopulateSoftmaxUInt8LookupTable(
+            &data->params, input->params.scale, params->beta);
+        break;
+#endif
+      case kTfLiteInt16:
+      default:
+        data->params.table = data->table;
+        optimized_ops::PopulateSoftmaxLookupTable(
+            &data->params, input->params.scale, params->beta);
+    }
+
     data->params.zero_point = output->params.zero_point;
     data->params.scale = output->params.scale;
   }
@@ -597,10 +596,10 @@ TfLiteStatus SoftmaxPrepare(TfLiteContext* context, TfLiteNode* node) {
     // exp LUT only used on nagative values
     // we consider exp(-10.0) is insignificant to accumulation
     gen_lut([](double value) { return std::exp(value); }, -10.0, 0.0,
-            data->params.exp_lut, data->size_of_lut);
+            data->params.exp_lut, data->kInt16LUTArraySize);
     data->params.one_over_one_plus_x_lut = data->one_over_one_plus_x_lut;
     gen_lut([](double value) { return 1.0 / (1.0 + value); }, 0.0, 1.0,
-            data->params.one_over_one_plus_x_lut, data->size_of_lut);
+            data->params.one_over_one_plus_x_lut, data->kInt16LUTArraySize);
     data->params.zero_point = output->params.zero_point;
     data->params.scale = output->params.scale;
 
@@ -1019,6 +1018,40 @@ TfLiteStatus SoftmaxQuantized(TfLiteContext* context, const TfLiteTensor* input,
   return kTfLiteOk;
 }
 
+template <>
+TfLiteStatus SoftmaxQuantized<int8_t, int8_t>(TfLiteContext* context,
+                                              const TfLiteTensor* input,
+                                              TfLiteTensor* output,
+                                              SoftmaxOpData* data) {
+#ifdef TFLITE_SOFTMAX_USE_UINT16_LUT
+  optimized_ops::SoftmaxInt8LUT(
+      data->params, GetTensorShape(input), GetTensorData<int8_t>(input),
+      GetTensorShape(output), GetTensorData<int8_t>(output));
+#else
+  optimized_ops::Softmax(data->params, GetTensorShape(input),
+                         GetTensorData<int8_t>(input), GetTensorShape(output),
+                         GetTensorData<int8_t>(output));
+#endif
+  return kTfLiteOk;
+}
+
+template <>
+TfLiteStatus SoftmaxQuantized<uint8_t, uint8_t>(TfLiteContext* context,
+                                                const TfLiteTensor* input,
+                                                TfLiteTensor* output,
+                                                SoftmaxOpData* data) {
+#ifdef TFLITE_SOFTMAX_USE_UINT16_LUT
+  optimized_ops::SoftmaxInt8LUT(
+      data->params, GetTensorShape(input), GetTensorData<uint8_t>(input),
+      GetTensorShape(output), GetTensorData<uint8_t>(output));
+#else
+  optimized_ops::Softmax(data->params, GetTensorShape(input),
+                         GetTensorData<uint8_t>(input), GetTensorShape(output),
+                         GetTensorData<uint8_t>(output));
+#endif
+  return kTfLiteOk;
+}
+
 template <>
 TfLiteStatus SoftmaxQuantized<int16, int16>(TfLiteContext* context,
                                             const TfLiteTensor* input,
diff --git a/tensorflow/lite/kernels/conv.cc b/tensorflow/lite/kernels/conv.cc
index 21ee5f806a8..1d610b2e068 100644
--- a/tensorflow/lite/kernels/conv.cc
+++ b/tensorflow/lite/kernels/conv.cc
@@ -28,7 +28,7 @@ limitations under the License.
 #include "tensorflow/lite/kernels/cpu_backend_context.h"
 #include "tensorflow/lite/kernels/eigen_support.h"
 // b/131835803 forces us to include multithreaded_conv.h before optimized_ops.h
-#ifndef TFLITE_WITH_RUY
+#ifndef TFLITE_WITH_RUY_ONLY
 #include "tensorflow/lite/kernels/internal/optimized/multithreaded_conv.h"
 #endif
 #include "tensorflow/lite/kernels/internal/optimized/optimized_ops.h"
@@ -768,8 +768,8 @@ void EvalFloat(TfLiteContext* context, TfLiteNode* node,
       break;
     }
     case kMultithreadOptimized: {
-#ifdef TFLITE_WITH_RUY
-      // See Register_CONV_2D: we should never be here when tflite_with_ruy
+#ifdef TFLITE_WITH_RUY_ONLY
+      // See Register_CONV_2D: we should never be here when TFLITE_WITH_RUY_ONLY
       // was enabled. We #if out this code in order to get the corresponding
       // binary size benefits.
       TFLITE_DCHECK(false);
@@ -1054,8 +1054,8 @@ TfLiteRegistration* Register_CONVOLUTION_CBLAS_OPT() {
 TfLiteRegistration* Register_CONV_2D() {
 #if defined TFLITE_USE_APPLE_ACCELERATE_FOR_CONV
   return Register_CONVOLUTION_CBLAS_OPT();
-#elif defined TFLITE_WITH_RUY
-  // tflite_with_ruy optimizes the generic kernel type.
+#elif defined TFLITE_WITH_RUY_ONLY
+  // TFLITE_WITH_RUY_ONLY optimizes the generic kernel type.
   return Register_CONVOLUTION_GENERIC_OPT();
 #else
   return Register_CONVOLUTION_MULTITHREADED_OPT();
@@ -1066,8 +1066,8 @@ TfLiteRegistration* Register_CONV_2D() {
 // models only need the UINT8 type. TFLite's op registration mechanism doesn't
 // yet allow for more nuanced registration mechanisms.
 TfLiteRegistration* Register_CONV_2D_UINT8() {
-#if defined TFLITE_WITH_RUY
-  // tflite_with_ruy optimizes the generic kernel type.
+#if defined TFLITE_WITH_RUY_ONLY
+  // TFLITE_WITH_RUY_ONLY optimizes the generic kernel type.
   return Register_CONVOLUTION_GENERIC_OPT_UINT8();
 #else
   return Register_CONV_2D();
diff --git a/tensorflow/lite/kernels/conv_test.cc b/tensorflow/lite/kernels/conv_test.cc
index a2201835195..ef1d5366255 100644
--- a/tensorflow/lite/kernels/conv_test.cc
+++ b/tensorflow/lite/kernels/conv_test.cc
@@ -113,7 +113,8 @@ class BaseConvolutionOpModel : public SingleOpModel {
     resolver_ = absl::make_unique<SingleOpResolver>(BuiltinOperator_CONV_2D,
                                                     registration);
     BuildInterpreter({GetShape(input_), GetShape(filter_), GetShape(bias_)},
-                     num_threads);
+                     num_threads, /*allow_fp32_relax_to_fp16=*/false,
+                     /*apply_delegate=*/true);
   }
 
  protected:
@@ -140,7 +141,7 @@ class ConvolutionOpModel : public BaseConvolutionOpModel<float> {
 const auto kKernelMap = new std::map<string, TfLiteRegistration*>({
     {"Reference", ops::builtin::Register_CONVOLUTION_REF()},
     {"GenericOptimized", ops::builtin::Register_CONVOLUTION_GENERIC_OPT()},
-#ifndef TFLITE_WITH_RUY
+#ifndef TFLITE_WITH_RUY_ONLY
     {"MultithreadedOptimized",
      ops::builtin::Register_CONVOLUTION_MULTITHREADED_OPT()},
 #endif
diff --git a/tensorflow/lite/kernels/cpu_backend_context.cc b/tensorflow/lite/kernels/cpu_backend_context.cc
index d6de9bf8d61..7a16bed0ead 100644
--- a/tensorflow/lite/kernels/cpu_backend_context.cc
+++ b/tensorflow/lite/kernels/cpu_backend_context.cc
@@ -55,6 +55,12 @@ CpuBackendContext::CpuBackendContext()
       ruy_context_(new ruy::Context),
       gemmlowp_context_(new gemmlowp::GemmContext) {
   SetMaxNumThreads(kDefaultNumThreadpoolThreads);
+// TODO(b/148289189) Remove when clients have transitioned to runtime flag.
+#ifdef TFLITE_WITH_RUY_GEMV
+  SetUseCaching(true);
+#else
+  SetUseCaching(false);
+#endif
 }
 
 CpuBackendContext::~CpuBackendContext() {}
@@ -67,4 +73,6 @@ void CpuBackendContext::SetMaxNumThreads(int max_num_threads) {
   gemmlowp_context_->set_max_num_threads(target_num_threads);
 }
 
+void CpuBackendContext::SetUseCaching(bool flag) { use_caching_ = flag; }
+
 }  // namespace tflite
diff --git a/tensorflow/lite/kernels/cpu_backend_context.h b/tensorflow/lite/kernels/cpu_backend_context.h
index 46abcd5e90f..b4973feb56f 100644
--- a/tensorflow/lite/kernels/cpu_backend_context.h
+++ b/tensorflow/lite/kernels/cpu_backend_context.h
@@ -43,6 +43,10 @@ class CpuBackendContext final : public TfLiteInternalBackendContext {
 
   int max_num_threads() const { return max_num_threads_; }
 
+  void SetUseCaching(bool flag);
+
+  bool use_caching() const { return use_caching_; }
+
   void ClearCaches() override { ruy_context_->ClearPrepackedCache(); }
 
  private:
@@ -51,7 +55,7 @@ class CpuBackendContext final : public TfLiteInternalBackendContext {
   // (see :cpu_backend_gemm), for now a CpuBackendContext always
   // stores both a gemmlowp context and a ruy context.
   // TODO(b/131416458): Once call sites all go through abstractions,
-  // elide what can be elided based on TFLITE_WITH_RUY.
+  // elide what can be elided based on TFLITE_WITH_RUY_ONLY.
   const std::unique_ptr<ruy::Context> ruy_context_;
   const std::unique_ptr<gemmlowp::GemmContext> gemmlowp_context_;
 
@@ -65,6 +69,12 @@ class CpuBackendContext final : public TfLiteInternalBackendContext {
   // This value also gets propagated to back-ends, where it plays the same
   // information-only role.
   int max_num_threads_;
+  // For matrix muliplications with constants parameters (i.e. weights), we can
+  // sometimes provide speedups by caching the "prepacked" data, for some
+  // additional memory cost. This flag permits the user to route all
+  // CpuBackendGem operations to a library that permits such an optimization
+  // (currently the Ruy library only).
+  bool use_caching_;
 
   CpuBackendContext(const CpuBackendContext&) = delete;
 };
diff --git a/tensorflow/lite/kernels/cpu_backend_gemm.h b/tensorflow/lite/kernels/cpu_backend_gemm.h
index 16ccc14557f..8e324c8b515 100644
--- a/tensorflow/lite/kernels/cpu_backend_gemm.h
+++ b/tensorflow/lite/kernels/cpu_backend_gemm.h
@@ -23,7 +23,7 @@ limitations under the License.
 #include "tensorflow/lite/kernels/cpu_backend_gemm_params.h"
 #include "tensorflow/lite/kernels/cpu_backend_gemm_ruy.h"
 
-#ifndef TFLITE_WITH_RUY
+#ifndef TFLITE_WITH_RUY_ONLY
 #include "tensorflow/lite/kernels/cpu_backend_gemm_eigen.h"
 #include "tensorflow/lite/kernels/cpu_backend_gemm_gemmlowp.h"
 #endif
@@ -41,7 +41,7 @@ template <typename LhsScalar, typename RhsScalar, typename AccumScalar,
 struct GemmImpl : detail::GemmImplUsingRuy<LhsScalar, RhsScalar, AccumScalar,
                                            DstScalar, quantization_flavor> {};
 
-#ifndef TFLITE_WITH_RUY
+#ifndef TFLITE_WITH_RUY_ONLY
 
 /* Specializations using gemmlowp */
 
@@ -81,7 +81,7 @@ template <>
 struct GemmImpl<float, float, float, float, QuantizationFlavor::kFloatingPoint>
     : detail::GemmImplUsingEigen {};
 
-#endif  // not TFLITE_WITH_RUY
+#endif  // not TFLITE_WITH_RUY_ONLY
 
 /* Public entry point */
 
@@ -94,12 +94,17 @@ void Gemm(const MatrixParams<LhsScalar>& lhs_params, const LhsScalar* lhs_data,
           CpuBackendContext* context) {
   ruy::profiler::ScopeLabel label("cpu_backend_gemm::Gemm");
   ValidateParams(lhs_params, rhs_params, dst_params, params);
-  bool do_custom_gemv = dst_params.cols == 1;
-#ifdef TFLITE_WITH_RUY_GEMV
-  // Prefer a Ruy GEMM to Custom GEMV unless we are doing float math.
-  // TODO(b/148692500): Add float GEMV kernels to Ruy.
-  do_custom_gemv = do_custom_gemv && std::is_floating_point<DstScalar>::value;
-#endif
+  if (context->use_caching()) {
+    // Dispatch to backend that supports caching of prepacked weights
+    // matrices.
+    detail::GemmImplUsingRuy<LhsScalar, RhsScalar, AccumScalar, DstScalar,
+                             quantization_flavor>::Run(lhs_params, lhs_data,
+                                                       rhs_params, rhs_data,
+                                                       dst_params, dst_data,
+                                                       params, context);
+    return;
+  }
+  const bool do_custom_gemv = (dst_params.cols == 1);
   if (do_custom_gemv) {
     // GEMV case: try a custom fast GEMV path.
     if (detail::CustomGemv(lhs_params, lhs_data, rhs_params, rhs_data,
diff --git a/tensorflow/lite/kernels/cpu_backend_gemm_custom_gemv.h b/tensorflow/lite/kernels/cpu_backend_gemm_custom_gemv.h
index f85a1715af2..1c3c0ca39c4 100644
--- a/tensorflow/lite/kernels/cpu_backend_gemm_custom_gemv.h
+++ b/tensorflow/lite/kernels/cpu_backend_gemm_custom_gemv.h
@@ -586,10 +586,10 @@ struct CustomGemvImpl<LhsScalar, RhsScalar, std::int32_t, DstScalar,
 // The float specialization below is unconditionally faster than ruy
 // because ruy does not currently have any Gemv path.
 // But it is not unconditionally faster than Eigen, which is what is used
-// unless TFLITE_WITH_RUY is defined. Indeed, Eigen has decently efficient
+// unless TFLITE_WITH_RUY_ONLY is defined. Indeed, Eigen has decently efficient
 // Gemv paths, and they may use AVX instructions, while the present
 // NEON intrinsics code maps at best to SSE4 on x86.
-#ifdef TFLITE_WITH_RUY
+#ifdef TFLITE_WITH_RUY_ONLY
 
 // We want to use fused multiply-add when it's available (that is, on A64
 // unconditionally and on A32 with VFPv4) because it's often faster, and
@@ -773,7 +773,7 @@ struct CustomGemvImpl<float, float, float, float,
   }
 };
 
-#endif  // TFLITE_WITH_RUY
+#endif  // TFLITE_WITH_RUY_ONLY
 
 #endif  // USE_NEON
 
diff --git a/tensorflow/lite/kernels/cpu_backend_gemm_eigen.cc b/tensorflow/lite/kernels/cpu_backend_gemm_eigen.cc
index ebde7a0b935..334baa5f7ed 100644
--- a/tensorflow/lite/kernels/cpu_backend_gemm_eigen.cc
+++ b/tensorflow/lite/kernels/cpu_backend_gemm_eigen.cc
@@ -13,7 +13,7 @@ See the License for the specific language governing permissions and
 limitations under the License.
 ==============================================================================*/
 
-#ifndef TFLITE_WITH_RUY
+#ifndef TFLITE_WITH_RUY_ONLY
 
 #include "tensorflow/lite/kernels/cpu_backend_gemm_eigen.h"
 
@@ -78,4 +78,4 @@ void GemmImplUsingEigen::Run(
 }  // namespace cpu_backend_gemm
 }  // namespace tflite
 
-#endif  // not TFLITE_WITH_RUY
+#endif  // not TFLITE_WITH_RUY_ONLY
diff --git a/tensorflow/lite/kernels/cpu_backend_gemm_eigen.h b/tensorflow/lite/kernels/cpu_backend_gemm_eigen.h
index bd4733dcfae..3a7fd9df624 100644
--- a/tensorflow/lite/kernels/cpu_backend_gemm_eigen.h
+++ b/tensorflow/lite/kernels/cpu_backend_gemm_eigen.h
@@ -16,7 +16,7 @@ limitations under the License.
 #ifndef TENSORFLOW_LITE_KERNELS_CPU_BACKEND_GEMM_EIGEN_H_
 #define TENSORFLOW_LITE_KERNELS_CPU_BACKEND_GEMM_EIGEN_H_
 
-#ifndef TFLITE_WITH_RUY
+#ifndef TFLITE_WITH_RUY_ONLY
 
 #include "tensorflow/lite/kernels/cpu_backend_context.h"
 #include "tensorflow/lite/kernels/cpu_backend_gemm_params.h"
@@ -37,6 +37,6 @@ struct GemmImplUsingEigen {
 }  // namespace cpu_backend_gemm
 }  // namespace tflite
 
-#endif  // not TFLITE_WITH_RUY
+#endif  // not TFLITE_WITH_RUY_ONLY
 
 #endif  // TENSORFLOW_LITE_KERNELS_CPU_BACKEND_GEMM_EIGEN_H_
diff --git a/tensorflow/lite/kernels/cpu_backend_gemm_gemmlowp.h b/tensorflow/lite/kernels/cpu_backend_gemm_gemmlowp.h
index ad9bbb75ae5..1e1074523ab 100644
--- a/tensorflow/lite/kernels/cpu_backend_gemm_gemmlowp.h
+++ b/tensorflow/lite/kernels/cpu_backend_gemm_gemmlowp.h
@@ -16,7 +16,7 @@ limitations under the License.
 #ifndef TENSORFLOW_LITE_KERNELS_CPU_BACKEND_GEMM_GEMMLOWP_H_
 #define TENSORFLOW_LITE_KERNELS_CPU_BACKEND_GEMM_GEMMLOWP_H_
 
-#ifndef TFLITE_WITH_RUY
+#ifndef TFLITE_WITH_RUY_ONLY
 
 #include <cstdint>
 #include <type_traits>
@@ -188,6 +188,6 @@ struct GemmImplUsingGemmlowp<LhsScalar, RhsScalar, AccumScalar, DstScalar,
 }  // namespace cpu_backend_gemm
 }  // namespace tflite
 
-#endif  // not TFLITE_WITH_RUY
+#endif  // not TFLITE_WITH_RUY_ONLY
 
 #endif  // TENSORFLOW_LITE_KERNELS_CPU_BACKEND_GEMM_GEMMLOWP_H_
diff --git a/tensorflow/lite/kernels/cpu_backend_gemm_ruy.h b/tensorflow/lite/kernels/cpu_backend_gemm_ruy.h
index b441628a67b..8aaedb6a13a 100644
--- a/tensorflow/lite/kernels/cpu_backend_gemm_ruy.h
+++ b/tensorflow/lite/kernels/cpu_backend_gemm_ruy.h
@@ -42,7 +42,7 @@ inline ruy::CachePolicy ToRuyCachePolicy(CachePolicy cache_policy) {
 
 template <typename Scalar, typename DataPointer>
 void MakeRuyMatrix(const MatrixParams<Scalar>& params, DataPointer data_ptr,
-                   ruy::Matrix<Scalar>* dst) {
+                   ruy::Matrix<Scalar>* dst, bool use_caching = false) {
   ruy::Order ruy_order = params.order == Order::kColMajor
                              ? ruy::Order::kColMajor
                              : ruy::Order::kRowMajor;
@@ -52,9 +52,9 @@ void MakeRuyMatrix(const MatrixParams<Scalar>& params, DataPointer data_ptr,
   // It does care whether we assign to it a Scalar* or a const Scalar*.
   dst->set_data(data_ptr);
   dst->set_zero_point(params.zero_point);
-#ifdef TFLITE_WITH_RUY_GEMV
-  dst->set_cache_policy(ToRuyCachePolicy(params.cache_policy));
-#endif
+  if (use_caching) {
+    dst->set_cache_policy(ToRuyCachePolicy(params.cache_policy));
+  }
 }
 
 template <typename GemmParamsType, typename RuySpecType>
@@ -88,8 +88,8 @@ struct GemmImplUsingRuy {
     ruy::Matrix<LhsScalar> ruy_lhs;
     ruy::Matrix<RhsScalar> ruy_rhs;
     ruy::Matrix<DstScalar> ruy_dst;
-    MakeRuyMatrix(lhs_params, lhs_data, &ruy_lhs);
-    MakeRuyMatrix(rhs_params, rhs_data, &ruy_rhs);
+    MakeRuyMatrix(lhs_params, lhs_data, &ruy_lhs, context->use_caching());
+    MakeRuyMatrix(rhs_params, rhs_data, &ruy_rhs, context->use_caching());
     MakeRuyMatrix(dst_params, dst_data, &ruy_dst);
 
     ruy::MulParams<AccumScalar, DstScalar> ruy_mul_params;
diff --git a/tensorflow/lite/kernels/cpu_backend_gemm_test.cc b/tensorflow/lite/kernels/cpu_backend_gemm_test.cc
index 110eb3a07ef..20334947dde 100644
--- a/tensorflow/lite/kernels/cpu_backend_gemm_test.cc
+++ b/tensorflow/lite/kernels/cpu_backend_gemm_test.cc
@@ -363,7 +363,8 @@ void TestSomeGemm(int rows, int depth, int cols,
   CpuBackendContext cpu_backend_context;
   std::default_random_engine random_engine;
   cpu_backend_context.SetMaxNumThreads(1 + (random_engine() % 8));
-
+  bool use_caching = static_cast<bool>(random_engine() % 2);
+  cpu_backend_context.SetUseCaching(use_caching);
   const bool use_golden = !golden.empty();
 
   std::vector<LhsScalar> lhs_data;
diff --git a/tensorflow/lite/kernels/cpu_backend_threadpool.h b/tensorflow/lite/kernels/cpu_backend_threadpool.h
index 39eafd51d6a..60a5ebfde29 100644
--- a/tensorflow/lite/kernels/cpu_backend_threadpool.h
+++ b/tensorflow/lite/kernels/cpu_backend_threadpool.h
@@ -19,7 +19,7 @@ limitations under the License.
 #include "tensorflow/lite/kernels/cpu_backend_context.h"
 #include "tensorflow/lite/kernels/internal/compatibility.h"
 
-#ifdef TFLITE_WITH_RUY
+#ifdef TFLITE_WITH_RUY_ONLY
 #include "ruy/context.h"  // from @ruy
 #include "ruy/thread_pool.h"  // from @ruy
 #else
@@ -29,7 +29,7 @@ limitations under the License.
 namespace tflite {
 namespace cpu_backend_threadpool {
 
-#ifdef TFLITE_WITH_RUY
+#ifdef TFLITE_WITH_RUY_ONLY
 
 using Task = ruy::Task;
 
@@ -41,7 +41,7 @@ void Execute(int tasks_count, TaskType* tasks,
       tasks_count, tasks);
 }
 
-#else  // not TFLITE_WITH_RUY
+#else  // not TFLITE_WITH_RUY_ONLY
 
 using Task = gemmlowp::Task;
 
diff --git a/tensorflow/lite/kernels/internal/optimized/depthwiseconv_multithread.h b/tensorflow/lite/kernels/internal/optimized/depthwiseconv_multithread.h
index 7d8838a076e..0e13222b28a 100644
--- a/tensorflow/lite/kernels/internal/optimized/depthwiseconv_multithread.h
+++ b/tensorflow/lite/kernels/internal/optimized/depthwiseconv_multithread.h
@@ -132,7 +132,7 @@ inline void DepthwiseConv(const DepthwiseParams& params,
   int thread_count = HowManyConvThreads(output_shape, filter_shape);
   const int max_threads = cpu_backend_context->max_num_threads();
   thread_count = std::max(1, std::min(thread_count, max_threads));
-#ifndef TFLITE_WITH_RUY
+#ifndef TFLITE_WITH_RUY_ONLY
   // Cap the number of threads to 2 for float path to avoid regression in
   // performance (b/132294857).
   if (std::is_floating_point<T>::value) {
diff --git a/tensorflow/lite/kernels/internal/optimized/optimized_ops.h b/tensorflow/lite/kernels/internal/optimized/optimized_ops.h
index 59d0ba7bf6f..528eea3d698 100644
--- a/tensorflow/lite/kernels/internal/optimized/optimized_ops.h
+++ b/tensorflow/lite/kernels/internal/optimized/optimized_ops.h
@@ -56,6 +56,10 @@ limitations under the License.
 #include "tensorflow/lite/kernels/internal/transpose_utils.h"
 #include "tensorflow/lite/kernels/internal/types.h"
 
+#if __aarch64__ && __clang__
+#define TFLITE_SOFTMAX_USE_UINT16_LUT
+#endif
+
 namespace tflite {
 namespace optimized_ops {
 
@@ -328,6 +332,29 @@ inline void BinaryBroadcastFiveFold(const ArithmeticParams& unswitched_params,
   }
 }
 
+#ifdef TFLITE_SOFTMAX_USE_UINT16_LUT
+
+// Looks up each element of <indices> in <table>, returns them in a vector.
+inline uint8x16_t aarch64_lookup_vector(const uint8x16x4_t table[4],
+                                        uint8x16_t indices) {
+  // Look up in 1st quarter of the table: top 2 bits of indices == 00
+  uint8x16_t output1 = vqtbl4q_u8(table[0], indices);
+  // Look up in 2nd quarter of the table: top 2 bits of indices == 01
+  uint8x16_t output2 =
+      vqtbl4q_u8(table[1], veorq_u8(indices, vdupq_n_u8(0x40)));
+  // Look up in 3rd quarter of the table: top 2 bits of indices == 10
+  uint8x16_t output3 =
+      vqtbl4q_u8(table[2], veorq_u8(indices, vdupq_n_u8(0x80)));
+  // Look up in 4th quarter of the table: top 2 bits of indices == 11
+  uint8x16_t output4 =
+      vqtbl4q_u8(table[3], veorq_u8(indices, vdupq_n_u8(0xc0)));
+
+  // Combine result of the 4 lookups.
+  return vorrq_u8(vorrq_u8(output1, output2), vorrq_u8(output3, output4));
+}
+
+#endif
+
 inline void AddBiasAndEvalActivationFunction(float output_activation_min,
                                              float output_activation_max,
                                              const RuntimeShape& bias_shape,
@@ -3969,6 +3996,271 @@ inline void Softmax(const SoftmaxParams& params,
   }
 }
 
+// Here's the softmax LUT optimization strategy:
+// For softmax, we can do some mathmetically equivalent transformation:
+//
+// softmax(x) = e^x / sum(e^x, 0...n)  ===> equals to
+// softmax(x) = e^(x - CONST) / sum(e^(x - CONST), 0...n)
+//
+// For quantization, `x` in our case is (input_q - input_zp) * input_s
+// For uint8 case (int8 can be handled similarly), the range is [0, 255]
+//
+// so if we let
+// CONST = (255 - input_zp) * input_s
+// then we will have:
+// softmax(x) = e^((input_q - 255) * input_s) --------- (1)
+//         /
+// sum(e^(input_q - 255) * input_s, 0...n)   -------- (2)
+//
+// the good thing about (1) is it's within the range of (0, 1), so we can
+// approximate its result with uint16.
+//  (1) = uint8_out * 1 / 2^16.
+//
+// so (1) is lookup_uint8_table(input_zp) * 1 / 2^16.
+// then (2) is essentially the following:
+// sum(lookup_uint8_table(input_zp), 0...n) / 2^16.
+//
+// since (output_q - output_zp) * output_s = softmax(x)
+// output_q = lookup_uint8_table(input_zp)
+//            /
+// (sum(lookup_uint8_table(input_zp), 0...n) * output_s)
+//             +
+//   output_zp
+//
+// We can actually further improve the performance by using uint8 instead of
+// uint16. But that we may lose some accuracy, so we need to pay attention
+// to that.
+inline void PopulateSoftmaxUInt8LookupTable(SoftmaxParams* data,
+                                            float input_scale, float beta) {
+  const float scale = input_scale * beta;
+  const int32_t max_uint8 = std::numeric_limits<uint8_t>::max();
+  const int32_t max_uint16 = std::numeric_limits<uint16_t>::max();
+
+  for (int32_t val = 0; val <= max_uint8; ++val) {
+    float input_to_exp = scale * (val - max_uint8);
+    int32_t temp = static_cast<int>(expf(input_to_exp) * max_uint16 + 0.5);
+    temp = std::min(max_uint16, temp);
+    uint8_t part1 = temp >> 8;
+    uint8_t part2 = temp & 0xff;
+    data->uint8_table1[val] = static_cast<uint8_t>(part1);
+    data->uint8_table2[val] = static_cast<uint8_t>(part2);
+  }
+}
+
+inline int FindMaxValue(int size, const uint8_t* input_data, uint8_t offset) {
+  int32_t max_val = std::numeric_limits<uint8_t>::min();
+  int j = 0;
+#ifdef TFLITE_SOFTMAX_USE_UINT16_LUT
+  uint8x16_t max_val_dup = vdupq_n_u8(max_val);
+  uint8x16_t offset_dup = vdupq_n_u8(offset);
+  for (; j <= size - 16; j += 16) {
+    uint8x16_t input_value = vld1q_u8(input_data + j);
+    input_value = veorq_u8(input_value, offset_dup);
+    max_val_dup = vmaxq_u8(input_value, max_val_dup);
+  }
+  max_val = std::max(max_val, static_cast<int32>(vmaxvq_u8(max_val_dup)));
+#endif
+
+  for (; j < size; ++j) {
+    max_val = std::max(max_val, static_cast<int32_t>(input_data[j] ^ offset));
+  }
+  return max_val;
+}
+
+#ifdef USE_NEON
+// Value_to_store layout:
+// [high_high, high_low, low_high, low_low].
+inline void StoreValue(int32x4x4_t value_to_store, int8_t* output) {
+  const int16x8_t result_1 = vcombine_s16(vqmovn_s32(value_to_store.val[1]),
+                                          vqmovn_s32(value_to_store.val[0]));
+  const int16x8_t result_2 = vcombine_s16(vqmovn_s32(value_to_store.val[3]),
+                                          vqmovn_s32(value_to_store.val[2]));
+  const int8x16_t result =
+      vcombine_s8(vqmovn_s16(result_2), vqmovn_s16(result_1));
+  vst1q_s8(output, result);
+}
+
+// Value_to_store layout:
+// [high_high, high_low, low_high, low_low].
+inline void StoreValue(int32x4x4_t value_to_store, uint8_t* output) {
+  const uint16x8_t result_1 =
+      vcombine_u16(vqmovn_u32(vreinterpretq_u32_s32(value_to_store.val[1])),
+                   vqmovn_u32(vreinterpretq_u32_s32(value_to_store.val[0])));
+  const uint16x8_t result_2 =
+      vcombine_u16(vqmovn_u32(vreinterpretq_u32_s32(value_to_store.val[3])),
+                   vqmovn_u32(vreinterpretq_u32_s32(value_to_store.val[2])));
+  const uint8x16_t result =
+      vcombine_u8(vqmovn_u16(result_2), vqmovn_u16(result_1));
+  vst1q_u8(output, result);
+}
+
+#endif
+
+template <typename In, typename Out>
+inline void SoftmaxInt8LUT(const SoftmaxParams& params,
+                           const RuntimeShape& input_shape,
+                           const In* input_data,
+                           const RuntimeShape& output_shape, Out* output_data) {
+  const int trailing_dim = input_shape.DimensionsCount() - 1;
+  const int excluding_last_dim =
+      MatchingFlatSizeSkipDim(input_shape, trailing_dim, output_shape);
+  const int last_dim =
+      MatchingDim(input_shape, trailing_dim, output_shape, trailing_dim);
+
+  const int32_t clamp_max = std::numeric_limits<Out>::max();
+  const int32_t clamp_min = std::numeric_limits<Out>::min();
+
+  // Offset is used to interpret the input data "correctly".
+  // If the input is uint8, the data will be unchanged.
+  // If the input is int8, since it will be reinterpret as uint8.
+  // e.g.,
+  // int8 127 will be applied "offset" to become 255 in uint8.
+  uint8_t offset = 0;
+  if (std::is_same<In, int8>::value) {
+    offset = 0x80;
+  }
+
+  const uint8_t* input_data_uint = reinterpret_cast<const uint8_t*>(input_data);
+
+#ifdef TFLITE_SOFTMAX_USE_UINT16_LUT
+  // This code uses ARM64-only instructions.
+  // TODO(b/143709993): Port to ARMv7
+
+  // Load the tables into registers. (4*4 128-bit registers)
+  uint8x16x4_t table1[4];
+  table1[0] = vld1q_u8_x4(params.uint8_table1 + 16 * 4 * 0);
+  table1[1] = vld1q_u8_x4(params.uint8_table1 + 16 * 4 * 1);
+  table1[2] = vld1q_u8_x4(params.uint8_table1 + 16 * 4 * 2);
+  table1[3] = vld1q_u8_x4(params.uint8_table1 + 16 * 4 * 3);
+
+  uint8x16x4_t table2[4];
+  table2[0] = vld1q_u8_x4(params.uint8_table2 + 16 * 4 * 0);
+  table2[1] = vld1q_u8_x4(params.uint8_table2 + 16 * 4 * 1);
+  table2[2] = vld1q_u8_x4(params.uint8_table2 + 16 * 4 * 2);
+  table2[3] = vld1q_u8_x4(params.uint8_table2 + 16 * 4 * 3);
+#endif
+
+  for (int i = 0; i < excluding_last_dim; ++i) {
+    // Find max quantized value.
+    int32_t max_val = FindMaxValue(last_dim, input_data_uint, offset);
+
+    int32 sum_exp = 0;
+    const int32_t max_uint8 = std::numeric_limits<uint8_t>::max();
+    const uint8_t table_offset = max_uint8 - max_val;
+
+    // Calculate normalizer sum(exp(x)).
+    int sum_j = 0;
+#ifdef TFLITE_SOFTMAX_USE_UINT16_LUT
+    uint8x16_t table_offset_dup = vdupq_n_u8(table_offset);
+    uint8x16_t offset_dup = vdupq_n_u8(offset);
+    uint32x4_t sum_4 = vdupq_n_u32(0);
+    const int multiplier_shift = 8;
+    for (; sum_j <= last_dim - 16; sum_j += 16) {
+      uint8x16_t input_value = vld1q_u8(input_data_uint + sum_j);
+      input_value = veorq_u8(input_value, offset_dup);
+      input_value = vaddq_u8(input_value, table_offset_dup);
+
+      const uint8x16_t output1 = aarch64_lookup_vector(table1, input_value);
+      const uint8x16_t output2 = aarch64_lookup_vector(table2, input_value);
+
+      uint16x8_t exp_value1 =
+          vshll_n_u8(vget_high_u8(output1), multiplier_shift);
+      uint16x8_t exp_value2 =
+          vshll_n_u8(vget_low_u8(output1), multiplier_shift);
+
+      exp_value1 = vaddw_u8(exp_value1, vget_high_u8(output2));
+      exp_value2 = vaddw_u8(exp_value2, vget_low_u8(output2));
+
+      sum_4 = vpadalq_u16(sum_4, exp_value1);
+      sum_4 = vpadalq_u16(sum_4, exp_value2);
+    }
+    int temp = vgetq_lane_u32(sum_4, 0) + vgetq_lane_u32(sum_4, 1) +
+               vgetq_lane_u32(sum_4, 2) + vgetq_lane_u32(sum_4, 3);
+    sum_exp += temp;
+
+#endif
+    for (; sum_j < last_dim; ++sum_j) {
+      const uint8_t index = (input_data_uint[sum_j] ^ offset) + table_offset;
+
+      uint8_t part1 = params.uint8_table1[index];
+      uint8_t part2 = params.uint8_table2[index];
+      sum_exp += ((part1 << 8) + part2);
+    }
+
+    const float inv_sum_exp = 1.0f / (sum_exp * params.scale);
+
+    int32 multiplier, shift;
+    QuantizeMultiplier(inv_sum_exp, &multiplier, &shift);
+
+    // Normalize and quantize probabilities.
+    int j = 0;
+#ifdef TFLITE_SOFTMAX_USE_UINT16_LUT
+    const int32x4_t output_zp_dup = vdupq_n_s32(params.zero_point);
+    const int32x4_t max_val_dup = vdupq_n_s32(clamp_max);
+    const int32x4_t min_val_dup = vdupq_n_s32(clamp_min);
+
+    for (; j <= last_dim - 16; j += 16) {
+      uint8x16_t input_value = vld1q_u8(input_data_uint + j);
+      input_value = veorq_u8(input_value, offset_dup);
+      input_value = vaddq_u8(input_value, table_offset_dup);
+
+      const uint8x16_t output1 = aarch64_lookup_vector(table1, input_value);
+      const uint8x16_t output2 = aarch64_lookup_vector(table2, input_value);
+
+      uint16x8_t exp_value1 =
+          vshll_n_u8(vget_high_u8(output1), multiplier_shift);
+      uint16x8_t exp_value2 =
+          vshll_n_u8(vget_low_u8(output1), multiplier_shift);
+
+      exp_value1 = vaddw_u8(exp_value1, vget_high_u8(output2));
+      exp_value2 = vaddw_u8(exp_value2, vget_low_u8(output2));
+
+      int32x4x4_t output_value;
+      output_value.val[0] =
+          vreinterpretq_s32_u32(vmovl_u16(vget_high_u16(exp_value1)));
+      output_value.val[1] =
+          vreinterpretq_s32_u32(vmovl_u16(vget_low_u16(exp_value1)));
+      output_value.val[2] =
+          vreinterpretq_s32_u32(vmovl_u16(vget_high_u16(exp_value2)));
+      output_value.val[3] =
+          vreinterpretq_s32_u32(vmovl_u16(vget_low_u16(exp_value2)));
+
+      int32x4x4_t temp_val =
+          MultiplyByQuantizedMultiplier4Rows(output_value, multiplier, shift);
+
+      temp_val.val[0] = vaddq_s32(temp_val.val[0], output_zp_dup);
+      temp_val.val[1] = vaddq_s32(temp_val.val[1], output_zp_dup);
+      temp_val.val[2] = vaddq_s32(temp_val.val[2], output_zp_dup);
+      temp_val.val[3] = vaddq_s32(temp_val.val[3], output_zp_dup);
+
+      temp_val.val[0] =
+          vmaxq_s32(vminq_s32(temp_val.val[0], max_val_dup), min_val_dup);
+      temp_val.val[1] =
+          vmaxq_s32(vminq_s32(temp_val.val[1], max_val_dup), min_val_dup);
+      temp_val.val[2] =
+          vmaxq_s32(vminq_s32(temp_val.val[2], max_val_dup), min_val_dup);
+      temp_val.val[3] =
+          vmaxq_s32(vminq_s32(temp_val.val[3], max_val_dup), min_val_dup);
+
+      StoreValue(temp_val, output_data + j);
+    }
+#endif
+    for (; j < last_dim; ++j) {
+      const uint8_t index = (input_data_uint[j] ^ offset) + table_offset;
+      const uint8_t part1 = params.uint8_table1[index];
+      const uint8_t part2 = params.uint8_table2[index];
+      const int32_t exp_value = (part1 << 8) + part2;
+      const int32_t output_value =
+          MultiplyByQuantizedMultiplier(exp_value, multiplier, shift);
+
+      output_data[j] = static_cast<Out>(std::max(
+          std::min(clamp_max, output_value + params.zero_point), clamp_min));
+    }
+    input_data_uint += last_dim;
+    output_data += last_dim;
+  }
+}
+
 // TODO(myenik): This is the same as the reference implementation, not actually
 // optimized yet.
 inline void LogSoftmax(const SoftmaxParams& params,
diff --git a/tensorflow/lite/kernels/internal/reference/integer_ops/transpose_conv.h b/tensorflow/lite/kernels/internal/reference/integer_ops/transpose_conv.h
index 422adc2a333..f28b7cbddb7 100644
--- a/tensorflow/lite/kernels/internal/reference/integer_ops/transpose_conv.h
+++ b/tensorflow/lite/kernels/internal/reference/integer_ops/transpose_conv.h
@@ -119,6 +119,102 @@ inline void TransposeConv(
   }
 }
 
+// int16 input (zero_point=0), int8 filter, int64 accumulator
+inline void TransposeConv(
+    const ConvParams& params, const int32* output_multiplier,
+    const int32* output_shift, const RuntimeShape& input_shape,
+    const int16* input_data, const RuntimeShape& filter_shape,
+    const int8* filter_data, const RuntimeShape& bias_shape,
+    const std::int64_t* bias_data, const RuntimeShape& output_shape,
+    int16* output_data, const RuntimeShape& im2col_shape, int8* im2col_data,
+    std::int64_t* scratch_buffer) {
+  const int stride_width = params.stride_width;
+  const int stride_height = params.stride_height;
+  const int pad_width = params.padding_values.width;
+  const int pad_height = params.padding_values.height;
+  TFLITE_DCHECK_EQ(input_shape.DimensionsCount(), 4);
+  TFLITE_DCHECK_EQ(filter_shape.DimensionsCount(), 4);
+  TFLITE_DCHECK_EQ(output_shape.DimensionsCount(), 4);
+  (void)im2col_data;   // only used in optimized code.
+  (void)im2col_shape;  // only used in optimized code.
+
+  const int batches = MatchingDim(input_shape, 0, output_shape, 0);
+  const int input_depth = MatchingDim(input_shape, 3, filter_shape, 3);
+  const int output_depth = MatchingDim(filter_shape, 0, output_shape, 3);
+  if (bias_data) {
+    TFLITE_DCHECK_EQ(bias_shape.FlatSize(), output_depth);
+  }
+  const int input_height = input_shape.Dims(1);
+  const int input_width = input_shape.Dims(2);
+  const int filter_height = filter_shape.Dims(1);
+  const int filter_width = filter_shape.Dims(2);
+  const int output_height = output_shape.Dims(1);
+  const int output_width = output_shape.Dims(2);
+  const int32 output_activation_min = std::numeric_limits<int16_t>::min();
+  const int32 output_activation_max = std::numeric_limits<int16_t>::max();
+  TFLITE_DCHECK_LE(output_activation_min, output_activation_max);
+
+  const int num_elements = output_shape.FlatSize();
+  // We need to initialize scratch_buffer to all 0s, as we apply the same
+  // 'scatter' based trick as in float version.
+  memset(scratch_buffer, 0, num_elements * sizeof(std::int64_t));
+
+  // Loop through input elements one at a time.
+  for (int batch = 0; batch < batches; ++batch) {
+    for (int in_y = 0; in_y < input_height; ++in_y) {
+      for (int in_x = 0; in_x < input_width; ++in_x) {
+        for (int in_channel = 0; in_channel < input_depth; ++in_channel) {
+          // Loop through the output elements it will influence.
+          const int out_x_origin = (in_x * stride_width) - pad_width;
+          const int out_y_origin = (in_y * stride_height) - pad_height;
+          for (int filter_y = 0; filter_y < filter_height; ++filter_y) {
+            for (int filter_x = 0; filter_x < filter_width; ++filter_x) {
+              for (int out_channel = 0; out_channel < output_depth;
+                   ++out_channel) {
+                // Compute output element location.
+                const int out_x = out_x_origin + filter_x;
+                const int out_y = out_y_origin + filter_y;
+                // We cannot accumulate out of bounds.
+                if ((out_x >= 0) && (out_x < output_width) && (out_y >= 0) &&
+                    (out_y < output_height)) {
+                  const int32 input_value = input_data[Offset(
+                      input_shape, batch, in_y, in_x, in_channel)];
+                  const int32 filter_value =
+                      filter_data[Offset(filter_shape, out_channel, filter_y,
+                                         filter_x, in_channel)];
+                  scratch_buffer[Offset(output_shape, batch, out_y, out_x,
+                                        out_channel)] +=
+                      input_value * filter_value;
+                }
+              }
+            }
+          }
+        }
+      }
+    }
+  }
+
+  for (int batch = 0; batch < batches; ++batch) {
+    for (int out_y = 0; out_y < output_height; ++out_y) {
+      for (int out_x = 0; out_x < output_width; ++out_x) {
+        for (int out_channel = 0; out_channel < output_depth; ++out_channel) {
+          std::int64_t acc = scratch_buffer[Offset(output_shape, batch, out_y,
+                                                   out_x, out_channel)];
+          if (bias_data) {
+            acc += bias_data[out_channel];
+          }
+          int32 scaled_acc = MultiplyByQuantizedMultiplier(
+              acc, output_multiplier[out_channel], output_shift[out_channel]);
+          scaled_acc = std::max(scaled_acc, output_activation_min);
+          scaled_acc = std::min(scaled_acc, output_activation_max);
+          output_data[Offset(output_shape, batch, out_y, out_x, out_channel)] =
+              static_cast<int16_t>(scaled_acc);
+        }
+      }
+    }
+  }
+}
+
 }  // namespace reference_integer_ops
 }  // namespace tflite
 
diff --git a/tensorflow/lite/kernels/internal/softmax_quantized_test.cc b/tensorflow/lite/kernels/internal/softmax_quantized_test.cc
index 4f9c6471a1c..9b5ef171eaf 100644
--- a/tensorflow/lite/kernels/internal/softmax_quantized_test.cc
+++ b/tensorflow/lite/kernels/internal/softmax_quantized_test.cc
@@ -133,6 +133,7 @@ void RunOneSoftmaxTest(const uint8* input_data,
   params.scale = 1.0f / 256;
   params.zero_point = 0;
   params.table = table;
+
   optimized_ops::PopulateSoftmaxLookupTable(&params, input_scale, beta);
   optimized_ops::Softmax(params, shape_common, input_data, shape_common,
                          optimized_softmax_output.data());
@@ -148,6 +149,19 @@ void RunOneSoftmaxTest(const uint8* input_data,
   CheckOutputData<uint8_t>(reference_quant_softmax_output.data(),
                            reference_float_softmax_output.data(), shape_common,
                            "Quant reference vs float reference", false);
+
+#if __aarch64__ && __clang__
+  uint8_t uint8_table1[256];
+  uint8_t uint8_table2[256];
+  params.uint8_table1 = uint8_table1;
+  params.uint8_table2 = uint8_table2;
+  optimized_ops::PopulateSoftmaxUInt8LookupTable(&params, input_scale, beta);
+  optimized_ops::SoftmaxInt8LUT(params, shape_common, input_data, shape_common,
+                                optimized_softmax_output.data());
+  CheckOutputData<uint8_t>(
+      optimized_softmax_output.data(), reference_quant_softmax_output.data(),
+      shape_common, "Optimized (Uint8 Lookup table) vs quant reference", false);
+#endif
 }
 
 // This function picks some random Softmax params, which are checked for
diff --git a/tensorflow/lite/kernels/internal/types.h b/tensorflow/lite/kernels/internal/types.h
index 52d74d1eca4..2a34f6608a3 100644
--- a/tensorflow/lite/kernels/internal/types.h
+++ b/tensorflow/lite/kernels/internal/types.h
@@ -1035,6 +1035,8 @@ struct SoftmaxParams {
   float* table;
   int16_t* exp_lut;
   int16_t* one_over_one_plus_x_lut;
+  uint8_t* uint8_table1;
+  uint8_t* uint8_table2;
 };
 
 struct SpaceToBatchParams {
diff --git a/tensorflow/lite/kernels/lstm.cc b/tensorflow/lite/kernels/lstm.cc
index e78ec95a9a8..e022bfb85ba 100644
--- a/tensorflow/lite/kernels/lstm.cc
+++ b/tensorflow/lite/kernels/lstm.cc
@@ -21,7 +21,6 @@ limitations under the License.
 #include <memory>
 #include <vector>
 
-#include "absl/memory/memory.h"
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
 #include "tensorflow/lite/kernels/cpu_backend_context.h"
@@ -1008,7 +1007,7 @@ TfLiteStatus PrecomputeZeroPointTimesWeightWithBias(
   TF_LITE_ENSURE_EQ(context, weight_shape.DimensionsCount(), 2);
   const int row = weight_shape.Dims(0);
   const int col = weight_shape.Dims(1);
-  *output = absl::make_unique<int32_t[]>(row);
+  output->reset(new int32_t[row]);
   if (bias_tensor == nullptr) {
     memset(output->get(), 0, row * sizeof(int32_t));
   } else {
diff --git a/tensorflow/lite/kernels/lstm_test.cc b/tensorflow/lite/kernels/lstm_test.cc
index 62634e6bfbd..74ec8d324c6 100644
--- a/tensorflow/lite/kernels/lstm_test.cc
+++ b/tensorflow/lite/kernels/lstm_test.cc
@@ -137,10 +137,7 @@ class LSTMOpModel : public SingleOpModel {
                           asymmetric_quantize_inputs)
             .Union());
 
-    // Do not apply delegate yet since tensor values are not known (and more
-    // specifically scales in quantized tensors are not known).
-    BuildInterpreter(input_shapes, /*allow_fp32_relax_to_fp16=*/false,
-                     /*apply_delegate=*/false);
+    BuildInterpreter(input_shapes);
   }
 
   void SetInputToInputWeights(const std::vector<float>& f) {
@@ -570,7 +567,8 @@ TEST_F(NoCifgNoPeepholeNoProjectionNoClippingOmittedLayerNormLstmTest,
 
 TEST_P(NoCifgNoPeepholeNoProjectionNoClippingLstmTest,
        HybridLstmBlackBoxTestUint8) {
-  if (SingleOpModel::GetForceUseNnapi() && GetParam()) {
+  // TODO(b/158205028): Fix this test if GetForceUseNnapi() && !GetParam().
+  if (SingleOpModel::GetForceUseNnapi()) {
     return;
   }
   const int n_batch = 1;
@@ -759,7 +757,8 @@ TEST_F(CifgNoPeepholeNoProjectionNoClippingLstmTest, LstmBlackBoxTest) {
 
 TEST_P(CifgNoPeepholeNoProjectionNoClippingLstmTest,
        HybridLstmBlackBoxTestUint8) {
-  if (SingleOpModel::GetForceUseNnapi() && GetParam()) {
+  // TODO(b/158205028): Fix this test if GetForceUseNnapi() && !GetParam().
+  if (SingleOpModel::GetForceUseNnapi()) {
     return;
   }
   const int n_batch = 1;
@@ -1496,7 +1495,8 @@ TEST_F(NoCifgPeepholeProjectionNoClippingLstmTest, LstmBlackBoxTest) {
 
 TEST_P(NoCifgPeepholeProjectionNoClippingLstmTest,
        HybridLstmBlackBoxTestUint8) {
-  if (SingleOpModel::GetForceUseNnapi() && GetParam()) {
+  // TODO(b/158205028): Fix this test if GetForceUseNnapi() && !GetParam().
+  if (SingleOpModel::GetForceUseNnapi()) {
     return;
   }
   const int n_batch = 2;
@@ -1725,7 +1725,8 @@ TEST_F(NoCifgPeepholeProjectionNoClippingLayerNormLstmTest,
 
 TEST_P(NoCifgPeepholeProjectionNoClippingLayerNormLstmTest,
        HybridLayerNormLstmBlackBoxTestUint8) {
-  if (SingleOpModel::GetForceUseNnapi() && GetParam()) {
+  // TODO(b/158205028): Fix this test if GetForceUseNnapi() && !GetParam().
+  if (SingleOpModel::GetForceUseNnapi()) {
     return;
   }
   const int n_batch = 2;
@@ -1983,7 +1984,7 @@ TEST_F(CifgPeepholeProjectionNoClippingLayerNormLstmTest,
 
 TEST_P(CifgPeepholeProjectionNoClippingLayerNormLstmTest,
        HybridLayerNormLstmBlackBoxTestUint8) {
-  if (SingleOpModel::GetForceUseNnapi() && GetParam()) {
+  if (SingleOpModel::GetForceUseNnapi()) {
     return;
   }
   const int n_batch = 2;
@@ -2257,10 +2258,7 @@ class LSTMIntegerOpModel : public SingleOpModel {
                                    cell_clip, proj_clip)
                      .Union());
 
-    // Do not apply delegate yet since tensor values are not known (and more
-    // specifically scales in quantized tensors are not known).
-    BuildInterpreter(input_shapes, /*allow_fp32_relax_to_fp16=*/false,
-                     /*apply_delegate=*/false);
+    BuildInterpreter(input_shapes);
   }
 
   void SetInputToInputWeights(const std::vector<float>& f) {
@@ -2936,10 +2934,7 @@ class LSTMIntegerOpModel8x8_8 : public SingleOpModel {
                                    cell_clip, proj_clip)
                      .Union());
 
-    // Do not apply delegate yet since tensor values are not known (and more
-    // specifically scales in quantized tensors are not known).
-    BuildInterpreter(input_shapes, /*allow_fp32_relax_to_fp16=*/false,
-                     /*apply_delegate=*/false);
+    BuildInterpreter(input_shapes);
   }
 
   void SetInputToInputWeights(const std::vector<float>& f) {
@@ -3359,6 +3354,7 @@ TEST(LSTMOpModel, InvalidTypeTest) {
 }
 #endif
 
+// Test parameter controls asymmetric_quantize_inputs in LSTMOpModel.
 #define QUANTIZE_PARAMETER_TEST(test) \
   INSTANTIATE_TEST_SUITE_P(test, test, ::testing::Bool())
 
diff --git a/tensorflow/lite/kernels/pad.cc b/tensorflow/lite/kernels/pad.cc
index 1bd4f65a043..cc735e4eede 100644
--- a/tensorflow/lite/kernels/pad.cc
+++ b/tensorflow/lite/kernels/pad.cc
@@ -76,9 +76,7 @@ TfLiteStatus ResizeOutputTensor(TfLiteContext* context,
                     op_context->dims);
   TF_LITE_ENSURE_EQ(context, SizeOfDimension(op_context->paddings, 1), 2);
 
-  // Determines the size of the output tensor.
-  TfLiteIntArray* input_size = op_context->input->dims;
-  TfLiteIntArray* output_size = TfLiteIntArrayCopy(input_size);
+  // Ensures all the elements of the paddings is non-negative.
   const int32* paddings_data = GetTensorData<int32>(op_context->paddings);
 
   for (int idx = 0; idx < op_context->dims; ++idx) {
@@ -87,6 +85,16 @@ TfLiteStatus ResizeOutputTensor(TfLiteContext* context,
 
     TF_LITE_ENSURE_MSG(context, (before_padding >= 0 && after_padding >= 0),
                        "Pad value has to be greater than equal to 0.");
+  }
+
+  // Determines the size of the output tensor.
+  TfLiteIntArray* input_size = op_context->input->dims;
+  TfLiteIntArray* output_size = TfLiteIntArrayCopy(input_size);
+  paddings_data = GetTensorData<int32>(op_context->paddings);
+
+  for (int idx = 0; idx < op_context->dims; ++idx) {
+    int before_padding = *paddings_data++;
+    int after_padding = *paddings_data++;
 
     output_size->data[idx] =
         (input_size->data[idx] + before_padding + after_padding);
@@ -110,9 +118,11 @@ TfLiteStatus Prepare(TfLiteContext* context, TfLiteNode* node) {
   TF_LITE_ENSURE(
       context, op_context.dims <= reference_ops::PadKernelMaxDimensionCount());
 
-  // Exit early if paddings is a non-const tensor. Set output tensor to
-  // dynamic so output size can be determined in Eval.
-  if (!IsConstantTensor(op_context.paddings)) {
+  // Exit early if paddings is a non-const tensor or the given input is an
+  // unranked input. Set output tensor to dynamic so output size can be
+  // determined in Eval.
+  if (NumDimensions(op_context.input) == 0 ||
+      !IsConstantTensor(op_context.paddings)) {
     SetTensorToDynamic(op_context.output);
     return kTfLiteOk;
   }
diff --git a/tensorflow/lite/kernels/pad_test.cc b/tensorflow/lite/kernels/pad_test.cc
index 8ef03290531..6e1e00cc3b8 100644
--- a/tensorflow/lite/kernels/pad_test.cc
+++ b/tensorflow/lite/kernels/pad_test.cc
@@ -271,6 +271,16 @@ TEST(PadOpTest, SimpleDynamicTest) {
   EXPECT_THAT(m.GetOutputShape(), ElementsAreArray({1, 4, 4, 1}));
 }
 
+TEST(PadOpTest, DynamicUnequalDimensions) {
+  if (SingleOpModel::GetForceUseNnapi()) {
+    return;
+  }
+  PadOpDynamicModel m({TensorType_FLOAT32, {}}, {3, 2}, {TensorType_FLOAT32});
+  m.SetInput({1, 2, 3, 4});
+  m.SetPaddings({0, 0, 1, 1, 1, 1, 0, 0});
+  ASSERT_NE(m.InvokeUnchecked(), kTfLiteOk) << "Unequal dimensions.";
+}
+
 TEST(PadOpTest, AdvancedConstTest) {
   PadOpConstModel m({TensorType_FLOAT32, {1, 2, 3, 1}}, {4, 2},
                     {1, 0, 0, 2, 0, 3, 0, 0}, {TensorType_FLOAT32});
@@ -528,6 +538,17 @@ TEST(PadV2OpTest, SimpleDynamicTest) {
   EXPECT_THAT(m.GetOutputShape(), ElementsAreArray({1, 4, 4, 1}));
 }
 
+TEST(PadV2OpTest, DynamicUnequalDimensions) {
+  if (SingleOpModel::GetForceUseNnapi()) {
+    return;
+  }
+  PadV2OpDynamicModel<float> m({TensorType_FLOAT32, {}}, {4, 2}, 0.0,
+                               {TensorType_FLOAT32});
+  m.SetInput({1, 2, 3, 4});
+  m.SetPaddings({0, 0, 1, 1, 1, 1, 0, 0});
+  ASSERT_NE(m.InvokeUnchecked(), kTfLiteOk) << "Unequal dimensions";
+}
+
 TEST(PadV2OpTest, SimpleDynamicValuedTest) {
   PadV2OpDynamicModel<float> m({TensorType_FLOAT32, {1, 2, 2, 1}}, {4, 2}, 5,
                                {TensorType_FLOAT32});
diff --git a/tensorflow/lite/kernels/test_util.cc b/tensorflow/lite/kernels/test_util.cc
index 9559140291b..010aed31dc6 100644
--- a/tensorflow/lite/kernels/test_util.cc
+++ b/tensorflow/lite/kernels/test_util.cc
@@ -199,15 +199,16 @@ void SingleOpModel::BuildInterpreter(std::vector<std::vector<int>> input_shapes,
   if (apply_delegate) ApplyDelegate();
 }
 
-void SingleOpModel::ApplyDelegate() {
+TfLiteStatus SingleOpModel::ApplyDelegate() {
   if (force_use_nnapi) {
-    interpreter_->ModifyGraphWithDelegate(TestNnApiDelegate());
+    delegate_ = TestNnApiDelegate();
   }
 
-  // Modify delegate with function.
-  if (apply_delegate_fn_) {
-    apply_delegate_fn_(interpreter_.get());
+  if (delegate_) {
+    return interpreter_->ModifyGraphWithDelegate(delegate_);
   }
+
+  return kTfLiteOk;
 }
 
 void SingleOpModel::Invoke() { ASSERT_EQ(interpreter_->Invoke(), kTfLiteOk); }
@@ -221,20 +222,6 @@ void SingleOpModel::BuildInterpreter(
                    /*apply_delegate=*/true);
 }
 
-void SingleOpModel::BuildInterpreter(std::vector<std::vector<int>> input_shapes,
-                                     bool allow_fp32_relax_to_fp16,
-                                     bool apply_delegate) {
-  BuildInterpreter(input_shapes, /*num_threads=*/-1, allow_fp32_relax_to_fp16,
-                   apply_delegate);
-}
-
-void SingleOpModel::BuildInterpreter(std::vector<std::vector<int>> input_shapes,
-                                     int num_threads) {
-  BuildInterpreter(input_shapes, num_threads,
-                   /*allow_fp32_relax_to_fp16=*/false,
-                   /*apply_delegate=*/true);
-}
-
 // static
 void SingleOpModel::SetForceUseNnapi(bool use_nnapi) {
   force_use_nnapi = use_nnapi;
diff --git a/tensorflow/lite/kernels/test_util.h b/tensorflow/lite/kernels/test_util.h
index d308dfee469..3d5df1a9cff 100644
--- a/tensorflow/lite/kernels/test_util.h
+++ b/tensorflow/lite/kernels/test_util.h
@@ -160,14 +160,11 @@ class SingleOpModel {
   SingleOpModel() {}
   ~SingleOpModel();
 
-  // Set a function callback that is run right after graph is prepared
-  // that allows applying external delegates. This is useful for testing
-  // other runtimes like NN API or GPU.
-  void SetApplyDelegate(std::function<void(Interpreter*)> apply_delegate_fn) {
-    apply_delegate_fn_ = apply_delegate_fn;
-  }
+  // Set a delegate that is applied right after graph is prepared. This is
+  // useful for testing other runtimes like NN API or GPU.
+  void SetDelegate(TfLiteDelegate* delegate) { delegate_ = delegate; }
 
-  void ApplyDelegate();
+  TfLiteStatus ApplyDelegate();
 
   // Copying or assignment is disallowed to simplify ownership semantics.
   SingleOpModel(const SingleOpModel&) = delete;
@@ -382,13 +379,7 @@ class SingleOpModel {
   // tensors given the shapes of the inputs.
   void BuildInterpreter(std::vector<std::vector<int>> input_shapes,
                         int num_threads, bool allow_fp32_relax_to_fp16,
-                        bool apply_delegate = true);
-
-  void BuildInterpreter(std::vector<std::vector<int>> input_shapes,
-                        int num_threads);
-
-  void BuildInterpreter(std::vector<std::vector<int>> input_shapes,
-                        bool allow_fp32_relax_to_fp16, bool apply_delegate);
+                        bool apply_delegate);
 
   void BuildInterpreter(std::vector<std::vector<int>> input_shapes);
 
@@ -761,9 +752,7 @@ class SingleOpModel {
   std::vector<int32_t> outputs_;
   std::vector<flatbuffers::Offset<Tensor>> tensors_;
   std::vector<flatbuffers::Offset<Buffer>> buffers_;
-  // A function pointer that gets called after the interpreter is created but
-  // before evaluation happens. This is useful for applying a delegate.
-  std::function<void(Interpreter*)> apply_delegate_fn_;
+  TfLiteDelegate* delegate_ = nullptr;
 };
 
 // Populate string tensors.
diff --git a/tensorflow/lite/kernels/transpose_conv.cc b/tensorflow/lite/kernels/transpose_conv.cc
index 9b2767f15a9..494433159d4 100644
--- a/tensorflow/lite/kernels/transpose_conv.cc
+++ b/tensorflow/lite/kernels/transpose_conv.cc
@@ -155,8 +155,9 @@ static TfLiteStatus AllocateTemporaryTensorsIfRequired(TfLiteContext* context,
     ++temporaries_count;
   }
 
-  // Allocate scratch buffer tensor for UInt8 inputs.
-  if (input_type == kTfLiteUInt8 || input_type == kTfLiteInt8) {
+  // Allocate scratch buffer tensor
+  if (input_type == kTfLiteUInt8 || input_type == kTfLiteInt8 ||
+      input_type == kTfLiteInt16) {
     if (data->scratch_tensor_id == kTensorNotAllocated) {
       context->AddTensors(context, 1, &data->scratch_tensor_id);
     }
@@ -227,13 +228,16 @@ TfLiteStatus ResizeAndTransposeWeights(TfLiteContext* context,
                              GetTensorShape(transposed_weights),
                              GetTensorData<uint8>(transposed_weights));
   } else if (weights->type == kTfLiteInt8) {
+    // int16 transpose_conv also with int8 weights
     optimized_ops::Transpose(transpose_params, input_shape,
                              GetTensorData<int8>(weights),
                              GetTensorShape(transposed_weights),
                              GetTensorData<int8>(transposed_weights));
   } else {
-    context->ReportError(
-        context, "Transpose conv only support float & uint8 & int8 right now.");
+    TF_LITE_KERNEL_LOG(
+        context,
+        "Only float32, uint8, int8, int16 is supported currently, got %s.",
+        TfLiteTypeGetName(weights->type));
     return kTfLiteError;
   }
 
@@ -263,9 +267,9 @@ TfLiteStatus Prepare(TfLiteContext* context, TfLiteNode* node) {
   TF_LITE_ENSURE_EQ(context, NumDimensions(output_shape), 1);
   TF_LITE_ENSURE_EQ(context, NumDimensions(input), 4);
   TF_LITE_ENSURE_EQ(context, NumDimensions(weights), 4);
-  TF_LITE_ENSURE(context, input->type == kTfLiteFloat32 ||
-                              input->type == kTfLiteUInt8 ||
-                              input->type == kTfLiteInt8);
+  TF_LITE_ENSURE(context,
+                 input->type == kTfLiteFloat32 || input->type == kTfLiteUInt8 ||
+                     input->type == kTfLiteInt8 || input->type == kTfLiteInt16);
 
   if (has_bias) {
     bias = GetOptionalInputTensor(context, node, kBiasTensor);
@@ -275,6 +279,9 @@ TfLiteStatus Prepare(TfLiteContext* context, TfLiteNode* node) {
         if (input->type == kTfLiteInt8) {
           TF_LITE_ENSURE_EQ(context, bias->params.zero_point, 0);
         }
+      } else if (input->type == kTfLiteInt16) {
+        TF_LITE_ENSURE_EQ(context, bias->type, kTfLiteInt64);
+        TF_LITE_ENSURE_EQ(context, bias->params.zero_point, 0);
       } else {
         TF_LITE_ENSURE_EQ(context, bias->type, input->type);
       }
@@ -283,6 +290,13 @@ TfLiteStatus Prepare(TfLiteContext* context, TfLiteNode* node) {
     }
   }
 
+  if (input->type == kTfLiteInt16) {
+    TF_LITE_ENSURE_EQ(context, weights->type, kTfLiteInt8);
+    TF_LITE_ENSURE_EQ(context, input->params.zero_point, 0);
+    TF_LITE_ENSURE_EQ(context, output->params.zero_point, 0);
+  } else {
+    TF_LITE_ENSURE_EQ(context, weights->type, input->type);
+  }
   TF_LITE_ENSURE_EQ(context, output->type, input->type);
   // Ensure that weights and inputs have the same channel dimension.
   // Note: TOCO will reorder weights in the following format: OHWI.
@@ -326,12 +340,18 @@ TfLiteStatus Prepare(TfLiteContext* context, TfLiteNode* node) {
     }
   }
 
-  if (input->type == kTfLiteUInt8 || input->type == kTfLiteInt8) {
+  if (input->type == kTfLiteUInt8 || input->type == kTfLiteInt8 ||
+      input->type == kTfLiteInt16) {
     node->temporaries->data[data->scratch_tensor_index] =
         data->scratch_tensor_id;
     TfLiteTensor* scratch_buffer =
         GetTemporary(context, node, data->scratch_tensor_index);
-    scratch_buffer->type = kTfLiteInt32;
+    if (input->type == kTfLiteInt16) {
+      scratch_buffer->type = kTfLiteInt64;
+    } else {
+      scratch_buffer->type = kTfLiteInt32;
+    }
+
     scratch_buffer->allocation_type = kTfLiteDynamic;
     if (!IsConstantTensor(output_shape)) {
       SetTensorToDynamic(scratch_buffer);
@@ -500,6 +520,37 @@ void EvalQuantizedPerChannel(
   }
 }
 
+void EvalQuantizedPerChannel16x8(
+    TfLiteContext* context, const TfLiteTransposeConvParams* params,
+    OpData* data, const TfLiteTensor* input, const TfLiteTensor* weights,
+    const TfLiteTensor* transposed_weights, const TfLiteTensor* bias,
+    TfLiteTensor* col2im, TfLiteTensor* output, TfLiteTensor* scratch_buffer) {
+  tflite::ConvParams op_params;
+  op_params.padding_type = PaddingType::kSame;
+  op_params.padding_values.width = data->padding.width;
+  op_params.padding_values.height = data->padding.height;
+  op_params.padding_values.width_offset = data->padding.width_offset;
+  op_params.padding_values.height_offset = data->padding.height_offset;
+  op_params.stride_width = params->stride_width;
+  op_params.stride_height = params->stride_height;
+  // Need to flip the sign of input offset to add it directly to the quantized
+  // buffer.
+  op_params.input_offset = -input->params.zero_point;
+  op_params.output_offset = output->params.zero_point;
+  op_params.quantized_activation_min = data->output_activation_min;
+  op_params.quantized_activation_max = data->output_activation_max;
+
+  // Need to add optimized kernel
+  reference_integer_ops::TransposeConv(
+      op_params, data->per_channel_output_multiplier.data(),
+      data->per_channel_output_shift.data(), GetTensorShape(input),
+      GetTensorData<int16>(input), GetTensorShape(weights),
+      GetTensorData<int8>(weights), GetTensorShape(bias),
+      GetTensorData<int64_t>(bias), GetTensorShape(output),
+      GetTensorData<int16>(output), GetTensorShape(col2im),
+      GetTensorData<int8>(col2im), GetTensorData<int64_t>(scratch_buffer));
+}
+
 template <KernelType kernel_type>
 TfLiteStatus Eval(TfLiteContext* context, TfLiteNode* node) {
   // Retrieve tensors (All should be allocated by now)
@@ -544,7 +595,7 @@ TfLiteStatus Eval(TfLiteContext* context, TfLiteNode* node) {
       filter_height, filter_width, params->padding, &unused_output_height,
       &unused_output_width);
 
-  // Currently support float32 and uint8.
+  // Currently support float32, uint8, int8, int16.
   switch (input->type) {
     case kTfLiteFloat32: {
       // Only for GenericOptimized path, we use transposed weights.
@@ -589,6 +640,21 @@ TfLiteStatus Eval(TfLiteContext* context, TfLiteNode* node) {
                                            col2im, output, scratch_buffer);
       break;
     }
+    case kTfLiteInt16: {
+      TfLiteTensor* scratch_buffer =
+          GetTemporary(context, node, data->scratch_tensor_index);
+      if (IsDynamicTensor(scratch_buffer)) {
+        TF_LITE_ENSURE_OK(context,
+                          ResizeTensor(context, output_shape, scratch_buffer));
+      }
+      if (data->weights_are_transposed && !IsConstantTensor(weights)) {
+        ResizeAndTransposeWeights(context, weights, transposed_weights);
+      }
+      EvalQuantizedPerChannel16x8(context, params, data, input, weights,
+                                  transposed_weights, bias, col2im, output,
+                                  scratch_buffer);
+      break;
+    }
     default:
       context->ReportError(context, "Type '%s' is not currently supported.",
                            TfLiteTypeGetName(input->type));
diff --git a/tensorflow/lite/kernels/transpose_conv_test.cc b/tensorflow/lite/kernels/transpose_conv_test.cc
index 77dc22b13e8..25c55c95412 100644
--- a/tensorflow/lite/kernels/transpose_conv_test.cc
+++ b/tensorflow/lite/kernels/transpose_conv_test.cc
@@ -76,7 +76,10 @@ class BaseTransposeConvOpModel : public SingleOpModel {
 
     if (test_type == TestType::kDynamic) {
       PopulateTensor<int32_t>(output_shape_, output_shape_data);
-      PopulateTensor<InputType>(filter_, filter_data);
+      if (!std::is_same<InputType, int16_t>::value &&
+          !std::is_same<InputType, int8_t>::value) {
+        PopulateTensor<InputType>(filter_, filter_data);
+      }
     }
   }
 
@@ -85,6 +88,8 @@ class BaseTransposeConvOpModel : public SingleOpModel {
       QuantizeAndPopulate<uint8_t>(input_, data);
     } else if (std::is_same<InputType, int8_t>::value) {
       QuantizeAndPopulate<int8_t>(input_, data);
+    } else if (std::is_same<InputType, int16_t>::value) {
+      QuantizeAndPopulate<int16_t>(input_, data);
     } else {
       PopulateTensor(input_, data);
     }
@@ -315,6 +320,56 @@ TEST_P(TransposeConvOpTest, SimpleTestQuantized) {
   EXPECT_THAT(model.GetOutputShape(), ElementsAreArray({1, 4, 4, 1}));
 }
 
+TEST_P(TransposeConvOpTest, TwoFiltersTestQuantized) {
+  // Float would be {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
+  // 18}
+  std::initializer_list<uint8_t> filter_data = {129, 131, 133, 135, 137, 139,
+                                                141, 143, 145, 147, 149, 151,
+                                                153, 155, 157, 159, 161, 163};
+  QuantizedTransposeConvOpModel model(
+      GetRegistration(), {1, 4, 4, 1},
+      {TensorType_UINT8, {1, 3, 3, 2}, -63.5, 64}, filter_data,
+      {TensorType_UINT8, {1, 4, 4, 2}, -63.5, 64},
+      {TensorType_UINT8, {}, -4064, 4096}, Padding_SAME, 1, 1, GetTestType());
+  model.SetInput({1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11,
+                  12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
+                  23, 24, 25, 26, 27, 28, 29, 30, 31, 32});
+  model.Invoke();
+
+  EXPECT_THAT(model.GetDequantizedOutput(),
+              ElementsAreArray(ArrayFloatNear(
+                  {192, 416, 576, 544, 672, 1344, 1696, 1440, 1504, 2720, 3072,
+                   2432, 1984, 3360, 3648, 2752},
+                  1e-5)));
+  EXPECT_THAT(model.GetOutputShape(), ElementsAreArray({1, 4, 4, 1}));
+}
+
+TEST_P(TransposeConvOpTest, PaddingValidTestQuantized) {
+  // Float would be {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
+  // 18}
+  std::initializer_list<uint8_t> filter_data = {129, 131, 133, 135, 137, 139,
+                                                141, 143, 145, 147, 149, 151,
+                                                153, 155, 157, 159, 161, 163};
+  QuantizedTransposeConvOpModel model(
+      GetRegistration(), {1, 6, 6, 1},
+      {TensorType_UINT8, {1, 3, 3, 2}, -63.5, 64}, filter_data,
+      {TensorType_UINT8, {1, 4, 4, 2}, -63.5, 64},
+      {TensorType_UINT8, {}, -4064, 4096}, Padding_VALID, 1, 1, GetTestType());
+  model.SetInput({1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11,
+                  12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
+                  23, 24, 25, 26, 27, 28, 29, 30, 31, 32});
+  model.Invoke();
+
+  EXPECT_THAT(model.GetDequantizedOutput(),
+              ElementsAreArray(ArrayFloatNear(
+                  {0,    32,   64,   96,   128,  96,   64,   192,  416,
+                   576,  544,  352,  224,  672,  1344, 1696, 1440, 864,
+                   608,  1504, 2720, 3072, 2432, 1440, 864,  1984, 3360,
+                   3648, 2752, 1536, 704,  1536, 2528, 2720, 2016, 1088},
+                  1e-5)));
+  EXPECT_THAT(model.GetOutputShape(), ElementsAreArray({1, 6, 6, 1}));
+}
+
 class PerChannelQuantizedTransposeConvOpModel
     : public BaseTransposeConvOpModel<int8_t> {
  public:
@@ -325,10 +380,6 @@ class PerChannelQuantizedTransposeConvOpModel
                               GetZeroPoint(output_));
   }
 
-  void SetInput(const std::initializer_list<float>& data) {
-    QuantizeAndPopulate<int8_t>(input_, data);
-  }
-
   void SetFilter(const std::initializer_list<float>& data) {
     PerChannelSymmetricQuantizeAndPopulate(filter_, data);
   }
@@ -391,54 +442,78 @@ TEST_P(TransposeConvOpTest, TestQuantizedPerChannelMultiChannel) {
   EXPECT_THAT(model.GetOutputShape(), ElementsAreArray({1, 5, 5, 2}));
 }
 
-TEST_P(TransposeConvOpTest, TwoFiltersTestQuantized) {
-  // Float would be {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
-  // 18}
-  std::initializer_list<uint8_t> filter_data = {129, 131, 133, 135, 137, 139,
-                                                141, 143, 145, 147, 149, 151,
-                                                153, 155, 157, 159, 161, 163};
-  QuantizedTransposeConvOpModel model(
-      GetRegistration(), {1, 4, 4, 1},
-      {TensorType_UINT8, {1, 3, 3, 2}, -63.5, 64}, filter_data,
-      {TensorType_UINT8, {1, 4, 4, 2}, -63.5, 64},
-      {TensorType_UINT8, {}, -4064, 4096}, Padding_SAME, 1, 1, GetTestType());
-  model.SetInput({1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11,
-                  12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
-                  23, 24, 25, 26, 27, 28, 29, 30, 31, 32});
+class PerChannelQuantizedTransposeConvOpModel16x8
+    : public BaseTransposeConvOpModel<int16_t> {
+ public:
+  using BaseTransposeConvOpModel::BaseTransposeConvOpModel;
+
+  std::vector<float> GetDequantizedOutput() {
+    return Dequantize<int16_t>(ExtractVector<int16_t>(output_),
+                               GetScale(output_), GetZeroPoint(output_));
+  }
+
+  void SetFilter(const std::initializer_list<float>& data) {
+    PerChannelSymmetricQuantizeAndPopulate(filter_, data);
+  }
+};
+
+TEST_P(TransposeConvOpTest, SimpleTestQuantizedPerChannel16x8) {
+  const std::initializer_list<float> filter_data = {
+      // [2 * 2 * 2 * 2] as [output_channel, y, x, input_channel]
+      1, 2,  // out channel = 0, y = 0, x = 0
+      3, 4,  // out channel = 0, y = 0, x = 1
+      3, 4,  // out channel = 0, y = 1, x = 0
+      5, 6,  // out channel = 0, y = 1, x = 1
+      7, 8,  // out channel = 1, y = 0, x = 0
+      5, 6,  // out channel = 1, y = 0, x = 1
+      3, 4,  // out channel = 1, y = 1, x = 0
+      1, 2,  // out channel = 1, y = 1, x = 1
+  };
+  PerChannelQuantizedTransposeConvOpModel16x8 model(
+      GetRegistration(),
+      /*output_shape_data=*/{1, 2, 3, 2},
+      /*filter=*/
+      {TensorType_INT8,
+       /*shape=*/{2, 2, 2, 2},
+       /*min=*/-64, /*max=*/64,
+       /*scale=*/0, /*zero_point=*/0,
+       /*per_channel_quantization=*/true,
+       /*per_channel_quantization_scales=*/{7.0 / 127, 8.0 / 127},
+       /*per_channel_quantization_offsets=*/{0, 0},
+       /*channel_index=*/0},
+      /*filter_data=*/{},
+      /*input=*/
+      {TensorType_INT16,
+       /*shape=*/{1, 2, 3, 2},
+       /*min=*/0, /*max=*/0,
+       /*scale=*/4.0 / 127,
+       /*zero_point=*/0},
+      /*output=*/
+      {TensorType_INT16,
+       /*shape=*/{},
+       /*min=*/0, /*max=*/0,
+       /*scale=*/1.0,
+       /*zero_point=*/0},
+      /*padding=*/Padding_SAME,
+      /*stride_w=*/1, /*stride_h=*/1, GetTestType());
+  model.SetInput({
+      // [1 * 2 * 3 * 2] as [batch, y, x, input_channel]
+      3, 2,    // batch = 0, y = 0, x = 0
+      1, -1,   // batch = 0, y = 0, x = 1
+      -2, -3,  // batch = 0, y = 0, x = 2
+      4, 3,    // batch = 0, y = 1, x = 0
+      2, -2,   // batch = 0, y = 1, x = 1
+      -3, -4,  // batch = 0, y = 1, x = 2
+  });
+  model.SetFilter(filter_data);
   model.Invoke();
 
   EXPECT_THAT(model.GetDequantizedOutput(),
               ElementsAreArray(ArrayFloatNear(
-                  {192, 416, 576, 544, 672, 1344, 1696, 1440, 1504, 2720, 3072,
-                   2432, 1984, 3360, 3648, 2752},
-                  1e-5)));
-  EXPECT_THAT(model.GetOutputShape(), ElementsAreArray({1, 4, 4, 1}));
-}
+                  {7, 37, 16, 26, -9, -39, 27, 69, 48, 42, -32, -74}, 1e-5)));
 
-TEST_P(TransposeConvOpTest, PaddingValidTestQuantized) {
-  // Float would be {1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17,
-  // 18}
-  std::initializer_list<uint8_t> filter_data = {129, 131, 133, 135, 137, 139,
-                                                141, 143, 145, 147, 149, 151,
-                                                153, 155, 157, 159, 161, 163};
-  QuantizedTransposeConvOpModel model(
-      GetRegistration(), {1, 6, 6, 1},
-      {TensorType_UINT8, {1, 3, 3, 2}, -63.5, 64}, filter_data,
-      {TensorType_UINT8, {1, 4, 4, 2}, -63.5, 64},
-      {TensorType_UINT8, {}, -4064, 4096}, Padding_VALID, 1, 1, GetTestType());
-  model.SetInput({1,  2,  3,  4,  5,  6,  7,  8,  9,  10, 11,
-                  12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
-                  23, 24, 25, 26, 27, 28, 29, 30, 31, 32});
-  model.Invoke();
-
-  EXPECT_THAT(model.GetDequantizedOutput(),
-              ElementsAreArray(ArrayFloatNear(
-                  {0,    32,   64,   96,   128,  96,   64,   192,  416,
-                   576,  544,  352,  224,  672,  1344, 1696, 1440, 864,
-                   608,  1504, 2720, 3072, 2432, 1440, 864,  1984, 3360,
-                   3648, 2752, 1536, 704,  1536, 2528, 2720, 2016, 1088},
-                  1e-5)));
-  EXPECT_THAT(model.GetOutputShape(), ElementsAreArray({1, 6, 6, 1}));
+  // GetOutputShape() should always be same as model.SetOutputShape(...);
+  EXPECT_THAT(model.GetOutputShape(), ElementsAreArray({1, 2, 3, 2}));
 }
 
 template <typename InputType>
diff --git a/tensorflow/lite/micro/BUILD b/tensorflow/lite/micro/BUILD
index ae4ba2fefd6..735952f9bee 100644
--- a/tensorflow/lite/micro/BUILD
+++ b/tensorflow/lite/micro/BUILD
@@ -36,8 +36,6 @@ cc_library(
         "memory_helpers.h",
         "micro_allocator.h",
         "micro_interpreter.h",
-        "micro_mutable_op_resolver.h",
-        "micro_op_resolver.h",
         "micro_optional_debug_tools.h",
         "simple_memory_allocator.h",
         "test_helpers.h",
@@ -47,6 +45,7 @@ cc_library(
     deps = [
         ":micro_compatibility",
         ":micro_utils",
+        ":op_resolvers",
         "//tensorflow/lite:type_to_tflitetype",
         "//tensorflow/lite/c:common",
         "//tensorflow/lite/core/api",
@@ -59,6 +58,53 @@ cc_library(
     ],
 )
 
+cc_library(
+    name = "op_resolvers",
+    srcs = [
+        "all_ops_resolver.cc",
+    ],
+    hdrs = [
+        "all_ops_resolver.h",
+        "micro_mutable_op_resolver.h",
+        "micro_op_resolver.h",
+    ],
+    build_for_embedded = True,
+    copts = micro_copts(),
+    deps = [
+        ":micro_compatibility",
+        "//tensorflow/lite/c:common",
+        "//tensorflow/lite/core/api",
+        "//tensorflow/lite/kernels:op_macros",
+        "//tensorflow/lite/kernels/internal:compatibility",
+        "//tensorflow/lite/micro/kernels:micro_ops",
+        "//tensorflow/lite/schema:schema_fbs",
+    ],
+)
+
+# TODO(b/144176795): This target should really be handled differently so that we
+# do not have a fork in the build graph. The bug has some initial ideas.
+cc_library(
+    name = "portable_optimized_op_resolver",
+    srcs = [
+        "all_ops_resolver.cc",
+        "micro_mutable_op_resolver.h",
+        "micro_op_resolver.h",
+    ],
+    hdrs = [
+        "all_ops_resolver.h",
+    ],
+    copts = micro_copts(),
+    deps = [
+        ":micro_compatibility",
+        "//tensorflow/lite/c:common",
+        "//tensorflow/lite/core/api",
+        "//tensorflow/lite/kernels:op_macros",
+        "//tensorflow/lite/kernels/internal:compatibility",
+        "//tensorflow/lite/micro/kernels:portable_optimized_micro_ops",
+        "//tensorflow/lite/schema:schema_fbs",
+    ],
+)
+
 cc_library(
     name = "debug_log",
     srcs = [
@@ -132,11 +178,13 @@ cc_library(
 )
 
 cc_library(
-    name = "recording_simple_memory_allocator",
+    name = "recording_allocators",
     srcs = [
+        "recording_micro_allocator.cc",
         "recording_simple_memory_allocator.cc",
     ],
     hdrs = [
+        "recording_micro_allocator.h",
         "recording_simple_memory_allocator.h",
     ],
     build_for_embedded = True,
@@ -144,6 +192,7 @@ cc_library(
     deps = [
         ":micro_compatibility",
         ":micro_framework",
+        "//tensorflow/lite/core/api",
     ],
 )
 
@@ -164,6 +213,7 @@ tflite_micro_cc_test(
     ],
     deps = [
         ":micro_framework",
+        ":op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -176,6 +226,7 @@ tflite_micro_cc_test(
     deps = [
         ":micro_framework",
         ":micro_utils",
+        ":op_resolvers",
         "//tensorflow/lite/core/api",
         "//tensorflow/lite/kernels:kernel_util",
         "//tensorflow/lite/micro/testing:micro_test",
@@ -200,7 +251,7 @@ tflite_micro_cc_test(
     ],
     deps = [
         ":micro_framework",
-        ":recording_simple_memory_allocator",
+        ":recording_allocators",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -216,6 +267,20 @@ tflite_micro_cc_test(
     ],
 )
 
+tflite_micro_cc_test(
+    name = "recording_micro_allocator_test",
+    srcs = [
+        "recording_micro_allocator_test.cc",
+    ],
+    deps = [
+        ":micro_framework",
+        ":op_resolvers",
+        ":recording_allocators",
+        "//tensorflow/lite/micro/testing:micro_test",
+        "//tensorflow/lite/micro/testing:test_conv_model",
+    ],
+)
+
 tflite_micro_cc_test(
     name = "memory_helpers_test",
     srcs = [
diff --git a/tensorflow/lite/micro/all_ops_resolver.cc b/tensorflow/lite/micro/all_ops_resolver.cc
new file mode 100644
index 00000000000..b0021a2e771
--- /dev/null
+++ b/tensorflow/lite/micro/all_ops_resolver.cc
@@ -0,0 +1,104 @@
+/* Copyright 2018 The TensorFlow Authors. All Rights Reserved.
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+    http://www.apache.org/licenses/LICENSE-2.0
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include "tensorflow/lite/micro/all_ops_resolver.h"
+
+#include "tensorflow/lite/micro/kernels/micro_ops.h"
+
+namespace tflite {
+namespace ops {
+namespace micro {
+namespace custom {
+TfLiteRegistration* Register_ETHOSU();
+const char* GetString_ETHOSU();
+}  // namespace custom
+}  // namespace micro
+}  // namespace ops
+
+AllOpsResolver::AllOpsResolver() {
+  // Please keep this list of Builtin Operators in alphabetical order.
+  AddBuiltin(BuiltinOperator_ABS, tflite::ops::micro::Register_ABS());
+  AddBuiltin(BuiltinOperator_ADD, tflite::ops::micro::Register_ADD());
+  AddBuiltin(BuiltinOperator_ARG_MAX, tflite::ops::micro::Register_ARG_MAX());
+  AddBuiltin(BuiltinOperator_ARG_MIN, tflite::ops::micro::Register_ARG_MIN());
+  AddBuiltin(BuiltinOperator_AVERAGE_POOL_2D,
+             tflite::ops::micro::Register_AVERAGE_POOL_2D());
+  AddBuiltin(BuiltinOperator_CEIL, tflite::ops::micro::Register_CEIL());
+  AddBuiltin(BuiltinOperator_CONCATENATION,
+             tflite::ops::micro::Register_CONCATENATION());
+  AddBuiltin(BuiltinOperator_CONV_2D, tflite::ops::micro::Register_CONV_2D());
+  AddBuiltin(BuiltinOperator_COS, tflite::ops::micro::Register_COS());
+  AddBuiltin(BuiltinOperator_DEPTHWISE_CONV_2D,
+             tflite::ops::micro::Register_DEPTHWISE_CONV_2D());
+  AddBuiltin(BuiltinOperator_DEQUANTIZE,
+             tflite::ops::micro::Register_DEQUANTIZE());
+  AddBuiltin(BuiltinOperator_EQUAL, tflite::ops::micro::Register_EQUAL());
+  AddBuiltin(BuiltinOperator_FLOOR, tflite::ops::micro::Register_FLOOR());
+  AddBuiltin(BuiltinOperator_FULLY_CONNECTED,
+             tflite::ops::micro::Register_FULLY_CONNECTED());
+  AddBuiltin(BuiltinOperator_GREATER, tflite::ops::micro::Register_GREATER());
+  AddBuiltin(BuiltinOperator_GREATER_EQUAL,
+             tflite::ops::micro::Register_GREATER_EQUAL());
+  AddBuiltin(BuiltinOperator_L2_NORMALIZATION,
+             tflite::ops::micro::Register_L2_NORMALIZATION());
+  AddBuiltin(BuiltinOperator_LESS, tflite::ops::micro::Register_LESS());
+  AddBuiltin(BuiltinOperator_LESS_EQUAL,
+             tflite::ops::micro::Register_LESS_EQUAL());
+  AddBuiltin(BuiltinOperator_LOG, tflite::ops::micro::Register_LOG());
+  AddBuiltin(BuiltinOperator_LOGICAL_AND,
+             tflite::ops::micro::Register_LOGICAL_AND());
+  AddBuiltin(BuiltinOperator_LOGICAL_NOT,
+             tflite::ops::micro::Register_LOGICAL_NOT());
+  AddBuiltin(BuiltinOperator_LOGICAL_OR,
+             tflite::ops::micro::Register_LOGICAL_OR());
+  AddBuiltin(BuiltinOperator_LOGISTIC, tflite::ops::micro::Register_LOGISTIC());
+  AddBuiltin(BuiltinOperator_MAX_POOL_2D,
+             tflite::ops::micro::Register_MAX_POOL_2D());
+  AddBuiltin(BuiltinOperator_MAXIMUM, tflite::ops::micro::Register_MAXIMUM());
+  AddBuiltin(BuiltinOperator_MEAN, tflite::ops::micro::Register_MEAN());
+  AddBuiltin(BuiltinOperator_MINIMUM, tflite::ops::micro::Register_MINIMUM());
+  AddBuiltin(BuiltinOperator_MUL, tflite::ops::micro::Register_MUL());
+  AddBuiltin(BuiltinOperator_NEG, tflite::ops::micro::Register_NEG());
+  AddBuiltin(BuiltinOperator_NOT_EQUAL,
+             tflite::ops::micro::Register_NOT_EQUAL());
+  AddBuiltin(BuiltinOperator_PACK, tflite::ops::micro::Register_PACK());
+  AddBuiltin(BuiltinOperator_PAD, tflite::ops::micro::Register_PAD());
+  AddBuiltin(BuiltinOperator_PADV2, tflite::ops::micro::Register_PADV2());
+  AddBuiltin(BuiltinOperator_PRELU, tflite::ops::micro::Register_PRELU());
+  AddBuiltin(BuiltinOperator_QUANTIZE, tflite::ops::micro::Register_QUANTIZE());
+  AddBuiltin(BuiltinOperator_RELU, tflite::ops::micro::Register_RELU());
+  AddBuiltin(BuiltinOperator_RELU6, tflite::ops::micro::Register_RELU6());
+  AddBuiltin(BuiltinOperator_RESHAPE, tflite::ops::micro::Register_RESHAPE());
+  AddBuiltin(BuiltinOperator_RESIZE_NEAREST_NEIGHBOR,
+             tflite::ops::micro::Register_RESIZE_NEAREST_NEIGHBOR());
+  AddBuiltin(BuiltinOperator_ROUND, tflite::ops::micro::Register_ROUND());
+  AddBuiltin(BuiltinOperator_RSQRT, tflite::ops::micro::Register_RSQRT());
+  AddBuiltin(BuiltinOperator_SIN, tflite::ops::micro::Register_SIN());
+  AddBuiltin(BuiltinOperator_SOFTMAX, tflite::ops::micro::Register_SOFTMAX());
+  AddBuiltin(BuiltinOperator_SPLIT, tflite::ops::micro::Register_SPLIT());
+  AddBuiltin(BuiltinOperator_SQRT, tflite::ops::micro::Register_SQRT());
+  AddBuiltin(BuiltinOperator_SQUARE, tflite::ops::micro::Register_SQUARE());
+  AddBuiltin(BuiltinOperator_STRIDED_SLICE,
+             tflite::ops::micro::Register_STRIDED_SLICE());
+  AddBuiltin(BuiltinOperator_SUB, tflite::ops::micro::Register_SUB());
+  AddBuiltin(BuiltinOperator_SVDF, tflite::ops::micro::Register_SVDF());
+  AddBuiltin(BuiltinOperator_TANH, tflite::ops::micro::Register_TANH());
+  AddBuiltin(BuiltinOperator_UNPACK, tflite::ops::micro::Register_UNPACK());
+
+  TfLiteRegistration* registration =
+      tflite::ops::micro::custom::Register_ETHOSU();
+  if (registration) {
+    AddCustom(tflite::ops::micro::custom::GetString_ETHOSU(), registration);
+  }
+}
+
+}  // namespace tflite
diff --git a/tensorflow/lite/micro/kernels/all_ops_resolver.h b/tensorflow/lite/micro/all_ops_resolver.h
similarity index 83%
rename from tensorflow/lite/micro/kernels/all_ops_resolver.h
rename to tensorflow/lite/micro/all_ops_resolver.h
index 5637316d0da..e8105b96e44 100644
--- a/tensorflow/lite/micro/kernels/all_ops_resolver.h
+++ b/tensorflow/lite/micro/all_ops_resolver.h
@@ -9,15 +9,13 @@ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 See the License for the specific language governing permissions and
 limitations under the License.
 ==============================================================================*/
-#ifndef TENSORFLOW_LITE_MICRO_KERNELS_ALL_OPS_RESOLVER_H_
-#define TENSORFLOW_LITE_MICRO_KERNELS_ALL_OPS_RESOLVER_H_
+#ifndef TENSORFLOW_LITE_MICRO_ALL_OPS_RESOLVER_H_
+#define TENSORFLOW_LITE_MICRO_ALL_OPS_RESOLVER_H_
 
 #include "tensorflow/lite/micro/compatibility.h"
 #include "tensorflow/lite/micro/micro_mutable_op_resolver.h"
 
 namespace tflite {
-namespace ops {
-namespace micro {
 
 // The magic number in the template parameter is the maximum number of ops that
 // can be added to AllOpsResolver. It can be increased if needed. And most
@@ -32,8 +30,6 @@ class AllOpsResolver : public MicroMutableOpResolver<128> {
   TF_LITE_REMOVE_VIRTUAL_DELETE
 };
 
-}  // namespace micro
-}  // namespace ops
 }  // namespace tflite
 
-#endif  // TENSORFLOW_LITE_MICRO_KERNELS_ALL_OPS_RESOLVER_H_
+#endif  // TENSORFLOW_LITE_MICRO_ALL_OPS_RESOLVER_H_
diff --git a/tensorflow/lite/micro/benchmarks/BUILD b/tensorflow/lite/micro/benchmarks/BUILD
index 73b288d2bc1..8c5238ef56b 100644
--- a/tensorflow/lite/micro/benchmarks/BUILD
+++ b/tensorflow/lite/micro/benchmarks/BUILD
@@ -48,6 +48,7 @@ cc_binary(
         "//tensorflow/lite/c:common",
         "//tensorflow/lite/micro:micro_error_reporter",
         "//tensorflow/lite/micro:micro_framework",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/kernels:micro_ops",
         "//tensorflow/lite/micro/testing:micro_benchmark",
     ],
@@ -62,6 +63,7 @@ cc_binary(
         "//tensorflow/lite/micro:micro_error_reporter",
         "//tensorflow/lite/micro:micro_framework",
         "//tensorflow/lite/micro:micro_utils",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/examples/person_detection:model_settings",
         "//tensorflow/lite/micro/examples/person_detection:person_detect_model_data",
         "//tensorflow/lite/micro/examples/person_detection:simple_images_test_data",
diff --git a/tensorflow/lite/micro/benchmarks/keyword_benchmark.cc b/tensorflow/lite/micro/benchmarks/keyword_benchmark.cc
index ec4e93f9506..958150aa6ab 100644
--- a/tensorflow/lite/micro/benchmarks/keyword_benchmark.cc
+++ b/tensorflow/lite/micro/benchmarks/keyword_benchmark.cc
@@ -38,65 +38,15 @@ uint8_t tensor_arena[tensor_arena_size];
 // A random number generator seed to generate input values.
 constexpr int kRandomSeed = 42;
 
-class KeywordRunner {
- public:
-  KeywordRunner()
-      : keyword_spotting_model_(
-            tflite::GetModel(g_keyword_scrambled_model_data)),
-        reporter_(&micro_reporter_),
-        interpreter_(keyword_spotting_model_, resolver_, tensor_arena,
-                     tensor_arena_size, reporter_) {
-    resolver_.AddBuiltin(tflite::BuiltinOperator_SVDF,
-                         tflite::ops::micro::Register_SVDF());
-    resolver_.AddBuiltin(tflite::BuiltinOperator_FULLY_CONNECTED,
-                         tflite::ops::micro::Register_FULLY_CONNECTED());
-    resolver_.AddBuiltin(tflite::BuiltinOperator_QUANTIZE,
-                         tflite::ops::micro::Register_QUANTIZE());
-    resolver_.AddBuiltin(tflite::BuiltinOperator_DEQUANTIZE,
-                         tflite::ops::micro::Register_DEQUANTIZE());
-    resolver_.AddBuiltin(tflite::BuiltinOperator_SOFTMAX,
-                         tflite::ops::micro::Register_SOFTMAX());
-    interpreter_.AllocateTensors();
-
-    // The pseudo-random number generator is initialized to a constant seed
-    std::srand(kRandomSeed);
-    TfLiteTensor* input = interpreter_.input(0);
-    TFLITE_CHECK_EQ(input->type, kTfLiteInt16);
-
-    // Pre-populate input tensor with random values.
-    int input_length = input->bytes / sizeof(int16_t);
-    int16_t* input_values = tflite::GetTensorData<int16_t>(input);
-    for (int i = 0; i < input_length; i++) {
-      // Pre-populate input tensor with a random value based on a constant seed.
-      input_values[i] = static_cast<int16_t>(std::rand() % INT16_MAX);
-    }
-  }
-
-  void RunSingleIteration() {
-    // Run the model on this input and make sure it succeeds.
-    TfLiteStatus invoke_status = interpreter_.Invoke();
-    if (invoke_status != kTfLiteOk) {
-      TF_LITE_REPORT_ERROR(reporter_, "Invoke failed.");
-    }
-  }
-
- private:
-  const tflite::Model* keyword_spotting_model_;
-  tflite::MicroErrorReporter micro_reporter_;
-  tflite::ErrorReporter* reporter_;
-  tflite::MicroMutableOpResolver<6> resolver_;
-  tflite::MicroInterpreter interpreter_;
-};
-
 // NOLINTNEXTLINE
-KeywordRunner runner;
-
-void KeywordRunFirstIteration() { runner.RunSingleIteration(); }
+MicroBenchmarkRunner<int16_t> runner(g_keyword_scrambled_model_data,
+                                     tensor_arena, tensor_arena_size,
+                                     kRandomSeed);
 
 void KeywordRunTenIerations() {
   // TODO(b/152644476): Add a way to run more than a single deterministic input.
   for (int i = 0; i < 10; i++) {
-    runner.RunSingleIteration();
+    runner.RunSingleIterationRandomInput();
   }
 }
 
@@ -104,8 +54,8 @@ void KeywordRunTenIerations() {
 
 TF_LITE_MICRO_BENCHMARKS_BEGIN
 
-TF_LITE_MICRO_BENCHMARK(KeywordRunFirstIteration);
+TF_LITE_MICRO_BENCHMARK(runner.RunSingleIterationRandomInput());
 
-TF_LITE_MICRO_BENCHMARK(KeywordRunTenIerations);
+TF_LITE_MICRO_BENCHMARK(KeywordRunTenIerations());
 
 TF_LITE_MICRO_BENCHMARKS_END
diff --git a/tensorflow/lite/micro/benchmarks/person_detection_benchmark.cc b/tensorflow/lite/micro/benchmarks/person_detection_benchmark.cc
index bec12ad8642..49b2b5908c0 100644
--- a/tensorflow/lite/micro/benchmarks/person_detection_benchmark.cc
+++ b/tensorflow/lite/micro/benchmarks/person_detection_benchmark.cc
@@ -40,84 +40,24 @@ uint8_t tensor_arena[tensor_arena_size];
 namespace {
 
 // Create an area of memory to use for input, output, and intermediate arrays.
-constexpr int tensor_arena_size = 73 * 1024;
+constexpr int tensor_arena_size = 95 * 1024;
 uint8_t tensor_arena[tensor_arena_size];
 
-class PersonDetectionRunner {
- public:
-  PersonDetectionRunner()
-      : person_detection_model_(tflite::GetModel(g_person_detect_model_data)),
-        reporter_(&micro_reporter_),
-        interpreter_(person_detection_model_, resolver_, tensor_arena,
-                     tensor_arena_size, reporter_) {
-    resolver_.AddBuiltin(tflite::BuiltinOperator_DEPTHWISE_CONV_2D,
-                         tflite::ops::micro::Register_DEPTHWISE_CONV_2D());
-    resolver_.AddBuiltin(tflite::BuiltinOperator_CONV_2D,
-                         tflite::ops::micro::Register_CONV_2D());
-    resolver_.AddBuiltin(tflite::BuiltinOperator_AVERAGE_POOL_2D,
-                         tflite::ops::micro::Register_AVERAGE_POOL_2D());
-    interpreter_.AllocateTensors();
-
-    TfLiteTensor* input = interpreter_.input(0);
-    TFLITE_CHECK_EQ(input->type, kTfLiteUInt8);
-  }
-
-  void RunSingleIterationWithPerson() {
-    // Populate input tensor with an image with a person
-    TfLiteTensor* input = interpreter_.input(0);
-    int8_t* input_buffer = tflite::GetTensorData<int8_t>(input);
-    int input_length = tflite::ElementCount(*input->dims);
-    for (int i = 0; i < input_length; i++) {
-      input_buffer[i] = g_person_data[i];
-    }
-
-    // Run the model on this input and make sure it succeeds.
-    TfLiteStatus invoke_status = interpreter_.Invoke();
-    if (invoke_status != kTfLiteOk) {
-      TF_LITE_REPORT_ERROR(reporter_, "Invoke failed.");
-    }
-  }
-
-  void RunSingleIterationWithoutPerson() {
-    // Populate input tensor with an image with no person.
-    TfLiteTensor* input = interpreter_.input(0);
-    int8_t* input_buffer = tflite::GetTensorData<int8_t>(input);
-    int input_length = tflite::ElementCount(*input->dims);
-    for (int i = 0; i < input_length; i++) {
-      input_buffer[i] = g_no_person_data[i];
-    }
-
-    // Run the model on this input and make sure it succeeds.
-    TfLiteStatus invoke_status = interpreter_.Invoke();
-    if (invoke_status != kTfLiteOk) {
-      TF_LITE_REPORT_ERROR(reporter_, "Invoke failed.");
-    }
-  }
-
- private:
-  const tflite::Model* person_detection_model_;
-  tflite::MicroErrorReporter micro_reporter_;
-  tflite::ErrorReporter* reporter_;
-  tflite::MicroMutableOpResolver<6> resolver_;
-  tflite::MicroInterpreter interpreter_;
-};
-
 // NOLINTNEXTLINE
-PersonDetectionRunner runner;
-
-void PersonDetectionFirstIteration() { runner.RunSingleIterationWithPerson(); }
+MicroBenchmarkRunner<uint8_t> runner(g_person_detect_model_data, tensor_arena,
+                                     tensor_arena_size, 0);
 
 void PersonDetectionTenIerationsWithPerson() {
   // TODO(b/152644476): Add a way to run more than a single deterministic input.
   for (int i = 0; i < 10; i++) {
-    runner.RunSingleIterationWithPerson();
+    runner.RunSingleIterationCustomInput(g_person_data);
   }
 }
 
 void PersonDetectionTenIerationsWithoutPerson() {
   // TODO(b/152644476): Add a way to run more than a single deterministic input.
   for (int i = 0; i < 10; i++) {
-    runner.RunSingleIterationWithoutPerson();
+    runner.RunSingleIterationCustomInput(g_no_person_data);
   }
 }
 
@@ -125,8 +65,8 @@ void PersonDetectionTenIerationsWithoutPerson() {
 
 TF_LITE_MICRO_BENCHMARKS_BEGIN
 
-TF_LITE_MICRO_BENCHMARK(PersonDetectionFirstIteration);
-TF_LITE_MICRO_BENCHMARK(PersonDetectionTenIerationsWithPerson);
-TF_LITE_MICRO_BENCHMARK(PersonDetectionTenIerationsWithoutPerson);
+TF_LITE_MICRO_BENCHMARK(runner.RunSingleIterationCustomInput(g_person_data));
+TF_LITE_MICRO_BENCHMARK(PersonDetectionTenIerationsWithPerson());
+TF_LITE_MICRO_BENCHMARK(PersonDetectionTenIerationsWithoutPerson());
 
 TF_LITE_MICRO_BENCHMARKS_END
diff --git a/tensorflow/lite/micro/examples/hello_world/BUILD b/tensorflow/lite/micro/examples/hello_world/BUILD
index 4488c192abb..7287ff41ff5 100644
--- a/tensorflow/lite/micro/examples/hello_world/BUILD
+++ b/tensorflow/lite/micro/examples/hello_world/BUILD
@@ -37,7 +37,7 @@ tflite_micro_cc_test(
         "//tensorflow/lite:schema_fbs_version",
         "//tensorflow/lite/micro:micro_error_reporter",
         "//tensorflow/lite/micro:micro_framework",
-        "//tensorflow/lite/micro/kernels:all_ops_resolver",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/kernels:micro_ops",
         "//tensorflow/lite/micro/testing:micro_test",
         "//tensorflow/lite/schema:schema_fbs",
@@ -89,7 +89,7 @@ cc_binary(
         "//tensorflow/lite:schema_fbs_version",
         "//tensorflow/lite/micro:micro_error_reporter",
         "//tensorflow/lite/micro:micro_framework",
-        "//tensorflow/lite/micro/kernels:all_ops_resolver",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/schema:schema_fbs",
     ],
 )
diff --git a/tensorflow/lite/micro/examples/hello_world/hello_world_test.cc b/tensorflow/lite/micro/examples/hello_world/hello_world_test.cc
index 46976f30ecb..0a447440aea 100644
--- a/tensorflow/lite/micro/examples/hello_world/hello_world_test.cc
+++ b/tensorflow/lite/micro/examples/hello_world/hello_world_test.cc
@@ -14,8 +14,8 @@ limitations under the License.
 ==============================================================================*/
 
 // #include "tensorflow/lite/c/common.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/examples/hello_world/model.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
 #include "tensorflow/lite/micro/micro_error_reporter.h"
 #include "tensorflow/lite/micro/micro_interpreter.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
@@ -40,7 +40,7 @@ TF_LITE_MICRO_TEST(LoadModelAndPerformInference) {
   }
 
   // This pulls in all the operation implementations we need
-  tflite::ops::micro::AllOpsResolver resolver;
+  tflite::AllOpsResolver resolver;
 
   // Create an area of memory to use for input, output, and intermediate arrays.
 
diff --git a/tensorflow/lite/micro/examples/hello_world/main_functions.cc b/tensorflow/lite/micro/examples/hello_world/main_functions.cc
index d1c2cafe850..62db659374d 100644
--- a/tensorflow/lite/micro/examples/hello_world/main_functions.cc
+++ b/tensorflow/lite/micro/examples/hello_world/main_functions.cc
@@ -15,10 +15,10 @@ limitations under the License.
 
 #include "tensorflow/lite/micro/examples/hello_world/main_functions.h"
 
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/examples/hello_world/constants.h"
 #include "tensorflow/lite/micro/examples/hello_world/model.h"
 #include "tensorflow/lite/micro/examples/hello_world/output_handler.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
 #include "tensorflow/lite/micro/micro_error_reporter.h"
 #include "tensorflow/lite/micro/micro_interpreter.h"
 #include "tensorflow/lite/schema/schema_generated.h"
@@ -64,7 +64,7 @@ void setup() {
 
   // This pulls in all the operation implementations we need.
   // NOLINTNEXTLINE(runtime-global-variables)
-  static tflite::ops::micro::AllOpsResolver resolver;
+  static tflite::AllOpsResolver resolver;
 
   // Build an interpreter to run the model with.
   static tflite::MicroInterpreter static_interpreter(
diff --git a/tensorflow/lite/micro/examples/hello_world/train/train_hello_world_model.ipynb b/tensorflow/lite/micro/examples/hello_world/train/train_hello_world_model.ipynb
index 129e278f540..d0fb0eaa1b5 100644
--- a/tensorflow/lite/micro/examples/hello_world/train/train_hello_world_model.ipynb
+++ b/tensorflow/lite/micro/examples/hello_world/train/train_hello_world_model.ipynb
@@ -69,7 +69,8 @@
         "# Define paths to model files\n",
         "import os\n",
         "MODELS_DIR = 'models/'\n",
-        "os.mkdir(MODELS_DIR)\n",
+        "if not os.path.exists(MODELS_DIR):\n",
+        "    os.mkdir(MODELS_DIR)\n",
         "MODEL_TF = MODELS_DIR + 'model.pb'\n",
         "MODEL_NO_QUANT_TFLITE = MODELS_DIR + 'model_no_quant.tflite'\n",
         "MODEL_TFLITE = MODELS_DIR + 'model.tflite'\n",
@@ -1590,7 +1591,7 @@
         "colab_type": "text"
       },
       "source": [
-        "From the plot, we can see that loss continues to reduce until around 500 epochs, at which point it is mostly stable. This means that there's no need to train our network beyond 500 epochs.\n",
+        "From the plot, we can see that loss continues to reduce until around 200 epochs, at which point it is mostly stable. This means that there's no need to train our network beyond 200 epochs.\n",
         "\n",
         "However, we can also see that the lowest loss value is still around 0.155. This means that our network's predictions are off by an average of ~15%. In addition, the validation loss values jump around a lot, and is sometimes even higher.\n",
         "\n",
@@ -1655,7 +1656,7 @@
         "\n",
         "**3. Actual vs Predicted Outputs**\n",
         "\n",
-        "To get more insight into what is happening, we can plot our network's predictions for the training data against the expected values:"
+        "To get more insight into what is happening, let's check its predictions against the test dataset we set aside earlier:"
       ]
     },
     {
@@ -1666,27 +1667,37 @@
         "outputId": "372e169f-f97d-47ee-e64c-162b8ba4e38c",
         "colab": {
           "base_uri": "https://localhost:8080/",
-          "height": 281
+          "height": 318
         }
       },
       "source": [
-        "# Use the model to make predictions from our validation data\n",
-        "predictions = model_1.predict(x_train)\n",
+        "# Calculate and print the loss on our test dataset\n",
+        "loss = model_1.evaluate(x_test, y_test)\n",
         "\n",
-        "# Plot the predictions along with to the test data\n",
+        "# Make predictions based on our test dataset\n",
+        "predictions = model_1.predict(x_test)\n",
+        "\n",
+        "# Graph the predictions against the actual values\n",
         "plt.clf()\n",
-        "plt.title('Training data predicted vs actual values')\n",
+        "plt.title('Comparison of predictions and actual values')\n",
         "plt.plot(x_test, y_test, 'b.', label='Actual')\n",
-        "plt.plot(x_train, predictions, 'r.', label='Predicted')\n",
+        "plt.plot(x_test, predictions, 'r.', label='Predicted')\n",
         "plt.legend()\n",
         "plt.show()"
       ],
       "execution_count": 13,
       "outputs": [
+        {
+          "output_type": "stream",
+          "text": [
+            "\r200/1 [================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================================] - 0s 57us/sample - loss: 0.1560 - mae: 0.3435\n"
+          ],
+          "name": "stdout"
+        },
         {
           "output_type": "display_data",
           "data": {
-            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEICAYAAABcVE8dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO2de3gV1bn/P28ugCKoRCxWRKiXVhRF\nQexGoPFyAFureGlPFYs3jFGxWk8FPed4So+KJu0pnKqVpCKFA6L+tKK2WijVCJgtiEprG2pFDRJF\njaACKpck6/fHmkkmO7N39v36fp5nnr3ntmbNzN7vrPm+73qXGGNQFEVR8p+iTFdAURRFSQ9q8BVF\nUQoENfiKoigFghp8RVGUAkENvqIoSoGgBl9RFKVAUIOf5YjIsyJyabK3TRQRMSJyZDqOlWq85yIi\nc0XktjQc8zIRWZ3q42QDItIoImemoNy8+Q2mi5JMVyAfEZGdntl9gd1AqzN/tTFmcbRlGWPOSsW2\n6UJEBgPvAKXGmJbM1qZ7jDGV0WwnInXAImPMA6mtUfrJ53MrdNTgpwBjzH7udxFpBKYaY1aEbici\nJblgBHMJvaaKEh6VdNKIiJSLSJOIzBCRD4D5InKgiPxeRJpF5BPn+0DPPnUiMtX5fpmIrBaRXzjb\nviMiZ8W57RARWSkiO0RkhYjcJyKLItT9ZhHZIiLvi8gVIeu+IyKvich2EdksIjM9q1c6n5+KyE4R\nCYjIESLynIhsFZGPRWSxiBwQ4dhGRH4kIm872/9cRIo85/miiMwWka3ATBHp6Zz3uyLyoSPT7BPl\nufxWRO7wzJ8rIuudc3tLRCaKyJ3AWOBe55zudbb9hoj8SUS2icgbIvJ9TzllIvKUU85a4IgI5/us\niEwLWfYXETlfLLNF5COnrNdF5Lgw5VwuIhuce/y2iFwdsj6qcxORwc49KPHs6/2txXQ/PWWcIiIf\niEixZ9l5IvJX5/soEQmKyKfO/bpXRHqEKau9Ps58J8msm3vzbRFpcK7TeyLyk+7qnrMYY3RK4QQ0\nAmc638uBFqAK6AnsA5QBF2Clnz7A/wOWevavw74hAFwG7AWuAoqBa4D3AYlj2yDwC6AHMAbYjn2N\n9zuHicCHwHFAb+AhwABHes5rGLYBcbyz7SRn3WBn2xJPeUcC/+Jcg/7Yh8KcCNfQAM8D/YBBwD9D\nzrMFuB77xroPMBt4ytm+D/A0cFeU5/Jb4A7n+yjgM6euRcChwDdCr7Uz3xvYDFzu1ONE4GNgqLP+\nYeBRZ7vjgPeA1WHOdwrwomd+KPCpc70mAK8ABwACHAMcEqac72AfLAJ8C/gCOCmOc/O7h+3bdHc/\n8fwHfOr4FvAvnvn/B9zifB8BfNO5noOBDcCNIb+LI8PU+TL3+kZxb7YAY53vB7rXKB+njFcg3ye6\nGvw9QK8I2w8HPvHMe/9YlwEbPev2dX70A2LZFms0W4B9PesXEd7gPwjc7Zk/2vtn89l+DjDb+d7F\nWPhsPwl4LcJ6A0z0zF8L/Nlznu961gnwOXCEZ1kAeCeac6Gzwa9xz8OnTqEG5l+BVSHb1AA/xT5w\n9+IYVGfdLMIb/D7OORzuzN8JPOh8Px37wPsmUBTjb3EpcEMc59blHoZuE+l+Etng3+E5t07n7bPt\njcATIb+LaAx+2HvjfH8XuBroG8v1zMVJJZ3002yM2eXOiMi+IlIjIptEZDu2dXSA9zU3hA/cL8aY\nL5yv+8W47VeBbZ5lYFtA4fhqyPpN3pXOq/nzYmWpz4BK4KBwhYnIV0TkYef1eTv2YRN2e5/6bXLq\n5LeuP/bh9oojBXwK/NFZ3u25hHAYtgUaDYcDp7jHdI47GfuA7Y9tWUZ1XGPMDuAPwA+cRRcBi511\nzwH3AvcBH4lIrYj09StHRM4SkZccGeNT4Nt0XOdYzi0icd5Pl4eA80WkJ3A+8KoxZpNT7tFiJc4P\nnHJnxVCul0j3Buwb9reBTSLygogE4jhGTqAGP/2Epif9N+DrwCnGmL7AOGe5pLAOW4B+IrKvZ9lh\n3WzvXT8oZP1DWAnlMGPM/sBcOurvl451lrN8mHPOl9D9+YYe/33PvPcYHwNfAscaYw5wpv1NhyO9\nu3PxspnwWnvoeW0GXvAc8wBjzH7GmGuAZuwbVbTHBVgCXOQYn15YScse2JhfGWNGYKWeo4GbQ3d2\nDOjjWNnuK8aYA4Bn6LjOsZzb586n9/cywPM9nvvpnksD9uF3FnAx9rfkcj/wD+Aop9x/j1Du5xHq\nF+neYIx52RhzLnAw9i3o0Wjqnouowc88fbAG6lMR6YeVAFKK04Jah3Vw9nCMyncj7PIocJmIDHUe\nEqF17IN9Y9glIqOwf1yXZqAN+FrI9juBz0TkUHwMlg83i3VwHwbcADwS5tzagN8As0XkYAAROVRE\nJkR5Ll7mAZeLyBkiUuSU8w1n3Ych5/R74GgR+aGIlDrTySJyjDGmFfgd9nrvKyJDge76SzyDbZn+\nN/CIc144ZZ4iIqVYI7cLe31D6YHV1JuBFrEO+/HxnJsxphnrc7hERIrFOrq9D4t47qeXh7D3dBxW\nw/eWux3Y6dTtmghlrMe+KewrNjb/Ss+6sPfG+f1PFpH9jTF7neP5Xc+8QA1+5pmDdTR+DLyElR/S\nwWSstr0Vq6M+gu0v0AVjzLPYej4HbHQ+vVwL/LeI7AD+C08LyZGN7gRedF6nvwn8DDgJ6zT8A9YY\ndseTWGflemefeRG2neHU8yVHCliBfYuK5ly8570W6+ib7dT1BawRBvhf4EKxEVC/cmSY8VgZ5n2s\nnOY65wGmYeW0D7B+gvmRTtYYsxt7Xc6kc6u3L/aB9gm2ZbwV+LnP/juAH2HvxSfYh/BT8Zybs+wq\nrCHfChwL1HsOF8/99LIE61R+zhjzsWf5T5x673DO2fch7zAb6x/7EFiAI4E559rdvfkh0Oj8Viqx\n/428xI3YUAocEXkE+IcxJuVvGLEiIgb7Wr8x03VRlFxGW/gFivNKe4TzOj8ROBerXyqKkqdoT9vC\nZQD21bsMaAKuMca8ltkqKYqSSlTSURRFKRBU0lEURSkQslbSOeigg8zgwYMzXQ1FUZSc4pVXXvnY\nGNPfb13WGvzBgwezbt26TFdDURQlpxCRsL24VdJRFEUpENTgK4qiFAhq8BVFUQqErNXwFUXJT/bu\n3UtTUxO7du3qfmMlLL169WLgwIGUlpZGvY8afEVR0kpTUxN9+vRh8ODBiKQyKWz+Yoxh69atNDU1\nMWTIkKj3U0lHUZS0smvXLsrKytTYJ4CIUFZWFvNbkhr8PCYYhLvusp+Kkk2osU+ceK6hSjp5SjAI\nZ5wBe/ZAjx7w5z9DIG/H8VEUJRq0hZ+n1NVZY9/aaj/r6jJdI0XJLpYuXYqI8I9//CPidnPmzOGL\nL76IuE0kfvvb3zJt2rS4908mavDzlPJy27IvLraf5eWZrpGiZBdLlixhzJgxLFmyJOJ2iRr8bEIN\nfp4SCFgZ5/bbo5NzVO9Xsplk/z537tzJ6tWrmTdvHg8//DAAra2t/OQnP+G4447j+OOP55577uFX\nv/oV77//PqeddhqnnXYaAPvtt197OY899hiXXXYZAE8//TSnnHIKJ554ImeeeSYffvhhciqbRFTD\nz2MCgeh0+1j1/mDQSkTl5eoXUFJPKvxRTz75JBMnTuToo4+mrKyMV155hbVr19LY2Mj69espKSlh\n27Zt9OvXj1/+8pc8//zzHHTQQRHLHDNmDC+99BIiwgMPPEB1dTX/8z//k1hFk4wafKWT3r97N8yc\naSe/P5U6g5V04+ePSvQ3t2TJEm644QYAfvCDH7BkyRLeeecdKisrKSmxZrFfv34xldnU1MS//uu/\nsmXLFvbs2RNTfHy6UIOvUF5utf62NjutWAGrVvkb80h/Pm35K6nA9Ue5jYxE/VHbtm3jueee4/XX\nX0dEaG1tRUQ4+eSTo9rfGw7pjYO//vrruemmmzjnnHOoq6tj5syZiVU0BaiGX2CE00JFwB38rK0N\ndu2ChQu77h/OGey2/G+7zX6qL0BJFrH6o7rjscce44c//CGbNm2isbGRzZs3M2TIEE444QRqampo\naWkB7IMBoE+fPuzYsaN9/6985Sts2LCBtrY2nnjiifbln332GYceeigACxYsSKySKUINfo6QDKeV\n1yifdhpcc01Hq9z5jbdjDMyf3/V44f58GgaqpJJAAG69NTlvjkuWLOG8887rtOyCCy5gy5YtDBo0\niOOPP54TTjiBhx56CICKigomTpzY7rS9++67Ofvssxk9ejSHHHJIexkzZ87ke9/7HiNGjOhW788Y\nxpiEJ+BB4CPgb2HWC/ArYCPwV+Ck7socMWKEUSz19cbss48xxcX2s74+tn1nzTKmpsaY8eONKSoy\nxppzY0RseTU1HeUXF3deX1nZtSy/4ydSR6WwaGhoyHQV8ga/awmsM2HsarI0/N8C9wI+IgAAZwFH\nOdMpwP3OpxIF8TqtgkHbkt+zx5pwV7ZxP42x67Zuta31ujooK4Prr+/YZ/58mDLFlhfJWeu2/FXD\nV5TsJSmSjjFmJbAtwibnAgudB9BLwAEickiE7RUP8XaiWrjQRt242rwxUFQEJ5/ctTz3lbmiAq64\nwj4UwEo9dXXRSTZuGaAx/YqSjaQrSudQYLNnvslZtsW7kYhUABUAgwYNSlPVsp9AAObMgccfhwsu\nSKz13LOnLQvCt8anTIEFC7pGRUQTKaFhm4qSvWRVWKYxphaoBRg5cqTJcHWyhmAQbrzRGtFVq2DY\nsOiM6JQp8OCDsHcvlJTAlVfaZe6+4coIJ89EI9mkImZaUZTkkC6D/x5wmGd+oLNMiYJ4jWgg0CHH\nxKqr+/XSjabnbrJjphVFSR7pMvhPAdNE5GGss/YzY8yWbvZRHLLViPp1tFLnraJkL0kx+CKyBCgH\nDhKRJuCnQCmAMWYu8AzwbWxY5hfA5ck4bqEQrxFNlZ4eDFqH8Pz51qkbWna0OXwUJVMUFxczbNgw\nWlpaOOaYY1iwYAH77rtvXGVddtllnH322Vx44YVMnTqVm266iaFDh/puW1dXR48ePRg9enRMxxg8\neDDr1q1LOL4/KQbfGHNRN+sNcF0yjpWPRJOSIB4jGq0UFEtKBPchsmtXR/SPavVKrrHPPvuwfv16\nACZPnszcuXO56aab2te3tLS059SJhQceeCDi+rq6Ovbbb7+YDX6y0J62GSaVKQmiCeeM9fjuQ8Q1\n9iLZJTMpeUoK83ePHTuWjRs3UldXx9ixYznnnHMYOnQora2t3HzzzZx88skcf/zx1NTUALaz6rRp\n0/j617/OmWeeyUcffdReVnl5OevWrQPgj3/8IyeddBInnHACZ5xxBo2NjcydO5fZs2czfPhwVq1a\nRXNzMxdccAEnn3wyJ598Mi+++CIAW7duZfz48Rx77LFMnTrV7cCaMFkVpVOIxOqQjaU1Ho0UFOvx\nvf6E4mIbs++N/FGUpJPCWN+WlhaeffZZJk6cCMCrr77K3/72N4YMGUJtbS37778/L7/8Mrt37+bU\nU09l/PjxvPbaa7zxxhs0NDTw4YcfMnToUK644opO5TY3N3PVVVexcuVKhgwZ0p5qubKykv3224+f\n/OQnAFx88cX8+Mc/ZsyYMbz77rtMmDCBDRs28LOf/YwxY8bwX//1X/zhD39g3rx5STlfNfgZJhaH\nbDy/e68U5PewiNUh3N1DRDNmKkknBbG+X375JcOHDwdsC//KK6+kvr6eUaNGtac1Xr58OX/96195\n7LHHAJsc7c0332TlypVcdNFFFBcX89WvfpXTTz+9S/kvvfQS48aNay8rXKrlFStW0NDQ0D6/fft2\ndu7cycqVK/nd734HwHe+8x0OPPDAhM7XRQ1+honFIZvI7z7cwyIeh3A4f4J2ulJSQgrC1Lwavpfe\nvXu3fzfGcM899zBhwoRO2zzzzDMJH9+lra2Nl156iV69eiWtzEiohp8FRJsJMFJq4u7kzUipEaI5\nvt8xQpdFmzFTh1NUYiLZ+ZGjZMKECdx///3s3bsXgH/+8598/vnnjBs3jkceeYTW1la2bNnC888/\n32Xfb37zm6xcuZJ33nkHCJ9qefz48dxzzz3t8+5DaNy4ce3ZOp999lk++eSTpJyTtvCznFCJJLQ1\n3l2r2t2/rCz+RpLfMaDrsmgaYvoWoMRFBmJ9p06dSmNjIyeddBLGGPr378/SpUs577zzeO655xg6\ndCiDBg0i4FOv/v37U1tby/nnn09bWxsHH3wwf/rTn/jud7/LhRdeyJNPPtk+Zu51113H8ccfT0tL\nC+PGjWPu3Ln89Kc/5aKLLuLYY49l9OjRyUs1Ey6NZqYnTY8cXcrhWbM6UhoXF9v5cPvX1IRPb+x3\nbHdbv2N4l3nTKEdKodxdfZXCQNMjJ49MpUdWUkA0mn2kVnXo/lu3dmSzjITbCt+922bXvOkm/2OU\nlNiyjbE5e9xonWijfDScU1HSixr8LMbPOLq9XKHDwIZzukYyrpGiaerqrLF3x7idPRvuvdc+MLzb\nX3451NRYg9/a2vWBpKkXFCW7UIOfxYQaR7Cfe/bY7/Pnw/PPh29VhzOu3eno5eW2Zd/WZudbW/3f\nDsKlUe7uGNHIsRremd8YYzoNBq7EjomjM5Ya/CwiXIvY/X7XXTbVsUs0oZl+xrU7qSgQgPvug2nT\n7DY9e/pLL5Fa66kIIVXyg169erF161bKysrU6MeJMYatW7fGHM6pBj9LqK3tbGD9jFx5OZSWdrTw\n49XAo9HRKyps3v14c/wkotVrTv38ZuDAgTQ1NdHc3JzpquQ0vXr1YuDAgTHtowY/CwgG4brrbOZJ\nsPq5n5Fz89uHavixEq2OHk8knPctJV6tXh27+U1paWl7D1QlvajBzyCucXz33Q69HGzHqnBGLlnh\nyKkIa/aTYqKJCvKrmzp2FSX5qMFPI97WL3QYx6Iim3VSxBr7e+/NTSOXDCnGe43ch4U6cBUlOajB\nTxOhrd9LL+0wjq2tdpuSEussrajIbF3jJVEpJtoevWr0FSU+1OCnidDWL1gD5h1IxBgb/pirJCrF\nhF6jhQvh7bc7+gSoA1dREkMNfgqIJg3xlCl2WrjQ9lJtbc0PB2UivoHQXPvz59sw1LY2K3vlw/VR\nlEwi8QTvp4ORI0cad+SYXCJSDHk4LVo16g68juza2g5jf+aZMHOm5t9XlO4QkVeMMSP91mkLP8lE\nclxG6hGrxsriXgvX2IP9vOCCrg/OsjK48UYr+bjO7lz1fyhKOlCDn2Q0hjw5bN3akd6hqKjDt+F9\ngxLpSN7W1mb7Mgwbpg9PRQmHGvwkE6vjUiUJf8rLbY/j0Aen9w3KDWd1Vcm2NusT0eupKP6ohp9B\nNGdMZPwehqHX7Prr4Ze/tMa+tLQjc6deT6VQKTwNPxiEW26BV1+1lqC1tSPrmDcGsqQEjjoKvvtd\n+Oc/7XT00XZ69FH49FM4/ngbO/nqq7ZJefjhsHOnzYNw4om2rP79obkZhg+3ZbzxBuzYAR9/DL17\nw2GHWQu0Y4eNM+zbF0pKGPHRVra3tgCG1i+L+XLCADisLxx0kC3344876vP738MXX9hjHH00/N//\nWZ3DjiUCBxxgLd6XX9r6lJbacj77DN57zzaFv/gCDjzQnrcxtv5ffAEvvQR9+th9wJbVo4e9Ns3N\nVkBPsTjeXeI4F783qEmTOhy9v/mN5uBRlHDkXws/GIRx4zoS02QR3istIfPe5VnJ9OlQVZWSopP1\npqNvTIpSaC38urqsNPYuEuZ71vPzn8P27fFnbItAsrJjag4eRYlM/hn88nIrWWSZ0Tchn5BjBt8Y\nmDsXHngAVq5MqjVNZmSThrgqSnjyz+AHAtYgZZmGv2NPD95t2MERvM2n9KVfWQk9d2y15RhjA8kH\nDLD6frZo+J98Am++2fn6trTAt74F3/8+LFqUtFumLXNFST35p+FnMTkXghkMdh5TMZRRo2DNmrRW\nKRI5d30VJQVE0vDV4CuRCQahuhqWLvVff8ghNudBhru4qsNWUSyRDH5Ruiuj5BiBADzxBNTUWMkp\nlC1b4OqrYcgQmw8hAwSD9pmze3dnx6+iKJ1Rgx8FwaAdQDwYzHRNMkhFhTXuo0b5r29stIZ/xoy0\nVstt2a9YoVk1FaU71OB3g2tQbrvNfkZj9PP6AbFmDUyeHH79z38O11yTtpN3Qzq9WTWjkXPy+h4p\nShiSYvBFZKKIvCEiG0XkFp/1l4lIs4isd6apyThuOvCLEY9Eba0NYvnP/4z+AZFzLFpkJZ7DD++6\nzg3fHD3aRhOl+AK4IZ3FxTb3jl8K5VDieYgrSj6QsMEXkWLgPuAsYChwkYgM9dn0EWPMcGd6INHj\npguvQelOKggGbcZGd9CO3bvzWEuuqLAyTn297dksPr0K3nzTGv5LLon7MN21xAMBmDPHGu45c6Jz\n1Mb6EFeUfCEZcfijgI3GmLcBRORh4FygIQllZ5xYYsTr6jpyuIN9SOS9lhwIwAsvWIu8cKFt3Yey\neLE1/jGGcEYTeRMM2pz4e/bAqlXRpUfWFNZKoZIMSedQYLNnvslZFsoFIvJXEXlMRA7zK0hEKkRk\nnYisa25uTkLVkkMgALfeGp0h6dnTasklJXZAjoIJDQwE4P77Yfx4//Vr18KECTEVGU1LPJ7WuvsQ\nv/12Dd9UCot0OW2fBgYbY44H/gQs8NvIGFNrjBlpjBnZv3//NFUtPvykBteQ3HGH7exbkKMvLVsW\n3ugvXx5T6GY0closkpuXaB/iipJPJNzxSkQCwExjzARn/lYAY8xdYbYvBrYZY/aPVG42d7zSTj5R\nEAzC975n0zp4GTUKevWyaaIvvrjbDJzR9J7VHraK0kGqs2W+DBwlIkOA94AfABeHVOAQY8wWZ/Yc\nYEMSjpsxkpXdMa8JBKCpyco4y5d3LF+3rsPRUV0NTz8NDeHdPZGSoXkN/a23Jq3mipK3JGzwjTEt\nIjINWAYUAw8aY/4uIv8NrDPGPAX8SETOAVqAbcBliR43k6jTLwaWLbMyzuOPw777dk3RsGGDTTA3\ne3ZMGljoW9acOTaXnLbyFSU8mksnTlRGiINg0IZphmPgQJulNIoLetddNo7eHdu2uNi+OMQqsel9\nVPKNwhoAJU1o3vU4CATsyFnV1f7rm5pgzBhYvbrbi+t9yxKxhr+trXuJzWvgQX0xSmGhBl9JL1VV\ncMQR8OMf21z8obS12bEMevWKOJaut39EWVlHLH4kiS1UBrr0UvXFKIWFGnwl/VRU2OmSS2ynrFBW\nrrSfy5fDW2+FjeTxvmUNG9a9NBPqbAf1xSiFhWr4DqrlZohg0Da13ZG1RDpGJXMZOhRuuCHhjg1+\n4bSg913JL3QAlG7QuPoswH3i/vGPHS38UCZPTnhYRX2wK/mOOm27QePqswBXnykvh7Fj7c0IZfFi\nm6gtgZa+OtuVQkbz4RN/93wlBQQCNgvapEn+6//zPzWfsaLEiRp8NJlW1uEdVjGU5mY74MDQoXDe\neXEbfx0ARSlEVMNXspsZM8LH7YN18s6dm1AvXX3IK/mEDmKeINoazCBVVbalP2oUlJZ2XW+MHUvX\np7Uf7r55fTa7d9tRsvTeKoWAtvC7QVuDWUQwaFv7ofl4XEpL7WAsgUDE++au2727Yyzcnj313ir5\ngbbwE0CHw8siXG1/1Cj/9Xv3wtSpbDnvGh65Mcju3f73zfXZnHmmNfbelAyKks+owe8GjeDJQtas\nCTvIimloYMDSufxy7WjuaJtBUZH/fQsErJTTs6feW6VwUEknCrSzTpbiSjxPPdV5MGHA/VU/evh0\nBi2pinnwFL3nSq6iPW3RP3Be4xlA3QDiLHZ/2VJUBBddFHUvXfXbKLlMQfe0dW3Bgw9aPVf/wHmI\n2312xw5k8eIOQ++ub2uzvXRXr4bGxm6L057XSr6S1xq+21KrqVHHa0GwaBFMn44ceGCHsfeyaROc\neGK3MZjqt1Hylbw2+G5LzVWtRPQPnPdUVcG2bXDMMf7r16+3rYAIRl97Xiv5Sl4bfG9LrWdP2z8n\nnj+wdrzKQRoawodvfvklnHqqHWN3xoxOq9x7DXZgdDX2Sj6R907bRJ216sDLAy65BJ5+Gnbu7BLN\nA9i3gYYGvddKXlDQHa8CgcRaatrxKg9YtAg++8w6bUt84hQ2bIAJEyKmXNC3PCUfyHuDnyjqwMsj\nAgE4/XT/dcuX86PaY3ncnMdoCdLWBitW2BZ/ba39vO22buV/Rclq1OB3gzrw8oxly2wvXbFxPK6g\naYDejQ2c07aUF8ypXEVte8qFefNg1y59y1Nyn7zX8BUlHNtOmcABa5cDNmbf22GrlSLGsZp1pfYJ\nv3evXdejh8blK9lNQWv4ihKOmknLGFdUz1Ls6FreDlvFtPF/XMyqvadw2d5au1zgiivU2Cu5S8EY\nfHW6KaGUl8OrPQN8v/gJflE8Hegw+gBfo5FRrKWGq7mbGfTqBVOmZKSqipIU8j61AmhopeKP65+p\nq4Mx5VXI60fQWnktYlq7SDw3U820r/ye3q/fAIH4B1FXlExSEC18Da1UwtEpbLeigoa5q5hXXEkL\nxRhoT8ZWhHXqcvXVcPDB+qqo5CQFYfA1tFKJlmEVAY5bdT9LKlexffi4Ljl5DGCam2kbPbpLL11F\nyXYKJkontMetpktWoqK21rbqQ2h38I4bB3ffrT8iJWvQfPghqKavxIRj9EPTLrfn3heBuXOhQrV9\nJfMUZFhmpKgc1fSVmKiogPp6dh80sF3X9w60gjH2LaC2NmNVVJRoyEuD77bgb7sNTjsNzjsPrrmm\nw/irpq/ETCBAr+bN/L2mnncPH+e/TWUl9O1rk7UpShaSFIMvIhNF5A0R2Sgit/is7ykijzjr14jI\n4GQcNxyhSbCWLrVv3KedZo2+pktQ4mVYRYDDG19AJk/uss4Yg9mxw46uNXy4RvIoWUfCBl9EioH7\ngLOAocBFIjI0ZLMrgU+MMUcCs4GqRI8bifJy23oPxSvfJJpFUylwnNG1dh/QnzaKOun7BuAvf4HR\no7W1r2QVyWjhjwI2GmPeNsbsAR4Gzg3Z5lxggfP9MeAMEfEdhS5ZiLTnx2pH5RslmQQnVXHg7o8Y\nw2pe4wSgc09dwLb2v/Utbf5AIlwAABlBSURBVO0rWUEyDP6hwGbPfJOzzHcbY0wL8BlQloRj+1JX\nBy0t1pdWVGQHPqqshOef1xa9kjxc6TBIgBGs5/+YTAs+r5YrV9oRtjRuX8kwWZVaQUQqgAqAQYMG\nxV2O65R1wy7nzFFDryQf7++spASCly/i1L7jOKK6a9w+xkB1tU3PfP/9+oNUuuD2DSorg61bU9NH\nKBkG/z3gMM/8QGeZ3zZNIlIC7A9sDS3IGFML1IKNw4+3Qt4cKdqxSkkV/r+zCjgCmDULPvzQJtL3\n8pe/wNixsGqV/jCVdtzIwt277SicRUV2HO5kB5UkQ9J5GThKRIaISA/gB8BTIds8BVzqfL8QeM6k\nuMeXOmWVdOD7O6uogMZGeO65ro4ksOFj48fbeGHV9hU65EF3yGV38J1k9xFK2OA7mvw0YBmwAXjU\nGPN3EflvETnH2WweUCYiG4GbgC6hm4qSdwQCNh7Yj507bbzw6NEwYUJ666VkHa48WORY5KKi1ASZ\nFGRqBUVJK8EgXHstrF8ffpvx462+rxQsydLwNZeOoqQIbxI+6MZvNGMG/Pzn1oHrx/TpUJXSLipK\nAaAGX1FSgDcJX3GxletbWuz3K66wo2O5ht99MJxdFmTYr6+xzls/evSAb35TM3AWAKnK2KsGX1FS\nwF132XxNra0dvln37yQCvXrZKAvwyc76eq319m7bFv4A2uLPW1KZsbcgs2UqSqrxJuErLbXfvYbf\njbLwzc5aUWGF2unTwx+gulo7a+UhwSDMnGlDML2/iXSMu51VHa8UJZcIjcMHWLgQ5s+30o43ysLb\nEbBT5EVVlXXmLl/uf5Dq6o7tlJzHL96+pATWroWf/azjd5OqpI5q8BUlAQKBzn/MQMBq96HabMSO\ngMuW2Zb8nDn2qRBKdTUccYQOsJIHhMbbf+MbsHEjPPlkhxzotvhTYfBVw1eUbGLGDDuQyqefdl5+\nyCHw+efwta/Br3+tDt0cJRi0D333uV5cbA29+wDw+n7ivcWq4StKrlBVBZ980lXb37IFtm+38o8O\noJ6zBAI2gsvr6ykq6hiM6eqrUztGh0o6ipKNVFVZGefxx2H1avjii87rVdvPSVyHbGmpddi6yR1T\nlSwtFJV0FCVBuounDrc+6jjsSy6xefVDKSqyDwOVd3ICbyhmSQlcfnnnvhrJIpKkoy18RUmA7uKp\nw62PKQ570SJobvaP5LnlFtiwAXr3tnH96tjNWrzhuQCDBqX/Wa0avqIkgG+MfRTru9uvC8uWQX09\njBtnW/ZFRVYIXrnSPgwaG60AfMopyT1BJWl4+21kavQ9NfiKkgDd/YnDrY/rzx8IwAsvWBnnjjts\n5E4oa9fC0NAhpZVsIBCwev0ZZ2RuUCbV8BUlQVKu4YdjxowO520offvC6afbaB/V+LOCVKZT8KK5\ndBQlDQSDtqctpMYZ50s4h66LiM3Jr9p+xvHmXiouhttvt26XZKNOW0VJMaEdaubPh1/9yobbpXKM\nUhYtguuug3PPtVp+KMZYbf+ttzSEM8OEjrWdCQ1fW/iKkgTuugv+4z86Z8ssLrY9KGMZozQhmWfC\nhPA5eQCGD9deuhkmVSmRvWhPW0VJMeXltjONS1FRh7GH6MYodTXe226znzFnTVy2DGpqYPBg//Vu\nL10dSzdjZHqsbTX4ipIEAgFrzCsr7fTrX9sWfSxjlMYcqulHRQW8805HCKcfS5fCmDE2Z49SUKiG\nryhJIjRz5rBhsY1RmlSN1w3hDOfUbWuz4+wOG6YSTwGhGr6ipJlIOm5KNN5IY+n26wdTp6pDN4/Q\nsExFyRLSFYvte+Dq6s6J170MGGDH0tW4/ZxHnbaKkiUkRaePh0AAnngCXnzROhn69u28/oMPrLav\nqZfzGtXwFSVNBIPw7rs2XBMyFIvtOhr69g3fS1dTL6eEdIRkdoe28BUlDbhSzm9+Y2P0r7rKyjmQ\n+oGrfamqsvLNQQf5r6+uhiFDNJInSSQccpsk1OArShrwSjktLTY1LmTYCFRV2d6548f7r3czcJ54\nosbtJ0jGpLwQ1OArShrwy46ZLUaAZctsa/+AA/zXr19v4/bV6EdNMNj5zS0bUiODaviKkhYCASvh\nhGq4mc6t0k5VlZ3CZeBsa4OxY+Hf/k21fYdIWVD9IrH87n/aMcZk5TRixAijKPlOfb0xs2bZz1jX\nd7dv3NTUGDNggDE2gLPrdNRRKThoblFfb8w++xhTXGw/vZdj1iy7HOznrFnprRuwzoSxqxk37OEm\nNfhKoRPJqERalzQmTw5v9EWMmTSpYA1/JKOelnsTgUgGXzV8RclSImn8adH/Fy2CUaP81xlj4/bH\nji1IbT+SJu/KN7ffnsaOdVGiGr6iZCmRcuukLbf6mjU2H88jj9jwolBaW+H88+F3v8suy5ZCXO1+\nzpzwOZJC8yplC5paQVGymLTn3YlEba21chs2dF1XVATnnJP3qRncgW727rXpsOvqsu90NbWComQh\noaF7sZL23OoVFdDQYI16KG1tBZGaYeFC+1ZljP10h7TMFRKSdESkH/AIMBhoBL5vjPnEZ7tW4HVn\n9l1jzDmJHFdRcp1okqhlLNFad1RVwXvvhR9Lt7ra5u1ZsCBLKpw8Pvig63w2pEyIlkRb+LcAfzbG\nHAX82Zn340tjzHBnUmOvFDzROF2zpmOWH4sW2dG1jjnGf/2bb+alQ3fAgK7LsiFlQrQkavDPBRY4\n3xcAkxIsT1EKgmh6XmZL78ywuBJPTU3H0F5eWlvhllvgqKPyQuYJBm2LvqjI5kPq0cM+ALL2oexD\nQk5bEfnUGHOA812AT9z5kO1agPVAC3C3MWZpmPIqgAqAQYMGjdi0aVPcdVOUbCdUCvCTBnJGLggG\n4dJLbcs+HP3723z8WX0inXGvf1kZ/OhHsHu3XV5cbIexHDYs+2S3hAZAEZEVgM+LDP8BLPAaeBH5\nxBhzoE8Zhxpj3hORrwHPAWcYY96KdFyN0lEKiazV62Nlxgx46CH42tfg7behqanrNtOn50R6Bu89\nKSqykTkuInDnndZpnm0P5YSidIwxZxpjjvOZngQ+FJFDnIMcAnwUpoz3nM+3gTrgxDjPRVHyEq9e\nv3s3zJyZ/XqwL1VVsHmzHU/34ov9t6muzgmJx3tPWls7xjEAG5LpSmxpj5ZKgEQ1/KeAS53vlwJP\nhm4gIgeKSE/n+0HAqUBDgsdVlLzC1euLimyE44oVueEEjEhVVXin7i9+kfUn5/Wh9OxpJZzKSjtl\nY/x9NCTa0/Zu4FERuRLYBHwfQERGApXGmKnAMUCNiLRhHzB3G2PU4CuKB7c7/syZ1ti3tXU4AXPR\nsLTT0ACnnAJr13Ze3tYGV14JX34J++wDN95oncBZRNZkuEwi2tNWUbKIvNHyQ6mthVmz7BiP4WzO\n+PE2N38ayDbdPZkk5LTNFGrwlUIln40RwaB9jVm+3H99Goy+96FaUgKXXw5Tpth1+XDd1eArSgGS\ntQ+OYBC+9a3OYS9eDj8c/v3fUybx3HWX7SjV2mrn3Zh6Y+yy0DerrL2OYYhk8DVbpqLkGNEYoNpa\nmDbNGrCePbNMGgoEbBTPtdfa4RND2bTJjqW7eDHcfXfSK+46Y3ft6kjwv2ePXefNkePG3994Yx5J\nbOES5Wd60gFQFKUr0QyuUV9vTElJx1glRUUdI2OlZISsRKipMaZfv/ADrRQVGTN9etIPW19vTGWl\nMT162GvZs2fH9x497HxxsTGlpXasF+91zHbQAVAUJT+INgdPW1vHfHGxbalmZc6XigqbVH76dKut\nhNLWZuP2+/a1ry1JIhCA+++31+r22+H55zu+X3GFTf3vxt+7qndbm72OuYwafEXJIaLNwdOzp43p\nLymBe++1NjWrc75UVcGLL8K4cf6Gf8cOK/OcckrEYmJNOR0I2OvlXo9bb7UOXPcaFxd3VKeoyF7H\nXEY1fEXJIaKJDffbJhhM0whZieBq+8GgTbq2cmXXbdauhYMP9s3JE09Ia7h93OsXquFn5XWLATX4\nipJjRDN8Xug2oQ8BsC3hrIw8cQ2/X4ctgOZm+yawcmWnyvvJXd2dW7h9vNdv2LDcG+gkHGrwFSUH\niSdU0DVikVrCWRWCuGYNDB3qP6RiS4ttep90ktVgAoG4xvmNdp8FC+w2CxbkdqSOGnxFyTES7Y0b\nrlWblb18GxpsorXZs7vG7a9da6eaGrj5ZgJVVTGnQnDffCK14ON5c8hW1GmrKDmGnwEKBuGaa+zk\ndVj6OTHDOX6zdoStqipboenT4dBDYfDgzuuNsZE8EybEnblywQL4zW/8I5iyfiCaGNAWvqLkGKEy\nRFmZXeZ2Hpo/34YZgn+LPZzjNx5JJK1UVdkpGITTTusYjcRl+XK45BJYtCisNOW3vLsWfD4lUVOD\nryg5RqgBqqvrrHZ4W+fhDJmf4zdnDFsgYJ9ofpE8ixezc+3fuXXTr1ndGuj0oAsnWUXzoIvGUZ4L\nqMFXlBwk1ACVlna08L1GK9YWe84YNjeSZ8KELonYer+5nhWcyvscypIvL6auropAIHJETk486JKA\nGnxFyXFcY+Y6Hp2gFaAADNmyZVbGWby4fZEAxRgOo4npVNNc9x7cuihsSz6rIpNSjGbLVBQl9wkx\n+gZr+A0gIrYXbyDgO3B81kUmJYhmy1SUAqKQWqztLFpkI3geeggAcQZPb0/S8P3vwwcfEDjpJAJr\n1rTvlk8hl9GgYZmKkke4LdbbbrMGPzRMM69xB1DfvBkmT7ZJcNypqcl21lq71jblnURs+RRyGQ1q\n8BUljwhtsdbUZFl2zHSxaJGVce68s2sytr17bSK22tp2h+3tt+eHnNMdKukoSh4RbnCPfJcqfHFD\ncJYu9c/JM28ebN1KoLycwK2FcXG0ha8oeYTbYr36apsiOVekiljTGsdU3po1MGBA141eecUOpTh6\nNAwcWBCvQRqloyh5Sq44b5MdKRO2vBkzbDfk3r1h0KCunbY80Ty5TKQoHW3hK0qeEppXJlIrurbW\n9mFK4qBSUZPsHD5hy6uqgo8+gnfesVk4QzEGrrwyrz3dquErSgEQqRVdW2slIOjotFpRkb66JTuH\nT1TlTZkCc+d2Xb5hg52cDJxUVSVWmSxDW/iKUgBEakU//njnbUPnXZKts7u4foerroJLL01eeaGR\nN53qHwhAfb3V7v1wM3AefXR+tfbDjW6e6WnEiBHJH85dUQqU+npj9tnHmOJi+1lf37GupsaN57FT\nTU1s+6e6fikvv77emMpKY4qKOl8I7zR9enIrlEKAdSaMXdUWvqIUAJHizSsqrIIxfrz99JNzEtHZ\ng0E47zw7YmE4H0Es5cfzphGx/EAA7r/fTkVhTGJ1NZx4Yu639sM9CTI9aQtfUTKH2+itrLTf422B\n19cbU1KSvDeIROoR1X719cYceWT4ln64E8gi0Ba+oijREgxaR+fcuXYaOxZefz2+Hql1dTajgRc/\nH0FUujvxv2kEAjBnjnVcz5kTof6BALz5ph1dq7jYf5urr7bJ2nKRcE+CTE/awleUzDBrljEinRu1\nJSXx6erRtvDD7RvaKk95C99LqHMjdDryyOQ7G5IA2sJXFCVaysvtgCpe2trii48PBGz/pkmTYNSo\n8D4CP8Jlsoz3TSPmNwPXudGvn//6jRvh1FOtgyJXtP1wT4JMT9rCV5TMUV9vzKRJtkVcVJSayJlo\n6pCsyJ36emN69rRvLj17xlHW9OnG9OkTucWfJZE8aAtfUZRYCATgiSdg1Sq44w6re9fVpbchG9qa\nh8T6ARjT+TMmqqpg+3YbyhSO6moYMiQz3ZWjJdyTIJoJ+B7wd6ANGBlhu4nAG8BG4JZoytYWvqJk\nB6mOkU9HHWbNsvuC/Zw1K4HKTJ8euaWf4dY+KWzh/w04H1gZbgMRKQbuA84ChgIXiYhPIgtFUbKR\nZOe6SUYdFi6MrbWf1IFOqqpsL91x48JvU11tk7VlGQkZfGPMBmPMG91sNgrYaIx52xizB3gYODeR\n4yqKkj7CGctoOkDF00nKbx9vHYqLbdLL226LfnCXpA90EgjACy9Yp+7hh/tvU11tvd/ZFMIZrukf\nywTUEUbSAS4EHvDM/xC4N8y2FcA6YN2gQYNS+NKjKEos1NdbGcSVUqKRWKLdJtpy3W0rK5MozySL\n7mSeyZPTVhUiSDrdZssUkRWAz+gB/Icx5skkPHPaMcbUArVg8+Ens2xFUeLHHTzKJZrBv7vbJjSD\n55w5tlOWO1rX7t2d93HrEAzCggXJy66ZFKqq4IgjYNYs2LSp6/pHHrESUDrTkPrQraRjjDnTGHOc\nzxStsX8POMwzP9BZpihKjhKNJh4qw7z7bmf5xftA2L0bpk2DP/2pI4qmrQ3KyrqWG408k6rMnhGp\nqIDGRv9InpYW20N36NDMRvGEa/rHMhFZ0ikB3gaGAD2AvwDHdlemRukoSnYTKseE26ay0pjSUhsD\n36OHv3xTUtI1WWVRUVe5JtpjZjqqyEyfbsx++4XPwDlpUsoqRqqidETkPBFpAgLAH0RkmbP8qyLy\njPNAaQGmAcuADcCjxpi/J3JcRVEyT+iIWuG2Adi7t2NA9YULO9ZfeqnNg3/ffXYMXjdZZVGRnfe+\nObgSUHfO2myIKqKqCnbssBk4/Vi61I6lm+a4/YRGvDLGPAE84bP8feDbnvlngGcSOZaiKPnBBx90\n1e+nTLHyTF2dlXG2bu06Fm80fgNI/ghaCeFq9v/7v9DQ0HV9Y6OVet56Ky2ja+kQh4qipJQpU2De\nPNvKB3j2WRgwoKvx7u5tIVpD7mr8WTOAe0WFnWpr7YV4+eWu3X2rq+1nio2+plZQFCWlBAJ2bHAR\nO++mS47k9PVzusYSSx+N3JR2KipgzRo7Vq4f1dXQp09KO2yJCX3SZAkjR44069aty3Q1FEVJAn6D\nqIN/KzzSgOt5Q22tNeyffuq/fvx4WLYsrqJF5BVjzEi/dSrpKIqSEoLBzgbdT2bxM+SRtPrQMnOW\nigoYNsyOLtPa2nX98uX2gZBkiUcNvqIoScFrjKGjlV5SApdfbrX8W2/tvpxwWn3etfwDAZuO9Npr\nYf36rut/8Qs7kEAST1I1fEVREiY0ZHLhws6dqmpqEs97kxXhlskmEIDXXvMfaMWYpJ+kGnxFURIm\n1BiDbYW7jlo3Bj+WMWhDna5JzXiZbVRU2FjU6dNtJwQR6NUr6SepBl9RlIQJNcZuXP3VVyfPSCc9\n42U2UlUFq1fDnXem5CQ1SkdRlKQQzqGaN47WHCFSlI4afEVRlDwiksFXSUdRlJSTkeyVShc0LFNR\nlJSSd+GUOYy28BVFSSl5GU6Zo6jBVxQlpeR1OGWOoZKOoigpJeuyVxYwavAVRUk5oWPiKplBJR1F\nUZQCQQ2+oihKgaAGX1EUpUBQg68oilIgqMFXFEUpENTgK4qiFAhZmzxNRJqBTQkUcRDwcZKqkwly\nvf6Q++eQ6/UHPYdsIN31P9wY099vRdYa/EQRkXXhMsblArlef8j9c8j1+oOeQzaQTfVXSUdRFKVA\nUIOvKIpSIOSzwa/NdAUSJNfrD7l/Drlef9BzyAaypv55q+EriqIoncnnFr6iKIriQQ2+oihKgZB3\nBl9EJorIGyKyUURuyXR9YkVEHhSRj0Tkb5muSzyIyGEi8ryINIjI30XkhkzXKVZEpJeIrBWRvzjn\n8LNM1ykeRKRYRF4Tkd9nui7xICKNIvK6iKwXkXWZrk88iMgBIvKYiPxDRDaISEaTROeVhi8ixcA/\ngX8BmoCXgYuMMQ0ZrVgMiMg4YCew0BhzXKbrEysicghwiDHmVRHpA7wCTMqxeyBAb2PMThEpBVYD\nNxhjXspw1WJCRG4CRgJ9jTFnZ7o+sSIijcBIY0zOdroSkQXAKmPMAyLSA9jXGPNppuqTby38UcBG\nY8zbxpg9wMPAuRmuU0wYY1YC2zJdj3gxxmwxxrzqfN8BbAAOzWytYsNYdjqzpc6UUy0jERkIfAd4\nINN1KVREZH9gHDAPwBizJ5PGHvLP4B8KbPbMN5FjxiafEJHBwInAmszWJHYcOWQ98BHwJ2NMrp3D\nHGA60JbpiiSAAZaLyCsiUpHpysTBEKAZmO9Iaw+ISO9MVijfDL6SJYjIfsDjwI3GmO2Zrk+sGGNa\njTHDgYHAKBHJGXlNRM4GPjLGvJLpuiTIGGPMScBZwHWO3JlLlAAnAfcbY04EPgcy6lfMN4P/HnCY\nZ36gs0xJI47u/Tiw2Bjzu0zXJxGcV/DngYmZrksMnAqc42jgDwOni8iizFYpdowx7zmfHwFPYCXb\nXKIJaPK8HT6GfQBkjHwz+C8DR4nIEMdB8gPgqQzXqaBwHJ7zgA3GmF9muj7xICL9ReQA5/s+2CCA\nf2S2VtFjjLnVGDPQGDMY+x94zhhzSYarFRMi0ttx+uPIIOOBnIpcM8Z8AGwWka87i84AMhq8UJLJ\ngycbY0yLiEwDlgHFwIPGmL9nuFoxISJLgHLgIBFpAn5qjJmX2VrFxKnAD4HXHQ0c4N+NMc9ksE6x\ncgiwwIn6KgIeNcbkZGhjDvMV4AnbfqAEeMgY88fMVikurgcWOw3Qt4HLM1mZvArLVBRFUcKTb5KO\noiiKEgY1+IqiKAWCGnxFUZQCQQ2+oihKgaAGX1EUpUBQg68oilIgqMFXFEUpEP4/gbBOHjqKOvUA\nAAAASUVORK5CYII=\n",
+            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEICAYAAABcVE8dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO2deZgU5bW43zM9w+ISl9FEFLmQBPMLBgXXtMI4Ri9g4oKSmCgGN+6Iijdcr0FNYiQxEdAkkrgxo0NkLorJFUW90cSgTMDQRhFJTCAqGoQRVDK4RwZm5vv98VVNV/dU793T1d3nfZ5+umv76tTSp06dc77ziTEGRVEUpfypKrYAiqIoSt+gCl9RFKVCUIWvKIpSIajCVxRFqRBU4SuKolQIqvAVRVEqBFX4ZYyITBaRJ4oth4uIDBSRR0XkPRH53yLsf5aILHJ+DxGRD0UklEU73xGRu/MvYd8gIveIyI+KLUcyvNcqz+0G/tgLiSr8NBCRc0VktaMgtorI4yIypthypcIYc68xZlyx5fDwVeBTQK0x5mvFFMQYs8kYs4cxpivZeiJSLyJtcdveaIyZWlgJSwsRuUBEni62HEpyVOGnQESuBOYBN2KV1RDgDuCMYsqVChGpLrYMPvwb8LIxpjPXhgJ6fIoSbIwx+knwAfYCPgS+lmSd/tgHwhbnMw/o7yyrB9qAmcDbwFZgIvBl4GVgO/AdT1uzgAeAXwEfAGuAwz3LrwFedZatA870LLsA+CNwC9AO/MiZ97SzXJxlbwPvAy8CX/AcZwuwDXgd+B5Q5Wn3aeAnwDvAP4BTkpyPzwOtwLvA34DTnfk/AHYCu5xzerHPtqmOfyNwNfAXoAOoBr4IrHL292eg3rP+MOAPTlu/B24DFjnLhgIGqHam9wV+6VzDd4ClwO7Ax0C3I/OHwIGOnIs8+zndOdZ3nWP/fJzMVzkyv+cc2wBn2X7A/znbbQdWuufd59z8HNjsXLvngbFx5+3XzjX8wJHlKM/y0c65/MDZ//3AjxLs5zPAU9h76J/AvcDenuUHAw8690q7c04/D+wAupxz9K6zbiswNe4efTqDY1qUQMb1wKme6WpHniOc6f8F3nTO9wrgUM+697jHHi+PM88An/X8t38CbALeAuYDAzO9dkH6qIWfnDAwAHgoyTrfxSqdUcDhwDFYhelygNPGQcD3gbuA84AjgbHAdSIyzLP+Gdgbdl/gPmCpiNQ4y151ttkLq0AXicggz7bHAq9h30R+HCfnOKAOOMTZ/mzsHxbgVmfep4ETgCnAhXHtvoS9yW8CmkVE4k+EI+ejwBPAJ4ErgHtF5HPGmOuxb0m/MtaV0hy/fRrHD3AO8BVgb+c4f4N9uO2LVaxLRGR/Z937sIpkP+AG4PwE+wT4H2A34FBH9luMMR8BpwBbHJn3MMZsiTvmQ4DFwAxgf+Ax4FER6edZ7WxgAvYBdBhW0QD8N9Yg2N85lu9gFY4fz2HvMfe8/K+IDPAsPx2ryPcGHsEqYhw5ljrHty/23E5Kch4EmI19sH0eq+BnOW2FsErudewD8yDgfmPMemAaEHHO0d5J2s/kmBKxGHsfuIwH/mmMWeNMPw4Mx17HNdiHVjbMwf5fRgGfJfofhsyuXXAo9hMnyB9gMvBminVeBb7smR4PbHR+12MtxJAzvSf2pjjWs/7zwETn9yzgGc+yKuxbwdgE+14LnOH8vgDYFLf8AqIW/pewbxVfxGOJACGs5T3CM+8SoNXTxgbPst2cYzjAR56xWMvK2/5iYJbn+HyttnSOH2stX+RZfjXwP3Ft/A6r2IcAncDunmX34WPhA4OwVvw+PjLVA20+crrtXAf8Ok7mN3DeNByZz/MsvwmY7/z+IfAwjkWZ4b35Ds7bjyPPMs+yEcDHzu867FuLeJavIoGF77OficALzu8w1pKu9lmv517zzGsliYWfxjElsvA/i31b2c2Zvhf4foJ193au817O9D2kYeFjH3wfAZ/xLAsD/8j12hXzoxZ+ctqB/VL4iw/EWjwurzvzetow0cDgx873W57lHwN7eKY3uz+MMd1YK+JAABGZIiJrReRdEXkX+ALWeu21bTzGmKewVt/twNsi0iQin3C2r/E5hoM802962vmX89Mrs8uBwGZH7kRtpSLh8ccvx8YEvuaeD+ecjMEq8AOBd4y10r2y+HEwsN0Y804GcrrEXH9H5s0kOH/Av4ieu5uBDcATIvKaiFyTaCcicpWIrHcynN7FvpF5r338PgY49+2BwBvG0VIOic4DIvIpEblfRN4QkfeBRZ79HAy8bvIQg0nzmHwxxmzAunVOE5HdsG839zlthkRkjoi86si/0dksZbtx7I81bp733Fu/deZDBtcuSKjCT04E6yuemGSdLVjF4zLEmZctB7s/RKQKGAxsEZF/w7qDpmOzXPYG/oq1RFySvlIaY35hjDkSawEeAnwb66fd5XMMb2Qh+xbgYEfubNvyPX7Pcu8xbsZa+Ht7PrsbY+Zg3wz2EZHd42TxYzOwr4j4uSJSvabHXH/H1XUwaRyzMeYDY8x/G2M+jVVaV4rISfHrichYbBzobOxbyN5Y/3Qvt5oPW4GD4lxwic4DWLebAUYaYz6BdT+6224GhiQwgPzO00dYpelygPsjx2OCqFvnDGCd8xAAONeZdzL2ATLU3WUq+UTkAM+yf2KNsUM999Zexpg9IP1rFzRU4SfBGPMe1md3u4hMFJHdRKRGRE4RkZuc1RYD3xOR/UVkP2f9XPKHjxSRs5w/1QzsA+cZbADRYF+pEZELsRZ+WojI0SJyrOMP/wgbZOt23j5+DfxYRPZ0HixXZnkMf8JalzOd81QPnIb1LadLouP3YxHWyhvvWHYDnDTKwcaY14HVwA9EpJ+TRnuaXyPGmK1Yv+8dIrKPI3uds/gtoFZE9kogw6+Br4jISc65/W9H5lWpDlREThWRzzrK+D1s0LPbZ9U9se6pbUC1iHwf+ESq9h0izrb/6RzXWdg4UyL2xAZe3xORg7BGgcuz2AfIHBHZ3TnfxzvL3gIGx8Uu1gJnOf+bzwIX5+mYwN5T44BLcax7T7sd2Lfz3bAPsET8GThUREY5sYNZ7gLnTe0u4BYR+SSAiBwkIuOd3+leu0ChCj8FxpifYhXg97A352aslb3UWeVHWMXyF2zmyxpnXrY8DHwd68/8JnCWMWaXMWYd8FPsH/gtYCQ2KyddPoG9gd/BvtK3Y19LwQZXP8IGfJ/G/oEWZCq4MWYnVqmegrWQ7gCmGGP+nkEzvsefYH+bsdbcd4hem28Tva/PxQactwPXY7NYEvFN7JvO37GZTDOcffwd+1B/zXm197qXMMa8hLWCb3WO+TTgNOdcpGI4sAyrYCPAHcaY5T7r/Q7rTngZe+12kMR9FyffTuAsrL96O/bcPphkkx8AR2CV2G+86zrGwWlYH/cmrLvt687ip7DZQW+KyD+debdg40NvAQuJDZ5mfUyOLFux5+w4bOaRS4vT3hvYTLZExgLGmJexvvhlwCvYe9/L1Vi3zTOOe2gZ8DlnWbrXLlBIrGtPKSYiMgsbBDqv2LIUg0o/fkUpNGrhK4qiVAiq8BVFUSoEdekoiqJUCGrhK4qiVAiBLUC13377maFDhxZbDEVRlJLi+eef/6cxZn+/ZYFV+EOHDmX16tXFFkNRFKWkEJGEPanVpaMoilIhqMJXFEWpEFThK4qiVAiB9eErilKe7Nq1i7a2Nnbs2FFsUUqaAQMGMHjwYGpqalKv7KAKX1GUPqWtrY0999yToUOH4jOOjpIGxhja29tpa2tj2LBhqTdwUJeOoih9yo4dO6itrVVlnwMiQm1tbcZvSarwy5RIBGbPtt+KEjRU2edONudQXTplSCQCJ50EO3dCv37w5JMQDhdbKkVRio1a+GVIa6tV9l1d9ru1tdgSKUrwWLp0KSLC3/+efLiGefPm8a9//SvpOsm45557mD59etbb5xNV+GVIfb217EMh+11fX2yJFCV4LF68mDFjxrB48eKk6+Wq8IOEKvwyJBy2bpwbbkjPnaP+fiXo5Pse/fDDD3n66adpbm7m/vvtCJxdXV1cddVVfOELX+Cwww7j1ltv5Re/+AVbtmzhxBNP5MQTTwRgjz326GnngQce4IILLgDg0Ucf5dhjj2X06NGcfPLJvPXWW/kRNo+oD79MCYfT89tn4++PRKybqL5eYwNK4SlETOrhhx9mwoQJHHLIIdTW1vL888/z7LPPsnHjRtauXUt1dTXbt29n33335Wc/+xnLly9nv/32S9rmmDFjeOaZZxAR7r77bm666SZ++tOf5iZonlGFX+HE+/tbWpIrcw0IK32NX0wq13tu8eLFfOtb3wLgG9/4BosXL+Yf//gH06ZNo7raqsV99903ozbb2tr4+te/ztatW9m5c2dG+fF9hSr8Cqe+3vr6u7uhqgoWLLB/rETKPNmfTy1/pRC4MSnXyMg1JrV9+3aeeuopXnzxRUSErq4uRISjjz46re296ZDePPgrrriCK6+8ktNPP53W1lZmzZqVm6AFQH34FUQiP6h7/3Z3Q2enVeY7dlhrP55EAWHX8r/uOvut8QAlX2Qak0rFAw88wDe/+U1ef/11Nm7cyObNmxk2bBiHH344jY2NdHZ2AvbBALDnnnvywQcf9Gz/qU99ivXr19Pd3c1DDz3UM/+9997joIMOAmDhwoW5CVkgVOGXCLkGrbwK+cQT4dJLoxZ5ZycYYz+u8jcGfvnL3vtL9OfTVFClkITDcO21+XlzXLx4MWeeeWbMvEmTJrF161aGDBnCYYcdxuGHH859990HQENDAxMmTOgJ2s6ZM4dTTz2V4447jkGDBvW0MWvWLL72ta9x5JFHpvT3Fw1jTM4fYAHwNvDXBMsF+AWwAfgLcESqNo888kijWFatMmbgQGNCIfu9alVm2954ozETJxoj4qp1+3vgQGMaG2Pbnjgxuk5Vld02vi2//ecio1JZrFu3rtgilA1+5xJYbRLo1Xz58O8BbgN8nAAAnAIMdz7HAnc630oaZBu0amqC6dOjFrwXY2xb7e3WUnd97y++CEuX2nW6u6G21v5OFax1LX/14StKcMmLwjfGrBCRoUlWOQNocZ4+z4jI3iIyyBizNR/7L3eyCVpFInD55VbZexGxwVmItuVN4WxttcvdIG57e3R+qoeOtw3vtKIowaCvsnQOAjZ7ptuceTEKX0QagAaAIUOG9JFopcH559vvKVPSU6StrVZpe6mqgv79Yd48q8j9LPH6ertO/MMlnYeOpmwqSrAJVFqmMaYJaAI46qijTIrVK4J4JTplSnrbuYq7o8Mq+iuvhL33Tu1uSeSaScdlU4h8aUVR8kdfKfw3gIM904OdeUoKslWiufjUE/XSTdV7N9/50oqi5Je+UviPANNF5H5ssPY99d+nRy5KtJA+db9OVhq4VZRgkxeFLyKLgXpgPxFpA64HagCMMfOBx4AvY9My/wVcmI/9VgK5KNFC+dTd7J+uLus28rabbg0fRSkmoVCIkSNH0tnZyec//3kWLlzIbrvtllVbF1xwAaeeeipf/epXmTp1KldeeSUjRozwXbe1tZV+/fpx3HHHZbSPoUOHsnr16pzz+/OVpXNOiuUGuDwf+yo30ilHkK0STccdlGk5hPjsn44O9dUrpcfAgQNZu3YtAJMnT2b+/PlceeWVPcs7Ozt7aupkwt133510eWtrK3vssUfGCj9faE/bIlLocgSp6uJns//47J9QSH31Sh9QwBreY8eOZcOGDbS2tjJ27FhOP/10RowYQVdXF9/+9rc5+uijOeyww2hsbARsZ9Xp06fzuc99jpNPPpm33367p636+npWr14NwG9/+1uOOOIIDj/8cE466SQ2btzI/PnzueWWWxg1ahQrV65k27ZtTJo0iaOPPpqjjz6aP/7xjwC0t7czbtw4Dj30UKZOnep2YM2ZQGXpVBqZBmQztcZTuYOyCQjHZ//cdpta90qBKWC+b2dnJ48//jgTJkwAYM2aNfz1r39l2LBhNDU1sddee/Hcc8/R0dHB8ccfz7hx43jhhRd46aWXWLduHW+99RYjRozgoosuiml327Zt/Md//AcrVqxg2LBhPaWWp02bxh577MFVV10FwLnnnst//dd/MWbMGDZt2sT48eNZv349P/jBDxgzZgzf//73+c1vfkNzc3NejlcVfhHJJCCb7T3vuoNcA8mr+LMJCCd7iGi1TKUgFCDf9+OPP2bUqFGAtfAvvvhiVq1axTHHHNNT1viJJ57gL3/5Cw888ABgi6O98sorrFixgnPOOYdQKMSBBx7Il770pV7tP/PMM9TV1fW0lajU8rJly1i3bl3P9Pvvv8+HH37IihUrePDBBwH4yle+wj777JPT8bqowi8imQRkc7nnEz0ssg0I+8UUtNOVUjAKkO/r9eF72X333Xt+G2O49dZbGT9+fMw6jz32WM77d+nu7uaZZ55hwIABeWszGerDLzLpVgFM5o9P5d5MVskynf0nat87P5NqmTqkopIR+a6PnCbjx4/nzjvvZNeuXQC8/PLLfPTRR9TV1fGrX/2Krq4utm7dyvLly3tt+8UvfpEVK1bwj3/8A0hcanncuHHceuutPdPuQ6iurq6nWufjjz/OO++8k5djUgs/4HjdJH7WeDLL2t22tjZ7AylR+/Hz581Lbx/6JqBkRRHyfadOncrGjRs54ogjMMaw//77s3TpUs4880yeeuopRowYwZAhQwj7yLX//vvT1NTEWWedRXd3N5/85Cf5/e9/z2mnncZXv/pVHn744Z4xcy+//HIOO+wwOjs7qaurY/78+Vx//fWcc845HHrooRx33HH5KzWTqIxmsT9aHjm9ksM33miXg/12yxnHb9vYmLi0sd9+3XUTte+dL2LMtGnJyyenklepHLQ8cv4oVnlkpQCk47dP5N6M37a93bpuUuFa4N4aPH7t19dDdbVt3xg7NOKUKan3oeUXFKV4qMIPMPHKsbbWjlQF0aqZiQKvqRRrooya1lar7Lu77eeWW2zqZXx1zXAYLrwQGhutwu/q6v1A0vILihIsVOEHGK9yrK2FK66wChzs8IPLl0eVvl9t+mTpk4n86PX10Xr4YBV5oreDKVNg4UL/h0qyfaRyx2p6Z/ljjIkZDFzJHJNFZyzN0gkIiTJX3Cya9nZwkgWA9MaNTZSBkypr5/bboaYmWj8/kdslWfJEtmPc6mDo5c+AAQNob2/PW+/RSsQYQ3t7e8bpnGrhB4BIxA4s7lrDruXupb7eKmHXws/F/53K3dPQACNHpmdlJ7LWs/XVa0398mfw4MG0tbWxbdu2YotS0gwYMIDBgwdntI0q/ADQ0mL95mC/W1r8XTStrXYZpD/ylR/p+NGzyYKLd8Vk46vXoG75U1NT09MDVelbVOEXCa9yTJd8piLnO605WW/eTOXSoK6iFAZV+H1EvIJ3lWN1NZxyinXXdHba73SHMQwS+XDFeM+RGyTWAK6i5A9V+H1AvPV7/vlR5djVBQ8/bBX9JZfk5qopJrm6YvzeEEB75SpKPlGF3wfEW79gFdiOHTaH3c1jHzKkdBVarq4YvzeETZui50gDuIqSO5qWWQDiUyzjC59NmWKV4yWXJB+gpNRItxCcH/HnqLbW9jVwM/d0oBVFyR218PNMouCln/UbDlvlrz7q3ueotTXa70AELrrI//yoj19R0kcVfp5JFLxMlLGig35H8Z6LF1+M9vY1BkaPjq7nrQI6Y4ZNZQ2FbAmIhoY+F1tRSgZV+HlG88jzQ3t7tMRDVZWdhtg3KJFo8bbubju4+siR+gBVlESoDz/PZDJWgw4Ekhh37NxQKLa8g/cNqrvbKn2Xri6YNUvPp6IkQoJaz+Koo44y7ujv5YgOBJIaP/98/Hm74gr42c+ilr5b/0fPp1KpiMjzxpij/JaVp0unqQmWLIFRo+Dll+GFF2D33eFb37JO3kSaxC/619QEzc1w4IEwc2Z0uKdEZShbWuDNN+GAA2KT6l2ZJk2ChgZeaYnwsx0t/D+zjoEf72DgZcPhk9t6lvu2uX07/POfcMghUVkSyR6JwE03RY999Gh45RV7HKecYueDnf/447BlC1x8sfWJXHMNvPYanHsuzJ2b32vjQ6LTmW4V0IkTrWW/bJm1+jWFU1ESkGhklGJ/sh7xqrHRTW33/8yc2XsYqURDS8W3VV1t5/mtu2qVMf36xa7fr5+d39houqHnY2bONF01/WPmdXu3a2xM3Kb7qalJLPuqVXZ5svOQ6CMSOz15cnbXIU3SGdWrL9tRlFKHJCNelZ8Pf8mS5MsffLB3Gk2iWr7xbXV22nl+63rzCF127YLWVt5ptu247uaP73uQqs6diDNPPMti9uvXZlzbvrIn2y4V8S6+++4rqFM82zLK8RRpnGtFKSnKT+FPmpR8+Vln9e7tFN/rx40QxrdVXW3n+a3r1i/2UlMD9fVEDrTtuKr0hU87MqQ6Br8249r2lT3ZdqnwG5SipaVg0eVEpz4bcun4pSiVQHkGbQPmw49E4H/qmjijcwkPV0/imysaCOOsu26drR8wfDhsC4gP/7LL4M9/tsv79YvWfqiutuMa5rngj3aeUpT8kSxoW54KP4CUnFJzBd60Ce66yyp8sG8AoRAccYR9QASkp1PJnV9FKRCq8JXscfMg3Spm8dTVwZw5RdWymuKqKFGSKfzy8+Er+cWNhrqV3uJZsQLGjrWuryLR0mKfR7kGfhWl3FGFnwLtDYtV+nfeaTXpxIm9l3d12boGRThJkQgsWBB9+aiu1nIWipIIVfhJcF0F111nv9PRZ2X9gAiH4aGHbMC4Ku7W6e62HbaGD4err+4zkVpbY8MLF16YnjunrK+ToiQgLwpfRCaIyEsiskFErvFZfoGIbBORtc5naj72W2gyzRFvaoITToDvfS/9B0RJMncuPP20tfZDIav8Rax7Z8MGmx103nl9Ioo3rXPAgPSGh8zmQa4o5UDOCl9EQsDtwCnACOAcERnhs+qvjDGjnM/due63L8gkRzwSsV6NXbussdvRUea+ZNfaX7kSfvQjGDQodvl999knYA5mdDpWeDgM8+ZZxT1vXnrWfb46eylKqZGPWjrHABuMMa8BiMj9wBnAujy0XVQyGbavtTVavx0qaIQmt+DNu+9ay97FGPsENCar1Jl0M28iEVsTf+dO++xJpzyylrBWKpV8uHQOAjZ7ptucefFMEpG/iMgDInKwX0Mi0iAiq0Vk9bZt2/IgWu6k23vTLedbVWUDh7fdVmGpgXPnwuTJ1rUjYnv6dndnbUana4VnY61rGQalUumrapmPAouNMR0icgmwEPhS/ErGmCagCWwefh/JljF+nXxyHcS7LFi0yFr13uGosjSj07XCs7XWdaQxpRLJueOViISBWcaY8c70tQDGmNkJ1g8B240xeyVrN6gdr7STTwYkK0GRxpMx3d6z2stWUaIUuh7+c8BwERkGvAF8Azg3ToBBxpitzuTpwPo87LcoJBqzVvHBz4zO4ImZygr3Kvprr82b1IpStuSs8I0xnSIyHfgdEAIWGGP+JiI/xNZlfgT4TxE5HegEtgMX5LrfYqEBvxzJ0xMz/rkxb54d91atfEVJjNbSyQJ1IeSAn4UPGZ/Q2bNtHn1Xlw2Uh0I2Rpypm02vpVJuVN4QhwVGA345EB/dhqyCIt43LZHooOapXhq8Cj7LXStKyaIKX+l7vE/M2bOjLp6ODjs47axZKTWv97mRbkJQ/MvF+edrPEapLFThK8XFNdU7OqyJvmwZ/OEPaQ204n1ujByZ2jUTHz4AjccolYX68B3Ul1tEIhFr1S9bFu2uLGKd84MGwbnn2o5dedhNHsIHihJodACUFGhufQBINdDK5Mm2Y1cedqMKXilndACUFGgxrQCQaqCV++7LS1lLHehcqWRU4ZNZVUylgHgHWqmri11mjD6JFSVHVOGjxbQCRzhsA7duMTaAgQOjT+I8jF6iA6AolYj68JVgE+90z0PARWM2SjmjPvwcUEuwyMQ73b0Bl44Om4B/6aW+FyjRtYtvYtYsvb5KZaAWfhLUEgwg7kVx8/a9HH64jQGEw0mvXXwTVVV2LAO9vko5oBZ+lmj2TgBxAy4nnxz177v8+c90jxlLy6URWloSXztvE1VVsSUZFKWcUYWfBM3eCSjhsPXD1NT0WiTdXRw5/2LWNUcIhRJfO7eJ/v31+iqVg7p0UqAddQJMJAKXXQZr1/bMcu/mDvrxi4mtdB0TTnrtchyjRVECR8X3tNU/b5lz3nlw770YwHXydCFsm3gJBxwzJOMLr7EbpZSp2PLIkQi0tMCCBdaXq3/eMmXRIqirQy67DNPVBYCEqjngsQXwaOYXXkc1U8qVslX4fqVZ9M9bxjQ0wMiRSEsL4Fj6d92VldbWUc2UcqVsFb5rpbnKXkT/vGWPt15yJAILF/bW2mn49+LHaFEDQSkXylbhe6206uq0yqsnRGMAJYif1vY6531uivjrrNdaKTfKVuHny0rTAF4JE6+1vc75ri6YP98GeFpbiRDW66yUPWWr8CE/VpoG8MoI97UvPrAzYQJm1DR27pzbc51bWnq/HOhbnlLqlLXCzwcawCsj3Ne+GTPg2Wej899/n/CKm5gdgmtDc6mujs3smjcvdsxctf6VUkV72qZASyeXGeGw1eD9+wPRjloA/7nvvSw7aTY/PCXS4/XZuROam+1LgZbYUEqdiuh4pSi9iER477Jr+MTaFT2zTKiaKgzdVdXc3XUh93RPYXWNfcLv2mXX6ddP3XpKsNHiaYoSTzjMHWf/gZuYySt8lhXUQbeBri5kVwdTu+fzNMfTvOs8nL5ciMBFF6myV0qXilD4WtNe8aO+Hn4wcC4jQq9wfb85mH79ME4FzipAMJzHvdxoriYUggEDbBanopQqZR+01bRKJRGxqbthQjzJmhktjHp2PmB76xrgsr3u498PeZ9BB8IgpgB6AymlSdkrfE2rVJIRm7obpmNemMVjPuDc7nt7Arp7frCVI5yHAI8t0JtIKVnK3qWjNe2VTAiH4dNPLyJSN5Mdgz+L1NXFjKxldu7ktSmzeLFJ/YNK6VERWTp+42BrJxolLSIROFUdAAMAABmaSURBVPFE6Ojosfi7qGIn/Xm18UlGNugNpASLii2P7BJfU0t9+krahMOwfDm0tLDl/9bwqbbVVNONYSe7mlvgBVudM+tCTYrSh5Slwk9mwatPX8kYx2LYPjrCPpechGEnXVRz+PN3w7Oddp3mZvjDH/RmUgJN2Sn8eAv+iivsCHiTJtmS6VoqQcmWkQ1hXuRJ2pe08rndNjHo4cbowl274NxzYcIEtfaVwJIXH76ITAB+DoSAu40xc+KW9wdagCOBduDrxpiNydrM1oc/ezZcd5214EWiNbIAGhut0lcfvpIzkQiMHYvbK8u9zQRshsAdd9ibTVH6mIL2tBWREHA7cAowAjhHREbErXYx8I4x5rPALcDcXPebiPp6+38T6b1syRL7HQ7DtdeqsldyIBzm1f++gy6kZyzdnluuqwsuvRSamoonn6L4kI+0zGOADcaY14wxO4H7gTPi1jkDWOj8fgA4ScRPJecHt+VQKHb+pEmF2qNSaUQiMPLWBi5lPp2EMNDzAWwq52WXWcWvXbyVgJAPhX8QsNkz3ebM813HGNMJvAfU5mHfvWhthc5O68oxBiZOhHHjou4cRckHbvD/LhqoYyXzmcZKqespzQBYS7+xEerq4Nhj1eJXik6ggrYi0gA0AAwZMiSrNuKDsjNnqutGyT/e++yF6jCjLgwzagpUvdgE06fHWh2dnbb+/rPPwooVsGhRscVXAogbW6ythfb2wsQY86Hw3wAO9kwPdub5rdMmItXAXtjgbQzGmCagCWzQNhthdABqpS9IeJ+FG2DkSDtk1oIF9ong5d57rcWvr5uKBze7sKPDegOrquyQDfnuJ5QPhf8cMFxEhmEV+zeAc+PWeQQ4H4gAXwWeMgXs4qsDUCt9QcL7zF0wZQrcdBMsXRq7vLm5cCacUpK4LkK3ikd3d2H6CeXsw3d88tOB3wHrgV8bY/4mIj8UkdOd1ZqBWhHZAFwJXJPrfhUl8ITD8NBDMHly7PwXXrC5wyeeqEFdBYi6CKscjVxVVZh+QhVRS0dRik5Tk80L3m03ePTRnvx9AKqrrW9frf2KJl8+/GR5+KrwFSUHMv6Tus7ajz+OnT9xon0bUJQcqfjiaYpSCBIF2ubN6638o727w4SffBLOPhva2qKNbdkSfQtw64AoZU0xevyrwleULPELtHV02KzM7u5oNVaIr9AaJnzddXDJJdHGhg+PTj/xBDz+uOYUlzHFqtqrCl9RssQNtHkt/Koq6573ZlmAT4XWax0L3rXo3bofLkuXwu9+p/W7y5BIBGbNit433vuk0Ba/KnxFyRJvLr7rw6+thRkzeldj9a3Q2tAQ67p54onYHezYYVM7v/1tdfGUCX5uwH794N134YQTrFFQiPx7F1X4ipIDfrn4I0f2ttRSdgZ0FXpzs03bdHvqbtgQdfWo0i954t2ARx0FF18Ml19uLznYh0GhxunQLB1FCRqRiLXsN2yIzhs3zrp4lJImErEPfbcDdv/+cOGFNl7vPgRqanIbS6eg5ZEVRckz4bB143jRUq9lQTgMF10UrejrWvX9+1v3TnU13Hab+vAVpbJoiAvqet05mr5ZsridqmtqrL++Xz/7MjdlSt+kaKpLR1FyIJ1c6kTrZJWH3dQUm845caKmb5YI3lTM6mrryinEaJja8UpRCkA6udSJ1sk6D1vTN0sWN2DrVtUYMqTvL5n68BUlS7x/YG8udTrrpLOtL36+/J07bVXO8eN1kJUA4/bbCIUKUxgtHdTCV5QsiR9sx+8PnGiddLb1xZu+uWaNTd0UiZZgdnP51bcfOMJhW3bDDb8U44VMffiKkgN97sP3a3jpUjualsuIETB4sAZ1A0ZflVNQH76iFAhvx6tIxA50BbHBuEQDpeQ8UI/bQG1trMJft85+tCZPoPBz4/X1ZVGFryh5IL5DzS9/Cb/4RbTcQkEHuPKmcG7eDOvXR5ctXWrr799xh1r7RSZrN14eUZeOouSB2bPhu9+1LnWwbvVQyPaeTHeM0ryUy41P23QJhWDlSrX0i0xflERWl46iFJj6etuZxrXwq6qiyh5Sj1GaN/+ua8XfcENsvf3ubpvJ869/qW+/iBR7vG1V+IqSB8Jhq8xdH/7o0bZqZnxVxESv8Xn17zY02ApuJ5wAu3bZeVVVmsmjqMJXlHwRb725VTPT8eHn3b8bDtsKXO4TaM2a2MBuc3OBAwtKEFEfvqL0MQVL00xGvG+/piZ2WC5V+mWD+vAVJSAk89UX1L/rzeTZbTebueP6j1pa+n5wVaUoaGkFRelDsi6pkA8aGmzdnZkzo338q6thwQL43vesz19LM5Q1auErSh/gumtqa4ufix0zNuOmTdHRN7q74bLL7IhbhSjjWOH0RUpmKtSHrygFJt6NM29eH3XISlc4bzYP2E4E/foVrn5vBdJXZRVAR7xSlKIS78ZxlfyMGXDddVYRuANj9DnhsB1iqdrzsm+MzSedPx/GjIGrry6ScOVDUV15HlThK0qB8SuLGxQFAFjf/ooVMG2a7Q7sjr8H0Q5b6tvPmEjE9sB2y24UuzQyqEtHUfqEeP9tX77iZ4RbAa6xMVonArQCpw/JfPJ+1xf6aBhDTctUlOISn3LpjZsmUgCpgnwFCQK6gn7iE9ayd/FW4ISKV/qpHth+b3DXXlv8h7oqfEUpEsny7lMplIK/IcydC5/5jH8FziVLKl7hpyqFEYTKmH6oD19RAkgqH3+fxADcvP0ZM2Ln+w2zWGGk8sm7b3A33BAgdx1q4StKIEllIfapBentpev14QchsbyInH++/U6UuVrsyph+aNBWUQJKUXz46RLYqHPhaWqCyy+3CUypxjgoBhq0VZQAkqvCLqoFmcinVOYWfyRiOyN3ddnpjo7iDFWYLTkpfBHZF/gVMBTYCJxtjHnHZ70u4EVncpMx5vRc9qsopU7Rg7K5Eu9Tqq0NuMD5oaUlquzBdlmorbX59qXwnMs1aHsN8KQxZjjwpDPtx8fGmFHOR5W9UvEEIiibC/FRyfb2WIFbWqK9jsqY448PSI/pNMnVpXMGUO/8Xgi0AtoPW1FSEKigbLbE+5RcgUMhO4p7Z6ct2VAmNXkiEXjzTTt4mDF2SIERI+CPf8zTSGV9QE5BWxF51xizt/NbgHfc6bj1OoG1QCcwxxizNEF7DUADwJAhQ458/fXXs5ZNUYKOX+/bZNOBxxV40ya4665Y3wfAoEEwa1bJ5fC7nY+bm6M15kIhuOMOO6pZ0DxZyYK2KRW+iCwDDvBZ9F1goVfBi8g7xph9fNo4yBjzhoh8GngKOMkY82qy/WqWjlJJBN5nnwnuwezYEVuewWXmTNuxqwRIdCgi8OMf296zQXsw51Qt0xhzsjHmCz6fh4G3RGSQs5NBwNsJ2njD+X4N6/YZneWxKEpZ4vXZd3RYQzjo/uCEuP5975CKXm6+uWQOzr0u8c+tmpqomy0cDkbZhHTINWj7COB0P+B84OH4FURkHxHp7/zeDzgeWJfjfhWlrHB99lVVNr972bLSCAImJByGO++EyZN7LzMmgFFof7w9avv3h4kTbVHRoPvqE5Fr0HYO8GsRuRh4HTgbQESOAqYZY6YCnwcaRaQb+4CZY4xRha8oHlyjeNYsq+y7u0sjCJiSRYtg27Zo0TWImseucxwCG9RNp8hdKaE9bRUlQJSVL99LU5ONeh54oPXhg9WgO3fa3/37w/LlfXawQfO755OcgrbFQhW+UqmUszLqYfZs+O53o85xETj6aDjiiIJb+96HqjdrFMrjvKvCV5QKJNAPDncYKNfC91JTA3/4Q8GEnj3bdpRys0bdIXyNsfPi36wCfR590Fo6ilJGpFNUraUl2vcpkK6hcNgeREsLrFkDzz4bXbZrF5x9ttXKBcjZdwOxbqqlMdHnjvu7pcWKV1tre9KWi4tNFb6ilBDp1uDx5o0HtraZ21M3EoGxY2M7arW12bTOV1/Ne86+G4htaYEFC+xuq6ujFr63o3BVlf12x3Uv9SC6KnxFKSFSjbQUnzfuuisCXdssHLbdVr1lKF1uvhnefz/vfn33WTNlSvQhCL07CrtvAGAzp2pr8yZCUdARrxSlhEg10lL88ksu8a9tFrg0+IYGWLnSJrp7McYOqJ5mp4RIJLOabeGwPWfu+bj2WvsQcM9hKGQfmmCt/fb2tI8okKiFryglRKq88GTLS6IY20MPwdVXw09+Yk1q6O1YT+CTyialNdE27jmM9+EH8rxlgCp8RSkxUg184rc8/kEAAa7hPneutfS9kefqautwd6uXjR0Lc+bECJ/K3eVHom2853DkyGj/sFJH0zIVpcTINU0wmSUcuBREbwXOxsbYojZVVbZ8g5PJk08LP9N1goSmZSpKmZAP5ZPIqg2kYvNm8jQ1xSr87m6YPt2a4OFwVmUQvBk7icjmzSGoaNBWUUqIRCNhNTXB+PH224tfEDNR4DfQo2yFw3DVVb3nd3XFCJpt5cqFC21mjl9sOFWgvJRQC19RSgi/kbCamqKViN0aZQ0NiS32RJZw4EfZmjsXPvMZmDcPXnrJzuvf31fQZK6p+GWpLPiyKqBmjAnk58gjjzSKovRm1SpjbrzRfhtjzLhxbra4/YwbZ+ffeKMxoZCdFwrZ6UzbDiyJBF21ymycdqM5od8qEwoZM3Bg7CqrVtl53mV+80oZYLVJoFfVwleUEiM+C2fSpNjqw5Mm2e9sLPZUGUCBwU9Q55Xm4B07ecz04ySe5Lmd4RiL3c+av/baMrLgU6AKX1FKHLfczJIlVtm702XlikgHR5tXmS768TGPcCr3MJUx9dHSDH4PwcBlJhUQTctUFKU8cIMWH3+MV6tJ3Bi6XgUPAcxMypGcxrRVFKW0yLS8QNngvtLsuy8CPR8efNAud05MmEhPJk+gM5MKgLp0FKWM8GbmhEJw0UWBHT2wMITDMHUq3HRTdN5ZZ9kTc8IJtqeup95+4DOT8owqfEUpI7wWa1eX7Zy6cGF5uCrSxnXfPPigVfZz58KZZ0bLMuzaBRdfDM3NhMPhiopzqMJXlDIi0eAepdw7NCvmzo2to79lS+zy9euhrg5uv51wQ0PFnBv14StKGeG6sS+5xPZJKpXeofmOO/Rq7+KLe6/U2Wlr8J95Jlx6aUUEPTRLR1HKlFJJN8x3DZ+E7TU1QXMzrF4dLb3sJa4YW6miWTqKUoHE15VJZEVHItbALZaRm+9MmYTtNTTAn/5klXpNTXRkE5fubpg2rXdBojJCffiKUgEksnojETjxRDteK9iS833t7893pkzK9hoaokXum5pirX1j4PLLeypwlhuq8BWlAkhUIMyd77Jrl7/CL6R7KJ0Sxdm0Fy9v7DE4pRlGj7avNl6l392dcnStkiVRkZ1if7R4mqLkj0QFwlatMqZ//2jhtX79fOuRFby4WKH3kbT9VauMmTjRLqyqsiehf/+SraZGkuJp6sNXlArAtXpvuCE2KBoOw/Ll1nU9bZq/dZ+rjz0SsYkwxx6b2D2eyT6yyehJ2r47lu7KlfCjH9neap2d0ZWvuQaGD7dj7ZY6iZ4Exf6oha8oxWPVKmOmTbOfxsbsre9Vq4ypro4t39zY6L9eOvvI9k0go+28K7v1pd3PqFGBt/jR8siKoqRLJGJd165vv6YGbrsN2tszd2m3tlpj2cuSJb0zH9Pzu2c/3GA4bMdNcSuKJt3GK8wdd0BbW3TZ2rUwZgycfjrMnFly/n1V+IqixNDaGq1CAPb3Cy/YbMZMqa+H6upYpe/W648nvsS9X2ZRthk9kQjMmGG3W7kyjSQcV5h3342tywM2qLt0KTzySMnl7asPX1GUGOrrbR+kfBAOw4oVMHEiHHOMre2Trn5MZM37xSKyaSst5s6FyZP9l5Vi3n4iX0+xP+rDV5Ti0dho3dci/pk7fUE+M3fcbCQR+51xW24mj9ef735CIRvsCIhvH83SURQlExoarOvjxz+OWsN9XWM/3prPVQZjYr8zFuahh+wrSvzrj1uW9KSTAl+PJyeFLyJfE5G/iUi3iPjWbnDWmyAiL4nIBhG5Jpd9KorSN7ilGcDqsuuu63udli8ZWlutXjbGfmddvqGhAZ5+2vqoQqHofGPyUxeiwORq4f8VOAtYkWgFEQkBtwOnACOAc0RkRI77VRSljwjCqFDxMrS0ZGbtu8HevFQP9ebtT5tWUmVJc8rSMcasB5D4IkSxHANsMMa85qx7P3AGsC6XfSuK0jckyoxJp9xCNiUZ/LbxyhAKwS9/aTN/0q2uWZAB3d1MnilT/BsOYLnSvkjLPAjY7JluA471W1FEGoAGgCFDhhReMkVRUuKnLNMpaZzuOum065Vh0ya4667scvELonf9Gs53zec8kVLhi8gy4ACfRd81xjycT2GMMU1AE9h6+PlsW1GU7InXael0gEq1TrxOdDtGuaN1dXTEbuPKEInYYRsDPQ5ttj3ECkxKhW+MOTnHfbwBHOyZHuzMUxSlREmnA1S8G2bTJqusXb3n1YkdHTB9unXTuFk03d1QW9u73XTcM0X3pvidoKIL1TcuneeA4SIyDKvovwGc2wf7VRSlQKSjdN11broJHn3U9k/yDqju1Yki0Swal6oqW87Bi1dnutk78QTCmxJ/gsAKtWOH/T12LMyZ0+eC5aTwReRM4FZgf+A3IrLWGDNeRA4E7jbGfNkY0yki04HfASFggTHmbzlLrihKUUnXJ/6b31hlDr3dNOefb79Hj7alDzo6rGVfVWWTX7xvDukq8sB4U7wnaPbsqK8KbPfj44+HM87o05o8uWbpPAQ85DN/C/Blz/RjwGO57EtRlNLDzX/3UlvbW3lPmRI1iGtr/Qu1pavI8z2CVl6or7evMd5XGGNsTZ7HH7c1qvtA6WvxNEVRCkZ9vbXUXePWGGvJn39+b+XtHX83UVvpKPKCpGDmSjgMV13VuxAbRDsW9IHAqvAVRSkYrvKdNQuWLbPuGrfscjLl7RffzESRFywFMxfmzoXPfAZ+/nNYvz5q7VdX28GEd+2ybwFXXWXXLQBisiosUXiOOuoos3r16mKLoShKHvDzv0Pi/kpFD7oWmkgkdhDfxsZYd08mZUXjEJHnjTG+pW7UwlcUJe/EW+iJrPNMg64ByGzMD95XkEjEpjB5Ff7NN6dRtD9zVOEripIXXGVcWxsdbMTtUOUGYROlUnpJVsqhLC1/P//+q6/ag83zQarCVxQlZ7zKWMT66ru7bZrl5Zdb4zXXujeBSbcsBK5//+abrbL3Vt9Uha8oSpDwKuOqKtuzVsT+7uqKBmtzqXsTyHTLfNLQYN043teYPB+kKnxFUXImXhm7bpx4904u+iuQ6Zb5psAHqVk6iqLkhUQB1bIJtJYIybJ0VOEriqKUEckUvo5pqyhKwYlE+n5MXKU36sNXFKWglG06ZQmiFr6iKAUlCGPiKhZV+IqiFJS8DiCu5IS6dBRFKSgVkU5ZIqjCVxSl4ASyemUFoi4dRVGUCkEVvqIoSoWgCl9RFKVCUIWvKIpSIajCVxRFqRBU4SuKolQIgS2eJiLbgNez3Hw/4J95FKcYlPoxlLr8oMcQBEpdfuj7Y/g3Y8z+fgsCq/BzQURWJ6oWVyqU+jGUuvygxxAESl1+CNYxqEtHURSlQlCFryiKUiGUq8JvKrYAeaDUj6HU5Qc9hiBQ6vJDgI6hLH34iqIoSm/K1cJXFEVR4lCFryiKUiGUlcIXkQki8pKIbBCRa4otT6aIyAIReVtE/lpsWbJFRA4WkeUisk5E/iYi3yq2TJkiIgNE5FkR+bNzDD8otkzZICIhEXlBRP6v2LJkg4hsFJEXRWStiKwutjzZICJ7i8gDIvJ3EVkvIkUtEl02PnwRCQEvA/8OtAHPAecYY9YVVbAMEJE64EOgxRjzhWLLkw0iMggYZIxZIyJ7As8DE0vsOgiwuzHmQxGpAZ4GvmWMeabIomWEiFwJHAV8whhzarHlyRQR2QgcZYwp2Y5XIrIQWGmMuVtE+gG7GWPeLZY85WThHwNsMMa8ZozZCdwPnFFkmTLCGLMC2F5sOXLBGLPVGLPG+f0BsB44qLhSZYaxfOhM1jifkrKMRGQw8BXg7mLLUqmIyF5AHdAMYIzZWUxlD+Wl8A8CNnum2ygxRVNuiMhQYDTwp+JKkjmOO2Qt8Dbwe2NMqR3DPGAm0F1sQXLAAE+IyPMi0lBsYbJgGLAN+KXjWrtbRHYvpkDlpPCVACEiewBLgBnGmPeLLU+mGGO6jDGjgMHAMSJSMi42ETkVeNsY83yxZcmRMcaYI4BTgMsdl2cpUQ0cAdxpjBkNfAQUNbZYTgr/DeBgz/RgZ57Sxzh+7yXAvcaYB4stTy44r+DLgQnFliUDjgdOd3zg9wNfEpFFxRUpc4wxbzjfbwMPYd22pUQb0OZ5O3wA+wAoGuWk8J8DhovIMCc48g3gkSLLVHE4Ac9mYL0x5mfFlicbRGR/Ednb+T0Qmwjw9+JKlT7GmGuNMYONMUOx/4OnjDHnFVmsjBCR3Z2gP44bZBxQUtlrxpg3gc0i8jln1klAUZMXqou583xijOkUkenA74AQsMAY87cii5URIrIYqAf2E5E24HpjTHNxpcqY44FvAi86PnCA7xhjHiuiTJkyCFjoZH5VAb82xpRkamMJ8yngIWs/UA3cZ4z5bXFFyoorgHsdI/Q14MJiClM2aZmKoihKcsrJpaMoiqIkQRW+oihKhaAKX1EUpUJQha8oilIhqMJXFEWpEFThK4qiVAiq8BVFUSqE/w/pdTMly6MnnAAAAABJRU5ErkJggg==\n",
             "text/plain": [
               "<Figure size 432x288 with 1 Axes>"
             ]
@@ -1704,7 +1715,7 @@
         "colab_type": "text"
       },
       "source": [
-        "Oh dear! The graph makes it clear that our network has learned to approximate the sine function in a very limited way. From `0 <= x <= 1.1` the line mostly fits, but for the rest of our `x` values it is a rough approximation at best.\n",
+        "Oh dear! The graph makes it clear that our network has learned to approximate the sine function in a very limited way.\n",
         "\n",
         "The rigidity of this fit suggests that the model does not have enough capacity to learn the full complexity of the sine wave function, so it's only able to approximate it in an overly simplistic way. By making our model bigger, we should be able to improve its performance."
       ]
@@ -2822,7 +2833,7 @@
         "outputId": "bdc6e8f7-480d-4d3e-c20b-94776722360f",
         "colab": {
           "base_uri": "https://localhost:8080/",
-          "height": 851
+          "height": 297
         }
       },
       "source": [
@@ -2833,18 +2844,11 @@
         "\n",
         "epochs = range(1, len(loss) + 1)\n",
         "\n",
-        "plt.plot(epochs, loss, 'g.', label='Training loss')\n",
-        "plt.plot(epochs, val_loss, 'b', label='Validation loss')\n",
-        "plt.title('Training and validation loss')\n",
-        "plt.xlabel('Epochs')\n",
-        "plt.ylabel('Loss')\n",
-        "plt.legend()\n",
-        "plt.show()\n",
-        "\n",
         "# Exclude the first few epochs so the graph is easier to read\n",
         "SKIP = 100\n",
         "\n",
-        "plt.clf()\n",
+        "plt.figure(figsize=(10, 4))\n",
+        "plt.subplot(1, 2, 1)\n",
         "\n",
         "plt.plot(epochs[SKIP:], loss[SKIP:], 'g.', label='Training loss')\n",
         "plt.plot(epochs[SKIP:], val_loss[SKIP:], 'b.', label='Validation loss')\n",
@@ -2852,9 +2856,8 @@
         "plt.xlabel('Epochs')\n",
         "plt.ylabel('Loss')\n",
         "plt.legend()\n",
-        "plt.show()\n",
         "\n",
-        "plt.clf()\n",
+        "plt.subplot(1, 2, 2)\n",
         "\n",
         "# Draw a graph of mean absolute error, which is another way of\n",
         "# measuring the amount of error in the prediction.\n",
@@ -2867,40 +2870,17 @@
         "plt.xlabel('Epochs')\n",
         "plt.ylabel('MAE')\n",
         "plt.legend()\n",
-        "plt.show()"
+        "\n",
+        "plt.tight_layout()"
       ],
-      "execution_count": 16,
+      "execution_count": 42,
       "outputs": [
         {
           "output_type": "display_data",
           "data": {
-            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYIAAAEWCAYAAABrDZDcAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nO3deXwU9fnA8c+TzQVJIJAAARIIKIcg\nkEAAFxSDUgtCwVupBSkKgrVerUBrPWq1KtLW+itW8UBpsWi1Ug+sChJAiMopAnKpwQQ5QgghIXfy\n/f0xk7jkTshmSeZ5+8oruzOzM88scZ/93mKMQSmllHP5+ToApZRSvqWJQCmlHE4TgVJKOZwmAqWU\ncjhNBEop5XCaCJRSyuE0EahGJSLvi8hNjX2sL4lIioiM8cJ5jYicaz9+VkTur8uxDbjOjSLyYUPj\nrOG8iSKS1tjnVU3P39cBKN8TkRyPp62BAqDEfn6rMWZpXc9ljBnnjWNbOmPMrMY4j4jEAt8CAcaY\nYvvcS4E6/xsq59FEoDDGhJY9FpEU4BZjzMqKx4mIf9mHi1Kq5dCqIVWtsqK/iMwVkcPAYhFpJyLv\niki6iGTaj6M9XpMkIrfYj6eJyCcissA+9lsRGdfAY3uIyFoRyRaRlSKyUET+WU3cdYnxDyKy3j7f\nhyIS6bF/iogcEJEMEbmvhvdnuIgcFhGXx7YrRWS7/XiYiCSLyAkROSQifxORwGrO9bKIPOLx/F77\nNd+LyPQKx44Xka0iclJEUkXkIY/da+3fJ0QkR0TcZe+tx+tHiMhGEcmyf4+o63tTExE5z379CRHZ\nKSITPfZdLiK77HMeFJFf29sj7X+fEyJyXETWiYh+LjUxfcNVbaKA9kB3YCbW38xi+3k3IA/4Ww2v\nHw7sASKB+cCLIiINOPZV4HMgAngImFLDNesS40+BnwMdgUCg7IOpH/B3+/xd7OtFUwVjzGfAKeCS\nCud91X5cAtxt348buBS4rYa4sWMYa8fzI6AXULF94hQwFQgHxgOzReQKe98o+3e4MSbUGJNc4dzt\ngfeAp+17+zPwnohEVLiHSu9NLTEHAO8AH9qv+yWwVET62Ie8iFXNGAacD3xsb/8VkAZ0ADoBvwV0\n3psmpolA1aYUeNAYU2CMyTPGZBhj3jTG5BpjsoFHgYtreP0BY8zzxpgS4BWgM9b/8HU+VkS6AUOB\nB4wxhcaYT4C3q7tgHWNcbIzZa4zJA14H4uzt1wDvGmPWGmMKgPvt96A6/wImA4hIGHC5vQ1jzGZj\nzKfGmGJjTArwXBVxVOU6O74dxphTWInP8/6SjDFfGmNKjTHb7evV5bxgJY59xph/2HH9C9gN/MTj\nmOrem5pcAIQCj9v/Rh8D72K/N0AR0E9E2hhjMo0xWzy2dwa6G2OKjDHrjE6A1uQ0EajapBtj8sue\niEhrEXnOrjo5iVUVEe5ZPVLB4bIHxphc+2FoPY/tAhz32AaQWl3AdYzxsMfjXI+Yunie2/4gzqju\nWljf/q8SkSDgKmCLMeaAHUdvu9rjsB3HH7FKB7U5LQbgQIX7Gy4iq+2qryxgVh3PW3buAxW2HQC6\nejyv7r2pNWZjjGfS9Dzv1VhJ8oCIrBERt739SWA/8KGIfCMi8+p2G6oxaSJQtan47exXQB9guDGm\nDT9URVRX3dMYDgHtRaS1x7aYGo4/kxgPeZ7bvmZEdQcbY3ZhfeCN4/RqIbCqmHYDvew4ftuQGLCq\ntzy9ilUiijHGtAWe9Thvbd+mv8eqMvPUDThYh7hqO29Mhfr98vMaYzYaYyZhVRstxyppYIzJNsb8\nyhjTE5gI3CMil55hLKqeNBGo+grDqnM/Ydc3P+jtC9rfsDcBD4lIoP1t8ic1vORMYnwDmCAiF9oN\nuw9T+/8nrwJ3YiWcf1eI4ySQIyJ9gdl1jOF1YJqI9LMTUcX4w7BKSPkiMgwrAZVJx6rK6lnNuVcA\nvUXkpyLiLyLXA/2wqnHOxGdYpYc5IhIgIolY/0bL7H+zG0WkrTGmCOs9KQUQkQkicq7dFpSF1a5S\nU1Wc8gJNBKq+ngJaAceAT4H/NdF1b8RqcM0AHgFewxrvUJUGx2iM2Qn8AuvD/RCQidWYWZOyOvqP\njTHHPLb/GutDOht43o65LjG8b9/Dx1jVJh9XOOQ24GERyQYewP52bb82F6tNZL3dE+eCCufOACZg\nlZoygDnAhApx15sxphDrg38c1vv+DDDVGLPbPmQKkGJXkc3C+vcEqzF8JZADJAPPGGNWn0ksqv5E\n22VUcyQirwG7jTFeL5Eo1dJpiUA1CyIyVETOERE/u3vlJKy6ZqXUGdKRxaq5iAL+g9VwmwbMNsZs\n9W1ISrUMWjWklFIOp1VDSinlcM2uaigyMtLExsb6OgyllGpWNm/efMwY06Gqfc0uEcTGxrJp0yZf\nh6GUUs2KiFQcUV5Oq4aUUsrhNBEopZTDaSJQSimHa3ZtBEqppldUVERaWhr5+fm1H6x8Kjg4mOjo\naAICAur8Gk0ESqlapaWlERYWRmxsLNWvK6R8zRhDRkYGaWlp9OjRo86v06ohpVSt8vPziYiI0CRw\nlhMRIiIi6l1y00SglKoTTQLNQ0P+nRyTCD75BH7zG9AZNZRS6nSOSQQbN8Ljj0NWlq8jUUrVV0ZG\nBnFxccTFxREVFUXXrl3LnxcWFtb42k2bNnHHHXfUeo0RI0Y0SqxJSUlMmDChUc7VVBzTWJwp+4Be\nfPTlVq69KN7X4Sil6iEiIoJt27YB8NBDDxEaGsqvf/3r8v3FxcX4+1f9cZaQkEBCQkKt19iwYUPj\nBNsMOaJEkJyazBNbrT+aKUvvIjk12ccRKdXyJacm89i6x7z2/9u0adOYNWsWw4cPZ86cOXz++ee4\n3W7i4+MZMWIEe/bsAU7/hv7QQw8xffp0EhMT6dmzJ08//XT5+UJDQ8uPT0xM5JprrqFv377ceOON\nlM3SvGLFCvr27cuQIUO44447av3mf/z4ca644goGDhzIBRdcwPbt2wFYs2ZNeYkmPj6e7OxsDh06\nxKhRo4iLi+P8889n3bp1jf6eVccRJYKklCSKgw8DUJQdTlJKEu4Yt4+jUqrlSk5N5tIll1JYUkig\nK5BVU1d55f+5tLQ0NmzYgMvl4uTJk6xbtw5/f39WrlzJb3/7W958881Kr9m9ezerV68mOzubPn36\nMHv27Ep97rdu3crOnTvp0qULI0eOZP369SQkJHDrrbeydu1aevToweTJk2uN78EHHyQ+Pp7ly5fz\n8ccfM3XqVLZt28aCBQtYuHAhI0eOJCcnh+DgYBYtWsSPf/xj7rvvPkpKSsjNzW2096k2jigRJMYm\nEhBmNQ648juRGJvo24CUauGSUpIoLCmkxJRQWFJIUkqSV65z7bXX4nK5AMjKyuLaa6/l/PPP5+67\n72bnzp1Vvmb8+PEEBQURGRlJx44dOXLkSKVjhg0bRnR0NH5+fsTFxZGSksLu3bvp2bNnef/8uiSC\nTz75hClTpgBwySWXkJGRwcmTJxk5ciT33HMPTz/9NCdOnMDf35+hQ4eyePFiHnroIb788kvCwsIa\n+rbUmyMSgTvGzXu3vALAzL73aWlAKS9LjE0k0BWIS1wEugK99uUrJCSk/PH999/P6NGj2bFjB++8\n8061femDgoLKH7tcLoqLixt0zJmYN28eL7zwAnl5eYwcOZLdu3czatQo1q5dS9euXZk2bRpLlixp\n1GvWxBFVQwCX9BlOUBCEFHf3dShKtXjuGDerpq4iKSWJxNjEJvnylZWVRdeuXQF4+eWXG/38ffr0\n4ZtvviElJYXY2Fhee+21Wl9z0UUXsXTpUu6//36SkpKIjIykTZs2fP311wwYMIABAwawceNGdu/e\nTatWrYiOjmbGjBkUFBSwZcsWpk6d2uj3URXHJAIRiIyEY8d8HYlSzuCOcTdp6XvOnDncdNNNPPLI\nI4wfP77Rz9+qVSueeeYZxo4dS0hICEOHDq31NWWN0wMHDqR169a88opVM/HUU0+xevVq/Pz86N+/\nP+PGjWPZsmU8+eSTBAQEEBoa2qQlgma3ZnFCQoJp6MI0cXHQvTv897+NHJRSLdxXX33Feeed5+sw\nfC4nJ4fQ0FCMMfziF7+gV69e3H333b4Oq5Kq/r1EZLMxpsp+tI5oIyijJQKl1Jl4/vnniYuLo3//\n/mRlZXHrrbf6OqRG4ZiqIQBaH2PvXiE5da82GCul6u3uu+8+K0sAZ8oxJYLk1GSSjr7BsWNw6ZJL\ndVCZUkrZvJoIRGSsiOwRkf0iMq+G464WESMitY8Db6CklCRKWx2FvHYUFJZ4rV+zUko1N15LBCLi\nAhYC44B+wGQR6VfFcWHAncBn3ooFrH7N/qEnAD8Ci3RQmVJKlfFmiWAYsN8Y840xphBYBkyq4rg/\nAE8AXl0Dzx3j5ndjZwHw4qVvaxuBUkrZvJkIugKpHs/T7G3lRGQwEGOMea+mE4nITBHZJCKb0tPT\nGxzQiD69AYgOiGvwOZRSTW/06NF88MEHp2176qmnmD17drWvSUxMpKyr+eWXX86JEycqHfPQQw+x\nYMGCGq+9fPlydu3aVf78gQceYOXKlfUJv0pn03TVPmssFhE/4M/Ar2o71hizyBiTYIxJ6NChQ4Ov\nGRlp/dYupEo1L5MnT2bZsmWnbVu2bFmd5vsBa9bQ8PDwBl27YiJ4+OGHGTNmTIPOdbbyZiI4CMR4\nPI+2t5UJA84HkkQkBbgAeNubDcZlOUQTgVLNyzXXXMN7771XvghNSkoK33//PRdddBGzZ88mISGB\n/v378+CDD1b5+tjYWI7Z/+M/+uij9O7dmwsvvLB8qmqwxggMHTqUQYMGcfXVV5Obm8uGDRt4++23\nuffee4mLi+Prr79m2rRpvPHGGwCsWrWK+Ph4BgwYwPTp0ykoKCi/3oMPPsjgwYMZMGAAu3fvrvH+\nfD1dtTfHEWwEeolID6wEcAPw07KdxpgsILLsuYgkAb82xjRs2HAdRERYv9/clMSAcUHaTqBUA9x1\nF9hrxDSauDh46qnq97dv355hw4bx/vvvM2nSJJYtW8Z1112HiPDoo4/Svn17SkpKuPTSS9m+fTsD\nBw6s8jybN29m2bJlbNu2jeLiYgYPHsyQIUMAuOqqq5gxYwYAv/vd73jxxRf55S9/ycSJE5kwYQLX\nXHPNaefKz89n2rRprFq1it69ezN16lT+/ve/c9dddwEQGRnJli1beOaZZ1iwYAEvvPBCtffn6+mq\nvVYiMMYUA7cDHwBfAa8bY3aKyMMiMtFb163J1vRkCMzmox3bdCyBUs2MZ/WQZ7XQ66+/zuDBg4mP\nj2fnzp2nVeNUtG7dOq688kpat25NmzZtmDjxh4+iHTt2cNFFFzFgwACWLl1a7TTWZfbs2UOPHj3o\n3dtqe7zppptYu3Zt+f6rrroKgCFDhpCSklLjuXw9XbVXRxYbY1YAKypse6CaYxO9GQtYYwloHYU5\n1b58jnQtFShVPzV9c/emSZMmcffdd7NlyxZyc3MZMmQI3377LQsWLGDjxo20a9eOadOmVTv9dG2m\nTZvG8uXLGTRoEC+//DJJSUlnFG/ZVNZnMo31vHnzGD9+PCtWrGDkyJF88MEH5dNVv/fee0ybNo17\n7rnnjGcpdczIYrDGEkjIccjr4NU50pVSjS80NJTRo0czffr08tLAyZMnCQkJoW3bthw5coT333+/\nxnOMGjWK5cuXk5eXR3Z2Nu+88075vuzsbDp37kxRURFLly4t3x4WFkZ2dnalc/Xp04eUlBT2798P\nwD/+8Q8uvvjiBt1b2XTVQJXTVc+dO5ehQ4eye/duDhw4QKdOnZgxYwa33HILW7ZsadA1PTlqriF3\njJvhvTJJO9yZ1720dJ5SynsmT57MlVdeWV5FNGjQIOLj4+nbty8xMTGMHDmyxtcPHjyY66+/nkGD\nBtGxY8fTppL+wx/+wPDhw+nQoQPDhw8v//C/4YYbmDFjBk8//XR5IzFAcHAwixcv5tprr6W4uJih\nQ4cya9asBt2Xr6erdtQ01ABTpsAnn8C33zZiUEq1cDoNdfOi01DXolMnOHIEmln+U0opr3FcIigI\nTiEvD1Z99bmvQ1FKqbOCoxJBcmoyz+15GIAJz9+i3UeVqofmVo3sVA35d3JUIkhKSaKktTW4uSgr\nQqeiVqqOgoODycjI0GRwljPGkJGRQXBwcL1e56heQ4mxiQS0fZMCwJXbVbuPKlVH0dHRpKWlcSaT\nPqqmERwcTHR0dL1e46hE4I5x89bNf+fy/4NfnPcH3DE9fB2SUs1CQEAAPXro/y8tlaOqhgB+PGAo\nLhe0LtQ/aqWUAgcmAj8/a/K5jAxfR6KUUmcHxyUCgJC2uXyye7f2GlJKKRyYCJJTkzlQtJmdB47o\nDKRKKYUDE0FSShKlrdLhVET5DKRKKeVkjksEibGJuEJOQF6kzkCqlFI4MBG4Y9zcOPwy/PI6snKK\nzkCqlFKOSwQAg3pGU1riR782mgSUUsqRiSDSXilZF7FXSimHJoJ08xUAq3d+6eNIlFLK9xyXCJJT\nk7nv01sBuP2Nh7T7qFLK8RyXCJJSkigKOgRAUU5b7T6qlHI8xyWCxNhEAsNOAuDK76TdR5VSjue4\nROCOcbNqxnJc/iXc0ON27T6qlHI8R01DXWZENzcdO0BQYVdfh6KUUj7nuBJBmchInYFUKaXA4YlA\nxxEopZSDE4G0Psae745p91GllOM5MhEkpyazJv0/pB8zOhW1UsrxHJkIrKmoj0JeewqKinUsgVLK\n0RyZCBJjE/EPzQTjIrCoo44lUEo5miMTgTvGzX0/ngXA4h+9o2MJlFKO5shEAODu0wuAmIB4H0ei\nlFK+5dhEoFNRK6WUxbGJICLC+r1kw3vaa0gp5WiOTQRf538GwFtbPtEupEopR3NsIvj0yMfgn4fJ\nbU9hSaF2IVVKOZYjJ50DGN0jEWmdAbkdCHQFahdSpZRjOTYRuGPcnBtzClfrkbw0dZV2IVVKOZZX\nq4ZEZKyI7BGR/SIyr4r9s0TkSxHZJiKfiEg/b8ZTUffOIYSX9tIkoJRyNK8lAhFxAQuBcUA/YHIV\nH/SvGmMGGGPigPnAn70VT1V0BlKllPJuiWAYsN8Y840xphBYBkzyPMAYc9LjaQhgvBhPJZGRkJ7e\nlFdUSqmzjzcTQVcg1eN5mr3tNCLyCxH5GqtEcEdVJxKRmSKySUQ2pTfiJ3d+YCpZWbDum08b7ZxK\nKdXc+Lz7qDFmoTHmHGAu8LtqjllkjEkwxiR06NChUa6bnJrMK/sWAHDZop/qOAKllGN5MxEcBGI8\nnkfb26qzDLjCi/GcJikliZJWhwAoPNlWxxEopRzLm4lgI9BLRHqISCBwA/C25wEi0svj6Xhgnxfj\nOU1ibCL+YScA8M/vrOMIlFKO5bVEYIwpBm4HPgC+Al43xuwUkYdFZKJ92O0islNEtgH3ADd5K56K\n3DFuFk+eD8B9Q57SLqRKKcfy6oAyY8wKYEWFbQ94PL7Tm9evzY8GxgHQrrS3L8NQSimf8nljsS+1\nbw8icPSoryNRSinfcXQicLmgbbsiPty+RXsNKaUcy9GJIDk1mSzXfj7fl6JTUSulHMvRiSApJQnT\nOh1ORepU1Eopx3J0IkiMTcQvVKeiVko5m6MTgTvGzaTBbloXdWeVTkWtlHIoRycCgAE9osg72Zph\nXTQJKKWcyfGJoEMHMAYyMnwdiVJK+YYmAnsOO52OWinlVI5PBEfNTgCSdu70cSRKKeUbjk4EyanJ\n3Lt+GgD3vPVHHUeglHIkRyeCpJQkioK/B6Aou52OI1BKOZKjE0FibCKBYdkAuPKidByBUsqRHJ0I\n3DFuPv75B7QKy2NSzC06jkAp5UhenYa6OXDHuInpDK68Vr4ORSmlfMLRJYIyHTpo91GllHNpIkAT\ngVLK2TQRAKWtjpDyfY52H1VKOZLjE0FyajIrvn+FnBPBXPLyGE0GSinHcXwiSEpJoqTVISj1p/BU\niI4lUEo5juMTQWJsIv5hmQAE5HfRsQRKKcdxfCJwx7h5ctIcAP5y4T91LIFSynEcnwgALu7fD4BO\nfuf7OBKllGp6dUoEIhIiIn72494iMlFEArwbWtPRqaiVUk5W1xLBWiBYRLoCHwJTgJe9FVRTi4y0\nfr+xcY32GlJKOU5dE4EYY3KBq4BnjDHXAv29F1bT2nI0GYKyWLXjCy5dcqkmA6WUo9Q5EYiIG7gR\neM/e5vJOSE0vKSUJQtIxpyIpLCnULqRKKUepayK4C/gN8JYxZqeI9ARWey+sppUYm4iEZEBuRwJd\ngdqFVCnlKHWafdQYswZYA2A3Gh8zxtzhzcCakjvGzcg+x/k6pYg3p67SLqRKKUepa6+hV0WkjYiE\nADuAXSJyr3dDa1q9u7VHcjtpElBKOU5dq4b6GWNOAlcA7wM9sHoOtRgdO1rdR43xdSRKKdW06poI\nAuxxA1cAbxtjioAW9ZHZpQsUFUFGhq8jUUqpplXXRPAckAKEAGtFpDtw0ltB+UJ24B4A3t/yhY8j\nUUqpplWnRGCMedoY09UYc7mxHABGezm2JpOcmszvt9wKwC1LH9RxBEopR6lrY3FbEfmziGyyf/6E\nVTpoEZJSkigO+Q6AohMddByBUspR6lo19BKQDVxn/5wEFnsrqKaWGJtIYNvjALhOxeg4AqWUo9Q1\nEZxjjHnQGPON/fN7oKc3A2tK7hg3H09/n9Ztc/lJ55nahVQp5Sh1TQR5InJh2RMRGQnkeSck33DH\nuDm3e2tKTkb5OhSllGpSdU0Es4CFIpIiIinA34Bba3uRiIwVkT0isl9E5lWx/x4R2SUi20Vkld0b\nyWdat89k895D2lislHKUuvYa+sIYMwgYCAw0xsQDl9T0GhFxAQuBcUA/YLKI9Ktw2FYgwRgzEHgD\nmF/P+BtNcmoyG0++zcGDpToDqVLKUeq1Qpkx5qQ9whjgnloOHwbst9sUCoFlwKQK51ttT28N8CkQ\nXZ94GlNSShKloWmQE0VBUbH2HFJKOcaZLFUptezvCqR6PE+zt1XnZqzpKypfSGRmWdfVdC8tI5YY\nm4h/26NgXATmR2vPIaWUY5xJImi0KSZE5GdAAvBklRcyZpExJsEYk9ChbF3JRuaOcfPIpNkA/G3k\nf7XnkFLKMWqchlpEsqn6A1+AVrWc+yAQ4/E82t5W8RpjgPuAi40xBbWc06u6ds8HIC0l2JdhKKVU\nk6qxRGCMCTPGtKniJ8wYU9taBhuBXiLSQ0QCgRuAtz0PEJF4rHmMJhpjjp7JjZyp5NRkbvlkNEgp\njyxfpo3FSinHOJOqoRoZY4qB24EPgK+A1+3VzR4WkYn2YU8CocC/RWSbiLxdzem8LikliSK/bAhL\no+RYD20sVko5Rp1WKGsoY8wKYEWFbQ94PB7jzevXR2JsIoGuQPIi9sPx3iTGnuPrkJRSqkl4rUTQ\n3Lhj3Kyauoqh57ejzal4bSxWSjmGJgIP7hg3142KJyszgMxMX0ejlFJNQxNBBeeea/3ev9+3cSil\nVFPRRFDBqdBtAKz4dJ+PI1FKqaahicBDcmoyt6xPBFcBj7z+jnYhVUo5giYCD0kpSRRJDkRtozg1\nQbuQKqUcQROBh7IupNJ1I3w/hAujE30dklJKeZ0mAg9lXUhHXxgCRSGk7Gvt65CUUsrrNBFUYX3p\nnwG4+e/PaTuBUqrF00RQQVJKEkXhX0HwcYoPaDuBUqrl8+oUE81RYmwiQf6B5MWug5TRJMYe9nVI\nSinlVVoiqMAd4+apsU/RNyENk9mDiDydakIp1bJpIqggOTWZu/53F3s7PgkBucy6O8PXISmllFdp\nIqggKSWJwpJCStscQIYtZM0H7cjQXKCUasE0EVRQNpbADz/8BrxBaYkfy5b5OiqllPIeTQQVlLUR\nuPxclEZtwq/bBn7z22L27PF1ZEop5R2aCKqQkZtBSWkJhlLMFVMpdRXw4x+jyUAp1SJpIqhCROsI\nSikFwLT/mjuf/oDsbBg8GO66C3JzfRygUko1Ik0EVcjIzcBPrLfGT/wI7b6H7dth2DD461/h4oth\nxYpaTqKUUs2EJoIqJMYmEuQKwg8/BOHz7z/nu9JkVq+GV1+FY8dg/HgYM8ZawMYYX0eslFINp4mg\nCmUNxiJCiSlh+e7ljH5lNMmpyUyeDHv3wn33waZN0KsXdO0KBw/6OmqllGoYTQTVyMjNoNSUlj8v\nKClgyRdLAAgIgEcegS++gNmz4dAh6NEDbr4ZSkp8FbFSSjWMJoJqJMYm4vJznbbtxa0vnjYbaffu\n8MwzsHUr/Pzn8NJLcN558NxzTR2tUko1nCaCarhj3NwSf8tp24pKi5i/fn6lY+Pi4NlnYfFi8PeH\nWbPg3nvh1KmmilYppRpOE0ENpg6aSqAr8LRty/csZ+7KuZWOFYFp02DzZpgxAxYsgP794aOPmihY\npZRqIE0ENXDHuEm6KYnosOjTts9fP5+5K+fy2LrHKi1c06oVLFoEa9dCcDBcdpmVIJKTtXeRUurs\nJKaZfTolJCSYTZs2Nek1F21exK3v3lrlPn8/fxZevpCZQ2ZW2pefD7//PTz+uPX82Wfh1qpPo5RS\nXiUim40xCVXt0xJBHcwcMpNR3UdVua+4tJjb3ruN2e/OrlQ6CA6Gxx6Dr7+GhASr7eCGGyAnpymi\nVkqputFEUEePX/o4AX4BVe4rMSU8u/lZLlp8EYs2L6q0v2dPWL8eHn4Y/v1v6NvXakNoZoUxpVQL\npYmgjtwxbtZMW8OoblWXDMBKCLe+e2uVjcmBgXD//bBmjTXm4N574bbb4MQJb0atlFK100RQD+4Y\nN2t+voY5I+eUz0VUlfnr59Pjrz2qLB1ceKHVkHzPPVabgdttVR0ppZSvaGNxAyWnJjN//XyW71le\n43GX9byMxNhEEmMTccecvv7xqlVw9dXWaOS//tUalCbizaiVUk5VU2OxJoIztGjzIl7c8iKZ+Zns\nO76vymMEIdg/mFVTV1VKBt99BzfdBElJcNVV1qjkyMgmCFwp5Sjaa8iLZg6ZyWczPmPvL/cyZ+Sc\nKo8xGPKK87jl7Vsq9Szq1s0qGTz5JLzzDgwaZCWDfVXnFKWUanRaImhkyanJ3PbebWw7sq3K/YLw\n0wE/JSwwDLBGL5eVErZts9E98lgAABZuSURBVLqX7tkDXbrAp59CTEyTha6UasG0asgH5q6cy5Pr\nn8RQ8/sb5Api9U2ry5NBSYk1Z9Gtt0JIiDWp3Y03atuBUurMaNWQDzwx5gmenfAsfrW8xZ7TWwO4\nXHDLLbBxI3TsCFOmwPTpkJXl7YiVUk6licCLZg6ZySfTP+GKPlcQFRJV7XHPbX6uUlfTwYNh926Y\nMweWLIELLoD583VUslKq8Xk1EYjIWBHZIyL7RWReFftHicgWESkWkWu8GYuvuGPcvHXDWxz69SFu\nHHBjlccYTJUD0fz94YknrBlMT52CuXNh4EBrHIJSSjUWryUCEXEBC4FxQD9gsoj0q3DYd8A04FVv\nxXE2+edV/+S5Cc8xrMswhMqV/vPXz6fznzpz5WtXnta76JJL4MABa3qK/HyYNMl6rJRSjcGbJYJh\nwH5jzDfGmEJgGTDJ8wBjTIoxZjtQWtUJWqKy7qbVtR8czjnM8t3LGfHSCC5++eLyyexE4JprrDmL\nzjkHrrsOJkyAr77ywU0opVoUbyaCrkCqx/M0e1u9ichMEdkkIpvS09MbJThfK2s/qGnuorUH1vLs\n5me5+OWLy0sIPXpY3UqfeAI2bIBhw+APf4DDh5sqcqVUS9MsGouNMYuMMQnGmIQOHTr4OpxG4zl3\nUU0qLpHp7281Im/bBmPGwAMPQNeuVonh4EFvR62Uamm8mQgOAp7DoaLtbaqCJ8Y8wYbpG2osHSzf\ns7xS+0G3bvDWW/D551ZD8v/+B/Hx1u9tVY9nU0qpSryZCDYCvUSkh4gEAjcAb3vxes1aWelgw/QN\nXNHniiobk8vaDy5cfOFp3U2HDoU//hE2bbLGHowbZyWEN99syjtQSjVXXksExphi4HbgA+Ar4HVj\nzE4ReVhEJgKIyFARSQOuBZ4TkZ3eiqe5KOtuun76eq7oc0WVx5SaUma9O6vS2IO+feGzz6wprsFq\nUL7vPsjL83bUSqnmTKeYOMvNXTn3tPaBikZ1H8Xjlz5eaVbTffvgwQfhX/+yVkibORNmzID27b0d\nsVLqbKRTTDRjT4x5gucmPMd5kecRGhhaaf/aA2sZ8dIIevy1x2ntB716wauvwsqV0KkTzJtnTWD3\n1lu6RKZS6nRaImhGklOTufjliykqLarxuH4d+nHn8DuZOWRm+baNG+Hmm+HLLyEhAW6/HcaOtZKE\nUqrl0xJBC1GXdZMBdqXvKp+yIjk1mcfWPUZxVDIbN1prHZw4AdOmWdNVvPoqFBQ0TfxKqbOTlgia\nqbpOc13W+8hzhbSSEmsxnDvusNY+6NDBmtDuxhshIKApoldKNTUtEbRAT4x5gvXT1zNryCziOsVV\ne5yx/8srzitvdHa54LLLYNcua0K7nj2t9ZLPOQf+/Gc4ebKp7kIpdTbQEkELkZyazJIvlvDfPf/l\nUM6hao8b1GkQtw29jYzcDBJjE3HHuCkthffft5bLXLMG2rSxehmNGQOjR0NgYBPeiFLKK3SFMgep\na4MygB9+TOw7kTkj5pR3P924Ef70J2t209JSOPdcK0FMmqSrpCnVnGnVkIOUNSjPGjKLUd1G0b1t\n92qPLaWU5buXM/KlkeVrIQwdCsuWWdNeL1tmlQauvNKa3O6557TaSKmWSEsEDvCz//yMpV8urfW4\nqNAoLoi+oLyEkJyazKr9a8j+7FpWLD2HHTusdZTHjYOf/Qx+8hPw068SSjULWjWkWLR5EX9c90cO\nZB2o0/FRoVEcPXUUgCBXECunrMJ1yM3zz1vtCd9/b01pcdddcMMN0LatN6NXSp0prRpSzBwyk5S7\nUspXSIuLqr6nEVgT3JWaUkpNKXnFefxj+xKGD4cXXrCqjf71L2jVCmbNgqgoq+vpRx9BSUkT3ZBS\nqtFoicDBklOTmbdyHmu/q9siyLHhscRFxZVXHRkDmzfD4sVWYsjMhOhomDoVJk6EuDgICvLyTSil\n6kSrhlSN6psQoPI0Fvn58M478PLL1noIpaVWUvjtb60Fc1rQekJKNUuaCFSdlI1F2JW+iwNZB+rU\nntA1rCttgtrQJ7JPeUnh+++t8Qj/93+QnGwNYOvfH4YMsaa2GFXzDBlKKS/QRKAaZNHmRcx+dzal\nlNb5NbHhsVzX/zrCg8K5uHsirkNuli+H1auttRIALr/cWivhRz+CLl28FLxS6jSaCFSDlZUSPk37\nlG1H6r/+ped6CdnZsHAhPPoo5ORAcDBceKGVFKZMsZ4rpbxDE4FqFGVJYdU3q9iXua9er40KjSIq\nNIpAv0Cm9JuJO/hmFi6E9eth715o184anzB2LAwfDr17e+kmlHIoTQSq0S3avIgXt7xIYWkhh7MP\nc/jU4Xq9vqxtIdAVRME+N+323cnONX3KRy5fcYVVdVRYaK2jEBbmhZtQykE0ESivq++AtapM7jeF\n3/VfwpIl8MorcNjOLdHRMH26VWIYPlznPFKqITQRqCbj2fNob8beepcU2rdqT5ugNsS06UaX/Evo\nVzqZNf/pzccfW/u7drXmPho1CgYMsEY3K6Vqp4lA+cyizYt46tOnyCvOo7ikmLTstHqfo1f7XpAb\nQei3N9LmwGQ2rIqgqMia56hfP2t6i/vus9ZYcLm8cBNKtQCaCNRZw7NtoaC4gOyC7Honh05B3emS\ndxknNlxJ4bFYSo714HBaMB07QkwMzJ5tdVHt3NlLN6FUM6SJQJ3V5q6cW756WoMUhBCZdjPB31yF\n68hQDuxvDUC3bmDMD6ObIyMbKWClmiFNBOqsl5yazPz189l6eCsFJQUczqlf20K5Ehed0n+K35Eh\nZH1zLkVHz6HoUF9cLkPPnsLw4TB5MowYAeHhjXsPSp3NNBGoZqcsMezJ2EOQf1CDuqgCUCpw4GL4\ndjTBJ+Io2nspJfkhBAWXMMLtYsQIqwG6c2ddhU21bJoIVItQ1vCcmZ/JkZwjGBrwt5sfBmkXwL7x\nBB28lIKD54GxWpjbxRxi4hUlXD8uml69rHUXJk+2qpRKSrQhWjVvmghUi1PWTfVwzmFSTqQ0aPoL\nAApbw6kO8O0lsG0apI4A41++O7jTAS4YlUPyu73pNfUv/HJGePmMqwcPWmMdhgxphBtSyss0EagW\nz3P8QnpuOkH+QWTmZZJdmM3xvON1P1FBKBwaDEcGwKmOsPcncDje2ifFELGXVtF7ad/9MAf/Owvx\nK8Y97xFGhEzld7/oqSu1qbOWJgLlaJ5VSg1qhC4OhJxO8PntcGSQlShy7QUWAk5BUQgAQee/T8SE\nP9E2NAg51ZE7r3aXlx6U8jVNBErZKjZCZ+ZlUlBSUL82h1KB7K5w9Hxovx9WPgZ7J0BJhelTBy6h\nTUwawd12ENh5HxGRJRSW5p+2doNSTUUTgVK1qLgoj4jUfyT0sd7w3YWQ2dNqc8joDXkRpx/T5jvo\nsAu6biQkqBWdB+ymsOsq/FwQHhxOZl4mIYEh5au/Jacmk5SSRGJsoiYOdUY0ESjVQGUjobu06ULv\niN4kfZtEZn4mXx//um4L9uS2g0NDIKOXVYrI6G1VLZ2ILe+thCsfWh0Hv2Jo9y1E7IWIPYS0zyE3\nvQOm43bo8za9InpRWFKIiJQnDRGhW9tu9Ivsx9RBU3HHuMuTR3BRF/7z70D6j1vHTfFTTtunicV5\nNBEo1cjKPlAjWkew9dDW+pckSv2gOBh23GAlh9xIKAyBY32t58WtK74Agk9YP5F7wD8fwlOg45cQ\nehiCsiHkKF26lnLo5BFMYDa89h/YfSVccx30/zdh6ZeRnZcH3dcB1hxO/n7+5VVkIYEhTOg9gb3H\n9rInY0+dq7C8lVw0aTUuTQRKNTHPOZXKvrmHB4fXbWCcAQrCIKs7hByB/ePg+LmQHw7Zna0SRUCu\nVQVVKWHYAnKgKPSH5x2/hKMDrMet06HXCuj5EbRNBSmxztdxJxS1spLHie4wZBG0OURUaBSh349H\nOu4iJDyv0v0cOfVD+0rZAkSexxQUF5QnG4wf4a3aciI/s1LJJjw4nPTPE2kdkUl2xw9qPW9mXiYg\nhAdGEhTgR6+IXmw5tIXcwnzatW5b6diyx3l5hk7hbWkf3B6A43nHyS/O5+bBNzOg44DybskVRYVG\nEd85nq2HtgKUl8Bq+ht4c9ebXN3v6vJOA+u++ZS7f5vBeT9ez21jflJeSlvyxZJK5/TsIh0VGlXr\n9WqjiUCps4jnjKwVP6TqNYK6OMBKBpk9QUohp7NV5QRQ0AaCM602ivTzrOeBOfDtmPoF2yrDKoVk\nnmN1n43aBqUBVgkm5Ah02Wwlj8Iw6xrHz4Xz/gNtUqFVJpQEWknFVQhtv4NNs63zjlgAWTEQdNIq\nzbQ+ZlWPPb/R2n/JfVaJ6bw3rdeGpFvXNGJ1501JhJwoqwSVEwUX/x76/xu+HQ0fLoAx86CotfV+\nZJ5jxRGx1zr3tptgVhy4iuDARdZ71+0TKxa/EgjIh5OdobiVlXz9iuCzOyA2Cc5/DfaOt96Htt/R\nKawTrQKCCQ8O51hGCX7+xbTxj+TA6kvIzi2ykqmrgM5tO1JKIUf+Nx3WPAQxn8DPxtE1Ipzvcw5i\nigPAvxAKW3NO+3M4ttVNVvgaK1EH5gIgCJP6TmpwRwNNBEo1I549mzqEdADDaWMjPL/l1nteJoP1\ngeoqsEocWTFWW0VBGzjWx/rQD0+BNgdhz0SrmiqnEwRnWR+kh+KhoK2VZHI6W/v988E/z/qwLWis\nCZxKAb/qdwecglJ/KAlqpOvZpMT6baoZRi7FPww4lGIrmfjnAWK9h2Alq6pi93xtGVeB9fqCcKsU\nV9yq8rX9c61t/nkw9m6CEv7F6ptW1zsZ1JQI/KvaqJTyHXeMm7dueKvOx1fsEhvoF0iviF7sy9hX\n3sj9zp53yCvOo1vbbj8kllh/CopTPOZyevX0E0d/XvvFS/2sb9QC5b1vSwKhOAjy21kN4IHZ1rf2\nUn8IOWqVFL4fYrVtpPe3PwjbWiWLkKNWoorZYH3If3sJlARYj4OyrCqssEPQbZ1VCtl1NfgXWNcr\nbgWFoXDu+1YpqfMWa7xHboR1zYI2cDgOIr+CNDeUuqwP2E5fwKlOdlL0s65h/KySSMQ+65zt91ml\nnazuVtwFbax4i4Ot+wI7IRZY289fZj3+crL1G6xj2++zShOHhlhx5bezSi5+RdZv/3zrPQvI+6HU\nVdDWiqWoFbTfR2FJIUkpSY3abqIlAqUUQKW66i+PfnlaFZZnXb9nqSQkMIT4qHi2HNpSZXVXda+r\n6rHnsbWdt+xxgF/Aab24qmtP8Hz8XdZ3DZur6iwQ5Apq9BKBVxOBiIwF/gq4gBeMMY9X2B8ELAGG\nABnA9caYlJrOqYlAKVVRfXsYVZX0XtzyIsEBwbQPbl+pYbjssWc13PG84+U9xapKaH0i+5R3OS7r\nNFCxZ1ZZCS6xR+Jp2zyPPZl/svy6Z9Jo7JNEICIuYC/wIyAN2AhMNsbs8jjmNmCgMWaWiNwAXGmM\nub6m82oiUEqp+qspEdTQGnPGhgH7jTHfGGMKgWXApArHTAJesR+/AVwqojPCK6VUU/JmIugKpHo8\nT7O3VXmMMaYYyAIqjMkHEZkpIptEZFN6erqXwlVKKWfyZiJoNMaYRcaYBGNMQocOHXwdjlJKtSje\nTAQHgRiP59H2tiqPERF/oC1Wo7FSSqkm4s1EsBHoJSI9RCQQuAF4u8IxbwM32Y+vAT42za0/q1JK\nNXNeG1BmjCkWkduBD7C6j75kjNkpIg8Dm4wxbwMvAv8Qkf3AcaxkoZRSqgk1uwFlIpIOHGjgyyOB\nY40YTnOg9+wMes/OcCb33N0YU2Uja7NLBGdCRDZV14+2pdJ7dga9Z2fw1j03i15DSimlvEcTgVJK\nOZzTEsEiXwfgA3rPzqD37AxeuWdHtREopZSqzGklAqWUUhVoIlBKKYdzRCIQkbEiskdE9ovIPF/H\n01hE5CUROSoiOzy2tReRj0Rkn/27nb1dRORp+z3YLiKDfRd5w4lIjIisFpFdIrJTRO60t7fY+xaR\nYBH5XES+sO/59/b2HiLymX1vr9kj+BGRIPv5fnt/rC/jPxMi4hKRrSLyrv28Rd+ziKSIyJcisk1E\nNtnbvP633eITgb0uwkJgHNAPmCwi/XwbVaN5GRhbYds8YJUxphewyn4O1v33sn9mAn9vohgbWzHw\nK2NMP+AC4Bf2v2dLvu8C4BJjzCAgDhgrIhcATwB/McacC2QCN9vH3wxk2tv/Yh/XXN0JfOXx3An3\nPNoYE+cxXsD7f9vGmBb9A7iBDzye/wb4ja/jasT7iwV2eDzfA3S2H3cG9tiPn8NaGKjScc35B/gv\n1uJHjrhvoDWwBRiONcLU395e/neONa2L237sbx8nvo69AfcabX/wXQK8i7Uycku/5xQgssI2r/9t\nt/gSAXVbF6El6WSMOWQ/Pgx0sh+3uPfBLv7HA5/Rwu/briLZBhwFPgK+Bk4Yax0POP2+6rTORzPw\nFDAH7MWIrXto6fdsgA9FZLOIzLS3ef1v22uTzinfM8YYEWmR/YNFJBR4E7jLGHPSc2G7lnjfxpgS\nIE5EwoG3gL4+DsmrRGQCcNQYs1lEEn0dTxO60BhzUEQ6Ah+JyG7Pnd7623ZCiaAu6yK0JEdEpDOA\n/fuovb3FvA8iEoCVBJYaY/5jb27x9w1gjDkBrMaqFgm31/GA0++rJazzMRKYKCIpWMvcXgL8lZZ9\nzxhjDtq/j2Il/GE0wd+2ExJBXdZFaEk813i4CasOvWz7VLunwQVAlkdxs9kQ66v/i8BXxpg/e+xq\nsfctIh3skgAi0gqrTeQrrIRwjX1YxXtu1ut8GGN+Y4yJNsbEYv0/+7Ex5kZa8D2LSIiIhJU9Bi4D\ndtAUf9u+bhxpogaYy4G9WPWq9/k6nka8r38Bh4AirPrBm7HqRVcB+4CVQHv7WMHqPfU18CWQ4Ov4\nG3jPF2LVo24Httk/l7fk+wYGAlvte94BPGBv7wl8DuwH/g0E2duD7ef77f09fX0PZ3j/icC7Lf2e\n7Xv7wv7ZWfZZ1RR/2zrFhFJKOZwTqoaUUkrVQBOBUko5nCYCpZRyOE0ESinlcJoIlFLK4TQRKGUT\nkRJ71seyn0abqVZEYsVjllilziY6xYRSP8gzxsT5OgilmpqWCJSqhT1H/Hx7nvjPReRce3usiHxs\nzwW/SkS62ds7ichb9voBX4jICPtULhF53l5T4EN7lDAicodY6ytsF5FlPrpN5WCaCJT6QasKVUPX\ne+zLMsYMAP6GNSsmwP8BrxhjBgJLgaft7U8Da4y1fsBgrFGiYM0bv9AY0x84AVxtb58HxNvnmeWt\nm1OqOjqyWCmbiOQYY0Kr2J6CtTDMN/aEd4eNMREicgxr/vcie/shY0ykiKQD0caYAo9zxAIfGWtx\nEURkLhBgjHlERP4H5ADLgeXGmBwv36pSp9ESgVJ1Y6p5XB8FHo9L+KGNbjzWnDGDgY0es2sq1SQ0\nEShVN9d7/E62H2/AmhkT4EZgnf14FTAbyheUaVvdSUXED4gxxqwG5mJNn1ypVKKUN+k3D6V+0Mpe\nBazM/4wxZV1I24nIdqxv9ZPtbb8EFovIvUA68HN7+53AIhG5Geub/2ysWWKr4gL+aScLAZ421poD\nSjUZbSNQqhZ2G0GCMeaYr2NRyhu0akgppRxOSwRKKeVwWiJQSimH00SglFIOp4lAKaUcThOBUko5\nnCYCpZRyuP8H8luf2Ik/xC4AAAAASUVORK5CYII=\n",
+            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAsMAAAEYCAYAAAC5nfszAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOzde3yU1bXw8d+aScJFVFrEUiUavEC9IAEC9pGCg2Ar3kBFK8c2cFACWNuip0atWjmixyN4XjlWFFBLpVVQ65GiolaQANVYbkapiC3aKNQbRhGshiQz6/1jP5PMTCbJEDK5ri+ffDLPfi6zZybs7Oxn7bVFVTHGGGOMMaYjCrR0BYwxxhhjjGkp1hk2xhhjjDEdlnWGjTHGGGNMh2WdYWOMMcYY02FZZ9gYY4wxxnRY1hk2xhhjjDEdlnWGTb1E5DkRmdjUx7YkESkVkdFpuK6KyHH+4/kicnMqxzbieS4TkT81tp71XDckIjub+rrGNDdrt/brum263WqrROS3InJbE19zkoj8uSmv2VFktHQFTNMTkS9jNrsC+4Cwvz1VVR9J9VqqOiYdx7Z3qjqtKa4jIjnAP4BMVa3yr/0IkPJnaExbYO1Wy7N2q+MQkUnAFar6vZauS2tgneF2SFW7RR+LSCnuB35l4nEikhFtqIwxpiVZu2VM+5Ts/+z+/j9O9/97C5PoQKK3wUXkOhH5CFgkIt8QkWdEZJeIfO4/7h1zTpGIXOE/niQifxaRu/xj/yEiYxp5bB8RWSsie0VkpYjME5Hf11HvVOo4S0Re9q/3JxE5LGb/j0XkPREpE5Eb63l/ThWRj0QkGFN2gYi84T8eKiLFIrJbRD4UkXtFJKuOa8XdAhORa/1zPhCRyQnHniMir4nIHhHZISIzY3av9b/vFpEvRcRLvBUmIqeJyAYR+cL/flqq7019ROQE//zdIvKmiJwfs+9sEdnqX/OfIvILv/ww//PZLSKficg6EbF2xjSatVvWbtXXbsX8fBSKyCd+fcf5bdTf/HbolzHHB0TkehF5x39vHxeRb8bsf8J/P7/wP+uTEt6feSLyrF+vv4jIsfV8NnVey3eYiLzoX2uNiBztnycicrf/evaIyBYROdnfd6iILPZ/rt4TkZskSRsrIjniwloyYsqKROQKETkBmA94/uez29/fyf/Zf19EPhYXNtOlntc3WUTe8n++X4jW39+nIvITEfk78HdJ/v+4k4jM9X++PvAfd0r4XKuPr6seTcF+SXU8vYBvAkcDBbifgUX+9lHA18C99Zx/KvA2cBgwG3hIRKQRxz4KrAd6ADOBH9fznKnU8d+AfwcOB7KAaOfsROB+//pH+M/XmyRU9S/Av4AzEq77qP84DFztvx4PGAVcWU+98etwll+fM4HjgcS4v38B+UB34BxguoiM8/eN8L93V9VuqlqccO1vAs8C9/iv7f8Bz4pIj4TXUOu9aaDOmcDTwJ/8834KPCIi/fxDHsLduj4YOBl4yS//D2An0BP4FvBLwNZ8NwfK2i1rt+prt3oBnYEjgV8BDwA/AgYDw4GbRaSPf+xPgXHA6bj39nNgXsy1nvNf7+HAZmqHdlwK/CfwDWA7cHs99WroWpcBs3CfTUnM/u/j3sO+wKHAJUCZv+/Xftkx/mvIx71PKVPVt4BpQLH/+XT3d/23/5y5wHHUvJ+1iMhYXPt+Ia69XwcsSThsHO7/04n+duL/4xuB7/rPNwAYCtwUc37i8emjqvbVjr+AUmC0/zgEVACd6zk+F/g8ZrsId7sSYBKwPWZfV1xHp9f+HIv7xVAFdI3Z/3vg9ym+pmR1vClm+0rgef/xr4ClMfsO8t+D0XVc+zbgN/7jg3EN/tF1HDsDeCpmW4Hj/Me/BW7zH/8G+O+Y4/rGHpvkunOBu/3HOf6xGTH7JwF/9h//GFifcH4xMKmh9ybJ84aAnf7j4cBHQCBm/xJgpv/4fWAqcEjCNW4F/ljXa7Mv+0rly9ota7f2s936GgjGvH4FTo05ZhMwzn/8FjAqZt+3gcrYusbs6+5f69CY9+fBmP1nA9tS/PyTXSv2M+6G+8MlG/eHzd9wHcXYNjjo/xycGFM2FShK8h4n+wyKiP9Z/3PMPvF/bo6NKfOAf9Txep4DLo/ZDgBfRX/u/Oc+I+Fzivt/DLwDnB2z/QOgNNX/9035ZSPDHc8uVS2PbohIVxFZ4N9u2YO7vdVdYm65Jfgo+kBVv/IfdtvPY48APospA9hRV4VTrONHMY+/iqnTEbHXVtV/UfMXdjKPAhf6t2ouBDar6nt+PfqKu9X5kV+P/8L9Rd+QuDoA7yW8vlNFZLV/2+sL3F/sKYUy+Nd+L6HsPdxf9FF1vTcN1llVI3Vc9yLcL4L3/Nt7nl8+Bzda8icReVdErk/tZRhTL2u3rN2qr90qU9XoZMuv/e8fx+z/Oub8o4GnxIWN7MZ1jsPAt0QkKCL/LS6EYg/ujzKIf10p1SvFa8V+xl8CnwFHqOpLuLsI84BPRGShiBzin5tJ/HuX+L41Vk/cH3+bYt6b5/3yZI4G/jfm2M9wHerYuiT+/4j7f0ztn4P3/LK6jk8b6wx3PIm3rP8D6If7K/oQam5v1XULsSl8CHxTRLrGlGXXc/yB1PHD2Gv7z9mjroNVdSvuP+QY4m81grttuQ043q/HLxtTB9wIU6xHgeVAtqoeiovlil63oRCDD3CNUqyjgH+mUK+GrpudEItWfV1V3aCqY3G3/5YBj/vle1X1P1T1GOB84BoRGXWAdTHG2i1rt5rKDmCMqnaP+eqsqv/EvXdjcSEhh+JGV6FxP1epXCv2M+6GCwn4AEBV71HVwbgQg77AtcCnuFHs2PeurvftX/732J/XXjGPEz+jT3F/NJwU874cqjETWxPswIXKxb6PXVT1lXqeI3E78efgKL+sruPTxjrD5mDcf4DdfhzXLel+Qn/EYiMwU0Sy/FHF89JUxz8A54rI98RNGrmVhn/uHwV+jvvl9URCPfYAX4rId4DpKdbhcWCSiJzo/1JLrP/BuBGnchEZimtEo3YBEVx8WDIrgL4i8m8ikiEiP8Q1ns+kWLe6/AU36lEoIpkiEsJ9Rkv9z+wyETlUVStx70kEQETOFZHj/BjLL3AjLpHkT2FMo1m7VZu1W6mZD9wuNZPVevrxr+Be0z7cKHxX3Ch6Y6VyrbNjPuNZwKuqukNEhvgj75m4Tm05EPFHvx/363+w/xquwYXrxFHVXbhO8o/8UerJQOxkv4+B3v5z498FfAC4W0QOBxCRI0XkB3W8vvnADeJPChQ3se/i/Xh/wIXe3eR/BofhwoOSTkhNN+sMm7lAF9xfha/ibos0h8tw8UhluHi3x3ANRzKNrqOqvgn8BPeL4kPcZImGFpZYgpuY8JKqfhpT/gtcg78X12g8lmIdnvNfw0u4EIKXEg65ErhVRPbiGoPHY879CjdB42X/dtR3E65dBpyLG4UqAwqBcxPqvd9UtQL3i34M7n2/D8hX1W3+IT8GSv3bf9Nwnye4ySIrgS9xMYD3qerqA6mLMUlYu1Vbh2+3UvS/uBHtP/l1fxU3yQtgMW6E/Z/AVn9fY6VyrUdxf2R8hpvs9yO//BDcZ/W5f40yXAgauAmA/wLeBf7sX+M3ddRhCm5EuQw4CYgdtX0JeBP4SESi7/t1uM/6Vb9tX4m7u1GLqj4F3IkbINkD/BX3+2J/3Ib7A/MNYAtukmGTLkSSKvEDlY1pUSLyGG4iQtpHeIwxpilYu2VM+2Ajw6ZF+LeBjhWX8/EsXGzVspaulzHG1MXaLWPaJ1uBzrSUXsD/4SaF7ASmq+prLVslY4ypl7VbxrRDFiZhjDHGGGM6LAuTMMYYY4wxHVa7CZM47LDDNCcnp6WrYYwxddq0adOnqlpXEvs2y9pfY0xrV1/72246wzk5OWzcuLGlq2GMMXUSkcRVt9oFa3+NMa1dfe2vhUkYY4wxxpgOyzrDxhhjjDGmw7LOsDHGGGOM6bDSGjPsJyX/XyAIPKiq/52wvxNuycLBuOUCf6iqpf5a2QuAPNz65j9X1aJ01tWY1q6yspKdO3dSXl7e0lUxDejcuTO9e/cmMzOzpatijDkA1u62PY1pf9PWGRaRIDAPOBOXnHyDiCxX1a0xh10OfK6qx4nIpbh1rn+IW08bVe0vIocDz4nIEFWNpKu+xrR2O3fu5OCDDyYnJwcRaenqmDqoKmVlZezcuZM+ffq0WD1SGIy4BrgCqAJ2AZNV9T0RyQXuBw4BwsDtqvpYs1bemFbC2t22pbHtbzrDJIYC21X1XVWtAJbilq6MNRZ42H/8B2CUuJ+2E4GXAFT1E2A3bpTYmA6rvLycHj16WIPcyokIPXr0aNGRpJjBiDG49nSCiJyYcNhrQJ6qnoJrf2f75V8B+ap6EnAWMFdEujdPzY1pXazdbVsa2/6mszN8JLAjZnunX5b0GFWtAr7ALXP5OnC+iGSISB9cGEV24hOISIGIbBSRjbt27drvChYXwx13uO/GtAXWILcNreBzanAwQlVXq+pX/uarQG+//G+q+nf/8QfAJ0CT50a29te0Fa3g/7PZD435vFprnuHfACcAG4H3gFdwt+viqOpCYCFAXl7efq0rXVwMo0ZBRQVkZcGqVeB5B15xY4xpBZINRpxaz/GXA88lForIUCALeCfJvgKgAOCoo47ar8pZ+2uMaU3SOTL8T+JHc3v7ZUmPEZEM4FCgTFWrVPVqVc1V1bFAd+BvTVm5oiLXEIfDUF4Oixc35dWNaX/KysrIzc0lNzeXXr16ceSRR1ZvV1RU1Hvuxo0b+dnPftbgc5x22mlNUteioiLOPffcJrlWeyciP8KFoc1JKP828Dvg35PN11DVhaqap6p5PXvu38BxbPu7bx/MnGkjxMYk09baXRHhwQcfrC4rKSlBRLjrrruqy6qqqujZsyfXX3993PmhUIh+/fpVv77x48c3Sb1Skc6R4Q3A8X6Ywz+BS4F/SzhmOTARKAbGAy+pqopIV0BU9V8iciZQlTDx7oCFQhAMusZYFRYtgvx8G50wpi49evSgpKQEgJkzZ9KtWzd+8YtfVO+vqqoiIyN5k5KXl0deXsNh/6+88krTVNakMhiBiIwGbgROV9V9MeWHAM8CN6rqq01duVDIjQjv2weRCKxcCevW2QixMYnaWrt78skn8/jjj3PFFVcAsGTJEgYMGBB3zIsvvkjfvn154oknuOOOO+LCGh555JGU6tzU0jYy7McAXwW8ALwFPK6qb4rIrSJyvn/YQ0APEdkOXANE/0w4HNgsIm8B1wE/bur6eR5MnlyzXVnpRiuMaU+KdxRzx7o7KN6RnmG3SZMmMW3aNE499VQKCwtZv349nucxcOBATjvtNN5++20gfqR25syZTJ48mVAoxDHHHMM999xTfb1u3bpVHx8KhRg/fjzf+c53uOyyy1B1kVArVqzgO9/5DoMHD+ZnP/tZgyPAn332GePGjeOUU07hu9/9Lm+88QYAa9asqR6BGDhwIHv37uXDDz9kxIgR5ObmcvLJJ7Nu3bomf8+aSfVghJ+q8lLc4EM1ERmIS2F5vj9ROVqeBTwFLFbVP6Sjcp7nOr6jR0Mg4DrEFRXWBpv2oSO3u0cffTTl5eV8/PHHqCrPP/88Y8aMiTtmyZIl/PznP+eoo46iuJXcEkprzLCqrgBWJJT9KuZxOXBxkvNKgX7prBvAwIE1jyMR2L073c9oTPMp3lHMqMWjqAhXkBXMYlX+Krzsph9227lzJ6+88grBYJA9e/awbt06MjIyWLlyJb/85S958skna52zbds2Vq9ezd69e+nXrx/Tp0+vlRPytdde48033+SII45g2LBhvPzyy+Tl5TF16lTWrl1Lnz59mDBhQoP1u+WWWxg4cCDLli3jpZdeIj8/n5KSEu666y7mzZvHsGHD+PLLL+ncuTMLFy7kBz/4ATfeeCPhcJivvvqqweu3RqpaJSLRwYgg8JvoYASwUVWX48IiugFP+CMz76vq+cAlwAjcQMUk/5KTVLWkKevoeS48Yt26mtjhUKgpn8GY5mftLowfP54nnniCgQMHMmjQIDp16lS9r7y8nJUrV7JgwQJ2797NkiVL4sI0LrvsMrp06QLAmWeeyZw5c2pdPx1a6wS6ZlFWBiIuTALg7rth3Di7TWfah6LSIirCFYQ1TEW4gqLSorQ0yhdffDHBYBCAL774gokTJ/L3v/8dEaGysjLpOeeccw6dOnWiU6dOHH744Xz88cf07t077pihQ4dWl+Xm5lJaWkq3bt045phjqvNHTpgwgYULF9Zbvz//+c/VvxjOOOMMysrK2LNnD8OGDeOaa67hsssu48ILL6R3794MGTKEyZMnU1lZybhx48jNzT2g96YlpTAYMbqO834P/D69tXOiI8RFRa4jbG2vaeus3YVLLrmEH/7wh2zbto0JEybEhWE888wzjBw5ki5dunDRRRcxa9Ys5s6dW/1a2l2YRFsQjRuOqqqyiXSm/QjlhMgKZhGUIFnBLEI5obQ8z0EHHVT9+Oabb2bkyJH89a9/5emnn64z12PsSEEwGKSqqqpRxxyI66+/ngcffJCvv/6aYcOGsW3bNkaMGMHatWs58sgjmTRpEoutQUib4mKYPt21udYRNu2FtbvQq1cvMjMzefHFFxk1alTcviVLlrBy5UpycnIYPHgwZWVlvPTSS/v9HE2tQ48Mex7MmwdXXlkzke6hh2winWkfvGyPVfmrKCotIpQTSsvoRKIvvviCI4906cR/+9vfNvn1+/Xrx7vvvktpaSk5OTk89ljDC6MNHz6cRx55hJtvvpmioiIOO+wwDjnkEN555x369+9P//792bBhA9u2baNLly707t2bKVOmsG/fPjZv3kx+fn6Tv46OrrjYdYCjk+EXLYLVq63dNW2ftbvOrbfeyieffFI94gtUh3Ps2LGjutO9aNEilixZwplnntnk9d4fHbozDFBQAM89B8uWue3KSjdSYY2yaQ+8bK9ZGuOowsJCJk6cyG233cY555zT5Nfv0qUL9913H2eddRYHHXQQQ4YMafCc6MSRU045ha5du/Lww27Ry7lz57J69WoCgQAnnXQSY8aMYenSpcyZM4fMzEy6detmI8NpUlTk2tqoaHq1mTOt7TVtn7W7ydO1PfXUU5xxxhlxo89jx46lsLCQfftcMpvYmOHDDjuMlStXNtGrqJ9EZwq2dXl5ebpx48ZGnTt9OsyfX7M9bRrcf38TVcyYJvLWW29xwgkntHQ1WtyXX35Jt27dUFV+8pOfcPzxx3P11Ve3dLVqSfZ5icgmVW13S8vvb/ubODIMLqtERobL8mN350xrYe2u01ba3aj9bX87dMxwVH4+RP9QCQbjs0wYY1qXBx54gNzcXE466SS++OILpk6d2tJVMvvJ89zo8LRpMHRofHq1BQvc6nStJOOSMYb23+5aZxjXMN9zD2RmurjhGTOsITamtbr66qspKSlh69atPPLII3Tt2rWlq2QawfPcHbjLL4+fyKwKX39t7bAxrUl7b3etM+wrK3MjE5GILc9sjDHNobjYdXqTTVhfvx5GjrQOsTEm/awz7ItNsxZdntkaYWOMSZ+iIhcaUdfUFVuVzhjTHKwz7LPlmY0xpnmFQm7luUAdv4lsVTpjTHOwznAMW57ZGGOaT3QFuttuc6t/ulWhnaFDLfewMaZ5WGc4RnR55qi777ZQCWOiRo4cyQsvvBBXNnfuXKZPn17nOaFQiGjKrbPPPpvdSf7CnDlzJnfddVe9z71s2TK2bt1avf2rX/2qSfJPFhUVce655x7wdUzjeR7ccAMUFkLnzi5crUsXmDvXOsLGtNd2V0R48MEHq8tKSkoQkbg6VVVV0bNnT66//vq480OhEP369SM3N5fc3FzGjx9/wHWyznCMxOWZKyth9uwWq44xrcqECRNYunRpXNnSpUuZMGFCSuevWLGC7t27N+q5ExvlW2+9ldGjRzfqWqZ1io4Sz5rlOsJFRbBwIdxxhw1KmI6rvba7J598Mo8//nj19pIlSxgwYEDcMS+++CJ9+/bliSeeIHFNjEceeYSSkhJKSkr4wx/+cMD1sc5wjOjyzLHxa8uWuQbZmLaouLjpOhPjx4/n2WefpcJfKaG0tJQPPviA4cOHM336dPLy8jjppJO45ZZbkp6fk5PDp59+CsDtt99O3759+d73vsfbb79dfcwDDzzAkCFDGDBgABdddBFfffUVr7zyCsuXL+faa68lNzeXd955h0mTJlU3gKtWrWLgwIH079+fyZMnV69klJOTwy233MKgQYPo378/27Ztq/f1ffbZZ4wbN45TTjmF7373u7zxxhsArFmzpnoEYuDAgezdu5cPP/yQESNGkJuby8knn8y6desO7M01gGuDQyGXYeKmm2DqVPfd8g6btsTa3Ybb3aOPPpry8nI+/vhjVJXnn3+eMWPGxB2zZMkSfv7zn3PUUUdRnOYGwDrDCQoKIC9hfZInn2yZuhhzIIqLXSfi5pubpjPxzW9+k6FDh/Lcc88BbnTikksuQUS4/fbb2bhxI2+88QZr1qyp7kgms2nTJpYuXUpJSQkrVqxgw4YN1fsuvPBCNmzYwOuvv84JJ5zAQw89xGmnncb555/PnDlzKCkp4dhjj60+vry8nEmTJvHYY4+xZcsWqqqquD9m+cjDDjuMzZs3M3369AZvCd5yyy0MHDiQN954g//6r/8iPz8fgLvuuot58+ZRUlLCunXr6NKlC48++ig/+MEPKCkp4fXXXyc3N7dR76mpLZphIhJx25bu0rQl1u6m3u6OHz+eJ554gldeeYVBgwbFLdNcXl7OypUrOe+885gwYQJLliyJO/eyyy6rHqS49tprU39D62Cd4SQuvzx+237PmbYo2qkIh5suRVXsLbvYW3WPP/44gwYNYuDAgbz55ptxt9YSrVu3jgsuuICuXbtyyCGHcP7551fv++tf/8rw4cPp378/jzzyCG+++Wa99Xn77bfp06cPffv2BWDixImsXbu2ev+FF14IwODBgyktLa33Wn/+85/58Y9/DMAZZ5xBWVkZe/bsYdiwYVxzzTXcc8897N69m4yMDIYMGcKiRYuYOXMmW7Zs4eCDD6732iZ10QwTsfM3LN2laSus3U293b3kkkt44oknWLJkSa2wj2eeeYaRI0fSpUsXLrroIpYtW0Y4HK7eHxsmMWfOnHrrmwrrDCdRUOAmcwQCrkH+9a+tETZtT7RTEQw2XYqqsWPHsmrVKjZv3sxXX33F4MGD+cc//sFdd93FqlWreOONNzjnnHMoLy9v1PUnTZrEvffey5YtW7jlllsafZ2o6EhDMBikKtnKDim4/vrrefDBB/n6668ZNmwY27ZtY8SIEaxdu5YjjzySSZMmsdiGLZuM57mY4YyM+PKqKkt3aVo/a3dTb3d79epFZmYmL774IqNGjYrbt2TJElauXElOTg6DBw+mrKyMl1566YDqVZ+0doZF5CwReVtEtovI9Un2dxKRx/z9fxGRHL88U0QeFpEtIvKWiNyQznom07276wir2i060zbFTkhatappZuZ369aNkSNHMnny5Oq/5Pfs2cNBBx3EoYceyscff1x9O68uI0aMYNmyZXz99dfs3buXp59+unrf3r17+fa3v01lZSWPPPJIdfnBBx/M3r17a12rX79+lJaWsn37dgB+97vfcfrppzfqtQ0fPrz6OYuKijjssMM45JBDeOedd+jfvz/XXXcdQ4YMYdu2bbz33nt861vfYsqUKVxxxRVs3ry5Uc/ZkRXvKOaOdXdQvKP2SEN0RdBYItCjRzNVzphGsnZ3/9x6663ceeedBGOyF+zZs4d169bx/vvvU1paSmlpKfPmzasVKtGUMho+pHFEJAjMA84EdgIbRGS5qsaO418OfK6qx4nIpcCdwA+Bi4FOqtpfRLoCW0VkiaqWpqu+iaKZJcJh1yF+6CHIz7dUP6Zt8bym/5mdMGECF1xwQfVtuwEDBjBw4EC+853vkJ2dzbBhw+o9f9CgQfzwhz9kwIABHH744QwZMqR636xZszj11FPp2bMnp556anVDfOmllzJlyhTuueeeuJnDnTt3ZtGiRVx88cVUVVUxZMgQpk2b1qjXNXPmTCZPnswpp5xC165defjhhwGXxmj16tUEAgFOOukkxowZw9KlS5kzZw6ZmZl069bNRob3U/GOYkYtHkVFuIKsYBar8lfhZdf8oEZH1yoqagYlIhE3sa5/f2uHTetm7W7qTjvttFplTz31FGeccUZcDPHYsWMpLCysnqh32WWX0aVLF8DFKB9oyjdJTFfRVETEA2aq6g/87RsAVPWOmGNe8I8pFpEM4COgJ3Ap8G/ABcChQDHwXVX9rK7ny8vL02hevaZywQUum0TUuHHw1FNN+hTGpOytt97ihBNOaOlqmBQl+7xEZJOq5tVxSpu1v+3vHevu4ObVNxPWMEEJMmvkLG4YHn8DsLjY3ZHbvBk2bnSd4WDQjbjd0Oz3Ck1HZe1u27S/7W86wySOBHbEbO/0y5Ieo6pVwBdAD+APwL+AD4H3gbuSdYRFpEBENorIxl27djX5C+jVK3776actdtgYYw5UKCdEVjCLoATJCmYRygklPe7hh2s6woGALc9sjEmP1jqBbigQBo4A+gD/ISLHJB6kqgtVNU9V83r27NnklcjPj1+EQ9UmcBhjzIHysj1W5a9i1shZtUIkomJTrAUCMHp008VgGmNMrHR2hv8JZMds9/bLkh7jh0kcCpThQiSeV9VKVf0EeBlIy63F+iZxeB7cdx9kZrq4tUDAJnCYlpWusCbTtFrD55TCBOZrRGSriLwhIqtE5OiYfRNF5O/+18R01M/L9rhh+A1JO8IQPys/IwO6dnVhE3Z3zjS31vD/2aSuMZ9XOjvDG4DjRaSPiGTh4oCXJxyzHIg2tOOBl9S9iveBMwBE5CDgu0D9y0c1QnQSx82rb2bU4lFJO8QFBXDvva4jHA7DT39qjbFpGZ07d6asrMwa5lZOVSkrK6Nz584tVoeYCcxjgBOBCSJyYsJhrwF5qnoKLjRttn/uN4FbgFNxd+luEZFvNFfdo6Kz8qdMcXflli2D+fNh2DC47suyifMAACAASURBVLrmro3pqKzdbVsa2/6mLZuEqlaJyFXAC0AQ+I2qvikitwIbVXU58BDwOxHZDnyG6zCDa8QXicibgACLVLXupVUaqai0iIpwBWENUxGuoKi0KOkoxWuvuY4wuNt2ixfbrTrT/Hr37s3OnTtJR3y8aVqdO3emd+/eLVmFocB2VX0XQESWAmOB6mw+qro65vhXgR/5j38AvBidpyEiLwJnAU2a16h4RzFFpUWEckJ1jg57nguXiE1VqgqzZ7vHd97ZlDUypjZrd9uexrS/aesMA6jqCmBFQtmvYh6X49KoJZ73ZbLyphadxBFN71PXJI5EH32U3noZk0xmZiZ9+vRp6WqYtiHZBOZT6zn+ciCaqDSVyc+ISAFQAHDUUUftV+UaSq0WKxRyoWoVFfHld90Fxx7rchKHQjZAYdLD2t2OobVOoGs2EwdMZMqgKfU2xvn5rjGOevppWLiwmSpojDFpJCI/ws3J2K81TQ9kAnPsXbl9VfuYWTQzaZga1IwOjxgRXx6JwJVXws03w6hRFr5mjGm8DtsZjo5MPLD5AR5+/eF6j/U8uPzymu1wGK66yhpfY0yrlcoEZkRkNHAjcL6q7tufcw9E9K5cgAARIqz8x8o6522Aa4PXrIHCQjeZOSocdl/79lmmH2NM43XYznCyeOH65Oe7Gc1RVVXW+BpjWq0GJzCLyEBgAa4j/EnMrheA74vIN/yJc9/3y5pMNLVa3hF5CEJEIym1w3feCVOnxneIwY0SW6YfY0xjddjOcKpJ36M8D665pmZbFXbvTm8djTGmMfxFjKITmN8CHo9OYBaR8/3D5gDdgCdEpERElvvnfgbMwnWoNwC31rf654Eo+bgExc3SzwhkpDRvIzFsDVznuKwsDRU0xnQIaZ1A15pFRyYams0cq3t31+hGM6zcfbdbotkmbhhjWpsUJjCPrufc3wC/SV/t3N25cCRcvT3muDF42V6DWSY8D84+26Vai8rIsJXpjDGN12E7w0B1Qxu9NddQhzgUcgngo2l+qqoszZoxxjRGKCdEMBAk7OetfPpvT3Pdyuv49V9+XW+WieJiWBHTxQ8G4eqra8LWrD02xuyvDhsmATWT6G566SZG/HYECzfVnyLC82DevJolmlXhoYdsIp0xxuwvL9tjcu7k6u2whpnz8hzKq8rrnctRVFST913ELcJx991w002WVcIY0zgdujNcVFrEvqp9RIhQFaniqhVX1TmbOaqgAM47r2a7shJmzLAG2Bhj9lf+gHwyAjU3KNX/F5BAnXM5YpdpzsyEl1927XAkYlkljDGN06E7w6GcEIFAzVsQ1nCDs5kBevWK316/3kYkjDFmf3nZHvPOnkdA4n8V9evRj4kDJiY/x1+medYsmDy5Zg4HuMeWVcIYs786dGc42hBnBmqmJu/e13CKiGSzmcvLXfywMcaY1BUMLuD+c+4nKMHqsrc+fYuFmxfWmXvY8+CGG2qnvFSFn/3MBiaMMfunQ3eGwTXEV3tXAxDRCLNfnp1S7HDsIhzgGuFFi6wRNsaY/VUwuIApg6Yg1CQQTiX3sOe50eFY+/bZwIQxZv90+M4wQMmHJXHbT259ssFz8vOhS5f45O+2EIcxxjRO/oB8MoM1t9zqixuOOy/fxRDHeuABuO46uOMOG6AwxjTMOsPARSdeFLed++3cBs+Jxq1NnepCJkTchA7LdWmMMY0j/r/MQCYFgwqYe9ZcikqL6p3Y7HluEGLo0JqycBhmz7YME8aY1FhnGHeLrnBYIQEJIAhzX53L9GemN5hZwvPcqEQg4MIkwmHYsqWZKm2MMe1IUWkRVZEqFCUcCbPmvTVc+eyV3LT6pjpjh6OSha6ByzBRUWF37Iwx9bPOsK97p+4IgqJUhCuYv2l+SrmHi4pcWh9wneGrrrJRCGOM2V+hnBBZwSwCBIgQ4a1P3yKsYSIaYV94X4OZfsrK4sPWwA1UZGXZHTtjTP2sM+wL5YSQhJY0ldzDoZBrcKvPqbLJG8YYs7+8bI9V+asYfczouIl0UT261p8zLRSCzp3jO8Qi8IMfNHFFjTHtTlo7wyJyloi8LSLbReT6JPs7ichj/v6/iEiOX36ZiJTEfEVEpOFA3gPgZXsM6jWoVnlDuYdtVTpjjGkaXrbHzNDMuIl0AKrKjOdnNBgqsWoVnHlmzQBFOAx//KPFDRtj6pe2zrCIBIF5wBjgRGCCiJyYcNjlwOeqehxwN3AngKo+oqq5qpoL/Bj4h6qWkGaXD4oPOhOETsFODc5mTrYqnY0OG2PM/vOyPYomFjFt8DSGHjGUgASqw9caCpXwPJg5s3buYVuZzhhTn4yGD2m0ocB2VX0XQESWAmOBrTHHjAVm+o//ANwrIqIau6YQE4ClaaxntYLBBYBLrdbzoJ78vezvHHHIESmdm7gq3UcfNXXtjDGmY/CyPbxsj4WbFvLaitcAUkqzBjW5h+fPrymzTD/GmPqkM0ziSGBHzPZOvyzpMapaBXwBJAaG/RBYkuwJRKRARDaKyMZdu3Y1SaULBhcwMzSTJ7Y+wfoP1rNs2zJGPjyywcwSiavSPf00LKx/7p0xxpg6FO8oZsbzMwhHwgQkwNyz5uJleymdG80DHwi4UeJ773XllnfYGJNMOkeGD5iInAp8pap/TbZfVRcCCwHy8vI02TGNUVRaRGW4snp7X3gfi19fXG9DHE3tEx2NCIfhyiuhf3+3zxhjTOqKSouoCFcQIYJGlDkvz+Gdz9+he6fu1SPERaVFhHJCtdrmaPxwURHs3u3mcbz2mku1lpXl9lm7bIyJSmdn+J9Adsx2b78s2TE7RSQDOBQoi9l/KXWMCqdTKCdEZjCTinBFddmikkXkD8ivt0Ocn+9WPgqH3XY08ftTT6W7xsYY075EU62VV5WjKNs/387sl2e7RTmCmQhCVaSKrGAWq/JXJe0Qb9kCv/xl/HWjeYetM2yMiUpnmMQG4HgR6SMiWbiO7fKEY5YDE/3H44GXovHCIhIALqGZ4oVjRSdwDD1iaHWKn6pIVUqTN2In0oGbyWzhEsYYs3+iqdaO/caxceWKUhmupCJcQVjD9U6se/LJ2mWBgMUPG2Pipa0z7McAXwW8ALwFPK6qb4rIrSJyvn/YQ0APEdkOXAPEpl8bAeyITsBrbl62x9yz5laPQAQDwZQmbxQW1qRZAzeT2RbiMMaY/edle1w77Npa5RmBDLKCWQQlWO/Euosuql0WiTRxJY0xbV5a8wyr6gpV7auqx6rq7X7Zr1R1uf+4XFUvVtXjVHVobMdXVYtU9bvprF8qoqvShSNhtnzS8FrLngf33Vd7IQ5L62OMMfuvYHAB474zrnpbEC4feDmrJ65myqApTBwwse5zC9wARexCHOEwzJhhAxTGmBq2Al09YifShTXM9GenN7g8M7gG+Be/qNlWdZM4jDHG7L/C0wrpktGFAAECEuCjf33Elk+28PDrD/PA5gcYtXhUnRl/7rzTTWyOvWO3fj0MHw7Tp1un2BhjneF6hXJCBGKGeCMaaXB55qju3eNHI+6+2xpdY0zzSWEF0BEisllEqkRkfMK+2SLypoi8JSL3SOJa9c0sGrYmIoQ1zLJty5j2zDTKq8objBsGN0AxZUp8WTjsOsm2Op0xxjrD9fCyPeadPY+A1LxNqUykAzdBI3YkoqrKVqUzxjSPFFcAfR+YBDyacO5pwDDgFOBkYAhwepqr3KCyr8qIaE3Ar/r/AhJIaUGO/Pz4Njnq66+tbTamo7POcAMKBhfwi9NqYh4UpUfXmnVBincUc8e6O2qNFnsezJtX0/iqulyXNgJhjGkG1SuAqmoFLivP2NgDVLVUVd8AEqeUKdAZyAI6AZnAx+mvcv2iKS9jCcLoPqOTplZLFJ3PkWyM29pmYzo26wynoHun7nGjww9tfojiHcUU7yhm1OJR3Lz65qQxawUF8anWKitd3mFjjEmzVFYATUpVi4HVwIf+1wuq+lbicelYAbQ+0ZSX4/qNIyhBAhKgc0ZnZoZmprwyXUGBC40IJPzmq6y0SXXGdGTWGU5BKCdERqBmfZL1H6xn5MMjWfz64gZzXfbqFb9teYeNMa2ZiBwHnIBbKOlI4AwRGZ54nKouVNU8Vc3r2bNns9TNy/Z46tKnuO+c+xjdZzQ/PfWnFJUWpTSPI6qgAO6/v3bIxPr1MHKka5+nT7fJdcZ0JNYZToGX7TE5d3JcWXR1uoZyXSbGqVneYWNMM0hlBdC6XAC8qqpfquqXwHNAq1mvrXhHMTOen8HKd1cy++XZ3PjSjYz47YiUMv1EFRTAunUwdGh8+b59cOWVbvR4/nzXOba22pj2zzrDKcofkB83Oqwoeyv2MnHARKYMmlJnzJrlHTbGtIBUVgCty/vA6SKSISKZuMlztcIkWkpRaREV4QoifqizolRFqlJOfRnleTB3LmRl1ZSJuCwTUdGlm40x7Zt1hlPkZXtcMfCKuLJHtjzCws0Lefj1h+s91/IOG2OaUyorgIrIEBHZCVwMLBCRN/3T/wC8A2wBXgdeV9Wnm/1F1CGUEyIrmEUg4dfX/qS+jPI819kdN84NWKjG78/KsqWbjekIMho+xETlD8jnwdcepCpSVV0W0Uh1vHB9kziieYejje3dd7sG2Gs1Nx+NMe2Jqq4AViSU/Srm8QZc+ETieWFgator2Ehetseq/FUUlRaxe99u7nrlruqUa2ENs/j1xRSVFhHKCaU8se6DD2ov05yb6zrDW7ZYO21Me2ed4f0QzTs8/Znp1bfoAAISaDDHZTTvcJXfj47mHbZG1hhj9o+X7VV3dI/9xrFc+eyVhDWMqrJwswuV6BTs1GDKteJit+hGeXntfSUl7vv69e57QUGTvgRjTCtiYRL7qWBwAXlH5MWVDew1MKUcl5Z32Bhjmlb/w/tXp75UlIhGiGiEfeF9DS6QVFTk4oITwyMScxE/+WTT1dcY0/pYZ7gRLh90edx2qE8o6cIbiSzvsDHGNK2i0qK40LWooARTumOXleUGKbKyXOjatGlw7bXxx110UdPV1xjT+liYRCMUDHb3y57c+iS5385l7qtzqQxXkhnMpGhi/bHDdeUdtltwxhiz/6Ir00XTXYLrCF/tXV09MlxXm+x5sGqVGyEOhdx2cbHbLix0oRIXXWTtszHtnWji/aE2Ki8vTzdu3Njszzv9menM3zS/enva4Gncf+79dR5fXAzDh8en78nMhDVrLH7YmPZORDapal7DR7YtLdX+RhXvKGbx64vZumsr5VXlhPqE+PVffk1FuIKsYFZKyzVDTQxxRYUbKV61ypXHdpaNMW1Tfe2vjQw3s2je4enTa2YvR/MOW0NrjDH7L9rRHbV4FBXhCjZ/tLk6djiVbD9R0RjicNh9X7wYHn44vnNs7bQx7Y/FDB+g/AH5dAp2AtytuYHfHtjgOcnyDj//vE2mM8aYxoouxhHWMOGIyywhCMFAw7HDUYkxxB995DJNRDvHtgCHMe1TWjvDInKWiLwtIttF5Pok+zuJyGP+/r+ISE7MvlNEpFhE3hSRLSLSOZ11bSwv2+OeMfeQGchEUWY8PyOlpO/RvMNRa9fa0p/GGNNYsYtxaOy//QgFjMYQz5rlVqdbsSI+00SPHmmouDGmxaWtMywiQWAeMAY4EZggIicmHHY58LmqHgfcDdzpn5sB/B6YpqonASGgMl11PVBlX5VV35L7uuprZr/ccIqIaN7hWPv22ciDMcY0RnQxjtHHjEaoGWmojFSmPEgBrkN8ww1QVhY/tyMSgZ/+1IW42aCFMe1LOkeGhwLbVfVdVa0AlgJjE44ZC0TXMv4DMEpEBPg+8Iaqvg6gqmX+qkitUignhMQM8y57exkLNy2s95xo3uFAwidgIw/GGNM4XrbHzNBMMoOZceXrP1jP6b89nenPTE+5UxwNmYg27aouVGLBAjfJzjrExrQf6ewMHwnsiNne6ZclPUZVq4AvgB5AX0BF5AUR2SwihcmeQEQKRGSjiGzctWtXk7+AVHnZHoN6DYorm7VmVkp5hwsKahpbEXjttXTV0hhj2j8v26NoYhFDjxgaV14ZqWTBpgWMWjwqpQ5xNGRi6lSX8SdKFb7+2k2uM8a0D611Al0G8D3gMv/7BSIyKvEgVV2oqnmqmtezZ8/mrmOcxIU4du7dyem/Pb3BRjc/v6ahVYUHHnB5h40xxjSOl+0x96y5ZATiEyYpWp1dIqXreHD//XDOObX32QqixrQf6ewM/xPIjtnu7ZclPcaPEz4UKMONIq9V1U9V9StgBTCIVqxgcAHjvjMurqwyUtlg/LDnweTJNdvhsItJsw6xMcY0npftccXAK+LihwEyAhkpZ5cA1+FdsaJ2eXQF0eJiuOMO6xgb05alszO8ATheRPqISBZwKbA84ZjlwET/8XjgJXVTf18A+otIV7+TfDqwNY11bRKFpxUSlPhZcU//7emURoczYgYwIhG3JKh1iI0xpvHyB+TXih+OaGS/rlFUFD+RLtayZfC978FNN1kcsTFtWdo6w34M8FW4ju1bwOOq+qaI3Coi5/uHPQT0EJHtwDXA9f65nwP/D9ehLgE2q+qz6aprU/GyPe475764kYiIRlj8ev3BZdHJdLGp1lStQ2yMMQfCy/aYnDs5rqwqUpVymATUzj18+OHx+yMR97Vvn4sjtlFiY9oeW445DRZuWsiVz15J2E+A0SnYidUTVze4AtIFF7iRhljBIKxbZ6seGdMe2HLMza94RzGhh0NUhCsAEIQTep7Az0/9OQWDC1K7RnHNksxbtrhJdYmCQXeHr6rKVqszpjWqr/1trRPo2rSCwQVMGTSlersiXNHg6DBAYWH8rGVwt+ds1rIxxjRONLvEuH7jEARF2bprK1OfmdpgCszqa/i5hz3PZQAaFz89hEAAhg2LX8rZcsYb03ZYZzhNYpdlVpSHXnuowdhhz4M1a+CEE+LLP/ooHTU0xpiOwcv2GHrkUJT4O6FzX53LHevuSNo2F+8ornNfYSF06eJC2wIBmDABXn21ZrW6jAw3imyMaRusM5wmZV+V1VoFKZXRYc9zKXtiJ9Q9/bTFDhtjzIEI5YRqpVp769O3uPGlG2vlHi7eUcyoxaO4efXNSfMSe55brjnaTj/2mAuPANdB/vd/txAJY9oS6wynSSgnVGsWcyqjw+Aa0SuuqNkOh+Gqq2xShjHGNJaX7bF20lpOPOzEuHJF2RfeFzeprqi0iIpwBWEN15mXuKysZvJcVZUbFQ4EoHNnlyHIGNN2WGc4TaJxaiccVhPzUBmp5IrlV6TUIU5Mt1ZVZTFoxhhzILxsjwfPf7BWCkxVpUfXHtXboZwQWcEsghIkK5iVNC9xsuWaIxH46U/dtmWVMKbtsM5wGnnZHqcffXpc2dZPt6a0Mp3nwTXX1GyrwvPPW+NqjEmNiJwlIm+LyHYRuT7J/hH+cvdVIjI+Yd9RIvInEXlLRLaKSE5z1TvdvGyP8/qeF1emKFetuIrpz0yneEcxXrbHqvxVzBo5i1X5q5JmAoou13zssfHlRUUu5/DNN1vuYWPaCusMp1n+gPxaoxCpxg937x6fe3jtWhg50hpXY0z9RCQIzAPGACcCE0TkxITD3gcmAY8mucRiYI6qngAMBT5JX22bX+GwQrKCWXFllZFKFmxaUB0j7GV73DD8hnpTYnoeXHttfNknn0B5uWWVMKYtsc5wmiVbiAPgoy8bThERCrnclbH27bPG1RjToKHAdlV9V1UrgKXA2NgDVLVUVd8A4pZk8zvNGar6on/cl6r6VTPVu1lEw9iGHjE0rlzROmOE61JQAAsWwNChrr0uLbWsEsa0NdYZbgYFgwuYf+58AjFv99N/e7rBHJfRlekCCZ/S+vU2OmyMqdeRwI6Y7Z1+WSr6ArtF5P9E5DURmeOPNMcRkQIR2SgiG3ft2tUEVW5eXrbH3LPm1sowoaqs/2B9SnM7oqK5hyMJKz2PGeMGL6y9NqZ1s85wMykYXBC32lFYw1z57JUNNrgFBe4rNlzij3+0WDRjTNpkAMOBXwBDgGNw4RRxVHWhquapal7Pnj2bt4ZNxMv2uGLgFXF37iJEWLZtWUpzO2KFQvGLJgUC8OyzFjtsTFtgneFmlBg/HNZwSrHD+fnxjayqi0mzcAljTB3+CWTHbPf2y1KxEyjxQyyqgGXAoCauX6uRPyC/VhpMcDHEs1+eHbfwRn0LcXiea5PHjXPhEqpQWelih6PhbcXFlmXCmNYopc6wiBwkIgH/cV8ROV9Earcepl5etsd5/eJnMW/dtbXOxrX6PA8mT44vU4UePZIfb4xp+0TkkHr2HdXA6RuA40Wkj4hkAZcCy1N86g1AdxGJDveeAWxN8dw2x8v2mJw7Oem+5W8v56bVNzFq8SgWblpY70Ic4NrqoX4YssYsdheJwO7dlmXCmNYq1ZHhtUBnETkS+BPwY+C36apUe1Z4WiGZgZq/I9a+vzbpCkiJEvMOi8Brr6WzpsaYFlYUfSAiqxL2LavvRH9E9yrgBeAt4HFVfVNEbhWR8/1rDhGRncDFwAIRedM/N4wLkVglIlsAAR5ompfUOuUPyKdLRpdaE50jRIhohPKqcua8PId94X31LsQBtfMPgwuZKClx2SUsy4QxrU+qnWHxZxNfCNynqhcDJ6WvWu2Xl+1x+cDL48qSrYBU6zx/Ml00u4QqPPCALdNsTDsW2zP7Zj37klLVFaraV1WPVdXb/bJfqepy//EGVe2tqgepag9VPSnm3BdV9RRV7a+qk/yMFO1WNK/w7WfczoJzFyTNMvHO5+8Q0QgBCdS5EAfULNUcO/E5MxNyc11ZIODa8ffft9FhY1qLlDvDIuIBlwHP+mW1Zheb1CTLPZy4AlIyBQUwZUrNdjgMV15pDaox7ZTW8TjZtjlA0bzC/Q/vT8nHJdXl4v9TlAABRvcZXedCHFFlZfHbvXrB//yPW0lUpGYww8IljGkdUu0MzwBuAJ7yb7UdA6xOX7Xat2Sxw4ry0+d+2uDs5fz8+NzD4TDMnp2OWhpjWtjhInKNiPxHzOPodttM39AGFJUWEY6EAdcRHttvLJ0zOhOUIJ0yOjEzNLPejjDUhEpER4ffe8+11arue3RinYVLGNM6pNQZVtU1qnq+qt7pT6T7VFV/lua6tWuFpxXSKdgprqwiXMHsl+vv2XoenBffj+aPf7RwCWPaoQeAg4FuMY+j2w+2YL3atVBOiKxgFkEJ0jmjM2OOH8PEARM5r+95TBwwMaVrRJdqHj06PnY4GZsIbUzLE9WG77aJyKPANCCMm2l8CPC/qjqngfPOAv4XF1LxoKr+d8L+TrhlPwcDZcAPVbVURHJwkz7e9g99VVWn1fdceXl5unHjxgZfS2tSvKOYGc/PYP0H66vLghJk3b+vw8v2KN5RTFFpEaGcUNxIRHExDB/uRhaiRGD+fBdKYYxpnURkk6rmNcF1hqjqhqaoU1Noi+1vfaJtb4+uPZjx/AzKq8pRFEHonNG5wTCJ6usUu1Hiinoirrt0cR1nr+HLGWMOQH3tb6phEieq6h5gHPAc0AeXUaK+Jw0C84AxwInABH+Zz1iXA5+r6nHA3cCdMfveUdVc/6vejnBbFV0BKTH38IznZ9Sbxsfz4L774kccVC1+2Jj2TEROFJFZIrIduL+l69OeReOHy74qY1/VPtQP0VaUr6u+ZsbzM1JakCOae3jo0LpHiC1UwpiWl2pnONPPKzwOWK6qlTQ8gWMosN1P3F4BLAXGJhwzFnjYf/wHYJRIQzeV2hcv2+O+c+6L6xCv/2A905+dXm8an4ICGJvwbobDsLjhNTyMMW2EiOSIyA0i8gbwO2A6MLopRpdNw0I5IQKB2r8m13+wnu/95ntc8NgFDXaKo9klOneOzzABroOckeFCJWwxDmNaTqqd4QVAKXAQsFZEjgb2NHDOkcCOmO2dflnSY/y8mF8A0QiqPiLymoisEZHhyZ5ARApEZKOIbNy1a1eKL6X1KRhcwJRBU+LKIuoWuQ9KsM40PoWF8SvTgYsdtvhhY9o+ESnGZe/JAC5S1cHAXlUtbdGKdSBetse8s+fF5YaPii7bPPLhkSl1iFetgttugwULYNq0mra7shKmT4cbb7TsEsa0lFQn0N2jqkeq6tnqvAeMTGO9PgSOUtWBwDXAo8lWY1LVhaqap6p5PXu27cnVydKtRTTCsKOG1Rmf5nmwZg2ccELMORHX0FqH2Jg272PchLlvUZM9wlKqNbOCwQWsmbSGaYOnJe0UV4QrWPz64pRWEr3hBndX76ijXFut6r5HH9uyzca0jFSXYz5URP5fdBRWRP4HN0pcn38C2THbvf2ypMeISAZwKFCmqvtUtQxAVTcB7wB9U6lrWxUNl0hcAWnte2vZ8smWus/z4PTT48ssftiYtk9VxwH9gU3ATBH5B/ANERla/5mmqXnZHvefez9rJq1hXL9xce20IDz02kP1LtOcKJp6LZkePWzZZmOaW6phEr8B9gKX+F97gEUNnLMBOF5E+ohIFnApsDzhmOVANFfNeOAlVVUR6elPwMPPaXw88G6KdW2zCgYXMP/c+bU6xHNentPgUs2J4RIWP2xM26eqX6jqIlX9PvBd4FfA3SKyo4FTTRp42R6FwwrJCtb0ZCNEqIxUVs/vSHWUeNUqGDEivvwXv3ALdtiyzcY0r1Q7w8eq6i3+ZLh3VfU/gWPqO8GPAb4KeAGXJu1xf8GOW0XkfP+wh4Ae/uzoa4Dr/fIRwBsiUoKbWDdNVT/bv5fWNhUMLuDaYdfGlW3/fDuhh0N1Nq7JwiUAPvooXbU0xjQ3Vf1YVX+tqsOA77V0fTqqotIiqiJVSfcFJMCikkUpjxJv2OAm0QUC8P3vQ0kJ7N7tRo2DQfc9FErDizDGxMlI8bivReR7qvpnABEZBnzd0EmqugJYkVD2q5jH5cDFSc57Engyxbq1O3eOvpNjv3EsuZP+VgAAIABJREFUt665lX/udZElFeEKZjw/g7lnza0zfvihh9xIQ5XfTj/9tIsdttzDxrQ9IpJ4Jy3R+Q3sN2kQXZQjmns4VkQjhDVMRCPVWYDqykdcVORGfqOp/v/0p5rvhYXQvbvrCFv+YWPSL9WR4WnAPBEpFZFS4F5gatpqZSgYXMB5feOXmlv/wXpG/HYECzclnx3neXDFFTXb4bCbpWyT6YxpkzzcXIt1wF3A/yR8mRbgZXusyl/FmcecWSukLZoFKCCBOrMARUXjhoPB2jmI/+//rCNsTHNKNZvE66o6ADgFOMXP8nBGWmtmyB+QX2vJ5qpIFdOfnV5nhzg/3+WtjIpEbDKdMW1UL+CXwMm4lTzPBD5V1TWquqZFa9bBedkeM0MzyQzGT9ZQFFUlIIE67+JVX8OPG541y8UKx3rnHZs8Z0xzSnVkGABV3eOvRAcuxtekkZftcc+YewgkfEwRjXDls1cmjUfzPJg3L36kIRyG2bPTXVtjTFNS1bCqPq+qE3GT57YDRSJyVQtXzeDa58m5k2uNDkc7xGVflTV8DT/d2p13utCIaLsdTbO2eLFLsbZwoaVaMyad9qsznKBDrRTXUsq+KqsVlwZu2ebFrydPF5Fsdbo//tHCJYxpa0Skk4hcCPwe+AlwD/BUy9bKROUPyKdzRudaAxYBCfD+F++nlGYtqnv3+EEMVXjgAbcYx9SpcNNNNlpsTLocSGfYkr83g1BOqNatuFQUFrpYtCjLPWxM2yIii4FiYBDwn6o6RFVnqWpivnbTQqLxw6OPGU1Aan6dVkWqWLBpASN+O4Lpz0xPOfdwbIibqrurF51gF4lYqjVj0qXezrCI7BWRPUm+9gJHNFMdOzQv26NoYhHTBk9jXL9x1SsgBSXIwG8PrPs8D+67z8IljGnDfoTLsf5z4JXY9ldE9jRwrmkm0fjhjEBNT1b9f1WRKuZvmp9SmjXPg8mTa0+miwoELNWaMelSb2dYVQ9W1UOSfB2sqqmmZTMHKLr60VOXPsW9Z99LZiATRZnx/Ix6G1gLlzCm7VLVgN/WJrbDB6tqreXpTcupK344qryqnKLSogavk58PnTsn3ycCQ4a4OGK7w2dM0zqQMAnTAsq+KiOiESIaobyqvM644ahk4RLTplmH2BhjmlJd8cPgRop7dO3R4DWiGSamTUu+qujatTB/PowcaR1iY5qSdYbbmFBOiGDA9W4VZeHmhXWmWYPk4RIWP2xM+yciZ4nI2yKyXUSuT7J/hIhsFpEqERmfZP8hIrJTRO5tnhq3bdH44dvOuI3CYYVxMcSC8NqHr8UdX7yjOOmyzZ4H99/vVhVN1ikGix02pqlZqEMb42V7nH3c2Sx7exng0qxNfWYq73z+DneOvjPpOQUF8NxzsGxZTVk4DDNmwNy5ltjdmPZGRILAPFxu4p3ABhFZrqpbYw57H5gE/KL2FQCYBaxNZz3bGy/bq84tfOw3juXKZ68krGE3cLFpIR/96yMKTysEYNTiUVSEK8gKZrEqf1WtnMTRdnnRotrPk5FhscPGNCUbGW6DenXrVats9suzuW7ldXWeU1hYe4Rh/XoYPtxCJoxph4YC21X1XVWtAJYCcTMIVLVU9f+3d+bxUZXn4v8+M1mwLqCRKyqBuC+UsgTRUYFQ1CJqxau31doGFR0Wbcvt9YJU7aVVa8G2Un9liyKS1ltt9ZoiCC5IgNJBZJUKLmjDolJZxBYlmcyc9/fHe87kzJIQNJOE5Pn6mc+c8573nPPMMTzzzPM+i3kDcFJPFpFi4ATgpeYQti0SLg5zW9/bEvsODhVvVTBgzgC+9advUR2rJm7iibbNmaishFgsffz0023ssNYfVpSmQY3hw5DSXqWJqhJ+GjKIQyG77HbOOcnj8TjccYcqU0VpY5wMbPft73DHDoqIBLDtnuvzGCuNpLRXaVKVCbA14nf8awcGQ4CG2zZ7LZsD7je1F+62ebONHR41ytYhHjhQnRqK8mVQY/gwJFQYYulNSxnYbWDasSkrptQbQxwKwaBB6eOxmMafKYqSYCzwgjFmR0OTRCQsIqtFZPWuXbuaSbTDi1BhiGnDpmVMqgPo1KFTg22bvYS6+++HWbPg0kvT5xhjdXiqUyMSUa+xojQWNYYPU0KFIZbevJTxF41POzZ15dSMiRlgS/fk5yePGQP79mVLUkVRWoAPgELffld3rDGEgDtEpAr4JVAqIr9InWSMKTPG9DPG9OvcufOXlbfNEi4OEy4OZzz2SfUnBy2R6bVsDodh0qTk6kB+YjEbOgHWAB4yBO69V7vWKUpjUGP4MGfyJZPTDOLNuzdz96t3Zyz0HgrBkiVw2WXJ13n4YVWYitKGeB04Q0ROEZE84HpgXmNONMbcaIzpZowpwoZKlBtj0qpRKI2ntFcpecG8tHGDoSZe06gaxFBXHSiQ4ZvbGJtsF4nYlb5o1IbBaeUJRTk4agy3ASZfMpnhZw9PGmtIyYZC1sPgb/1ZWwu33qoGsaK0BYwxMeAO4EVgM/BHY8ybIvIzEfkmgIicJyI7gP8AZonImy0ncdvG30m0/0n9k5pzCMK2T7c1qmUzWA9xOLOjmdpaa/h6scbBoH0vKNCQCUVpiKwaw42oc5kvIk+7x18TkaKU491EZL+IaCLHQRh/4XiCkrx+Zkz9hd5DIZg2LdnDsGmTjSlWhakohz/GmBeMMWcaY04zxjzgjv3EGDPP3X7dGNPVGHOkMabAGNMjwzWeMMbc0dyyt0W8TqJTh05NNOcISpCABChbU8bAJwY2WDPeT2lpsjPDw3Gs4evFGt93ny2fOW6chkwoSkNkzRj21bm8HDgXuEFEzk2ZNhL4xBhzOvAwkFoo99fAwmzJ2JYIFYa46qyrksYO1rI5HIZ+/ZLHPM+CoiiK0vT4m3NcddZV1Dq1ODjEnBhjFoxplEHsOTNyc5MbKgUCsGdP3ZyJE+2+hkwoSsNk0zN80DqX7v5cd/sZYIiI/actIsOBvwO6dNdIxl84nvxgcnbcgdgBbp13a70G8ciRyfsimkynKIqSTUKFIUqKSpj/zvykccc4jJ4/ulEGcThsy2WOGmWTogMB+9q3LzkkIjVkQpt1KEo62TSGG1PnMjHHjXH7FCgQkaOACcBPG7qBlvZJJlQYYsmIJYwuHp0UMrFp9yYunnNxRgUbDtuSPd27231jYMoU+O53NcZMURQlW1RWVRJ34mnjBsPYBWMbFUPstW5+5BFr7MZiVn/ffXddSIQ/ZGLxYu04qiiZaK0JdJOAh40x+xuapKV90vHi0opPLE4ad4zDHS/ckVHBhsNw1lnJY08+maxQFUVRlKajpKiE3GB68ySwjTnKN9g6aZHtkXpLZXrs2WPDIDyMgerqulJrXshEJkNY6xErCmQIwW8yGlPn0puzQ0RygI7AHuB84DoRmQJ0AhwRqTbG/DaL8rYpRvYdyaoPVyWN1Tq1lG8oz1jg/dpr4aWUxqt+hareBEVRlKbDqzBRvqGcnft38vw7zxM3dRbto2sfZef+nSzcspCYEyMvmMfi0sUZ9XdJSXLsMNSVWistrV9/e/WIo1EbQqGeY6W9kk3PcGPqXM4DRrjb1wGvGssAY0yRW+dyKvBzNYQPjXBxmFlXzqJ7x+5J42Vry+oNlxif3r8jqXaloiiK0nR4K3nPXf8cy29ezrnH1+WYx02circrqInXEDdxovFovfWIQyG46qr08dpaW0azPv2t9YgVxZI1Y7gxdS6B2dgY4S3AjwAt7N6EhIvDjCoelVTT0jEOo+aPosf0Hkx4ZULS8tvkyZkNYq0woSiKkl1ChSEGdh9Y7/FgIEhJUUm9x8ePT+8u6jjwyiv1h7tpcp2iWLIZJoEx5gXghZSxn/i2q7EF3xu6xqSsCNdO8OLSovFo0vimXZvYtGsTgtAhp0Ni+W3yZDjtNFubcvNmO9dxtMKEoihKtintVcqjax9NCpfw8Ds1MuF1Fy0vh9mzrRMDrP6uL9wtFLK6/tlnbaichkgo7ZXWmkCnNBFeXJp/+c1Ppk514TB873vJMWgPPQTXXKPhEoqiKNkiVBhi+hXTCUj6V3PMiR20bXMoBN26WQPYjzHw6KMwZozV4V7SXFmZbcixeLF9V/2utFfUGG4HhApDPPbNx8gNZM5cFiRt+a2kxC6deRgDFRXaoU5RFCWbhIvDzLhiRsaOoqs+XHXQkmte6ENqQl08bstolpTA4MFwzz3WOK6u1phhRVFjuJ0QKgyx9KaljC4eTe8TeictuRkMGz/emDw/Q7tmsEtvU6Y0h8SKoijtk3BxmNv63pac74FDxVsVDHpiUIMGsVdXeNQo26HOjzFWh9fUWO+x49ixQEBjhpX2jRrD7Qgvc3nd6HWMKh6VGK+vBnE4bAu6p3oYnn9evcOKoijZpLRXKR1yOqTFCtc6tYxbNC6hrzPVIfaacSxdCsOHJzs1jEm+jwhccomWVVPaN2oMt1NKe5WSE6jLn6x1ajO2bQ6HYebMZIM4HlfvsKIoSjYJFYZYXLqYUcWj0mKIV324ioFPDGTCKxMYUj6Ee5fcy5DyIWn6OxSC/v3THRoeItChgy2/poaw0p5RY7idEioMMW3YNAK+P4FNuzdlXILzDGK/d6GiAiZMaC5pFUVR2h/eat6MK2Yk6WqwCXUPrXiI6lh1g3WIvRjiVETg6qthxIj0Y4rS3lBjuB0TLg7T76R+SWO1Ti1TVqS7fcNh6Jc8lYcestnIiqIoSvYIF4cJF4fTxo37H0BAAmz7dFtG7/DixTZcwk+vXrBwoa0yMWSI1eXalllpr6gx3M4Z2Xdk2ljF2xVMeCXd7TsyZaoxMHasKk9FUZRsU9qrlCNyjqj3eNyJU7a2rN5wieees9Uk+ve3iXUbNthEunjcvo8dC3ffbStNqE5X2htqDLdzvLbNx+QdkzQ+ZcUUrnn6miSlmqllczwO3/lOXf1KRVEUpenxYohHF4/OWIfYwcExTlLd+NTkunDYeoi9KhJQF08cj9uxmhrboENR2hNqDCuEi8M8dNlDaeMVb1UwYM4AytbUxUJMnpy+3FZVZWOK1aOgKIqSPfwxxKl1iP0UfKWAyPZIxuS61BbMV1+dfv7OnVn6AIrSSlFjWAGsQTz+ovFp43ET544X7qBsTVnCwzB+fHJDDg/1KCiKomSfTHWIPYwxjFs0jvIN5UTj0bTkOi+G+L77bJONLmduxTHJLesWLNDVPqV9ocawkmDyJZMZfvbwtPFap5Yx88dwz5J7GFI+BLpGmD49c7meOXNUgSqKomQbrw5xapUJg6E6Vs3O/TvJC+YRlCB5wbykLqOhEEycCHSN8Pi+ERCsBuLgJuPV1tr44iFDVJ8r7QM1hpUkxl84nvxgftp4ajyaV24t1UNcW6stPRVFUbKNF0N8yamXpMUQGwzz3p7HeSefx219b2Nx6WJChemFhCurKomf/BcYMQT6PUowN153DaMtmpX2gxrDShKhwhBLRizhslMvq3dOwVcKAJuMsXx5cgyx48CiRdabEIloqR5FaSlEZKiIvC0iW0TkrgzHB4rIWhGJich1vvHeIhIRkTdF5A0R+XbzSq40llBhiEklk8gP5qd5iB0clm1dxpz1c4DMnepKikqs97jb6xwx/EdcPzI5WNgYKCjI/udQlJZGjWElDU/B5gXTK7V78WieQs3U4WjZMhgwwCbU3XuvLrUpSnMjIkFgGnA5cC5wg4icmzJtG3AT8L8p458DpcaYHsBQYKqIdMquxMoXxe8hzhRDHI1HmbJiCoOeGJQIdUvo78IQU4dOZcgpQ5g6dCq73u+adK7jwA9+oPpbafuoMaxkJFQYonJEJaOLRzOw28CEkjUYDsQOJBnEJSXp4RJe7cp4XJfaFKUF6A9sMca8b4yJAk8BSXUDjDFVxpg3ACdl/B1jzLvu9ofAx0Dn5hFb+SJ4DowOOR0yHp/3zjxqndqMpdfGLRrH4r8vZtyicfQe/F7auTU1cOutahArbRs1hpV68cr4DD19aJrHYdWHqxg8dzCR7RFCIZg2LXNCHUBOjl1q05AJRWk2Tga2+/Z3uGOHhIj0B/KAdCtJaVV4HuLLTr0sSV8bTFq1CC/UrbKqMqniRKeL/sisWXByyl/Kpk0waJDqb6XtklVjuBExa/ki8rR7/DURKXLH+4vIeve1QUSuyaacSsOUFJWQn5OeVFcTr0m0bvYS6jIZxCeeaJfaNGRCUQ4fRORE4HfAzcakWFP2eFhEVovI6l27djW/gEoafg9xppAJSA51S8QM+ypOhMPwpz9lTo5WD7HSVsmaMdzImLWRwCfGmNOBh4HJ7vjfgH7GmN7YmLVZIpKTLVmVhvE8Dv1P6p92rOLtCvrM6sOY+WPoeXkko0FcVZXc9lNDJhQl63wAFPr2u7pjjUJEjgEWAHcbY1ZmmmOMKTPG9DPG9OvcWaMoWguevh5VPCpjYw6DoSZWw6TKSQAsLl3MfYPvS6o4EQqRsXzmpk1w0UVwzTVqFCtti2x6hg8as+buz3W3nwGGiIgYYz43xsTc8Q54xQ+VFsNLtMhUdm39zvXMXDOTAXMGQHFZxpJrHo6Tnp2sVScUpcl5HThDRE4RkTzgemBeY0505z8HlBtjnsmijEqW8ELcpl8xvd7Wza/8/RVbNx6YOGBiWum1+lb7jIGKCu04qrQtsmkMNyZmLTHHNX4/BQoAROR8EXkT2AiM9hnHCXSZrnnxyq4NPyu9MQfYbnVjFoyB4jKWL7dVJlIRgT176vYjERs6oSEUitJ0uPryDuBFYDPwR2PMmyLyMxH5JoCInCciO4D/wK6+veme/i1gIHCTL1ytdwt8DOVLEi4OM+OKGeQGctPCJhzjUB2rZlLlpKRya0nnNxD+ph1HlbZEq02gM8a85pb2OQ+YKCJpabK6TNf8hApDPHf9cxlbN4NVsHe8cAd0jTB1arqHWCTZM1xZaatNaNUJRWlajDEvGGPONMacZox5wB37iTFmnrv9ujGmqzHmSGNMgatvMcb83hiTa4zp7Xutb8nPonxxwsVhlt60lEtPvTRjc46X3385UW4tUy1izyAOZLAWZs9WB4bSNsimMdyYmLXEHDcmuCOwxz/BGLMZ2A98NWuSKofM5EsmM+vKWRlj0mJOjPIN5Ym4s9zcumOOA2PH1sWclZRAXp41mvPy7L6iKIrSdHiJdTmB9NQbL4Z43KJxDJ47mHuX3JtUixisQfyXv6Sv9tXWwpQp2ZZeUbJPNo3hxsSszQNGuNvXAa8aY4x7Tg6AiHQHzgaqsiir8gUIF4dZfvPytG51BkPZ2jLK1pTR8/III39TTv+BnySW2uJxG3N28cWwcSNMnWpDJKZOtYkbiqIoStMSKgxxS+9b0sYFwcFh1YerqInXJMqsebWIE+eHrI7OS+nFVFGhZdeUwx8xJnu5aSIyDJgKBIHHjTEPiMjPgNXGmHlu6MPvgD7AXuB6Y8z7IvI94C6gFlsQ/mfGmIqG7tWvXz+zevXqrH0WpX4i2yMMKR/CgdiBtGM5gRyMMQQ/uJj4468Sj6X//srNtR7jYBBuuQVKS9UoVtomIrLGGNOvpeVoalT/Hh54uro6Vo1x89IDEkirQ5wfzOeRyx9hz+d7KCkqSUqui0TgW9+CHTuSr52bC0uXqu5WWi8N6d+sGsPNiSrjlsXrZLTqw1UZjwcIULjlPrY+OQFMAOqpgSkCHTrA4sWqVJW2hxrDSksT2R5hUuUkXvn7KzjGIUAAhCSDePhZw3nxvReJxqPkBfOSyq4BlJXBqFHp1z73XHjsMdXdSuukIf3bahPolMMLr/RabiA343EHh62n3w1XjAbiJFfLM4mXMZpIpyiKki28+OH8YD5BCZKfk88NX70hac47e9+hOlZN3MST2jd7hMMwPkMO9aZNMGCANZYV5XBCjWGlyQgVhlh601IGdhtY/6R+j8HIgXTuvjvlgAMSJxAwmkinKIqSRbzGHF6zjR6deyRVmti0a1MijMIxTqJ9s5/Jk2HWrPSya/G49Rr36KFGsXL4oMaw0qSECkMsvXkp4y8aX287UClcyc3/s8Itu2awIRMBMIIxhu9/X5fZFEVRskmoMJRotlFSVEJ+MD+jzg4QYM/nezJcoa7sWqYmS5s2WaN4woSmllxRmh41hpWsMPmSycy8cmbG7kcGwy+3X0vxbY8hwTg2bAIgiDHCww9rZrKiKEpz4XmKzzvpvLRjIpLRM+wRDsPy5TZeOBMPPVTnIdZuo0prRY1hJWukdj/yG8aOcVjV5TbMTQOgXxlIDM9LHIvZzkaqOBVFUZoHL+8jtXa8YxzGLRpXb5c6sCt5jz2WXFPewxjrIR40yLZw1m6jSmtEjWElq3jdjx74+gPMuGJGuqe4cCVcORauGJswiI2xsWgXXwz33KOKU1EUpTkIFYaYfsX0pPbNBkN1rJryDQ33Xg6FbGm10aOhd4bm3cuW2RbOX7bbqDpJlGygxrCSdbzYtHBxmDsvvDPjHOk3m3OH/jWxb4zBcWz94ZoarS6hKIrSHHgOjFHFoxLVgQyGmWtm0mN6D8rW2JiHTK2bQyGYMQPWrYPhw+u/RyAA27YdukEbiVjniHqXlaYmvTejomSRyZdMBuChFQ8lspXBNufI6/M0LLoQTA7+OsQiWl1CURSluQgVhggVhtj52U4q3qrrd7Vp1yZGzR/FxMUT+eTAJwDkBnOpHFEJQGVVZaJJx/jxMH8+xGLp16+ttat/c+ceWk35ykrrVfZ7lzXZWmkK1DOsNDuTL5nMqOJRSZnLcSfO+rwZcNbzvpnWWDbGtm1WFEVRmo8uR3bJOL73wF6M+180HmXKiikMKR/CvUvuZUj5ECLbI4RCNjRi+PD08mtg9Xp1NUya1HgPb0mJbQcdDKIlOJUmRY1hpUUo7VVKh5wOBCVoWzZ7XuKLHoJgDbYLt8VxDKNHa4keRVGU5qS0Vyk5gYMvIFdWVSaadETj0USTjlAInnuu/vJrxsBLL9nEusYYxKGQ9STfd592KVWaFjWGlRbBX/R92rBpBAOupixcCTcNJu/8J5IqTBhjmDLFZiRrnJiiKEr2CRWGWHbTMkYXj6b3CRmy4lz21exLODRyAjmUFJUkHffKrw28bDciDskdSG1eyLhxjTeIJ05UQ1hpWjRmWGkxvLg0gHUfrWPWmllWoRauJFq4EjpHYP5M7G82u862bJmtMjFjhlWwiqIoSvbw6+nI9ghTVkzh7T1vcyB2gKp9VWnze53QKzHXH0NM1wivDxwC+d+D+dPABPHnhqxaZZ0dI0dCaWnDxm7atRXlSyLGmIPPOgzo16+fWb16dUuLoXxBItsjDCkfwoHYgeQDL/8cVtzl7iQHnt14o235WVKiXgLl8EBE1hhj+rW0HE2N6t/2R2R7hAFzBhA38aRxcf8DW4WiQ04HFpcuprKqknuX3EvcxJE1YVgwHeNkiJ0Ajjii/jAI77siGo+SF8xjceliNYiVRtGQ/tUwCaVV4IVNDD8rpR7PpT+Gi36BXVZL/uH25JNw991aYkdRFKW58WoSpzbpMBgc9z+DoSZWk/Di5gXzCEqQDuf/jpnPbqJ//8zXPnDANl7KRGVVJdF4NC0+WVG+DOoZVlodZWvKmL12NlEnyoadG2zoxOpbYf4MwFO8yV7igQNh6FD1EiutG/UMK20NL2Rh1Yerksqw+endpTcXnHwBfU7sw57P9yRiisvnv8vscTdSG033EAcCmcPhUj3DU3u8xp7NPVX3KwelIf2rxrDSqjn/0fNZ9eEqu7P9AnjlQdg6yD0q1HmLrXGcl6e1J5XWixrDSlulvrAJP4IwoPsAjutwHAu3LCTmxAh+cDEXVD3D8pePJ5M50rs3FBVBly51scSeAV6w50rGfacnNVFDMCfGb596i/Dwntn7kMphTYuFSYjIUBF5W0S2iMhdGY7ni8jT7vHXRKTIHb9URNaIyEb3/evZlFNpvYzsO7Jup3Al3DzYhk2IQ135tTqjOBo1TJn2UXOLqSiK0q7xt3KuD4Nh2dZlVLxdQU28xoY6nLSUbZefR6+R05GAV0GojvXroaLClmcbPBgm/OI9Jt1fQ8GeK9mz2RrCTlyojcLt0/+U1BFPURpL1oxhEQkC04DLgXOBG0Tk3JRpI4FPjDGnAw8Dk93x3cBVxpiewAjgd9mSU2ndhIvDjL9ofPLgpT+GWy6Gs70lOVt+zaOiIsB3x+zQ/vVKu6YRzoiBIrJWRGIicl3KsREi8q77GtF8UiuHM14r559//efpersBqj6tYn3X2zE3D0DO/jOpBrFHTY1hyo+LeOnRixl17ZmUzfnMOkakFgJx4p+cRPn8d5vo0yjtiayFSYhICJhkjPmGuz8RwBjzoG/Oi+6ciIjkADuBzsYnlIgIsAc40RhTU9/9dJmubRPZHuGuV+5i2bZlyQde/jmsGE/m33VCTg7ceuvBS/UoSnPQXGESrjPiHeBSYAfwOnCDMWaTb04RcAxwJzDPGPOMO34csBroh7VK1gDFxphP6ruf6l8lE2Vrypi6cipv7X6rrrFSI+i9cSHrnx2aMuo/X5L3uy+HHf3BySE/X1jyalD1vZJGS4VJnAxs9+3vcMcyzjHGxIBPgYKUOdcCaxsyhJW2T6gwxNKbl/LXW/7KwG4DCXh/upf+GEZeDCe/Rp2HuM5LHIsZZs40WnFCaW/0B7YYY943xkSBp4Cr/ROMMVXGmDfwt3u0fAN42Riz1zWAXwZSLRNFOSjh4jCbbt/EiltWMLp4NMPPGk5Rx6KDnreh5zB63zqd7md8hgQMySFx6bki7PwaOLlgcqitDVBZ2dSfRGnrtOrSaiLSAxs6Maqe42ERWS0iq3ft2tW8wiktgmcU/+WWvzC6eLQt61O4Eob+JwRqqSvB5q82IRw4YBg3Dr47Zgdn9H+fCb947wvLEImgIRhKa6cxzogvda7qX6WxhApDzLhyBs8MdmBtAAAgAElEQVRd/xxDT0/+XdUpv1OiLrGHwbC+6+1su/FoGxJ38qqkoxafUVzTEUwAJIYTOEDBORuz9lmUtkk2jeEPgELffld3LOMcN0yiIzYkAhHpCjwHlBpjMlouxpgyY0w/Y0y/zp07N7H4SmvGU67Lb17O6OLRDLwoj05jroHuS7FeBL9RbBXmqlWGJ2eezJbXT2HKxFO/kEEcidi6xvfcaxg0uJayClW6SvtE9a/yRSjtVUp+MB9ByA/m88KNL7DilhX0Pym96LDBYLr+1To7gjVA3L6CMXcbkrzFBW/BaS8ye05tvc6K1uLM8MsR2R7hweUPavJfC5LNdsyvA2eIyClYo/d64Dspc+ZhE+QiwHXAq8YYIyKdgAXAXcaYFVmUUTnMSWoVekmEAf82gPi282DFf8NbV+Nv5WypM5Bn/vp4hg86tFjiykoS2cuOY7OXexbv1w5ISmujMc6Ihs4tSTm3skmkUto9ocIQS0YsSWunPHXoVAY+MZCYE0s/qXAl3DQYqkqgqNKObSiFtbeAk+dOEtjdA3b3YNVbMGABXHVVSkk215kRjdoynPV1ucs2fjlycuOY0onET/6LdtRrQbLmGXZjgO8AXgQ2A380xrwpIj8TkW+602YDBSKyBfgR4GU83wGcDvxERNa7r3/LlqxK28Ar7RMoXAXXXwtXjgbxSvWkL639c9cxXHihwylnfkZZWePuUVICwZyYzV4O1uJ0f1U7ICmtkYQzQkTysM6IeY0890XgMhE5VkSOBS5zxxSlSQgVhpg4YGKS0RcqDLHspmWMLh6dOa64cCUM+IV9L1wJV47l6FHfTMkXAc9THI8nl2SLRGxXu+pqiMehuiZOecXW7H/YDFRWWkM4Hrfvte9d1K466rUW77wfbbqhtDki2yNMWTGFdTvXsfVvJ7pe4m9S170O6hRn3d9/wdlvcsIpe/hh+LgGC7eXVWzk9ul/wun+KvlFa/WXvNJomrPphogMA6Zi//AfN8Y8ICI/A1YbY+aJyHnYULRjgWpgpzGmh3vuLcCP3Us9YIyZ09C9VP8qTUlke4SSuSVE41ECBPhal6/xxs43cFJyPQXBbD8f5iy1CXQpsceJeWIYNUp4/HFbix6AYA15I4dSec+DsCNEZWXzdTBN9wwPaTee4Zb0zmsHOqXd4pX22byuE2z4Huw/wWcYp4ZPuASizHr2nXoN4sj2COUbygEb/9bcilQ5fNEOdIrSOLwuc14ohad3d+7fyYf/+pDVH63GMQ4BApy19Ve887sfEI9lNoaROAMv+5QVrxQQjwPE4eTXCVx+J+G+Yeb+V2mzG2eRCInvDbpG0sJG2ioPPgj33mu94sEg3HcfTJzYPPdWY1hp93iKdNOuTSz7vzNh/gySPcXg9xbLkbvodcMzTP9Jn7qYZPcac9bPIebEyAvmMbXHa9oOVGk0agwrypcnsj3CkPIhROPRhDe14pV/8NCTqzAf9IO3hlMXBWrqXiKu30NA4hCMMvz6fzLvDyfgxIVA0HD/fZJmnKUa5soXpz7PcHM844b0bzYT6BSl1eBPtJtQOIEpJwywCRhbvgH7TqEultgaxOazzqx/bAwXPvUhR57yIkd/fRYfH/dnjDGJ4vE1VX2579kAB6rjYIKaUKcoitIMhApDLC5dnDCeAH694z8wA2Kw/QJ4dxjE831nBACT3LfD5EBMePtvX8ExUSAHh1r2dVkEDE8YZwVfKWDconEJw3tqj9fYs7nnIa8EHszYay8GdyhkDWD/amqmHzfN/QzUGFbaHZMvmcxpx5bx7KZnYcc/eOV/7sKpzcF6iv2JGAb2n8RnG0/is42XwHFbbC3jI3fDEXtx3h3GjngOVtE6EIgT6/YK5Rt2tWllpiiK0tL4HRwPLn8Qx3Hjib3KExtK3bC4+qoKGSDA5vVHkXCGOEF++dwiAH79h7U43V8l0O01HOPgGIeaqr7ccd/ZOLFDC6k4mLHX0PHI9ohtMV01iNLh3dPu54VbFJyzkT0F8zMa05EINlmwaCmlV57R4t9PoRCJ0BC2l1BZVUk0Hk1KIlRjWFGagXBxmHBxGIBIiVUmlW+t46XyntSFT/iVZwD2nmk3d/uv5Cv8bgQ2fI/Zgaco7RVpcYWjKIrSHigpKiE/J5+amG1U+2/nVPGPwtvtKt7LP4cVd5HsFoY63R2oezeC8/xvmbIAcK4CuRfnitsJnDfbNgapGkSsNohx7DJ/ZWVmY9gfDxwKcVBjr77jke0RSu6fSPTxFyCex+ypca741i66XPgSpVeeATtCDBnilvsMnEZgxALyi+5LNqYjMPjrcWpqTobgdTy+fhiV9zzYot9Pqcb/1KFTyQnk4MQdcgI5CW9/c6LGsNLuCYXsayJ9KLtmIxMnwt63eqTMypSY4S/ZFrD1LleHqV07kikdX+G5hzLfr7X8Sk9V2O2J9vzZFaWtkRo24U+4mx34H2qPfR/WjYScajhiL7x9lQ2TSGrt7K0KBt3uz2K72i2YhoOBA8djjvgYAtUEpAN5eQGb/JaCZ3xGo0JenuGRP2xiW842cgI54EBeMC/N2CspKiEvmJcwDr3jlVWV1L53EcTzbKvpqKHi953hKWvU3tJpLtFod5y4gJOL8/cBRAtXUj7/XcrfsIpt5/6d1EQ7288bN9S+d+EheV6zoStTjf91H61LhB+atB8tzYMaw4riIzy8J+HhVgGM/fHf+duaI4n9qzPpXgUPT5l6x4PgBKj41Tco2vlzLr5mE7s+28W1515LuDjMhF+8x0P3dMc4LfsrPTWJYer/1r/E1tZoLYX3FUVpOvxhE/790l6llPcpZ+f+yQB8+K8PWTVvESyYDsZfVSi9a6k1iIOwYCYYAwEHQr+CIz5j6u3fJhRKT5Yur9hqvbAmSE1NLaMf+Ct0PI7gqf257Zs96HNin0QtYU/eTMY8WCM5cGQ5cXHAxLFe7CDEc62RfO1ScnJvJO44EKyFokqc7efz6P3fIV7rfYbjIRAHsXNM0RIKvnJzo55ptnRlqvEPEHfiGAxxJ65hEorSWgiFYN2SUwAoK4PZs+GT/fvZ/a9PqakO8vmuf6NueS2Dd8EE2Pr78Wxd/hfovJmXes3hzvxn+dfM58FxFXCcxK/0g5Vna+rkCn/R95qo4fbpf8Jc/PN2UecyteB9fUudiqIc/qQayZHtEYZ8PIQDGNcg9sygOMkhcn6D2G357BiI/BfOsNt58PcRFm5ZSJez/05pr1I2rjmKZxfuoTp3BwT/HeIGAnHMuhHg5BBbGmVT158xN2iT8YKBIMNOH0aXo7pQ2qs0TU4AdoRgUTE47neNOPY9ECfwz+70ObEP/PpJZj37FqZoCRSuxMyfTrzWHyMdtIZ88WPQq5xAt1Xs+fybNIYvoysb+s7KlAA5d8PcNM94c6LGsKIchHDYvuAo4Ki6GKzqGFZ5Onhdj5K9C0HYOsi+Vt/GvzptA8e3NGcEU300ZX/eyL2P1BKPBQkE48z4ky3PlprNXBOrIRAIMG3YtES8c31kWtryK6eSkhB5eVbBBXJixLu/itOI5IWM121kAkdrCUsoKSHx2fPySCx1tpdsbkVpzyQMsYGVVF74AC/93wn2QJe18MJvfe2dHZIT71y97lhPcZUxVP0f8NX/ZWbtR27ohdR5j2s6wUd94IPzsN5cWLYsgAyoth7QeJyKtytg+wXM+mlvzukcTTR88nTRqqeHEK8tJpHcbQDiCEGcNSP5wQ3wyB8gOGikbWO9+lZYcytpxjwCHbcR6LaK/GA+BXuuZMyEg4fqFZyzkUDO2RhyyMsTCs7ZyIPLD76CaBtTLXQbU92XlhDo6dmJA+pq2GXyjDfn94bWGVaUL4AX97vTeRM+6sO8p2ydynQl5NHAv7NO78O+U+vOPfs5+v/HMl5feA4GQ6D373G6rkhMlx0XcnX+w4y/sX+9yRs2Zg0COXGmP/02PYv3p2Ure97ognM2Mu7N8+s1thM1mtd2ZMUDP8PEc8nPExYvtscTCRxyAEZcQl7RGipHVCYpNG9Oa6nFnBq3DTRLaR+tM6worQt/E6V/vf9VXnq2CzmBHKKfHsuedQNI1ul+Mo17Y3EQY2OOE/WOgbOfg4seshUvwJaBe2JJXRm4QJSiwUv44NQHiZ28vJ7ueq53mCBILcPHruPym9cz+qdrMPOn+UI/4u4pBoJRbvzVHHr0+Sf7tpzDr0YPs97jYJS8WzKH6kW2Rxg8dzA1VX0Jbh3Cf32nmKkrp1L73kXknrai3vC+SAQGDIoRrxXryb7idkaPCjLjyhmHVELN/70hwVquemAq47894EvpZa0zrChNjE266w50B6BsIIy93SEed8v7GH/Bd89I9uMLr9h3avKht4az6r669tHO2puh7+OQvw+qBmM+6kuFCfD8ozVc9cBvEst0nnG7atNOamqOB5NDvNZhzLSn6fetl6mJ1+AYh2g8SvmGcrp1rKTgsgL2fL6H75//fR6OPEzcxBm3aBw9/61nXTbz3BKiVX2h8n+g1mZcV9fEKa/YQbdO3YlGsT8EJBeqBhEtjHDXnD9z3DtFvLP3HT6PHuBAzaXg2FrMY6c9fUi1mLPiHegaYe4xQ4juijK3PI8RvUYknk9NvKZFYtYURWl+0sITfmDfxswoZ+b3+9vktaSVvxRPccaxYF1zD/+8t66GLUNhxBBrEG8o9V0fcPKoWnwpLB1o5wCcOd/XNRVb3lMMuHHCFdU/5M3/9++wYEZdOIcbSsGw2+ks53LRwFpuv2YAAAPvW0i89kpfQt1FSfqurGIjs597j4+P/yM1x9RAYYQ4hjlP9Ca6YhE4OUSXRplyxm957s50HVleDvFa1yB3ExDLugymz4ll7Pl8TyJxrqaqL5Pur+HayzeyLmc6QCJcBKzOr65xME4AHKFi0ScsODCIkX1GJs1rKtQYVpQmIByGnj0D1tNaAD/4AdTU+OOIId2TkDruV6q+xA4nD1aPItmgFuJRoeLpo6HXemb+9GvI+r6YeBACx0IgZkPgxOAcOJJVT38diqyBHq8azKPbl+N0nWUzd7dfYJUyv4Fe5VQXvsaUFVPof3J/Vn24iuhrpSnJJnGM1PLokhf5r38fQk5uUVICB6tvZdmC+9xYvC5APCmBI959MVOePoYuu0466DLdl0ngKCuDZ5+Fa6/1wlzca26PMKlyUtKPg537d+IY+0PGMQ4FXylI3L+1hHcoitJ8lF55Bo+vH0p07fWw7ibrnTVenohHqiHsH/PP80LpghDLg0UPw9EfwjtXZJgbhFg+/PlRW87TCVp9fuY8OOof0Mt6sakqsfoWePd3PwTHZwhL3IZqHOjM3lP+j3mf/ZX5TwS4sPBCYvlnubHHMQjWIqcsZV/NRYyZP4aVEWH9lF9C7ByQK+Cs6+D0hbDoN+yK5ZMIG4kbnn/xX5QNtgZuSVFJXcz01p5AgftZbAKi8/cBjF0wlqvOvIrAjouIr7seZ93NvOTk8NLjURixHgpXMmf9HJaMWAI7Qqx68yOMHAvkWOP/iN3UOrXMWjOLuRvmNvnqnYZJKEoWSMTRFsDChTBvnv0hLwGHcy56l+j+o9my7kQyh1KkeBPqHYM6JevNcZfH8vdBTUd331XgCY9CDgSj1vPwj68mJ5EEa2zBev8y3uPLUsoQ2QYjmACSU8ux1/yUve93twXuDxwHWweQFmsnsUQCB4CUv4qJ5UIwSvCmbzB99PcIF4cTBeZ3vnk2HLGb9UtOo2rt6dYQF4feg7aS120DJ/V8p8ElswkTYMqUuuc0/sH3mXzXaYlluppYDc72/kjVYHJP+yu3XHUOZWvKcHAIEOD+r99PSc7EhCEeyInRZ/ydlAzMp1N+py8cV6xhEopy+JDI29hzJXs296SgANats+XK3txRxbuV/Ul1UqTpaXE4vus/2b29Uz13qS8MI+V4v5lw5di64e0XWIP4026w+jasb9PVz92Xw44LrK4PxKDPnDojeu5ia5CLgW4rbKk5z8iuKoHF9/muBW6dOeqaUjmQUwMjhhAoXIXBIDsuxHniJevlDsTs94XjJSPG4ex51qje2RfW3uIL+3C/r05+HfrMhgPHc063E3j/D9+npgab+JfwD0UTHvUAAS459RImlUw6JD3ckP5VY1hRmoFUD2MkYrejUahTmnHX+wrpCvJghvHB4tpS58bhK7vgwPEp5YVcxXT0h3Z3Z29fu2oPX8wacTj7zxnan2a4X78yq8znT4fV4brzXUV4VPwU9udUwcKpVlknkhN9MXfec3IVY+/zqinqWARA1cYT+WDjGXQ6Lsa75T9yvTiuHMe9yzlXv8A/dsXY+0nMKv2P+roGfYwTxn4n0W47GAjyo65/ovKJQaxa3tG9TgxOewVKfgqFK8kJ5DQqkTHt/4Yaw4rSZohEYMq0j1j/3k62re6BEw9afSGGQMDYRDfH2nT1h1V4ZMo3qTvn5K/P58bvxnny90E+eO8Y2HahGxbhTQn6ruVP/vMcGDHo8Al87q+E5CMQhTMXpNRg9skocfs67t1EF1bAGtIAa25zz3O/EyA5vCPp89T3feVWzEjEWqd8h3SsgqN2QZ/ZSL/ZdMjpcEgeYjWGFaUVEonY+KqdO6FLFygthYoK+OUvDY5jfAkYB/MU10eq8j1YreTGEnffPSXneaf9Rmum68Wh41b4tDvJCtLD80A05jO7xvwxH1rDOScK//iabxkzQObPlfrM3Gsd8wF0r7RfFF/5GN683i1n5G/R7f4I6LQVOm6Hzm8xfmwXJt88vJ7nlI4aw4rSNvGvBu7ZA9u2waOP2rJk9ZPJQPbGqTsmMcb/fBtT/+c0otH6DGj/fmroXUPOEv89XcM5SZ+75+bvdVcbM+lurwayZ5zH4ax5bitsb35930eZYrDr+3w+ui9FLr2bB0qvTKpK0RBqDCvKYYRfqY4bB9XV1rMQCNiXMRCPH8q/24YM5/oUaYZ5GZXhwZRXfcZ4Q8koBzu3IRrrIW/MNRuhmIM1zHrm3UZXx1BjWFHaB4lqCDU2RC4VEcjJccPnxL7XzavTMxJw+O8HquhkTuPuuz0vs3+e1aEB13Z1HLsdzIlz0tk72PpGN/9dU67fkI6rz5Bu7PeImydiclPGG1rRbEg/Z/ix0IT6VxPoFKWV4bWHBujZM9nb4NXErawU9u2zxzp0gOOOqzt/715rQJ9xBvzxj1BbC5m9Dh6N+5V+bv/dbPrLsXVLcxnPSb2mP6Y509xMntpMMqZ+hoaoTz7/e+oxvywNLWemyBDP5dmFewg33jmsKEo7IBSyCb9+/e3FHINdCYS68DmwK4Vr18Lq1ZIwkkeFgzbfIQK5uV5onUedPrrzTujUyf9dEaSysjv3/C2TMZ6qS1N1sLe65nmL/XMyrdRlOhbgxLM+4KO3upGuV+MEcwQnHnSN+9TvnVRZ/Nf2zXfy2bO5JzSB/s2qMSwiQ4HfYF1JjxljfpFyPB8oB4qBPcC3jTFVIlIAPAOcBzxhjLkjm3IqSmvFbxinjjeG22+vU8br1tmQDBD27oVdu2DLFs8zISmeCatsRIRzzoEf/hB69jzLF+fsUaf8cnOFK66AhQuFWAyCQWHYMGHBAr9BnskjUJ8nwMkw7xARN5kjDccNQ8nk5W6MB8MlWMu1lxekjyuK0u6pT3+nzvFvp1bQ8YzmUMjqcn9o3THHwPr16RVz/OTn22uJwLHHWr0P1nvcrx+sWyeufk42NgMByMkRhg3Dp8Otg0NEfB7qTKFokJsHk/6zO+PGwYFq4w47BIKGO+/fyvBBp1FeDrNne/e3902+nr+hFRx3fA17d+cmjubnSeKHxJcla2ESIhIE3gEuBXYArwM3GGM2+eaMBb5mjBktItcD1xhjvi0iRwJ9gK8CX22MMazLdIpy6PgT+8AqWoA+feo80X5l7cU5e3M8A9uLefaUeWqyYGUl7NsHzz8Pb79tz8/Ph+9/Hx5+GGIxq5x79qzrCuf3ciQb8ySM+ZjsZ99nBzi5Sx5FJ3YEoOqjT9n9z/0MumYLR3fdys4Vl7J3dx67PttNfk4uecfsY+TNufQ8oSfl5bByJbzxRqYlSmswi9jayoGAcOpZ9n4dj4tyXOE/GHlz7iE1ENEwCUVRDkZTlnRM1fGppSqhzsAGq8tT9b9f7/s92p5e3rTJrkaWlMA//1k3z6///aubB/tO8W/PmWO/H1Ll9d+jsbRIzLCIhIBJxphvuPsTAYwxD/rmvOjOiYhIDrAT6GxcoUTkJqCfGsOK0naoz1huyVq+qQo7PSylaeRTY1hRlJakNejbQ6Ep5W2pmOGTge2+/R3A+fXNMcbERORTbLXm3Y25gYiEgTBAt27dDjJbUZTWQOrSYWOWErPNwWRoafkURVGagtagbw+F5pI3cPAprRdjTJkxpp8xpl/nzp1bWhxFURRFURTlMCObxvAHQKFvv6s7lnGOGybREZtIpyiKonwJRGSoiLwtIltE5K4Mx/NF5Gn3+GsiUuSO54rIXBHZKCKbvRA3RVGUtko2jeHXgTNE5BQRyQOuB+alzJkHjHC3rwNeNW2l8LGiKEoL4SYwTwMuB84FbhCRc1OmjQQ+McacDjwMTHbH/wPIN8b0xFb6GeUZyoqiKG2RrBnDxpgYcAfwIrAZ+KMx5k0R+ZmIfNOdNhsoEJEtwI+AhPdCRKqAXwM3iciODIpcURRFyUx/YIsx5n1jTBR4Crg6Zc7VwFx3+xlgiIh4tY2OdFfrjgCiwD+bR2xFUZTmJ6t1ho0xLwAvpIz9xLddjfVCZDq3KJuyKYqitGG+TALzM1hD+SPgK8B/GmP2pt5AE5gVRWkrHNYJdIqiKEqT0x+IAycBpwD/JSKnpk7SBGZFUdoKbaYd85o1a3aLyNZDPO14GlnGLcuoHMmoHMmoHMkcznJ0z4YgGTiUBOYdKQnM3wEWGWNqgY9FZAXQD3i/vpup/m0SVI5kVI5kVI5kmlT/thlj2BhzyK4JEVndGgrgqxwqh8qhcjQxiQRmrNF7PdbI9eMlMEfwJTCLyDbg68Dv3G6gFwBTG7qZ6l+VQ+VQOQ5nOTRMQlEUpY3xJROYpwFHicibWKN6jjHmjeb9BIqiKM1Hm/EMK4qiKHV80QRmY8z+TOOKoihtlfbuGS5raQFcVI5kVI5kVI5kVI62QWt5fipHMipHMipHMm1SDtEeF4qiKIqiKEp7pb17hhVFURRFUZR2jBrDiqIoiqIoSrulTRvDIvK4iHwsIn/zjR0nIi+LyLvu+7HuuIjIIyKyRUTeEJG+WZZjkoh8ICLr3dcw37GJrhxvi8g3mkiGQhFZIiKbRORNEfmhO96sz6MBOZr7eXQQkVUissGV46fu+Cki8pp7v6dFJM8dz3f3t7jHi7IsxxMi8nff8+jtjmft79S9flBE1onIfHe/WZ9HA3I0+/MQkSoR2ejeb7U71uz643ClHr2n+lf1r+rf+uVR/VsnQ/PqX2NMm30BA4G+wN98Y1OAu9ztu4DJ7vYwYCEg2Lqar2VZjknAnRnmngtsAPKx3Z/eA4JNIMOJQF93+2jgHfdezfo8GpCjuZ+HAEe527nAa+7n/CNwvTs+Exjjbo8FZrrb1wNPN9HzqE+OJ4DrMszP2t+pe/0fAf8LzHf3m/V5NCBHsz8PoAo4PmWs2fXH4fpC9a//uqp/k6+r+jezPKp/665dRTPq3zbtGTbGLAP2pgxfDcx1t+cCw33j5cayEugkIidmUY76uBp4yhhTY4z5O7AF2x71y8rwkTFmrbv9L2zt0ZNp5ufRgBz1ka3nYYwtIQVWCeYCBtts4Bl3PPV5eM/pGWCIiEgW5aiPrP2dikhX4ArgMXdfaObnkUmOg5C159HA/ZpVfxyuqP5NkkH1b7Icqn9TUP3bKLL276VNG8P1cIIx5iN3eydwgrt9MrDdN28HDSuJpuAO16X/uOfubw453CWVPthfwS32PFLkgGZ+Hu5S0HrgY+BlrNdjn7ENC1LvlZDDPf4pUJANOYwx3vN4wH0eD4tIfqocGWT8skwFxgOOu19ACzyPDHJ4NPfzMMBLIrJGRMLuWGvSH4cjren5qf5V/av69+ByeLRp/dsejeEExvrXW6q23AzgNKA38BHwq+a4qYgcBTwLjDPG/NN/rDmfRwY5mv15GGPixpjeQFest+PsbN+zMXKIyFeBia485wHHAROyKYOIXAl8bIxZk837fAk5mvV5uFxsjOkLXA7cLiID/QdbWH8c9qj+Vf2r+tei+jcjzap/26Mx/A/Pfe6+f+yOfwAU+uZ1dceygjHmH+4/Qgd4lLqlp6zJISK5WAX4pDHm/9zhZn8emeRoiefhYYzZBywBQtjlFa8zo/9eCTnc4x2BPVmSY6i7nGmMMTXAHLL/PC4CvikiVcBT2OW539D8zyNNDhH5fQs8D4wxH7jvHwPPufdsFfrjMKZVPD/Vv6p/G5BD9W871L/t0RieB4xwt0cAf/aNl7pZiRcAn/rc8U1OSjzLNYCX6TwPuF5stugpwBnAqia4nwCzgc3GmF/7DjXr86hPjhZ4Hp1FpJO7fQRwKTZ+bglwnTst9Xl4z+k64FX3l2k25HjL9w9esHFR/ufR5P9fjDETjTFdjTFF2ISMV40xN9LMz6MeOb7b3M9DRI4UkaO9beAy956tQn8cxrSK56f6V/VvA3Ko/m2P+tc0YQZia3sBf8Au+dRiY0hGYuNqFgPvAq8Ax7lzBZiGjVvaCPTLshy/c+/zhvs/8kTf/LtdOd4GLm8iGS7GLim8Aax3X8Oa+3k0IEdzP4+vAevc+/0N+Ik7fipW2W8B/gTku+Md3P0t7vFTsyzHq+7z+Bvwe+oynrP2d+qTqYS6LOJmfR4NyNGsz8P93Bvc15vA3e54s+uPw/WF6l+/DKp/k+VQ/Vu/TCWo/m12/avtmBVFURRFUZR2S3sMk1AURVEURVEUQI1hRVEURVEUpR2jxrCiKIqiKP+FhGgAAAJYSURBVIrSblFjWFEURVEURWm3qDGsKIqiKIqitFvUGFbaJCISF5H1vtddTXjtIhH528FnKoqitD9U/yqHGzkHn6IohyUHjG2xqSiKojQvqn+Vwwr1DCvtChGpEpEpIrJRRFaJyOnueJGIvCoib4jIYhHp5o6fICLPicgG93Whe6mgiDwqIm+KyEtu9yJE5Acissm9zlMt9DEVRVFaHap/ldaKGsNKW+WIlGW6b/uOfWqM6Qn8Fpjqjv0/YK4x5mvAk8Aj7vgjwFJjTC+gL7YbDti2pNOMMT2AfcC17vhdQB/3OqOz9eEURVFaMap/lcMK7UCntElEZL8x5qgM41XA140x74tILrDTGFMgIruxLUhr3fGPjDHHi8guoKsxpsZ3jSLgZWPMGe7+BCDXGHO/iCwC9gMVQIUxZn+WP6qiKEqrQvWvcrihnmGlPWLq2T4Uanzbceri76/A9kjvC7wuIhqXryiKUofqX6XVocaw0h75tu894m7/Fbje3b4RWO5uLwbGAIhIUEQ61ndREQkAhcaYJcAEoCOQ5h1RFEVpx6j+VVod+qtJaascISLrffuLjDFeeZ9jReQNrHfhBnfs+8AcEflvYBdwszv+Q6BMREZiPRBjgI/quWcQ+L2rsAV4xBizr8k+kaIoyuGB6l/lsEJjhpV2hRuz1s8Ys7ulZVEURWlPqP5VWisaJqEoiqIoiqK0W9QzrCiKoiiKorRb1DOsKIqiKIqitFvUGFYURVEURVHaLWoMK4qiKIqiKO0WNYYVRVEURVGUdosaw4qiKIqiKEq75f8Dse0eSz6Q6GgAAAAASUVORK5CYII=\n",
             "text/plain": [
-              "<Figure size 432x288 with 1 Axes>"
-            ]
-          },
-          "metadata": {
-            "tags": []
-          }
-        },
-        {
-          "output_type": "display_data",
-          "data": {
-            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOyde3hU5bX/P2smCYh4qQGLQjR4F0UI\n4dIRgVBqi0oFvFRsTwNFHUGxRauAtipHT2tBzynHIxdRRPFgwZ9Wioq1RySAOOUeRBB60AaDgmIU\nhAPkMrN+f+w9ycxkkkxCJtf1eZ482fvd7957zQ7s77zvetdaoqoYhmEYRiyexjbAMAzDaJqYQBiG\nYRhxMYEwDMMw4mICYRiGYcTFBMIwDMOIiwmEYRiGERcTCKNBEJG3RGR0ffdtTESkQER+kITrqoic\n527PEZEHE+lbh/v8TET+Vlc7q7lujojsqe/rGg1PSmMbYDRdRORwxG47oBgIuvu3q+rCRK+lqlcl\no29LR1XH1cd1RCQT+CeQqqpl7rUXAgn/DY3WhwmEUSWq2j68LSIFwK2q+k5sPxFJCb90DMNoOdgU\nk1FrwlMIIjJZRPYB80XkOyLyhojsF5Fv3O0uEefkicit7vYYEXlPRJ5w+/5TRK6qY9+uIrJKRA6J\nyDsiMlNE/rsKuxOx8VERWeNe728i0iHi+M9FZLeIFInIb6p5Pv1EZJ+IeCPaRorIB+52XxEJiMgB\nEdkrIk+JSFoV13peRP4tYv8+95zPRWRsTN9rRGSziHwrIoUiMjXi8Cr39wEROSwivvCzjTj/chFZ\nLyIH3d+XJ/psqkNELnbPPyAi20Tk2ohjV4vIdvean4nIvW57B/fvc0BEvhaR1SJi76sGxh64UVc6\nAacBZwN+nH9L8939s4CjwFPVnN8P2Al0AKYD80RE6tD3JWAdkA5MBX5ezT0TsfGnwC+A04E0IPzC\n6gbMdq9/pnu/LsRBVdcC/wd8P+a6L7nbQeBu9/P4gCHAHdXYjWvDUNeeK4HzgVj/x/8BucCpwDXA\neBEZ4R4b6P4+VVXbq2og5tqnAW8CT7qf7T+AN0UkPeYzVHo2NdicCrwO/M097y5goYhc6HaZhzNd\neRJwKfCu2/5rYA/QEfgu8ABgeYEaGBMIo66EgIdVtVhVj6pqkaq+qqpHVPUQ8DtgUDXn71bVZ1Q1\nCLwAnIHzIki4r4icBfQBHlLVElV9D1ha1Q0TtHG+qv5DVY8CLwM93fYbgDdUdZWqFgMPus+gKv4E\n3AwgIicBV7ttqOpGVf27qpapagHwdBw74vET174PVfX/cAQx8vPlqepWVQ2p6gfu/RK5LjiC8r+q\n+qJr15+AHcCPI/pU9Wyq43tAe+AP7t/oXeAN3GcDlALdRORkVf1GVTdFtJ8BnK2qpaq6Wi1xXINj\nAmHUlf2qeiy8IyLtRORpdwrmW5wpjVMjp1li2BfeUNUj7mb7WvY9E/g6og2gsCqDE7RxX8T2kQib\nzoy8tvuCLqrqXjijhetEpA1wHbBJVXe7dlzgTp/sc+34Pc5ooiaibAB2x3y+fiKywp1COwiMS/C6\n4WvvjmnbDXSO2K/q2dRos6pGimnkda/HEc/dIrJSRHxu++PALuBvIvKJiExJ7GMY9YkJhFFXYr/N\n/Rq4EOinqidTMaVR1bRRfbAXOE1E2kW0ZVTT/3hs3Bt5bfee6VV1VtXtOC/Cq4ieXgJnqmoHcL5r\nxwN1sQFnmiySl3BGUBmqegowJ+K6NX37/hxn6i2Ss4DPErCrputmxPgPyq+rqutVdTjO9NMSnJEJ\nqnpIVX+tqucA1wL3iMiQ47TFqCUmEEZ9cRLOnP4Bdz774WTf0P1GvgGYKiJp7rfPH1dzyvHY+Aow\nTESucB3Kj1Dz/5+XgF/hCNH/i7HjW+CwiFwEjE/QhpeBMSLSzRWoWPtPwhlRHRORvjjCFGY/zpTY\nOVVcexlwgYj8VERSROQmoBvOdNDxsBZntDFJRFJFJAfnb7TI/Zv9TEROUdVSnGcSAhCRYSJynutr\nOojjt6luSs9IAiYQRn0xAzgB+Ar4O/DXBrrvz3AcvUXAvwGLceI14lFnG1V1G3Anzkt/L/ANjhO1\nOsI+gHdV9auI9ntxXt6HgGdcmxOx4S33M7yLM/3ybkyXO4BHROQQ8BDut3H33CM4Ppc17sqg78Vc\nuwgYhjPKKgImAcNi7K41qlqCIwhX4Tz3WUCuqu5wu/wcKHCn2sbh/D3BccK/AxwGAsAsVV1xPLYY\ntUfM72O0JERkMbBDVZM+gjGMlo6NIIxmjYj0EZFzRcTjLgMdjjOXbRjGcWKR1EZzpxPwZxyH8R5g\nvKpublyTDKNlYFNMhmEYRlxsiskwDMOIS4uZYurQoYNmZmY2thmGYRjNio0bN36lqh3jHWsxApGZ\nmcmGDRsa2wzDMIxmhYjERtCXY1NMhmEYRlxMIAzDMIy4mEAYhmEYcUmqD8INXPpPwAs8q6p/iDne\nBlgAZOOE99+kqgVurpungd44+Vd+pap5ybTVMIzaU1payp49ezh27FjNnY1GpW3btnTp0oXU1NSE\nz0maQLgplGfiFDfZA6wXkaVulsswtwDfqOp5IjIKmAbcBNwGoKrdReR04C0R6ROTMtgwjEZmz549\nnHTSSWRmZlJ1vSejsVFVioqK2LNnD127dk34vGROMfUFdqnqJ27CrkU4aRAiGY5TAAacbJlD3OyN\n3XATkanql8ABnNGEYRhNiGPHjpGenm7i0MQREdLT02s90kumQHQmurjJHqKLj0T1cYveH8RJmbAF\nuNZNO9wVZwqqUp5/EfGLyAYR2bB///46GxoIwGOPOb8Nw6gdJg7Ng7r8nZpqHMRzwMU4uf53A+/j\n5IOPQlXnAnMBevfuXaecIYEADBkCJSWQlgbLl4PPV/N5hmEYLZ1kjiA+I/pbfxcqV6cq7yMiKcAp\nQJFbE/duVe3pVps6FfhHMozMy4PiYggGnd95ecm4i2EYyaCoqIiePXvSs2dPOnXqROfOncv3S0pK\nqj13w4YN/PKXv6zxHpdffnm92JqXl8ewYcPq5VoNRTJHEOuB890pos+AUURXuAKnPOJonIIgN+AU\nVlG3Wpao6v+JyJVAWYxzu95IT4eQ6/oOhZx9wzCaB+np6eTn5wMwdepU2rdvz7333lt+vKysjJSU\n+K+53r1707t3za7N999/v36MbYYkbQTh+hQmAG8DHwEvq+o2EXlERK51u80D0kVkF3APEC5Mfjqw\nSUQ+AibjVJ1KCkVF4HGfgghstkTRhpFUAoUBHlv9GIHC5Dj9xowZw7hx4+jXrx+TJk1i3bp1+Hw+\nsrKyuPzyy9m5cycQ/Y1+6tSpjB07lpycHM455xyefPLJ8uu1b9++vH9OTg433HADF110ET/72c8I\nZ8NetmwZF110EdnZ2fzyl7+scaTw9ddfM2LECC677DK+973v8cEHHwCwcuXK8hFQVlYWhw4dYu/e\nvQwcOJCePXty6aWXsnr16np/ZlWRVB+Eqi7DqXUb2fZQxPYx4MY45xXgFJdPOjk5kJLi+CBU4Zln\nICsL/P6GuLthtC4ChQGGLBhCSbCENG8ay3OX48uof6ffnj17eP/99/F6vXz77besXr2alJQU3nnn\nHR544AFeffXVSufs2LGDFStWcOjQIS688ELGjx9fKWZg8+bNbNu2jTPPPJP+/fuzZs0aevfuze23\n386qVavo2rUrN998c432Pfzww2RlZbFkyRLeffddcnNzyc/P54knnmDmzJn079+fw4cP07ZtW+bO\nncuPfvQjfvOb3xAMBjly5Ei9PaeaaPWR1D4fjB1bsR8MwoQJtqLJMJJBXkEeJcESghqkJFhCXkFe\nUu5z44034vV6ATh48CA33ngjl156KXfffTfbtm2Le84111xDmzZt6NChA6effjpffPFFpT59+/al\nS5cueDweevbsSUFBATt27OCcc84pjy9IRCDee+89fv5zZ2Lk+9//PkVFRXz77bf079+fe+65hyef\nfJIDBw6QkpJCnz59mD9/PlOnTmXr1q2cdNJJdX0stabVCwRAbq4zighTVgYLFjSePYbRUsnJzCHN\nm4ZXvKR508jJzEnKfU488cTy7QcffJDBgwfz4Ycf8vrrr1cZC9CmTZvyba/XS1lZWZ36HA9Tpkzh\n2Wef5ejRo/Tv358dO3YwcOBAVq1aRefOnRkzZgwLGvDlZAKBM4qYORPcLxyowrx5NoowjPrGl+Fj\nee5yHh38aNKml2I5ePAgnTs7IVjPP/98vV//wgsv5JNPPqGgoACAxYsX13jOgAEDWLhwIeD4Njp0\n6MDJJ5/Mxx9/TPfu3Zk8eTJ9+vRhx44d7N69m+9+97vcdttt3HrrrWzatKneP0NVmEC4+P3w4x9X\n7JeW2ijCMJKBL8PH/QPubxBxAJg0aRL3338/WVlZ9f6NH+CEE05g1qxZDB06lOzsbE466SROOeWU\nas+ZOnUqGzdu5LLLLmPKlCm88IKTUGLGjBlceumlXHbZZaSmpnLVVVeRl5dHjx49yMrKYvHixfzq\nV7+q989QFS2mJnXv3r31eAsGjR8Pc+ZU7I8bB7NnH6dhhtGC+eijj7j44osb24xG5/Dhw7Rv3x5V\n5c477+T888/n7rvvbmyzKhHv7yUiG1U17npfG0FEkJsL4SlGr9dZzWQYhlETzzzzDD179uSSSy7h\n4MGD3H777Y1tUr1gAhGBzwdPPgmpqY4fYuJE80MYhlEzd999N/n5+Wzfvp2FCxfSrl27xjapXjCB\niKGoyImoDoXg2DHzQxiG0XoxgYghJyd6NdMzz8DcuY1qkmEYRqNgAhGDBc4ZhmE4mEDEIV7gnGV5\nNQyjtWECEQefD+65p2JfFQ4caDx7DMOIz+DBg3n77bej2mbMmMH48eOrPCcnJ4fwkvirr76aA3H+\nc0+dOpUnnnii2nsvWbKE7dsrkkw/9NBDvPPOO7UxPy5NKS24CUQVnHqqk901zBNPmC/CMJoaN998\nM4sWLYpqW7RoUUL5kMDJwnrqqafW6d6xAvHII4/wgx/8oE7XaqqYQFRBpLManFVN5oswjOOnPkv8\n3nDDDbz55pvlxYEKCgr4/PPPGTBgAOPHj6d3795ccsklPPzww3HPz8zM5KuvvgLgd7/7HRdccAFX\nXHFFeUpwcGIc+vTpQ48ePbj++us5cuQI77//PkuXLuW+++6jZ8+efPzxx4wZM4ZXXnkFgOXLl5OV\nlUX37t0ZO3YsxcXF5fd7+OGH6dWrF927d2fHjh3Vfr7GTgtuAlEF4fxMnognFAyaL8Iwjodwid8H\nH3R+H69InHbaafTt25e33noLcEYPP/nJTxARfve737FhwwY++OADVq5cWf5yjcfGjRtZtGgR+fn5\nLFu2jPXr15cfu+6661i/fj1btmzh4osvZt68eVx++eVce+21PP744+Tn53PuueeW9z927Bhjxoxh\n8eLFbN26lbKyMmZHpGTo0KEDmzZtYvz48TVOY4XTgn/wwQf8/ve/Jzc3F6A8LXh+fj6rV6/mhBNO\n4KWXXuJHP/oR+fn5bNmyhZ49e9bpmUZiAlENfr+TaiM11Zlu8nis4pxhHA95eU7tlWDQ+V0fX7gi\np5kip5defvllevXqRVZWFtu2bYuaDopl9erVjBw5knbt2nHyySdz7bXXlh/78MMPGTBgAN27d2fh\nwoVVpgsPs3PnTrp27coFF1wAwOjRo1m1alX58euuuw6A7Ozs8gR/VdHYacGTKhAiMlREdorILhGZ\nEud4GxFZ7B5fKyKZbnuqiLwgIltF5CMRuT+ZdlaH3w9PPeWIQzAId91l00yGUVdyciAtzZm+TUtz\n9o+X4cOHs3z5cjZt2sSRI0fIzs7mn//8J0888QTLly/ngw8+4JprrqkyzXdNjBkzhqeeeoqtW7fy\n8MMP1/k6YcIpw48nXXhDpQVPmkCIiBeYCVwFdANuFpFuMd1uAb5R1fOAPwLT3PYbgTaq2h3IBm4P\ni0djsHmzIw6qzrcei642jLrh88Hy5fDoo85vXz0kdG3fvj2DBw9m7Nix5aOHb7/9lhNPPJFTTjmF\nL774onwKqioGDhzIkiVLOHr0KIcOHeL1118vP3bo0CHOOOMMSktLy1N0A5x00kkcOnSo0rUuvPBC\nCgoK2LVrFwAvvvgigwYNqtNna+y04MksOdoX2KWqnwCIyCJgOBA5zhsOTHW3XwGeEhEBFDhRRFKA\nE4AS4Nsk2lor9u1rbAsMo/ni89WPMERy8803M3LkyPKppnB67IsuuoiMjAz69+9f7fm9evXipptu\nokePHpx++un06dOn/Nijjz5Kv3796NixI/369SsXhVGjRnHbbbfx5JNPljunAdq2bcv8+fO58cYb\nKSsro0+fPowbN65OnytcK/uyyy6jXbt2UWnBV6xYgcfj4ZJLLuGqq65i0aJFPP7446SmptK+fft6\nGUEkLd23iNwADFXVW939nwP9VHVCRJ8P3T573P2PgX7AQeBFYAjQDrhbVatdZFof6b6rIhCAQYOc\nGhHgDI9nzbK61YZh6b6bFy0l3XdfIAicCXQFfi0i58R2EhG/iGwQkQ379+9PmjE+H9xyS8W+pd8w\nDKM1kEyB+AzIiNjv4rbF7eNOJ50CFAE/Bf6qqqWq+iWwBqikcKo6V1V7q2rvjh07JuEjVGB1qw3D\naG0kUyDWA+eLSFcRSQNGAUtj+iwFRrvbNwDvqjPn9SnwfQARORH4HlB9RMlxECgM8NjqxwgUVj0k\niFe3ev58G0UYRkupStnSqcvfKWlOalUtE5EJwNuAF3hOVbeJyCPABlVdCswDXhSRXcDXOCICzuqn\n+SKyDRBgvqpWHeVyHAQKAwxZMISSYAlp3rRqC6n7/c6Kpqefjl7RVN8ON8NoLrRt25aioiLS09OR\nyNw0RpNCVSkqKqJt27a1Oi+Zq5hQ1WXAspi2hyK2j+EsaY0973C89mSQV5BHSbCEoAYpCZaQV5BX\nbTH13Fx47jlHHMKjiNxcEwmjddKlSxf27NlDMn2ARv3Qtm1bunTpUqtzkioQzYGczBzSvGnlI4ic\nzJxq+4frRcyZ4+yXljrRoCYQRmskNTWVrl27NrYZRpJo9QLhy/AxY+gMXt3+Ktd3u77a0UOYrKyK\n7VDIUoEbhtEyafUCESgMMPGvEykJlrD609V0P717jSJRVOTkZgr7fJ54As491+IiDMNoWTTVOIgG\nI54PoiYsFbhhGK2BVi8QYR+EV7wJ+SAgfipwi4swDKOlkbRUGw3N8aTaCBQGyCvIIyczJyEfRJi5\nc+GOO5zIanDSgq9caQ5rwzCaD80x1UaD4svwkZOZQ15BXrXBcrH4/fDjH1fsl5baKMIwjJZDq3dS\nQ0WwXHFZMR6Ph5lXz8SfnZjHuVOn6H3L9GoYRkvBRhA4jurismJChCgLlTFh2YSERxK5uc7UUpil\nS2HkSHNYG4bR/DGBwHFUeyI8zkENJrSaCSpneg2FYMkSGDzYRMIwjOaNCQSOD2Lm1TNJ9VQMBQ4U\nJx79FpvpFaC4uH7q7RqGYTQWJhAu/mw/d/vuBiCkIaavmc7cjdXWKCon3rJXgPT0+rbSMAyj4TCB\niCB/b37U/rxN8xI+1+93fsIJLUWczK+GYRjNFROICK7vdn3U/sa9Gxn/xvg6OaxV4ZlnnFgJwzCM\n5ogJRAT+bD8jLhpRvh/UIHM2zmHA/AEJTTeFM72Wn2+lSQ3DaMaYQMQw6fJJpHnTotqCGuSON+9I\naCRhpUkNw2gpmEDE4MvwMbbn2ErtIQ0ltPQ1XmnSefNsFGEYRvMjqQIhIkNFZKeI7BKRKXGOtxGR\nxe7xtSKS6bb/TETyI35CItIzmbZGktsjt9IoItFEfmApOAzDaBkkLdWGiHhxaktfCewB1ovIUlXd\nHtHtFuAbVT1PREYB04CbVHUhsNC9TndgiapGLzFKIr4MH3mj81iwZQH7DlfkzliwZUH58ZqwFByG\nYTR3kpbNVUR8wFRV/ZG7fz+Aqj4W0edtt09ARFKAfUBHjTBKRH7vnKa/qe5+x5PNtToChQFyXsih\nJFgCQBtvG1aMXlGjSAQCMGiQM3oAZ8pp1iwrKmQYRtOisbK5dgYKI/b3uG1x+6hqGXAQiA0vuwn4\nU7wbiIhfRDaIyIZkFU3PK8ijNFhavl8cLE7YFxGZgsNWNBmG0dxo0k5qEekHHFHVD+MdV9W5qtpb\nVXt37NgxKTbkZObg9Xij2tLbJRYibSuaDMNoziRTID4DMiL2u7htcfu4U0ynAEURx0dRxeihofBl\n+Lg161YEJ0TaIx6KjhTVcJZ7bpwVTXPnWvCcYRjNg2QKxHrgfBHpKiJpOC/7pTF9lgKj3e0bgHfD\n/gcR8QA/ARYl0caEyO2RS9uUtnjFSxtvm4RXM0HlFU2hkFOFzqaaDMNo6iRtFZOqlonIBOBtwAs8\np6rbROQRYIOqLgXmAS+KyC7gaxwRCTMQKFTVT5JlY6L4Mnwsz11evqqpNquZoPKKpmDQmWqy0qSG\nYTRlrCZ1gsSuZvKKl1nXzEqo8lzsiiawVU2GYTQNrCZ1PRC7mimowYQrz/l8sHIl9O1b0WarmgzD\naOqYQCRITmYOqd7UqLayUFn5dFNN+HwwY4atajIMo/lgApEg4ejqEReOwOM+NkWZnz8/ahQRKAzw\n2OrH4o4sLE+TYRjNCROIWuDL8PHaqNei/A6lwdLywLlAYYAhC4bw4IoHGbJgSFyRsDxNhmE0F0wg\n6kDWGVnl2yFCrPt8HYHCAHkFeZQESwhqkJJgSZUR15anyTCM5oAJRB0oOlKERyoe3V92/IUhC4aQ\n3i6dNG8aXvFWm/01svIcwOuvW/CcYRhNDxOIOpCTmUOKp8LbrCjFwWKKjhSxPHc5jw5+lOW5y6uM\nk7A8TYZhNAdMIOpAVUWFDhQfIK8gj5zMnBqD6GLzNJWWwvTp9W2pYRhG3TGBqCO5PXI5IeWE8hxN\nIQ0xfc10fvPub6p0UEcSXtHkifgLLFliU02GYTQdTCDqSDj9Rp8z+0S1h6ebEkkJ7vdD75j4xXnz\n6tFIwzCM48AE4jjwZfjodUavSu1e8Sac0C/SFwGwYYONIgzDaBqYQBwnuT1ySfVULEnyipenrn4q\n4UR+fj+MGFGxHwrB+PEmEoZhND6WrK8eCBQGylNuZJ2Rxea9mwFHPBIRikAABg50Um+E8Xph9WrL\n+GoYRnKpLlmfCUQ9Utf61eCMGMaNc9JvhBk3DmbPTpa1hmEYls21wYjN+FpdNHUsfj8MHx7dZhHW\nhmE0JiYQ9UhsxlePeBKuXw0waVJ0hPXSpTBypAXQGYbROCRVIERkqIjsFJFdIjIlzvE2IrLYPb5W\nRDIjjl0mIgER2SYiW0WkbTJtrQ8iM756xYuiTPzrxIRqRkDlCOtQyImNGDzYRMIwjIYnaQIhIl5g\nJnAV0A24WUS6xXS7BfhGVc8D/ghMc89NAf4bGKeqlwA5QCnNAF+Gj76dncpAIQ1xtOwo09ckHiId\nG2ENUFwMeXn1aKRhGEYCJHME0RfYpaqfqGoJsAiImWVnOPCCu/0KMEREBPgh8IGqbgFQ1SJVDSbR\n1nolJzMH52M4LNm5hLkbE1u3Gi/CGiA98ZkqwzCMeiGZAtEZKIzY3+O2xe2jqmXAQSAduABQEXlb\nRDaJyKR4NxARv4hsEJEN+/fvr/cPUFd8GT56dYoOoHt05aMJTzX5/c5PWGNEYPPm+rbSMAyjepqq\nkzoFuAL4mft7pIgMie2kqnNVtbeq9u7YsWND21gtt/SKDpHec2gPA+YPSHgkEZkSXBWeecaC5wzD\naFiSKRCfARkR+13ctrh9XL/DKUARzmhjlap+papHgGVA5ZwWTRh/tp8RF42IagtqkDvevCOhkYTP\nB2MjEsYGgxZhbRhGw5JMgVgPnC8iXUUkDRgFLI3psxQY7W7fALyrTuTe20B3EWnnCscgYHsSbU0K\nky6fRJo3LaotqMHyqOuaiHVYh0JO8JyJhGEYDUHSBML1KUzAedl/BLysqttE5BERudbtNg9IF5Fd\nwD3AFPfcb4D/wBGZfGCTqr6ZLFuTRXjZa7cOsYu3EjzfdVhH+LtRdUYS48fb0lfDMJKLpdpoAAKF\nAQa/MJjiYDFe8TLrmln4s/0Jnz9ypBMPEYkItG0Ly5dbvibDMOqOpdpoZHwZPp686klSPamENMSd\ny+5M2FkNlSOswRlJWHyEYRjJxASigSg6UkQwFERRykJljH9zfK1iI1auhL59Kx+z+AjDMJKFCUQD\nkZOZgyci+i2koVqLxIwZcMIJFT6JUAjuust8EYZhJAcTiAbCl+Fj5tUz8Ui0SCS67BUckVi+HPpE\nVDktKYEFiS2KMgzDqBUmEA2IP9vP7GtmI1QsSwpqkFuX3lorkegVExFiacENw0gGJhANjD/bz/CL\nolNSbf9qe52jrMHSghuGkRxMIBqBSZdPwiveqLagBpmwbELCUdaWFtwwjGRjAtEI+DJ8zLpmViWR\nKAuVJVyBztKCG4aRbEwgGgl/tp/Vv1jNwLMGlrcpmnAFOksLbhhGsjGBaER8GT6Gnjc0amXTvE3z\n6pwWHGDePJtmMgyjfjCBaGRyMnNI8VTMFa37fB2DXxicsEjEOqzXrYMBAyyhn2EYx09CAiEiJ4o4\nX3NF5AIRuVZEUms6z6gZX4aPsT3HRrUVB4uZvmY6j61+rEahiE0LDk5q8AkTbCRhGMbxkegIYhXQ\nVkQ6A38Dfg48nyyjWhu5PXIrpQVfsnMJv13xW4YsGFKjSOTmQlr06ZSVmcPaMIzjI1GBELdwz3XA\nLFW9EbgkeWa1LsJpwS/ucHFUe0hDlARLalzZ5PM5YjCwwt+NKhw4UP+2GobRekhYIETEh1MCNFyX\nwVtNf6OW+DJ8DDp7UNxjiaxs8vlg6NBoh/UTT5gvwjCMupOoQEwE7gdec4v+nAOsSJ5ZrZPcHrlx\nA+jueuuuhJzWOTngjTg9FLIypYZh1J2EBEJVV6rqtao6zXVWf6Wqv0yyba2OcABdqifa/18SLGH6\nmuk1nx8nNiIUgjvuMIe1YRi1J9FVTC+JyMkiciLwIbBdRO5L4LyhIrJTRHaJyJQ4x9uIyGL3+FoR\nyXTbM0XkqIjkuz9zavexmsV+5DkAACAASURBVC/+bD8rx6yk75nRxR9e/8frUaOIQGEg7ionvx9m\nz46eagoG4dZbTSQMw6gdCZUcFZF8Ve0pIj8DeuHUjt6oqpdVc44X+AdwJbAHp770zaq6PaLPHcBl\nqjpOREYBI1X1Jlco3lDVSxP9IE255GhdCBQGGDB/AEENlrf1PbMvM4bOAGDIgiGUBEtI86axPHc5\nvozouqPxypS2aQMrVliJUsMwKqiPkqOpbtzDCGCpqpYCNSlLX2CXqn6iqiXAImB4TJ/hwAvu9ivA\nEJHI776tl3j5mtZ9vo5Bzw9iwZYFlARLCGqwylVOkyZF+yPAydVktSMMw0iURAXiaaAAOBFYJSJn\nA9/WcE5noDBif4/bFrePqpYBB4Hwkp2uIrJZRFaKyIB4NxARv4hsEJEN+/fvT/CjNB/82X5u63Vb\nVFtpqJTt+7eT5k3DK17SvGnkZOZUOtfng1mzKovE00/D5MlJNNowjBZDok7qJ1W1s6perQ67gcFJ\ntGsvcJaqZgH3AC+JyMlx7Jqrqr1VtXfHjh2TaE7jEW9l06pPV9Gncx9u63Vb3OmlMH4/rF4dXcta\nFaZPt/oRhmHUTKJO6lNE5D/C39ZF5N9xRhPV8RmQEbHfxW2L20dEUoBTgCJVLVbVIgBV3Qh8DFyQ\niK0tjfBUU2QVOoBVu1cxP39+zee7taxjs74uWQJDhphIGIZRNYlOMT0HHAJ+4v58C9T0dloPnC8i\nXUUkDRgFLI3psxQY7W7fALyrqioiHV0nN27MxfnAJwna2uLwZ/uZM2xOVNZXcHI2LdhSs1PB54N7\n763cbvUjDMOojkQF4lxVfdh1OH+iqv8KnFPdCa5PYQLwNvAR8LIbZPeIiFzrdpsHpIvILpyppPBS\n2IHAByKSj+O8HqeqX9fuo7UswvWsY6ebntn0TEKlSqdNgxEjKrdb/QjDMKoi0WWuAeA+VX3P3e8P\nPKGqTWbBZEtb5loVgcIAE/86kXWfrytv84qX23rdRm6P3Cr9EeBMJw0ZAseOOb4IcJL85eXZ0lfD\naK3UxzLXccBMESkQkQLgKeD2erLPqAW+DB8zhs6IqiER1CBzNs5h4PMDqx1N+HywfDn06VPRVlLi\nOK0NwzBiSXQV0xZV7QFchhPYlgV8P6mWGVXiy/Ax8+qZlVJylIXKGPfGOEYuHlll7iafD3r1im77\ny18sX5NhGJWpVUU5Vf1WVcPxD/ckwR4jQfzZfp66+ik8MX9CRVmyY0m1Velyc6PjI1QtqZ9hGJU5\nnpKjFvHcyBQdKUKrCGivro5EOIjOkvoZhlEdxyMQNXu3jaSSk5lDqjd+5deqIqzDVJXUz/wRhmGE\nqXYVk4gcIr4QCHCCqqbEOdYotJZVTLEECgPlsRBZZ2Tx1v++xeeHPueWXrfgz/bXeH5sUj8RmDPH\nERDDMFo+1a1iSmiZa3OgtQpEJIHCQI1ZXiudE4ABA5zRQxiPxxldmEgYRsunPpa5Gs2AvIK88iyv\nR8uOMvGvE2usRFeVP2LcOHNaG0ZrxwSiBZGTmYPXE50efMD8ATVGWsfzR6ia09owWjsmEC0IX4aP\nsT3HRrUFNci4N8YlJBLDY6p1BIMwcaKJhGG0VkwgWhi5PXJJ86ZFtSnKuDfGMfmdyXHLlIaZNAlS\nYxZFrVsHV1xh002G0RoxJ3ULJFAY4Nalt7L9q+2VjglC25S2VTqwAwFn1LBuXXS71+vUlrCcTYbR\nsjAndSvDl+Hj2WufrZSKA5zRRHGwuNoguhkzKleiCwatXKlhtDZMIFoovgwfK8esZMSFIyoVG1JV\nDhQfqHK6Kd7KJoBnnrGpJsNoTdgUUytg7sa53PHmHQQ1GNVel+kmi5EwjJaFTTG1cvzZfm7rdVul\n9kSnm1Ii4uUtRsIwWg9JFQgRGSoiO0Vkl4hMiXO8jYgsdo+vFZHMmONnichhEYlTMNOoDfFWN4Ez\niqguZ5PPBzNnWoyEYbRGkiYQbk3pmcBVQDfgZhHpFtPtFuAbVT0P+CMwLeb4fwBvJcvG1oQvw0fe\n6DxGXDgiKkV4SENMf396tRHXVcVImNPaMFo2yRxB9AV2uTWsS4BFQMxrhuHAC+72K8AQEee7qoiM\nAP4JbEuija0KX4aP10a9FpXEL1w/oqaI63gxEk8/DZMnJ8tawzAam2QKRGegMGJ/j9sWt4+qlgEH\ngXQRaQ9MBv61uhuIiF9ENojIhv3799eb4S2d3B65USVLwYm4vuPNO6qtRLdyJfTtW9Gm6qQHnzwZ\nHnvMppwMo6XRVJ3UU4E/qurh6jqp6lxV7a2qvTt27NgwlrUAwiVLvRId7BDUYHnq8LjnuU7r2OWv\njz8Ov/0tDBliImEYLYlkCsRnQEbEfhe3LW4fEUkBTgGKgH7AdBEpACYCD4jIhCTa2urwZ/tZ/YvV\ndOsQ7RZatXtVtf4Inw/ujVkyoOqsbjp2DPLykmCsYRiNQjIFYj1wvoh0FZE0YBSwNKbPUmC0u30D\n8K46DFDVTFXNBGYAv1fVp5Joa6skXsT19q+21+iPmDYNRoyo3K4K6enJsNQwjMYgaQLh+hQmAG8D\nHwEvq+o2EXlERK51u83D8TnsAu4BKi2FNZKLL8PHLVm3RLUFNcjtb9zOoOcHMXLRSMa/Mb7SqGLS\nJEirvGqWefNsmskwWgoWSW0QKAyQ80IOJcGSKvu08bZhxegVURHXgYDjpF661JliKu/bBlassMR+\nhtEcsEhqo1rCMRKx/ohISoIllSKufT547bXKaTeKi+HWW20kYRjNHRMIA6g+A2yY9HbxHQy5uZWn\nm7Zvh0GDTCQMozljAmGUE84AOy57HAPPGohHKv55KMqdy+6M67z2+ZzVS126RLeXllq0tWE0Z0wg\njCh8GT5mD5vNyl+s5L1fvMcPz/lhebrwslAZE5ZNqDJF+IMPVr7e3LkwcqSNJAyjOWICYVSJL8PH\n1JypeD0VAXWloVJuXXprXJHw+530G5EjiVAIliyBwYNNJAyjuWECYVRLOOo6MsHf9q+2c8X8K+JO\nN/n98PLL0SnCwXFcWxCdYTQvTCCMGvFn++l9ZvQquJCGGP/m+Cp9EjNnVk7JceBAMq00DKO+MYEw\nEuKWXrdUagtpqMoEf35/5eWvjz/u+CPmzrXkfobRHLBAOSNh5m6cy4y/z+Cjrz6Kas88JZOh5w0l\nt0dupUC6gQOhrKzytUSgbVtYvtwC6gyjMbFAOaNe8Gf72X7ndkZcFJ2IqeBgAXM2zmHQ84OiRhNV\nTTWBk7fJ/BKG0bQxgTBqzaTLJ1VKFQ7OCqfrFl9Hv2f6lfsm/H6YPTu6ZGkkltzPMJouJhBGrfFl\n+Jh1zay4IrHv//ax7vN13P7G7VEiMWdO5ZFEKAR33WW+CMNoqphAGHUiXE9ixIUjygPpYnl1+6sV\n/f3w3nswbhxcfHFFn5ISJ+GfYRhNDxMIo86Ea1zPGTYn7mjiWPBYJZ/E7NlOjqZI/vIXZ2UTOKMJ\nW+FkGE0DW8Vk1AuBwgALtizg73v+zpYvtqA4/6684mXWNbPwZ1eseQ0EYMAACAYrzheBn/4U/vxn\nZ1SRlmYrnAyjIbBVTEbSCedw+sklP4lqD2qQcW+MY/I7kyv6+mDWrGifhCosXAhHjzrCUVLirHCy\nEYVhNB5JFQgRGSoiO0Vkl4hUqhYnIm1EZLF7fK2IZLrtfUUk3/3ZIiIjk2mnUX/kZOaQ6o1OGa4o\n09dMj4q6rml1k8fjrHAaMsRJAjhkiImEYTQ0SRMIEfECM4GrgG7AzSISW5HmFuAbVT0P+CMwzW3/\nEOitqj2BocDTIhKT3cdoioSLD424sHLR6kdXPhrlk/D74b774l8nFILNm52RROSIwjCMhiOZI4i+\nwC5V/URVS4BFwPCYPsOBF9ztV4AhIiKqesStaQ3QFmgZjpJWQth5Pan/pKj2PYf20P+5/vzLn/+F\nx1Y/RqAwwLRpTn3r2JFEKOQUHfJ4nJ+0NMjJabjPYBhGcgWiM1AYsb/HbYvbxxWEg0A6gIj0E5Ft\nwFZgXIRglCMifhHZICIb9u/fn4SPYBwP034wrVLUtaIs3LqQB959gIHPD2TuxrlMmwZr1sCIERV+\nCVVYtcopOuTxwIwZ0Q5r800YRvJpsk5qVV2rqpcAfYD7RaRtnD5zVbW3qvbu2LFjwxtp1MikyyeR\n5k2Le6wsVMa4N8Yxd+PcKutbgzOaKCqq2A8EzDdhGA1BMgXiMyAjYr+L2xa3j+tjOAUoiuygqh8B\nh4FLk2apkTTCPomBZw2Me1zRcpEAp751bC0JVVi3rkII8vLMN2EYDUEyBWI9cL6IdBWRNGAUsDSm\nz1JgtLt9A/Cuqqp7TgqAiJwNXAQUJNFWI4n4Mnys/MVKJvWfFFXnOoyiFbUlugQYNnEZ3pRQxXF1\nqtINGOAE1OXkOD4Jr9d8E4aRTJIaKCciVwMzAC/wnKr+TkQeATao6lJ32uhFIAv4Ghilqp+IyM+B\nKUApEAIeUdUl1d3LAuWaB+GAumc2PUNQg5WOp3pSCWkI72dX0HPHa6xf/R0i/4mKOCufRoyABQuc\ntqwsZwoqJ8cC6wyjtlQXKGeR1EajECgMMH3NdJbsjK/7HvHg7zifeRN/RmlJeNRRsdSpZ0/46CPH\niR0KOY7sNm0s+towaotFUhtNjvBS2NhVTmFCGuLQd/8Go78PF72Gs9K54stMfr5TTyLkzkSFQuaP\nMIz6xgTCaFQmXT6JVE9q3GMvbX2J0s6rYNT10H9anB5a/mOxEoZR/5hAGI2KL8PHyjErGXHhCDwx\n/xw1Mj7yygfoef3bEQF14WNB5OzVXPvTfTa9ZBj1jAmE0eiEp5veG/teXKEIk3nj08z581a69fgW\nRyAE8KK7+7P0Tx3ZurUhrTaMlo8JhNFkCAtFZGrwSJbsWMLtWy7j65wxkHIMCK+C8hIKehg/vqKu\nhGEYx48JhNHkyO2RywkpJ+DBEzduYt93XoPRQ6D3XJAywqOJUIhykbBUHIZx/NgyV6NJEigMkFeQ\nR3q7dO566y5KgiVx+3k3jSP0xkw0JDhTTo5YpKQ4AXZWeMgwqseWuRrNDl+Gj/sH3I8/2199qo7s\nufQYOwckSIVfQikrs1QchnG8mEAYTZ5wqo6nhz1Nl5O6RB9U2NJlAlwzHifoPiwSFaSnN5SlhtGy\nMIEwmg3+bD8v3/hyuX/CK14Qdzls72eh/3S3p5b/DgaV8eNh8uSqrmoYRlVYlTajWeHL8LE8dzl5\nBXl8evBTnt74dMXBKx/g1Lbf4cC7t4GGfRJCKKRMn+6MKqbFi7czDCMu5qQ2mi2BwgADnx9IWSim\nllTh92BLLmzw4wySK6acRoxwKtiZ09owHMxJbbRIfBk+Zl49k1RPKhLpd8j4Owy7I86Uk5M2vH9/\nGDnSlsAaRk2YQBjNGn+2n5VjVnJ79u2OTyKSKx+A/n8gNtFfuL5E//7mmzCM6rApJqPFEE4hvrNo\nJyLCR/s/chzYG26FN+YQO90UZtIk800YrRerB2G0OgKFAQbMH1BRlGjDrfDGbJzaVWEqxCIz06kx\nYf4Jo7VhPgij1eHL8PHjC39c0dD7WbhlAPSeA502u40Vy2ELCpxpp0GDzDdhGGGSKhAiMlREdorI\nLhGZEud4GxFZ7B5fKyKZbvuVIrJRRLa6v7+fTDuNlkmlWhNh5/W47BjfRMVIorRUuXX8tyYShkES\nBUJEvMBM4CqgG3CziHSL6XYL8I2qngf8EQjPBH8F/FhVuwOjcepWG0atCNeaGJc9joFnDYxe6XTl\nAzDsdpyMsEqkI3v7lpO4/PIQWVlO8j8TC6O1kjQfhIj4gKmq+iN3/34AVX0sos/bbp+AiKQA+4CO\nGmGUiAhQBJyhqsVV3c98EEZNhJ3Yf9n5l4piRIXfgzX3waeXw5HTqfjOFP4nKIjAffeZI9tomTSW\nD6IzUBixv8dti9tHVcuAg0Bs5pzrgU3xxEFE/CKyQUQ27N+/v94MN1om4XoTa8auYcSFIzj7lLOd\naadR18PNI8FbQvS0kzPiUFWmT9fyEcXcuTDyX/bSb/gm5i45/ipFlprcaKo06VQbInIJzrTTD+Md\nV9W5wFxwRhANaJrRjAkLBcDkdyYzfc10RyjGDIZ3HoPdg4iMmwhniM3PV/LzwynFOwGdWPdmMbyy\nFf+I7nWyJRCAIUOcrLOWmtxoaiRzBPEZkBGx38Vti9vHnWI6BWc6CRHpArwG5Krqx0m002jFTPvB\nNJ4e9jR9z+zLwCvS8IwdAsP80HktnPYPt1f0iMLB3Q+m8vjvT6zzt/+8PCguUYJB57elJjeaEskU\niPXA+SLSVUTSgFHA0pg+S3Gc0AA3AO+qqorIqcCbwBRVXZNEGw0Df7aftbetZeWYlcy+Zjae3s/B\nbT745UXuaqdwGvEKv0TFvodd67vSv3+Ifxm/p9b3Tr94KyHPUZBSQp6jpF9shbWNpkPSpphUtUxE\nJgBv40QnPaeq20TkEWCDqi4F5gEvisgu4GscEQGYAJwHPCQiD7ltP1TVL5Nlr2GAIxbdT+/Ogi0L\n2L5/O6uv/A160VLHkb3jWqID7cIIqrBwTmf+lv82Hc/+kgt672XSTQPwZVQ/X1SU/gae0W8S+ucA\nPF1XU5R+DVC36SrDqG8sktowqiFq5VNhPydL7N4s+KwPFWJRUcnOIQSiyOX/zpo/DaxWJAKFAYYs\nGEJJsIQ0bxrLc5fXKCqGUZ9Yqg3DOE4ChQGmvDOF9z59j1BhX3h+BQTbxPSqqIkdFot22X/m0ku8\n3DLy3ChHdmTN7c17ncju3B65+DJ8BAKwYMluyFxJ7rDzTTCMpGICYRj1RPjFvm3zySx80QsoFJ8I\nW/+FCkd27IhCgRBd+m7mpMFz6HjRLtbuWUtpsJQQITziIeWzAYw99QWyzj2bX/4qSHGxgreEtLFX\nk/fbx0wkjKRhAmEYSSBQGGDBlgXsO7yPguVXsmW+Hw0KFWs/4olFCLovhNM/ghO+gn294PB34X+v\nhlAKHo8QCgHqBSmF7z/MuInfMHvY7Eb4hEZrwATCMBqAQMBZtrpk9U7WvXU+lZfGQvSoIvaYAEEQ\ndUqmesqg13OkZi1i5YN/sFGEkRQsm6thNAA+H9x/P6xddiGTHvsneGLzPEWKQuQoI9zmLJtFvU5b\nKA023EbpvP9h+pPfVHvvQADGT97N+NkLCBRaSHZzoDlE0NsIwjCSRCAA02fu5e87d7NvTxvY19M9\nEm/qiSqOub8lRM9bniZzyP8A0Kl9pyin9oBBpQRLPeAtJe2WoU3GbxEeVeXkOAIa9uHkZOY0Cfsa\ni6YUQV/dCKJJp9owjOaMzwev+c4AznBWQU0J8N4iH6FQ5DRTmMjgu1gHt4B6yH/WT/57neC8t+Do\nqczJvIeefY7xxaKpBEuvdfoFPZRsGkVeQV65eES+oOORrJd27EtwxktbmbjNlvSC8zcpKYFg0Pmd\nl9c0U6yYQBhGA+DL8LFyIQQmOC+DAwcgP9+pYvdf/+Wk2QhpCajH+SkvjxopGl7YMdL5IQQSIv+d\nNXAwI/pmu6/gyVcf4q95U1jz6L8RLPPg8QaZ/f92lC+1jVxmO/GvE+v00q5JfGJfgq++VURJ5xKC\nGqQkWFIuYrW9R7gt/eKtFKW/UaWwJSKOjUVOjiOaYfHMyWlsi+JjAmEYDYjPV/llNWIE5OUJ6Rf/\ng817N7NvzZW89coZlJSEUI1dOou77XWEZPegiCu5o479l7DvycXsO2sNlDr+jFCZcPv9/8vCrQs5\n7YvreGPPiwQPfwfp+iba5RiKcrTsKLfMfJZBeia5I86u9qVa1RRJ5GgkJ8cX9RK8/qp0Vm9Lo7is\nGBEhvV1s4ubolWHs8fHWg7+mrNRbfg9w7ltcooQ85+IZ/SZtMh+tJGxh+4pLFG9KGU8t2lHnhIrJ\nwOdzPk843oUu5wOO/U1pGs58EIbRBAkEYMECeHZeGWWlUBG1HSkS8VZIRR6PTDLoxGI4K6TC1wuB\ntxSy5kObA1AwGPb2AvXgSQ1y75y3OPW8j8pfVJHfyPPy4De/DaEhDx5vCP+vC2HAH5ifP5+yUFn5\naIQ9vqigv61fbmXCsgkENUgbb5uoF3ugMEDOCzmUFPSCghw4eBZsvA00BY9X+bdHnc/24IPOqMRZ\nBvwQMmAaw0/8A5323wSZKzn5/G38ee6FfPzqaDTkLBf2DvlXVj93Ta1fuMkchcSLogcY/MLg8rYV\no1ckXSTMB2EYzYzwSCM3N6V8SurxJ0JoKLZn7Be8yISCkW1ufIaGj7mjkKAHNtwe018IlQjTZ+2F\nYQ8gCOcd+Tkf//FpQmUpeFLLOPOG6ahMAlIJUcqc7b+DjemQmQXA0YLBTAn+hdMumM7rJ76O7lfm\n3nkbqTtHUXrBGOj9DMfKjrFgywIA8gryWPf5OkrW5sKbs9yVXCHwhEDLCGmIA1LIiEHnkpIaJBhy\nxS0zD91wC0uWTXRGVJ6fwPnL3I9RCuL0C569nAVbimCPL+EX/vE6ksMiD5CbW/ncBVsWcKzMGb2F\np9w+PfgpxUGn9E1xsJgFWxZUCGgjTJnZCMIwmgnhF86+ffDmm1BaCh4PXNbnIGnf/Zgc3ym8PK8T\nBbvaUVkgoPLoIzY7beyqqiB0+gDK0qCsHRzIxPGNBOGUAvj2bNdfEgKPQsjjxG6IQijFKcA0eohT\na+N/fg9rIsrSD/ND72cB8IiHkIac6n7PrQJNibDFjQtB8KSUceE9d3K07CgFKwc4AYZHT4PdA4j2\n2bh4SqDXc9BjAWT8nYEymfd+96+ESr14UoPMXvyP6PQn7gv4gHxM3kdb+fKzthS8eyWoF/Eot9/7\nKfs6LubzrRdUSp0Sy9y5cMcdEAw69qSkBlm1MqXCjxIzWko9531WPvgHFmxZwJyNc8qvMy57HLOH\nzY47ZdY9+3C9TEVZoJxhtDCq+jYZCMDAgVBWBuUvS0/QeXlHTTeFibfUtrr26vrFbgehw0fQ4R+w\nYzhRL/EO2yBztdO10ybYdVWcsq9QkWrdC5TBue/Axa/CW/8Vkwsr9t7uub2fhmF3QKEP8h6Cj3+A\nM3FSBp030P7sXZw3+H1Q2PL4v6Olqe69FKQMNLXiFv2nwdpfQjANvCUMfPARTrtgp/MR2nfi5P0/\nIi8P2p50mPfm3Ewo6ImwJcipl2zkGv969n9nKV/uPJf8V38AO38MCJJSyll3jeU7F+zgg30fECKE\nV7zMumYWFPp4/PcnsmtDpivIQeg9l9Rrf0UwFMTj8TDz6pn4s/0J/L0qYwJhGK2IyJFGp07O9MbW\nrTD+jhChIBWR2nioehQR698IE3ssTsxGlSQqOpEEI05zX9yRU2ZRdsS7ZSmcXOiOdsLXiMFTAid/\nBge6Ev+zCFAGJ38O33ahfBQVFj9wRjKf9nefa9i22M/ril37fXC4E9F+pSC02w9nvQ/pO2FfFnTa\nDMWnwuZfQDAl2nYpg0v/5Ahqp81I20Pc97O+TPvFiOofZ7xHZAJhGEb58tB0mDgRiotBxJmmCgYh\nFHL2q38n1DTiqE5Yquobe17ImcI6eBaVX+g1jYCqGtHE3jf2/lV9ntpS3Xnx7Knq/kQcix35xbuH\nQsoxnv5/H9d6tZY5qQ3DiFpi2717xRQVVAhHURGkpwubNzsjkK+/hv37oU0bZxQSCoHHI/TvD926\nwcknwx//6PhDan7hhlwBil26G/3CS0mFXoMOsP71rm5fiH6hO9cKpyYREVfUqpvuirQr3mgnnn8m\n9ryqxC+S6s6LN/qKvX+sDeFRXmRbvFGbB4JpvPpWEf7aDyKqJKkCISJDgf/E+RrwrKr+IeZ4G2AB\nkI1Ti/omVS0QkXTgFaAP8LyqTkimnYbR2oiNx0hkVUxVfg8njsMRGEdYKl52X38Na9aAKrRp4+Gu\nu+IJCogIZ5/tBA5OmiRAL3L+6qwgqqDihepNDfHrR3Zzqp7rjojEHREJHg+UBRXVoLsaqrpv7fFE\nLUTdfDVxriVlrtO98meAIOkX7eCbf1xEKBQ5UqpppFN1Opbrr6ocW3I8JG2KSUS8wD+AK4E9ODWq\nb1bV7RF97gAuU9VxIjIKGKmqN4nIiUAWcClwaSICYVNMhtE0qZSPKRAtKBB/GWjkMtGsLMpHNWG/\nSqxzPnZElJPjjHrmzYO2bWHtWsd57/XC1VdXrAQDZ2rt4oth2DA49dRIsYO33nL6iUD//sAJReze\ne5hPP8xAQwKizitawesNcc5lX/DFwW/53rXbOOeCI+xbcyWdTjqDk8/+mNeXlfHNno6c1mU/v7qn\nBP+I7k7OrunO/QoLnVGaQ8y7WUKkpAjBMo87YqoQD483xL3/VsC0KefW+u/TKD4IEfEBU1X1R+7+\n/QCq+lhEn7fdPgERSQH2AR3VNUpExgC9TSAMwzhe4glVdXEKVZ0Xrx3qJ0YhUjyd6b5oEQ3fJ/J4\nUdHx3bexBOIGYKiq3uru/xzoF/myF5EP3T573P2P3T5fuftjqEYgRMQP+AHOOuus7N27dyflsxiG\nYbRUWmw9CFWdq6q9VbV3x44dG9scwzCMFkUyBeIzIDLNZBe3LW4fd4rpFBxntWEYhtHIJFMg1gPn\ni0hXEUkDRgFLY/osBUa72zcA72qy5rwMwzCMWpG0Za6qWiYiE4C3cZa5Pqeq20TkEWCDqi4F5gEv\nisgu4GscEQFARAqAk4E0ERkB/DByBZRhGIaRXJIaB6Gqy4BlMW0PRWwfA26s4tzMZNpmGIZhVE+z\ndlIbhmEYyaPF5GISkf1AXde5dgC+qkdz6guzq3aYXbWjqdoFTde2lmjX2aoadxloixGI40FENlS1\nDrgxMbtqh9lVO5qqXdB0bWttdtkUk2EYhhEXEwjDMAwjLiYQDnMb24AqMLtqh9lVO5qqXdB0bWtV\ndpkPwjAMw4iLjSAMwzCMuJhAGIZhGHFp8QIhIs+JyJduavFw22ki8j8i8r/u7++47SIiT4rILhH5\nQER6NYJtU0XkMxHJ8wqvjQAABk1JREFUd3+ujjh2v2vbThH5UZJsyhCRFSKyXUS2iciv3PZGfWbV\n2NWoz8u9T1sRWSciW1zb/tVt7yoia10bFrs5yRCRNu7+Lvd4ZgPb9byI/DPimfV02xv6379XRDaL\nyBvufqM+r2rsavTnJSIFIrLVvf8Gty35/ydVtUX/AAOBXsCHEW3TgSnu9hRgmrt9NfAWTpmm7wFr\nG8G2qcC9cfp2A7YAbYCuwMeANwk2nQH0crdPwqkK2K2xn1k1djXq83LvJUB7dzsVWOs+i5eBUW77\nHGC8u30HMMfdHgUsbmC7ngduiNO/of/93wO8BLzh7jfq86rGrkZ/XkAB0CGmLen/J1v8CEJVV+Ek\nAoxkOPCCu/0CMCKifYE6/B04VUTOaGDbqmI4sEhVi1X1n8AuoG8SbNqrqpvc7UPAR0BnGvmZVWNX\nVTTI83LtUVU97O6muj8KfB+ntjpUfmbhZ/kKMEREIqvWJ9uuqmiwf/8i0gW4BnjW3Rca+XnFs6sG\nGvR9UcX9k/p/ssULRBV8V1X3utv7gO+6252Bwoh+e6j+JZQsJrhDw+fCw0YawTZ3KJ+F882zyTyz\nGLugCTwvd1oiH/gS+B+cEcsBVS2Lc/9y29zjB4H6rTZfhV2qGn5mv3Of2R9FpE2sXXFsrm9mAJOA\ncAXmdJrA84pjV5jGfl4K/E1ENopTSRMa4P9kaxWIctQZkzWltb6zgXOBnsBe4N8bwwgRaQ+8CkxU\n1W8jjzXmM4tjV5N4XqoaVNWeOIWx+gIXNYYdscTaJSKXAvfj2NcHOA2Y3JA2icgw4EtV3diQ962J\nauxq1OflcoWq9gKuAu4UkYGRB5P1f7K1CsQX4SGX+/tLtz2RKnhJRVW/cP9Th4BnqJgWaTDbRCQV\n5yW8UFX/7DY3+jOLZ1dTeF6RqOoBYAXgwxnah1PqR96/wSspRtg11J2uU1UtBubT8M+sP3CtODVf\nFuFMLf0njf+8KtklIv/dBJ4XqvqZ+/tL4DXXhqT/n2ytAhFZyW408JeI9v/f3h2EWFXFcRz//hoj\nBxMtCxFMJmlWkZG4CHERQUG2ChcqSiBucpGtIkNo5cpFi1E3uhDRKHCRtJJwlAgUdKEzzmDhEO4K\nxkWCICLyb3H+z3fR+3w4et9N5veBx9x33uPe/ztv3vznnHvf/3yeVwG8D9yqDOEG4qG5ws+AzhVO\nvwBb8oqON4FR4GIDxxdlIadrEfF95aFW+6xXXG33V8bwuqSluT0MfEQ5R3KOslIiPNpnja+k2COu\nPyp/VESZt672WePvZUR8GxEro6z5soXy+rfRcn/1iGt72/0laZGkxZ1t4OOMofnP5FzPbj8vN+BH\nytTDPcpc3E7K/OU4cB04A7yazxVwiDJ/fBVY10Jsx/PYk/lGr6g8f2/G9ifwSUMxbaAMVSeBK3nb\n2HafPSauVvsrj7MGuJwxTAHfZftqSlKaAU4CL2X7wrw/k4+vHnBcZ7PPpoATdK90Gujvfx7zA7pX\nC7XaX4+Jq9X+yn6ZyNs0sDfbG/9MutSGmZnVmq9TTGZm1ocThJmZ1XKCMDOzWk4QZmZWywnCzMxq\nOUGY9SHpvrqVPK9I2vMM9z2iSjVfs/+TBf2fYjbv3YlSrsJsXvEIwmyOskb/fpU6/RclvZXtI5LO\nZnG3cUmrsn25pJ9V1meYkLQ+dzUk6YjKmg2/5reekbRbZf2LSUk/tfQybR5zgjDrb/ihKabNlcdu\nRcQ7wEFKJVCAA8CxiFgD/ACMZfsY8FtEvEtZB2Q620eBQxHxNvAvsCnb9wDv5X6+aOrFmfXib1Kb\n9SHpdkS8XNN+A/gwIv7KQoL/RMQySTcpJT/uZfvfEfGapFlgZZSib519jFDKcI/m/W+AFyNin6TT\nwG3gFHAqums7mA2ERxBmTyd6bD+Ju5Xt+3TPDX5KqamzFrhUqXRqNhBOEGZPZ3Pl54XcPk+pBgqw\nDfg9t8eBXfBgIZ8lvXYq6QXgjYg4R1l/YAnwyCjGrEn+j8Ssv+Fcla3jdER0LnV9RdIkZRSwNdu+\nBI5K+hqYBXZk+1fAYUk7KSOFXZRqvnWGgBOZRASMRVnTwWxgfA7CbI7yHMS6iLjZdixmTfAUk5mZ\n1fIIwszMankEYWZmtZwgzMyslhOEmZnVcoIwM7NaThBmZlbrP163unCMZAiLAAAAAElFTkSuQmCC\n",
-            "text/plain": [
-              "<Figure size 432x288 with 1 Axes>"
-            ]
-          },
-          "metadata": {
-            "tags": []
-          }
-        },
-        {
-          "output_type": "display_data",
-          "data": {
-            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAYgAAAEWCAYAAAB8LwAVAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOydeXxU5bn4v89MFqC41IiiEAy2iksp\nOzouGEStoFWsS6W2AVHCIt5yvZbWqpUreqlob6mKQFCRtBbU+pMiAipIgOp42RVBrKBhUVFMRbSY\nZea8vz/ecyZnJpNkghkSyPPNJ5+cc95z3nlnyXnm2cUYg6IoiqIkEmjqBSiKoijNExUQiqIoSlJU\nQCiKoihJUQGhKIqiJEUFhKIoipIUFRCKoihKUlRAHAKIyCIRGdrY5zYlIlIqIhelYV4jIt93t6eL\nyN2pnHsAj3ODiLxyoOs83BCRp0Tkvkaec5iI/KMx51QaRkZTL+BwRUS+9u22ASqAqLs/0hjzdKpz\nGWMGpuPcwx1jzKjGmEdE8oAPgUxjTMSd+2kg5fdQSS8iMgy42RhzXlOv5XBCBUSaMMa09bZFpBT7\n4V2SeJ6IZHg3HUVRmj/J/mcb+n98qPzfq4npICMi+SKyS0R+LSK7gVki8l0RWSAie0TkC3e7o++a\nEhG52d0eJiL/EJGH3HM/FJGBB3huZxFZISJficgSEZkqIn+pZd2prHGiiLzuzveKiBzrG/+FiGwX\nkTIRubOO1+csEdktIkHfsatE5G13u6+IhEVkr4h8IiKPikhWLXPFmT1E5FfuNR+LyPCEcy8TkfUi\nsk9EdorIBN/wCvfvXhH5WkRCieYPETlHRFaLyJfu33NSfW0S1uF9PsaLyGfuegeLyCAR+aeI/EtE\nfus7PyAivxGRbe5r+6yIHOMbf859Pb903+szE16fqSLykruu/xOR79Xx3tQ6l8uxIvKqO9dyETnJ\nvU5E5I/u89knIhtF5Afu2FEiUux+rraLyF0iUuO+JCJ5Yk2CGb5jJSJys4icDkwHQu77s9cdz3Y/\n+ztE5FOxJsfWdTy/4SLyrvv5ftlbvztmROQWEXkfeF+S/x9ni8gU9/P1sbudnfC+xs6vbR3NCRUQ\nTUN74BjgJKAQ+z7Mcvc7Ad8Aj9Zx/VnAe8CxwGTgCRGRAzj3r8AqIAeYAPyijsdMZY0/A24EjgOy\ngNsBROQMYJo7/4nu43UkCcaY/wP+DVyYMO9f3e0o8J/u8wkBA4Axdawbdw2Xuuu5GDgFSPR//Bso\nAI4GLgNGi8hgd6yf+/doY0xbY0w4Ye5jgJeAh93n9r/ASyKSk/Acarw2tdAeaAV0AH4HzAR+DvQC\nzgfuFpHO7rm3AoOBC7Cv7RfAVN9ci9znexywjppmseuB/wa+C2wF7q9jXfXNdQMwEfvebPCNX4J9\nDU8FjgKuA8rcsUfcYye7z6EA+zqljDHmXWAUEHbfn6Pdod+7j9kd+D7Vr2cNRORK4LfAT4B2wEpg\nTsJpg7H/T2e4+4n/x3cCZ7uP1w3oC9zluz7x/OaPMUZ/0/wLlAIXudv5QCXQqo7zuwNf+PZLsCYq\ngGHAVt9YG8AA7RtyLvYmHwHa+Mb/AvwlxeeUbI13+fbHAIvd7d8Bc31j33Ffg4tqmfs+4El3+wjs\nzfukWs4dB7zg2zfA993tp4D73O0ngd/7zjvVf26SeacAf3S389xzM3zjw4B/uNu/AFYlXB8GhtX3\n2iR53Hys8A36nr8BzvKdsxYY7G6/CwzwjZ0AVPnX6hs72p3rKN/r87hvfBCwJcX3P9lc/ve4LVaY\n52KF/T+xN8+A75yg+zk4w3dsJFCS5DVO9h6UEP9Z/4dvTNzPzfd8x0LAh7U8n0XATb79ALDf+9y5\nj31hwvsU938MbAMG+fZ/BJSm+n/fHH9Vg2ga9hhjyr0dEWkjIjNcFXsf1qRxtPjMLAns9jaMMfvd\nzbYNPPdE4F++YwA7a1twimvc7dve71vTif65jTH/pvobZDL+CvzEVc9/Aqwzxmx313GqWPPWbncd\n/4P9xlofcWsAtic8v7NEZJlr6vgS+400lXm9ubcnHNuO/cbqUdtrk4wyY4wX0PCN+/dT3/g3vutP\nAl4Qa3LbixUYUeB4EQmKyO9d89M+7BcViH9eKa0rxbn87/HXwL+AE40xr2G1zanAZyJSJCJHutdm\nEv/aJb5uB0o77Beitb7XZrF7PBknAX/ynfsvrJDxryXx/yPu/5ian4Pt7rHazm/2qIBoGhJL6P4X\n0AX7LfFIqk0atZmNGoNPgGNEpI3vWG4d53+bNX7in9t9zJzaTjbGbMb+cw0k3rwE1lS1BTjFXcdv\nD2QNWA3Kz1+B+UCuMeYorE3bm7e+kscfY28wfjoBH6Wwrm/LTmCgMeZo328rY8xH2NfuSqw57Sjs\nt3A4sM9VKnP53+O2WHPKxwDGmIeNMb2w5plTgV8Bn2O1Hf9rV9vr9m/3r//z2t63nfgefY4VpGf6\nXpejjC94JIGd2OhC/+vY2hjzRh2Pkbif+Dno5B6r7fxmjwqI5sER2A/zXteefU+6H9D9Rr4GmCAi\nWSISAn6cpjX+DbhcRM4T61C+l/o/e38FfokVRM8lrGMf8LWInAaMTnENzwLDROQMV0Alrv8IrEZV\nLiJ9sTdEjz2Ag7WTJ2MhcKqI/ExEMkTkp9gb4YIU1/ZtmA7c73MIt3Pt6WCfUwVWW2uD1bYOlFTm\nGuR7jycCbxpjdopIH1dDy8Te6MsBx9WSnnXXf4T7HG7DmjrjMMbswQqOn7vazHDA71D/FOjoPjbG\nGAfru/mjiBwHICIdRORHtTy/6cAd4jreXef5tQ14fcD6LO5y34NjsabVpEEfhwoqIJoHU4DW2G89\nb2JV4YPBDVi7bBnW7v8M9iaQjANeozFmE3AL9qb/CdaRuquey+ZgnZavGWM+9x2/HXvz/gp7A3gm\nxTUscp/Da1hn7GsJp4wB7hWRr7D/2M/6rt2Pdd6+7pogzk6Yuwy4HKtllQHjgcsT1p0u/oTVfF5x\n1/4m1pEKUIzVxD4CNrtjB0oqc/0VK3j/hXWo/9w9fiT2vfrCnaMMeNAduxUrND4A/uHO8WQtaxiB\n1TzKgDMB/7f714BNwG4R8V73X2Pf6zdds9gSrBZcA2PMC8ADwFz33HewGmxDuA/7pettYCPWkd+o\nyYMHG3EdKIqCiDyDdVKmXYNRFKX5oxpEC8ZV/b8nNpb+UqyNeV5Tr0tRlOaBZlK3bNoD/w/rMN4F\njDbGrG/aJSmK0lxQE5OiKIqSFDUxKYqiKEk5bExMxx57rMnLy2vqZSiKohxSrF279nNjTNIEwsNG\nQOTl5bFmzZqmXoaiKMohhYgkVgGIoSYmRVEUJSkqIBRFUZSkqIBQFEVRknLY+CAURTl4VFVVsWvX\nLsrLD6nipC2aVq1a0bFjRzIzM1O+Jq0Cws3O/RO27vvjxpjfJ4zfBtyM7UuwBxhujNkuIt2xVTuP\nxJYuvt8Yk1LNHUVR0s+uXbs44ogjyMvLo/ZeVUpzwRhDWVkZu3btonPnzvVf4JI2E5PbJ2AqtuDV\nGcAQt7OYn/VAb2PMD7EVPye7x/cDBcaYM4FLgSkicjSKojQLysvLycnJUeFwiCAi5OTkNFjjS6cP\noi+2m9kHxphKYC621k8MY8wyX8OaN3HbUBpj/mmMed/d/hj4jNobfXxrwmGYNMn+VRQlNVQ4HFoc\nyPuVThNTB+I7MO2iugxxMm7Ctv2Lw63Nn4Vt55c4Vojb27VTp8T+L6kRDsOAAVBZCVlZsHQphEIH\nNJWiKMphRbOIYhKRnwO9qa4R7x0/AfgzcKPbACQOY0yRMaa3MaZ3u3YHpmCUlFjhEI1CRQVMmKCa\nhKI0d8rKyujevTvdu3enffv2dOjQIbZfWVlZ57Vr1qzhP/7jP+p9jHPOOadR1lpSUoKI8Pjjj8eO\nbdiwARHhoYceih2LRCK0a9eO3/zmN3HX5+fn06VLl9jzu+aaaxplXamQTg3iI+JbPHYkSStBEbkI\nuBO4wBhT4Tt+JPAScKcx5ts0OqmT/HyrOVRUgOPAkiWwcqVqEorSnMnJyWHDhg0ATJgwgbZt23L7\n7bfHxiORCBkZyW9vvXv3pnfv3vU+xhtvvFHvOanygx/8gGeffZabb74ZgDlz5tCtW7e4c1599VVO\nPfVUnnvuOSZNmhRnEnr66adTWnNjk04NYjVwioh0dtsAXo/tfBVDRHoAM4ArjDGf+Y5nAS8AxcaY\nv6VxjYRCVhhcdBEEAlZIVFZazUJRlMYjvDPMpJWTCO9Mj4o+bNgwRo0axVlnncX48eNZtWoVoVCI\nHj16cM455/Dee+8B9hv95ZdfDljhMnz4cPLz8zn55JN5+OGHY/O1bds2dn5+fj7XXHMNp512Gjfc\ncANeFeyFCxdy2mmn0atXL/7jP/4jNm8iJ510EuXl5Xz66acYY1i8eDEDB8Y3rJszZw6//OUv6dSp\nE+FmYsZImwZhjImIyFjgZWyY65PGmE0ici+wxhgzH2tSags850rLHcaYK4DrsL2Ic0RkmDvlMGPM\nhnSsNRSypqWVK6t9Efn56XgkRWmZhHeGGVA8gMpoJVnBLJYWLCWU2/gq+q5du3jjjTcIBoPs27eP\nlStXkpGRwZIlS/jtb3/L888/X+OaLVu2sGzZMr766iu6dOnC6NGja+QKrF+/nk2bNnHiiSdy7rnn\n8vrrr9O7d29GjhzJihUr6Ny5M0OGDKlzbddccw3PPfccPXr0oGfPnmRnZ8fGysvLWbJkCTNmzGDv\n3r3MmTMnzsR1ww030Lp1awAuvvhiHnzwwRrzp4O05kEYYxZiG7r7j/3Ot31RLdf9hSZo9j10qP1b\nUKDmJUVpTEpKS6iMVhI1USqjlZSUlqRFQFx77bUEg0EAvvzyS4YOHcr777+PiFBVVZX0mssuu4zs\n7Gyys7M57rjj+PTTT+nYsWPcOX379o0d6969O6WlpbRt25aTTz45llcwZMgQioqKal3bddddx09/\n+lO2bNnCkCFD4kxYCxYsoH///rRu3Zqrr76aiRMnMmXKlNhzORxNTIcMRUVwwQX27+zZTb0aRTn8\nyM/LJyuYRVCCZAWzyM/LT8vjfOc734lt33333fTv35933nmHF198sdYcAP83+WAwSCQSOaBz6qN9\n+/ZkZmby6quvMmDAgLixOXPmsGTJEvLy8ujVqxdlZWW89tprDX6MxqbFl9oIh+GWW8B7vysqrP9B\nNQhFaTxCuSGWFiylpLSE/Lz8tGgPiXz55Zd06NABgKeeeqrR5+/SpQsffPABpaWl5OXl8cwz9Rd7\nuPfee/nss89imgEQM4Xt3LkzJohmzZrFnDlzuPjiixt93Q2hxQuIkhLrmPYQgR07rOBQIaEojUco\nN3RQBIPH+PHjGTp0KPfddx+XXXZZo8/funVrHnvsMS699FK+853v0KdPn3qvSRY6+8ILL3DhhRfG\naSlXXnkl48ePp6LCBnb6fRDHHnssS5YsaaRnUTeHTU/q3r17mwNpGOQlylVUWOHgRTIFgzB8uPoj\nFCUZ7777LqeffnpTL6PJ+frrr2nbti3GGG655RZOOeUU/vM//7Opl1Uryd43EVlrjEnq4GjxPggv\nzPW++2DECCscolEbzTRjhhUezSTiTFGUZsbMmTPp3r07Z555Jl9++SUjR45s6iU1Ki1eg/ATDkP/\n/lab8AgEoHdv6NlTtQlF8VAN4tBENYhvSTQav+84sGoVTJ9uhYdqE4qitBRUQPgoKakpIPxohrWi\nKC0JFRA+8vOhrmZLmmGtKEpLQgWEj1DIagijRkHfvtb/4NG3Lyxbpj4IRVFaDiogEgiFYNo0mDIF\nsrNtuGvr1nZfhYOiNA/69+/Pyy+/HHdsypQpjB49utZr8vPz8QJZBg0axN69e2ucM2HChLgS3MmY\nN28emzdvju3/7ne/a5S8hOZYFlwFRC144a8jRlTXaNLOc4rSPBgyZAhz586NOzZ37tx6C+Z5LFy4\nkKOPPrAuxokC4t577+Wii5KWlWswXllwj/rKgidGoT799NNs2LCBDRs28Le/fftC2Cog6mH2bJg5\n0/oe+veHu+6Cfv1s3SZFUVKnMb9gXXPNNbz00kux5kClpaV8/PHHnH/++YwePZrevXtz5plncs89\n9yS9Pi8vj88//xyA+++/n1NPPZXzzjsvVhIcbI5Dnz596NatG1dffTX79+/njTfeYP78+fzqV7+i\ne/fubNu2jWHDhsVuxkuXLqVHjx507dqV4cOHxzKh8/LyuOeee+jZsyddu3Zly5YtSdfV3MqCq4Co\nA3+3uaqq6qZCkQiMHauahKKkilex4O67Gyf59JhjjqFv374sWmS7FM+dO5frrrsOEeH+++9nzZo1\nvP322yxfvpy333671nnWrl3L3Llz2bBhAwsXLmT16tWxsZ/85CesXr2at956i9NPP50nnniCc845\nhyuuuIIHH3yQDRs28L3vfS92fnl5OcOGDeOZZ55h48aNRCIRpk2bFhs/9thjWbduHaNHj67TjOWV\nBX/jjTdqLQv+4x//mCFDhjBnzpy4a2+44YaYielXv/pV6i9oLaiAqAOv21wwaH/9Pb+jUQ15VZRU\n8X/Zaqxwcb+ZyW9eevbZZ+nZsyc9evRg06ZNceagRFauXMlVV11FmzZtOPLII7niiitiY++88w7n\nn38+Xbt25emnn2bTpk11rue9996jc+fOnHrqqQAMHTqUFStWxMZ/8pOfANCrVy9KS0trnee6667j\nueeeY86cOTVMZollwefNm0fUF5vvNzE1Rs8IFRB14PdD+IovEghYB7aGvCpKavi/bDVWuPiVV17J\n0qVLWbduHfv376dXr158+OGHPPTQQyxdupS3336byy67rNYy3/UxbNgwHn30UTZu3Mg999xzwPN4\neJpAfeXCm1NZcBUQ1N0KMRSCTp2sWckYKxwuushGNZWUqJlJUVLB+7I1cWLj9Xtv27Yt/fv3Z/jw\n4bFv2vv27eM73/kORx11FJ9++mnMBFUb/fr1Y968eXzzzTd89dVXvPjii7Gxr776ihNOOIGqqiqe\nfvrp2PEjjjiCr776qsZcXbp0obS0lK1btwLw5z//mQsuuOCAntu9997LAw88kLQs+I4dOygtLaW0\ntJSpU6fWMDM1Ji2+3HcqrRC9bz9eO9Krr4Zx46r3G+sDryiHM6FQ4/+fDBkyhKuuuipmaurWrRs9\nevTgtNNOIzc3l3PPPbfO63v27MlPf/pTunXrxnHHHRdXsnvixImcddZZtGvXjrPOOismFK6//npG\njBjBww8/HBcp1KpVK2bNmsW1115LJBKhT58+jBo16oCeV3MpC97ii/VNWjmJu5fdTdRECUqQif0n\ncsf5d9Q4LxyG4uLq/ZkzrT01GLTfiu6oeYmiHLZosb5Dk4YW62vxGoTXCtHTIOpqhTh7ttUagkHI\ncF85Lb+hKMrhSlp9ECJyqYi8JyJbReQ3ScZvE5HNIvK2iCwVkZN8Y0NF5H33d2i61ui1QpzYf2JS\n85KHPwojEoFu3azzWs1LiqIcrqRNgxCRIDAVuBjYBawWkfnGGH/M2XqgtzFmv4iMBiYDPxWRY4B7\ngN6AAda6136RjrWm0grR80N4uRCrV8PatbB7N4wfr0JCaXkYYxB/7LfSrDkQd0I6NYi+wFZjzAfG\nmEpgLnCl/wRjzDJjzH53902go7v9I+BVY8y/XKHwKnBpGtdaL14UxkUX2XwIY6w2MW8enH++ZlYr\nLYtWrVpRVlZ2QDcd5eBjjKGsrIxWrVo16Lp0+iA6ADt9+7uAs+o4/ybAi0lLdm2HRl2dj/DOMCWl\nJeTn5depSYRCMGECvPaaNTN5RKPg1QgrLEzXKhWl+dCxY0d27drFnj17mnopSoq0atWKjh071n+i\nj2bhpBaRn2PNSQ0KGhaRQqAQoFOnTgf02KmEufoJhWDqVBgzJr65kONUC4muXa3PIj9fTU/K4Ulm\nZiadO3du6mUoaSadJqaPgFzffkf3WBwichFwJ3CFMaaiIdcaY4qMMb2NMb3btWt3QIssKS2hMlpJ\n1ESpiFQwoWRC0oQ5P4WFsHIlDB4cX37Dcazg6N+/8WrOKIqiNBXpFBCrgVNEpLOIZAHXA/P9J4hI\nD2AGVjh85ht6GbhERL4rIt8FLnGPNTpemGuAAA4OSz5cwoDiAfUKiVAIXnjB9qr2NxaKRq0juzFr\nziiKojQFaRMQxpgIMBZ7Y38XeNYYs0lE7hURryLWg0Bb4DkR2SAi891r/wVMxAqZ1cC97rFGxwtz\n7X1ibwTBMQ6V0UpKSktSur6w0DYY8tdq8sjI0BwJRVEOXdLqgzDGLAQWJhz7nW+71i4bxpgngSfT\nt7p4Nny6AYONyMgIZNSZMJdI167QqxesWhV//MYb1QehKMqhS7NwUjc1JaUlRJ1qj3O346s7ONUX\n4eTVua+oiD+enQ0FBWlbsqIoStpRAUG1H6IiUoGDw+qPV5M/O59HBj7CuMXj6oxw8jKsHcf6Inr3\nhp49oUePav+DahGKohyKaLlv4v0QAAZDZbSSP735p1iEU21+CX+d++xsWwa8Rw/bce6uuzSSSVGU\nQxcVEC6h3BA9T+gZd2zz55sJSIAAAUSEnDY5Na9LqHMPcMsttkWp41jTk0YyKYpyKKICwkdBtwKC\nEh+OFHWiiNjopnGLx9XaVOiOO+zfkhIrGDyMsc5r1SIURTnUUAHhI5Qb4rHLHiMg1S+LweAYJ+Xw\n1/x8a2ryEuiMsfWaLrhAhYSiKIcWKiASKOxVyLTLppEZyCQgATKDmWQFswhKsN5+EVBtcrr44vjj\nVVUweTJMmqSCQlGUQ4MW31GuNsI7wxS/ZVvI9TihB2X7y+ot5hd3fdhWefXXaxKxv9nZ2kdCUZTm\nQV0d5VSDqIPZb82maG0Rtyy8JeagnrRyUr1lOMDe/B97LL4MhzHquFYU5dBB8yBqoaS0JJYX4TgO\noxeMRkQwGLKD2fVWfQVbhmP9eluvyY+IluBQFKX5oxpELeTn5RPwff13cIiaKI5xKI+Ux8xP9VFQ\nYPMkEikuVl+EoijNGxUQtRDKDTF10FQyA5kI8W0VDYZZG2albGoqKYG+fasjm6JRmDFDk+gURWne\nqICog8JehSwftpyRvUaSHcyOExQRJ5JyxddQyGZYt2oVH/6q5cAVRWnOqICoh1BuiGmXT2PZ0GVc\n2eVKghIkIIGUQl7j5nGFhN9preXAFUVpzqiAaAAvb3sZYwwBCTDl0ikph7x6lJVVb4toOXBFUZo3\nGsWUIl5rUgcHMcL6T9YzaeUkctrkpJwj4RX2q6y0AmLdOvj1r+Hoo7V/taIozQ8VECnilQSvjFYi\nIsxcNxPHOBisRpEdzGbKpVPqFBZelvXkybb8xqpV9lfE+ic0eU5RlOaEmphSxCsJPqLnCIwxRE00\n1oHOMQ4VkQrGLhzL3cvurrOndSgE+/fHHzNGk+cURWl+qIBoAKHcEJ2O6kRieZKABAgEAkRNtM7e\nER5XX538eE7NauKKoihNhgqIBpKfl09GsNoyF5QghT0LmTpoKtnB7JSK+hUWwvjx8RFNjmP7SIwe\nrbkRiqI0D7RY3wEwesFoZqydgcEgCFd2uZK+HfrGHNapOq7DYRg3zvoh/LRurf4IRVEODk1WrE9E\nLhWR90Rkq4j8Jsl4PxFZJyIREbkmYWyyiGwSkXdF5GERkcTrm4qCbgVkBjMBm1U977153Pnandy6\n6FZy2uQwbvG4en0RYAVAz541j5eXw4QJqkkoitK0pE1AiEgQmAoMBM4AhojIGQmn7QCGAX9NuPYc\n4Fzgh8APgD7ABelaa0MJ5YYY3n14XGa118f6iXVP1NvH2k9Bge1n7ccYeOUVWy68qCgNT0BRFCUF\n0qlB9AW2GmM+MMZUAnOBK/0nGGNKjTFvA07CtQZoBWQB2UAm8Gka19pgCroV0CqjVY06TScecWKD\nGww99hhkZtYci0ZhzBjVJBRFaRrSKSA6ADt9+7vcY/VijAkDy4BP3N+XjTHvJp4nIoUiskZE1uzZ\ns6cRlpw6XtjryF4jYwX9soPZjD93PEsLljKx/8SUSoKDdVovXw6XXFJzLBq1WoZqEoqiHGyaZaKc\niHwfOB3o6B56VUTON8as9J9njCkCisA6qQ/uKq2QCOWGKOhWECv/vfGzjQ3uPgdWk5gwweZCVFbG\nj23dCiNHwrZtmnWtKMrBI50C4iMg17ff0T2WClcBbxpjvgYQkUVACFhZ51VNyOy3ZscaDAlCMBBk\n6qCpFPYqTHkOrzR4cTHs3g1r1sCuXdXjDz1ks66DQRg+3GoWKigURUkX6TQxrQZOEZHOIpIFXA/M\nT/HaHcAFIpIhIplYB3UNE1NzwV+nCazDOuJEGLtwbEo9I/yEQjBtGrzwAtx9d/yYMdbkVFmp/SQU\nRUk/aRMQxpgIMBZ4GXtzf9YYs0lE7hWRKwBEpI+I7AKuBWaIyCb38r8B24CNwFvAW8aYF9O11m+L\nV6cpkPByRpxIyp3nklFYaAVB377QvbtNrNN+EoqiHCw0Ua6RCO8MU1Jawt6KvfzhjT8QNVEAsoPZ\nLBu6DLCaRkN8E+GwNTc98QRUVVUfz8iwAiIrSxPqFEX5dtSVKNcsndSHIp7DGmBf+b5YpnVFtIKb\n5t/EB198QMSJkBXMSim6KRy2JqTycisM/BgD555rxzZuVAGhKEp60FpMacCfaQ3w7ufvUhGtSDl5\nDqqjmZIpeNEorFhhS3SMHKkhsIqipAcVEGnAy7RORJCUW5V6zYWCQfu3Xz+7HQjEF/kDeP75xlm3\noiiKHxUQaaKgWwFBia+hcdqxpzG029CUrveaC02caLWJ5cth5UrruD7vvPhzaysfriiK8m1QJ3Ua\nKVpbxJiXxhA1UQIECAaCOMZJ2Q+RiOeX8FqW9uxpNQ1NnlMU5UBRJ3UT4SXJjV04logTocqxoUie\nH6KhAsLzS0Sj1tzUsyc88og9phFNiqI0NmpiSjNl+8tivas9RIScNg1vH5fol9i920YyeclzmhOh\nKEpjohpEmvGS6CqjtsCSMYaoE2Xc4nF0Pa5rg+s1LV1qBUFODtx6a3WUU0aGFSCKoiiNhWoQacar\n+jqi5wgAHKw28U3kG8YtHlR5k2gAACAASURBVHdApTjuuAPKyqzm4NGtm82JmDRJy28oitI4qAZx\nEAjlhigpLSExIGDVx6voP7s/Dw98uMEVYD1zU0WF7We9apX9FYFWrdQfoSjKt0c1iINEfl4+2RnZ\nNRoMVUQrGLtwbEotSv145qaLLqquzwTW5PTNN7bXtWoSiqJ8G1RAHCQSGwx5BCRA1EQblGUdm9Pt\nIZHYshSsNtG/vwoJRVEOHBUQB5FQbohpl0/j0UGPxpLoPLNTQAIpZ1nHzRmC226rmV0N1vxUXGyF\nhPomFEVpKOqDaALK9pfFBIPBYIwhKEGmXDrlgJLnHnnEbgeD1sTk+Dp8FxXBzJn2eHa2+iYURUkd\n1SCagPy8fAIJX/kd41C2v6zBc3nJc55Q6J2QD+k4NtrJcaxGUVKiGoWiKKmhAqIJCOWGmDpoalyt\nJkFYvG0xoxeMblDoa2Ly3E032b/JCAZt/sSAAbZbnXakUxSlLlRANBGFvQpZeeNKBncZTIAADg4r\ntq9g+trp9J/dv8HRTBMn2r+FhVZL6Ns3/rxAAB591OZPeOU6NPtaUZS6UAHRhIRyQ/Tt0DeuDAfY\n0NfJr09m0spJtQqK8M5wbNxLnvN8C6EQTJkCrVvbENhAAG6/Hbp2hR07bNa1p3Fo9rWiKLWh1Vyb\nmPDOMPmz82OlOPwIQquMVjUqv4Z3hhlQPIDKaGWdlWGLimDsWKstZGRYYRGJWOEwfDgUFKjDWlFa\nOnVVc1UNookJ5YYoGVrCqF6jOOPYM+LGvJalibkRJaUlVEYr682dKCuzzmnHsT2tKyqssIhEoFMn\nFQ6KotSNCohmgJcf8fgVj9doMoSBHV/uiDM1eQUAgxKsM3fCc2AHAvGtSx0H9u7VSCZFUeomrSYm\nEbkU+BMQBB43xvw+YbwfMAX4IXC9MeZvvrFOwONALmCAQcaY0toe61A1MSVy1dyrmPfevNi+V5oj\nGAgyddDUWI+J8M4wJaUl9dZvCodttvWrr8YLicxMKyi0j4SitGyaxMQkIkFgKjAQOAMYIiJnJJy2\nAxgG/DXJFMXAg8aY04G+wGfpWmtzYvy548kKVsepGvcn4kQYu3BsTJMI5Ya44/w76k2s88pxZFZX\n94j5IqLR6twIRVGURNJpYuoLbDXGfGCMqQTmAlf6TzDGlBpj3gYc/3FXkGQYY151z/vaGLM/jWtt\nNng+iUtOvqRGYb+oiTaoVlNszpAVAoMHV2dbe9qE49jcCEVRlETSKSA6ADt9+7vcY6lwKrBXRP6f\niKwXkQddjSQOESkUkTUismbPnj2NsOTmQSg3xIT8CWQGM+OOByVYwx+R8pyhmrkRYP0TZWWaXa0o\nSk2aq5M6AzgfuB3oA5yMNUXFYYwpMsb0Nsb0bteu3cFdYZoJ5YYY3n14nBYRcSJMXzud82edT9Ha\nogbP6XdaQ3WOxN69ml2tKEpN0lms7yOsg9mjo3ssFXYBG4wxHwCIyDzgbOCJRl1hM6egWwGz35pN\nRaQi1okOrKlpzEtjAGo0GqrLee1vWbp3L/zxj9YP8Yc/WFOTMTa7urjYnpOfr85rRWnJ1CkgRORI\nY8y+WsY6GWN21HH5auAUEemMFQzXAz9LcV2rgaNFpJ0xZg9wIXDohyg1EK+HxISSCbzywStxY1ET\nZfRLowHIDmaztGApQL0JdKGQ/Z00qTpHwk8gALNmWSe2RjgpSsumPhNTibchIksTxuZRB8aYCDAW\neBl4F3jWGLNJRO4VkSvcOfuIyC7gWmCGiGxyr41izUtLRWQjIMDMlJ/VYYTnj/BHNnk4xsExDuWR\nciaUTKD4reKUEuig2tzk70YnAj16VEc4aa0mRWnZ1Gdi8ofRHFPHWFKMMQuBhQnHfufbXo01PSW7\n9lVsfkSLx4tsKn6rmN1f7+afZf9k8+ebY+MGw5IPl5CxPYOMQAY41Nt8yKvXNGaMFQZQXZtp/Xpr\nbsrKshFOkyapuUlRWiL1CQhTy3ayfSWNhHJDhHJDsdpNHp4T2zEOUSfKiJ4j6HRUp3oT6MBGL/lp\n377aHxEMwq232t7WlZVqblKUlkh9AuI4EbkNqy1427j7h1fY0CFCSWkJUcd+5ReEK7tcycvbXo75\nHQq6FaTclc4zM1VUWKGwfXv1mONY81J5ebXzuqREBYSitCTqExAzgSOSbIMtg6EcZLw6TJ5AGHjK\nQNq3bc/ur3fTvm37Bs3lRTUlK8XhOLB6dfwxTahTlJbFAddiEpE+rg+hWXC41GJKBS+UNadNDuMW\nj6M8Uh4Lgc0IZMTVbEppvrDVJiprVhyPIQKtWqmZSVEONxqtFpOInCEiE0VkKzCtUVanNBivDlPZ\n/jIqIhVxDYciToQxL41pUOtSrxTHqFE221qShB/4zUyKorQM6tUgRCQPGOL+VgEnAb3rqqzaFLQk\nDcIjvDNMv6f6EXEiScczAhnc3OPmBvklErUJkWqBkZ1tI5/KyjSqSVEOF+rSIOoUECISBo7EFtqb\na4x5X0Q+NMZ0Ts9SD5yWKCAAitYWMXbhWKqcqqTjtXWlq4tw2GZT794NixbZZkOBAFx4ISxZYrUJ\nNTcpyuFBXQKiPif1p9gCe8djo5beR8NbmxWFvQrpelzXmE/iiXVPsOrjVbFxg4lLmkulh4Q/2/rF\nF6vLcLziS+b2yoSHQlagaGkORTn8qFNAGGMGi8hRwE+ACSJyCrYERl9jzKq6rlUOHl6OBEDX47rS\nf3Z/KqIVgNUgRIS9FXtT6mPtxwuDrayMLxHuHw+HbYE/zZVQlMOPep3UxpgvjTGzjDGXYAvm/Q74\no4jsrOdSpQkI5YZ4eODDZAZsqXCDIepE+d/w/1IRrYiV4Sh+q5hJKyfV6cj2wmAnToQhQ+LHbr+9\n2rldWamlORTlcKRB1VyNMZ8CjwCPiMhJ6VmS8m0p21+GY6qr8Hkd6QISICABgoEgszbMIuJE6tUm\nPG1g4sRqh/WQIbBvH4webWs3eVqGV6pDUZTDg/qquc6v5/orGnEtSiPhJdP58yPAluPICGQw6JRB\nvPjei3FF/eoyN3lagjFWQMydW12/KTsbHn5YI5sU5XCkPg0ihO0KNwf4P1Io0Kc0PXWWCXeifLzv\n45SL+kG8L0KkWjiAdVavXw/TNCtGUQ476gtzDQIXY3Mgfgi8BMwxxmw6OMtLnZYa5loXteVJCEJm\nMJPh3YennCPhRSrl5Ngifv6s6+xsWLbMbms0k6IcWhxwmKvbl2ExsFhEsrGCokRE/tsY82jjL1Vp\nTEK5IaYOmlojT8LzSXQ6qlPKuRFe6KvHqFHVUU1VVTZvYvZsKziCQRg+HAoKVFAoyqFMvVFMIpIt\nIj8B/gLcAjwMvJDuhSmNQ2GvQpYPW84lJ18S199aEHZ8uSPlchx+yspqluPYvNlWfvWimWbM0P7W\ninKoU5+JqRj4Abbpz1xjzDsHa2ENRU1MdRPeGWZA8QAqIjY/AgFjDMFAkNtCt3F09tEp9ZCA6twH\nrxR4bQSDNvrpjjsa6UkoitLofJtSGw7wb3fXf6IAxhhzZKOt8luiAqJ+wjvDFL9VzMx1M4maaNxY\nQ0tyhMO2TPiSJTX7WnvhsNnZmjinKM2dA67maowJGGOOcH+P9P0e0ZyEg5IaodwQnY7qRLIvBQZD\neaSc4reKU5srZAVEdnbNMS8c9kc/go0bbckONTUpyqFHg8p9K4c++Xn5ZGdkx/kjPAyGWRtmNahM\n+NKl1mGdmRk/Fo3CvHkwciTceaf6IxTlUCStAkJELhWR90Rkq4j8Jsl4PxFZJyIREbkmyfiRIrJL\nRDRiqpHwciTuv/B+xp87PlaSw6MqWhUr7AfWLFVXSY5QyOZALF9uBUV2dk0HtjHVxf0URTl0aFCp\njYbg5lBMxeZR7AJWi8h8Y8xm32k7gGHA7bVMMxFYka41tlT8xf0GdxnM5NcnM++9eQA4OGzas4lJ\nKyext2Ivfwz/kaiJkh3MrrckRyhkS2+MGROfTOehLUsV5dAibQIC6AtsNcZ8ACAic4ErgZiA8JoO\nuc7wOESkF7bM+GIgqQNF+faEckP07dCXv7/391hZjqc3Pl3jvIpoRb0lOcCGwCbDGBg3zm6vX2//\nap6EojRv0ikgOmDLdHjsAs5K5UIRCQB/AH4OXFTHeYVAIUCnTp0OeKEtnfy8fIKBYK2d6QCCEqy3\nJAckL8vhlQr/5htb4M+Lepo1y2Zgq5BQlOZJc3VSjwEWGmN21XWSMabIGNPbGNO7Xbt2B2lphx9e\nxnVQgknHBeGyUy9LbS5fifCbb67pj/CHxCYrDx4Oa9STojQX0qlBfATk+vY7usdSIQScLyJjgLZA\nloh8bYyp4ehWGofCXoWs/2Q909dOrzEmIsx/bz4vb305pTwJzx8RDtvyG7Ul1AUC8X4JbT6kKM2L\ndGoQq4FTRKSziGQB1wP1lQ8HwBhzgzGmkzEmD+vALlbhkH4KuhWQFcyK7Qew/SMc4+AYJ+aHSBVP\nmxg50mZV+wkEqv0SnragzYcUpXmRNgFhjIkAY4GXgXeBZ40xm0TkXhG5AkBE+ojILuBaYIaINLsq\nsS2JUG6IkqEljOo1ilG9RlHYqzAuf/5A6jd5YbC9esUfdxz76w9/9fwXwWB18yE1OSlK01FnqY1D\nCS210fgkq9/kGIegBHnsssesAEmRoiKrSSRjxgwodKfyyop7nenU5KQo6eWAS20oLRsvqa6wV2FM\nOABETZRRC0ZRtLYo5bkKC60guOQSGDzYmpjA/vWHxoZCtrif9rtWlKZHBYRSJ179Jn+Pa7BlOUYt\nGMWvl/w65bkKC+Hll2H8eJtxHQjYKKdVq6yGkWhKSmZyUhTl4KEmJqVewjvD5M/OpzJamXR8xuUz\nGmRuAisQEjOuRWxNp5KSalOS3+SUaF6qa0xRlNQ44I5yigLVzuvit4rZvGczK3bEVz958PUHASjb\nX0ZOmxzK9pfV21uirKxmmXBjrCmpuLj6hp/Yyc5DQ2IVJf2ogFBSwl+/6ddLfs3k1yfHxrZ+sZWR\nC0YiCAZDQAL11m7Kz7faQmUSpWT3bmtuqkszSOafUAGhKI2L+iCUBvPARQ8w/tzxNUqGe7WcHONQ\nGa2sM2fCc0L37Vtz7KWX4O676y4Rrv4JRUk/KiCUA+Lo7KOT9pQAmy8hIuS0qbt8aygEU6ZA69bx\nJTmqqqxmUFeJcH9JDzUvKUp6UAGhHBC1NR4SJJZ9PW7xuHqT6rwb/cUXV4e+ejgO7N3b2CtXFCVV\n1AehHBBejkRij2uDwTEOBhMzM6VSu2nCBFi5smbdpj/8Afbts6XBQZPoFOVgogJCOWA8x3WPE3ow\n5qUxcULCoz4zU2wuV5MoLoaZM6vDX6NRm2D35JPWDBWJWL9D9+7WBOU46qRWlHShJiblW1PYq5AR\nPUfUMDdFTZTRC0Zz1TNXpVS/yavb9Nhj8cX9vPDXiorqqKVVq6xwCATUSa0o6UIFhNIoFHQroFVG\nqxpCwsFh3pZ5XPDUBTEhUV+f68JCGDGi/scMBOCii9S8pCjpQjOplUYjvDNM8VvFPLH+Caqcqhrj\nfU/sy009b2Lc4nFURivJCmbVmivhJcLV1ktCBFq1UuGgKN8WLdanHBRCuSGmXT6N5cOWM7jL4Brj\nqz5exagFoyiPlBM10TpzJfzRTYld6QIBOP98GDq0+piWBVeUxkc1CCVtFK0tYvSC0Tg4Nca8cNgf\nd/kx488ZX2ukk6dJVFRUH/N6XAcCtujflCm28ZBGNClKw1ENQmkSCnsV1lrEz2CImijztsyj/+z+\ntfojPE3ivvusb0Kk2uTkRTA98YQ1RWlZcEVpXFRAKGmloFsBrTNa13lOoqkp0Ynt9YgoKLAagr+X\nhAisWVMtNDIyNKJJURoLzYNQ0oqXUFdSWsLibYtZsX1FjXMCEojlS3hd7JI5sT1toqQEcnJg/Xqb\nM+GvCjtwoJqXFKWxUAGhpB0voS4/Lz928xcROhzRgZ37dmIwjFs8jq7HdaWktITKaGWcE9vvn/CX\n/x796+1EnVz8ivCiRbbXRFmZ9olQlG+LmpiUg4anTYzoOYKgBNnx5Q4c4+AYh28i3zBu8Thy2uSQ\nFcwiKEGyglnk5+UnnSu8M8yTe4dCsByIgpu9XVUFY8fWXw1WUZT6UQGhHFS8FqYRJxJXkgNsGOzo\nBaPp06EPI3qOqLOfRElpCdEO/4ChA6D3TIKZUQIB64vwqsGqw1pRvh1pFRAicqmIvCciW0XkN0nG\n+4nIOhGJiMg1vuPdRSQsIptE5G0R+Wk616kcXPLz8mNaQlCCcWMODiu2r2DWhlls/GwjoxeMZvSC\n0TWinGJzdFpN68G38V/3bo+LcAK7nZNaKShFUZKQtjwIEQkC/wQuBnYBq4EhxpjNvnPygCOB24H5\nxpi/ucdPBYwx5n0RORFYC5xujKm1+LPmQRxahHeGKSktIadNDrcuujVpv+ugBGMFALOD2SwbuixO\noyhaW8Tzm5/n6jOupuyVQu68s2bWdXY2LFumvghFqY2m6kndF9hqjPnAXcRc4EogJiCMMaXuWFwm\nlTHmn77tj0XkM6AdoN0BDhP8LUyBGgl1gsSEA0BFtILit4pj14R3hmMlO1buWMmUM0NkZnat0cK0\nosIm0U2ZokJCURpKOk1MHYCdvv1d7rEGISJ9gSxgW5KxQhFZIyJr9uzZc8ALVZqWsv1liK+eRoBA\nDf8EwKwNs2KmpsRop7KcBZSUwODBNRsPrVoF551no5sURUmdZu2kFpETgD8DNxpjatRrMMYUGWN6\nG2N6t2vX7uAvUGkU/D6JjEBGUuEAEHEiFL9VzKSVk5JGO4VC8MIL8I9/1Ox17TgwciRcdZVGNilK\nqqTTxPQRkOvb7+geSwkRORJ4CbjTGPNmI69NaUb4k+ly2uRwy8JbiDiRGucZY3hi/RM4xiErmMWU\nS6dQtr/MCoeEXIkpU2xBv2g0fo5582D+fNt3ojB5FRBFUVzSqUGsBk4Rkc4ikgVcD8xP5UL3/BeA\nYs9xrRzehHJD3HH+HRT2KmTqoKlkBjKT9paocqqImigVkQqe3/x8DeEQmy9kGw8lmptAtQlFSZW0\nVnMVkUHAFCAIPGmMuV9E7gXWGGPmi0gfrCD4LlAO7DbGnCkiPwdmAZt80w0zxmyo7bE0iunwwust\nMWvDLCqjlUnNTgEJkB3MZmnBUsD6JRIFRjgMkydbzSEZGuWktHTqimLSct9Ks8YTFDPXzYyLahIE\ngyEoQUb0HMHst2bX2YSoqAhGj46v2wS22N/999tigIrSEtFy38ohi9eEqNcJvWqMBSRAMBBkxfYV\n9TYhKiy0zutkUU57NXhaUZKiAkI5JLip501x+wZjcyWcKJs/3xwzQWUEMsjPy0/a99of5dSvnzuP\nsSaoHj2shqE+CUWpRqu5KocEXuOhB19/kG1fbMNgcIxTwzdxY/cbAWotGQ5WULRqFT//hg32d+ZM\n69zWCCdFUQ1COYQo7FVI8VXFtMpoRVCCZAYzyQxkxsaDEqTHCT2SlgxP5Oqrkz9GNGo1CU2qUxTV\nIJRDDH/OhFcKfPLrk3nxny9iMNy66FbO7nA2YH0UtZUMLyyEbdvgwQdr1m9yHBg1ClasgDPP1L4S\nSstFo5iUQ55JKydx97K746KcwGoUj132WK19scH6HMaNs+U4aiMjA6ZOVbOTcniiUUzKYU1+Xj7B\nQLDGccc4lO0vq/NaL+u6dWsb8pqMSKSm2SkchkmT1KmtHN6oiUk55AnlhhjefTjT106PO24wrPp4\nFeGd4VobD0F8r+tNm+Dpp2ue4zhWSGzbBvv2waxZVnBkZdlr1QSlHI6oBqEcFhR0K6B1Rusa5Tnm\nbZlHv6f6UbS2bq9zKGST5f7yFxg/vvYSHZMnw/Tptox4Y3WtU21Eaa6ogFAOCzzn9cheI8kOZscJ\niogTYdSCUTWERLJcCYAHHrC5EpdcUvdjikAwCDt2HPjNPRy2vbO1h7bSHFEntXLY4ZXnKFpXhJNQ\nJf6Mdmdw+amXs698H7M2zCLiRGotz+HdvMvLa0Y6eXiaRnb2gZmaJk2ywiEatcJm4kQt+6EcXJqq\no5yiNAlet7rdX+9m3nvxVfo279nM5j2b445VRCpiuRL+gn9+38TevfDQQzVrOXn7FRX2vIYKiPx8\n68eorLR/8/Mbdr2ipBPVIJTDlvDOMP2e6pe0t0QifU/sy5pP1mCMoVVGq1oL/o0ZU7PHhMcZZ8Av\nf9nwcNhw2AoXzbdQmgKt5qq0WMI7w0x+fTJ/f+/vtXaqSyRAgPsuvI87zq9p6wmHobgYFi/9N6Xv\ntwFqxsaOHw9HH603fOXQQPMglBZLKDfEC9e/wOvDX6dfp34EUvjIGww5bXKAmo7sUAgKfhvm4/6X\nQbACcCBB8Dz4INx1V8OczrU5zBWlKVEfhNIiCOWGWH7jcsI7w7HWpoveX8T89+bjEO9YMBhuWXgL\nK7av4NlNzxI10VhjolBuiJLSEqId/gHD+sNbQ5F1N2Oc6n8lY+xvebnVNurTIsI7w3UWF1SUpkI1\nCKVF4W9t+sL1L1DYq7BG7gTY0NinNz5NlVOFYxwqotWO7Py8fLKCWQQ7rab14NuY/vy7nHFGzccy\nxvot6mttmkpxQUVpClRAKC2agm4FseqwWcGsWk1QjnFYvG0xoxeMBmBpwVIm9p/I0oKlFA7uyuOP\n2yikGtc5tt1p//61C4mYwHHXkKy4oKI0BeqkVlo8ntkpPy+fyW9MZt6WWhpYuwQIcMVpVzDw+wMp\n219GTpsc+7fscp6Y1LXWwn+nnw5dukD79lBQEG968q+BXSGNalIOGhrFpCgpEt4Z5oKnLqDKqUrp\nfK83dkACZAQyGNRqIgvuGkekIgAESRblBDbBbto06No1PsTVS86rqDQEMyI8OncLhYO7NtbTU5Qa\nqIBQlAbgZWLv/no3L73/UsrCIsbOs6E0Hz7qA1uuojYhARAIGIwxZGQabhpuK9IWzTQ4UQEiBL6/\njGkPtlchoaSNJhMQInIp8CfsV6nHjTG/TxjvB0wBfghcb4z5m29sKHCXu3ufMWZ2XY+lAkJJB35h\nsWH3Bkq/LE394p1nI7NLMJHsWk7w/vckti0BAxLFRAUIgkTJzDIsX5ap5iYlLTRJqQ0RCQJTgYuB\nXcBqEZlvjPHXOdgBDANuT7j2GOAeoDf2P2ete+0X6VqvoiTDK9sBVlj0n92fimhFzLRUJ7lv8v3b\nCsnYOJx2zg/gmxxWroTqL2WeZlG9b0tHBeA7e+Df7cBkEI2YAyrjoSjflnTmQfQFthpjPgAQkbnA\nlUBMQBhjSt2xhAo3/Ah41RjzL3f8VeBSYE4a16sodRLKDbFs6LJYHsUtC2+pt4zH+22K4axitgYy\neXTQo/DSB7z5zHlUbh6IDSJMND+JPf7v493dKJmZkJ9fsyGSoqSbdAqIDsBO3/4u4KxvcW2HxJNE\npBAoBOjUqdOBrVJRGoBfowAYu3AsURMlIAF6tu9Jfud8NnyygVc/eDVOw6hyqhi5YKTdue4BWHMz\nvPQYmAAEolaJMJnu2Z7QcCDnXarabWXyY2cznvaHpBbhrzVFx3BcQUSleXNIZ1IbY4qAIrA+iCZe\njtLCKOxVSNfjuta44YV3hinZbpPfaqX343D8O9aZnVdij73+K9hyBdZlByDw+Zk4n5/JvHfhxTnw\n2GPJiwE214J/XlRWZSVkZEYxBXcQ7fAPzRg/REingPgIyPXtd3SPpXptfsK1JY2yKkVpRBI1Cu9Y\nydCS+osE5r5pf13k+ms47ZsbefcvI+CjvlgTlMHTKKJRW0120aL4XIpw2CbieSXDly1rPkKipMSu\nKxoFxwDbzsWcuDyWMa4ConmTzkzq1cApItJZRLKA64H5KV77MnCJiHxXRL4LXOIeU5RDAq9I4PTL\npxOUav+BIOQdnUf39t3jzg8QIDOYSZfu/yJz0K8hoxzw/BvVAiYatZnZ06dXZ2cXF9t+FMZARYVh\n8mO7Y+c3dTtTr99FMGj/Zn7vdc0Yd2nq9yYV0h3mOggbxhoEnjTG3C8i9wJrjDHzRaQP8ALwXaAc\n2G2MOdO9djjwW3eq+40xs+p6LA1zVZorXqgs2NIe3rfmorVFPL/5ebqf0D2uw10wEGRQq4l8/OII\nVq08Gkx1GKylOvpp1Ci7PX26p2kYgn0eZ+XzP4BdoZh5JyvrwDreNQbqg6iJ3/TWlO8NNGFHOWPM\nQmBhwrHf+bZXY81Hya59EngynetTlINBMjMUWB9GYS/rUJi0chIRJ0LURMGBvmdFyfn+Lla9kQ1R\n13ltglQr/VZgTJ/ucGzul8ARgEDAzlG84H067Q3FzDuVlQfW8a4xCIX8j5v8tWhp+E1vyd4bf+mV\npny9DmkntaIcLngF+7yS3/l5+RR/+RgM3WAd2a0/h0WPQNSfdGe1h893Hu3uR60QWXMzMzcE+K/b\n7LfT8gqDwWGvlALfAxJqP0Gj3Yyay42tuVNXq9nmVP5dBYSiNANCuSGWFiyNu7kWv1Uc78g+/h14\nqwC+Ph62XEnNPIqgq1gI0YjhoT84DBnxMU/POB7jBJh8VwdgG/u+yODJvTaaKBgIIggRJ0JWMIsp\nZ/4fZe92PaBoqFRvbHUJkVi/jrLL611HfcIoHIbiedshbzkFl5/SrASWv9954nNMVv5dBYSitHAS\nTVEF3Qp4csOTVEYrCRDgh33KeTt3rG1w9Or/wOu/IbGbXXXZDsGJwtziI8Fk2OPRAJN/29meEVgM\nPWYR7fZnyLVe0m8+7M6Ye7tANDW7eGJobSo3trqEiDdWUdoTZ/YvCTiGjKAw6LrdtD/nldhN3rvx\ne0IumTAKh6H/hVEqKjpA8Bqe3DCIkrsmNTsh4flk2Fkt5JJpk02FCghFaaZ44bL+b8mew/uJwD1U\nffcDWH+TjXhq/S94f5Drr6gOTozuPyJ+UiMYxJqq1oyEDUNh6AA7VvI7olUBMPX7LBKdrFP+upEd\nGTvICGSAQ603trqE/DbYugAAEl1JREFUiDfmfHg+RLJwjFAZNcz7SzuYa2/yjwx8hHE/60p5RUdM\nYCEMHUBlp9UUL3ifkr2hmLAqnredisqOVjhGDZXbzqH4reIGC4h05pfUJSyHdhsKxAc1NAUqIBSl\nGZOoVXj7Bd0KKO5RzO6vHwCgfdv27N6ylXnPHAFrR7gObX+tp5qFASEAkSxY/Ef4pAc4GfaYRMjI\nlDrLe5SUQHmFg3EClFdEGfM/b+IceQzBk/sy4oozKehWAFjnu98EVNe34/y8fGTXOfBlJwhEIOqW\nHSEI0Uyqtp3L84vKqKg0GCdoM89L83EkwBP/8zOiEYMEonTu+y5by7aBHA/iQLAK8pYxa8P6Bt1w\n0x1pVFsnQb/Q8F7HpkIFhKIcgiSLjAp3C7OwPJ/K9uvcMh7uv7dEbUmPWOIdVAuKoJuUJ9XHcrZw\n5BnvMW9LkOIym35U0K2AjWvb8vyiMq4emMPe9tswgUvAZGIkSnTdL8DJILK8kt2nPgrdqm90IkLP\n9j25qedNFPYqrOFribErhJm9BCpdDejoUtiXC04AxBBs+wVXD8xh2ewIjoN74y/BbCigqsJdfzTI\n1td/APwAApXQ63HoZn05ESfYIHt+fZFGqVCXnySZsGxO/gdQAaEohw2eSaq4ezFv/vCXbHjF7SHR\nfh0s/hNEsql2bPt9F4H4/bLT2bPidCavMHCaA99fxPStn8B7PwYjvPK4wxFXPQWXvgTvXg2Z++G9\nK1xzDvx98Zd8fNQ4KqIVOMYBA6s+XsWq/wsw7dOuZAWyuenGywmdb9fn3URXPTOAaFUvbNqUgb0n\ng1QhAcGYILL4Ybr+OoNH525k9NS5OCe9Bp/+ANbeTE3hBzgZyFG7yMpbT8SxyXl7t57Oj/5SwtUD\nc+rtsZFz+kYCGadhyCArS8g5fSOTVi5IOUKraN5GbnlsEc5Jr5GdNzGp0z6ZKam5+B9AGwYpymGL\nP0HvyD0/4sXnvsuWV0OYaMDVKHxaQw3qOx4FMW4SH1TXjwLO/T202mdrTHkRWDvPhqeWVYfpBisY\n/PuHGdj/aMYtHkd5pByz8yyYtRycTOIKFnqajlTR9xcvMeW+49n42UZG/fdazIKpPnNatPpc9zHG\nz1zM4IuOp6S0hE3rj+Tp/7oRolmQUcmMZ7fVKiSK1hYxduFYIjv6ENw+gNuG9OSRj39GRWlPAtsv\nZOqYa+sUMEVFMHpMFCdqIODAoLGMGhlk2uXTYu9NXc76ZFqH55zf3e4Z2p/2YaP5J5osUU5RlKYj\n0Qz1wI3VTtfF77zBir+GqGlyEmp8E/fVg6rerg6pjT8/AuH/sgIoWGkd4Llv2lyOqO/GH81k3vQf\nMm/LvZD7TfWiT10QX7AwUGUFkWN9CasyH+C8Wau46Itn4KVproBy1xCIwqCxdOdGV0vJpHDw4NjU\nd927wAoHkwERw/OLyiisHqZo3kaeX1RG97P38r+73FLuHd8gCvy/RXmUl/8Cs+iPONEsxix36FqS\n3OQUDsMtt4ATdYWwY2Dho8xsP4CCbmGr6XmmpB19KC+9kOJj3ofLSZpx781ZHZU1FoYOYNaGWTw8\n8GHK9pelLe9EBYSitCC8rOb8nUK/f99CZE0BfNITIQPjiHXq1lHaI/5volCJ2k3P2R3BOsBPWA/Z\ne925vQirAHxwEWzvVx1FNXupvYEHI3DKfGj7qfUfQHXV29w3cXaczSuzrgDHJxwkCqE/wDc5VPR+\nnNHXnsX6T9YzeoG92U5+ZiXO3o7W+e0YCFbR/ey9MS1r87qjWDHxdxA9nVeeiMApz9jHb78Os/hP\nbI1mgfzcrt9kEK2qsjkWHT+Ofdv3fDRt9p+G47SPf31MgOiH5zH59cn07dCXvVtPx7w4FdYPxTgZ\nFK2IMGPtAEzuGwDM2jCLZUOXVUd4lVg/iBVuAm8VUJH7JmMXjsUxTtoS6tTEpCgtlMSktJwcKCuD\nTV8v56+vbMHsOQ22n0u1+cgh1tAIqBYUjr35H/8O8lk3K2iS4hBfodY1C532dyg7FfacCQgSiBIc\n8N9Ez/3/7Z17sFVVHcc/33OAqyMCymUQBXmIVhYGSA6VvXAsH6k11kiPsRonjLR0TA3G0bHSP3R6\nGEU2FCqm+aq0sjSNRy9Rg3gEEkoIKIGgBkUpXO759cda+559z93nXi7cfc6J+/vMnDlrr73OWt/z\nO7P376zH/q0b2kfCTfb63nksLPks4f+thR7GW+6C1VPLPZczLoPXmmHUIgoqUrrjseB81Boc1qiF\nsDs+gf7WO0O9C74WJ/ZTbao1ftU+wN4wXGSCYgsjv3gRLw58gJKV0IvvKLdR2EtBBUp74wqsxD5v\n+CU69RthR8F5v62YE9oLxywJ2qJTHPuvz3LVx9/GuKHjuHn2Fh6650go9Qt6irvh01PQiCcxDCEu\nPvnitiGs7lC3PalriTsIx+k50s7jkZ808+yrz9I85gX+NGcqrS3BYahQ4vRzX2H+w8207oXyTR+q\nD1NVDmN1pNCnxK0PPMOyLcvYuvqNrP3vE6xZfGzbJHlbtRZ0FIpGqdRafiCw7UZeCD2Gw/8BO0an\n2k0cVdLgHjjhVxXPkaR0K85tFFph+JPwn2ZofhbGPtLmhNo5GLWE1VMDN8HrA+CJK8srymiFQZtg\nxyg6LA5I65FBax+QoQJxj/LkVWjfRuxZNRWb2vU69hV3EI7j9AhJeHEI+1EsWgTXXhuWgnZNV04j\n5L1p/L+Z+70BnHZaCGNeKlUOd6WHugpIyT7fqeEmi/MknU60Vz4nUoL+W2HXMNqv9mqFgRth50ja\nTca3fbYER62El06KQ2glGPlHGLImFFkyLfW5ak++p+2S9NQqzyXp1o5ZI/+ITr+GGy/8IDPfNTPj\nO1fHJ6kdx+kR2kdmDfTrl9zIO5aXiDdwKBbFBRfA/fdDSwu0jyNVvnFePn1A25h7qLOyXLgzFgpC\nCntNSKKlBVQwxr3vGZY/Po5sh9S+jjLRIexK72ycmpDfOaairuR8/NzWCan8Imx8T3iphcwluG2f\nr+YMKnUm6dZym+lFAhvfg922kMFnr6UncQfhOM5+kw46l8xhDB4My5aF8xfGB4HT4SouuST0QubM\nae8Ahg8PvZFp00JPJdvxqO39yith0KByJNTQRoFFi05i5fz05zqbcM9aqRWOJQtzIG0rparVVW3O\nhdTQUuWNPvRMxr93E2sWj2HPnuBEO9ZV2dMppOqrOF/qyytrxsGH6DHcQTiOc0Bk9SqyylSWnzAB\nLr00DE81NYWeRVKu0vEkDmfAAFi+HM4/v+Pe3Ok2mppCD6RYhMmTxfr1sHlzuAkXCjBpkli2LOnJ\nJJSdRlOTmDUrlJk7t1xOBeOIkS/w6obhqRVZ1Sn2gT5FsXcvoBKlkrBSmL+56saN3DTjuLZhu7lz\nVUVPfG9bBZa9qqxvX7ULG94T+ByE4zh1I69geJX1ZsVVgvJ8yoQJwQlt3dp+v++krvS8S1LfnXeG\n8tVI6oHUjnpU/76VdR51VFlXovHyy0OvSoJzzoETTgj1HX00XH31/tnQJ6kdx+n15BmZtVbk8R3c\nQTiO4ziZdOYguh5EcxzHcXoluToISWdIWitpnaQZGeebJN0Xzz8laVTM7ytpnqS/SlojqXsLex3H\ncZwDJjcHIakIzAbOBE4EPibpxIpiFwH/NLOxwLeAm2L+R4EmMxsHnAxcnDgPx3Ecpzbk2YM4BVhn\nZuvNbA9wL3BeRZnzgHkx/RPgNEnJIuHDJPUBDgX2AP/KUavjOI5TQZ4O4hjghdTxizEvs4yZ7QV2\nAoMJzuI/wBZgE/B1M3u1sgFJ0yQtkbRk+/btPf8NHMdxejGNOkl9CuGZ8qOB0cCXJI2pLGRmc8xs\nkplNGjJkSK01Oo7jHNTk+ST1ZmBE6nh4zMsq82IcThoIvAJ8HHjUzFqAbZL+BEwC1ldrbOnSpS9L\n2rifWpuBl/fzs3niurqH6+o+jarNdXWPA9E1stqJPB3En4HjJY0mOIKphBt/ml8AnwIWAx8BFpiZ\nSdoETAF+JOkwYDJwS2eNmdl+dyEkLam2DrieuK7u4bq6T6Nqc13dIy9duQ0xxTmFS4HfAGuA+81s\ntaSvSjo3FpsLDJa0DrgCSJbCzgb6S1pNcDS3m9nKvLQ6juM4Hck1WJ+Z/Rr4dUXedan064QlrZWf\n25WV7ziO49SORp2krjVz6i2gCq6re7iu7tOo2lxX98hF10ETi8lxHMfpWbwH4TiO42TiDsJxHMfJ\npFc4CEm3SdomaVUq70hJj0t6Lr4fEfMlaVYMILhS0sQa67pe0mZJy+PrrNS5mVHXWkkfyEnTCEkL\nJT0jabWky2J+I9irmrZ62+wQSU9LWhF1fSXmj45BKNfFoJT9Yn5mkMoa6rpD0vMpe42P+TX7LWN7\nRUnLJD0cj+tqr0501d1ekjYoBC9dLmlJzMv/mjSzg/4FvBuYCKxK5d0MzIjpGcBNMX0W8AhhX7/J\nwFM11nU9cGVG2ROBFUAT4enyvwPFHDQNAybG9OHAs7HtRrBXNW31tpmA/jHdF3gq2uJ+YGrM/z4w\nPaY/D3w/pqcC9+Vkr2q67gA+klG+Zr9lbO8K4MfAw/G4rvbqRFfd7QVsAJor8nK/JntFD8LMfg9U\nxnJKBwqcR3mr7/OAOy3wJDBI0rAa6qrGecC9ZrbbzJ4H1hFCkvS0pi1m9peY/jfhGZZjaAx7VdNW\njVrZzCwszYZwI+5LCDg5hRBXDDraLCtIZa10VaNmv6Wk4cDZwA/jsaizvbJ0dUHN7NVJ+7lek73C\nQVRhqJltiemtwNCY3pcgg3lzaewa3pZ0G+uhK3blJxD+eTaUvSq0QZ1tFocllgPbgMcJvZUdFh4Y\nrWy7WpDK3HWZWWKvG6O9viWpqVJXhuae5hbgaqAUjwfTAPbK0JVQb3sZ8JikpZKmxbzcr8ne7CDa\nsNAva5T1vrcCxwHjCdFsv1EPEZL6Az8FLjezdqHW622vDG11t5mZtZrZeELMsVOAN9ZaQxaVuiS9\nBZhJ0Pc24Ejgy7XUJOmDwDYzW1rLdruiE111tVfkVDObSNhf5xJJ706fzOua7M0O4qWk2xXft8X8\nfQkymBtm9lK8qEvADygPidRMl6S+hBvw3Wb2s5jdEPbK0tYINkswsx3AQuDthK59Eq0g3XabLrUP\nUlkLXWfEoTozs93A7dTeXu8EzpW0gbBPzBTg29TfXh10SbqrAeyFmW2O79uAB6OG3K/J3uwgkkCB\nxPefp/IvjCsBJgM7U9243KkYK/wwkKxw+gUwNa7oGA0cDzydQ/sixMhaY2bfTJ2qu72qaWsAmw2R\nNCimDwVOJ8yPLCQEoYSONkts2Rakska6/pa6qYgwbp22V+6/pZnNNLPhZjaKMOm8wMw+QZ3tVUXX\nJ+ttL0mHSTo8SQPvjxryvyb3d3b7/+kF3EMYemghjMddRBjDnA88B/wWODKWFSFY4N+BvwKTaqzr\nR7HdlfGHHpYqf03UtRY4MydNpxK6qiuB5fF1VoPYq5q2etvsJGBZbH8VcF3MH0NwSOuABwjb6AIc\nEo/XxfNjaqxrQbTXKuAuyiudavZbpjS+l/JqobraqxNddbVXtMuK+FoNXBPzc78mPdSG4ziOk0lv\nHmJyHMdxOsEdhOM4jpOJOwjHcRwnE3cQjuM4TibuIBzHcZxM3EE4ThdIalU5kudySTO6/tQ+1z1K\nqWi+jtNI5LonteMcJLxmIVyF4/QqvAfhOPtJjNF/s0Kc/qcljY35oyQtiMHd5ks6NuYPlfSgwv4M\nKyS9I1ZVlPQDhT0bHotPPSPpiwp7X6yUdG+dvqbTi3EH4Thdc2jFENMFqXM7zWwc8F1CJFCA7wDz\nzOwk4G5gVsyfBfzOzN5K2Adkdcw/HphtZm8GdgDnx/wZwIRYz+fy+nKOUw1/ktpxukDSLjPrn5G/\nAZhiZutjEMGtZjZY0suEcB8tMX+LmTVL2g4MtxD0LaljFCEM9/Hx+MtAXzO7QdKjwC7gIeAhK+/t\n4Dg1wXsQjnNgWJV0d9idSrdSnhs8mxBTZyLw51SkU8epCe4gHOfAuCD1vjimnyBEAwX4BPCHmJ4P\nTIe2jXwGVqtUUgEYYWYLCfsPDAQ69GIcJ0/8H4njdM2hcVe2hEfNLFnqeoSklYRewMdi3heA2yVd\nBWwHPhPzLwPmSLqI0FOYTojmm0URuCs6EQGzLOzp4Dg1w+cgHGc/iXMQk8zs5XprcZw88CEmx3Ec\nJxPvQTiO4ziZeA/CcRzHycQdhOM4jpOJOwjHcRwnE3cQjuM4TibuIBzHcZxM/gd2guItfXq1MgAA\nAABJRU5ErkJggg==\n",
-            "text/plain": [
-              "<Figure size 432x288 with 1 Axes>"
+              "<Figure size 720x288 with 2 Axes>"
             ]
           },
           "metadata": {
@@ -2918,7 +2898,6 @@
       "source": [
         "Great results! From these graphs, we can see several exciting things:\n",
         "\n",
-        "*   Our network has reached its peak accuracy much more quickly (within 200 epochs instead of 500)\n",
         "*   The overall loss and MAE are much better than our previous network\n",
         "*   Metrics are better for validation than training, which means the network is not overfitting\n",
         "\n",
@@ -3171,7 +3150,7 @@
         "# See how they line up with the data\n",
         "plt.clf()\n",
         "plt.title('Comparison of various models against actual values')\n",
-        "plt.plot(x_test, y_test, 'bo', label='Actual predictions')\n",
+        "plt.plot(x_test, y_test, 'bo', label='Actual values')\n",
         "plt.plot(x_test, predictions, 'ro', label='Original predictions')\n",
         "plt.plot(x_test, model_no_quant_predictions, 'bx', label='Lite predictions')\n",
         "plt.plot(x_test, model_predictions, 'gx', label='Lite quantized predictions')\n",
@@ -3183,7 +3162,7 @@
         {
           "output_type": "display_data",
           "data": {
-            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEICAYAAABcVE8dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOydd3gUVdfAfzebQEioAiIt2ajUdCAU\nIyUIRBAEhFAEpEhVXl8/pCiIYHuFABZABBsoPUFpii+CCVXzmtAERGlZQlOQHhJCsjnfH7O7bJJN\ng0BIMr/nmWd3Zu7cuXPnzpk755x7rhIRdHR0dHSKP06FXQAdHR0dnXuDLvB1dHR0Sgi6wNfR0dEp\nIegCX0dHR6eEoAt8HR0dnRKCLvB1dHR0Sgi6wC9klFL9lFI/FnY5rCilyiil1iulriilIu/B+Q4q\npdrc7fPcC5RSRqWUKKWc85B2kFJqx70oV15QSnkopRKVUobCLsu9QCnVRil16i7ke1/d18wUG4Gv\nlHpWKRVnabRnlVI/KKUeL+xy5YaILBWRDoVdDjt6AtWAyiISdrdPJiLeIrLlbp9HJ2dEJEFEyoqI\n+U7yUUptUUoNLahy2eWb55epTvYUC4GvlBoDfAj8B01YeQDzgK6FWa7cuE8brydwWETS7uZJ7tNr\n19Ep3ohIkV6ACkAiEJZDmtJoL4QzluVDoLRlXxvgFDAeOAecBboBnYDDwEVgol1eU4FVwErgGrAb\n8Lfb/ypwzLLvd6C73b5BwE7gA+AC8I5l2w7LfmXZdw64CuwHfOyu82vgPHACeB1wsst3BzATuATE\nAx1zqI8GwBbgMnAQeNqy/U3gJpBqqdPnMx1XA0gGHrDbFgj8A7gAjwBRlmv7B1gKVLRLawImAL8B\nKYCzZVu7PNwnWz3Z5SfAo5b/nSz1fQ04DYzN5trt78Fl4DjwmGX7SUvdD8zUvrKrd4Olzv+x5POi\npUzOdsd+gdamTlvutyHz9eR03x2UfzBwyHKdx4ERmfaPt5zvDDA0Ux09BeyxnOMkMNXuOGOmsm8B\n3rbU1TXgR6CKZZ8rsMRyny8DsWgdrXcBM3ADrf3MzeYaIoG/gCvANsDbbl8ZYJalrq+gtesyQIKl\nfImWpQXas7gkh2vItq6wPPfZlO8TYGambWuBMXl8xnc4Ko9dvQ61Wx9iKeMlYCPgmd82kS95eTeE\n8L1cgCeBNPtKdZDmLSAGeBCoCvwMvG1349OAN9CE1jC0h3sZUA7wRhNyXpb0U9EEYk9L+rFoAtbF\nsj8MTTA6Ab2B60B1u8aQBvwLTdiVydRAQoFdQEXLDW9gd+zXlkZXztKQDmMRyJY8Ui1lNwCj0B54\n5aAuXICjwESgFNDW0nDr2V3fkhzqMgoYZrc+A5hv+f8o0B5NcFdFe5g/tEtrAvYCtYEydtva5eE+\n2erJLj97YXYWaGn5XwlolE35rfdgsKWu3kETJh9byt3BUh9l81DvI4E/LNfzABBNRoGzGlgAuFuu\n6VcsQiev991B+Z9Ce7EqoDWQZL1WtGfhL7Q264YmlO3rqA3gi9Y2/YC/gW6OhBOaYDoG1EVrp1uA\naZZ9I4D1lnMYgMZAeUcCLZtrGGKpT+sLfq/dvo8tedS05P2YJV2G8jlqqw6uIae6akP2Ar8V2gtR\n2bWnZKBGHp/xPAl8NA3EUcv9dkbrTPyc3zaRL3l5rwTz3VqAfsBfuaQ5BnSyWw8FTHY3PplbPa9y\nlpvUzC79LrsHYyoQY7fPCTth4+Dce4Gudo0hIdN++wbSFk2gNMfSi7RsN6D1vBvabRsBbLHL46jd\nPjfLNTzkoDwt0YSCff7LsfT2yF3gDwWiLP+V5cFolU3absAeu3UTMCRTGhO3BH5O98lWT3b77YVZ\ngqVOyufSFgYBR+zWfS35VLPbdgEIyEO9RwEj7fZ1sOTljNbjTcHyYrPs7wtE5/W+57H9rwH+bfn/\nJfCe3b5H7evIwbEfAh9Y/hvJKvBft0v7AvBfy/8haC9jPwd5biEXgZ8pfUXLeSugPUvJ2H0x26XL\nUD5HbdVRmhzqqg3ZC3xlaU+tLOvDsLT5bNJnfsbzKvB/wO4r2nL9SWhq1dtuEzktxUGHfwGokotO\nuAbaJ6KVE5ZttjzklrEq2fL7t93+ZKCs3fpJ6x8RSUdTCdUAUEo9p5Taq5S6rJS6DPgAVRwdmxkR\niQLmovVyzimlPlVKlbcc7+LgGmrarf9ll0+S5a99ma3UAE5ayp1dXjnxDdBCKVUdrSeUDmwHUEpV\nU0qtUEqdVkpdRethVsl0fLbXT+73KSd6oKl1TiiltiqlWuSQNvO9RUQc3e/c6r0GGa/HPp2n5diz\ndm1hAVpPPwM53PcsKKU6KqVilFIXLXl24lYdZy7PyUzHNlNKRSulziulrqB9oWS+P/b8Zfc/iVvt\naTGa+mGFUuqMUipcKeWSQz72ZTAopaYppY5Z2ojJsquKZXFFe/HfMbnUVbaIJn1XoL2gAZ5FU09a\n883tGc8rnsBHdvlcRHvZ1MxPm8gPxUHg/4LWk+qWQ5ozaJVrxcOy7Xapbf2jlHICagFnlFKewGfA\naDQvl4rAAbSbaEVyylhEZotIY6Ah2uf0ODQdcaqDazh9G2U/A9S2lDvfeYnIJTR9bm+0B2GF5QEB\nzWgugK+IlAf6k/HaIefrz+k+XUf7cgFAKfVQpnLFikhXNIG6BojIy/XkQm71fha7tmDZZ+UkWrus\nIiIVLUt5EfF2dKJs7nsGlFKl0V64M9G+SCoCG7hVx2fR2qKV2hlzYBmwDqgtIhWA+WS9P7kiIqki\n8qaINERTuXQGnrPuzuXwZ9FUGe3QevVGy3aFVt830NQwWU7rYFuGNgHY2kQe6io3lgM9Lc90M0te\n5PEZty8f2ZURrY2MsGsfFUWkjIj8DHlrE/mlyAt8EbmCpn//WCnVTSnlppRysbzdwy3JlgOvK6Wq\nKqWqWNIvuYPTNlZKPWP5qngZ7cGOQdPVCpoNAKXUYLS3f55QSgVZemEuaI3lBpBu+fqIAN5VSpWz\nNLoxt3kN/0PrrY231FMboAtajyavLEN7wHta/lsph2ZQu6KUqkn+G2hO92kf4K2UClBKuaJ9zgOg\nlCplGc9QQURS0Yxc6dwheaj3COAlpVQtpVQlNGOe9dizaC/GWUqp8kopJ6XUI0qp1pnPk919d1Ck\nUmj67PNAmlKqI5oayUoEMFgp1UAp5QZMznR8OeCiiNxQSjVFE775RikVopTytfjsX0V7KVrL+zfw\ncA6Hl0N7Xi6gCcL/WHdYvjq/BN5XStWwfA20sAjv85Zz2Oe9F2hlGUNQAXjNbl9udZUjIrIH7QX0\nObBRRC5bduX5GReR82idg/6WaxlCxpfZfOA1pZS3Ja8KSqkwy/+8tol8UeQFPoCIzEJ7EF9HuxEn\n0d7AayxJ3gHi0LxD9qN51rxzB6dci9bDvQQMAJ6x9Hp+R/Mw+AWt4fuieTnklfJovYdLaOqBC2hG\nUdAMvdfRvA12oAnaL/NbcBG5iSbgO6I16HnAcyLyRz6yWQfUQbOd7LPb/ibQCM274nvg23wWL9v7\nJCKH0Yy6m4EjaHVgzwDAZFETjESz7RQEOdX7Z2iqjX2Wsma+3ufQBM/vaPd0FVDdwTlyuu82ROQa\n8BKaYL+EJrDX2e3/AZiNZjw+itYJAU3AgqaHf0spdQ3tZXq7X0EPWa7lKpqHyVY0NQ/AR2g940tK\nqdkOjv3aco2n0eolJtP+sWj3PhZNxTEdTYedhOYFtNOiAmkuIpvQvOV+Q7OzfWfNJLe6yiPL0L5E\nbJ2a23jGh6F1fC6gGdN/tstrteX6Vlja7QG05xLy2Cbyi9UKrZNHlFJT0Yxg/Qu7LDo6OaGUaoAm\nRErLXR5XoVM0KBY9fB0dHQ2lVHelVGmLimk6sF4X9jpWdIGvo1O8GIE2WOcY2iCoUYVbHJ37CV2l\no6Ojo1NC0Hv4Ojo6OiWE+zaAVZUqVcRoNBZ2MXR0dHSKFLt27fpHRKo62nffCnyj0UhcXFxhF0NH\nR0enSKGUOpHdPl2lo6Ojo1NC0AW+jo6OTglBF/g6Ojo6JYT7Voevo3O/kZqayqlTp7hx40ZhF0VH\nB1dXV2rVqoWLS54ClQK6wNfRyTOnTp2iXLlyGI1GlMp3kEkdnQJDRLhw4QKnTp3Cy8srz8fpKp1i\nyNKlYDSCk5P2u3Rpbkfo5IUbN25QuXJlXdjrFDpKKSpXrpzvr029h1/MWLoUhg+HJMsUKCdOaOsA\n/QoqfmQJRhf2OvcLt9MW9R5+MWPSpFvC3kpSkrZdR0enZKML/GJGQkL+tusUPdasWYNSij/+yH0K\ngw8//JCkzD2AfLBo0SJGjx5928fnlUGDBrFq1SoAhg4dyu+//55t2i1btvDzz7aw8syfP5+vv/76\nrpexOKAL/GKGh0f+tuv6/rvH3arb5cuX8/jjj7N8+fJc096pwL8T0tJuLyrz559/TsOGDbPdn1ng\njxw5kueeey7b9Dq30AV+MePdd8HNLeM2Nzdte2as+v4TJ0Dklr4/J8GkvyDyxu3UbV5ITExkx44d\nfPHFF6xYcWtWSrPZzNixY/Hx8cHPz485c+Ywe/Zszpw5Q0hICCEhIQCULXtrXvtVq1YxaNAgANav\nX0+zZs0IDAykXbt2/P333+TE1KlTGTBgAC1atKBOnTp89tlngCaMW7ZsydNPP03Dhg0xm82MGzeO\noKAg/Pz8WLBgAaB5mYwePZp69erRrl07zp07Z8u7TZs2trAq//3vf2nUqBH+/v488cQTmEwm5s+f\nzwcffEBAQADbt29n6tSpzJw5E4C9e/fSvHlz/Pz86N69O5cuXbLlOWHCBJo2bUrdunXZvn07AAcP\nHqRp06YEBATg5+fHkSNHbvveFAlE5L5cGjduLDq3x5IlIp6eIkppv0uW3No3fcd0iToeJdOni1Qz\nnJd3mSA/GZFpwcgm2oo3+wSyHmfN181NRBNh2uLmljVdceX333/Pc1pPz4z1ZF08Pe+sDEuWLJEh\nQ4aIiEiLFi0kLi5ORETmzZsnPXr0kNTUVBERuXDhgqUcnnL+/Hnb8e7u7rb/kZGRMnDgQBERuXjx\noqSnp4uIyGeffSZjxowREZGFCxfKiy++mKUcU6ZMET8/P0lKSpLz589LrVq15PTp0xIdHS1ubm5y\n/PhxERFZsGCBvP322yIicuPGDWncuLEcP35cvvnmG2nXrp2kpaXJ6dOnpUKFChIZGSkiIq1bt5bY\n2Fg5d+6c1KpVy5aX9ZqmTJkiM2bMyFAW67qvr69s2bJFREQmT54s//73v215Wq/p+++/lyeeeEJE\nREaPHi1LLA04JSVFkpKS8ngn7g8ctUkgTrKRq3oPvxjSrx+YTJCerv2eNoYTHR9NeDhsn1Wd7vOf\nxLT6Uc6NfIxJna4QGlaRRafn0oXvONh8E/Wf9WLLCSOLB26mb99b+eZkENZ7/hm5W7aU5cuX06dP\nHwD69OljU+ts3ryZESNG4OysOd498MAD+cr31KlThIaG4uvry4wZMzh48GCux3Tt2pUyZcpQpUoV\nQkJC+PXXXwFo2rSpzTf8xx9/5OuvvyYgIIBmzZpx4cIFjhw5wrZt2+jbty8Gg4EaNWrQtm3bLPnH\nxMTQqlUrW165XdOVK1e4fPkyrVtr88QPHDiQbdu22fY/88wzADRu3BiTyQRAixYt+M9//sP06dM5\nceIEZcqUyfW6izK6wC8BHNsWRJcvuvPTdiPReypyY9sbfBJ6jDJch6D5pF2ox2HTSG5MqA6hYxl2\n3EQ8Xuwy+7Pt6tOUeqU+kL2wsqorClp9UZTJry0lL1y8eJGoqCiGDh2K0WhkxowZREREIPmYxMje\nlc/eh/tf//oXo0ePZv/+/SxYsCBP/t2Z3QKt6+7u7rZtIsKcOXPYu3cve/fuJT4+ng4dOuS5vAVJ\n6dKlATAYDDb7wrPPPsu6desoU6YMnTp1IioqqlDKdq/QBX4R4HZ6z52WduL9X96n6sud2LX5awxL\nFxLtn0DSc0+T8sQ7kFSZpKpnIN0AtX8l/ZVa4HoFgDcf6EMvIrgxKpAzQet56EhdwsMtwsoYDcHh\nGc5lMOiuoJnJjy0lr6xatYoBAwZw4sQJTCYTJ0+exMvLi+3bt9O+fXsWLFhgE2QXL14EoFy5cly7\nds2WR7Vq1Th06BDp6emsXr3atv3KlSvUrFkTgK+++ipP5Vm7di03btzgwoULbNmyhaCgoCxpQkND\n+eSTT0hNTQXg8OHDXL9+nVatWrFy5UrMZjNnz54lOjo6y7HNmzdn27ZtxMfH53hNVipUqEClSpVs\n+vnFixfbevvZcfz4cR5++GFeeuklunbtym+//Zanay+q6AOv7nPyO5Cq07vhxG+owwXx44cOr9Dw\nLy92tfwvhnrbMTsLKEDdACczpJUGQwpcfxDK/QVmJ1wOt+Nq0Apo+BO4n0eZnbmyfzBN1ocw/eFD\nDOhxg9TIW4LCzS2rsLdSkl1Brfdm0iStHjw8NGF/J4Pfli9fzoQJEzJs69GjB8uXL2fOnDkcPnwY\nPz8/XFxcGDZsGKNHj2b48OE8+eST1KhRg+joaKZNm0bnzp2pWrUqTZo0ITExEdCMsGFhYVSqVIm2\nbdvahGxO+Pn5ERISwj///MPkyZOpUaMGhw8fzpBm6NChmEwmGjVqhIhQtWpV1qxZQ/fu3YmKiqJh\nw4Z4eHjQokWLLPlXrVqVTz/9lGeeeYb09HQefPBBNm3aRJcuXejZsydr165lzpw5GY756quvGDly\nJElJSTz88MMsXLgwx2uIiIhg8eLFuLi48NBDDzFx4sRcr7soUyBz2iqlvgQ6A+dExMfBfgV8BHQC\nkoBBIrI7pzybNGki+gQoWo/+hIPpDDw9Nf18Zt5/fSavqImw+3m4+CiEjgVxApWeMaECtr4Oj26A\nmrupmOjCZfc0uFkGxACu1yCxKqxaTqmwZ+gdV4nlwQkYdg9k+YZLzGU0MW7t6Db9fb7du5kbX2zI\ncxmLKocOHaJBgwaFXYz7gqlTp1K2bFnGjh1b2EUp0Thqk0qpXSLSxFH6glLpLAKezGF/R6COZRkO\nfFJA5y325Nf4N2bJXJ7eXRWC5sOj/4V0Z3CyCHsFJD4Eqe4YbpaG4GlQYzfqtD/XSwkusYOhVJIm\n7G+UA/fzuDVcRNoVI4tbnyDN1JaUoEVsan6AOOM1fLtWYdk/Y3ngUrss6guloFOnAqsGHR2dAqBA\nVDoisk0pZcwhSVfga4vLUIxSqqJSqrqInC2I8xdnHngALlzIut3DA8LDIejKZkKWDrXpDaJPeBF8\nIoj9D3xD/KObQdAWQF2tipT7C2NsKOcuPkZyhyk4pzlRddMrnKEW9G+vJRRAFG5/hJAUtETblloK\nasVB7Eg+CV2AIf0Z/ucEbJxJ14SH2eMH//ufZrQF7ferryA4WI/hUxyZOnVqYRdB5za4V0bbmsBJ\nu/VTlm0ZUEoNV0rFKaXizp8/f4+Kdv+ydCk4sE3h4qLpg49dG0a3ZdeIPuEFIkSf8KKb8SVWd/6B\neI+/NMFtdaT4owtS7jzGhIcwBf1IWa91yNdROH2zmVavnCUqCnBKx/Vcbcp/9S3u6jppj1oMader\n0n7pGE2KN/oCRDAbgITHGRXjypeNvuWS1yNUemz8rUIao0kKDC/RhlsdnfuN+8pLR0Q+FZEmItKk\nalWHk66XKCZNgps3s24vX17rNffZtB4V1ptuxpd4gzfpZnyJm336EOOdANcrA6BSLLqWazVh40xc\nU1zx2TiAc+pBPCWEL14PYfno8cSeiWVirZ8wf55AE1M5hvyvLDddwHDWl1kz++FlMlLq1xHgnKq9\nRMxO4LGDzzvtRvyW82fd41w3V6AtmzVPnrBecDqoRBtudXTuN+6Vl85poLbdei3LNp0cyCAsg8Ph\ndBCYQrB4p8G5v+l9CJaE9eftuFdwa9IfvwP1iU9/hPNBa+iysSVrY7bTtVMN1gfNp+zGKZi+jefz\nz7OqWcYHj4dgaOgGr8wxENXEhdIxk3FuPBu8f2XF9VY4NX9B+2pILY3WdG6SGvQlpLnitPFdXFu+\nzmNlUvmtCTwa+RYxphA8PO9JVeno6OSBe9XDXwc8pzSaA1d0/X3u2AbpPNuJ8hX2Uj7sSX4yKsyi\n6Nrdi3b9DJw58H9I3Eho/TYSN5Jp3z1Ak+o/8Ni1mXz3v204IazfcBo2ziKpzq8Ohb09NR6Lxtyj\nF1GjIrjxw1usf341k8IOYh44gAf/qQYbZ/L0shGaO6fB0tu/WpP0mHGUO/IY77QGz7gniTG9Tht+\nwoG3nY6OTiFRIAJfKbUc+AWop5Q6pZR6Xik1Uik10pJkA3AcOAp8BrxQEOct6uQ2oOrddzV9fbXj\nDbkatILkg/15JsyF+v1rs87fhOx+nmhCKNVkDpO3Qqkmc+hmfIlx1dZz+psxZPC4jRlD+uINGXTq\njs4feyaWiJ4RhHhpwbZCvEJ4rlFv+jXuR6Vfz9A55hG20AbndNF6+2mloeIJeMmTU/5bKb23J7ua\nxFGt0zNsHzqe8icfAaOR6EmbCc84XkvnNjh16hRdu3alTp06PPLII/z73//mpiO9H3DmzBl69uyZ\na56dOnXi8uXLt1Ue+8BldxP7gGq5lXfNmjUZwiu/8cYbbN68+a6XsUiQXZCdwl6Ke/C0vAQimz5d\nZMKjkVKeS+LS/D/CFCW8XEuYitC/nTQzviMVxpeSqGbVRJSSqGbVpMKUCjL8P1GilOPgXUrl/fyZ\n8fQUwRglzq+6CRPKS+nm74j7BIOoyQhTECYbBGOUGDo9L0xBnCc5SZQRiaKNVOGcRE3cdFfr9G6T\nn+BpIpJzFLvbID09XYKCguTLL78UEZG0tDQZMmSIjB07NktaaxC1u03mQGb5IT9ltAZUywsDBw60\nBWIr7ujB04oIuc1M1endcFYfbs+H9dfRzziUMjGj4JInVDwFKe44e2zH3+d1VkfeJCTmL0hPJyTm\nL1YPXM0jrWJzjeVyOzNjJSQANWNJO9CfGivnsjwmlv/71YwYwOV6OW307oAOmIO+ALMLhn39iDa9\nSS8ieM3Ynth9YbdVV0WSuxAfOSoqCldXVwYPHgxoMWE++OADvvzyS5KSkli0aBFPP/00bdu2tYUS\n9vHRxkEmJSXRq1cvGjZsSPfu3WnWrJmtx2w0Gvnnn38wmUw0aNCAYcOG4e3tTYcOHUhOTgbgs88+\nIygoCH9/f3r06JFrjP1BgwYxcuRImjRpQt26dfnuu+8AspTx+vXrDBkyhKZNmxIYGMjatWsBSE5O\npk+fPjRo0IDu3bvbymFfXoCvv/4aPz8//P39GTBgAD///DPr1q1j3LhxBAQEcOzYsQyTq/z0008E\nBgbi6+vLkCFDSElJseU5ZcoUGjVqhK+vr21yma1btxIQEEBAQACBgYEOQzoUKbJ7ExT2Utx7+Ln1\nwGdNmiFMKC+lJjoLE8qLy+BArRf9hhKmIIbeHaXCOBetd+8ARz14EKlcWduX2/kdkSXkrzFKGFdF\n6oSECeOqSLmRHtrXx1SEiWWklnGRgMgA4yCpMg6J8qLgK/Iekq8e/l2Ij/zRRx/Jyy+/nGV7QECA\n7Nu3TxYuXCg1a9a0hRGOj48Xb29vERGZMWOGDB8+XERE9u/fLwaDwdZjtoZQjo+PF4PBIHv27BER\nkbCwMFm8eLGIiPzzzz+2802aNElmz54tItn38AcOHCihoaFiNpvl8OHDUrNmTUlOTs5Sxtdee812\njkuXLkmdOnUkMTFRZs2aJYMHDxYRkX379jks74EDB6ROnTq28M/WPDP38K3rycnJUqtWLfnzzz9F\nRGTAgAHywQcf2PK0XtPHH38szz//vIiIdO7cWXbs2CEiIteuXbtnX055Re/hFxFsPfDgcM2N0W57\ndHw0aXvfZdZKL26a3aDUdVI99mghD1LKYYh9HnP9/5J0cAAr2ndxmH+/fvDpp1C5csbtFy5oHc3s\nIs3mFM0xQ0Awq+vlqghO/y+CGgdbcK1agjaoV0CpNE71fgnfkG4sCVvPa5H+hKSXIJedQpprsn37\n9g7DCO/YscMWVtk6SYojvLy8CAgIADKGET5w4AAtW7bE19eXpUuX5il8cq9evXBycqJOnTo8/PDD\ntl6zfRl//PFHpk2bRkBAAG3atOHGjRskJCSwbds2+vfvD2gxexyVNyoqirCwMKpUqQLkHj75zz//\nxMvLi7p16wJ5C58cHBzMmDFjmD17NpcvX7aFny6q6AK/kLAJz9NBmuA0RuPmBv0mRdNrVS+Cfr/M\nGNM+vH7tDAaz5g3jZKb9rw1w3zATl43vUrriYa4e/Sxbw2+/fmA3wZEN69d4XqI52ht2J02CgQO1\nGDnUjMXpmwiIDyHZ/33OBH2HSnOh1253Rm58BHFOw6lUIvtbr6V/XFneM21ixMObmN11M6ecjaQr\nJ045G5ndtZgac+9CfOSGDRuya9euDNuuXr1KQkICjz76KJAxNPHtYA0hDBnDCA8aNIi5c+eyf/9+\npkyZUqDhk7/55htb+OSEhIRCi1fkKHzyq6++yueff05ycjLBwcF5mkf4fkYX+IWEtQfu+lcIj0e+\nSJWwtoxppvj0SFvCtr7IjLM/8YKxB/FNv0OllQIB5zQDm5oeIsw4Co+YnqSv2c66dY7VxFZB7Sjw\nGsDFi9r5PT21uDeentq6vcumIzX0V19pMXLc9own/bjmySNem+HwU8jSjaz4LpH5MUepEfsUinSe\nOA4/NDvJa5O+4uxf8NY6f46YvXBCOGL24q11/rjuLIYeFHchPvITTzxBUlKSbcJus9nMK6+8wqBB\ng3DLfK5MBAcHExERAcDvv//O/v3783Xua9euUb16dVJTU1maRztEZGQk6enpHDt2jOPHj1OvXr0s\naUJDQ5kzZw6aJgL27NkDQKtWrVi2bBmgfV04Clvctm1bIiMjuWCJPZJb+OR69ephMpk4evQokLfw\nyceOHcPX15cJEyYQFBSkC3yd26dfP3jimQHsoCWBcc14pzUExjXjk4MtOfrUdD7pvYlShiTKpd5k\n1MZHSEt1x2BI4YveGzhV/xRlyjg2vP7737cEdXZ4eGSdGatfv4w9+oEDHef/6aeZti/bAMvXg0l7\nAWCM5ox3DFV+jGLzV0LE8H1owNAAACAASURBVM28V2E65R5txERje3oRwRtoxtyJxvb8fbEYGnOt\nb/Sc3qj5RCnF6tWriYyMpE6dOtStWxdXV1f+85//5HrsCy+8wPnz52nYsCGvv/463t7eVKhQIc/n\nfvvtt2nWrBnBwcHUr18/T8d4eHjQtGlTOnbsyPz583F1dc2SZvLkyaSmpuLn54e3tzeTJ08GYNSo\nUSQmJtKgQQPeeOMNGjdunOVYb29vJk2aROvWrfH392fMmDGANhPYjBkzCAwM5NixY7b0rq6uLFy4\nkLCwMHx9fXFycmLkyJFZ8rXnww8/tKnAXFxc6NixY56u/b4lO+V+YS/F3WjbsaPIrMciJcqIuE4o\nJbxaQSqGjBZerSCuE0pJUGd/qdG5k/Tp7C6bjUg8nhJonCF0Hi5l+wyXPnOmZ2t4zW3Jzv0yO0Nv\nvpfg6VK6flSGc0Qdj5JhnZEq45ABxkEZjLmbjUXDmJtvt8z7iLS0NElOThYRkaNHj4rRaJSUlJS7\ndr6S5BpZmOTXaFu0LRBFlE7vhrPL+Dk/7BnBKHpQWn3DDScDl703gksipVPNTDuwjydMe1kBrLAe\naNKWRGDTpuwjaeaEp6emkpk0CQYMyDgxhyNXTUcYDGA253COU+MdnCOEN37wpP6BiowNW0/LuDYs\nbhHLqKhHqHcyzXZsdHw0sWditVAPOgVGUlISISEhpKamIiLMmzePUqVKFXaxdO412b0JCnspzj38\nWZNmCBPdNDfLvk9J2eaThckutgFVo4w95L1glWtP2mAQKVUqa++9cmXH6a1jf7IbcJWXLwY3N5FR\no7Lfr1T25xgXuEkqc04GhHgKU5E63ZqJGldZptd9zDZwzH1SBXH3iSqosUoFSlHu4esUT3S3zPuc\n8HAInLODWcvqQKob1P2exPbvglMq7Y+Ce/WdfMIoLv8dkcXmlxmzGcqVy6om/uij7O2FOQ24ys2B\nxJr/vHlZ3T2teHhkf47P4tvR99mv+L7JSV7fCn/X2UOp7f/HlK6/8UYboXurixiWLqTZATNSMGOV\ndHR07NAF/j3m2LVhdHtgIIGmSnj90k1ztzSkQ2I1nlzyMusikykV1p1VnpVtNr+cuHgxq+E1J3th\nTu7h776rpXeEdbpCq80xp5dKdue4XCmaFQHTWTVqM29HCWu2VcK15ZukHu3M263h+t/NmcKbbORJ\nLcwykPRgNKOXF0e/TR2de48u8O8x1hj27Zs/SXyLtdhmpCr7N680r8VcUyQ3I9dy1inW5kWTk9DP\nrlfuyAMnp/RWr52RI7MKfUfehJlfKpUrQ5kyms7eKZtWVbFhpsBsv57jje2p0OBb2DoZVSOON3vv\nY7bRh334E2icCWG9uPx7UPYVoKOjk2d0gX+PCfn1HM9u98Ac+iq4XMcl1YVRGx/R1DuhY1ndPAFM\nISRtumW0tEbNzEypUvl3687NPXzePFi8OG/ehNaXyuLFkJysGZBFHBt03dxgbt/xNmEPEN30Qd5q\n6YLzb71pH59K6orvSFLuvNI7nkbdffnz2XEERk6g4qWQrBnq6OjkG13g3wPsfdtPKg92GfzgWnVQ\n4PzLvwiLqc2bywLgcGd4OOsgpH79YOHCjHrzypXhyy/z79adF/fw7L4OsiM77x6DIeeXxor2XZDI\nlfznwG/sCZtGY+JIW7GeUpdqscn/b4IPVeOkaSBfXu7GMxU207dv/q61OFLWwdDp+fPn2wZjLVq0\niDNnztzrYmVAD2V8H5OdNbewl+LipZPZY6Utm6S88VupMM5FJocgFca5SHnjt9KWTVmCnBUVbicQ\nm4gW/jlq4iYxOxlkltFf1LjKUr3bE1oY6O7PihpXWWYZ/SWKNuLONanC+UL12smPl8706SJRURm3\nRUVp2+8Ed3f3HPfnJ4xwftBDGd+f5NdLp9AFe3ZLcRH4rs93FJrPyhBh0mmiq/g/7yJmlCwzVhOX\ncRW0yJOWNC4u95c7Ym7caWDI7uU1d8323aoJU5EHunUUgqeLah4ubuPcxNX4vSjM8ryxj1R86g4l\n5h2QH4EfFSVSpcotoZ95/XZxJPCtESsjIyPF3d1d6tatK/7+/pKUlCRxcXHSqlUradSokXTo0EHO\nnDmT5fiBAwfKiBEjpHHjxlKnTh1Zv369iIgsXLhQunTpIiEhIdKqVStJTEyUwYMHS1BQkAQEBMia\nNWtERCQpKUl69+4t9evXl27duknTpk2zRLYUEfnqq6/E19dX/Pz8pH///rJz506pVKmSGI1G8ff3\nl6NHj2Z4AWzevFkCAgLEx8dHBg8eLDdu3LDl+cYbb0hgYKD4+PjIoUOHRERky5Yt4u/vL/7+/hIQ\nECBXr169s8ouAugCv5DJPOcFzWcJU5TMaK4kHaRTX8tkIZleAgRPt60XJWEvcnuTqdgDIoHGGeI+\nEU3oT3YSl8GNhHFVtHoKnibevb3EaTKC8Q4l5h2QXz98q5CfPLlghL1IzgJfJGPv+ubNm9KiRQs5\nd+6ciIisWLHCFnLYHj2UcdFF98MvRBwFG+se4wEbZzAuVGg9GDbUBTbO1LZbMYXATs1I6+l5R+FW\nCoU7DRtTrVk0e8KmU3fZDHav2c9jhyuR6rEb/qkDLd+Dhis5WD8ew+FQul+8ZRHObYrIwiYkBEaN\ngrff1n5D7rHt+c8//+TAgQO0b9+egIAA3nnnHU6dOuUwrR7KuGSg10gB4sh4GUEfZsf8i1caPM52\nzx1w4nFmxZziJSbgQlqGtHcYTLFQsfr/3w4h/WNZ/XEEe0whtGUzk1b60r53GdLr/wAp5aHmHjjd\niGkrGzCw7FCMRhMnTmgvFxEtjxMnNJfQnTs1T6P7geho+OQTmDxZ+w0JubdCX0Tw9vbml19+yTVt\nfkIZO4p6ea/JLpTxU089xYYNGwgODmbjxo15DvRWUtB7+AWIowFHBszQ/EPw2AknWmq/zT/EGTNL\nlhRoMMUiy/LR4/ni9RA8PSFateMlPkJWfkfZ62XA9SrcKA8VE4g0VqFSYoItCqhV2FsRgfnz74+e\nfnQ09OoFERHw1lvab69e2va7iX1o4Hr16nH+/HmbwE9NTc124hI9lHHJQBf4BYijQU2zmiteCQU2\nzqDlwndg4wxeCYVZLRSTJmkvCfsAZiUVe1fQg/hRr3ddEt2TIbEKlL6KU0IQcWEzWWl8MMd8RLSw\nzoUt9GNjNSFv7dGHhGjrsbF3lm9SUhK1atWyLe+//36G/da5ZAMCAjCbzaxatYoJEybg7+9PQEAA\nP//8s8N89VDGJYTslPuFvRRFo60j4+VDzwYKzWfKLF4WAZnFy0Lzmdr22zRyFnfK9h4lTEFceodK\nFG1kVCds648ED81TiOa7UZ/FNXia7hpZdNGNtoWII+NltYO7meXkyRjDHADGGObQZ5cnfy3bneFY\nawCzkoYjw2s5vyjUrlG0XDkWL+KZu0HRP64spR48yLGdn+Up35Janzo6OaEbbQsYR8bLpUt7Yjyd\npqlvamU/E9Vdnt/6vsPq1WQ1dFujY3766R/gCZP2w8MJJk3l1RfSvoNLhs3MNw/FgwQS8OB5PieK\ndg7zL2n1ebssWrSosIugc4/QBf5dxpFQs/cusecO5rcukuQUqtlRSIcav2+ml9mfeLwwcoJ4vPgN\nf55gMz85EPolrT51dHJDF/gFRHg4HPvvEfrsGkfI1bUARJfvynspM0hKqZMhrUhWoV+UXTJvl5xC\nNTsiZOlQIvCic3BL6le6xtEDL7HG1IugyvFUqbqQlEdXwKVHYOf4ElmfOjq5oevwC4igK5tZGV2V\nblcXEU0bomlD96uLOJlS1Rbb3R4R3SUzp1DNDklIIIQt9Dh9kt3eR0jp0xeMW4gtfwKnvt1Qfivh\nTFCJrU8dndzQe/gFQPjOcI4df5uWncsRfWABT5m+I924lXSfl+nAj/hcSiZq56UMx1gnFCnJvPtu\nRnUX5PylE96pAs4HPfnWNIf2Kx9iU+95hPa/iSiBG2Zk+Xo8JaTEu7jq6GSH3sMvAI5tC2KpZzLR\nfmdJ6x1GcvMFpPTui9lvJdu9z9L+dMbwsLq6QSO/IRmcAyYxNuwUzxkHsdM0ibqH65HqLKQZwCVm\nBN1NF4r9tIhFITyyI/bu3cuGDRts6+vWrWPatGl3nK99KOa7ibXez5w5Q8+ePXNM++GHH5Jk14vJ\nLUT0vUQX+AVAn0QzhpWRmM2lSHEBQl8BlxTSzaWYstKfZtc8S7z6JjvyE3s/rfxYZpZ6lciwb2jQ\nLYjDfrGoNBe4WQZpNpefjAbasvm+cMkM3xlOdHzGYbXR8dGE7yz46RpHjhzJc889BxQdgf/000/z\n6quvFmKJsIVkyA81atRg1apVOabJLPA3bNhAxYoV832uu4Eu8HMgr8G5QpYOZYppGzd+HQ/OKdo8\ntc4plP51BG+athA74vN8TSii45jx42HMO2PpeBR2BfwB6Qbk0iM8tLcDN8QVc58wrjdfAH27cCK4\nU6GWNahGEL1W9bIJ/ej4aHqt6kVQjYKfrnHq1KnMnDmTVatWERcXR79+/QgICCA5OZldu3bRunVr\nGjduTGhoKGfPns1yfHx8PC1atMDX15fXX3/d1pvdsmULnTt3tqUbPXq0zYXzrbfeIigoCB8fH4YP\nH24Lt9CmTRsmTJhA06ZNqVu3Ltu3b+fmzZu88cYbrFy5koCAAFauXMmiRYsYPXo0AAEBAbalTJky\nbN26levXrzNkyBCaNm1KYGAga9dqjhDJycn06dOHBg0a0L17d5KTkx3WidFoZPz48fj6+tK0aVNb\niAbrSORmzZoxfvx4jh07xpNPPknjxo1p2bKlLRxD5jqxYjKZ8PHxAcBsNjN27Fjb6N45c+Ywe/Zs\nzpw5Q0hICCGWYdZGo5F//vkHgPfffx8fHx98fHz48MMPbXk2aNCAYcOG4e3tTYcOHWzXNXv2bBo2\nbIifnx99+vTJV7twSHYjsgp7KeyRtvkK+auUDDf2FqcJ7sLrpYUpSOnXEfcJBunc/es7nvRC5xZR\nx6PEfZKSOsPLChPdhImlhSlK6nZqKx59vYXXXYQpiGvIrGzzyBzCOq8jcvMdHvl4lFQJryKToyZL\nlfAqEnX8zuMj343wyF26dJGvvvpKRETmzp1rO0d0dLQ89dRTtnQvvviiLFy4UERuhTcWEenfv7+s\nW7fOdv4xY8aIiMj3338vTzzxhIhosfVffPFF2zGZ10VE1q1bJ48//rjcvHnztkIx2+Pp6SnvvPOO\niGhx+K3XMXDgQHnqqackLS1NRETatm0rhw8fFhGRmJgYCQkJybFO4uPjxdvbW0RE5s2bJz169LCF\nYbbWif0cAPbrcXFx4uPjI4mJiXLt2jVp2LCh7N69W+Lj48VgMMiePXtERCQsLMx27dWrV7fNA3Dp\n0qUs16mPtC0gcvIRz0x00wdZ2nsV6QahdCq4bXwT51QDYkjnp4DRBIXd5YhZJQRrL/ktwrn06XFq\n/11O05OlluJwUBQJj/4JhlTU+Tp0+KuLwzwchbC+Wzr/EK8QRjUZxdvb3mZUk1EZ5vO9F+Q1PPLO\nnTvpa5k/csCAAXnKOzo6mmbNmuHr60tUVFSGoGyOQhfnxpEjRxg3bhwRERG4uLjcUShmK9Zr6tu3\nb4aIoWFhYRgMBhITE/n5558JCwsjICCAESNG2L6A8lInmzdvZsSIEbYwzLmFfN6xYwfdu3fH3d2d\nsmXL8swzz7B9+3YAvLy8CAgIADLWm5+fH/369WPJkiUFEu65QAS+UupJpdSfSqmjSqksijml1CCl\n1Hml1F7LMrQgzns3yY+P+Id+XUg7+Cydf6vADytT+C4mGsPKSCr+1hHDoT7EnrnDiFklEEfqtNgz\nsUT0jCCt/Fhee2wrFw6+AM434GptTY1m0HSypXcP5OXT4xzmm58X+Z0SHR/NJ3GfMLnVZD6J+ySL\nTv9uI5bwyHv37mXv3r3s37+fH3/80WHazOGRAZydnUlPT7et37hxw/b7wgsvsGrVKvbv38+wYcNs\n+8Bx6OKcSExMpFevXnz22WdUr17dVvZvvvnGVvaEhAQaNGiQ94vPdE32/60hn9PT06lYsaLtHHv3\n7uXQoUMOj7nbWOsMMtbb999/z4svvsju3bsJCgq6LbuDPXcs8JVSBuBjoCPQEOirlGroIOlKEQmw\nLJ/f6XnvNvnxEd8W+RnB3z3HnO9K0dqk8CKeJqZynPnue66vXMD44PF3t7DFjOx64TVN4wnxCmH8\neEjr2pP1MdF4xz4OlY+CoC3AYBbC1SuEO7CP5new1+1i/RqJ6BnBWyFvEdEzIoNO/25xO+GRg4OD\nWbFiBQBL7T51PD09+f3330lJSeHy5cv89NNPwC3BX6VKFRITE3M1YmYuV2aGDBnC4MGDadmypW3b\nnYRitrJy5Urbb4sWLbLsL1++PF5eXkRGRgLaS2bfvn1A9nViT/v27VmwYIFNCOcW8rlly5asWbOG\npKQkrl+/zurVqzNcc2bS09M5efIkISEhTJ8+nStXrpCYmJht+rxQED38psBRETkuIjeBFUDXAsi3\nUHn3Xc190h6nluF0CZ6Zoev5/uszuewdThTt8MKEgXS8MNniu+jD+/NPXnrh48fDZK+2HPS3BKFT\nwN++cNONT0KP0bl5CM5rV2UR+vke7HWbWL9GrGqcEK8QInpG3PHX3t0Ij/zRRx/x8ccf4+vry+nT\np23ba9euTa9evfDx8aFXr14EBgYCULFiRYYNG4aPjw+hoaEEBeVuiA4JCeH333+3GW2tnDhxglWr\nVvHll1/aDLdxcXF3FIrZyqVLl/Dz8+Ojjz7igw8+cJhm6dKlfPHFF/j7++Pt7W0zDmdXJ/YMHToU\nDw8P/Pz88Pf3t72Ihg8fzpNPPmkz2lpp1KgRgwYNomnTpjRr1oyhQ4fa6tQRZrOZ/v374+vrS2Bg\nIC+99NKde/tkp9zP6wL0BD63Wx8AzM2UZhBwFvgNWAXUziav4UAcEOfh4ZHFGHGvyWzcG/3sDFHj\nKssso78W6tjoL2pcZXnce4bDEL1K6SGPbwelHIc8VipjOoKni+u/q2hzBL9STfvtNELo+5Q8ONRT\nSnceIMNfH5rhmDuZf7e4hkfOjCPDcFEjs+G0uHK/Gm3XA0YR8QM2AV85SiQin4pIExFpUrVq1XtU\ntOzJ7CP+3sa5zIysxdiwU7QKac3YsFPMjKzFmhNzs3wNKAUjR+oumLdDXnvhlco7c6PiBVxih+Du\n8g9OaS4QtACu1OJclUs4ey+jz6b1GY650/l3dXSKMgUh8E8Dte3Wa1m22RCRCyKSYln9HMj+O+w+\nxu1CAmNM+3g8zoftrbfyeJwPY0z7qJSYkEWILF58/8ytWtRwpE7LPDp56VJIrLYZNs5k5YYLvLXC\nBzGXgXQDNPkMJzGzfqWZkF/PZck/P4O9SiJ3qie+HzCZTLYJ03VuURACPxaoo5TyUkqVAvoA6+wT\nKKWq260+DRyiCBAeDtGTNtt09uk48YKxB9ubHKDl1tbsaHKA943+JOChC5ECJLdeuNWom7poA8SM\nYS6jec+0iXb/awAGMzilk/7r/7HH9HKBK+fFUVxrHZ1C4Hba4h0LfBFJA0YDG9EEeYSIHFRKvaWU\netqS7CWl1EGl1D7gJTSd/n1N1Zc78fEfQXRbdo3oE14gQp1O7fhkwFqaHarOtuitNvXOAO/RhV3c\nYkdOL9DMRt0o2lHf+DGbmx2i9E0nnMxOOLeYzkRje+3eWYzrnd7NaMHN60hqK66urly4cEEX+jqF\njohw4cIFh3MP54S6XxtvkyZN5F4ERcqOrn0Hsq7eYgw3nXFfvpKHHvovh0M/RaW68OMyM21NQgIe\nhD08mtQn09n7se56ea9wcso0gYwxGvp0xyBpLF5Zlm0PlWV+6HFcUg0MXtaDevzB2LBTvHj6Vdbv\nHEtCAjzwAFy7Bjdv3srGzS1nfX5qaiqnTp3K4HOuo1NYuLq6UqtWLVxcXDJsV0rtEpEmjo7RBX52\nODvTNegx1oXugHQncDLjdNOVzctv0NqkcFbp2tR7eijee47RmGmayOBwqHQMDvQBUwjxGAlv7swn\nT5zCeLIGJ6pdZWZkLXqeuIynmHLMWw9brVPUyUng6/Hws8NsZm3MdtwbNCLJU/P1Lh3zAph24+QZ\nT7qpcItXkskSR39nxq8rDxKYFyMcKNOa7a230nJra8aYtpJO7iMn9XlwdYozeiwdstHlGgwEN+9O\nksceMBtA4GbzOXQzvkR0v/t+oHCxxt6oC5ph155TyoP3jf7scGBczw19oJxOcabEC/zshvG37diP\nn0PXYLjpTNRiM09vbIm5VBrXn+3LirKGwi52icdq1BXRXGDtPXpm9B1tGyNhb1x/xpizcV2fmEan\nuFPiBX52w/i3VjpP/fOBbFppJsQEa2N/5uk/B1A2xZ9HWunB0O4nMnv0HGuYzsxSrzJGLoNSjJHL\ntEhqxl6/IxmOMzwajVv7cH0Alk6JocQbbbN4fFhQShMgOsWDEe9Fs/R6d75c7krP4+dY9fCDDOl7\ng37uq1nw2r0NW6yjczfJyWhb4nv4pduGE2icSTxGzDgRj5FA40xKty34qeh0Co8+iWYMSxcyvMdF\npoYIw3tcxLB0IeenmYvt/Lc6Opkp8QJ/aDUn9oZN41tjRZwQvjVWZG/YNIZWK/FVU6wIWTqUNabZ\n3Iz7F2+3hptx/2KNaTbvXx1arCc919Gxp8RLtTk7HQdEm7Nzbr5HYurcP2S+d3IiAYxbSGvyBWyd\njGoyH4xb8OQEzZM2M2CAfn91ij8lXodvVeK3Crnls70teiuCoqxbegaDbm4jMXXuD6yeV/b3brnX\nQ4zseZGrF/1RB8OQ5rNxdj/DtKV+THwoDKeHf+RmojcPNovizMQ/Cq/wOjp3iK7DzwkPxz7bpw0e\n92wqPJ2Cwdqr798/q+fV6zW6IJEr6XKwDOmhr8KVmqQ6C688fZGU0MnULr0fc+AnXPutbaGUXUfn\nXlDiBf77/R37bD9d27HPtj4S8/7EfjyFI47t/IwgUznWxGxn1MaHEY9YuPYQVDpJ2RtOHPa4BLGj\nuB6hx7TWKb6UeIG/uUxWn+2ZpV7l0COOfTL1kZj3J47GU2TmqGc7FBAWUxsSgqH8X2AuRWIZM7Wv\nQOcNHTAY8h9FU0enqFDsdfhLl2rCICGBfAU7c6QH1nX49y/ZjaewYr13NV7oRseGQaSEToZr1aDc\nX3CjArhegdiR9N4Synq66fddp8hSInX4S5dClSqaPjdz2IS89Nj0qfCKFjl9ednfu4lPNyAldDLe\nCeU0YZ/QAlyvar9B86nUprtuu9EpthRLgW/tnV+4kGlHcDj1HpxJ64FG2/e6o4kxrOizWBUdspsW\nccmSjPfuUt19jDIPJ83tKqNiofbC5bBxJs4p7oyKhWgvx/nrthud4kCxFPjZ6XMDT2uDrCJqVwQR\n3lcVGXtzGu2Si2U1lCjy+kX2x+QNzHt7Pn98DGyYy0k8qB3Tg7RlP8KGuRz62HH+uu1Gp1ggIvfl\n0rhxY7ldlBLRlDgZl3g8ZZbRX9S4ytIypLWocZVlltFfxNMz27yWLNF2K6X9Llly28XSuY8YVWO1\nQLqMYq4IyCjmCqRLZ1ZnaT9ubvp91yk6AHGSjVwtll3b7HpjHiQwxrSPx+N82N56K4/H+TDGtC/b\n7/XsQifrXhtFn6hy3RhVYy3zGE2n4FY8YvyMYXzKEepzXLR4SgSH67YbnWJFsRT4jvS5AKedHA+y\nyu4NkV3oZN2AV/T54w+Yd7obiHDtchfGhZ2ivvETeLEB/XonsjdsGoGntcfjvX0v4PJ/9XU3TZ0i\nT7Gc4tDaG8vsjhl5aDRjb05jZmQtxpi28n68P2PDTkGpVxnjIJ/sDHW6Aa94sfjgXL69XotXwk7x\nwGUjf9Y38dgfldlqehXPTls56PYdxI4Cu6880Hv9OkWPYtnDB8ceNtkNstpcJn+DrHQDXvHAOsDK\nqup76O9qXKxp4oEzRn6uf4GaLztzJug7yp+sDxtujcDVv/J0iirFfuDVnaAPviq+2N/beIx8a6zI\n2D7xiFMqpJeidFo6KWWvodKh49f/xwbT+xmO1yfI0blfKZEDrwCqvtyJrn0HgrOz9oQ6O9O170Cq\nvtwpT8frg6+KL/b2mWeMlnhKK7x4etkwcE4ixf0aCIiCDQ2T6cwaLbExGoLD9a88nSJJsRX44TvD\nqXvNzLp6i+ka9BgAwaENWVfvax77u2qe89EHXxVP7O0we2qmExD5Ki+ZDkDDVWBIBQWcDYR0Zwia\nT61O3TVhH9YLzM60GKvPiKZT9Ci2Aj+oRhCHK//IY7E+rAvdgfvIh/k5aD+PxfqyOkJ3syjpZOih\n7xzPHtNYQvkv6+uCS5qi/NaXoMJJ2PU8AEt90YT99tdQrd9j+FNBhVJuHZ07odgK/NjIEF6N9Oew\n935KXalK0kPHKf13HbpueBKVbi7s4ukUMo5cd6OMBsq7nKfj0n9xNfpD3CO/AO9vKH8siGtlIOif\na7i0fIs25yMI8dInPtcpehRbgX/sv0d427QF17P1uFnxHM6XHyKl2lE+bp6MGUNhF0+nkHFkn3Gr\nE0uDyMmsM33IKOax3vQBhu1juerxO9VPPEKsZwrqaHucvjTr/vg6RZJiKfA7vRvOvnJhXO3dn1OP\n/Emlo4GklU6Cc/VICJ1LYPN++iAanSz2mU8Hjud/JybTmbV8zGjEuAWXlu/gFPUGZ6ufxfloK1L9\nVlG7+cucOAHPvxNN37m6Ll+n6FAsBX67ZCf+53MEqf89/NGZSztmgcEMD/5Bwz+8OPDweT1Ugk4W\n+vWDxYthv2c3DEqIrQmDt1cjvWU4RL1NqepxdImtwaK28TzafCQpXXoRvUTX5esUHYqdH/7SpdB6\noJFRHW/ynd8VQAAFZmdGba2Kp+EYr+7MeM2enloPT0cnA0YjI2oZWHr6Q9JNbUk3buVm2HO0O+LM\njobnSF72E+pEiO6PHFWRkwAAIABJREFUr3NfUWL88K2DaWqYExjzXT0Mv/wbSiVDqSRK/TqCZTFx\n1N3ZNctxeqgEHUdE9/ucb3fGsN70PmOZRYqpE4a4oWwK+JtXfhEwhej++DpFimIl8K2DaRLwYKKx\nHeZm8zDcLAU3y+DU9CNSjdsZx4wsx+kPrY4jYiu0I6L8MABmMYb2xtcwN/mcRls7Mb8JNHl4JtWr\n6/Pf6hQdCkTgK6WeVEr9qZQ6qpR61cH+0kqplZb9/1NKGQvivJk5EdwJn+YDWWV8gJg+4bhKMs32\n1sP9nBcpypWbvXtzzHgqwzFubpqLno5OZsaPB0aPphcRDDQOYnPYZ4yMbENC9CLCInuwq8c0Uv6a\nqYfP1iky3LHAV0oZgI+BjkBDoK9SqmGmZM8Dl0TkUeADYPqdntcRfvFVORC6mBktDHQ+4E6j/fX4\nOegArQ5UI2DFZNIODoCasXqoBJ08E1uhHRET92Gs9S0zI2sRafqEjvzA16ZFzIysRYeaGXsLemA1\nnfuZOzbaKqVaAFNFJNSy/hqAiLxnl2ajJc0vSiln4C+gquRw8tsx2qYpZ3o0f4x1oTson+DLVY/9\nPL3xcb6J+RkX0gAwGCAtLb9XqVPicXICEd7gTd4OdqVRpQhmHthFa5PCQLoWdsFnBVx6BPXzeN2Q\nq1No3G2jbU3gpN36Kcs2h2lEJA24AlQugHNnwICZtTHbNWHv+RvlE3xZG7MdA7dG1lpjmevo5AsP\nD6JpwyeMYsDpQ+z2PkKXPgZWGh/UhH3vbuCzEk4H6TYhnXxjDdVtifGIUnfHJnRfGW2VUsOVUnFK\nqbjz58/n+3gzBro2b8lVj/2UP+HHVY/9dG3eEjMGDAYYNQrmzcs9Hx2dzET3+5xeRBBBL3xMVRm1\nsj3XxZ3Bz57H9dm2OCszrFiN27kQ3Sakky/sp1IFMFv6p3fDJlQQAv80UNtuvZZlm8M0FpVOBeBC\n5oxE5FMRaSIiTapWzXtESysd/r+9O4+LstofOP45MwwquC+5w6hppuaSkChuGEaSG3ZBE83qmmV1\nq6tImpk3jTJSb7fbT8tMrwumomFqdkli3KNA09T0asqAS+67qAwz5/fHA8iuCDjMcN6v17wYZp55\nnjOgX86c8z3f0z+UtQHbGBjbncsLf2NgbHfWBmzjif6hZGSoYK/cu6yxfD/PZLxIZJH5P3T+pRu3\nXG3cdAXDzy8RnHqegQPVnJBSPAVtpZqltOeESiPgJwIthRDNhBCuwDBgbZ5j1gKjMu//BYgvavz+\nXu1tcRa/nSNZnbADCaxO2IHfzpHsbVH8TwuKklN4OPhF+IPZTB/PZNob/8XOx3Zkp/3KLp+xwaMS\n1U8ftndTFQdzp3VApblOqFRW2gohAoFPAD2wQEoZIYSYBiRJKdcKISoDS4BOwAVgmJTyaFHnLA87\nXilKQUzNBQNC9Fx3McDRx6nFeS622Ak2AwM3d+FUQEs6dtSObVG7BeG+4fZtsFKuGY23h3MKUtxK\nAEVN2pbKJuZSyg3AhjyPvZvj/k0guDSupSj2ltimJk3PWzl8MgTrI9Fc1FvBpkPo01jbZzOulp/Y\nt1eHQWcgZmiMvZurlHMREfm3Us1S2uuEytWkraI4Au8O0ZyMW0TltssQm98Bqx70NqQO0NmQtlvc\nvKonIyqGkzv8sjMw1GpcJUvOfxOTJ8OoUVpPHrTUcSibdUKl0sNXlIoksYY/E80TiYhegQx+Hs60\nB89t2c9bDBLXHS/TZZ+VF17QVuFaLNpzWZkXoCZ3K6qsrJysHn1KCixadH8WgaoevqIUU3g4LOCv\n6M096Hy4nhbsbXqtMKsELK4Yusyml3Eq6em3g30WtRq34oqK0nrzeYdv0tJgxNxIHm02k+MuRqTQ\nPg7OfmcmgRGlt+eCCviKcg/O1WyJh88b7OxwEBerAGHVNjy3VMJFCqxC8o9h+3jQ52Xwzf8fVlVo\nrXiyevbWQnZY7XRCx+7gGaxsWhOBZLaoSVj6DPxvlF6YVgFfUe7BK5Em9vVZTedD9Xny1wboDz0B\n1kqACxnJfbDuHUmXFBeO9ZkPJ/JvkqJW41Y8ReXb9yGOX8wTmRndhLDg47T3G8T44OPMjG7CuKWf\nlVobVMBXlHtQo00ibxvX4fHdUrau/x33r1cwdtmT6H4bCk0SsV5vyN4mN6i0LBr/lNxdOlWhtWIq\n7FNdCw6zEy+20oNx5j20S+rO3l7f0iApiHHmPaX6cVBN2irKPQj3DQdfiLwI9f57mGGbXgazjSjz\nfK5fr4+114fc3DyRjeZP8a6TTLuqZlJTtZ59RISasK2IPDwKzrfXD3+Sm0dfICghhseMH7HXaz78\n8TinfBcxO7kD4+SlUmuD6uErSgmEh8MX8S1h0iRCWMlUY290XnNg8xSsvrP4JPAQVc+nsDA0jhkz\n4PWvZxNFoL2brdhBYAG/9ha+L5JyqSe3At7hSmA4G4PnQ2pXaPEjA3fVIyz4OLNHvFZqbVABX1Hu\nUc5c6iFz/JnQajDTgn/HNXoJVUxvod/1LGu9TzIosBHBH3RgyfcvE/ZDGP7N/VVufgUTFaWlXuY1\n/cQ6KreNwpD4AtL7S7hVDVqvY2BiI77dcJKZrhOJq1J6tbadbhNzRbkf8uZSA7Tq/iKnjgcyrHEY\nD52oxnvmTVwPDMPq/RXNTruTXP86BtMsRrcdx6JFuV/r5qY243FmhZVPsKLjE2N7xgefAEsVqHkM\ncaoN1T/fTkz15/C7vKbY16owm5gryv1SUMbFoW1f8viFagw7cZQPg/dQ1bgO64b5uJ9uRnKD6zxy\nGvpv8WDu3ILzsFVuvvPKOe/agsN8w2BsCASSaPMc+LOjFuwvNUbW/52bPv/H8s75998uKRXwFeUe\nFJY4seaqP37Sk0nRHTgZ/HcY0Zfr9ZNpdsqdvfUhxieFIFYV65yK48tKw+1DHGepx/P8h0305p+8\nSULgEmgRR+c/alHdcIZKic9xK2AKJx9fV+rtUAFfUe5BYXn0Hh7aZikfmjfS+U8bPBgHR/w59vkF\niJ0JAWE84DOpWOdUHF/WhO1rfMYw385YjFvpz3omGAPh0a/odrAOIckXidlSm8reMfjoX8ZqjCv1\ndqiAryj3ICJCG3fPKyUF+nzgT4tWi/jNeJHGf7SGhrvJMG7HmBDE2NgWzG9upJXIXTdf5eY7t4UL\noT9rGOHbHr1VQPAw0oy/YGv8K6129WBPi/N4nwS/hFPEjIohqI+RDaEb7nziYlKTtopyj6KitHH3\nfJNxRhMEh7Ax+gJ/7daa1Es9oe0qOBCEYd8Q9A128kjLTzmz5TQpOhM12yTy2TPhasLWSQVGRPKT\neT6XrreiW7IrP/XYgtz/F3h0oZaV43aesbEtmPNnRvEK3xdCTdoqShkIDdX+f2aVtc3WOBGiVzLH\nvILUoy+A9+fo9w8CBJbhf+FmwDsMO3yGhSYTdceG8M2/tdILKk3TOfnf0HGp3glo9R07/GKR+4PB\n+wvQp4P7efRHerMsIQlT6Pwyb4vq4StKCel0Wgnkgszk7xzxWcfcgKOQ2h08tqGzVGLyTzeZ20XH\nJP1HJB4JY+1alabptIxGZouajH/mMLjm+CULaHbKnQvV0unxx1f08BlJeClsjqZ6+IpShoqabG1O\nMlEJSehTfcBzK6T6ovvpDab3gn4/N+XDiFHc2hCn0jSdWWoq48x7aJjwDAiyb7VPNcJcrTLDK71A\nQtdxeAebyrwpKuArSglFRIDBUPBzE/iYdJ//w+bxEz1SAI9tZPSIpMqJh1nqdY1Jxr7MvjIaAl+B\nV1vneq1K03QSHh4MMr7Onz5f394zQcKFWpfoumcAi11WM6n7JBJPJpZ5U1TAV5QSCg3VsjDq1Mn/\n3BGfddwMmMLbsbX4hwncLYCQ3Gh0gJapdfgweA+vD00B77mQ3CfXa1WapmMKjIhk9jszsydlZldt\nzNrhX4BrGgYrDExsBOlu4JpGQpeVTHtiEhm2jPuy2b0K+IpSCkJD4dw5bSxfSli6NHMyt3kc4oeZ\nRCRcIK5JTaYt64Au9iMqXarHodaHyJAG1rWG5olPwIY52edTaZqOy/+GjrD0GcwWNUFKvmymA72V\nh0/WIjYKvt1vYJbuPVrr+uPVtN19C/agJm0V5b4ZUiOOLVc68CCH+Rkfqr5Zl2s1L1L1Ui2ufXKO\nDrp9/CbbqxLKji5zkjYs+Djdk9qxzWuftpGJvFQqaZd3oiZtFcWOsipjxlzxpwN7+IDJ6APHcK3G\nRVwuNeBajYvoA8fwT9ub2GxaTFDB3oFlTtJ2T2rH1l6b6Z7UjnHmPdhSUu2edqsCvqKUoayqmlmL\ns+LxJzpwE1bvryBxDBmf/AnHHsPq/RXRgbezNEzJJiK3l97m1cp95OHBbGMHtnnto8fmXmzz2sds\nYwdS8UBK7d/CmDH2Cfoq4CtKGSqoqqapGXRLfAQ2fK49EP8hLhl61rfKfD7ZRMiqELwb5d8LVyn/\nZo94jbDM/Wi3mDZn71M7xHh7IxN7pd2qgK8oZaig1Mpn/+8tdmzYjTvXmcI03M2PkREVy3lDFd41\nvUvIqhBW/mUlfs387n+DlbsWGQmmyXG5lkibJscx708br56YSMixS9gQDDFfomP0RH5tnHsjE3uk\n3ao9bRWlDBW0j+ksxuNOGusYgB+bOE19FpufpXrScKa7T2fK7hpwyUpkDUpl5aVS+qKiYHHsi0Qc\nDWRNSjP8SMGU0ozBy67Su+9hFqz5ks+sYbdfYM685WCPtFvVw1eUMlRQVc0L1KMLCTQjGRuCG1QG\no4lzXZcwcjd82jKNActusPjdw4jmJmr1j8w33qu2SLSfrHmZyUfXIYKHMtj4Ou/yHoONryOChxL6\n47p8w3h52S3tVkpZLm+dO3eWiuIMli6V0tNTSiG0r2PHSunmlpWxL2Xf/p6SidVlZZ/3pdsEN1nZ\n533JxOqy8zP1JRPqSozx0s1NO0/W+XK+HmSu55WyVbOmlH3YKG0g441ItwluEr8p0jChhow3Iq2I\nXL+bnLesfwNl+bsCkmQhcVXl4SuKHWSVVk5NhdFPCZa11WNZsZZ0qsCwwRj0V9EjuRkVD2ZtLN/T\nU0vZLGx/1KznlbLlL+LYTQeiCQGgn583t3p9TKXNE/jelEhLfTJNreZ8r7tfv5+i8vDVGL6i2EFo\naI5cewGt97VjfPAoSBoLOgsWg+StzbDDbCU+87CsSb7CJvtU7Z2yFRkJR/57mIl8gB7JYGK4YfwZ\ni9cIHt0cyBGvTxicvILp7avhFpe/+ml5WDmtxvAVxc5M1QcxzbwJl6S/Qq/pAFTaPIF/elUhyXiV\nPmhb3WVN8hW1vaJSNgIjIvkg5RHmN3idwca/A+DqMwPLiAGQVgev9IN8E21BjHqe/T565s3TevRC\naF/LS6lrFfAVxQ5yTrr+7dbHWIxbqfTYTCql68BqwJrch/TobxDBQxltHIHBcLuHWNBEcHnpQTqj\nyO2RGG8e5XLNo9ha/ZfrI56mz9CqnOv7MegtUOMYeqsgqW1NYkbF0KJnYvbmOOVt5bQK+Ipyn+Vc\nfSsl7G94HEvwSIbvt/L9Mhuu541kPPM07dhHTLSF1Manad15FGO3aTthh4ZSbnuQziRyeyQvrXuJ\nrZtcWJa+kLHxDSGjkhbkW68HnQ29xZWx8Q35vMcl1pknc3KH330rhHYvShTwhRC1hRAbhRCHM7/W\nKuQ4qxBid+ZtbUmuqSiOLt/q28aJWKJjePd7TzD3Ru4bBq432NM1Dsy92WHtwd6AJbTYWy/7JeW1\nB+lMvBt5s3jXcuJuTMGyZTKLe5tBl55rExP+7MyyHqn8I/ohtu0Ps1vJhLtVoiwdIUQkcEFKOUMI\nMRGoJaV8q4Djrkkpqxbn3CpLR3FWhW2J+Dhx7KEDKwnhEx8rawO2QWo38NjBwNjurE7YgYvMuP8N\nrqACA8FwZjpr/Wdi0Kdh0dtAb9M2MMkiwLBnKD1iRhOPP2D/bKmyrJY5CFiUeX8RMLiE51MUp1fY\n5Gq80Kpp9mYT3yZspXrqI+C5nUqpXnybsBU91vvb0ApO9/tvrN05mc6/dMPimpEr2Hc7mLnbjQRL\n+xXE+/yW/brynC1V0oBfX0r5Z+b9U0D9Qo6rLIRIEkIkCCEK/aMghBiTeVzS2bNnS9g0RSmfCpp0\nBa3X/yP+WNEzyKcHVzz2Uj2lPbc8khjk0wOp09//xlZgZ3390AeOYafPFi3Q24R2u1aXn1qfp39i\nI6oc6g3nWkOfKWDUqp2W52ypO+bhCyHigAYFPJWr1puUUgohChsf8pRSnhBCNAfihRB7pZRH8h4k\npZwHzANtSOeOrVcUB5Q13p618EqnA2uOznsnn1D2BSxhYGx3vk3YyiCfHqwN2EaQcSTf2qfJFdKj\nly7wi/dXYNND7CyofRi8P0fndh5D4nMcuOzCjQ1fagcbTdA4EbczfuU6W+qOAV9K6V/Yc0KI00KI\nhlLKP4UQDYEzhZzjRObXo0KITUAnIF/AV5SKIufCK12ez9n7mp+lXexIVidEIYGYX3YQZBzJjvrq\nU29Zi4wE78tx+EWNxtgE6h8zcrppCjy8Bjy20TaxO3/omvHg5Qz2b19InTpQtSqkpvjhIf2IKOfZ\nUiVdabsWGAXMyPyarwOSmbmTJqW8JYSoC/gCamcHRcmUr6Lmsg3sAwyZ02OeTcH8tV2aVuEcufoi\nEcu0CpiGlCBOb58FL3cEz63oUrqxf8NmXuRLvuQlDAb417/Kd4DPq6Rj+DOAvkKIw4B/5vcIIbyE\nEPMzj3kYSBJC7AFMwAwp5e8lvK6iOI3CxvSzlOdJQEdUVKXRYRtvV8CcxXgIfBXq7+WRUyA9dmDw\n+YjVDKFOHVi40LGCPaCqZSpKebB0qZR6fcEVFj09tWP69ZNyVrdo+VF3IeONSKnXy1ndomW/flLG\nH42XH237yK7vwREUVGkUpAxw2ShTdZ7ZFTArTagmGdFXMhU5NhBpAzmlay3JVCH7vz/L3m+jSBRR\nLVOttFWUciA0FBYtKrpkQnLzRxlvS8F8fAghwfBK08GMt6Ww/6GW+bZEVPXyC1bQlpNBrGJnRgf+\nsDVDANHmz7iV9AY8uBGXI70I3tCbFDxZfPICw+vOxGqMs0vbS4Mqj6wo5UjOsskeHlqwzxo2mN1N\nx/gnJMTOpNmpGiQ/8wa4puGWDtN1H5NRPYzGjeGNN+D8+dzndXNT5Rcg/6K3PsTxGx2YxAd8yNt0\nYhcbjQYqBw9iQtI1PvUyIKNX4GWuRjz+DvFzLGrhlQr4iuIgpBD80wfGBwhI7Q4eW0HAyM2efG9K\nZMrAPUyK8y90tyW9XivFkPcPSUWScy8Bl+H+PCiP8ODVW2zZ9394mpuzN/AL6DyP/n9Y6Z4KHifq\nMyr4JpbomHz7EpRXZbnSVlGU+8SKHhLehFRf8NwKNheabR7OUq9rTDL2Zch3o4vcWs9q1Xq3KSmU\n+5ovZSVrgrwPcYijj3OwVQrfdzjN9aEjtWDvPRekDpOHHo8T9RluPqUF+8aJ2ecoaPMZR6ECvqI4\niBCWM96nCXhsw8UK6DJIbrOLxvu78mHwHv7XNDMS+cyG4YFFnistTRs6ciZ3M2+RVWm0S/dgZpxa\ngSE2AquLxFrpphbsbToqWwT6FdHMNy/VXmT2g+23K2DqHXjBswr4iuIg1nZPhYAJPHUIPlrSAbfY\n96DuQY57r6fb/kYkNYbJPrUhIAyOFrpeMpszpXvmLTld2KeYyEho9HscfY9f4sPgPfQ+dUUbHtNZ\nteqXehvil9d43GzNLoaWl9WBSxqpgK8oDqLdwDgMppnc+HojH5g3sj7BxNjYFoizLVnr/SdRrWrw\nQcBF3GL/QZ+E9tmvy7uSN4uUzpPBU1D2TUGfYrwvxxHyQQcw92ZSdAc2Dv+3NhciAQm6DIHusX8R\nY6xT6LXqFP5UuacCvqI4iN0TNjC67ThMmVU1m5OMZ8IQZs55Cn2qD3s9L6NP7cL0hIt4k4inJyxd\nCosXF76wK2dP2JFTOe9mn9/ISOCzz1hJCEHEMKHBcDBc10ocZwj0iX/Fpoc0Fx1i+ODsYmh5Xb3q\nWD+bnFTAVxQHERWl5epLCfH4Y8SMF4lM8amF1SMBUnqQ4fELU3xqEcAP2Ruj5NwhqyBpaVoq590M\niZRXRe3zGxgIs31XcWTtg/TrWZ1fjRd5gDPY2q0Gm57q113QI5nx+y7qx45DnunEiz7D+GhpYoG9\n+fR0x53/UAFfURxEQcMWv/psJi1gKsR+jMstVzjYn7SAqfzqswmb3oVO3UchQgOZPFnLUBGi4HOf\nP393QyLlVVH7/CY16sF4Wwr6Ex1xabuM8cOSOTw0DBrugoxKWPWS8T/WYkLwMU6fegrDf36m55Uv\nCPcN58KFgq/nqPMfKuArioMoKMhEN6+FW+x7jEyoR8bRQG2v1YP9+bK5kaDHurHbfwn9jt6kRUoc\nzz8PtWuX/JrlUd5PMXr97T9YzdMOQMAE5jboygMrPgd9OrReByIDd3kL/fJofkpYCdErte0mLbf/\n0BX1ycERqYCvKA6ioCDjtyyM6QkX+Z5+TEm4hiE2Alp/x8FKTVgbsI2Bsd2ZkCBJwotWlt+4caPg\nnnBhE5GOFNhCQ2/39LMyaVJS4IOfzlPZIiBgAsl+i8HlppaRo4MhPzfhP+ZFWkZOjvTLrD90RX1y\ncEQq4CuKgygo+CTizQe8zUpCmMZUYhN+gBs1wXMbnG5LjYTRDCaGGz5zOTD6BRp1ejG7JyyE9nXe\nPK3MrzMEtoKGvfzM8PyyIG0jE8+tALhk6KmUrmNpl4tMN3bPd56sP3Q5Pznk/Hk56irlktbDVxTl\nPsm7U5aHB5hS/fGTcTQjGQl84mOFKhe1NMP6+1gSmAAXzkDAZCpbBF/E7aTP5I00Cp1PYg1/wsNz\nX6OwOj6OoqAhqA95i7kN6oFudfZjGbteZPTvZ/ly6H/5ddj7sLxzdumEvH/ocm5W4+hULR1FcWCv\nvAJz52r32/mMyt4asTEnmRuQY1O5jMrMinqIceY9mOjNAOM42vbcys+LnGsvIoMBnsxYw1qCsh9z\n85nGzYCpuFih/RnJ3noCiwtUiX2PG6e6Q7vlcLEFbA9HCHj5ZZgzx45vooSKqqWjeviK4sCyAtO8\nebe3RnwjIZUgktA9/BQ2zx2ZR9qYyAwu8QuzjZ24PnQE9Q7Xwlk2n4vcHol3I2+eqXaZJRcH8Sqf\nEWx8jbcbD+Lmw+vRWSrx7foaPPnbGVYYH2DE8MvcaLceEqZk9+xBS0ndsMGOb6SMqTF8RXFwc+ZA\nRgYsDdzA0d8WMYZ53PCZi83jJ/RWtOEdl3QszzzNdL8Mrg8dgbu4zvhdKWA0Ypocpy1KcmBHtngT\ntCiITaOCaBvYk7nGh3k8uDYJJ17nAdsZGqfdJHDPKb5eamOSPIXt6w1w4OkCz+UomUn3QgV8RXES\nWROWR3zWYQl4m0oWwQ9LYGZs5gGuadBrOrjc4tnlg0lsDAMeSad//A6OXH0RAFOyiZfWvUTkdsf6\nCzDsmhW5aCGn3AT7vbfB8H7I6FVUbrOYM03N9D+kHRcaqpU2lkf98DweXuC5HCkzqbhUwFcUJ5Hd\nM20eB4f6c2tZHI+bJc0TBmFIfAFk5qorl1vMbdCVuBMTWd/+MmkBU3lo52pMySaCVgSxfP/yXLtn\nOQK/qNGsMX+Ky09/11IuDemIwSO56b2ItondmVPAMI2zpVzeDRXwFcVJZPdMl22Ar9dlj02/Znwa\nyyOrcE/XMWUzVLboIGACGzueQG/TUdmi451OF+k/vw8Zl6/T85fPSIz2K/xCpaxUavikpoJxExav\nBbB5CkiBrHmCqpdrsX/DFl5pFJPvJc6Wcnk3VMBXFCdRUI8V4GS7bbgIKwtW1mWaCSKWPQJWA3Rc\ngvXnN8n4aQI3XCHNFTJ+/jtbYgbgffn+7Nt6t2WN78T02AMEBRuwRkdTt8p+EBJhg2s1LuIR2I81\ncnCBr8sa4rHZyK495MxUwFcUJ5HVY823avZiC1i9jlcun8KfjbzHVNwzrDx+FCp3iSSj62eQXgXS\nq3Cr26eE+njh99GT2ZO5pmRTmY3p321Z4ztZ3ncAMnoFA9qM4pz3NwxMbES1xd9Q/VhrUr1judrz\nldJrtANTAV9RnEhoKJw7p5VFzhqqqHMwHGH24/x5+NGo50bw88xfUZe3t4Bel66VCI5/H5Z9h4uE\nuQFHGBRQn5APOuByZSYhq0Jw0bmUSdC/m7LGd6NFtS9ZM7wa+5qd4eVE+NcGA17malxZcAASx3Kt\nbnzJG+sE1MIrRXFyOTfuxjcSTniD2Y+JvrXYV6sKcedeJb35FvQ7xmHBQKU+47jVdA+dj9TigMdF\nRmxpy1fdTmJduRpP6VeqK3BztS2HOnW0P1x5ZeXb+zW7PcdgSjaReDKRcN/wIs9XtapjryK+W2oT\nc0WpwHL1lreHZ0/m/rI9mq3rf8c14VU+3nEat+AB6Mng1oLdcORxdj54kfoX3Pmy6ykMK5fSwtyk\n1OvkR0SAq2v+x69cKfgaWfn2Jp8GoNNh8mlA0KIgjmzxzj5f3nkMg0HbtMRRa/2XJhXwFcXJFZZX\nbhL+1OUs/+E5/m7ew7vRbbAFD6fK4KehRTycaktyg+vo/uiLq7kbHzMBKN4Ye1SU1rMWQrvp9Vo5\niCyhoVCtWv7X5SxRnCUyEmotOI9ctJCgnhd4t7dkQM8r3Fq0jGHXrNnny5t5U726tmlJTo5U679U\nSSnL5a1z585SUZSSW7pUSjc3KbX+rXZzc5Ny7FgpXV217/uwUdbhjOw7uL7kH8jaIx6VTKgrCRop\nmSrkWJ8W0gbSgl4GES3h7q6r0+W+btZt7NjbxwlR8DFC3D6mXz8pR9aKkXU5I2fxpqziN07yDyR+\n78hZvCmlp2eh7bib8zsTIEkWEldVD19RnFxh+eZz5sCCBdr4djz+eBgXsaPNaRqfq8KFFrsQW9+C\nmMUQ+zFfPJ4YJnB5AAAN30lEQVTCy/3hU/7GGoYwzLDqjtedPFlLdyzIvHm37xf2CaR2bW2MX3SP\n5Hf9Eyzx3UA34/u8Z+zJzS5fwslHEd0i+V//T4qc5XW2TUxKQgV8RakACss3z8rqiT9q4tgrH9Hp\n2485Yw4Bixuy13T6GifBqY7YrO5EM4QwZjGT8URZh93xmkVl2mRtUAJ3HnfvdEJHSoct6DosYO3w\nr7gyfARSWKHePqSQRLXTY3rsgUKvFRhYvMedmQr4iqKQeDKRlX9Zybb9YbRaPw63ZctxF9fZFTwD\n1+FPwvJVXFy/mu5sw/TMJ3R94XbEjtweyeyfZudK2zQlm6gRWHgap15/+/6dxt2/MX/G2B+bYHOx\ngf4GGNLAJR1cLAyM80G/PJrlfQcUeq3Cql86c1XMwqiArygK4b7h+DXzw9MT9tMeH3MVXvi5Kufd\nId2QAQ32YOQoW30SWd8Kfquny86UcYmYQVhsGFs3adXWTckmQlaF8Mogb3SFRJgxY3J/n/cTSM7N\nwz1IZU7CEWr/9iTobVqtHH0GrfZ4syNhNWNdrLSo9mWh7620cv2dgQr4iqJkyyocFm/U828vAw9t\nfhos7hAQRtPnW0BAGMTORH69LjtTZlq7a1SJ/QdbLdN41/QuIatCWPmXlUS86MfixeDufvv8Oh2M\nHXvnDUZq1IA+xJGMEYFkkPF1LrTaDtbMkGUTHGr9Pzo98jm1X/xLvp27cips4/bibujuDFTAVxQl\nW2goDHzdBMEhEL2S/5lW0X1ZGC422OoJ3VMhIuEsHcy1SU/6G9N7QXrS31ifYOL1PTB9y3TGeo3N\nXhgVGgrXrt3OjbFa7243qZ7BL5JkvEoyzfinsQNrh/4HDDdAZ6PzH7VAgE6fhmnQLLyDTWX7Q3Ei\nJQr4QohgIcR+IYRNCFHgyq7M454UQvxPCPGHEGJiSa6pKErZ8g1J5O2WK/GUWtDe1qA6GTrgopFt\nHoJKPh8xw9gV4fU5bJ5ChtdXRPscY27Ly0zZDHN/iGD2OzPvaVOVyEgwTY7jzR8XIIKHMtj4OrMa\n94SzrUFv4eVEWLW0Og/GjsGGHpeTPZj3XWKR58w5PHQ3jzu1wvI17+YGPAw8BGwCvAo5Rg8cAZoD\nrsAeoM2dzq3y8BXF/ug6SzJVSHxmaX10n1lSTEVWnqST1Y3fyCm8Jyv7vJ+dqy9BzjJ2kGJCHfna\n8I+lp6eW7+7pqeXlF6VfPykf9X9WVjd+I+PpLeONSJcJNSWDR0rdO3o5ywdpRdzOpTfGS3w/km5u\nRZ/b07PgPPwiUvcdGmWVhy+lPCCl/N8dDnsM+ENKeVRKmQ4sBwaV5LqKotwfldvEQexMSBinPZAw\njtqHumA525E15k+ZxlSe1S+gcux0FuhH4u/bl/eYylMHXFn9wFukNJyN9DSR0iSSv75volPkAAKj\ntHzIvHXw9zd4lF2ubqQHP8tg4+tMMP9CxqXm0HEJLvuG0CmhN6nkSJ43+8H28Duumq2IG50U5n5s\nYt4YOJbj++NAl4IOFEKMAcYAeFTEVRGKUs7M99vAmK8hZwXj818n0Ic4mjEaG4LPtx9lGHFEMJkf\njT2pFDwIv61XMbUHAsajs1TGFh/BrSED2J2WxsjmM7Pr4KelAb6R1D6hI/TAr4QF/MrNxJe5OfQ5\ndqZXg+on6HwC/mi5hsHGFXiZC6jDQNEZN1lrDiZPrhjF04pyxx6+ECJOCLGvgFup99KllPOklF5S\nSq969eqV9ukVRSmmgnLkQVuZ2wwzemykoD24hw6MNB/jVnQM7/aowpDftWNthpvQN1zLn4+dSf2j\n43LVwW90oiG7g2cgTnVgVizg/TlUugI1TqA72Z6Pv+zN6tVWboY8T7xRn7+R5F41W9AOWhVto5PC\n3DHgSyn9pZTtCrh9e5fXOAE0zfF9k8zHFEVxAHmDZVbQz/JX5hPMSlYSwmJGMcu8nutJ41nSEfSp\nXTPz5q08nFqLFgkDeP753CWMXzXvxy16IWHBx5lZ5a8g9VpkulYP15pHGWx8Hd2wWMIejMGtZf4J\n2pzDM6W1g5azuh9pmYlASyFEMyGEKzAMWHsfrqsoShnIOyYejz9e+j1410kGoJPxEwxe/4bdI7F6\n/AQS9FY44HERs883VLHcTo/pQxz/ZDzTzPGIpJf4s9dXIKw0PFUH3M9yc/8IbgY/x/RTej5504+0\njbkT7uvUyb0PbWntoOWsSpqWGSSEOA50Bb4TQsRmPt5ICLEBQEqZAbwGxAIHgJVSyv0la7aiKPZS\n0DDPyEX+VD1nxtSlPkHBBgxbx6FrE629wOqKdedLYKmCNWASV3wWUse4nNq+4fT1fYIu/Tsw2acu\ntq7/AglYDXhcljx1CPD+goyDofx6OjFfIAet9HLO4Rm1qrZoascrRVFKzUtTXmT50kCGNQ7je++j\nHHc3IM+3gtrJED8N2kRDg93gko774m+ZxhQmD9/LTYOEcw9RK/kRLnb8Dgw3eC62DVGuL9Cmfxy/\nvbWBgkKVELkrcha245WnpzYcVRGoHa8URbkvsvaW/SIhhTfXdEAKoP5++LMj9J4OjRPBcAuu1eFZ\nFvGheSO634ZBRiWofZiLnWOobMugcux01uoDqb5/HBMabbjrEscqBbNoKuArilJqwsPBL8If01v/\n5T3zJgxL14LVAB47oNJl0NsQl+vjbrjI58M2Us34LWn7RgNWbaBfn4H4+TX6JTzEhe2RnD+vTboG\nBt5dIC+s9n9FzcrJSw3pKIpSqqKi4LXXoMGlfRykDfhNhV7va09muMKt6hi3DuNEr6+w6AUIC7hY\nMFgFLjaJxeZOxvJ12Xvvgha4AwO14G21auWVx4y5u7o8FY0a0lEUpUxl5b4LASNHwqVLcJB2uBg3\nQvcPtMlYCejT0Z1rhrnHciyHngbXNDBYcLHqqbJ0Na2XfUyG1MOwIDDeLoqWkgKLFt3eOMVq1b5X\n6ZbFowK+oiglkjP3Hbg9uWo0kTFiAOhsVE99mGWL6mOwgs0jEa7WRddmBQYr6KyQYatEaNs9JF8K\ngxVrYN9Qbbw/k16v0i1LgxrSURSlRArLjME3Uuvdm3vBisx1mkYTBI2C6seoYRHEfC3hgfoMePIa\nVuHCuEYxWr59juDu5pY/2GfJm6WjqCEdRVHKUKE57tvD4aNLt4M9aOPyv7zGmM5jiBn9I35HJX4J\np1g3ah3PPjqUGm0SC5x0zbu6N4squVU8qoevKEqJFNrDL8S95MTnKraWyc1NZeAURPXwFUUpMwXl\nvguR+2uWe82JV+mWpUMFfEVRSqSgYLxkiTZ5u2SJVu8mS5UqJbuOqnhZMvejHr6iKE4uNLTwAHzj\nxu37WQupsl6j3F+qh68oSplR1SvLFxXwFUUpM6p6ZfmiAr6iKGXmboueKfeHCviKopQZVb2yfFEB\nX1GUMqPSKcsXlaWjKEqZKiqDR7m/VA9fURSlglABX1EUpYJQAV9RFKWCUAFfURSlglABX1EUpYIo\nt+WRhRBngWIUXc2lLnCuFJtjD47+Hhy9/aDeQ3ng6O2H+/8ePKWU9Qp6otwG/JIQQiQVVg/aUTj6\ne3D09oN6D+WBo7cfytd7UEM6iqIoFYQK+IqiKBWEswb8efZuQClw9Pfg6O0H9R7KA0dvP5Sj9+CU\nY/iKoihKfs7aw1cURVHyUAFfURSlgnCqgC+EeFII8T8hxB9CiIn2bk9xCSEWCCHOCCH22bst90oI\n0VQIYRJC/C6E2C+EeMPebSouIURlIcQvQog9me/hPXu36V4IIfRCiF+FEOvt3ZZ7IYQwCyH2CiF2\nCyGS7N2eeyGEqCmEWCWEOCiEOCCE6GrX9jjLGL4QQg8cAvoCx4FE4Bkp5e92bVgxCCF6AteAxVLK\ndvZuz70QQjQEGkopdwkhqgE7gcEO9nsQgLuU8poQwgBsA96QUibYuWnFIoQYB3gB1aWU/e3dnuIS\nQpgBLymlwy68EkIsArZKKecLIVwBNynlJXu1x5l6+I8Bf0gpj0op04HlwCA7t6lYpJRbgAv2bkdJ\nSCn/lFLuyrx/FTgANLZvq4pHaq5lfmvIvDlUz0gI0QR4Cphv77ZUVEKIGkBP4CsAKWW6PYM9OFfA\nbwwcy/H9cRws0DgbIYQR6AT8bN+WFF/mcMhu4AywUUrpaO/hEyAcsNm7ISUggR+EEDuFEGPs3Zh7\n0Aw4CyzMHFqbL4Rwt2eDnCngK+WIEKIqsBp4U0p5xd7tKS4ppVVK2RFoAjwmhHCYITYhRH/gjJRy\np73bUkLdpZSPAv2AVzOHPB2JC/AoMFdK2Qm4Dth1btGZAv4JoGmO75tkPqbcZ5nj3quBKCnlN/Zu\nT0lkfgQ3AU/auy3F4AsMzBwDXw70EUIstW+Tik9KeSLz6xkgBm3Y1pEcB47n+HS4Cu0PgN04U8BP\nBFoKIZplTo4MA9bauU0VTuaE51fAASnlbHu3514IIeoJIWpm3q+Clghw0L6tuntSyklSyiZSSiPa\n/4N4KeUIOzerWIQQ7pmT/mQOgzwBOFT2mpTyFHBMCPFQ5kOPA3ZNXnCaTcyllBlCiNeAWEAPLJBS\n7rdzs4pFCPE10BuoK4Q4DkyVUn5l31YVmy8wEtibOQYO8LaUcoMd21RcDYFFmZlfOmCllNIhUxsd\nWH0gRus/4AIsk1L+175Nuid/A6IyO6FHgeft2RinSctUFEVRiuZMQzqKoihKEVTAVxRFqSBUwFcU\nRakgVMBXFEWpIFTAVxRFqSBUwFcURakgVMBXFEWpIP4fTVXk+niDyvEAAAAASUVORK5CYII=\n",
+            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXwAAAEICAYAAABcVE8dAAAABHNCSVQICAgIfAhkiAAAAAlwSFlzAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4yLjEsIGh0dHA6Ly9tYXRwbG90bGliLm9yZy+j8jraAAAgAElEQVR4nOydd3wURfvAv5PLUUIXlJ5cRHpCghCKoR2CwYAUNQQMSFGqvuovQlAQQZFXqVIUFBXxJYEUFARFUUxAimhCk6YI5AIhUqRJCJBy8/tj745LcmkQSJvv57Ofu92dnZ2dnX322WeeeUZIKVEoFApF6cepqAugUCgUinuDEvgKhUJRRlACX6FQKMoISuArFApFGUEJfIVCoSgjKIGvUCgUZQQl8IsYIUSQEOKHoi6HFSFERSHEBiHEFSFE1D043yEhRLe7fZ57gRDCIISQQgjnfKQdLoTYfi/KlR+EEK5CiGQhhK6oy3IvEEJ0E0Ik3oV8i9V9zUqpEfhCiGeEEHGWRvu3EOI7IUSnoi5XXkgpw6SUjxV1Oex4GqgN1JRSBtztk0kpW0opt9zt8yhyR0p5UkpZWUqZcSf5CCG2CCGeL6xy2eWb75epImdKhcAXQgQDC4D/ogkrV2AJ0K8oy5UXxbTxugFHpZTpd/MkxfTaFYrSjZSyRC9ANSAZCMglTXm0F0KSZVkAlLfs6wYkAiHAOeBvoD/gDxwFLgKT7fKaDqwBIoCrwB7Ay27/a8Bxy77DwAC7fcOBHcD7wAXgHcu27Zb9wrLvHPAvcADwsLvO/wHngQTgDcDJLt/twFzgEhAPPJ5LfTQHtgCXgUNAX8v2t4BUIM1Sp89lOa4ecB24z25ba+AfQA80AqIt1/YPEAZUt0trAiYBvwM3AWfLth75uE+2erLLTwIPWf77W+r7KnAamJDDtdvfg8vACeARy/ZTlroflqV95VTvOkud/2PJ5wVLmZztjv0MrU2dttxvXdbrye2+Oyj/COCI5TpPAGOy7A+xnC8JeD5LHfUG9lrOcQqYbnecIUvZtwAzLHV1FfgBqGXZVwEItdzny0AsmqI1E8gAbqC1nw9yuIYo4AxwBfgZaGm3ryIwz1LXV9DadUXgpKV8yZalI9qzGJrLNeRYV1ie+xzKtxSYm2Xb10BwPp/x7Y7KY1evz9utj7SU8RKwCXAraJsokLy8G0L4Xi5ALyDdvlIdpHkb2AU8ANwP7ARm2N34dOBNNKE1Cu3hXgVUAVqiCTl3S/rpaALxaUv6CWgCVm/ZH4AmGJ2AQOAaUNeuMaQD/0ETdhWzNBA/YDdQ3XLDm9sd+z9Lo6tiaUhHsQhkSx5plrLrgHFoD7xwUBd64BgwGSgHdLc03KZ21xeaS11GA6Ps1ucAH1n+PwT0RBPc96M9zAvs0pqAfUBDoKLdth75uE+2erLLz16Y/Q10tvyvATycQ/mt92CEpa7eQRMmH1rK/ZilPirno97HAn9Yruc+IIbMAmct8DFQyXJNv2EROvm97w7K3xvtxSqArkCK9VrRnoUzaG3WBU0o29dRN8ATrW22As4C/R0JJzTBdBxogtZOtwDvWfaNATZYzqED2gBVHQm0HK5hpKU+rS/4fXb7PrTkUd+S9yOWdJnK56itOriG3OqqGzkL/C5oL0Rh156uA/Xy+YznS+CjWSCOWe63M5oysbOgbaJA8vJeCea7tQBBwJk80hwH/O3W/QCT3Y2/zi3Nq4rlJrW3S7/b7sGYDuyy2+eEnbBxcO59QD+7xnAyy377BtIdTaB0wKJFWrbr0DTvFnbbxgBb7PI4ZrfPxXINdRyUpzOaULDPfzUWbY+8Bf7zQLTlv7A8GF1ySNsf2Gu3bgJGZklj4pbAz+0+2erJbr+9MDtpqZOqebSF4cBfduuelnxq2227AHjno96jgbF2+x6z5OWMpvHexPJis+wfDMTk977ns/2vA162/F8OvGu37yH7OnJw7ALgfct/A9kF/ht2accD31v+j0R7GbdykOcW8hD4WdJXt5y3GtqzdB27L2a7dJnK56itOkqTS111I2eBLyztqYtlfRSWNp9D+qzPeH4F/nfYfUVbrj8Fzax6220it6U02PAvALXysAnXQ/tEtJJg2WbLQ97qrLpu+T1rt/86UNlu/ZT1j5TSjGYSqgcghHhWCLFPCHFZCHEZ8ABqOTo2K1LKaOADNC3nnBBimRCiquV4vYNrqG+3fsYunxTLX/syW6kHnLKUO6e8cuNLoKMQoi6aJmQGtgEIIWoLIcKFEKeFEP+iaZi1shyf4/WT933KjafQzDoJQoitQoiOuaTNem+RUjq633nVez0yX499OjfLsX/btYWP0TT9TORy37MhhHhcCLFLCHHRkqc/t+o4a3lOZTm2vRAiRghxXghxBe0LJev9seeM3f8UbrWnlWjmh3AhRJIQYrYQQp9LPvZl0Akh3hNCHLe0EZNlVy3LUgHtxX/H5FFXOSI16RuO9oIGeAbNPGnNN69nPL+4AQvt8rmI9rKpX5A2URBKg8D/BU2T6p9LmiS0yrXiatl2uzS0/hFCOAENgCQhhBvwCfAimpdLdeAg2k20InPLWEq5SErZBmiB9jk9Ec1GnObgGk7fRtmTgIaWchc4LynlJTR7biDagxBueUBA6zSXgKeUsiowhMzXDrlff2736RralwsAQog6WcoVK6XshyZQ1wGR+bmePMir3v/Gri1Y9lk5hdYua0kpq1uWqlLKlo5OlMN9z4QQojzaC3cu2hdJdWAjt+r4b7S2aKVh5hxYBawHGkopqwEfkf3+5ImUMk1K+ZaUsgWayaUP8Kx1dx6HP4NmyuiBptUbLNsFWn3fQDPDZDutg22Z2gRgaxP5qKu8WA08bXmm21vyIp/PuH35yKmMaG1kjF37qC6lrCil3An5axMFpcQLfCnlFTT7+4dCiP5CCBchhN7ydp9tSbYaeEMIcb8QopYlfegdnLaNEOJJy1fFK2gP9i40W61E6wNACDEC7e2fL4QQPhYtTI/WWG4AZsvXRyQwUwhRxdLogm/zGn5F09ZCLPXUDXgCTaPJL6vQHvCnLf+tVEHrULsihKhPwRtobvdpP9BSCOEthKiA9jkPgBCinGU8QzUpZRpaJ5eZOyQf9R4JvCSEaCCEqIHWmWc99m+0F+M8IURVIYSTEKKREKJr1vPkdN8dFKkcmj37PJAuhHgczYxkJRIYIYRoLoRwAaZmOb4KcFFKeUMI0Q5N+BYYIYRRCOFp8dn/F+2laC3vWeDBXA6vgva8XEAThP+17rB8dS4H5gsh6lm+BjpahPd5yzns894HdLGMIagGvG63L6+6yhUp5V60F9CnwCYp5WXLrnw/41LK82jKwRDLtYwk88vsI+B1IURLS17VhBABlv/5bRMFosQLfAAp5Ty0B/ENtBtxCu0NvM6S5B0gDs075ACaZ807d3DKr9E03EvAUOBJi9ZzGM3D4Be0hu+J5uWQX6qiaQ+X0MwDF9A6RUHr6L2G5m2wHU3QLi9owaWUqWgC/nG0Br0EeFZK+UcBslkPNEbrO9lvt/0t4GE074pvga8KWLwc75OU8ihap+5m4C+0OrBnKGCymAnGovXtFAa51fsnaKaN/ZayZr3eZ9EEz2G0e7oGqOvgHLnddxtSyqvAS2iC/RKawF5vt/87YBFa5/ExNCUENAELmh3+bSHEVbSX6e1+BdWxXMu/aB4mW9HMPAAL0TTjS0KIRQ6O/Z/lGk+j1cuuLPsnoN37WDQTxyw0G3YKmhfQDosJpIOU8kc0b7nf0frZvrFmkldd5ZNVaF8iNqXmNp7xUWiKzwW0zvSddnmttVxfuKXdHkR7LiGfbaKgWHuhFflECDEdrRNsSFGXRaHIDSFEczQhUl7e5XEVipJBqdDwFQqFhhBigBCivMXENAvYoIS9wooS+ApF6WIM2mCd42iDoMYVbXEUxQll0lEoFIoygtLwFQqFooxQbANY1apVSxoMhqIuhkKhUJQodu/e/Y+U8n5H+4qtwDcYDMTFxRV1MRQKhaJEIYRIyGmfMukoFApFGUEJfIVCoSgjKIGvUCgUZYRia8NXKIobaWlpJCYmcuPGjaIuikJBhQoVaNCgAXp9vgKVAkrgKxT5JjExkSpVqmAwGBCiwEEmFYpCQ0rJhQsXSExMxN3dPd/HKZNOKSQsDAwGcHLSfsPC8jpCkR9u3LhBzZo1lbBXFDlCCGrWrFngr02l4ZcywsJg9GhIsUyBkpCgrQMEFVb8yDKMEvaK4sLttEWl4Zcypky5JeytpKRo2xUKRdlGCfxSxsmTBduuKHmsW7cOIQR//JH3FAYLFiwgJasGUABWrFjBiy++eNvHF3Y+ijtDCfxShqtrwbYre//d427V7erVq+nUqROrV6/OM+2dCnxF6UIJ/FLGzJng4pJ5m4uLtj0rVnt/QgJIecven5tgUi+I/HE7dZsfkpOT2b59O5999hnh4bdmpczIyGDChAl4eHjQqlUrFi9ezKJFi0hKSsJoNGI0GgGoXPnWvPZr1qxh+PDhAGzYsIH27dvTunVrevTowdmzZ8kJs9mMwWDg8uXLtm2NGzfm7Nmz+cpn+PDhrFmzxrZuX6Y5c+bg4+NDq1atmDZtGgDXrl2jd+/eeHl54eHhQURERAFrTWFFCfxSRlAQLFsGbm4ghPa7bJnjDtuc7P1DhjgW5ndLiJVG7lZfytdff02vXr1o0qQJNWvWZPfu3QAsW7YMk8nEvn37+P333wkKCuKll16iXr16xMTEEBMTk2u+nTp1YteuXezdu5dBgwYxe/bsHNM6OTnRr18/1q5dC8Cvv/6Km5sbtWvXLlA+Wfnhhx/466+/+O2339i3bx+7d+/m559/5vvvv6devXrs37+fgwcP0qtXr3znqciMEvilkCDCMF2oglkKTAmCoCFCk/7WpUoVZg/eS4KvP3SYf+vAF5rBCF94xp+EBHjuOXjo5fE0+6AZkLsQU5p/Zu5WX8rq1asZNGgQAIMGDbKZdTZv3syYMWNwdtYc7+67774C5ZuYmIifnx+enp7MmTOHQ4cO5Zo+MDDQpmmHh4cTGBh4W/nY88MPP/DDDz/QunVrHn74Yf744w/++usvPD09+fHHH5k0aRLbtm2jWrVqBbo2xS2UW2ZpIywMhg+HdMez2s1mIj7JsRw79TS6akYy/CZQk3Nc0N0HupvguhNOPgLAzf79OF5jPZW2jWN2Ss7CyqrpK1fQW7i6avXgaPvtcvHiRaKjozlw4ABCCDIyMhBCMGdO/ue2tnfls/fh/s9//kNwcDB9+/Zly5YtTJ8+Pdd8OnbsyLFjxzh//jzr1q3jjTfeyHc+zs7OmM1mQDMPpaamAtpgotdff50xY8ZkO2bPnj1s3LiRN954g0cffZQ333wz39esuIXS8EsABdKep0zJJuz9fbsw3+DFbCbys+8v9De8xNmM+gjvlYjYUVzwmw2+70ENE1xy04T+hNrQbD380ZfUtUvw8clZWOl0yhU0KwXpS8kva9asYejQoSQkJGAymTh16hTu7u5s27aNnj178vHHH5NuufcXL14EoEqVKly9etWWR+3atTly5Ahms9lmkgG4cuUK9evXB+CLL77IsyxCCAYMGEBwcDDNmzenZs2a+c7HYDDYTFHr168nLS0NAD8/P5YvX05ycjIAp0+f5ty5cyQlJeHi4sKQIUOYOHEie/bsyX+lKTKhNPxiTp4DqcLCNMl68iS4ujK7QQI+AuY8Aj1OwLsPtqFyxkW+63oC1zNnSKxYFdktkPVOkiY/juSoMRSQ4HKJ8hfrklb5LOY0F6h8DpIfwDliDbqAMczYC8nJH98qmCEG6sfisjckm7C3UpZdQa1fNna3hpkz7+yLZ/Xq1UyaNCnTtqeeeorVq1ezePFijh49SqtWrdDr9YwaNYoXX3yR0aNH06tXL5st/7333qNPnz7cf//9tG3b1iZcp0+fTkBAADVq1KB79+7Ex8fnWZ7AwEB8fHxYsWKFbVt+8hk1ahT9+vXDy8uLXr16UalSJQAee+wxjhw5QseOHQGtMzc0NJRjx44xceJEnJyc0Ov1LF269HarsMxTKHPaCiGWA32Ac1JKDwf7BbAQ8AdSgOFSylxf023btpVqAhRNo3dkGnBzA9PMLG8DYIwhkOUBm6h0yI8rPhEYTtbB5HoG3dmHyKh9DMxO4GSG9HJwOBBarQQBXG4A1U7Dtfug8gVIdQF9CuKPx9G5byUdZwhfByajJuwDBlJ+QySfvWFkypRcymi6WzVz7zly5AjNmzcv6mIoFDYctUkhxG4pZVtH6QvLpLMCyK3r/HGgsWUZDahXdD7JtfPPQS/qINNZMi424or3N3C8BybXM3C1jkXY60BnBrMzJLUHr5UANDjWlEr6C3DJFSpdgORaoL8O12ohm32HOb4r5cJXIwIC6DGgDuWe6Yk+KpTql4wEBTk2XwgB/v53oUIUCsVtUygCX0r5M3AxlyT9gP9JjV1AdSFE3cI4d2knJ2cLV1ccvg2MbGHMocugvwYPbYYrDaHqGW2nLgNxuT5IHbhtAzOIdD3/bn8Xz229oUaCZsNPakvP/Q9A5X9AOmGue4BUkz8yvTybvc7CkQF8b3qP02d19Bs8jFdi/Rk2TBPyVqSEL75Q3joKRXHiXnXa1gdO2a0nWrZlQggxWggRJ4SIO3/+/D0qWvElLAzs+tts6PWaVu3f3Y/5Bq9M++YbvPil1n2QVgkkUO2U9gvwbx1ktdOaN06GDn26MxV/eoOMgCDMtY5y/6YQXHf3Zt7Ov9nb+CxD94FThg7K/0ud/j2hahJISG32LfsMl3iygy/rm66kya8VuLb0C1rJvZnKUtY7bhWK4kax8tKRUi6TUraVUra9/36Hk66XKaZMAYvHGgCDCSMeAzfSnAiaYqBH7QeZEJBoE/rzDV5MGBTPAY/DsO9Z7SCr1n22JVQ5q9nwAXaP5r7DfdF3fgfjtvY8del3zu6azYrTS3g3YD+RUTBiXTcqh0Wg11/ljPdmnPcHIjbNhnI3eHXoIdb7beeRTf05tGs56+jHM4ThRWahX5Y7bhWK4sa9EvingYZ26w0s2xS5YC8sFzOeUIZiIAEntGGuwetWMPd0IBMCEuli7MqEgETa3XgU3ZcbwN0ysjKlhvZ7xQBHe6M715bW5+dRq7GJM2u/ZO24TXSe+zgh2yUiNJRf6wsiosBoglh8mMZbIEGX5Emlxl8x90wYupPtQZcOybXZvWsVEpjGW8w0+NLO92nScGYx44E78ztXKBSFy71yy1wPvCiECAfaA1eklH/fo3OXWKyDdwYTxjXfpWw9rQliKzEPpJB+ZRWdHvoP27rOoLN5Kn1qvU1S6/mcqvUnbJoHu4K10bR+E2DTXDyuB7NnH0AwAEZ3I0Z3Lc4KQUG8FhRkc/X0EXMYGAAjwgIZZDoLhjR6DjlAhs6M00VXzDVOctM/mNR6sbxaLQF9hYUMDpM4A6s6nEHv0ZqOXoOBkHtbcQqFwiGFouELIVYDvwBNhRCJQojnhBBjhRBjLUk2AieAY8AnYFH/yjh5DaiaOROG68P48ZkFLKn2OAMC9MQYtH2+/p48OsSZtRd6sP3mUjqbp7L95lKcG8fg0XczNX6bqwl70H43zYUHN2MX7yrn8wcFgclEbOgsIhtP5mO5CyNbeL9OFzJ0ZoTZCXPFy/BHH/BZiqy3ByqfJ11ozcltcEt+8VtLxgNHqDpvDxgMxHwyhdk78h9XReGYxMRE+vXrR+PGjWnUqBEvv/yybaRqVpKSknj66afzzNPf3z9TILSCMH36dObOnXtbxxaEbt26YXXTzqu869at4/Dhw7b1N998k82bN9/1MpYIpJTFcmnTpo0szYSGSuniIqXmz6ItLi7adiuzBu2Rz4tP5CMdBkimCanzf05WmlhRVh3SVjINKQJ7SzGxppz3VbSUUsp5X0VLMamW9isy521dhMj/+bPiNPRx2bzDcFnJsF7yWlWpn+wsedNJMh3JC40l04RkqrNkGpJpSP3gnjKabjLagKwWopejR358F2v07nP48OGCHRAaKqWbm1bpbm65V24+MJvN0sfHRy5fvlxKKWV6erocOXKknDBhQra0aWlpd3Su/DJt2jQ5Z86c2zq2IGXs2rWrjI2NzVfaYcOGyaioqNsqU0nDUZsE4mQOcrVYddqWJfKKpjh7Nvx8ZgJf9N7OvjMjeGRTfzJ8lnNNVubfh+LgQiN8Essxt9xrBA/QTDLBA4zMbR/J5sOxecbFv51ojjJ0I0d2fU4dUzO6/tqKtHLp4GTG/UwlqHUMLhk0274AMsqjc91KjHELAwL0yMgIBn2be8TGUsVdCC0aHR1NhQoVGDFiBAA6nY7333+f5cuXk5KSwooVK+jbty/du3fn0UcfxWQy4eGhjYNMSUlh4MCBtGjRggEDBtC+fXubxmwwGPjnn38wmUw0b96cUaNG0bJlSx577DGuX78OwCeffIKPjw9eXl489dRTecbYHz58OGPHjqVt27Y0adKEb775BiBbGa9du8bIkSNp164drVu35uuvvwbg+vXrDBo0iObNmzNgwABbOezLC/C///2PVq1a4eXlxdChQ9m5cyfr169n4sSJeHt7c/z48UzhmH/66Sdat26Np6cnI0eO5ObNm7Y8p02bxsMPP4ynp6dtcpmtW7fi7e2Nt7c3rVu3zhSmoiSiBH4RkVc0xeNXRxFzoys6j9WkBA7htzNj4GptqHwezFC+ciLvnP+O4HcmZDo+eICRjVNCHA6GAkhO1mTO7URztL4sjhsS2dpxr+bumV6eC9WuUfukG9SIBym07VJiPtqXGV0hNe4/rDMtwniuDMUxvwvxkQ8dOkSbNm0ybatatSqurq4cO3YM0IKMrVmzhq1bt2ZKt2TJEmrUqMHhw4eZMWOGLZZNVv766y9eeOEFDh06RPXq1fnyyy8BePLJJ4mNjWX//v00b96czz77LM/ymkwmfvvtN7799lvGjh1rC9ZmX8aZM2fSvXt3fvvtN2JiYpg4cSLXrl1j6dKluLi4cOTIEd566y2H5T106BDvvPMO0dHR7N+/n4ULF/LII4/Qt29f5syZw759+2jUqJEt/Y0bNxg+fDgREREcOHCA9PT0TGEaatWqxZ49exg3bpzNTDV37lw+/PBD9u3bx7Zt26hYsWKe112cUQK/iMhLAx/04wb0nd/BacsUdCKD9KH+UOUMZABCkP77EPr0L09MvGOt2RoX3xLTysaFC5qimeuArhyYORPKN4uBQQNAZGidwmHfcdW5HGddTWAWICRVjrcD51RSW62B/UMRbT8CwxYt87ISR7mI5prs2bOnw9DI27dvt4VVtk6S4gh3d3e8vb0BaNOmDSZLbIyDBw/SuXNnPD09CQsLy1fY44EDB+Lk5ETjxo158MEHbVqzfRl/+OEH3nvvPby9venWrRs3btzg5MmT/PzzzwwZMgSAVq1aOSxvdHQ0AQEB1KpVC8g7JPSff/6Ju7s7TZo0AWDYsGH8/PPPtv1PPvlktuv29fUlODiYRYsWcfnyZVv46ZKKEvhFRF7RFI2/nWNtVBoZneeSkfKAFhJBgPP2SXS0mHdSDz7DjOWxOcrPoCCwm0zIhlXxzE80R3v5PGUKtO4dSyVTIIRtRBcbDPFG5FlvRHJNMJdjyFY3nBr8hkjXwz/NePjcTZyjQhkwUE9MPy/SR2Y2c6SPLKUzqBR0rsl80KJFi2ya7r///svJkyd56KGHAGyByG6X8uXL2/7rdDpb9M3hw4fzwQcfcODAAaZNm5YptHJO2Iditl+3L6OUki+//JJ9+/axb98+Tp48WWTxiqzXbn/dr732Gp9++inXr1/H19c3X/MIF2eUwC8irBp4hUdngyEm08xU89fG4N/dj72mV7h5rB/cFw9mgXOqnvR2S/ntzFjN66aGie2zQhyaia2C2lFQM4CLF/OeGcuRGfr3pSE8W+NjXM4ZyciwJNz8HtJJQNh3hMaYuHJwNDLDhaBvOxC4w8C665HIbzYw9GICi+o1zlSORfUa89inoXeljouUuxAf+dFHHyUlJYX//e9/gDat4auvvsrw4cNxcWS/s8PX15fIyEgADh8+zIEDBwp07qtXr1K3bl3S0tIIy+cLOioqCrPZzPHjxzlx4gRNmzbNlsbPz4/FixcjLUEc9+7VBu516dKFVatWAdrXxe+//57t2O7duxMVFcWFCxeAnENCW2natCkmk8lm/lq5ciVdu3bN9RqOHz+Op6cnkyZNwsfHRwl8xe0TFAQ9mvsgBg7kpfdjbMJ+wq6BJFwdwWsdHoBWoejTBJVSnSgX/QZOwkxGYAC6f1pT8/uNWEKJ20hJgZdfviWoc8LV1eZ9idms/Vpd8K0a/bBhjs3Qy5Zl2V4/FqIitUiaAN98DOFr+bZlc0LkLIxnVrPuAz+8t3fPPjI4IBE/05k7rMliSEHmmswnQgjWrl1LVFQUjRs3pkmTJlSoUIH//ve/eR47fvx4zp8/T4sWLXjjjTdo2bJlgWaOmjFjBu3bt8fX15dmzZrl6xhXV1fatWvH448/zkcffUSFChWypZk6dSppaWm0atWKli1bMnXqVADGjRtHcnIyzZs3580338zWdwHQsmVLpkyZQteuXfHy8iI4WHNDHjRoEHPmzKF169YcP37clr5ChQp8/vnnBAQE4OnpiZOTE2PHjs2Wrz0LFiywmcD0ej2PP/54vq692JKT+05RL6XdLdNKdLSULk03STGxpuxs7CrFxJrSpekmWaF5tHSaXEEaB1eQmw3IVYbaUj+xmqTDPFl50Gg5aPGsHF0v81pycr905Kp5u4ujc8TjJucZvDJd6zyDl0xDV2iui3eTArtlFiPS09Pl9evXpZRSHjt2TBoMBnnz5s27dr6y5BpZlCi3zBLC7NkQEwOxhwYzI30yurjn2dZ1KyJuDDPSJ1Oj9f9hXrWRmNXX6WGSPGM6Q1rUWtClkxz+MT9OD8mx4zU33Nw0zX3KlOx2f0eOJY7Q6W7vHPNrzmSM6S86xXmwretW3M5WxZv9OJNhsxnFvPMcsz8YXPALU+RKSkoKnTp1wsvLiwEDBrBkyRLKlStX1MVS3GtyehMU9VKaNfxZs6Ts3VaS/KkAACAASURBVFvKqlWlnNfyEVlpkk7yWjWJcarktWqy0iSdXPdg1Ty1aJ1OynLlsmvWNWs6Tm9VoHMacJWfLwYXFynHjct5vxA5n2PcOCnbPjhHiok1ZSdjVyleqyorTdLJaIOWKNqArDURGfFg7WKp8JdkDV9ROlEafgnAxwe+v68X11u8y+RrU7gmKmtysf5O0CdzTVTimNndoR+9PRkZUKVKdjPxwoU59xfm5h6elwOJNf8lS7K7e1pxdc35HOG7Ytj99Cyqb45ix5Yt+Ic/xzVRiScCdbxphIEB8HqUF/EnhlkV/jsdq6RQKOxQAr8IiD00mN4XD5DmN4WbHT+E8LVw8SF46CfaxFfFJTyUyPoP2vr8cuPixewdr7n1F+bmHj5zZuZJTOyxTldo7XPM7aWS0zkuucQyt0MkF/cYMZvh1et/4xIeSrW/H2JGV3g8zo13TT/Sht0MRpPyKqa+QlF4KIF/j5k9G5w/OsnOFkk0iTVCk++g73NQbw9NTlcmoe4lZjCVXju8bF40uQn9nLRyRx44uaW3eu2MHZtd6DvyJsz6UqlZEypWhKFDNbu9I9wSQ2xhIACM8/owQz+DpAancN/3CKFtk3nd0JMe/ISnYSF1fV8GVEx9haKwUAL/HnP86ijeujaBR6Je4mjL3+FaTc3P/noNznxyitejvHg3YD/bDbciIM6cqc1ylZVy5Qru1p2Xe/iSJbByZf68Ca0vlZUr4fp1bRSvlNzyz8/hHFZiHqnHu0P/oE10P+IbH+XhbY/xbsB+xndoxJSAE4w8rfmKq5j6CkXhoAT+PcDet737qg1kBASxnn7wt5c2b6wZqHiJ6x2W8odpHK2iJhNd/5aPdFAQfP55Zrt5zZqwfHnB3brz4x6e09dBTuTk3aPT5f7SiE2K5XWPr0nYtZCeUc+zu/OPVDn2CEu7/83cqAbMMMXwDGEkJJTuKAwFobKDodMfffSRbTDWihUrSEpKutfFyoQKZVyMyak3t6iX0uKlk9VjJQMhRxsCpZhSTjIN6TTVSVadhOzrX08yDdmyw3AJmqdNSSGvUMw5ER0tZa1aUn5bdZCUICs/6yOZjvQ09rNl8l+Dj6zr+1Ku4wfuFQXx0pk1S7s+e6Kjte13QqVKlXLdX5AwwgVBhTIunigvnWLGlCkwK2U8aThjRiCQNOUPpCWMcPntrzAtwoudLZPoFNuSQw+eRa/XOkVLCrcbNiY2FiIj4Xt9H941+JBc709IrciBzhtwD/RgvsGLKQEnePX0Vpr7dyFlRLMS04Hr4wMDB2pjLUD7HThQ217YWCchWbNmDXFxcQQFBeHt7c3169fZvXs3Xbt2pU2bNvj5+fH339knmlOhjMsQOb0JinopqRp+1jkvFjNOmu3U3mi6yfJ9hspKr+nkVCOy2kS9rGr4SrY2zJH4zpJQvHzP88PtTKZiD4ZoycRacq7BS84zeEmm6LVJVKbo5TyDl3zE31MyDXm/f/88vxruJgX1w7d+wUydqv1m1fhvB0cavv0kJPbadWpqquzYsaM8d+6clFLK8PBwOWLEiGzHDxs2TPr5+cmMjAx59OhRWb9+fXn9+nX5+eefy/r168sLFy5IKaV8/fXX5cqVK6WUUl66dEk2btxYJicny3nz5tny3b9/v9TpdLYyuLm5yfPnz8uDBw/Kxo0by/Pnz0sppS3PrBq+df369euyQYMG8s8//5RSSjl06FD5/vvv2/JctGiRlFLKDz/8UD733HNSSin79Okjt2/fLqWU8urVq/dsEpiiRGn4RYijYGPf0of3ecWWJtxQG9F8LS3DZzA9RrA06j6uB4xgL21gRwhubncUbqVIuNOwMRUaabF4njJdJti0H8+d/tokKs5pvN7/b3b6HEAX+xyTNhqwD/9S3CMtG40wbhzMmKH9Go15H1OY/Pnnnxw8eJCePXvi7e3NO++8Q2JiosO0KpRx2UDVSCHiqPOyBz8ygXkABLOA0/VPcyNqHQNN36DDDCYgKgbqx+JyzngnwRSLFKv//+3w6YgQRo+GyczE07CQg223U+OEN5cM+0itfg4uN2D2xiq8ZehC/Z6zMRi0CKFCaC9W0F6uQ4fCjh2ap1FxICYGli6FqVO1X6Px3gp9KSUtW7bkl19+yTNtQUIZO4p6ea/JKZRx79692bhxI76+vmzatCnfgd7KCkrDL0Qc+YtvfmYBT3ToxgTm0YWtbNyxlb513uKnoAWEhlq04gQjbokhdxpMscRi/UL4vnU9JgfEMyeqAfVuXtG0fDNQLZGJgX8iAgJxjbliiwJqFfZWpISPPioemr7VZh8ZCW+/rf3a2/TvFvahgZs2bcr58+dtAj8tLS3HiUtUKOOygRL4hYijTsqKJx5hg982DB3+wza6YOjwHzb4baOV0yNMmaK9JFxdNR/1sijsrQQFQYenYpnXMZLPWlTlULN4nP54HKfUyvBvPczNvqPByQZ8dCx3aS6lFritqIW+tUPaqtEbjdp6bOyd5ZuSkkKDBg1sy/z58zPtt3bAent7k5GRwZo1a5g0aRJeXl54e3uzc+dOh/mqUMZlhJyM+0W9lMRO29BQKYfrQ2U8bjIDIeNxk8P1odLY51nJNCGrjmglmSaksc+zd9TJWdrhxaZS5/+cnMcrsqIxWDIdyaiHpdOEGnKzgczun4ZoW2f33a7P0ho8TblGllxUp20RErRjPMvThmIgASckBhJomfE7W75dgbuuE/+6/Y67rhMx33xR2PNbl1gcdbwO+ucPXt/oxluGLpRru5ipW6FS9SOw/TX6BriAwWIXMcRAwEA4nd3XsazWp0KRG6rTtrAIC4OPPkKQ2bAcbe5Cry6P813GdjjZmXjX7dBhPuwKzpZFWYsZY/Vqsr78rNExly2DiH9rQ8tA1kalYTSBMf46TwRMJ2XbW5qQjxsHbZdmnmkrC2WtPm+XFStWFHURFPcIpeEXFi+/nL0XEejRoQ/fGzdpc9B+/rP26zdBE/pZKGsxY3IL1dxp8mXWNZ6IUWq+nkbpxobG/8cTnkepEDcMus6gQtwwJtbIyHFClrJWnwpFXiiBXxiEhcGFC8xmIjF0y7Qr4sH76Lbp8Vsa/a5gTeg/mDleyB3Ob10iyS1Uc4hvCMZRMzMF9TG6tOCVbStxarsUtk7Fqe1SHr/ehy2jwwp7vnCFolSiBH5hMGUKs5mIM2kMJNIm9OfzCgdXJSCymm92BcOqjYU5v3WJpKAhGWIWv8qAJ8yYo6IYGvMQzlGhDHjCTNqeV3nlFahevWzXp0KRF0rgFwYnT3Lc90sm97lAgGEcA4nkWb7gVUMf6vQdQLTvnmyHWCcUyW9EytJIXqGasxLuXAkZFcFM0498yZMEmZKQURHMc27Me+/B5ctQrZpycVUockIJ/ELAv7sfugyBs8cqlgb+SF1DGCs7/IMI8ufvNr9R/kJmLxJlbtAoaEiGRoeeZ51pEem+C2jVpyNLn/sfdep8T8yO73jQ9WPoM4bLj/fh2U3+Re6Hf7coCeGRHbFv3z42btxoW1+/fj3vvffeHedrH4r5bmKt96SkJJ5++ulc0y5YsIAUu86pvEJE31Ny8tcs6qUk+eHPm6JNzD2uQyOpf62ixBL6mMkuct5X0dkCqil/+9vEEqUt2oCsOgnJlAqSacg6/v0kk6pK3RRnrd47zJNuboV/+gKFR94+S0afyBwtLfpEtJy1/c7iIxdVeOQ75fPPP5cvvPBCoed7J9dbkOBqedW7PdaAcfcC5YdfiOQ3OFfwOxOYW+41lna+TNrZNqBPBQE9K7/Ku6ON1KunzDeFguWTwHhKx7QIL0gvB2Y9Z3y+RpRPJsM5XesQ3xVsC79QVPjU82HgmoHExGtjBmLiYxi4ZiA+9Qo/PvKdhkeOj4+nY8eOeHp68sYbb9i02S1bttCnTx9buhdffNHmwvn222/j4+ODh4cHo0ePtoVb6NatG5MmTaJdu3Y0adKEbdu2kZqayptvvklERATe3t5ERESwYsUKXnzxRQBbSGNvb28qVqzI1q1bbysUsz0Gg4GQkBA8PT1p166dLUSDdSRy+/btCQkJ4fjx4/Tq1Ys2bdrQuXNnWziGrHVixWQy4eHhAUBGRgYTJkywje5dvHgxixYtIikpCaPRiNEyzNo+RPT8+fPx8PDAw8ODBQsW2PJs3rw5o0aNomXLljz22GO261q0aBEtWrSgVatWDBo0qEDtwiE5vQmKeilqDb+gIX+jo6VkwFBtVOhUZ8kUF8lrVeW4WdF3POmFIjPRGGUtzsmexvZafVuXsR6yN+slSFmjdbQcvX60Q436dr+4Chwe+US0rDW7lpwaPVXWml0rm8Z/O9yN8MhPPPGE/OKLL6SUUn7wwQe2c8TExMjevXvb0r3wwgvy888/l1LeCm8spZRDhgyR69evt50/ODhYSinlt99+Kx999FEpZXYN35HGv379etmpUyeZmpp6W6GY7XFzc5PvvPOOlFLKL774wnYdw4YNk71795bp6elSSim7d+8ujx49KqWUcteuXdJoNOZaJ/Hx8bJly5ZSSimXLFkin3rqKduXgrVOsmr41vW4uDjp4eEhk5OT5dWrV2WLFi3knj17ZHx8vNTpdHLv3r1SSikDAgJs1163bl1548YNWz1kRWn4hURuPuKOGLRwPrQKhbRykFYJfpoBCD5N7o9PwF2OmFXGiK3ekwDDOH5sdwSnDB1ItKX2QX7wX0eAYTTJfQYQfig8m0btKIT16NF3J/aO0d3IuLbjmPHzDMa1HYfR/d7GR85veOQdO3YwePBgAIYOHZqvvGNiYmjfvj2enp5ER0dnCsrmKHRxXvz1119MnDiRyMhI9Hr9HYVitmK9psGDB2eKGBoQEIBOpyM5OZmdO3cSEBCAt7c3Y8aMsX0B5adONm/ezJgxY2xhmPMK+bx9+3YGDBhApUqVqFy5Mk8++STbtm0DwN3dHW9vbyBzvbVq1YqgoCBCQ0MLJdxzoYy0FUL0AhYCOuBTKeV7WfYPB+YApy2bPpBSfloY575b5OYjnpWgN2I413IqHO0Nv1hcMAMGwpY3KW/4k9ik2Hv+sJd0wsLIMbicz+xkph3fQDnndFKdzPDHE9DoJ9CnkOaznKg2OpyFE4Pkpmz1ntuLvLBNbTHxMSyNW8rULlNZGrcUo8F4T9uBlLcfHhnA2dkZs9lsW79x44btd/z48cTFxdGwYUOmT59u2weOQxfnRnJyMgMHDuSTTz6hbt26trLfaShm+2uy/28N+Ww2m6levTr79u3L8/i7jbXOQKs3q0nn22+/5eeff2bDhg3MnDmTAwcO3JHgv2MNXwihAz4EHgdaAIOFEC0cJI2QUnpblmIt7KFgPuJf7YqFVd/A6g3aMH+TURvyr0vnWsTHhPiG3N3CljLy0sJjW1TjWdkI/bWqcK4ZTu4xEP02Tn97aiGVdRmYzzRjUIfswrUgL/I7wWqzj3w6kreNbxP5dGQmm/7d4nbCI/v6+hIeHg5AmN2njpubG4cPH+bmzZtcvnyZn376Cbgl+GvVqkVycrJtSsL8lisrI0eOZMSIEXTu3Nm27U5CMVuJiIiw/Xbs2DHb/qpVq+Lu7k5UVBSgvWT2798P5Fwn9vTs2ZOPP/7Y9lLLK+Rz586dWbduHSkpKVy7do21a9dmuuasmM1mTp06hdFoZNasWVy5coXk5OQc0+eHwjDptAOOSSlPSClTgXCgXyHkW6Tk5CMe6p+9J/dmdEj2eC4mI+wIUcP7b4O8zGkhviF8POsw9617H+eqJzE7mRHdp2Ku/YfNvKOreYS9l2OYPTtzPrc7/25BiU2KJfLpSJtGb3Q3Evl0JLFJdxYf+W6ER164cCEffvghnp6enD592ra9YcOGDBw4EA8PDwYOHEjr1q0BqF69OqNGjcLDwwM/Pz988jFRr9Fo5PDhw7ZOWysJCQmsWbOG5cuX2zpu4+Li7igUs5VLly7RqlUrFi5cyPvvv+8wTVhYGJ999hleXl60bNnS1jmcU53Y8/zzz+Pq6mqbp9f6Iho9ejS9evWyddpaefjhhxk+fDjt2rWjffv2PP/887Y6dURGRgZDhgzB09OT1q1b89JLL1G9evUc0+eLnIz7+V2Ap9HMONb1oWgmG/s0w4G/gd+BNUDDHPIaDcQBca6urtk6I+41WTv32g9+Sc5p5JOpJ3dOIx/p1u2lbOF5QTtOuWAWHCGy16W1Pu3ZxKNylCFQ6idb3DGnI5mql006jJblXqsoea2qnPdV5o7SO5l/t7SGR85KQVwQiyv30jWyKCmunbYbAIOUshXwI/CFo0RSymVSyrZSyrb333//PSpazgQFZXan7Pf7AUKePMF8gxcA8w1ehDx5gueSDmT7GhACxo5VLpi3Q3618NFum3na9A8Z/zTXTDkAGc4cPROICF9D74OVSD+9LNMxdzr/rkJRkikMgX8aaGi33oBbnbMASCkvSClvWlY/BXL+DivGTDq0hblRDZgQkEgXY1cmBCQyN6oBU45uySZEVq4sPnOrljTyE3IhLAySk2FEBy/MdQ9AWjnKpzqhJx0GPUk6zrz6TVNC5mbvsMz6IlfCPjN3aicuDphMJtuE6YpbFIbAjwUaCyHchRDlgEHAevsEQoi6dqt9gSOFcN57zklcCTbtp1OcB9u6bqVTnAfBpv2cxFUJkUIkLy3c2ql7oUoMSY9+iD5Nz7yw5vx3lSdpGRVxcrqJ8AgnnEG23tiYGLLZ828HKbOHwFYoioLbaYt3LPCllOnAi8AmNEEeKaU8JIR4WwjR15LsJSHEISHEfuAlNJt+sWf27MyTTs+vOZPnDYFsa3uQzlu7sr3tQd41+DC/pgqMU9jk9gK1derWj4X9w+i8aiJepvt41/Qj8yLc8f+9BrpLbkQQSMwDgbYJxbP2LeZ3JLWVChUqcOHCBSX0FUWOlJILFy44nHs4N0Rxbbxt27aV9yIoUk7c/4o/LhV1XF7/IusurMB4LgLX3r051fp7Wu/1Iu6bPbxp6MbMgAM8Uy6SsHeUn/29wskp+1wz/VjLf1jMo1hCGdCN/qzFhzj2O7Um8rW9JLXoYfPtv+8+uHoVUlNv5eHikrs9Py0tjcTExEw+5wpFUVGhQgUaNGiAXq/PtF0IsVtK2dbRMWqKwxx4pHYP1t+cgK7fJvqvjqBO+6qcarMMkabHcNIPZxGHq4RnysVwqWIsoAT+vcLVlWyxcr5mAC6k0IgTuJJAV7byMouYwZtMNb9N57mzGCWXkZCmSfMLF7Lnm9cALL1ej7u7eyFfjUJx71Aafk6MH0+/vQdZ77cdzE7glIFTagU2r76hTbuXzyHjisIn61y4WZnkW4MHTrvxrulHxrGUpYzjdUNPztVPYNaOS7nmLYRmRlIoSiq5afgqlg452HKXLePrXdtwOdkadBkgoPyu8WDqpmbHLmLsO3VBE9L23H/awISARF439ORtpvG6oScTAhK5/7Qhz7zVQDlFaabMC/ychvHLjAx8OwwgxXUvWAJ0pXZYTH/DS8Q8EFjUxS7zWDt1pdRcYO09elLODWRuVAPeDdjPm0Z4N2A/c6MacN40MNc81cQ0itJOmRf4OQ3j79uhCzv91qFLdSZ6ZQZ9N3Umo1w61wYHEt5b2euLE1k9eqYucyX43F+Mi4MZXWFcHLyU9Bdn9ZnVd70eatZUA7AUZYcy32mbk3Um5kEzzY66seQXE0YTGE3b6EdntnpdpNHIYjJdmcIxQUHEpBxm6Yn3mLrVzNL2Tpx1f596dYJwC7sVgTMoSJsDN0TFtlOUEcq8wK9WTZv8Oiv6jds4Mng8nFoGZIBOx9etPdTw2RJATHwMAy8vI3L0ZozvGjHGx+Af2genny7j0X0B/znyNw1/G8SQVcMp33Q/+12cuVRjMxuDNuaduUJRginzJp3x43PZvmQJpKdrhuL0dCXsSwiOIlXOrNEfs+9k9p/1Y4IfPNurAhUDnqCO7gdW/TOBGpd6FHGpFYq7T5kX+DNnwuTJmocOaL+TJ2vbCzoSU1E8CPENIWmnMdO9Gz19BxtXZZDW8ms43pObPitwu5rGcZ+fYNNcVv0nWN1fRamnzJt0Zu+YTQ/DFXRVHmDGlZeZUmUhRsM5Bn9QjfWTQmwdulbvHVAde8WdrH76CQngwkm40A1dXHvMXWfB5QYcqJOIe0J94nf9H6Dur6L0U+Y1fJ/DVxhwbA4La7gylbdZWMOVAcfmcPOTKwWa01ZR9Fi/yIYMye55FUEg/Q0v4dJ2Pk2ONYRqiXCmJfGuSfTpYGQwYer+Kko9ZV7DZ+oJZMUIREAgxKUh2uqRkREEmCJZ6yC5GnNVPMlr9O1UgxECAnnmUBof+ZxCd70SGTWPUzl2MN/6rWI6sXDfDhLco4E/7mnZFYp7RZnX8GPPurLOtIiX4tKY0RVeiktjnWkRJ3E85FKNxCyeOBpPYc/x+pd5dNNETNXhiU2dyTg4FJxvkFznOE9s6synHVLAZymYuqu+G0WppdQL/Lwe3hC3CDBsYWlbmLoVlrYFDFt4oWZEnpNwKIoPeX15uewN4amQmfRY9Qobdm3hkY1BEDsWGv7K+g4mTlUDYsfSd2dgrhOoKxQlmVIr8MPCoFYtzZ6b28MbMyWIgQMhMgrejtF+Bw6E2HeD1FR4JYjcvrzs791mvT9zeZUddGbcRg+40hCqn4IrDRm30YOvLnVXfTeKUkupFPi2GZGyhsD1nU3TB+bSdZjBpvLPO7WPB28O0CJgCoFRuhH50GRiW1RTs1iVIHKaFjE0NPO92/j5OYLLWcZT+L8I1U7B5Ybar/+LOOE4VKbqu1GUBkplp21O9tzWp53YF/AekVENCDYlMF9UZ+ONX5lb+TUwfWVLZ0RFty9pWAW6dYITV1ftJZDtJW3ZMH7VEJb6ALFjqXalOlcaHGKpzwYkwEbAEKPNqLVDi7ug+m4UpYFSqeHnpI19ZfrA4STkwaEf5JiX6sArOeT7iywoiNUP1obYsYzb6EHg6XjKuW6FP55gtXttcI+BgIFwWpsTUfXdKEoNUspiubRp00beLm5uUmpW+8xLBkJKkJ2NXSXTtV8JUgrhMJ/QUCldXDLn4eKibVeUbJrWvSzH8aGUIKPpJqsavpLlJlaVNY3jZJWJ5aXe8L0ErS2p+60oSQBxMge5Wio1fEf2XIDTTq7MN3ix3W4S8vkGrxy/13MKnaw68Eo+fyRVY0loNXBzoytb+Mq0GF3c81zoupT0uBdYZ1rAYLTPuR071FeeopSQ05ugqJc70fCl1LQyNzdNebdqafOmzJFiYk05z+AlJch5Bi9tfcoch3kI4fhLIYcPAkUJJR43GW1A6idWkxinSpeJLjLagIzHzeH9V195iuIMZU3DBzt77tjxmBKdCRoi2LzlW+buMxIsL4MQBMvLzC33GpsrOvbMyKmjTnXglQ6s/TPHDQkMCNDDkf6Ui++AiApnQICeLw3V0Rs2ge/sTMeprzxFSaXUCnwA/8luzN+7FDIyANi4YwtcXYP/M9LWsxf8zgQ2TnE8A0ZOrn6qA6/kYz+15Wf1vZBREbx3cB+pQ5/gWvtl3Ihax+se3rgEPMEDDaLghWaZjldumoqSSKkW+PExPXnVD+Z30Nbnd4BX/bTt+cF+smw1+Kp0Yd8/c3jH56w2LSPYtB/D0RbQ7Btu9nwTmq+jxsmmnGsWR+34zAJffeUpSiKlV+D36MGoXVVg01xe9YMuIzRhz6a52vZ8ogZflU7sNfT9tCaUIUhgeUQtSGoN9XeTppOYmh2kZWwnTm/ckOl49ZWnKImUToEfFgY//UQwC5i3KxFOdmKbG3CyE/N2JfKK0+KiLqGiiMmqoa8miHAG0d/wEtWqH4QblaHCv3DtfhZvdM40ArdmTfXiV5RMSqfAnzIFf98umstlhwXgugMSOoPrdj4ZvJ6VFUcXdQkVRYyj/pmpBiMiIJAaJ5tC+WRIrgWVzvPEiPO2NOXLQ6t+MczeMRuFoqRR+gT++PGQkIDudBteHfyXxYwzB/eYZyHVhT+anGB4q4eKupSKIsZR/8zfjS9T9WQzmxln9C9V4Wwzrrkewj3Qg3AxmPTWs4mp04f5r/oof3xFiaN0Cfzx42HpUmb7Ah7h8G9DSHXBveIe4gNfRZi6wtHe4L5ZDaJRZOufWTYshMRaN2ge25mDG7cz6PQJKlRJhJMdSGhymGf7p2P2m0Sn6J6c/dWowiYrShylS+AvWwbA8dOBxHicw6lyIhzpT3zXVaC/iZPrVvjlVVi1UcU6V2QjKAhWdjxKj1/HYMKNbiaYGdUIav0Fib6keq2hx++1+WPXJ3Rns/LHV5Q4SpfAt/jbDzKdRRcehbPuJnitggxn0N0k4+cZYMocB1M9tAp7goJg0T9BGKQJIQStTTXQ/9UD3LYhEh5h80PpvG7oyWc8Dyh/fEXJolQJfLOTDgAjWwgijFT0IABdOvw+FDq9q4W9zYJ6aBWOiHkgkD4djKR7RVAnqS7ygYM4b3uVtwMOc9yQAIYYqvmrzltFyaFUCfyVFUdr8cyBGI+zgBOkVsQptTw0+Rq2va7FOM+CGkSjcER4byPm7m8xdlMjLv3wKXqRRlrX92i+rR/hHqAPGEBd6aMCqylKDIUi8IUQvYQQfwohjgkhXnOwv7wQIsKy/1chhKEwzpuV4deW8DLvs9ngxAmP3ZCho+mqOZhXfae5YnR9yxbj3IoKlaDIiUYjLzPTNJaoXb/wX9OPVAwPo4K4QVyTs6xuXpGKUZ9Td2OGmv9WUWIQWnC1O8hACB1wFOgJJAKxwGAp5WG7NOOBVlLKsUKIQcAAKWVgbvm2bdtWxsXFFagsTkP9kcd6aCacGsfh4CCosxce3Aw7J2qeO5ca4ZYYkvusSAqFhdmzwefKZoyzehGT0Znexoe53nU++q2T2BTzK+7E444p0zFubprXj0JRFAghdksp2zraVxhTHLYDjkkpT1hOd+lIqgAAIABJREFUFg70Aw7bpekHTLf8XwN8IIQWZLgQzm+jXGIPbvpNgE1z4ZuPocN8sK6bjGAyotOBKb0wz6oozYSEAPSAd81g2IJo+xtsnYq+7TyIT8HVJLIdo/qEFMWVwjDp1AdO2a0nWrY5TCOlTAeuADUL4dyZSN0arAl3vwkwosstYb8r2JZmtBpkq7gNYto9wIAAPc5RoTyauou0RF+eGKQjwvCAlsAQA33GgO9s1SekKDDWUN1CgLOz9ns3+oSKVaetEGK0ECJOCBF3/vz5vA/IgqsrmnA/2Qnctmm/FmGv08G4cbBkSSEXWlEmCO/5BDIqgnWmRTQ6fR86161ccypPmIcTqwx10AU+AR4RlL/go/qEFAXCPlQ32LzL70qfUGEI/NNAQ7v1BpZtDtMIIZyBasCFrBlJKZdJKdtKKdvef//9BS5IUBCaGcd1uy12Dh3mM3kypKcrYa+4fRpV+YR1z1TB6BZPoOksMmIt+gwzP7Q6z4hnzmMWOrpGvsIAb6PqE1IUCEdTqVop7HFChSHwY4HGQgh3IUQ5YBCwPkua9cAwy/+ngejCtt8DRF/XbPaVd85FrPiZyjs180709fmFfSpFGSMkBIwze4DJRHe3eHqaMkj7bQJp5dK5Wc5Mj1+bs+/EK1Q9+1dRF1VRwsirz6cw+4TuWOBbbPIvApuAI0CklPKQEOJtIURfS7LPgJpCiGNAMJDNdbMwuFRjM+MazeXqD8GYzXD1h2DGNZrLpRqb78bpFGWVkycxGqZCu0WQ6oJILc+P7Y+QZtjGoN0Ti7p0ihJGXn0+hdkndMdumXeL23HLVCjuBTEd6vBEt3+4JirhFP4l5v9v797joq7SB45/zgyDCN61vMOga21pkSmFd0chirS0FRTRXDez7LLbz5Qy7WJFJl5+9dtWzWpdE0zB0swoAhkLL9jgqmlleWHwnpe8pSIwc35/fAcEGVAEHGY479drXs7AzPd7Ju2Z7zznOc9BD8OH4CfOM31ZF7JDtvHJJ64epeIuinL4ztI6vr6V32WvorLMWjVpqyjuYFnYYPJOBtFz3UDs1oHa6u39fWhwrD1Terbg0rIViEAzDaKfIPq9+OIKDLUaVylS8t/E1KkwZgw0aQIDSMeKERs6DuiNzAhN59CVM6JVoAK+olRSx4Yf8Nf0W9g4IBUR/SDY9GA081v7XG45fYk10QvRxdzHeeNyVvxfMGPHolbjKsVKVuUU/ZtYvBj6Rj5OtvEc+whEh2S3LZCXfzjH3nOPV9u5VUpHUa5DJ7GbAyFJXAqfhs5mwG73BmEDrzztCQW+9F46mfXW15y+Xq3GrZsSE7Wr+aLSyyJDWcEzxkgeiTQgk5dzr9WXDCM0iBzMyu+aYco6es3nUCkdRalmJ5p04pMsC2HbW2L3KgD9JdAVat1ZBYRtuoNd1qcZgPOCAbUat+4purJ3FuxX8QjbrM+xMrmAi5FjSTNtQBc5nJXJBZi+P1ZtY1ABX1Guw3vvwVsdepPeqZDA3LZa/yZ9obb3Qr4P6ff+zBRjGM/wntPXq9W4dY/Tevte8fw07m8MDunPJObwD+s2CnYPhn5vYMtrCtb+1fqPRQV8RbkObXqa2TFyJkEbHySn1SmQaDe7F2TEoZM2Xhmxk78axxAqSl/lqw6tdZOzb3VdD+n45SYbq8MzaRkSy46QtRCUABK8G+5niPHvmGM+rLYxqICvKNfBctjCWxFT2BW2Ci+vAgy/hhFwoJWWx+//OrZvp5O3czR9uzxN39DIUpulV7bMTvEMzi7UP7O+x5xPOkG+L0fD50D48wA8lNqHL5faEWPGsqyBvtrGoAK+olyH2F6xFNoLefTuEfytfipv547nVMY8/Art/Cm3JehtsDOazNtO0Hv3aaxxicWbpatgXzdFRFy+P4B0cjASQK7WyTdrYvH8T+D+tnyRtY5vvJJZOWYlHfuW3bTpeqmAryjXqa01ltRn3+eDqSbe+GkYSUcX8/qyLuxpd5I7vDdijxxJn+RnMFkhZVQCvfy2Fpdjqtr8uiUxUSu9BOjcazTZxnPkEIgAZhrvgR6ztJSgzYsc/0NE9Ajlh07DMAWaiO0VW23jUGWZinIdnK2ODNens8UWRFfTYNL6bab5ycZctEzkjaxTzOAl/o+/M6ZvMK17pXPi3ZRSr72eFZWK+zAatXr7jr0e54itPaLP2+iTE+nEbrbETAOvAiZY4E8/BfF89G7wvsCc8DlM7DHxqse+kirLVJRq5qziItUWyp87L2Zjj82EbWvJyYYFXAh/lRdDbmaKMYzE6GUUmJ7nwLrQMq+t7q6ISu2Smwsd2c2Th7Zg6PMmMvNFzkc+xpb7loNXAQ/tgnkpMFGeZo5uOn/WDSJ9X/X3AFNX+IpyHXQ6bZVkKUYzREYxZ6eNGV1OITNf5OSAf4LhPF52sOlgUGof9mbN4ye6lDmmEGC335jxKzdOfDy888X/cKzJaertfIQ3eJkXInMpPNceWu1Ad/hO0hc2wxSQUy2r8dQVvqJUM6el0W0ttPwuiYkx/2TKqmBO9vkAfh4KUkehHoz727Imax1jWXTtx1TcXvCZdM4cHICty2dcGD6Kycyi8Eh3aLUDbDq8m+2q9vLL8qiAryjXIS5Oy7uXsiGW3zabEKNimLr7O0ZlN9BqqoUdcnuR43+YwSH9EdhpK0p3xFK1+Z7LlDiONda5+C5LACGwj4qAjumIQh1+BQLduqnYRo6p1vLL8qiAryjXISZGm2QNCHD++zzjJj4LydUqL/J9wfwGTVOnsDo8k0kh7fi713xVm19HxLfLZdmgdfRiI+x6CLy0FhziWBf677wZQ984TL6vVWv5ZXlUwFeU6xQTo6VcywR9Ry7/puM3c9OJprD9UYiM4tTRgZA6m5u6xbEuMg6rleLafFBlmp4quLAlH3f2Jm3kP6HLMrAZQIK99Q+kn3iaVwxx9OlfWK3ll+XxqvEzKIqHK7Nkvq0FkpPItZogZA6ET6KV5WGO9pwNpwM43uIUoVsAoxHz1BgWXmrM6hdiiyt3iloog7rq9wgDEyjI2ALh2kZ/hkJBgU3LBxaa3uYXv9W838t0Q4airvAVpYrKTLZuiNVWTwLTs07xUGofjgavxtBoHwQvoLFlOBOzwCxyGbpnFjv+1USVaXqQ+HgwT00v/spmmWehU/OjNDvqDwIKDJKoTe3osWEB3XxH3pBUThEV8BWliuLiwGBw/rtVDOWLrHUE7m9DQatfEL915kzntdxnupehkQZk0nKm7jI7fa1qoeyegs+kE/VWEObcQJCS4NPfcPjHvlxqdRJfgy/1veqTet9vxC1qx+Zp79+QVE4RFfAVpYpiYmDRImjevOzvttKN1iGTsPofoncu0PJHdEfuIK3fZvKzn2WV9f8YznKnx1Vlmu7Jsj2SKcYwokjiFaYzxPh38qOHYbOdZ030Gr4c+SUSydDlQzHnOP+wrykq4CtKNYiJgRMntMVYUkJCgmMyN2Quh8PfQabOYeniAJ60gL2jGfaEIbovAOM6BJIFuqdKHU+Vabqv4J9OMyNyOw8YY3mDV+jYJQ50duLWginQhCnQxKrhqxjeeTiWwzcunQNqpa2i1KiIxAhCO4QyscdEHgmaytrwWZzV1adeXj3qff4+wrGr0dTb72BLh3MU/CsHf38t2KsJWzdlNDJXNGFS5EF6Z3dhffedzE5ux0R5+obsa6lW2iqKi6TEpNBy30SMRljZsDEXk7/gph0DuNTkOO1vn4tMXk70ACObgnfQct9tqoWyBzDHfMgMaxqjshuQ2e9bRmU3YIY1jUdOfejyslsV8BWlBhV11czNBTbEUmANZ1nKafSWx/gxeD0XhjzFb+2t6C2PsTjloquHq1TCldU4WpltOrO2hjJl6mK+uvcAL38LKfccwNhhMSvPhiLl5bJbVwR9FfAVpQY566ppYh1pKXvhTDsKmxyFM+1IS9lLf9a5ZIzK9bmyGsecG0jUW0GE3j2bGY1nkjQ+ndczJF7r0sn+y0xtQZ6Dq8puVcBXlBrkrLTyHA2YFnESGh+E0+2h8UGmRZzkvGhw4weoXDdT4jiSiCquxokiiSSiKNwWx/gmSYw1mdDp4LfNJkhO0hbkleCKslu10lZRapC/vyOdU0KHiPs5GbwCveUxXkrx582/rWFjsAUTXbHodODvj3lqDJbbG9/QGm3l2sTHg8+GdC62y+UekcsE63ze4BUGkg7GdcjT8M5zptLf7Kym4sV4RVxRdquu8BWlBjnrqnkycAdYxrMk5Tyv8RqhGQOg0IfttxzSUgMil6G7Z/HxuCblTvKpLRJdx2dDOq+vDsJwKIihkQbmGrtSn/NkGS8yNNJA4JGWZdJ4V3JZ2a2UslbeunXrJhXFEyQkSBkQIKUQ2p8hIVLWq1dUsS/laP4jfYxfSp/JDeTLJmTjyQbpZ1wtR/Of4uf4+mrHKTqer+/l11/5e6VmzJwp5XjTr3Jcrw5yjjFINuKUNBi/lkxuITsNuVf6vKSTjYyfyYGklfq7KXkr+jdQk39XQLYsJ666PLCXd1MBX/FkJT8E1tJfNuKU1D/aV/Iasp5psmzEKbmW/nKA8SVJr5kStOdLqf3pLJgU/V6pfjNnSvlgwHbpyznpZ1wtG082yLZGxwfykBjJa8hRQ5AZL6XJJk1c+/dTUcBXKR1FcYGi1sp2OwxgHa8a+2Nrsx3y63Pp3vd51dgfYVxHduQsOh5qAlye5Ctvsk/13qk5wWfSycz1RwDSauKP5DUcipwEQ0dD0FLCtrUkpZMOxul5772yabzasnJaBXxFcbEnhtXjlRE7MSxbDku/BAkvjvqB8BgByct5w6qV8xVN8pU32ad679SM+HjgvfdYxVD0FHIJH2zW+2DPAxCUQNgPN7N11Q5GHppJ1Ioo2vQ0F2+OU9s2uFEBX1FcoOSk6zf2gZyXfhjIZ7T1APzyEAVeEtvxO4qbqxkMl68QnU0E15YrSE8THw9en69g0B138tS4LTSM7o/NmKnV1N+eDGdas7bLb3QdFEz7oEkkDUvCcthS6htcbVo5rcoyFeUGK1p9W1TJYf3sS/TGbzBEDoI9BXCngG2j0XX6Aow/IKzwbuFTwDzgcvCYOlVL46jeOzXD74U/0/Cs5HSzewk/YWX1zTYwbIdOA0A69p9tdAR7YT0yOp/gh8/MzGlrIjbmxmxmcj2qdIUvhGgmhEgTQux2/Nm0nOfZhBDbHLfVVTmnorg7Z6tvbdb7aLX/HpYEgfghmm6rnsUrOZGhkQbWGeGCrEf8ozuLn19bryA9SfPcO/mt5a/k35XI6v7b4UiQ9gsB6GzarcAXY+J72JZ/wW9eFpe1TLhWVU3pvAislVJ2AtY6HjtzUUp5l+P2UBXPqShuzenkqtHMLx234v3jSOr9aTVbjGcZa81BJi/npbYPM5k5jLYvuuFjrcsWLz+B3vIYUm8H77Pgvwmk0AK+49Z001is1nHaoqoNsbV+p7KqBvyHgcWO+4uBIVU8nqJ4vDKTq45Nz/lkDfnJidx56g8M0fcz3/hnAqwdyNqwksEh/ckYOdcl461zHBMsJsyOnkfttUgpAJ0ESfHtVI//lOqRA7W7WqqqAb+llPKI4/5RoGU5z/MRQmQLIbKEEOV+KAghxjuel338+PEqDk1Raqcyk66OTc/J0XK/kTsFhd6FNO0xjR0EERjyDF+EZzIwR7hmwHVIfPRWzI8lFPfDeDaiEBofKBXkAYyW+6DADwznYeTgUkG/NldLXXXSVgiRDrRy8qtSX1yklFIIUd5uKgFSykNCiA5AhhBih5Ry75VPklIuBBaCtgHKVUevKG7oyklXXVYsNtvl3/tkPckgdvJF+HoajQ0ix38Hg1P78HzXLq4ZcB2y99RY4lq/yiprHtMiTvJj8Hqw6xFI5KWGUP8M2AXWOzdDxusQaAa/Y9qHttVU66ulqrTjlRDiF6C/lPKIEKI1sE5KeetVXvMfYI2UckVFz1M7Xil1hU6nrcUszU7gWH9yAg4RmNsW66L9zJ6jY+JEV4zQs8VviCe4TTCmQBPmDoKhwwwUZE7jYu93kedvxqvZbmautdE1qz9hER2x3/IVcs8gONWR5rtiadCgdlVL1eSOV6uBMY77Y4DPnZy8qRCinuN+C6AX8FMVz6soHsNpCiDkHXL8D0NuH3L8DzP4rXdIT7/hQ6sT9n4XzNDFQzGHtMKUA69kFnAh/DXkngep1+AQ3yTa+J8sCCSHfikjkO8cgjXvY/g+lnffda9qqaoG/LeBMCHEbiDU8RghRHchxIeO59wGZAshtgNm4G0ppQr4iuJQJqcfMhfCJ0HqbFj0HXwzmy/yJxH6spq0rQ4lF701bQpN/30SuXgRQ/v+zismmNqnAWyPgbuWoMt+nIvW+4khgUCsZBAKQPPmsGhR7Q/wZZTXZMfVN9U8TalLEhKk1OsdjbZGPiAJmVOq6dacjXPkra8/IDNeSivVejPjpTQ5c6arR+8+ruw0OoA02YJjcg7PyfqmiZLXkAwdVdy5tFFsPWkI/Lr478EdOpKimqcpSu0WEwOLFzuu9JemQJaWrC+aBJzYYyIXjp8nImMzT7TTYw7QttSLyNjMewf7Yc4xE78hvvh4ql++cyUXvbXmEBF8SRJRTDf2paD7R7B9NNyZSFzmH0w3C57501cURo0qU3rprlTAV5RaIiaGCptuDdpxkrzwl1nUuD+DIn0Jj+hAXvjLdD2wn6gVUQS3CSYxEVq0gFGjtMpC6eJNs2ubohr5ILbiNTKcSdG7eSfExqXIMRQmrwSf3+F0W6YO0LHMeDMznjAhl2vbE3rCf8cqVenUJFWloyilSSF4OqQj88P3wW9doOUOuu1tSm7rU0zxnoVl7yRWry7btqGIXq9NLtaWahJXMBq1D8DoXnfxWeOuXAr+D9gMkD4Dmu2B4AUY8r0wmF+mtf4Aezd8UOYYAQHaBG1tVZNVOoqi3ECRWe0R+3tCqx1wph1b/nSKB7IDmBE3hksp6RVurWezqSv+Hj0gXJ/OuEPb8emciMHyN9AXQlisFuwLBPU/SSI863anwR7K7lHsTlTAVxQ3sYqHiQgJRfpvpMnRdtD4IGLPQBK6/8EUYxhzz4675mPV9p4v1+Na5i3Gd0hniy0IYe3PyuQCbJ1XwZl2oLeDgP6b7uFT63usZFi559Hra+wt1DgV8BXFTUzofRt54S/zkKU1Xg0PcotlALLjWuqfaMuMyO3sNTouPUPmwsiIqx6vNvd8qayiltMVzVsUbWSSRBRRJDHZ+j32I3dDkwNgFyAhLWQH0419KjxXyVXR7kYFfEVxE7qQ7dzyzXh6nDnClOQgfk9ZRqNdfbng/wM9f2zDxk5NMPR11PDvC738unL+L5fScyp4nLWcLvUtJjGR4LghDLrjTrYaT/EAX7ElYhF0TEdnA4MN6ln+Ct4XWB89q8KqnObNa+xt1DgV8BXFTRyelcLAuxaQvuEb3rKmkUQUI5aPxJD6FquDjzDv9osUmJ5Hl/o2ZE0kIABeWmgm6t34MjtkFSl5JezOpZwV7fP75zZneGrUGSx3fM4Y21ImRR5kScgxuPtDONUOL7uO2LVNuNR5DVgmwPEuNOlsKfdc586513+bklSVjqK4iZI7ZQ0gnY8Yx14C+QsrOTf2QewBG8Hmhd+Sz/hiZH0YpydqRRRJw5I4vNHE1KnlTzg2bw4XL5a+Svb1rT17sV5NUfXNlQIC4MzgVpzeN5SHfvLm28j5nM2MQ4a9hLDpwXCJ+qnTCcnqSYZRD20tNN8Vy4kT2utbtICTJ50ft7ZW6lRUpaMCvqK4CWdB7RgteDWkiVaqub83+K/HJ18weXs93gnxw/fLJI59byouxRw92lmjtvLV5sBW0pXbRsLlD6wNSwXzgwHLk/j8NIi8mGHglQcCwra1ZMKqEB5hVfHrDIbLbROcN7bT1knY7TX/vq6HKstUFA/gLG3xcchJFoTvxSs1DhZ9hy71bfK87bxxz0XGmgtos7kRsmc8ucLM+PHQrJnjhUYz9Iove8BrOGdtVHLRGmiVNEU5/H+lwAQLELyAvCETioP9HbmNSe9UyBvG3qWOVVBwOfdfXm/72tzzviIq4CuKm3AWZJI6NMM79U38siYQRir2o93BrsOQ78XCey6y1Xia5k2zYOQDXBh7Gxfvjqfen80wfAg01bak8PUtfyLSnQJbTMzlRnRFlTS5uWBHBynvaTtXNTkAwC3bgznS4gwdM4exNXJmubtWlWlsB7W+531FVMBXFDfhLPjcunQuPllPs5KhBA66H8PwB2HLeAoMNvK8dBii78e3/ddguAQtdnGhwU500UMQegE/jihu3/Duu54R2MaNg1EX5mFHFN+C2cT8iJ3azlWONMyvl7rRKvl/2dPnU8icom1gUkLRB93V2l24GxXwFcVNOAs+O7iT5URhYh0AviKPVj/10lore+VR4F3IgZsvQmE9KPSGoCXYdZdYO24lcp8JqxXa9DRzyBjvEYHtzrxNfMCT/C/PIYD/5Tm2RiyC4AXcewDWfgxPONI7O2//VdtaUl8IG2KLj3HlB11MjHv1vK+ImrRVFDf21FNwen4ibzGVAHJ52vgX5kd+y83Zj3Cs97+1YAbot43A6/YkLnlrl7gTOszBeGQiwZHm4koeU6DJlW+lWvg/48vhnJHYUxbSm/WspzfyhWb4cY4Wy2yXJ70jnoLADPjXrlKvFwKefBLmzbvxY68uqkpHUTzYU09pV+Ov2KbxGq8TaupJWr/N2obbdi/QaUHfp0DH5E12ZvXQkWeQdLo4ilMtvvKIYB8fD8Fn0knaFsaCYMAyHlLeh4gnIfh9nrRA75GyTCWPM+5SmVQeVaWjKB5s3jwoLIQU/wk8bowmM8SiBXtAlxYHB+4FIE94Y835K/qlq8BmYLfvEiZ0n+D2wR60YB/1VhBRKf3pabkDghfCc/4Q/D49LXcwP6VsSqw87lKZdD28XD0ARVGqh7X1r2zr+xUNjweQf/5WDDm98ekzjT4/tyDl6OPY22xnSdvb8DpkwK/QRqszMCvjDU5kHMPYcAGxsWDOMbNs5zI6NutIbK/Yq5+0lrBsj2SKMYBB1vVcSPGDWwK0ipyLjdiUso25f1rARLSgX5SDL2+xljtVJlWWCviK4iF+87JA8udcsproSSbTeQ3d0QIsbY9g2uBHLOuxGddTGBnF48uGEMmnRIzUMd+wkHv/2IbXpihe//Z1JJJVw1dhzjFjOWxxi8Af/NNpoiJPUz85hQu3r4PGBxB2kD5n0T34JMuPL2TiFa+Ji3O+WMvdKpMqQ6V0FMVDBByMBauWntlIH8JYy0CrpNuGgbzOq/hxnoFtZ1AveTELrElMsmajX7oKnwJBttjM8988z6WCAkZIbdVp0S5aNa06eviY7AFMSQ7i5MhHIXg+FNRj9sdBjM5ugK37B+gGPVXmNZ5WcnktVMBXFA/hrE4f4FVeRQKrGEr6hjS+ss7CQD7/pRv51nD6bArG5ogEeXmSnCNLGfzxjancuZa2xtfCHPMhM6xpNCywgYDRm1rxljWNQ1+uRL91Arn6DKev86SSy2uhAr6ieIiiK9YrV81upA9j+Q99yCyay6U+eQwkHb0xnbR7dyHy60F+ffTiEmnNP+SRrGbwoU3rIV+DrtrW+BpZGocyZepiDNiZ9i182f0A/sbFZBCKbfU8vBfuuvpB6gAV8BXFg8TEwIkTkJBwOVXRvDnMMzyHgUJCSSOSJFYylKnGMPQjHgIpkUu/QmS8hs1LorNB8t17GLL0HMFn0gFtMjd+Q/VH/4raGpfLSQ4oONLMjMYz+X1FOm+aJb8np5dqmeDJlTeVoQK+onigkqmKBg20hmAAGYQSxHYCyeH7tmDaeTO+yxNoyWFkn1mI1Jk02NObPGHANiISloYRf2svwhb8hRdGBVd7n/zyKmKKm7xdIT56K+bHEkrlgMyPJTBr9iqShiURYHekoKwmbRWto2VCs2bu2+u/OqmAryge7sqr2wxCCcSK/4YRZK3ZzhvWDM61/YWw5HFIvZ2z525Ht/Z1kPBwpBcvDNuCPHAvzdp+RW4ujF04l7tmXX0LxWsRFwfe3mV/fvZs2e0Jnxiwm+8O/IMhrcdjpj9m+jOY1US0fpb2e7/FFGgqPY9hNcGGWAwGbdOSqs4TeAK10lZRPFx59eZ3sZVoljKLWJKIAiDCOJm84dEgBOKXB5FBS7VFXIU+TE+8m1db/QXCJ9H0+9n8nnJloWNZiYnwxBNw/rz2WKfTHpdsXXAtm4z4/70fh0744p3TG32fOEhezoVWPyPvXoR3kz18/Ukhpn2y+JxTp2ofdP7+8Mcf7reJSVWo1gqKUoeVtznImDGwecFWpsnpNOYMw0kiiSi2Gk8xaeSvSMPFUq0Zeu+HDf7QOfVRdmYtvupGKomJ8OijzjcKmTDhctC/2iYj8fGwYfUAVoeaocAXQ8Y0CgbEgUH7FJmTChOPlB+93XETk6pQrRUUpQ4rr9583jyYuKQrjzdfxUDM3Ml2+pAJODozCLTma9tHIew61geAcX9bfsxaxAjDiqued+rU8gPqwoWX71eUxzca4ZsX0tm4YTkPpfYBwwUKwl7Rgr0A7+2R3JXVv8LVUp62iUlVqICvKHVAefXmRVU9UsJaGUpUo695sctdgNDKNPO9ofMKpM6O4VQbcvwPMzikP4m2EVc9Z0WVMUUblIDz9QMl8+4fMo4koliXtVrbxlFfCAL0uT0wdFrDUOM/MLcpv4A+opzphvJ+7slUwFcUpdjKZnpsnT8Fuxdzlt7C+G352paAhT4UfJ5AYOoEvgjP5J17bFc9VkVX0Hr95fvOvoE0agT5+Y7joH1y/BGyEPzXa18/JNhab8eWOZULUWNZlmUuexKHlJTK/dyTqYCvKEqxJp0t2H8cSddlL/OI9TTWJmjdJ7ePpmnbr7Fm/ZMWv3bngy4Ni+uBCTgJAAAL9UlEQVQc4wc15amXnyQi7nKdvjnHTI9J8ejKiTDjx5d+fOU3kJMnYQDp5GBEIHk6pAP28BfA5oVv6vTi9M6lAXF0OfcyHftanJ0GuM5afw+lAr6iKMXei46FNe+z1TqJQKz4LE1mU8o25qzx4/cN8czmeY5veptdTb2ZK5qAlFhPNme+fiHGvH2AFuyjVkQx/sFgPv4Y/PwuH1+nKz1h61RiIo/26ky28Rw5BPJkrw7s6vID4kAwD267iTVZZtYd/R/a7L6bprY7iI4prLDBW3k1/eX93JOpKh1FUUp56imYP//y46GsIIkR6LFhQ088k1hibM8vka/SO7sLmd138lDm7Wzsk8mEbJh/r44p+pkUNppEbCUbbUY8/Q9C0zbRtcDC0EgDMnk5DVtlcmjgPPxshaxeZuM3WjImMo/eJ1eS8dHVe/2UV/bZvLk2f+FpaqxKRwgRKYT4UQhhF0I4PYHjefcLIX4RQuwRQrxYlXMqilKz5s273JoBYCXDMFCIDomBQl4knp+tz2jBvt+33JHdm41Zn/JAdgBv9IMHNrdnRtyY4rYM1yoiAgxfezFpyD62EsTK5ALOj3iUQwMW4CVBL22YA2FkpI2C5JWY/226plWzv/9euZ97sqqmdHYCjwDflfcEIYQe+BfwAHA7EC2EuL2K51UUpQYV5dSd7Qy1H3/mGoNY330nfb7tx87u6+kZ8hcSuv9Bn2/7kdD9D6YYwwh+f1yl2hmEnlnBF/tmMTg5hkmRB4kJfAybzgbeFzFsepqHvw/gzX5A9oTiNtDXsmpWlWWWIKWs8g1YB3Qv53c9gNQSj6cAU652zG7duklFUVwrIEBKrWjz8q2rcZYUk5vLOcYgKUFOCOkoeVXIbiHREqQMM74oW0xGrjUicwiQA0iTIKWvr5QJCZePnZBw+fhDG6VJO8g5PCcFNlnPNEnyGlL3ko982YT0e1EveaGRNJpiJJNbSIwZpcYUEFD+e0hI0M5d8vlXjsWTANmynLh6IyZt2wIHSjw+6PhZGUKI8UKIbCFE9vHjx2/A0BRFqYizGvmtbe3clfwij1hPY0fgr9/HQ6m9+UF/G6P5mHRrHPV/vo8RQ7yJD/Eiq9d3TDf24tabZzMmdTARiRFM/cDM2A/jyc2FnmSy9mx3QkY2g5B3aGlcyqV7FkJ+fexI/nN7ffTShpewYc15TGuKFhlV3AkTKq64qYsbnZTnqpO2Qoh0oJWTX02VUn7ueM46YJKUsswsqxBiGHC/lHKc4/Fo4F4p5TMVnVdN2ipK7XBlb5or+/IMIJ3tBJFMFJZe67Ae+gvzmQCjHgB9AYZzjaD+OQrwBq9LcKEx1D8LH6+lddtVeN22nOM7n0UCl8KnQUE9OH+zttK38QGwGRjI2+xM66pt47ghVgv2bR33Kd0X58rxxsXVreBepUlbKWWolLKLk9vn13j+Q0D7Eo/bOX6mKIobuLJGvmgyt0hRy+VO+hyCD0Fy5KeEkQ7pbwNQ0PAsBXqpBXs74HsGw4G7wGrib4d2cODmc+SFT6MAAxwIAcMlbQPyxgcgdTZdEuLx9ilkzrMmmu9ylP04OmFC6X1oq2sHLU91I1I6FqCTECJQCOENjABW34DzKopSA5ylebJ8Q/l2sRWT1PaWTY/8gD71V4OtnnalXhRp9MCJP+G7KINneYc3rGbmLO0EBb7awqq2jm/1QmvWNjQrgJ3W54gxxjJ+fNnyyubNS6dnqmsHLU9V1bLMoUKIg2gTs18KIVIdP28jhEgBkFIWAs8AqcDPQJKU8seqDVtRFFepKCdetLfsqOwGZPb7FoSN4n0VBWDTI3xP0cf4OkP4HAF0tTZFt+nvl5u1AbrcHmzwh5Uh+wkIcB7IQdvcpWS6Rq2qrZhaeKUoSrWJjwevs7N5xT6F815orZXtlL60vNAIL+/zeK99hTVZZl4yhpIV86bWs8fBJ/VN/PiDk+EzGdliNp/8feI1tTgur/e/p/a+d0a1R1YU5YYo2lu2c/2/Uf/31mAXWpT57TZIneXI4Z+lUCfJH/g6ESGXg72Q0GNXcyjwJS98Gie5Cd/M2fx4Mf2aa+mdpZtK5vjrOhXwFUWpNpbDFpKGJbH55ffx2vsM5DeE/SH47BhORtaX9FzyOhz7M+Q3pLDQj7yw10B/CV2hgbUfwzfpDTAkr4FfB0GHdC5kTGT3aylERFxbIFclmBVTKR1FUapVybLI+vWhzYXdxDOZIWiFfWsZQDJRrDZlc7TfhwDU/3Yia8xbWdX8Mf55smx0DgjQWi8sXKj10tfrteqbCpuw1VFqi0NFUWpUUZDPzdWurCsKK9Ek4mv8go+Gf4Ven4cNPT52G96ffcbZXx8s93W+vmW3aVRX72WpHL6iKDWmZO07VBzsAT7v3IaPhn+Fj+4SaZ/kM+erEPLwIX9kDH5dnG9kotercsvqoAK+oihVUl7JZHnsrSw8GDiClMe/wrRPMvG/Gcy5ZxV36oYz+AmL01y9rZwNtlS5ZeWolI6iKFWi0139qr6kK0spr+SsNUJRuuhKdanc8lpVlNLxutGDURTFszjrr3O151ckJsZ5Xn78+LI5fFVuWTkqpaMoSpU4q30v6qN/ZT/96w3SqtyyeqiAryhKlTgLxkuWaGmeJUu0fjdF6tev2nlKNnFTwb7yVEpHUZQqKy8NA3Dx4uX7J09qqZmi1yg3lrrCVxSlxqjulbWLCviKotQY1b2ydlEBX1GUGqM2EK9dVMBXFKXGqO6VtYsK+Iqi1BhVTlm7qCodRVFqVEUVPMqNpa7wFUVR6ggV8BVFUeoIFfAVRVHqCBXwFUVR6ggV8BVFUeqIWtsPXwhxHKhE09VSWgAnqnE4ruDu78Hdxw/qPdQG7j5+uPHvIUBKeZOzX9TagF8VQojs8jYAcBfu/h7cffyg3kNt4O7jh9r1HlRKR1EUpY5QAV9RFKWO8NSAv9DVA6gG7v4e3H38oN5DbeDu44da9B48MoevKIqilOWpV/iKoijKFVTAVxRFqSM8KuALIe4XQvwihNgjhHjR1eOpLCHEv4UQx4QQO109luslhGgvhDALIX4SQvwohPiHq8dUWUIIHyHE90KI7Y73MN3VY7oeQgi9EGKrEGKNq8dyPYQQViHEDiHENiFEtqvHcz2EEE2EECuEELuEED8LIXq4dDyeksMXQuiBX4Ew4CBgAaKllD+5dGCVIIToC/wBfCyl7OLq8VwPIURroLWU8r9CiIbAFmCIm/09CMBPSvmHEMIArAf+IaXMcvHQKkUIMRHoDjSSUg5y9XgqSwhhBbpLKd124ZUQYjGQKaX8UAjhDfhKKU+7ajyedIV/D7BHSrlPSpkPLAMedvGYKkVK+R3wu6vHURVSyiNSyv867p8DfgbaunZUlSM1fzgeGhw3t7oyEkK0Ax4EPnT1WOoqIURjoC/wEYCUMt+VwR48K+C3BQ6UeHwQNws0nkYIYQS6AptdO5LKc6RDtgHHgDQppbu9h3eAWMDu6oFUgQS+EUJsEUKMd/VgrkMgcBxY5EitfSiE8HPlgDwp4Cu1iBCiAfAp8JyU8qyrx1NZUkqblPIuoB1wjxDCbVJsQohBwDEp5RZXj6WKeksp7wYeAJ52pDzdiRdwNzBfStkVOA+4dG7RkwL+IaB9icftHD9TbjBH3vtTIFFK+Zmrx1MVjq/gZuB+V4+lEnoBDzly4MuAAUKIBNcOqfKklIccfx4DVqKlbd3JQeBgiW+HK9A+AFzGkwK+BegkhAh0TI6MAFa7eEx1jmPC8yPgZynlXFeP53oIIW4SQjRx3K+PVgiwy7WjunZSyilSynZSSiPa/wcZUspRLh5WpQgh/ByT/jjSIPcBblW9JqU8ChwQQtzq+NFAwKXFCx6zibmUslAI8QyQCuiBf0spf3TxsCpFCPEJ0B9oIYQ4CLwqpfzItaOqtF7AaGCHIwcO8JKUMsWFY6qs1sBiR+WXDkiSUrplaaMbawms1K4f8AKWSim/du2QrsuzQKLjInQfMNaVg/GYskxFURSlYp6U0lEURVEqoAK+oihKHaECvqIoSh2hAr6iKEodoQK+oihKHaECvqIoSh2hAr6iKEod8f8oF2uugx1fwAAAAABJRU5ErkJggg==\n",
             "text/plain": [
               "<Figure size 432x288 with 1 Axes>"
             ]
diff --git a/tensorflow/lite/micro/examples/magic_wand/BUILD b/tensorflow/lite/micro/examples/magic_wand/BUILD
index b0be47c1eeb..3822e2b05af 100644
--- a/tensorflow/lite/micro/examples/magic_wand/BUILD
+++ b/tensorflow/lite/micro/examples/magic_wand/BUILD
@@ -43,7 +43,7 @@ tflite_micro_cc_test(
         "//tensorflow/lite:schema_fbs_version",
         "//tensorflow/lite/micro:micro_error_reporter",
         "//tensorflow/lite/micro:micro_framework",
-        "//tensorflow/lite/micro/kernels:all_ops_resolver",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/kernels:micro_ops",
         "//tensorflow/lite/micro/testing:micro_test",
         "//tensorflow/lite/schema:schema_fbs",
@@ -81,6 +81,7 @@ tflite_micro_cc_test(
         "//tensorflow/lite/c:common",
         "//tensorflow/lite/micro:micro_error_reporter",
         "//tensorflow/lite/micro:micro_framework",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -159,6 +160,7 @@ cc_binary(
         "//tensorflow/lite:schema_fbs_version",
         "//tensorflow/lite/micro:micro_error_reporter",
         "//tensorflow/lite/micro:micro_framework",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/kernels:micro_ops",
         "//tensorflow/lite/schema:schema_fbs",
     ],
diff --git a/tensorflow/lite/micro/examples/micro_speech/BUILD b/tensorflow/lite/micro/examples/micro_speech/BUILD
index b487b895f7a..522b68e205f 100644
--- a/tensorflow/lite/micro/examples/micro_speech/BUILD
+++ b/tensorflow/lite/micro/examples/micro_speech/BUILD
@@ -52,9 +52,9 @@ tflite_micro_cc_test(
         "//tensorflow/lite:schema_fbs_version",
         "//tensorflow/lite/micro:micro_error_reporter",
         "//tensorflow/lite/micro:micro_framework",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/examples/micro_speech/micro_features:micro_features_test_data",
         "//tensorflow/lite/micro/examples/micro_speech/micro_features:model",
-        "//tensorflow/lite/micro/kernels:all_ops_resolver",
         "//tensorflow/lite/micro/kernels:micro_ops",
         "//tensorflow/lite/micro/testing:micro_test",
         "//tensorflow/lite/schema:schema_fbs",
@@ -364,6 +364,7 @@ cc_binary(
         "//tensorflow/lite:schema_fbs_version",
         "//tensorflow/lite/micro:micro_error_reporter",
         "//tensorflow/lite/micro:micro_framework",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/examples/micro_speech/micro_features:micro_model_settings",
         "//tensorflow/lite/micro/examples/micro_speech/micro_features:model",
         "//tensorflow/lite/micro/kernels:micro_ops",
@@ -386,6 +387,7 @@ cc_binary(
         "//tensorflow/lite:schema_fbs_version",
         "//tensorflow/lite/micro:micro_error_reporter",
         "//tensorflow/lite/micro:micro_framework",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/examples/micro_speech/micro_features:micro_model_settings",
         "//tensorflow/lite/micro/examples/micro_speech/micro_features:model",
         "//tensorflow/lite/micro/kernels:micro_ops",
diff --git a/tensorflow/lite/micro/examples/micro_speech/apollo3/pushbutton_test.cc b/tensorflow/lite/micro/examples/micro_speech/apollo3/pushbutton_test.cc
index 1126d7db4d1..a9ca638905f 100644
--- a/tensorflow/lite/micro/examples/micro_speech/apollo3/pushbutton_test.cc
+++ b/tensorflow/lite/micro/examples/micro_speech/apollo3/pushbutton_test.cc
@@ -17,9 +17,9 @@ limitations under the License.
  * micro_speech_test.cc */
 
 #include "tensorflow/lite/c/common.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/examples/micro_speech/simple_features/model.h"
 #include "tensorflow/lite/micro/examples/micro_speech/simple_features/simple_features_generator.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
 #include "tensorflow/lite/micro/micro_error_reporter.h"
 #include "tensorflow/lite/micro/micro_interpreter.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
@@ -69,7 +69,7 @@ TF_LITE_MICRO_TEST(TestSimpleFeaturesGenerator) {
   }
 
   // This pulls in all the operation implementations we need.
-  tflite::ops::micro::AllOpsResolver resolver;
+  tflite::AllOpsResolver resolver;
 
   // Create an area of memory to use for input, output, and intermediate arrays.
   const int tensor_arena_size = 10 * 1024;
diff --git a/tensorflow/lite/micro/examples/micro_speech/main_functions.cc b/tensorflow/lite/micro/examples/micro_speech/main_functions.cc
index 5369008182b..30c5022b2d6 100644
--- a/tensorflow/lite/micro/examples/micro_speech/main_functions.cc
+++ b/tensorflow/lite/micro/examples/micro_speech/main_functions.cc
@@ -72,30 +72,26 @@ void setup() {
   // incur some penalty in code space for op implementations that are not
   // needed by this graph.
   //
-  // tflite::ops::micro::AllOpsResolver resolver;
+  // tflite::AllOpsResolver resolver;
   // NOLINTNEXTLINE(runtime-global-variables)
   static tflite::MicroMutableOpResolver<4> micro_op_resolver(error_reporter);
   if (micro_op_resolver.AddBuiltin(
           tflite::BuiltinOperator_DEPTHWISE_CONV_2D,
-          tflite::ops::micro::Register_DEPTHWISE_CONV_2D(),
-          tflite::MicroOpResolverAnyVersion()) != kTfLiteOk) {
+          tflite::ops::micro::Register_DEPTHWISE_CONV_2D()) != kTfLiteOk) {
     return;
   }
   if (micro_op_resolver.AddBuiltin(
           tflite::BuiltinOperator_FULLY_CONNECTED,
-          tflite::ops::micro::Register_FULLY_CONNECTED(),
-          tflite::MicroOpResolverAnyVersion()) != kTfLiteOk) {
+          tflite::ops::micro::Register_FULLY_CONNECTED()) != kTfLiteOk) {
     return;
   }
   if (micro_op_resolver.AddBuiltin(tflite::BuiltinOperator_SOFTMAX,
-                                   tflite::ops::micro::Register_SOFTMAX(),
-                                   tflite::MicroOpResolverAnyVersion()) !=
+                                   tflite::ops::micro::Register_SOFTMAX()) !=
       kTfLiteOk) {
     return;
   }
   if (micro_op_resolver.AddBuiltin(tflite::BuiltinOperator_RESHAPE,
-                                   tflite::ops::micro::Register_RESHAPE(),
-                                   tflite::MicroOpResolverAnyVersion()) !=
+                                   tflite::ops::micro::Register_RESHAPE()) !=
       kTfLiteOk) {
     return;
   }
diff --git a/tensorflow/lite/micro/examples/micro_speech/micro_speech_test.cc b/tensorflow/lite/micro/examples/micro_speech/micro_speech_test.cc
index b1b224c9391..2c442f955cc 100644
--- a/tensorflow/lite/micro/examples/micro_speech/micro_speech_test.cc
+++ b/tensorflow/lite/micro/examples/micro_speech/micro_speech_test.cc
@@ -47,20 +47,17 @@ TF_LITE_MICRO_TEST(TestInvoke) {
   // incur some penalty in code space for op implementations that are not
   // needed by this graph.
   //
-  // tflite::ops::micro::AllOpsResolver resolver;
+  // tflite::AllOpsResolver resolver;
   tflite::MicroMutableOpResolver<4> micro_op_resolver;
-  micro_op_resolver.AddBuiltin(tflite::BuiltinOperator_DEPTHWISE_CONV_2D,
-                               tflite::ops::micro::Register_DEPTHWISE_CONV_2D(),
-                               tflite::MicroOpResolverAnyVersion());
+  micro_op_resolver.AddBuiltin(
+      tflite::BuiltinOperator_DEPTHWISE_CONV_2D,
+      tflite::ops::micro::Register_DEPTHWISE_CONV_2D());
   micro_op_resolver.AddBuiltin(tflite::BuiltinOperator_FULLY_CONNECTED,
-                               tflite::ops::micro::Register_FULLY_CONNECTED(),
-                               tflite::MicroOpResolverAnyVersion());
+                               tflite::ops::micro::Register_FULLY_CONNECTED());
   micro_op_resolver.AddBuiltin(tflite::BuiltinOperator_SOFTMAX,
-                               tflite::ops::micro::Register_SOFTMAX(),
-                               tflite::MicroOpResolverAnyVersion());
+                               tflite::ops::micro::Register_SOFTMAX());
   micro_op_resolver.AddBuiltin(tflite::BuiltinOperator_RESHAPE,
-                               tflite::ops::micro::Register_RESHAPE(),
-                               tflite::MicroOpResolverAnyVersion());
+                               tflite::ops::micro::Register_RESHAPE());
 
   // Create an area of memory to use for input, output, and intermediate arrays.
   const int tensor_arena_size = 10 * 1024;
diff --git a/tensorflow/lite/micro/examples/network_tester/network_tester_test.cc b/tensorflow/lite/micro/examples/network_tester/network_tester_test.cc
index 0650222b970..9295c87063b 100644
--- a/tensorflow/lite/micro/examples/network_tester/network_tester_test.cc
+++ b/tensorflow/lite/micro/examples/network_tester/network_tester_test.cc
@@ -10,10 +10,10 @@ See the License for the specific language governing permissions and
 limitations under the License.
 ==============================================================================*/
 
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/examples/network_tester/expected_output_data.h"
 #include "tensorflow/lite/micro/examples/network_tester/input_data.h"
 #include "tensorflow/lite/micro/examples/network_tester/network_model.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
 #include "tensorflow/lite/micro/micro_error_reporter.h"
 #include "tensorflow/lite/micro/micro_interpreter.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
@@ -66,7 +66,7 @@ TF_LITE_MICRO_TEST(TestInvoke) {
     return 1;
   }
 
-  tflite::ops::micro::AllOpsResolver resolver;
+  tflite::AllOpsResolver resolver;
 
   tflite::MicroInterpreter interpreter(model, resolver, tensor_arena,
                                        TENSOR_ARENA_SIZE, error_reporter);
diff --git a/tensorflow/lite/micro/examples/person_detection/BUILD b/tensorflow/lite/micro/examples/person_detection/BUILD
index 84eddba73d4..8617373cb41 100644
--- a/tensorflow/lite/micro/examples/person_detection/BUILD
+++ b/tensorflow/lite/micro/examples/person_detection/BUILD
@@ -114,6 +114,7 @@ cc_binary(
         "//tensorflow/lite:schema_fbs_version",
         "//tensorflow/lite/micro:micro_error_reporter",
         "//tensorflow/lite/micro:micro_framework",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/kernels:micro_ops",
         "//tensorflow/lite/schema:schema_fbs",
     ],
diff --git a/tensorflow/lite/micro/examples/person_detection/main_functions.cc b/tensorflow/lite/micro/examples/person_detection/main_functions.cc
index 6b07f6514d5..aa4d83a3334 100644
--- a/tensorflow/lite/micro/examples/person_detection/main_functions.cc
+++ b/tensorflow/lite/micro/examples/person_detection/main_functions.cc
@@ -63,7 +63,7 @@ void setup() {
   // incur some penalty in code space for op implementations that are not
   // needed by this graph.
   //
-  // tflite::ops::micro::AllOpsResolver resolver;
+  // tflite::AllOpsResolver resolver;
   // NOLINTNEXTLINE(runtime-global-variables)
   static tflite::MicroMutableOpResolver<3> micro_op_resolver;
   micro_op_resolver.AddBuiltin(
diff --git a/tensorflow/lite/micro/examples/person_detection/person_detection_test.cc b/tensorflow/lite/micro/examples/person_detection/person_detection_test.cc
index dafed8089e3..abe87636284 100644
--- a/tensorflow/lite/micro/examples/person_detection/person_detection_test.cc
+++ b/tensorflow/lite/micro/examples/person_detection/person_detection_test.cc
@@ -53,7 +53,7 @@ TF_LITE_MICRO_TEST(TestInvoke) {
   // incur some penalty in code space for op implementations that are not
   // needed by this graph.
   //
-  // tflite::ops::micro::AllOpsResolver resolver;
+  // tflite::AllOpsResolver resolver;
   tflite::MicroMutableOpResolver<3> micro_op_resolver;
   micro_op_resolver.AddBuiltin(
       tflite::BuiltinOperator_DEPTHWISE_CONV_2D,
diff --git a/tensorflow/lite/micro/examples/person_detection_experimental/BUILD b/tensorflow/lite/micro/examples/person_detection_experimental/BUILD
index 49f10c814cb..fd726544be3 100644
--- a/tensorflow/lite/micro/examples/person_detection_experimental/BUILD
+++ b/tensorflow/lite/micro/examples/person_detection_experimental/BUILD
@@ -71,6 +71,7 @@ tflite_micro_cc_test(
         "//tensorflow/lite/c:common",
         "//tensorflow/lite/micro:micro_error_reporter",
         "//tensorflow/lite/micro:micro_framework",
+        "//tensorflow/lite/micro/kernels:micro_ops",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -115,6 +116,7 @@ cc_binary(
         "//tensorflow/lite:schema_fbs_version",
         "//tensorflow/lite/micro:micro_error_reporter",
         "//tensorflow/lite/micro:micro_framework",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/kernels:micro_ops",
         "//tensorflow/lite/schema:schema_fbs",
     ],
diff --git a/tensorflow/lite/micro/examples/person_detection_experimental/main_functions.cc b/tensorflow/lite/micro/examples/person_detection_experimental/main_functions.cc
index 3090356ee0d..ac47e36ff8f 100644
--- a/tensorflow/lite/micro/examples/person_detection_experimental/main_functions.cc
+++ b/tensorflow/lite/micro/examples/person_detection_experimental/main_functions.cc
@@ -70,7 +70,7 @@ void setup() {
   // incur some penalty in code space for op implementations that are not
   // needed by this graph.
   //
-  // tflite::ops::micro::AllOpsResolver resolver;
+  // tflite::AllOpsResolver resolver;
   // NOLINTNEXTLINE(runtime-global-variables)
   static tflite::MicroMutableOpResolver<5> micro_op_resolver;
   micro_op_resolver.AddBuiltin(
diff --git a/tensorflow/lite/micro/kernels/BUILD b/tensorflow/lite/micro/kernels/BUILD
index bbb5c67d9e5..229b764a42d 100644
--- a/tensorflow/lite/micro/kernels/BUILD
+++ b/tensorflow/lite/micro/kernels/BUILD
@@ -8,18 +8,18 @@ load(
     "micro_copts",
 )
 
-package(
-    default_visibility = [
-        "//visibility:public",
-    ],
-    licenses = ["notice"],  # Apache 2.0
-)
+licenses(["notice"])  # Apache 2.0
 
 config_setting(
     name = "xtensa_hifimini",
     define_values = {"tflm_build": "xtensa_hifimini"},
 )
 
+package_group(
+    name = "micro_top_level",
+    packages = ["//tensorflow/lite/micro"],
+)
+
 # LINT.IfChange(micro_ops)
 cc_library(
     name = "micro_ops",
@@ -33,6 +33,7 @@ cc_library(
         "concatenation.cc",
         "dequantize.cc",
         "elementwise.cc",
+        "ethosu.cc",
         "floor.cc",
         "l2norm.cc",
         "logical.cc",
@@ -76,6 +77,14 @@ cc_library(
     # TODO(b/153609488): enable embedded build once we can properly support it.
     #build_for_embedded = True,
     copts = micro_copts(),
+    visibility = [
+        # Needed for micro:op_resolvers but visibility can not be
+        # finer-grained than a package.
+        #":micro_top_level",
+        # Currently setting to public for legacy reasons. Can be made more
+        # restrictive once cl/314173854 lands.
+        "//visibility:public",
+    ],
     deps = [
         ":activation_utils",
         ":micro_utils",
@@ -99,24 +108,6 @@ cc_library(
 )
 # LINT.ThenChange(//tensorflow/lite/micro/kernels/BUILD:portable_optimized_micro_ops)
 
-cc_library(
-    name = "all_ops_resolver",
-    srcs = [
-        "all_ops_resolver.cc",
-    ],
-    hdrs = [
-        "all_ops_resolver.h",
-    ],
-    # TODO(b/153609488): enable embedded build once we can properly support it.
-    #build_for_embedded = True,
-    copts = micro_copts(),
-    deps = [
-        ":micro_ops",
-        "//tensorflow/lite/micro:micro_compatibility",
-        "//tensorflow/lite/micro:micro_framework",
-    ],
-)
-
 # LINT.IfChange(portable_optimized_micro_ops)
 cc_library(
     name = "portable_optimized_micro_ops",
@@ -131,6 +122,7 @@ cc_library(
         "conv.cc",
         "dequantize.cc",
         "elementwise.cc",
+        "ethosu.cc",
         "floor.cc",
         "fully_connected.cc",
         "l2norm.cc",
@@ -159,6 +151,11 @@ cc_library(
     ],
     hdrs = ["micro_ops.h"],
     copts = micro_copts(),
+    visibility = [
+        # Needed for micro:portable_optimized_ops_resolver but visibility can not be
+        # finer-grained than a package.
+        ":micro_top_level",
+    ],
     deps = [
         ":activation_utils",
         ":micro_utils",
@@ -175,23 +172,7 @@ cc_library(
         "//tensorflow/lite/micro:micro_utils",
     ],
 )
-
 # LINT.ThenChange(//tensorflow/lite/micro/kernels/BUILD:micro_ops)
-cc_library(
-    name = "portable_optimized_ops_resolver",
-    srcs = [
-        "all_ops_resolver.cc",
-    ],
-    hdrs = [
-        "all_ops_resolver.h",
-    ],
-    copts = micro_copts(),
-    deps = [
-        ":portable_optimized_micro_ops",
-        "//tensorflow/lite/micro:micro_compatibility",
-        "//tensorflow/lite/micro:micro_framework",
-    ],
-)
 
 test_suite(
     name = "all_tests",
@@ -201,9 +182,9 @@ tflite_micro_cc_test(
     name = "elementwise_test",
     srcs = ["elementwise_test.cc"],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
         "//tensorflow/lite/micro:debug_log",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -214,8 +195,8 @@ tflite_micro_cc_test(
         "pooling_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -226,9 +207,9 @@ tflite_micro_cc_test(
         "depthwise_conv_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
         "//tensorflow/lite/kernels/internal:tensor",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -239,9 +220,9 @@ tflite_micro_cc_test(
         "depthwise_conv_test.cc",
     ],
     deps = [
-        ":portable_optimized_ops_resolver",
         "//tensorflow/lite/c:common",
         "//tensorflow/lite/kernels/internal:tensor",
+        "//tensorflow/lite/micro:portable_optimized_op_resolver",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -252,10 +233,10 @@ tflite_micro_cc_test(
         "fully_connected_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
         "//tensorflow/lite/micro:micro_framework",
         "//tensorflow/lite/micro:micro_utils",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -266,8 +247,8 @@ tflite_micro_cc_test(
         "softmax_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -278,8 +259,8 @@ tflite_micro_cc_test(
         "logistic_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -290,8 +271,8 @@ tflite_micro_cc_test(
         "svdf_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -302,9 +283,9 @@ tflite_micro_cc_test(
         "conv_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
         "//tensorflow/lite/micro:micro_utils",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -315,8 +296,8 @@ tflite_micro_cc_test(
         "prelu_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -327,8 +308,8 @@ tflite_micro_cc_test(
         "floor_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -339,8 +320,8 @@ tflite_micro_cc_test(
         "logical_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -351,8 +332,8 @@ tflite_micro_cc_test(
         "neg_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -363,8 +344,8 @@ tflite_micro_cc_test(
         "maximum_minimum_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -375,8 +356,8 @@ tflite_micro_cc_test(
         "mul_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -387,8 +368,8 @@ tflite_micro_cc_test(
         "sub_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -399,8 +380,8 @@ tflite_micro_cc_test(
         "arg_min_max_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -411,8 +392,8 @@ tflite_micro_cc_test(
         "comparisons_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -423,8 +404,8 @@ tflite_micro_cc_test(
         "ceil_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -435,8 +416,8 @@ tflite_micro_cc_test(
         "round_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -447,8 +428,8 @@ tflite_micro_cc_test(
         "strided_slice_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -459,9 +440,9 @@ tflite_micro_cc_test(
         "pack_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
         "//tensorflow/lite/micro:debug_log",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -472,9 +453,9 @@ tflite_micro_cc_test(
         "unpack_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
         "//tensorflow/lite/micro:debug_log",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -485,9 +466,9 @@ tflite_micro_cc_test(
         "split_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
         "//tensorflow/lite/micro:debug_log",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -498,8 +479,8 @@ tflite_micro_cc_test(
         "add_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -532,9 +513,9 @@ tflite_micro_cc_test(
         "quantize_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
         "//tensorflow/lite/micro:micro_framework",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -545,9 +526,9 @@ tflite_micro_cc_test(
         "dequantize_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
         "//tensorflow/lite/micro:micro_framework",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -564,11 +545,11 @@ tflite_micro_cc_test(
         "reshape_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
         "//tensorflow/lite/kernels/internal:tensor",
         "//tensorflow/lite/micro:micro_framework",
         "//tensorflow/lite/micro:micro_utils",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -579,8 +560,8 @@ tflite_micro_cc_test(
         "activations_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -591,8 +572,8 @@ tflite_micro_cc_test(
         "concatenation_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -603,9 +584,9 @@ tflite_micro_cc_test(
         "pad_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
         "//tensorflow/lite/micro:micro_framework",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -616,8 +597,8 @@ tflite_micro_cc_test(
         "reduce_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -628,9 +609,9 @@ tflite_micro_cc_test(
         "circular_buffer_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         ":micro_ops",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -641,8 +622,8 @@ tflite_micro_cc_test(
         "resize_nearest_neighbor_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -653,8 +634,8 @@ tflite_micro_cc_test(
         "l2norm_test.cc",
     ],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
@@ -663,9 +644,9 @@ tflite_micro_cc_test(
     name = "tanh_test",
     srcs = ["tanh_test.cc"],
     deps = [
-        ":all_ops_resolver",
         "//tensorflow/lite/c:common",
         "//tensorflow/lite/micro:micro_framework",
+        "//tensorflow/lite/micro:op_resolvers",
         "//tensorflow/lite/micro/testing:micro_test",
     ],
 )
diff --git a/tensorflow/lite/micro/kernels/activations_test.cc b/tensorflow/lite/micro/kernels/activations_test.cc
index 008be1039e1..686139e51a5 100644
--- a/tensorflow/lite/micro/kernels/activations_test.cc
+++ b/tensorflow/lite/micro/kernels/activations_test.cc
@@ -15,7 +15,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -41,7 +41,7 @@ void TestReluFloat(const int* input_dims_data, const float* input_data,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_RELU);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -97,7 +97,7 @@ void TestRelu6Float(const int* input_dims_data, const float* input_data,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_RELU6);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -158,7 +158,7 @@ void TestReluUint8(const int* input_dims_data, const float* input_data,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_RELU);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -223,7 +223,7 @@ void TestRelu6Uint8(const int* input_dims_data, const float* input_data,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_RELU6);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -287,7 +287,7 @@ void TestReluInt8(const int* input_dims_data, const float* input_data,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_RELU);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -353,7 +353,7 @@ void TestRelu6Int8(const int* input_dims_data, const float* input_data,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_RELU6);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/add_test.cc b/tensorflow/lite/micro/kernels/add_test.cc
index 07dc25222b6..d97739a345b 100644
--- a/tensorflow/lite/micro/kernels/add_test.cc
+++ b/tensorflow/lite/micro/kernels/add_test.cc
@@ -17,7 +17,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -69,7 +69,7 @@ void ValidateAddGoldens(TfLiteTensor* tensors, int tensors_size,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(::tflite::BuiltinOperator_ADD);
 
diff --git a/tensorflow/lite/micro/kernels/all_ops_resolver.cc b/tensorflow/lite/micro/kernels/all_ops_resolver.cc
deleted file mode 100644
index e427511c1f2..00000000000
--- a/tensorflow/lite/micro/kernels/all_ops_resolver.cc
+++ /dev/null
@@ -1,80 +0,0 @@
-/* Copyright 2018 The TensorFlow Authors. All Rights Reserved.
-Licensed under the Apache License, Version 2.0 (the "License");
-you may not use this file except in compliance with the License.
-You may obtain a copy of the License at
-    http://www.apache.org/licenses/LICENSE-2.0
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
-==============================================================================*/
-
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
-
-#include "tensorflow/lite/micro/kernels/micro_ops.h"
-
-namespace tflite {
-namespace ops {
-namespace micro {
-
-AllOpsResolver::AllOpsResolver() {
-  // Please keep this list of Builtin Operators in alphabetical order.
-  AddBuiltin(BuiltinOperator_ABS, Register_ABS());
-  AddBuiltin(BuiltinOperator_ADD, Register_ADD());
-  AddBuiltin(BuiltinOperator_ARG_MAX, Register_ARG_MAX());
-  AddBuiltin(BuiltinOperator_ARG_MIN, Register_ARG_MIN());
-  AddBuiltin(BuiltinOperator_AVERAGE_POOL_2D, Register_AVERAGE_POOL_2D());
-  AddBuiltin(BuiltinOperator_CEIL, Register_CEIL());
-  AddBuiltin(BuiltinOperator_CONCATENATION, Register_CONCATENATION());
-  AddBuiltin(BuiltinOperator_CONV_2D, Register_CONV_2D());
-  AddBuiltin(BuiltinOperator_COS, Register_COS());
-  AddBuiltin(BuiltinOperator_DEPTHWISE_CONV_2D, Register_DEPTHWISE_CONV_2D());
-  AddBuiltin(BuiltinOperator_DEQUANTIZE, Register_DEQUANTIZE());
-  AddBuiltin(BuiltinOperator_EQUAL, Register_EQUAL());
-  AddBuiltin(BuiltinOperator_FLOOR, Register_FLOOR());
-  AddBuiltin(BuiltinOperator_FULLY_CONNECTED, Register_FULLY_CONNECTED());
-  AddBuiltin(BuiltinOperator_GREATER, Register_GREATER());
-  AddBuiltin(BuiltinOperator_GREATER_EQUAL, Register_GREATER_EQUAL());
-  AddBuiltin(BuiltinOperator_L2_NORMALIZATION, Register_L2_NORMALIZATION());
-  AddBuiltin(BuiltinOperator_LESS, Register_LESS());
-  AddBuiltin(BuiltinOperator_LESS_EQUAL, Register_LESS_EQUAL());
-  AddBuiltin(BuiltinOperator_LOG, Register_LOG());
-  AddBuiltin(BuiltinOperator_LOGICAL_AND, Register_LOGICAL_AND());
-  AddBuiltin(BuiltinOperator_LOGICAL_NOT, Register_LOGICAL_NOT());
-  AddBuiltin(BuiltinOperator_LOGICAL_OR, Register_LOGICAL_OR());
-  AddBuiltin(BuiltinOperator_LOGISTIC, Register_LOGISTIC());
-  AddBuiltin(BuiltinOperator_MAX_POOL_2D, Register_MAX_POOL_2D());
-  AddBuiltin(BuiltinOperator_MAXIMUM, Register_MAXIMUM());
-  AddBuiltin(BuiltinOperator_MEAN, Register_MEAN());
-  AddBuiltin(BuiltinOperator_MINIMUM, Register_MINIMUM());
-  AddBuiltin(BuiltinOperator_MUL, Register_MUL());
-  AddBuiltin(BuiltinOperator_NEG, Register_NEG());
-  AddBuiltin(BuiltinOperator_NOT_EQUAL, Register_NOT_EQUAL());
-  AddBuiltin(BuiltinOperator_PACK, Register_PACK());
-  AddBuiltin(BuiltinOperator_PAD, Register_PAD());
-  AddBuiltin(BuiltinOperator_PADV2, Register_PADV2());
-  AddBuiltin(BuiltinOperator_PRELU, Register_PRELU());
-  AddBuiltin(BuiltinOperator_QUANTIZE, Register_QUANTIZE());
-  AddBuiltin(BuiltinOperator_RELU, Register_RELU());
-  AddBuiltin(BuiltinOperator_RELU6, Register_RELU6());
-  AddBuiltin(BuiltinOperator_RESHAPE, Register_RESHAPE());
-  AddBuiltin(BuiltinOperator_RESIZE_NEAREST_NEIGHBOR,
-             Register_RESIZE_NEAREST_NEIGHBOR());
-  AddBuiltin(BuiltinOperator_ROUND, Register_ROUND());
-  AddBuiltin(BuiltinOperator_RSQRT, Register_RSQRT());
-  AddBuiltin(BuiltinOperator_SIN, Register_SIN());
-  AddBuiltin(BuiltinOperator_SOFTMAX, Register_SOFTMAX());
-  AddBuiltin(BuiltinOperator_SPLIT, Register_SPLIT());
-  AddBuiltin(BuiltinOperator_SQRT, Register_SQRT());
-  AddBuiltin(BuiltinOperator_SQUARE, Register_SQUARE());
-  AddBuiltin(BuiltinOperator_STRIDED_SLICE, Register_STRIDED_SLICE());
-  AddBuiltin(BuiltinOperator_SUB, Register_SUB());
-  AddBuiltin(BuiltinOperator_SVDF, Register_SVDF());
-  AddBuiltin(BuiltinOperator_TANH, Register_TANH());
-  AddBuiltin(BuiltinOperator_UNPACK, Register_UNPACK());
-}
-
-}  // namespace micro
-}  // namespace ops
-}  // namespace tflite
diff --git a/tensorflow/lite/micro/kernels/arc_mli/conv_slicing_test.cc b/tensorflow/lite/micro/kernels/arc_mli/conv_slicing_test.cc
index 2c32b45bc26..25d5377009e 100644
--- a/tensorflow/lite/micro/kernels/arc_mli/conv_slicing_test.cc
+++ b/tensorflow/lite/micro/kernels/arc_mli/conv_slicing_test.cc
@@ -24,7 +24,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/micro_utils.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
@@ -136,7 +136,7 @@ TfLiteStatus ValidateConvGoldens(TfLiteTensor* tensors, int tensors_size,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
 
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_CONV_2D);
diff --git a/tensorflow/lite/micro/kernels/arc_mli/depthwise_conv_slicing_test.cc b/tensorflow/lite/micro/kernels/arc_mli/depthwise_conv_slicing_test.cc
index 10f2f857c01..d9efb3ae709 100644
--- a/tensorflow/lite/micro/kernels/arc_mli/depthwise_conv_slicing_test.cc
+++ b/tensorflow/lite/micro/kernels/arc_mli/depthwise_conv_slicing_test.cc
@@ -25,7 +25,7 @@ limitations under the License.
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
 #include "tensorflow/lite/kernels/internal/tensor_ctypes.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -55,7 +55,7 @@ TfLiteStatus ValidateDepthwiseConvGoldens(const T* expected_output_data,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_DEPTHWISE_CONV_2D);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/arc_mli/fully_connected_slicing_test.cc b/tensorflow/lite/micro/kernels/arc_mli/fully_connected_slicing_test.cc
index a64e7bdff4a..d0c7143b18b 100644
--- a/tensorflow/lite/micro/kernels/arc_mli/fully_connected_slicing_test.cc
+++ b/tensorflow/lite/micro/kernels/arc_mli/fully_connected_slicing_test.cc
@@ -26,7 +26,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -69,7 +69,7 @@ void TestFullyConnectedQuantized(
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_FULLY_CONNECTED, 4);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/arc_mli/pooling_slicing_test.cc b/tensorflow/lite/micro/kernels/arc_mli/pooling_slicing_test.cc
index e8667874120..7f21a67d9f7 100644
--- a/tensorflow/lite/micro/kernels/arc_mli/pooling_slicing_test.cc
+++ b/tensorflow/lite/micro/kernels/arc_mli/pooling_slicing_test.cc
@@ -25,7 +25,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -60,7 +60,7 @@ void TestAveragePoolingQuantized(
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_AVERAGE_POOL_2D);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -131,7 +131,7 @@ void TestMaxPoolQuantized(const int* input_dims_data, const T* input_data,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_MAX_POOL_2D);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/arg_min_max_test.cc b/tensorflow/lite/micro/kernels/arg_min_max_test.cc
index 2b4d22d0f4e..c5bbd537fc8 100644
--- a/tensorflow/lite/micro/kernels/arg_min_max_test.cc
+++ b/tensorflow/lite/micro/kernels/arg_min_max_test.cc
@@ -15,7 +15,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -28,7 +28,7 @@ void ValidateArgMinMaxGoldens(TfLiteTensor* tensors, int tensors_size,
                               int output_size, bool using_min) {
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration;
   if (using_min) {
     registration = resolver.FindOp(tflite::BuiltinOperator_ARG_MIN);
diff --git a/tensorflow/lite/micro/kernels/ceil_test.cc b/tensorflow/lite/micro/kernels/ceil_test.cc
index fc21b0141b5..db876a37fcb 100644
--- a/tensorflow/lite/micro/kernels/ceil_test.cc
+++ b/tensorflow/lite/micro/kernels/ceil_test.cc
@@ -15,7 +15,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -37,7 +37,7 @@ void TestCeil(const int* input_dims_data, const float* input_data,
   };
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_CEIL);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/circular_buffer_test.cc b/tensorflow/lite/micro/kernels/circular_buffer_test.cc
index 1fd19bb9d19..cb123c50b19 100644
--- a/tensorflow/lite/micro/kernels/circular_buffer_test.cc
+++ b/tensorflow/lite/micro/kernels/circular_buffer_test.cc
@@ -15,7 +15,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/kernels/micro_ops.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
diff --git a/tensorflow/lite/micro/kernels/cmsis-nn/mul.cc b/tensorflow/lite/micro/kernels/cmsis-nn/mul.cc
index b11fffefacf..d746166ebd9 100644
--- a/tensorflow/lite/micro/kernels/cmsis-nn/mul.cc
+++ b/tensorflow/lite/micro/kernels/cmsis-nn/mul.cc
@@ -50,14 +50,16 @@ TfLiteStatus CalculateOpData(TfLiteContext* context, TfLiteNode* node,
 
   TF_LITE_ENSURE_EQ(context, input1->type, input2->type);
 
-  TF_LITE_ENSURE_STATUS(CalculateActivationRangeQuantized(
-      context, params->activation, output, &data->output_activation_min,
-      &data->output_activation_max));
+  if (output->type == kTfLiteUInt8 || output->type == kTfLiteInt8) {
+    TF_LITE_ENSURE_STATUS(CalculateActivationRangeQuantized(
+        context, params->activation, output, &data->output_activation_min,
+        &data->output_activation_max));
 
-  double real_multiplier =
-      input1->params.scale * input2->params.scale / output->params.scale;
-  QuantizeMultiplier(real_multiplier, &data->output_multiplier,
-                     &data->output_shift);
+    double real_multiplier =
+        input1->params.scale * input2->params.scale / output->params.scale;
+    QuantizeMultiplier(real_multiplier, &data->output_multiplier,
+                       &data->output_shift);
+  }
 
   return kTfLiteOk;
 }
diff --git a/tensorflow/lite/micro/kernels/comparisons_test.cc b/tensorflow/lite/micro/kernels/comparisons_test.cc
index 731d9d778e7..198bb5c9d19 100644
--- a/tensorflow/lite/micro/kernels/comparisons_test.cc
+++ b/tensorflow/lite/micro/kernels/comparisons_test.cc
@@ -17,7 +17,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -36,7 +36,7 @@ void TestComparison(tflite::BuiltinOperator op, TfLiteTensor* tensors,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration = resolver.FindOp(op);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
 
diff --git a/tensorflow/lite/micro/kernels/concatenation_test.cc b/tensorflow/lite/micro/kernels/concatenation_test.cc
index 040cd43a400..feb79804981 100644
--- a/tensorflow/lite/micro/kernels/concatenation_test.cc
+++ b/tensorflow/lite/micro/kernels/concatenation_test.cc
@@ -16,7 +16,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -47,7 +47,7 @@ void TestConcatenateTwoInputs(std::initializer_list<int> input1_dims_data,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_CONCATENATION);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -109,7 +109,7 @@ void TestConcatenateQuantizedTwoInputs(
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_CONCATENATION);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/conv_test.cc b/tensorflow/lite/micro/kernels/conv_test.cc
index 9b1add6d94a..a8e7b12ac99 100644
--- a/tensorflow/lite/micro/kernels/conv_test.cc
+++ b/tensorflow/lite/micro/kernels/conv_test.cc
@@ -15,7 +15,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/micro_utils.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
@@ -57,7 +57,7 @@ TfLiteStatus ValidateConvGoldens(TfLiteTensor* tensors, int tensors_size,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
 
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_CONV_2D);
diff --git a/tensorflow/lite/micro/kernels/depthwise_conv_test.cc b/tensorflow/lite/micro/kernels/depthwise_conv_test.cc
index 07177452ed7..b696cea22b0 100644
--- a/tensorflow/lite/micro/kernels/depthwise_conv_test.cc
+++ b/tensorflow/lite/micro/kernels/depthwise_conv_test.cc
@@ -16,7 +16,7 @@ limitations under the License.
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
 #include "tensorflow/lite/kernels/internal/tensor_ctypes.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -46,7 +46,7 @@ TfLiteStatus ValidateDepthwiseConvGoldens(const T* expected_output_data,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_DEPTHWISE_CONV_2D);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/dequantize.cc b/tensorflow/lite/micro/kernels/dequantize.cc
index 4b87c0eb04c..1fa136ae117 100644
--- a/tensorflow/lite/micro/kernels/dequantize.cc
+++ b/tensorflow/lite/micro/kernels/dequantize.cc
@@ -142,6 +142,9 @@ TfLiteStatus Eval(TfLiteContext* context, TfLiteNode* node) {
 }  // namespace dequantize
 
 TfLiteRegistration* Register_DEQUANTIZE() {
+  // TODO(b/149408647): Once we remove AddBuiltin from MicroOpResolver and
+  // completely switch to the templated AddBuiltin from MicroMutableOpResolver,
+  // this struct no longer needs to be static and can be returned by value.
   static TfLiteRegistration r = {/*init=*/dequantize::Init,
                                  /*free=*/nullptr,
                                  /*prepare=*/dequantize::Prepare,
diff --git a/tensorflow/lite/micro/kernels/dequantize_test.cc b/tensorflow/lite/micro/kernels/dequantize_test.cc
index c90f745a778..c51c48cc23f 100644
--- a/tensorflow/lite/micro/kernels/dequantize_test.cc
+++ b/tensorflow/lite/micro/kernels/dequantize_test.cc
@@ -15,7 +15,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/test_helpers.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
@@ -29,7 +29,7 @@ void ValidateDequantizeGoldens(TfLiteTensor* tensors, int tensors_size,
                                const T* expected_output_data, T* output_data,
                                int output_length, float tolerance = 1e-5) {
   TfLiteContext context;
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
   const TfLiteRegistration* registration =
diff --git a/tensorflow/lite/micro/kernels/elementwise_test.cc b/tensorflow/lite/micro/kernels/elementwise_test.cc
index 923739b5bb4..4086b91b0a9 100644
--- a/tensorflow/lite/micro/kernels/elementwise_test.cc
+++ b/tensorflow/lite/micro/kernels/elementwise_test.cc
@@ -14,8 +14,8 @@ limitations under the License.
 ==============================================================================*/
 
 #include "tensorflow/lite/c/common.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/debug_log.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -46,7 +46,7 @@ void TestElementwiseFloat(tflite::BuiltinOperator op,
 
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  tflite::ops::micro::AllOpsResolver resolver;
+  tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration = resolver.FindOp(op);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
 
@@ -111,7 +111,7 @@ void TestElementwiseBool(tflite::BuiltinOperator op,
 
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  tflite::ops::micro::AllOpsResolver resolver;
+  tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration = resolver.FindOp(op);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
 
diff --git a/tensorflow/lite/micro/kernels/ethos-u/README.md b/tensorflow/lite/micro/kernels/ethos-u/README.md
new file mode 100644
index 00000000000..becf270e4a0
--- /dev/null
+++ b/tensorflow/lite/micro/kernels/ethos-u/README.md
@@ -0,0 +1,44 @@
+# Info
+
+To use Ethos-U kernel add TAGS="ethos-u" to the make line. A tflite file
+compiled by ARM's offline tool Vela is required for it to work. Armclang 6.14 is
+required as compiler as well.
+
+## Vela example workflow
+
+```
+     | tensor0
+     |
+     v
++------------+
+| ethos-u    |
+| custom op  |
++------------+
+     +
+     |
+     | tensor1
+     |
+     v
++---------+
+| softmax |
+|         |
++----|----+
+     |
+     | tensor2
+     |
+     v
+```
+
+Note that ethousu_init() need to be called once during startup.
+
+(TODO: Add link to driver readme.) __FPU_PRESENT need to be set in target
+makefile.
+
+# Example 1
+
+Compile a binary with Ethos-U kernel.
+
+```
+make -f tensorflow/lite/micro/tools/make/Makefile network_tester_test TAGS="ethos-u" \
+TARGET=<ethos_u_enabled_target> NETWORK_MODEL=<ethos_u_enabled_tflite>
+```
diff --git a/tensorflow/lite/micro/kernels/ethos-u/ethosu.cc b/tensorflow/lite/micro/kernels/ethos-u/ethosu.cc
new file mode 100644
index 00000000000..9af6ebb37dd
--- /dev/null
+++ b/tensorflow/lite/micro/kernels/ethos-u/ethosu.cc
@@ -0,0 +1,103 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include <ethosu_driver.h>
+
+#include "tensorflow/lite/c/common.h"
+#include "tensorflow/lite/micro/tools/make/downloads/flatbuffers/include/flatbuffers/flexbuffers.h"
+
+namespace tflite {
+namespace ops {
+namespace micro {
+namespace custom {
+namespace ethosu {
+
+constexpr uint8_t CO_TYPE_ETHOSU = 1;
+
+void* Init(TfLiteContext* context, const char* buffer, size_t length) {
+  return nullptr;
+}
+
+void Free(TfLiteContext* context, void* buffer) {}
+
+TfLiteStatus Prepare(TfLiteContext* context, TfLiteNode* node) {
+  TF_LITE_ENSURE(context, node->inputs->size > 0);
+  TF_LITE_ENSURE(context, context->tensors);
+  TF_LITE_ENSURE(context, node->custom_initial_data_size > 0);
+  return kTfLiteOk;
+}
+
+TfLiteStatus Eval(TfLiteContext* context, TfLiteNode* node) {
+  // Get base addresses
+  TfLiteTensor* tensor;
+  int num_base_addr = node->inputs->size + node->outputs->size;
+  int i = 0;
+  int num_tensors = 0;
+  uint64_t base_addrs[num_base_addr];
+  void* cms_data;
+  int cms_data_size;
+  uint8_t co_type;
+  int result;
+
+  const uint8_t* custom_data =
+      static_cast<uint8_t const*>(node->custom_initial_data);
+  auto root = flexbuffers::GetRoot(custom_data, node->custom_initial_data_size);
+  co_type = root.AsInt8();
+  if (co_type != CO_TYPE_ETHOSU) {
+    TF_LITE_KERNEL_LOG(context, "CO_TYPE != ETHOSU");
+    return kTfLiteError;
+  }
+
+  // Get command stream data address and size
+  tensor = &(context->tensors[node->inputs->data[0]]);
+  cms_data = reinterpret_cast<void*>(tensor->data.uint8);
+  cms_data_size = tensor->bytes;
+
+  // Get adresses to weights/scratch/input data
+  for (i = 1; i < node->inputs->size; ++i) {
+    tensor = &(context->tensors[node->inputs->data[i]]);
+    base_addrs[num_tensors] = reinterpret_cast<uint64_t>(tensor->data.uint8);
+    num_tensors++;
+  }
+
+  // Get adresses to output data
+  for (i = 0; i < node->outputs->size; ++i) {
+    tensor = &(context->tensors[node->outputs->data[i]]);
+    base_addrs[num_tensors] = reinterpret_cast<uint64_t>(tensor->data.uint8);
+    num_tensors++;
+  }
+
+  result = ethosu_invoke(cms_data, cms_data_size, base_addrs, num_tensors);
+  if (-1 == result) {
+    return kTfLiteError;
+  } else {
+    return kTfLiteOk;
+  }
+}
+
+}  // namespace ethosu
+
+TfLiteRegistration* Register_ETHOSU() {
+  static TfLiteRegistration r = {ethosu::Init, ethosu::Free, ethosu::Prepare,
+                                 ethosu::Eval};
+  return &r;
+}
+
+const char* GetString_ETHOSU() { return "ethos-u"; }
+
+}  // namespace custom
+}  // namespace micro
+}  // namespace ops
+}  // namespace tflite
diff --git a/tensorflow/lite/micro/kernels/ethosu.cc b/tensorflow/lite/micro/kernels/ethosu.cc
new file mode 100644
index 00000000000..eac6cea8324
--- /dev/null
+++ b/tensorflow/lite/micro/kernels/ethosu.cc
@@ -0,0 +1,32 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+//
+// This is a stub file for non-Ethos platforms
+//
+#include "tensorflow/lite/c/common.h"
+
+namespace tflite {
+namespace ops {
+namespace micro {
+namespace custom {
+TfLiteRegistration* Register_ETHOSU() { return nullptr; }
+
+const char* GetString_ETHOSU() { return ""; }
+
+}  // namespace custom
+}  // namespace micro
+}  // namespace ops
+}  // namespace tflite
diff --git a/tensorflow/lite/micro/kernels/floor_test.cc b/tensorflow/lite/micro/kernels/floor_test.cc
index eea959d54ce..e6f477bf44d 100644
--- a/tensorflow/lite/micro/kernels/floor_test.cc
+++ b/tensorflow/lite/micro/kernels/floor_test.cc
@@ -15,7 +15,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -38,7 +38,7 @@ void TestFloor(const int* input_dims_data, const float* input_data,
   };
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_FLOOR);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/fully_connected.cc b/tensorflow/lite/micro/kernels/fully_connected.cc
index 66b8379739d..bd949e6f552 100644
--- a/tensorflow/lite/micro/kernels/fully_connected.cc
+++ b/tensorflow/lite/micro/kernels/fully_connected.cc
@@ -217,6 +217,9 @@ TfLiteStatus Eval(TfLiteContext* context, TfLiteNode* node) {
 }  // namespace fully_connected
 
 TfLiteRegistration* Register_FULLY_CONNECTED() {
+  // TODO(b/149408647): Once we remove AddBuiltin from MicroOpResolver and
+  // completely switch to the templated AddBuiltin from MicroMutableOpResolver,
+  // this struct no longer needs to be static and can be returned by value.
   static TfLiteRegistration r = {/*init=*/fully_connected::Init,
                                  /*free=*/nullptr,
                                  /*prepare=*/fully_connected::Prepare,
diff --git a/tensorflow/lite/micro/kernels/fully_connected_test.cc b/tensorflow/lite/micro/kernels/fully_connected_test.cc
index b464d977385..b3066be5eb0 100644
--- a/tensorflow/lite/micro/kernels/fully_connected_test.cc
+++ b/tensorflow/lite/micro/kernels/fully_connected_test.cc
@@ -18,7 +18,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/micro_utils.h"
 #include "tensorflow/lite/micro/test_helpers.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
@@ -53,7 +53,7 @@ TfLiteStatus TestFullyConnectedFloat(
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_FULLY_CONNECTED);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -133,7 +133,7 @@ TfLiteStatus TestFullyConnectedQuantized(
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_FULLY_CONNECTED);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/l2norm_test.cc b/tensorflow/lite/micro/kernels/l2norm_test.cc
index 4b034b211db..9ad3481c582 100644
--- a/tensorflow/lite/micro/kernels/l2norm_test.cc
+++ b/tensorflow/lite/micro/kernels/l2norm_test.cc
@@ -15,7 +15,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -104,7 +104,7 @@ void TestL2Normalization(const int* input_dims_data,
 
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_L2_NORMALIZATION);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/logical_test.cc b/tensorflow/lite/micro/kernels/logical_test.cc
index ec5503bc8f3..262de56ee72 100644
--- a/tensorflow/lite/micro/kernels/logical_test.cc
+++ b/tensorflow/lite/micro/kernels/logical_test.cc
@@ -14,7 +14,7 @@ limitations under the License.
 ==============================================================================*/
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -47,7 +47,7 @@ void TestLogicalOp(tflite::BuiltinOperator op,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration = resolver.FindOp(op);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
 
diff --git a/tensorflow/lite/micro/kernels/logistic_test.cc b/tensorflow/lite/micro/kernels/logistic_test.cc
index 6473858f9fb..7e6484b5154 100644
--- a/tensorflow/lite/micro/kernels/logistic_test.cc
+++ b/tensorflow/lite/micro/kernels/logistic_test.cc
@@ -15,7 +15,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -43,7 +43,7 @@ void TestLogisticFloat(std::initializer_list<int> input_dims_data,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_LOGISTIC);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -105,7 +105,7 @@ void TestLogisticInt8(std::initializer_list<int> input_dims_data,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_LOGISTIC);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/maximum_minimum_test.cc b/tensorflow/lite/micro/kernels/maximum_minimum_test.cc
index c60ca906a5d..2c50552ad79 100644
--- a/tensorflow/lite/micro/kernels/maximum_minimum_test.cc
+++ b/tensorflow/lite/micro/kernels/maximum_minimum_test.cc
@@ -15,7 +15,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -48,7 +48,7 @@ void TestMaxMinFloat(tflite::BuiltinOperator op,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration = resolver.FindOp(op);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
 
@@ -107,7 +107,7 @@ void TestMaxMinQuantized(
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration = resolver.FindOp(op);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
 
@@ -164,7 +164,7 @@ void TestMaxMinQuantizedInt32(
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration = resolver.FindOp(op);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
 
diff --git a/tensorflow/lite/micro/kernels/mul.cc b/tensorflow/lite/micro/kernels/mul.cc
index e89b58a3f62..fb47728a1a4 100644
--- a/tensorflow/lite/micro/kernels/mul.cc
+++ b/tensorflow/lite/micro/kernels/mul.cc
@@ -50,11 +50,11 @@ TfLiteStatus CalculateOpData(TfLiteContext* context, TfLiteNode* node,
 
   TF_LITE_ENSURE_EQ(context, input1->type, input2->type);
 
-  TF_LITE_ENSURE_STATUS(CalculateActivationRangeQuantized(
-      context, params->activation, output, &data->output_activation_min,
-      &data->output_activation_max));
-
   if (output->type == kTfLiteUInt8 || output->type == kTfLiteInt8) {
+    TF_LITE_ENSURE_STATUS(CalculateActivationRangeQuantized(
+        context, params->activation, output, &data->output_activation_min,
+        &data->output_activation_max));
+
     double real_multiplier = static_cast<double>(input1->params.scale) *
                              static_cast<double>(input2->params.scale) /
                              static_cast<double>(output->params.scale);
@@ -138,7 +138,7 @@ TfLiteStatus Eval(TfLiteContext* context, TfLiteNode* node) {
   const TfLiteTensor* input2 = GetInput(context, node, kInput2Tensor);
   TfLiteTensor* output = GetOutput(context, node, kOutputTensor);
 
-  CalculateOpData(context, node, params, &data);
+  TF_LITE_ENSURE_STATUS(CalculateOpData(context, node, params, &data));
 
   switch (input1->type) {
     case kTfLiteUInt8:
diff --git a/tensorflow/lite/micro/kernels/mul_test.cc b/tensorflow/lite/micro/kernels/mul_test.cc
index 446512a7c6c..d58d5eaf715 100644
--- a/tensorflow/lite/micro/kernels/mul_test.cc
+++ b/tensorflow/lite/micro/kernels/mul_test.cc
@@ -15,7 +15,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -35,7 +35,7 @@ void TestMulFloat(std::initializer_list<int> input1_dims_data,
   TfLiteIntArray* output_dims = IntArrayFromInitializer(output_dims_data);
   const int output_dims_count = ElementCount(*output_dims);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
 
   constexpr int inputs_size = 2;
   constexpr int outputs_size = 1;
@@ -107,7 +107,7 @@ void TestMulQuantized(std::initializer_list<int> input1_dims_data,
   TfLiteIntArray* output_dims = IntArrayFromInitializer(output_dims_data);
   const int output_dims_count = ElementCount(*output_dims);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
 
   constexpr int inputs_size = 2;
   constexpr int outputs_size = 1;
diff --git a/tensorflow/lite/micro/kernels/neg_test.cc b/tensorflow/lite/micro/kernels/neg_test.cc
index 9e8f0c842ea..9608e31d73c 100644
--- a/tensorflow/lite/micro/kernels/neg_test.cc
+++ b/tensorflow/lite/micro/kernels/neg_test.cc
@@ -15,7 +15,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -41,7 +41,7 @@ void TestNegFloat(std::initializer_list<int> input_dims_data,
 
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_NEG);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/pack_test.cc b/tensorflow/lite/micro/kernels/pack_test.cc
index 216cd615e4c..187fc2a0ddd 100644
--- a/tensorflow/lite/micro/kernels/pack_test.cc
+++ b/tensorflow/lite/micro/kernels/pack_test.cc
@@ -15,8 +15,8 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/debug_log.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -50,7 +50,7 @@ void TestPackTwoInputsFloat(std::initializer_list<int> input1_dims_data,
 
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  tflite::ops::micro::AllOpsResolver resolver;
+  tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_PACK);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -127,7 +127,7 @@ void TestPackThreeInputsFloat(std::initializer_list<int> input1_dims_data,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  tflite::ops::micro::AllOpsResolver resolver;
+  tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_PACK);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -200,7 +200,7 @@ void TestPackTwoInputsQuantized(
 
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  tflite::ops::micro::AllOpsResolver resolver;
+  tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_PACK);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -270,7 +270,7 @@ void TestPackTwoInputsQuantized32(
 
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  tflite::ops::micro::AllOpsResolver resolver;
+  tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_PACK);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/pad_test.cc b/tensorflow/lite/micro/kernels/pad_test.cc
index eac5b980837..28b2069bc18 100644
--- a/tensorflow/lite/micro/kernels/pad_test.cc
+++ b/tensorflow/lite/micro/kernels/pad_test.cc
@@ -15,7 +15,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/test_helpers.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
@@ -30,7 +30,7 @@ TfLiteStatus ValidatePadGoldens(TfLiteTensor* tensors, int tensors_size,
                                 int output_length) {
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_PAD);
   TF_LITE_ENSURE(&context, registration != nullptr);
@@ -67,7 +67,7 @@ TfLiteStatus ValidatePadV2Goldens(TfLiteTensor* tensors, int tensors_size,
                                   int output_length) {
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_PADV2);
   TF_LITE_ENSURE(&context, registration != nullptr);
diff --git a/tensorflow/lite/micro/kernels/pooling_test.cc b/tensorflow/lite/micro/kernels/pooling_test.cc
index 1ac74fca644..e656fb802ed 100644
--- a/tensorflow/lite/micro/kernels/pooling_test.cc
+++ b/tensorflow/lite/micro/kernels/pooling_test.cc
@@ -17,7 +17,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -49,7 +49,7 @@ void TestAveragePoolingFloat(std::initializer_list<int> input_dims_data,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_AVERAGE_POOL_2D);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -123,7 +123,7 @@ void TestAveragePoolingQuantized(
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_AVERAGE_POOL_2D);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -190,7 +190,7 @@ void TestMaxPoolFloat(std::initializer_list<int> input_dims_data,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_MAX_POOL_2D);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -266,7 +266,7 @@ void TestMaxPoolQuantized(std::initializer_list<int> input_dims_data,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_MAX_POOL_2D);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/prelu_test.cc b/tensorflow/lite/micro/kernels/prelu_test.cc
index 1a41e75501d..c60e33313bb 100644
--- a/tensorflow/lite/micro/kernels/prelu_test.cc
+++ b/tensorflow/lite/micro/kernels/prelu_test.cc
@@ -15,7 +15,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -44,7 +44,7 @@ void TestPreluFloat(std::initializer_list<int> input_dims_data,
   };
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_PRELU);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -111,7 +111,7 @@ void TestPreluQuantized(std::initializer_list<int> input_dims_data,
   };
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_PRELU);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/quantize_test.cc b/tensorflow/lite/micro/kernels/quantize_test.cc
index 3089357040c..33d356f9191 100644
--- a/tensorflow/lite/micro/kernels/quantize_test.cc
+++ b/tensorflow/lite/micro/kernels/quantize_test.cc
@@ -15,7 +15,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/test_helpers.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
@@ -33,7 +33,7 @@ void ValidateQuantizeGoldens(TfLiteTensor* tensors, int tensors_size,
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
   // Version 1 of quantize supports int8 and uint8 quantization.
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_QUANTIZE);
 
diff --git a/tensorflow/lite/micro/kernels/reduce_test.cc b/tensorflow/lite/micro/kernels/reduce_test.cc
index 965e45adb44..a65ba8cd376 100644
--- a/tensorflow/lite/micro/kernels/reduce_test.cc
+++ b/tensorflow/lite/micro/kernels/reduce_test.cc
@@ -15,7 +15,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -51,7 +51,7 @@ TfLiteStatus ValidateReduceGoldens(TfLiteTensor* tensors, int tensors_size,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
 
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_MEAN);
diff --git a/tensorflow/lite/micro/kernels/reshape_test.cc b/tensorflow/lite/micro/kernels/reshape_test.cc
index 7e7f58f0a03..1a88997cf65 100644
--- a/tensorflow/lite/micro/kernels/reshape_test.cc
+++ b/tensorflow/lite/micro/kernels/reshape_test.cc
@@ -19,7 +19,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/common.h"
 #include "tensorflow/lite/kernels/internal/tensor_ctypes.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/micro_utils.h"
 #include "tensorflow/lite/micro/test_helpers.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
@@ -60,7 +60,7 @@ void TestReshapeImpl(TfLiteTensor* input_tensor, TfLiteTensor* shape_tensor,
     node.outputs = IntArrayFromInitializer({1, 2});
   }
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_RESHAPE);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/resize_nearest_neighbor_test.cc b/tensorflow/lite/micro/kernels/resize_nearest_neighbor_test.cc
index 72e7252b31b..fda4a1f5173 100644
--- a/tensorflow/lite/micro/kernels/resize_nearest_neighbor_test.cc
+++ b/tensorflow/lite/micro/kernels/resize_nearest_neighbor_test.cc
@@ -14,7 +14,7 @@ limitations under the License.
 ==============================================================================*/
 
 #include "tensorflow/lite/c/builtin_op_data.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -69,7 +69,7 @@ void TestResizeNearestNeighbor(const int* input_dims_data, const T* input_data,
 
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_RESIZE_NEAREST_NEIGHBOR);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/round_test.cc b/tensorflow/lite/micro/kernels/round_test.cc
index 10eb2759a4b..c1421e099ea 100644
--- a/tensorflow/lite/micro/kernels/round_test.cc
+++ b/tensorflow/lite/micro/kernels/round_test.cc
@@ -15,7 +15,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -37,7 +37,7 @@ void TestRound(const int* input_dims_data, const float* input_data,
   };
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_ROUND);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/softmax.cc b/tensorflow/lite/micro/kernels/softmax.cc
index 3d8e7cd87f7..616017ecc5b 100644
--- a/tensorflow/lite/micro/kernels/softmax.cc
+++ b/tensorflow/lite/micro/kernels/softmax.cc
@@ -137,6 +137,9 @@ TfLiteStatus SoftmaxEval(TfLiteContext* context, TfLiteNode* node) {
 }  // namespace activations
 
 TfLiteRegistration* Register_SOFTMAX() {
+  // TODO(b/149408647): Once we remove AddBuiltin from MicroOpResolver and
+  // completely switch to the templated AddBuiltin from MicroMutableOpResolver,
+  // this struct no longer needs to be static and can be returned by value.
   static TfLiteRegistration r = {/*init=*/nullptr,
                                  /*free=*/nullptr,
                                  /*prepare=*/activations::SoftmaxPrepare,
diff --git a/tensorflow/lite/micro/kernels/softmax_test.cc b/tensorflow/lite/micro/kernels/softmax_test.cc
index afd97f1f015..b8d89673837 100644
--- a/tensorflow/lite/micro/kernels/softmax_test.cc
+++ b/tensorflow/lite/micro/kernels/softmax_test.cc
@@ -15,7 +15,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -43,7 +43,7 @@ void TestSoftmaxFloat(std::initializer_list<int> input_dims_data,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_SOFTMAX);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -109,7 +109,7 @@ void TestSoftmaxQuantized(std::initializer_list<int> input_dims_data,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_SOFTMAX);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -175,7 +175,7 @@ void TestSoftmaxQuantizedSigned(
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_SOFTMAX);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/split_test.cc b/tensorflow/lite/micro/kernels/split_test.cc
index 807929ca4d5..fb46dfbfb49 100644
--- a/tensorflow/lite/micro/kernels/split_test.cc
+++ b/tensorflow/lite/micro/kernels/split_test.cc
@@ -15,8 +15,8 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/debug_log.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -63,7 +63,7 @@ void TestSplitTwoOutputsFloat(
 
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  tflite::ops::micro::AllOpsResolver resolver;
+  tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_SPLIT);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -166,7 +166,7 @@ void TestSplitFourOutputsFloat(
 
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  tflite::ops::micro::AllOpsResolver resolver;
+  tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_SPLIT);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -264,7 +264,7 @@ void TestSplitTwoOutputsQuantized(
 
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  tflite::ops::micro::AllOpsResolver resolver;
+  tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_SPLIT);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -353,7 +353,7 @@ void TestSplitTwoOutputsQuantized32(
 
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  tflite::ops::micro::AllOpsResolver resolver;
+  tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_SPLIT);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/strided_slice_test.cc b/tensorflow/lite/micro/kernels/strided_slice_test.cc
index a36a90b858e..308c20ec0d8 100644
--- a/tensorflow/lite/micro/kernels/strided_slice_test.cc
+++ b/tensorflow/lite/micro/kernels/strided_slice_test.cc
@@ -14,7 +14,7 @@ limitations under the License.
 ==============================================================================*/
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -86,7 +86,7 @@ void TestStrideSlide(std::initializer_list<int> input_shape,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_STRIDED_SLICE);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/sub_test.cc b/tensorflow/lite/micro/kernels/sub_test.cc
index e4875c3bd1e..fd298e55276 100644
--- a/tensorflow/lite/micro/kernels/sub_test.cc
+++ b/tensorflow/lite/micro/kernels/sub_test.cc
@@ -17,7 +17,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -69,7 +69,7 @@ void ValidateSubGoldens(TfLiteTensor* tensors, int tensors_size,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(::tflite::BuiltinOperator_SUB);
 
diff --git a/tensorflow/lite/micro/kernels/svdf.cc b/tensorflow/lite/micro/kernels/svdf.cc
index 8c33fde5a87..ba7cb05da57 100644
--- a/tensorflow/lite/micro/kernels/svdf.cc
+++ b/tensorflow/lite/micro/kernels/svdf.cc
@@ -528,6 +528,9 @@ TfLiteStatus Eval(TfLiteContext* context, TfLiteNode* node) {
 }  // namespace svdf
 
 TfLiteRegistration* Register_SVDF() {
+  // TODO(b/149408647): Once we remove AddBuiltin from MicroOpResolver and
+  // completely switch to the templated AddBuiltin from MicroMutableOpResolver,
+  // this struct no longer needs to be static and can be returned by value.
   static TfLiteRegistration r = {/*init=*/svdf::Init,
                                  /*free=*/nullptr,
                                  /*prepare=*/svdf::Prepare,
diff --git a/tensorflow/lite/micro/kernels/svdf_test.cc b/tensorflow/lite/micro/kernels/svdf_test.cc
index fead3ab2fab..31c610a72c7 100644
--- a/tensorflow/lite/micro/kernels/svdf_test.cc
+++ b/tensorflow/lite/micro/kernels/svdf_test.cc
@@ -17,7 +17,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -173,7 +173,7 @@ void ValidateSVDFGoldens(const int batch_size, const int num_units,
   TfLiteContext context;
   PopulateContext(tensors, tensor_count, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_SVDF);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -248,7 +248,7 @@ void ValidateIntegerSVDFGoldens(const int batch_size, const int num_units,
   TfLiteContext context;
   PopulateContext(tensors, tensor_count, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_SVDF);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/tanh_test.cc b/tensorflow/lite/micro/kernels/tanh_test.cc
index 517d8a9c4a8..cfdef61b271 100644
--- a/tensorflow/lite/micro/kernels/tanh_test.cc
+++ b/tensorflow/lite/micro/kernels/tanh_test.cc
@@ -15,7 +15,7 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -43,7 +43,7 @@ void TestTanhFloat(std::initializer_list<int> input_dims_data,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_TANH);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -105,7 +105,7 @@ void TestTanhInt8(std::initializer_list<int> input_dims_data,
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
 
-  ::tflite::ops::micro::AllOpsResolver resolver;
+  ::tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_TANH);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/kernels/unpack_test.cc b/tensorflow/lite/micro/kernels/unpack_test.cc
index 015c92c208a..fc524e90cd8 100644
--- a/tensorflow/lite/micro/kernels/unpack_test.cc
+++ b/tensorflow/lite/micro/kernels/unpack_test.cc
@@ -15,8 +15,8 @@ limitations under the License.
 
 #include "tensorflow/lite/c/builtin_op_data.h"
 #include "tensorflow/lite/c/common.h"
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/debug_log.h"
-#include "tensorflow/lite/micro/kernels/all_ops_resolver.h"
 #include "tensorflow/lite/micro/testing/micro_test.h"
 #include "tensorflow/lite/micro/testing/test_utils.h"
 
@@ -65,7 +65,7 @@ void TestUnpackThreeOutputsFloat(
 
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  tflite::ops::micro::AllOpsResolver resolver;
+  tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_UNPACK);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -142,7 +142,7 @@ void TestUnpackOneOutputFloat(std::initializer_list<int> input_dims_data,
 
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  tflite::ops::micro::AllOpsResolver resolver;
+  tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_UNPACK);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -234,7 +234,7 @@ void TestUnpackThreeOutputsQuantized(
 
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  tflite::ops::micro::AllOpsResolver resolver;
+  tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_UNPACK);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
@@ -330,7 +330,7 @@ void TestUnpackThreeOutputsQuantized32(
 
   TfLiteContext context;
   PopulateContext(tensors, tensors_size, micro_test::reporter, &context);
-  tflite::ops::micro::AllOpsResolver resolver;
+  tflite::AllOpsResolver resolver;
   const TfLiteRegistration* registration =
       resolver.FindOp(tflite::BuiltinOperator_UNPACK);
   TF_LITE_MICRO_EXPECT_NE(nullptr, registration);
diff --git a/tensorflow/lite/micro/micro_allocator.cc b/tensorflow/lite/micro/micro_allocator.cc
index ad26483ec3c..2289e8390bc 100644
--- a/tensorflow/lite/micro/micro_allocator.cc
+++ b/tensorflow/lite/micro/micro_allocator.cc
@@ -404,7 +404,10 @@ TfLiteStatus InitializeTfLiteTensorFromFlatbuffer(
 MicroAllocator::MicroAllocator(TfLiteContext* context, const Model* model,
                                uint8_t* tensor_arena, size_t arena_size,
                                ErrorReporter* error_reporter)
-    : model_(model), error_reporter_(error_reporter), context_(context) {
+    : model_(model),
+      context_(context),
+      error_reporter_(error_reporter),
+      active_(false) {
   uint8_t* aligned_arena = AlignPointerUp(tensor_arena, kBufferAlignment);
   if (aligned_arena != tensor_arena) {
     TF_LITE_REPORT_ERROR(
@@ -419,7 +422,20 @@ MicroAllocator::MicroAllocator(TfLiteContext* context, const Model* model,
   // destructed as it's the root allocator.
   memory_allocator_ = SimpleMemoryAllocator::Create(
       error_reporter, aligned_arena, aligned_arena_size);
+}
 
+MicroAllocator::MicroAllocator(TfLiteContext* context, const Model* model,
+                               SimpleMemoryAllocator* memory_allocator,
+                               ErrorReporter* error_reporter)
+    : memory_allocator_(memory_allocator),
+      model_(model),
+      context_(context),
+      error_reporter_(error_reporter),
+      active_(false) {}
+
+MicroAllocator::~MicroAllocator() {}
+
+TfLiteStatus MicroAllocator::Init() {
   TfLiteStatus status = InitGraphAndContextTensorData();
   // TODO(b/147871299): Consider improving this code. A better way of handling
   // failures in the constructor is to have a static function that returns a
@@ -431,9 +447,10 @@ MicroAllocator::MicroAllocator(TfLiteContext* context, const Model* model,
   } else {
     active_ = true;
   }
+  return status;
 }
 
-TfLiteStatus MicroAllocator::InitializeFromFlatbuffer(
+TfLiteStatus MicroAllocator::PrepareFromFlatbuffer(
     const MicroOpResolver& op_resolver,
     NodeAndRegistration** node_and_registrations) {
   if (!active_) {
@@ -580,12 +597,6 @@ size_t MicroAllocator::used_bytes() const {
   if (active_) {
     return 0;
   }
-  TF_LITE_REPORT_ERROR(error_reporter_, "Total buffer usage: %d bytes",
-                       memory_allocator_->GetUsedBytes());
-  TF_LITE_REPORT_ERROR(error_reporter_, "Head usage: %d bytes",
-                       memory_allocator_->GetHeadUsedBytes());
-  TF_LITE_REPORT_ERROR(error_reporter_, "Tail usage: %d bytes",
-                       memory_allocator_->GetTailUsedBytes());
   return memory_allocator_->GetUsedBytes();
 }
 
@@ -684,25 +695,35 @@ TfLiteStatus MicroAllocator::PrepareNodeAndRegistrationDataFromFlatbuffer(
     BuiltinOperator op_type =
         static_cast<BuiltinOperator>(registration->builtin_code);
 
-    if (op_type != BuiltinOperator_CUSTOM && op->custom_options()) {
-      TF_LITE_REPORT_ERROR(
-          error_reporter_,
-          "Unsupported behavior: found builtin operator %s with custom "
-          "options.\n",
-          EnumNameBuiltinOperator(op_type));
-      return kTfLiteError;
-    }
-
     const char* custom_data = nullptr;
     size_t custom_data_size = 0;
     unsigned char* builtin_data = nullptr;
-    if (op->custom_options()) {
-      custom_data = reinterpret_cast<const char*>(op->custom_options()->data());
-      custom_data_size = op->custom_options()->size();
+
+    if (op_type == BuiltinOperator_CUSTOM) {
+      // Custom Ops may or may not have a non-null custom_options field.
+      if (op->custom_options() != nullptr) {
+        custom_data =
+            reinterpret_cast<const char*>(op->custom_options()->data());
+        custom_data_size = op->custom_options()->size();
+      }
     } else {
+      if (op->custom_options() != nullptr) {
+        TF_LITE_REPORT_ERROR(
+            error_reporter_,
+            "Unsupported behavior: found builtin operator %s with custom "
+            "options.\n",
+            EnumNameBuiltinOperator(op_type));
+        return kTfLiteError;
+      }
+
       MicroOpResolver::BuiltinParseFunction parser =
           op_resolver.GetOpDataParser(op_type);
-      TFLITE_DCHECK(parser != nullptr);
+      if (parser == nullptr) {
+        TF_LITE_REPORT_ERROR(error_reporter_, "Did not find a parser for %s",
+                             EnumNameBuiltinOperator(op_type));
+
+        return kTfLiteError;
+      }
       TF_LITE_ENSURE_STATUS(parser(op, op_type, error_reporter_,
                                    &builtin_data_allocator,
                                    (void**)(&builtin_data)));
@@ -724,6 +745,16 @@ TfLiteStatus MicroAllocator::PrepareNodeAndRegistrationDataFromFlatbuffer(
   }
 
   return kTfLiteOk;
+}  // namespace tflite
+
+size_t MicroAllocator::GetTensorsCount() const {
+  return context_->tensors_size;
 }
 
+size_t MicroAllocator::GetOperatorsCount() const {
+  return subgraph_->operators()->size();
+}
+
+ErrorReporter* MicroAllocator::error_reporter() { return error_reporter_; }
+
 }  // namespace tflite
diff --git a/tensorflow/lite/micro/micro_allocator.h b/tensorflow/lite/micro/micro_allocator.h
index d7f7e4c9d6c..a56bef02bc8 100644
--- a/tensorflow/lite/micro/micro_allocator.h
+++ b/tensorflow/lite/micro/micro_allocator.h
@@ -21,6 +21,7 @@ limitations under the License.
 #include "flatbuffers/flatbuffers.h"  // from @flatbuffers
 #include "tensorflow/lite/c/common.h"
 #include "tensorflow/lite/core/api/error_reporter.h"
+#include "tensorflow/lite/micro/compatibility.h"
 #include "tensorflow/lite/micro/micro_op_resolver.h"
 #include "tensorflow/lite/micro/simple_memory_allocator.h"
 #include "tensorflow/lite/schema/schema_generated.h"
@@ -85,13 +86,18 @@ class MicroAllocator {
   MicroAllocator(TfLiteContext* context, const Model* model,
                  uint8_t* tensor_arena, size_t arena_size,
                  ErrorReporter* error_reporter);
+  virtual ~MicroAllocator();
+
+  // Initializes the allocator by allocating required internal structs required
+  // to prepare the model from the flatbuffer data in PrepareFromFlatbuffer.
+  TfLiteStatus Init();
 
   // Run through the model flatbuffer data (loaded from the TfLiteModel
   // instance) to allocate nodes and registrations. We need to keep them for the
   // entire life time of the model to allow persistent tensors. This method
   // needs to be called before FinishTensorAllocation method. This method also
   // allocates any internal Op data that is required from the flatbuffer.
-  TfLiteStatus InitializeFromFlatbuffer(
+  TfLiteStatus PrepareFromFlatbuffer(
       const MicroOpResolver& op_resolver,
       NodeAndRegistration** node_and_registrations);
 
@@ -123,39 +129,52 @@ class MicroAllocator {
   size_t used_bytes() const;
 
  protected:
+  MicroAllocator(TfLiteContext* context, const Model* model,
+                 SimpleMemoryAllocator* memory_allocator,
+                 ErrorReporter* error_reporter);
+
   // Allocates an array in the arena to hold pointers to the tensors required
   // to initialize and prepare a model. These allocations are stored and
   // populated on the context.
-  TfLiteStatus AllocateTfLiteTensorArray();
+  virtual TfLiteStatus AllocateTfLiteTensorArray();
 
   // Populates content on the list of tensor pointers required to initialize and
   // prepare a model from data in the flatbuffer (loaded from the TfLiteModel
   // instance). Persistent data (e.g. quantization params) is allocated from the
   // arena.
-  TfLiteStatus PopulateTfLiteTensorArrayFromFlatbuffer();
+  virtual TfLiteStatus PopulateTfLiteTensorArrayFromFlatbuffer();
 
   // Allocates an array in the arena to hold pointers to the node and
   // registration pointers required to represent the inference graph of the
   // model.
-  TfLiteStatus AllocateNodeAndRegistrations(
+  virtual TfLiteStatus AllocateNodeAndRegistrations(
       NodeAndRegistration** node_and_registrations);
 
   // Populates node and registration pointers representing the inference graph
   // of the model from values inside the flatbuffer (loaded from the TfLiteModel
   // instance). Persistent data (e.g. operator data) is allocated from the
   // arena.
-  TfLiteStatus PrepareNodeAndRegistrationDataFromFlatbuffer(
+  virtual TfLiteStatus PrepareNodeAndRegistrationDataFromFlatbuffer(
       const MicroOpResolver& op_resolver,
       NodeAndRegistration* node_and_registrations);
 
+  // Returns the number of tensors in the model subgraph.
+  size_t GetTensorsCount() const;
+
+  // Returns the number of operators in the model subgraph.
+  size_t GetOperatorsCount() const;
+
+  ErrorReporter* error_reporter();
+
  private:
   TfLiteStatus InitGraphAndContextTensorData();
 
-  const Model* model_;
   // A simple memory allocator that always allocate from the arena tail.
   SimpleMemoryAllocator* memory_allocator_;
-  ErrorReporter* error_reporter_;
+
+  const Model* model_;
   TfLiteContext* context_;
+  ErrorReporter* error_reporter_;
   // Indicating if the allocator is ready for allocation.
   bool active_ = false;
 
@@ -167,6 +186,8 @@ class MicroAllocator {
   size_t scratch_buffer_count_ = 0;
 
   const SubGraph* subgraph_;
+
+  TF_LITE_REMOVE_VIRTUAL_DELETE
 };
 
 }  // namespace tflite
diff --git a/tensorflow/lite/micro/micro_allocator_test.cc b/tensorflow/lite/micro/micro_allocator_test.cc
index 013c1cefabf..a85d30dab46 100644
--- a/tensorflow/lite/micro/micro_allocator_test.cc
+++ b/tensorflow/lite/micro/micro_allocator_test.cc
@@ -151,6 +151,7 @@ TF_LITE_MICRO_TEST(TestFinishTensorAllocation) {
   uint8_t arena[arena_size];
   tflite::MicroAllocator allocator(&context, model, arena, arena_size,
                                    micro_test::reporter);
+  TF_LITE_MICRO_EXPECT_EQ(kTfLiteOk, allocator.Init());
   TF_LITE_MICRO_EXPECT_EQ(4, context.tensors_size);
   // Memory planning hasn't been finalized, so the used bytes is unknown.
   TF_LITE_MICRO_EXPECT_EQ(0, allocator.used_bytes());
@@ -187,6 +188,7 @@ TF_LITE_MICRO_TEST(TestAllocationForModelsWithBranches) {
   uint8_t arena[arena_size];
   tflite::MicroAllocator allocator(&context, model, arena, arena_size,
                                    micro_test::reporter);
+  TF_LITE_MICRO_EXPECT_EQ(kTfLiteOk, allocator.Init());
   TF_LITE_MICRO_EXPECT_EQ(kTfLiteOk, allocator.FinishTensorAllocation());
 
   uint8_t* start = context.tensors[0].data.uint8;
@@ -211,6 +213,7 @@ TF_LITE_MICRO_TEST(TestFinishComplexTensorAllocation) {
   uint8_t arena[arena_size];
   tflite::MicroAllocator allocator(&context, model, arena, arena_size,
                                    micro_test::reporter);
+  TF_LITE_MICRO_EXPECT_EQ(kTfLiteOk, allocator.Init());
   TF_LITE_MICRO_EXPECT_EQ(10, context.tensors_size);
 
   TF_LITE_MICRO_EXPECT_EQ(kTfLiteOk, allocator.FinishTensorAllocation());
@@ -238,4 +241,15 @@ TF_LITE_MICRO_TEST(TestFinishComplexTensorAllocation) {
   tflite::testing::EnsureUniqueVariableTensorBuffer(&context, 7);
 }
 
+TF_LITE_MICRO_TEST(TestDoubleInitFails) {
+  const tflite::Model* model = tflite::testing::GetComplexMockModel();
+  TfLiteContext context;
+  constexpr size_t arena_size = 2048;
+  uint8_t arena[arena_size];
+  tflite::MicroAllocator allocator(&context, model, arena, arena_size,
+                                   micro_test::reporter);
+  TF_LITE_MICRO_EXPECT_EQ(kTfLiteOk, allocator.Init());
+  TF_LITE_MICRO_EXPECT_EQ(10, context.tensors_size);
+}
+
 TF_LITE_MICRO_TESTS_END
diff --git a/tensorflow/lite/micro/micro_interpreter.cc b/tensorflow/lite/micro/micro_interpreter.cc
index 7e2e56e417d..9f0f073af10 100644
--- a/tensorflow/lite/micro/micro_interpreter.cc
+++ b/tensorflow/lite/micro/micro_interpreter.cc
@@ -91,6 +91,12 @@ MicroInterpreter::MicroInterpreter(const Model* model,
     initialization_status_ = kTfLiteError;
     return;
   }
+  if (allocator_.Init() != kTfLiteOk) {
+    TF_LITE_REPORT_ERROR(error_reporter,
+                         "Failed to initialize the allocator.\n");
+    initialization_status_ = kTfLiteError;
+    return;
+  }
   subgraph_ = (*subgraphs)[0];
 
   context_.impl_ = static_cast<void*>(&context_helper_);
@@ -166,7 +172,7 @@ void MicroInterpreter::CorrectTensorDataEndianness(T* data, int32_t size) {
 }
 
 TfLiteStatus MicroInterpreter::AllocateTensors() {
-  TF_LITE_ENSURE_OK(&context_, allocator_.InitializeFromFlatbuffer(
+  TF_LITE_ENSURE_OK(&context_, allocator_.PrepareFromFlatbuffer(
                                    op_resolver_, &node_and_registrations_));
 
   // Only allow AllocatePersistentBuffer in Init stage.
diff --git a/tensorflow/lite/micro/micro_interpreter_test.cc b/tensorflow/lite/micro/micro_interpreter_test.cc
index aa25593c508..bd4e536218d 100644
--- a/tensorflow/lite/micro/micro_interpreter_test.cc
+++ b/tensorflow/lite/micro/micro_interpreter_test.cc
@@ -174,8 +174,13 @@ class MockOpResolver : public MicroOpResolver {
   }
 
   TfLiteStatus AddBuiltin(tflite::BuiltinOperator op,
-                          TfLiteRegistration* registration,
-                          int version) override {
+                          TfLiteRegistration* registration) override {
+    // This function is currently not used in the tests.
+    return kTfLiteError;
+  }
+
+  TfLiteStatus AddCustom(const char* name,
+                         TfLiteRegistration* registration) override {
     // This function is currently not used in the tests.
     return kTfLiteError;
   }
diff --git a/tensorflow/lite/micro/micro_mutable_op_resolver.h b/tensorflow/lite/micro/micro_mutable_op_resolver.h
index 54de9e0e518..34768ae3cb8 100644
--- a/tensorflow/lite/micro/micro_mutable_op_resolver.h
+++ b/tensorflow/lite/micro/micro_mutable_op_resolver.h
@@ -15,22 +15,21 @@ limitations under the License.
 #ifndef TENSORFLOW_LITE_MICRO_MICRO_MUTABLE_OP_RESOLVER_H_
 #define TENSORFLOW_LITE_MICRO_MICRO_MUTABLE_OP_RESOLVER_H_
 
+#include <cstdio>
 #include <cstring>
 
 #include "tensorflow/lite/c/common.h"
 #include "tensorflow/lite/core/api/error_reporter.h"
 #include "tensorflow/lite/core/api/flatbuffer_conversions.h"
+#include "tensorflow/lite/kernels/internal/compatibility.h"
+#include "tensorflow/lite/kernels/op_macros.h"
 #include "tensorflow/lite/micro/compatibility.h"
+#include "tensorflow/lite/micro/kernels/micro_ops.h"
 #include "tensorflow/lite/micro/micro_op_resolver.h"
 #include "tensorflow/lite/schema/schema_generated.h"
 
 namespace tflite {
 
-// TODO(b/151245712) TODO(b/149408647): remove any version once we no longer
-// support op versions in the API or we switch most users and AllOpsResolver to
-// the new selective registration API, whichever seems more appropriate.
-inline int MicroOpResolverAnyVersion() { return 0; }
-
 template <unsigned int tOpCount>
 class MicroMutableOpResolver : public MicroOpResolver {
  public:
@@ -61,47 +60,71 @@ class MicroMutableOpResolver : public MicroOpResolver {
   }
 
   MicroOpResolver::BuiltinParseFunction GetOpDataParser(
-      tflite::BuiltinOperator) const override {
-    // TODO(b/149408647): Replace with the more selective builtin parser.
-    return ParseOpData;
+      BuiltinOperator op) const override {
+    TFLITE_DCHECK(num_buitin_ops_ <= tOpCount);
+    for (unsigned int i = 0; i < num_buitin_ops_; ++i) {
+      if (builtin_codes_[i] == op) return builtin_parsers_[i];
+    }
+    return nullptr;
+  }
+
+  // The Add* functions below add the various Builtin operators to the
+  // MicroMutableOpResolver object.
+  //
+  // This API is currently experimental (and only supported for a small subset
+  // of operators). It will soon be preferred over the AddBuiltin override of
+  // the MicroOpResolver interface for the following reason:
+  //  * If all calls to AddBuiltin for an application use this API, the code
+  //    size will be smaller by 5-8K (compared to the using the AddBuiltin
+  //    override).
+
+  TfLiteStatus AddDequantize() {
+    // TODO(b/149408647): Replace ParseOpData with the operator specific parse
+    // function once cl/313453102 lands.
+    return AddBuiltin(BuiltinOperator_DEQUANTIZE,
+                      *tflite::ops::micro::Register_DEQUANTIZE(), ParseOpData);
+  }
+
+  TfLiteStatus AddFullyConnected() {
+    // TODO(b/149408647): Replace ParseOpData with the operator specific parse
+    // function once cl/313453102 lands.
+    return AddBuiltin(BuiltinOperator_FULLY_CONNECTED,
+                      *tflite::ops::micro::Register_FULLY_CONNECTED(),
+                      ParseOpData);
+  }
+
+  TfLiteStatus AddQuantize() {
+    // TODO(b/149408647): Replace ParseOpData with the operator specific parse
+    // function once cl/313453102 lands.
+    return AddBuiltin(BuiltinOperator_QUANTIZE,
+                      *tflite::ops::micro::Register_QUANTIZE(), ParseOpData);
+  }
+
+  TfLiteStatus AddSoftmax() {
+    // TODO(b/149408647): Replace ParseOpData with the operator specific parse
+    // function once cl/313453102 lands.
+    return AddBuiltin(BuiltinOperator_SOFTMAX,
+                      *tflite::ops::micro::Register_SOFTMAX(), ParseOpData);
+  }
+
+  TfLiteStatus AddSvdf() {
+    // TODO(b/149408647): Replace ParseOpData with the operator specific parse
+    // function once cl/313453102 lands.
+    return AddBuiltin(BuiltinOperator_SVDF,
+                      *tflite::ops::micro::Register_SVDF(), ParseOpData);
   }
 
   TfLiteStatus AddBuiltin(tflite::BuiltinOperator op,
-                          TfLiteRegistration* registration,
-                          int version = 1) override {
-    if (registrations_len_ >= tOpCount) {
-      if (error_reporter_) {
-        TF_LITE_REPORT_ERROR(error_reporter_,
-                             "Couldn't register builtin op #%d, resolver size "
-                             "is too small (%d)",
-                             op, tOpCount);
-      }
-      return kTfLiteError;
-    }
-
-    if (FindOp(op) != nullptr) {
-      if (error_reporter_ != nullptr) {
-        TF_LITE_REPORT_ERROR(error_reporter_,
-                             "Registering multiple versions of the same op is "
-                             "not supported (Op: #%d, version: %d).",
-                             op, version);
-      }
-      return kTfLiteError;
-    }
-
-    TfLiteRegistration* new_registration = &registrations_[registrations_len_];
-    registrations_len_ += 1;
-
-    *new_registration = *registration;
-    new_registration->builtin_code = op;
-    new_registration->version = version;
-
-    return kTfLiteOk;
+                          TfLiteRegistration* registration) override {
+    TFLITE_DCHECK(registration != nullptr);
+    // For code that is not switched over to the new selective registration of
+    // the parse function, we pass in ParseOpData. This allows for backwards
+    // compatibility.
+    return AddBuiltin(op, *registration, ParseOpData);
   }
 
-  TfLiteStatus AddCustom(const char* name, TfLiteRegistration* registration,
-                         int version = 1) {
-    printf("registrations_len_: %d\n", registrations_len_);
+  TfLiteStatus AddCustom(const char* name,
+                         TfLiteRegistration* registration) override {
     if (registrations_len_ >= tOpCount) {
       if (error_reporter_) {
         TF_LITE_REPORT_ERROR(
@@ -115,9 +138,9 @@ class MicroMutableOpResolver : public MicroOpResolver {
     if (FindOp(name) != nullptr) {
       if (error_reporter_ != nullptr) {
         TF_LITE_REPORT_ERROR(error_reporter_,
-                             "Registering multiple versions of the same op is "
-                             "not supported (Op: %s, version: %d).",
-                             name, version);
+                             "Calling AddCustom for the same op more than once "
+                             "is not supported (Op: %s).",
+                             name);
       }
       return kTfLiteError;
     }
@@ -128,16 +151,66 @@ class MicroMutableOpResolver : public MicroOpResolver {
     *new_registration = *registration;
     new_registration->builtin_code = BuiltinOperator_CUSTOM;
     new_registration->custom_name = name;
-    new_registration->version = version;
-
     return kTfLiteOk;
   }
 
   unsigned int GetRegistrationLength() { return registrations_len_; }
 
  private:
+  TfLiteStatus AddBuiltin(tflite::BuiltinOperator op,
+                          const TfLiteRegistration& registration,
+                          MicroOpResolver::BuiltinParseFunction parser) {
+    if (op == BuiltinOperator_CUSTOM) {
+      if (error_reporter_ != nullptr) {
+        TF_LITE_REPORT_ERROR(error_reporter_,
+                             "Invalid parameter BuiltinOperator_CUSTOM to the "
+                             "AddBuiltin function.");
+      }
+      return kTfLiteError;
+    }
+
+    if (FindOp(op) != nullptr) {
+      if (error_reporter_ != nullptr) {
+        TF_LITE_REPORT_ERROR(error_reporter_,
+                             "Calling AddBuiltin with the same op more than "
+                             "once is not supported (Op: #%d).",
+                             op);
+      }
+      return kTfLiteError;
+    }
+
+    if (registrations_len_ >= tOpCount) {
+      if (error_reporter_) {
+        TF_LITE_REPORT_ERROR(error_reporter_,
+                             "Couldn't register builtin op #%d, resolver size "
+                             "is too small (%d).",
+                             op, tOpCount);
+      }
+      return kTfLiteError;
+    }
+
+    registrations_[registrations_len_] = registration;
+    // Strictly speaking, the builtin_code is not necessary for TFLM but filling
+    // it in regardless.
+    registrations_[registrations_len_].builtin_code = op;
+    registrations_len_++;
+
+    builtin_codes_[num_buitin_ops_] = op;
+    builtin_parsers_[num_buitin_ops_] = parser;
+    num_buitin_ops_++;
+
+    return kTfLiteOk;
+  }
+
   TfLiteRegistration registrations_[tOpCount];
   unsigned int registrations_len_ = 0;
+
+  // Arrays (and counter) to store the builtin codes and their corresponding
+  // parse functions as these are registered with the Op Resolver.
+  BuiltinOperator builtin_codes_[tOpCount];
+  MicroOpResolver::BuiltinParseFunction builtin_parsers_[tOpCount];
+  unsigned int num_buitin_ops_ = 0;
+
   ErrorReporter* error_reporter_;
 
   TF_LITE_REMOVE_VIRTUAL_DELETE
diff --git a/tensorflow/lite/micro/micro_mutable_op_resolver_test.cc b/tensorflow/lite/micro/micro_mutable_op_resolver_test.cc
index 82f1c4bf1db..ff5dfdf3a9a 100644
--- a/tensorflow/lite/micro/micro_mutable_op_resolver_test.cc
+++ b/tensorflow/lite/micro/micro_mutable_op_resolver_test.cc
@@ -70,24 +70,18 @@ TF_LITE_MICRO_TEST(TestOperations) {
 
   MicroMutableOpResolver<2> micro_op_resolver;
   TF_LITE_MICRO_EXPECT_EQ(
-      kTfLiteOk, micro_op_resolver.AddBuiltin(BuiltinOperator_CONV_2D, &r, 1));
+      kTfLiteOk, micro_op_resolver.AddBuiltin(BuiltinOperator_CONV_2D, &r));
 
-  // Only one AddBuiltin per operator should return kTfLiteOk, regardless of
-  // what the version parameter is.
-  TF_LITE_MICRO_EXPECT_EQ(kTfLiteError, micro_op_resolver.AddBuiltin(
-                                            BuiltinOperator_CONV_2D, &r, 1));
-  TF_LITE_MICRO_EXPECT_EQ(kTfLiteError, micro_op_resolver.AddBuiltin(
-                                            BuiltinOperator_CONV_2D, &r, 2));
+  // Only one AddBuiltin per operator should return kTfLiteOk.
+  TF_LITE_MICRO_EXPECT_EQ(
+      kTfLiteError, micro_op_resolver.AddBuiltin(BuiltinOperator_CONV_2D, &r));
 
   TF_LITE_MICRO_EXPECT_EQ(kTfLiteOk,
-                          micro_op_resolver.AddCustom("mock_custom", &r, 2));
+                          micro_op_resolver.AddCustom("mock_custom", &r));
 
-  // Only one AddCustom per operator should return kTfLiteOk, regardless of
-  // what the version parameter is.
+  // Only one AddCustom per operator should return kTfLiteOk.
   TF_LITE_MICRO_EXPECT_EQ(kTfLiteError,
-                          micro_op_resolver.AddCustom("mock_custom", &r, 2));
-  TF_LITE_MICRO_EXPECT_EQ(kTfLiteError,
-                          micro_op_resolver.AddCustom("mock_custom", &r, 1));
+                          micro_op_resolver.AddCustom("mock_custom", &r));
 
   tflite::MicroOpResolver* resolver = &micro_op_resolver;
 
diff --git a/tensorflow/lite/micro/micro_op_resolver.h b/tensorflow/lite/micro/micro_op_resolver.h
index 68035b3256e..0f5528d7b70 100644
--- a/tensorflow/lite/micro/micro_op_resolver.h
+++ b/tensorflow/lite/micro/micro_op_resolver.h
@@ -46,30 +46,24 @@ class MicroOpResolver : public OpResolver {
 
   // Registers a Builtin Operator with the MicroOpResolver.
   //
-  // Note that the version parameter is ignored and only a first call for a
-  // given BuiltinOperator enum will be successful. i.e. if this function is
-  // called again for a previously added BuiltinOperator (even with a different
-  // version parameter), the MicroOpResolver will be unchanged and this function
+  // Only the first call for a given BuiltinOperator enum will be successful.
+  // i.e. if this function is called again for a previously added
+  // BuiltinOperator, the MicroOpResolver will be unchanged and this function
   // will return kTfLiteError.
   //
-  // TODO(b/151245712): The version param is kept to avoid breaking the legacy
-  // API but it should be removed eventually.
+  // TODO(b/149408647): remove this API once the templated AddBuiltin API in
+  // MicroMutableOpResolver is properly implemented.
   virtual TfLiteStatus AddBuiltin(tflite::BuiltinOperator op,
-                                  TfLiteRegistration* registration,
-                                  int version) = 0;
+                                  TfLiteRegistration* registration) = 0;
 
   // Registers a Custom Operator with the MicroOpResolver.
   //
-  // Note that the version parameter is ignored and only a first call for a
-  // given name will be successful. i.e. if this function is called again for a
-  // previously added Custom Operator (even with a different version parameter),
-  // the MicroOpResolver will be unchanged and this function will return
+  // Only the first call for a given name will be successful. i.e. if this
+  // function is called again for a previously added Custom Operator, the
+  // MicroOpResolver will be unchanged and this function will return
   // kTfLiteError.
-  //
-  // TODO(b/151245712): The version param is kept to avoid breaking the legacy
-  // API but it should be removed eventually.
-  TfLiteStatus AddCustom(const char* name, TfLiteRegistration* registration,
-                         int version);
+  virtual TfLiteStatus AddCustom(const char* name,
+                                 TfLiteRegistration* registration) = 0;
 
   // Returns the Op registration struct corresponding to the enum code from the
   // flatbuffer schema. Returns nullptr if the op is not found or if op ==
diff --git a/tensorflow/lite/micro/recording_micro_allocator.cc b/tensorflow/lite/micro/recording_micro_allocator.cc
new file mode 100644
index 00000000000..96cbb6e00e5
--- /dev/null
+++ b/tensorflow/lite/micro/recording_micro_allocator.cc
@@ -0,0 +1,149 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include "tensorflow/lite/micro/recording_micro_allocator.h"
+
+#include "tensorflow/lite/core/api/error_reporter.h"
+#include "tensorflow/lite/micro/compatibility.h"
+#include "tensorflow/lite/micro/recording_simple_memory_allocator.h"
+
+namespace tflite {
+
+RecordingMicroAllocator::RecordingMicroAllocator(
+    TfLiteContext* context, const Model* model,
+    RecordingSimpleMemoryAllocator* recording_memory_allocator,
+    ErrorReporter* error_reporter)
+    : MicroAllocator(context, model, recording_memory_allocator,
+                     error_reporter),
+      recording_memory_allocator_(recording_memory_allocator) {}
+
+RecordedAllocation RecordingMicroAllocator::GetRecordedAllocation(
+    RecordedAllocationType allocation_type) {
+  switch (allocation_type) {
+    case RecordedAllocationType::kTfLiteTensorArray:
+      return recorded_tflite_tensor_array_data_;
+    case RecordedAllocationType::kTfLiteTensorArrayQuantizationData:
+      return recorded_tflite_tensor_array_quantization_data_;
+    case RecordedAllocationType::kNodeAndRegistrationArray:
+      return recorded_node_and_registration_array_data_;
+    case RecordedAllocationType::kOpData:
+      return recorded_op_data_;
+  }
+  TF_LITE_REPORT_ERROR(error_reporter(), "Invalid allocation type supplied: %d",
+                       allocation_type);
+  return RecordedAllocation();
+}
+
+void RecordingMicroAllocator::PrintAllocations() {
+  TF_LITE_REPORT_ERROR(
+      error_reporter(),
+      "[RecordingMicroAllocator] Arena allocation total %d bytes",
+      recording_memory_allocator_->GetUsedBytes());
+  TF_LITE_REPORT_ERROR(
+      error_reporter(),
+      "[RecordingMicroAllocator] Arena allocation head %d bytes",
+      recording_memory_allocator_->GetHeadUsedBytes());
+  TF_LITE_REPORT_ERROR(
+      error_reporter(),
+      "[RecordingMicroAllocator] Arena allocation tail %d bytes",
+      recording_memory_allocator_->GetTailUsedBytes());
+  PrintRecordedAllocation(RecordedAllocationType::kTfLiteTensorArray,
+                          "TfLiteTensor struct allocation");
+  PrintRecordedAllocation(
+      RecordedAllocationType::kTfLiteTensorArrayQuantizationData,
+      "TfLiteTensor quantization data allocations");
+  PrintRecordedAllocation(RecordedAllocationType::kNodeAndRegistrationArray,
+                          "NodeAndRegistration struct allocation");
+  PrintRecordedAllocation(RecordedAllocationType::kOpData,
+                          "Operator runtime data allocation");
+}
+
+void RecordingMicroAllocator::PrintRecordedAllocation(
+    RecordedAllocationType allocation_type, const char* allocation_name) {
+  RecordedAllocation allocation = GetRecordedAllocation(allocation_type);
+  TF_LITE_REPORT_ERROR(error_reporter(),
+                       "[RecordingMicroAllocator] '%s' used %d bytes "
+                       "(requested %d bytes %d times)",
+                       allocation_name, allocation.used_bytes,
+                       allocation.requested_bytes, allocation.count);
+}
+
+TfLiteStatus RecordingMicroAllocator::AllocateTfLiteTensorArray() {
+  SnapshotAllocationUsage(recorded_tflite_tensor_array_data_);
+
+  TfLiteStatus status = MicroAllocator::AllocateTfLiteTensorArray();
+
+  RecordAllocationUsage(recorded_tflite_tensor_array_data_);
+  recorded_tflite_tensor_array_data_.count = GetTensorsCount();
+  return status;
+}
+
+TfLiteStatus
+RecordingMicroAllocator::PopulateTfLiteTensorArrayFromFlatbuffer() {
+  SnapshotAllocationUsage(recorded_tflite_tensor_array_quantization_data_);
+
+  TfLiteStatus status =
+      MicroAllocator::PopulateTfLiteTensorArrayFromFlatbuffer();
+
+  RecordAllocationUsage(recorded_tflite_tensor_array_quantization_data_);
+  return status;
+}
+
+TfLiteStatus RecordingMicroAllocator::AllocateNodeAndRegistrations(
+    NodeAndRegistration** node_and_registrations) {
+  SnapshotAllocationUsage(recorded_node_and_registration_array_data_);
+
+  TfLiteStatus status =
+      MicroAllocator::AllocateNodeAndRegistrations(node_and_registrations);
+
+  RecordAllocationUsage(recorded_node_and_registration_array_data_);
+  recorded_node_and_registration_array_data_.count = GetOperatorsCount();
+  return status;
+}
+
+TfLiteStatus
+RecordingMicroAllocator::PrepareNodeAndRegistrationDataFromFlatbuffer(
+    const MicroOpResolver& op_resolver,
+    NodeAndRegistration* node_and_registrations) {
+  SnapshotAllocationUsage(recorded_op_data_);
+
+  TfLiteStatus status =
+      MicroAllocator::PrepareNodeAndRegistrationDataFromFlatbuffer(
+          op_resolver, node_and_registrations);
+
+  RecordAllocationUsage(recorded_op_data_);
+  return status;
+}
+
+void RecordingMicroAllocator::SnapshotAllocationUsage(
+    RecordedAllocation& recorded_allocation) {
+  recorded_allocation.requested_bytes =
+      recording_memory_allocator_->GetRequestedBytes();
+  recorded_allocation.used_bytes = recording_memory_allocator_->GetUsedBytes();
+  recorded_allocation.count = recording_memory_allocator_->GetAllocatedCount();
+}
+
+void RecordingMicroAllocator::RecordAllocationUsage(
+    RecordedAllocation& recorded_allocation) {
+  recorded_allocation.requested_bytes =
+      recording_memory_allocator_->GetRequestedBytes() -
+      recorded_allocation.requested_bytes;
+  recorded_allocation.used_bytes = recording_memory_allocator_->GetUsedBytes() -
+                                   recorded_allocation.used_bytes;
+  recorded_allocation.count = recording_memory_allocator_->GetAllocatedCount() -
+                              recorded_allocation.count;
+}
+
+}  // namespace tflite
diff --git a/tensorflow/lite/micro/recording_micro_allocator.h b/tensorflow/lite/micro/recording_micro_allocator.h
new file mode 100644
index 00000000000..4a70c955cda
--- /dev/null
+++ b/tensorflow/lite/micro/recording_micro_allocator.h
@@ -0,0 +1,92 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#ifndef TENSORFLOW_LITE_MICRO_RECORDING_MICRO_ALLOCATOR_H_
+#define TENSORFLOW_LITE_MICRO_RECORDING_MICRO_ALLOCATOR_H_
+
+#include "tensorflow/lite/micro/compatibility.h"
+#include "tensorflow/lite/micro/micro_allocator.h"
+#include "tensorflow/lite/micro/recording_simple_memory_allocator.h"
+
+namespace tflite {
+
+// List of buckets currently recorded by this class. Each type keeps a list of
+// allocated information during model initialization.
+enum class RecordedAllocationType {
+  kTfLiteTensorArray,
+  kTfLiteTensorArrayQuantizationData,
+  kNodeAndRegistrationArray,
+  kOpData
+};
+
+// Container for holding information about allocation recordings by a given
+// type. Each recording contains the number of bytes requested, the actual bytes
+// allocated (can defer from requested by alignment), and the number of items
+// allocated.
+typedef struct RecordedAllocation {
+  RecordedAllocation() : requested_bytes(0), used_bytes(0), count(0) {}
+  size_t requested_bytes;
+  size_t used_bytes;
+  size_t count;
+} RecordedAllocation;
+
+// Utility subclass of MicroAllocator that records all allocations
+// inside the arena. A summary of allocations can be logged through the
+// ErrorReporter by invoking LogAllocations(). Individual allocation recordings
+// can be retrieved by type through the GetRecordedAllocation() function. This
+// class should only be used for auditing memory usage or integration testing.
+class RecordingMicroAllocator : public MicroAllocator {
+ public:
+  RecordingMicroAllocator(TfLiteContext* context, const Model* model,
+                          RecordingSimpleMemoryAllocator* memory_allocator,
+                          ErrorReporter* error_reporter);
+
+  // Returns the recorded allocations information for a given allocation type.
+  RecordedAllocation GetRecordedAllocation(
+      RecordedAllocationType allocation_type);
+
+  // Logs out through the ErrorReporter all allocation recordings by type
+  // defined in RecordedAllocationType.
+  void PrintAllocations();
+
+ protected:
+  TfLiteStatus AllocateTfLiteTensorArray() override;
+  TfLiteStatus PopulateTfLiteTensorArrayFromFlatbuffer() override;
+  TfLiteStatus AllocateNodeAndRegistrations(
+      NodeAndRegistration** node_and_registrations) override;
+  TfLiteStatus PrepareNodeAndRegistrationDataFromFlatbuffer(
+      const MicroOpResolver& op_resolver,
+      NodeAndRegistration* node_and_registrations) override;
+
+  void SnapshotAllocationUsage(RecordedAllocation& recorded_allocation);
+  void RecordAllocationUsage(RecordedAllocation& recorded_allocation);
+
+ private:
+  void PrintRecordedAllocation(RecordedAllocationType allocation_type,
+                               const char* allocation_name);
+
+  RecordingSimpleMemoryAllocator* recording_memory_allocator_;
+
+  RecordedAllocation recorded_tflite_tensor_array_data_;
+  RecordedAllocation recorded_tflite_tensor_array_quantization_data_;
+  RecordedAllocation recorded_node_and_registration_array_data_;
+  RecordedAllocation recorded_op_data_;
+
+  TF_LITE_REMOVE_VIRTUAL_DELETE
+};
+
+}  // namespace tflite
+
+#endif  // TENSORFLOW_LITE_MICRO_RECORDING_MICRO_ALLOCATOR_H_
diff --git a/tensorflow/lite/micro/recording_micro_allocator_test.cc b/tensorflow/lite/micro/recording_micro_allocator_test.cc
new file mode 100644
index 00000000000..7e1a1beeaeb
--- /dev/null
+++ b/tensorflow/lite/micro/recording_micro_allocator_test.cc
@@ -0,0 +1,185 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include "tensorflow/lite/micro/recording_micro_allocator.h"
+
+#include "tensorflow/lite/micro/all_ops_resolver.h"
+#include "tensorflow/lite/micro/test_helpers.h"
+#include "tensorflow/lite/micro/testing/micro_test.h"
+#include "tensorflow/lite/micro/testing/test_conv_model.h"
+
+#define TF_LITE_TENSOR_STRUCT_SIZE sizeof(TfLiteTensor)
+#define TF_LITE_AFFINE_QUANTIZATION_SIZE sizeof(TfLiteAffineQuantization)
+#define NODE_AND_REGISTRATION_STRUCT_SIZE sizeof(tflite::NodeAndRegistration)
+
+// TODO(b/158303868): Move tests into anonymous namespace.
+namespace {
+
+constexpr int kTestConvArenaSize = 1024 * 12;
+
+}  // namespace
+
+TF_LITE_MICRO_TESTS_BEGIN
+
+TF_LITE_MICRO_TEST(TestRecordedValuesDefaultToZero) {
+  TfLiteContext context;
+  const tflite::Model* model = tflite::testing::GetSimpleMockModel();
+  constexpr size_t arena_size = 1024;
+  uint8_t arena[arena_size];
+
+  tflite::RecordingSimpleMemoryAllocator memory_allocator(micro_test::reporter,
+                                                          arena, arena_size);
+  tflite::RecordingMicroAllocator micro_allocator(
+      &context, model, &memory_allocator, micro_test::reporter);
+
+  tflite::RecordedAllocation recorded_allocation;
+
+  recorded_allocation = micro_allocator.GetRecordedAllocation(
+      tflite::RecordedAllocationType::kTfLiteTensorArray);
+  TF_LITE_MICRO_EXPECT_EQ(0, recorded_allocation.requested_bytes);
+  TF_LITE_MICRO_EXPECT_EQ(0, recorded_allocation.used_bytes);
+  TF_LITE_MICRO_EXPECT_EQ(0, recorded_allocation.count);
+
+  recorded_allocation = micro_allocator.GetRecordedAllocation(
+      tflite::RecordedAllocationType::kTfLiteTensorArrayQuantizationData);
+  TF_LITE_MICRO_EXPECT_EQ(0, recorded_allocation.requested_bytes);
+  TF_LITE_MICRO_EXPECT_EQ(0, recorded_allocation.used_bytes);
+  TF_LITE_MICRO_EXPECT_EQ(0, recorded_allocation.count);
+
+  recorded_allocation = micro_allocator.GetRecordedAllocation(
+      tflite::RecordedAllocationType::kNodeAndRegistrationArray);
+  TF_LITE_MICRO_EXPECT_EQ(0, recorded_allocation.requested_bytes);
+  TF_LITE_MICRO_EXPECT_EQ(0, recorded_allocation.used_bytes);
+  TF_LITE_MICRO_EXPECT_EQ(0, recorded_allocation.count);
+
+  recorded_allocation = micro_allocator.GetRecordedAllocation(
+      tflite::RecordedAllocationType::kOpData);
+  TF_LITE_MICRO_EXPECT_EQ(0, recorded_allocation.requested_bytes);
+  TF_LITE_MICRO_EXPECT_EQ(0, recorded_allocation.used_bytes);
+  TF_LITE_MICRO_EXPECT_EQ(0, recorded_allocation.count);
+}
+
+TF_LITE_MICRO_TEST(TestRecordsTfLiteTensorArrayData) {
+  TfLiteContext context;
+  const tflite::Model* model = tflite::GetModel(kTestConvModelData);
+  uint8_t arena[kTestConvArenaSize];
+  tflite::RecordingSimpleMemoryAllocator memory_allocator(
+      micro_test::reporter, arena, kTestConvArenaSize);
+  tflite::RecordingMicroAllocator allocator(&context, model, &memory_allocator,
+                                            micro_test::reporter);
+  TfLiteStatus status = allocator.Init();
+
+  // TODO(b/158102673): Ugly workaround for not having fatal test assertions:
+  TF_LITE_MICRO_EXPECT_EQ(kTfLiteOk, status);
+  if (status != kTfLiteOk) return 1;
+
+  tflite::RecordedAllocation recorded_allocation =
+      allocator.GetRecordedAllocation(
+          tflite::RecordedAllocationType::kTfLiteTensorArray);
+  TF_LITE_MICRO_EXPECT_EQ(recorded_allocation.count, context.tensors_size);
+  TF_LITE_MICRO_EXPECT_EQ(recorded_allocation.requested_bytes,
+                          context.tensors_size * TF_LITE_TENSOR_STRUCT_SIZE);
+  TF_LITE_MICRO_EXPECT_GE(recorded_allocation.used_bytes,
+                          context.tensors_size * TF_LITE_TENSOR_STRUCT_SIZE);
+}
+
+TF_LITE_MICRO_TEST(TestRecordsTensorArrayQuantizationData) {
+  TfLiteContext context;
+  const tflite::Model* model = tflite::GetModel(kTestConvModelData);
+  uint8_t arena[kTestConvArenaSize];
+  tflite::RecordingSimpleMemoryAllocator memory_allocator(
+      micro_test::reporter, arena, kTestConvArenaSize);
+  tflite::RecordingMicroAllocator allocator(&context, model, &memory_allocator,
+                                            micro_test::reporter);
+  TfLiteStatus status = allocator.Init();
+
+  // TODO(b/158102673): Ugly workaround for not having fatal test assertions:
+  TF_LITE_MICRO_EXPECT_EQ(kTfLiteOk, status);
+  if (status != kTfLiteOk) return 1;
+
+  // Walk the model subgraph to find all tensors with quantization params and
+  // keep a tally.
+  size_t quantized_tensor_count = 0;
+  size_t quantized_channel_bytes = 0;
+  for (size_t i = 0; i < context.tensors_size; ++i) {
+    const tflite::Tensor* cur_tensor =
+        model->subgraphs()->Get(0)->tensors()->Get(i);
+    const tflite::QuantizationParameters* quantization_params =
+        cur_tensor->quantization();
+    if (quantization_params && quantization_params->scale() &&
+        quantization_params->scale()->size() > 0 &&
+        quantization_params->zero_point() &&
+        quantization_params->zero_point()->size() > 0) {
+      quantized_tensor_count++;
+      size_t num_channels = quantization_params->scale()->size();
+      quantized_channel_bytes += TfLiteIntArrayGetSizeInBytes(num_channels);
+      quantized_channel_bytes += TfLiteFloatArrayGetSizeInBytes(num_channels);
+    }
+  }
+
+  // Calculate the expected allocation bytes with subgraph quantization data:
+  size_t expected_requested_bytes =
+      quantized_tensor_count * TF_LITE_AFFINE_QUANTIZATION_SIZE +
+      quantized_channel_bytes;
+
+  tflite::RecordedAllocation recorded_allocation =
+      allocator.GetRecordedAllocation(
+          tflite::RecordedAllocationType::kTfLiteTensorArrayQuantizationData);
+
+  // Each quantized tensors has 3 mallocs (quant struct, scale dimensions, zero
+  // point dimensions):
+  TF_LITE_MICRO_EXPECT_EQ(recorded_allocation.count,
+                          quantized_tensor_count * 3);
+  TF_LITE_MICRO_EXPECT_EQ(recorded_allocation.requested_bytes,
+                          expected_requested_bytes);
+  TF_LITE_MICRO_EXPECT_GE(recorded_allocation.used_bytes,
+                          expected_requested_bytes);
+}
+
+TF_LITE_MICRO_TEST(TestRecordsNodeAndRegistrationArrayData) {
+  TfLiteContext context;
+  const tflite::Model* model = tflite::GetModel(kTestConvModelData);
+  uint8_t arena[kTestConvArenaSize];
+  tflite::RecordingSimpleMemoryAllocator memory_allocator(
+      micro_test::reporter, arena, kTestConvArenaSize);
+  tflite::RecordingMicroAllocator allocator(&context, model, &memory_allocator,
+                                            micro_test::reporter);
+  TfLiteStatus status = allocator.Init();
+
+  // TODO(b/158102673): Ugly workaround for not having fatal test assertions:
+  TF_LITE_MICRO_EXPECT_EQ(kTfLiteOk, status);
+  if (status != kTfLiteOk) return 1;
+
+  tflite::AllOpsResolver ops_resolver;
+  tflite::NodeAndRegistration* node_and_registrations;
+  TF_LITE_MICRO_EXPECT_EQ(
+      kTfLiteOk,
+      allocator.PrepareFromFlatbuffer(ops_resolver, &node_and_registrations));
+
+  size_t num_ops = model->subgraphs()->Get(0)->operators()->size();
+  tflite::RecordedAllocation recorded_allocation =
+      allocator.GetRecordedAllocation(
+          tflite::RecordedAllocationType::kNodeAndRegistrationArray);
+  TF_LITE_MICRO_EXPECT_EQ(recorded_allocation.count, num_ops);
+  TF_LITE_MICRO_EXPECT_EQ(recorded_allocation.requested_bytes,
+                          num_ops * NODE_AND_REGISTRATION_STRUCT_SIZE);
+  TF_LITE_MICRO_EXPECT_GE(recorded_allocation.used_bytes,
+                          num_ops * NODE_AND_REGISTRATION_STRUCT_SIZE);
+}
+
+// TODO(b/158124094): Find a way to audit OpData allocations on
+// cross-architectures.
+
+TF_LITE_MICRO_TESTS_END
diff --git a/tensorflow/lite/micro/simple_memory_allocator.cc b/tensorflow/lite/micro/simple_memory_allocator.cc
index 65d60161e9a..3b416047c8f 100644
--- a/tensorflow/lite/micro/simple_memory_allocator.cc
+++ b/tensorflow/lite/micro/simple_memory_allocator.cc
@@ -17,6 +17,7 @@ limitations under the License.
 
 #include <cstddef>
 #include <cstdint>
+#include <new>
 
 #include "tensorflow/lite/core/api/error_reporter.h"
 #include "tensorflow/lite/micro/memory_helpers.h"
@@ -48,8 +49,8 @@ SimpleMemoryAllocator* SimpleMemoryAllocator::Create(
   // allocator instance.
   uint8_t* allocator_buffer = tmp.AllocateFromTail(
       sizeof(SimpleMemoryAllocator), alignof(SimpleMemoryAllocator));
-  return new (allocator_buffer)
-      SimpleMemoryAllocator(error_reporter, tmp.head_, tmp.tail_);
+  // Use the default copy constructor to populate internal states.
+  return new (allocator_buffer) SimpleMemoryAllocator(tmp);
 }
 
 SimpleMemoryAllocator::~SimpleMemoryAllocator() {}
diff --git a/tensorflow/lite/micro/testing/BUILD b/tensorflow/lite/micro/testing/BUILD
index 29baad15e87..3d97d2fec1e 100644
--- a/tensorflow/lite/micro/testing/BUILD
+++ b/tensorflow/lite/micro/testing/BUILD
@@ -45,7 +45,9 @@ cc_library(
     ],
     deps = [
         "//tensorflow/lite/micro:micro_error_reporter",
+        "//tensorflow/lite/micro:micro_framework",
         "//tensorflow/lite/micro:micro_time",
+        "//tensorflow/lite/micro:op_resolvers",
     ],
 )
 
diff --git a/tensorflow/lite/micro/testing/micro_benchmark.h b/tensorflow/lite/micro/testing/micro_benchmark.h
index f059842ffd1..4f660308f06 100644
--- a/tensorflow/lite/micro/testing/micro_benchmark.h
+++ b/tensorflow/lite/micro/testing/micro_benchmark.h
@@ -18,7 +18,9 @@ limitations under the License.
 
 #include <climits>
 
+#include "tensorflow/lite/micro/all_ops_resolver.h"
 #include "tensorflow/lite/micro/micro_error_reporter.h"
+#include "tensorflow/lite/micro/micro_interpreter.h"
 #include "tensorflow/lite/micro/micro_time.h"
 
 namespace micro_benchmark {
@@ -46,7 +48,7 @@ extern tflite::ErrorReporter* reporter;
     return 0;                                                                 \
   }                                                                           \
   start_ticks = tflite::GetCurrentTimeTicks();                                \
-  func();                                                                     \
+  func;                                                                       \
   duration_ticks = tflite::GetCurrentTimeTicks() - start_ticks;               \
   if (duration_ticks > INT_MAX / 1000) {                                      \
     duration_ms = duration_ticks / (tflite::ticks_per_second() / 1000);       \
@@ -56,4 +58,62 @@ extern tflite::ErrorReporter* reporter;
   TF_LITE_REPORT_ERROR(micro_benchmark::reporter, "%s took %d ticks (%d ms)", \
                        #func, duration_ticks, duration_ms);
 
+template <typename inputT>
+class MicroBenchmarkRunner {
+ public:
+  MicroBenchmarkRunner(const uint8_t* model, uint8_t* tensor_arena,
+                       int tensor_arena_size, int random_seed)
+      : model_(tflite::GetModel(model)),
+        reporter_(&micro_reporter_),
+        interpreter_(model_, resolver_, tensor_arena, tensor_arena_size,
+                     reporter_) {
+    interpreter_.AllocateTensors();
+
+    // The pseudo-random number generator is initialized to a constant seed
+    std::srand(random_seed);
+    TfLiteTensor* input = interpreter_.input(0);
+
+    // Pre-populate input tensor with random values.
+    int input_length = input->bytes / sizeof(inputT);
+    inputT* input_values = tflite::GetTensorData<inputT>(input);
+    for (int i = 0; i < input_length; i++) {
+      // Pre-populate input tensor with a random value based on a constant seed.
+      input_values[i] = static_cast<inputT>(
+          std::rand() % (std::numeric_limits<inputT>::max() -
+                         std::numeric_limits<inputT>::min() + 1));
+    }
+  }
+
+  void RunSingleIterationRandomInput() {
+    // Run the model on this input and make sure it succeeds.
+    TfLiteStatus invoke_status = interpreter_.Invoke();
+    if (invoke_status != kTfLiteOk) {
+      TF_LITE_REPORT_ERROR(reporter_, "Invoke failed.");
+    }
+  }
+
+  void RunSingleIterationCustomInput(const inputT* custom_input) {
+    // Populate input tensor with an image with no person.
+    TfLiteTensor* input = interpreter_.input(0);
+    inputT* input_buffer = tflite::GetTensorData<inputT>(input);
+    int input_length = input->bytes / sizeof(inputT);
+    for (int i = 0; i < input_length; i++) {
+      input_buffer[i] = custom_input[i];
+    }
+
+    // Run the model on this input and make sure it succeeds.
+    TfLiteStatus invoke_status = interpreter_.Invoke();
+    if (invoke_status != kTfLiteOk) {
+      TF_LITE_REPORT_ERROR(reporter_, "Invoke failed.");
+    }
+  }
+
+ private:
+  const tflite::Model* model_;
+  tflite::MicroErrorReporter micro_reporter_;
+  tflite::ErrorReporter* reporter_;
+  tflite::AllOpsResolver resolver_;
+  tflite::MicroInterpreter interpreter_;
+};
+
 #endif  // TENSORFLOW_LITE_MICRO_TESTING_MICRO_BENCHMARK_H_
diff --git a/tensorflow/lite/micro/tools/make/Makefile b/tensorflow/lite/micro/tools/make/Makefile
index c00a8a8ae89..b804868fbb1 100644
--- a/tensorflow/lite/micro/tools/make/Makefile
+++ b/tensorflow/lite/micro/tools/make/Makefile
@@ -116,6 +116,7 @@ MICROLITE_CC_BASE_SRCS := \
 $(wildcard tensorflow/lite/micro/*.cc) \
 $(wildcard tensorflow/lite/micro/kernels/*.cc) \
 $(wildcard tensorflow/lite/micro/memory_planner/*.cc) \
+$(wildcard tensorflow/lite/micro/testing/*model.cc) \
 tensorflow/lite/c/common.c \
 tensorflow/lite/core/api/error_reporter.cc \
 tensorflow/lite/core/api/flatbuffer_conversions.cc \
diff --git a/tensorflow/lite/micro/tools/make/ext_libs/ethosu.inc b/tensorflow/lite/micro/tools/make/ext_libs/ethosu.inc
new file mode 100644
index 00000000000..11dc93e688f
--- /dev/null
+++ b/tensorflow/lite/micro/tools/make/ext_libs/ethosu.inc
@@ -0,0 +1,29 @@
+ifneq ($(filter ethos-u,$(ALL_TAGS)),)
+    # Don't want -lm flag
+    MICROLITE_LIBS :=
+
+    ifneq (,$(filter $(TARGET_ARCH), x86_64))
+        $(error target architecture x86_64 not supported)
+    endif
+
+    THIRD_PARTY_DOWNLOADS += \
+      $(eval $(call add_third_party_download,$(ETHOSU_URL),$(ETHOSU_MD5),ethosu,))
+    ETHOSU_DRIVER_PATH = $(MAKEFILE_DIR)/downloads/ethosu
+
+    # Currently there is a dependency to CMSIS-NN
+    THIRD_PARTY_DOWNLOADS += \
+        $(eval $(call add_third_party_download,$(CMSIS_URL),$(CMSIS_MD5),cmsis,))
+    CMSIS_PATH = $(MAKEFILE_DIR)/downloads/cmsis/
+    THIRD_PARTY_CC_HDRS += $(call recursive_find,$(CMSIS_PATH)/CMSIS/Core/Include,*.h)
+
+    THIRD_PARTY_CC_HDRS += $(call recursive_find,$(ETHOSU_DRIVER_PATH)/include,*.h)
+    ifeq (,$(ETHOSU_DRIVER_LIBS))
+        THIRD_PARTY_CC_SRCS += $(call recursive_find,$(ETHOSU_DRIVER_PATH)/src,*.c)
+    else
+        MICROLITE_LIBS += $(ETHOSU_DRIVER_LIBS)
+    endif
+
+    INCLUDES += -I$(ETHOSU_DRIVER_PATH)/include \
+                -I$(CMSIS_PATH)/CMSIS/Core/Include
+    GENERATED_PROJECT_INCLUDES += -I./$(ETHOSU_DRIVER_PATH)/include
+endif
diff --git a/tensorflow/lite/micro/tools/make/targets/stm32f4_makefile.inc b/tensorflow/lite/micro/tools/make/targets/stm32f4_makefile.inc
index 5114aa4620c..8dd20d4de5b 100644
--- a/tensorflow/lite/micro/tools/make/targets/stm32f4_makefile.inc
+++ b/tensorflow/lite/micro/tools/make/targets/stm32f4_makefile.inc
@@ -58,10 +58,12 @@ ifeq ($(TARGET), stm32f4)
   MICROLITE_CC_SRCS := $(filter-out $(EXCLUDED_SRCS), $(MICROLITE_CC_SRCS))
   TEST_SCRIPT := tensorflow/lite/micro/testing/test_stm32f4_binary.sh
   # TODO, non working tests.. the micro_speech example partly works
+  # TODO(b/158324045): Examine why some tests fail here.
   EXCLUDED_TESTS := \
     tensorflow/lite/micro/micro_interpreter_test.cc \
     tensorflow/lite/micro/micro_allocator_test.cc \
     tensorflow/lite/micro/memory_helpers_test.cc \
+    tensorflow/lite/micro/recording_micro_allocator_test.cc \
     tensorflow/lite/micro/kernels/logistic_test.cc \
     tensorflow/lite/micro/kernels/logical_test.cc \
     tensorflow/lite/micro/kernels/maximum_minimum_test.cc \
diff --git a/tensorflow/lite/micro/tools/make/third_party_downloads.inc b/tensorflow/lite/micro/tools/make/third_party_downloads.inc
index 1543abc63f8..8071901d3e7 100644
--- a/tensorflow/lite/micro/tools/make/third_party_downloads.inc
+++ b/tensorflow/lite/micro/tools/make/third_party_downloads.inc
@@ -83,3 +83,5 @@ ZEPHYR_MD5 := "755622eb4812fde918a6382b65d50c3b"
 XTENSA_HIFI4_URL :="https://github.com/foss-xtensa/nnlib-hifi4/raw/master/archive/xa_nnlib_04_07.zip"
 XTENSA_HIFI4_MD5 :="f234764928f9a42901df33a27e118c8b"
 
+ETHOSU_URL := "https://git.mlplatform.org/ml/ethos-u/ethos-u-core-driver.git/snapshot/ethos-u-core-driver-bcb5aaa99756f1b5c1295b079ebdd60996bc75a5.tar.gz"
+ETHOSU_MD5 := "d2073c8d88fc167fd5c46b5dcda58ea1"
diff --git a/tensorflow/lite/python/convert.py b/tensorflow/lite/python/convert.py
index a5fbb88132e..9ce88ec6c96 100644
--- a/tensorflow/lite/python/convert.py
+++ b/tensorflow/lite/python/convert.py
@@ -108,15 +108,19 @@ class ConverterError(Exception):
   pass
 
 
-def mlir_quantize(input_data_str, disable_per_channel=False,
+def mlir_quantize(input_data_str,
+                  disable_per_channel=False,
+                  fully_quantize=False,
                   inference_type=_types_pb2.INT8):
   """Quantize `input_data_str` with calibration results.
 
   Args:
     input_data_str: Input data in serialized form (e.g. a TFLITE model with
-                    calibration results).
-    disable_per_channel: Bool indicating whether to do per-channel or
-                         per-tensor quantization
+      calibration results).
+    disable_per_channel: Bool indicating whether to do per-channel or per-tensor
+      quantization
+    fully_quantize: Bool indicating whether to fully quantize the model. Besides
+      model body, the input/output will be quantized as well.
     inference_type: Data type for the activations. The default value is int8.
 
   Returns:
@@ -125,6 +129,7 @@ def mlir_quantize(input_data_str, disable_per_channel=False,
   """
   return wrap_toco.wrapped_experimental_mlir_quantize(input_data_str,
                                                       disable_per_channel,
+                                                      fully_quantize,
                                                       inference_type)
 
 
diff --git a/tensorflow/lite/python/lite.py b/tensorflow/lite/python/lite.py
index a68077296dd..93cca1a6af5 100644
--- a/tensorflow/lite/python/lite.py
+++ b/tensorflow/lite/python/lite.py
@@ -201,6 +201,11 @@ class QuantizationMode(object):
             self._representative_dataset is not None and
             self._smallest_supported_type() == constants.INT8)
 
+  def is_post_training_integer_quantize(self):
+    """Post training integer quantization."""
+    return (self.post_training_int8_no_float() or
+            self.post_training_int8_allow_float())
+
   def training_time_int8_allow_float(self):
     """Training-time int8 quantize, allow float fallback."""
     return (self._any_optimization_enabled() and
@@ -231,10 +236,9 @@ class QuantizationMode(object):
 
   def converter_flags(self, inference_ty=None, inference_input_ty=None):
     """Flags to the converter."""
-    if (self.post_training_int8_no_float() or
-        self.post_training_int8_allow_float()):
+    if self.is_post_training_integer_quantize():
       # The inference_input_type is for the quantizer, then we need to keep the
-      # converter inference_iput_type to float.
+      # converter inference_input_type to float.
       inference_input_ty = constants.FLOAT
 
     if self.training_time_int8_allow_float():
@@ -472,7 +476,56 @@ class TFLiteConverterBase(object):
 
 
 class TFLiteConverterBaseV2(TFLiteConverterBase):
-  """Converter subclass to share functionality between V2 converters."""
+  """Converter subclass to share functionality between V2 converters.
+
+  Attributes:
+    allow_custom_ops: Boolean indicating whether to allow custom operations.
+      When False, any unknown operation is an error. When True, custom ops are
+      created for any op that is unknown. The developer needs to provide these
+      to the TensorFlow Lite runtime with a custom resolver. (default False)
+    optimizations: Experimental flag, subject to change. A list of optimizations
+      to apply when converting the model. E.g. `[Optimize.DEFAULT]`
+    representative_dataset: A representative dataset that can be used to
+      generate input and output samples for the model. The converter can use the
+      dataset to evaluate different optimizations. Note that this is an optional
+      attribute but it is necessary if INT8 is the only support builtin ops in
+      target ops.
+    target_spec: Experimental flag, subject to change. Specification of target
+      device.
+    inference_input_type: Data type of the input layer. Note that integer types
+      (tf.int8 and tf.uint8) are currently only supported for post training
+      integer quantization. (default tf.float32, must be in {tf.float32,
+      tf.int8, tf.uint8})
+    inference_output_type: Data type of the output layer. Note that integer
+      types (tf.int8 and tf.uint8) are currently only supported for post
+      training integer quantization. (default tf.float32, must be in
+      {tf.float32, tf.int8, tf.uint8})
+    experimental_new_converter: Experimental flag, subject to change. Enables
+      MLIR-based conversion instead of TOCO conversion.
+  """
+
+  def __init__(self):
+    """Constructor for TFLiteConverter."""
+    super(TFLiteConverterBaseV2, self).__init__()
+    self.inference_input_type = constants.FLOAT
+    self.inference_output_type = constants.FLOAT
+
+  def _validate_inference_input_output_types(self, quant_mode):
+    """Validate inference_input_type and inference_output_type flags."""
+    default_types = [constants.FLOAT, None]
+    # We only support integer types for post training integer quantization
+    # as we have statistical information to quantize the input and output.
+    if quant_mode.is_post_training_integer_quantize():
+      all_types = default_types + [constants.INT8, constants.QUANTIZED_UINT8]
+      if self.inference_input_type not in all_types or \
+          self.inference_output_type not in all_types:
+        all_types_names = ["tf." + t.name for t in all_types]
+        raise ValueError("The inference_input_type and inference_output_type "
+                         "must be in {}.".format(all_types_names))
+    elif self.inference_input_type not in default_types or \
+        self.inference_output_type not in default_types:
+      raise ValueError("The inference_input_type and inference_output_type "
+                       "must be tf.float32.")
 
   def convert(self, graph_def, input_tensors, output_tensors):
     """Converts a TensorFlow GraphDef based on instance variables.
@@ -496,6 +549,8 @@ class TFLiteConverterBaseV2(TFLiteConverterBase):
     quant_mode = QuantizationMode(self.optimizations, self.target_spec,
                                   self.representative_dataset, graph_def)
 
+    self._validate_inference_input_output_types(quant_mode)
+
     if not self._is_unknown_shapes_allowed():
       # Checks dimensions in input tensor.
       for tensor in input_tensors:
@@ -522,13 +577,11 @@ class TFLiteConverterBaseV2(TFLiteConverterBase):
 
     converter_kwargs = self._get_base_converter_args()
     converter_kwargs.update(quant_mode.converter_flags())
-
     if not self.experimental_new_converter:
       logging.warning(
-          "Please consider switching to use new converter by setting "
-          "experimental_new_converter to true. "
-          "Old converter (TOCO) is deprecated and flow will be switched on "
-          "by default to use new converter soon.")
+          "Please consider switching to the new converter by setting "
+          "experimental_new_converter=True. "
+          "The old converter (TOCO) is deprecated.")
     else:
       logging.info("Using experimental converter: If you encountered a problem "
                    "please file a bug. You can opt-out "
@@ -541,7 +594,8 @@ class TFLiteConverterBaseV2(TFLiteConverterBase):
         output_tensors=output_tensors,
         **converter_kwargs)
 
-    calibrate_and_quantize, flags = quant_mode.quantizer_flags()
+    calibrate_and_quantize, flags = quant_mode.quantizer_flags(
+        self.inference_input_type, self.inference_output_type)
     if calibrate_and_quantize:
       result = self._calibrate_quantize_model(result, **flags)
 
@@ -799,12 +853,9 @@ class TFLiteConverterV2(TFLiteFrozenGraphConverterV2):
 
   Attributes:
     allow_custom_ops: Boolean indicating whether to allow custom operations.
-      When false any unknown operation is an error. When true, custom ops are
-      created for any op that is unknown. The developer will need to provide
-      these to the TensorFlow Lite runtime with a custom resolver.
-      (default False)
-    target_spec: Experimental flag, subject to change. Specification of target
-      device.
+      When False, any unknown operation is an error. When True, custom ops are
+      created for any op that is unknown. The developer needs to provide these
+      to the TensorFlow Lite runtime with a custom resolver. (default False)
     optimizations: Experimental flag, subject to change. A list of optimizations
       to apply when converting the model. E.g. `[Optimize.DEFAULT]`
     representative_dataset: A representative dataset that can be used to
@@ -812,21 +863,32 @@ class TFLiteConverterV2(TFLiteFrozenGraphConverterV2):
       dataset to evaluate different optimizations. Note that this is an optional
       attribute but it is necessary if INT8 is the only support builtin ops in
       target ops.
-    experimental_new_converter: Experimental flag, subject to change.
-      Enables MLIR-based conversion instead of TOCO conversion.
+    target_spec: Experimental flag, subject to change. Specification of target
+      device.
+    inference_input_type: Data type of the input layer. Note that integer types
+      (tf.int8 and tf.uint8) are currently only supported for post training
+      integer quantization. (default tf.float32, must be in {tf.float32,
+      tf.int8, tf.uint8})
+    inference_output_type: Data type of the output layer. Note that integer
+      types (tf.int8 and tf.uint8) are currently only supported for post
+      training integer quantization. (default tf.float32, must be in
+      {tf.float32, tf.int8, tf.uint8})
+    experimental_new_converter: Experimental flag, subject to change. Enables
+      MLIR-based conversion instead of TOCO conversion. (default True)
+
   Example usage:
 
     ```python
     # Converting a SavedModel to a TensorFlow Lite model.
-    converter = lite.TFLiteConverter.from_saved_model(saved_model_dir)
+    converter = tf.lite.TFLiteConverter.from_saved_model(saved_model_dir)
     tflite_model = converter.convert()
 
     # Converting a tf.Keras model to a TensorFlow Lite model.
-    converter = lite.TFLiteConverter.from_keras_model(model)
+    converter = tf.lite.TFLiteConverter.from_keras_model(model)
     tflite_model = converter.convert()
 
     # Converting ConcreteFunctions to a TensorFlow Lite model.
-    converter = lite.TFLiteConverter.from_concrete_functions([func])
+    converter = tf.lite.TFLiteConverter.from_concrete_functions([func])
     tflite_model = converter.convert()
     ```
   """
@@ -1032,7 +1094,7 @@ class TFLiteConverterBaseV1(TFLiteConverterBase):
       generate input and output samples for the model. The converter can use the
       dataset to evaluate different optimizations.
     experimental_new_converter: Experimental flag, subject to change. Enables
-      MLIR-based conversion instead of TOCO conversion.
+      MLIR-based conversion instead of TOCO conversion. (default True)
   """
 
   def __init__(self, experimental_debug_info_func):
@@ -1193,10 +1255,9 @@ class TFLiteConverterBaseV1(TFLiteConverterBase):
 
     if not self.experimental_new_converter:
       logging.warning(
-          "Please consider switching to use new converter by setting "
-          "experimental_new_converter to true. "
-          "Old converter (TOCO) is deprecated and flow will be switched on "
-          "by default to use new converter soon.")
+          "Please consider switching to the new converter by setting "
+          "experimental_new_converter=True. "
+          "The old converter (TOCO) is deprecated.")
     else:
       logging.info("Using experimental converter: If you encountered a problem "
                    "please file a bug. You can opt-out "
@@ -1574,29 +1635,30 @@ class TFLiteConverter(TFLiteFrozenGraphConverter):
       generate input and output samples for the model. The converter can use
       the dataset to evaluate different optimizations.
     experimental_new_converter: Experimental flag, subject to change.
-      Enables MLIR-based conversion instead of TOCO conversion.
+      Enables MLIR-based conversion instead of TOCO conversion. (default True)
 
   Example usage:
 
     ```python
     # Converting a GraphDef from session.
-    converter = lite.TFLiteConverter.from_session(sess, in_tensors, out_tensors)
+    converter = tf.compat.v1.TFLiteConverter.from_session(
+      sess, in_tensors, out_tensors)
     tflite_model = converter.convert()
     open("converted_model.tflite", "wb").write(tflite_model)
 
     # Converting a GraphDef from file.
-    converter = lite.TFLiteConverter.from_frozen_graph(
+    converter = tf.compat.v1.TFLiteConverter.from_frozen_graph(
       graph_def_file, input_arrays, output_arrays)
     tflite_model = converter.convert()
     open("converted_model.tflite", "wb").write(tflite_model)
 
     # Converting a SavedModel.
-    converter = lite.TFLiteConverter.from_saved_model(saved_model_dir)
+    converter = tf.compat.v1.TFLiteConverter.from_saved_model(saved_model_dir)
     tflite_model = converter.convert()
     open("converted_model.tflite", "wb").write(tflite_model)
 
     # Converting a tf.keras model.
-    converter = lite.TFLiteConverter.from_keras_model_file(keras_model)
+    converter = tf.compat.v1.TFLiteConverter.from_keras_model_file(keras_model)
     tflite_model = converter.convert()
     open("converted_model.tflite", "wb").write(tflite_model)
     ```
diff --git a/tensorflow/lite/python/lite_test.py b/tensorflow/lite/python/lite_test.py
index 9611bda2594..a20d7e83562 100644
--- a/tensorflow/lite/python/lite_test.py
+++ b/tensorflow/lite/python/lite_test.py
@@ -32,6 +32,7 @@ from six.moves import range
 from tensorflow.lite.python import lite
 from tensorflow.lite.python import lite_constants
 from tensorflow.lite.python.convert import ConverterError
+from tensorflow.lite.python.convert import mlir_quantize
 from tensorflow.lite.python.interpreter import Interpreter
 from tensorflow.python import keras
 from tensorflow.python.client import session
@@ -1175,9 +1176,21 @@ class FromSessionTest(TestModels, parameterized.TestCase):
     converter._experimental_new_quantizer = True
     quantized_tflite = converter.convert()
     self.assertTrue(quantized_tflite)
-
     self.assertLess(len(quantized_tflite), len(float_tflite))
 
+    # calibration only api
+    converter._experimental_calibrate_only = True
+    calibrated_tflite = converter.convert()
+    quantized_tflite = mlir_quantize(calibrated_tflite, fully_quantize=True)
+    interpreter = Interpreter(model_content=quantized_tflite)
+    interpreter.allocate_tensors()
+    input_details = interpreter.get_input_details()
+    self.assertEqual(np.int8, input_details[0]['dtype'])
+    self.assertEqual((1., 0.), input_details[0]['quantization'])
+
+    output_details = interpreter.get_output_details()
+    self.assertEqual(np.int8, output_details[0]['dtype'])
+
   def testFloatTocoConverter(self):
     """Tests deprecated test TocoConverter."""
     with ops.Graph().as_default():
@@ -1754,7 +1767,7 @@ class FromSavedModelTest(TestModels):
     log = io.BytesIO() if six.PY2 else io.StringIO()
     handler = logging.StreamHandler(log)
     logging.root.addHandler(handler)
-    warning_message = 'Please consider switching to use new converter'
+    warning_message = 'Please consider switching to the new converter'
     # Convert model and ensure model is not None.
     converter = lite.TFLiteConverter.from_saved_model(saved_model_dir)
     converter.experimental_new_converter = False
diff --git a/tensorflow/lite/python/lite_v2_test.py b/tensorflow/lite/python/lite_v2_test.py
index 9af37df2975..f56f85d0ba4 100644
--- a/tensorflow/lite/python/lite_v2_test.py
+++ b/tensorflow/lite/python/lite_v2_test.py
@@ -71,6 +71,27 @@ class FromConcreteFunctionTest(lite_v2_test_util.ModelTest):
     actual_value = self._evaluateTFLiteModel(tflite_model, [input_data])
     self.assertEqual(expected_value.numpy(), actual_value)
 
+  @parameterized.named_parameters(
+      ('_INT8InputOutput', lite.constants.INT8),
+      ('_UINT8InputOutput', lite.constants.QUANTIZED_UINT8))
+  @test_util.run_v2_only
+  def testInvalidFloat(self, inference_input_output_type):
+    root = self._getSimpleVariableModel()
+    input_data = tf.constant(1., shape=[1])
+    concrete_func = root.f.get_concrete_function(input_data)
+
+    # Convert model.
+    converter = lite.TFLiteConverterV2.from_concrete_functions([concrete_func])
+    # We don't support integer types as we don't have statistical information
+    # to quantize (only supported for post training integer quantization).
+    with self.assertRaises(ValueError) as error:
+      converter.inference_input_type = inference_input_output_type
+      converter.inference_output_type = inference_input_output_type
+      converter.convert()
+    self.assertEqual(
+        'The inference_input_type and inference_output_type '
+        'must be tf.float32.', str(error.exception))
+
   @test_util.run_v2_only
   def testScalarInput(self):
     root = self._getSimpleVariableModel()
@@ -172,39 +193,112 @@ class FromConcreteFunctionTest(lite_v2_test_util.ModelTest):
     self.assertLess(len(quantized_tflite), len(float_tflite))
 
   @parameterized.named_parameters(
-      ('EnableMlirQuantizer', True),  # enable mlir quantizer
-      ('DisableMlirQuantizer', False))  # disable mlir quantizer
-  def testCalibrateAndQuantizeBuiltinInt8(self, mlir_quantizer):
+      ('_INT8InputOutput', lite.constants.INT8),
+      ('_UINT8InputOutput', lite.constants.QUANTIZED_UINT8))
+  @test_util.run_v2_only
+  def testInvalidPostTrainingDynamicRangeQuantization(
+      self, inference_input_output_type):
+    func, _ = self._getCalibrationQuantizeModel()
+
+    # Convert float model.
+    converter = lite.TFLiteConverterV2.from_concrete_functions([func])
+    tflite_model = converter.convert()
+    self.assertTrue(tflite_model)
+
+    # Convert quantized model.
+    quantized_converter = lite.TFLiteConverterV2.from_concrete_functions([func])
+    quantized_converter.optimizations = [lite.Optimize.DEFAULT]
+    # We don't support integer types as we don't have statistical information
+    # to quantize (only supported for post training integer quantization).
+    with self.assertRaises(ValueError) as error:
+      quantized_converter.inference_input_type = inference_input_output_type
+      quantized_converter.inference_output_type = inference_input_output_type
+      quantized_converter.convert()
+    self.assertEqual(
+        'The inference_input_type and inference_output_type '
+        'must be tf.float32.', str(error.exception))
+
+  @parameterized.named_parameters(
+      ('_DefaultFLOAT32InputOutput', lite.constants.FLOAT),
+      ('_INT8InputOutput', lite.constants.INT8),
+      ('_UINT8InputOutput', lite.constants.QUANTIZED_UINT8))
+  def testPostTrainingIntegerAllowFloatQuantization(
+      self, inference_input_output_type):
     func, calibration_gen = self._getCalibrationQuantizeModel()
 
     # Convert float model.
-    float_converter = lite.TFLiteConverterV2.from_concrete_functions([func])
-    float_tflite = float_converter.convert()
-    self.assertTrue(float_tflite)
+    converter = lite.TFLiteConverterV2.from_concrete_functions([func])
+    tflite_model = converter.convert()
+    self.assertTrue(tflite_model)
+
+    # Convert quantized model.
+    quantized_converter = lite.TFLiteConverterV2.from_concrete_functions([func])
+    quantized_converter.optimizations = [lite.Optimize.DEFAULT]
+    quantized_converter.representative_dataset = calibration_gen
+    quantized_converter.inference_input_type = inference_input_output_type
+    quantized_converter.inference_output_type = inference_input_output_type
+    quantized_tflite_model = quantized_converter.convert()
+    self.assertTrue(quantized_tflite_model)
+
+    interpreter = Interpreter(model_content=quantized_tflite_model)
+    interpreter.allocate_tensors()
+    input_details = interpreter.get_input_details()
+    self.assertLen(input_details, 1)
+    self.assertEqual(inference_input_output_type.as_numpy_dtype,
+                     input_details[0]['dtype'])
+    output_details = interpreter.get_output_details()
+    self.assertLen(output_details, 1)
+    self.assertEqual(inference_input_output_type.as_numpy_dtype,
+                     output_details[0]['dtype'])
+
+    # Ensure that the quantized tflite model is smaller.
+    self.assertLess(len(quantized_tflite_model), len(tflite_model))
+
+  @parameterized.named_parameters(
+      ('_DefaultFLOAT32InputOutput_UseTargetTypesFlag', lite.constants.FLOAT,
+       False), ('_DefaultFLOAT32InputOutput', lite.constants.FLOAT, True),
+      ('_INT8InputOutput', lite.constants.INT8, True),
+      ('_UINT8InputOutput', lite.constants.QUANTIZED_UINT8, True))
+  @test_util.run_v2_only
+  def testPostTrainingIntegerNoFloatQuantization(self,
+                                                 inference_input_output_type,
+                                                 use_target_ops_flag):
+    func, calibration_gen = self._getCalibrationQuantizeModel()
+
+    # Convert float model.
+    converter = lite.TFLiteConverterV2.from_concrete_functions([func])
+    tflite_model = converter.convert()
+    self.assertTrue(tflite_model)
 
     # Convert model by specifying target spec (instead of optimizations), since
     # when targeting an integer only backend, quantization is mandatory.
     quantized_converter = lite.TFLiteConverterV2.from_concrete_functions([func])
-    quantized_converter.target_spec.supported_ops = [
-        lite.OpsSet.TFLITE_BUILTINS_INT8
-    ]
+    quantized_converter.optimizations = [lite.Optimize.DEFAULT]
     quantized_converter.representative_dataset = calibration_gen
-    quantized_converter._experimental_new_quantizer = mlir_quantizer
-    quantized_tflite = quantized_converter.convert()
-    self.assertTrue(quantized_tflite)
+    if use_target_ops_flag:
+      quantized_converter.target_spec.supported_ops = [
+          lite.OpsSet.TFLITE_BUILTINS_INT8
+      ]
+    else:
+      quantized_converter.target_spec.supported_types = [lite.constants.INT8]
+    quantized_converter.inference_input_type = inference_input_output_type
+    quantized_converter.inference_output_type = inference_input_output_type
+    quantized_tflite_model = quantized_converter.convert()
+    self.assertTrue(quantized_tflite_model)
 
-    # The default input and output types should be float.
-    interpreter = Interpreter(model_content=quantized_tflite)
+    interpreter = Interpreter(model_content=quantized_tflite_model)
     interpreter.allocate_tensors()
     input_details = interpreter.get_input_details()
     self.assertLen(input_details, 1)
-    self.assertEqual(np.float32, input_details[0]['dtype'])
+    self.assertEqual(inference_input_output_type.as_numpy_dtype,
+                     input_details[0]['dtype'])
     output_details = interpreter.get_output_details()
     self.assertLen(output_details, 1)
-    self.assertEqual(np.float32, output_details[0]['dtype'])
+    self.assertEqual(inference_input_output_type.as_numpy_dtype,
+                     output_details[0]['dtype'])
 
-    # Ensure that the quantized weights tflite model is smaller.
-    self.assertLess(len(quantized_tflite), len(float_tflite))
+    # Ensure that the quantized tflite model is smaller.
+    self.assertLess(len(quantized_tflite_model), len(tflite_model))
 
   def testCalibrateAndQuantizeBuiltinInt16(self):
     func, calibration_gen = self._getCalibrationQuantizeModel()
@@ -279,7 +373,7 @@ class FromConcreteFunctionTest(lite_v2_test_util.ModelTest):
     return tf.keras.Sequential(QLinear(3, input_shape=(2,)))
 
   @test_util.run_v2_only
-  def testTrainingTimeQuantizeConversion(self):
+  def testTrainingTimeQuantization(self):
     model = self._getTrainingTimeQuantizedModel()
 
     float_converter = lite.TFLiteConverterV2.from_keras_model(model)
@@ -297,6 +391,29 @@ class FromConcreteFunctionTest(lite_v2_test_util.ModelTest):
     interpreter = Interpreter(model_content=quantized_tflite)
     self.assertEqual(np.float32, interpreter.get_input_details()[0]['dtype'])
 
+  @parameterized.named_parameters(
+      ('_INT8InputOutput', lite.constants.INT8),
+      ('_UINT8InputOutput', lite.constants.QUANTIZED_UINT8))
+  def testInvalidTrainingTimeQuantization(self, inference_input_output_type):
+    # We currently don't support integer inference_input_type and
+    # inference_output_type flags for training time quantization.
+
+    model = self._getTrainingTimeQuantizedModel()
+
+    converter = lite.TFLiteConverterV2.from_keras_model(model)
+    tflite_model = converter.convert()
+    self.assertTrue(tflite_model)
+
+    quantized_converter = lite.TFLiteConverterV2.from_keras_model(model)
+    quantized_converter.optimizations = [lite.Optimize.DEFAULT]
+    with self.assertRaises(ValueError) as error:
+      quantized_converter.inference_input_type = inference_input_output_type
+      quantized_converter.inference_output_type = inference_input_output_type
+      quantized_converter.convert()
+    self.assertEqual(
+        'The inference_input_type and inference_output_type '
+        'must be tf.float32.', str(error.exception))
+
   @test_util.run_v2_only
   def testNewQuantizer(self):
     """Test the model quantized by the new converter."""
diff --git a/tensorflow/lite/python/tflite_convert.py b/tensorflow/lite/python/tflite_convert.py
index c7504a3a638..b5eb66eedc4 100644
--- a/tensorflow/lite/python/tflite_convert.py
+++ b/tensorflow/lite/python/tflite_convert.py
@@ -13,7 +13,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 # ==============================================================================
-"""Python command line interface for running TOCO."""
+"""Python command line interface for converting TF models to TFLite models."""
 
 from __future__ import absolute_import
 from __future__ import division
@@ -71,7 +71,7 @@ def _parse_inference_type(value, flag):
                    "QUANTIZED_UINT8 are supported.".format(flag))
 
 
-def _get_toco_converter(flags):
+def _get_tflite_converter(flags):
   """Makes a TFLiteConverter object based on the flags provided.
 
   Args:
@@ -129,7 +129,7 @@ def _convert_tf1_model(flags):
     ValueError: Invalid flags.
   """
   # Create converter.
-  converter = _get_toco_converter(flags)
+  converter = _get_tflite_converter(flags)
   if flags.inference_type:
     converter.inference_type = _parse_inference_type(flags.inference_type,
                                                      "inference_type")
@@ -589,7 +589,7 @@ def _get_parser(use_v2_converter):
       action=_ParseExperimentalNewConverter,
       nargs="?",
       help=("Experimental flag, subject to change. Enables MLIR-based "
-            "conversion instead of TOCO conversion."))
+            "conversion instead of TOCO conversion. (default True)"))
   return parser
 
 
diff --git a/tensorflow/lite/python/tflite_convert_test.py b/tensorflow/lite/python/tflite_convert_test.py
index d6a35ba9248..17d466db326 100644
--- a/tensorflow/lite/python/tflite_convert_test.py
+++ b/tensorflow/lite/python/tflite_convert_test.py
@@ -281,11 +281,11 @@ class TfLiteConvertV1Test(TestModels):
                                                 self._input_shapes,
                                                 custom_opdefs_str))
 
-    # Ensure --experimental_new_converter.
+    # Ensure --allow_custom_ops.
     flags_str_final = ('{} --allow_custom_ops').format(flags_str)
     self._run(flags_str_final, should_succeed=False)
 
-    # Ensure --allow_custom_ops.
+    # Ensure --experimental_new_converter.
     flags_str_final = ('{} --experimental_new_converter').format(flags_str)
     self._run(flags_str_final, should_succeed=False)
 
@@ -344,15 +344,18 @@ class ArgParserTest(test_util.TensorFlowTestCase):
         '--output_file=/tmp/output.tflite',
     ]
 
+    # Note that when the flag parses to None, the converter uses the default
+    # value, which is True.
+
     # V1 parser.
-    parser = tflite_convert._get_parser(False)
+    parser = tflite_convert._get_parser(use_v2_converter=False)
     parsed_args = parser.parse_args(args)
-    self.assertFalse(parsed_args.experimental_new_converter)
+    self.assertIsNone(parsed_args.experimental_new_converter)
 
     # V2 parser.
-    parser = tflite_convert._get_parser(True)
+    parser = tflite_convert._get_parser(use_v2_converter=True)
     parsed_args = parser.parse_args(args)
-    self.assertFalse(parsed_args.experimental_new_converter)
+    self.assertIsNone(parsed_args.experimental_new_converter)
 
   def test_experimental_new_converter(self):
     args = [
@@ -362,12 +365,12 @@ class ArgParserTest(test_util.TensorFlowTestCase):
     ]
 
     # V1 parser.
-    parser = tflite_convert._get_parser(False)
+    parser = tflite_convert._get_parser(use_v2_converter=False)
     parsed_args = parser.parse_args(args)
     self.assertTrue(parsed_args.experimental_new_converter)
 
     # V2 parser.
-    parser = tflite_convert._get_parser(True)
+    parser = tflite_convert._get_parser(use_v2_converter=True)
     parsed_args = parser.parse_args(args)
     self.assertTrue(parsed_args.experimental_new_converter)
 
@@ -396,12 +399,12 @@ class ArgParserTest(test_util.TensorFlowTestCase):
     ]
 
     # V1 parser.
-    parser = tflite_convert._get_parser(False)
+    parser = tflite_convert._get_parser(use_v2_converter=False)
     parsed_args = parser.parse_args(args)
     self.assertFalse(parsed_args.experimental_new_converter)
 
     # V2 parser.
-    parser = tflite_convert._get_parser(True)
+    parser = tflite_convert._get_parser(use_v2_converter=True)
     parsed_args = parser.parse_args(args)
     self.assertFalse(parsed_args.experimental_new_converter)
 
diff --git a/tensorflow/lite/python/util.py b/tensorflow/lite/python/util.py
index 32a2d596629..9d1b55e5092 100644
--- a/tensorflow/lite/python/util.py
+++ b/tensorflow/lite/python/util.py
@@ -117,6 +117,12 @@ def get_tensors_from_tensor_names(graph, tensor_names):
   tensors = []
   invalid_tensors = []
   for name in tensor_names:
+    if not isinstance(name, six.string_types):
+      raise ValueError("Invalid type for a tensor name in the provided graph. "
+                       "Expected type for a tensor name is 'str', instead got "
+                       "type '{}' for tensor name '{}'".format(
+                           type(name), name))
+
     tensor = tensor_name_to_tensor.get(name)
     if tensor is None:
       invalid_tensors.append(name)
diff --git a/tensorflow/lite/python/wrap_toco.py b/tensorflow/lite/python/wrap_toco.py
index 8f72cc8cbbd..c6176275d81 100644
--- a/tensorflow/lite/python/wrap_toco.py
+++ b/tensorflow/lite/python/wrap_toco.py
@@ -44,10 +44,11 @@ def wrapped_get_potentially_supported_ops():
 
 
 def wrapped_experimental_mlir_quantize(input_data_str, disable_per_channel,
-                                       inference_type):
+                                       fully_quantize, inference_type):
   """Wraps experimental mlir quantize model."""
   return _pywrap_toco_api.ExperimentalMlirQuantizeModel(input_data_str,
                                                         disable_per_channel,
+                                                        fully_quantize,
                                                         inference_type)
 
 
diff --git a/tensorflow/lite/toco/python/toco_python_api.cc b/tensorflow/lite/toco/python/toco_python_api.cc
index 441aabf0ffe..3f3d301a40d 100644
--- a/tensorflow/lite/toco/python/toco_python_api.cc
+++ b/tensorflow/lite/toco/python/toco_python_api.cc
@@ -264,11 +264,13 @@ PyObject* MlirQuantizeModel(PyObject* data, bool disable_per_channel,
     default:
       return nullptr;
   }
+  tflite::TensorType inference_io_type =
+      fully_quantize ? inference_tensor_type : tflite::TensorType_FLOAT32;
   flatbuffers::FlatBufferBuilder builder;
   auto status = mlir::lite::QuantizeModel(
-      *tflite_model, tflite::TensorType::TensorType_FLOAT32,
-      tflite::TensorType::TensorType_FLOAT32, inference_tensor_type, {},
-      disable_per_channel, fully_quantize, &builder, error_reporter.get());
+      *tflite_model, inference_io_type, inference_io_type,
+      inference_tensor_type, {}, disable_per_channel, fully_quantize, &builder,
+      error_reporter.get());
 
   if (status != kTfLiteOk) {
     error_reporter->exception();
diff --git a/tensorflow/lite/tools/benchmark/BUILD b/tensorflow/lite/tools/benchmark/BUILD
index f6cb71749f8..c94e1dba7fe 100644
--- a/tensorflow/lite/tools/benchmark/BUILD
+++ b/tensorflow/lite/tools/benchmark/BUILD
@@ -144,13 +144,13 @@ cc_library(
         "//tensorflow/lite:string_util",
         "//tensorflow/lite/c:common",
         "//tensorflow/lite/kernels:builtin_ops",
+        "//tensorflow/lite/kernels:cpu_backend_context",
         "//tensorflow/lite/profiling:platform_profiler",
         "//tensorflow/lite/profiling:profile_summary_formatter",
         "//tensorflow/lite/profiling:profiler",
         "//tensorflow/lite/tools:logging",
         "//tensorflow/lite/tools/delegates:delegate_provider_hdr",
         "//tensorflow/lite/tools/delegates:tflite_execution_providers",
-        "//tensorflow/lite/tools/evaluation:utils",
         "@com_google_absl//absl/base:core_headers",
         "@com_google_absl//absl/strings",
         "@ruy//ruy/profiler",
diff --git a/tensorflow/lite/tools/benchmark/benchmark_model.cc b/tensorflow/lite/tools/benchmark/benchmark_model.cc
index 4c6fb0eb86e..2a858e7a326 100644
--- a/tensorflow/lite/tools/benchmark/benchmark_model.cc
+++ b/tensorflow/lite/tools/benchmark/benchmark_model.cc
@@ -34,6 +34,7 @@ BenchmarkParams BenchmarkModel::DefaultParams() {
   params.AddParam("max_secs", BenchmarkParam::Create<float>(150.0f));
   params.AddParam("run_delay", BenchmarkParam::Create<float>(-1.0f));
   params.AddParam("num_threads", BenchmarkParam::Create<int32_t>(1));
+  params.AddParam("use_caching", BenchmarkParam::Create<bool>(false));
   params.AddParam("benchmark_name", BenchmarkParam::Create<std::string>(""));
   params.AddParam("output_prefix", BenchmarkParam::Create<std::string>(""));
   params.AddParam("warmup_runs", BenchmarkParam::Create<int32_t>(1));
@@ -82,6 +83,11 @@ std::vector<Flag> BenchmarkModel::GetFlags() {
           "the end of the run but will not start the next run."),
       CreateFlag<float>("run_delay", &params_, "delay between runs in seconds"),
       CreateFlag<int32_t>("num_threads", &params_, "number of threads"),
+      CreateFlag<bool>(
+          "use_caching", &params_,
+          "Enable caching of prepacked weights matrices in matrix "
+          "multiplication routines. Currently implies the use of the Ruy "
+          "library."),
       CreateFlag<std::string>("benchmark_name", &params_, "benchmark name"),
       CreateFlag<std::string>("output_prefix", &params_,
                               "benchmark output prefix"),
@@ -108,6 +114,8 @@ void BenchmarkModel::LogParams() {
                    << params_.Get<float>("run_delay") << "]";
   TFLITE_LOG(INFO) << "Num threads: [" << params_.Get<int32_t>("num_threads")
                    << "]";
+  TFLITE_LOG(INFO) << "Use caching: [" << params_.Get<bool>("use_caching")
+                   << "]";
   TFLITE_LOG(INFO) << "Benchmark name: ["
                    << params_.Get<std::string>("benchmark_name") << "]";
   TFLITE_LOG(INFO) << "Output prefix: ["
diff --git a/tensorflow/lite/tools/benchmark/benchmark_test.cc b/tensorflow/lite/tools/benchmark/benchmark_test.cc
index 33ccacc0451..37eddf4faf7 100644
--- a/tensorflow/lite/tools/benchmark/benchmark_test.cc
+++ b/tensorflow/lite/tools/benchmark/benchmark_test.cc
@@ -53,6 +53,7 @@ BenchmarkParams CreateParams(int32_t num_runs, float min_secs, float max_secs,
   params.AddParam("max_secs", BenchmarkParam::Create<float>(max_secs));
   params.AddParam("run_delay", BenchmarkParam::Create<float>(-1.0f));
   params.AddParam("num_threads", BenchmarkParam::Create<int32_t>(1));
+  params.AddParam("use_caching", BenchmarkParam::Create<bool>(false));
   params.AddParam("benchmark_name", BenchmarkParam::Create<std::string>(""));
   params.AddParam("output_prefix", BenchmarkParam::Create<std::string>(""));
   params.AddParam("warmup_runs", BenchmarkParam::Create<int32_t>(1));
@@ -397,6 +398,14 @@ TEST(BenchmarkTest, RunWithWrongFlags) {
   EXPECT_EQ(kTfLiteError, status);
 }
 
+TEST(BenchmarkTest, RunWithUseCaching) {
+  ASSERT_THAT(g_fp32_model_path, testing::NotNull());
+  TestBenchmark benchmark(CreateFp32Params());
+  ScopedCommandlineArgs scoped_argv({"--use_caching=false"});
+  auto status = benchmark.Run(scoped_argv.argc(), scoped_argv.argv());
+  EXPECT_EQ(kTfLiteOk, status);
+}
+
 class MaxDurationWorksTestListener : public BenchmarkListener {
   void OnBenchmarkEnd(const BenchmarkResults& results) override {
     const int64_t num_actual_runs = results.inference_time_us().count();
diff --git a/tensorflow/lite/tools/benchmark/benchmark_tflite_model.cc b/tensorflow/lite/tools/benchmark/benchmark_tflite_model.cc
index 969713cce73..9114910ad73 100644
--- a/tensorflow/lite/tools/benchmark/benchmark_tflite_model.cc
+++ b/tensorflow/lite/tools/benchmark/benchmark_tflite_model.cc
@@ -30,6 +30,7 @@ limitations under the License.
 #include "absl/strings/numbers.h"
 #include "ruy/profiler/profiler.h"  // from @ruy
 #include "tensorflow/lite/c/common.h"
+#include "tensorflow/lite/kernels/cpu_backend_context.h"
 #include "tensorflow/lite/kernels/register.h"
 #include "tensorflow/lite/model.h"
 #include "tensorflow/lite/op_resolver.h"
@@ -600,11 +601,24 @@ TfLiteStatus BenchmarkTfLiteModel::ResetInputsAndOutputs() {
 TfLiteStatus BenchmarkTfLiteModel::InitInterpreter() {
   auto resolver = GetOpResolver();
   const int32_t num_threads = params_.Get<int32_t>("num_threads");
+  const bool use_caching = params_.Get<bool>("use_caching");
   tflite::InterpreterBuilder(*model_, *resolver)(&interpreter_, num_threads);
   if (!interpreter_) {
     TFLITE_LOG(ERROR) << "Failed to initialize the interpreter";
     return kTfLiteError;
   }
+  // Manually enable caching behavior in TF Lite interpreter.
+  if (use_caching) {
+    external_context_.reset(new tflite::ExternalCpuBackendContext());
+    std::unique_ptr<tflite::CpuBackendContext> cpu_backend_context(
+        new tflite::CpuBackendContext());
+    cpu_backend_context->SetUseCaching(true);
+    external_context_->set_internal_backend_context(
+        std::move(cpu_backend_context));
+    interpreter_->SetExternalContext(kTfLiteCpuBackendContext,
+                                     external_context_.get());
+  }
+
   return kTfLiteOk;
 }
 
diff --git a/tensorflow/lite/tools/benchmark/benchmark_tflite_model.h b/tensorflow/lite/tools/benchmark/benchmark_tflite_model.h
index cc87743b531..e3307601d73 100644
--- a/tensorflow/lite/tools/benchmark/benchmark_tflite_model.h
+++ b/tensorflow/lite/tools/benchmark/benchmark_tflite_model.h
@@ -85,6 +85,7 @@ class BenchmarkTfLiteModel : public BenchmarkModel {
 
   std::unique_ptr<tflite::FlatBufferModel> model_;
   std::unique_ptr<tflite::Interpreter> interpreter_;
+  std::unique_ptr<tflite::ExternalCpuBackendContext> external_context_;
 
  private:
   // Implement type erasure with unique_ptr with custom deleter.
diff --git a/tensorflow/lite/tools/benchmark/experimental/c/c_api_types.h b/tensorflow/lite/tools/benchmark/experimental/c/c_api_types.h
index 31d8d53d563..ab769fec249 100644
--- a/tensorflow/lite/tools/benchmark/experimental/c/c_api_types.h
+++ b/tensorflow/lite/tools/benchmark/experimental/c/c_api_types.h
@@ -58,7 +58,7 @@ typedef enum TfLiteExternalContextType {
   kTfLiteEigenContext = 0,       // include eigen_support.h to use.
   kTfLiteGemmLowpContext = 1,    // include gemm_support.h to use.
   kTfLiteEdgeTpuContext = 2,     // Placeholder for Edge TPU support.
-  kTfLiteCpuBackendContext = 3,  // include cpu_backend_support.h to use.
+  kTfLiteCpuBackendContext = 3,  // include cpu_backend_context.h to use.
   kTfLiteMaxExternalContexts = 4
 } TfLiteExternalContextType;
 
diff --git a/tensorflow/lite/tools/make/Makefile b/tensorflow/lite/tools/make/Makefile
index 738d50b1771..e3776f8e056 100644
--- a/tensorflow/lite/tools/make/Makefile
+++ b/tensorflow/lite/tools/make/Makefile
@@ -177,9 +177,9 @@ ifeq ($(BUILD_TYPE),windows)
 	BUILD_WITH_MMAP=false
 endif
 ifeq ($(BUILD_WITH_MMAP),true)
-	CORE_CC_EXCLUDE_SRCS += tensorflow/lite/mmap_allocation.cc
-else
 	CORE_CC_EXCLUDE_SRCS += tensorflow/lite/mmap_allocation_disabled.cc
+else
+	CORE_CC_EXCLUDE_SRCS += tensorflow/lite/mmap_allocation.cc
 endif
 
 BUILD_WITH_RUY ?= false
@@ -187,7 +187,7 @@ ifeq ($(TARGET_ARCH),aarch64)
 	BUILD_WITH_RUY=true
 endif
 ifeq ($(BUILD_WITH_RUY),true)
-  CXXFLAGS += -DTFLITE_WITH_RUY
+  CXXFLAGS += -DTFLITE_WITH_RUY_ONLY
 endif
 
 BUILD_WITH_RUY_PROFILER ?= false
diff --git a/tensorflow/lite/tools/optimize/quantize_model.cc b/tensorflow/lite/tools/optimize/quantize_model.cc
index b7b99d9c393..c15808c532c 100644
--- a/tensorflow/lite/tools/optimize/quantize_model.cc
+++ b/tensorflow/lite/tools/optimize/quantize_model.cc
@@ -38,6 +38,15 @@ namespace optimize {
 
 namespace {
 
+bool IsFloatTensor(const SubGraphT* subgraph, int32_t tensor_idx) {
+  TensorT* tensor = subgraph->tensors[tensor_idx].get();
+  if (tensor->type != TensorType_FLOAT32) {
+    // Skip non-real-valued tensor.
+    return false;
+  }
+  return true;
+}
+
 // Gets the operator property from the operator_property list and additionally
 // modifies the quantizable parameter based on the user's specified
 // operator_names.
@@ -46,8 +55,8 @@ operator_property::OperatorProperty GetOperatorProperty(
     int subgraph_index, int op_idx, const string& operator_name) {
   operator_property::OperatorProperty property =
       operator_property::GetOperatorProperty(model, subgraph_index, op_idx);
-  const OperatorT* op =
-      model->subgraphs[subgraph_index]->operators[op_idx].get();
+  const SubGraphT* subgraph = model->subgraphs[subgraph_index].get();
+  const OperatorT* op = subgraph->operators[op_idx].get();
   const BuiltinOperator op_code =
       model->operator_codes[op->opcode_index]->builtin_code;
   // The algorithm adds Dequantize and Quantize, so we don't require them to be
@@ -61,6 +70,67 @@ operator_property::OperatorProperty GetOperatorProperty(
   return property;
 }
 
+bool IsRealValueOp(const std::unordered_set<string>& real_value_op_set,
+                   const string& operator_name) {
+  return real_value_op_set.find(operator_name) != real_value_op_set.end();
+}
+
+// Creates a set that contains all quantizable ops that happen to take a
+// non-float type in the source graph.
+std::unordered_set<string> PopulateRealValueOpSet(
+    ModelT* model, const std::unordered_set<string>& operator_names) {
+  std::unordered_set<string> real_value_op_set;
+  for (size_t subgraph_idx = 0; subgraph_idx < model->subgraphs.size();
+       subgraph_idx++) {
+    SubGraphT* subgraph = model->subgraphs.at(subgraph_idx).get();
+    for (size_t op_idx = 0; op_idx < subgraph->operators.size(); op_idx++) {
+      OperatorT* op = subgraph->operators[op_idx].get();
+      const string operator_name = subgraph->tensors[op->outputs[0]]->name;
+      operator_property::OperatorProperty property = GetOperatorProperty(
+          operator_names, model, subgraph_idx, op_idx, operator_name);
+
+      if (!property.quantizable) {
+        real_value_op_set.insert(operator_name);
+        continue;
+      }
+
+      for (const std::pair<int, operator_property::TensorProperty>& input :
+           property.inputs) {
+        const int32_t input_idx = input.first;
+        const int32_t tensor_idx = op->inputs[input_idx];
+        if (IsFloatTensor(subgraph, tensor_idx)) {
+          real_value_op_set.insert(operator_name);
+          break;
+        }
+      }
+      for (const std::pair<int, operator_property::TensorProperty>& output :
+           property.outputs) {
+        const int32_t output_idx = output.first;
+        const int32_t tensor_idx = op->outputs[output_idx];
+        if (IsFloatTensor(subgraph, tensor_idx)) {
+          real_value_op_set.insert(operator_name);
+          break;
+        }
+      }
+
+      if (property.arbitrary_inputs) {
+        const int32_t tensor_idx = op->inputs[0];
+        if (IsFloatTensor(subgraph, tensor_idx)) {
+          real_value_op_set.insert(operator_name);
+        }
+      }
+
+      if (property.arbitrary_outputs) {
+        const int32_t tensor_idx = op->outputs[0];
+        if (IsFloatTensor(subgraph, tensor_idx)) {
+          real_value_op_set.insert(operator_name);
+        }
+      }
+    }
+  }
+  return real_value_op_set;
+}
+
 TfLiteStatus QuantizeBias(ModelT* model, const TensorT* input_tensor,
                           const TensorT* weight_tensor, TensorT* bias_tensor,
                           bool is_per_channel, int channel_dim_index,
@@ -291,19 +361,21 @@ TfLiteStatus SetInputAndOutputTypes(ModelT* model, const TensorType& input_type,
 // We have made the restriction that for int8 quantized concat, minimum, and
 // maximum, the inputs and outputs must have the same scale and zero point.
 // The other ones with constraints are handled in QuantizeWeightsAndInput.
-TfLiteStatus ApplyConstraints(ModelT* model,
-                              const std::unordered_set<string>& operator_names,
-                              ErrorReporter* error_reporter) {
+TfLiteStatus ApplyConstraints(
+    ModelT* model, const std::unordered_set<string>& operator_names,
+    const std::unordered_set<string>& real_value_op_set,
+    ErrorReporter* error_reporter) {
   for (int subgraph_idx = 0; subgraph_idx < model->subgraphs.size();
        subgraph_idx++) {
     SubGraphT* subgraph = model->subgraphs.at(subgraph_idx).get();
     // Iterate backward to avoid messing with index.
     for (int op_idx = subgraph->operators.size() - 1; op_idx >= 0; op_idx--) {
       OperatorT* op = subgraph->operators[op_idx].get();
-      operator_property::OperatorProperty property =
-          GetOperatorProperty(operator_names, model, subgraph_idx, op_idx,
-                              subgraph->tensors[op->outputs[0]]->name);
-      if (!property.quantizable) {
+      const string operator_name = subgraph->tensors[op->outputs[0]]->name;
+      operator_property::OperatorProperty property = GetOperatorProperty(
+          operator_names, model, subgraph_idx, op_idx, operator_name);
+      if (!property.quantizable ||
+          !IsRealValueOp(real_value_op_set, operator_name)) {
         continue;
       }
       if (!property.arbitrary_inputs ||
@@ -562,11 +634,11 @@ TfLiteStatus QuantizeOpInput(
         *op_idx += 1;
       }
     } else {
-      TF_LITE_REPORT_ERROR(
-          error_reporter,
-          "Unable to find buffer or min/max value for input activation "
-          "%d in %s in subgraph %d, node: %d",
-          input_idx, EnumNameBuiltinOperator(op_code), subgraph_idx, *op_idx);
+      TF_LITE_REPORT_ERROR(error_reporter,
+                           "Unable to find buffer or min/max value for input "
+                           "%d in %s in subgraph %d, node: %d",
+                           input_idx, EnumNameBuiltinOperator(op_code),
+                           subgraph_idx, *op_idx);
       return kTfLiteError;
     }
   } else if (!property.quantizable && is_input_quantized) {
@@ -811,6 +883,7 @@ TfLiteStatus QuantizeSharedRange(ModelT* model, ErrorReporter* error_reporter) {
 TfLiteStatus QuantizeWeightsInputOutput(
     ModelT* model, bool allow_float,
     const std::unordered_set<string>& operator_names,
+    const std::unordered_set<string>& real_value_op_set,
     ErrorReporter* error_reporter) {
   for (size_t subgraph_idx = 0; subgraph_idx < model->subgraphs.size();
        subgraph_idx++) {
@@ -819,9 +892,12 @@ TfLiteStatus QuantizeWeightsInputOutput(
       OperatorT* op = subgraph->operators[op_idx].get();
       const BuiltinOperator op_code =
           model->operator_codes[op->opcode_index]->builtin_code;
-      operator_property::OperatorProperty property =
-          GetOperatorProperty(operator_names, model, subgraph_idx, op_idx,
-                              subgraph->tensors[op->outputs[0]]->name);
+      const string operator_name = subgraph->tensors[op->outputs[0]]->name;
+      operator_property::OperatorProperty property = GetOperatorProperty(
+          operator_names, model, subgraph_idx, op_idx, operator_name);
+      if (!IsRealValueOp(real_value_op_set, operator_name)) {
+        continue;
+      }
 
       if (!property.quantizable && !allow_float) {
         TF_LITE_REPORT_ERROR(error_reporter,
@@ -851,6 +927,7 @@ TfLiteStatus QuantizeWeightsInputOutput(
 // Quantize bias.
 TfLiteStatus QuantizeBiases(ModelT* model,
                             const std::unordered_set<string>& operator_names,
+                            const std::unordered_set<string>& real_value_op_set,
                             ErrorReporter* error_reporter) {
   for (size_t subgraph_idx = 0; subgraph_idx < model->subgraphs.size();
        subgraph_idx++) {
@@ -859,10 +936,11 @@ TfLiteStatus QuantizeBiases(ModelT* model,
       OperatorT* op = subgraph->operators[op_idx].get();
       const BuiltinOperator op_code =
           model->operator_codes[op->opcode_index]->builtin_code;
-      operator_property::OperatorProperty property =
-          GetOperatorProperty(operator_names, model, subgraph_idx, op_idx,
-                              subgraph->tensors[op->outputs[0]]->name);
-      if (!property.quantizable) {
+      const string operator_name = subgraph->tensors[op->outputs[0]]->name;
+      operator_property::OperatorProperty property = GetOperatorProperty(
+          operator_names, model, subgraph_idx, op_idx, operator_name);
+      if (!property.quantizable ||
+          !IsRealValueOp(real_value_op_set, operator_name)) {
         continue;
       }
       for (const int bias_idx : property.biases) {
@@ -922,15 +1000,19 @@ std::unordered_set<string> GetAllOperatorOutputs(ModelT* model) {
 // will not be filled by this function.
 TfLiteStatus FillQuantizationParams(
     ModelT* model, const std::unordered_set<string>& operator_names,
+    const std::unordered_set<string>& real_value_op_set,
     ErrorReporter* error_reporter) {
   for (size_t subgraph_idx = 0; subgraph_idx < model->subgraphs.size();
        subgraph_idx++) {
     SubGraphT* subgraph = model->subgraphs.at(subgraph_idx).get();
     for (size_t op_idx = 0; op_idx < subgraph->operators.size(); op_idx++) {
       OperatorT* op = subgraph->operators[op_idx].get();
-      operator_property::OperatorProperty property =
-          GetOperatorProperty(operator_names, model, subgraph_idx, op_idx,
-                              subgraph->tensors[op->outputs[0]]->name);
+      const string operator_name = subgraph->tensors[op->outputs[0]]->name;
+      operator_property::OperatorProperty property = GetOperatorProperty(
+          operator_names, model, subgraph_idx, op_idx, operator_name);
+      if (!IsRealValueOp(real_value_op_set, operator_name)) {
+        continue;
+      }
 
       // Populate max, min for each input tensor.
       for (const std::pair<int, operator_property::TensorProperty>& input :
@@ -1027,15 +1109,19 @@ TfLiteStatus FillQuantizationParams(
 // Check compatibility of activation, weight and bias scales. Adjust if needed.
 TfLiteStatus EnsureBiasScaleCompatibility(
     ModelT* model, const std::unordered_set<string>& operator_names,
+    const std::unordered_set<string>& real_value_op_set,
     ErrorReporter* error_reporter) {
   for (size_t subgraph_idx = 0; subgraph_idx < model->subgraphs.size();
        subgraph_idx++) {
     SubGraphT* subgraph = model->subgraphs.at(subgraph_idx).get();
     for (size_t op_idx = 0; op_idx < subgraph->operators.size(); op_idx++) {
       OperatorT* op = subgraph->operators[op_idx].get();
-      operator_property::OperatorProperty property =
-          GetOperatorProperty(operator_names, model, subgraph_idx, op_idx,
-                              subgraph->tensors[op->outputs[0]]->name);
+      const string operator_name = subgraph->tensors[op->outputs[0]]->name;
+      operator_property::OperatorProperty property = GetOperatorProperty(
+          operator_names, model, subgraph_idx, op_idx, operator_name);
+      if (!IsRealValueOp(real_value_op_set, operator_name)) {
+        continue;
+      }
 
       // Loop over all bias tensors.
       for (const int bias_idx : property.biases) {
@@ -1171,17 +1257,19 @@ TfLiteStatus QuantizeModel(flatbuffers::FlatBufferBuilder* builder,
                            const TensorType& output_type, bool allow_float,
                            const std::unordered_set<string>& operator_names,
                            ErrorReporter* error_reporter) {
-  TF_LITE_ENSURE_STATUS(
-      FillQuantizationParams(model, operator_names, error_reporter));
-  TF_LITE_ENSURE_STATUS(
-      EnsureBiasScaleCompatibility(model, operator_names, error_reporter));
+  auto real_value_op_set = PopulateRealValueOpSet(model, operator_names);
+  TF_LITE_ENSURE_STATUS(FillQuantizationParams(
+      model, operator_names, real_value_op_set, error_reporter));
+  TF_LITE_ENSURE_STATUS(EnsureBiasScaleCompatibility(
+      model, operator_names, real_value_op_set, error_reporter));
   TF_LITE_ENSURE_STATUS(QuantizeIntemediateTensors(model, error_reporter));
   TF_LITE_ENSURE_STATUS(QuantizeSharedRange(model, error_reporter));
   TF_LITE_ENSURE_STATUS(QuantizeWeightsInputOutput(
-      model, allow_float, operator_names, error_reporter));
+      model, allow_float, operator_names, real_value_op_set, error_reporter));
+  TF_LITE_ENSURE_STATUS(ApplyConstraints(model, operator_names,
+                                         real_value_op_set, error_reporter));
   TF_LITE_ENSURE_STATUS(
-      ApplyConstraints(model, operator_names, error_reporter));
-  TF_LITE_ENSURE_STATUS(QuantizeBiases(model, operator_names, error_reporter));
+      QuantizeBiases(model, operator_names, real_value_op_set, error_reporter));
   utils::SetOperatorCodeVersion(model);
   TF_LITE_ENSURE_STATUS(
       SetInputAndOutputTypes(model, input_type, output_type, error_reporter));
diff --git a/tensorflow/opensource_only.files b/tensorflow/opensource_only.files
index 41750ea02b4..d49b1c1e381 100644
--- a/tensorflow/opensource_only.files
+++ b/tensorflow/opensource_only.files
@@ -4,6 +4,7 @@ tensorflow/api_template_v1.__init__.py
 tensorflow/compat_template.__init__.py
 tensorflow/compat_template_v1.__init__.py
 tensorflow/compiler/mlir/glob_lit_test.bzl
+tensorflow/go/op/wrappers.go
 tensorflow/lite/micro/build_def.bzl
 tensorflow/python/autograph/core/config.py
 tensorflow/python/eager/benchmarks_test_base.py
@@ -12,6 +13,7 @@ tensorflow/python/tpu/profiler/pip_package/README
 tensorflow/python/tpu/profiler/pip_package/build_pip_package.sh
 tensorflow/python/tpu/profiler/pip_package/setup.py
 tensorflow/python/tpu/tpu.bzl
+tensorflow/security/fuzzing/tf_fuzzing.bzl
 tensorflow/stream_executor/build_defs.bzl
 tensorflow/third_party/BUILD
 tensorflow/third_party/__init__.py
@@ -95,6 +97,7 @@ tensorflow/third_party/gpus/cuda/BUILD.windows.tpl
 tensorflow/third_party/gpus/cuda/LICENSE
 tensorflow/third_party/gpus/cuda/build_defs.bzl.tpl
 tensorflow/third_party/gpus/cuda/cuda_config.h.tpl
+tensorflow/third_party/gpus/cuda/cuda_config.py.tpl
 tensorflow/third_party/gpus/cuda_configure.bzl
 tensorflow/third_party/gpus/find_cuda_config.py
 tensorflow/third_party/gpus/find_cuda_config.py.gz.base64
diff --git a/tensorflow/opensource_only/go/op_wrappers.go b/tensorflow/opensource_only/go/op_wrappers.go
new file mode 100644
index 00000000000..f118e2bd494
--- /dev/null
+++ b/tensorflow/opensource_only/go/op_wrappers.go
@@ -0,0 +1,49728 @@
+// Copyright 2017 The TensorFlow Authors. All Rights Reserved.
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+// DO NOT EDIT
+// This file was machine generated by github.com/tensorflow/tensorflow/tensorflow/go/genop/internal
+//
+// WARNING: This generation of wrapper function for TensorFlow ops is in an
+// experimental state. The generated API can change without notice.
+
+package op
+
+import tf "github.com/tensorflow/tensorflow/tensorflow/go"
+
+// optionalAttr is an intentionally un-exported type to hide
+// details of how optional attributes to operations are implemented.
+type optionalAttr map[string]interface{}
+
+func makeOutputList(op *tf.Operation, start int, output string) ([]tf.Output, int, error) {
+	size, err := op.OutputListSize(output)
+	if err != nil {
+		return nil, start, err
+	}
+	list := make([]tf.Output, size)
+	for i := 0; i < size; i++ {
+		list[i] = op.Output(start + i)
+	}
+	return list, start + size, nil
+}
+
+// FakeQuantWithMinMaxVarsGradientAttr is an optional argument to FakeQuantWithMinMaxVarsGradient.
+type FakeQuantWithMinMaxVarsGradientAttr func(optionalAttr)
+
+// FakeQuantWithMinMaxVarsGradientNumBits sets the optional num_bits attribute to value.
+//
+// value: The bitwidth of the quantization; between 2 and 8, inclusive.
+// If not specified, defaults to 8
+func FakeQuantWithMinMaxVarsGradientNumBits(value int64) FakeQuantWithMinMaxVarsGradientAttr {
+	return func(m optionalAttr) {
+		m["num_bits"] = value
+	}
+}
+
+// FakeQuantWithMinMaxVarsGradientNarrowRange sets the optional narrow_range attribute to value.
+//
+// value: Whether to quantize into 2^num_bits - 1 distinct values.
+// If not specified, defaults to false
+func FakeQuantWithMinMaxVarsGradientNarrowRange(value bool) FakeQuantWithMinMaxVarsGradientAttr {
+	return func(m optionalAttr) {
+		m["narrow_range"] = value
+	}
+}
+
+// Compute gradients for a FakeQuantWithMinMaxVars operation.
+//
+// Arguments:
+//	gradients: Backpropagated gradients above the FakeQuantWithMinMaxVars operation.
+//	inputs: Values passed as inputs to the FakeQuantWithMinMaxVars operation.
+// min, max: Quantization interval, scalar floats.
+//
+//
+//
+// Returns:
+//	backprops_wrt_input: Backpropagated gradients w.r.t. inputs:
+// `gradients * (inputs >= min && inputs <= max)`.
+//	backprop_wrt_min: Backpropagated gradients w.r.t. min parameter:
+// `sum(gradients * (inputs < min))`.
+//	backprop_wrt_max: Backpropagated gradients w.r.t. max parameter:
+// `sum(gradients * (inputs > max))`.
+func FakeQuantWithMinMaxVarsGradient(scope *Scope, gradients tf.Output, inputs tf.Output, min tf.Output, max tf.Output, optional ...FakeQuantWithMinMaxVarsGradientAttr) (backprops_wrt_input tf.Output, backprop_wrt_min tf.Output, backprop_wrt_max tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "FakeQuantWithMinMaxVarsGradient",
+		Input: []tf.Input{
+			gradients, inputs, min, max,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// FakeQuantWithMinMaxArgsGradientAttr is an optional argument to FakeQuantWithMinMaxArgsGradient.
+type FakeQuantWithMinMaxArgsGradientAttr func(optionalAttr)
+
+// FakeQuantWithMinMaxArgsGradientMin sets the optional min attribute to value.
+// If not specified, defaults to -6
+func FakeQuantWithMinMaxArgsGradientMin(value float32) FakeQuantWithMinMaxArgsGradientAttr {
+	return func(m optionalAttr) {
+		m["min"] = value
+	}
+}
+
+// FakeQuantWithMinMaxArgsGradientMax sets the optional max attribute to value.
+// If not specified, defaults to 6
+func FakeQuantWithMinMaxArgsGradientMax(value float32) FakeQuantWithMinMaxArgsGradientAttr {
+	return func(m optionalAttr) {
+		m["max"] = value
+	}
+}
+
+// FakeQuantWithMinMaxArgsGradientNumBits sets the optional num_bits attribute to value.
+// If not specified, defaults to 8
+func FakeQuantWithMinMaxArgsGradientNumBits(value int64) FakeQuantWithMinMaxArgsGradientAttr {
+	return func(m optionalAttr) {
+		m["num_bits"] = value
+	}
+}
+
+// FakeQuantWithMinMaxArgsGradientNarrowRange sets the optional narrow_range attribute to value.
+// If not specified, defaults to false
+func FakeQuantWithMinMaxArgsGradientNarrowRange(value bool) FakeQuantWithMinMaxArgsGradientAttr {
+	return func(m optionalAttr) {
+		m["narrow_range"] = value
+	}
+}
+
+// Compute gradients for a FakeQuantWithMinMaxArgs operation.
+//
+// Arguments:
+//	gradients: Backpropagated gradients above the FakeQuantWithMinMaxArgs operation.
+//	inputs: Values passed as inputs to the FakeQuantWithMinMaxArgs operation.
+//
+// Returns Backpropagated gradients below the FakeQuantWithMinMaxArgs operation:
+// `gradients * (inputs >= min && inputs <= max)`.
+func FakeQuantWithMinMaxArgsGradient(scope *Scope, gradients tf.Output, inputs tf.Output, optional ...FakeQuantWithMinMaxArgsGradientAttr) (backprops tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "FakeQuantWithMinMaxArgsGradient",
+		Input: []tf.Input{
+			gradients, inputs,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Applies sparse addition to `input` using individual values or slices
+//
+// from `updates` according to indices `indices`.  The updates are non-aliasing:
+// `input` is only modified in-place if no other operations will use it.
+// Otherwise, a copy of `input` is made.  This operation has a gradient with
+// respect to both `input` and `updates`.
+//
+// `input` is a `Tensor` with rank `P` and `indices` is a `Tensor` of rank `Q`.
+//
+// `indices` must be integer tensor, containing indices into `input`.
+// It must be shape \\([d_0, ..., d_{Q-2}, K]\\) where `0 < K <= P`.
+//
+// The innermost dimension of `indices` (with length `K`) corresponds to
+// indices into elements (if `K = P`) or `(P-K)`-dimensional slices
+// (if `K < P`) along the `K`th dimension of `input`.
+//
+// `updates` is `Tensor` of rank `Q-1+P-K` with shape:
+//
+// $$[d_0, ..., d_{Q-2}, input.shape[K], ..., input.shape[P-1]].$$
+//
+// For example, say we want to add 4 scattered elements to a rank-1 tensor to 8
+// elements. In Python, that addition would look like this:
+//
+//     input = tf.constant([1, 2, 3, 4, 5, 6, 7, 8])
+//     indices = tf.constant([[4], [3], [1], [7]])
+//     updates = tf.constant([9, 10, 11, 12])
+//     output = tf.scatter_nd_non_aliasing_add(input, indices, updates)
+//     with tf.Session() as sess:
+//       print(sess.run(output))
+//
+// The resulting value `output` would look like this:
+//
+//     [1, 13, 3, 14, 14, 6, 7, 20]
+//
+// See `tf.scatter_nd` for more details about how to make updates to slices.
+//
+// Arguments:
+//	input: A Tensor.
+//	indices: A Tensor. Must be one of the following types: `int32`, `int64`.
+// A tensor of indices into `input`.
+//	updates: A Tensor. Must have the same type as ref. A tensor of updated values
+// to add to `input`.
+//
+// Returns A `Tensor` with the same shape as `input`, containing values of `input`
+// updated with `updates`.
+func ScatterNdNonAliasingAdd(scope *Scope, input tf.Output, indices tf.Output, updates tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ScatterNdNonAliasingAdd",
+		Input: []tf.Input{
+			input, indices, updates,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Subtracts sparse `updates` from an existing tensor according to `indices`.
+//
+// This operation creates a new tensor by subtracting sparse `updates` from the
+// passed in `tensor`.
+// This operation is very similar to `tf.scatter_nd_sub`, except that the updates
+// are subtracted from an existing tensor (as opposed to a variable). If the memory
+// for the existing tensor cannot be re-used, a copy is made and updated.
+//
+// `indices` is an integer tensor containing indices into a new tensor of shape
+// `shape`.  The last dimension of `indices` can be at most the rank of `shape`:
+//
+//     indices.shape[-1] <= shape.rank
+//
+// The last dimension of `indices` corresponds to indices into elements
+// (if `indices.shape[-1] = shape.rank`) or slices
+// (if `indices.shape[-1] < shape.rank`) along dimension `indices.shape[-1]` of
+// `shape`.  `updates` is a tensor with shape
+//
+//     indices.shape[:-1] + shape[indices.shape[-1]:]
+//
+// The simplest form of tensor_scatter_sub is to subtract individual elements
+// from a tensor by index. For example, say we want to insert 4 scattered elements
+// in a rank-1 tensor with 8 elements.
+//
+// In Python, this scatter subtract operation would look like this:
+//
+// ```python
+//     indices = tf.constant([[4], [3], [1], [7]])
+//     updates = tf.constant([9, 10, 11, 12])
+//     tensor = tf.ones([8], dtype=tf.int32)
+//     updated = tf.tensor_scatter_nd_sub(tensor, indices, updates)
+//     print(updated)
+// ```
+//
+// The resulting tensor would look like this:
+//
+//     [1, -10, 1, -9, -8, 1, 1, -11]
+//
+// We can also, insert entire slices of a higher rank tensor all at once. For
+// example, if we wanted to insert two slices in the first dimension of a
+// rank-3 tensor with two matrices of new values.
+//
+// In Python, this scatter add operation would look like this:
+//
+// ```python
+//     indices = tf.constant([[0], [2]])
+//     updates = tf.constant([[[5, 5, 5, 5], [6, 6, 6, 6],
+//                             [7, 7, 7, 7], [8, 8, 8, 8]],
+//                            [[5, 5, 5, 5], [6, 6, 6, 6],
+//                             [7, 7, 7, 7], [8, 8, 8, 8]]])
+//     tensor = tf.ones([4, 4, 4],dtype=tf.int32)
+//     updated = tf.tensor_scatter_nd_sub(tensor, indices, updates)
+//     print(updated)
+// ```
+//
+// The resulting tensor would look like this:
+//
+//     [[[-4, -4, -4, -4], [-5, -5, -5, -5], [-6, -6, -6, -6], [-7, -7, -7, -7]],
+//      [[1, 1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1]],
+//      [[-4, -4, -4, -4], [-5, -5, -5, -5], [-6, -6, -6, -6], [-7, -7, -7, -7]],
+//      [[1, 1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1]]]
+//
+// Note that on CPU, if an out of bound index is found, an error is returned.
+// On GPU, if an out of bound index is found, the index is ignored.
+//
+// Arguments:
+//	tensor: Tensor to copy/update.
+//	indices: Index tensor.
+//	updates: Updates to scatter into output.
+//
+// Returns A new tensor copied from tensor and updates subtracted according to the indices.
+func TensorScatterSub(scope *Scope, tensor tf.Output, indices tf.Output, updates tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorScatterSub",
+		Input: []tf.Input{
+			tensor, indices, updates,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Adds sparse `updates` to an existing tensor according to `indices`.
+//
+// This operation creates a new tensor by adding sparse `updates` to the passed
+// in `tensor`.
+// This operation is very similar to `tf.scatter_nd_add`, except that the updates
+// are added onto an existing tensor (as opposed to a variable). If the memory
+// for the existing tensor cannot be re-used, a copy is made and updated.
+//
+// `indices` is an integer tensor containing indices into a new tensor of shape
+// `tensor.shape`.  The last dimension of `indices` can be at most the rank of
+// `tensor.shape`:
+//
+//     indices.shape[-1] <= tensor.shape.rank
+//
+// The last dimension of `indices` corresponds to indices into elements
+// (if `indices.shape[-1] = tensor.shape.rank`) or slices
+// (if `indices.shape[-1] < tensor.shape.rank`) along dimension
+// `indices.shape[-1]` of `tensor.shape`.  `updates` is a tensor with shape
+//
+//     indices.shape[:-1] + tensor.shape[indices.shape[-1]:]
+//
+// The simplest form of tensor_scatter_add is to add individual elements to a
+// tensor by index. For example, say we want to add 4 elements in a rank-1
+// tensor with 8 elements.
+//
+// In Python, this scatter add operation would look like this:
+//
+// ```python
+//     indices = tf.constant([[4], [3], [1], [7]])
+//     updates = tf.constant([9, 10, 11, 12])
+//     tensor = tf.ones([8], dtype=tf.int32)
+//     updated = tf.tensor_scatter_nd_add(tensor, indices, updates)
+//     print(updated)
+// ```
+//
+// The resulting tensor would look like this:
+//
+//     [1, 12, 1, 11, 10, 1, 1, 13]
+//
+// We can also, insert entire slices of a higher rank tensor all at once. For
+// example, if we wanted to insert two slices in the first dimension of a
+// rank-3 tensor with two matrices of new values.
+//
+// In Python, this scatter add operation would look like this:
+//
+// ```python
+//     indices = tf.constant([[0], [2]])
+//     updates = tf.constant([[[5, 5, 5, 5], [6, 6, 6, 6],
+//                             [7, 7, 7, 7], [8, 8, 8, 8]],
+//                            [[5, 5, 5, 5], [6, 6, 6, 6],
+//                             [7, 7, 7, 7], [8, 8, 8, 8]]])
+//     tensor = tf.ones([4, 4, 4],dtype=tf.int32)
+//     updated = tf.tensor_scatter_nd_add(tensor, indices, updates)
+//     print(updated)
+// ```
+//
+// The resulting tensor would look like this:
+//
+//     [[[6, 6, 6, 6], [7, 7, 7, 7], [8, 8, 8, 8], [9, 9, 9, 9]],
+//      [[1, 1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1]],
+//      [[6, 6, 6, 6], [7, 7, 7, 7], [8, 8, 8, 8], [9, 9, 9, 9]],
+//      [[1, 1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1], [1, 1, 1, 1]]]
+//
+// Note that on CPU, if an out of bound index is found, an error is returned.
+// On GPU, if an out of bound index is found, the index is ignored.
+//
+// Arguments:
+//	tensor: Tensor to copy/update.
+//	indices: Index tensor.
+//	updates: Updates to scatter into output.
+//
+// Returns A new tensor copied from tensor and updates added according to the indices.
+func TensorScatterAdd(scope *Scope, tensor tf.Output, indices tf.Output, updates tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorScatterAdd",
+		Input: []tf.Input{
+			tensor, indices, updates,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Reshapes a quantized tensor as per the Reshape op.
+//
+// ```
+//
+// Arguments:
+//
+//	shape: Defines the shape of the output tensor.
+//	input_min: The minimum value of the input.
+//	input_max: The maximum value of the input.
+//
+// Returns:
+//	output
+//	output_min: This value is copied from input_min.
+//	output_max: This value is copied from input_max.
+func QuantizedReshape(scope *Scope, tensor tf.Output, shape tf.Output, input_min tf.Output, input_max tf.Output) (output tf.Output, output_min tf.Output, output_max tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizedReshape",
+		Input: []tf.Input{
+			tensor, shape, input_min, input_max,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// QuantizeAndDequantizeV2Attr is an optional argument to QuantizeAndDequantizeV2.
+type QuantizeAndDequantizeV2Attr func(optionalAttr)
+
+// QuantizeAndDequantizeV2SignedInput sets the optional signed_input attribute to value.
+//
+// value: Whether the quantization is signed or unsigned. (actually this parameter should
+// have been called <b>`signed_output`</b>)
+// If not specified, defaults to true
+func QuantizeAndDequantizeV2SignedInput(value bool) QuantizeAndDequantizeV2Attr {
+	return func(m optionalAttr) {
+		m["signed_input"] = value
+	}
+}
+
+// QuantizeAndDequantizeV2NumBits sets the optional num_bits attribute to value.
+//
+// value: The bitwidth of the quantization.
+// If not specified, defaults to 8
+func QuantizeAndDequantizeV2NumBits(value int64) QuantizeAndDequantizeV2Attr {
+	return func(m optionalAttr) {
+		m["num_bits"] = value
+	}
+}
+
+// QuantizeAndDequantizeV2RangeGiven sets the optional range_given attribute to value.
+//
+// value: Whether the range is given or should be determined from the `input` tensor.
+// If not specified, defaults to false
+func QuantizeAndDequantizeV2RangeGiven(value bool) QuantizeAndDequantizeV2Attr {
+	return func(m optionalAttr) {
+		m["range_given"] = value
+	}
+}
+
+// QuantizeAndDequantizeV2RoundMode sets the optional round_mode attribute to value.
+//
+// value: The 'round_mode' attribute controls which rounding tie-breaking algorithm is
+// used when rounding float values to their quantized equivalents. The following
+// rounding modes are currently supported:
+//
+// *   HALF_TO_EVEN: this is the default round_mode.
+// *   HALF_UP: round towards positive. In this mode 7.5 rounds up to 8 and -7.5
+//     rounds up to -7.
+//
+// If not specified, defaults to "HALF_TO_EVEN"
+func QuantizeAndDequantizeV2RoundMode(value string) QuantizeAndDequantizeV2Attr {
+	return func(m optionalAttr) {
+		m["round_mode"] = value
+	}
+}
+
+// QuantizeAndDequantizeV2NarrowRange sets the optional narrow_range attribute to value.
+//
+// value: If True, then the absolute value of the quantized minimum value is the same as
+// the quantized maximum value, instead of 1 greater.
+// i.e. for 8 bit quantization, the minimum value is -127 instead of -128.
+// If not specified, defaults to false
+func QuantizeAndDequantizeV2NarrowRange(value bool) QuantizeAndDequantizeV2Attr {
+	return func(m optionalAttr) {
+		m["narrow_range"] = value
+	}
+}
+
+// QuantizeAndDequantizeV2Axis sets the optional axis attribute to value.
+//
+// value: If specified, this axis is treated as a channel or slice axis, and a separate
+// quantization range is used for each channel or slice along this axis.
+// If not specified, defaults to -1
+func QuantizeAndDequantizeV2Axis(value int64) QuantizeAndDequantizeV2Attr {
+	return func(m optionalAttr) {
+		m["axis"] = value
+	}
+}
+
+// Quantizes then dequantizes a tensor.
+//
+// This op simulates the precision loss from the quantized forward pass by:
+//
+// 1. Quantizing the tensor to fixed point numbers, which should match the target
+//    quantization method when it is used in inference.
+// 2. Dequantizing it back to floating point numbers for the following ops, most
+//    likely matmul.
+//
+// There are different ways to quantize. This version uses only scaling, so 0.0
+// maps to 0.
+//
+// From the specified 'num_bits' in the quantized output type, it determines
+// minimum and maximum representable quantized values.
+//
+// e.g.
+//
+// *   [-128, 127] for signed, num_bits = 8, or
+// *   [0, 255] for unsigned, num_bits = 8.
+//
+// If range_given == False, the initial input_min, input_max will be determined
+// automatically as the minimum and maximum values in the input tensor, otherwise
+// the specified values of input_min, input_max are used.
+//
+// Note: If the input_min, input_max are specified, they do not need to equal the
+// actual minimum and maximum values in the tensor. e.g. in some cases it may be
+// beneficial to specify these values such that the low probability extremes of the
+// input distribution are clipped.
+//
+// This op determines the maximum scale_factor that would map the initial
+// [input_min, input_max] range to a range that lies within the representable
+// quantized range.
+//
+// It determines the scale from one of input_min and input_max, then updates the
+// other one to maximize the representable range.
+//
+// e.g.
+//
+// *   if the output is signed, num_bits = 8, [input_min, input_max] = [-10.0,
+//     5.0]: it would use a scale_factor of -128 / -10.0 = 12.8 In this case, it
+//     would update input_max to be 127 / 12.8 = 9.921875
+// *   if the output is signed, num_bits = 8, [input_min, input_max] = [-10.0,
+//     10.0]: it would use a scale_factor of 127 / 10.0 = 12.7 In this case, it
+//     would update input_min to be 128.0 / 12.7 = -10.07874
+// *   if the output is unsigned, input_min is forced to be 0, and only the
+//     specified input_max is used.
+//
+// After determining the scale_factor and updating the input range, it applies the
+// following to each value in the 'input' tensor.
+//
+// output = round(clamp(value, input_min, input_max) * scale_factor) / scale_factor.
+//
+// The above round function rounds the value based on the given round_mode.
+//
+//
+// Arguments:
+//	input: Tensor to quantize and then dequantize.
+//	input_min: If `range_given == True`, this specifies the minimum input value that needs to
+// be represented, otherwise it is determined from the min value of the `input`
+// tensor.
+//	input_max: If `range_given == True`, this specifies the maximum input value that needs to
+// be represented, otherwise it is determined from the max value of the `input`
+// tensor.
+func QuantizeAndDequantizeV2(scope *Scope, input tf.Output, input_min tf.Output, input_max tf.Output, optional ...QuantizeAndDequantizeV2Attr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizeAndDequantizeV2",
+		Input: []tf.Input{
+			input, input_min, input_max,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// QuantizeAndDequantizeAttr is an optional argument to QuantizeAndDequantize.
+type QuantizeAndDequantizeAttr func(optionalAttr)
+
+// QuantizeAndDequantizeSignedInput sets the optional signed_input attribute to value.
+// If not specified, defaults to true
+func QuantizeAndDequantizeSignedInput(value bool) QuantizeAndDequantizeAttr {
+	return func(m optionalAttr) {
+		m["signed_input"] = value
+	}
+}
+
+// QuantizeAndDequantizeNumBits sets the optional num_bits attribute to value.
+// If not specified, defaults to 8
+func QuantizeAndDequantizeNumBits(value int64) QuantizeAndDequantizeAttr {
+	return func(m optionalAttr) {
+		m["num_bits"] = value
+	}
+}
+
+// QuantizeAndDequantizeRangeGiven sets the optional range_given attribute to value.
+// If not specified, defaults to false
+func QuantizeAndDequantizeRangeGiven(value bool) QuantizeAndDequantizeAttr {
+	return func(m optionalAttr) {
+		m["range_given"] = value
+	}
+}
+
+// QuantizeAndDequantizeInputMin sets the optional input_min attribute to value.
+// If not specified, defaults to 0
+func QuantizeAndDequantizeInputMin(value float32) QuantizeAndDequantizeAttr {
+	return func(m optionalAttr) {
+		m["input_min"] = value
+	}
+}
+
+// QuantizeAndDequantizeInputMax sets the optional input_max attribute to value.
+// If not specified, defaults to 0
+func QuantizeAndDequantizeInputMax(value float32) QuantizeAndDequantizeAttr {
+	return func(m optionalAttr) {
+		m["input_max"] = value
+	}
+}
+
+// Use QuantizeAndDequantizeV2 instead.
+//
+// DEPRECATED at GraphDef version 22: Replaced by QuantizeAndDequantizeV2
+func QuantizeAndDequantize(scope *Scope, input tf.Output, optional ...QuantizeAndDequantizeAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizeAndDequantize",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// OneHotAttr is an optional argument to OneHot.
+type OneHotAttr func(optionalAttr)
+
+// OneHotAxis sets the optional axis attribute to value.
+//
+// value: The axis to fill (default: -1, a new inner-most axis).
+// If not specified, defaults to -1
+func OneHotAxis(value int64) OneHotAttr {
+	return func(m optionalAttr) {
+		m["axis"] = value
+	}
+}
+
+// Returns a one-hot tensor.
+//
+// The locations represented by indices in `indices` take value `on_value`,
+// while all other locations take value `off_value`.
+//
+// If the input `indices` is rank `N`, the output will have rank `N+1`,
+// The new axis is created at dimension `axis` (default: the new axis is
+// appended at the end).
+//
+// If `indices` is a scalar the output shape will be a vector of length `depth`.
+//
+// If `indices` is a vector of length `features`, the output shape will be:
+// ```
+//   features x depth if axis == -1
+//   depth x features if axis == 0
+// ```
+//
+// If `indices` is a matrix (batch) with shape `[batch, features]`,
+// the output shape will be:
+// ```
+//   batch x features x depth if axis == -1
+//   batch x depth x features if axis == 1
+//   depth x batch x features if axis == 0
+// ```
+//
+//
+// Examples
+// =========
+//
+// Suppose that
+// ```
+//   indices = [0, 2, -1, 1]
+//   depth = 3
+//   on_value = 5.0
+//   off_value = 0.0
+//   axis = -1
+// ```
+//
+// Then output is `[4 x 3]`:
+// ```
+// output =
+//   [5.0 0.0 0.0]  // one_hot(0)
+//   [0.0 0.0 5.0]  // one_hot(2)
+//   [0.0 0.0 0.0]  // one_hot(-1)
+//   [0.0 5.0 0.0]  // one_hot(1)
+// ```
+//
+// Suppose that
+// ```
+//   indices = [0, 2, -1, 1]
+//   depth = 3
+//   on_value = 0.0
+//   off_value = 3.0
+//   axis = 0
+// ```
+//
+// Then output is `[3 x 4]`:
+// ```
+// output =
+//   [0.0 3.0 3.0 3.0]
+//   [3.0 3.0 3.0 0.0]
+//   [3.0 3.0 3.0 3.0]
+//   [3.0 0.0 3.0 3.0]
+// //  ^                one_hot(0)
+// //      ^            one_hot(2)
+// //          ^        one_hot(-1)
+// //              ^    one_hot(1)
+// ```
+//
+// Suppose that
+// ```
+//   indices = [[0, 2], [1, -1]]
+//   depth = 3
+//   on_value = 1.0
+//   off_value = 0.0
+//   axis = -1
+// ```
+//
+// Then output is `[2 x 2 x 3]`:
+// ```
+// output =
+//   [
+//     [1.0, 0.0, 0.0]  // one_hot(0)
+//     [0.0, 0.0, 1.0]  // one_hot(2)
+//   ][
+//     [0.0, 1.0, 0.0]  // one_hot(1)
+//     [0.0, 0.0, 0.0]  // one_hot(-1)
+//   ]
+// ```
+//
+// Arguments:
+//	indices: A tensor of indices.
+//	depth: A scalar defining the depth of the one hot dimension.
+//	on_value: A scalar defining the value to fill in output when `indices[j] = i`.
+//	off_value: A scalar defining the value to fill in output when `indices[j] != i`.
+//
+// Returns The one-hot tensor.
+func OneHot(scope *Scope, indices tf.Output, depth tf.Output, on_value tf.Output, off_value tf.Output, optional ...OneHotAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "OneHot",
+		Input: []tf.Input{
+			indices, depth, on_value, off_value,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Extract `patches` from `input` and put them in the "depth" output dimension. 3D extension of `extract_image_patches`.
+//
+// Arguments:
+//	input: 5-D Tensor with shape `[batch, in_planes, in_rows, in_cols, depth]`.
+//	ksizes: The size of the sliding window for each dimension of `input`.
+//	strides: 1-D of length 5. How far the centers of two consecutive patches are in
+// `input`. Must be: `[1, stride_planes, stride_rows, stride_cols, 1]`.
+//	padding: The type of padding algorithm to use.
+//
+// We specify the size-related attributes as:
+//
+// ```python
+//       ksizes = [1, ksize_planes, ksize_rows, ksize_cols, 1]
+//       strides = [1, stride_planes, strides_rows, strides_cols, 1]
+// ```
+//
+// Returns 5-D Tensor with shape `[batch, out_planes, out_rows, out_cols,
+// ksize_planes * ksize_rows * ksize_cols * depth]` containing patches
+// with size `ksize_planes x ksize_rows x ksize_cols x depth` vectorized
+// in the "depth" dimension. Note `out_planes`, `out_rows` and `out_cols`
+// are the dimensions of the output patches.
+func ExtractVolumePatches(scope *Scope, input tf.Output, ksizes []int64, strides []int64, padding string) (patches tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"ksizes": ksizes, "strides": strides, "padding": padding}
+	opspec := tf.OpSpec{
+		Type: "ExtractVolumePatches",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// DepthToSpaceAttr is an optional argument to DepthToSpace.
+type DepthToSpaceAttr func(optionalAttr)
+
+// DepthToSpaceDataFormat sets the optional data_format attribute to value.
+// If not specified, defaults to "NHWC"
+func DepthToSpaceDataFormat(value string) DepthToSpaceAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// DepthToSpace for tensors of type T.
+//
+// Rearranges data from depth into blocks of spatial data.
+// This is the reverse transformation of SpaceToDepth. More specifically,
+// this op outputs a copy of the input tensor where values from the `depth`
+// dimension are moved in spatial blocks to the `height` and `width` dimensions.
+// The attr `block_size` indicates the input block size and how the data is moved.
+//
+//   * Chunks of data of size `block_size * block_size` from depth are rearranged
+//     into non-overlapping blocks of size `block_size x block_size`
+//   * The width the output tensor is `input_depth * block_size`, whereas the
+//     height is `input_height * block_size`.
+//   * The Y, X coordinates within each block of the output image are determined
+//     by the high order component of the input channel index.
+//   * The depth of the input tensor must be divisible by
+//     `block_size * block_size`.
+//
+// The `data_format` attr specifies the layout of the input and output tensors
+// with the following options:
+//   "NHWC": `[ batch, height, width, channels ]`
+//   "NCHW": `[ batch, channels, height, width ]`
+//   "NCHW_VECT_C":
+//       `qint8 [ batch, channels / 4, height, width, 4 ]`
+//
+// It is useful to consider the operation as transforming a 6-D Tensor.
+// e.g. for data_format = NHWC,
+//      Each element in the input tensor can be specified via 6 coordinates,
+//      ordered by decreasing memory layout significance as:
+//      n,iY,iX,bY,bX,oC  (where n=batch index, iX, iY means X or Y coordinates
+//                         within the input image, bX, bY means coordinates
+//                         within the output block, oC means output channels).
+//      The output would be the input transposed to the following layout:
+//      n,iY,bY,iX,bX,oC
+//
+// This operation is useful for resizing the activations between convolutions
+// (but keeping all data), e.g. instead of pooling. It is also useful for training
+// purely convolutional models.
+//
+// For example, given an input of shape `[1, 1, 1, 4]`, data_format = "NHWC" and
+// block_size = 2:
+//
+// ```
+// x = [[[[1, 2, 3, 4]]]]
+//
+// ```
+//
+// This operation will output a tensor of shape `[1, 2, 2, 1]`:
+//
+// ```
+//    [[[[1], [2]],
+//      [[3], [4]]]]
+// ```
+//
+// Here, the input has a batch of 1 and each batch element has shape `[1, 1, 4]`,
+// the corresponding output will have 2x2 elements and will have a depth of
+// 1 channel (1 = `4 / (block_size * block_size)`).
+// The output element shape is `[2, 2, 1]`.
+//
+// For an input tensor with larger depth, here of shape `[1, 1, 1, 12]`, e.g.
+//
+// ```
+// x = [[[[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]]]]
+// ```
+//
+// This operation, for block size of 2, will return the following tensor of shape
+// `[1, 2, 2, 3]`
+//
+// ```
+//    [[[[1, 2, 3], [4, 5, 6]],
+//      [[7, 8, 9], [10, 11, 12]]]]
+//
+// ```
+//
+// Similarly, for the following input of shape `[1 2 2 4]`, and a block size of 2:
+//
+// ```
+// x =  [[[[1, 2, 3, 4],
+//        [5, 6, 7, 8]],
+//       [[9, 10, 11, 12],
+//        [13, 14, 15, 16]]]]
+// ```
+//
+// the operator will return the following tensor of shape `[1 4 4 1]`:
+//
+// ```
+// x = [[[ [1],   [2],  [5],  [6]],
+//       [ [3],   [4],  [7],  [8]],
+//       [ [9],  [10], [13],  [14]],
+//       [ [11], [12], [15],  [16]]]]
+//
+// ```
+//
+// Arguments:
+//
+//	block_size: The size of the spatial block, same as in Space2Depth.
+func DepthToSpace(scope *Scope, input tf.Output, block_size int64, optional ...DepthToSpaceAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"block_size": block_size}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DepthToSpace",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SpaceToDepthAttr is an optional argument to SpaceToDepth.
+type SpaceToDepthAttr func(optionalAttr)
+
+// SpaceToDepthDataFormat sets the optional data_format attribute to value.
+// If not specified, defaults to "NHWC"
+func SpaceToDepthDataFormat(value string) SpaceToDepthAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// SpaceToDepth for tensors of type T.
+//
+// Rearranges blocks of spatial data, into depth. More specifically,
+// this op outputs a copy of the input tensor where values from the `height`
+// and `width` dimensions are moved to the `depth` dimension.
+// The attr `block_size` indicates the input block size.
+//
+//   * Non-overlapping blocks of size `block_size x block size` are rearranged
+//     into depth at each location.
+//   * The depth of the output tensor is `block_size * block_size * input_depth`.
+//   * The Y, X coordinates within each block of the input become the high order
+//     component of the output channel index.
+//   * The input tensor's height and width must be divisible by block_size.
+//
+// The `data_format` attr specifies the layout of the input and output tensors
+// with the following options:
+//   "NHWC": `[ batch, height, width, channels ]`
+//   "NCHW": `[ batch, channels, height, width ]`
+//   "NCHW_VECT_C":
+//       `qint8 [ batch, channels / 4, height, width, 4 ]`
+//
+// It is useful to consider the operation as transforming a 6-D Tensor.
+// e.g. for data_format = NHWC,
+//      Each element in the input tensor can be specified via 6 coordinates,
+//      ordered by decreasing memory layout significance as:
+//      n,oY,bY,oX,bX,iC  (where n=batch index, oX, oY means X or Y coordinates
+//                         within the output image, bX, bY means coordinates
+//                         within the input block, iC means input channels).
+//      The output would be a transpose to the following layout:
+//      n,oY,oX,bY,bX,iC
+//
+// This operation is useful for resizing the activations between convolutions
+// (but keeping all data), e.g. instead of pooling. It is also useful for training
+// purely convolutional models.
+//
+// For example, given an input of shape `[1, 2, 2, 1]`, data_format = "NHWC" and
+// block_size = 2:
+//
+// ```
+// x = [[[[1], [2]],
+//       [[3], [4]]]]
+// ```
+//
+// This operation will output a tensor of shape `[1, 1, 1, 4]`:
+//
+// ```
+// [[[[1, 2, 3, 4]]]]
+// ```
+//
+// Here, the input has a batch of 1 and each batch element has shape `[2, 2, 1]`,
+// the corresponding output will have a single element (i.e. width and height are
+// both 1) and will have a depth of 4 channels (1 * block_size * block_size).
+// The output element shape is `[1, 1, 4]`.
+//
+// For an input tensor with larger depth, here of shape `[1, 2, 2, 3]`, e.g.
+//
+// ```
+// x = [[[[1, 2, 3], [4, 5, 6]],
+//       [[7, 8, 9], [10, 11, 12]]]]
+// ```
+//
+// This operation, for block_size of 2, will return the following tensor of shape
+// `[1, 1, 1, 12]`
+//
+// ```
+// [[[[1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]]]]
+// ```
+//
+// Similarly, for the following input of shape `[1 4 4 1]`, and a block size of 2:
+//
+// ```
+// x = [[[[1],   [2],  [5],  [6]],
+//       [[3],   [4],  [7],  [8]],
+//       [[9],  [10], [13],  [14]],
+//       [[11], [12], [15],  [16]]]]
+// ```
+//
+// the operator will return the following tensor of shape `[1 2 2 4]`:
+//
+// ```
+// x = [[[[1, 2, 3, 4],
+//        [5, 6, 7, 8]],
+//       [[9, 10, 11, 12],
+//        [13, 14, 15, 16]]]]
+// ```
+//
+// Arguments:
+//
+//	block_size: The size of the spatial block.
+func SpaceToDepth(scope *Scope, input tf.Output, block_size int64, optional ...SpaceToDepthAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"block_size": block_size}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SpaceToDepth",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// BatchToSpace for 4-D tensors of type T.
+//
+// This is a legacy version of the more general BatchToSpaceND.
+//
+// Rearranges (permutes) data from batch into blocks of spatial data, followed by
+// cropping. This is the reverse transformation of SpaceToBatch. More specifically,
+// this op outputs a copy of the input tensor where values from the `batch`
+// dimension are moved in spatial blocks to the `height` and `width` dimensions,
+// followed by cropping along the `height` and `width` dimensions.
+//
+// Arguments:
+//	input: 4-D tensor with shape
+// `[batch*block_size*block_size, height_pad/block_size, width_pad/block_size,
+//   depth]`. Note that the batch size of the input tensor must be divisible by
+// `block_size * block_size`.
+//	crops: 2-D tensor of non-negative integers with shape `[2, 2]`. It specifies
+// how many elements to crop from the intermediate result across the spatial
+// dimensions as follows:
+//
+//     crops = [[crop_top, crop_bottom], [crop_left, crop_right]]
+//
+//
+// Returns 4-D with shape `[batch, height, width, depth]`, where:
+//
+//       height = height_pad - crop_top - crop_bottom
+//       width = width_pad - crop_left - crop_right
+//
+// The attr `block_size` must be greater than one. It indicates the block size.
+//
+// Some examples:
+//
+// (1) For the following input of shape `[4, 1, 1, 1]` and block_size of 2:
+//
+// ```
+// [[[[1]]], [[[2]]], [[[3]]], [[[4]]]]
+// ```
+//
+// The output tensor has shape `[1, 2, 2, 1]` and value:
+//
+// ```
+// x = [[[[1], [2]], [[3], [4]]]]
+// ```
+//
+// (2) For the following input of shape `[4, 1, 1, 3]` and block_size of 2:
+//
+// ```
+// [[[[1, 2, 3]]], [[[4, 5, 6]]], [[[7, 8, 9]]], [[[10, 11, 12]]]]
+// ```
+//
+// The output tensor has shape `[1, 2, 2, 3]` and value:
+//
+// ```
+// x = [[[[1, 2, 3], [4, 5, 6]],
+//       [[7, 8, 9], [10, 11, 12]]]]
+// ```
+//
+// (3) For the following input of shape `[4, 2, 2, 1]` and block_size of 2:
+//
+// ```
+// x = [[[[1], [3]], [[9], [11]]],
+//      [[[2], [4]], [[10], [12]]],
+//      [[[5], [7]], [[13], [15]]],
+//      [[[6], [8]], [[14], [16]]]]
+// ```
+//
+// The output tensor has shape `[1, 4, 4, 1]` and value:
+//
+// ```
+// x = [[[[1],   [2],  [3],  [4]],
+//      [[5],   [6],  [7],  [8]],
+//      [[9],  [10], [11],  [12]],
+//      [[13], [14], [15],  [16]]]]
+// ```
+//
+// (4) For the following input of shape `[8, 1, 2, 1]` and block_size of 2:
+//
+// ```
+// x = [[[[1], [3]]], [[[9], [11]]], [[[2], [4]]], [[[10], [12]]],
+//      [[[5], [7]]], [[[13], [15]]], [[[6], [8]]], [[[14], [16]]]]
+// ```
+//
+// The output tensor has shape `[2, 2, 4, 1]` and value:
+//
+// ```
+// x = [[[[1], [3]], [[5], [7]]],
+//      [[[2], [4]], [[10], [12]]],
+//      [[[5], [7]], [[13], [15]]],
+//      [[[6], [8]], [[14], [16]]]]
+// ```
+func BatchToSpace(scope *Scope, input tf.Output, crops tf.Output, block_size int64) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"block_size": block_size}
+	opspec := tf.OpSpec{
+		Type: "BatchToSpace",
+		Input: []tf.Input{
+			input, crops,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SpaceToBatch for 4-D tensors of type T.
+//
+// This is a legacy version of the more general SpaceToBatchND.
+//
+// Zero-pads and then rearranges (permutes) blocks of spatial data into batch.
+// More specifically, this op outputs a copy of the input tensor where values from
+// the `height` and `width` dimensions are moved to the `batch` dimension. After
+// the zero-padding, both `height` and `width` of the input must be divisible by the
+// block size.
+//
+// Arguments:
+//	input: 4-D with shape `[batch, height, width, depth]`.
+//	paddings: 2-D tensor of non-negative integers with shape `[2, 2]`. It specifies
+//   the padding of the input with zeros across the spatial dimensions as follows:
+//
+//       paddings = [[pad_top, pad_bottom], [pad_left, pad_right]]
+//
+//   The effective spatial dimensions of the zero-padded input tensor will be:
+//
+//       height_pad = pad_top + height + pad_bottom
+//       width_pad = pad_left + width + pad_right
+//
+// The attr `block_size` must be greater than one. It indicates the block size.
+//
+//   * Non-overlapping blocks of size `block_size x block size` in the height and
+//     width dimensions are rearranged into the batch dimension at each location.
+//   * The batch of the output tensor is `batch * block_size * block_size`.
+//   * Both height_pad and width_pad must be divisible by block_size.
+//
+// The shape of the output will be:
+//
+//     [batch*block_size*block_size, height_pad/block_size, width_pad/block_size,
+//      depth]
+//
+// Some examples:
+//
+// (1) For the following input of shape `[1, 2, 2, 1]` and block_size of 2:
+//
+// ```
+// x = [[[[1], [2]], [[3], [4]]]]
+// ```
+//
+// The output tensor has shape `[4, 1, 1, 1]` and value:
+//
+// ```
+// [[[[1]]], [[[2]]], [[[3]]], [[[4]]]]
+// ```
+//
+// (2) For the following input of shape `[1, 2, 2, 3]` and block_size of 2:
+//
+// ```
+// x = [[[[1, 2, 3], [4, 5, 6]],
+//       [[7, 8, 9], [10, 11, 12]]]]
+// ```
+//
+// The output tensor has shape `[4, 1, 1, 3]` and value:
+//
+// ```
+// [[[[1, 2, 3]]], [[[4, 5, 6]]], [[[7, 8, 9]]], [[[10, 11, 12]]]]
+// ```
+//
+// (3) For the following input of shape `[1, 4, 4, 1]` and block_size of 2:
+//
+// ```
+// x = [[[[1],   [2],  [3],  [4]],
+//       [[5],   [6],  [7],  [8]],
+//       [[9],  [10], [11],  [12]],
+//       [[13], [14], [15],  [16]]]]
+// ```
+//
+// The output tensor has shape `[4, 2, 2, 1]` and value:
+//
+// ```
+// x = [[[[1], [3]], [[9], [11]]],
+//      [[[2], [4]], [[10], [12]]],
+//      [[[5], [7]], [[13], [15]]],
+//      [[[6], [8]], [[14], [16]]]]
+// ```
+//
+// (4) For the following input of shape `[2, 2, 4, 1]` and block_size of 2:
+//
+// ```
+// x = [[[[1],   [2],  [3],  [4]],
+//       [[5],   [6],  [7],  [8]]],
+//      [[[9],  [10], [11],  [12]],
+//       [[13], [14], [15],  [16]]]]
+// ```
+//
+// The output tensor has shape `[8, 1, 2, 1]` and value:
+//
+// ```
+// x = [[[[1], [3]]], [[[9], [11]]], [[[2], [4]]], [[[10], [12]]],
+//      [[[5], [7]]], [[[13], [15]]], [[[6], [8]]], [[[14], [16]]]]
+// ```
+//
+// Among others, this operation is useful for reducing atrous convolution into
+// regular convolution.
+//
+func SpaceToBatch(scope *Scope, input tf.Output, paddings tf.Output, block_size int64) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"block_size": block_size}
+	opspec := tf.OpSpec{
+		Type: "SpaceToBatch",
+		Input: []tf.Input{
+			input, paddings,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SqueezeAttr is an optional argument to Squeeze.
+type SqueezeAttr func(optionalAttr)
+
+// SqueezeAxis sets the optional axis attribute to value.
+//
+// value: If specified, only squeezes the dimensions listed. The dimension
+// index starts at 0. It is an error to squeeze a dimension that is not 1. Must
+// be in the range `[-rank(input), rank(input))`.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func SqueezeAxis(value []int64) SqueezeAttr {
+	return func(m optionalAttr) {
+		m["squeeze_dims"] = value
+	}
+}
+
+// Removes dimensions of size 1 from the shape of a tensor.
+//
+// Given a tensor `input`, this operation returns a tensor of the same type with
+// all dimensions of size 1 removed. If you don't want to remove all size 1
+// dimensions, you can remove specific size 1 dimensions by specifying
+// `axis`.
+//
+// For example:
+//
+// ```
+// # 't' is a tensor of shape [1, 2, 1, 3, 1, 1]
+// shape(squeeze(t)) ==> [2, 3]
+// ```
+//
+// Or, to remove specific size 1 dimensions:
+//
+// ```
+// # 't' is a tensor of shape [1, 2, 1, 3, 1, 1]
+// shape(squeeze(t, [2, 4])) ==> [1, 2, 3, 1]
+// ```
+//
+// Arguments:
+//	input: The `input` to squeeze.
+//
+// Returns Contains the same data as `input`, but has one or more dimensions of
+// size 1 removed.
+func Squeeze(scope *Scope, input tf.Output, optional ...SqueezeAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Squeeze",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// A placeholder op that passes through `input` when its output is not fed.
+//
+// Arguments:
+//	input: The default value to produce when `output` is not fed.
+//	shape: The (possibly partial) shape of the tensor.
+//
+// Returns A placeholder tensor that defaults to `input` if it is not fed.
+func PlaceholderWithDefault(scope *Scope, input tf.Output, shape tf.Shape) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"shape": shape}
+	opspec := tf.OpSpec{
+		Type: "PlaceholderWithDefault",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// PlaceholderAttr is an optional argument to Placeholder.
+type PlaceholderAttr func(optionalAttr)
+
+// PlaceholderShape sets the optional shape attribute to value.
+//
+// value: (Optional) The shape of the tensor. If the shape has 0 dimensions, the
+// shape is unconstrained.
+// If not specified, defaults to <unknown_rank:true >
+func PlaceholderShape(value tf.Shape) PlaceholderAttr {
+	return func(m optionalAttr) {
+		m["shape"] = value
+	}
+}
+
+// A placeholder op for a value that will be fed into the computation.
+//
+// N.B. This operation will fail with an error if it is executed. It is
+// intended as a way to represent a value that will always be fed, and to
+// provide attrs that enable the fed value to be checked at runtime.
+//
+// Arguments:
+//	dtype: The type of elements in the tensor.
+//
+// Returns A placeholder tensor that must be replaced using the feed mechanism.
+func Placeholder(scope *Scope, dtype tf.DataType, optional ...PlaceholderAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Placeholder",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Return the reduction indices for computing gradients of s0 op s1 with broadcast.
+//
+// This is typically used by gradient computations for a broadcasting operation.
+func BroadcastGradientArgs(scope *Scope, s0 tf.Output, s1 tf.Output) (r0 tf.Output, r1 tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "BroadcastGradientArgs",
+		Input: []tf.Input{
+			s0, s1,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Return the shape of s0 op s1 with broadcast.
+//
+// Given `s0` and `s1`, tensors that represent shapes, compute `r0`, the
+// broadcasted shape. `s0`, `s1` and `r0` are all integer vectors.
+func BroadcastArgs(scope *Scope, s0 tf.Output, s1 tf.Output) (r0 tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "BroadcastArgs",
+		Input: []tf.Input{
+			s0, s1,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// TensorStridedSliceUpdateAttr is an optional argument to TensorStridedSliceUpdate.
+type TensorStridedSliceUpdateAttr func(optionalAttr)
+
+// TensorStridedSliceUpdateBeginMask sets the optional begin_mask attribute to value.
+// If not specified, defaults to 0
+func TensorStridedSliceUpdateBeginMask(value int64) TensorStridedSliceUpdateAttr {
+	return func(m optionalAttr) {
+		m["begin_mask"] = value
+	}
+}
+
+// TensorStridedSliceUpdateEndMask sets the optional end_mask attribute to value.
+// If not specified, defaults to 0
+func TensorStridedSliceUpdateEndMask(value int64) TensorStridedSliceUpdateAttr {
+	return func(m optionalAttr) {
+		m["end_mask"] = value
+	}
+}
+
+// TensorStridedSliceUpdateEllipsisMask sets the optional ellipsis_mask attribute to value.
+// If not specified, defaults to 0
+func TensorStridedSliceUpdateEllipsisMask(value int64) TensorStridedSliceUpdateAttr {
+	return func(m optionalAttr) {
+		m["ellipsis_mask"] = value
+	}
+}
+
+// TensorStridedSliceUpdateNewAxisMask sets the optional new_axis_mask attribute to value.
+// If not specified, defaults to 0
+func TensorStridedSliceUpdateNewAxisMask(value int64) TensorStridedSliceUpdateAttr {
+	return func(m optionalAttr) {
+		m["new_axis_mask"] = value
+	}
+}
+
+// TensorStridedSliceUpdateShrinkAxisMask sets the optional shrink_axis_mask attribute to value.
+// If not specified, defaults to 0
+func TensorStridedSliceUpdateShrinkAxisMask(value int64) TensorStridedSliceUpdateAttr {
+	return func(m optionalAttr) {
+		m["shrink_axis_mask"] = value
+	}
+}
+
+// Assign `value` to the sliced l-value reference of `input`.
+//
+// The values of `value` are assigned to the positions in the tensor `input` that
+// are selected by the slice parameters. The slice parameters `begin` `end`
+// `strides` etc. work exactly as in `StridedSlice`.
+//
+// NOTE this op currently does not support broadcasting and so `value`'s shape
+// must be exactly the shape produced by the slice of `input`.
+func TensorStridedSliceUpdate(scope *Scope, input tf.Output, begin tf.Output, end tf.Output, strides tf.Output, value tf.Output, optional ...TensorStridedSliceUpdateAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorStridedSliceUpdate",
+		Input: []tf.Input{
+			input, begin, end, strides, value,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Ensures that the tensor's shape matches the expected shape.
+//
+// Raises an error if the input tensor's shape does not match the specified shape.
+// Returns the input tensor otherwise.
+//
+// Arguments:
+//	input: A tensor, whose shape is to be validated.
+//	shape: The expected (possibly partially specified) shape of the input tensor.
+//
+// Returns A tensor with the same shape and contents as the input tensor or value.
+func EnsureShape(scope *Scope, input tf.Output, shape tf.Shape) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"shape": shape}
+	opspec := tf.OpSpec{
+		Type: "EnsureShape",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ShapeAttr is an optional argument to Shape.
+type ShapeAttr func(optionalAttr)
+
+// ShapeOutType sets the optional out_type attribute to value.
+// If not specified, defaults to DT_INT32
+func ShapeOutType(value tf.DataType) ShapeAttr {
+	return func(m optionalAttr) {
+		m["out_type"] = value
+	}
+}
+
+// Returns the shape of a tensor.
+//
+// This operation returns a 1-D integer tensor representing the shape of `input`.
+//
+// For example:
+//
+// ```
+// # 't' is [[[1, 1, 1], [2, 2, 2]], [[3, 3, 3], [4, 4, 4]]]
+// shape(t) ==> [2, 2, 3]
+// ```
+func Shape(scope *Scope, input tf.Output, optional ...ShapeAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Shape",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// UniqueWithCountsV2Attr is an optional argument to UniqueWithCountsV2.
+type UniqueWithCountsV2Attr func(optionalAttr)
+
+// UniqueWithCountsV2OutIdx sets the optional out_idx attribute to value.
+// If not specified, defaults to DT_INT32
+func UniqueWithCountsV2OutIdx(value tf.DataType) UniqueWithCountsV2Attr {
+	return func(m optionalAttr) {
+		m["out_idx"] = value
+	}
+}
+
+// Finds unique elements along an axis of a tensor.
+//
+// This operation either returns a tensor `y` containing unique elements
+// along the `axis` of a tensor. The returned unique elements is sorted
+// in the same order as they occur along `axis` in `x`.
+// This operation also returns a tensor `idx` and a tensor `count`
+// that are the same size as the number of the elements in `x` along the
+// `axis` dimension. The `idx` contains the index in the unique output `y`
+// and the `count` contains the count in the unique output `y`.
+// In other words, for an `1-D` tensor `x` with `axis = None:
+//
+// `y[idx[i]] = x[i] for i in [0, 1,...,rank(x) - 1]`
+//
+// For example:
+//
+// ```
+// # tensor 'x' is [1, 1, 2, 4, 4, 4, 7, 8, 8]
+// y, idx, count = unique_with_counts(x)
+// y ==> [1, 2, 4, 7, 8]
+// idx ==> [0, 0, 1, 2, 2, 2, 3, 4, 4]
+// count ==> [2, 1, 3, 1, 2]
+// ```
+//
+// For an `2-D` tensor `x` with `axis = 0`:
+//
+// ```
+// # tensor 'x' is [[1, 0, 0],
+// #                [1, 0, 0],
+// #                [2, 0, 0]]
+// y, idx, count = unique_with_counts(x, axis=0)
+// y ==> [[1, 0, 0],
+//        [2, 0, 0]]
+// idx ==> [0, 0, 1]
+// count ==> [2, 1]
+// ```
+//
+// For an `2-D` tensor `x` with `axis = 1`:
+//
+// ```
+// # tensor 'x' is [[1, 0, 0],
+// #                [1, 0, 0],
+// #                [2, 0, 0]]
+// y, idx, count = unique_with_counts(x, axis=1)
+// y ==> [[1, 0],
+//        [1, 0],
+//        [2, 0]]
+// idx ==> [0, 1, 1]
+// count ==> [1, 2]
+// ```
+//
+// Arguments:
+//	x: A `Tensor`.
+//	axis: A `Tensor` of type `int32` (default: None). The axis of the Tensor to
+// find the unique elements.
+//
+// Returns:
+//	y: A `Tensor`. Unique elements along the `axis` of `Tensor` x.
+//	idx: A 1-D Tensor. Has the same type as x that contains the index of each
+// value of x in the output y.
+//	count: A 1-D Tensor. The count of each value of x in the output y.
+func UniqueWithCountsV2(scope *Scope, x tf.Output, axis tf.Output, optional ...UniqueWithCountsV2Attr) (y tf.Output, idx tf.Output, count tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "UniqueWithCountsV2",
+		Input: []tf.Input{
+			x, axis,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Shuffle dimensions of x according to a permutation and conjugate the result.
+//
+// The output `y` has the same rank as `x`. The shapes of `x` and `y` satisfy:
+//   `y.shape[i] == x.shape[perm[i]] for i in [0, 1, ..., rank(x) - 1]`
+//   `y[i,j,k,...,s,t,u] == conj(x[perm[i], perm[j], perm[k],...,perm[s], perm[t], perm[u]])`
+func ConjugateTranspose(scope *Scope, x tf.Output, perm tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ConjugateTranspose",
+		Input: []tf.Input{
+			x, perm,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the inverse permutation of a tensor.
+//
+// This operation computes the inverse of an index permutation. It takes a 1-D
+// integer tensor `x`, which represents the indices of a zero-based array, and
+// swaps each value with its index position. In other words, for an output tensor
+// `y` and an input tensor `x`, this operation computes the following:
+//
+// `y[x[i]] = i for i in [0, 1, ..., len(x) - 1]`
+//
+// The values must include 0. There can be no duplicate values or negative values.
+//
+// For example:
+//
+// ```
+// # tensor `x` is [3, 4, 0, 2, 1]
+// invert_permutation(x) ==> [2, 4, 3, 0, 1]
+// ```
+//
+// Arguments:
+//	x: 1-D.
+//
+// Returns 1-D.
+func InvertPermutation(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "InvertPermutation",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// PreventGradientAttr is an optional argument to PreventGradient.
+type PreventGradientAttr func(optionalAttr)
+
+// PreventGradientMessage sets the optional message attribute to value.
+//
+// value: Will be printed in the error when anyone tries to differentiate
+// this operation.
+// If not specified, defaults to ""
+func PreventGradientMessage(value string) PreventGradientAttr {
+	return func(m optionalAttr) {
+		m["message"] = value
+	}
+}
+
+// An identity op that triggers an error if a gradient is requested.
+//
+// When executed in a graph, this op outputs its input tensor as-is.
+//
+// When building ops to compute gradients, the TensorFlow gradient system
+// will return an error when trying to lookup the gradient of this op,
+// because no gradient must ever be registered for this function.  This
+// op exists to prevent subtle bugs from silently returning unimplemented
+// gradients in some corner cases.
+//
+// Arguments:
+//	input: any tensor.
+//
+// Returns the same input tensor.
+func PreventGradient(scope *Scope, input tf.Output, optional ...PreventGradientAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "PreventGradient",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Stops gradient computation.
+//
+// When executed in a graph, this op outputs its input tensor as-is.
+//
+// When building ops to compute gradients, this op prevents the contribution of
+// its inputs to be taken into account.  Normally, the gradient generator adds ops
+// to a graph to compute the derivatives of a specified 'loss' by recursively
+// finding out inputs that contributed to its computation.  If you insert this op
+// in the graph it inputs are masked from the gradient generator.  They are not
+// taken into account for computing gradients.
+//
+// This is useful any time you want to compute a value with TensorFlow but need
+// to pretend that the value was a constant. Some examples include:
+//
+// *  The *EM* algorithm where the *M-step* should not involve backpropagation
+//    through the output of the *E-step*.
+// *  Contrastive divergence training of Boltzmann machines where, when
+//    differentiating the energy function, the training must not backpropagate
+//    through the graph that generated the samples from the model.
+// *  Adversarial training, where no backprop should happen through the adversarial
+//    example generation process.
+func StopGradient(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "StopGradient",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Identity op for gradient debugging.
+//
+// This op is hidden from public in Python. It is used by TensorFlow Debugger to
+// register gradient tensors for gradient debugging.
+// This op operates on non-reference-type tensors.
+func DebugGradientIdentity(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "DebugGradientIdentity",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Gather slices from `params` into a Tensor with shape specified by `indices`.
+//
+// `indices` is a K-dimensional integer tensor, best thought of as a
+// (K-1)-dimensional tensor of indices into `params`, where each element defines a
+// slice of `params`:
+//
+//     output[\\(i_0, ..., i_{K-2}\\)] = params[indices[\\(i_0, ..., i_{K-2}\\)]]
+//
+// Whereas in `tf.gather` `indices` defines slices into the `axis`
+// dimension of `params`, in `tf.gather_nd`, `indices` defines slices into the
+// first `N` dimensions of `params`, where `N = indices.shape[-1]`.
+//
+// The last dimension of `indices` can be at most the rank of
+// `params`:
+//
+//     indices.shape[-1] <= params.rank
+//
+// The last dimension of `indices` corresponds to elements
+// (if `indices.shape[-1] == params.rank`) or slices
+// (if `indices.shape[-1] < params.rank`) along dimension `indices.shape[-1]`
+// of `params`.  The output tensor has shape
+//
+//     indices.shape[:-1] + params.shape[indices.shape[-1]:]
+//
+// Note that on CPU, if an out of bound index is found, an error is returned.
+// On GPU, if an out of bound index is found, a 0 is stored in the
+// corresponding output value.
+//
+// Some examples below.
+//
+// Simple indexing into a matrix:
+//
+// ```python
+//     indices = [[0, 0], [1, 1]]
+//     params = [['a', 'b'], ['c', 'd']]
+//     output = ['a', 'd']
+// ```
+//
+// Slice indexing into a matrix:
+//
+// ```python
+//     indices = [[1], [0]]
+//     params = [['a', 'b'], ['c', 'd']]
+//     output = [['c', 'd'], ['a', 'b']]
+// ```
+//
+// Indexing into a 3-tensor:
+//
+// ```python
+//     indices = [[1]]
+//     params = [[['a0', 'b0'], ['c0', 'd0']],
+//               [['a1', 'b1'], ['c1', 'd1']]]
+//     output = [[['a1', 'b1'], ['c1', 'd1']]]
+//
+//
+//     indices = [[0, 1], [1, 0]]
+//     params = [[['a0', 'b0'], ['c0', 'd0']],
+//               [['a1', 'b1'], ['c1', 'd1']]]
+//     output = [['c0', 'd0'], ['a1', 'b1']]
+//
+//
+//     indices = [[0, 0, 1], [1, 0, 1]]
+//     params = [[['a0', 'b0'], ['c0', 'd0']],
+//               [['a1', 'b1'], ['c1', 'd1']]]
+//     output = ['b0', 'b1']
+// ```
+//
+// Batched indexing into a matrix:
+//
+// ```python
+//     indices = [[[0, 0]], [[0, 1]]]
+//     params = [['a', 'b'], ['c', 'd']]
+//     output = [['a'], ['b']]
+// ```
+//
+// Batched slice indexing into a matrix:
+//
+// ```python
+//     indices = [[[1]], [[0]]]
+//     params = [['a', 'b'], ['c', 'd']]
+//     output = [[['c', 'd']], [['a', 'b']]]
+// ```
+//
+// Batched indexing into a 3-tensor:
+//
+// ```python
+//     indices = [[[1]], [[0]]]
+//     params = [[['a0', 'b0'], ['c0', 'd0']],
+//               [['a1', 'b1'], ['c1', 'd1']]]
+//     output = [[[['a1', 'b1'], ['c1', 'd1']]],
+//               [[['a0', 'b0'], ['c0', 'd0']]]]
+//
+//     indices = [[[0, 1], [1, 0]], [[0, 0], [1, 1]]]
+//     params = [[['a0', 'b0'], ['c0', 'd0']],
+//               [['a1', 'b1'], ['c1', 'd1']]]
+//     output = [[['c0', 'd0'], ['a1', 'b1']],
+//               [['a0', 'b0'], ['c1', 'd1']]]
+//
+//
+//     indices = [[[0, 0, 1], [1, 0, 1]], [[0, 1, 1], [1, 1, 0]]]
+//     params = [[['a0', 'b0'], ['c0', 'd0']],
+//               [['a1', 'b1'], ['c1', 'd1']]]
+//     output = [['b0', 'b1'], ['d0', 'c1']]
+// ```
+//
+// See also `tf.gather` and `tf.batch_gather`.
+//
+// Arguments:
+//	params: The tensor from which to gather values.
+//	indices: Index tensor.
+//
+// Returns Values from `params` gathered from indices given by `indices`, with
+// shape `indices.shape[:-1] + params.shape[indices.shape[-1]:]`.
+func GatherNd(scope *Scope, params tf.Output, indices tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "GatherNd",
+		Input: []tf.Input{
+			params, indices,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// GatherV2Attr is an optional argument to GatherV2.
+type GatherV2Attr func(optionalAttr)
+
+// GatherV2BatchDims sets the optional batch_dims attribute to value.
+// If not specified, defaults to 0
+func GatherV2BatchDims(value int64) GatherV2Attr {
+	return func(m optionalAttr) {
+		m["batch_dims"] = value
+	}
+}
+
+// Gather slices from `params` axis `axis` according to `indices`.
+//
+// `indices` must be an integer tensor of any dimension (usually 0-D or 1-D).
+// Produces an output tensor with shape `params.shape[:axis] +
+// indices.shape[batch_dims:] + params.shape[axis + 1:]` where:
+//
+// ```python
+//     # Scalar indices (output is rank(params) - 1).
+//     output[a_0, ..., a_n, b_0, ..., b_n] =
+//       params[a_0, ..., a_n, indices, b_0, ..., b_n]
+//
+//     # Vector indices (output is rank(params)).
+//     output[a_0, ..., a_n, i, b_0, ..., b_n] =
+//       params[a_0, ..., a_n, indices[i], b_0, ..., b_n]
+//
+//     # Higher rank indices (output is rank(params) + rank(indices) - 1).
+//     output[a_0, ..., a_n, i, ..., j, b_0, ... b_n] =
+//       params[a_0, ..., a_n, indices[i, ..., j], b_0, ..., b_n]
+// ```
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src="https://www.tensorflow.org/images/Gather.png" alt>
+// </div>
+//
+// Note that on CPU, if an out of bound index is found, an error is returned.
+// On GPU, if an out of bound index is found, a 0 is stored in the
+// corresponding output value.
+//
+// See also `tf.batch_gather` and `tf.gather_nd`.
+//
+// Arguments:
+//	params: The tensor from which to gather values. Must be at least rank
+// `axis + 1`.
+//	indices: Index tensor. Must be in range `[0, params.shape[axis])`.
+//	axis: The axis in `params` to gather `indices` from. Defaults to the first
+// dimension. Supports negative indexes.
+//
+// Returns Values from `params` gathered from indices given by `indices`, with
+// shape `params.shape[:axis] + indices.shape + params.shape[axis + 1:]`.
+func GatherV2(scope *Scope, params tf.Output, indices tf.Output, axis tf.Output, optional ...GatherV2Attr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "GatherV2",
+		Input: []tf.Input{
+			params, indices, axis,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Reverses specific dimensions of a tensor.
+//
+// NOTE `tf.reverse` has now changed behavior in preparation for 1.0.
+// `tf.reverse_v2` is currently an alias that will be deprecated before TF 1.0.
+//
+// Given a `tensor`, and a `int32` tensor `axis` representing the set of
+// dimensions of `tensor` to reverse. This operation reverses each dimension
+// `i` for which there exists `j` s.t. `axis[j] == i`.
+//
+// `tensor` can have up to 8 dimensions. The number of dimensions specified
+// in `axis` may be 0 or more entries. If an index is specified more than
+// once, a InvalidArgument error is raised.
+//
+// For example:
+//
+// ```
+// # tensor 't' is [[[[ 0,  1,  2,  3],
+// #                  [ 4,  5,  6,  7],
+// #                  [ 8,  9, 10, 11]],
+// #                 [[12, 13, 14, 15],
+// #                  [16, 17, 18, 19],
+// #                  [20, 21, 22, 23]]]]
+// # tensor 't' shape is [1, 2, 3, 4]
+//
+// # 'dims' is [3] or 'dims' is [-1]
+// reverse(t, dims) ==> [[[[ 3,  2,  1,  0],
+//                         [ 7,  6,  5,  4],
+//                         [ 11, 10, 9, 8]],
+//                        [[15, 14, 13, 12],
+//                         [19, 18, 17, 16],
+//                         [23, 22, 21, 20]]]]
+//
+// # 'dims' is '[1]' (or 'dims' is '[-3]')
+// reverse(t, dims) ==> [[[[12, 13, 14, 15],
+//                         [16, 17, 18, 19],
+//                         [20, 21, 22, 23]
+//                        [[ 0,  1,  2,  3],
+//                         [ 4,  5,  6,  7],
+//                         [ 8,  9, 10, 11]]]]
+//
+// # 'dims' is '[2]' (or 'dims' is '[-2]')
+// reverse(t, dims) ==> [[[[8, 9, 10, 11],
+//                         [4, 5, 6, 7],
+//                         [0, 1, 2, 3]]
+//                        [[20, 21, 22, 23],
+//                         [16, 17, 18, 19],
+//                         [12, 13, 14, 15]]]]
+// ```
+//
+// Arguments:
+//	tensor: Up to 8-D.
+//	axis: 1-D. The indices of the dimensions to reverse. Must be in the range
+// `[-rank(tensor), rank(tensor))`.
+//
+// Returns The same shape as `tensor`.
+func ReverseV2(scope *Scope, tensor tf.Output, axis tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ReverseV2",
+		Input: []tf.Input{
+			tensor, axis,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns the batched diagonal part of a batched tensor.
+//
+// This operation returns a tensor with the `diagonal` part
+// of the batched `input`. The `diagonal` part is computed as follows:
+//
+// Assume `input` has `k` dimensions `[I, J, K, ..., M, N]`, then the output is a
+// tensor of rank `k - 1` with dimensions `[I, J, K, ..., min(M, N)]` where:
+//
+// `diagonal[i, j, k, ..., n] = input[i, j, k, ..., n, n]`.
+//
+// The input must be at least a matrix.
+//
+// For example:
+//
+// ```
+// # 'input' is [[[1, 0, 0, 0]
+//                [0, 2, 0, 0]
+//                [0, 0, 3, 0]
+//                [0, 0, 0, 4]],
+//               [[5, 0, 0, 0]
+//                [0, 6, 0, 0]
+//                [0, 0, 7, 0]
+//                [0, 0, 0, 8]]]
+//
+// and input.shape = (2, 4, 4)
+//
+// tf.matrix_diag_part(input) ==> [[1, 2, 3, 4], [5, 6, 7, 8]]
+//
+// which has shape (2, 4)
+// ```
+//
+// Arguments:
+//	input: Rank `k` tensor where `k >= 2`.
+//
+// Returns The extracted diagonal(s) having shape
+// `diagonal.shape = input.shape[:-2] + [min(input.shape[-2:])]`.
+func MatrixDiagPart(scope *Scope, input tf.Output) (diagonal tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "MatrixDiagPart",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MatrixSetDiagV3Attr is an optional argument to MatrixSetDiagV3.
+type MatrixSetDiagV3Attr func(optionalAttr)
+
+// MatrixSetDiagV3Align sets the optional align attribute to value.
+//
+// value: Some diagonals are shorter than `max_diag_len` and need to be padded. `align` is
+// a string specifying how superdiagonals and subdiagonals should be aligned,
+// respectively. There are four possible alignments: "RIGHT_LEFT" (default),
+// "LEFT_RIGHT", "LEFT_LEFT", and "RIGHT_RIGHT". "RIGHT_LEFT" aligns superdiagonals
+// to the right (left-pads the row) and subdiagonals to the left (right-pads the
+// row). It is the packing format LAPACK uses. cuSPARSE uses "LEFT_RIGHT", which is
+// the opposite alignment.
+// If not specified, defaults to "RIGHT_LEFT"
+func MatrixSetDiagV3Align(value string) MatrixSetDiagV3Attr {
+	return func(m optionalAttr) {
+		m["align"] = value
+	}
+}
+
+// Returns a batched matrix tensor with new batched diagonal values.
+//
+// Given `input` and `diagonal`, this operation returns a tensor with the
+// same shape and values as `input`, except for the specified diagonals of the
+// innermost matrices. These will be overwritten by the values in `diagonal`.
+//
+// `input` has `r+1` dimensions `[I, J, ..., L, M, N]`. When `k` is scalar or
+// `k[0] == k[1]`, `diagonal` has `r` dimensions `[I, J, ..., L, max_diag_len]`.
+// Otherwise, it has `r+1` dimensions `[I, J, ..., L, num_diags, max_diag_len]`.
+// `num_diags` is the number of diagonals, `num_diags = k[1] - k[0] + 1`.
+// `max_diag_len` is the longest diagonal in the range `[k[0], k[1]]`,
+// `max_diag_len = min(M + min(k[1], 0), N + min(-k[0], 0))`
+//
+// The output is a tensor of rank `k+1` with dimensions `[I, J, ..., L, M, N]`.
+// If `k` is scalar or `k[0] == k[1]`:
+//
+// ```
+// output[i, j, ..., l, m, n]
+//   = diagonal[i, j, ..., l, n-max(k[1], 0)] ; if n - m == k[1]
+//     input[i, j, ..., l, m, n]              ; otherwise
+// ```
+//
+// Otherwise,
+//
+// ```
+// output[i, j, ..., l, m, n]
+//   = diagonal[i, j, ..., l, diag_index, index_in_diag] ; if k[0] <= d <= k[1]
+//     input[i, j, ..., l, m, n]                         ; otherwise
+// ```
+// where `d = n - m`, `diag_index = k[1] - d`, and
+// `index_in_diag = n - max(d, 0) + offset`.
+//
+// `offset` is zero except when the alignment of the diagonal is to the right.
+// ```
+// offset = max_diag_len - diag_len(d) ; if (`align` in {RIGHT_LEFT, RIGHT_RIGHT}
+//                                            and `d >= 0`) or
+//                                          (`align` in {LEFT_RIGHT, RIGHT_RIGHT}
+//                                            and `d <= 0`)
+//          0                          ; otherwise
+// ```
+// where `diag_len(d) = min(cols - max(d, 0), rows + min(d, 0))`.
+//
+// For example:
+//
+// ```
+// # The main diagonal.
+// input = np.array([[[7, 7, 7, 7],              # Input shape: (2, 3, 4)
+//                    [7, 7, 7, 7],
+//                    [7, 7, 7, 7]],
+//                   [[7, 7, 7, 7],
+//                    [7, 7, 7, 7],
+//                    [7, 7, 7, 7]]])
+// diagonal = np.array([[1, 2, 3],               # Diagonal shape: (2, 3)
+//                      [4, 5, 6]])
+// tf.matrix_set_diag(input, diagonal)
+//   ==> [[[1, 7, 7, 7],  # Output shape: (2, 3, 4)
+//         [7, 2, 7, 7],
+//         [7, 7, 3, 7]],
+//        [[4, 7, 7, 7],
+//         [7, 5, 7, 7],
+//         [7, 7, 6, 7]]]
+//
+// # A superdiagonal (per batch).
+// tf.matrix_set_diag(input, diagonal, k = 1)
+//   ==> [[[7, 1, 7, 7],  # Output shape: (2, 3, 4)
+//         [7, 7, 2, 7],
+//         [7, 7, 7, 3]],
+//        [[7, 4, 7, 7],
+//         [7, 7, 5, 7],
+//         [7, 7, 7, 6]]]
+//
+// # A band of diagonals.
+// diagonals = np.array([[[0, 9, 1],  # Diagonal shape: (2, 4, 3)
+//                        [6, 5, 8],
+//                        [1, 2, 3],
+//                        [4, 5, 0]],
+//                       [[0, 1, 2],
+//                        [5, 6, 4],
+//                        [6, 1, 2],
+//                        [3, 4, 0]]])
+// tf.matrix_set_diag(input, diagonals, k = (-1, 2))
+//   ==> [[[1, 6, 9, 7],  # Output shape: (2, 3, 4)
+//         [4, 2, 5, 1],
+//         [7, 5, 3, 8]],
+//        [[6, 5, 1, 7],
+//         [3, 1, 6, 2],
+//         [7, 4, 2, 4]]]
+//
+// # LEFT_RIGHT alignment.
+// diagonals = np.array([[[9, 1, 0],  # Diagonal shape: (2, 4, 3)
+//                        [6, 5, 8],
+//                        [1, 2, 3],
+//                        [0, 4, 5]],
+//                       [[1, 2, 0],
+//                        [5, 6, 4],
+//                        [6, 1, 2],
+//                        [0, 3, 4]]])
+// tf.matrix_set_diag(input, diagonals, k = (-1, 2), align="LEFT_RIGHT")
+//   ==> [[[1, 6, 9, 7],  # Output shape: (2, 3, 4)
+//         [4, 2, 5, 1],
+//         [7, 5, 3, 8]],
+//        [[6, 5, 1, 7],
+//         [3, 1, 6, 2],
+//         [7, 4, 2, 4]]]
+//
+// ```
+//
+// Arguments:
+//	input: Rank `r+1`, where `r >= 1`.
+//	diagonal: Rank `r` when `k` is an integer or `k[0] == k[1]`. Otherwise, it has rank `r+1`.
+// `k >= 1`.
+//	k: Diagonal offset(s). Positive value means superdiagonal, 0 refers to the main
+// diagonal, and negative value means subdiagonals. `k` can be a single integer
+// (for a single diagonal) or a pair of integers specifying the low and high ends
+// of a matrix band. `k[0]` must not be larger than `k[1]`.
+//
+// Returns Rank `r+1`, with `output.shape = input.shape`.
+func MatrixSetDiagV3(scope *Scope, input tf.Output, diagonal tf.Output, k tf.Output, optional ...MatrixSetDiagV3Attr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MatrixSetDiagV3",
+		Input: []tf.Input{
+			input, diagonal, k,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns a batched matrix tensor with new batched diagonal values.
+//
+// Given `input` and `diagonal`, this operation returns a tensor with the
+// same shape and values as `input`, except for the specified diagonals of the
+// innermost matrices. These will be overwritten by the values in `diagonal`.
+//
+// `input` has `r+1` dimensions `[I, J, ..., L, M, N]`. When `k` is scalar or
+// `k[0] == k[1]`, `diagonal` has `r` dimensions `[I, J, ..., L, max_diag_len]`.
+// Otherwise, it has `r+1` dimensions `[I, J, ..., L, num_diags, max_diag_len]`.
+// `num_diags` is the number of diagonals, `num_diags = k[1] - k[0] + 1`.
+// `max_diag_len` is the longest diagonal in the range `[k[0], k[1]]`,
+// `max_diag_len = min(M + min(k[1], 0), N + min(-k[0], 0))`
+//
+// The output is a tensor of rank `k+1` with dimensions `[I, J, ..., L, M, N]`.
+// If `k` is scalar or `k[0] == k[1]`:
+//
+// ```
+// output[i, j, ..., l, m, n]
+//   = diagonal[i, j, ..., l, n-max(k[1], 0)] ; if n - m == k[1]
+//     input[i, j, ..., l, m, n]              ; otherwise
+// ```
+//
+// Otherwise,
+//
+// ```
+// output[i, j, ..., l, m, n]
+//   = diagonal[i, j, ..., l, diag_index, index_in_diag] ; if k[0] <= d <= k[1]
+//     input[i, j, ..., l, m, n]                         ; otherwise
+// ```
+// where `d = n - m`, `diag_index = k[1] - d`, and `index_in_diag = n - max(d, 0)`.
+//
+// For example:
+//
+// ```
+// # The main diagonal.
+// input = np.array([[[7, 7, 7, 7],              # Input shape: (2, 3, 4)
+//                    [7, 7, 7, 7],
+//                    [7, 7, 7, 7]],
+//                   [[7, 7, 7, 7],
+//                    [7, 7, 7, 7],
+//                    [7, 7, 7, 7]]])
+// diagonal = np.array([[1, 2, 3],               # Diagonal shape: (2, 3)
+//                      [4, 5, 6]])
+// tf.matrix_set_diag(diagonal) ==> [[[1, 7, 7, 7],  # Output shape: (2, 3, 4)
+//                                    [7, 2, 7, 7],
+//                                    [7, 7, 3, 7]],
+//                                   [[4, 7, 7, 7],
+//                                    [7, 5, 7, 7],
+//                                    [7, 7, 6, 7]]]
+//
+// # A superdiagonal (per batch).
+// tf.matrix_set_diag(diagonal, k = 1)
+//   ==> [[[7, 1, 7, 7],  # Output shape: (2, 3, 4)
+//         [7, 7, 2, 7],
+//         [7, 7, 7, 3]],
+//        [[7, 4, 7, 7],
+//         [7, 7, 5, 7],
+//         [7, 7, 7, 6]]]
+//
+// # A band of diagonals.
+// diagonals = np.array([[[1, 2, 3],  # Diagonal shape: (2, 2, 3)
+//                        [4, 5, 0]],
+//                       [[6, 1, 2],
+//                        [3, 4, 0]]])
+// tf.matrix_set_diag(diagonals, k = (-1, 0))
+//   ==> [[[1, 7, 7, 7],  # Output shape: (2, 3, 4)
+//         [4, 2, 7, 7],
+//         [0, 5, 3, 7]],
+//        [[6, 7, 7, 7],
+//         [3, 1, 7, 7],
+//         [7, 4, 2, 7]]]
+//
+// ```
+//
+// Arguments:
+//	input: Rank `r+1`, where `r >= 1`.
+//	diagonal: Rank `r` when `k` is an integer or `k[0] == k[1]`. Otherwise, it has rank `r+1`.
+// `k >= 1`.
+//	k: Diagonal offset(s). Positive value means superdiagonal, 0 refers to the main
+// diagonal, and negative value means subdiagonals. `k` can be a single integer
+// (for a single diagonal) or a pair of integers specifying the low and high ends
+// of a matrix band. `k[0]` must not be larger than `k[1]`.
+//
+// Returns Rank `r+1`, with `output.shape = input.shape`.
+func MatrixSetDiagV2(scope *Scope, input tf.Output, diagonal tf.Output, k tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "MatrixSetDiagV2",
+		Input: []tf.Input{
+			input, diagonal, k,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns a diagonal tensor with a given diagonal values.
+//
+// Given a `diagonal`, this operation returns a tensor with the `diagonal` and
+// everything else padded with zeros. The diagonal is computed as follows:
+//
+// Assume `diagonal` has dimensions [D1,..., Dk], then the output is a tensor of
+// rank 2k with dimensions [D1,..., Dk, D1,..., Dk] where:
+//
+// `output[i1,..., ik, i1,..., ik] = diagonal[i1, ..., ik]` and 0 everywhere else.
+//
+// For example:
+//
+// ```
+// # 'diagonal' is [1, 2, 3, 4]
+// tf.diag(diagonal) ==> [[1, 0, 0, 0]
+//                        [0, 2, 0, 0]
+//                        [0, 0, 3, 0]
+//                        [0, 0, 0, 4]]
+// ```
+//
+// Arguments:
+//	diagonal: Rank k tensor where k is at most 1.
+func Diag(scope *Scope, diagonal tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Diag",
+		Input: []tf.Input{
+			diagonal,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns a tensor of ones with the same shape and type as x.
+//
+// Arguments:
+//	x: a tensor of type T.
+//
+// Returns a tensor of the same shape and type as x but filled with ones.
+func OnesLike(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "OnesLike",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns a constant tensor on the host. Only for writing C++ tests.
+//
+// Arguments:
+//	value: Attr `value` is the tensor to return.
+//
+func HostConst(scope *Scope, value tf.Tensor, dtype tf.DataType) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"value": value, "dtype": dtype}
+	opspec := tf.OpSpec{
+		Type: "HostConst",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Splits a tensor into `num_split` tensors along one dimension.
+//
+// Arguments:
+//	axis: 0-D.  The dimension along which to split.  Must be in the range
+// `[-rank(value), rank(value))`.
+//	value: The tensor to split.
+//	num_split: The number of ways to split.  Must evenly divide
+// `value.shape[split_dim]`.
+//
+// Returns They are identically shaped tensors, whose shape matches that of `value`
+// except along `axis`, where their sizes are
+// `values.shape[split_dim] / num_split`.
+func Split(scope *Scope, axis tf.Output, value tf.Output, num_split int64) (output []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_split": num_split}
+	opspec := tf.OpSpec{
+		Type: "Split",
+		Input: []tf.Input{
+			axis, value,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if output, idx, err = makeOutputList(op, idx, "output"); err != nil {
+		scope.UpdateErr("Split", err)
+		return
+	}
+	return output
+}
+
+// Computes offsets of concat inputs within its output.
+//
+// For example:
+//
+// ```
+// # 'x' is [2, 2, 7]
+// # 'y' is [2, 3, 7]
+// # 'z' is [2, 5, 7]
+// concat_offset(2, [x, y, z]) => [0, 0, 0], [0, 2, 0], [0, 5, 0]
+// ```
+//
+// This is typically used by gradient computations for a concat operation.
+//
+// Arguments:
+//	concat_dim: The dimension along which to concatenate.
+//	shape: The `N` int32 vectors representing shape of tensors being concatenated.
+//
+// Returns The `N` int32 vectors representing the starting offset
+// of input tensors within the concatenated output.
+func ConcatOffset(scope *Scope, concat_dim tf.Output, shape []tf.Output) (offset []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ConcatOffset",
+		Input: []tf.Input{
+			concat_dim, tf.OutputList(shape),
+		},
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if offset, idx, err = makeOutputList(op, idx, "offset"); err != nil {
+		scope.UpdateErr("ConcatOffset", err)
+		return
+	}
+	return offset
+}
+
+// Checks a tensor for NaN and Inf values.
+//
+// When run, reports an `InvalidArgument` error if `tensor` has any values
+// that are not a number (NaN) or infinity (Inf). Otherwise, passes `tensor` as-is.
+//
+// Arguments:
+//
+//	message: Prefix of the error message.
+func CheckNumerics(scope *Scope, tensor tf.Output, message string) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"message": message}
+	opspec := tf.OpSpec{
+		Type: "CheckNumerics",
+		Input: []tf.Input{
+			tensor,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Broadcast an array for a compatible shape.
+//
+// Broadcasting is the process of making arrays to have compatible shapes
+// for arithmetic operations. Two shapes are compatible if for each
+// dimension pair they are either equal or one of them is one. When trying
+// to broadcast a Tensor to a shape, it starts with the trailing dimensions,
+// and works its way forward.
+//
+// For example,
+//
+// >>> x = tf.constant([1, 2, 3])
+// >>> y = tf.broadcast_to(x, [3, 3])
+// >>> print(y)
+// tf.Tensor(
+//     [[1 2 3]
+//      [1 2 3]
+//      [1 2 3]], shape=(3, 3), dtype=int32)
+//
+// In the above example, the input Tensor with the shape of `[1, 3]`
+// is broadcasted to output Tensor with shape of `[3, 3]`.
+//
+// When doing broadcasted operations such as multiplying a tensor
+// by a scalar, broadcasting (usually) confers some time or space
+// benefit, as the broadcasted tensor is never materialized.
+//
+// However, `broadcast_to` does not carry with it any such benefits.
+// The newly-created tensor takes the full memory of the broadcasted
+// shape. (In a graph context, `broadcast_to` might be fused to
+// subsequent operation and then be optimized away, however.)
+//
+// Arguments:
+//	input: A Tensor to broadcast.
+//	shape: An 1-D `int` Tensor. The shape of the desired output.
+//
+// Returns A Tensor.
+func BroadcastTo(scope *Scope, input tf.Output, shape tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "BroadcastTo",
+		Input: []tf.Input{
+			input, shape,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Converts an array of flat indices into a tuple of coordinate arrays.
+//
+//
+// Example:
+//
+// ```
+// y = tf.unravel_index(indices=[2, 5, 7], dims=[3, 3])
+// # 'dims' represent a hypothetical (3, 3) tensor of indices:
+// # [[0, 1, *2*],
+// #  [3, 4, *5*],
+// #  [6, *7*, 8]]
+// # For each entry from 'indices', this operation returns
+// # its coordinates (marked with '*'), such as
+// # 2 ==> (0, 2)
+// # 5 ==> (1, 2)
+// # 7 ==> (2, 1)
+// y ==> [[0, 1, 2], [2, 2, 1]]
+// ```
+//
+// @compatibility(numpy)
+// Equivalent to np.unravel_index
+// @end_compatibility
+//
+// Arguments:
+//	indices: An 0-D or 1-D `int` Tensor whose elements are indices into the
+// flattened version of an array of dimensions dims.
+//	dims: An 1-D `int` Tensor. The shape of the array to use for unraveling
+// indices.
+//
+// Returns An 2-D (or 1-D if indices is 0-D) tensor where each row has the
+// same shape as the indices array.
+func UnravelIndex(scope *Scope, indices tf.Output, dims tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "UnravelIndex",
+		Input: []tf.Input{
+			indices, dims,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// EmptyAttr is an optional argument to Empty.
+type EmptyAttr func(optionalAttr)
+
+// EmptyInit sets the optional init attribute to value.
+//
+// value: If True, initialize the returned tensor with the default value of dtype.  Otherwise, the implementation is free not to initializethe tensor's content.
+// If not specified, defaults to false
+func EmptyInit(value bool) EmptyAttr {
+	return func(m optionalAttr) {
+		m["init"] = value
+	}
+}
+
+// Creates a tensor with the given shape.
+//
+// This operation creates a tensor of `shape` and `dtype`.
+//
+// Arguments:
+//	shape: 1-D. Represents the shape of the output tensor.
+//
+//
+// Returns A `Tensor` of type `T`.
+func Empty(scope *Scope, shape tf.Output, dtype tf.DataType, optional ...EmptyAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Empty",
+		Input: []tf.Input{
+			shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+//     Subtracts `v` into specified rows of `x`.
+//
+//     Computes y = x; y[i, :] -= v; return y.
+//
+// Arguments:
+//	x: A `Tensor` of type T.
+//	i: A vector. Indices into the left-most dimension of `x`.
+//	v: A `Tensor` of type T. Same dimension sizes as x except the first dimension, which must be the same as i's size.
+//
+// Returns A `Tensor` of type T. An alias of `x`. The content of `y` is undefined if there are duplicates in `i`.
+func InplaceSub(scope *Scope, x tf.Output, i tf.Output, v tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "InplaceSub",
+		Input: []tf.Input{
+			x, i, v,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+//     Adds v into specified rows of x.
+//
+//     Computes y = x; y[i, :] += v; return y.
+//
+// Arguments:
+//	x: A `Tensor` of type T.
+//	i: A vector. Indices into the left-most dimension of `x`.
+//	v: A `Tensor` of type T. Same dimension sizes as x except the first dimension, which must be the same as i's size.
+//
+// Returns A `Tensor` of type T. An alias of `x`. The content of `y` is undefined if there are duplicates in `i`.
+func InplaceAdd(scope *Scope, x tf.Output, i tf.Output, v tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "InplaceAdd",
+		Input: []tf.Input{
+			x, i, v,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Makes a copy of `x`.
+//
+// Arguments:
+//	x: The source tensor of type `T`.
+//
+// Returns     y: A `Tensor` of type `T`. A copy of `x`. Guaranteed that `y`
+//       is not an alias of `x`.
+func DeepCopy(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "DeepCopy",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// PackAttr is an optional argument to Pack.
+type PackAttr func(optionalAttr)
+
+// PackAxis sets the optional axis attribute to value.
+//
+// value: Dimension along which to pack.  Negative values wrap around, so the
+// valid range is `[-(R+1), R+1)`.
+// If not specified, defaults to 0
+func PackAxis(value int64) PackAttr {
+	return func(m optionalAttr) {
+		m["axis"] = value
+	}
+}
+
+// Packs a list of `N` rank-`R` tensors into one rank-`(R+1)` tensor.
+//
+// Packs the `N` tensors in `values` into a tensor with rank one higher than each
+// tensor in `values`, by packing them along the `axis` dimension.
+// Given a list of tensors of shape `(A, B, C)`;
+//
+// if `axis == 0` then the `output` tensor will have the shape `(N, A, B, C)`.
+// if `axis == 1` then the `output` tensor will have the shape `(A, N, B, C)`.
+// Etc.
+//
+// For example:
+//
+// ```
+// # 'x' is [1, 4]
+// # 'y' is [2, 5]
+// # 'z' is [3, 6]
+// pack([x, y, z]) => [[1, 4], [2, 5], [3, 6]]  # Pack along first dim.
+// pack([x, y, z], axis=1) => [[1, 2, 3], [4, 5, 6]]
+// ```
+//
+// This is the opposite of `unpack`.
+//
+// Arguments:
+//	values: Must be of same shape and type.
+//
+// Returns The packed tensor.
+func Pack(scope *Scope, values []tf.Output, optional ...PackAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Pack",
+		Input: []tf.Input{
+			tf.OutputList(values),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MfccAttr is an optional argument to Mfcc.
+type MfccAttr func(optionalAttr)
+
+// MfccUpperFrequencyLimit sets the optional upper_frequency_limit attribute to value.
+//
+// value: The highest frequency to use when calculating the
+// ceptstrum.
+// If not specified, defaults to 4000
+func MfccUpperFrequencyLimit(value float32) MfccAttr {
+	return func(m optionalAttr) {
+		m["upper_frequency_limit"] = value
+	}
+}
+
+// MfccLowerFrequencyLimit sets the optional lower_frequency_limit attribute to value.
+//
+// value: The lowest frequency to use when calculating the
+// ceptstrum.
+// If not specified, defaults to 20
+func MfccLowerFrequencyLimit(value float32) MfccAttr {
+	return func(m optionalAttr) {
+		m["lower_frequency_limit"] = value
+	}
+}
+
+// MfccFilterbankChannelCount sets the optional filterbank_channel_count attribute to value.
+//
+// value: Resolution of the Mel bank used internally.
+// If not specified, defaults to 40
+func MfccFilterbankChannelCount(value int64) MfccAttr {
+	return func(m optionalAttr) {
+		m["filterbank_channel_count"] = value
+	}
+}
+
+// MfccDctCoefficientCount sets the optional dct_coefficient_count attribute to value.
+//
+// value: How many output channels to produce per time slice.
+// If not specified, defaults to 13
+func MfccDctCoefficientCount(value int64) MfccAttr {
+	return func(m optionalAttr) {
+		m["dct_coefficient_count"] = value
+	}
+}
+
+// Transforms a spectrogram into a form that's useful for speech recognition.
+//
+// Mel Frequency Cepstral Coefficients are a way of representing audio data that's
+// been effective as an input feature for machine learning. They are created by
+// taking the spectrum of a spectrogram (a 'cepstrum'), and discarding some of the
+// higher frequencies that are less significant to the human ear. They have a long
+// history in the speech recognition world, and https://en.wikipedia.org/wiki/Mel-frequency_cepstrum
+// is a good resource to learn more.
+//
+// Arguments:
+//	spectrogram: Typically produced by the Spectrogram op, with magnitude_squared
+// set to true.
+//	sample_rate: How many samples per second the source audio used.
+func Mfcc(scope *Scope, spectrogram tf.Output, sample_rate tf.Output, optional ...MfccAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Mfcc",
+		Input: []tf.Input{
+			spectrogram, sample_rate,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// AudioSpectrogramAttr is an optional argument to AudioSpectrogram.
+type AudioSpectrogramAttr func(optionalAttr)
+
+// AudioSpectrogramMagnitudeSquared sets the optional magnitude_squared attribute to value.
+//
+// value: Whether to return the squared magnitude or just the
+// magnitude. Using squared magnitude can avoid extra calculations.
+// If not specified, defaults to false
+func AudioSpectrogramMagnitudeSquared(value bool) AudioSpectrogramAttr {
+	return func(m optionalAttr) {
+		m["magnitude_squared"] = value
+	}
+}
+
+// Produces a visualization of audio data over time.
+//
+// Spectrograms are a standard way of representing audio information as a series of
+// slices of frequency information, one slice for each window of time. By joining
+// these together into a sequence, they form a distinctive fingerprint of the sound
+// over time.
+//
+// This op expects to receive audio data as an input, stored as floats in the range
+// -1 to 1, together with a window width in samples, and a stride specifying how
+// far to move the window between slices. From this it generates a three
+// dimensional output. The first dimension is for the channels in the input, so a
+// stereo audio input would have two here for example. The second dimension is time,
+// with successive frequency slices. The third dimension has an amplitude value for
+// each frequency during that time slice.
+//
+// This means the layout when converted and saved as an image is rotated 90 degrees
+// clockwise from a typical spectrogram. Time is descending down the Y axis, and
+// the frequency decreases from left to right.
+//
+// Each value in the result represents the square root of the sum of the real and
+// imaginary parts of an FFT on the current window of samples. In this way, the
+// lowest dimension represents the power of each frequency in the current window,
+// and adjacent windows are concatenated in the next dimension.
+//
+// To get a more intuitive and visual look at what this operation does, you can run
+// tensorflow/examples/wav_to_spectrogram to read in an audio file and save out the
+// resulting spectrogram as a PNG image.
+//
+// Arguments:
+//	input: Float representation of audio data.
+//	window_size: How wide the input window is in samples. For the highest efficiency
+// this should be a power of two, but other values are accepted.
+//	stride: How widely apart the center of adjacent sample windows should be.
+//
+// Returns 3D representation of the audio frequencies as an image.
+func AudioSpectrogram(scope *Scope, input tf.Output, window_size int64, stride int64, optional ...AudioSpectrogramAttr) (spectrogram tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"window_size": window_size, "stride": stride}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "AudioSpectrogram",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// DecodeWavAttr is an optional argument to DecodeWav.
+type DecodeWavAttr func(optionalAttr)
+
+// DecodeWavDesiredChannels sets the optional desired_channels attribute to value.
+//
+// value: Number of sample channels wanted.
+// If not specified, defaults to -1
+func DecodeWavDesiredChannels(value int64) DecodeWavAttr {
+	return func(m optionalAttr) {
+		m["desired_channels"] = value
+	}
+}
+
+// DecodeWavDesiredSamples sets the optional desired_samples attribute to value.
+//
+// value: Length of audio requested.
+// If not specified, defaults to -1
+func DecodeWavDesiredSamples(value int64) DecodeWavAttr {
+	return func(m optionalAttr) {
+		m["desired_samples"] = value
+	}
+}
+
+// Decode a 16-bit PCM WAV file to a float tensor.
+//
+// The -32768 to 32767 signed 16-bit values will be scaled to -1.0 to 1.0 in float.
+//
+// When desired_channels is set, if the input contains fewer channels than this
+// then the last channel will be duplicated to give the requested number, else if
+// the input has more channels than requested then the additional channels will be
+// ignored.
+//
+// If desired_samples is set, then the audio will be cropped or padded with zeroes
+// to the requested length.
+//
+// The first output contains a Tensor with the content of the audio samples. The
+// lowest dimension will be the number of channels, and the second will be the
+// number of samples. For example, a ten-sample-long stereo WAV file should give an
+// output shape of [10, 2].
+//
+// Arguments:
+//	contents: The WAV-encoded audio, usually from a file.
+//
+// Returns:
+//	audio: 2-D with shape `[length, channels]`.
+//	sample_rate: Scalar holding the sample rate found in the WAV header.
+func DecodeWav(scope *Scope, contents tf.Output, optional ...DecodeWavAttr) (audio tf.Output, sample_rate tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DecodeWav",
+		Input: []tf.Input{
+			contents,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// UnbatchGradAttr is an optional argument to UnbatchGrad.
+type UnbatchGradAttr func(optionalAttr)
+
+// UnbatchGradContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func UnbatchGradContainer(value string) UnbatchGradAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// UnbatchGradSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func UnbatchGradSharedName(value string) UnbatchGradAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Gradient of Unbatch.
+//
+// Acts like Batch but using the given batch_index index of batching things as they
+// become available. This ensures that the gradients are propagated back in the
+// same session which did the forward pass.
+//
+// original_input: The input to the Unbatch operation this is the gradient of.
+// batch_index: The batch_index given to the Unbatch operation this is the gradient
+// of.
+// grad: The downstream gradient.
+// id: The id scalar emitted by Batch.
+// batched_grad: The return value, either an empty tensor or the batched gradient.
+// container: Container to control resource sharing.
+// shared_name: Instances of UnbatchGrad with the same container and shared_name
+//  are assumed to possibly belong to the same batch. If left empty, the op name
+//  will be used as the shared name.
+func UnbatchGrad(scope *Scope, original_input tf.Output, batch_index tf.Output, grad tf.Output, id tf.Output, optional ...UnbatchGradAttr) (batched_grad tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "UnbatchGrad",
+		Input: []tf.Input{
+			original_input, batch_index, grad, id,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes element-wise population count (a.k.a. popcount, bitsum, bitcount).
+//
+// For each entry in `x`, calculates the number of `1` (on) bits in the binary
+// representation of that entry.
+//
+// **NOTE**: It is more efficient to first `tf.bitcast` your tensors into
+// `int32` or `int64` and perform the bitcount on the result, than to feed in
+// 8- or 16-bit inputs and then aggregate the resulting counts.
+func PopulationCount(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "PopulationCount",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Bucketize each feature based on bucket boundaries.
+//
+// An op that returns a list of float tensors, where each tensor represents the
+// bucketized values for a single feature.
+//
+// Arguments:
+//	float_values: float; List of Rank 1 Tensor each containing float values for a single feature.
+//	bucket_boundaries: float; List of Rank 1 Tensors each containing the bucket boundaries for a single
+// feature.
+//
+// Returns int; List of Rank 1 Tensors each containing the bucketized values for a single feature.
+func BoostedTreesBucketize(scope *Scope, float_values []tf.Output, bucket_boundaries []tf.Output) (buckets []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesBucketize",
+		Input: []tf.Input{
+			tf.OutputList(float_values), tf.OutputList(bucket_boundaries),
+		},
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if buckets, idx, err = makeOutputList(op, idx, "buckets"); err != nil {
+		scope.UpdateErr("BoostedTreesBucketize", err)
+		return
+	}
+	return buckets
+}
+
+// Returns immutable tensor from memory region.
+//
+// The current implementation memmaps the tensor from a file.
+//
+// Arguments:
+//	dtype: Type of the returned tensor.
+//	shape: Shape of the returned tensor.
+//	memory_region_name: Name of readonly memory region used by the tensor, see
+// NewReadOnlyMemoryRegionFromFile in tensorflow::Env.
+func ImmutableConst(scope *Scope, dtype tf.DataType, shape tf.Shape, memory_region_name string) (tensor tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype, "shape": shape, "memory_region_name": memory_region_name}
+	opspec := tf.OpSpec{
+		Type: "ImmutableConst",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Add the quantile summaries to each quantile stream resource.
+//
+// An op that adds a list of quantile summaries to a quantile stream resource. Each
+// summary Tensor is rank 2, containing summaries (value, weight, min_rank, max_rank)
+// for a single feature.
+//
+// Arguments:
+//	quantile_stream_resource_handle: resource handle referring to a QuantileStreamResource.
+//	summaries: string; List of Rank 2 Tensor each containing the summaries for a single feature.
+//
+// Returns the created operation.
+func BoostedTreesQuantileStreamResourceAddSummaries(scope *Scope, quantile_stream_resource_handle tf.Output, summaries []tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesQuantileStreamResourceAddSummaries",
+		Input: []tf.Input{
+			quantile_stream_resource_handle, tf.OutputList(summaries),
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// BoostedTreesCreateQuantileStreamResourceAttr is an optional argument to BoostedTreesCreateQuantileStreamResource.
+type BoostedTreesCreateQuantileStreamResourceAttr func(optionalAttr)
+
+// BoostedTreesCreateQuantileStreamResourceMaxElements sets the optional max_elements attribute to value.
+//
+// value: int; The maximum number of data points that can be fed to the stream.
+// If not specified, defaults to 1099511627776
+func BoostedTreesCreateQuantileStreamResourceMaxElements(value int64) BoostedTreesCreateQuantileStreamResourceAttr {
+	return func(m optionalAttr) {
+		m["max_elements"] = value
+	}
+}
+
+// Create the Resource for Quantile Streams.
+//
+// Arguments:
+//	quantile_stream_resource_handle: resource; Handle to quantile stream resource.
+//	epsilon: float; The required approximation error of the stream resource.
+//	num_streams: int; The number of streams managed by the resource that shares the same epsilon.
+//
+// Returns the created operation.
+func BoostedTreesCreateQuantileStreamResource(scope *Scope, quantile_stream_resource_handle tf.Output, epsilon tf.Output, num_streams tf.Output, optional ...BoostedTreesCreateQuantileStreamResourceAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesCreateQuantileStreamResource",
+		Input: []tf.Input{
+			quantile_stream_resource_handle, epsilon, num_streams,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// BoostedTreesUpdateEnsembleV2Attr is an optional argument to BoostedTreesUpdateEnsembleV2.
+type BoostedTreesUpdateEnsembleV2Attr func(optionalAttr)
+
+// BoostedTreesUpdateEnsembleV2LogitsDimension sets the optional logits_dimension attribute to value.
+//
+// value: scalar, dimension of the logits
+// If not specified, defaults to 1
+func BoostedTreesUpdateEnsembleV2LogitsDimension(value int64) BoostedTreesUpdateEnsembleV2Attr {
+	return func(m optionalAttr) {
+		m["logits_dimension"] = value
+	}
+}
+
+// Updates the tree ensemble by adding a layer to the last tree being grown
+//
+// or by starting a new tree.
+//
+// Arguments:
+//	tree_ensemble_handle: Handle to the ensemble variable.
+//	feature_ids: Rank 1 tensor with ids for each feature. This is the real id of
+// the feature that will be used in the split.
+//	dimension_ids: List of rank 1 tensors representing the dimension in each feature.
+//	node_ids: List of rank 1 tensors representing the nodes for which this feature
+// has a split.
+//	gains: List of rank 1 tensors representing the gains for each of the feature's
+// split.
+//	thresholds: List of rank 1 tensors representing the thesholds for each of the
+// feature's split.
+//	left_node_contribs: List of rank 2 tensors with left leaf contribs for each of
+// the feature's splits. Will be added to the previous node values to constitute
+// the values of the left nodes.
+//	right_node_contribs: List of rank 2 tensors with right leaf contribs for each
+// of the feature's splits. Will be added to the previous node values to constitute
+// the values of the right nodes.
+//	split_types: List of rank 1 tensors representing the split type for each feature.
+//	max_depth: Max depth of the tree to build.
+//	learning_rate: shrinkage const for each new tree.
+//	pruning_mode: 0-No pruning, 1-Pre-pruning, 2-Post-pruning.
+//
+// Returns the created operation.
+func BoostedTreesUpdateEnsembleV2(scope *Scope, tree_ensemble_handle tf.Output, feature_ids []tf.Output, dimension_ids []tf.Output, node_ids []tf.Output, gains []tf.Output, thresholds []tf.Output, left_node_contribs []tf.Output, right_node_contribs []tf.Output, split_types []tf.Output, max_depth tf.Output, learning_rate tf.Output, pruning_mode tf.Output, optional ...BoostedTreesUpdateEnsembleV2Attr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesUpdateEnsembleV2",
+		Input: []tf.Input{
+			tree_ensemble_handle, tf.OutputList(feature_ids), tf.OutputList(dimension_ids), tf.OutputList(node_ids), tf.OutputList(gains), tf.OutputList(thresholds), tf.OutputList(left_node_contribs), tf.OutputList(right_node_contribs), tf.OutputList(split_types), max_depth, learning_rate, pruning_mode,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Updates the tree ensemble by either adding a layer to the last tree being grown
+//
+// or by starting a new tree.
+//
+// Arguments:
+//	tree_ensemble_handle: Handle to the ensemble variable.
+//	feature_ids: Rank 1 tensor with ids for each feature. This is the real id of
+// the feature that will be used in the split.
+//	node_ids: List of rank 1 tensors representing the nodes for which this feature
+// has a split.
+//	gains: List of rank 1 tensors representing the gains for each of the feature's
+// split.
+//	thresholds: List of rank 1 tensors representing the thesholds for each of the
+// feature's split.
+//	left_node_contribs: List of rank 2 tensors with left leaf contribs for each of
+// the feature's splits. Will be added to the previous node values to constitute
+// the values of the left nodes.
+//	right_node_contribs: List of rank 2 tensors with right leaf contribs for each
+// of the feature's splits. Will be added to the previous node values to constitute
+// the values of the right nodes.
+//	max_depth: Max depth of the tree to build.
+//	learning_rate: shrinkage const for each new tree.
+//	pruning_mode: 0-No pruning, 1-Pre-pruning, 2-Post-pruning.
+//
+// Returns the created operation.
+func BoostedTreesUpdateEnsemble(scope *Scope, tree_ensemble_handle tf.Output, feature_ids tf.Output, node_ids []tf.Output, gains []tf.Output, thresholds []tf.Output, left_node_contribs []tf.Output, right_node_contribs []tf.Output, max_depth tf.Output, learning_rate tf.Output, pruning_mode int64) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"pruning_mode": pruning_mode}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesUpdateEnsemble",
+		Input: []tf.Input{
+			tree_ensemble_handle, feature_ids, tf.OutputList(node_ids), tf.OutputList(gains), tf.OutputList(thresholds), tf.OutputList(left_node_contribs), tf.OutputList(right_node_contribs), max_depth, learning_rate,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Runs multiple additive regression ensemble predictors on input instances and
+//
+// computes the update to cached logits. It is designed to be used during training.
+// It traverses the trees starting from cached tree id and cached node id and
+// calculates the updates to be pushed to the cache.
+//
+// Arguments:
+//
+//	cached_tree_ids: Rank 1 Tensor containing cached tree ids which is the starting
+// tree of prediction.
+//	cached_node_ids: Rank 1 Tensor containing cached node id which is the starting
+// node of prediction.
+//	bucketized_features: A list of rank 1 Tensors containing bucket id for each
+// feature.
+//	logits_dimension: scalar, dimension of the logits, to be used for partial logits
+// shape.
+//
+// Returns:
+//	partial_logits: Rank 2 Tensor containing logits update (with respect to cached
+// values stored) for each example.
+//	tree_ids: Rank 1 Tensor containing new tree ids for each example.
+//	node_ids: Rank 1 Tensor containing new node ids in the new tree_ids.
+func BoostedTreesTrainingPredict(scope *Scope, tree_ensemble_handle tf.Output, cached_tree_ids tf.Output, cached_node_ids tf.Output, bucketized_features []tf.Output, logits_dimension int64) (partial_logits tf.Output, tree_ids tf.Output, node_ids tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"logits_dimension": logits_dimension}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesTrainingPredict",
+		Input: []tf.Input{
+			tree_ensemble_handle, cached_tree_ids, cached_node_ids, tf.OutputList(bucketized_features),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Aggregates the summary of accumulated stats for the batch.
+//
+// The summary stats contains gradients and hessians accumulated for each node, feature dimension id and bucket.
+//
+// Arguments:
+//	node_ids: int32; Rank 1 Tensor containing node ids for each example, shape [batch_size].
+//	gradients: float32; Rank 2 Tensor (shape=[batch_size, logits_dimension]) with gradients for each example.
+//	hessians: float32; Rank 2 Tensor (shape=[batch_size, hessian_dimension]) with hessians for each example.
+//	feature: int32; Rank 2 feature Tensors (shape=[batch_size, feature_dimension]).
+//	max_splits: int; the maximum number of splits possible in the whole tree.
+//	num_buckets: int; equals to the maximum possible value of bucketized feature.
+//
+// Returns output Rank 4 Tensor (shape=[splits, feature_dimension, buckets, logits_dimension + hessian_dimension])
+// containing accumulated stats for each node, feature dimension and bucket.
+func BoostedTreesAggregateStats(scope *Scope, node_ids tf.Output, gradients tf.Output, hessians tf.Output, feature tf.Output, max_splits int64, num_buckets int64) (stats_summary tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"max_splits": max_splits, "num_buckets": num_buckets}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesAggregateStats",
+		Input: []tf.Input{
+			node_ids, gradients, hessians, feature,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Makes the summary of accumulated stats for the batch.
+//
+// The summary stats contains gradients and hessians accumulated into the corresponding node and bucket for each example.
+//
+// Arguments:
+//	node_ids: int32 Rank 1 Tensor containing node ids, which each example falls into for the requested layer.
+//	gradients: float32; Rank 2 Tensor (shape=[#examples, 1]) for gradients.
+//	hessians: float32; Rank 2 Tensor (shape=[#examples, 1]) for hessians.
+//	bucketized_features_list: int32 list of Rank 1 Tensors, each containing the bucketized feature (for each feature column).
+//	max_splits: int; the maximum number of splits possible in the whole tree.
+//	num_buckets: int; equals to the maximum possible value of bucketized feature.
+//
+// Returns output Rank 4 Tensor (shape=[#features, #splits, #buckets, 2]) containing accumulated stats put into the corresponding node and bucket. The first index of 4th dimension refers to gradients, and the second to hessians.
+func BoostedTreesMakeStatsSummary(scope *Scope, node_ids tf.Output, gradients tf.Output, hessians tf.Output, bucketized_features_list []tf.Output, max_splits int64, num_buckets int64) (stats_summary tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"max_splits": max_splits, "num_buckets": num_buckets}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesMakeStatsSummary",
+		Input: []tf.Input{
+			node_ids, gradients, hessians, tf.OutputList(bucketized_features_list),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Deserializes a serialized tree ensemble config and replaces current tree
+//
+// ensemble.
+//
+// Arguments:
+//	tree_ensemble_handle: Handle to the tree ensemble.
+//	stamp_token: Token to use as the new value of the resource stamp.
+//	tree_ensemble_serialized: Serialized proto of the ensemble.
+//
+// Returns the created operation.
+func BoostedTreesDeserializeEnsemble(scope *Scope, tree_ensemble_handle tf.Output, stamp_token tf.Output, tree_ensemble_serialized tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesDeserializeEnsemble",
+		Input: []tf.Input{
+			tree_ensemble_handle, stamp_token, tree_ensemble_serialized,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Flush the quantile summaries from each quantile stream resource.
+//
+// An op that outputs a list of quantile summaries of a quantile stream resource.
+// Each summary Tensor is rank 2, containing summaries (value, weight, min_rank,
+// max_rank) for a single feature.
+//
+// Arguments:
+//	quantile_stream_resource_handle: resource handle referring to a QuantileStreamResource.
+//
+func BoostedTreesFlushQuantileSummaries(scope *Scope, quantile_stream_resource_handle tf.Output, num_features int64) (summaries []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_features": num_features}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesFlushQuantileSummaries",
+		Input: []tf.Input{
+			quantile_stream_resource_handle,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if summaries, idx, err = makeOutputList(op, idx, "summaries"); err != nil {
+		scope.UpdateErr("BoostedTreesFlushQuantileSummaries", err)
+		return
+	}
+	return summaries
+}
+
+// BoostedTreesSparseCalculateBestFeatureSplitAttr is an optional argument to BoostedTreesSparseCalculateBestFeatureSplit.
+type BoostedTreesSparseCalculateBestFeatureSplitAttr func(optionalAttr)
+
+// BoostedTreesSparseCalculateBestFeatureSplitSplitType sets the optional split_type attribute to value.
+//
+// value: A string indicating if this Op should perform inequality split or equality split.
+// If not specified, defaults to "inequality"
+func BoostedTreesSparseCalculateBestFeatureSplitSplitType(value string) BoostedTreesSparseCalculateBestFeatureSplitAttr {
+	return func(m optionalAttr) {
+		m["split_type"] = value
+	}
+}
+
+// Calculates gains for each feature and returns the best possible split information for the feature.
+//
+// The split information is the best threshold (bucket id), gains and left/right node contributions per node for each feature.
+//
+// It is possible that not all nodes can be split on each feature. Hence, the list of possible nodes can differ between the features. Therefore, we return `node_ids_list` for each feature, containing the list of nodes that this feature can be used to split.
+//
+// In this manner, the output is the best split per features and per node, so that it needs to be combined later to produce the best split for each node (among all possible features).
+//
+// The output shapes are compatible in a way that the first dimension of all tensors are the same and equal to the number of possible split nodes for each feature.
+//
+// Arguments:
+//	node_id_range: A Rank 1 tensor (shape=[2]) to specify the range [first, last) of node ids to process within `stats_summary_list`. The nodes are iterated between the two nodes specified by the tensor, as like `for node_id in range(node_id_range[0], node_id_range[1])` (Note that the last index node_id_range[1] is exclusive).
+//	stats_summary_indices: A Rank 2 int64 tensor of dense shape [N, 4] (N specifies the number of non-zero values) for accumulated stats summary (gradient/hessian) per node per bucket for each feature. The second dimension contains node id, feature dimension, bucket id, and stats dim.
+// stats dim is the sum of logits dimension and hessian dimension, hessian dimension can either be logits dimension if diagonal hessian is used, or logits dimension^2 if full hessian is used.
+//	stats_summary_values: A Rank 1 float tensor of dense shape [N] (N specifies the number of non-zero values), which supplies the values for each element in summary_indices.
+//	stats_summary_shape: A Rank 1 float tensor of dense shape [4], which specifies the dense shape of the sparse tensor, which is [num tree nodes, feature dimensions, num buckets, stats dim].
+//	l1: l1 regularization factor on leaf weights, per instance based.
+//	l2: l2 regularization factor on leaf weights, per instance based.
+//	tree_complexity: adjustment to the gain, per leaf based.
+//	min_node_weight: minimum avg of hessians in a node before required for the node to be considered for splitting.
+//	logits_dimension: The dimension of logit, i.e., number of classes.
+//
+// Returns:
+//	node_ids: A Rank 1 tensor indicating possible node ids that can be split.
+//	gains: A Rank 1 tensor indicating the best gains to split each node.
+//	feature_dimensions: A Rank 1 tensor indicating the best feature dimension for each feature to split for each node.
+//	thresholds: A Rank 1 tensor indicating the bucket id to compare with (as a threshold) for split in each node.
+//	left_node_contribs: A Rank 2 tensor indicating the contribution of the left nodes when branching from parent nodes to the left direction by the given threshold for each feature.
+// This value will be used to make the left node value by adding to the parent node value. Second dimension size is logits dimension.
+//	right_node_contribs: A Rank 2 tensor, with the same shape/conditions as left_node_contribs_list, but just that the value is for the right node.
+//	split_with_default_directions: A Rank 1 tensor indicating which direction to go if data is missing.
+// Inequality with default left returns 0, inequality with default right returns 1, equality with default right returns 2.
+func BoostedTreesSparseCalculateBestFeatureSplit(scope *Scope, node_id_range tf.Output, stats_summary_indices tf.Output, stats_summary_values tf.Output, stats_summary_shape tf.Output, l1 tf.Output, l2 tf.Output, tree_complexity tf.Output, min_node_weight tf.Output, logits_dimension int64, optional ...BoostedTreesSparseCalculateBestFeatureSplitAttr) (node_ids tf.Output, gains tf.Output, feature_dimensions tf.Output, thresholds tf.Output, left_node_contribs tf.Output, right_node_contribs tf.Output, split_with_default_directions tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"logits_dimension": logits_dimension}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesSparseCalculateBestFeatureSplit",
+		Input: []tf.Input{
+			node_id_range, stats_summary_indices, stats_summary_values, stats_summary_shape, l1, l2, tree_complexity, min_node_weight,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3), op.Output(4), op.Output(5), op.Output(6)
+}
+
+// Calculates gains for each feature and returns the best possible split information for each node. However, if no split is found, then no split information is returned for that node.
+//
+// The split information is the best threshold (bucket id), gains and left/right node contributions per node for each feature.
+//
+// It is possible that not all nodes can be split on each feature. Hence, the list of possible nodes can differ between the features. Therefore, we return `node_ids_list` for each feature, containing the list of nodes that this feature can be used to split.
+//
+// In this manner, the output is the best split per features and per node, so that it needs to be combined later to produce the best split for each node (among all possible features).
+//
+// The output shapes are compatible in a way that the first dimension of all tensors are the same and equal to the number of possible split nodes for each feature.
+//
+// Arguments:
+//	node_id_range: A Rank 1 tensor (shape=[2]) to specify the range [first, last) of node ids to process within `stats_summary_list`. The nodes are iterated between the two nodes specified by the tensor, as like `for node_id in range(node_id_range[0], node_id_range[1])` (Note that the last index node_id_range[1] is exclusive).
+//	stats_summaries_list: A list of Rank 4 tensor (#shape=[max_splits, feature_dims, bucket, stats_dims]) for accumulated stats summary (gradient/hessian) per node, per dimension, per buckets for each feature.
+// The first dimension of the tensor is the maximum number of splits, and thus not all elements of it will be used, but only the indexes specified by node_ids will be used.
+//	split_types: A Rank 1 tensor indicating if this Op should perform inequality split or equality split per feature.
+//	candidate_feature_ids: Rank 1 tensor with ids for each feature. This is the real id of the feature.
+//	l1: l1 regularization factor on leaf weights, per instance based.
+//	l2: l2 regularization factor on leaf weights, per instance based.
+//	tree_complexity: adjustment to the gain, per leaf based.
+//	min_node_weight: minimum avg of hessians in a node before required for the node to be considered for splitting.
+//	logits_dimension: The dimension of logit, i.e., number of classes.
+//
+// Returns:
+//	node_ids: A Rank 1 tensors indicating possible split node ids for each feature. The length of the list is num_features, but each tensor has different size as each feature provides different possible nodes. See above for details like shapes and sizes.
+//	gains: A Rank 1 tensor indicating the best gains for each feature to split for certain nodes. See above for details like shapes and sizes.
+//	feature_ids: A Rank 1 tensors indicating the best feature id for each node. See above for details like shapes and sizes.
+//	feature_dimensions: A Rank 1 tensors indicating the best feature dimension for each feature to split for certain nodes if the feature is multi-dimension. See above for details like shapes and sizes.
+//	thresholds: A Rank 1 tensors indicating the bucket id to compare with (as a threshold) for split in each node. See above for details like shapes and sizes.
+//	left_node_contribs: A Rank 2 tensors indicating the contribution of the left nodes when branching from parent nodes (given by the tensor element in the output node_ids_list) to the left direction by the given threshold for each feature. This value will be used to make the left node value by adding to the parent node value. Second dimension size is 1 for 1-dimensional logits, but would be larger for multi-class problems. See above for details like shapes and sizes.
+//	right_node_contribs: A Rank 2 tensors, with the same shape/conditions as left_node_contribs_list, but just that the value is for the right node.
+//	split_with_default_directions: A Rank 1 tensors indicating the which direction to go if data is missing. See above for details like shapes and sizes.
+// Inequality with default left returns 0, inequality with default right returns 1, equality with default right returns 2.
+func BoostedTreesCalculateBestFeatureSplitV2(scope *Scope, node_id_range tf.Output, stats_summaries_list []tf.Output, split_types tf.Output, candidate_feature_ids tf.Output, l1 tf.Output, l2 tf.Output, tree_complexity tf.Output, min_node_weight tf.Output, logits_dimension int64) (node_ids tf.Output, gains tf.Output, feature_ids tf.Output, feature_dimensions tf.Output, thresholds tf.Output, left_node_contribs tf.Output, right_node_contribs tf.Output, split_with_default_directions tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"logits_dimension": logits_dimension}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesCalculateBestFeatureSplitV2",
+		Input: []tf.Input{
+			node_id_range, tf.OutputList(stats_summaries_list), split_types, candidate_feature_ids, l1, l2, tree_complexity, min_node_weight,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3), op.Output(4), op.Output(5), op.Output(6), op.Output(7)
+}
+
+// Calculates gains for each feature and returns the best possible split information for the feature.
+//
+// The split information is the best threshold (bucket id), gains and left/right node contributions per node for each feature.
+//
+// It is possible that not all nodes can be split on each feature. Hence, the list of possible nodes can differ between the features. Therefore, we return `node_ids_list` for each feature, containing the list of nodes that this feature can be used to split.
+//
+// In this manner, the output is the best split per features and per node, so that it needs to be combined later to produce the best split for each node (among all possible features).
+//
+// The length of output lists are all of the same length, `num_features`.
+// The output shapes are compatible in a way that the first dimension of all tensors of all lists are the same and equal to the number of possible split nodes for each feature.
+//
+// Arguments:
+//	node_id_range: A Rank 1 tensor (shape=[2]) to specify the range [first, last) of node ids to process within `stats_summary_list`. The nodes are iterated between the two nodes specified by the tensor, as like `for node_id in range(node_id_range[0], node_id_range[1])` (Note that the last index node_id_range[1] is exclusive).
+//	stats_summary_list: A list of Rank 3 tensor (#shape=[max_splits, bucket, 2]) for accumulated stats summary (gradient/hessian) per node per buckets for each feature. The first dimension of the tensor is the maximum number of splits, and thus not all elements of it will be used, but only the indexes specified by node_ids will be used.
+//	l1: l1 regularization factor on leaf weights, per instance based.
+//	l2: l2 regularization factor on leaf weights, per instance based.
+//	tree_complexity: adjustment to the gain, per leaf based.
+//	min_node_weight: minimum avg of hessians in a node before required for the node to be considered for splitting.
+//	max_splits: the number of nodes that can be split in the whole tree. Used as a dimension of output tensors.
+//
+// Returns:
+//	node_ids_list: An output list of Rank 1 tensors indicating possible split node ids for each feature. The length of the list is num_features, but each tensor has different size as each feature provides different possible nodes. See above for details like shapes and sizes.
+//	gains_list: An output list of Rank 1 tensors indicating the best gains for each feature to split for certain nodes. See above for details like shapes and sizes.
+//	thresholds_list: An output list of Rank 1 tensors indicating the bucket id to compare with (as a threshold) for split in each node. See above for details like shapes and sizes.
+//	left_node_contribs_list: A list of Rank 2 tensors indicating the contribution of the left nodes when branching from parent nodes (given by the tensor element in the output node_ids_list) to the left direction by the given threshold for each feature. This value will be used to make the left node value by adding to the parent node value. Second dimension size is 1 for 1-dimensional logits, but would be larger for multi-class problems. See above for details like shapes and sizes.
+//	right_node_contribs_list: A list of Rank 2 tensors, with the same shape/conditions as left_node_contribs_list, but just that the value is for the right node.
+func BoostedTreesCalculateBestGainsPerFeature(scope *Scope, node_id_range tf.Output, stats_summary_list []tf.Output, l1 tf.Output, l2 tf.Output, tree_complexity tf.Output, min_node_weight tf.Output, max_splits int64) (node_ids_list []tf.Output, gains_list []tf.Output, thresholds_list []tf.Output, left_node_contribs_list []tf.Output, right_node_contribs_list []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"max_splits": max_splits}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesCalculateBestGainsPerFeature",
+		Input: []tf.Input{
+			node_id_range, tf.OutputList(stats_summary_list), l1, l2, tree_complexity, min_node_weight,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if node_ids_list, idx, err = makeOutputList(op, idx, "node_ids_list"); err != nil {
+		scope.UpdateErr("BoostedTreesCalculateBestGainsPerFeature", err)
+		return
+	}
+	if gains_list, idx, err = makeOutputList(op, idx, "gains_list"); err != nil {
+		scope.UpdateErr("BoostedTreesCalculateBestGainsPerFeature", err)
+		return
+	}
+	if thresholds_list, idx, err = makeOutputList(op, idx, "thresholds_list"); err != nil {
+		scope.UpdateErr("BoostedTreesCalculateBestGainsPerFeature", err)
+		return
+	}
+	if left_node_contribs_list, idx, err = makeOutputList(op, idx, "left_node_contribs_list"); err != nil {
+		scope.UpdateErr("BoostedTreesCalculateBestGainsPerFeature", err)
+		return
+	}
+	if right_node_contribs_list, idx, err = makeOutputList(op, idx, "right_node_contribs_list"); err != nil {
+		scope.UpdateErr("BoostedTreesCalculateBestGainsPerFeature", err)
+		return
+	}
+	return node_ids_list, gains_list, thresholds_list, left_node_contribs_list, right_node_contribs_list
+}
+
+// Checks whether a tree ensemble has been initialized.
+//
+// Arguments:
+//	tree_ensemble_handle: Handle to the tree ensemble resource.
+//
+// Returns output boolean on whether it is initialized or not.
+func IsBoostedTreesEnsembleInitialized(scope *Scope, tree_ensemble_handle tf.Output) (is_initialized tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "IsBoostedTreesEnsembleInitialized",
+		Input: []tf.Input{
+			tree_ensemble_handle,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// BoostedTreesEnsembleResourceHandleOpAttr is an optional argument to BoostedTreesEnsembleResourceHandleOp.
+type BoostedTreesEnsembleResourceHandleOpAttr func(optionalAttr)
+
+// BoostedTreesEnsembleResourceHandleOpContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func BoostedTreesEnsembleResourceHandleOpContainer(value string) BoostedTreesEnsembleResourceHandleOpAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// BoostedTreesEnsembleResourceHandleOpSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func BoostedTreesEnsembleResourceHandleOpSharedName(value string) BoostedTreesEnsembleResourceHandleOpAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Creates a handle to a BoostedTreesEnsembleResource
+func BoostedTreesEnsembleResourceHandleOp(scope *Scope, optional ...BoostedTreesEnsembleResourceHandleOpAttr) (resource tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesEnsembleResourceHandleOp",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Deserializes a proto into the tree handle
+//
+// Arguments:
+//	tree_handle: Handle to the tree resource to be restored.
+//	tree_config: Serialied proto string of the boosted_trees.Tree proto.
+//
+// Returns the created operation.
+func TensorForestTreeDeserialize(scope *Scope, tree_handle tf.Output, tree_config tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorForestTreeDeserialize",
+		Input: []tf.Input{
+			tree_handle, tree_config,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Serializes the tree handle to a proto
+//
+// Arguments:
+//	tree_handle: Handle to the tree resource to be serialized.
+//
+// Returns Serialied proto string of the tree resource.
+func TensorForestTreeSerialize(scope *Scope, tree_handle tf.Output) (tree_config tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorForestTreeSerialize",
+		Input: []tf.Input{
+			tree_handle,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a tree resource and returns a handle to it.
+//
+// Arguments:
+//	tree_handle: Handle to the tree resource to be created.
+//	tree_config: Serialized proto string of the boosted_trees.Tree.
+//
+// Returns the created operation.
+func TensorForestCreateTreeVariable(scope *Scope, tree_handle tf.Output, tree_config tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorForestCreateTreeVariable",
+		Input: []tf.Input{
+			tree_handle, tree_config,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Checks whether a tree has been initialized.
+//
+// Arguments:
+//	tree_handle: Handle to the tree.
+//
+// Returns Whether the tree is initialized.
+func TensorForestTreeIsInitializedOp(scope *Scope, tree_handle tf.Output) (is_initialized tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorForestTreeIsInitializedOp",
+		Input: []tf.Input{
+			tree_handle,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// TensorForestTreeResourceHandleOpAttr is an optional argument to TensorForestTreeResourceHandleOp.
+type TensorForestTreeResourceHandleOpAttr func(optionalAttr)
+
+// TensorForestTreeResourceHandleOpContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func TensorForestTreeResourceHandleOpContainer(value string) TensorForestTreeResourceHandleOpAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// TensorForestTreeResourceHandleOpSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func TensorForestTreeResourceHandleOpSharedName(value string) TensorForestTreeResourceHandleOpAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Creates a handle to a TensorForestTreeResource
+func TensorForestTreeResourceHandleOp(scope *Scope, optional ...TensorForestTreeResourceHandleOpAttr) (resource tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorForestTreeResourceHandleOp",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// AllCandidateSamplerAttr is an optional argument to AllCandidateSampler.
+type AllCandidateSamplerAttr func(optionalAttr)
+
+// AllCandidateSamplerSeed sets the optional seed attribute to value.
+//
+// value: If either seed or seed2 are set to be non-zero, the random number
+// generator is seeded by the given seed.  Otherwise, it is seeded by a
+// random seed.
+// If not specified, defaults to 0
+func AllCandidateSamplerSeed(value int64) AllCandidateSamplerAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// AllCandidateSamplerSeed2 sets the optional seed2 attribute to value.
+//
+// value: An second seed to avoid seed collision.
+// If not specified, defaults to 0
+func AllCandidateSamplerSeed2(value int64) AllCandidateSamplerAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Generates labels for candidate sampling with a learned unigram distribution.
+//
+// See explanations of candidate sampling and the data formats at
+// go/candidate-sampling.
+//
+// For each batch, this op picks a single set of sampled candidate labels.
+//
+// The advantages of sampling candidates per-batch are simplicity and the
+// possibility of efficient dense matrix multiplication. The disadvantage is that
+// the sampled candidates must be chosen independently of the context and of the
+// true labels.
+//
+// Arguments:
+//	true_classes: A batch_size * num_true matrix, in which each row contains the
+// IDs of the num_true target_classes in the corresponding original label.
+//	num_true: Number of true labels per context.
+//	num_sampled: Number of candidates to produce.
+//	unique: If unique is true, we sample with rejection, so that all sampled
+// candidates in a batch are unique. This requires some approximation to
+// estimate the post-rejection sampling probabilities.
+//
+// Returns:
+//	sampled_candidates: A vector of length num_sampled, in which each element is
+// the ID of a sampled candidate.
+//	true_expected_count: A batch_size * num_true matrix, representing
+// the number of times each candidate is expected to occur in a batch
+// of sampled candidates. If unique=true, then this is a probability.
+//	sampled_expected_count: A vector of length num_sampled, for each sampled
+// candidate representing the number of times the candidate is expected
+// to occur in a batch of sampled candidates.  If unique=true, then this is a
+// probability.
+func AllCandidateSampler(scope *Scope, true_classes tf.Output, num_true int64, num_sampled int64, unique bool, optional ...AllCandidateSamplerAttr) (sampled_candidates tf.Output, true_expected_count tf.Output, sampled_expected_count tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_true": num_true, "num_sampled": num_sampled, "unique": unique}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "AllCandidateSampler",
+		Input: []tf.Input{
+			true_classes,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// FixedUnigramCandidateSamplerAttr is an optional argument to FixedUnigramCandidateSampler.
+type FixedUnigramCandidateSamplerAttr func(optionalAttr)
+
+// FixedUnigramCandidateSamplerVocabFile sets the optional vocab_file attribute to value.
+//
+// value: Each valid line in this file (which should have a CSV-like format)
+// corresponds to a valid word ID. IDs are in sequential order, starting from
+// num_reserved_ids. The last entry in each line is expected to be a value
+// corresponding to the count or relative probability. Exactly one of vocab_file
+// and unigrams needs to be passed to this op.
+// If not specified, defaults to ""
+func FixedUnigramCandidateSamplerVocabFile(value string) FixedUnigramCandidateSamplerAttr {
+	return func(m optionalAttr) {
+		m["vocab_file"] = value
+	}
+}
+
+// FixedUnigramCandidateSamplerDistortion sets the optional distortion attribute to value.
+//
+// value: The distortion is used to skew the unigram probability distribution.
+// Each weight is first raised to the distortion's power before adding to the
+// internal unigram distribution. As a result, distortion = 1.0 gives regular
+// unigram sampling (as defined by the vocab file), and distortion = 0.0 gives
+// a uniform distribution.
+// If not specified, defaults to 1
+func FixedUnigramCandidateSamplerDistortion(value float32) FixedUnigramCandidateSamplerAttr {
+	return func(m optionalAttr) {
+		m["distortion"] = value
+	}
+}
+
+// FixedUnigramCandidateSamplerNumReservedIds sets the optional num_reserved_ids attribute to value.
+//
+// value: Optionally some reserved IDs can be added in the range [0,
+// ..., num_reserved_ids) by the users. One use case is that a special unknown
+// word token is used as ID 0. These IDs will have a sampling probability of 0.
+// If not specified, defaults to 0
+func FixedUnigramCandidateSamplerNumReservedIds(value int64) FixedUnigramCandidateSamplerAttr {
+	return func(m optionalAttr) {
+		m["num_reserved_ids"] = value
+	}
+}
+
+// FixedUnigramCandidateSamplerNumShards sets the optional num_shards attribute to value.
+//
+// value: A sampler can be used to sample from a subset of the original range
+// in order to speed up the whole computation through parallelism. This parameter
+// (together with 'shard') indicates the number of partitions that are being
+// used in the overall computation.
+// If not specified, defaults to 1
+//
+// REQUIRES: value >= 1
+func FixedUnigramCandidateSamplerNumShards(value int64) FixedUnigramCandidateSamplerAttr {
+	return func(m optionalAttr) {
+		m["num_shards"] = value
+	}
+}
+
+// FixedUnigramCandidateSamplerShard sets the optional shard attribute to value.
+//
+// value: A sampler can be used to sample from a subset of the original range
+// in order to speed up the whole computation through parallelism. This parameter
+// (together with 'num_shards') indicates the particular partition number of a
+// sampler op, when partitioning is being used.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func FixedUnigramCandidateSamplerShard(value int64) FixedUnigramCandidateSamplerAttr {
+	return func(m optionalAttr) {
+		m["shard"] = value
+	}
+}
+
+// FixedUnigramCandidateSamplerUnigrams sets the optional unigrams attribute to value.
+//
+// value: A list of unigram counts or probabilities, one per ID in sequential
+// order. Exactly one of vocab_file and unigrams should be passed to this op.
+// If not specified, defaults to <>
+func FixedUnigramCandidateSamplerUnigrams(value []float32) FixedUnigramCandidateSamplerAttr {
+	return func(m optionalAttr) {
+		m["unigrams"] = value
+	}
+}
+
+// FixedUnigramCandidateSamplerSeed sets the optional seed attribute to value.
+//
+// value: If either seed or seed2 are set to be non-zero, the random number
+// generator is seeded by the given seed.  Otherwise, it is seeded by a
+// random seed.
+// If not specified, defaults to 0
+func FixedUnigramCandidateSamplerSeed(value int64) FixedUnigramCandidateSamplerAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// FixedUnigramCandidateSamplerSeed2 sets the optional seed2 attribute to value.
+//
+// value: An second seed to avoid seed collision.
+// If not specified, defaults to 0
+func FixedUnigramCandidateSamplerSeed2(value int64) FixedUnigramCandidateSamplerAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Generates labels for candidate sampling with a learned unigram distribution.
+//
+// A unigram sampler could use a fixed unigram distribution read from a
+// file or passed in as an in-memory array instead of building up the distribution
+// from data on the fly. There is also an option to skew the distribution by
+// applying a distortion power to the weights.
+//
+// The vocabulary file should be in CSV-like format, with the last field
+// being the weight associated with the word.
+//
+// For each batch, this op picks a single set of sampled candidate labels.
+//
+// The advantages of sampling candidates per-batch are simplicity and the
+// possibility of efficient dense matrix multiplication. The disadvantage is that
+// the sampled candidates must be chosen independently of the context and of the
+// true labels.
+//
+// Arguments:
+//	true_classes: A batch_size * num_true matrix, in which each row contains the
+// IDs of the num_true target_classes in the corresponding original label.
+//	num_true: Number of true labels per context.
+//	num_sampled: Number of candidates to randomly sample.
+//	unique: If unique is true, we sample with rejection, so that all sampled
+// candidates in a batch are unique. This requires some approximation to
+// estimate the post-rejection sampling probabilities.
+//	range_max: The sampler will sample integers from the interval [0, range_max).
+//
+// Returns:
+//	sampled_candidates: A vector of length num_sampled, in which each element is
+// the ID of a sampled candidate.
+//	true_expected_count: A batch_size * num_true matrix, representing
+// the number of times each candidate is expected to occur in a batch
+// of sampled candidates. If unique=true, then this is a probability.
+//	sampled_expected_count: A vector of length num_sampled, for each sampled
+// candidate representing the number of times the candidate is expected
+// to occur in a batch of sampled candidates.  If unique=true, then this is a
+// probability.
+func FixedUnigramCandidateSampler(scope *Scope, true_classes tf.Output, num_true int64, num_sampled int64, unique bool, range_max int64, optional ...FixedUnigramCandidateSamplerAttr) (sampled_candidates tf.Output, true_expected_count tf.Output, sampled_expected_count tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_true": num_true, "num_sampled": num_sampled, "unique": unique, "range_max": range_max}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "FixedUnigramCandidateSampler",
+		Input: []tf.Input{
+			true_classes,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// ThreadUnsafeUnigramCandidateSamplerAttr is an optional argument to ThreadUnsafeUnigramCandidateSampler.
+type ThreadUnsafeUnigramCandidateSamplerAttr func(optionalAttr)
+
+// ThreadUnsafeUnigramCandidateSamplerSeed sets the optional seed attribute to value.
+//
+// value: If either seed or seed2 are set to be non-zero, the random number
+// generator is seeded by the given seed.  Otherwise, it is seeded by a
+// random seed.
+// If not specified, defaults to 0
+func ThreadUnsafeUnigramCandidateSamplerSeed(value int64) ThreadUnsafeUnigramCandidateSamplerAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// ThreadUnsafeUnigramCandidateSamplerSeed2 sets the optional seed2 attribute to value.
+//
+// value: An second seed to avoid seed collision.
+// If not specified, defaults to 0
+func ThreadUnsafeUnigramCandidateSamplerSeed2(value int64) ThreadUnsafeUnigramCandidateSamplerAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Generates labels for candidate sampling with a learned unigram distribution.
+//
+// See explanations of candidate sampling and the data formats at
+// go/candidate-sampling.
+//
+// For each batch, this op picks a single set of sampled candidate labels.
+//
+// The advantages of sampling candidates per-batch are simplicity and the
+// possibility of efficient dense matrix multiplication. The disadvantage is that
+// the sampled candidates must be chosen independently of the context and of the
+// true labels.
+//
+// Arguments:
+//	true_classes: A batch_size * num_true matrix, in which each row contains the
+// IDs of the num_true target_classes in the corresponding original label.
+//	num_true: Number of true labels per context.
+//	num_sampled: Number of candidates to randomly sample.
+//	unique: If unique is true, we sample with rejection, so that all sampled
+// candidates in a batch are unique. This requires some approximation to
+// estimate the post-rejection sampling probabilities.
+//	range_max: The sampler will sample integers from the interval [0, range_max).
+//
+// Returns:
+//	sampled_candidates: A vector of length num_sampled, in which each element is
+// the ID of a sampled candidate.
+//	true_expected_count: A batch_size * num_true matrix, representing
+// the number of times each candidate is expected to occur in a batch
+// of sampled candidates. If unique=true, then this is a probability.
+//	sampled_expected_count: A vector of length num_sampled, for each sampled
+// candidate representing the number of times the candidate is expected
+// to occur in a batch of sampled candidates.  If unique=true, then this is a
+// probability.
+func ThreadUnsafeUnigramCandidateSampler(scope *Scope, true_classes tf.Output, num_true int64, num_sampled int64, unique bool, range_max int64, optional ...ThreadUnsafeUnigramCandidateSamplerAttr) (sampled_candidates tf.Output, true_expected_count tf.Output, sampled_expected_count tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_true": num_true, "num_sampled": num_sampled, "unique": unique, "range_max": range_max}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ThreadUnsafeUnigramCandidateSampler",
+		Input: []tf.Input{
+			true_classes,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// MatrixDiagPartV3Attr is an optional argument to MatrixDiagPartV3.
+type MatrixDiagPartV3Attr func(optionalAttr)
+
+// MatrixDiagPartV3Align sets the optional align attribute to value.
+//
+// value: Some diagonals are shorter than `max_diag_len` and need to be padded. `align` is
+// a string specifying how superdiagonals and subdiagonals should be aligned,
+// respectively. There are four possible alignments: "RIGHT_LEFT" (default),
+// "LEFT_RIGHT", "LEFT_LEFT", and "RIGHT_RIGHT". "RIGHT_LEFT" aligns superdiagonals
+// to the right (left-pads the row) and subdiagonals to the left (right-pads the
+// row). It is the packing format LAPACK uses. cuSPARSE uses "LEFT_RIGHT", which is
+// the opposite alignment.
+// If not specified, defaults to "RIGHT_LEFT"
+func MatrixDiagPartV3Align(value string) MatrixDiagPartV3Attr {
+	return func(m optionalAttr) {
+		m["align"] = value
+	}
+}
+
+// Returns the batched diagonal part of a batched tensor.
+//
+// Returns a tensor with the `k[0]`-th to `k[1]`-th diagonals of the batched
+// `input`.
+//
+// Assume `input` has `r` dimensions `[I, J, ..., L, M, N]`.
+// Let `max_diag_len` be the maximum length among all diagonals to be extracted,
+// `max_diag_len = min(M + min(k[1], 0), N + min(-k[0], 0))`
+// Let `num_diags` be the number of diagonals to extract,
+// `num_diags = k[1] - k[0] + 1`.
+//
+// If `num_diags == 1`, the output tensor is of rank `r - 1` with shape
+// `[I, J, ..., L, max_diag_len]` and values:
+//
+// ```
+// diagonal[i, j, ..., l, n]
+//   = input[i, j, ..., l, n+y, n+x] ; if 0 <= n+y < M and 0 <= n+x < N,
+//     padding_value                 ; otherwise.
+// ```
+// where `y = max(-k[1], 0)`, `x = max(k[1], 0)`.
+//
+// Otherwise, the output tensor has rank `r` with dimensions
+// `[I, J, ..., L, num_diags, max_diag_len]` with values:
+//
+// ```
+// diagonal[i, j, ..., l, m, n]
+//   = input[i, j, ..., l, n+y, n+x] ; if 0 <= n+y < M and 0 <= n+x < N,
+//     padding_value                 ; otherwise.
+// ```
+// where `d = k[1] - m`, `y = max(-d, 0) - offset`, and `x = max(d, 0) - offset`.
+//
+// `offset` is zero except when the alignment of the diagonal is to the right.
+// ```
+// offset = max_diag_len - diag_len(d) ; if (`align` in {RIGHT_LEFT, RIGHT_RIGHT}
+//                                            and `d >= 0`) or
+//                                          (`align` in {LEFT_RIGHT, RIGHT_RIGHT}
+//                                            and `d <= 0`)
+//          0                          ; otherwise
+// ```
+// where `diag_len(d) = min(cols - max(d, 0), rows + min(d, 0))`.
+//
+// The input must be at least a matrix.
+//
+// For example:
+//
+// ```
+// input = np.array([[[1, 2, 3, 4],  # Input shape: (2, 3, 4)
+//                    [5, 6, 7, 8],
+//                    [9, 8, 7, 6]],
+//                   [[5, 4, 3, 2],
+//                    [1, 2, 3, 4],
+//                    [5, 6, 7, 8]]])
+//
+// # A main diagonal from each batch.
+// tf.matrix_diag_part(input) ==> [[1, 6, 7],  # Output shape: (2, 3)
+//                                 [5, 2, 7]]
+//
+// # A superdiagonal from each batch.
+// tf.matrix_diag_part(input, k = 1)
+//   ==> [[2, 7, 6],  # Output shape: (2, 3)
+//        [4, 3, 8]]
+//
+// # A band from each batch.
+// tf.matrix_diag_part(input, k = (-1, 2))
+//   ==> [[[0, 3, 8],  # Output shape: (2, 4, 3)
+//         [2, 7, 6],
+//         [1, 6, 7],
+//         [5, 8, 0]],
+//        [[0, 3, 4],
+//         [4, 3, 8],
+//         [5, 2, 7],
+//         [1, 6, 0]]]
+//
+// # LEFT_RIGHT alignment.
+// tf.matrix_diag_part(input, k = (-1, 2), align="LEFT_RIGHT")
+//   ==> [[[3, 8, 0],  # Output shape: (2, 4, 3)
+//         [2, 7, 6],
+//         [1, 6, 7],
+//         [0, 5, 8]],
+//        [[3, 4, 0],
+//         [4, 3, 8],
+//         [5, 2, 7],
+//         [0, 1, 6]]]
+//
+// # max_diag_len can be shorter than the main diagonal.
+// tf.matrix_diag_part(input, k = (-2, -1))
+//   ==> [[[5, 8],
+//         [9, 0]],
+//        [[1, 6],
+//         [5, 0]]]
+//
+// # padding_value = 9
+// tf.matrix_diag_part(input, k = (1, 3), padding_value = 9)
+//   ==> [[[9, 9, 4],  # Output shape: (2, 3, 3)
+//         [9, 3, 8],
+//         [2, 7, 6]],
+//        [[9, 9, 2],
+//         [9, 3, 4],
+//         [4, 3, 8]]]
+//
+// ```
+//
+// Arguments:
+//	input: Rank `r` tensor where `r >= 2`.
+//	k: Diagonal offset(s). Positive value means superdiagonal, 0 refers to the main
+// diagonal, and negative value means subdiagonals. `k` can be a single integer
+// (for a single diagonal) or a pair of integers specifying the low and high ends
+// of a matrix band. `k[0]` must not be larger than `k[1]`.
+//	padding_value: The value to fill the area outside the specified diagonal band with.
+// Default is 0.
+//
+// Returns The extracted diagonal(s).
+func MatrixDiagPartV3(scope *Scope, input tf.Output, k tf.Output, padding_value tf.Output, optional ...MatrixDiagPartV3Attr) (diagonal tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MatrixDiagPartV3",
+		Input: []tf.Input{
+			input, k, padding_value,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// LearnedUnigramCandidateSamplerAttr is an optional argument to LearnedUnigramCandidateSampler.
+type LearnedUnigramCandidateSamplerAttr func(optionalAttr)
+
+// LearnedUnigramCandidateSamplerSeed sets the optional seed attribute to value.
+//
+// value: If either seed or seed2 are set to be non-zero, the random number
+// generator is seeded by the given seed.  Otherwise, it is seeded by a
+// random seed.
+// If not specified, defaults to 0
+func LearnedUnigramCandidateSamplerSeed(value int64) LearnedUnigramCandidateSamplerAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// LearnedUnigramCandidateSamplerSeed2 sets the optional seed2 attribute to value.
+//
+// value: An second seed to avoid seed collision.
+// If not specified, defaults to 0
+func LearnedUnigramCandidateSamplerSeed2(value int64) LearnedUnigramCandidateSamplerAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Generates labels for candidate sampling with a learned unigram distribution.
+//
+// See explanations of candidate sampling and the data formats at
+// go/candidate-sampling.
+//
+// For each batch, this op picks a single set of sampled candidate labels.
+//
+// The advantages of sampling candidates per-batch are simplicity and the
+// possibility of efficient dense matrix multiplication. The disadvantage is that
+// the sampled candidates must be chosen independently of the context and of the
+// true labels.
+//
+// Arguments:
+//	true_classes: A batch_size * num_true matrix, in which each row contains the
+// IDs of the num_true target_classes in the corresponding original label.
+//	num_true: Number of true labels per context.
+//	num_sampled: Number of candidates to randomly sample.
+//	unique: If unique is true, we sample with rejection, so that all sampled
+// candidates in a batch are unique. This requires some approximation to
+// estimate the post-rejection sampling probabilities.
+//	range_max: The sampler will sample integers from the interval [0, range_max).
+//
+// Returns:
+//	sampled_candidates: A vector of length num_sampled, in which each element is
+// the ID of a sampled candidate.
+//	true_expected_count: A batch_size * num_true matrix, representing
+// the number of times each candidate is expected to occur in a batch
+// of sampled candidates. If unique=true, then this is a probability.
+//	sampled_expected_count: A vector of length num_sampled, for each sampled
+// candidate representing the number of times the candidate is expected
+// to occur in a batch of sampled candidates.  If unique=true, then this is a
+// probability.
+func LearnedUnigramCandidateSampler(scope *Scope, true_classes tf.Output, num_true int64, num_sampled int64, unique bool, range_max int64, optional ...LearnedUnigramCandidateSamplerAttr) (sampled_candidates tf.Output, true_expected_count tf.Output, sampled_expected_count tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_true": num_true, "num_sampled": num_sampled, "unique": unique, "range_max": range_max}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LearnedUnigramCandidateSampler",
+		Input: []tf.Input{
+			true_classes,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// LogUniformCandidateSamplerAttr is an optional argument to LogUniformCandidateSampler.
+type LogUniformCandidateSamplerAttr func(optionalAttr)
+
+// LogUniformCandidateSamplerSeed sets the optional seed attribute to value.
+//
+// value: If either seed or seed2 are set to be non-zero, the random number
+// generator is seeded by the given seed.  Otherwise, it is seeded by a
+// random seed.
+// If not specified, defaults to 0
+func LogUniformCandidateSamplerSeed(value int64) LogUniformCandidateSamplerAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// LogUniformCandidateSamplerSeed2 sets the optional seed2 attribute to value.
+//
+// value: An second seed to avoid seed collision.
+// If not specified, defaults to 0
+func LogUniformCandidateSamplerSeed2(value int64) LogUniformCandidateSamplerAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Generates labels for candidate sampling with a log-uniform distribution.
+//
+// See explanations of candidate sampling and the data formats at
+// go/candidate-sampling.
+//
+// For each batch, this op picks a single set of sampled candidate labels.
+//
+// The advantages of sampling candidates per-batch are simplicity and the
+// possibility of efficient dense matrix multiplication. The disadvantage is that
+// the sampled candidates must be chosen independently of the context and of the
+// true labels.
+//
+// Arguments:
+//	true_classes: A batch_size * num_true matrix, in which each row contains the
+// IDs of the num_true target_classes in the corresponding original label.
+//	num_true: Number of true labels per context.
+//	num_sampled: Number of candidates to randomly sample.
+//	unique: If unique is true, we sample with rejection, so that all sampled
+// candidates in a batch are unique. This requires some approximation to
+// estimate the post-rejection sampling probabilities.
+//	range_max: The sampler will sample integers from the interval [0, range_max).
+//
+// Returns:
+//	sampled_candidates: A vector of length num_sampled, in which each element is
+// the ID of a sampled candidate.
+//	true_expected_count: A batch_size * num_true matrix, representing
+// the number of times each candidate is expected to occur in a batch
+// of sampled candidates. If unique=true, then this is a probability.
+//	sampled_expected_count: A vector of length num_sampled, for each sampled
+// candidate representing the number of times the candidate is expected
+// to occur in a batch of sampled candidates.  If unique=true, then this is a
+// probability.
+func LogUniformCandidateSampler(scope *Scope, true_classes tf.Output, num_true int64, num_sampled int64, unique bool, range_max int64, optional ...LogUniformCandidateSamplerAttr) (sampled_candidates tf.Output, true_expected_count tf.Output, sampled_expected_count tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_true": num_true, "num_sampled": num_sampled, "unique": unique, "range_max": range_max}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LogUniformCandidateSampler",
+		Input: []tf.Input{
+			true_classes,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Selects the k nearest centers for each point.
+//
+// Rows of points are assumed to be input points. Rows of centers are assumed to be
+// the list of candidate centers. For each point, the k centers that have least L2
+// distance to it are computed.
+//
+// Arguments:
+//	points: Matrix of shape (n, d). Rows are assumed to be input points.
+//	centers: Matrix of shape (m, d). Rows are assumed to be centers.
+//	k: Number of nearest centers to return for each point. If k is larger than m, then
+// only m centers are returned.
+//
+// Returns:
+//	nearest_center_indices: Matrix of shape (n, min(m, k)). Each row contains the indices of the centers
+// closest to the corresponding point, ordered by increasing distance.
+//	nearest_center_distances: Matrix of shape (n, min(m, k)). Each row contains the squared L2 distance to the
+// corresponding center in nearest_center_indices.
+func NearestNeighbors(scope *Scope, points tf.Output, centers tf.Output, k tf.Output) (nearest_center_indices tf.Output, nearest_center_distances tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "NearestNeighbors",
+		Input: []tf.Input{
+			points, centers, k,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Returns the index of a data point that should be added to the seed set.
+//
+// Entries in distances are assumed to be squared distances of candidate points to
+// the already sampled centers in the seed set. The op constructs one Markov chain
+// of the k-MC^2 algorithm and returns the index of one candidate point to be added
+// as an additional cluster center.
+//
+// Arguments:
+//	distances: Vector with squared distances to the closest previously sampled cluster center
+// for each candidate point.
+//	seed: Scalar. Seed for initializing the random number generator.
+//
+// Returns Scalar with the index of the sampled point.
+func KMC2ChainInitialization(scope *Scope, distances tf.Output, seed tf.Output) (index tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "KMC2ChainInitialization",
+		Input: []tf.Input{
+			distances, seed,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Selects num_to_sample rows of input using the KMeans++ criterion.
+//
+// Rows of points are assumed to be input points. One row is selected at random.
+// Subsequent rows are sampled with probability proportional to the squared L2
+// distance from the nearest row selected thus far till num_to_sample rows have
+// been sampled.
+//
+// Arguments:
+//	points: Matrix of shape (n, d). Rows are assumed to be input points.
+//	num_to_sample: Scalar. The number of rows to sample. This value must not be larger than n.
+//	seed: Scalar. Seed for initializing the random number generator.
+//	num_retries_per_sample: Scalar. For each row that is sampled, this parameter
+// specifies the number of additional points to draw from the current
+// distribution before selecting the best. If a negative value is specified, a
+// heuristic is used to sample O(log(num_to_sample)) additional points.
+//
+// Returns Matrix of shape (num_to_sample, d). The sampled rows.
+func KmeansPlusPlusInitialization(scope *Scope, points tf.Output, num_to_sample tf.Output, seed tf.Output, num_retries_per_sample tf.Output) (samples tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "KmeansPlusPlusInitialization",
+		Input: []tf.Input{
+			points, num_to_sample, seed, num_retries_per_sample,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// AbortAttr is an optional argument to Abort.
+type AbortAttr func(optionalAttr)
+
+// AbortErrorMsg sets the optional error_msg attribute to value.
+//
+// value: A string which is the message associated with the exception.
+// If not specified, defaults to ""
+func AbortErrorMsg(value string) AbortAttr {
+	return func(m optionalAttr) {
+		m["error_msg"] = value
+	}
+}
+
+// AbortExitWithoutError sets the optional exit_without_error attribute to value.
+// If not specified, defaults to false
+func AbortExitWithoutError(value bool) AbortAttr {
+	return func(m optionalAttr) {
+		m["exit_without_error"] = value
+	}
+}
+
+// Raise a exception to abort the process when called.
+//
+// If exit_without_error is true, the process will exit normally,
+// otherwise it will exit with a SIGABORT signal.
+//
+// Returns nothing but an exception.
+//
+// Returns the created operation.
+func Abort(scope *Scope, optional ...AbortAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Abort",
+
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Makes its input available to the next iteration.
+//
+// Arguments:
+//	data: The tensor to be made available to the next iteration.
+//
+// Returns The same tensor as `data`.
+func NextIteration(scope *Scope, data tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "NextIteration",
+		Input: []tf.Input{
+			data,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Exits the current frame to its parent frame.
+//
+// Exit makes its input `data` available to the parent frame.
+//
+// Arguments:
+//	data: The tensor to be made available to the parent frame.
+//
+// Returns The same tensor as `data`.
+func Exit(scope *Scope, data tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Exit",
+		Input: []tf.Input{
+			data,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// EnterAttr is an optional argument to Enter.
+type EnterAttr func(optionalAttr)
+
+// EnterIsConstant sets the optional is_constant attribute to value.
+//
+// value: If true, the output is constant within the child frame.
+// If not specified, defaults to false
+func EnterIsConstant(value bool) EnterAttr {
+	return func(m optionalAttr) {
+		m["is_constant"] = value
+	}
+}
+
+// EnterParallelIterations sets the optional parallel_iterations attribute to value.
+//
+// value: The number of iterations allowed to run in parallel.
+// If not specified, defaults to 10
+func EnterParallelIterations(value int64) EnterAttr {
+	return func(m optionalAttr) {
+		m["parallel_iterations"] = value
+	}
+}
+
+// Creates or finds a child frame, and makes `data` available to the child frame.
+//
+// This op is used together with `Exit` to create loops in the graph.
+// The unique `frame_name` is used by the `Executor` to identify frames. If
+// `is_constant` is true, `output` is a constant in the child frame; otherwise
+// it may be changed in the child frame. At most `parallel_iterations` iterations
+// are run in parallel in the child frame.
+//
+// Arguments:
+//	data: The tensor to be made available to the child frame.
+//	frame_name: The name of the child frame.
+//
+// Returns The same tensor as `data`.
+func Enter(scope *Scope, data tf.Output, frame_name string, optional ...EnterAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"frame_name": frame_name}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Enter",
+		Input: []tf.Input{
+			data,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// DenseCountSparseOutputAttr is an optional argument to DenseCountSparseOutput.
+type DenseCountSparseOutputAttr func(optionalAttr)
+
+// DenseCountSparseOutputMinlength sets the optional minlength attribute to value.
+//
+// value: Minimum value to count. Can be set to -1 for no minimum.
+// If not specified, defaults to -1
+//
+// REQUIRES: value >= -1
+func DenseCountSparseOutputMinlength(value int64) DenseCountSparseOutputAttr {
+	return func(m optionalAttr) {
+		m["minlength"] = value
+	}
+}
+
+// DenseCountSparseOutputMaxlength sets the optional maxlength attribute to value.
+//
+// value: Maximum value to count. Can be set to -1 for no maximum.
+// If not specified, defaults to -1
+//
+// REQUIRES: value >= -1
+func DenseCountSparseOutputMaxlength(value int64) DenseCountSparseOutputAttr {
+	return func(m optionalAttr) {
+		m["maxlength"] = value
+	}
+}
+
+// Performs sparse-output bin counting for a tf.tensor input.
+//
+//   Counts the number of times each value occurs in the input.
+//
+// Arguments:
+//	values: Tensor containing data to count.
+//	weights: A Tensor of the same shape as indices containing per-index weight values. May
+// also be the empty tensor if no weights are used.
+//	binary_output: Whether to output the number of occurrences of each value or 1.
+//
+// Returns:
+//	output_indices: Indices tensor for the resulting sparse tensor object.
+//	output_values: Values tensor for the resulting sparse tensor object.
+//	output_dense_shape: Shape tensor for the resulting sparse tensor object.
+func DenseCountSparseOutput(scope *Scope, values tf.Output, weights tf.Output, binary_output bool, optional ...DenseCountSparseOutputAttr) (output_indices tf.Output, output_values tf.Output, output_dense_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"binary_output": binary_output}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DenseCountSparseOutput",
+		Input: []tf.Input{
+			values, weights,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// CTCBeamSearchDecoderAttr is an optional argument to CTCBeamSearchDecoder.
+type CTCBeamSearchDecoderAttr func(optionalAttr)
+
+// CTCBeamSearchDecoderMergeRepeated sets the optional merge_repeated attribute to value.
+//
+// value: If true, merge repeated classes in output.
+// If not specified, defaults to true
+func CTCBeamSearchDecoderMergeRepeated(value bool) CTCBeamSearchDecoderAttr {
+	return func(m optionalAttr) {
+		m["merge_repeated"] = value
+	}
+}
+
+// Performs beam search decoding on the logits given in input.
+//
+// A note about the attribute merge_repeated: For the beam search decoder,
+// this means that if consecutive entries in a beam are the same, only
+// the first of these is emitted.  That is, when the top path is "A B B B B",
+// "A B" is returned if merge_repeated = True but "A B B B B" is
+// returned if merge_repeated = False.
+//
+// Arguments:
+//	inputs: 3-D, shape: `(max_time x batch_size x num_classes)`, the logits.
+//	sequence_length: A vector containing sequence lengths, size `(batch)`.
+//	beam_width: A scalar >= 0 (beam search beam width).
+//	top_paths: A scalar >= 0, <= beam_width (controls output size).
+//
+// Returns:
+//	decoded_indices: A list (length: top_paths) of indices matrices.  Matrix j,
+// size `(total_decoded_outputs[j] x 2)`, has indices of a
+// `SparseTensor<int64, 2>`.  The rows store: [batch, time].
+//	decoded_values: A list (length: top_paths) of values vectors.  Vector j,
+// size `(length total_decoded_outputs[j])`, has the values of a
+// `SparseTensor<int64, 2>`.  The vector stores the decoded classes for beam j.
+//	decoded_shape: A list (length: top_paths) of shape vector.  Vector j,
+// size `(2)`, stores the shape of the decoded `SparseTensor[j]`.
+// Its values are: `[batch_size, max_decoded_length[j]]`.
+//	log_probability: A matrix, shaped: `(batch_size x top_paths)`.  The
+// sequence log-probabilities.
+func CTCBeamSearchDecoder(scope *Scope, inputs tf.Output, sequence_length tf.Output, beam_width int64, top_paths int64, optional ...CTCBeamSearchDecoderAttr) (decoded_indices []tf.Output, decoded_values []tf.Output, decoded_shape []tf.Output, log_probability tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"beam_width": beam_width, "top_paths": top_paths}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CTCBeamSearchDecoder",
+		Input: []tf.Input{
+			inputs, sequence_length,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if decoded_indices, idx, err = makeOutputList(op, idx, "decoded_indices"); err != nil {
+		scope.UpdateErr("CTCBeamSearchDecoder", err)
+		return
+	}
+	if decoded_values, idx, err = makeOutputList(op, idx, "decoded_values"); err != nil {
+		scope.UpdateErr("CTCBeamSearchDecoder", err)
+		return
+	}
+	if decoded_shape, idx, err = makeOutputList(op, idx, "decoded_shape"); err != nil {
+		scope.UpdateErr("CTCBeamSearchDecoder", err)
+		return
+	}
+	log_probability = op.Output(idx)
+	return decoded_indices, decoded_values, decoded_shape, log_probability
+}
+
+// CTCGreedyDecoderAttr is an optional argument to CTCGreedyDecoder.
+type CTCGreedyDecoderAttr func(optionalAttr)
+
+// CTCGreedyDecoderMergeRepeated sets the optional merge_repeated attribute to value.
+//
+// value: If True, merge repeated classes in output.
+// If not specified, defaults to false
+func CTCGreedyDecoderMergeRepeated(value bool) CTCGreedyDecoderAttr {
+	return func(m optionalAttr) {
+		m["merge_repeated"] = value
+	}
+}
+
+// Performs greedy decoding on the logits given in inputs.
+//
+// A note about the attribute merge_repeated: if enabled, when
+// consecutive logits' maximum indices are the same, only the first of
+// these is emitted.  Labeling the blank '*', the sequence "A B B * B B"
+// becomes "A B B" if merge_repeated = True and "A B B B B" if
+// merge_repeated = False.
+//
+// Regardless of the value of merge_repeated, if the maximum index of a given
+// time and batch corresponds to the blank, index `(num_classes - 1)`, no new
+// element is emitted.
+//
+// Arguments:
+//	inputs: 3-D, shape: `(max_time x batch_size x num_classes)`, the logits.
+//	sequence_length: A vector containing sequence lengths, size `(batch_size)`.
+//
+// Returns:
+//	decoded_indices: Indices matrix, size `(total_decoded_outputs x 2)`,
+// of a `SparseTensor<int64, 2>`.  The rows store: [batch, time].
+//	decoded_values: Values vector, size: `(total_decoded_outputs)`,
+// of a `SparseTensor<int64, 2>`.  The vector stores the decoded classes.
+//	decoded_shape: Shape vector, size `(2)`, of the decoded SparseTensor.
+// Values are: `[batch_size, max_decoded_length]`.
+//	log_probability: Matrix, size `(batch_size x 1)`, containing sequence
+// log-probabilities.
+func CTCGreedyDecoder(scope *Scope, inputs tf.Output, sequence_length tf.Output, optional ...CTCGreedyDecoderAttr) (decoded_indices tf.Output, decoded_values tf.Output, decoded_shape tf.Output, log_probability tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CTCGreedyDecoder",
+		Input: []tf.Input{
+			inputs, sequence_length,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3)
+}
+
+// CTCLossAttr is an optional argument to CTCLoss.
+type CTCLossAttr func(optionalAttr)
+
+// CTCLossPreprocessCollapseRepeated sets the optional preprocess_collapse_repeated attribute to value.
+//
+// value: Scalar, if true then repeated labels are
+// collapsed prior to the CTC calculation.
+// If not specified, defaults to false
+func CTCLossPreprocessCollapseRepeated(value bool) CTCLossAttr {
+	return func(m optionalAttr) {
+		m["preprocess_collapse_repeated"] = value
+	}
+}
+
+// CTCLossCtcMergeRepeated sets the optional ctc_merge_repeated attribute to value.
+//
+// value: Scalar.  If set to false, *during* CTC calculation
+// repeated non-blank labels will not be merged and are interpreted as
+// individual labels.  This is a simplified version of CTC.
+// If not specified, defaults to true
+func CTCLossCtcMergeRepeated(value bool) CTCLossAttr {
+	return func(m optionalAttr) {
+		m["ctc_merge_repeated"] = value
+	}
+}
+
+// CTCLossIgnoreLongerOutputsThanInputs sets the optional ignore_longer_outputs_than_inputs attribute to value.
+//
+// value: Scalar. If set to true, during CTC
+// calculation, items that have longer output sequences than input sequences
+// are skipped: they don't contribute to the loss term and have zero-gradient.
+// If not specified, defaults to false
+func CTCLossIgnoreLongerOutputsThanInputs(value bool) CTCLossAttr {
+	return func(m optionalAttr) {
+		m["ignore_longer_outputs_than_inputs"] = value
+	}
+}
+
+// Calculates the CTC Loss (log probability) for each batch entry.  Also calculates
+//
+// the gradient.  This class performs the softmax operation for you, so inputs
+// should be e.g. linear projections of outputs by an LSTM.
+//
+// Arguments:
+//	inputs: 3-D, shape: `(max_time x batch_size x num_classes)`, the logits.
+//	labels_indices: The indices of a `SparseTensor<int32, 2>`.
+// `labels_indices(i, :) == [b, t]` means `labels_values(i)` stores the id for
+// `(batch b, time t)`.
+//	labels_values: The values (labels) associated with the given batch and time.
+//	sequence_length: A vector containing sequence lengths (batch).
+//
+// Returns:
+//	loss: A vector (batch) containing log-probabilities.
+//	gradient: The gradient of `loss`.  3-D, shape:
+// `(max_time x batch_size x num_classes)`.
+func CTCLoss(scope *Scope, inputs tf.Output, labels_indices tf.Output, labels_values tf.Output, sequence_length tf.Output, optional ...CTCLossAttr) (loss tf.Output, gradient tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CTCLoss",
+		Input: []tf.Input{
+			inputs, labels_indices, labels_values, sequence_length,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// CudnnRNNCanonicalToParamsAttr is an optional argument to CudnnRNNCanonicalToParams.
+type CudnnRNNCanonicalToParamsAttr func(optionalAttr)
+
+// CudnnRNNCanonicalToParamsRnnMode sets the optional rnn_mode attribute to value.
+// If not specified, defaults to "lstm"
+func CudnnRNNCanonicalToParamsRnnMode(value string) CudnnRNNCanonicalToParamsAttr {
+	return func(m optionalAttr) {
+		m["rnn_mode"] = value
+	}
+}
+
+// CudnnRNNCanonicalToParamsInputMode sets the optional input_mode attribute to value.
+// If not specified, defaults to "linear_input"
+func CudnnRNNCanonicalToParamsInputMode(value string) CudnnRNNCanonicalToParamsAttr {
+	return func(m optionalAttr) {
+		m["input_mode"] = value
+	}
+}
+
+// CudnnRNNCanonicalToParamsDirection sets the optional direction attribute to value.
+// If not specified, defaults to "unidirectional"
+func CudnnRNNCanonicalToParamsDirection(value string) CudnnRNNCanonicalToParamsAttr {
+	return func(m optionalAttr) {
+		m["direction"] = value
+	}
+}
+
+// CudnnRNNCanonicalToParamsDropout sets the optional dropout attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNCanonicalToParamsDropout(value float32) CudnnRNNCanonicalToParamsAttr {
+	return func(m optionalAttr) {
+		m["dropout"] = value
+	}
+}
+
+// CudnnRNNCanonicalToParamsSeed sets the optional seed attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNCanonicalToParamsSeed(value int64) CudnnRNNCanonicalToParamsAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// CudnnRNNCanonicalToParamsSeed2 sets the optional seed2 attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNCanonicalToParamsSeed2(value int64) CudnnRNNCanonicalToParamsAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Converts CudnnRNN params from canonical form to usable form.
+//
+// Writes a set of weights into the opaque params buffer so they can be used in
+// upcoming training or inferences.
+//
+// Note that the params buffer may not be compatible across different GPUs. So any
+// save and restoration should be converted to and from the canonical weights and
+// biases.
+//
+// num_layers: Specifies the number of layers in the RNN model.
+// num_units: Specifies the size of the hidden state.
+// input_size: Specifies the size of the input state.
+// weights: the canonical form of weights that can be used for saving
+//     and restoration. They are more likely to be compatible across different
+//     generations.
+// biases: the canonical form of biases that can be used for saving
+//     and restoration. They are more likely to be compatible across different
+//     generations.
+// num_params: number of parameter sets for all layers.
+//     Each layer may contain multiple parameter sets, with each set consisting of
+//     a weight matrix and a bias vector.
+// rnn_mode: Indicates the type of the RNN model.
+// input_mode: Indicate whether there is a linear projection between the input and
+//     The actual computation before the first layer. 'skip_input' is only allowed
+//     when input_size == num_units; 'auto_select' implies 'skip_input' when
+//     input_size == num_units; otherwise, it implies 'linear_input'.
+// direction: Indicates whether a bidirectional model will be used.
+//     dir = (direction == bidirectional) ? 2 : 1
+// dropout: dropout probability. When set to 0., dropout is disabled.
+// seed: the 1st part of a seed to initialize dropout.
+// seed2: the 2nd part of a seed to initialize dropout.
+func CudnnRNNCanonicalToParams(scope *Scope, num_layers tf.Output, num_units tf.Output, input_size tf.Output, weights []tf.Output, biases []tf.Output, optional ...CudnnRNNCanonicalToParamsAttr) (params tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CudnnRNNCanonicalToParams",
+		Input: []tf.Input{
+			num_layers, num_units, input_size, tf.OutputList(weights), tf.OutputList(biases),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// CudnnRNNParamsToCanonicalV2Attr is an optional argument to CudnnRNNParamsToCanonicalV2.
+type CudnnRNNParamsToCanonicalV2Attr func(optionalAttr)
+
+// CudnnRNNParamsToCanonicalV2RnnMode sets the optional rnn_mode attribute to value.
+// If not specified, defaults to "lstm"
+func CudnnRNNParamsToCanonicalV2RnnMode(value string) CudnnRNNParamsToCanonicalV2Attr {
+	return func(m optionalAttr) {
+		m["rnn_mode"] = value
+	}
+}
+
+// CudnnRNNParamsToCanonicalV2InputMode sets the optional input_mode attribute to value.
+// If not specified, defaults to "linear_input"
+func CudnnRNNParamsToCanonicalV2InputMode(value string) CudnnRNNParamsToCanonicalV2Attr {
+	return func(m optionalAttr) {
+		m["input_mode"] = value
+	}
+}
+
+// CudnnRNNParamsToCanonicalV2Direction sets the optional direction attribute to value.
+// If not specified, defaults to "unidirectional"
+func CudnnRNNParamsToCanonicalV2Direction(value string) CudnnRNNParamsToCanonicalV2Attr {
+	return func(m optionalAttr) {
+		m["direction"] = value
+	}
+}
+
+// CudnnRNNParamsToCanonicalV2Dropout sets the optional dropout attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNParamsToCanonicalV2Dropout(value float32) CudnnRNNParamsToCanonicalV2Attr {
+	return func(m optionalAttr) {
+		m["dropout"] = value
+	}
+}
+
+// CudnnRNNParamsToCanonicalV2Seed sets the optional seed attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNParamsToCanonicalV2Seed(value int64) CudnnRNNParamsToCanonicalV2Attr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// CudnnRNNParamsToCanonicalV2Seed2 sets the optional seed2 attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNParamsToCanonicalV2Seed2(value int64) CudnnRNNParamsToCanonicalV2Attr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// CudnnRNNParamsToCanonicalV2NumProj sets the optional num_proj attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNParamsToCanonicalV2NumProj(value int64) CudnnRNNParamsToCanonicalV2Attr {
+	return func(m optionalAttr) {
+		m["num_proj"] = value
+	}
+}
+
+// Retrieves CudnnRNN params in canonical form. It supports the projection in LSTM.
+//
+// Retrieves a set of weights from the opaque params buffer that can be saved and
+// restored in a way compatible with future runs.
+//
+// Note that the params buffer may not be compatible across different GPUs. So any
+// save and restoration should be converted to and from the canonical weights and
+// biases.
+//
+// num_layers: Specifies the number of layers in the RNN model.
+// num_units: Specifies the size of the hidden state.
+// input_size: Specifies the size of the input state.
+// num_params_weights: number of weight parameter matrix for all layers.
+// num_params_biases: number of bias parameter vector for all layers.
+// weights: the canonical form of weights that can be used for saving
+//     and restoration. They are more likely to be compatible across different
+//     generations.
+// biases: the canonical form of biases that can be used for saving
+//     and restoration. They are more likely to be compatible across different
+//     generations.
+// rnn_mode: Indicates the type of the RNN model.
+// input_mode: Indicate whether there is a linear projection between the input and
+//     The actual computation before the first layer. 'skip_input' is only allowed
+//     when input_size == num_units; 'auto_select' implies 'skip_input' when
+//     input_size == num_units; otherwise, it implies 'linear_input'.
+// direction: Indicates whether a bidirectional model will be used.
+//     dir = (direction == bidirectional) ? 2 : 1
+// dropout: dropout probability. When set to 0., dropout is disabled.
+// seed: the 1st part of a seed to initialize dropout.
+// seed2: the 2nd part of a seed to initialize dropout.
+// num_proj: The output dimensionality for the projection matrices. If None or 0,
+//     no projection is performed.
+func CudnnRNNParamsToCanonicalV2(scope *Scope, num_layers tf.Output, num_units tf.Output, input_size tf.Output, params tf.Output, num_params_weights int64, num_params_biases int64, optional ...CudnnRNNParamsToCanonicalV2Attr) (weights []tf.Output, biases []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_params_weights": num_params_weights, "num_params_biases": num_params_biases}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CudnnRNNParamsToCanonicalV2",
+		Input: []tf.Input{
+			num_layers, num_units, input_size, params,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if weights, idx, err = makeOutputList(op, idx, "weights"); err != nil {
+		scope.UpdateErr("CudnnRNNParamsToCanonicalV2", err)
+		return
+	}
+	if biases, idx, err = makeOutputList(op, idx, "biases"); err != nil {
+		scope.UpdateErr("CudnnRNNParamsToCanonicalV2", err)
+		return
+	}
+	return weights, biases
+}
+
+// Returns the diagonal part of the tensor.
+//
+// This operation returns a tensor with the `diagonal` part
+// of the `input`. The `diagonal` part is computed as follows:
+//
+// Assume `input` has dimensions `[D1,..., Dk, D1,..., Dk]`, then the output is a
+// tensor of rank `k` with dimensions `[D1,..., Dk]` where:
+//
+// `diagonal[i1,..., ik] = input[i1, ..., ik, i1,..., ik]`.
+//
+// For example:
+//
+// ```
+// # 'input' is [[1, 0, 0, 0]
+//               [0, 2, 0, 0]
+//               [0, 0, 3, 0]
+//               [0, 0, 0, 4]]
+//
+// tf.diag_part(input) ==> [1, 2, 3, 4]
+// ```
+//
+// Arguments:
+//	input: Rank k tensor where k is even and not zero.
+//
+// Returns The extracted diagonal.
+func DiagPart(scope *Scope, input tf.Output) (diagonal tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "DiagPart",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// CudnnRNNParamsToCanonicalAttr is an optional argument to CudnnRNNParamsToCanonical.
+type CudnnRNNParamsToCanonicalAttr func(optionalAttr)
+
+// CudnnRNNParamsToCanonicalRnnMode sets the optional rnn_mode attribute to value.
+// If not specified, defaults to "lstm"
+func CudnnRNNParamsToCanonicalRnnMode(value string) CudnnRNNParamsToCanonicalAttr {
+	return func(m optionalAttr) {
+		m["rnn_mode"] = value
+	}
+}
+
+// CudnnRNNParamsToCanonicalInputMode sets the optional input_mode attribute to value.
+// If not specified, defaults to "linear_input"
+func CudnnRNNParamsToCanonicalInputMode(value string) CudnnRNNParamsToCanonicalAttr {
+	return func(m optionalAttr) {
+		m["input_mode"] = value
+	}
+}
+
+// CudnnRNNParamsToCanonicalDirection sets the optional direction attribute to value.
+// If not specified, defaults to "unidirectional"
+func CudnnRNNParamsToCanonicalDirection(value string) CudnnRNNParamsToCanonicalAttr {
+	return func(m optionalAttr) {
+		m["direction"] = value
+	}
+}
+
+// CudnnRNNParamsToCanonicalDropout sets the optional dropout attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNParamsToCanonicalDropout(value float32) CudnnRNNParamsToCanonicalAttr {
+	return func(m optionalAttr) {
+		m["dropout"] = value
+	}
+}
+
+// CudnnRNNParamsToCanonicalSeed sets the optional seed attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNParamsToCanonicalSeed(value int64) CudnnRNNParamsToCanonicalAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// CudnnRNNParamsToCanonicalSeed2 sets the optional seed2 attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNParamsToCanonicalSeed2(value int64) CudnnRNNParamsToCanonicalAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Retrieves CudnnRNN params in canonical form.
+//
+// Retrieves a set of weights from the opaque params buffer that can be saved and
+// restored in a way compatible with future runs.
+//
+// Note that the params buffer may not be compatible across different GPUs. So any
+// save and restoration should be converted to and from the canonical weights and
+// biases.
+//
+// num_layers: Specifies the number of layers in the RNN model.
+// num_units: Specifies the size of the hidden state.
+// input_size: Specifies the size of the input state.
+// num_params: number of parameter sets for all layers.
+//     Each layer may contain multiple parameter sets, with each set consisting of
+//     a weight matrix and a bias vector.
+// weights: the canonical form of weights that can be used for saving
+//     and restoration. They are more likely to be compatible across different
+//     generations.
+// biases: the canonical form of biases that can be used for saving
+//     and restoration. They are more likely to be compatible across different
+//     generations.
+// rnn_mode: Indicates the type of the RNN model.
+// input_mode: Indicate whether there is a linear projection between the input and
+//     The actual computation before the first layer. 'skip_input' is only allowed
+//     when input_size == num_units; 'auto_select' implies 'skip_input' when
+//     input_size == num_units; otherwise, it implies 'linear_input'.
+// direction: Indicates whether a bidirectional model will be used.
+//     dir = (direction == bidirectional) ? 2 : 1
+// dropout: dropout probability. When set to 0., dropout is disabled.
+// seed: the 1st part of a seed to initialize dropout.
+// seed2: the 2nd part of a seed to initialize dropout.
+func CudnnRNNParamsToCanonical(scope *Scope, num_layers tf.Output, num_units tf.Output, input_size tf.Output, params tf.Output, num_params int64, optional ...CudnnRNNParamsToCanonicalAttr) (weights []tf.Output, biases []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_params": num_params}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CudnnRNNParamsToCanonical",
+		Input: []tf.Input{
+			num_layers, num_units, input_size, params,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if weights, idx, err = makeOutputList(op, idx, "weights"); err != nil {
+		scope.UpdateErr("CudnnRNNParamsToCanonical", err)
+		return
+	}
+	if biases, idx, err = makeOutputList(op, idx, "biases"); err != nil {
+		scope.UpdateErr("CudnnRNNParamsToCanonical", err)
+		return
+	}
+	return weights, biases
+}
+
+// CudnnRNNBackpropV3Attr is an optional argument to CudnnRNNBackpropV3.
+type CudnnRNNBackpropV3Attr func(optionalAttr)
+
+// CudnnRNNBackpropV3RnnMode sets the optional rnn_mode attribute to value.
+// If not specified, defaults to "lstm"
+func CudnnRNNBackpropV3RnnMode(value string) CudnnRNNBackpropV3Attr {
+	return func(m optionalAttr) {
+		m["rnn_mode"] = value
+	}
+}
+
+// CudnnRNNBackpropV3InputMode sets the optional input_mode attribute to value.
+// If not specified, defaults to "linear_input"
+func CudnnRNNBackpropV3InputMode(value string) CudnnRNNBackpropV3Attr {
+	return func(m optionalAttr) {
+		m["input_mode"] = value
+	}
+}
+
+// CudnnRNNBackpropV3Direction sets the optional direction attribute to value.
+// If not specified, defaults to "unidirectional"
+func CudnnRNNBackpropV3Direction(value string) CudnnRNNBackpropV3Attr {
+	return func(m optionalAttr) {
+		m["direction"] = value
+	}
+}
+
+// CudnnRNNBackpropV3Dropout sets the optional dropout attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNBackpropV3Dropout(value float32) CudnnRNNBackpropV3Attr {
+	return func(m optionalAttr) {
+		m["dropout"] = value
+	}
+}
+
+// CudnnRNNBackpropV3Seed sets the optional seed attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNBackpropV3Seed(value int64) CudnnRNNBackpropV3Attr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// CudnnRNNBackpropV3Seed2 sets the optional seed2 attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNBackpropV3Seed2(value int64) CudnnRNNBackpropV3Attr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// CudnnRNNBackpropV3NumProj sets the optional num_proj attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNBackpropV3NumProj(value int64) CudnnRNNBackpropV3Attr {
+	return func(m optionalAttr) {
+		m["num_proj"] = value
+	}
+}
+
+// CudnnRNNBackpropV3TimeMajor sets the optional time_major attribute to value.
+// If not specified, defaults to true
+func CudnnRNNBackpropV3TimeMajor(value bool) CudnnRNNBackpropV3Attr {
+	return func(m optionalAttr) {
+		m["time_major"] = value
+	}
+}
+
+// Backprop step of CudnnRNNV3.
+//
+// Compute the backprop of both data and weights in a RNN. Takes an extra
+//     "sequence_lengths" input than CudnnRNNBackprop.
+//
+// rnn_mode: Indicates the type of the RNN model.
+// input_mode: Indicates whether there is a linear projection between the input and
+//     the actual computation before the first layer. 'skip_input' is only allowed
+//     when input_size == num_units; 'auto_select' implies 'skip_input' when
+//     input_size == num_units; otherwise, it implies 'linear_input'.
+// direction: Indicates whether a bidirectional model will be used. Should be
+//   "unidirectional" or "bidirectional".
+// dropout: Dropout probability. When set to 0., dropout is disabled.
+// seed: The 1st part of a seed to initialize dropout.
+// seed2: The 2nd part of a seed to initialize dropout.
+// input: If time_major is true, this is a 3-D tensor with the shape of
+//     [seq_length, batch_size, input_size]. If time_major is false, the shape is
+//     [batch_size, seq_length, input_size].
+// input_h: If time_major is true, this is a 3-D tensor with the shape of
+//     [num_layer * dir, batch_size, num_units]. If time_major is false, the shape
+//     is [batch_size, num_layer * dir, num_units].
+// input_c: For LSTM, a 3-D tensor with the shape of
+//     [num_layer * dir, batch, num_units]. For other models, it is ignored.
+// params: A 1-D tensor that contains the weights and biases in an opaque layout.
+//     The size must be created through CudnnRNNParamsSize, and initialized
+//     separately. Note that they might not be compatible across different
+//     generations. So it is a good idea to save and restore
+// sequence_lengths: a vector of lengths of each input sequence.
+// output: If time_major is true, this is a 3-D tensor with the shape of
+//     [seq_length, batch_size, dir * num_units]. If time_major is false, the
+//     shape is [batch_size, seq_length, dir * num_units].
+// output_h: The same shape has input_h.
+// output_c: The same shape as input_c for LSTM. An empty tensor for other models.
+// output_backprop: A 3-D tensor with the same shape as output in the forward pass.
+// output_h_backprop: A 3-D tensor with the same shape as output_h in the forward
+//     pass.
+// output_c_backprop: A 3-D tensor with the same shape as output_c in the forward
+//     pass.
+// time_major: Indicates whether the input/output format is time major or batch
+//     major.
+// reserve_space: The same reserve_space produced in the forward operation.
+// input_backprop: The backprop to input in the forward pass. Has the same shape
+//     as input.
+// input_h_backprop: The backprop to input_h in the forward pass. Has the same
+//     shape as input_h.
+// input_c_backprop: The backprop to input_c in the forward pass. Has the same
+//     shape as input_c.
+// params_backprop: The backprop to the params buffer in the forward pass. Has the
+//     same shape as params.
+func CudnnRNNBackpropV3(scope *Scope, input tf.Output, input_h tf.Output, input_c tf.Output, params tf.Output, sequence_lengths tf.Output, output tf.Output, output_h tf.Output, output_c tf.Output, output_backprop tf.Output, output_h_backprop tf.Output, output_c_backprop tf.Output, reserve_space tf.Output, host_reserved tf.Output, optional ...CudnnRNNBackpropV3Attr) (input_backprop tf.Output, input_h_backprop tf.Output, input_c_backprop tf.Output, params_backprop tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CudnnRNNBackpropV3",
+		Input: []tf.Input{
+			input, input_h, input_c, params, sequence_lengths, output, output_h, output_c, output_backprop, output_h_backprop, output_c_backprop, reserve_space, host_reserved,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3)
+}
+
+// CudnnRNNBackpropAttr is an optional argument to CudnnRNNBackprop.
+type CudnnRNNBackpropAttr func(optionalAttr)
+
+// CudnnRNNBackpropRnnMode sets the optional rnn_mode attribute to value.
+// If not specified, defaults to "lstm"
+func CudnnRNNBackpropRnnMode(value string) CudnnRNNBackpropAttr {
+	return func(m optionalAttr) {
+		m["rnn_mode"] = value
+	}
+}
+
+// CudnnRNNBackpropInputMode sets the optional input_mode attribute to value.
+// If not specified, defaults to "linear_input"
+func CudnnRNNBackpropInputMode(value string) CudnnRNNBackpropAttr {
+	return func(m optionalAttr) {
+		m["input_mode"] = value
+	}
+}
+
+// CudnnRNNBackpropDirection sets the optional direction attribute to value.
+// If not specified, defaults to "unidirectional"
+func CudnnRNNBackpropDirection(value string) CudnnRNNBackpropAttr {
+	return func(m optionalAttr) {
+		m["direction"] = value
+	}
+}
+
+// CudnnRNNBackpropDropout sets the optional dropout attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNBackpropDropout(value float32) CudnnRNNBackpropAttr {
+	return func(m optionalAttr) {
+		m["dropout"] = value
+	}
+}
+
+// CudnnRNNBackpropSeed sets the optional seed attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNBackpropSeed(value int64) CudnnRNNBackpropAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// CudnnRNNBackpropSeed2 sets the optional seed2 attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNBackpropSeed2(value int64) CudnnRNNBackpropAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Backprop step of CudnnRNN.
+//
+// Compute the backprop of both data and weights in a RNN.
+//
+// rnn_mode: Indicates the type of the RNN model.
+// input_mode: Indicate whether there is a linear projection between the input and
+//     the actual computation before the first layer. 'skip_input' is only allowed
+//     when input_size == num_units; 'auto_select' implies 'skip_input' when
+//     input_size == num_units; otherwise, it implies 'linear_input'.
+// direction: Indicates whether a bidirectional model will be used. Should be
+//   "unidirectional" or "bidirectional".
+// dropout: Dropout probability. When set to 0., dropout is disabled.
+// seed: The 1st part of a seed to initialize dropout.
+// seed2: The 2nd part of a seed to initialize dropout.
+// input: A 3-D tensor with the shape of [seq_length, batch_size, input_size].
+// input_h: A 3-D tensor with the shape of [num_layer * dir, batch_size,
+//     num_units].
+// input_c: For LSTM, a 3-D tensor with the shape of
+//     [num_layer * dir, batch, num_units]. For other models, it is ignored.
+// params: A 1-D tensor that contains the weights and biases in an opaque layout.
+//     The size must be created through CudnnRNNParamsSize, and initialized
+//     separately. Note that they might not be compatible across different
+//     generations. So it is a good idea to save and restore
+// output: A 3-D tensor with the shape of [seq_length, batch_size,
+//     dir * num_units].
+// output_h: The same shape has input_h.
+// output_c: The same shape as input_c for LSTM. An empty tensor for other models.
+// output_backprop: A 3-D tensor with the same shape as output in the forward pass.
+// output_h_backprop: A 3-D tensor with the same shape as output_h in the forward
+//     pass.
+// output_c_backprop: A 3-D tensor with the same shape as output_c in the forward
+//     pass.
+// reserve_space: The same reserve_space produced in for forward operation.
+// input_backprop: The backprop to input in the forward pass. Has the same shape
+//     as input.
+// input_h_backprop: The backprop to input_h in the forward pass. Has the same
+//     shape as input_h.
+// input_c_backprop: The backprop to input_c in the forward pass. Has the same
+//     shape as input_c.
+// params_backprop: The backprop to the params buffer in the forward pass. Has the
+//     same shape as params.
+func CudnnRNNBackprop(scope *Scope, input tf.Output, input_h tf.Output, input_c tf.Output, params tf.Output, output tf.Output, output_h tf.Output, output_c tf.Output, output_backprop tf.Output, output_h_backprop tf.Output, output_c_backprop tf.Output, reserve_space tf.Output, optional ...CudnnRNNBackpropAttr) (input_backprop tf.Output, input_h_backprop tf.Output, input_c_backprop tf.Output, params_backprop tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CudnnRNNBackprop",
+		Input: []tf.Input{
+			input, input_h, input_c, params, output, output_h, output_c, output_backprop, output_h_backprop, output_c_backprop, reserve_space,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3)
+}
+
+// CudnnRNNV3Attr is an optional argument to CudnnRNNV3.
+type CudnnRNNV3Attr func(optionalAttr)
+
+// CudnnRNNV3RnnMode sets the optional rnn_mode attribute to value.
+// If not specified, defaults to "lstm"
+func CudnnRNNV3RnnMode(value string) CudnnRNNV3Attr {
+	return func(m optionalAttr) {
+		m["rnn_mode"] = value
+	}
+}
+
+// CudnnRNNV3InputMode sets the optional input_mode attribute to value.
+// If not specified, defaults to "linear_input"
+func CudnnRNNV3InputMode(value string) CudnnRNNV3Attr {
+	return func(m optionalAttr) {
+		m["input_mode"] = value
+	}
+}
+
+// CudnnRNNV3Direction sets the optional direction attribute to value.
+// If not specified, defaults to "unidirectional"
+func CudnnRNNV3Direction(value string) CudnnRNNV3Attr {
+	return func(m optionalAttr) {
+		m["direction"] = value
+	}
+}
+
+// CudnnRNNV3Dropout sets the optional dropout attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNV3Dropout(value float32) CudnnRNNV3Attr {
+	return func(m optionalAttr) {
+		m["dropout"] = value
+	}
+}
+
+// CudnnRNNV3Seed sets the optional seed attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNV3Seed(value int64) CudnnRNNV3Attr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// CudnnRNNV3Seed2 sets the optional seed2 attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNV3Seed2(value int64) CudnnRNNV3Attr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// CudnnRNNV3NumProj sets the optional num_proj attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNV3NumProj(value int64) CudnnRNNV3Attr {
+	return func(m optionalAttr) {
+		m["num_proj"] = value
+	}
+}
+
+// CudnnRNNV3IsTraining sets the optional is_training attribute to value.
+// If not specified, defaults to true
+func CudnnRNNV3IsTraining(value bool) CudnnRNNV3Attr {
+	return func(m optionalAttr) {
+		m["is_training"] = value
+	}
+}
+
+// CudnnRNNV3TimeMajor sets the optional time_major attribute to value.
+// If not specified, defaults to true
+func CudnnRNNV3TimeMajor(value bool) CudnnRNNV3Attr {
+	return func(m optionalAttr) {
+		m["time_major"] = value
+	}
+}
+
+// A RNN backed by cuDNN.
+//
+// Computes the RNN from the input and initial states, with respect to the params
+// buffer. Accepts one extra input "sequence_lengths" than CudnnRNN.
+//
+// rnn_mode: Indicates the type of the RNN model.
+// input_mode: Indicates whether there is a linear projection between the input and
+//   the actual computation before the first layer. 'skip_input' is only allowed
+//   when input_size == num_units; 'auto_select' implies 'skip_input' when
+//   input_size == num_units; otherwise, it implies 'linear_input'.
+// direction: Indicates whether a bidirectional model will be used. Should be
+//   "unidirectional" or "bidirectional".
+// dropout: Dropout probability. When set to 0., dropout is disabled.
+// seed: The 1st part of a seed to initialize dropout.
+// seed2: The 2nd part of a seed to initialize dropout.
+// input: If time_major is true, this is a 3-D tensor with the shape of
+//     [seq_length, batch_size, input_size]. If time_major is false, the shape is
+//     [batch_size, seq_length, input_size].
+// input_h: If time_major is true, this is a 3-D tensor with the shape of
+//     [num_layer * dir, batch_size, num_units]. If time_major is false, the shape
+//     is [batch_size, num_layer * dir, num_units].
+// input_c: For LSTM, a 3-D tensor with the shape of
+//     [num_layer * dir, batch, num_units]. For other models, it is ignored.
+// params: A 1-D tensor that contains the weights and biases in an opaque layout.
+//     The size must be created through CudnnRNNParamsSize, and initialized
+//     separately. Note that they might not be compatible across different
+//     generations. So it is a good idea to save and restore
+// sequence_lengths: a vector of lengths of each input sequence.
+// output: If time_major is true, this is a 3-D tensor with the shape of
+//     [seq_length, batch_size, dir * num_units]. If time_major is false, the
+//     shape is [batch_size, seq_length, dir * num_units].
+// output_h: The same shape has input_h.
+// output_c: The same shape as input_c for LSTM. An empty tensor for other models.
+// is_training: Indicates whether this operation is used for inference or
+//   training.
+// time_major: Indicates whether the input/output format is time major or batch
+//     major.
+// reserve_space: An opaque tensor that can be used in backprop calculation. It
+//   is only produced if is_training is true.
+func CudnnRNNV3(scope *Scope, input tf.Output, input_h tf.Output, input_c tf.Output, params tf.Output, sequence_lengths tf.Output, optional ...CudnnRNNV3Attr) (output tf.Output, output_h tf.Output, output_c tf.Output, reserve_space tf.Output, host_reserved tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CudnnRNNV3",
+		Input: []tf.Input{
+			input, input_h, input_c, params, sequence_lengths,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3), op.Output(4)
+}
+
+// Pads a tensor with zeros.
+//
+// This operation pads a `input` with zeros according to the `paddings` you
+// specify. `paddings` is an integer tensor with shape `[Dn, 2]`, where n is the
+// rank of `input`. For each dimension D of `input`, `paddings[D, 0]` indicates
+// how many zeros to add before the contents of `input` in that dimension, and
+// `paddings[D, 1]` indicates how many zeros to add after the contents of `input`
+// in that dimension.
+//
+// The padded size of each dimension D of the output is:
+//
+// `paddings(D, 0) + input.dim_size(D) + paddings(D, 1)`
+//
+// For example:
+//
+// ```
+// # 't' is [[1, 1], [2, 2]]
+// # 'paddings' is [[1, 1], [2, 2]]
+// # rank of 't' is 2
+// pad(t, paddings) ==> [[0, 0, 0, 0, 0, 0]
+//                       [0, 0, 1, 1, 0, 0]
+//                       [0, 0, 2, 2, 0, 0]
+//                       [0, 0, 0, 0, 0, 0]]
+// ```
+//
+func Pad(scope *Scope, input tf.Output, paddings tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Pad",
+		Input: []tf.Input{
+			input, paddings,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// CudnnRNNV2Attr is an optional argument to CudnnRNNV2.
+type CudnnRNNV2Attr func(optionalAttr)
+
+// CudnnRNNV2RnnMode sets the optional rnn_mode attribute to value.
+// If not specified, defaults to "lstm"
+func CudnnRNNV2RnnMode(value string) CudnnRNNV2Attr {
+	return func(m optionalAttr) {
+		m["rnn_mode"] = value
+	}
+}
+
+// CudnnRNNV2InputMode sets the optional input_mode attribute to value.
+// If not specified, defaults to "linear_input"
+func CudnnRNNV2InputMode(value string) CudnnRNNV2Attr {
+	return func(m optionalAttr) {
+		m["input_mode"] = value
+	}
+}
+
+// CudnnRNNV2Direction sets the optional direction attribute to value.
+// If not specified, defaults to "unidirectional"
+func CudnnRNNV2Direction(value string) CudnnRNNV2Attr {
+	return func(m optionalAttr) {
+		m["direction"] = value
+	}
+}
+
+// CudnnRNNV2Dropout sets the optional dropout attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNV2Dropout(value float32) CudnnRNNV2Attr {
+	return func(m optionalAttr) {
+		m["dropout"] = value
+	}
+}
+
+// CudnnRNNV2Seed sets the optional seed attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNV2Seed(value int64) CudnnRNNV2Attr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// CudnnRNNV2Seed2 sets the optional seed2 attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNV2Seed2(value int64) CudnnRNNV2Attr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// CudnnRNNV2IsTraining sets the optional is_training attribute to value.
+// If not specified, defaults to true
+func CudnnRNNV2IsTraining(value bool) CudnnRNNV2Attr {
+	return func(m optionalAttr) {
+		m["is_training"] = value
+	}
+}
+
+// A RNN backed by cuDNN.
+//
+// Computes the RNN from the input and initial states, with respect to the params
+// buffer. Produces one extra output "host_reserved" than CudnnRNN.
+//
+// rnn_mode: Indicates the type of the RNN model.
+// input_mode: Indicates whether there is a linear projection between the input and
+//   the actual computation before the first layer. 'skip_input' is only allowed
+//   when input_size == num_units; 'auto_select' implies 'skip_input' when
+//   input_size == num_units; otherwise, it implies 'linear_input'.
+// direction: Indicates whether a bidirectional model will be used. Should be
+//   "unidirectional" or "bidirectional".
+// dropout: Dropout probability. When set to 0., dropout is disabled.
+// seed: The 1st part of a seed to initialize dropout.
+// seed2: The 2nd part of a seed to initialize dropout.
+// input: A 3-D tensor with the shape of [seq_length, batch_size, input_size].
+// input_h: A 3-D tensor with the shape of [num_layer * dir, batch_size,
+//     num_units].
+// input_c: For LSTM, a 3-D tensor with the shape of
+//     [num_layer * dir, batch, num_units]. For other models, it is ignored.
+// params: A 1-D tensor that contains the weights and biases in an opaque layout.
+//     The size must be created through CudnnRNNParamsSize, and initialized
+//     separately. Note that they might not be compatible across different
+//     generations. So it is a good idea to save and restore
+// output: A 3-D tensor with the shape of [seq_length, batch_size,
+//     dir * num_units].
+// output_h: The same shape has input_h.
+// output_c: The same shape as input_c for LSTM. An empty tensor for other models.
+// is_training: Indicates whether this operation is used for inference or
+//   training.
+// reserve_space: An opaque tensor that can be used in backprop calculation. It
+//   is only produced if is_training is true.
+// host_reserved: An opaque tensor that can be used in backprop calculation. It is
+//   only produced if is_training is true. It is output on host memory rather than
+//   device memory.
+func CudnnRNNV2(scope *Scope, input tf.Output, input_h tf.Output, input_c tf.Output, params tf.Output, optional ...CudnnRNNV2Attr) (output tf.Output, output_h tf.Output, output_c tf.Output, reserve_space tf.Output, host_reserved tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CudnnRNNV2",
+		Input: []tf.Input{
+			input, input_h, input_c, params,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3), op.Output(4)
+}
+
+// CudnnRNNParamsSizeAttr is an optional argument to CudnnRNNParamsSize.
+type CudnnRNNParamsSizeAttr func(optionalAttr)
+
+// CudnnRNNParamsSizeRnnMode sets the optional rnn_mode attribute to value.
+// If not specified, defaults to "lstm"
+func CudnnRNNParamsSizeRnnMode(value string) CudnnRNNParamsSizeAttr {
+	return func(m optionalAttr) {
+		m["rnn_mode"] = value
+	}
+}
+
+// CudnnRNNParamsSizeInputMode sets the optional input_mode attribute to value.
+// If not specified, defaults to "linear_input"
+func CudnnRNNParamsSizeInputMode(value string) CudnnRNNParamsSizeAttr {
+	return func(m optionalAttr) {
+		m["input_mode"] = value
+	}
+}
+
+// CudnnRNNParamsSizeDirection sets the optional direction attribute to value.
+// If not specified, defaults to "unidirectional"
+func CudnnRNNParamsSizeDirection(value string) CudnnRNNParamsSizeAttr {
+	return func(m optionalAttr) {
+		m["direction"] = value
+	}
+}
+
+// CudnnRNNParamsSizeDropout sets the optional dropout attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNParamsSizeDropout(value float32) CudnnRNNParamsSizeAttr {
+	return func(m optionalAttr) {
+		m["dropout"] = value
+	}
+}
+
+// CudnnRNNParamsSizeSeed sets the optional seed attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNParamsSizeSeed(value int64) CudnnRNNParamsSizeAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// CudnnRNNParamsSizeSeed2 sets the optional seed2 attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNParamsSizeSeed2(value int64) CudnnRNNParamsSizeAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// CudnnRNNParamsSizeNumProj sets the optional num_proj attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNParamsSizeNumProj(value int64) CudnnRNNParamsSizeAttr {
+	return func(m optionalAttr) {
+		m["num_proj"] = value
+	}
+}
+
+// Computes size of weights that can be used by a Cudnn RNN model.
+//
+// Return the params size that can be used by the Cudnn RNN model. Subsequent
+// weight allocation and initialization should use this size.
+//
+// num_layers: Specifies the number of layers in the RNN model.
+// num_units: Specifies the size of the hidden state.
+// input_size: Specifies the size of the input state.
+// rnn_mode: Indicates the type of the RNN model.
+// input_mode: Indicate whether there is a linear projection between the input and
+//   The actual computation before the first layer. 'skip_input' is only allowed
+//   when input_size == num_units; 'auto_select' implies 'skip_input' when
+//   input_size == num_units; otherwise, it implies 'linear_input'.
+// direction: Indicates whether a bidirectional model will be used.
+//   dir = (direction == bidirectional) ? 2 : 1
+// dropout: dropout probability. When set to 0., dropout is disabled.
+// seed: the 1st part of a seed to initialize dropout.
+// seed2: the 2nd part of a seed to initialize dropout.
+// params_size: The size of the params buffer that should be allocated and
+//   initialized for this RNN model. Note that this params buffer may not be
+//   compatible across GPUs. Please use CudnnRNNParamsWeights and
+//   CudnnRNNParamsBiases to save and restore them in a way that is compatible
+//   across different runs.
+func CudnnRNNParamsSize(scope *Scope, num_layers tf.Output, num_units tf.Output, input_size tf.Output, T tf.DataType, S tf.DataType, optional ...CudnnRNNParamsSizeAttr) (params_size tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"T": T, "S": S}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CudnnRNNParamsSize",
+		Input: []tf.Input{
+			num_layers, num_units, input_size,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RecordInputAttr is an optional argument to RecordInput.
+type RecordInputAttr func(optionalAttr)
+
+// RecordInputFileRandomSeed sets the optional file_random_seed attribute to value.
+//
+// value: Random seeds used to produce randomized records.
+// If not specified, defaults to 301
+func RecordInputFileRandomSeed(value int64) RecordInputAttr {
+	return func(m optionalAttr) {
+		m["file_random_seed"] = value
+	}
+}
+
+// RecordInputFileShuffleShiftRatio sets the optional file_shuffle_shift_ratio attribute to value.
+//
+// value: Shifts the list of files after the list is randomly
+// shuffled.
+// If not specified, defaults to 0
+func RecordInputFileShuffleShiftRatio(value float32) RecordInputAttr {
+	return func(m optionalAttr) {
+		m["file_shuffle_shift_ratio"] = value
+	}
+}
+
+// RecordInputFileBufferSize sets the optional file_buffer_size attribute to value.
+//
+// value: The randomization shuffling buffer.
+// If not specified, defaults to 10000
+func RecordInputFileBufferSize(value int64) RecordInputAttr {
+	return func(m optionalAttr) {
+		m["file_buffer_size"] = value
+	}
+}
+
+// RecordInputFileParallelism sets the optional file_parallelism attribute to value.
+//
+// value: How many sstables are opened and concurrently iterated over.
+// If not specified, defaults to 16
+func RecordInputFileParallelism(value int64) RecordInputAttr {
+	return func(m optionalAttr) {
+		m["file_parallelism"] = value
+	}
+}
+
+// RecordInputBatchSize sets the optional batch_size attribute to value.
+//
+// value: The batch size.
+// If not specified, defaults to 32
+func RecordInputBatchSize(value int64) RecordInputAttr {
+	return func(m optionalAttr) {
+		m["batch_size"] = value
+	}
+}
+
+// RecordInputCompressionType sets the optional compression_type attribute to value.
+//
+// value: The type of compression for the file. Currently ZLIB and
+// GZIP are supported. Defaults to none.
+// If not specified, defaults to ""
+func RecordInputCompressionType(value string) RecordInputAttr {
+	return func(m optionalAttr) {
+		m["compression_type"] = value
+	}
+}
+
+// Emits randomized records.
+//
+// Arguments:
+//	file_pattern: Glob pattern for the data files.
+//
+// Returns A tensor of shape [batch_size].
+func RecordInput(scope *Scope, file_pattern string, optional ...RecordInputAttr) (records tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"file_pattern": file_pattern}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RecordInput",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// OrderedMapIncompleteSizeAttr is an optional argument to OrderedMapIncompleteSize.
+type OrderedMapIncompleteSizeAttr func(optionalAttr)
+
+// OrderedMapIncompleteSizeCapacity sets the optional capacity attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func OrderedMapIncompleteSizeCapacity(value int64) OrderedMapIncompleteSizeAttr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// OrderedMapIncompleteSizeMemoryLimit sets the optional memory_limit attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func OrderedMapIncompleteSizeMemoryLimit(value int64) OrderedMapIncompleteSizeAttr {
+	return func(m optionalAttr) {
+		m["memory_limit"] = value
+	}
+}
+
+// OrderedMapIncompleteSizeContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func OrderedMapIncompleteSizeContainer(value string) OrderedMapIncompleteSizeAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// OrderedMapIncompleteSizeSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func OrderedMapIncompleteSizeSharedName(value string) OrderedMapIncompleteSizeAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Op returns the number of incomplete elements in the underlying container.
+func OrderedMapIncompleteSize(scope *Scope, dtypes []tf.DataType, optional ...OrderedMapIncompleteSizeAttr) (size tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "OrderedMapIncompleteSize",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// OrderedMapSizeAttr is an optional argument to OrderedMapSize.
+type OrderedMapSizeAttr func(optionalAttr)
+
+// OrderedMapSizeCapacity sets the optional capacity attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func OrderedMapSizeCapacity(value int64) OrderedMapSizeAttr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// OrderedMapSizeMemoryLimit sets the optional memory_limit attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func OrderedMapSizeMemoryLimit(value int64) OrderedMapSizeAttr {
+	return func(m optionalAttr) {
+		m["memory_limit"] = value
+	}
+}
+
+// OrderedMapSizeContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func OrderedMapSizeContainer(value string) OrderedMapSizeAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// OrderedMapSizeSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func OrderedMapSizeSharedName(value string) OrderedMapSizeAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Op returns the number of elements in the underlying container.
+func OrderedMapSize(scope *Scope, dtypes []tf.DataType, optional ...OrderedMapSizeAttr) (size tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "OrderedMapSize",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// OrderedMapUnstageNoKeyAttr is an optional argument to OrderedMapUnstageNoKey.
+type OrderedMapUnstageNoKeyAttr func(optionalAttr)
+
+// OrderedMapUnstageNoKeyCapacity sets the optional capacity attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func OrderedMapUnstageNoKeyCapacity(value int64) OrderedMapUnstageNoKeyAttr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// OrderedMapUnstageNoKeyMemoryLimit sets the optional memory_limit attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func OrderedMapUnstageNoKeyMemoryLimit(value int64) OrderedMapUnstageNoKeyAttr {
+	return func(m optionalAttr) {
+		m["memory_limit"] = value
+	}
+}
+
+// OrderedMapUnstageNoKeyContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func OrderedMapUnstageNoKeyContainer(value string) OrderedMapUnstageNoKeyAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// OrderedMapUnstageNoKeySharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func OrderedMapUnstageNoKeySharedName(value string) OrderedMapUnstageNoKeyAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Op removes and returns the (key, value) element with the smallest
+//
+// key from the underlying container.   If the underlying container
+// does not contain elements, the op will block until it does.
+func OrderedMapUnstageNoKey(scope *Scope, indices tf.Output, dtypes []tf.DataType, optional ...OrderedMapUnstageNoKeyAttr) (key tf.Output, values []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "OrderedMapUnstageNoKey",
+		Input: []tf.Input{
+			indices,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	key = op.Output(idx)
+	if values, idx, err = makeOutputList(op, idx, "values"); err != nil {
+		scope.UpdateErr("OrderedMapUnstageNoKey", err)
+		return
+	}
+	return key, values
+}
+
+// OrderedMapPeekAttr is an optional argument to OrderedMapPeek.
+type OrderedMapPeekAttr func(optionalAttr)
+
+// OrderedMapPeekCapacity sets the optional capacity attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func OrderedMapPeekCapacity(value int64) OrderedMapPeekAttr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// OrderedMapPeekMemoryLimit sets the optional memory_limit attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func OrderedMapPeekMemoryLimit(value int64) OrderedMapPeekAttr {
+	return func(m optionalAttr) {
+		m["memory_limit"] = value
+	}
+}
+
+// OrderedMapPeekContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func OrderedMapPeekContainer(value string) OrderedMapPeekAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// OrderedMapPeekSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func OrderedMapPeekSharedName(value string) OrderedMapPeekAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Op peeks at the values at the specified key.  If the
+//
+// underlying container does not contain this key
+// this op will block until it does.   This Op is optimized for
+// performance.
+func OrderedMapPeek(scope *Scope, key tf.Output, indices tf.Output, dtypes []tf.DataType, optional ...OrderedMapPeekAttr) (values []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "OrderedMapPeek",
+		Input: []tf.Input{
+			key, indices,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if values, idx, err = makeOutputList(op, idx, "values"); err != nil {
+		scope.UpdateErr("OrderedMapPeek", err)
+		return
+	}
+	return values
+}
+
+// MapIncompleteSizeAttr is an optional argument to MapIncompleteSize.
+type MapIncompleteSizeAttr func(optionalAttr)
+
+// MapIncompleteSizeCapacity sets the optional capacity attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func MapIncompleteSizeCapacity(value int64) MapIncompleteSizeAttr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// MapIncompleteSizeMemoryLimit sets the optional memory_limit attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func MapIncompleteSizeMemoryLimit(value int64) MapIncompleteSizeAttr {
+	return func(m optionalAttr) {
+		m["memory_limit"] = value
+	}
+}
+
+// MapIncompleteSizeContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func MapIncompleteSizeContainer(value string) MapIncompleteSizeAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// MapIncompleteSizeSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func MapIncompleteSizeSharedName(value string) MapIncompleteSizeAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Op returns the number of incomplete elements in the underlying container.
+func MapIncompleteSize(scope *Scope, dtypes []tf.DataType, optional ...MapIncompleteSizeAttr) (size tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MapIncompleteSize",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MapSizeAttr is an optional argument to MapSize.
+type MapSizeAttr func(optionalAttr)
+
+// MapSizeCapacity sets the optional capacity attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func MapSizeCapacity(value int64) MapSizeAttr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// MapSizeMemoryLimit sets the optional memory_limit attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func MapSizeMemoryLimit(value int64) MapSizeAttr {
+	return func(m optionalAttr) {
+		m["memory_limit"] = value
+	}
+}
+
+// MapSizeContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func MapSizeContainer(value string) MapSizeAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// MapSizeSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func MapSizeSharedName(value string) MapSizeAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Op returns the number of elements in the underlying container.
+func MapSize(scope *Scope, dtypes []tf.DataType, optional ...MapSizeAttr) (size tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MapSize",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MapUnstageNoKeyAttr is an optional argument to MapUnstageNoKey.
+type MapUnstageNoKeyAttr func(optionalAttr)
+
+// MapUnstageNoKeyCapacity sets the optional capacity attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func MapUnstageNoKeyCapacity(value int64) MapUnstageNoKeyAttr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// MapUnstageNoKeyMemoryLimit sets the optional memory_limit attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func MapUnstageNoKeyMemoryLimit(value int64) MapUnstageNoKeyAttr {
+	return func(m optionalAttr) {
+		m["memory_limit"] = value
+	}
+}
+
+// MapUnstageNoKeyContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func MapUnstageNoKeyContainer(value string) MapUnstageNoKeyAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// MapUnstageNoKeySharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func MapUnstageNoKeySharedName(value string) MapUnstageNoKeyAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Op removes and returns a random (key, value)
+//
+// from the underlying container.   If the underlying container
+// does not contain elements, the op will block until it does.
+func MapUnstageNoKey(scope *Scope, indices tf.Output, dtypes []tf.DataType, optional ...MapUnstageNoKeyAttr) (key tf.Output, values []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MapUnstageNoKey",
+		Input: []tf.Input{
+			indices,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	key = op.Output(idx)
+	if values, idx, err = makeOutputList(op, idx, "values"); err != nil {
+		scope.UpdateErr("MapUnstageNoKey", err)
+		return
+	}
+	return key, values
+}
+
+// UnbatchAttr is an optional argument to Unbatch.
+type UnbatchAttr func(optionalAttr)
+
+// UnbatchContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func UnbatchContainer(value string) UnbatchAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// UnbatchSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func UnbatchSharedName(value string) UnbatchAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Reverses the operation of Batch for a single output Tensor.
+//
+// An instance of Unbatch either receives an empty batched_tensor, in which case it
+// asynchronously waits until the values become available from a concurrently
+// running instance of Unbatch with the same container and shared_name, or receives
+// a non-empty batched_tensor in which case it finalizes all other concurrently
+// running instances and outputs its own element from the batch.
+//
+// batched_tensor: The possibly transformed output of Batch. The size of the first
+//  dimension should remain unchanged by the transformations for the operation to
+//  work.
+// batch_index: The matching batch_index obtained from Batch.
+// id: The id scalar emitted by Batch.
+// unbatched_tensor: The Tensor corresponding to this execution.
+// timeout_micros: Maximum amount of time (in microseconds) to wait to receive the
+//  batched input tensor associated with a given invocation of the op.
+// container: Container to control resource sharing.
+// shared_name: Instances of Unbatch with the same container and shared_name are
+//  assumed to possibly belong to the same batch. If left empty, the op name will
+//  be used as the shared name.
+func Unbatch(scope *Scope, batched_tensor tf.Output, batch_index tf.Output, id tf.Output, timeout_micros int64, optional ...UnbatchAttr) (unbatched_tensor tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"timeout_micros": timeout_micros}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Unbatch",
+		Input: []tf.Input{
+			batched_tensor, batch_index, id,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MapUnstageAttr is an optional argument to MapUnstage.
+type MapUnstageAttr func(optionalAttr)
+
+// MapUnstageCapacity sets the optional capacity attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func MapUnstageCapacity(value int64) MapUnstageAttr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// MapUnstageMemoryLimit sets the optional memory_limit attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func MapUnstageMemoryLimit(value int64) MapUnstageAttr {
+	return func(m optionalAttr) {
+		m["memory_limit"] = value
+	}
+}
+
+// MapUnstageContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func MapUnstageContainer(value string) MapUnstageAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// MapUnstageSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func MapUnstageSharedName(value string) MapUnstageAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Op removes and returns the values associated with the key
+//
+// from the underlying container.   If the underlying container
+// does not contain this key, the op will block until it does.
+func MapUnstage(scope *Scope, key tf.Output, indices tf.Output, dtypes []tf.DataType, optional ...MapUnstageAttr) (values []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MapUnstage",
+		Input: []tf.Input{
+			key, indices,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if values, idx, err = makeOutputList(op, idx, "values"); err != nil {
+		scope.UpdateErr("MapUnstage", err)
+		return
+	}
+	return values
+}
+
+// StageSizeAttr is an optional argument to StageSize.
+type StageSizeAttr func(optionalAttr)
+
+// StageSizeCapacity sets the optional capacity attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func StageSizeCapacity(value int64) StageSizeAttr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// StageSizeMemoryLimit sets the optional memory_limit attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func StageSizeMemoryLimit(value int64) StageSizeAttr {
+	return func(m optionalAttr) {
+		m["memory_limit"] = value
+	}
+}
+
+// StageSizeContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func StageSizeContainer(value string) StageSizeAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// StageSizeSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func StageSizeSharedName(value string) StageSizeAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Op returns the number of elements in the underlying container.
+func StageSize(scope *Scope, dtypes []tf.DataType, optional ...StageSizeAttr) (size tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StageSize",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// StagePeekAttr is an optional argument to StagePeek.
+type StagePeekAttr func(optionalAttr)
+
+// StagePeekCapacity sets the optional capacity attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func StagePeekCapacity(value int64) StagePeekAttr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// StagePeekMemoryLimit sets the optional memory_limit attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func StagePeekMemoryLimit(value int64) StagePeekAttr {
+	return func(m optionalAttr) {
+		m["memory_limit"] = value
+	}
+}
+
+// StagePeekContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func StagePeekContainer(value string) StagePeekAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// StagePeekSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func StagePeekSharedName(value string) StagePeekAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Op peeks at the values at the specified index.  If the
+//
+// underlying container does not contain sufficient elements
+// this op will block until it does.   This Op is optimized for
+// performance.
+func StagePeek(scope *Scope, index tf.Output, dtypes []tf.DataType, optional ...StagePeekAttr) (values []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StagePeek",
+		Input: []tf.Input{
+			index,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if values, idx, err = makeOutputList(op, idx, "values"); err != nil {
+		scope.UpdateErr("StagePeek", err)
+		return
+	}
+	return values
+}
+
+// UnstageAttr is an optional argument to Unstage.
+type UnstageAttr func(optionalAttr)
+
+// UnstageCapacity sets the optional capacity attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func UnstageCapacity(value int64) UnstageAttr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// UnstageMemoryLimit sets the optional memory_limit attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func UnstageMemoryLimit(value int64) UnstageAttr {
+	return func(m optionalAttr) {
+		m["memory_limit"] = value
+	}
+}
+
+// UnstageContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func UnstageContainer(value string) UnstageAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// UnstageSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func UnstageSharedName(value string) UnstageAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Op is similar to a lightweight Dequeue.
+//
+// The basic functionality is similar to dequeue with many fewer
+// capabilities and options.  This Op is optimized for performance.
+func Unstage(scope *Scope, dtypes []tf.DataType, optional ...UnstageAttr) (values []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Unstage",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if values, idx, err = makeOutputList(op, idx, "values"); err != nil {
+		scope.UpdateErr("Unstage", err)
+		return
+	}
+	return values
+}
+
+// StageAttr is an optional argument to Stage.
+type StageAttr func(optionalAttr)
+
+// StageCapacity sets the optional capacity attribute to value.
+//
+// value: Maximum number of elements in the Staging Area. If > 0, inserts
+// on the container will block when the capacity is reached.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func StageCapacity(value int64) StageAttr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// StageMemoryLimit sets the optional memory_limit attribute to value.
+//
+// value: The maximum number of bytes allowed for Tensors in the Staging Area.
+// If > 0, inserts will block until sufficient space is available.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func StageMemoryLimit(value int64) StageAttr {
+	return func(m optionalAttr) {
+		m["memory_limit"] = value
+	}
+}
+
+// StageContainer sets the optional container attribute to value.
+//
+// value: If non-empty, this queue is placed in the given container. Otherwise,
+// a default container is used.
+// If not specified, defaults to ""
+func StageContainer(value string) StageAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// StageSharedName sets the optional shared_name attribute to value.
+//
+// value: It is necessary to match this name to the matching Unstage Op.
+// If not specified, defaults to ""
+func StageSharedName(value string) StageAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Stage values similar to a lightweight Enqueue.
+//
+// The basic functionality of this Op is similar to a queue with many
+// fewer capabilities and options.  This Op is optimized for performance.
+//
+// Arguments:
+//	values: a list of tensors
+// dtypes A list of data types that inserted values should adhere to.
+//
+// Returns the created operation.
+func Stage(scope *Scope, values []tf.Output, optional ...StageAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Stage",
+		Input: []tf.Input{
+			tf.OutputList(values),
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Delete the tensor specified by its handle in the session.
+//
+// Arguments:
+//	handle: The handle for a tensor stored in the session state.
+//
+// Returns the created operation.
+func DeleteSessionTensor(scope *Scope, handle tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "DeleteSessionTensor",
+		Input: []tf.Input{
+			handle,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Store the input tensor in the state of the current session.
+//
+// Arguments:
+//	value: The tensor to be stored.
+//
+// Returns The handle for the tensor stored in the session state, represented
+// as a ResourceHandle object.
+func GetSessionHandleV2(scope *Scope, value tf.Output) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "GetSessionHandleV2",
+		Input: []tf.Input{
+			value,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Store the input tensor in the state of the current session.
+//
+// Arguments:
+//	value: The tensor to be stored.
+//
+// Returns The handle for the tensor stored in the session state, represented
+// as a string.
+func GetSessionHandle(scope *Scope, value tf.Output) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "GetSessionHandle",
+		Input: []tf.Input{
+			value,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Copy a tensor setting everything outside a central band in each innermost matrix to zero.
+//
+// The `band` part is computed as follows:
+// Assume `input` has `k` dimensions `[I, J, K, ..., M, N]`, then the output is a
+// tensor with the same shape where
+//
+// `band[i, j, k, ..., m, n] = in_band(m, n) * input[i, j, k, ..., m, n]`.
+//
+// The indicator function
+//
+// `in_band(m, n) = (num_lower < 0 || (m-n) <= num_lower)) &&
+//                  (num_upper < 0 || (n-m) <= num_upper)`.
+//
+// For example:
+//
+// ```
+// # if 'input' is [[ 0,  1,  2, 3]
+//                  [-1,  0,  1, 2]
+//                  [-2, -1,  0, 1]
+//                  [-3, -2, -1, 0]],
+//
+// tf.matrix_band_part(input, 1, -1) ==> [[ 0,  1,  2, 3]
+//                                        [-1,  0,  1, 2]
+//                                        [ 0, -1,  0, 1]
+//                                        [ 0,  0, -1, 0]],
+//
+// tf.matrix_band_part(input, 2, 1) ==> [[ 0,  1,  0, 0]
+//                                       [-1,  0,  1, 0]
+//                                       [-2, -1,  0, 1]
+//                                       [ 0, -2, -1, 0]]
+// ```
+//
+// Useful special cases:
+//
+// ```
+//  tf.matrix_band_part(input, 0, -1) ==> Upper triangular part.
+//  tf.matrix_band_part(input, -1, 0) ==> Lower triangular part.
+//  tf.matrix_band_part(input, 0, 0) ==> Diagonal.
+// ```
+//
+// Arguments:
+//	input: Rank `k` tensor.
+//	num_lower: 0-D tensor. Number of subdiagonals to keep. If negative, keep entire
+// lower triangle.
+//	num_upper: 0-D tensor. Number of superdiagonals to keep. If negative, keep
+// entire upper triangle.
+//
+// Returns Rank `k` tensor of the same shape as input. The extracted banded tensor.
+func MatrixBandPart(scope *Scope, input tf.Output, num_lower tf.Output, num_upper tf.Output) (band tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "MatrixBandPart",
+		Input: []tf.Input{
+			input, num_lower, num_upper,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ListDiffAttr is an optional argument to ListDiff.
+type ListDiffAttr func(optionalAttr)
+
+// ListDiffOutIdx sets the optional out_idx attribute to value.
+// If not specified, defaults to DT_INT32
+func ListDiffOutIdx(value tf.DataType) ListDiffAttr {
+	return func(m optionalAttr) {
+		m["out_idx"] = value
+	}
+}
+
+// Computes the difference between two lists of numbers or strings.
+//
+// Given a list `x` and a list `y`, this operation returns a list `out` that
+// represents all values that are in `x` but not in `y`. The returned list `out`
+// is sorted in the same order that the numbers appear in `x` (duplicates are
+// preserved). This operation also returns a list `idx` that represents the
+// position of each `out` element in `x`. In other words:
+//
+// `out[i] = x[idx[i]] for i in [0, 1, ..., len(out) - 1]`
+//
+// For example, given this input:
+//
+// ```
+// x = [1, 2, 3, 4, 5, 6]
+// y = [1, 3, 5]
+// ```
+//
+// This operation would return:
+//
+// ```
+// out ==> [2, 4, 6]
+// idx ==> [1, 3, 5]
+// ```
+//
+// Arguments:
+//	x: 1-D. Values to keep.
+//	y: 1-D. Values to remove.
+//
+// Returns:
+//	out: 1-D. Values present in `x` but not in `y`.
+//	idx: 1-D. Positions of `x` values preserved in `out`.
+func ListDiff(scope *Scope, x tf.Output, y tf.Output, optional ...ListDiffAttr) (out tf.Output, idx tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ListDiff",
+		Input: []tf.Input{
+			x, y,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Deprecated. Use TensorArrayScatterV3
+//
+// DEPRECATED at GraphDef version 26: Use TensorArrayScatterV3
+func TensorArrayScatterV2(scope *Scope, handle tf.Output, indices tf.Output, value tf.Output, flow_in tf.Output) (flow_out tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorArrayScatterV2",
+		Input: []tf.Input{
+			handle, indices, value, flow_in,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Deprecated. Use TensorArrayReadV3
+//
+// DEPRECATED at GraphDef version 26: Use TensorArrayReadV3
+func TensorArrayReadV2(scope *Scope, handle tf.Output, index tf.Output, flow_in tf.Output, dtype tf.DataType) (value tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	opspec := tf.OpSpec{
+		Type: "TensorArrayReadV2",
+		Input: []tf.Input{
+			handle, index, flow_in,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Deprecated. Use TensorArrayGradV3
+//
+// DEPRECATED at GraphDef version 26: Use TensorArrayGradV3
+func TensorArrayGradV2(scope *Scope, handle tf.Output, flow_in tf.Output, source string) (grad_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"source": source}
+	opspec := tf.OpSpec{
+		Type: "TensorArrayGradV2",
+		Input: []tf.Input{
+			handle, flow_in,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// TensorArrayV2Attr is an optional argument to TensorArrayV2.
+type TensorArrayV2Attr func(optionalAttr)
+
+// TensorArrayV2ElementShape sets the optional element_shape attribute to value.
+// If not specified, defaults to <unknown_rank:true >
+func TensorArrayV2ElementShape(value tf.Shape) TensorArrayV2Attr {
+	return func(m optionalAttr) {
+		m["element_shape"] = value
+	}
+}
+
+// TensorArrayV2DynamicSize sets the optional dynamic_size attribute to value.
+// If not specified, defaults to false
+func TensorArrayV2DynamicSize(value bool) TensorArrayV2Attr {
+	return func(m optionalAttr) {
+		m["dynamic_size"] = value
+	}
+}
+
+// TensorArrayV2ClearAfterRead sets the optional clear_after_read attribute to value.
+// If not specified, defaults to true
+func TensorArrayV2ClearAfterRead(value bool) TensorArrayV2Attr {
+	return func(m optionalAttr) {
+		m["clear_after_read"] = value
+	}
+}
+
+// TensorArrayV2TensorArrayName sets the optional tensor_array_name attribute to value.
+// If not specified, defaults to ""
+func TensorArrayV2TensorArrayName(value string) TensorArrayV2Attr {
+	return func(m optionalAttr) {
+		m["tensor_array_name"] = value
+	}
+}
+
+// Deprecated. Use TensorArrayV3
+//
+// DEPRECATED at GraphDef version 26: Use TensorArrayV3
+func TensorArrayV2(scope *Scope, size tf.Output, dtype tf.DataType, optional ...TensorArrayV2Attr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorArrayV2",
+		Input: []tf.Input{
+			size,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Get the current size of the TensorArray.
+//
+// Arguments:
+//	handle: The handle to a TensorArray (output of TensorArray or TensorArrayGrad).
+//	flow_in: A float scalar that enforces proper chaining of operations.
+//
+// Returns The current size of the TensorArray.
+func TensorArraySizeV3(scope *Scope, handle tf.Output, flow_in tf.Output) (size tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorArraySizeV3",
+		Input: []tf.Input{
+			handle, flow_in,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Split the data from the input value into TensorArray elements.
+//
+// Assuming that `lengths` takes on values
+//
+//   ```(n0, n1, ..., n(T-1))```
+//
+// and that `value` has shape
+//
+//   ```(n0 + n1 + ... + n(T-1) x d0 x d1 x ...)```,
+//
+// this splits values into a TensorArray with T tensors.
+//
+// TensorArray index t will be the subtensor of values with starting position
+//
+//   ```(n0 + n1 + ... + n(t-1), 0, 0, ...)```
+//
+// and having size
+//
+//   ```nt x d0 x d1 x ...```
+//
+// Arguments:
+//	handle: The handle to a TensorArray.
+//	value: The concatenated tensor to write to the TensorArray.
+//	lengths: The vector of lengths, how to split the rows of value into the
+// TensorArray.
+//	flow_in: A float scalar that enforces proper chaining of operations.
+//
+// Returns A float scalar that enforces proper chaining of operations.
+func TensorArraySplitV3(scope *Scope, handle tf.Output, value tf.Output, lengths tf.Output, flow_in tf.Output) (flow_out tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorArraySplitV3",
+		Input: []tf.Input{
+			handle, value, lengths, flow_in,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// TensorArrayConcatV3Attr is an optional argument to TensorArrayConcatV3.
+type TensorArrayConcatV3Attr func(optionalAttr)
+
+// TensorArrayConcatV3ElementShapeExcept0 sets the optional element_shape_except0 attribute to value.
+//
+// value: The expected shape of an element, if known,
+// excluding the first dimension. Used to validate the shapes of
+// TensorArray elements. If this shape is not fully specified, concatenating
+// zero-size TensorArrays is an error.
+// If not specified, defaults to <unknown_rank:true >
+func TensorArrayConcatV3ElementShapeExcept0(value tf.Shape) TensorArrayConcatV3Attr {
+	return func(m optionalAttr) {
+		m["element_shape_except0"] = value
+	}
+}
+
+// Concat the elements from the TensorArray into value `value`.
+//
+// Takes `T` elements of shapes
+//
+//   ```
+//   (n0 x d0 x d1 x ...), (n1 x d0 x d1 x ...), ..., (n(T-1) x d0 x d1 x ...)
+//   ```
+//
+// and concatenates them into a Tensor of shape:
+//
+//   ```(n0 + n1 + ... + n(T-1) x d0 x d1 x ...)```
+//
+// All elements must have the same shape (excepting the first dimension).
+//
+// Arguments:
+//	handle: The handle to a TensorArray.
+//	flow_in: A float scalar that enforces proper chaining of operations.
+//	dtype: The type of the elem that is returned.
+//
+// Returns:
+//	value: All of the elements in the TensorArray, concatenated along the first
+// axis.
+//	lengths: A vector of the row sizes of the original T elements in the
+// value output.  In the example above, this would be the values:
+// `(n1, n2, ..., n(T-1))`.
+func TensorArrayConcatV3(scope *Scope, handle tf.Output, flow_in tf.Output, dtype tf.DataType, optional ...TensorArrayConcatV3Attr) (value tf.Output, lengths tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorArrayConcatV3",
+		Input: []tf.Input{
+			handle, flow_in,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// TensorArrayGatherV3Attr is an optional argument to TensorArrayGatherV3.
+type TensorArrayGatherV3Attr func(optionalAttr)
+
+// TensorArrayGatherV3ElementShape sets the optional element_shape attribute to value.
+//
+// value: The expected shape of an element, if known. Used to
+// validate the shapes of TensorArray elements. If this shape is not
+// fully specified, gathering zero-size TensorArrays is an error.
+// If not specified, defaults to <unknown_rank:true >
+func TensorArrayGatherV3ElementShape(value tf.Shape) TensorArrayGatherV3Attr {
+	return func(m optionalAttr) {
+		m["element_shape"] = value
+	}
+}
+
+// Gather specific elements from the TensorArray into output `value`.
+//
+// All elements selected by `indices` must have the same shape.
+//
+// Arguments:
+//	handle: The handle to a TensorArray.
+//	indices: The locations in the TensorArray from which to read tensor elements.
+//	flow_in: A float scalar that enforces proper chaining of operations.
+//	dtype: The type of the elem that is returned.
+//
+// Returns All of the elements in the TensorArray, concatenated along a new
+// axis (the new dimension 0).
+func TensorArrayGatherV3(scope *Scope, handle tf.Output, indices tf.Output, flow_in tf.Output, dtype tf.DataType, optional ...TensorArrayGatherV3Attr) (value tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorArrayGatherV3",
+		Input: []tf.Input{
+			handle, indices, flow_in,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// GatherAttr is an optional argument to Gather.
+type GatherAttr func(optionalAttr)
+
+// GatherValidateIndices sets the optional validate_indices attribute to value.
+// If not specified, defaults to true
+func GatherValidateIndices(value bool) GatherAttr {
+	return func(m optionalAttr) {
+		m["validate_indices"] = value
+	}
+}
+
+// Gather slices from `params` according to `indices`.
+//
+// `indices` must be an integer tensor of any dimension (usually 0-D or 1-D).
+// Produces an output tensor with shape `indices.shape + params.shape[1:]` where:
+//
+// ```python
+//     # Scalar indices
+//     output[:, ..., :] = params[indices, :, ... :]
+//
+//     # Vector indices
+//     output[i, :, ..., :] = params[indices[i], :, ... :]
+//
+//     # Higher rank indices
+//     output[i, ..., j, :, ... :] = params[indices[i, ..., j], :, ..., :]
+// ```
+//
+// If `indices` is a permutation and `len(indices) == params.shape[0]` then
+// this operation will permute `params` accordingly.
+//
+// `validate_indices`: DEPRECATED. If this operation is assigned to CPU, values in
+// `indices` are always validated to be within range. If assigned to GPU,
+// out-of-bound indices result in safe but unspecified behavior, which may include
+// raising an error.
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src="https://www.tensorflow.org/images/Gather.png" alt>
+// </div>
+func Gather(scope *Scope, params tf.Output, indices tf.Output, optional ...GatherAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Gather",
+		Input: []tf.Input{
+			params, indices,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Read an element from the TensorArray into output `value`.
+//
+// Arguments:
+//	handle: The handle to a TensorArray.
+//
+//	flow_in: A float scalar that enforces proper chaining of operations.
+//	dtype: The type of the elem that is returned.
+//
+// Returns The tensor that is read from the TensorArray.
+func TensorArrayReadV3(scope *Scope, handle tf.Output, index tf.Output, flow_in tf.Output, dtype tf.DataType) (value tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	opspec := tf.OpSpec{
+		Type: "TensorArrayReadV3",
+		Input: []tf.Input{
+			handle, index, flow_in,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Push an element onto the tensor_array.
+//
+// Arguments:
+//	handle: The handle to a TensorArray.
+//	index: The position to write to inside the TensorArray.
+//	value: The tensor to write to the TensorArray.
+//	flow_in: A float scalar that enforces proper chaining of operations.
+//
+// Returns A float scalar that enforces proper chaining of operations.
+func TensorArrayWriteV3(scope *Scope, handle tf.Output, index tf.Output, value tf.Output, flow_in tf.Output) (flow_out tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorArrayWriteV3",
+		Input: []tf.Input{
+			handle, index, value, flow_in,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a TensorArray for storing multiple gradients of values in the given handle.
+//
+// Similar to TensorArrayGradV3. However it creates an accumulator with an
+// expanded shape compared to the input TensorArray whose gradient is being
+// computed. This enables multiple gradients for the same TensorArray to be
+// calculated using the same accumulator.
+//
+// Arguments:
+//	handle: The handle to the forward TensorArray.
+//	flow_in: A float scalar that enforces proper chaining of operations.
+//	shape_to_prepend: An int32 vector representing a shape. Elements in the gradient accumulator will
+// have shape which is this shape_to_prepend value concatenated with shape of the
+// elements in the TensorArray corresponding to the input handle.
+//	source: The gradient source string, used to decide which gradient TensorArray
+// to return.
+func TensorArrayGradWithShape(scope *Scope, handle tf.Output, flow_in tf.Output, shape_to_prepend tf.Output, source string) (grad_handle tf.Output, flow_out tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"source": source}
+	opspec := tf.OpSpec{
+		Type: "TensorArrayGradWithShape",
+		Input: []tf.Input{
+			handle, flow_in, shape_to_prepend,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Delete the stack from its resource container.
+//
+// Arguments:
+//	handle: The handle to a stack.
+//
+// Returns the created operation.
+func StackCloseV2(scope *Scope, handle tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "StackCloseV2",
+		Input: []tf.Input{
+			handle,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Pop the element at the top of the stack.
+//
+// Arguments:
+//	handle: The handle to a stack.
+//	elem_type: The type of the elem that is popped.
+//
+// Returns The tensor that is popped from the top of the stack.
+func StackPopV2(scope *Scope, handle tf.Output, elem_type tf.DataType) (elem tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"elem_type": elem_type}
+	opspec := tf.OpSpec{
+		Type: "StackPopV2",
+		Input: []tf.Input{
+			handle,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// StackPushV2Attr is an optional argument to StackPushV2.
+type StackPushV2Attr func(optionalAttr)
+
+// StackPushV2SwapMemory sets the optional swap_memory attribute to value.
+//
+// value: Swap `elem` to CPU. Default to false.
+// If not specified, defaults to false
+func StackPushV2SwapMemory(value bool) StackPushV2Attr {
+	return func(m optionalAttr) {
+		m["swap_memory"] = value
+	}
+}
+
+// Push an element onto the stack.
+//
+// Arguments:
+//	handle: The handle to a stack.
+//	elem: The tensor to be pushed onto the stack.
+//
+// Returns The same tensor as the input 'elem'.
+func StackPushV2(scope *Scope, handle tf.Output, elem tf.Output, optional ...StackPushV2Attr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StackPushV2",
+		Input: []tf.Input{
+			handle, elem,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// StackV2Attr is an optional argument to StackV2.
+type StackV2Attr func(optionalAttr)
+
+// StackV2StackName sets the optional stack_name attribute to value.
+//
+// value: Overrides the name used for the temporary stack resource. Default
+// value is the name of the 'Stack' op (which is guaranteed unique).
+// If not specified, defaults to ""
+func StackV2StackName(value string) StackV2Attr {
+	return func(m optionalAttr) {
+		m["stack_name"] = value
+	}
+}
+
+// A stack that produces elements in first-in last-out order.
+//
+// Arguments:
+//	max_size: The maximum size of the stack if non-negative. If negative, the stack
+// size is unlimited.
+//	elem_type: The type of the elements on the stack.
+//
+// Returns The handle to the stack.
+func StackV2(scope *Scope, max_size tf.Output, elem_type tf.DataType, optional ...StackV2Attr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"elem_type": elem_type}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StackV2",
+		Input: []tf.Input{
+			max_size,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Checks a tensor for NaN, -Inf and +Inf values.
+//
+// When run, reports an `InvalidArgument` error if `tensor` has any values
+// that are not a number (NaN) or infinity (Inf). Otherwise, passes `tensor` as-is.
+// Unlike CheckNumerics (V1), CheckNumericsV2 distinguishes -Inf and +Inf in the
+// errors it throws.
+//
+// Arguments:
+//
+//	message: Prefix of the error message.
+func CheckNumericsV2(scope *Scope, tensor tf.Output, message string) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"message": message}
+	opspec := tf.OpSpec{
+		Type: "CheckNumericsV2",
+		Input: []tf.Input{
+			tensor,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Applies a gradient to a given accumulator.
+//
+// Does not add if local_step is lesser than the accumulator's global_step.
+//
+// Arguments:
+//	handle: The handle to a accumulator.
+//	local_step: The local_step value at which the gradient was computed.
+//	gradient: A tensor of the gradient to be accumulated.
+//
+// Returns the created operation.
+func ResourceAccumulatorApplyGradient(scope *Scope, handle tf.Output, local_step tf.Output, gradient tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceAccumulatorApplyGradient",
+		Input: []tf.Input{
+			handle, local_step, gradient,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Updates the accumulator with a new value for global_step.
+//
+// Logs warning if the accumulator's value is already higher than
+// new_global_step.
+//
+// Arguments:
+//	handle: The handle to an accumulator.
+//	new_global_step: The new global_step value to set.
+//
+// Returns the created operation.
+func ResourceAccumulatorSetGlobalStep(scope *Scope, handle tf.Output, new_global_step tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceAccumulatorSetGlobalStep",
+		Input: []tf.Input{
+			handle, new_global_step,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Computes the number of elements in the given queue.
+//
+// Arguments:
+//	handle: The handle to a queue.
+//
+// Returns The number of elements in the given queue.
+func QueueSizeV2(scope *Scope, handle tf.Output) (size tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "QueueSizeV2",
+		Input: []tf.Input{
+			handle,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// QueueEnqueueManyV2Attr is an optional argument to QueueEnqueueManyV2.
+type QueueEnqueueManyV2Attr func(optionalAttr)
+
+// QueueEnqueueManyV2TimeoutMs sets the optional timeout_ms attribute to value.
+//
+// value: If the queue is too full, this operation will block for up
+// to timeout_ms milliseconds.
+// Note: This option is not supported yet.
+// If not specified, defaults to -1
+func QueueEnqueueManyV2TimeoutMs(value int64) QueueEnqueueManyV2Attr {
+	return func(m optionalAttr) {
+		m["timeout_ms"] = value
+	}
+}
+
+// Enqueues zero or more tuples of one or more tensors in the given queue.
+//
+// This operation slices each component tensor along the 0th dimension to
+// make multiple queue elements. All of the tuple components must have the
+// same size in the 0th dimension.
+//
+// The components input has k elements, which correspond to the components of
+// tuples stored in the given queue.
+//
+// N.B. If the queue is full, this operation will block until the given
+// elements have been enqueued (or 'timeout_ms' elapses, if specified).
+//
+// Arguments:
+//	handle: The handle to a queue.
+//	components: One or more tensors from which the enqueued tensors should
+// be taken.
+//
+// Returns the created operation.
+func QueueEnqueueManyV2(scope *Scope, handle tf.Output, components []tf.Output, optional ...QueueEnqueueManyV2Attr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QueueEnqueueManyV2",
+		Input: []tf.Input{
+			handle, tf.OutputList(components),
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// QueueEnqueueV2Attr is an optional argument to QueueEnqueueV2.
+type QueueEnqueueV2Attr func(optionalAttr)
+
+// QueueEnqueueV2TimeoutMs sets the optional timeout_ms attribute to value.
+//
+// value: If the queue is full, this operation will block for up to
+// timeout_ms milliseconds.
+// Note: This option is not supported yet.
+// If not specified, defaults to -1
+func QueueEnqueueV2TimeoutMs(value int64) QueueEnqueueV2Attr {
+	return func(m optionalAttr) {
+		m["timeout_ms"] = value
+	}
+}
+
+// Enqueues a tuple of one or more tensors in the given queue.
+//
+// The components input has k elements, which correspond to the components of
+// tuples stored in the given queue.
+//
+// N.B. If the queue is full, this operation will block until the given
+// element has been enqueued (or 'timeout_ms' elapses, if specified).
+//
+// Arguments:
+//	handle: The handle to a queue.
+//	components: One or more tensors from which the enqueued tensors should be taken.
+//
+// Returns the created operation.
+func QueueEnqueueV2(scope *Scope, handle tf.Output, components []tf.Output, optional ...QueueEnqueueV2Attr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QueueEnqueueV2",
+		Input: []tf.Input{
+			handle, tf.OutputList(components),
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// PriorityQueueV2Attr is an optional argument to PriorityQueueV2.
+type PriorityQueueV2Attr func(optionalAttr)
+
+// PriorityQueueV2ComponentTypes sets the optional component_types attribute to value.
+//
+// value: The type of each component in a value.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func PriorityQueueV2ComponentTypes(value []tf.DataType) PriorityQueueV2Attr {
+	return func(m optionalAttr) {
+		m["component_types"] = value
+	}
+}
+
+// PriorityQueueV2Capacity sets the optional capacity attribute to value.
+//
+// value: The upper bound on the number of elements in this queue.
+// Negative numbers mean no limit.
+// If not specified, defaults to -1
+func PriorityQueueV2Capacity(value int64) PriorityQueueV2Attr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// PriorityQueueV2Container sets the optional container attribute to value.
+//
+// value: If non-empty, this queue is placed in the given container.
+// Otherwise, a default container is used.
+// If not specified, defaults to ""
+func PriorityQueueV2Container(value string) PriorityQueueV2Attr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// PriorityQueueV2SharedName sets the optional shared_name attribute to value.
+//
+// value: If non-empty, this queue will be shared under the given name
+// across multiple sessions.
+// If not specified, defaults to ""
+func PriorityQueueV2SharedName(value string) PriorityQueueV2Attr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// A queue that produces elements sorted by the first component value.
+//
+// Note that the PriorityQueue requires the first component of any element
+// to be a scalar int64, in addition to the other elements declared by
+// component_types.  Therefore calls to Enqueue and EnqueueMany (resp. Dequeue
+// and DequeueMany) on a PriorityQueue will all require (resp. output) one extra
+// entry in their input (resp. output) lists.
+//
+// Arguments:
+//	shapes: The shape of each component in a value. The length of this attr must
+// be either 0 or the same as the length of component_types. If the length of
+// this attr is 0, the shapes of queue elements are not constrained, and
+// only one element may be dequeued at a time.
+//
+// Returns The handle to the queue.
+func PriorityQueueV2(scope *Scope, shapes []tf.Shape, optional ...PriorityQueueV2Attr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"shapes": shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "PriorityQueueV2",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Does nothing. Serves as a control trigger for scheduling.
+//
+// Only useful as a placeholder for control edges.
+//
+// Returns the created operation.
+func ControlTrigger(scope *Scope) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ControlTrigger",
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Interleave the values from the `data` tensors into a single tensor.
+//
+// Builds a merged tensor such that
+//
+// ```python
+//     merged[indices[m][i, ..., j], ...] = data[m][i, ..., j, ...]
+// ```
+//
+// For example, if each `indices[m]` is scalar or vector, we have
+//
+// ```python
+//     # Scalar indices:
+//     merged[indices[m], ...] = data[m][...]
+//
+//     # Vector indices:
+//     merged[indices[m][i], ...] = data[m][i, ...]
+// ```
+//
+// Each `data[i].shape` must start with the corresponding `indices[i].shape`,
+// and the rest of `data[i].shape` must be constant w.r.t. `i`.  That is, we
+// must have `data[i].shape = indices[i].shape + constant`.  In terms of this
+// `constant`, the output shape is
+//
+//     merged.shape = [max(indices)] + constant
+//
+// Values may be merged in parallel, so if an index appears in both `indices[m][i]`
+// and `indices[n][j]`, the result may be invalid. This differs from the normal
+// DynamicStitch operator that defines the behavior in that case.
+//
+// For example:
+//
+// ```python
+//     indices[0] = 6
+//     indices[1] = [4, 1]
+//     indices[2] = [[5, 2], [0, 3]]
+//     data[0] = [61, 62]
+//     data[1] = [[41, 42], [11, 12]]
+//     data[2] = [[[51, 52], [21, 22]], [[1, 2], [31, 32]]]
+//     merged = [[1, 2], [11, 12], [21, 22], [31, 32], [41, 42],
+//               [51, 52], [61, 62]]
+// ```
+//
+// This method can be used to merge partitions created by `dynamic_partition`
+// as illustrated on the following example:
+//
+// ```python
+//     # Apply function (increments x_i) on elements for which a certain condition
+//     # apply (x_i != -1 in this example).
+//     x=tf.constant([0.1, -1., 5.2, 4.3, -1., 7.4])
+//     condition_mask=tf.not_equal(x,tf.constant(-1.))
+//     partitioned_data = tf.dynamic_partition(
+//         x, tf.cast(condition_mask, tf.int32) , 2)
+//     partitioned_data[1] = partitioned_data[1] + 1.0
+//     condition_indices = tf.dynamic_partition(
+//         tf.range(tf.shape(x)[0]), tf.cast(condition_mask, tf.int32) , 2)
+//     x = tf.dynamic_stitch(condition_indices, partitioned_data)
+//     # Here x=[1.1, -1., 6.2, 5.3, -1, 8.4], the -1. values remain
+//     # unchanged.
+// ```
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src="https://www.tensorflow.org/images/DynamicStitch.png" alt>
+// </div>
+func ParallelDynamicStitch(scope *Scope, indices []tf.Output, data []tf.Output) (merged tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ParallelDynamicStitch",
+		Input: []tf.Input{
+			tf.OutputList(indices), tf.OutputList(data),
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Partitions `data` into `num_partitions` tensors using indices from `partitions`.
+//
+// For each index tuple `js` of size `partitions.ndim`, the slice `data[js, ...]`
+// becomes part of `outputs[partitions[js]]`.  The slices with `partitions[js] = i`
+// are placed in `outputs[i]` in lexicographic order of `js`, and the first
+// dimension of `outputs[i]` is the number of entries in `partitions` equal to `i`.
+// In detail,
+//
+// ```python
+//     outputs[i].shape = [sum(partitions == i)] + data.shape[partitions.ndim:]
+//
+//     outputs[i] = pack([data[js, ...] for js if partitions[js] == i])
+// ```
+//
+// `data.shape` must start with `partitions.shape`.
+//
+// For example:
+//
+// ```python
+//     # Scalar partitions.
+//     partitions = 1
+//     num_partitions = 2
+//     data = [10, 20]
+//     outputs[0] = []  # Empty with shape [0, 2]
+//     outputs[1] = [[10, 20]]
+//
+//     # Vector partitions.
+//     partitions = [0, 0, 1, 1, 0]
+//     num_partitions = 2
+//     data = [10, 20, 30, 40, 50]
+//     outputs[0] = [10, 20, 50]
+//     outputs[1] = [30, 40]
+// ```
+//
+// See `dynamic_stitch` for an example on how to merge partitions back.
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src="https://www.tensorflow.org/images/DynamicPartition.png" alt>
+// </div>
+//
+// Arguments:
+//
+//	partitions: Any shape.  Indices in the range `[0, num_partitions)`.
+//	num_partitions: The number of partitions to output.
+func DynamicPartition(scope *Scope, data tf.Output, partitions tf.Output, num_partitions int64) (outputs []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_partitions": num_partitions}
+	opspec := tf.OpSpec{
+		Type: "DynamicPartition",
+		Input: []tf.Input{
+			data, partitions,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if outputs, idx, err = makeOutputList(op, idx, "outputs"); err != nil {
+		scope.UpdateErr("DynamicPartition", err)
+		return
+	}
+	return outputs
+}
+
+// ResourceConditionalAccumulatorAttr is an optional argument to ResourceConditionalAccumulator.
+type ResourceConditionalAccumulatorAttr func(optionalAttr)
+
+// ResourceConditionalAccumulatorContainer sets the optional container attribute to value.
+//
+// value: If non-empty, this accumulator is placed in the given container.
+// Otherwise, a default container is used.
+// If not specified, defaults to ""
+func ResourceConditionalAccumulatorContainer(value string) ResourceConditionalAccumulatorAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// ResourceConditionalAccumulatorSharedName sets the optional shared_name attribute to value.
+//
+// value: If non-empty, this accumulator will be shared under the
+// given name across multiple sessions.
+// If not specified, defaults to ""
+func ResourceConditionalAccumulatorSharedName(value string) ResourceConditionalAccumulatorAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// ResourceConditionalAccumulatorReductionType sets the optional reduction_type attribute to value.
+// If not specified, defaults to "MEAN"
+func ResourceConditionalAccumulatorReductionType(value string) ResourceConditionalAccumulatorAttr {
+	return func(m optionalAttr) {
+		m["reduction_type"] = value
+	}
+}
+
+// A conditional accumulator for aggregating gradients.
+//
+// The accumulator accepts gradients marked with local_step greater or
+// equal to the most recent global_step known to the accumulator. The
+// average can be extracted from the accumulator, provided sufficient
+// gradients have been accumulated. Extracting the average automatically
+// resets the aggregate to 0, and increments the global_step recorded by
+// the accumulator.
+// This is a resource version of ConditionalAccumulator that will work in TF2.0
+// with tf.cond version 2.
+//
+// Arguments:
+//	dtype: The type of the value being accumulated.
+//	shape: The shape of the values, can be [], in which case shape is unknown.
+//
+// Returns The handle to the accumulator.
+func ResourceConditionalAccumulator(scope *Scope, dtype tf.DataType, shape tf.Shape, optional ...ResourceConditionalAccumulatorAttr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype, "shape": shape}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceConditionalAccumulator",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MultiDeviceIteratorFromStringHandleAttr is an optional argument to MultiDeviceIteratorFromStringHandle.
+type MultiDeviceIteratorFromStringHandleAttr func(optionalAttr)
+
+// MultiDeviceIteratorFromStringHandleOutputTypes sets the optional output_types attribute to value.
+//
+// value: The type list for the return values.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func MultiDeviceIteratorFromStringHandleOutputTypes(value []tf.DataType) MultiDeviceIteratorFromStringHandleAttr {
+	return func(m optionalAttr) {
+		m["output_types"] = value
+	}
+}
+
+// MultiDeviceIteratorFromStringHandleOutputShapes sets the optional output_shapes attribute to value.
+//
+// value: The list of shapes being produced.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func MultiDeviceIteratorFromStringHandleOutputShapes(value []tf.Shape) MultiDeviceIteratorFromStringHandleAttr {
+	return func(m optionalAttr) {
+		m["output_shapes"] = value
+	}
+}
+
+// Generates a MultiDeviceIterator resource from its provided string handle.
+//
+// Arguments:
+//	string_handle: String representing the resource.
+//
+// Returns A MultiDeviceIterator resource.
+func MultiDeviceIteratorFromStringHandle(scope *Scope, string_handle tf.Output, optional ...MultiDeviceIteratorFromStringHandleAttr) (multi_device_iterator tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MultiDeviceIteratorFromStringHandle",
+		Input: []tf.Input{
+			string_handle,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a TensorArray for storing the gradients of values in the given handle.
+//
+// If the given TensorArray gradient already exists, returns a reference to it.
+//
+// Locks the size of the original TensorArray by disabling its dynamic size flag.
+//
+// **A note about the input flow_in:**
+//
+// The handle flow_in forces the execution of the gradient lookup to occur
+// only after certain other operations have occurred.  For example, when
+// the forward TensorArray is dynamically sized, writes to this TensorArray
+// may resize the object.  The gradient TensorArray is statically sized based
+// on the size of the forward TensorArray when this operation executes.
+// Furthermore, the size of the forward TensorArray is frozen by this call.
+// As a result, the flow is used to ensure that the call to generate the gradient
+// TensorArray only happens after all writes are executed.
+//
+// In the case of dynamically sized TensorArrays, gradient computation should
+// only be performed on read operations that have themselves been chained via
+// flow to occur only after all writes have executed. That way the final size
+// of the forward TensorArray is known when this operation is called.
+//
+// **A note about the source attribute:**
+//
+// TensorArray gradient calls use an accumulator TensorArray object.  If
+// multiple gradients are calculated and run in the same session, the multiple
+// gradient nodes may accidentally flow through the same accumulator TensorArray.
+// This double counts and generally breaks the TensorArray gradient flow.
+//
+// The solution is to identify which gradient call this particular
+// TensorArray gradient is being called in.  This is performed by identifying
+// a unique string (e.g. "gradients", "gradients_1", ...) from the input
+// gradient Tensor's name.  This string is used as a suffix when creating
+// the TensorArray gradient object here (the attribute `source`).
+//
+// The attribute `source` is added as a suffix to the forward TensorArray's
+// name when performing the creation / lookup, so that each separate gradient
+// calculation gets its own TensorArray accumulator.
+//
+// Arguments:
+//	handle: The handle to the forward TensorArray.
+//	flow_in: A float scalar that enforces proper chaining of operations.
+//	source: The gradient source string, used to decide which gradient TensorArray
+// to return.
+func TensorArrayGradV3(scope *Scope, handle tf.Output, flow_in tf.Output, source string) (grad_handle tf.Output, flow_out tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"source": source}
+	opspec := tf.OpSpec{
+		Type: "TensorArrayGradV3",
+		Input: []tf.Input{
+			handle, flow_in,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Produces a string handle for the given MultiDeviceIterator.
+//
+// Arguments:
+//	multi_device_iterator: A MultiDeviceIterator resource.
+//
+// Returns A string representing the resource.
+func MultiDeviceIteratorToStringHandle(scope *Scope, multi_device_iterator tf.Output) (string_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "MultiDeviceIteratorToStringHandle",
+		Input: []tf.Input{
+			multi_device_iterator,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Gets next element for the provided shard number.
+//
+// Arguments:
+//	multi_device_iterator: A MultiDeviceIterator resource.
+//	shard_num: Integer representing which shard to fetch data for.
+//	incarnation_id: Which incarnation of the MultiDeviceIterator is running.
+//	output_types: The type list for the return values.
+//	output_shapes: The list of shapes being produced.
+//
+// Returns Result of the get_next on the dataset.
+func MultiDeviceIteratorGetNextFromShard(scope *Scope, multi_device_iterator tf.Output, shard_num tf.Output, incarnation_id tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (components []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "MultiDeviceIteratorGetNextFromShard",
+		Input: []tf.Input{
+			multi_device_iterator, shard_num, incarnation_id,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if components, idx, err = makeOutputList(op, idx, "components"); err != nil {
+		scope.UpdateErr("MultiDeviceIteratorGetNextFromShard", err)
+		return
+	}
+	return components
+}
+
+// Creates a MultiDeviceIterator resource.
+//
+// Arguments:
+//	devices: A list of devices the iterator works across.
+//	shared_name: If non-empty, this resource will be shared under the given name
+// across multiple sessions.
+//	container: If non-empty, this resource is placed in the given container.
+// Otherwise, a default container is used.
+//	output_types: The type list for the return values.
+//	output_shapes: The list of shapes being produced.
+//
+// Returns Handle to the resource created.
+func MultiDeviceIterator(scope *Scope, devices []string, shared_name string, container string, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"devices": devices, "shared_name": shared_name, "container": container, "output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "MultiDeviceIterator",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// BoostedTreesCalculateBestFeatureSplitAttr is an optional argument to BoostedTreesCalculateBestFeatureSplit.
+type BoostedTreesCalculateBestFeatureSplitAttr func(optionalAttr)
+
+// BoostedTreesCalculateBestFeatureSplitSplitType sets the optional split_type attribute to value.
+//
+// value: A string indicating if this Op should perform inequality split or equality split.
+// If not specified, defaults to "inequality"
+func BoostedTreesCalculateBestFeatureSplitSplitType(value string) BoostedTreesCalculateBestFeatureSplitAttr {
+	return func(m optionalAttr) {
+		m["split_type"] = value
+	}
+}
+
+// Calculates gains for each feature and returns the best possible split information for the feature.
+//
+// The split information is the best threshold (bucket id), gains and left/right node contributions per node for each feature.
+//
+// It is possible that not all nodes can be split on each feature. Hence, the list of possible nodes can differ between the features. Therefore, we return `node_ids_list` for each feature, containing the list of nodes that this feature can be used to split.
+//
+// In this manner, the output is the best split per features and per node, so that it needs to be combined later to produce the best split for each node (among all possible features).
+//
+// The output shapes are compatible in a way that the first dimension of all tensors are the same and equal to the number of possible split nodes for each feature.
+//
+// Arguments:
+//	node_id_range: A Rank 1 tensor (shape=[2]) to specify the range [first, last) of node ids to process within `stats_summary_list`. The nodes are iterated between the two nodes specified by the tensor, as like `for node_id in range(node_id_range[0], node_id_range[1])` (Note that the last index node_id_range[1] is exclusive).
+//	stats_summary: A Rank 4 tensor (#shape=[max_splits, feature_dims, bucket, stats_dims]) for accumulated stats summary (gradient/hessian) per node, per dimension, per buckets for each feature.
+// The first dimension of the tensor is the maximum number of splits, and thus not all elements of it will be used, but only the indexes specified by node_ids will be used.
+//	l1: l1 regularization factor on leaf weights, per instance based.
+//	l2: l2 regularization factor on leaf weights, per instance based.
+//	tree_complexity: adjustment to the gain, per leaf based.
+//	min_node_weight: minimum avg of hessians in a node before required for the node to be considered for splitting.
+//	logits_dimension: The dimension of logit, i.e., number of classes.
+//
+// Returns:
+//	node_ids: A Rank 1 tensors indicating possible split node ids for each feature. The length of the list is num_features, but each tensor has different size as each feature provides different possible nodes. See above for details like shapes and sizes.
+//	gains: A Rank 1 tensors indicating the best gains for each feature to split for certain nodes. See above for details like shapes and sizes.
+//	feature_dimensions: A Rank 1 tensors indicating the best feature dimension for each feature to split for certain nodes if the feature is multi-dimension. See above for details like shapes and sizes.
+//	thresholds: A Rank 1 tensors indicating the bucket id to compare with (as a threshold) for split in each node. See above for details like shapes and sizes.
+//	left_node_contribs: A Rank 2 tensors indicating the contribution of the left nodes when branching from parent nodes (given by the tensor element in the output node_ids_list) to the left direction by the given threshold for each feature. This value will be used to make the left node value by adding to the parent node value. Second dimension size is 1 for 1-dimensional logits, but would be larger for multi-class problems. See above for details like shapes and sizes.
+//	right_node_contribs: A Rank 2 tensors, with the same shape/conditions as left_node_contribs_list, but just that the value is for the right node.
+//	split_with_default_directions: A Rank 1 tensors indicating the which direction to go if data is missing. See above for details like shapes and sizes.
+// Inequality with default left returns 0, inequality with default right returns 1, equality with default right returns 2.
+func BoostedTreesCalculateBestFeatureSplit(scope *Scope, node_id_range tf.Output, stats_summary tf.Output, l1 tf.Output, l2 tf.Output, tree_complexity tf.Output, min_node_weight tf.Output, logits_dimension int64, optional ...BoostedTreesCalculateBestFeatureSplitAttr) (node_ids tf.Output, gains tf.Output, feature_dimensions tf.Output, thresholds tf.Output, left_node_contribs tf.Output, right_node_contribs tf.Output, split_with_default_directions tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"logits_dimension": logits_dimension}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesCalculateBestFeatureSplit",
+		Input: []tf.Input{
+			node_id_range, stats_summary, l1, l2, tree_complexity, min_node_weight,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3), op.Output(4), op.Output(5), op.Output(6)
+}
+
+// ModelDatasetAttr is an optional argument to ModelDataset.
+type ModelDatasetAttr func(optionalAttr)
+
+// ModelDatasetAlgorithm sets the optional algorithm attribute to value.
+// If not specified, defaults to 0
+func ModelDatasetAlgorithm(value int64) ModelDatasetAttr {
+	return func(m optionalAttr) {
+		m["algorithm"] = value
+	}
+}
+
+// ModelDatasetCpuBudget sets the optional cpu_budget attribute to value.
+// If not specified, defaults to 0
+func ModelDatasetCpuBudget(value int64) ModelDatasetAttr {
+	return func(m optionalAttr) {
+		m["cpu_budget"] = value
+	}
+}
+
+// Identity transformation that models performance.
+//
+// Identity transformation that models performance.
+//
+// Arguments:
+//	input_dataset: A variant tensor representing the input dataset.
+//
+//
+func ModelDataset(scope *Scope, input_dataset tf.Output, output_types []tf.DataType, output_shapes []tf.Shape, optional ...ModelDatasetAttr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ModelDataset",
+		Input: []tf.Input{
+			input_dataset,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns a list of tensors with the same shapes and contents as the input
+//
+// tensors.
+//
+// This op can be used to override the gradient for complicated functions. For
+// example, suppose y = f(x) and we wish to apply a custom function g for backprop
+// such that dx = g(dy). In Python,
+//
+// ```python
+// with tf.get_default_graph().gradient_override_map(
+//     {'IdentityN': 'OverrideGradientWithG'}):
+//   y, _ = identity_n([f(x), x])
+//
+// @tf.RegisterGradient('OverrideGradientWithG')
+// def ApplyG(op, dy, _):
+//   return [None, g(dy)]  # Do not backprop to f(x).
+// ```
+func IdentityN(scope *Scope, input []tf.Output) (output []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "IdentityN",
+		Input: []tf.Input{
+			tf.OutputList(input),
+		},
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if output, idx, err = makeOutputList(op, idx, "output"); err != nil {
+		scope.UpdateErr("IdentityN", err)
+		return
+	}
+	return output
+}
+
+// Returns true if and only if the given Optional variant has a value.
+func OptionalHasValue(scope *Scope, optional tf.Output) (has_value tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "OptionalHasValue",
+		Input: []tf.Input{
+			optional,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Constructs an Optional variant from a tuple of tensors.
+func OptionalFromValue(scope *Scope, components []tf.Output) (optional tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "OptionalFromValue",
+		Input: []tf.Input{
+			tf.OutputList(components),
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// OptimizeDatasetAttr is an optional argument to OptimizeDataset.
+type OptimizeDatasetAttr func(optionalAttr)
+
+// OptimizeDatasetOptimizationConfigs sets the optional optimization_configs attribute to value.
+// If not specified, defaults to <>
+func OptimizeDatasetOptimizationConfigs(value []string) OptimizeDatasetAttr {
+	return func(m optionalAttr) {
+		m["optimization_configs"] = value
+	}
+}
+
+// Creates a dataset by applying optimizations to `input_dataset`.
+//
+// Creates a dataset by applying optimizations to `input_dataset`.
+//
+// Arguments:
+//	input_dataset: A variant tensor representing the input dataset.
+//	optimizations: A `tf.string` vector `tf.Tensor` identifying optimizations to use.
+//
+//
+func OptimizeDataset(scope *Scope, input_dataset tf.Output, optimizations tf.Output, output_types []tf.DataType, output_shapes []tf.Shape, optional ...OptimizeDatasetAttr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "OptimizeDataset",
+		Input: []tf.Input{
+			input_dataset, optimizations,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Converts the given `resource_handle` representing an iterator to a string.
+//
+// Arguments:
+//	resource_handle: A handle to an iterator resource.
+//
+// Returns A string representation of the given handle.
+func IteratorToStringHandle(scope *Scope, resource_handle tf.Output) (string_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "IteratorToStringHandle",
+		Input: []tf.Input{
+			resource_handle,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Gets the next output from the given iterator.
+//
+// This operation is a synchronous version IteratorGetNext. It should only be used
+// in situations where the iterator does not block the calling thread, or where
+// the calling thread is not a member of the thread pool used to execute parallel
+// operations (e.g. in eager mode).
+func IteratorGetNextSync(scope *Scope, iterator tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (components []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "IteratorGetNextSync",
+		Input: []tf.Input{
+			iterator,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if components, idx, err = makeOutputList(op, idx, "components"); err != nil {
+		scope.UpdateErr("IteratorGetNextSync", err)
+		return
+	}
+	return components
+}
+
+// RaggedCountSparseOutputAttr is an optional argument to RaggedCountSparseOutput.
+type RaggedCountSparseOutputAttr func(optionalAttr)
+
+// RaggedCountSparseOutputMinlength sets the optional minlength attribute to value.
+//
+// value: Minimum value to count. Can be set to -1 for no minimum.
+// If not specified, defaults to -1
+//
+// REQUIRES: value >= -1
+func RaggedCountSparseOutputMinlength(value int64) RaggedCountSparseOutputAttr {
+	return func(m optionalAttr) {
+		m["minlength"] = value
+	}
+}
+
+// RaggedCountSparseOutputMaxlength sets the optional maxlength attribute to value.
+//
+// value: Maximum value to count. Can be set to -1 for no maximum.
+// If not specified, defaults to -1
+//
+// REQUIRES: value >= -1
+func RaggedCountSparseOutputMaxlength(value int64) RaggedCountSparseOutputAttr {
+	return func(m optionalAttr) {
+		m["maxlength"] = value
+	}
+}
+
+// Performs sparse-output bin counting for a ragged tensor input.
+//
+//   Counts the number of times each value occurs in the input.
+//
+// Arguments:
+//	splits: Tensor containing the row splits of the ragged tensor to count.
+//	values: Tensor containing values of the sparse tensor to count.
+//	weights: A Tensor of the same shape as indices containing per-index weight values.
+// May also be the empty tensor if no weights are used.
+//	binary_output: Whether to output the number of occurrences of each value or 1.
+//
+// Returns:
+//	output_indices: Indices tensor for the resulting sparse tensor object.
+//	output_values: Values tensor for the resulting sparse tensor object.
+//	output_dense_shape: Shape tensor for the resulting sparse tensor object.
+//   END
+//   }
+//   attr {
+//     name: "T"
+//     description: <<END
+// Dtype of the input values tensor.
+func RaggedCountSparseOutput(scope *Scope, splits tf.Output, values tf.Output, weights tf.Output, binary_output bool, optional ...RaggedCountSparseOutputAttr) (output_indices tf.Output, output_values tf.Output, output_dense_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"binary_output": binary_output}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RaggedCountSparseOutput",
+		Input: []tf.Input{
+			splits, values, weights,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Gets the next output from the given iterator .
+func IteratorGetNext(scope *Scope, iterator tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (components []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "IteratorGetNext",
+		Input: []tf.Input{
+			iterator,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if components, idx, err = makeOutputList(op, idx, "components"); err != nil {
+		scope.UpdateErr("IteratorGetNext", err)
+		return
+	}
+	return components
+}
+
+// Makes a new iterator from the given `dataset` and stores it in `iterator`.
+//
+// This operation may be executed multiple times. Each execution will reset the
+// iterator in `iterator` to the first element of `dataset`.
+//
+// Returns the created operation.
+func MakeIterator(scope *Scope, dataset tf.Output, iterator tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "MakeIterator",
+		Input: []tf.Input{
+			dataset, iterator,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// A container for an iterator resource.
+//
+// Arguments:
+//	handle: A handle to the iterator to delete.
+//	deleter: A variant deleter.
+//
+// Returns the created operation.
+func DeleteIterator(scope *Scope, handle tf.Output, deleter tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "DeleteIterator",
+		Input: []tf.Input{
+			handle, deleter,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Splits a tensor into `num_split` tensors along one dimension.
+//
+// Arguments:
+//	value: The tensor to split.
+//	size_splits: list containing the sizes of each output tensor along the split
+// dimension. Must sum to the dimension of value along split_dim.
+// Can contain one -1 indicating that dimension is to be inferred.
+//	axis: 0-D.  The dimension along which to split.  Must be in the range
+// `[-rank(value), rank(value))`.
+//
+//
+// Returns Tensors whose shape matches that of `value`
+// except along `axis`, where their sizes are
+// `size_splits[i]`.
+func SplitV(scope *Scope, value tf.Output, size_splits tf.Output, axis tf.Output, num_split int64) (output []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_split": num_split}
+	opspec := tf.OpSpec{
+		Type: "SplitV",
+		Input: []tf.Input{
+			value, size_splits, axis,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if output, idx, err = makeOutputList(op, idx, "output"); err != nil {
+		scope.UpdateErr("SplitV", err)
+		return
+	}
+	return output
+}
+
+// A container for an iterator resource.
+//
+// Returns:
+//	handle: A handle to the iterator that can be passed to a "MakeIterator" or
+// "IteratorGetNext" op. In contrast to Iterator, AnonymousIterator prevents
+// resource sharing by name, and does not keep a reference to the resource
+// container.
+//	deleter: A variant deleter that should be passed into the op that deletes the iterator.
+func AnonymousIteratorV2(scope *Scope, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output, deleter tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "AnonymousIteratorV2",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// A container for an iterator resource.
+//
+// Returns A handle to the iterator that can be passed to a "MakeIterator" or
+// "IteratorGetNext" op. In contrast to Iterator, AnonymousIterator prevents
+// resource sharing by name, and does not keep a reference to the resource
+// container.
+func AnonymousIterator(scope *Scope, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "AnonymousIterator",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// A container for an iterator resource.
+//
+// Returns A handle to the iterator that can be passed to a "MakeIterator"
+// or "IteratorGetNext" op.
+func Iterator(scope *Scope, shared_name string, container string, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"shared_name": shared_name, "container": container, "output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "Iterator",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that emits the records from one or more binary files.
+//
+// Arguments:
+//	filenames: A scalar or a vector containing the name(s) of the file(s) to be
+// read.
+//	header_bytes: A scalar representing the number of bytes to skip at the
+// beginning of a file.
+//	record_bytes: A scalar representing the number of bytes in each record.
+//	footer_bytes: A scalar representing the number of bytes to skip at the end
+// of a file.
+//	buffer_size: A scalar representing the number of bytes to buffer. Must be > 0.
+func FixedLengthRecordDataset(scope *Scope, filenames tf.Output, header_bytes tf.Output, record_bytes tf.Output, footer_bytes tf.Output, buffer_size tf.Output) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "FixedLengthRecordDataset",
+		Input: []tf.Input{
+			filenames, header_bytes, record_bytes, footer_bytes, buffer_size,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that emits the lines of one or more text files.
+//
+// Arguments:
+//	filenames: A scalar or a vector containing the name(s) of the file(s) to be
+// read.
+//	compression_type: A scalar containing either (i) the empty string (no
+// compression), (ii) "ZLIB", or (iii) "GZIP".
+//	buffer_size: A scalar containing the number of bytes to buffer.
+func TextLineDataset(scope *Scope, filenames tf.Output, compression_type tf.Output, buffer_size tf.Output) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TextLineDataset",
+		Input: []tf.Input{
+			filenames, compression_type, buffer_size,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// A container for an iterator resource.
+//
+// Arguments:
+//	multi_device_iterator: A handle to the multi device iterator to delete.
+//	iterators: A list of iterator handles (unused). This is added so that automatic control dependencies get added during function tracing that ensure this op runs after all the dependent iterators are deleted.
+//	deleter: A variant deleter.
+//
+// Returns the created operation.
+func DeleteMultiDeviceIterator(scope *Scope, multi_device_iterator tf.Output, iterators []tf.Output, deleter tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "DeleteMultiDeviceIterator",
+		Input: []tf.Input{
+			multi_device_iterator, tf.OutputList(iterators), deleter,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Creates a dataset with a range of values. Corresponds to python's xrange.
+//
+// Arguments:
+//	start: corresponds to start in python's xrange().
+//	stop: corresponds to stop in python's xrange().
+//	step: corresponds to step in python's xrange().
+//
+//
+func RangeDataset(scope *Scope, start tf.Output, stop tf.Output, step tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "RangeDataset",
+		Input: []tf.Input{
+			start, stop, step,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that batches and pads `batch_size` elements from the input.
+//
+// Arguments:
+//
+//	batch_size: A scalar representing the number of elements to accumulate in a
+// batch.
+//	padded_shapes: A list of int64 tensors representing the desired padded shapes
+// of the corresponding output components. These shapes may be partially
+// specified, using `-1` to indicate that a particular dimension should be
+// padded to the maximum size of all batch elements.
+//	padding_values: A list of scalars containing the padding value to use for
+// each of the outputs.
+//
+func PaddedBatchDataset(scope *Scope, input_dataset tf.Output, batch_size tf.Output, padded_shapes []tf.Output, padding_values []tf.Output, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "PaddedBatchDataset",
+		Input: []tf.Input{
+			input_dataset, batch_size, tf.OutputList(padded_shapes), tf.OutputList(padding_values),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// BatchDatasetV2Attr is an optional argument to BatchDatasetV2.
+type BatchDatasetV2Attr func(optionalAttr)
+
+// BatchDatasetV2ParallelCopy sets the optional parallel_copy attribute to value.
+// If not specified, defaults to false
+func BatchDatasetV2ParallelCopy(value bool) BatchDatasetV2Attr {
+	return func(m optionalAttr) {
+		m["parallel_copy"] = value
+	}
+}
+
+// Creates a dataset that batches `batch_size` elements from `input_dataset`.
+//
+// Arguments:
+//
+//	batch_size: A scalar representing the number of elements to accumulate in a batch.
+//	drop_remainder: A scalar representing whether the last batch should be dropped in case its size
+// is smaller than desired.
+//
+//
+func BatchDatasetV2(scope *Scope, input_dataset tf.Output, batch_size tf.Output, drop_remainder tf.Output, output_types []tf.DataType, output_shapes []tf.Shape, optional ...BatchDatasetV2Attr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "BatchDatasetV2",
+		Input: []tf.Input{
+			input_dataset, batch_size, drop_remainder,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ShuffleDatasetAttr is an optional argument to ShuffleDataset.
+type ShuffleDatasetAttr func(optionalAttr)
+
+// ShuffleDatasetReshuffleEachIteration sets the optional reshuffle_each_iteration attribute to value.
+//
+// value: If true, each iterator over this dataset will be given
+// a different pseudorandomly generated seed, based on a sequence seeded by the
+// `seed` and `seed2` inputs. If false, each iterator will be given the same
+// seed, and repeated iteration over this dataset will yield the exact same
+// sequence of results.
+// If not specified, defaults to true
+func ShuffleDatasetReshuffleEachIteration(value bool) ShuffleDatasetAttr {
+	return func(m optionalAttr) {
+		m["reshuffle_each_iteration"] = value
+	}
+}
+
+// Creates a dataset that shuffles elements from `input_dataset` pseudorandomly.
+//
+// Arguments:
+//
+//	buffer_size: The number of output elements to buffer in an iterator over
+// this dataset. Compare with the `min_after_dequeue` attr when creating a
+// `RandomShuffleQueue`.
+//	seed: A scalar seed for the random number generator. If either `seed` or
+// `seed2` is set to be non-zero, the random number generator is seeded
+// by the given seed.  Otherwise, a random seed is used.
+//	seed2: A second scalar seed to avoid seed collision.
+//
+//
+func ShuffleDataset(scope *Scope, input_dataset tf.Output, buffer_size tf.Output, seed tf.Output, seed2 tf.Output, output_types []tf.DataType, output_shapes []tf.Shape, optional ...ShuffleDatasetAttr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ShuffleDataset",
+		Input: []tf.Input{
+			input_dataset, buffer_size, seed, seed2,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset containing elements of first component of `input_dataset` having true in the last component.
+func FilterByLastComponentDataset(scope *Scope, input_dataset tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "FilterByLastComponentDataset",
+		Input: []tf.Input{
+			input_dataset,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// PrefetchDatasetAttr is an optional argument to PrefetchDataset.
+type PrefetchDatasetAttr func(optionalAttr)
+
+// PrefetchDatasetSlackPeriod sets the optional slack_period attribute to value.
+// If not specified, defaults to 0
+func PrefetchDatasetSlackPeriod(value int64) PrefetchDatasetAttr {
+	return func(m optionalAttr) {
+		m["slack_period"] = value
+	}
+}
+
+// PrefetchDatasetLegacyAutotune sets the optional legacy_autotune attribute to value.
+// If not specified, defaults to true
+func PrefetchDatasetLegacyAutotune(value bool) PrefetchDatasetAttr {
+	return func(m optionalAttr) {
+		m["legacy_autotune"] = value
+	}
+}
+
+// Creates a dataset that asynchronously prefetches elements from `input_dataset`.
+//
+// Arguments:
+//
+//	buffer_size: The maximum number of elements to buffer in an iterator over
+// this dataset.
+//
+//
+func PrefetchDataset(scope *Scope, input_dataset tf.Output, buffer_size tf.Output, output_types []tf.DataType, output_shapes []tf.Shape, optional ...PrefetchDatasetAttr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "PrefetchDataset",
+		Input: []tf.Input{
+			input_dataset, buffer_size,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Forwards the input to the output.
+//
+// This operator represents the loop termination condition used by the
+// "pivot" switches of a loop.
+//
+// Arguments:
+//	input: A boolean scalar, representing the branch predicate of the Switch op.
+//
+// Returns The same tensor as `input`.
+func LoopCond(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "LoopCond",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that skips `count` elements from the `input_dataset`.
+//
+// Arguments:
+//
+//	count: A scalar representing the number of elements from the `input_dataset`
+// that should be skipped.  If count is -1, skips everything.
+//
+//
+func SkipDataset(scope *Scope, input_dataset tf.Output, count tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "SkipDataset",
+		Input: []tf.Input{
+			input_dataset, count,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that emits the outputs of `input_dataset` `count` times.
+//
+// Arguments:
+//
+//	count: A scalar representing the number of times that `input_dataset` should
+// be repeated. A value of `-1` indicates that it should be repeated infinitely.
+//
+//
+func RepeatDataset(scope *Scope, input_dataset tf.Output, count tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "RepeatDataset",
+		Input: []tf.Input{
+			input_dataset, count,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// UnpackAttr is an optional argument to Unpack.
+type UnpackAttr func(optionalAttr)
+
+// UnpackAxis sets the optional axis attribute to value.
+//
+// value: Dimension along which to unpack.  Negative values wrap around, so the
+// valid range is `[-R, R)`.
+// If not specified, defaults to 0
+func UnpackAxis(value int64) UnpackAttr {
+	return func(m optionalAttr) {
+		m["axis"] = value
+	}
+}
+
+// Unpacks a given dimension of a rank-`R` tensor into `num` rank-`(R-1)` tensors.
+//
+// Unpacks `num` tensors from `value` by chipping it along the `axis` dimension.
+// For example, given a tensor of shape `(A, B, C, D)`;
+//
+// If `axis == 0` then the i'th tensor in `output` is the slice `value[i, :, :, :]`
+//   and each tensor in `output` will have shape `(B, C, D)`. (Note that the
+//   dimension unpacked along is gone, unlike `split`).
+//
+// If `axis == 1` then the i'th tensor in `output` is the slice `value[:, i, :, :]`
+//   and each tensor in `output` will have shape `(A, C, D)`.
+// Etc.
+//
+// This is the opposite of `pack`.
+//
+// Arguments:
+//	value: 1-D or higher, with `axis` dimension size equal to `num`.
+//
+//
+// Returns The list of tensors unpacked from `value`.
+func Unpack(scope *Scope, value tf.Output, num int64, optional ...UnpackAttr) (output []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num": num}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Unpack",
+		Input: []tf.Input{
+			value,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if output, idx, err = makeOutputList(op, idx, "output"); err != nil {
+		scope.UpdateErr("Unpack", err)
+		return
+	}
+	return output
+}
+
+// Creates a dataset that concatenates `input_dataset` with `another_dataset`.
+func ConcatenateDataset(scope *Scope, input_dataset tf.Output, another_dataset tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "ConcatenateDataset",
+		Input: []tf.Input{
+			input_dataset, another_dataset,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// A placeholder op for a value that will be fed into the computation.
+//
+// DEPRECATED at GraphDef version 23: Placeholder now behaves the same as PlaceholderV2.
+//
+// N.B. This operation will fail with an error if it is executed. It is
+// intended as a way to represent a value that will always be fed, and to
+// provide attrs that enable the fed value to be checked at runtime.
+//
+// Arguments:
+//	dtype: The type of elements in the tensor.
+//	shape: The shape of the tensor. The shape can be any partially-specified
+// shape.  To be unconstrained, pass in a shape with unknown rank.
+//
+// Returns A placeholder tensor that must be replaced using the feed mechanism.
+func PlaceholderV2(scope *Scope, dtype tf.DataType, shape tf.Shape) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype, "shape": shape}
+	opspec := tf.OpSpec{
+		Type: "PlaceholderV2",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RandomShuffleQueueV2Attr is an optional argument to RandomShuffleQueueV2.
+type RandomShuffleQueueV2Attr func(optionalAttr)
+
+// RandomShuffleQueueV2Shapes sets the optional shapes attribute to value.
+//
+// value: The shape of each component in a value. The length of this attr must
+// be either 0 or the same as the length of component_types. If the length of
+// this attr is 0, the shapes of queue elements are not constrained, and
+// only one element may be dequeued at a time.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func RandomShuffleQueueV2Shapes(value []tf.Shape) RandomShuffleQueueV2Attr {
+	return func(m optionalAttr) {
+		m["shapes"] = value
+	}
+}
+
+// RandomShuffleQueueV2Capacity sets the optional capacity attribute to value.
+//
+// value: The upper bound on the number of elements in this queue.
+// Negative numbers mean no limit.
+// If not specified, defaults to -1
+func RandomShuffleQueueV2Capacity(value int64) RandomShuffleQueueV2Attr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// RandomShuffleQueueV2MinAfterDequeue sets the optional min_after_dequeue attribute to value.
+//
+// value: Dequeue will block unless there would be this
+// many elements after the dequeue or the queue is closed. This
+// ensures a minimum level of mixing of elements.
+// If not specified, defaults to 0
+func RandomShuffleQueueV2MinAfterDequeue(value int64) RandomShuffleQueueV2Attr {
+	return func(m optionalAttr) {
+		m["min_after_dequeue"] = value
+	}
+}
+
+// RandomShuffleQueueV2Seed sets the optional seed attribute to value.
+//
+// value: If either seed or seed2 is set to be non-zero, the random number
+// generator is seeded by the given seed.  Otherwise, a random seed is used.
+// If not specified, defaults to 0
+func RandomShuffleQueueV2Seed(value int64) RandomShuffleQueueV2Attr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// RandomShuffleQueueV2Seed2 sets the optional seed2 attribute to value.
+//
+// value: A second seed to avoid seed collision.
+// If not specified, defaults to 0
+func RandomShuffleQueueV2Seed2(value int64) RandomShuffleQueueV2Attr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// RandomShuffleQueueV2Container sets the optional container attribute to value.
+//
+// value: If non-empty, this queue is placed in the given container.
+// Otherwise, a default container is used.
+// If not specified, defaults to ""
+func RandomShuffleQueueV2Container(value string) RandomShuffleQueueV2Attr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// RandomShuffleQueueV2SharedName sets the optional shared_name attribute to value.
+//
+// value: If non-empty, this queue will be shared under the given name
+// across multiple sessions.
+// If not specified, defaults to ""
+func RandomShuffleQueueV2SharedName(value string) RandomShuffleQueueV2Attr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// A queue that randomizes the order of elements.
+//
+// Arguments:
+//	component_types: The type of each component in a value.
+//
+// Returns The handle to the queue.
+func RandomShuffleQueueV2(scope *Scope, component_types []tf.DataType, optional ...RandomShuffleQueueV2Attr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"component_types": component_types}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RandomShuffleQueueV2",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that splits a SparseTensor into elements row-wise.
+func SparseTensorSliceDataset(scope *Scope, indices tf.Output, values tf.Output, dense_shape tf.Output) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseTensorSliceDataset",
+		Input: []tf.Input{
+			indices, values, dense_shape,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that emits `components` as a tuple of tensors once.
+func TensorDataset(scope *Scope, components []tf.Output, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "TensorDataset",
+		Input: []tf.Input{
+			tf.OutputList(components),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// QueueCloseV2Attr is an optional argument to QueueCloseV2.
+type QueueCloseV2Attr func(optionalAttr)
+
+// QueueCloseV2CancelPendingEnqueues sets the optional cancel_pending_enqueues attribute to value.
+//
+// value: If true, all pending enqueue requests that are
+// blocked on the given queue will be canceled.
+// If not specified, defaults to false
+func QueueCloseV2CancelPendingEnqueues(value bool) QueueCloseV2Attr {
+	return func(m optionalAttr) {
+		m["cancel_pending_enqueues"] = value
+	}
+}
+
+// Closes the given queue.
+//
+// This operation signals that no more elements will be enqueued in the
+// given queue. Subsequent Enqueue(Many) operations will fail.
+// Subsequent Dequeue(Many) operations will continue to succeed if
+// sufficient elements remain in the queue. Subsequent Dequeue(Many)
+// operations that would block will fail immediately.
+//
+// Arguments:
+//	handle: The handle to a queue.
+//
+// Returns the created operation.
+func QueueCloseV2(scope *Scope, handle tf.Output, optional ...QueueCloseV2Attr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QueueCloseV2",
+		Input: []tf.Input{
+			handle,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// DebugIdentityV2Attr is an optional argument to DebugIdentityV2.
+type DebugIdentityV2Attr func(optionalAttr)
+
+// DebugIdentityV2TfdbgContextId sets the optional tfdbg_context_id attribute to value.
+//
+// value: A tfdbg-generated ID for the context that the op belongs to,
+//   e.g., a concrete compiled tf.function.
+// If not specified, defaults to ""
+func DebugIdentityV2TfdbgContextId(value string) DebugIdentityV2Attr {
+	return func(m optionalAttr) {
+		m["tfdbg_context_id"] = value
+	}
+}
+
+// DebugIdentityV2OpName sets the optional op_name attribute to value.
+//
+// value: Optional. Name of the op that the debug op is concerned with.
+//   Used only for single-tensor trace.
+// If not specified, defaults to ""
+func DebugIdentityV2OpName(value string) DebugIdentityV2Attr {
+	return func(m optionalAttr) {
+		m["op_name"] = value
+	}
+}
+
+// DebugIdentityV2OutputSlot sets the optional output_slot attribute to value.
+//
+// value: Optional. Output slot index of the tensor that the debug op
+//   is concerned with. Used only for single-tensor trace.
+// If not specified, defaults to -1
+func DebugIdentityV2OutputSlot(value int64) DebugIdentityV2Attr {
+	return func(m optionalAttr) {
+		m["output_slot"] = value
+	}
+}
+
+// DebugIdentityV2TensorDebugMode sets the optional tensor_debug_mode attribute to value.
+//
+// value: TensorDebugMode enum value. See debug_event.proto for details.
+// If not specified, defaults to -1
+func DebugIdentityV2TensorDebugMode(value int64) DebugIdentityV2Attr {
+	return func(m optionalAttr) {
+		m["tensor_debug_mode"] = value
+	}
+}
+
+// DebugIdentityV2DebugUrls sets the optional debug_urls attribute to value.
+//
+// value: List of URLs to debug targets, e.g., file:///foo/tfdbg_dump.
+// If not specified, defaults to <>
+func DebugIdentityV2DebugUrls(value []string) DebugIdentityV2Attr {
+	return func(m optionalAttr) {
+		m["debug_urls"] = value
+	}
+}
+
+// DebugIdentityV2CircularBufferSize sets the optional circular_buffer_size attribute to value.
+// If not specified, defaults to 1000
+func DebugIdentityV2CircularBufferSize(value int64) DebugIdentityV2Attr {
+	return func(m optionalAttr) {
+		m["circular_buffer_size"] = value
+	}
+}
+
+// Debug Identity V2 Op.
+//
+// Provides an identity mapping from input to output, while writing the content of
+// the input tensor by calling DebugEventsWriter.
+//
+// The semantics of the input tensor depends on tensor_debug_mode. In typical
+// usage, the input tensor comes directly from the user computation only when
+// graph_debug_mode is FULL_TENSOR (see protobuf/debug_event.proto for a
+// list of all the possible values of graph_debug_mode). For the other debug modes,
+// the input tensor should be produced by an additional op or subgraph that
+// computes summary information about one or more tensors.
+//
+// Arguments:
+//	input: Input tensor, non-Reference type
+func DebugIdentityV2(scope *Scope, input tf.Output, optional ...DebugIdentityV2Attr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DebugIdentityV2",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// DebugNanCountAttr is an optional argument to DebugNanCount.
+type DebugNanCountAttr func(optionalAttr)
+
+// DebugNanCountDeviceName sets the optional device_name attribute to value.
+// If not specified, defaults to ""
+func DebugNanCountDeviceName(value string) DebugNanCountAttr {
+	return func(m optionalAttr) {
+		m["device_name"] = value
+	}
+}
+
+// DebugNanCountTensorName sets the optional tensor_name attribute to value.
+//
+// value: Name of the input tensor.
+// If not specified, defaults to ""
+func DebugNanCountTensorName(value string) DebugNanCountAttr {
+	return func(m optionalAttr) {
+		m["tensor_name"] = value
+	}
+}
+
+// DebugNanCountDebugUrls sets the optional debug_urls attribute to value.
+//
+// value: List of URLs to debug targets, e.g.,
+//   file:///foo/tfdbg_dump, grpc:://localhost:11011.
+// If not specified, defaults to <>
+func DebugNanCountDebugUrls(value []string) DebugNanCountAttr {
+	return func(m optionalAttr) {
+		m["debug_urls"] = value
+	}
+}
+
+// DebugNanCountGatedGrpc sets the optional gated_grpc attribute to value.
+//
+// value:  Whether this op will be gated. If any of the debug_urls of this
+//   debug node is of the grpc:// scheme, when the value of this attribute is set
+//   to True, the data will not actually be sent via the grpc stream unless this
+//   debug op has been enabled at the debug_url. If all of the debug_urls of this
+//   debug node are of the grpc:// scheme and the debug op is enabled at none of
+//   them, the output will be an empty Tensor.
+// If not specified, defaults to false
+func DebugNanCountGatedGrpc(value bool) DebugNanCountAttr {
+	return func(m optionalAttr) {
+		m["gated_grpc"] = value
+	}
+}
+
+// Debug NaN Value Counter Op.
+//
+// Counts number of NaNs in the input tensor, for debugging.
+//
+// Arguments:
+//	input: Input tensor, non-Reference type.
+func DebugNanCount(scope *Scope, input tf.Output, optional ...DebugNanCountAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DebugNanCount",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// DebugIdentityAttr is an optional argument to DebugIdentity.
+type DebugIdentityAttr func(optionalAttr)
+
+// DebugIdentityDeviceName sets the optional device_name attribute to value.
+//
+// value: Name of the device on which the tensor resides.
+// If not specified, defaults to ""
+func DebugIdentityDeviceName(value string) DebugIdentityAttr {
+	return func(m optionalAttr) {
+		m["device_name"] = value
+	}
+}
+
+// DebugIdentityTensorName sets the optional tensor_name attribute to value.
+//
+// value: Name of the input tensor.
+// If not specified, defaults to ""
+func DebugIdentityTensorName(value string) DebugIdentityAttr {
+	return func(m optionalAttr) {
+		m["tensor_name"] = value
+	}
+}
+
+// DebugIdentityDebugUrls sets the optional debug_urls attribute to value.
+//
+// value: List of URLs to debug targets, e.g.,
+//   file:///foo/tfdbg_dump, grpc:://localhost:11011
+// If not specified, defaults to <>
+func DebugIdentityDebugUrls(value []string) DebugIdentityAttr {
+	return func(m optionalAttr) {
+		m["debug_urls"] = value
+	}
+}
+
+// DebugIdentityGatedGrpc sets the optional gated_grpc attribute to value.
+//
+// value: Whether this op will be gated. If any of the debug_urls of this
+//   debug node is of the grpc:// scheme, when the value of this attribute is set
+//   to True, the data will not actually be sent via the grpc stream unless this
+//   debug op has been enabled at the debug_url. If all of the debug_urls of this
+//   debug node are of the grpc:// scheme and the debug op is enabled at none of
+//   them, the output will be an empty Tensor.
+// If not specified, defaults to false
+func DebugIdentityGatedGrpc(value bool) DebugIdentityAttr {
+	return func(m optionalAttr) {
+		m["gated_grpc"] = value
+	}
+}
+
+// Provides an identity mapping of the non-Ref type input tensor for debugging.
+//
+// Provides an identity mapping of the non-Ref type input tensor for debugging.
+//
+// Arguments:
+//	input: Input tensor, non-Reference type
+func DebugIdentity(scope *Scope, input tf.Output, optional ...DebugIdentityAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DebugIdentity",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Aggregates the summary of accumulated stats for the batch.
+//
+// The summary stats contains gradients and hessians accumulated for each node, bucket and dimension id.
+//
+// Arguments:
+//	node_ids: int32; Rank 1 Tensor containing node ids for each example, shape [batch_size].
+//	gradients: float32; Rank 2 Tensor (shape=[batch_size, logits_dimension]) with gradients for each example.
+//	hessians: float32; Rank 2 Tensor (shape=[batch_size, hessian_dimension]) with hessians for each example.
+//	feature_indices: int32; Rank 2 indices of feature sparse Tensors (shape=[number of sparse entries, 2]).
+// Number of sparse entries across all instances from the batch. The first value is
+// the index of the instance, the second is dimension of the feature. The second axis
+// can only have 2 values, i.e., the input dense version of Tensor can only be matrix.
+//	feature_values: int32; Rank 1 values of feature sparse Tensors (shape=[number of sparse entries]).
+// Number of sparse entries across all instances from the batch. The first value is
+// the index of the instance, the second is dimension of the feature.
+//	feature_shape: int32; Rank 1 dense shape of feature sparse Tensors (shape=[2]).
+// The first axis can only have 2 values, [batch_size, feature_dimension].
+//	max_splits: int; the maximum number of splits possible in the whole tree.
+//	num_buckets: int; equals to the maximum possible value of bucketized feature + 1.
+//
+// Returns:
+//	stats_summary_indices: int32; Rank 2 indices of summary sparse Tensors (shape=[number of non zero statistics, 4])
+// The second axis can only be 4 including node id, feature dimension, bucket id, and statistics_dimension.
+// statistics_dimension = logits_dimension + hessian_dimension.
+//	stats_summary_values: output Rank 1 Tensor (shape=[number of non zero statistics])
+//	stats_summary_shape: output Rank 1 Tensor (shape=[4])
+// The tensor has following 4 values: [max_splits, feature_dimension, num_buckets, statistics_dimension],
+// where statistics_dimension = gradient_dimension + hessian_dimension. gradient_dimension
+// is the same as label_dimension, i.e., the output space. hessian_dimension can be the same
+// as logits dimension when diagonal hessian is used, or label_dimension^2 when full
+// hessian is used.
+func BoostedTreesSparseAggregateStats(scope *Scope, node_ids tf.Output, gradients tf.Output, hessians tf.Output, feature_indices tf.Output, feature_values tf.Output, feature_shape tf.Output, max_splits int64, num_buckets int64) (stats_summary_indices tf.Output, stats_summary_values tf.Output, stats_summary_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"max_splits": max_splits, "num_buckets": num_buckets}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesSparseAggregateStats",
+		Input: []tf.Input{
+			node_ids, gradients, hessians, feature_indices, feature_values, feature_shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// DecodeProtoV2Attr is an optional argument to DecodeProtoV2.
+type DecodeProtoV2Attr func(optionalAttr)
+
+// DecodeProtoV2DescriptorSource sets the optional descriptor_source attribute to value.
+//
+// value: Either the special value `local://` or a path to a file containing
+// a serialized `FileDescriptorSet`.
+// If not specified, defaults to "local://"
+func DecodeProtoV2DescriptorSource(value string) DecodeProtoV2Attr {
+	return func(m optionalAttr) {
+		m["descriptor_source"] = value
+	}
+}
+
+// DecodeProtoV2MessageFormat sets the optional message_format attribute to value.
+//
+// value: Either `binary` or `text`.
+// If not specified, defaults to "binary"
+func DecodeProtoV2MessageFormat(value string) DecodeProtoV2Attr {
+	return func(m optionalAttr) {
+		m["message_format"] = value
+	}
+}
+
+// DecodeProtoV2Sanitize sets the optional sanitize attribute to value.
+//
+// value: Whether to sanitize the result or not.
+// If not specified, defaults to false
+func DecodeProtoV2Sanitize(value bool) DecodeProtoV2Attr {
+	return func(m optionalAttr) {
+		m["sanitize"] = value
+	}
+}
+
+// The op extracts fields from a serialized protocol buffers message into tensors.
+//
+// The `decode_proto` op extracts fields from a serialized protocol buffers
+// message into tensors.  The fields in `field_names` are decoded and converted
+// to the corresponding `output_types` if possible.
+//
+// A `message_type` name must be provided to give context for the field names.
+// The actual message descriptor can be looked up either in the linked-in
+// descriptor pool or a filename provided by the caller using the
+// `descriptor_source` attribute.
+//
+// Each output tensor is a dense tensor. This means that it is padded to hold
+// the largest number of repeated elements seen in the input minibatch. (The
+// shape is also padded by one to prevent zero-sized dimensions). The actual
+// repeat counts for each example in the minibatch can be found in the `sizes`
+// output. In many cases the output of `decode_proto` is fed immediately into
+// tf.squeeze if missing values are not a concern. When using tf.squeeze, always
+// pass the squeeze dimension explicitly to avoid surprises.
+//
+// For the most part, the mapping between Proto field types and TensorFlow dtypes
+// is straightforward. However, there are a few special cases:
+//
+// - A proto field that contains a submessage or group can only be converted
+// to `DT_STRING` (the serialized submessage). This is to reduce the complexity
+// of the API. The resulting string can be used as input to another instance of
+// the decode_proto op.
+//
+// - TensorFlow lacks support for unsigned integers. The ops represent uint64
+// types as a `DT_INT64` with the same twos-complement bit pattern (the obvious
+// way). Unsigned int32 values can be represented exactly by specifying type
+// `DT_INT64`, or using twos-complement if the caller specifies `DT_INT32` in
+// the `output_types` attribute.
+//
+// Both binary and text proto serializations are supported, and can be
+// chosen using the `format` attribute.
+//
+// The `descriptor_source` attribute selects the source of protocol
+// descriptors to consult when looking up `message_type`. This may be:
+//
+// - An empty string  or "local://", in which case protocol descriptors are
+// created for C++ (not Python) proto definitions linked to the binary.
+//
+// - A file, in which case protocol descriptors are created from the file,
+// which is expected to contain a `FileDescriptorSet` serialized as a string.
+// NOTE: You can build a `descriptor_source` file using the `--descriptor_set_out`
+// and `--include_imports` options to the protocol compiler `protoc`.
+//
+// - A "bytes://<bytes>", in which protocol descriptors are created from `<bytes>`,
+// which is expected to be a `FileDescriptorSet` serialized as a string.
+//
+// Arguments:
+//	bytes: Tensor of serialized protos with shape `batch_shape`.
+//	message_type: Name of the proto message type to decode.
+//	field_names: List of strings containing proto field names. An extension field can be decoded
+// by using its full name, e.g. EXT_PACKAGE.EXT_FIELD_NAME.
+//	output_types: List of TF types to use for the respective field in field_names.
+//
+// Returns:
+//	sizes: Tensor of int32 with shape `[batch_shape, len(field_names)]`.
+// Each entry is the number of values found for the corresponding field.
+// Optional fields may have 0 or 1 values.
+//	values: List of tensors containing values for the corresponding field.
+// `values[i]` has datatype `output_types[i]`
+// and shape `[batch_shape, max(sizes[...,i])]`.
+func DecodeProtoV2(scope *Scope, bytes tf.Output, message_type string, field_names []string, output_types []tf.DataType, optional ...DecodeProtoV2Attr) (sizes tf.Output, values []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"message_type": message_type, "field_names": field_names, "output_types": output_types}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DecodeProtoV2",
+		Input: []tf.Input{
+			bytes,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	sizes = op.Output(idx)
+	if values, idx, err = makeOutputList(op, idx, "values"); err != nil {
+		scope.UpdateErr("DecodeProtoV2", err)
+		return
+	}
+	return sizes, values
+}
+
+// Output the logits for the given input data
+//
+// Arguments:
+//	tree_handle: Handle to the tree resource.
+//	dense_features: Rank 2 dense features tensor.
+//	logits_dimension: Scalar, dimension of the logits.
+//
+// Returns The logits predictions from the tree for each instance in the batch.
+func TensorForestTreePredict(scope *Scope, tree_handle tf.Output, dense_features tf.Output, logits_dimension int64) (logits tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"logits_dimension": logits_dimension}
+	opspec := tf.OpSpec{
+		Type: "TensorForestTreePredict",
+		Input: []tf.Input{
+			tree_handle, dense_features,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// EncodeProtoAttr is an optional argument to EncodeProto.
+type EncodeProtoAttr func(optionalAttr)
+
+// EncodeProtoDescriptorSource sets the optional descriptor_source attribute to value.
+// If not specified, defaults to "local://"
+func EncodeProtoDescriptorSource(value string) EncodeProtoAttr {
+	return func(m optionalAttr) {
+		m["descriptor_source"] = value
+	}
+}
+
+// The op serializes protobuf messages provided in the input tensors.
+//
+// The types of the tensors in `values` must match the schema for the fields
+// specified in `field_names`. All the tensors in `values` must have a common
+// shape prefix, *batch_shape*.
+//
+// The `sizes` tensor specifies repeat counts for each field.  The repeat count
+// (last dimension) of a each tensor in `values` must be greater than or equal
+// to corresponding repeat count in `sizes`.
+//
+// A `message_type` name must be provided to give context for the field names.
+// The actual message descriptor can be looked up either in the linked-in
+// descriptor pool or a filename provided by the caller using the
+// `descriptor_source` attribute.
+//
+// For the most part, the mapping between Proto field types and TensorFlow dtypes
+// is straightforward. However, there are a few special cases:
+//
+// - A proto field that contains a submessage or group can only be converted
+// to `DT_STRING` (the serialized submessage). This is to reduce the complexity
+// of the API. The resulting string can be used as input to another instance of
+// the decode_proto op.
+//
+// - TensorFlow lacks support for unsigned integers. The ops represent uint64
+// types as a `DT_INT64` with the same twos-complement bit pattern (the obvious
+// way). Unsigned int32 values can be represented exactly by specifying type
+// `DT_INT64`, or using twos-complement if the caller specifies `DT_INT32` in
+// the `output_types` attribute.
+//
+// The `descriptor_source` attribute selects the source of protocol
+// descriptors to consult when looking up `message_type`. This may be:
+//
+// - An empty string  or "local://", in which case protocol descriptors are
+// created for C++ (not Python) proto definitions linked to the binary.
+//
+// - A file, in which case protocol descriptors are created from the file,
+// which is expected to contain a `FileDescriptorSet` serialized as a string.
+// NOTE: You can build a `descriptor_source` file using the `--descriptor_set_out`
+// and `--include_imports` options to the protocol compiler `protoc`.
+//
+// - A "bytes://<bytes>", in which protocol descriptors are created from `<bytes>`,
+// which is expected to be a `FileDescriptorSet` serialized as a string.
+//
+// Arguments:
+//	sizes: Tensor of int32 with shape `[batch_shape, len(field_names)]`.
+//	values: List of tensors containing values for the corresponding field.
+//	field_names: List of strings containing proto field names.
+//	message_type: Name of the proto message type to decode.
+//
+// Returns Tensor of serialized protos with shape `batch_shape`.
+func EncodeProto(scope *Scope, sizes tf.Output, values []tf.Output, field_names []string, message_type string, optional ...EncodeProtoAttr) (bytes tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"field_names": field_names, "message_type": message_type}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "EncodeProto",
+		Input: []tf.Input{
+			sizes, tf.OutputList(values),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Registers a dataset with the tf.data service.
+func RegisterDataset(scope *Scope, dataset tf.Output, address tf.Output, protocol tf.Output, external_state_policy int64) (dataset_id tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"external_state_policy": external_state_policy}
+	opspec := tf.OpSpec{
+		Type: "RegisterDataset",
+		Input: []tf.Input{
+			dataset, address, protocol,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// DataServiceDatasetAttr is an optional argument to DataServiceDataset.
+type DataServiceDatasetAttr func(optionalAttr)
+
+// DataServiceDatasetTaskRefreshIntervalHintMs sets the optional task_refresh_interval_hint_ms attribute to value.
+// If not specified, defaults to -1
+func DataServiceDatasetTaskRefreshIntervalHintMs(value int64) DataServiceDatasetAttr {
+	return func(m optionalAttr) {
+		m["task_refresh_interval_hint_ms"] = value
+	}
+}
+
+// Creates a dataset that reads data from the tf.data service.
+func DataServiceDataset(scope *Scope, dataset_id tf.Output, processing_mode tf.Output, address tf.Output, protocol tf.Output, job_name tf.Output, max_outstanding_requests tf.Output, iteration_counter tf.Output, output_types []tf.DataType, output_shapes []tf.Shape, optional ...DataServiceDatasetAttr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DataServiceDataset",
+		Input: []tf.Input{
+			dataset_id, processing_mode, address, protocol, job_name, max_outstanding_requests, iteration_counter,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that contains the unique elements of `input_dataset`.
+func UniqueDataset(scope *Scope, input_dataset tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "UniqueDataset",
+		Input: []tf.Input{
+			input_dataset,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// A dataset that splits the elements of its input into multiple elements.
+func ExperimentalUnbatchDataset(scope *Scope, input_dataset tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalUnbatchDataset",
+		Input: []tf.Input{
+			input_dataset,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// A dataset that splits the elements of its input into multiple elements.
+func UnbatchDataset(scope *Scope, input_dataset tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "UnbatchDataset",
+		Input: []tf.Input{
+			input_dataset,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that uses a custom thread pool to compute `input_dataset`.
+//
+// Arguments:
+//
+//	thread_pool: A resource produced by the ThreadPoolHandle op.
+//
+//
+func ExperimentalThreadPoolDataset(scope *Scope, input_dataset tf.Output, thread_pool tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalThreadPoolDataset",
+		Input: []tf.Input{
+			input_dataset, thread_pool,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Gets the next output from the given iterator as an Optional variant.
+func IteratorGetNextAsOptional(scope *Scope, iterator tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (optional tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "IteratorGetNextAsOptional",
+		Input: []tf.Input{
+			iterator,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Produces a summary of any statistics recorded by the given statistics manager.
+func StatsAggregatorSummary(scope *Scope, iterator tf.Output) (summary tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "StatsAggregatorSummary",
+		Input: []tf.Input{
+			iterator,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ExperimentalStatsAggregatorHandleAttr is an optional argument to ExperimentalStatsAggregatorHandle.
+type ExperimentalStatsAggregatorHandleAttr func(optionalAttr)
+
+// ExperimentalStatsAggregatorHandleContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func ExperimentalStatsAggregatorHandleContainer(value string) ExperimentalStatsAggregatorHandleAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// ExperimentalStatsAggregatorHandleSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func ExperimentalStatsAggregatorHandleSharedName(value string) ExperimentalStatsAggregatorHandleAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Creates a statistics manager resource.
+func ExperimentalStatsAggregatorHandle(scope *Scope, optional ...ExperimentalStatsAggregatorHandleAttr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalStatsAggregatorHandle",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// StatsAggregatorHandleAttr is an optional argument to StatsAggregatorHandle.
+type StatsAggregatorHandleAttr func(optionalAttr)
+
+// StatsAggregatorHandleContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func StatsAggregatorHandleContainer(value string) StatsAggregatorHandleAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// StatsAggregatorHandleSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func StatsAggregatorHandleSharedName(value string) StatsAggregatorHandleAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Creates a statistics manager resource.
+func StatsAggregatorHandle(scope *Scope, optional ...StatsAggregatorHandleAttr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StatsAggregatorHandle",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that executes a SQL query and emits rows of the result set.
+//
+// Arguments:
+//	driver_name: The database type. Currently, the only supported type is 'sqlite'.
+//	data_source_name: A connection string to connect to the database.
+//	query: A SQL query to execute.
+//
+//
+func ExperimentalSqlDataset(scope *Scope, driver_name tf.Output, data_source_name tf.Output, query tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalSqlDataset",
+		Input: []tf.Input{
+			driver_name, data_source_name, query,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Generate the bucket boundaries for each feature based on accumulated summaries.
+//
+// An op that returns a list of float tensors for a quantile stream resource. Each
+// tensor is Rank 1 containing bucket boundaries for a single feature.
+//
+// Arguments:
+//	quantile_stream_resource_handle: resource handle referring to a QuantileStreamResource.
+//	num_features: inferred int; number of features to get bucket boundaries for.
+//
+// Returns float; List of Rank 1 Tensors each containing the bucket boundaries for a feature.
+func BoostedTreesQuantileStreamResourceGetBucketBoundaries(scope *Scope, quantile_stream_resource_handle tf.Output, num_features int64) (bucket_boundaries []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_features": num_features}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesQuantileStreamResourceGetBucketBoundaries",
+		Input: []tf.Input{
+			quantile_stream_resource_handle,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if bucket_boundaries, idx, err = makeOutputList(op, idx, "bucket_boundaries"); err != nil {
+		scope.UpdateErr("BoostedTreesQuantileStreamResourceGetBucketBoundaries", err)
+		return
+	}
+	return bucket_boundaries
+}
+
+// Creates a dataset that passes a sliding window over `input_dataset`.
+//
+// Arguments:
+//
+//	window_size: A scalar representing the number of elements in the
+// sliding window.
+//	window_shift: A scalar representing the steps moving the sliding window
+// forward in one iteration. It must be positive.
+//	window_stride: A scalar representing the stride of the input elements of the sliding window.
+// It must be positive.
+//
+//
+func ExperimentalSlidingWindowDataset(scope *Scope, input_dataset tf.Output, window_size tf.Output, window_shift tf.Output, window_stride tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalSlidingWindowDataset",
+		Input: []tf.Input{
+			input_dataset, window_size, window_shift, window_stride,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Deprecated. Use TensorArraySizeV3
+//
+// DEPRECATED at GraphDef version 26: Use TensorArraySizeV3
+func TensorArraySizeV2(scope *Scope, handle tf.Output, flow_in tf.Output) (size tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorArraySizeV2",
+		Input: []tf.Input{
+			handle, flow_in,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Elementwise computes the bitwise right-shift of `x` and `y`.
+//
+// Performs a logical shift for unsigned integer types, and an arithmetic shift
+// for signed integer types.
+//
+// If `y` is negative, or greater than or equal to than the width of `x` in bits
+// the result is implementation defined.
+//
+// Example:
+//
+// ```python
+// import tensorflow as tf
+// from tensorflow.python.ops import bitwise_ops
+// import numpy as np
+// dtype_list = [tf.int8, tf.int16, tf.int32, tf.int64]
+//
+// for dtype in dtype_list:
+//   lhs = tf.constant([-1, -5, -3, -14], dtype=dtype)
+//   rhs = tf.constant([5, 0, 7, 11], dtype=dtype)
+//
+//   right_shift_result = bitwise_ops.right_shift(lhs, rhs)
+//
+//   print(right_shift_result)
+//
+// # This will print:
+// # tf.Tensor([-1 -5 -1 -1], shape=(4,), dtype=int8)
+// # tf.Tensor([-1 -5 -1 -1], shape=(4,), dtype=int16)
+// # tf.Tensor([-1 -5 -1 -1], shape=(4,), dtype=int32)
+// # tf.Tensor([-1 -5 -1 -1], shape=(4,), dtype=int64)
+//
+// lhs = np.array([-2, 64, 101, 32], dtype=np.int8)
+// rhs = np.array([-1, -5, -3, -14], dtype=np.int8)
+// bitwise_ops.right_shift(lhs, rhs)
+// # <tf.Tensor: shape=(4,), dtype=int8, numpy=array([ -2,  64, 101,  32], dtype=int8)>
+// ```
+//
+func RightShift(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "RightShift",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RebatchDatasetAttr is an optional argument to RebatchDataset.
+type RebatchDatasetAttr func(optionalAttr)
+
+// RebatchDatasetUseFallback sets the optional use_fallback attribute to value.
+// If not specified, defaults to true
+func RebatchDatasetUseFallback(value bool) RebatchDatasetAttr {
+	return func(m optionalAttr) {
+		m["use_fallback"] = value
+	}
+}
+
+// Creates a dataset that changes the batch size.
+//
+// Creates a dataset that changes the batch size of the dataset to current batch
+// size // num_workers.
+//
+// Arguments:
+//	input_dataset: A variant tensor representing the input dataset.
+//	num_replicas: A scalar representing the number of replicas to distribute this batch across. As
+// a result of this transformation the current batch size would end up being
+// divided  by this parameter.
+//
+//
+func RebatchDataset(scope *Scope, input_dataset tf.Output, num_replicas tf.Output, output_types []tf.DataType, output_shapes []tf.Shape, optional ...RebatchDatasetAttr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RebatchDataset",
+		Input: []tf.Input{
+			input_dataset, num_replicas,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that uses a custom thread pool to compute `input_dataset`.
+//
+// Arguments:
+//
+//	num_threads: Identifies the number of threads to use for the private threadpool.
+//
+//
+func ExperimentalPrivateThreadPoolDataset(scope *Scope, input_dataset tf.Output, num_threads tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalPrivateThreadPoolDataset",
+		Input: []tf.Input{
+			input_dataset, num_threads,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that uses a custom thread pool to compute `input_dataset`.
+//
+// Arguments:
+//
+//	num_threads: Identifies the number of threads to use for the private threadpool.
+//
+//
+func PrivateThreadPoolDataset(scope *Scope, input_dataset tf.Output, num_threads tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "PrivateThreadPoolDataset",
+		Input: []tf.Input{
+			input_dataset, num_threads,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ExperimentalParseExampleDatasetAttr is an optional argument to ExperimentalParseExampleDataset.
+type ExperimentalParseExampleDatasetAttr func(optionalAttr)
+
+// ExperimentalParseExampleDatasetSloppy sets the optional sloppy attribute to value.
+// If not specified, defaults to false
+func ExperimentalParseExampleDatasetSloppy(value bool) ExperimentalParseExampleDatasetAttr {
+	return func(m optionalAttr) {
+		m["sloppy"] = value
+	}
+}
+
+// Transforms `input_dataset` containing `Example` protos as vectors of DT_STRING into a dataset of `Tensor` or `SparseTensor` objects representing the parsed features.
+//
+// Arguments:
+//
+//
+//	dense_defaults: A dict mapping string keys to `Tensor`s.
+// The keys of the dict must match the dense_keys of the feature.
+//	sparse_keys: A list of string keys in the examples features.
+// The results for these keys will be returned as `SparseTensor` objects.
+//	dense_keys: A list of Ndense string Tensors (scalars).
+// The keys expected in the Examples features associated with dense values.
+//	sparse_types: A list of `DTypes` of the same length as `sparse_keys`.
+// Only `tf.float32` (`FloatList`), `tf.int64` (`Int64List`),
+// and `tf.string` (`BytesList`) are supported.
+//	dense_shapes: List of tuples with the same length as `dense_keys`.
+// The shape of the data for each dense feature referenced by `dense_keys`.
+// Required for any input tensors identified by `dense_keys`.  Must be
+// either fully defined, or may contain an unknown first dimension.
+// An unknown first dimension means the feature is treated as having
+// a variable number of blocks, and the output shape along this dimension
+// is considered unknown at graph build time.  Padding is applied for
+// minibatch elements smaller than the maximum number of blocks for the
+// given feature along this dimension.
+//	output_types: The type list for the return values.
+//	output_shapes: The list of shapes being produced.
+func ExperimentalParseExampleDataset(scope *Scope, input_dataset tf.Output, num_parallel_calls tf.Output, dense_defaults []tf.Output, sparse_keys []string, dense_keys []string, sparse_types []tf.DataType, dense_shapes []tf.Shape, output_types []tf.DataType, output_shapes []tf.Shape, optional ...ExperimentalParseExampleDatasetAttr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"sparse_keys": sparse_keys, "dense_keys": dense_keys, "sparse_types": sparse_types, "dense_shapes": dense_shapes, "output_types": output_types, "output_shapes": output_shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalParseExampleDataset",
+		Input: []tf.Input{
+			input_dataset, num_parallel_calls, tf.OutputList(dense_defaults),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns a batched matrix tensor with new batched diagonal values.
+//
+// Given `input` and `diagonal`, this operation returns a tensor with the
+// same shape and values as `input`, except for the main diagonal of the
+// innermost matrices.  These will be overwritten by the values in `diagonal`.
+//
+// The output is computed as follows:
+//
+// Assume `input` has `k+1` dimensions `[I, J, K, ..., M, N]` and `diagonal` has
+// `k` dimensions `[I, J, K, ..., min(M, N)]`.  Then the output is a
+// tensor of rank `k+1` with dimensions `[I, J, K, ..., M, N]` where:
+//
+//   * `output[i, j, k, ..., m, n] = diagonal[i, j, k, ..., n]` for `m == n`.
+//   * `output[i, j, k, ..., m, n] = input[i, j, k, ..., m, n]` for `m != n`.
+//
+// Arguments:
+//	input: Rank `k+1`, where `k >= 1`.
+//	diagonal: Rank `k`, where `k >= 1`.
+//
+// Returns Rank `k+1`, with `output.shape = input.shape`.
+func MatrixSetDiag(scope *Scope, input tf.Output, diagonal tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "MatrixSetDiag",
+		Input: []tf.Input{
+			input, diagonal,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ParseExampleDatasetV2Attr is an optional argument to ParseExampleDatasetV2.
+type ParseExampleDatasetV2Attr func(optionalAttr)
+
+// ParseExampleDatasetV2Deterministic sets the optional deterministic attribute to value.
+//
+// value: A string indicating the op-level determinism to use. Deterministic controls
+// whether the dataset is allowed to return elements out of order if the next
+// element to be returned isn't available, but a later element is. Options are
+// "true", "false", and "default". "default" indicates that determinism should be
+// decided by the `experimental_deterministic` parameter of `tf.data.Options`.
+// If not specified, defaults to "default"
+func ParseExampleDatasetV2Deterministic(value string) ParseExampleDatasetV2Attr {
+	return func(m optionalAttr) {
+		m["deterministic"] = value
+	}
+}
+
+// ParseExampleDatasetV2RaggedKeys sets the optional ragged_keys attribute to value.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseExampleDatasetV2RaggedKeys(value []string) ParseExampleDatasetV2Attr {
+	return func(m optionalAttr) {
+		m["ragged_keys"] = value
+	}
+}
+
+// ParseExampleDatasetV2RaggedValueTypes sets the optional ragged_value_types attribute to value.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseExampleDatasetV2RaggedValueTypes(value []tf.DataType) ParseExampleDatasetV2Attr {
+	return func(m optionalAttr) {
+		m["ragged_value_types"] = value
+	}
+}
+
+// ParseExampleDatasetV2RaggedSplitTypes sets the optional ragged_split_types attribute to value.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseExampleDatasetV2RaggedSplitTypes(value []tf.DataType) ParseExampleDatasetV2Attr {
+	return func(m optionalAttr) {
+		m["ragged_split_types"] = value
+	}
+}
+
+// Transforms `input_dataset` containing `Example` protos as vectors of DT_STRING into a dataset of `Tensor` or `SparseTensor` objects representing the parsed features.
+//
+// Arguments:
+//
+//
+//	dense_defaults: A dict mapping string keys to `Tensor`s.
+// The keys of the dict must match the dense_keys of the feature.
+//	sparse_keys: A list of string keys in the examples features.
+// The results for these keys will be returned as `SparseTensor` objects.
+//	dense_keys: A list of Ndense string Tensors (scalars).
+// The keys expected in the Examples features associated with dense values.
+//	sparse_types: A list of `DTypes` of the same length as `sparse_keys`.
+// Only `tf.float32` (`FloatList`), `tf.int64` (`Int64List`),
+// and `tf.string` (`BytesList`) are supported.
+//	dense_shapes: List of tuples with the same length as `dense_keys`.
+// The shape of the data for each dense feature referenced by `dense_keys`.
+// Required for any input tensors identified by `dense_keys`.  Must be
+// either fully defined, or may contain an unknown first dimension.
+// An unknown first dimension means the feature is treated as having
+// a variable number of blocks, and the output shape along this dimension
+// is considered unknown at graph build time.  Padding is applied for
+// minibatch elements smaller than the maximum number of blocks for the
+// given feature along this dimension.
+//	output_types: The type list for the return values.
+//	output_shapes: The list of shapes being produced.
+func ParseExampleDatasetV2(scope *Scope, input_dataset tf.Output, num_parallel_calls tf.Output, dense_defaults []tf.Output, sparse_keys []string, dense_keys []string, sparse_types []tf.DataType, dense_shapes []tf.Shape, output_types []tf.DataType, output_shapes []tf.Shape, optional ...ParseExampleDatasetV2Attr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"sparse_keys": sparse_keys, "dense_keys": dense_keys, "sparse_types": sparse_types, "dense_shapes": dense_shapes, "output_types": output_types, "output_shapes": output_shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ParseExampleDatasetV2",
+		Input: []tf.Input{
+			input_dataset, num_parallel_calls, tf.OutputList(dense_defaults),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// GenerateVocabRemappingAttr is an optional argument to GenerateVocabRemapping.
+type GenerateVocabRemappingAttr func(optionalAttr)
+
+// GenerateVocabRemappingOldVocabSize sets the optional old_vocab_size attribute to value.
+//
+// value: Number of entries in the old vocab file to consider.  If -1,
+// use the entire old vocabulary.
+// If not specified, defaults to -1
+//
+// REQUIRES: value >= -1
+func GenerateVocabRemappingOldVocabSize(value int64) GenerateVocabRemappingAttr {
+	return func(m optionalAttr) {
+		m["old_vocab_size"] = value
+	}
+}
+
+// Given a path to new and old vocabulary files, returns a remapping Tensor of
+//
+// length `num_new_vocab`, where `remapping[i]` contains the row number in the old
+// vocabulary that corresponds to row `i` in the new vocabulary (starting at line
+// `new_vocab_offset` and up to `num_new_vocab` entities), or `-1` if entry `i`
+// in the new vocabulary is not in the old vocabulary.  The old vocabulary is
+// constrained to the first `old_vocab_size` entries if `old_vocab_size` is not the
+// default value of -1.
+//
+// `num_vocab_offset` enables
+// use in the partitioned variable case, and should generally be set through
+// examining partitioning info.  The format of the files should be a text file,
+// with each line containing a single entity within the vocabulary.
+//
+// For example, with `new_vocab_file` a text file containing each of the following
+// elements on a single line: `[f0, f1, f2, f3]`, old_vocab_file = [f1, f0, f3],
+// `num_new_vocab = 3, new_vocab_offset = 1`, the returned remapping would be
+// `[0, -1, 2]`.
+//
+// The op also returns a count of how many entries in the new vocabulary
+// were present in the old vocabulary, which is used to calculate the number of
+// values to initialize in a weight matrix remapping
+//
+// This functionality can be used to remap both row vocabularies (typically,
+// features) and column vocabularies (typically, classes) from TensorFlow
+// checkpoints.  Note that the partitioning logic relies on contiguous vocabularies
+// corresponding to div-partitioned variables.  Moreover, the underlying remapping
+// uses an IndexTable (as opposed to an inexact CuckooTable), so client code should
+// use the corresponding index_table_from_file() as the FeatureColumn framework
+// does (as opposed to tf.feature_to_id(), which uses a CuckooTable).
+//
+// Arguments:
+//	new_vocab_file: Path to the new vocab file.
+//	old_vocab_file: Path to the old vocab file.
+//	new_vocab_offset: How many entries into the new vocab file to start reading.
+//	num_new_vocab: Number of entries in the new vocab file to remap.
+//
+// Returns:
+//	remapping: A Tensor of length num_new_vocab where the element at index i
+// is equal to the old ID that maps to the new ID i.  This element is -1 for any
+// new ID that is not found in the old vocabulary.
+//	num_present: Number of new vocab entries found in old vocab.
+func GenerateVocabRemapping(scope *Scope, new_vocab_file tf.Output, old_vocab_file tf.Output, new_vocab_offset int64, num_new_vocab int64, optional ...GenerateVocabRemappingAttr) (remapping tf.Output, num_present tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"new_vocab_offset": new_vocab_offset, "num_new_vocab": num_new_vocab}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "GenerateVocabRemapping",
+		Input: []tf.Input{
+			new_vocab_file, old_vocab_file,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Creates a dataset that overrides the maximum intra-op parallelism.
+//
+// Arguments:
+//
+//	max_intra_op_parallelism: Identifies the maximum intra-op parallelism to use.
+//
+//
+func ExperimentalMaxIntraOpParallelismDataset(scope *Scope, input_dataset tf.Output, max_intra_op_parallelism tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalMaxIntraOpParallelismDataset",
+		Input: []tf.Input{
+			input_dataset, max_intra_op_parallelism,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SpaceToBatch for N-D tensors of type T.
+//
+// This operation divides "spatial" dimensions `[1, ..., M]` of the input into a
+// grid of blocks of shape `block_shape`, and interleaves these blocks with the
+// "batch" dimension (0) such that in the output, the spatial dimensions
+// `[1, ..., M]` correspond to the position within the grid, and the batch
+// dimension combines both the position within a spatial block and the original
+// batch position.  Prior to division into blocks, the spatial dimensions of the
+// input are optionally zero padded according to `paddings`.  See below for a
+// precise description.
+//
+// Arguments:
+//	input: N-D with shape `input_shape = [batch] + spatial_shape + remaining_shape`,
+// where spatial_shape has `M` dimensions.
+//	block_shape: 1-D with shape `[M]`, all values must be >= 1.
+//	paddings: 2-D with shape `[M, 2]`, all values must be >= 0.
+//   `paddings[i] = [pad_start, pad_end]` specifies the padding for input dimension
+//   `i + 1`, which corresponds to spatial dimension `i`.  It is required that
+//   `block_shape[i]` divides `input_shape[i + 1] + pad_start + pad_end`.
+//
+// This operation is equivalent to the following steps:
+//
+// 1. Zero-pad the start and end of dimensions `[1, ..., M]` of the
+//    input according to `paddings` to produce `padded` of shape `padded_shape`.
+//
+// 2. Reshape `padded` to `reshaped_padded` of shape:
+//
+//      [batch] +
+//      [padded_shape[1] / block_shape[0],
+//        block_shape[0],
+//       ...,
+//       padded_shape[M] / block_shape[M-1],
+//       block_shape[M-1]] +
+//      remaining_shape
+//
+// 3. Permute dimensions of `reshaped_padded` to produce
+//    `permuted_reshaped_padded` of shape:
+//
+//      block_shape +
+//      [batch] +
+//      [padded_shape[1] / block_shape[0],
+//       ...,
+//       padded_shape[M] / block_shape[M-1]] +
+//      remaining_shape
+//
+// 4. Reshape `permuted_reshaped_padded` to flatten `block_shape` into the batch
+//    dimension, producing an output tensor of shape:
+//
+//      [batch * prod(block_shape)] +
+//      [padded_shape[1] / block_shape[0],
+//       ...,
+//       padded_shape[M] / block_shape[M-1]] +
+//      remaining_shape
+//
+// Some examples:
+//
+// (1) For the following input of shape `[1, 2, 2, 1]`, `block_shape = [2, 2]`, and
+//     `paddings = [[0, 0], [0, 0]]`:
+//
+// ```
+// x = [[[[1], [2]], [[3], [4]]]]
+// ```
+//
+// The output tensor has shape `[4, 1, 1, 1]` and value:
+//
+// ```
+// [[[[1]]], [[[2]]], [[[3]]], [[[4]]]]
+// ```
+//
+// (2) For the following input of shape `[1, 2, 2, 3]`, `block_shape = [2, 2]`, and
+//     `paddings = [[0, 0], [0, 0]]`:
+//
+// ```
+// x = [[[[1, 2, 3], [4, 5, 6]],
+//       [[7, 8, 9], [10, 11, 12]]]]
+// ```
+//
+// The output tensor has shape `[4, 1, 1, 3]` and value:
+//
+// ```
+// [[[[1, 2, 3]]], [[[4, 5, 6]]], [[[7, 8, 9]]], [[[10, 11, 12]]]]
+// ```
+//
+// (3) For the following input of shape `[1, 4, 4, 1]`, `block_shape = [2, 2]`, and
+//     `paddings = [[0, 0], [0, 0]]`:
+//
+// ```
+// x = [[[[1],   [2],  [3],  [4]],
+//       [[5],   [6],  [7],  [8]],
+//       [[9],  [10], [11],  [12]],
+//       [[13], [14], [15],  [16]]]]
+// ```
+//
+// The output tensor has shape `[4, 2, 2, 1]` and value:
+//
+// ```
+// x = [[[[1], [3]], [[9], [11]]],
+//      [[[2], [4]], [[10], [12]]],
+//      [[[5], [7]], [[13], [15]]],
+//      [[[6], [8]], [[14], [16]]]]
+// ```
+//
+// (4) For the following input of shape `[2, 2, 4, 1]`, block_shape = `[2, 2]`, and
+//     paddings = `[[0, 0], [2, 0]]`:
+//
+// ```
+// x = [[[[1],   [2],  [3],  [4]],
+//       [[5],   [6],  [7],  [8]]],
+//      [[[9],  [10], [11],  [12]],
+//       [[13], [14], [15],  [16]]]]
+// ```
+//
+// The output tensor has shape `[8, 1, 3, 1]` and value:
+//
+// ```
+// x = [[[[0], [1], [3]]], [[[0], [9], [11]]],
+//      [[[0], [2], [4]]], [[[0], [10], [12]]],
+//      [[[0], [5], [7]]], [[[0], [13], [15]]],
+//      [[[0], [6], [8]]], [[[0], [14], [16]]]]
+// ```
+//
+// Among others, this operation is useful for reducing atrous convolution into
+// regular convolution.
+func SpaceToBatchND(scope *Scope, input tf.Output, block_shape tf.Output, paddings tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SpaceToBatchND",
+		Input: []tf.Input{
+			input, block_shape, paddings,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns a batched diagonal tensor with given batched diagonal values.
+//
+// Returns a tensor with the contents in `diagonal` as `k[0]`-th to `k[1]`-th
+// diagonals of a matrix, with everything else padded with `padding`. `num_rows`
+// and `num_cols` specify the dimension of the innermost matrix of the output. If
+// both are not specified, the op assumes the innermost matrix is square and infers
+// its size from `k` and the innermost dimension of `diagonal`. If only one of them
+// is specified, the op assumes the unspecified value is the smallest possible
+// based on other criteria.
+//
+// Let `diagonal` have `r` dimensions `[I, J, ..., L, M, N]`. The output tensor has
+// rank `r+1` with shape `[I, J, ..., L, M, num_rows, num_cols]` when only one
+// diagonal is given (`k` is an integer or `k[0] == k[1]`). Otherwise, it has rank
+// `r` with shape `[I, J, ..., L, num_rows, num_cols]`.
+//
+// The second innermost dimension of `diagonal` has double meaning.
+// When `k` is scalar or `k[0] == k[1]`, `M` is part of the batch size
+// [I, J, ..., M], and the output tensor is:
+//
+// ```
+// output[i, j, ..., l, m, n]
+//   = diagonal[i, j, ..., l, n-max(d_upper, 0)] ; if n - m == d_upper
+//     padding_value                             ; otherwise
+// ```
+//
+// Otherwise, `M` is treated as the number of diagonals for the matrix in the
+// same batch (`M = k[1]-k[0]+1`), and the output tensor is:
+//
+// ```
+// output[i, j, ..., l, m, n]
+//   = diagonal[i, j, ..., l, diag_index, index_in_diag] ; if k[0] <= d <= k[1]
+//     padding_value                                     ; otherwise
+// ```
+// where `d = n - m`, `diag_index = k[1] - d`, and `index_in_diag = n - max(d, 0)`.
+//
+// For example:
+//
+// ```
+// # The main diagonal.
+// diagonal = np.array([[1, 2, 3, 4],            # Input shape: (2, 4)
+//                      [5, 6, 7, 8]])
+// tf.matrix_diag(diagonal) ==> [[[1, 0, 0, 0],  # Output shape: (2, 4, 4)
+//                                [0, 2, 0, 0],
+//                                [0, 0, 3, 0],
+//                                [0, 0, 0, 4]],
+//                               [[5, 0, 0, 0],
+//                                [0, 6, 0, 0],
+//                                [0, 0, 7, 0],
+//                                [0, 0, 0, 8]]]
+//
+// # A superdiagonal (per batch).
+// diagonal = np.array([[1, 2, 3],  # Input shape: (2, 3)
+//                      [4, 5, 6]])
+// tf.matrix_diag(diagonal, k = 1)
+//   ==> [[[0, 1, 0, 0],  # Output shape: (2, 4, 4)
+//         [0, 0, 2, 0],
+//         [0, 0, 0, 3],
+//         [0, 0, 0, 0]],
+//        [[0, 4, 0, 0],
+//         [0, 0, 5, 0],
+//         [0, 0, 0, 6],
+//         [0, 0, 0, 0]]]
+//
+// # A band of diagonals.
+// diagonals = np.array([[[1, 2, 3],  # Input shape: (2, 2, 3)
+//                        [4, 5, 0]],
+//                       [[6, 7, 9],
+//                        [9, 1, 0]]])
+// tf.matrix_diag(diagonals, k = (-1, 0))
+//   ==> [[[1, 0, 0],  # Output shape: (2, 3, 3)
+//         [4, 2, 0],
+//         [0, 5, 3]],
+//        [[6, 0, 0],
+//         [9, 7, 0],
+//         [0, 1, 9]]]
+//
+// # Rectangular matrix.
+// diagonal = np.array([1, 2])  # Input shape: (2)
+// tf.matrix_diag(diagonal, k = -1, num_rows = 3, num_cols = 4)
+//   ==> [[0, 0, 0, 0],  # Output shape: (3, 4)
+//        [1, 0, 0, 0],
+//        [0, 2, 0, 0]]
+//
+// # Rectangular matrix with inferred num_cols and padding_value = 9.
+// tf.matrix_diag(diagonal, k = -1, num_rows = 3, padding_value = 9)
+//   ==> [[9, 9],  # Output shape: (3, 2)
+//        [1, 9],
+//        [9, 2]]
+// ```
+//
+// Arguments:
+//	diagonal: Rank `r`, where `r >= 1`
+//	k: Diagonal offset(s). Positive value means superdiagonal, 0 refers to the main
+// diagonal, and negative value means subdiagonals. `k` can be a single integer
+// (for a single diagonal) or a pair of integers specifying the low and high ends
+// of a matrix band. `k[0]` must not be larger than `k[1]`.
+//	num_rows: The number of rows of the output matrix. If it is not provided, the op assumes
+// the output matrix is a square matrix and infers the matrix size from k and the
+// innermost dimension of `diagonal`.
+//	num_cols: The number of columns of the output matrix. If it is not provided, the op
+// assumes the output matrix is a square matrix and infers the matrix size from
+// k and the innermost dimension of `diagonal`.
+//	padding_value: The number to fill the area outside the specified diagonal band with.
+// Default is 0.
+//
+// Returns Has rank `r+1` when `k` is an integer or `k[0] == k[1]`, rank `r` otherwise.
+func MatrixDiagV2(scope *Scope, diagonal tf.Output, k tf.Output, num_rows tf.Output, num_cols tf.Output, padding_value tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "MatrixDiagV2",
+		Input: []tf.Input{
+			diagonal, k, num_rows, num_cols, padding_value,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that overrides the maximum intra-op parallelism.
+//
+// Arguments:
+//
+//	max_intra_op_parallelism: Identifies the maximum intra-op parallelism to use.
+//
+//
+func MaxIntraOpParallelismDataset(scope *Scope, input_dataset tf.Output, max_intra_op_parallelism tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "MaxIntraOpParallelismDataset",
+		Input: []tf.Input{
+			input_dataset, max_intra_op_parallelism,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// StageClearAttr is an optional argument to StageClear.
+type StageClearAttr func(optionalAttr)
+
+// StageClearCapacity sets the optional capacity attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func StageClearCapacity(value int64) StageClearAttr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// StageClearMemoryLimit sets the optional memory_limit attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func StageClearMemoryLimit(value int64) StageClearAttr {
+	return func(m optionalAttr) {
+		m["memory_limit"] = value
+	}
+}
+
+// StageClearContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func StageClearContainer(value string) StageClearAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// StageClearSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func StageClearSharedName(value string) StageClearAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Op removes all elements in the underlying container.
+//
+// Returns the created operation.
+func StageClear(scope *Scope, dtypes []tf.DataType, optional ...StageClearAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StageClear",
+
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Records the latency of producing `input_dataset` elements in a StatsAggregator.
+func ExperimentalLatencyStatsDataset(scope *Scope, input_dataset tf.Output, tag tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalLatencyStatsDataset",
+		Input: []tf.Input{
+			input_dataset, tag,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns the name of the device on which `resource` has been placed.
+func IteratorGetDevice(scope *Scope, resource tf.Output) (device tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "IteratorGetDevice",
+		Input: []tf.Input{
+			resource,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a Dataset that returns pseudorandom numbers.
+//
+// Arguments:
+//	seed: A scalar seed for the random number generator. If either seed or
+// seed2 is set to be non-zero, the random number generator is seeded
+// by the given seed.  Otherwise, a random seed is used.
+//	seed2: A second scalar seed to avoid seed collision.
+//
+//
+func ExperimentalRandomDataset(scope *Scope, seed tf.Output, seed2 tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalRandomDataset",
+		Input: []tf.Input{
+			seed, seed2,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that contains the elements of `input_dataset` ignoring errors.
+func ExperimentalIgnoreErrorsDataset(scope *Scope, input_dataset tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalIgnoreErrorsDataset",
+		Input: []tf.Input{
+			input_dataset,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// CudnnRNNBackpropV2Attr is an optional argument to CudnnRNNBackpropV2.
+type CudnnRNNBackpropV2Attr func(optionalAttr)
+
+// CudnnRNNBackpropV2RnnMode sets the optional rnn_mode attribute to value.
+// If not specified, defaults to "lstm"
+func CudnnRNNBackpropV2RnnMode(value string) CudnnRNNBackpropV2Attr {
+	return func(m optionalAttr) {
+		m["rnn_mode"] = value
+	}
+}
+
+// CudnnRNNBackpropV2InputMode sets the optional input_mode attribute to value.
+// If not specified, defaults to "linear_input"
+func CudnnRNNBackpropV2InputMode(value string) CudnnRNNBackpropV2Attr {
+	return func(m optionalAttr) {
+		m["input_mode"] = value
+	}
+}
+
+// CudnnRNNBackpropV2Direction sets the optional direction attribute to value.
+// If not specified, defaults to "unidirectional"
+func CudnnRNNBackpropV2Direction(value string) CudnnRNNBackpropV2Attr {
+	return func(m optionalAttr) {
+		m["direction"] = value
+	}
+}
+
+// CudnnRNNBackpropV2Dropout sets the optional dropout attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNBackpropV2Dropout(value float32) CudnnRNNBackpropV2Attr {
+	return func(m optionalAttr) {
+		m["dropout"] = value
+	}
+}
+
+// CudnnRNNBackpropV2Seed sets the optional seed attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNBackpropV2Seed(value int64) CudnnRNNBackpropV2Attr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// CudnnRNNBackpropV2Seed2 sets the optional seed2 attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNBackpropV2Seed2(value int64) CudnnRNNBackpropV2Attr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Backprop step of CudnnRNN.
+//
+// Compute the backprop of both data and weights in a RNN. Takes an extra
+//     "host_reserved" inupt than CudnnRNNBackprop, which is used to determine RNN
+//     cudnnRNNAlgo_t and cudnnMathType_t.
+//
+// rnn_mode: Indicates the type of the RNN model.
+// input_mode: Indicates whether there is a linear projection between the input and
+//     the actual computation before the first layer. 'skip_input' is only allowed
+//     when input_size == num_units; 'auto_select' implies 'skip_input' when
+//     input_size == num_units; otherwise, it implies 'linear_input'.
+// direction: Indicates whether a bidirectional model will be used. Should be
+//   "unidirectional" or "bidirectional".
+// dropout: Dropout probability. When set to 0., dropout is disabled.
+// seed: The 1st part of a seed to initialize dropout.
+// seed2: The 2nd part of a seed to initialize dropout.
+// input: A 3-D tensor with the shape of [seq_length, batch_size, input_size].
+// input_h: A 3-D tensor with the shape of [num_layer * dir, batch_size,
+//     num_units].
+// input_c: For LSTM, a 3-D tensor with the shape of
+//     [num_layer * dir, batch, num_units]. For other models, it is ignored.
+// params: A 1-D tensor that contains the weights and biases in an opaque layout.
+//     The size must be created through CudnnRNNParamsSize, and initialized
+//     separately. Note that they might not be compatible across different
+//     generations. So it is a good idea to save and restore
+// output: A 3-D tensor with the shape of [seq_length, batch_size,
+//     dir * num_units].
+// output_h: The same shape has input_h.
+// output_c: The same shape as input_c for LSTM. An empty tensor for other models.
+// output_backprop: A 3-D tensor with the same shape as output in the forward pass.
+// output_h_backprop: A 3-D tensor with the same shape as output_h in the forward
+//     pass.
+// output_c_backprop: A 3-D tensor with the same shape as output_c in the forward
+//     pass.
+// reserve_space: The same reserve_space produced in the forward operation.
+// host_reserved: The same host_reserved produced in the forward operation.
+// input_backprop: The backprop to input in the forward pass. Has the same shape
+//     as input.
+// input_h_backprop: The backprop to input_h in the forward pass. Has the same
+//     shape as input_h.
+// input_c_backprop: The backprop to input_c in the forward pass. Has the same
+//     shape as input_c.
+// params_backprop: The backprop to the params buffer in the forward pass. Has the
+//     same shape as params.
+func CudnnRNNBackpropV2(scope *Scope, input tf.Output, input_h tf.Output, input_c tf.Output, params tf.Output, output tf.Output, output_h tf.Output, output_c tf.Output, output_backprop tf.Output, output_h_backprop tf.Output, output_c_backprop tf.Output, reserve_space tf.Output, host_reserved tf.Output, optional ...CudnnRNNBackpropV2Attr) (input_backprop tf.Output, input_h_backprop tf.Output, input_c_backprop tf.Output, params_backprop tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CudnnRNNBackpropV2",
+		Input: []tf.Input{
+			input, input_h, input_c, params, output, output_h, output_c, output_backprop, output_h_backprop, output_c_backprop, reserve_space, host_reserved,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3)
+}
+
+// A substitute for `InterleaveDataset` on a fixed list of `N` datasets.
+//
+// Arguments:
+//	selector_input_dataset: A dataset of scalar `DT_INT64` elements that determines which of the
+// `N` data inputs should produce the next output element.
+//	data_input_datasets: `N` datasets with the same type that will be interleaved according to
+// the values of `selector_input_dataset`.
+//
+//
+func DirectedInterleaveDataset(scope *Scope, selector_input_dataset tf.Output, data_input_datasets []tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "DirectedInterleaveDataset",
+		Input: []tf.Input{
+			selector_input_dataset, tf.OutputList(data_input_datasets),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that batches input elements into a SparseTensor.
+//
+// Arguments:
+//	input_dataset: A handle to an input dataset. Must have a single component.
+//	batch_size: A scalar representing the number of elements to accumulate in a
+// batch.
+//	row_shape: A vector representing the dense shape of each row in the produced
+// SparseTensor. The shape may be partially specified, using `-1` to indicate
+// that a particular dimension should use the maximum size of all batch elements.
+//
+//
+func ExperimentalDenseToSparseBatchDataset(scope *Scope, input_dataset tf.Output, batch_size tf.Output, row_shape tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalDenseToSparseBatchDataset",
+		Input: []tf.Input{
+			input_dataset, batch_size, row_shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Writes the given dataset to the given file using the TFRecord format.
+//
+// Arguments:
+//	input_dataset: A variant tensor representing the dataset to write.
+//	filename: A scalar string tensor representing the filename to use.
+//	compression_type: A scalar string tensor containing either (i) the empty string (no
+// compression), (ii) "ZLIB", or (iii) "GZIP".
+//
+// Returns the created operation.
+func ExperimentalDatasetToTFRecord(scope *Scope, input_dataset tf.Output, filename tf.Output, compression_type tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalDatasetToTFRecord",
+		Input: []tf.Input{
+			input_dataset, filename, compression_type,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Creates a dataset from the given `graph_def`.
+//
+// Creates a dataset from the provided `graph_def`.
+//
+// Arguments:
+//	graph_def: The graph representation of the dataset (as serialized GraphDef).
+//
+// Returns A variant tensor representing the dataset.
+func DatasetFromGraph(scope *Scope, graph_def tf.Output) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "DatasetFromGraph",
+		Input: []tf.Input{
+			graph_def,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns the cardinality of `input_dataset`.
+//
+// Returns the cardinality of `input_dataset`.
+//
+// Arguments:
+//	input_dataset: A variant tensor representing the dataset to return cardinality for.
+//
+// Returns The cardinality of `input_dataset`. Named constants are used to represent
+// infinite and unknown cardinality.
+func ExperimentalDatasetCardinality(scope *Scope, input_dataset tf.Output) (cardinality tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalDatasetCardinality",
+		Input: []tf.Input{
+			input_dataset,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Interleave the values from the `data` tensors into a single tensor.
+//
+// Builds a merged tensor such that
+//
+// ```python
+//     merged[indices[m][i, ..., j], ...] = data[m][i, ..., j, ...]
+// ```
+//
+// For example, if each `indices[m]` is scalar or vector, we have
+//
+// ```python
+//     # Scalar indices:
+//     merged[indices[m], ...] = data[m][...]
+//
+//     # Vector indices:
+//     merged[indices[m][i], ...] = data[m][i, ...]
+// ```
+//
+// Each `data[i].shape` must start with the corresponding `indices[i].shape`,
+// and the rest of `data[i].shape` must be constant w.r.t. `i`.  That is, we
+// must have `data[i].shape = indices[i].shape + constant`.  In terms of this
+// `constant`, the output shape is
+//
+//     merged.shape = [max(indices)] + constant
+//
+// Values are merged in order, so if an index appears in both `indices[m][i]` and
+// `indices[n][j]` for `(m,i) < (n,j)` the slice `data[n][j]` will appear in the
+// merged result. If you do not need this guarantee, ParallelDynamicStitch might
+// perform better on some devices.
+//
+// For example:
+//
+// ```python
+//     indices[0] = 6
+//     indices[1] = [4, 1]
+//     indices[2] = [[5, 2], [0, 3]]
+//     data[0] = [61, 62]
+//     data[1] = [[41, 42], [11, 12]]
+//     data[2] = [[[51, 52], [21, 22]], [[1, 2], [31, 32]]]
+//     merged = [[1, 2], [11, 12], [21, 22], [31, 32], [41, 42],
+//               [51, 52], [61, 62]]
+// ```
+//
+// This method can be used to merge partitions created by `dynamic_partition`
+// as illustrated on the following example:
+//
+// ```python
+//     # Apply function (increments x_i) on elements for which a certain condition
+//     # apply (x_i != -1 in this example).
+//     x=tf.constant([0.1, -1., 5.2, 4.3, -1., 7.4])
+//     condition_mask=tf.not_equal(x,tf.constant(-1.))
+//     partitioned_data = tf.dynamic_partition(
+//         x, tf.cast(condition_mask, tf.int32) , 2)
+//     partitioned_data[1] = partitioned_data[1] + 1.0
+//     condition_indices = tf.dynamic_partition(
+//         tf.range(tf.shape(x)[0]), tf.cast(condition_mask, tf.int32) , 2)
+//     x = tf.dynamic_stitch(condition_indices, partitioned_data)
+//     # Here x=[1.1, -1., 6.2, 5.3, -1, 8.4], the -1. values remain
+//     # unchanged.
+// ```
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src="https://www.tensorflow.org/images/DynamicStitch.png" alt>
+// </div>
+func DynamicStitch(scope *Scope, indices []tf.Output, data []tf.Output) (merged tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "DynamicStitch",
+		Input: []tf.Input{
+			tf.OutputList(indices), tf.OutputList(data),
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Uncompresses a compressed dataset element.
+func UncompressElement(scope *Scope, compressed tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (components []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "UncompressElement",
+		Input: []tf.Input{
+			compressed,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if components, idx, err = makeOutputList(op, idx, "components"); err != nil {
+		scope.UpdateErr("UncompressElement", err)
+		return
+	}
+	return components
+}
+
+// Records the bytes size of each element of `input_dataset` in a StatsAggregator.
+func BytesProducedStatsDataset(scope *Scope, input_dataset tf.Output, tag tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "BytesProducedStatsDataset",
+		Input: []tf.Input{
+			input_dataset, tag,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ExperimentalAutoShardDatasetAttr is an optional argument to ExperimentalAutoShardDataset.
+type ExperimentalAutoShardDatasetAttr func(optionalAttr)
+
+// ExperimentalAutoShardDatasetAutoShardPolicy sets the optional auto_shard_policy attribute to value.
+// If not specified, defaults to 0
+func ExperimentalAutoShardDatasetAutoShardPolicy(value int64) ExperimentalAutoShardDatasetAttr {
+	return func(m optionalAttr) {
+		m["auto_shard_policy"] = value
+	}
+}
+
+// Creates a dataset that shards the input dataset.
+//
+// Creates a dataset that shards the input dataset by num_workers, returning a
+// sharded dataset for the index-th worker. This attempts to automatically shard
+// a dataset by examining the Dataset graph and inserting a shard op before the
+// inputs to a reader Dataset (e.g. CSVDataset, TFRecordDataset).
+//
+// This dataset will throw a NotFound error if we cannot shard the dataset
+// automatically.
+//
+// Arguments:
+//	input_dataset: A variant tensor representing the input dataset.
+//	num_workers: A scalar representing the number of workers to distribute this dataset across.
+//	index: A scalar representing the index of the current worker out of num_workers.
+//
+//
+func ExperimentalAutoShardDataset(scope *Scope, input_dataset tf.Output, num_workers tf.Output, index tf.Output, output_types []tf.DataType, output_shapes []tf.Shape, optional ...ExperimentalAutoShardDatasetAttr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalAutoShardDataset",
+		Input: []tf.Input{
+			input_dataset, num_workers, index,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// A transformation that asserts which transformations happen next.
+//
+// This transformation checks whether the camel-case names (i.e. "FlatMap", not
+// "flat_map") of the transformations following this transformation match the list
+// of names in the `transformations` argument. If there is a mismatch, the
+// transformation raises an exception.
+//
+// The check occurs when iterating over the contents of the dataset, which
+// means that the check happens *after* any static optimizations are applied
+// to the dataset graph.
+//
+// Arguments:
+//	input_dataset: A variant tensor representing the input dataset.
+// `AssertNextDataset` passes through the outputs of its input dataset.
+//	transformations: A `tf.string` vector `tf.Tensor` identifying the transformations that are
+// expected to happen next.
+//
+//
+func AssertNextDataset(scope *Scope, input_dataset tf.Output, transformations tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "AssertNextDataset",
+		Input: []tf.Input{
+			input_dataset, transformations,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Return the index of device the op runs.
+func DeviceIndex(scope *Scope, device_names []string) (index tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"device_names": device_names}
+	opspec := tf.OpSpec{
+		Type: "DeviceIndex",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ShardDatasetAttr is an optional argument to ShardDataset.
+type ShardDatasetAttr func(optionalAttr)
+
+// ShardDatasetRequireNonEmpty sets the optional require_non_empty attribute to value.
+// If not specified, defaults to false
+func ShardDatasetRequireNonEmpty(value bool) ShardDatasetAttr {
+	return func(m optionalAttr) {
+		m["require_non_empty"] = value
+	}
+}
+
+// Creates a `Dataset` that includes only 1/`num_shards` of this dataset.
+//
+// Arguments:
+//
+//	num_shards: An integer representing the number of shards operating in parallel.
+//	index: An integer representing the current worker index.
+//
+//
+func ShardDataset(scope *Scope, input_dataset tf.Output, num_shards tf.Output, index tf.Output, output_types []tf.DataType, output_shapes []tf.Shape, optional ...ShardDatasetAttr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ShardDataset",
+		Input: []tf.Input{
+			input_dataset, num_shards, index,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// NonMaxSuppressionV5Attr is an optional argument to NonMaxSuppressionV5.
+type NonMaxSuppressionV5Attr func(optionalAttr)
+
+// NonMaxSuppressionV5PadToMaxOutputSize sets the optional pad_to_max_output_size attribute to value.
+//
+// value: If true, the output `selected_indices` is padded to be of length
+// `max_output_size`. Defaults to false.
+// If not specified, defaults to false
+func NonMaxSuppressionV5PadToMaxOutputSize(value bool) NonMaxSuppressionV5Attr {
+	return func(m optionalAttr) {
+		m["pad_to_max_output_size"] = value
+	}
+}
+
+// Greedily selects a subset of bounding boxes in descending order of score,
+//
+// pruning away boxes that have high intersection-over-union (IOU) overlap
+// with previously selected boxes.  Bounding boxes with score less than
+// `score_threshold` are removed.  Bounding boxes are supplied as
+// [y1, x1, y2, x2], where (y1, x1) and (y2, x2) are the coordinates of any
+// diagonal pair of box corners and the coordinates can be provided as normalized
+// (i.e., lying in the interval [0, 1]) or absolute.  Note that this algorithm
+// is agnostic to where the origin is in the coordinate system and more
+// generally is invariant to orthogonal transformations and translations
+// of the coordinate system; thus translating or reflections of the coordinate
+// system result in the same boxes being selected by the algorithm.
+// The output of this operation is a set of integers indexing into the input
+// collection of bounding boxes representing the selected boxes.  The bounding
+// box coordinates corresponding to the selected indices can then be obtained
+// using the `tf.gather operation`.  For example:
+//   selected_indices = tf.image.non_max_suppression_v2(
+//       boxes, scores, max_output_size, iou_threshold, score_threshold)
+//   selected_boxes = tf.gather(boxes, selected_indices)
+// This op also supports a Soft-NMS (with Gaussian weighting) mode (c.f.
+// Bodla et al, https://arxiv.org/abs/1704.04503) where boxes reduce the score
+// of other overlapping boxes instead of directly causing them to be pruned.
+// To enable this Soft-NMS mode, set the `soft_nms_sigma` parameter to be
+// larger than 0.
+//
+// Arguments:
+//	boxes: A 2-D float tensor of shape `[num_boxes, 4]`.
+//	scores: A 1-D float tensor of shape `[num_boxes]` representing a single
+// score corresponding to each box (each row of boxes).
+//	max_output_size: A scalar integer tensor representing the maximum number of
+// boxes to be selected by non max suppression.
+//	iou_threshold: A 0-D float tensor representing the threshold for deciding whether
+// boxes overlap too much with respect to IOU.
+//	score_threshold: A 0-D float tensor representing the threshold for deciding when to remove
+// boxes based on score.
+//	soft_nms_sigma: A 0-D float tensor representing the sigma parameter for Soft NMS; see Bodla et
+// al (c.f. https://arxiv.org/abs/1704.04503).  When `soft_nms_sigma=0.0` (which
+// is default), we fall back to standard (hard) NMS.
+//
+// Returns:
+//	selected_indices: A 1-D integer tensor of shape `[M]` representing the selected
+// indices from the boxes tensor, where `M <= max_output_size`.
+//	selected_scores: A 1-D float tensor of shape `[M]` representing the corresponding
+// scores for each selected box, where `M <= max_output_size`.  Scores only differ
+// from corresponding input scores when using Soft NMS (i.e. when
+// `soft_nms_sigma>0`)
+//	valid_outputs: A 0-D integer tensor representing the number of valid elements in
+// `selected_indices`, with the valid elements appearing first.
+func NonMaxSuppressionV5(scope *Scope, boxes tf.Output, scores tf.Output, max_output_size tf.Output, iou_threshold tf.Output, score_threshold tf.Output, soft_nms_sigma tf.Output, optional ...NonMaxSuppressionV5Attr) (selected_indices tf.Output, selected_scores tf.Output, valid_outputs tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "NonMaxSuppressionV5",
+		Input: []tf.Input{
+			boxes, scores, max_output_size, iou_threshold, score_threshold, soft_nms_sigma,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// NonMaxSuppressionV4Attr is an optional argument to NonMaxSuppressionV4.
+type NonMaxSuppressionV4Attr func(optionalAttr)
+
+// NonMaxSuppressionV4PadToMaxOutputSize sets the optional pad_to_max_output_size attribute to value.
+//
+// value: If true, the output `selected_indices` is padded to be of length
+// `max_output_size`. Defaults to false.
+// If not specified, defaults to false
+func NonMaxSuppressionV4PadToMaxOutputSize(value bool) NonMaxSuppressionV4Attr {
+	return func(m optionalAttr) {
+		m["pad_to_max_output_size"] = value
+	}
+}
+
+// Greedily selects a subset of bounding boxes in descending order of score,
+//
+// pruning away boxes that have high intersection-over-union (IOU) overlap
+// with previously selected boxes.  Bounding boxes with score less than
+// `score_threshold` are removed.  Bounding boxes are supplied as
+// [y1, x1, y2, x2], where (y1, x1) and (y2, x2) are the coordinates of any
+// diagonal pair of box corners and the coordinates can be provided as normalized
+// (i.e., lying in the interval [0, 1]) or absolute.  Note that this algorithm
+// is agnostic to where the origin is in the coordinate system and more
+// generally is invariant to orthogonal transformations and translations
+// of the coordinate system; thus translating or reflections of the coordinate
+// system result in the same boxes being selected by the algorithm.
+// The output of this operation is a set of integers indexing into the input
+// collection of bounding boxes representing the selected boxes.  The bounding
+// box coordinates corresponding to the selected indices can then be obtained
+// using the `tf.gather operation`.  For example:
+//   selected_indices = tf.image.non_max_suppression_v2(
+//       boxes, scores, max_output_size, iou_threshold, score_threshold)
+//   selected_boxes = tf.gather(boxes, selected_indices)
+//
+// Arguments:
+//	boxes: A 2-D float tensor of shape `[num_boxes, 4]`.
+//	scores: A 1-D float tensor of shape `[num_boxes]` representing a single
+// score corresponding to each box (each row of boxes).
+//	max_output_size: A scalar integer tensor representing the maximum number of
+// boxes to be selected by non max suppression.
+//	iou_threshold: A 0-D float tensor representing the threshold for deciding whether
+// boxes overlap too much with respect to IOU.
+//	score_threshold: A 0-D float tensor representing the threshold for deciding when to remove
+// boxes based on score.
+//
+// Returns:
+//	selected_indices: A 1-D integer tensor of shape `[M]` representing the selected
+// indices from the boxes tensor, where `M <= max_output_size`.
+//	valid_outputs: A 0-D integer tensor representing the number of valid elements in
+// `selected_indices`, with the valid elements appearing first.
+func NonMaxSuppressionV4(scope *Scope, boxes tf.Output, scores tf.Output, max_output_size tf.Output, iou_threshold tf.Output, score_threshold tf.Output, optional ...NonMaxSuppressionV4Attr) (selected_indices tf.Output, valid_outputs tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "NonMaxSuppressionV4",
+		Input: []tf.Input{
+			boxes, scores, max_output_size, iou_threshold, score_threshold,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Greedily selects a subset of bounding boxes in descending order of score,
+//
+// pruning away boxes that have high intersection-over-union (IOU) overlap
+// with previously selected boxes.  Bounding boxes with score less than
+// `score_threshold` are removed.  Bounding boxes are supplied as
+// [y1, x1, y2, x2], where (y1, x1) and (y2, x2) are the coordinates of any
+// diagonal pair of box corners and the coordinates can be provided as normalized
+// (i.e., lying in the interval [0, 1]) or absolute.  Note that this algorithm
+// is agnostic to where the origin is in the coordinate system and more
+// generally is invariant to orthogonal transformations and translations
+// of the coordinate system; thus translating or reflections of the coordinate
+// system result in the same boxes being selected by the algorithm.
+// The output of this operation is a set of integers indexing into the input
+// collection of bounding boxes representing the selected boxes.  The bounding
+// box coordinates corresponding to the selected indices can then be obtained
+// using the `tf.gather operation`.  For example:
+//   selected_indices = tf.image.non_max_suppression_v2(
+//       boxes, scores, max_output_size, iou_threshold, score_threshold)
+//   selected_boxes = tf.gather(boxes, selected_indices)
+//
+// Arguments:
+//	boxes: A 2-D float tensor of shape `[num_boxes, 4]`.
+//	scores: A 1-D float tensor of shape `[num_boxes]` representing a single
+// score corresponding to each box (each row of boxes).
+//	max_output_size: A scalar integer tensor representing the maximum number of
+// boxes to be selected by non max suppression.
+//	iou_threshold: A 0-D float tensor representing the threshold for deciding whether
+// boxes overlap too much with respect to IOU.
+//	score_threshold: A 0-D float tensor representing the threshold for deciding when to remove
+// boxes based on score.
+//
+// Returns A 1-D integer tensor of shape `[M]` representing the selected
+// indices from the boxes tensor, where `M <= max_output_size`.
+func NonMaxSuppressionV3(scope *Scope, boxes tf.Output, scores tf.Output, max_output_size tf.Output, iou_threshold tf.Output, score_threshold tf.Output) (selected_indices tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "NonMaxSuppressionV3",
+		Input: []tf.Input{
+			boxes, scores, max_output_size, iou_threshold, score_threshold,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Greedily selects a subset of bounding boxes in descending order of score,
+//
+// pruning away boxes that have high intersection-over-union (IOU) overlap
+// with previously selected boxes.  Bounding boxes are supplied as
+// [y1, x1, y2, x2], where (y1, x1) and (y2, x2) are the coordinates of any
+// diagonal pair of box corners and the coordinates can be provided as normalized
+// (i.e., lying in the interval [0, 1]) or absolute.  Note that this algorithm
+// is agnostic to where the origin is in the coordinate system.  Note that this
+// algorithm is invariant to orthogonal transformations and translations
+// of the coordinate system; thus translating or reflections of the coordinate
+// system result in the same boxes being selected by the algorithm.
+//
+// The output of this operation is a set of integers indexing into the input
+// collection of bounding boxes representing the selected boxes.  The bounding
+// box coordinates corresponding to the selected indices can then be obtained
+// using the `tf.gather operation`.  For example:
+//
+//   selected_indices = tf.image.non_max_suppression_v2(
+//       boxes, scores, max_output_size, iou_threshold)
+//   selected_boxes = tf.gather(boxes, selected_indices)
+//
+// Arguments:
+//	boxes: A 2-D float tensor of shape `[num_boxes, 4]`.
+//	scores: A 1-D float tensor of shape `[num_boxes]` representing a single
+// score corresponding to each box (each row of boxes).
+//	max_output_size: A scalar integer tensor representing the maximum number of
+// boxes to be selected by non max suppression.
+//	iou_threshold: A 0-D float tensor representing the threshold for deciding whether
+// boxes overlap too much with respect to IOU.
+//
+// Returns A 1-D integer tensor of shape `[M]` representing the selected
+// indices from the boxes tensor, where `M <= max_output_size`.
+func NonMaxSuppressionV2(scope *Scope, boxes tf.Output, scores tf.Output, max_output_size tf.Output, iou_threshold tf.Output) (selected_indices tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "NonMaxSuppressionV2",
+		Input: []tf.Input{
+			boxes, scores, max_output_size, iou_threshold,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// NonMaxSuppressionAttr is an optional argument to NonMaxSuppression.
+type NonMaxSuppressionAttr func(optionalAttr)
+
+// NonMaxSuppressionIouThreshold sets the optional iou_threshold attribute to value.
+//
+// value: A float representing the threshold for deciding whether boxes
+// overlap too much with respect to IOU.
+// If not specified, defaults to 0.5
+func NonMaxSuppressionIouThreshold(value float32) NonMaxSuppressionAttr {
+	return func(m optionalAttr) {
+		m["iou_threshold"] = value
+	}
+}
+
+// Greedily selects a subset of bounding boxes in descending order of score,
+//
+// pruning away boxes that have high intersection-over-union (IOU) overlap
+// with previously selected boxes.  Bounding boxes are supplied as
+// [y1, x1, y2, x2], where (y1, x1) and (y2, x2) are the coordinates of any
+// diagonal pair of box corners and the coordinates can be provided as normalized
+// (i.e., lying in the interval [0, 1]) or absolute.  Note that this algorithm
+// is agnostic to where the origin is in the coordinate system.  Note that this
+// algorithm is invariant to orthogonal transformations and translations
+// of the coordinate system; thus translating or reflections of the coordinate
+// system result in the same boxes being selected by the algorithm.
+// The output of this operation is a set of integers indexing into the input
+// collection of bounding boxes representing the selected boxes.  The bounding
+// box coordinates corresponding to the selected indices can then be obtained
+// using the `tf.gather operation`.  For example:
+//   selected_indices = tf.image.non_max_suppression(
+//       boxes, scores, max_output_size, iou_threshold)
+//   selected_boxes = tf.gather(boxes, selected_indices)
+//
+// Arguments:
+//	boxes: A 2-D float tensor of shape `[num_boxes, 4]`.
+//	scores: A 1-D float tensor of shape `[num_boxes]` representing a single
+// score corresponding to each box (each row of boxes).
+//	max_output_size: A scalar integer tensor representing the maximum number of
+// boxes to be selected by non max suppression.
+//
+// Returns A 1-D integer tensor of shape `[M]` representing the selected
+// indices from the boxes tensor, where `M <= max_output_size`.
+func NonMaxSuppression(scope *Scope, boxes tf.Output, scores tf.Output, max_output_size tf.Output, optional ...NonMaxSuppressionAttr) (selected_indices tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "NonMaxSuppression",
+		Input: []tf.Input{
+			boxes, scores, max_output_size,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// CropAndResizeGradBoxesAttr is an optional argument to CropAndResizeGradBoxes.
+type CropAndResizeGradBoxesAttr func(optionalAttr)
+
+// CropAndResizeGradBoxesMethod sets the optional method attribute to value.
+//
+// value: A string specifying the interpolation method. Only 'bilinear' is
+// supported for now.
+// If not specified, defaults to "bilinear"
+func CropAndResizeGradBoxesMethod(value string) CropAndResizeGradBoxesAttr {
+	return func(m optionalAttr) {
+		m["method"] = value
+	}
+}
+
+// Computes the gradient of the crop_and_resize op wrt the input boxes tensor.
+//
+// Arguments:
+//	grads: A 4-D tensor of shape `[num_boxes, crop_height, crop_width, depth]`.
+//	image: A 4-D tensor of shape `[batch, image_height, image_width, depth]`.
+// Both `image_height` and `image_width` need to be positive.
+//	boxes: A 2-D tensor of shape `[num_boxes, 4]`. The `i`-th row of the tensor
+// specifies the coordinates of a box in the `box_ind[i]` image and is specified
+// in normalized coordinates `[y1, x1, y2, x2]`. A normalized coordinate value of
+// `y` is mapped to the image coordinate at `y * (image_height - 1)`, so as the
+// `[0, 1]` interval of normalized image height is mapped to
+// `[0, image_height - 1] in image height coordinates. We do allow y1 > y2, in
+// which case the sampled crop is an up-down flipped version of the original
+// image. The width dimension is treated similarly. Normalized coordinates
+// outside the `[0, 1]` range are allowed, in which case we use
+// `extrapolation_value` to extrapolate the input image values.
+//	box_ind: A 1-D tensor of shape `[num_boxes]` with int32 values in `[0, batch)`.
+// The value of `box_ind[i]` specifies the image that the `i`-th box refers to.
+//
+// Returns A 2-D tensor of shape `[num_boxes, 4]`.
+func CropAndResizeGradBoxes(scope *Scope, grads tf.Output, image tf.Output, boxes tf.Output, box_ind tf.Output, optional ...CropAndResizeGradBoxesAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CropAndResizeGradBoxes",
+		Input: []tf.Input{
+			grads, image, boxes, box_ind,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ExtractGlimpseV2Attr is an optional argument to ExtractGlimpseV2.
+type ExtractGlimpseV2Attr func(optionalAttr)
+
+// ExtractGlimpseV2Centered sets the optional centered attribute to value.
+//
+// value: indicates if the offset coordinates are centered relative to
+// the image, in which case the (0, 0) offset is relative to the center
+// of the input images. If false, the (0,0) offset corresponds to the
+// upper left corner of the input images.
+// If not specified, defaults to true
+func ExtractGlimpseV2Centered(value bool) ExtractGlimpseV2Attr {
+	return func(m optionalAttr) {
+		m["centered"] = value
+	}
+}
+
+// ExtractGlimpseV2Normalized sets the optional normalized attribute to value.
+//
+// value: indicates if the offset coordinates are normalized.
+// If not specified, defaults to true
+func ExtractGlimpseV2Normalized(value bool) ExtractGlimpseV2Attr {
+	return func(m optionalAttr) {
+		m["normalized"] = value
+	}
+}
+
+// ExtractGlimpseV2UniformNoise sets the optional uniform_noise attribute to value.
+//
+// value: indicates if the noise should be generated using a
+// uniform distribution or a Gaussian distribution.
+// If not specified, defaults to true
+func ExtractGlimpseV2UniformNoise(value bool) ExtractGlimpseV2Attr {
+	return func(m optionalAttr) {
+		m["uniform_noise"] = value
+	}
+}
+
+// ExtractGlimpseV2Noise sets the optional noise attribute to value.
+//
+// value: indicates if the noise should `uniform`, `gaussian`, or
+// `zero`. The default is `uniform` which means the the noise type
+// will be decided by `uniform_noise`.
+// If not specified, defaults to "uniform"
+func ExtractGlimpseV2Noise(value string) ExtractGlimpseV2Attr {
+	return func(m optionalAttr) {
+		m["noise"] = value
+	}
+}
+
+// Extracts a glimpse from the input tensor.
+//
+// Returns a set of windows called glimpses extracted at location
+// `offsets` from the input tensor. If the windows only partially
+// overlaps the inputs, the non overlapping areas will be filled with
+// random noise.
+//
+// The result is a 4-D tensor of shape `[batch_size, glimpse_height,
+// glimpse_width, channels]`. The channels and batch dimensions are the
+// same as that of the input tensor. The height and width of the output
+// windows are specified in the `size` parameter.
+//
+// The argument `normalized` and `centered` controls how the windows are built:
+//
+// * If the coordinates are normalized but not centered, 0.0 and 1.0
+//   correspond to the minimum and maximum of each height and width
+//   dimension.
+// * If the coordinates are both normalized and centered, they range from
+//   -1.0 to 1.0. The coordinates (-1.0, -1.0) correspond to the upper
+//   left corner, the lower right corner is located at (1.0, 1.0) and the
+//   center is at (0, 0).
+// * If the coordinates are not normalized they are interpreted as
+//   numbers of pixels.
+//
+// Arguments:
+//	input: A 4-D float tensor of shape `[batch_size, height, width, channels]`.
+//	size: A 1-D tensor of 2 elements containing the size of the glimpses
+// to extract.  The glimpse height must be specified first, following
+// by the glimpse width.
+//	offsets: A 2-D integer tensor of shape `[batch_size, 2]` containing
+// the y, x locations of the center of each window.
+//
+// Returns A tensor representing the glimpses `[batch_size,
+// glimpse_height, glimpse_width, channels]`.
+func ExtractGlimpseV2(scope *Scope, input tf.Output, size tf.Output, offsets tf.Output, optional ...ExtractGlimpseV2Attr) (glimpse tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ExtractGlimpseV2",
+		Input: []tf.Input{
+			input, size, offsets,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ExtractGlimpseAttr is an optional argument to ExtractGlimpse.
+type ExtractGlimpseAttr func(optionalAttr)
+
+// ExtractGlimpseCentered sets the optional centered attribute to value.
+//
+// value: indicates if the offset coordinates are centered relative to
+// the image, in which case the (0, 0) offset is relative to the center
+// of the input images. If false, the (0,0) offset corresponds to the
+// upper left corner of the input images.
+// If not specified, defaults to true
+func ExtractGlimpseCentered(value bool) ExtractGlimpseAttr {
+	return func(m optionalAttr) {
+		m["centered"] = value
+	}
+}
+
+// ExtractGlimpseNormalized sets the optional normalized attribute to value.
+//
+// value: indicates if the offset coordinates are normalized.
+// If not specified, defaults to true
+func ExtractGlimpseNormalized(value bool) ExtractGlimpseAttr {
+	return func(m optionalAttr) {
+		m["normalized"] = value
+	}
+}
+
+// ExtractGlimpseUniformNoise sets the optional uniform_noise attribute to value.
+//
+// value: indicates if the noise should be generated using a
+// uniform distribution or a Gaussian distribution.
+// If not specified, defaults to true
+func ExtractGlimpseUniformNoise(value bool) ExtractGlimpseAttr {
+	return func(m optionalAttr) {
+		m["uniform_noise"] = value
+	}
+}
+
+// ExtractGlimpseNoise sets the optional noise attribute to value.
+//
+// value: indicates if the noise should `uniform`, `gaussian`, or
+// `zero`. The default is `uniform` which means the the noise type
+// will be decided by `uniform_noise`.
+// If not specified, defaults to "uniform"
+func ExtractGlimpseNoise(value string) ExtractGlimpseAttr {
+	return func(m optionalAttr) {
+		m["noise"] = value
+	}
+}
+
+// Extracts a glimpse from the input tensor.
+//
+// Returns a set of windows called glimpses extracted at location
+// `offsets` from the input tensor. If the windows only partially
+// overlaps the inputs, the non overlapping areas will be filled with
+// random noise.
+//
+// The result is a 4-D tensor of shape `[batch_size, glimpse_height,
+// glimpse_width, channels]`. The channels and batch dimensions are the
+// same as that of the input tensor. The height and width of the output
+// windows are specified in the `size` parameter.
+//
+// The argument `normalized` and `centered` controls how the windows are built:
+//
+// * If the coordinates are normalized but not centered, 0.0 and 1.0
+//   correspond to the minimum and maximum of each height and width
+//   dimension.
+// * If the coordinates are both normalized and centered, they range from
+//   -1.0 to 1.0. The coordinates (-1.0, -1.0) correspond to the upper
+//   left corner, the lower right corner is located at (1.0, 1.0) and the
+//   center is at (0, 0).
+// * If the coordinates are not normalized they are interpreted as
+//   numbers of pixels.
+//
+// Arguments:
+//	input: A 4-D float tensor of shape `[batch_size, height, width, channels]`.
+//	size: A 1-D tensor of 2 elements containing the size of the glimpses
+// to extract.  The glimpse height must be specified first, following
+// by the glimpse width.
+//	offsets: A 2-D integer tensor of shape `[batch_size, 2]` containing
+// the y, x locations of the center of each window.
+//
+// Returns A tensor representing the glimpses `[batch_size,
+// glimpse_height, glimpse_width, channels]`.
+func ExtractGlimpse(scope *Scope, input tf.Output, size tf.Output, offsets tf.Output, optional ...ExtractGlimpseAttr) (glimpse tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ExtractGlimpse",
+		Input: []tf.Input{
+			input, size, offsets,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SampleDistortedBoundingBoxAttr is an optional argument to SampleDistortedBoundingBox.
+type SampleDistortedBoundingBoxAttr func(optionalAttr)
+
+// SampleDistortedBoundingBoxSeed sets the optional seed attribute to value.
+//
+// value: If either `seed` or `seed2` are set to non-zero, the random number
+// generator is seeded by the given `seed`.  Otherwise, it is seeded by a random
+// seed.
+// If not specified, defaults to 0
+func SampleDistortedBoundingBoxSeed(value int64) SampleDistortedBoundingBoxAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// SampleDistortedBoundingBoxSeed2 sets the optional seed2 attribute to value.
+//
+// value: A second seed to avoid seed collision.
+// If not specified, defaults to 0
+func SampleDistortedBoundingBoxSeed2(value int64) SampleDistortedBoundingBoxAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// SampleDistortedBoundingBoxMinObjectCovered sets the optional min_object_covered attribute to value.
+//
+// value: The cropped area of the image must contain at least this
+// fraction of any bounding box supplied. The value of this parameter should be
+// non-negative. In the case of 0, the cropped area does not need to overlap
+// any of the bounding boxes supplied.
+// If not specified, defaults to 0.1
+func SampleDistortedBoundingBoxMinObjectCovered(value float32) SampleDistortedBoundingBoxAttr {
+	return func(m optionalAttr) {
+		m["min_object_covered"] = value
+	}
+}
+
+// SampleDistortedBoundingBoxAspectRatioRange sets the optional aspect_ratio_range attribute to value.
+//
+// value: The cropped area of the image must have an aspect ratio =
+// width / height within this range.
+// If not specified, defaults to <f:0.75 f:1.33 >
+func SampleDistortedBoundingBoxAspectRatioRange(value []float32) SampleDistortedBoundingBoxAttr {
+	return func(m optionalAttr) {
+		m["aspect_ratio_range"] = value
+	}
+}
+
+// SampleDistortedBoundingBoxAreaRange sets the optional area_range attribute to value.
+//
+// value: The cropped area of the image must contain a fraction of the
+// supplied image within this range.
+// If not specified, defaults to <f:0.05 f:1 >
+func SampleDistortedBoundingBoxAreaRange(value []float32) SampleDistortedBoundingBoxAttr {
+	return func(m optionalAttr) {
+		m["area_range"] = value
+	}
+}
+
+// SampleDistortedBoundingBoxMaxAttempts sets the optional max_attempts attribute to value.
+//
+// value: Number of attempts at generating a cropped region of the image
+// of the specified constraints. After `max_attempts` failures, return the entire
+// image.
+// If not specified, defaults to 100
+func SampleDistortedBoundingBoxMaxAttempts(value int64) SampleDistortedBoundingBoxAttr {
+	return func(m optionalAttr) {
+		m["max_attempts"] = value
+	}
+}
+
+// SampleDistortedBoundingBoxUseImageIfNoBoundingBoxes sets the optional use_image_if_no_bounding_boxes attribute to value.
+//
+// value: Controls behavior if no bounding boxes supplied.
+// If true, assume an implicit bounding box covering the whole input. If false,
+// raise an error.
+// If not specified, defaults to false
+func SampleDistortedBoundingBoxUseImageIfNoBoundingBoxes(value bool) SampleDistortedBoundingBoxAttr {
+	return func(m optionalAttr) {
+		m["use_image_if_no_bounding_boxes"] = value
+	}
+}
+
+// Generate a single randomly distorted bounding box for an image.
+//
+// Bounding box annotations are often supplied in addition to ground-truth labels
+// in image recognition or object localization tasks. A common technique for
+// training such a system is to randomly distort an image while preserving
+// its content, i.e. *data augmentation*. This Op outputs a randomly distorted
+// localization of an object, i.e. bounding box, given an `image_size`,
+// `bounding_boxes` and a series of constraints.
+//
+// The output of this Op is a single bounding box that may be used to crop the
+// original image. The output is returned as 3 tensors: `begin`, `size` and
+// `bboxes`. The first 2 tensors can be fed directly into `tf.slice` to crop the
+// image. The latter may be supplied to `tf.image.draw_bounding_boxes` to visualize
+// what the bounding box looks like.
+//
+// Bounding boxes are supplied and returned as `[y_min, x_min, y_max, x_max]`. The
+// bounding box coordinates are floats in `[0.0, 1.0]` relative to the width and
+// height of the underlying image.
+//
+// For example,
+//
+// ```python
+//     # Generate a single distorted bounding box.
+//     begin, size, bbox_for_draw = tf.image.sample_distorted_bounding_box(
+//         tf.shape(image),
+//         bounding_boxes=bounding_boxes)
+//
+//     # Draw the bounding box in an image summary.
+//     image_with_box = tf.image.draw_bounding_boxes(tf.expand_dims(image, 0),
+//                                                   bbox_for_draw)
+//     tf.summary.image('images_with_box', image_with_box)
+//
+//     # Employ the bounding box to distort the image.
+//     distorted_image = tf.slice(image, begin, size)
+// ```
+//
+// Note that if no bounding box information is available, setting
+// `use_image_if_no_bounding_boxes = true` will assume there is a single implicit
+// bounding box covering the whole image. If `use_image_if_no_bounding_boxes` is
+// false and no bounding boxes are supplied, an error is raised.
+//
+// Arguments:
+//	image_size: 1-D, containing `[height, width, channels]`.
+//	bounding_boxes: 3-D with shape `[batch, N, 4]` describing the N bounding boxes
+// associated with the image.
+//
+// Returns:
+//	begin: 1-D, containing `[offset_height, offset_width, 0]`. Provide as input to
+// `tf.slice`.
+//	size: 1-D, containing `[target_height, target_width, -1]`. Provide as input to
+// `tf.slice`.
+//	bboxes: 3-D with shape `[1, 1, 4]` containing the distorted bounding box.
+// Provide as input to `tf.image.draw_bounding_boxes`.
+func SampleDistortedBoundingBox(scope *Scope, image_size tf.Output, bounding_boxes tf.Output, optional ...SampleDistortedBoundingBoxAttr) (begin tf.Output, size tf.Output, bboxes tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SampleDistortedBoundingBox",
+		Input: []tf.Input{
+			image_size, bounding_boxes,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Draw bounding boxes on a batch of images.
+//
+// Outputs a copy of `images` but draws on top of the pixels zero or more bounding
+// boxes specified by the locations in `boxes`. The coordinates of the each
+// bounding box in `boxes` are encoded as `[y_min, x_min, y_max, x_max]`. The
+// bounding box coordinates are floats in `[0.0, 1.0]` relative to the width and
+// height of the underlying image.
+//
+// For example, if an image is 100 x 200 pixels (height x width) and the bounding
+// box is `[0.1, 0.2, 0.5, 0.9]`, the upper-left and bottom-right coordinates of
+// the bounding box will be `(40, 10)` to `(100, 50)` (in (x,y) coordinates).
+//
+// Parts of the bounding box may fall outside the image.
+//
+// Arguments:
+//	images: 4-D with shape `[batch, height, width, depth]`. A batch of images.
+//	boxes: 3-D with shape `[batch, num_bounding_boxes, 4]` containing bounding
+// boxes.
+//	colors: 2-D. A list of RGBA colors to cycle through for the boxes.
+//
+// Returns 4-D with the same shape as `images`. The batch of input images with
+// bounding boxes drawn on the images.
+func DrawBoundingBoxesV2(scope *Scope, images tf.Output, boxes tf.Output, colors tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "DrawBoundingBoxesV2",
+		Input: []tf.Input{
+			images, boxes, colors,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Draw bounding boxes on a batch of images.
+//
+// Outputs a copy of `images` but draws on top of the pixels zero or more bounding
+// boxes specified by the locations in `boxes`. The coordinates of the each
+// bounding box in `boxes` are encoded as `[y_min, x_min, y_max, x_max]`. The
+// bounding box coordinates are floats in `[0.0, 1.0]` relative to the width and
+// height of the underlying image.
+//
+// For example, if an image is 100 x 200 pixels (height x width) and the bounding
+// box is `[0.1, 0.2, 0.5, 0.9]`, the upper-left and bottom-right coordinates of
+// the bounding box will be `(40, 10)` to `(180, 50)` (in (x,y) coordinates).
+//
+// Parts of the bounding box may fall outside the image.
+//
+// Arguments:
+//	images: 4-D with shape `[batch, height, width, depth]`. A batch of images.
+//	boxes: 3-D with shape `[batch, num_bounding_boxes, 4]` containing bounding
+// boxes.
+//
+// Returns 4-D with the same shape as `images`. The batch of input images with
+// bounding boxes drawn on the images.
+func DrawBoundingBoxes(scope *Scope, images tf.Output, boxes tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "DrawBoundingBoxes",
+		Input: []tf.Input{
+			images, boxes,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Convert one or more images from HSV to RGB.
+//
+// Outputs a tensor of the same shape as the `images` tensor, containing the RGB
+// value of the pixels. The output is only well defined if the value in `images`
+// are in `[0,1]`.
+//
+// See `rgb_to_hsv` for a description of the HSV encoding.
+//
+// Arguments:
+//	images: 1-D or higher rank. HSV data to convert. Last dimension must be size 3.
+//
+// Returns `images` converted to RGB.
+func HSVToRGB(scope *Scope, images tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "HSVToRGB",
+		Input: []tf.Input{
+			images,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Converts one or more images from RGB to HSV.
+//
+// Outputs a tensor of the same shape as the `images` tensor, containing the HSV
+// value of the pixels. The output is only well defined if the value in `images`
+// are in `[0,1]`.
+//
+// `output[..., 0]` contains hue, `output[..., 1]` contains saturation, and
+// `output[..., 2]` contains value. All HSV values are in `[0,1]`. A hue of 0
+// corresponds to pure red, hue 1/3 is pure green, and 2/3 is pure blue.
+//
+// Usage Example:
+//
+// >>> blue_image = tf.stack([
+// ...    tf.zeros([5,5]),
+// ...    tf.zeros([5,5]),
+// ...    tf.ones([5,5])],
+// ...    axis=-1)
+// >>> blue_hsv_image = tf.image.rgb_to_hsv(blue_image)
+// >>> blue_hsv_image[0,0].numpy()
+// array([0.6666667, 1. , 1. ], dtype=float32)
+//
+//
+// Arguments:
+//	images: 1-D or higher rank. RGB data to convert. Last dimension must be size 3.
+//
+// Returns `images` converted to HSV.
+func RGBToHSV(scope *Scope, images tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "RGBToHSV",
+		Input: []tf.Input{
+			images,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Decode the frame(s) of a GIF-encoded image to a uint8 tensor.
+//
+// GIF images with frame or transparency compression are not supported.
+// On Linux and MacOS systems, convert animated GIFs from compressed to
+// uncompressed by running:
+//
+//     convert $src.gif -coalesce $dst.gif
+//
+// This op also supports decoding JPEGs and PNGs, though it is cleaner to use
+// `tf.io.decode_image`.
+//
+// Arguments:
+//	contents: 0-D.  The GIF-encoded image.
+//
+// Returns 4-D with shape `[num_frames, height, width, 3]`. RGB channel order.
+func DecodeGif(scope *Scope, contents tf.Output) (image tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "DecodeGif",
+		Input: []tf.Input{
+			contents,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// DecodeBmpAttr is an optional argument to DecodeBmp.
+type DecodeBmpAttr func(optionalAttr)
+
+// DecodeBmpChannels sets the optional channels attribute to value.
+// If not specified, defaults to 0
+func DecodeBmpChannels(value int64) DecodeBmpAttr {
+	return func(m optionalAttr) {
+		m["channels"] = value
+	}
+}
+
+// Decode the first frame of a BMP-encoded image to a uint8 tensor.
+//
+// The attr `channels` indicates the desired number of color channels for the
+// decoded image.
+//
+// Accepted values are:
+//
+// *   0: Use the number of channels in the BMP-encoded image.
+// *   3: output an RGB image.
+// *   4: output an RGBA image.
+//
+// Arguments:
+//	contents: 0-D.  The BMP-encoded image.
+//
+// Returns 3-D with shape `[height, width, channels]`. RGB order
+func DecodeBmp(scope *Scope, contents tf.Output, optional ...DecodeBmpAttr) (image tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DecodeBmp",
+		Input: []tf.Input{
+			contents,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// EncodePngAttr is an optional argument to EncodePng.
+type EncodePngAttr func(optionalAttr)
+
+// EncodePngCompression sets the optional compression attribute to value.
+//
+// value: Compression level.
+// If not specified, defaults to -1
+func EncodePngCompression(value int64) EncodePngAttr {
+	return func(m optionalAttr) {
+		m["compression"] = value
+	}
+}
+
+// PNG-encode an image.
+//
+// `image` is a 3-D uint8 or uint16 Tensor of shape `[height, width, channels]`
+// where `channels` is:
+//
+// *   1: for grayscale.
+// *   2: for grayscale + alpha.
+// *   3: for RGB.
+// *   4: for RGBA.
+//
+// The ZLIB compression level, `compression`, can be -1 for the PNG-encoder
+// default or a value from 0 to 9.  9 is the highest compression level, generating
+// the smallest output, but is slower.
+//
+// Arguments:
+//	image: 3-D with shape `[height, width, channels]`.
+//
+// Returns 0-D. PNG-encoded image.
+func EncodePng(scope *Scope, image tf.Output, optional ...EncodePngAttr) (contents tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "EncodePng",
+		Input: []tf.Input{
+			image,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Invert (flip) each bit of supported types; for example, type `uint8` value 01010101 becomes 10101010.
+//
+// Flip each bit of supported types.  For example, type `int8` (decimal 2) binary 00000010 becomes (decimal -3) binary 11111101.
+// This operation is performed on each element of the tensor argument `x`.
+//
+// Example:
+// ```python
+// import tensorflow as tf
+// from tensorflow.python.ops import bitwise_ops
+//
+// # flip 2 (00000010) to -3 (11111101)
+// tf.assert_equal(-3, bitwise_ops.invert(2))
+//
+// dtype_list = [dtypes.int8, dtypes.int16, dtypes.int32, dtypes.int64,
+//               dtypes.uint8, dtypes.uint16, dtypes.uint32, dtypes.uint64]
+//
+// inputs = [0, 5, 3, 14]
+// for dtype in dtype_list:
+//   # Because of issues with negative numbers, let's test this indirectly.
+//   # 1. invert(a) and a = 0
+//   # 2. invert(a) or a = invert(0)
+//   input_tensor = tf.constant([0, 5, 3, 14], dtype=dtype)
+//   not_a_and_a, not_a_or_a, not_0 = [bitwise_ops.bitwise_and(
+//                                       input_tensor, bitwise_ops.invert(input_tensor)),
+//                                     bitwise_ops.bitwise_or(
+//                                       input_tensor, bitwise_ops.invert(input_tensor)),
+//                                     bitwise_ops.invert(
+//                                       tf.constant(0, dtype=dtype))]
+//
+//   expected = tf.constant([0, 0, 0, 0], dtype=tf.float32)
+//   tf.assert_equal(tf.cast(not_a_and_a, tf.float32), expected)
+//
+//   expected = tf.cast([not_0] * 4, tf.float32)
+//   tf.assert_equal(tf.cast(not_a_or_a, tf.float32), expected)
+//
+//   # For unsigned dtypes let's also check the result directly.
+//   if dtype.is_unsigned:
+//     inverted = bitwise_ops.invert(input_tensor)
+//     expected = tf.constant([dtype.max - x for x in inputs], dtype=tf.float32)
+//     tf.assert_equal(tf.cast(inverted, tf.float32), tf.cast(expected, tf.float32))
+// ```
+func Invert(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Invert",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// DecodePngAttr is an optional argument to DecodePng.
+type DecodePngAttr func(optionalAttr)
+
+// DecodePngChannels sets the optional channels attribute to value.
+//
+// value: Number of color channels for the decoded image.
+// If not specified, defaults to 0
+func DecodePngChannels(value int64) DecodePngAttr {
+	return func(m optionalAttr) {
+		m["channels"] = value
+	}
+}
+
+// DecodePngDtype sets the optional dtype attribute to value.
+// If not specified, defaults to DT_UINT8
+func DecodePngDtype(value tf.DataType) DecodePngAttr {
+	return func(m optionalAttr) {
+		m["dtype"] = value
+	}
+}
+
+// Decode a PNG-encoded image to a uint8 or uint16 tensor.
+//
+// The attr `channels` indicates the desired number of color channels for the
+// decoded image.
+//
+// Accepted values are:
+//
+// *   0: Use the number of channels in the PNG-encoded image.
+// *   1: output a grayscale image.
+// *   3: output an RGB image.
+// *   4: output an RGBA image.
+//
+// If needed, the PNG-encoded image is transformed to match the requested number
+// of color channels.
+//
+// This op also supports decoding JPEGs and non-animated GIFs since the interface
+// is the same, though it is cleaner to use `tf.io.decode_image`.
+//
+// Arguments:
+//	contents: 0-D.  The PNG-encoded image.
+//
+// Returns 3-D with shape `[height, width, channels]`.
+func DecodePng(scope *Scope, contents tf.Output, optional ...DecodePngAttr) (image tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DecodePng",
+		Input: []tf.Input{
+			contents,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Adjust the saturation of one or more images.
+//
+// `images` is a tensor of at least 3 dimensions.  The last dimension is
+// interpreted as channels, and must be three.
+//
+// The input image is considered in the RGB colorspace. Conceptually, the RGB
+// colors are first mapped into HSV. A scale is then applied all the saturation
+// values, and then remapped back to RGB colorspace.
+//
+// Arguments:
+//	images: Images to adjust.  At least 3-D.
+//	scale: A float scale to add to the saturation.
+//
+// Returns The hue-adjusted image or images.
+func AdjustSaturation(scope *Scope, images tf.Output, scale tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "AdjustSaturation",
+		Input: []tf.Input{
+			images, scale,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ExtractJpegShapeAttr is an optional argument to ExtractJpegShape.
+type ExtractJpegShapeAttr func(optionalAttr)
+
+// ExtractJpegShapeOutputType sets the optional output_type attribute to value.
+//
+// value: (Optional) The output type of the operation (int32 or int64).
+// Defaults to int32.
+// If not specified, defaults to DT_INT32
+func ExtractJpegShapeOutputType(value tf.DataType) ExtractJpegShapeAttr {
+	return func(m optionalAttr) {
+		m["output_type"] = value
+	}
+}
+
+// Extract the shape information of a JPEG-encoded image.
+//
+// This op only parses the image header, so it is much faster than DecodeJpeg.
+//
+// Arguments:
+//	contents: 0-D. The JPEG-encoded image.
+//
+// Returns 1-D. The image shape with format [height, width, channels].
+func ExtractJpegShape(scope *Scope, contents tf.Output, optional ...ExtractJpegShapeAttr) (image_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ExtractJpegShape",
+		Input: []tf.Input{
+			contents,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RandomCropAttr is an optional argument to RandomCrop.
+type RandomCropAttr func(optionalAttr)
+
+// RandomCropSeed sets the optional seed attribute to value.
+//
+// value: If either seed or seed2 are set to be non-zero, the random number
+// generator is seeded by the given seed.  Otherwise, it is seeded by a
+// random seed.
+// If not specified, defaults to 0
+func RandomCropSeed(value int64) RandomCropAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// RandomCropSeed2 sets the optional seed2 attribute to value.
+//
+// value: An second seed to avoid seed collision.
+// If not specified, defaults to 0
+func RandomCropSeed2(value int64) RandomCropAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Randomly crop `image`.
+//
+// DEPRECATED at GraphDef version 8: Random crop is now pure Python
+//
+// `size` is a 1-D int64 tensor with 2 elements representing the crop height and
+// width.  The values must be non negative.
+//
+// This Op picks a random location in `image` and crops a `height` by `width`
+// rectangle from that location.  The random location is picked so the cropped
+// area will fit inside the original image.
+//
+// Arguments:
+//	image: 3-D of shape `[height, width, channels]`.
+//	size: 1-D of length 2 containing: `crop_height`, `crop_width`..
+//
+// Returns 3-D of shape `[crop_height, crop_width, channels].`
+func RandomCrop(scope *Scope, image tf.Output, size tf.Output, optional ...RandomCropAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RandomCrop",
+		Input: []tf.Input{
+			image, size,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResizeNearestNeighborGradAttr is an optional argument to ResizeNearestNeighborGrad.
+type ResizeNearestNeighborGradAttr func(optionalAttr)
+
+// ResizeNearestNeighborGradAlignCorners sets the optional align_corners attribute to value.
+//
+// value: If true, the centers of the 4 corner pixels of the input and grad tensors are
+// aligned. Defaults to false.
+// If not specified, defaults to false
+func ResizeNearestNeighborGradAlignCorners(value bool) ResizeNearestNeighborGradAttr {
+	return func(m optionalAttr) {
+		m["align_corners"] = value
+	}
+}
+
+// ResizeNearestNeighborGradHalfPixelCenters sets the optional half_pixel_centers attribute to value.
+// If not specified, defaults to false
+func ResizeNearestNeighborGradHalfPixelCenters(value bool) ResizeNearestNeighborGradAttr {
+	return func(m optionalAttr) {
+		m["half_pixel_centers"] = value
+	}
+}
+
+// Computes the gradient of nearest neighbor interpolation.
+//
+// Arguments:
+//	grads: 4-D with shape `[batch, height, width, channels]`.
+//	size: = A 1-D int32 Tensor of 2 elements: `orig_height, orig_width`. The
+// original input size.
+//
+// Returns 4-D with shape `[batch, orig_height, orig_width, channels]`. Gradients
+// with respect to the input image.
+func ResizeNearestNeighborGrad(scope *Scope, grads tf.Output, size tf.Output, optional ...ResizeNearestNeighborGradAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResizeNearestNeighborGrad",
+		Input: []tf.Input{
+			grads, size,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Initializes the multi device iterator with the given dataset.
+//
+// Arguments:
+//	dataset: Dataset to be iterated upon.
+//	multi_device_iterator: A MultiDeviceIteratorResource.
+//	max_buffer_size: The maximum size of the host side per device buffer to keep.
+//
+// Returns An int64 indicating which incarnation of the MultiDeviceIterator
+// is running.
+func MultiDeviceIteratorInit(scope *Scope, dataset tf.Output, multi_device_iterator tf.Output, max_buffer_size tf.Output) (incarnation_id tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "MultiDeviceIteratorInit",
+		Input: []tf.Input{
+			dataset, multi_device_iterator, max_buffer_size,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Deprecated. Disallowed in GraphDef version >= 2.
+//
+// DEPRECATED at GraphDef version 2: Use AdjustContrastv2 instead
+func AdjustContrast(scope *Scope, images tf.Output, contrast_factor tf.Output, min_value tf.Output, max_value tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "AdjustContrast",
+		Input: []tf.Input{
+			images, contrast_factor, min_value, max_value,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// A placeholder op for a value that will be fed into the computation.
+//
+// Arguments:
+//	dtype: The type of elements in the tensor.
+//	shape: The shape of the tensor.
+//
+// Returns A tensor that will be provided using the infeed mechanism.
+func InfeedDequeue(scope *Scope, dtype tf.DataType, shape tf.Shape) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype, "shape": shape}
+	opspec := tf.OpSpec{
+		Type: "InfeedDequeue",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Encodes a `RaggedTensor` into a `variant` Tensor.
+//
+//
+// Encodes the given `RaggedTensor` and returns a `variant` Tensor. If
+// `batched_input` is True, then input `RaggedTensor` is unbatched along the
+// zero-th dimension, each component `RaggedTensor` is encoded into a scalar
+// `variant` Tensor, and these are stacked to return a 1-D `variant` Tensor.
+// If `batched_input` is False, then the input `RaggedTensor` is encoded as is and
+// a scalar `variant` Tensor is returned. A `RaggedTensor` is encoded by first
+// creating a 1-D `variant` Tensor with `ragged_rank + 1` elements, containing the
+// splits and values Tensors of the `RaggedTensor`. Then the 1-D `variant` Tensor
+// is wrapped in a scalar `variant` Tensor. See `RaggedTensorFromVariant` for the
+// corresponding decoding logic.
+//
+//
+// Arguments:
+//	rt_nested_splits: A list of one or more Tensors representing the splits of the input
+// `RaggedTensor`.
+//	rt_dense_values: A Tensor representing the values of the input `RaggedTensor`.
+//	batched_input: A `bool` denoting whether the input is a batched `RaggedTensor`.
+//
+// Returns A `variant` Tensor that containing encoded `RaggedTensor`.
+func RaggedTensorToVariant(scope *Scope, rt_nested_splits []tf.Output, rt_dense_values tf.Output, batched_input bool) (encoded_ragged tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"batched_input": batched_input}
+	opspec := tf.OpSpec{
+		Type: "RaggedTensorToVariant",
+		Input: []tf.Input{
+			tf.OutputList(rt_nested_splits), rt_dense_values,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResizeNearestNeighborAttr is an optional argument to ResizeNearestNeighbor.
+type ResizeNearestNeighborAttr func(optionalAttr)
+
+// ResizeNearestNeighborAlignCorners sets the optional align_corners attribute to value.
+//
+// value: If true, the centers of the 4 corner pixels of the input and output tensors are
+// aligned, preserving the values at the corner pixels. Defaults to false.
+// If not specified, defaults to false
+func ResizeNearestNeighborAlignCorners(value bool) ResizeNearestNeighborAttr {
+	return func(m optionalAttr) {
+		m["align_corners"] = value
+	}
+}
+
+// ResizeNearestNeighborHalfPixelCenters sets the optional half_pixel_centers attribute to value.
+// If not specified, defaults to false
+func ResizeNearestNeighborHalfPixelCenters(value bool) ResizeNearestNeighborAttr {
+	return func(m optionalAttr) {
+		m["half_pixel_centers"] = value
+	}
+}
+
+// Resize `images` to `size` using nearest neighbor interpolation.
+//
+// Arguments:
+//	images: 4-D with shape `[batch, height, width, channels]`.
+//	size: = A 1-D int32 Tensor of 2 elements: `new_height, new_width`.  The
+// new size for the images.
+//
+// Returns 4-D with shape
+// `[batch, new_height, new_width, channels]`.
+func ResizeNearestNeighbor(scope *Scope, images tf.Output, size tf.Output, optional ...ResizeNearestNeighborAttr) (resized_images tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResizeNearestNeighbor",
+		Input: []tf.Input{
+			images, size,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Runs multiple additive regression ensemble predictors on input instances and
+//
+// computes the logits. It is designed to be used during prediction.
+// It traverses all the trees and calculates the final score for each instance.
+//
+// Arguments:
+//
+//	bucketized_features: A list of rank 1 Tensors containing bucket id for each
+// feature.
+//	logits_dimension: scalar, dimension of the logits, to be used for partial logits
+// shape.
+//
+// Returns Output rank 2 Tensor containing logits for each example.
+func BoostedTreesPredict(scope *Scope, tree_ensemble_handle tf.Output, bucketized_features []tf.Output, logits_dimension int64) (logits tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"logits_dimension": logits_dimension}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesPredict",
+		Input: []tf.Input{
+			tree_ensemble_handle, tf.OutputList(bucketized_features),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RandomGammaAttr is an optional argument to RandomGamma.
+type RandomGammaAttr func(optionalAttr)
+
+// RandomGammaSeed sets the optional seed attribute to value.
+//
+// value: If either `seed` or `seed2` are set to be non-zero, the random number
+// generator is seeded by the given seed.  Otherwise, it is seeded by a
+// random seed.
+// If not specified, defaults to 0
+func RandomGammaSeed(value int64) RandomGammaAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// RandomGammaSeed2 sets the optional seed2 attribute to value.
+//
+// value: A second seed to avoid seed collision.
+// If not specified, defaults to 0
+func RandomGammaSeed2(value int64) RandomGammaAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Outputs random values from the Gamma distribution(s) described by alpha.
+//
+// This op uses the algorithm by Marsaglia et al. to acquire samples via
+// transformation-rejection from pairs of uniform and normal random variables.
+// See http://dl.acm.org/citation.cfm?id=358414
+//
+// Arguments:
+//	shape: 1-D integer tensor. Shape of independent samples to draw from each
+// distribution described by the shape parameters given in alpha.
+//	alpha: A tensor in which each scalar is a "shape" parameter describing the
+// associated gamma distribution.
+//
+// Returns A tensor with shape `shape + shape(alpha)`. Each slice
+// `[:, ..., :, i0, i1, ...iN]` contains the samples drawn for
+// `alpha[i0, i1, ...iN]`. The dtype of the output matches the dtype of alpha.
+func RandomGamma(scope *Scope, shape tf.Output, alpha tf.Output, optional ...RandomGammaAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RandomGamma",
+		Input: []tf.Input{
+			shape, alpha,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns 0 if x == 0, and x * log1p(y) otherwise, elementwise.
+func Xlog1py(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Xlog1py",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// QuantizedResizeBilinearAttr is an optional argument to QuantizedResizeBilinear.
+type QuantizedResizeBilinearAttr func(optionalAttr)
+
+// QuantizedResizeBilinearAlignCorners sets the optional align_corners attribute to value.
+//
+// value: If true, the centers of the 4 corner pixels of the input and output tensors are
+// aligned, preserving the values at the corner pixels. Defaults to false.
+// If not specified, defaults to false
+func QuantizedResizeBilinearAlignCorners(value bool) QuantizedResizeBilinearAttr {
+	return func(m optionalAttr) {
+		m["align_corners"] = value
+	}
+}
+
+// QuantizedResizeBilinearHalfPixelCenters sets the optional half_pixel_centers attribute to value.
+// If not specified, defaults to false
+func QuantizedResizeBilinearHalfPixelCenters(value bool) QuantizedResizeBilinearAttr {
+	return func(m optionalAttr) {
+		m["half_pixel_centers"] = value
+	}
+}
+
+// Resize quantized `images` to `size` using quantized bilinear interpolation.
+//
+// Input images and output images must be quantized types.
+//
+// Arguments:
+//	images: 4-D with shape `[batch, height, width, channels]`.
+//	size: = A 1-D int32 Tensor of 2 elements: `new_height, new_width`.  The
+// new size for the images.
+//
+//
+//
+// Returns:
+//	resized_images: 4-D with shape
+// `[batch, new_height, new_width, channels]`.
+//	out_min
+//	out_max
+func QuantizedResizeBilinear(scope *Scope, images tf.Output, size tf.Output, min tf.Output, max tf.Output, optional ...QuantizedResizeBilinearAttr) (resized_images tf.Output, out_min tf.Output, out_max tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizedResizeBilinear",
+		Input: []tf.Input{
+			images, size, min, max,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// ResizeAreaAttr is an optional argument to ResizeArea.
+type ResizeAreaAttr func(optionalAttr)
+
+// ResizeAreaAlignCorners sets the optional align_corners attribute to value.
+//
+// value: If true, the centers of the 4 corner pixels of the input and output tensors are
+// aligned, preserving the values at the corner pixels. Defaults to false.
+// If not specified, defaults to false
+func ResizeAreaAlignCorners(value bool) ResizeAreaAttr {
+	return func(m optionalAttr) {
+		m["align_corners"] = value
+	}
+}
+
+// Resize `images` to `size` using area interpolation.
+//
+// Input images can be of different types but output images are always float.
+//
+// The range of pixel values for the output image might be slightly different
+// from the range for the input image because of limited numerical precision.
+// To guarantee an output range, for example `[0.0, 1.0]`, apply
+// `tf.clip_by_value` to the output.
+//
+// Each output pixel is computed by first transforming the pixel's footprint into
+// the input tensor and then averaging the pixels that intersect the footprint. An
+// input pixel's contribution to the average is weighted by the fraction of its
+// area that intersects the footprint.  This is the same as OpenCV's INTER_AREA.
+//
+// Arguments:
+//	images: 4-D with shape `[batch, height, width, channels]`.
+//	size: = A 1-D int32 Tensor of 2 elements: `new_height, new_width`.  The
+// new size for the images.
+//
+// Returns 4-D with shape
+// `[batch, new_height, new_width, channels]`.
+func ResizeArea(scope *Scope, images tf.Output, size tf.Output, optional ...ResizeAreaAttr) (resized_images tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResizeArea",
+		Input: []tf.Input{
+			images, size,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Restore a reader to a previously saved state.
+//
+// Not all Readers support being restored, so this can produce an
+// Unimplemented error.
+//
+// Arguments:
+//	reader_handle: Handle to a Reader.
+//	state: Result of a ReaderSerializeState of a Reader with type
+// matching reader_handle.
+//
+// Returns the created operation.
+func ReaderRestoreStateV2(scope *Scope, reader_handle tf.Output, state tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ReaderRestoreStateV2",
+		Input: []tf.Input{
+			reader_handle, state,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Computes rectified linear 6: `min(max(features, 0), 6)`.
+func Relu6(scope *Scope, features tf.Output) (activations tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Relu6",
+		Input: []tf.Input{
+			features,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RaggedRangeAttr is an optional argument to RaggedRange.
+type RaggedRangeAttr func(optionalAttr)
+
+// RaggedRangeTsplits sets the optional Tsplits attribute to value.
+// If not specified, defaults to DT_INT64
+func RaggedRangeTsplits(value tf.DataType) RaggedRangeAttr {
+	return func(m optionalAttr) {
+		m["Tsplits"] = value
+	}
+}
+
+// Returns a `RaggedTensor` containing the specified sequences of numbers.
+//
+//
+// Returns a `RaggedTensor` `result` composed from `rt_dense_values` and
+// `rt_nested_splits`, such that
+// `result[i] = range(starts[i], limits[i], deltas[i])`.
+//
+// ```python
+// (rt_nested_splits, rt_dense_values) = ragged_range(
+//       starts=[2, 5, 8], limits=[3, 5, 12], deltas=1)
+// result = tf.ragged.from_row_splits(rt_dense_values, rt_nested_splits)
+// print(result)
+// <tf.RaggedTensor [[2], [], [8, 9, 10, 11]] >
+// ```
+//
+// The input tensors `starts`, `limits`, and `deltas` may be scalars or vectors.
+// The vector inputs must all have the same size.  Scalar inputs are broadcast
+// to match the size of the vector inputs.
+//
+// Arguments:
+//	starts: The starts of each range.
+//	limits: The limits of each range.
+//	deltas: The deltas of each range.
+//
+// Returns:
+//	rt_nested_splits: The `row_splits` for the returned `RaggedTensor`.
+//	rt_dense_values: The `flat_values` for the returned `RaggedTensor`.
+func RaggedRange(scope *Scope, starts tf.Output, limits tf.Output, deltas tf.Output, optional ...RaggedRangeAttr) (rt_nested_splits tf.Output, rt_dense_values tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RaggedRange",
+		Input: []tf.Input{
+			starts, limits, deltas,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Split a `SparseTensor` into `num_split` tensors along one dimension.
+//
+// If the `shape[split_dim]` is not an integer multiple of `num_split`. Slices
+// `[0 : shape[split_dim] % num_split]` gets one extra dimension.
+// For example, if `split_dim = 1` and `num_split = 2` and the input is
+//
+//     input_tensor = shape = [2, 7]
+//     [    a   d e  ]
+//     [b c          ]
+//
+// Graphically the output tensors are:
+//
+//     output_tensor[0] = shape = [2, 4]
+//     [    a  ]
+//     [b c    ]
+//
+//     output_tensor[1] = shape = [2, 3]
+//     [ d e  ]
+//     [      ]
+//
+// Arguments:
+//	split_dim: 0-D.  The dimension along which to split.  Must be in the range
+// `[0, rank(shape))`.
+//	indices: 2-D tensor represents the indices of the sparse tensor.
+//	values: 1-D tensor represents the values of the sparse tensor.
+//	shape: 1-D. tensor represents the shape of the sparse tensor.
+// output indices: A list of 1-D tensors represents the indices of the output
+// sparse tensors.
+//	num_split: The number of ways to split.
+//
+// Returns:
+//	output_indices
+//	output_values: A list of 1-D tensors represents the values of the output sparse
+// tensors.
+//	output_shape: A list of 1-D tensors represents the shape of the output sparse
+// tensors.
+func SparseSplit(scope *Scope, split_dim tf.Output, indices tf.Output, values tf.Output, shape tf.Output, num_split int64) (output_indices []tf.Output, output_values []tf.Output, output_shape []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_split": num_split}
+	opspec := tf.OpSpec{
+		Type: "SparseSplit",
+		Input: []tf.Input{
+			split_dim, indices, values, shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if output_indices, idx, err = makeOutputList(op, idx, "output_indices"); err != nil {
+		scope.UpdateErr("SparseSplit", err)
+		return
+	}
+	if output_values, idx, err = makeOutputList(op, idx, "output_values"); err != nil {
+		scope.UpdateErr("SparseSplit", err)
+		return
+	}
+	if output_shape, idx, err = makeOutputList(op, idx, "output_shape"); err != nil {
+		scope.UpdateErr("SparseSplit", err)
+		return
+	}
+	return output_indices, output_values, output_shape
+}
+
+// Produce a string tensor that encodes the state of a Reader.
+//
+// Not all Readers support being serialized, so this can produce an
+// Unimplemented error.
+//
+// Arguments:
+//	reader_handle: Handle to a Reader.
+func ReaderSerializeStateV2(scope *Scope, reader_handle tf.Output) (state tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ReaderSerializeStateV2",
+		Input: []tf.Input{
+			reader_handle,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns up to `num_records` (key, value) pairs produced by a Reader.
+//
+// Will dequeue from the input queue if necessary (e.g. when the
+// Reader needs to start reading from a new file since it has finished
+// with the previous file).
+// It may return less than `num_records` even before the last batch.
+//
+// Arguments:
+//	reader_handle: Handle to a `Reader`.
+//	queue_handle: Handle to a `Queue`, with string work items.
+//	num_records: number of records to read from `Reader`.
+//
+// Returns:
+//	keys: A 1-D tensor.
+//	values: A 1-D tensor.
+func ReaderReadUpToV2(scope *Scope, reader_handle tf.Output, queue_handle tf.Output, num_records tf.Output) (keys tf.Output, values tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ReaderReadUpToV2",
+		Input: []tf.Input{
+			reader_handle, queue_handle, num_records,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// QueueDequeueV2Attr is an optional argument to QueueDequeueV2.
+type QueueDequeueV2Attr func(optionalAttr)
+
+// QueueDequeueV2TimeoutMs sets the optional timeout_ms attribute to value.
+//
+// value: If the queue is empty, this operation will block for up to
+// timeout_ms milliseconds.
+// Note: This option is not supported yet.
+// If not specified, defaults to -1
+func QueueDequeueV2TimeoutMs(value int64) QueueDequeueV2Attr {
+	return func(m optionalAttr) {
+		m["timeout_ms"] = value
+	}
+}
+
+// Dequeues a tuple of one or more tensors from the given queue.
+//
+// This operation has k outputs, where k is the number of components
+// in the tuples stored in the given queue, and output i is the ith
+// component of the dequeued tuple.
+//
+// N.B. If the queue is empty, this operation will block until an element
+// has been dequeued (or 'timeout_ms' elapses, if specified).
+//
+// Arguments:
+//	handle: The handle to a queue.
+//	component_types: The type of each component in a tuple.
+//
+// Returns One or more tensors that were dequeued as a tuple.
+func QueueDequeueV2(scope *Scope, handle tf.Output, component_types []tf.DataType, optional ...QueueDequeueV2Attr) (components []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"component_types": component_types}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QueueDequeueV2",
+		Input: []tf.Input{
+			handle,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if components, idx, err = makeOutputList(op, idx, "components"); err != nil {
+		scope.UpdateErr("QueueDequeueV2", err)
+		return
+	}
+	return components
+}
+
+// Return a slice from 'input'.
+//
+// The output tensor is a tensor with dimensions described by 'size'
+// whose values are extracted from 'input' starting at the offsets in
+// 'begin'.
+//
+// *Requirements*:
+//   0 <= begin[i] <= begin[i] + size[i] <= Di  for i in [0, n)
+//
+// Arguments:
+//
+//	begin: begin[i] specifies the offset into the 'i'th dimension of
+// 'input' to slice from.
+//	size: size[i] specifies the number of elements of the 'i'th dimension
+// of 'input' to slice. If size[i] is -1, all remaining elements in dimension
+// i are included in the slice (i.e. this is equivalent to setting
+// size[i] = input.dim_size(i) - begin[i]).
+func Slice(scope *Scope, input tf.Output, begin tf.Output, size tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Slice",
+		Input: []tf.Input{
+			input, begin, size,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// TFRecordReaderV2Attr is an optional argument to TFRecordReaderV2.
+type TFRecordReaderV2Attr func(optionalAttr)
+
+// TFRecordReaderV2Container sets the optional container attribute to value.
+//
+// value: If non-empty, this reader is placed in the given container.
+// Otherwise, a default container is used.
+// If not specified, defaults to ""
+func TFRecordReaderV2Container(value string) TFRecordReaderV2Attr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// TFRecordReaderV2SharedName sets the optional shared_name attribute to value.
+//
+// value: If non-empty, this reader is named in the given bucket
+// with this shared_name. Otherwise, the node name is used instead.
+// If not specified, defaults to ""
+func TFRecordReaderV2SharedName(value string) TFRecordReaderV2Attr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// TFRecordReaderV2CompressionType sets the optional compression_type attribute to value.
+// If not specified, defaults to ""
+func TFRecordReaderV2CompressionType(value string) TFRecordReaderV2Attr {
+	return func(m optionalAttr) {
+		m["compression_type"] = value
+	}
+}
+
+// A Reader that outputs the records from a TensorFlow Records file.
+//
+// Returns The handle to reference the Reader.
+func TFRecordReaderV2(scope *Scope, optional ...TFRecordReaderV2Attr) (reader_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TFRecordReaderV2",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ParseExampleDatasetAttr is an optional argument to ParseExampleDataset.
+type ParseExampleDatasetAttr func(optionalAttr)
+
+// ParseExampleDatasetSloppy sets the optional sloppy attribute to value.
+// If not specified, defaults to false
+func ParseExampleDatasetSloppy(value bool) ParseExampleDatasetAttr {
+	return func(m optionalAttr) {
+		m["sloppy"] = value
+	}
+}
+
+// ParseExampleDatasetRaggedKeys sets the optional ragged_keys attribute to value.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseExampleDatasetRaggedKeys(value []string) ParseExampleDatasetAttr {
+	return func(m optionalAttr) {
+		m["ragged_keys"] = value
+	}
+}
+
+// ParseExampleDatasetRaggedValueTypes sets the optional ragged_value_types attribute to value.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseExampleDatasetRaggedValueTypes(value []tf.DataType) ParseExampleDatasetAttr {
+	return func(m optionalAttr) {
+		m["ragged_value_types"] = value
+	}
+}
+
+// ParseExampleDatasetRaggedSplitTypes sets the optional ragged_split_types attribute to value.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseExampleDatasetRaggedSplitTypes(value []tf.DataType) ParseExampleDatasetAttr {
+	return func(m optionalAttr) {
+		m["ragged_split_types"] = value
+	}
+}
+
+// Transforms `input_dataset` containing `Example` protos as vectors of DT_STRING into a dataset of `Tensor` or `SparseTensor` objects representing the parsed features.
+//
+// Arguments:
+//
+//
+//	dense_defaults: A dict mapping string keys to `Tensor`s.
+// The keys of the dict must match the dense_keys of the feature.
+//	sparse_keys: A list of string keys in the examples features.
+// The results for these keys will be returned as `SparseTensor` objects.
+//	dense_keys: A list of Ndense string Tensors (scalars).
+// The keys expected in the Examples features associated with dense values.
+//	sparse_types: A list of `DTypes` of the same length as `sparse_keys`.
+// Only `tf.float32` (`FloatList`), `tf.int64` (`Int64List`),
+// and `tf.string` (`BytesList`) are supported.
+//	dense_shapes: List of tuples with the same length as `dense_keys`.
+// The shape of the data for each dense feature referenced by `dense_keys`.
+// Required for any input tensors identified by `dense_keys`.  Must be
+// either fully defined, or may contain an unknown first dimension.
+// An unknown first dimension means the feature is treated as having
+// a variable number of blocks, and the output shape along this dimension
+// is considered unknown at graph build time.  Padding is applied for
+// minibatch elements smaller than the maximum number of blocks for the
+// given feature along this dimension.
+//	output_types: The type list for the return values.
+//	output_shapes: The list of shapes being produced.
+func ParseExampleDataset(scope *Scope, input_dataset tf.Output, num_parallel_calls tf.Output, dense_defaults []tf.Output, sparse_keys []string, dense_keys []string, sparse_types []tf.DataType, dense_shapes []tf.Shape, output_types []tf.DataType, output_shapes []tf.Shape, optional ...ParseExampleDatasetAttr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"sparse_keys": sparse_keys, "dense_keys": dense_keys, "sparse_types": sparse_types, "dense_shapes": dense_shapes, "output_types": output_types, "output_shapes": output_shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ParseExampleDataset",
+		Input: []tf.Input{
+			input_dataset, num_parallel_calls, tf.OutputList(dense_defaults),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// IdentityReaderV2Attr is an optional argument to IdentityReaderV2.
+type IdentityReaderV2Attr func(optionalAttr)
+
+// IdentityReaderV2Container sets the optional container attribute to value.
+//
+// value: If non-empty, this reader is placed in the given container.
+// Otherwise, a default container is used.
+// If not specified, defaults to ""
+func IdentityReaderV2Container(value string) IdentityReaderV2Attr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// IdentityReaderV2SharedName sets the optional shared_name attribute to value.
+//
+// value: If non-empty, this reader is named in the given bucket
+// with this shared_name. Otherwise, the node name is used instead.
+// If not specified, defaults to ""
+func IdentityReaderV2SharedName(value string) IdentityReaderV2Attr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// A Reader that outputs the queued work as both the key and value.
+//
+// To use, enqueue strings in a Queue.  ReaderRead will take the front
+// work string and output (work, work).
+//
+// Returns The handle to reference the Reader.
+func IdentityReaderV2(scope *Scope, optional ...IdentityReaderV2Attr) (reader_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "IdentityReaderV2",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// FixedLengthRecordReaderV2Attr is an optional argument to FixedLengthRecordReaderV2.
+type FixedLengthRecordReaderV2Attr func(optionalAttr)
+
+// FixedLengthRecordReaderV2HeaderBytes sets the optional header_bytes attribute to value.
+//
+// value: Number of bytes in the header, defaults to 0.
+// If not specified, defaults to 0
+func FixedLengthRecordReaderV2HeaderBytes(value int64) FixedLengthRecordReaderV2Attr {
+	return func(m optionalAttr) {
+		m["header_bytes"] = value
+	}
+}
+
+// FixedLengthRecordReaderV2FooterBytes sets the optional footer_bytes attribute to value.
+//
+// value: Number of bytes in the footer, defaults to 0.
+// If not specified, defaults to 0
+func FixedLengthRecordReaderV2FooterBytes(value int64) FixedLengthRecordReaderV2Attr {
+	return func(m optionalAttr) {
+		m["footer_bytes"] = value
+	}
+}
+
+// FixedLengthRecordReaderV2HopBytes sets the optional hop_bytes attribute to value.
+//
+// value: Number of bytes to hop before each read. Default of 0 means using
+// record_bytes.
+// If not specified, defaults to 0
+func FixedLengthRecordReaderV2HopBytes(value int64) FixedLengthRecordReaderV2Attr {
+	return func(m optionalAttr) {
+		m["hop_bytes"] = value
+	}
+}
+
+// FixedLengthRecordReaderV2Container sets the optional container attribute to value.
+//
+// value: If non-empty, this reader is placed in the given container.
+// Otherwise, a default container is used.
+// If not specified, defaults to ""
+func FixedLengthRecordReaderV2Container(value string) FixedLengthRecordReaderV2Attr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// FixedLengthRecordReaderV2SharedName sets the optional shared_name attribute to value.
+//
+// value: If non-empty, this reader is named in the given bucket
+// with this shared_name. Otherwise, the node name is used instead.
+// If not specified, defaults to ""
+func FixedLengthRecordReaderV2SharedName(value string) FixedLengthRecordReaderV2Attr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// FixedLengthRecordReaderV2Encoding sets the optional encoding attribute to value.
+//
+// value: The type of encoding for the file. Currently ZLIB and GZIP
+// are supported. Defaults to none.
+// If not specified, defaults to ""
+func FixedLengthRecordReaderV2Encoding(value string) FixedLengthRecordReaderV2Attr {
+	return func(m optionalAttr) {
+		m["encoding"] = value
+	}
+}
+
+// A Reader that outputs fixed-length records from a file.
+//
+// Arguments:
+//	record_bytes: Number of bytes in the record.
+//
+// Returns The handle to reference the Reader.
+func FixedLengthRecordReaderV2(scope *Scope, record_bytes int64, optional ...FixedLengthRecordReaderV2Attr) (reader_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"record_bytes": record_bytes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "FixedLengthRecordReaderV2",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Saves the input tensors to disk.
+//
+// The size of `tensor_names` must match the number of tensors in `data`. `data[i]`
+// is written to `filename` with name `tensor_names[i]`.
+//
+// See also `SaveSlices`.
+//
+// Arguments:
+//	filename: Must have a single element. The name of the file to which we write
+// the tensor.
+//	tensor_names: Shape `[N]`. The names of the tensors to be saved.
+//	data: `N` tensors to save.
+//
+// Returns the created operation.
+func Save(scope *Scope, filename tf.Output, tensor_names tf.Output, data []tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Save",
+		Input: []tf.Input{
+			filename, tensor_names, tf.OutputList(data),
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// DatasetToGraphV2Attr is an optional argument to DatasetToGraphV2.
+type DatasetToGraphV2Attr func(optionalAttr)
+
+// DatasetToGraphV2ExternalStatePolicy sets the optional external_state_policy attribute to value.
+// If not specified, defaults to 0
+func DatasetToGraphV2ExternalStatePolicy(value int64) DatasetToGraphV2Attr {
+	return func(m optionalAttr) {
+		m["external_state_policy"] = value
+	}
+}
+
+// DatasetToGraphV2StripDeviceAssignment sets the optional strip_device_assignment attribute to value.
+// If not specified, defaults to false
+func DatasetToGraphV2StripDeviceAssignment(value bool) DatasetToGraphV2Attr {
+	return func(m optionalAttr) {
+		m["strip_device_assignment"] = value
+	}
+}
+
+// Returns a serialized GraphDef representing `input_dataset`.
+//
+// Returns a graph representation for `input_dataset`.
+//
+// Arguments:
+//	input_dataset: A variant tensor representing the dataset to return the graph representation for.
+//
+// Returns The graph representation of the dataset (as serialized GraphDef).
+func DatasetToGraphV2(scope *Scope, input_dataset tf.Output, optional ...DatasetToGraphV2Attr) (graph tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DatasetToGraphV2",
+		Input: []tf.Input{
+			input_dataset,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Restores tensors from a V2 checkpoint.
+//
+// For backward compatibility with the V1 format, this Op currently allows
+// restoring from a V1 checkpoint as well:
+//   - This Op first attempts to find the V2 index file pointed to by "prefix", and
+//     if found proceed to read it as a V2 checkpoint;
+//   - Otherwise the V1 read path is invoked.
+// Relying on this behavior is not recommended, as the ability to fall back to read
+// V1 might be deprecated and eventually removed.
+//
+// By default, restores the named tensors in full.  If the caller wishes to restore
+// specific slices of stored tensors, "shape_and_slices" should be non-empty
+// strings and correspondingly well-formed.
+//
+// Callers must ensure all the named tensors are indeed stored in the checkpoint.
+//
+// Arguments:
+//	prefix: Must have a single element.  The prefix of a V2 checkpoint.
+//	tensor_names: shape {N}.  The names of the tensors to be restored.
+//	shape_and_slices: shape {N}.  The slice specs of the tensors to be restored.
+// Empty strings indicate that they are non-partitioned tensors.
+//	dtypes: shape {N}.  The list of expected dtype for the tensors.  Must match
+// those stored in the checkpoint.
+//
+// Returns shape {N}.  The restored tensors, whose shapes are read from the
+// checkpoint directly.
+func RestoreV2(scope *Scope, prefix tf.Output, tensor_names tf.Output, shape_and_slices tf.Output, dtypes []tf.DataType) (tensors []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes}
+	opspec := tf.OpSpec{
+		Type: "RestoreV2",
+		Input: []tf.Input{
+			prefix, tensor_names, shape_and_slices,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if tensors, idx, err = makeOutputList(op, idx, "tensors"); err != nil {
+		scope.UpdateErr("RestoreV2", err)
+		return
+	}
+	return tensors
+}
+
+// Delete the TensorArray from its resource container.
+//
+// This enables the user to close and release the resource in the middle
+// of a step/run.
+//
+// Arguments:
+//	handle: The handle to a TensorArray (output of TensorArray or TensorArrayGrad).
+//
+// Returns the created operation.
+func TensorArrayCloseV3(scope *Scope, handle tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorArrayCloseV3",
+		Input: []tf.Input{
+			handle,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Saves tensors in V2 checkpoint format.
+//
+// By default, saves the named tensors in full.  If the caller wishes to save
+// specific slices of full tensors, "shape_and_slices" should be non-empty strings
+// and correspondingly well-formed.
+//
+// Arguments:
+//	prefix: Must have a single element. The prefix of the V2 checkpoint to which we
+// write the tensors.
+//	tensor_names: shape {N}. The names of the tensors to be saved.
+//	shape_and_slices: shape {N}.  The slice specs of the tensors to be saved.
+// Empty strings indicate that they are non-partitioned tensors.
+//	tensors: `N` tensors to save.
+//
+// Returns the created operation.
+func SaveV2(scope *Scope, prefix tf.Output, tensor_names tf.Output, shape_and_slices tf.Output, tensors []tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SaveV2",
+		Input: []tf.Input{
+			prefix, tensor_names, shape_and_slices, tf.OutputList(tensors),
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// SparseCountSparseOutputAttr is an optional argument to SparseCountSparseOutput.
+type SparseCountSparseOutputAttr func(optionalAttr)
+
+// SparseCountSparseOutputMinlength sets the optional minlength attribute to value.
+//
+// value: Minimum value to count. Can be set to -1 for no minimum.
+// If not specified, defaults to -1
+//
+// REQUIRES: value >= -1
+func SparseCountSparseOutputMinlength(value int64) SparseCountSparseOutputAttr {
+	return func(m optionalAttr) {
+		m["minlength"] = value
+	}
+}
+
+// SparseCountSparseOutputMaxlength sets the optional maxlength attribute to value.
+//
+// value: Maximum value to count. Can be set to -1 for no maximum.
+// If not specified, defaults to -1
+//
+// REQUIRES: value >= -1
+func SparseCountSparseOutputMaxlength(value int64) SparseCountSparseOutputAttr {
+	return func(m optionalAttr) {
+		m["maxlength"] = value
+	}
+}
+
+// Performs sparse-output bin counting for a sparse tensor input.
+//
+//   Counts the number of times each value occurs in the input.
+//
+// Arguments:
+//	indices: Tensor containing the indices of the sparse tensor to count.
+//	values: Tensor containing values of the sparse tensor to count.
+//	dense_shape: Tensor containing the dense shape of the sparse tensor to count.
+//	weights: A Tensor of the same shape as indices containing per-index weight values.
+// May also be the empty tensor if no weights are used.
+//	binary_output: Whether to output the number of occurrences of each value or 1.
+//
+// Returns:
+//	output_indices: Indices tensor for the resulting sparse tensor object.
+//	output_values: Values tensor for the resulting sparse tensor object.
+//	output_dense_shape: Shape tensor for the resulting sparse tensor object.
+func SparseCountSparseOutput(scope *Scope, indices tf.Output, values tf.Output, dense_shape tf.Output, weights tf.Output, binary_output bool, optional ...SparseCountSparseOutputAttr) (output_indices tf.Output, output_values tf.Output, output_dense_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"binary_output": binary_output}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseCountSparseOutput",
+		Input: []tf.Input{
+			indices, values, dense_shape, weights,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// DebugNumericSummaryV2Attr is an optional argument to DebugNumericSummaryV2.
+type DebugNumericSummaryV2Attr func(optionalAttr)
+
+// DebugNumericSummaryV2OutputDtype sets the optional output_dtype attribute to value.
+//
+// value: Optional. The type of the output. Can be float32 or float64 (default: float32).
+// If not specified, defaults to DT_FLOAT
+func DebugNumericSummaryV2OutputDtype(value tf.DataType) DebugNumericSummaryV2Attr {
+	return func(m optionalAttr) {
+		m["output_dtype"] = value
+	}
+}
+
+// DebugNumericSummaryV2TensorDebugMode sets the optional tensor_debug_mode attribute to value.
+//
+// value: Tensor debug mode: the mode in which the input tensor is summarized
+//   by the op. See the TensorDebugMode enum in
+//   tensorflow/core/protobuf/debug_event.proto for details.
+//
+// Supported values:
+//   2 (CURT_HEALTH): Output a float32/64 tensor of shape [2]. The 1st
+//   element is the tensor_id, if provided, and -1 otherwise. The 2nd
+//   element is a bit which is set to 1 if the input tensor has an
+//   infinity or nan value, or zero otherwise.
+//
+//   3 (CONCISE_HEALTH): Output a float32/64 tensor of shape [5]. The 1st
+//   element is the tensor_id, if provided, and -1 otherwise. The
+//   remaining four slots are the total number of elements, -infs,
+//   +infs, and nans in the input tensor respectively.
+//
+//   4 (FULL_HEALTH): Output a float32/64 tensor of shape [11]. The 1st
+//   element is the tensor_id, if provided, and -1 otherwise. The 2nd
+//   element is the device_id, if provided, and -1 otherwise. The 3rd
+//   element holds the datatype value of the input tensor as according
+//   to the enumerated type in tensorflow/core/framework/types.proto.
+//   The remaining elements hold the total number of elements, -infs,
+//   +infs, nans, negative finite numbers, zeros, and positive finite
+//   numbers in the input tensor respectively.
+//
+//   5 (SHAPE): Output a float32/64 tensor of shape [10]. The 1st
+//   element is the tensor_id, if provided, and -1 otherwise. The 2nd
+//   element holds the datatype value of the input tensor as according
+//   to the enumerated type in tensorflow/core/framework/types.proto.
+//   The 3rd element holds the rank of the tensor. The 4th element holds
+//   the number of elements within the tensor. Finally the remaining 6
+//   elements hold the shape of the tensor. If the rank of the tensor
+//   is lower than 6, the shape is right padded with zeros. If the rank
+//   is greater than 6, the head of the shape is truncated.
+//
+//   6 (FULL_NUMERICS): Output a float32/64 tensor of shape [22]. The 1st
+//   element is the tensor_id, if provided, and -1 otherwise. The 2nd
+//   element is the device_id, if provided, and -1 otherwise. The 3rd
+//   element holds the datatype value of the input tensor as according
+//   to the enumerated type in tensorflow/core/framework/types.proto.
+//   The 4th element holds the rank of the tensor. The 5th to 11th
+//   elements hold the shape of the tensor. If the rank of the tensor
+//   is lower than 6, the shape is right padded with zeros. If the rank
+//   is greater than 6, the head of the shape is truncated. The 12th to
+//   18th elements hold the number of elements, -infs, +infs, nans,
+//   denormal floats, negative finite numbers, zeros, and positive
+//   finite numbers in the input tensor respectively. The final four
+//   elements hold the min value, max value, mean, and variance of the
+//   input tensor.
+//
+//   8 (REDUCE_INF_NAN_THREE_SLOTS): Output a float32/64 tensor of shape
+//   [3]. The 1st element is -inf if any elements of the input tensor
+//   is -inf, or zero otherwise. The 2nd element is +inf if any elements
+//   of the input tensor is +inf, or zero otherwise.  The 3rd element is
+//   nan if any element of the input tensor is nan, or zero otherwise.
+// If not specified, defaults to -1
+func DebugNumericSummaryV2TensorDebugMode(value int64) DebugNumericSummaryV2Attr {
+	return func(m optionalAttr) {
+		m["tensor_debug_mode"] = value
+	}
+}
+
+// DebugNumericSummaryV2TensorId sets the optional tensor_id attribute to value.
+//
+// value: Optional. An integer identifier for the tensor being summarized by this op.
+// If not specified, defaults to -1
+func DebugNumericSummaryV2TensorId(value int64) DebugNumericSummaryV2Attr {
+	return func(m optionalAttr) {
+		m["tensor_id"] = value
+	}
+}
+
+// Debug Numeric Summary V2 Op.
+//
+// Computes a numeric summary of the input tensor. The shape of the output
+// depends on the tensor_debug_mode attribute.
+// This op is used internally by TensorFlow Debugger (tfdbg) v2.
+//
+// Arguments:
+//	input: Input tensor, to be summarized by the op.
+func DebugNumericSummaryV2(scope *Scope, input tf.Output, optional ...DebugNumericSummaryV2Attr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DebugNumericSummaryV2",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// DebugNumericSummaryAttr is an optional argument to DebugNumericSummary.
+type DebugNumericSummaryAttr func(optionalAttr)
+
+// DebugNumericSummaryDeviceName sets the optional device_name attribute to value.
+// If not specified, defaults to ""
+func DebugNumericSummaryDeviceName(value string) DebugNumericSummaryAttr {
+	return func(m optionalAttr) {
+		m["device_name"] = value
+	}
+}
+
+// DebugNumericSummaryTensorName sets the optional tensor_name attribute to value.
+//
+// value: Name of the input tensor.
+// If not specified, defaults to ""
+func DebugNumericSummaryTensorName(value string) DebugNumericSummaryAttr {
+	return func(m optionalAttr) {
+		m["tensor_name"] = value
+	}
+}
+
+// DebugNumericSummaryDebugUrls sets the optional debug_urls attribute to value.
+//
+// value: List of URLs to debug targets, e.g.,
+//   file:///foo/tfdbg_dump, grpc:://localhost:11011.
+// If not specified, defaults to <>
+func DebugNumericSummaryDebugUrls(value []string) DebugNumericSummaryAttr {
+	return func(m optionalAttr) {
+		m["debug_urls"] = value
+	}
+}
+
+// DebugNumericSummaryLowerBound sets the optional lower_bound attribute to value.
+//
+// value: (float) The lower bound <= which values will be included in the
+//   generalized -inf count. Default: -inf.
+// If not specified, defaults to -inf
+func DebugNumericSummaryLowerBound(value float32) DebugNumericSummaryAttr {
+	return func(m optionalAttr) {
+		m["lower_bound"] = value
+	}
+}
+
+// DebugNumericSummaryUpperBound sets the optional upper_bound attribute to value.
+//
+// value: (float) The upper bound >= which values will be included in the
+//   generalized +inf count. Default: +inf.
+// If not specified, defaults to inf
+func DebugNumericSummaryUpperBound(value float32) DebugNumericSummaryAttr {
+	return func(m optionalAttr) {
+		m["upper_bound"] = value
+	}
+}
+
+// DebugNumericSummaryMuteIfHealthy sets the optional mute_if_healthy attribute to value.
+//
+// value: (bool) Do not send data to the debug URLs unless at least one
+//   of elements [2], [3] and [7] (i.e., the nan count and the generalized -inf and
+//   inf counts) is non-zero.
+// If not specified, defaults to false
+func DebugNumericSummaryMuteIfHealthy(value bool) DebugNumericSummaryAttr {
+	return func(m optionalAttr) {
+		m["mute_if_healthy"] = value
+	}
+}
+
+// DebugNumericSummaryGatedGrpc sets the optional gated_grpc attribute to value.
+//
+// value: Whether this op will be gated. If any of the debug_urls of this
+//   debug node is of the grpc:// scheme, when the value of this attribute is set
+//   to True, the data will not actually be sent via the grpc stream unless this
+//   debug op has been enabled at the debug_url. If all of the debug_urls of this
+//   debug node are of the grpc:// scheme and the debug op is enabled at none of
+//   them, the output will be an empty Tensor.
+// If not specified, defaults to false
+func DebugNumericSummaryGatedGrpc(value bool) DebugNumericSummaryAttr {
+	return func(m optionalAttr) {
+		m["gated_grpc"] = value
+	}
+}
+
+// Debug Numeric Summary Op.
+//
+// Provide a basic summary of numeric value types, range and distribution.
+//
+// output: A double tensor of shape [14 + nDimensions], where nDimensions is the
+//   number of dimensions of the tensor's shape. The elements of output are:
+//   [0]: is initialized (1.0) or not (0.0).
+//   [1]: total number of elements
+//   [2]: NaN element count
+//   [3]: generalized -inf count: elements <= lower_bound. lower_bound is -inf by
+//     default.
+//   [4]: negative element count (excluding -inf), if lower_bound is the default
+//     -inf. Otherwise, this is the count of elements > lower_bound and < 0.
+//   [5]: zero element count
+//   [6]: positive element count (excluding +inf), if upper_bound is the default
+//     +inf. Otherwise, this is the count of elements < upper_bound and > 0.
+//   [7]: generalized +inf count, elements >= upper_bound. upper_bound is +inf by
+//     default.
+// Output elements [1:8] are all zero, if the tensor is uninitialized.
+//   [8]: minimum of all non-inf and non-NaN elements.
+//        If uninitialized or no such element exists: +inf.
+//   [9]: maximum of all non-inf and non-NaN elements.
+//        If uninitialized or no such element exists: -inf.
+//   [10]: mean of all non-inf and non-NaN elements.
+//         If uninitialized or no such element exists: NaN.
+//   [11]: variance of all non-inf and non-NaN elements.
+//         If uninitialized or no such element exists: NaN.
+//   [12]: Data type of the tensor encoded as an enum integer. See the DataType
+//         proto for more details.
+//   [13]: Number of dimensions of the tensor (ndims).
+//   [14+]: Sizes of the dimensions.
+//
+//
+// Arguments:
+//	input: Input tensor, non-Reference type.
+func DebugNumericSummary(scope *Scope, input tf.Output, optional ...DebugNumericSummaryAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DebugNumericSummary",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Outputs random integers from a uniform distribution.
+//
+// The generated values are uniform integers in the range `[minval, maxval)`.
+// The lower bound `minval` is included in the range, while the upper bound
+// `maxval` is excluded.
+//
+// The random integers are slightly biased unless `maxval - minval` is an exact
+// power of two.  The bias is small for values of `maxval - minval` significantly
+// smaller than the range of the output (either `2^32` or `2^64`).
+//
+// Arguments:
+//	resource: The handle of the resource variable that stores the state of the RNG.
+//	algorithm: The RNG algorithm.
+//	shape: The shape of the output tensor.
+//	minval: Minimum value (inclusive, scalar).
+//	maxval: Maximum value (exclusive, scalar).
+//
+// Returns Random values with specified shape.
+func StatefulUniformInt(scope *Scope, resource tf.Output, algorithm tf.Output, shape tf.Output, minval tf.Output, maxval tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "StatefulUniformInt",
+		Input: []tf.Input{
+			resource, algorithm, shape, minval, maxval,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// An Op to exchange data across TPU replicas.
+//
+// On each replica, the input is split into `split_count` blocks along
+// `split_dimension` and send to the other replicas given group_assignment. After
+// receiving `split_count` - 1 blocks from other replicas, we concatenate the
+// blocks along `concat_dimension` as the output.
+//
+// For example, suppose there are 2 TPU replicas:
+// replica 0 receives input: `[[A, B]]`
+// replica 1 receives input: `[[C, D]]`
+//
+// group_assignment=`[[0, 1]]`
+// concat_dimension=0
+// split_dimension=1
+// split_count=2
+//
+// replica 0's output: `[[A], [C]]`
+// replica 1's output: `[[B], [D]]`
+//
+// Arguments:
+//	input: The local input to the sum.
+//	group_assignment: An int32 tensor with shape
+// [num_groups, num_replicas_per_group]. `group_assignment[i]` represents the
+// replica ids in the ith subgroup.
+//	concat_dimension: The dimension number to concatenate.
+//	split_dimension: The dimension number to split.
+//	split_count: The number of splits, this number must equal to the sub-group
+// size(group_assignment.get_shape()[1])
+//
+// Returns The exchanged result.
+func AllToAll(scope *Scope, input tf.Output, group_assignment tf.Output, concat_dimension int64, split_dimension int64, split_count int64) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"concat_dimension": concat_dimension, "split_dimension": split_dimension, "split_count": split_count}
+	opspec := tf.OpSpec{
+		Type: "AllToAll",
+		Input: []tf.Input{
+			input, group_assignment,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// TridiagonalSolveAttr is an optional argument to TridiagonalSolve.
+type TridiagonalSolveAttr func(optionalAttr)
+
+// TridiagonalSolvePartialPivoting sets the optional partial_pivoting attribute to value.
+//
+// value: Whether to apply partial pivoting. Partial pivoting makes the procedure more
+// stable, but slower.
+// If not specified, defaults to true
+func TridiagonalSolvePartialPivoting(value bool) TridiagonalSolveAttr {
+	return func(m optionalAttr) {
+		m["partial_pivoting"] = value
+	}
+}
+
+// Solves tridiagonal systems of equations.
+//
+//   Solves tridiagonal systems of equations.
+//   Supports batch dimensions and multiple right-hand sides per each left-hand
+//   side.
+//   On CPU, solution is computed via Gaussian elimination with or without partial
+//   pivoting, depending on `partial_pivoting` attribute. On GPU, Nvidia's cuSPARSE
+//   library is used: https://docs.nvidia.com/cuda/cusparse/index.html#gtsv
+//   Partial pivoting is not yet supported by XLA backends.
+//
+// Arguments:
+//	diagonals: Tensor of shape `[..., 3, M]` whose innermost 2 dimensions represent the
+// tridiagonal matrices with three rows being the superdiagonal, diagonals, and
+// subdiagonals, in order. The last element of the superdiagonal and the first
+// element of the subdiagonal is ignored.
+//	rhs: Tensor of shape `[..., M, K]`, representing K right-hand sides per each
+// left-hand side.
+//
+// Returns Tensor of shape `[..., M, K]` containing the solutions
+func TridiagonalSolve(scope *Scope, diagonals tf.Output, rhs tf.Output, optional ...TridiagonalSolveAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TridiagonalSolve",
+		Input: []tf.Input{
+			diagonals, rhs,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes gradients for SparseSegmentMean.
+//
+// Returns tensor "output" with same shape as grad, except for dimension 0 whose
+// value is output_dim0.
+//
+// Arguments:
+//	grad: gradient propagated to the SparseSegmentMean op.
+//	indices: indices passed to the corresponding SparseSegmentMean op.
+//	segment_ids: segment_ids passed to the corresponding SparseSegmentMean op.
+//	output_dim0: dimension 0 of "data" passed to SparseSegmentMean op.
+func SparseSegmentMeanGrad(scope *Scope, grad tf.Output, indices tf.Output, segment_ids tf.Output, output_dim0 tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseSegmentMeanGrad",
+		Input: []tf.Input{
+			grad, indices, segment_ids, output_dim0,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SvdAttr is an optional argument to Svd.
+type SvdAttr func(optionalAttr)
+
+// SvdComputeUv sets the optional compute_uv attribute to value.
+//
+// value: If true, left and right singular vectors will be
+// computed and returned in `u` and `v`, respectively.
+// If false, `u` and `v` are not set and should never referenced.
+// If not specified, defaults to true
+func SvdComputeUv(value bool) SvdAttr {
+	return func(m optionalAttr) {
+		m["compute_uv"] = value
+	}
+}
+
+// SvdFullMatrices sets the optional full_matrices attribute to value.
+//
+// value: If true, compute full-sized `u` and `v`. If false
+// (the default), compute only the leading `P` singular vectors.
+// Ignored if `compute_uv` is `False`.
+// If not specified, defaults to false
+func SvdFullMatrices(value bool) SvdAttr {
+	return func(m optionalAttr) {
+		m["full_matrices"] = value
+	}
+}
+
+// Computes the singular value decompositions of one or more matrices.
+//
+// Computes the SVD of each inner matrix in `input` such that
+// `input[..., :, :] = u[..., :, :] * diag(s[..., :, :]) * transpose(v[..., :, :])`
+//
+// ```python
+// # a is a tensor containing a batch of matrices.
+// # s is a tensor of singular values for each matrix.
+// # u is the tensor containing the left singular vectors for each matrix.
+// # v is the tensor containing the right singular vectors for each matrix.
+// s, u, v = svd(a)
+// s, _, _ = svd(a, compute_uv=False)
+// ```
+//
+// Arguments:
+//	input: A tensor of shape `[..., M, N]` whose inner-most 2 dimensions
+// form matrices of size `[M, N]`. Let `P` be the minimum of `M` and `N`.
+//
+// Returns:
+//	s: Singular values. Shape is `[..., P]`.
+//	u: Left singular vectors. If `full_matrices` is `False` then shape is
+// `[..., M, P]`; if `full_matrices` is `True` then shape is
+// `[..., M, M]`. Undefined if `compute_uv` is `False`.
+//	v: Left singular vectors. If `full_matrices` is `False` then shape is
+// `[..., N, P]`. If `full_matrices` is `True` then shape is `[..., N, N]`.
+// Undefined if `compute_uv` is false.
+func Svd(scope *Scope, input tf.Output, optional ...SvdAttr) (s tf.Output, u tf.Output, v tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Svd",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// QrAttr is an optional argument to Qr.
+type QrAttr func(optionalAttr)
+
+// QrFullMatrices sets the optional full_matrices attribute to value.
+//
+// value: If true, compute full-sized `q` and `r`. If false
+// (the default), compute only the leading `P` columns of `q`.
+// If not specified, defaults to false
+func QrFullMatrices(value bool) QrAttr {
+	return func(m optionalAttr) {
+		m["full_matrices"] = value
+	}
+}
+
+// Computes the QR decompositions of one or more matrices.
+//
+// Computes the QR decomposition of each inner matrix in `tensor` such that
+// `tensor[..., :, :] = q[..., :, :] * r[..., :,:])`
+//
+// ```python
+// # a is a tensor.
+// # q is a tensor of orthonormal matrices.
+// # r is a tensor of upper triangular matrices.
+// q, r = qr(a)
+// q_full, r_full = qr(a, full_matrices=True)
+// ```
+//
+// Arguments:
+//	input: A tensor of shape `[..., M, N]` whose inner-most 2 dimensions
+// form matrices of size `[M, N]`. Let `P` be the minimum of `M` and `N`.
+//
+// Returns:
+//	q: Orthonormal basis for range of `a`. If `full_matrices` is `False` then
+// shape is `[..., M, P]`; if `full_matrices` is `True` then shape is
+// `[..., M, M]`.
+//	r: Triangular factor. If `full_matrices` is `False` then shape is
+// `[..., P, N]`. If `full_matrices` is `True` then shape is `[..., M, N]`.
+func Qr(scope *Scope, input tf.Output, optional ...QrAttr) (q tf.Output, r tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Qr",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// MatrixTriangularSolveAttr is an optional argument to MatrixTriangularSolve.
+type MatrixTriangularSolveAttr func(optionalAttr)
+
+// MatrixTriangularSolveLower sets the optional lower attribute to value.
+//
+// value: Boolean indicating whether the innermost matrices in `matrix` are
+// lower or upper triangular.
+// If not specified, defaults to true
+func MatrixTriangularSolveLower(value bool) MatrixTriangularSolveAttr {
+	return func(m optionalAttr) {
+		m["lower"] = value
+	}
+}
+
+// MatrixTriangularSolveAdjoint sets the optional adjoint attribute to value.
+//
+// value: Boolean indicating whether to solve with `matrix` or its (block-wise)
+//          adjoint.
+//
+// @compatibility(numpy)
+// Equivalent to scipy.linalg.solve_triangular
+// @end_compatibility
+// If not specified, defaults to false
+func MatrixTriangularSolveAdjoint(value bool) MatrixTriangularSolveAttr {
+	return func(m optionalAttr) {
+		m["adjoint"] = value
+	}
+}
+
+// Solves systems of linear equations with upper or lower triangular matrices by backsubstitution.
+//
+//
+// `matrix` is a tensor of shape `[..., M, M]` whose inner-most 2 dimensions form
+// square matrices. If `lower` is `True` then the strictly upper triangular part
+// of each inner-most matrix is assumed to be zero and not accessed.
+// If `lower` is False then the strictly lower triangular part of each inner-most
+// matrix is assumed to be zero and not accessed.
+// `rhs` is a tensor of shape `[..., M, N]`.
+//
+// The output is a tensor of shape `[..., M, N]`. If `adjoint` is
+// `True` then the innermost matrices in `output` satisfy matrix equations
+// `matrix[..., :, :] * output[..., :, :] = rhs[..., :, :]`.
+// If `adjoint` is `False` then the strictly then the  innermost matrices in
+// `output` satisfy matrix equations
+// `adjoint(matrix[..., i, k]) * output[..., k, j] = rhs[..., i, j]`.
+//
+// Note, the batch shapes for the inputs only need to broadcast.
+//
+// Example:
+// ```python
+//
+// a = tf.constant([[3,  0,  0,  0],
+//                  [2,  1,  0,  0],
+//                  [1,  0,  1,  0],
+//                  [1,  1,  1,  1]], dtype=tf.float32)
+//
+// b = tf.constant([[4],
+//                  [2],
+//                  [4],
+//                  [2]], dtype=tf.float32)
+//
+// x = tf.linalg.triangular_solve(a, b, lower=True)
+// x
+// # <tf.Tensor: shape=(4, 1), dtype=float32, numpy=
+// # array([[ 1.3333334 ],
+// #        [-0.66666675],
+// #        [ 2.6666665 ],
+// #        [-1.3333331 ]], dtype=float32)>
+//
+// # in python3 one can use `a@x`
+// tf.matmul(a, x)
+// # <tf.Tensor: shape=(4, 1), dtype=float32, numpy=
+// # array([[4.       ],
+// #        [2.       ],
+// #        [4.       ],
+// #        [1.9999999]], dtype=float32)>
+// ```
+//
+// Arguments:
+//	matrix: Shape is `[..., M, M]`.
+//	rhs: Shape is `[..., M, K]`.
+//
+// Returns Shape is `[..., M, K]`.
+func MatrixTriangularSolve(scope *Scope, matrix tf.Output, rhs tf.Output, optional ...MatrixTriangularSolveAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MatrixTriangularSolve",
+		Input: []tf.Input{
+			matrix, rhs,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SelfAdjointEigV2Attr is an optional argument to SelfAdjointEigV2.
+type SelfAdjointEigV2Attr func(optionalAttr)
+
+// SelfAdjointEigV2ComputeV sets the optional compute_v attribute to value.
+//
+// value: If `True` then eigenvectors will be computed and returned in `v`.
+// Otherwise, only the eigenvalues will be computed.
+// If not specified, defaults to true
+func SelfAdjointEigV2ComputeV(value bool) SelfAdjointEigV2Attr {
+	return func(m optionalAttr) {
+		m["compute_v"] = value
+	}
+}
+
+// Computes the eigen decomposition of one or more square self-adjoint matrices.
+//
+// Computes the eigenvalues and (optionally) eigenvectors of each inner matrix in
+// `input` such that `input[..., :, :] = v[..., :, :] * diag(e[..., :])`. The eigenvalues
+// are sorted in non-decreasing order.
+//
+// ```python
+// # a is a tensor.
+// # e is a tensor of eigenvalues.
+// # v is a tensor of eigenvectors.
+// e, v = self_adjoint_eig(a)
+// e = self_adjoint_eig(a, compute_v=False)
+// ```
+//
+// Arguments:
+//	input: `Tensor` input of shape `[N, N]`.
+//
+// Returns:
+//	e: Eigenvalues. Shape is `[N]`.
+//	v: Eigenvectors. Shape is `[N, N]`.
+func SelfAdjointEigV2(scope *Scope, input tf.Output, optional ...SelfAdjointEigV2Attr) (e tf.Output, v tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SelfAdjointEigV2",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Computes the Eigen Decomposition of a batch of square self-adjoint matrices.
+//
+// DEPRECATED at GraphDef version 11: Use SelfAdjointEigV2 instead.
+//
+// The input is a tensor of shape `[..., M, M]` whose inner-most 2 dimensions
+// form square matrices, with the same constraints as the single matrix
+// SelfAdjointEig.
+//
+// The result is a [..., M+1, M] matrix with [..., 0,:] containing the
+// eigenvalues, and subsequent [...,1:, :] containing the eigenvectors. The eigenvalues
+// are sorted in non-decreasing order.
+//
+// Arguments:
+//	input: Shape is `[..., M, M]`.
+//
+// Returns Shape is `[..., M+1, M]`.
+func SelfAdjointEig(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SelfAdjointEig",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that emits the key-value pairs in one or more LMDB files.
+//
+// The Lightning Memory-Mapped Database Manager, or LMDB, is an embedded binary
+// key-value database. This dataset can read the contents of LMDB database files,
+// the names of which generally have the `.mdb` suffix.
+//
+// Each output element consists of a key-value pair represented as a pair of
+// scalar string `Tensor`s, where the first `Tensor` contains the key and the
+// second `Tensor` contains the value.
+//
+// LMDB uses different file formats on big- and little-endian machines.
+// `LMDBDataset` can only read files in the format of the host machine.
+//
+// Arguments:
+//	filenames: A scalar or a vector containing the name(s) of the binary file(s) to be
+// read.
+//
+//
+func LMDBDataset(scope *Scope, filenames tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "LMDBDataset",
+		Input: []tf.Input{
+			filenames,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MatrixInverseAttr is an optional argument to MatrixInverse.
+type MatrixInverseAttr func(optionalAttr)
+
+// MatrixInverseAdjoint sets the optional adjoint attribute to value.
+// If not specified, defaults to false
+func MatrixInverseAdjoint(value bool) MatrixInverseAttr {
+	return func(m optionalAttr) {
+		m["adjoint"] = value
+	}
+}
+
+// Computes the inverse of one or more square invertible matrices or their
+//
+// adjoints (conjugate transposes).
+//
+// The input is a tensor of shape `[..., M, M]` whose inner-most 2 dimensions
+// form square matrices. The output is a tensor of the same shape as the input
+// containing the inverse for all input submatrices `[..., :, :]`.
+//
+// The op uses LU decomposition with partial pivoting to compute the inverses.
+//
+// If a matrix is not invertible there is no guarantee what the op does. It
+// may detect the condition and raise an exception or it may simply return a
+// garbage result.
+//
+// Arguments:
+//	input: Shape is `[..., M, M]`.
+//
+// Returns Shape is `[..., M, M]`.
+//
+// @compatibility(numpy)
+// Equivalent to np.linalg.inv
+// @end_compatibility
+func MatrixInverse(scope *Scope, input tf.Output, optional ...MatrixInverseAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MatrixInverse",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the sign and the log of the absolute value of the determinant of
+//
+// one or more square matrices.
+//
+// The input is a tensor of shape `[N, M, M]` whose inner-most 2 dimensions
+// form square matrices. The outputs are two tensors containing the signs and
+// absolute values of the log determinants for all N input submatrices
+// `[..., :, :]` such that the determinant = sign*exp(log_abs_determinant).
+// The log_abs_determinant is computed as det(P)*sum(log(diag(LU))) where LU
+// is the LU decomposition of the input and P is the corresponding
+// permutation matrix.
+//
+// Arguments:
+//	input: Shape is `[N, M, M]`.
+//
+// Returns:
+//	sign: The signs of the log determinants of the inputs. Shape is `[N]`.
+//	log_abs_determinant: The logs of the absolute values of the determinants
+// of the N input matrices.  Shape is `[N]`.
+func LogMatrixDeterminant(scope *Scope, input tf.Output) (sign tf.Output, log_abs_determinant tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "LogMatrixDeterminant",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Computes the determinant of one or more square matrices.
+//
+// The input is a tensor of shape `[..., M, M]` whose inner-most 2 dimensions
+// form square matrices. The output is a tensor containing the determinants
+// for all input submatrices `[..., :, :]`.
+//
+// Arguments:
+//	input: Shape is `[..., M, M]`.
+//
+// Returns Shape is `[...]`.
+func MatrixDeterminant(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "MatrixDeterminant",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResizeBilinearAttr is an optional argument to ResizeBilinear.
+type ResizeBilinearAttr func(optionalAttr)
+
+// ResizeBilinearAlignCorners sets the optional align_corners attribute to value.
+//
+// value: If true, the centers of the 4 corner pixels of the input and output tensors are
+// aligned, preserving the values at the corner pixels. Defaults to false.
+// If not specified, defaults to false
+func ResizeBilinearAlignCorners(value bool) ResizeBilinearAttr {
+	return func(m optionalAttr) {
+		m["align_corners"] = value
+	}
+}
+
+// ResizeBilinearHalfPixelCenters sets the optional half_pixel_centers attribute to value.
+// If not specified, defaults to false
+func ResizeBilinearHalfPixelCenters(value bool) ResizeBilinearAttr {
+	return func(m optionalAttr) {
+		m["half_pixel_centers"] = value
+	}
+}
+
+// Resize `images` to `size` using bilinear interpolation.
+//
+// Input images can be of different types but output images are always float.
+//
+// Arguments:
+//	images: 4-D with shape `[batch, height, width, channels]`.
+//	size: = A 1-D int32 Tensor of 2 elements: `new_height, new_width`.  The
+// new size for the images.
+//
+// Returns 4-D with shape
+// `[batch, new_height, new_width, channels]`.
+func ResizeBilinear(scope *Scope, images tf.Output, size tf.Output, optional ...ResizeBilinearAttr) (resized_images tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResizeBilinear",
+		Input: []tf.Input{
+			images, size,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a TensorList by indexing into a Tensor.
+//
+// Each member of the TensorList corresponds to one row of the input tensor,
+// specified by the given index (see `tf.gather`).
+//
+// tensor: The input tensor.
+// indices: The indices used to index into the list.
+// element_shape: The shape of the elements in the list (can be less specified than
+//   the shape of the tensor).
+// num_elements: The size of the output list. Must be large enough to accommodate
+//   the largest index in indices. If -1, the list is just large enough to include
+//   the largest index in indices.
+// output_handle: The TensorList.
+func TensorListScatterV2(scope *Scope, tensor tf.Output, indices tf.Output, element_shape tf.Output, num_elements tf.Output) (output_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorListScatterV2",
+		Input: []tf.Input{
+			tensor, indices, element_shape, num_elements,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a TensorList by indexing into a Tensor.
+//
+// Each member of the TensorList corresponds to one row of the input tensor,
+// specified by the given index (see `tf.gather`).
+//
+// tensor: The input tensor.
+// indices: The indices used to index into the list.
+// element_shape: The shape of the elements in the list (can be less specified than
+//   the shape of the tensor).
+// output_handle: The TensorList.
+func TensorListScatter(scope *Scope, tensor tf.Output, indices tf.Output, element_shape tf.Output) (output_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorListScatter",
+		Input: []tf.Input{
+			tensor, indices, element_shape,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the gradient of the sigmoid of `x` wrt its input.
+//
+// Specifically, `grad = dy * y * (1 - y)`, where `y = sigmoid(x)`, and
+// `dy` is the corresponding input gradient.
+func SigmoidGrad(scope *Scope, y tf.Output, dy tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SigmoidGrad",
+		Input: []tf.Input{
+			y, dy,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a Tensor by indexing into the TensorList.
+//
+// Each row in the produced Tensor corresponds to the element in the TensorList
+// specified by the given index (see `tf.gather`).
+//
+// input_handle: The input tensor list.
+// indices: The indices used to index into the list.
+// values: The tensor.
+func TensorListGather(scope *Scope, input_handle tf.Output, indices tf.Output, element_shape tf.Output, element_dtype tf.DataType) (values tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"element_dtype": element_dtype}
+	opspec := tf.OpSpec{
+		Type: "TensorListGather",
+		Input: []tf.Input{
+			input_handle, indices, element_shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// The shape of the elements of the given list, as a tensor.
+//
+//   input_handle: the list
+//   element_shape: the shape of elements of the list
+func TensorListElementShape(scope *Scope, input_handle tf.Output, shape_type tf.DataType) (element_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"shape_type": shape_type}
+	opspec := tf.OpSpec{
+		Type: "TensorListElementShape",
+		Input: []tf.Input{
+			input_handle,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ExperimentalThreadPoolHandleAttr is an optional argument to ExperimentalThreadPoolHandle.
+type ExperimentalThreadPoolHandleAttr func(optionalAttr)
+
+// ExperimentalThreadPoolHandleMaxIntraOpParallelism sets the optional max_intra_op_parallelism attribute to value.
+//
+// value: The maximum degree of parallelism to use within operations that execute on this
+// threadpool.
+// If not specified, defaults to 1
+func ExperimentalThreadPoolHandleMaxIntraOpParallelism(value int64) ExperimentalThreadPoolHandleAttr {
+	return func(m optionalAttr) {
+		m["max_intra_op_parallelism"] = value
+	}
+}
+
+// ExperimentalThreadPoolHandleContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func ExperimentalThreadPoolHandleContainer(value string) ExperimentalThreadPoolHandleAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// ExperimentalThreadPoolHandleSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func ExperimentalThreadPoolHandleSharedName(value string) ExperimentalThreadPoolHandleAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Creates a dataset that uses a custom thread pool to compute `input_dataset`.
+//
+// Arguments:
+//	num_threads: The number of threads in the thread pool.
+//	display_name: A human-readable name for the threads that may be visible in some
+// visualizations.
+// threadpool.
+//
+// Returns A resource that can be consumed by one or more ExperimentalThreadPoolDataset
+// ops.
+func ExperimentalThreadPoolHandle(scope *Scope, num_threads int64, display_name string, optional ...ExperimentalThreadPoolHandleAttr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_threads": num_threads, "display_name": display_name}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalThreadPoolHandle",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a TensorList which, when stacked, has the value of `tensor`.
+//
+// Each tensor in the result list corresponds to one row of the input tensor.
+//
+// tensor: The input tensor.
+// output_handle: The list.
+func TensorListFromTensor(scope *Scope, tensor tf.Output, element_shape tf.Output) (output_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorListFromTensor",
+		Input: []tf.Input{
+			tensor, element_shape,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// TensorListStackAttr is an optional argument to TensorListStack.
+type TensorListStackAttr func(optionalAttr)
+
+// TensorListStackNumElements sets the optional num_elements attribute to value.
+// If not specified, defaults to -1
+func TensorListStackNumElements(value int64) TensorListStackAttr {
+	return func(m optionalAttr) {
+		m["num_elements"] = value
+	}
+}
+
+// Stacks all tensors in the list.
+//
+// Requires that all tensors have the same shape.
+//
+// input_handle: the input list
+// tensor: the gathered result
+// num_elements: optional. If not -1, the number of elements in the list.
+//
+func TensorListStack(scope *Scope, input_handle tf.Output, element_shape tf.Output, element_dtype tf.DataType, optional ...TensorListStackAttr) (tensor tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"element_dtype": element_dtype}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorListStack",
+		Input: []tf.Input{
+			input_handle, element_shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns the number of tensors in the input tensor list.
+//
+// input_handle: the input list
+// length: the number of tensors in the list
+func TensorListLength(scope *Scope, input_handle tf.Output) (length tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorListLength",
+		Input: []tf.Input{
+			input_handle,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Merges summaries.
+//
+// This op creates a
+// [`Summary`](https://www.tensorflow.org/code/tensorflow/core/framework/summary.proto)
+// protocol buffer that contains the union of all the values in the input
+// summaries.
+//
+// When the Op is run, it reports an `InvalidArgument` error if multiple values
+// in the summaries to merge use the same tag.
+//
+// Arguments:
+//	inputs: Can be of any shape.  Each must contain serialized `Summary` protocol
+// buffers.
+//
+// Returns Scalar. Serialized `Summary` protocol buffer.
+func MergeSummary(scope *Scope, inputs []tf.Output) (summary tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "MergeSummary",
+		Input: []tf.Input{
+			tf.OutputList(inputs),
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// AvgPoolAttr is an optional argument to AvgPool.
+type AvgPoolAttr func(optionalAttr)
+
+// AvgPoolDataFormat sets the optional data_format attribute to value.
+//
+// value: Specify the data format of the input and output data. With the
+// default format "NHWC", the data is stored in the order of:
+//     [batch, in_height, in_width, in_channels].
+// Alternatively, the format could be "NCHW", the data storage order of:
+//     [batch, in_channels, in_height, in_width].
+// If not specified, defaults to "NHWC"
+func AvgPoolDataFormat(value string) AvgPoolAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// Performs average pooling on the input.
+//
+// Each entry in `output` is the mean of the corresponding size `ksize`
+// window in `value`.
+//
+// Arguments:
+//	value: 4-D with shape `[batch, height, width, channels]`.
+//	ksize: The size of the sliding window for each dimension of `value`.
+//	strides: The stride of the sliding window for each dimension of `value`.
+//	padding: The type of padding algorithm to use.
+//
+// Returns The average pooled output tensor.
+func AvgPool(scope *Scope, value tf.Output, ksize []int64, strides []int64, padding string, optional ...AvgPoolAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"ksize": ksize, "strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "AvgPool",
+		Input: []tf.Input{
+			value,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// AudioSummaryV2Attr is an optional argument to AudioSummaryV2.
+type AudioSummaryV2Attr func(optionalAttr)
+
+// AudioSummaryV2MaxOutputs sets the optional max_outputs attribute to value.
+//
+// value: Max number of batch elements to generate audio for.
+// If not specified, defaults to 3
+//
+// REQUIRES: value >= 1
+func AudioSummaryV2MaxOutputs(value int64) AudioSummaryV2Attr {
+	return func(m optionalAttr) {
+		m["max_outputs"] = value
+	}
+}
+
+// Outputs a `Summary` protocol buffer with audio.
+//
+// The summary has up to `max_outputs` summary values containing audio. The
+// audio is built from `tensor` which must be 3-D with shape `[batch_size,
+// frames, channels]` or 2-D with shape `[batch_size, frames]`. The values are
+// assumed to be in the range of `[-1.0, 1.0]` with a sample rate of `sample_rate`.
+//
+// The `tag` argument is a scalar `Tensor` of type `string`.  It is used to
+// build the `tag` of the summary values:
+//
+// *  If `max_outputs` is 1, the summary value tag is '*tag*/audio'.
+// *  If `max_outputs` is greater than 1, the summary value tags are
+//    generated sequentially as '*tag*/audio/0', '*tag*/audio/1', etc.
+//
+// Arguments:
+//	tag: Scalar. Used to build the `tag` attribute of the summary values.
+//	tensor: 2-D of shape `[batch_size, frames]`.
+//	sample_rate: The sample rate of the signal in hertz.
+//
+// Returns Scalar. Serialized `Summary` protocol buffer.
+func AudioSummaryV2(scope *Scope, tag tf.Output, tensor tf.Output, sample_rate tf.Output, optional ...AudioSummaryV2Attr) (summary tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "AudioSummaryV2",
+		Input: []tf.Input{
+			tag, tensor, sample_rate,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Outputs a `Summary` protocol buffer with a histogram.
+//
+// The generated
+// [`Summary`](https://www.tensorflow.org/code/tensorflow/core/framework/summary.proto)
+// has one summary value containing a histogram for `values`.
+//
+// This op reports an `InvalidArgument` error if any value is not finite.
+//
+// Arguments:
+//	tag: Scalar.  Tag to use for the `Summary.Value`.
+//	values: Any shape. Values to use to build the histogram.
+//
+// Returns Scalar. Serialized `Summary` protocol buffer.
+func HistogramSummary(scope *Scope, tag tf.Output, values tf.Output) (summary tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "HistogramSummary",
+		Input: []tf.Input{
+			tag, values,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// StringLengthAttr is an optional argument to StringLength.
+type StringLengthAttr func(optionalAttr)
+
+// StringLengthUnit sets the optional unit attribute to value.
+//
+// value: The unit that is counted to compute string length.  One of: `"BYTE"` (for
+// the number of bytes in each string) or `"UTF8_CHAR"` (for the number of UTF-8
+// encoded Unicode code points in each string).  Results are undefined
+// if `unit=UTF8_CHAR` and the `input` strings do not contain structurally
+// valid UTF-8.
+// If not specified, defaults to "BYTE"
+func StringLengthUnit(value string) StringLengthAttr {
+	return func(m optionalAttr) {
+		m["unit"] = value
+	}
+}
+
+// String lengths of `input`.
+//
+// Computes the length of each string given in the input tensor.
+//
+// >>> strings = tf.constant(['Hello','TensorFlow', '\U0001F642'])
+// >>> tf.strings.length(strings).numpy() # default counts bytes
+// array([ 5, 10, 4], dtype=int32)
+// >>> tf.strings.length(strings, unit="UTF8_CHAR").numpy()
+// array([ 5, 10, 1], dtype=int32)
+//
+//
+// Arguments:
+//	input: The strings for which to compute the length for each element.
+//
+// Returns Integer tensor that has the same shape as `input`. The output contains the
+// element-wise string lengths of `input`.
+func StringLength(scope *Scope, input tf.Output, optional ...StringLengthAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StringLength",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// TensorSummaryAttr is an optional argument to TensorSummary.
+type TensorSummaryAttr func(optionalAttr)
+
+// TensorSummaryDescription sets the optional description attribute to value.
+//
+// value: A json-encoded SummaryDescription proto.
+// If not specified, defaults to ""
+func TensorSummaryDescription(value string) TensorSummaryAttr {
+	return func(m optionalAttr) {
+		m["description"] = value
+	}
+}
+
+// TensorSummaryLabels sets the optional labels attribute to value.
+//
+// value: An unused list of strings.
+// If not specified, defaults to <>
+func TensorSummaryLabels(value []string) TensorSummaryAttr {
+	return func(m optionalAttr) {
+		m["labels"] = value
+	}
+}
+
+// TensorSummaryDisplayName sets the optional display_name attribute to value.
+//
+// value: An unused string.
+// If not specified, defaults to ""
+func TensorSummaryDisplayName(value string) TensorSummaryAttr {
+	return func(m optionalAttr) {
+		m["display_name"] = value
+	}
+}
+
+// Outputs a `Summary` protocol buffer with a tensor.
+//
+// This op is being phased out in favor of TensorSummaryV2, which lets callers pass
+// a tag as well as a serialized SummaryMetadata proto string that contains
+// plugin-specific data. We will keep this op to maintain backwards compatibility.
+//
+// Arguments:
+//	tensor: A tensor to serialize.
+func TensorSummary(scope *Scope, tensor tf.Output, optional ...TensorSummaryAttr) (summary tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorSummary",
+		Input: []tf.Input{
+			tensor,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Scatters tensor at indices in an input list.
+//
+// Each member of the TensorList corresponds to one row of the input tensor,
+// specified by the given index (see `tf.gather`).
+//
+// input_handle: The list to scatter into.
+// tensor: The input tensor.
+// indices: The indices used to index into the list.
+// output_handle: The TensorList.
+func TensorListScatterIntoExistingList(scope *Scope, input_handle tf.Output, tensor tf.Output, indices tf.Output) (output_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorListScatterIntoExistingList",
+		Input: []tf.Input{
+			input_handle, tensor, indices,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Outputs a `Summary` protocol buffer with a tensor and per-plugin data.
+//
+// Arguments:
+//	tag: A string attached to this summary. Used for organization in TensorBoard.
+//	tensor: A tensor to serialize.
+//	serialized_summary_metadata: A serialized SummaryMetadata proto. Contains plugin
+// data.
+func TensorSummaryV2(scope *Scope, tag tf.Output, tensor tf.Output, serialized_summary_metadata tf.Output) (summary tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorSummaryV2",
+		Input: []tf.Input{
+			tag, tensor, serialized_summary_metadata,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the gradient for the sqrt of `x` wrt its input.
+//
+// Specifically, `grad = dy * 0.5 / y`, where `y = sqrt(x)`, and `dy`
+// is the corresponding input gradient.
+func SqrtGrad(scope *Scope, y tf.Output, dy tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SqrtGrad",
+		Input: []tf.Input{
+			y, dy,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MutableHashTableOfTensorsV2Attr is an optional argument to MutableHashTableOfTensorsV2.
+type MutableHashTableOfTensorsV2Attr func(optionalAttr)
+
+// MutableHashTableOfTensorsV2Container sets the optional container attribute to value.
+//
+// value: If non-empty, this table is placed in the given container.
+// Otherwise, a default container is used.
+// If not specified, defaults to ""
+func MutableHashTableOfTensorsV2Container(value string) MutableHashTableOfTensorsV2Attr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// MutableHashTableOfTensorsV2SharedName sets the optional shared_name attribute to value.
+//
+// value: If non-empty, this table is shared under the given name across
+// multiple sessions.
+// If not specified, defaults to ""
+func MutableHashTableOfTensorsV2SharedName(value string) MutableHashTableOfTensorsV2Attr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// MutableHashTableOfTensorsV2UseNodeNameSharing sets the optional use_node_name_sharing attribute to value.
+// If not specified, defaults to false
+func MutableHashTableOfTensorsV2UseNodeNameSharing(value bool) MutableHashTableOfTensorsV2Attr {
+	return func(m optionalAttr) {
+		m["use_node_name_sharing"] = value
+	}
+}
+
+// MutableHashTableOfTensorsV2ValueShape sets the optional value_shape attribute to value.
+// If not specified, defaults to <>
+func MutableHashTableOfTensorsV2ValueShape(value tf.Shape) MutableHashTableOfTensorsV2Attr {
+	return func(m optionalAttr) {
+		m["value_shape"] = value
+	}
+}
+
+// Creates an empty hash table.
+//
+// This op creates a mutable hash table, specifying the type of its keys and
+// values. Each value must be a vector. Data can be inserted into the table using
+// the insert operations. It does not support the initialization operation.
+//
+// Arguments:
+//	key_dtype: Type of the table keys.
+//	value_dtype: Type of the table values.
+//
+// Returns Handle to a table.
+func MutableHashTableOfTensorsV2(scope *Scope, key_dtype tf.DataType, value_dtype tf.DataType, optional ...MutableHashTableOfTensorsV2Attr) (table_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"key_dtype": key_dtype, "value_dtype": value_dtype}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MutableHashTableOfTensorsV2",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the grayscale dilation of 4-D `input` and 3-D `filter` tensors.
+//
+// The `input` tensor has shape `[batch, in_height, in_width, depth]` and the
+// `filter` tensor has shape `[filter_height, filter_width, depth]`, i.e., each
+// input channel is processed independently of the others with its own structuring
+// function. The `output` tensor has shape
+// `[batch, out_height, out_width, depth]`. The spatial dimensions of the output
+// tensor depend on the `padding` algorithm. We currently only support the default
+// "NHWC" `data_format`.
+//
+// In detail, the grayscale morphological 2-D dilation is the max-sum correlation
+// (for consistency with `conv2d`, we use unmirrored filters):
+//
+//     output[b, y, x, c] =
+//        max_{dy, dx} input[b,
+//                           strides[1] * y + rates[1] * dy,
+//                           strides[2] * x + rates[2] * dx,
+//                           c] +
+//                     filter[dy, dx, c]
+//
+// Max-pooling is a special case when the filter has size equal to the pooling
+// kernel size and contains all zeros.
+//
+// Note on duality: The dilation of `input` by the `filter` is equal to the
+// negation of the erosion of `-input` by the reflected `filter`.
+//
+// Arguments:
+//	input: 4-D with shape `[batch, in_height, in_width, depth]`.
+//	filter: 3-D with shape `[filter_height, filter_width, depth]`.
+//	strides: The stride of the sliding window for each dimension of the input
+// tensor. Must be: `[1, stride_height, stride_width, 1]`.
+//	rates: The input stride for atrous morphological dilation. Must be:
+// `[1, rate_height, rate_width, 1]`.
+//	padding: The type of padding algorithm to use.
+//
+// Returns 4-D with shape `[batch, out_height, out_width, depth]`.
+func Dilation2D(scope *Scope, input tf.Output, filter tf.Output, strides []int64, rates []int64, padding string) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"strides": strides, "rates": rates, "padding": padding}
+	opspec := tf.OpSpec{
+		Type: "Dilation2D",
+		Input: []tf.Input{
+			input, filter,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes softplus: `log(exp(features) + 1)`.
+func Softplus(scope *Scope, features tf.Output) (activations tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Softplus",
+		Input: []tf.Input{
+			features,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MutableHashTableV2Attr is an optional argument to MutableHashTableV2.
+type MutableHashTableV2Attr func(optionalAttr)
+
+// MutableHashTableV2Container sets the optional container attribute to value.
+//
+// value: If non-empty, this table is placed in the given container.
+// Otherwise, a default container is used.
+// If not specified, defaults to ""
+func MutableHashTableV2Container(value string) MutableHashTableV2Attr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// MutableHashTableV2SharedName sets the optional shared_name attribute to value.
+//
+// value: If non-empty, this table is shared under the given name across
+// multiple sessions.
+// If not specified, defaults to ""
+func MutableHashTableV2SharedName(value string) MutableHashTableV2Attr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// MutableHashTableV2UseNodeNameSharing sets the optional use_node_name_sharing attribute to value.
+//
+// value: If true and shared_name is empty, the table is shared
+// using the node name.
+// If not specified, defaults to false
+func MutableHashTableV2UseNodeNameSharing(value bool) MutableHashTableV2Attr {
+	return func(m optionalAttr) {
+		m["use_node_name_sharing"] = value
+	}
+}
+
+// Creates an empty hash table.
+//
+// This op creates a mutable hash table, specifying the type of its keys and
+// values. Each value must be a scalar. Data can be inserted into the table using
+// the insert operations. It does not support the initialization operation.
+//
+// Arguments:
+//	key_dtype: Type of the table keys.
+//	value_dtype: Type of the table values.
+//
+// Returns Handle to a table.
+func MutableHashTableV2(scope *Scope, key_dtype tf.DataType, value_dtype tf.DataType, optional ...MutableHashTableV2Attr) (table_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"key_dtype": key_dtype, "value_dtype": value_dtype}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MutableHashTableV2",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Calculates the prior from the training data (the bias) and fills in the first node with the logits' prior. Returns a boolean indicating whether to continue centering.
+//
+// Arguments:
+//	tree_ensemble_handle: Handle to the tree ensemble.
+//	mean_gradients: A tensor with shape=[logits_dimension] with mean of gradients for a first node.
+//	mean_hessians: A tensor with shape=[logits_dimension] mean of hessians for a first node.
+//	l1: l1 regularization factor on leaf weights, per instance based.
+//	l2: l2 regularization factor on leaf weights, per instance based.
+//
+// Returns Bool, whether to continue bias centering.
+func BoostedTreesCenterBias(scope *Scope, tree_ensemble_handle tf.Output, mean_gradients tf.Output, mean_hessians tf.Output, l1 tf.Output, l2 tf.Output) (continue_centering tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesCenterBias",
+		Input: []tf.Input{
+			tree_ensemble_handle, mean_gradients, mean_hessians, l1, l2,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// HashTableV2Attr is an optional argument to HashTableV2.
+type HashTableV2Attr func(optionalAttr)
+
+// HashTableV2Container sets the optional container attribute to value.
+//
+// value: If non-empty, this table is placed in the given container.
+// Otherwise, a default container is used.
+// If not specified, defaults to ""
+func HashTableV2Container(value string) HashTableV2Attr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// HashTableV2SharedName sets the optional shared_name attribute to value.
+//
+// value: If non-empty, this table is shared under the given name across
+// multiple sessions.
+// If not specified, defaults to ""
+func HashTableV2SharedName(value string) HashTableV2Attr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// HashTableV2UseNodeNameSharing sets the optional use_node_name_sharing attribute to value.
+//
+// value: If true and shared_name is empty, the table is shared
+// using the node name.
+// If not specified, defaults to false
+func HashTableV2UseNodeNameSharing(value bool) HashTableV2Attr {
+	return func(m optionalAttr) {
+		m["use_node_name_sharing"] = value
+	}
+}
+
+// Creates a non-initialized hash table.
+//
+// This op creates a hash table, specifying the type of its keys and values.
+// Before using the table you will have to initialize it.  After initialization the
+// table will be immutable.
+//
+// Arguments:
+//	key_dtype: Type of the table keys.
+//	value_dtype: Type of the table values.
+//
+// Returns Handle to a table.
+func HashTableV2(scope *Scope, key_dtype tf.DataType, value_dtype tf.DataType, optional ...HashTableV2Attr) (table_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"key_dtype": key_dtype, "value_dtype": value_dtype}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "HashTableV2",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Check if the input matches the regex pattern.
+//
+// The input is a string tensor of any shape. The pattern is a scalar
+// string tensor which is applied to every element of the input tensor.
+// The boolean values (True or False) of the output tensor indicate
+// if the input matches the regex pattern provided.
+//
+// The pattern follows the re2 syntax (https://github.com/google/re2/wiki/Syntax)
+//
+// Examples:
+//
+// >>> tf.strings.regex_full_match(["TF lib", "lib TF"], ".*lib$")
+// <tf.Tensor: shape=(2,), dtype=bool, numpy=array([ True, False])>
+// >>> tf.strings.regex_full_match(["TF lib", "lib TF"], ".*TF$")
+// <tf.Tensor: shape=(2,), dtype=bool, numpy=array([False,  True])>
+//
+// Arguments:
+//	input: A string tensor of the text to be processed.
+//	pattern: A scalar string tensor containing the regular expression to match the input.
+//
+// Returns A bool tensor with the same shape as `input`.
+func RegexFullMatch(scope *Scope, input tf.Output, pattern tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "RegexFullMatch",
+		Input: []tf.Input{
+			input, pattern,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MatrixDiagV3Attr is an optional argument to MatrixDiagV3.
+type MatrixDiagV3Attr func(optionalAttr)
+
+// MatrixDiagV3Align sets the optional align attribute to value.
+//
+// value: Some diagonals are shorter than `max_diag_len` and need to be padded. `align` is
+// a string specifying how superdiagonals and subdiagonals should be aligned,
+// respectively. There are four possible alignments: "RIGHT_LEFT" (default),
+// "LEFT_RIGHT", "LEFT_LEFT", and "RIGHT_RIGHT". "RIGHT_LEFT" aligns superdiagonals
+// to the right (left-pads the row) and subdiagonals to the left (right-pads the
+// row). It is the packing format LAPACK uses. cuSPARSE uses "LEFT_RIGHT", which is
+// the opposite alignment.
+// If not specified, defaults to "RIGHT_LEFT"
+func MatrixDiagV3Align(value string) MatrixDiagV3Attr {
+	return func(m optionalAttr) {
+		m["align"] = value
+	}
+}
+
+// Returns a batched diagonal tensor with given batched diagonal values.
+//
+// Returns a tensor with the contents in `diagonal` as `k[0]`-th to `k[1]`-th
+// diagonals of a matrix, with everything else padded with `padding`. `num_rows`
+// and `num_cols` specify the dimension of the innermost matrix of the output. If
+// both are not specified, the op assumes the innermost matrix is square and infers
+// its size from `k` and the innermost dimension of `diagonal`. If only one of them
+// is specified, the op assumes the unspecified value is the smallest possible
+// based on other criteria.
+//
+// Let `diagonal` have `r` dimensions `[I, J, ..., L, M, N]`. The output tensor has
+// rank `r+1` with shape `[I, J, ..., L, M, num_rows, num_cols]` when only one
+// diagonal is given (`k` is an integer or `k[0] == k[1]`). Otherwise, it has rank
+// `r` with shape `[I, J, ..., L, num_rows, num_cols]`.
+//
+// The second innermost dimension of `diagonal` has double meaning.
+// When `k` is scalar or `k[0] == k[1]`, `M` is part of the batch size
+// [I, J, ..., M], and the output tensor is:
+//
+// ```
+// output[i, j, ..., l, m, n]
+//   = diagonal[i, j, ..., l, n-max(d_upper, 0)] ; if n - m == d_upper
+//     padding_value                             ; otherwise
+// ```
+//
+// Otherwise, `M` is treated as the number of diagonals for the matrix in the
+// same batch (`M = k[1]-k[0]+1`), and the output tensor is:
+//
+// ```
+// output[i, j, ..., l, m, n]
+//   = diagonal[i, j, ..., l, diag_index, index_in_diag] ; if k[0] <= d <= k[1]
+//     padding_value                                     ; otherwise
+// ```
+// where `d = n - m`, `diag_index = [k] - d`, and
+// `index_in_diag = n - max(d, 0) + offset`.
+//
+// `offset` is zero except when the alignment of the diagonal is to the right.
+// ```
+// offset = max_diag_len - diag_len(d) ; if (`align` in {RIGHT_LEFT, RIGHT_RIGHT}
+//                                            and `d >= 0`) or
+//                                          (`align` in {LEFT_RIGHT, RIGHT_RIGHT}
+//                                            and `d <= 0`)
+//          0                          ; otherwise
+// ```
+// where `diag_len(d) = min(cols - max(d, 0), rows + min(d, 0))`.
+//
+// For example:
+//
+// ```
+// # The main diagonal.
+// diagonal = np.array([[1, 2, 3, 4],            # Input shape: (2, 4)
+//                      [5, 6, 7, 8]])
+// tf.matrix_diag(diagonal) ==> [[[1, 0, 0, 0],  # Output shape: (2, 4, 4)
+//                                [0, 2, 0, 0],
+//                                [0, 0, 3, 0],
+//                                [0, 0, 0, 4]],
+//                               [[5, 0, 0, 0],
+//                                [0, 6, 0, 0],
+//                                [0, 0, 7, 0],
+//                                [0, 0, 0, 8]]]
+//
+// # A superdiagonal (per batch).
+// diagonal = np.array([[1, 2, 3],  # Input shape: (2, 3)
+//                      [4, 5, 6]])
+// tf.matrix_diag(diagonal, k = 1)
+//   ==> [[[0, 1, 0, 0],  # Output shape: (2, 4, 4)
+//         [0, 0, 2, 0],
+//         [0, 0, 0, 3],
+//         [0, 0, 0, 0]],
+//        [[0, 4, 0, 0],
+//         [0, 0, 5, 0],
+//         [0, 0, 0, 6],
+//         [0, 0, 0, 0]]]
+//
+// # A tridiagonal band (per batch).
+// diagonals = np.array([[[0, 8, 9],  # Input shape: (2, 2, 3)
+//                        [1, 2, 3],
+//                        [4, 5, 0]],
+//                       [[0, 2, 3],
+//                        [6, 7, 9],
+//                        [9, 1, 0]]])
+// tf.matrix_diag(diagonals, k = (-1, 1))
+//   ==> [[[1, 8, 0],  # Output shape: (2, 3, 3)
+//         [4, 2, 9],
+//         [0, 5, 3]],
+//        [[6, 2, 0],
+//         [9, 7, 3],
+//         [0, 1, 9]]]
+//
+// # LEFT_RIGHT alignment.
+// diagonals = np.array([[[8, 9, 0],  # Input shape: (2, 2, 3)
+//                        [1, 2, 3],
+//                        [0, 4, 5]],
+//                       [[2, 3, 0],
+//                        [6, 7, 9],
+//                        [0, 9, 1]]])
+// tf.matrix_diag(diagonals, k = (-1, 1), align="LEFT_RIGHT")
+//   ==> [[[1, 8, 0],  # Output shape: (2, 3, 3)
+//         [4, 2, 9],
+//         [0, 5, 3]],
+//        [[6, 2, 0],
+//         [9, 7, 3],
+//         [0, 1, 9]]]
+//
+// # Rectangular matrix.
+// diagonal = np.array([1, 2])  # Input shape: (2)
+// tf.matrix_diag(diagonal, k = -1, num_rows = 3, num_cols = 4)
+//   ==> [[0, 0, 0, 0],  # Output shape: (3, 4)
+//        [1, 0, 0, 0],
+//        [0, 2, 0, 0]]
+//
+// # Rectangular matrix with inferred num_cols and padding_value = 9.
+// tf.matrix_diag(diagonal, k = -1, num_rows = 3, padding_value = 9)
+//   ==> [[9, 9],  # Output shape: (3, 2)
+//        [1, 9],
+//        [9, 2]]
+//
+// ```
+//
+// Arguments:
+//	diagonal: Rank `r`, where `r >= 1`
+//	k: Diagonal offset(s). Positive value means superdiagonal, 0 refers to the main
+// diagonal, and negative value means subdiagonals. `k` can be a single integer
+// (for a single diagonal) or a pair of integers specifying the low and high ends
+// of a matrix band. `k[0]` must not be larger than `k[1]`.
+//	num_rows: The number of rows of the output matrix. If it is not provided, the op assumes
+// the output matrix is a square matrix and infers the matrix size from k and the
+// innermost dimension of `diagonal`.
+//	num_cols: The number of columns of the output matrix. If it is not provided, the op
+// assumes the output matrix is a square matrix and infers the matrix size from
+// k and the innermost dimension of `diagonal`.
+//	padding_value: The number to fill the area outside the specified diagonal band with.
+// Default is 0.
+//
+// Returns Has rank `r+1` when `k` is an integer or `k[0] == k[1]`, rank `r` otherwise.
+func MatrixDiagV3(scope *Scope, diagonal tf.Output, k tf.Output, num_rows tf.Output, num_cols tf.Output, padding_value tf.Output, optional ...MatrixDiagV3Attr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MatrixDiagV3",
+		Input: []tf.Input{
+			diagonal, k, num_rows, num_cols, padding_value,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Greedily selects a subset of bounding boxes in descending order of score,
+//
+// pruning away boxes that have high overlaps
+// with previously selected boxes.  Bounding boxes with score less than
+// `score_threshold` are removed. N-by-n overlap values are supplied as square matrix,
+// which allows for defining a custom overlap criterium (eg. intersection over union,
+// intersection over area, etc.).
+//
+// The output of this operation is a set of integers indexing into the input
+// collection of bounding boxes representing the selected boxes.  The bounding
+// box coordinates corresponding to the selected indices can then be obtained
+// using the `tf.gather operation`.  For example:
+//
+//   selected_indices = tf.image.non_max_suppression_with_overlaps(
+//       overlaps, scores, max_output_size, overlap_threshold, score_threshold)
+//   selected_boxes = tf.gather(boxes, selected_indices)
+//
+// Arguments:
+//	overlaps: A 2-D float tensor of shape `[num_boxes, num_boxes]` representing
+// the n-by-n box overlap values.
+//	scores: A 1-D float tensor of shape `[num_boxes]` representing a single
+// score corresponding to each box (each row of boxes).
+//	max_output_size: A scalar integer tensor representing the maximum number of
+// boxes to be selected by non max suppression.
+//	overlap_threshold: A 0-D float tensor representing the threshold for deciding whether
+// boxes overlap too.
+//	score_threshold: A 0-D float tensor representing the threshold for deciding when to remove
+// boxes based on score.
+//
+// Returns A 1-D integer tensor of shape `[M]` representing the selected
+// indices from the boxes tensor, where `M <= max_output_size`.
+func NonMaxSuppressionWithOverlaps(scope *Scope, overlaps tf.Output, scores tf.Output, max_output_size tf.Output, overlap_threshold tf.Output, score_threshold tf.Output) (selected_indices tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "NonMaxSuppressionWithOverlaps",
+		Input: []tf.Input{
+			overlaps, scores, max_output_size, overlap_threshold, score_threshold,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Outputs all keys and values in the table.
+//
+// Arguments:
+//	table_handle: Handle to the table.
+//
+//
+//
+// Returns:
+//	keys: Vector of all keys present in the table.
+//	values: Tensor of all values in the table. Indexed in parallel with `keys`.
+func LookupTableExportV2(scope *Scope, table_handle tf.Output, Tkeys tf.DataType, Tvalues tf.DataType) (keys tf.Output, values tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"Tkeys": Tkeys, "Tvalues": Tvalues}
+	opspec := tf.OpSpec{
+		Type: "LookupTableExportV2",
+		Input: []tf.Input{
+			table_handle,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// RetrieveTPUEmbeddingAdagradParametersAttr is an optional argument to RetrieveTPUEmbeddingAdagradParameters.
+type RetrieveTPUEmbeddingAdagradParametersAttr func(optionalAttr)
+
+// RetrieveTPUEmbeddingAdagradParametersTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func RetrieveTPUEmbeddingAdagradParametersTableId(value int64) RetrieveTPUEmbeddingAdagradParametersAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingAdagradParametersTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingAdagradParametersTableName(value string) RetrieveTPUEmbeddingAdagradParametersAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingAdagradParametersConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingAdagradParametersConfig(value string) RetrieveTPUEmbeddingAdagradParametersAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Retrieve Adagrad embedding parameters.
+//
+// An op that retrieves optimization parameters from embedding to host
+// memory. Must be preceded by a ConfigureTPUEmbeddingHost op that sets up
+// the correct embedding table configuration. For example, this op is
+// used to retrieve updated parameters before saving a checkpoint.
+//
+// Returns:
+//	parameters: Parameter parameters updated by the Adagrad optimization algorithm.
+//	accumulators: Parameter accumulators updated by the Adagrad optimization algorithm.
+func RetrieveTPUEmbeddingAdagradParameters(scope *Scope, num_shards int64, shard_id int64, optional ...RetrieveTPUEmbeddingAdagradParametersAttr) (parameters tf.Output, accumulators tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RetrieveTPUEmbeddingAdagradParameters",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Computes the sum along sparse segments of a tensor divided by the sqrt of N.
+//
+// N is the size of the segment being reduced.
+//
+// Like `SparseSegmentSqrtN`, but allows missing ids in `segment_ids`. If an id is
+// missing, the `output` tensor at that position will be zeroed.
+//
+// Read
+// [the section on segmentation](https://tensorflow.org/api_docs/python/tf/math#Segmentation)
+// for an explanation of segments.
+//
+// Arguments:
+//
+//	indices: A 1-D tensor. Has same rank as `segment_ids`.
+//	segment_ids: A 1-D tensor. Values should be sorted and can be repeated.
+//	num_segments: Should equal the number of distinct segment IDs.
+//
+// Returns Has same shape as data, except for dimension 0 which
+// has size `k`, the number of segments.
+func SparseSegmentSqrtNWithNumSegments(scope *Scope, data tf.Output, indices tf.Output, segment_ids tf.Output, num_segments tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseSegmentSqrtNWithNumSegments",
+		Input: []tf.Input{
+			data, indices, segment_ids, num_segments,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the Cholesky decomposition of one or more square matrices.
+//
+// The input is a tensor of shape `[..., M, M]` whose inner-most 2 dimensions
+// form square matrices.
+//
+// The input has to be symmetric and positive definite. Only the lower-triangular
+// part of the input will be used for this operation. The upper-triangular part
+// will not be read.
+//
+// The output is a tensor of the same shape as the input
+// containing the Cholesky decompositions for all input submatrices `[..., :, :]`.
+//
+// **Note**: The gradient computation on GPU is faster for large matrices but
+// not for large batch dimensions when the submatrices are small. In this
+// case it might be faster to use the CPU.
+//
+// Arguments:
+//	input: Shape is `[..., M, M]`.
+//
+// Returns Shape is `[..., M, M]`.
+func Cholesky(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Cholesky",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Splits a tensor into a list.
+//
+// list[i] corresponds to lengths[i] tensors from the input tensor.
+// The tensor must have rank at least 1 and contain exactly sum(lengths) elements.
+//
+// tensor: The input tensor.
+// element_shape: A shape compatible with that of elements in the tensor.
+// lengths: Vector of sizes of the 0th dimension of tensors in the list.
+// output_handle: The list.
+func TensorListSplit(scope *Scope, tensor tf.Output, element_shape tf.Output, lengths tf.Output) (output_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorListSplit",
+		Input: []tf.Input{
+			tensor, element_shape, lengths,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Check if the input matches the regex pattern.
+//
+// The input is a string tensor of any shape. The pattern is the
+// regular expression to be matched with every element of the input tensor.
+// The boolean values (True or False) of the output tensor indicate
+// if the input matches the regex pattern provided.
+//
+// The pattern follows the re2 syntax (https://github.com/google/re2/wiki/Syntax)
+//
+// Arguments:
+//	input: A string tensor of the text to be processed.
+//	pattern: The regular expression to match the input.
+//
+// Returns A bool tensor with the same shape as `input`.
+func StaticRegexFullMatch(scope *Scope, input tf.Output, pattern string) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"pattern": pattern}
+	opspec := tf.OpSpec{
+		Type: "StaticRegexFullMatch",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ParseSingleSequenceExampleAttr is an optional argument to ParseSingleSequenceExample.
+type ParseSingleSequenceExampleAttr func(optionalAttr)
+
+// ParseSingleSequenceExampleContextSparseTypes sets the optional context_sparse_types attribute to value.
+//
+// value: A list of Ncontext_sparse types; the data types of data in
+// each context Feature given in context_sparse_keys.
+// Currently the ParseSingleSequenceExample supports DT_FLOAT (FloatList),
+// DT_INT64 (Int64List), and DT_STRING (BytesList).
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseSingleSequenceExampleContextSparseTypes(value []tf.DataType) ParseSingleSequenceExampleAttr {
+	return func(m optionalAttr) {
+		m["context_sparse_types"] = value
+	}
+}
+
+// ParseSingleSequenceExampleFeatureListDenseTypes sets the optional feature_list_dense_types attribute to value.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseSingleSequenceExampleFeatureListDenseTypes(value []tf.DataType) ParseSingleSequenceExampleAttr {
+	return func(m optionalAttr) {
+		m["feature_list_dense_types"] = value
+	}
+}
+
+// ParseSingleSequenceExampleContextDenseShapes sets the optional context_dense_shapes attribute to value.
+//
+// value: A list of Ncontext_dense shapes; the shapes of data in
+// each context Feature given in context_dense_keys.
+// The number of elements in the Feature corresponding to context_dense_key[j]
+// must always equal context_dense_shapes[j].NumEntries().
+// The shape of context_dense_values[j] will match context_dense_shapes[j].
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseSingleSequenceExampleContextDenseShapes(value []tf.Shape) ParseSingleSequenceExampleAttr {
+	return func(m optionalAttr) {
+		m["context_dense_shapes"] = value
+	}
+}
+
+// ParseSingleSequenceExampleFeatureListSparseTypes sets the optional feature_list_sparse_types attribute to value.
+//
+// value: A list of Nfeature_list_sparse types; the data types
+// of data in each FeatureList given in feature_list_sparse_keys.
+// Currently the ParseSingleSequenceExample supports DT_FLOAT (FloatList),
+// DT_INT64 (Int64List), and DT_STRING (BytesList).
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseSingleSequenceExampleFeatureListSparseTypes(value []tf.DataType) ParseSingleSequenceExampleAttr {
+	return func(m optionalAttr) {
+		m["feature_list_sparse_types"] = value
+	}
+}
+
+// ParseSingleSequenceExampleFeatureListDenseShapes sets the optional feature_list_dense_shapes attribute to value.
+//
+// value: A list of Nfeature_list_dense shapes; the shapes of
+// data in each FeatureList given in feature_list_dense_keys.
+// The shape of each Feature in the FeatureList corresponding to
+// feature_list_dense_key[j] must always equal
+// feature_list_dense_shapes[j].NumEntries().
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseSingleSequenceExampleFeatureListDenseShapes(value []tf.Shape) ParseSingleSequenceExampleAttr {
+	return func(m optionalAttr) {
+		m["feature_list_dense_shapes"] = value
+	}
+}
+
+// Transforms a scalar brain.SequenceExample proto (as strings) into typed tensors.
+//
+// Arguments:
+//	serialized: A scalar containing a binary serialized SequenceExample proto.
+//	feature_list_dense_missing_assumed_empty: A vector listing the
+// FeatureList keys which may be missing from the SequenceExample.  If the
+// associated FeatureList is missing, it is treated as empty.  By default,
+// any FeatureList not listed in this vector must exist in the SequenceExample.
+//	context_sparse_keys: A list of Ncontext_sparse string Tensors (scalars).
+// The keys expected in the Examples' features associated with context_sparse
+// values.
+//	context_dense_keys: A list of Ncontext_dense string Tensors (scalars).
+// The keys expected in the SequenceExamples' context features associated with
+// dense values.
+//	feature_list_sparse_keys: A list of Nfeature_list_sparse string Tensors
+// (scalars).  The keys expected in the FeatureLists associated with sparse
+// values.
+//	feature_list_dense_keys: A list of Nfeature_list_dense string Tensors (scalars).
+// The keys expected in the SequenceExamples' feature_lists associated
+// with lists of dense values.
+//	context_dense_defaults: A list of Ncontext_dense Tensors (some may be empty).
+// context_dense_defaults[j] provides default values
+// when the SequenceExample's context map lacks context_dense_key[j].
+// If an empty Tensor is provided for context_dense_defaults[j],
+// then the Feature context_dense_keys[j] is required.
+// The input type is inferred from context_dense_defaults[j], even when it's
+// empty.  If context_dense_defaults[j] is not empty, its shape must match
+// context_dense_shapes[j].
+//	debug_name: A scalar containing the name of the serialized proto.
+// May contain, for example, table key (descriptive) name for the
+// corresponding serialized proto.  This is purely useful for debugging
+// purposes, and the presence of values here has no effect on the output.
+// May also be an empty scalar if no name is available.
+func ParseSingleSequenceExample(scope *Scope, serialized tf.Output, feature_list_dense_missing_assumed_empty tf.Output, context_sparse_keys []tf.Output, context_dense_keys []tf.Output, feature_list_sparse_keys []tf.Output, feature_list_dense_keys []tf.Output, context_dense_defaults []tf.Output, debug_name tf.Output, optional ...ParseSingleSequenceExampleAttr) (context_sparse_indices []tf.Output, context_sparse_values []tf.Output, context_sparse_shapes []tf.Output, context_dense_values []tf.Output, feature_list_sparse_indices []tf.Output, feature_list_sparse_values []tf.Output, feature_list_sparse_shapes []tf.Output, feature_list_dense_values []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ParseSingleSequenceExample",
+		Input: []tf.Input{
+			serialized, feature_list_dense_missing_assumed_empty, tf.OutputList(context_sparse_keys), tf.OutputList(context_dense_keys), tf.OutputList(feature_list_sparse_keys), tf.OutputList(feature_list_dense_keys), tf.OutputList(context_dense_defaults), debug_name,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if context_sparse_indices, idx, err = makeOutputList(op, idx, "context_sparse_indices"); err != nil {
+		scope.UpdateErr("ParseSingleSequenceExample", err)
+		return
+	}
+	if context_sparse_values, idx, err = makeOutputList(op, idx, "context_sparse_values"); err != nil {
+		scope.UpdateErr("ParseSingleSequenceExample", err)
+		return
+	}
+	if context_sparse_shapes, idx, err = makeOutputList(op, idx, "context_sparse_shapes"); err != nil {
+		scope.UpdateErr("ParseSingleSequenceExample", err)
+		return
+	}
+	if context_dense_values, idx, err = makeOutputList(op, idx, "context_dense_values"); err != nil {
+		scope.UpdateErr("ParseSingleSequenceExample", err)
+		return
+	}
+	if feature_list_sparse_indices, idx, err = makeOutputList(op, idx, "feature_list_sparse_indices"); err != nil {
+		scope.UpdateErr("ParseSingleSequenceExample", err)
+		return
+	}
+	if feature_list_sparse_values, idx, err = makeOutputList(op, idx, "feature_list_sparse_values"); err != nil {
+		scope.UpdateErr("ParseSingleSequenceExample", err)
+		return
+	}
+	if feature_list_sparse_shapes, idx, err = makeOutputList(op, idx, "feature_list_sparse_shapes"); err != nil {
+		scope.UpdateErr("ParseSingleSequenceExample", err)
+		return
+	}
+	if feature_list_dense_values, idx, err = makeOutputList(op, idx, "feature_list_dense_values"); err != nil {
+		scope.UpdateErr("ParseSingleSequenceExample", err)
+		return
+	}
+	return context_sparse_indices, context_sparse_values, context_sparse_shapes, context_dense_values, feature_list_sparse_indices, feature_list_sparse_values, feature_list_sparse_shapes, feature_list_dense_values
+}
+
+// Computes the number of elements in the given table.
+//
+// Arguments:
+//	table_handle: Handle to the table.
+//
+// Returns Scalar that contains number of elements in the table.
+func LookupTableSizeV2(scope *Scope, table_handle tf.Output) (size tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "LookupTableSizeV2",
+		Input: []tf.Input{
+			table_handle,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes inverse hyperbolic sine of x element-wise.
+//
+//   Given an input tensor, this function computes inverse hyperbolic sine
+//   for every element in the tensor. Both input and output has a range of
+//   `[-inf, inf]`.
+//
+//   ```python
+//   x = tf.constant([-float("inf"), -2, -0.5, 1, 1.2, 200, 10000, float("inf")])
+//   tf.math.asinh(x) ==> [-inf -1.4436355 -0.4812118 0.8813736 1.0159732 5.991471 9.903487 inf]
+//   ```
+func Asinh(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Asinh",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Looks up keys in a table, outputs the corresponding values.
+//
+// The tensor `keys` must of the same type as the keys of the table.
+// The output `values` is of the type of the table values.
+//
+// The scalar `default_value` is the value output for keys not present in the
+// table. It must also be of the same type as the table values.
+//
+// Arguments:
+//	table_handle: Handle to the table.
+//	keys: Any shape.  Keys to look up.
+//
+//
+// Returns Same shape as `keys`.  Values found in the table, or `default_values`
+// for missing keys.
+func LookupTableFindV2(scope *Scope, table_handle tf.Output, keys tf.Output, default_value tf.Output) (values tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "LookupTableFindV2",
+		Input: []tf.Input{
+			table_handle, keys, default_value,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MaxPoolGradAttr is an optional argument to MaxPoolGrad.
+type MaxPoolGradAttr func(optionalAttr)
+
+// MaxPoolGradDataFormat sets the optional data_format attribute to value.
+//
+// value: Specify the data format of the input and output data. With the
+// default format "NHWC", the data is stored in the order of:
+//     [batch, in_height, in_width, in_channels].
+// Alternatively, the format could be "NCHW", the data storage order of:
+//     [batch, in_channels, in_height, in_width].
+// If not specified, defaults to "NHWC"
+func MaxPoolGradDataFormat(value string) MaxPoolGradAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// Computes gradients of the maxpooling function.
+//
+// Arguments:
+//	orig_input: The original input tensor.
+//	orig_output: The original output tensor.
+//	grad: 4-D.  Gradients w.r.t. the output of `max_pool`.
+//	ksize: The size of the window for each dimension of the input tensor.
+//	strides: The stride of the sliding window for each dimension of the
+// input tensor.
+//	padding: The type of padding algorithm to use.
+//
+// Returns Gradients w.r.t. the input to `max_pool`.
+func MaxPoolGrad(scope *Scope, orig_input tf.Output, orig_output tf.Output, grad tf.Output, ksize []int64, strides []int64, padding string, optional ...MaxPoolGradAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"ksize": ksize, "strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MaxPoolGrad",
+		Input: []tf.Input{
+			orig_input, orig_output, grad,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Rolls the elements of a tensor along an axis.
+//
+// The elements are shifted positively (towards larger indices) by the offset of
+// `shift` along the dimension of `axis`. Negative `shift` values will shift
+// elements in the opposite direction. Elements that roll passed the last position
+// will wrap around to the first and vice versa. Multiple shifts along multiple
+// axes may be specified.
+//
+// For example:
+//
+// ```
+// # 't' is [0, 1, 2, 3, 4]
+// roll(t, shift=2, axis=0) ==> [3, 4, 0, 1, 2]
+//
+// # shifting along multiple dimensions
+// # 't' is [[0, 1, 2, 3, 4], [5, 6, 7, 8, 9]]
+// roll(t, shift=[1, -2], axis=[0, 1]) ==> [[7, 8, 9, 5, 6], [2, 3, 4, 0, 1]]
+//
+// # shifting along the same axis multiple times
+// # 't' is [[0, 1, 2, 3, 4], [5, 6, 7, 8, 9]]
+// roll(t, shift=[2, -3], axis=[1, 1]) ==> [[1, 2, 3, 4, 0], [6, 7, 8, 9, 5]]
+// ```
+//
+// Arguments:
+//
+//	shift: Dimension must be 0-D or 1-D. `shift[i]` specifies the number of places by which
+// elements are shifted positively (towards larger indices) along the dimension
+// specified by `axis[i]`. Negative shifts will roll the elements in the opposite
+// direction.
+//	axis: Dimension must be 0-D or 1-D. `axis[i]` specifies the dimension that the shift
+// `shift[i]` should occur. If the same axis is referenced more than once, the
+// total shift for that axis will be the sum of all the shifts that belong to that
+// axis.
+//
+// Returns Has the same shape and size as the input. The elements are shifted
+// positively (towards larger indices) by the offsets of `shift` along the
+// dimensions of `axis`.
+func Roll(scope *Scope, input tf.Output, shift tf.Output, axis tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Roll",
+		Input: []tf.Input{
+			input, shift, axis,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RestoreSliceAttr is an optional argument to RestoreSlice.
+type RestoreSliceAttr func(optionalAttr)
+
+// RestoreSlicePreferredShard sets the optional preferred_shard attribute to value.
+//
+// value: Index of file to open first if multiple files match
+// `file_pattern`. See the documentation for `Restore`.
+// If not specified, defaults to -1
+func RestoreSlicePreferredShard(value int64) RestoreSliceAttr {
+	return func(m optionalAttr) {
+		m["preferred_shard"] = value
+	}
+}
+
+// Restores a tensor from checkpoint files.
+//
+// This is like `Restore` except that restored tensor can be listed as filling
+// only a slice of a larger tensor.  `shape_and_slice` specifies the shape of the
+// larger tensor and the slice that the restored tensor covers.
+//
+// The `shape_and_slice` input has the same format as the
+// elements of the `shapes_and_slices` input of the `SaveSlices` op.
+//
+// Arguments:
+//	file_pattern: Must have a single element. The pattern of the files from
+// which we read the tensor.
+//	tensor_name: Must have a single element. The name of the tensor to be
+// restored.
+//	shape_and_slice: Scalar. The shapes and slice specifications to use when
+// restoring a tensors.
+//	dt: The type of the tensor to be restored.
+//
+// Returns The restored tensor.
+func RestoreSlice(scope *Scope, file_pattern tf.Output, tensor_name tf.Output, shape_and_slice tf.Output, dt tf.DataType, optional ...RestoreSliceAttr) (tensor tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dt": dt}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RestoreSlice",
+		Input: []tf.Input{
+			file_pattern, tensor_name, shape_and_slice,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// OrderedMapUnstageAttr is an optional argument to OrderedMapUnstage.
+type OrderedMapUnstageAttr func(optionalAttr)
+
+// OrderedMapUnstageCapacity sets the optional capacity attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func OrderedMapUnstageCapacity(value int64) OrderedMapUnstageAttr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// OrderedMapUnstageMemoryLimit sets the optional memory_limit attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func OrderedMapUnstageMemoryLimit(value int64) OrderedMapUnstageAttr {
+	return func(m optionalAttr) {
+		m["memory_limit"] = value
+	}
+}
+
+// OrderedMapUnstageContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func OrderedMapUnstageContainer(value string) OrderedMapUnstageAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// OrderedMapUnstageSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func OrderedMapUnstageSharedName(value string) OrderedMapUnstageAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Op removes and returns the values associated with the key
+//
+// from the underlying container.   If the underlying container
+// does not contain this key, the op will block until it does.
+func OrderedMapUnstage(scope *Scope, key tf.Output, indices tf.Output, dtypes []tf.DataType, optional ...OrderedMapUnstageAttr) (values []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "OrderedMapUnstage",
+		Input: []tf.Input{
+			key, indices,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if values, idx, err = makeOutputList(op, idx, "values"); err != nil {
+		scope.UpdateErr("OrderedMapUnstage", err)
+		return
+	}
+	return values
+}
+
+// SobolSampleAttr is an optional argument to SobolSample.
+type SobolSampleAttr func(optionalAttr)
+
+// SobolSampleDtype sets the optional dtype attribute to value.
+//
+// value: The type of the sample. One of: `float32` or `float64`.
+// If not specified, defaults to DT_FLOAT
+func SobolSampleDtype(value tf.DataType) SobolSampleAttr {
+	return func(m optionalAttr) {
+		m["dtype"] = value
+	}
+}
+
+// Generates points from the Sobol sequence.
+//
+// Creates a Sobol sequence with `num_results` samples. Each sample has dimension
+// `dim`. Skips the first `skip` samples.
+//
+// Arguments:
+//	dim: Positive scalar `Tensor` representing each sample's dimension.
+//	num_results: Positive scalar `Tensor` of dtype int32. The number of Sobol points to return
+// in the output.
+//	skip: Positive scalar `Tensor` of dtype int32. The number of initial points of the
+// Sobol sequence to skip.
+//
+// Returns `Tensor` of samples from Sobol sequence with `shape` [num_results, dim].
+func SobolSample(scope *Scope, dim tf.Output, num_results tf.Output, skip tf.Output, optional ...SobolSampleAttr) (samples tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SobolSample",
+		Input: []tf.Input{
+			dim, num_results, skip,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// QuantizedReluAttr is an optional argument to QuantizedRelu.
+type QuantizedReluAttr func(optionalAttr)
+
+// QuantizedReluOutType sets the optional out_type attribute to value.
+// If not specified, defaults to DT_QUINT8
+func QuantizedReluOutType(value tf.DataType) QuantizedReluAttr {
+	return func(m optionalAttr) {
+		m["out_type"] = value
+	}
+}
+
+// Computes Quantized Rectified Linear: `max(features, 0)`
+//
+// Arguments:
+//
+//	min_features: The float value that the lowest quantized value represents.
+//	max_features: The float value that the highest quantized value represents.
+//
+// Returns:
+//	activations: Has the same output shape as "features".
+//	min_activations: The float value that the lowest quantized value represents.
+//	max_activations: The float value that the highest quantized value represents.
+func QuantizedRelu(scope *Scope, features tf.Output, min_features tf.Output, max_features tf.Output, optional ...QuantizedReluAttr) (activations tf.Output, min_activations tf.Output, max_activations tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizedRelu",
+		Input: []tf.Input{
+			features, min_features, max_features,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Returns the next representable value of `x1` in the direction of `x2`, element-wise.
+//
+// This operation returns the same result as the C++ std::nextafter function.
+//
+// It can also return a subnormal number.
+//
+// @compatibility(cpp)
+// Equivalent to C++ std::nextafter function.
+// @end_compatibility
+func NextAfter(scope *Scope, x1 tf.Output, x2 tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "NextAfter",
+		Input: []tf.Input{
+			x1, x2,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Bucketizes 'input' based on 'boundaries'.
+//
+// For example, if the inputs are
+//     boundaries = [0, 10, 100]
+//     input = [[-5, 10000]
+//              [150,   10]
+//              [5,    100]]
+//
+// then the output will be
+//     output = [[0, 3]
+//               [3, 2]
+//               [1, 3]]
+//
+// Arguments:
+//	input: Any shape of Tensor contains with int or float type.
+//	boundaries: A sorted list of floats gives the boundary of the buckets.
+//
+// Returns Same shape with 'input', each value of input replaced with bucket index.
+//
+// @compatibility(numpy)
+// Equivalent to np.digitize.
+// @end_compatibility
+func Bucketize(scope *Scope, input tf.Output, boundaries []float32) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"boundaries": boundaries}
+	opspec := tf.OpSpec{
+		Type: "Bucketize",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the log of the absolute value of `Gamma(x)` element-wise.
+//
+//   For positive numbers, this function computes log((input - 1)!) for every element in the tensor.
+//   `lgamma(5) = log((5-1)!) = log(4!) = log(24) = 3.1780539`
+//
+// Example:
+//
+// ```python
+// x = tf.constant([0, 0.5, 1, 4.5, -4, -5.6])
+// tf.math.lgamma(x) ==> [inf, 0.5723649, 0., 2.4537368, inf, -4.6477685]
+// ```
+func Lgamma(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Lgamma",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Reads the value of a variable.
+//
+// The tensor returned by this operation is immutable.
+//
+// The value returned by this operation is guaranteed to be influenced by all the
+// writes on which this operation depends directly or indirectly, and to not be
+// influenced by any of the writes which depend directly or indirectly on this
+// operation.
+//
+// Arguments:
+//	resource: handle to the resource in which to store the variable.
+//	dtype: the dtype of the value.
+func ReadVariableOp(scope *Scope, resource tf.Output, dtype tf.DataType) (value tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	opspec := tf.OpSpec{
+		Type: "ReadVariableOp",
+		Input: []tf.Input{
+			resource,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes a range that covers the actual values present in a quantized tensor.
+//
+// Given a quantized tensor described by `(input, input_min, input_max)`, outputs a
+// range that covers the actual values present in that tensor. This op is typically
+// used to produce the `requested_output_min` and `requested_output_max` for
+// `Requantize`.
+//
+// Arguments:
+//
+//	input_min: The float value that the minimum quantized input value represents.
+//	input_max: The float value that the maximum quantized input value represents.
+//
+// Returns:
+//	output_min: The computed min output.
+//	output_max: the computed max output.
+func RequantizationRange(scope *Scope, input tf.Output, input_min tf.Output, input_max tf.Output) (output_min tf.Output, output_max tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "RequantizationRange",
+		Input: []tf.Input{
+			input, input_min, input_max,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Compare values of `input` to `threshold` and pack resulting bits into a `uint8`.
+//
+// Each comparison returns a boolean `true` (if `input_value > threshold`)
+// or and `false` otherwise.
+//
+// This operation is useful for Locality-Sensitive-Hashing (LSH) and other
+// algorithms that use hashing approximations of cosine and `L2` distances;
+// codes can be generated from an input via:
+//
+// ```python
+// codebook_size = 50
+// codebook_bits = codebook_size * 32
+// codebook = tf.get_variable('codebook', [x.shape[-1].value, codebook_bits],
+//                            dtype=x.dtype,
+//                            initializer=tf.orthogonal_initializer())
+// codes = compare_and_threshold(tf.matmul(x, codebook), threshold=0.)
+// codes = tf.bitcast(codes, tf.int32)  # go from uint8 to int32
+// # now codes has shape x.shape[:-1] + [codebook_size]
+// ```
+//
+// **NOTE**: Currently, the innermost dimension of the tensor must be divisible
+// by 8.
+//
+// Given an `input` shaped `[s0, s1, ..., s_n]`, the output is
+// a `uint8` tensor shaped `[s0, s1, ..., s_n / 8]`.
+//
+// Arguments:
+//	input: Values to compare against `threshold` and bitpack.
+//	threshold: Threshold to compare against.
+//
+// Returns The bitpacked comparisons.
+func CompareAndBitpack(scope *Scope, input tf.Output, threshold tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "CompareAndBitpack",
+		Input: []tf.Input{
+			input, threshold,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Tensor contraction according to Einstein summation convention.
+//
+// Implements generalized Tensor contraction and reduction. Each input Tensor must
+// have a corresponding input subscript appearing in the comma-separated left-hand
+// side of the equation. The right-hand side of the equation consists of the
+// output subscript. The input subscripts and the output subscript should consist
+// of zero or more named axis labels and at most one ellipsis (`...`).
+//
+// The named axis labels may be any single character other than those having
+// special meaning, namely `,.->`. The behavior of this Op is undefined if it
+// receives an ill-formatted equation; since the validation is done at
+// graph-building time, we omit format validation checks at runtime.
+//
+// Note: This Op is *not* intended to be called by the user; instead users should
+// call `tf.einsum` directly. It is a hidden Op used by `tf.einsum`.
+//
+// Operations are applied to the input(s) according to the following rules:
+//
+//  (a) Generalized Diagonals: For input dimensions corresponding to axis labels
+//      appearing more than once in the same input subscript, we take the
+//      generalized (`k`-dimensional) diagonal.
+//      For example, in the equation `iii->i` with input shape `[3, 3, 3]`, the
+//      generalized diagonal would consist of `3` elements at indices `(0, 0, 0)`,
+//      `(1, 1, 1)` and `(2, 2, 2)` to create a Tensor of shape `[3]`.
+//
+//  (b) Reduction: Axes corresponding to labels appearing only in one input
+//      subscript but not in the output subscript are summed over prior to Tensor
+//      contraction.
+//      For example, in the equation `ab,bc->b`, the axis labels `a` and `c` are
+//      the reduction axis labels.
+//
+//  (c) Batch Dimensions: Axes corresponding to labels appearing in each of the
+//      input subscripts and also in the output subscript make up the batch
+//      dimensions in Tensor contraction. Unnamed axis labels corresponding to
+//      ellipsis (`...`) also correspond to batch dimensions.
+//      For example, for the equation denoting batch matrix multiplication,
+//      `bij,bjk->bik`, the axis label `b` corresponds to a batch dimension.
+//
+//  (d) Contraction: In case of binary einsum, axes corresponding to labels
+//      appearing in two different inputs (and not in the output) are contracted
+//      against each other.
+//      Considering the batch matrix multiplication equation again
+//      (`bij,bjk->bik`), the contracted axis label is `j`.
+//
+//  (e) Expand Diagonal: If the output subscripts contain repeated (explicit) axis
+//      labels, the opposite operation of (a) is applied. For example, in the
+//      equation `i->iii`, and input shape `[3]`, the output of shape `[3, 3, 3]`
+//      are all zeros, except for the (generalized) diagonal which is populated
+//      with values from the input.
+//      Note: This operation is not supported by `np.einsum` or `tf.einsum`; it is
+//      provided to enable computing the symbolic gradient of `tf.einsum`.
+//
+// The output subscripts must contain only labels appearing in at least one of the
+// input subscripts. Furthermore, all dimensions mapping to the same axis label
+// must be equal.
+//
+// Any of the input and output subscripts may contain at most a single ellipsis
+// (`...`). These ellipsis are mapped against dimensions not corresponding to any
+// named axis label. If two inputs contain ellipsis, then they are broadcasted
+// according to standard NumPy broadcasting
+// [rules](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html).
+//
+// The broadcasted dimensions are placed in the corresponding location of the
+// ellipsis in the output subscript. If the broadcasted dimensions are non-empty
+// and the output subscripts do not contain ellipsis, then an InvalidArgument error
+// is raised.
+//
+// @compatibility(numpy)
+// Similar to [`numpy.einsum`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.einsum.html).
+//
+// Comparison with `numpy.einsum`:
+//
+//  * This Op only supports unary and binary forms of `numpy.einsum`.
+//  * This Op does not support implicit form. (i.e. equations without `->`).
+//  * This Op also supports repeated indices in the output subscript, which is not
+//    supported by `numpy.einsum`.
+// @end_compatibility
+//
+//
+// Arguments:
+//	inputs: List of 1 or 2 Tensors.
+//	equation: String describing the Einstein Summation operation; in the format of np.einsum.
+//
+// Returns Output Tensor with shape depending upon `equation`.
+func Einsum(scope *Scope, inputs []tf.Output, equation string) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"equation": equation}
+	opspec := tf.OpSpec{
+		Type: "Einsum",
+		Input: []tf.Input{
+			tf.OutputList(inputs),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Convert the quantized 'input' tensor into a lower-precision 'output', using the
+//
+// actual distribution of the values to maximize the usage of the lower bit depth
+// and adjusting the output min and max ranges accordingly.
+//
+// [input_min, input_max] are scalar floats that specify the range for the float
+// interpretation of the 'input' data. For example, if input_min is -1.0f and
+// input_max is 1.0f, and we are dealing with quint16 quantized data, then a 0
+// value in the 16-bit data should be interpreted as -1.0f, and a 65535 means 1.0f.
+//
+// This operator tries to squeeze as much precision as possible into an output with
+// a lower bit depth by calculating the actual min and max values found in the
+// data. For example, maybe that quint16 input has no values lower than 16,384 and
+// none higher than 49,152. That means only half the range is actually needed, all
+// the float interpretations are between -0.5f and 0.5f, so if we want to compress
+// the data into a quint8 output, we can use that range rather than the theoretical
+// -1.0f to 1.0f that is suggested by the input min and max.
+//
+// In practice, this is most useful for taking output from operations like
+// QuantizedMatMul that can produce higher bit-depth outputs than their inputs and
+// may have large potential output ranges, but in practice have a distribution of
+// input values that only uses a small fraction of the possible range. By feeding
+// that output into this operator, we can reduce it from 32 bits down to 8 with
+// minimal loss of accuracy.
+//
+// Arguments:
+//
+//	input_min: The float value that the minimum quantized input value represents.
+//	input_max: The float value that the maximum quantized input value represents.
+//	out_type: The type of the output. Should be a lower bit depth than Tinput.
+//
+// Returns:
+//	output
+//	output_min: The float value that the minimum quantized output value represents.
+//	output_max: The float value that the maximum quantized output value represents.
+func QuantizeDownAndShrinkRange(scope *Scope, input tf.Output, input_min tf.Output, input_max tf.Output, out_type tf.DataType) (output tf.Output, output_min tf.Output, output_max tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"out_type": out_type}
+	opspec := tf.OpSpec{
+		Type: "QuantizeDownAndShrinkRange",
+		Input: []tf.Input{
+			input, input_min, input_max,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Converts each string in the input Tensor to its hash mod by a number of buckets.
+//
+// The hash function is deterministic on the content of the string within the
+// process.
+//
+// Note that the hash function may change from time to time.
+// This functionality will be deprecated and it's recommended to use
+// `tf.string_to_hash_bucket_fast()` or `tf.string_to_hash_bucket_strong()`.
+//
+// Arguments:
+//
+//	num_buckets: The number of buckets.
+//
+// Returns A Tensor of the same shape as the input `string_tensor`.
+func StringToHashBucket(scope *Scope, string_tensor tf.Output, num_buckets int64) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_buckets": num_buckets}
+	opspec := tf.OpSpec{
+		Type: "StringToHashBucket",
+		Input: []tf.Input{
+			string_tensor,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes softsign: `features / (abs(features) + 1)`.
+func Softsign(scope *Scope, features tf.Output) (activations tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Softsign",
+		Input: []tf.Input{
+			features,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// QuantizedAddAttr is an optional argument to QuantizedAdd.
+type QuantizedAddAttr func(optionalAttr)
+
+// QuantizedAddToutput sets the optional Toutput attribute to value.
+// If not specified, defaults to DT_QINT32
+func QuantizedAddToutput(value tf.DataType) QuantizedAddAttr {
+	return func(m optionalAttr) {
+		m["Toutput"] = value
+	}
+}
+
+// Returns x + y element-wise, working on quantized buffers.
+//
+// Arguments:
+//
+//
+//	min_x: The float value that the lowest quantized `x` value represents.
+//	max_x: The float value that the highest quantized `x` value represents.
+//	min_y: The float value that the lowest quantized `y` value represents.
+//	max_y: The float value that the highest quantized `y` value represents.
+//
+// Returns:
+//	z
+//	min_z: The float value that the lowest quantized output value represents.
+//	max_z: The float value that the highest quantized output value represents.
+//
+// *NOTE*: `QuantizedAdd` supports limited forms of broadcasting. More about
+// broadcasting [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func QuantizedAdd(scope *Scope, x tf.Output, y tf.Output, min_x tf.Output, max_x tf.Output, min_y tf.Output, max_y tf.Output, optional ...QuantizedAddAttr) (z tf.Output, min_z tf.Output, max_z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizedAdd",
+		Input: []tf.Input{
+			x, y, min_x, max_x, min_y, max_y,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// ShuffleAndRepeatDatasetAttr is an optional argument to ShuffleAndRepeatDataset.
+type ShuffleAndRepeatDatasetAttr func(optionalAttr)
+
+// ShuffleAndRepeatDatasetReshuffleEachIteration sets the optional reshuffle_each_iteration attribute to value.
+// If not specified, defaults to true
+func ShuffleAndRepeatDatasetReshuffleEachIteration(value bool) ShuffleAndRepeatDatasetAttr {
+	return func(m optionalAttr) {
+		m["reshuffle_each_iteration"] = value
+	}
+}
+
+// Creates a dataset that shuffles and repeats elements from `input_dataset`
+//
+// pseudorandomly.
+//
+// Arguments:
+//
+//	buffer_size: The number of output elements to buffer in an iterator over
+// this dataset. Compare with the `min_after_dequeue` attr when creating a
+// `RandomShuffleQueue`.
+//	seed: A scalar seed for the random number generator. If either `seed` or
+// `seed2` is set to be non-zero, the random number generator is seeded
+// by the given seed.  Otherwise, a random seed is used.
+//	seed2: A second scalar seed to avoid seed collision.
+//	count: A scalar representing the number of times the underlying dataset
+// should be repeated. The default is `-1`, which results in infinite repetition.
+//
+//
+func ShuffleAndRepeatDataset(scope *Scope, input_dataset tf.Output, buffer_size tf.Output, seed tf.Output, seed2 tf.Output, count tf.Output, output_types []tf.DataType, output_shapes []tf.Shape, optional ...ShuffleAndRepeatDatasetAttr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ShuffleAndRepeatDataset",
+		Input: []tf.Input{
+			input_dataset, buffer_size, seed, seed2, count,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// QuantizedMulAttr is an optional argument to QuantizedMul.
+type QuantizedMulAttr func(optionalAttr)
+
+// QuantizedMulToutput sets the optional Toutput attribute to value.
+// If not specified, defaults to DT_QINT32
+func QuantizedMulToutput(value tf.DataType) QuantizedMulAttr {
+	return func(m optionalAttr) {
+		m["Toutput"] = value
+	}
+}
+
+// Returns x * y element-wise, working on quantized buffers.
+//
+// Arguments:
+//
+//
+//	min_x: The float value that the lowest quantized `x` value represents.
+//	max_x: The float value that the highest quantized `x` value represents.
+//	min_y: The float value that the lowest quantized `y` value represents.
+//	max_y: The float value that the highest quantized `y` value represents.
+//
+// Returns:
+//	z
+//	min_z: The float value that the lowest quantized output value represents.
+//	max_z: The float value that the highest quantized output value represents.
+//
+// *NOTE*: `QuantizedMul` supports limited forms of broadcasting. More about
+// broadcasting [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func QuantizedMul(scope *Scope, x tf.Output, y tf.Output, min_x tf.Output, max_x tf.Output, min_y tf.Output, max_y tf.Output, optional ...QuantizedMulAttr) (z tf.Output, min_z tf.Output, max_z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizedMul",
+		Input: []tf.Input{
+			x, y, min_x, max_x, min_y, max_y,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// CumulativeLogsumexpAttr is an optional argument to CumulativeLogsumexp.
+type CumulativeLogsumexpAttr func(optionalAttr)
+
+// CumulativeLogsumexpExclusive sets the optional exclusive attribute to value.
+//
+// value: If `True`, perform exclusive cumulative log-sum-exp.
+// If not specified, defaults to false
+func CumulativeLogsumexpExclusive(value bool) CumulativeLogsumexpAttr {
+	return func(m optionalAttr) {
+		m["exclusive"] = value
+	}
+}
+
+// CumulativeLogsumexpReverse sets the optional reverse attribute to value.
+//
+// value: A `bool` (default: False).
+// If not specified, defaults to false
+func CumulativeLogsumexpReverse(value bool) CumulativeLogsumexpAttr {
+	return func(m optionalAttr) {
+		m["reverse"] = value
+	}
+}
+
+// Compute the cumulative product of the tensor `x` along `axis`.
+//
+// By default, this op performs an inclusive cumulative log-sum-exp,
+// which means that the first
+// element of the input is identical to the first element of the output:
+// ```python
+// tf.math.cumulative_logsumexp([a, b, c])  # => [a, log(exp(a) + exp(b)), log(exp(a) + exp(b) + exp(c))]
+// ```
+//
+// By setting the `exclusive` kwarg to `True`, an exclusive cumulative log-sum-exp is
+// performed instead:
+// ```python
+// tf.cumulative_logsumexp([a, b, c], exclusive=True)  # => [-inf, a, log(exp(a) * exp(b))]
+// ```
+// Note that the neutral element of the log-sum-exp operation is `-inf`,
+// however, for performance reasons, the minimal value representable by the
+// floating point type is used instead.
+//
+// By setting the `reverse` kwarg to `True`, the cumulative log-sum-exp is performed in the
+// opposite direction.
+//
+// Arguments:
+//	x: A `Tensor`. Must be one of the following types: `float16`, `float32`, `float64`.
+//	axis: A `Tensor` of type `int32` (default: 0). Must be in the range
+// `[-rank(x), rank(x))`.
+func CumulativeLogsumexp(scope *Scope, x tf.Output, axis tf.Output, optional ...CumulativeLogsumexpAttr) (out tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CumulativeLogsumexp",
+		Input: []tf.Input{
+			x, axis,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResourceApplyGradientDescentAttr is an optional argument to ResourceApplyGradientDescent.
+type ResourceApplyGradientDescentAttr func(optionalAttr)
+
+// ResourceApplyGradientDescentUseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, the subtraction will be protected by a lock;
+// otherwise the behavior is undefined, but may exhibit less contention.
+// If not specified, defaults to false
+func ResourceApplyGradientDescentUseLocking(value bool) ResourceApplyGradientDescentAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// Update '*var' by subtracting 'alpha' * 'delta' from it.
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	alpha: Scaling factor. Must be a scalar.
+//	delta: The change.
+//
+// Returns the created operation.
+func ResourceApplyGradientDescent(scope *Scope, var_ tf.Output, alpha tf.Output, delta tf.Output, optional ...ResourceApplyGradientDescentAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceApplyGradientDescent",
+		Input: []tf.Input{
+			var_, alpha, delta,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Computes the matrix logarithm of one or more square matrices:
+//
+//
+// \\(log(exp(A)) = A\\)
+//
+// This op is only defined for complex matrices. If A is positive-definite and
+// real, then casting to a complex matrix, taking the logarithm and casting back
+// to a real matrix will give the correct result.
+//
+// This function computes the matrix logarithm using the Schur-Parlett algorithm.
+// Details of the algorithm can be found in Section 11.6.2 of:
+// Nicholas J. Higham, Functions of Matrices: Theory and Computation, SIAM 2008.
+// ISBN 978-0-898716-46-7.
+//
+// The input is a tensor of shape `[..., M, M]` whose inner-most 2 dimensions
+// form square matrices. The output is a tensor of the same shape as the input
+// containing the exponential for all input submatrices `[..., :, :]`.
+//
+// Arguments:
+//	input: Shape is `[..., M, M]`.
+//
+// Returns Shape is `[..., M, M]`.
+//
+// @compatibility(scipy)
+// Equivalent to scipy.linalg.logm
+// @end_compatibility
+func MatrixLogarithm(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "MatrixLogarithm",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SparseBincountAttr is an optional argument to SparseBincount.
+type SparseBincountAttr func(optionalAttr)
+
+// SparseBincountBinaryOutput sets the optional binary_output attribute to value.
+//
+// value: bool; Whether the kernel should count the appearance or number of occurrences.
+// If not specified, defaults to false
+func SparseBincountBinaryOutput(value bool) SparseBincountAttr {
+	return func(m optionalAttr) {
+		m["binary_output"] = value
+	}
+}
+
+// Counts the number of occurrences of each value in an integer array.
+//
+// Outputs a vector with length `size` and the same dtype as `weights`. If
+// `weights` are empty, then index `i` stores the number of times the value `i` is
+// counted in `arr`. If `weights` are non-empty, then index `i` stores the sum of
+// the value in `weights` at each index where the corresponding value in `arr` is
+// `i`.
+//
+// Values in `arr` outside of the range [0, size) are ignored.
+//
+// Arguments:
+//	indices: 2D int64 `Tensor`.
+//	values: 1D int `Tensor`.
+//	dense_shape: 1D int64 `Tensor`.
+//	size: non-negative int scalar `Tensor`.
+//	weights: is an int32, int64, float32, or float64 `Tensor` with the same
+// shape as `input`, or a length-0 `Tensor`, in which case it acts as all weights
+// equal to 1.
+//
+// Returns 1D `Tensor` with length equal to `size` or 2D `Tensor` with [batch_size, `size`].
+// The counts or summed weights for each value in the range [0, size).
+func SparseBincount(scope *Scope, indices tf.Output, values tf.Output, dense_shape tf.Output, size tf.Output, weights tf.Output, optional ...SparseBincountAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseBincount",
+		Input: []tf.Input{
+			indices, values, dense_shape, size, weights,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Calculate product with tridiagonal matrix.
+//
+// Calculates product of two matrices, where left matrix is a tridiagonal matrix.
+//
+// Arguments:
+//	superdiag: Tensor of shape `[..., 1, M]`, representing superdiagonals of
+// tri-diagonal matrices to the left of multiplication. Last element is ignored.
+//	maindiag: Tensor of shape `[..., 1, M]`, representing main diagonals of tri-diagonal
+// matrices to the left of multiplication.
+//	subdiag: Tensor of shape `[..., 1, M]`, representing subdiagonals of tri-diagonal
+// matrices to the left of multiplication. First element is ignored.
+//	rhs: Tensor of shape `[..., M, N]`, representing MxN matrices to the right of
+// multiplication.
+//
+// Returns Tensor of shape `[..., M, N]` containing the product.
+func TridiagonalMatMul(scope *Scope, superdiag tf.Output, maindiag tf.Output, subdiag tf.Output, rhs tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TridiagonalMatMul",
+		Input: []tf.Input{
+			superdiag, maindiag, subdiag, rhs,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes scaled exponential linear: `scale * alpha * (exp(features) - 1)`
+//
+// if < 0, `scale * features` otherwise.
+//
+// To be used together with
+// `initializer = tf.variance_scaling_initializer(factor=1.0, mode='FAN_IN')`.
+// For correct dropout, use `tf.contrib.nn.alpha_dropout`.
+//
+// See [Self-Normalizing Neural Networks](https://arxiv.org/abs/1706.02515)
+func Selu(scope *Scope, features tf.Output) (activations tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Selu",
+		Input: []tf.Input{
+			features,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// DenseBincountAttr is an optional argument to DenseBincount.
+type DenseBincountAttr func(optionalAttr)
+
+// DenseBincountBinaryOutput sets the optional binary_output attribute to value.
+//
+// value: bool; Whether the kernel should count the appearance or number of occurrences.
+// If not specified, defaults to false
+func DenseBincountBinaryOutput(value bool) DenseBincountAttr {
+	return func(m optionalAttr) {
+		m["binary_output"] = value
+	}
+}
+
+// Counts the number of occurrences of each value in an integer array.
+//
+// Outputs a vector with length `size` and the same dtype as `weights`. If
+// `weights` are empty, then index `i` stores the number of times the value `i` is
+// counted in `arr`. If `weights` are non-empty, then index `i` stores the sum of
+// the value in `weights` at each index where the corresponding value in `arr` is
+// `i`.
+//
+// Values in `arr` outside of the range [0, size) are ignored.
+//
+// Arguments:
+//	input: 1D or 2D int `Tensor`.
+//	size: non-negative int scalar `Tensor`.
+//	weights: is an int32, int64, float32, or float64 `Tensor` with the same
+// shape as `arr`, or a length-0 `Tensor`, in which case it acts as all weights
+// equal to 1.
+//
+// Returns 1D `Tensor` with length equal to `size` or 2D `Tensor` with [batch_size, `size`].
+// The counts or summed weights for each value in the range [0, size).
+func DenseBincount(scope *Scope, input tf.Output, size tf.Output, weights tf.Output, optional ...DenseBincountAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DenseBincount",
+		Input: []tf.Input{
+			input, size, weights,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns the complex conjugate of a complex number.
+//
+// Given a tensor `input` of complex numbers, this operation returns a tensor of
+// complex numbers that are the complex conjugate of each element in `input`. The
+// complex numbers in `input` must be of the form \\(a + bj\\), where *a* is the
+// real part and *b* is the imaginary part.
+//
+// The complex conjugate returned by this operation is of the form \\(a - bj\\).
+//
+// For example:
+//
+// ```
+// # tensor 'input' is [-2.25 + 4.75j, 3.25 + 5.75j]
+// tf.conj(input) ==> [-2.25 - 4.75j, 3.25 - 5.75j]
+// ```
+func Conj(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Conj",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ImagAttr is an optional argument to Imag.
+type ImagAttr func(optionalAttr)
+
+// ImagTout sets the optional Tout attribute to value.
+// If not specified, defaults to DT_FLOAT
+func ImagTout(value tf.DataType) ImagAttr {
+	return func(m optionalAttr) {
+		m["Tout"] = value
+	}
+}
+
+// Returns the imaginary part of a complex number.
+//
+// Given a tensor `input` of complex numbers, this operation returns a tensor of
+// type `float` that is the imaginary part of each element in `input`. All
+// elements in `input` must be complex numbers of the form \\(a + bj\\), where *a*
+// is the real part and *b* is the imaginary part returned by this operation.
+//
+// For example:
+//
+// ```
+// # tensor 'input' is [-2.25 + 4.75j, 3.25 + 5.75j]
+// tf.imag(input) ==> [4.75, 5.75]
+// ```
+func Imag(scope *Scope, input tf.Output, optional ...ImagAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Imag",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RealAttr is an optional argument to Real.
+type RealAttr func(optionalAttr)
+
+// RealTout sets the optional Tout attribute to value.
+// If not specified, defaults to DT_FLOAT
+func RealTout(value tf.DataType) RealAttr {
+	return func(m optionalAttr) {
+		m["Tout"] = value
+	}
+}
+
+// Returns the real part of a complex number.
+//
+// Given a tensor `input` of complex numbers, this operation returns a tensor of
+// type `float` that is the real part of each element in `input`. All elements in
+// `input` must be complex numbers of the form \\(a + bj\\), where *a* is the real
+//  part returned by this operation and *b* is the imaginary part.
+//
+// For example:
+//
+// ```
+// # tensor 'input' is [-2.25 + 4.75j, 3.25 + 5.75j]
+// tf.real(input) ==> [-2.25, 3.25]
+// ```
+func Real(scope *Scope, input tf.Output, optional ...RealAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Real",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// DequantizeAttr is an optional argument to Dequantize.
+type DequantizeAttr func(optionalAttr)
+
+// DequantizeMode sets the optional mode attribute to value.
+// If not specified, defaults to "MIN_COMBINED"
+func DequantizeMode(value string) DequantizeAttr {
+	return func(m optionalAttr) {
+		m["mode"] = value
+	}
+}
+
+// DequantizeNarrowRange sets the optional narrow_range attribute to value.
+// If not specified, defaults to false
+func DequantizeNarrowRange(value bool) DequantizeAttr {
+	return func(m optionalAttr) {
+		m["narrow_range"] = value
+	}
+}
+
+// DequantizeAxis sets the optional axis attribute to value.
+// If not specified, defaults to -1
+func DequantizeAxis(value int64) DequantizeAttr {
+	return func(m optionalAttr) {
+		m["axis"] = value
+	}
+}
+
+// DequantizeDtype sets the optional dtype attribute to value.
+//
+// value: Type of the output tensor. Currently Dequantize supports float and bfloat16.
+// If 'dtype' is 'bfloat16', it only supports 'MIN_COMBINED' mode.
+// If not specified, defaults to DT_FLOAT
+func DequantizeDtype(value tf.DataType) DequantizeAttr {
+	return func(m optionalAttr) {
+		m["dtype"] = value
+	}
+}
+
+// Dequantize the 'input' tensor into a float or bfloat16 Tensor.
+//
+// [min_range, max_range] are scalar floats that specify the range for
+// the output. The 'mode' attribute controls exactly which calculations are
+// used to convert the float values to their quantized equivalents.
+//
+// In 'MIN_COMBINED' mode, each value of the tensor will undergo the following:
+//
+// ```
+// if T == qint8: in[i] += (range(T) + 1)/ 2.0
+// out[i] = min_range + (in[i]* (max_range - min_range) / range(T))
+// ```
+// here `range(T) = numeric_limits<T>::max() - numeric_limits<T>::min()`
+//
+// *MIN_COMBINED Mode Example*
+//
+// If the input comes from a QuantizedRelu6, the output type is
+// quint8 (range of 0-255) but the possible range of QuantizedRelu6 is
+// 0-6.  The min_range and max_range values are therefore 0.0 and 6.0.
+// Dequantize on quint8 will take each value, cast to float, and multiply
+// by 6 / 255.
+// Note that if quantizedtype is qint8, the operation will additionally add
+// each value by 128 prior to casting.
+//
+// If the mode is 'MIN_FIRST', then this approach is used:
+//
+// ```c++
+// num_discrete_values = 1 << (# of bits in T)
+// range_adjust = num_discrete_values / (num_discrete_values - 1)
+// range = (range_max - range_min) * range_adjust
+// range_scale = range / num_discrete_values
+// const double offset_input = static_cast<double>(input) - lowest_quantized;
+// result = range_min + ((input - numeric_limits<T>::min()) * range_scale)
+// ```
+//
+// If the mode is `SCALED`, dequantization is performed by multiplying each
+// input value by a scaling_factor. (Thus an input of 0 always maps to 0.0).
+//
+// The scaling_factor is determined from `min_range`, `max_range`, and
+// `narrow_range` in a way that is compatible with `QuantizeAndDequantize{V2|V3}`
+// and `QuantizeV2`, using the following algorithm:
+//
+// ```c++
+//
+//   const int min_expected_T = std::numeric_limits<T>::min() +
+//     (narrow_range ? 1 : 0);
+//   const int max_expected_T = std::numeric_limits<T>::max();
+//   const float max_expected_T = std::numeric_limits<float>::max();
+//
+//   const float scale_factor =
+//     (std::numeric_limits<T>::min() == 0) ? (max_range / max_expected_T)
+//                                          : std::max(min_range / min_expected_T,
+//                                                     max_range / max_expected_T);
+// ```
+//
+// Arguments:
+//
+//	min_range: The minimum scalar value possibly produced for the input.
+//	max_range: The maximum scalar value possibly produced for the input.
+func Dequantize(scope *Scope, input tf.Output, min_range tf.Output, max_range tf.Output, optional ...DequantizeAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Dequantize",
+		Input: []tf.Input{
+			input, min_range, max_range,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ComplexAttr is an optional argument to Complex.
+type ComplexAttr func(optionalAttr)
+
+// ComplexTout sets the optional Tout attribute to value.
+// If not specified, defaults to DT_COMPLEX64
+func ComplexTout(value tf.DataType) ComplexAttr {
+	return func(m optionalAttr) {
+		m["Tout"] = value
+	}
+}
+
+// Converts two real numbers to a complex number.
+//
+// Given a tensor `real` representing the real part of a complex number, and a
+// tensor `imag` representing the imaginary part of a complex number, this
+// operation returns complex numbers elementwise of the form \\(a + bj\\), where
+// *a* represents the `real` part and *b* represents the `imag` part.
+//
+// The input tensors `real` and `imag` must have the same shape.
+//
+// For example:
+//
+// ```
+// # tensor 'real' is [2.25, 3.25]
+// # tensor `imag` is [4.75, 5.75]
+// tf.complex(real, imag) ==> [[2.25 + 4.75j], [3.25 + 5.75j]]
+// ```
+func Complex(scope *Scope, real tf.Output, imag tf.Output, optional ...ComplexAttr) (out tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Complex",
+		Input: []tf.Input{
+			real, imag,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// CudnnRNNCanonicalToParamsV2Attr is an optional argument to CudnnRNNCanonicalToParamsV2.
+type CudnnRNNCanonicalToParamsV2Attr func(optionalAttr)
+
+// CudnnRNNCanonicalToParamsV2RnnMode sets the optional rnn_mode attribute to value.
+// If not specified, defaults to "lstm"
+func CudnnRNNCanonicalToParamsV2RnnMode(value string) CudnnRNNCanonicalToParamsV2Attr {
+	return func(m optionalAttr) {
+		m["rnn_mode"] = value
+	}
+}
+
+// CudnnRNNCanonicalToParamsV2InputMode sets the optional input_mode attribute to value.
+// If not specified, defaults to "linear_input"
+func CudnnRNNCanonicalToParamsV2InputMode(value string) CudnnRNNCanonicalToParamsV2Attr {
+	return func(m optionalAttr) {
+		m["input_mode"] = value
+	}
+}
+
+// CudnnRNNCanonicalToParamsV2Direction sets the optional direction attribute to value.
+// If not specified, defaults to "unidirectional"
+func CudnnRNNCanonicalToParamsV2Direction(value string) CudnnRNNCanonicalToParamsV2Attr {
+	return func(m optionalAttr) {
+		m["direction"] = value
+	}
+}
+
+// CudnnRNNCanonicalToParamsV2Dropout sets the optional dropout attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNCanonicalToParamsV2Dropout(value float32) CudnnRNNCanonicalToParamsV2Attr {
+	return func(m optionalAttr) {
+		m["dropout"] = value
+	}
+}
+
+// CudnnRNNCanonicalToParamsV2Seed sets the optional seed attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNCanonicalToParamsV2Seed(value int64) CudnnRNNCanonicalToParamsV2Attr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// CudnnRNNCanonicalToParamsV2Seed2 sets the optional seed2 attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNCanonicalToParamsV2Seed2(value int64) CudnnRNNCanonicalToParamsV2Attr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// CudnnRNNCanonicalToParamsV2NumProj sets the optional num_proj attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNCanonicalToParamsV2NumProj(value int64) CudnnRNNCanonicalToParamsV2Attr {
+	return func(m optionalAttr) {
+		m["num_proj"] = value
+	}
+}
+
+// Converts CudnnRNN params from canonical form to usable form. It supports the projection in LSTM.
+//
+// Writes a set of weights into the opaque params buffer so they can be used in
+// upcoming training or inferences.
+//
+// Note that the params buffer may not be compatible across different GPUs. So any
+// save and restoration should be converted to and from the canonical weights and
+// biases.
+//
+// num_layers: Specifies the number of layers in the RNN model.
+// num_units: Specifies the size of the hidden state.
+// input_size: Specifies the size of the input state.
+// weights: the canonical form of weights that can be used for saving
+//     and restoration. They are more likely to be compatible across different
+//     generations.
+// biases: the canonical form of biases that can be used for saving
+//     and restoration. They are more likely to be compatible across different
+//     generations.
+// num_params_weights: number of weight parameter matrix for all layers.
+// num_params_biases: number of bias parameter vector for all layers.
+// rnn_mode: Indicates the type of the RNN model.
+// input_mode: Indicate whether there is a linear projection between the input and
+//     The actual computation before the first layer. 'skip_input' is only allowed
+//     when input_size == num_units; 'auto_select' implies 'skip_input' when
+//     input_size == num_units; otherwise, it implies 'linear_input'.
+// direction: Indicates whether a bidirectional model will be used.
+//     dir = (direction == bidirectional) ? 2 : 1
+// dropout: dropout probability. When set to 0., dropout is disabled.
+// seed: the 1st part of a seed to initialize dropout.
+// seed2: the 2nd part of a seed to initialize dropout.
+// num_proj: The output dimensionality for the projection matrices. If None or 0,
+//     no projection is performed.
+func CudnnRNNCanonicalToParamsV2(scope *Scope, num_layers tf.Output, num_units tf.Output, input_size tf.Output, weights []tf.Output, biases []tf.Output, optional ...CudnnRNNCanonicalToParamsV2Attr) (params tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CudnnRNNCanonicalToParamsV2",
+		Input: []tf.Input{
+			num_layers, num_units, input_size, tf.OutputList(weights), tf.OutputList(biases),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a sequence of numbers.
+//
+// This operation creates a sequence of numbers that begins at `start` and
+// extends by increments of `delta` up to but not including `limit`.
+//
+// For example:
+//
+// ```
+// # 'start' is 3
+// # 'limit' is 18
+// # 'delta' is 3
+// tf.range(start, limit, delta) ==> [3, 6, 9, 12, 15]
+// ```
+//
+// Arguments:
+//	start: 0-D (scalar). First entry in the sequence.
+//	limit: 0-D (scalar). Upper limit of sequence, exclusive.
+//	delta: 0-D (scalar). Optional. Default is 1. Number that increments `start`.
+//
+// Returns 1-D.
+func Range(scope *Scope, start tf.Output, limit tf.Output, delta tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Range",
+		Input: []tf.Input{
+			start, limit, delta,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// AnyAttr is an optional argument to Any.
+type AnyAttr func(optionalAttr)
+
+// AnyKeepDims sets the optional keep_dims attribute to value.
+//
+// value: If true, retain reduced dimensions with length 1.
+// If not specified, defaults to false
+func AnyKeepDims(value bool) AnyAttr {
+	return func(m optionalAttr) {
+		m["keep_dims"] = value
+	}
+}
+
+// Computes the "logical or" of elements across dimensions of a tensor.
+//
+// Reduces `input` along the dimensions given in `axis`. Unless
+// `keep_dims` is true, the rank of the tensor is reduced by 1 for each entry in
+// `axis`. If `keep_dims` is true, the reduced dimensions are
+// retained with length 1.
+//
+// Arguments:
+//	input: The tensor to reduce.
+//	axis: The dimensions to reduce. Must be in the range
+// `[-rank(input), rank(input))`.
+//
+// Returns The reduced tensor.
+func Any(scope *Scope, input tf.Output, axis tf.Output, optional ...AnyAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Any",
+		Input: []tf.Input{
+			input, axis,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the gradient of morphological 2-D dilation with respect to the input.
+//
+// Arguments:
+//	input: 4-D with shape `[batch, in_height, in_width, depth]`.
+//	filter: 3-D with shape `[filter_height, filter_width, depth]`.
+//	out_backprop: 4-D with shape `[batch, out_height, out_width, depth]`.
+//	strides: 1-D of length 4. The stride of the sliding window for each dimension of
+// the input tensor. Must be: `[1, stride_height, stride_width, 1]`.
+//	rates: 1-D of length 4. The input stride for atrous morphological dilation.
+// Must be: `[1, rate_height, rate_width, 1]`.
+//	padding: The type of padding algorithm to use.
+//
+// Returns 4-D with shape `[batch, in_height, in_width, depth]`.
+func Dilation2DBackpropInput(scope *Scope, input tf.Output, filter tf.Output, out_backprop tf.Output, strides []int64, rates []int64, padding string) (in_backprop tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"strides": strides, "rates": rates, "padding": padding}
+	opspec := tf.OpSpec{
+		Type: "Dilation2DBackpropInput",
+		Input: []tf.Input{
+			input, filter, out_backprop,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// AllAttr is an optional argument to All.
+type AllAttr func(optionalAttr)
+
+// AllKeepDims sets the optional keep_dims attribute to value.
+//
+// value: If true, retain reduced dimensions with length 1.
+// If not specified, defaults to false
+func AllKeepDims(value bool) AllAttr {
+	return func(m optionalAttr) {
+		m["keep_dims"] = value
+	}
+}
+
+// Computes the "logical and" of elements across dimensions of a tensor.
+//
+// Reduces `input` along the dimensions given in `axis`. Unless
+// `keep_dims` is true, the rank of the tensor is reduced by 1 for each entry in
+// `axis`. If `keep_dims` is true, the reduced dimensions are
+// retained with length 1.
+//
+// Arguments:
+//	input: The tensor to reduce.
+//	axis: The dimensions to reduce. Must be in the range
+// `[-rank(input), rank(input))`.
+//
+// Returns The reduced tensor.
+func All(scope *Scope, input tf.Output, axis tf.Output, optional ...AllAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "All",
+		Input: []tf.Input{
+			input, axis,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes gradients for SparseSegmentSqrtN.
+//
+// Returns tensor "output" with same shape as grad, except for dimension 0 whose
+// value is output_dim0.
+//
+// Arguments:
+//	grad: gradient propagated to the SparseSegmentSqrtN op.
+//	indices: indices passed to the corresponding SparseSegmentSqrtN op.
+//	segment_ids: segment_ids passed to the corresponding SparseSegmentSqrtN op.
+//	output_dim0: dimension 0 of "data" passed to SparseSegmentSqrtN op.
+func SparseSegmentSqrtNGrad(scope *Scope, grad tf.Output, indices tf.Output, segment_ids tf.Output, output_dim0 tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseSegmentSqrtNGrad",
+		Input: []tf.Input{
+			grad, indices, segment_ids, output_dim0,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the mean along sparse segments of a tensor.
+//
+// See `tf.sparse.segment_sum` for usage examples.
+//
+// Like `SegmentMean`, but `segment_ids` can have rank less than `data`'s first
+// dimension, selecting a subset of dimension 0, specified by `indices`.
+//
+// Arguments:
+//
+//	indices: A 1-D tensor. Has same rank as `segment_ids`.
+//	segment_ids: A 1-D tensor. Values should be sorted and can be repeated.
+//
+// Returns Has same shape as data, except for dimension 0 which
+// has size `k`, the number of segments.
+func SparseSegmentMean(scope *Scope, data tf.Output, indices tf.Output, segment_ids tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseSegmentMean",
+		Input: []tf.Input{
+			data, indices, segment_ids,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the sum along sparse segments of a tensor.
+//
+// Like `SparseSegmentSum`, but allows missing ids in `segment_ids`. If an id is
+// missing, the `output` tensor at that position will be zeroed.
+//
+// Read
+// [the section on segmentation](https://tensorflow.org/api_docs/python/tf/sparse#Segmentation)
+// for an explanation of segments.
+//
+// For example:
+//
+// ```python
+// c = tf.constant([[1,2,3,4], [-1,-2,-3,-4], [5,6,7,8]])
+//
+// tf.sparse_segment_sum_with_num_segments(
+//     c, tf.constant([0, 1]), tf.constant([0, 0]), num_segments=3)
+// # => [[0 0 0 0]
+// #     [0 0 0 0]
+// #     [0 0 0 0]]
+//
+// tf.sparse_segment_sum_with_num_segments(c,
+//                                         tf.constant([0, 1]),
+//                                         tf.constant([0, 2],
+//                                         num_segments=4))
+// # => [[ 1  2  3  4]
+// #     [ 0  0  0  0]
+// #     [-1 -2 -3 -4]
+// #     [ 0  0  0  0]]
+// ```
+//
+// Arguments:
+//
+//	indices: A 1-D tensor. Has same rank as `segment_ids`.
+//	segment_ids: A 1-D tensor. Values should be sorted and can be repeated.
+//	num_segments: Should equal the number of distinct segment IDs.
+//
+// Returns Has same shape as data, except for dimension 0 which
+// has size `num_segments`.
+func SparseSegmentSumWithNumSegments(scope *Scope, data tf.Output, indices tf.Output, segment_ids tf.Output, num_segments tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseSegmentSumWithNumSegments",
+		Input: []tf.Input{
+			data, indices, segment_ids, num_segments,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the sum along sparse segments of a tensor.
+//
+// Read
+// [the section on segmentation](https://tensorflow.org/api_docs/python/tf/math#Segmentation)
+// for an explanation of segments.
+//
+// Like `SegmentSum`, but `segment_ids` can have rank less than `data`'s first
+// dimension, selecting a subset of dimension 0, specified by `indices`.
+//
+// For example:
+//
+// ```python
+// c = tf.constant([[1,2,3,4], [-1,-2,-3,-4], [5,6,7,8]])
+//
+// # Select two rows, one segment.
+// tf.sparse_segment_sum(c, tf.constant([0, 1]), tf.constant([0, 0]))
+// # => [[0 0 0 0]]
+//
+// # Select two rows, two segment.
+// tf.sparse_segment_sum(c, tf.constant([0, 1]), tf.constant([0, 1]))
+// # => [[ 1  2  3  4]
+// #     [-1 -2 -3 -4]]
+//
+// # Select all rows, two segments.
+// tf.sparse_segment_sum(c, tf.constant([0, 1, 2]), tf.constant([0, 0, 1]))
+// # => [[0 0 0 0]
+// #     [5 6 7 8]]
+//
+// # Which is equivalent to:
+// tf.segment_sum(c, tf.constant([0, 0, 1]))
+// ```
+//
+// Arguments:
+//
+//	indices: A 1-D tensor. Has same rank as `segment_ids`.
+//	segment_ids: A 1-D tensor. Values should be sorted and can be repeated.
+//
+// Returns Has same shape as data, except for dimension 0 which
+// has size `k`, the number of segments.
+func SparseSegmentSum(scope *Scope, data tf.Output, indices tf.Output, segment_ids tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseSegmentSum",
+		Input: []tf.Input{
+			data, indices, segment_ids,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the sum along segments of a tensor.
+//
+// Read
+// [the section on segmentation](https://tensorflow.org/api_docs/python/tf/math#Segmentation)
+// for an explanation of segments.
+//
+// Computes a tensor such that
+// \\(output[i] = \sum_{j...} data[j...]\\) where the sum is over tuples `j...` such
+// that `segment_ids[j...] == i`.  Unlike `SegmentSum`, `segment_ids`
+// need not be sorted and need not cover all values in the full
+// range of valid values.
+//
+// If the sum is empty for a given segment ID `i`, `output[i] = 0`.
+// If the given segment ID `i` is negative, the value is dropped and will not be
+// added to the sum of the segment.
+//
+// `num_segments` should equal the number of distinct segment IDs.
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src="https://www.tensorflow.org/images/UnsortedSegmentSum.png" alt>
+// </div>
+//
+// ``` python
+// c = tf.constant([[1,2,3,4], [5,6,7,8], [4,3,2,1]])
+// tf.unsorted_segment_sum(c, tf.constant([0, 1, 0]), num_segments=2)
+// # ==> [[ 5,  5, 5, 5],
+// #       [5,  6, 7, 8]]
+// ```
+//
+//
+// Arguments:
+//
+//	segment_ids: A tensor whose shape is a prefix of `data.shape`.
+//
+//
+// Returns Has same shape as data, except for the first `segment_ids.rank`
+// dimensions, which are replaced with a single dimension which has size
+// `num_segments`.
+func UnsortedSegmentSum(scope *Scope, data tf.Output, segment_ids tf.Output, num_segments tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "UnsortedSegmentSum",
+		Input: []tf.Input{
+			data, segment_ids, num_segments,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes arctangent of `y/x` element-wise, respecting signs of the arguments.
+//
+// This is the angle \( \theta \in [-\pi, \pi] \) such that
+// \[ x = r \cos(\theta) \]
+// and
+// \[ y = r \sin(\theta) \]
+// where \(r = \sqrt(x^2 + y^2) \).
+func Atan2(scope *Scope, y tf.Output, x tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Atan2",
+		Input: []tf.Input{
+			y, x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the product along segments of a tensor.
+//
+// Read
+// [the section on segmentation](https://tensorflow.org/api_docs/python/tf/math#Segmentation)
+// for an explanation of segments.
+//
+// Computes a tensor such that
+// \\(output_i = \prod_j data_j\\) where the product is over `j` such
+// that `segment_ids[j] == i`.
+//
+// If the product is empty for a given segment ID `i`, `output[i] = 1`.
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src="https://www.tensorflow.org/images/SegmentProd.png" alt>
+// </div>
+//
+// For example:
+//
+// ```
+// c = tf.constant([[1,2,3,4], [4, 3, 2, 1], [5,6,7,8]])
+// tf.segment_prod(c, tf.constant([0, 0, 1]))
+// # ==> [[4, 6, 6, 4],
+// #      [5, 6, 7, 8]]
+// ```
+//
+//
+// Arguments:
+//
+//	segment_ids: A 1-D tensor whose size is equal to the size of `data`'s
+// first dimension.  Values should be sorted and can be repeated.
+//
+// Returns Has same shape as data, except for dimension 0 which
+// has size `k`, the number of segments.
+func SegmentProd(scope *Scope, data tf.Output, segment_ids tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SegmentProd",
+		Input: []tf.Input{
+			data, segment_ids,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResourceApplyAdamWithAmsgradAttr is an optional argument to ResourceApplyAdamWithAmsgrad.
+type ResourceApplyAdamWithAmsgradAttr func(optionalAttr)
+
+// ResourceApplyAdamWithAmsgradUseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var, m, and v tensors will be protected
+// by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceApplyAdamWithAmsgradUseLocking(value bool) ResourceApplyAdamWithAmsgradAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// Update '*var' according to the Adam algorithm.
+//
+// $$\text{lr}_t := \mathrm{learning_rate} * \sqrt{1 - \beta_2^t} / (1 - \beta_1^t)$$
+// $$m_t := \beta_1 * m_{t-1} + (1 - \beta_1) * g$$
+// $$v_t := \beta_2 * v_{t-1} + (1 - \beta_2) * g * g$$
+// $$\hat{v}_t := max{\hat{v}_{t-1}, v_t}$$
+// $$\text{variable} := \text{variable} - \text{lr}_t * m_t / (\sqrt{\hat{v}_t} + \epsilon)$$
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	m: Should be from a Variable().
+//	v: Should be from a Variable().
+//	vhat: Should be from a Variable().
+//	beta1_power: Must be a scalar.
+//	beta2_power: Must be a scalar.
+//	lr: Scaling factor. Must be a scalar.
+//	beta1: Momentum factor. Must be a scalar.
+//	beta2: Momentum factor. Must be a scalar.
+//	epsilon: Ridge term. Must be a scalar.
+//	grad: The gradient.
+//
+// Returns the created operation.
+func ResourceApplyAdamWithAmsgrad(scope *Scope, var_ tf.Output, m tf.Output, v tf.Output, vhat tf.Output, beta1_power tf.Output, beta2_power tf.Output, lr tf.Output, beta1 tf.Output, beta2 tf.Output, epsilon tf.Output, grad tf.Output, optional ...ResourceApplyAdamWithAmsgradAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceApplyAdamWithAmsgrad",
+		Input: []tf.Input{
+			var_, m, v, vhat, beta1_power, beta2_power, lr, beta1, beta2, epsilon, grad,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Computes the sum along segments of a tensor.
+//
+// Read
+// [the section on segmentation](https://tensorflow.org/api_docs/python/tf/math#Segmentation)
+// for an explanation of segments.
+//
+// Computes a tensor such that
+// \\(output_i = \sum_j data_j\\) where sum is over `j` such
+// that `segment_ids[j] == i`.
+//
+// If the sum is empty for a given segment ID `i`, `output[i] = 0`.
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src="https://www.tensorflow.org/images/SegmentSum.png" alt>
+// </div>
+//
+// For example:
+//
+// ```
+// c = tf.constant([[1,2,3,4], [4, 3, 2, 1], [5,6,7,8]])
+// tf.segment_sum(c, tf.constant([0, 0, 1]))
+// # ==> [[5, 5, 5, 5],
+// #      [5, 6, 7, 8]]
+// ```
+//
+//
+// Arguments:
+//
+//	segment_ids: A 1-D tensor whose size is equal to the size of `data`'s
+// first dimension.  Values should be sorted and can be repeated.
+//
+// Returns Has same shape as data, except for dimension 0 which
+// has size `k`, the number of segments.
+func SegmentSum(scope *Scope, data tf.Output, segment_ids tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SegmentSum",
+		Input: []tf.Input{
+			data, segment_ids,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ArgMinAttr is an optional argument to ArgMin.
+type ArgMinAttr func(optionalAttr)
+
+// ArgMinOutputType sets the optional output_type attribute to value.
+// If not specified, defaults to DT_INT64
+func ArgMinOutputType(value tf.DataType) ArgMinAttr {
+	return func(m optionalAttr) {
+		m["output_type"] = value
+	}
+}
+
+// Returns the index with the smallest value across dimensions of a tensor.
+//
+// Note that in case of ties the identity of the return value is not guaranteed.
+//
+// Usage:
+//   ```python
+//   import tensorflow as tf
+//   a = [1, 10, 26.9, 2.8, 166.32, 62.3]
+//   b = tf.math.argmin(input = a)
+//   c = tf.keras.backend.eval(b)
+//   # c = 0
+//   # here a[0] = 1 which is the smallest element of a across axis 0
+//   ```
+//
+// Arguments:
+//
+//	dimension: int32 or int64, must be in the range `[-rank(input), rank(input))`.
+// Describes which dimension of the input Tensor to reduce across. For vectors,
+// use dimension = 0.
+func ArgMin(scope *Scope, input tf.Output, dimension tf.Output, optional ...ArgMinAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ArgMin",
+		Input: []tf.Input{
+			input, dimension,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the reverse mode backpropagated gradient of the Cholesky algorithm.
+//
+// For an explanation see "Differentiation of the Cholesky algorithm" by
+// Iain Murray http://arxiv.org/abs/1602.07527.
+//
+// Arguments:
+//	l: Output of batch Cholesky algorithm l = cholesky(A). Shape is `[..., M, M]`.
+// Algorithm depends only on lower triangular part of the innermost matrices of
+// this tensor.
+//	grad: df/dl where f is some scalar function. Shape is `[..., M, M]`.
+// Algorithm depends only on lower triangular part of the innermost matrices of
+// this tensor.
+//
+// Returns Symmetrized version of df/dA . Shape is `[..., M, M]`
+func CholeskyGrad(scope *Scope, l tf.Output, grad tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "CholeskyGrad",
+		Input: []tf.Input{
+			l, grad,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Reshapes a tensor.
+//
+// Given `tensor`, this operation returns a tensor that has the same values
+// as `tensor` with shape `shape`.
+//
+// If one component of 1-D tensor `shape` is the special value -1, the size of that
+// dimension is computed so that the total size remains constant.  In particular, a
+// `shape` of `[-1]` flattens into 1-D.  At most one component of `shape` may be
+// unknown.
+//
+// The `shape` must be 1-D and the operation returns a tensor with shape
+// `shape` filled with the values of `tensor`. In this case, the number of elements
+// implied by `shape` must be the same as the number of elements in `tensor`.
+//
+// It is an error if `shape` is not 1-D.
+//
+// For example:
+//
+// ```
+// # tensor 't' is [1, 2, 3, 4, 5, 6, 7, 8, 9]
+// # tensor 't' has shape [9]
+// reshape(t, [3, 3]) ==> [[1, 2, 3],
+//                         [4, 5, 6],
+//                         [7, 8, 9]]
+//
+// # tensor 't' is [[[1, 1], [2, 2]],
+// #                [[3, 3], [4, 4]]]
+// # tensor 't' has shape [2, 2, 2]
+// reshape(t, [2, 4]) ==> [[1, 1, 2, 2],
+//                         [3, 3, 4, 4]]
+//
+// # tensor 't' is [[[1, 1, 1],
+// #                 [2, 2, 2]],
+// #                [[3, 3, 3],
+// #                 [4, 4, 4]],
+// #                [[5, 5, 5],
+// #                 [6, 6, 6]]]
+// # tensor 't' has shape [3, 2, 3]
+// # pass '[-1]' to flatten 't'
+// reshape(t, [-1]) ==> [1, 1, 1, 2, 2, 2, 3, 3, 3, 4, 4, 4, 5, 5, 5, 6, 6, 6]
+//
+// # -1 can also be used to infer the shape
+//
+// # -1 is inferred to be 9:
+// reshape(t, [2, -1]) ==> [[1, 1, 1, 2, 2, 2, 3, 3, 3],
+//                          [4, 4, 4, 5, 5, 5, 6, 6, 6]]
+// # -1 is inferred to be 2:
+// reshape(t, [-1, 9]) ==> [[1, 1, 1, 2, 2, 2, 3, 3, 3],
+//                          [4, 4, 4, 5, 5, 5, 6, 6, 6]]
+// # -1 is inferred to be 3:
+// reshape(t, [ 2, -1, 3]) ==> [[[1, 1, 1],
+//                               [2, 2, 2],
+//                               [3, 3, 3]],
+//                              [[4, 4, 4],
+//                               [5, 5, 5],
+//                               [6, 6, 6]]]
+//
+// # tensor 't' is [7]
+// # shape `[]` reshapes to a scalar
+// reshape(t, []) ==> 7
+// ```
+//
+// Arguments:
+//
+//	shape: Defines the shape of the output tensor.
+func Reshape(scope *Scope, tensor tf.Output, shape tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Reshape",
+		Input: []tf.Input{
+			tensor, shape,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SnapshotDatasetAttr is an optional argument to SnapshotDataset.
+type SnapshotDatasetAttr func(optionalAttr)
+
+// SnapshotDatasetCompression sets the optional compression attribute to value.
+// If not specified, defaults to ""
+func SnapshotDatasetCompression(value string) SnapshotDatasetAttr {
+	return func(m optionalAttr) {
+		m["compression"] = value
+	}
+}
+
+// SnapshotDatasetReaderPathPrefix sets the optional reader_path_prefix attribute to value.
+// If not specified, defaults to ""
+func SnapshotDatasetReaderPathPrefix(value string) SnapshotDatasetAttr {
+	return func(m optionalAttr) {
+		m["reader_path_prefix"] = value
+	}
+}
+
+// SnapshotDatasetWriterPathPrefix sets the optional writer_path_prefix attribute to value.
+// If not specified, defaults to ""
+func SnapshotDatasetWriterPathPrefix(value string) SnapshotDatasetAttr {
+	return func(m optionalAttr) {
+		m["writer_path_prefix"] = value
+	}
+}
+
+// SnapshotDatasetShardSizeBytes sets the optional shard_size_bytes attribute to value.
+// If not specified, defaults to 10737418240
+func SnapshotDatasetShardSizeBytes(value int64) SnapshotDatasetAttr {
+	return func(m optionalAttr) {
+		m["shard_size_bytes"] = value
+	}
+}
+
+// SnapshotDatasetPendingSnapshotExpirySeconds sets the optional pending_snapshot_expiry_seconds attribute to value.
+// If not specified, defaults to 86400
+func SnapshotDatasetPendingSnapshotExpirySeconds(value int64) SnapshotDatasetAttr {
+	return func(m optionalAttr) {
+		m["pending_snapshot_expiry_seconds"] = value
+	}
+}
+
+// SnapshotDatasetNumReaderThreads sets the optional num_reader_threads attribute to value.
+// If not specified, defaults to 1
+func SnapshotDatasetNumReaderThreads(value int64) SnapshotDatasetAttr {
+	return func(m optionalAttr) {
+		m["num_reader_threads"] = value
+	}
+}
+
+// SnapshotDatasetReaderBufferSize sets the optional reader_buffer_size attribute to value.
+// If not specified, defaults to 1
+func SnapshotDatasetReaderBufferSize(value int64) SnapshotDatasetAttr {
+	return func(m optionalAttr) {
+		m["reader_buffer_size"] = value
+	}
+}
+
+// SnapshotDatasetNumWriterThreads sets the optional num_writer_threads attribute to value.
+// If not specified, defaults to 1
+func SnapshotDatasetNumWriterThreads(value int64) SnapshotDatasetAttr {
+	return func(m optionalAttr) {
+		m["num_writer_threads"] = value
+	}
+}
+
+// SnapshotDatasetWriterBufferSize sets the optional writer_buffer_size attribute to value.
+// If not specified, defaults to 1
+func SnapshotDatasetWriterBufferSize(value int64) SnapshotDatasetAttr {
+	return func(m optionalAttr) {
+		m["writer_buffer_size"] = value
+	}
+}
+
+// SnapshotDatasetShuffleOnRead sets the optional shuffle_on_read attribute to value.
+// If not specified, defaults to false
+func SnapshotDatasetShuffleOnRead(value bool) SnapshotDatasetAttr {
+	return func(m optionalAttr) {
+		m["shuffle_on_read"] = value
+	}
+}
+
+// SnapshotDatasetSeed sets the optional seed attribute to value.
+// If not specified, defaults to 0
+func SnapshotDatasetSeed(value int64) SnapshotDatasetAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// SnapshotDatasetSeed2 sets the optional seed2 attribute to value.
+// If not specified, defaults to 0
+func SnapshotDatasetSeed2(value int64) SnapshotDatasetAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// SnapshotDatasetMode sets the optional mode attribute to value.
+// If not specified, defaults to "auto"
+func SnapshotDatasetMode(value string) SnapshotDatasetAttr {
+	return func(m optionalAttr) {
+		m["mode"] = value
+	}
+}
+
+// SnapshotDatasetSnapshotName sets the optional snapshot_name attribute to value.
+// If not specified, defaults to ""
+func SnapshotDatasetSnapshotName(value string) SnapshotDatasetAttr {
+	return func(m optionalAttr) {
+		m["snapshot_name"] = value
+	}
+}
+
+// Creates a dataset that will write to / read from a snapshot.
+//
+// This dataset attempts to determine whether a valid snapshot exists at the
+// `snapshot_path`, and reads from the snapshot in lieu of using `input_dataset`.
+// If not, it will run the preprocessing pipeline as usual, and write out a
+// snapshot of the data processed for future use.
+//
+// Arguments:
+//	input_dataset: A variant tensor representing the input dataset.
+//	path: The path we should write snapshots to / read snapshots from.
+//
+//
+func SnapshotDataset(scope *Scope, input_dataset tf.Output, path tf.Output, output_types []tf.DataType, output_shapes []tf.Shape, optional ...SnapshotDatasetAttr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SnapshotDataset",
+		Input: []tf.Input{
+			input_dataset, path,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ArgMaxAttr is an optional argument to ArgMax.
+type ArgMaxAttr func(optionalAttr)
+
+// ArgMaxOutputType sets the optional output_type attribute to value.
+// If not specified, defaults to DT_INT64
+func ArgMaxOutputType(value tf.DataType) ArgMaxAttr {
+	return func(m optionalAttr) {
+		m["output_type"] = value
+	}
+}
+
+// Returns the index with the largest value across dimensions of a tensor.
+//
+// Note that in case of ties the identity of the return value is not guaranteed.
+//
+// Usage:
+//   ```python
+//   import tensorflow as tf
+//   a = [1, 10, 26.9, 2.8, 166.32, 62.3]
+//   b = tf.math.argmax(input = a)
+//   c = tf.keras.backend.eval(b)
+//   # c = 4
+//   # here a[4] = 166.32 which is the largest element of a across axis 0
+//   ```
+//
+// Arguments:
+//
+//	dimension: int32 or int64, must be in the range `[-rank(input), rank(input))`.
+// Describes which dimension of the input Tensor to reduce across. For vectors,
+// use dimension = 0.
+func ArgMax(scope *Scope, input tf.Output, dimension tf.Output, optional ...ArgMaxAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ArgMax",
+		Input: []tf.Input{
+			input, dimension,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResizeBilinearGradAttr is an optional argument to ResizeBilinearGrad.
+type ResizeBilinearGradAttr func(optionalAttr)
+
+// ResizeBilinearGradAlignCorners sets the optional align_corners attribute to value.
+//
+// value: If true, the centers of the 4 corner pixels of the input and grad tensors are
+// aligned. Defaults to false.
+// If not specified, defaults to false
+func ResizeBilinearGradAlignCorners(value bool) ResizeBilinearGradAttr {
+	return func(m optionalAttr) {
+		m["align_corners"] = value
+	}
+}
+
+// ResizeBilinearGradHalfPixelCenters sets the optional half_pixel_centers attribute to value.
+// If not specified, defaults to false
+func ResizeBilinearGradHalfPixelCenters(value bool) ResizeBilinearGradAttr {
+	return func(m optionalAttr) {
+		m["half_pixel_centers"] = value
+	}
+}
+
+// Computes the gradient of bilinear interpolation.
+//
+// Arguments:
+//	grads: 4-D with shape `[batch, height, width, channels]`.
+//	original_image: 4-D with shape `[batch, orig_height, orig_width, channels]`,
+// The image tensor that was resized.
+//
+// Returns 4-D with shape `[batch, orig_height, orig_width, channels]`.
+// Gradients with respect to the input image. Input image must have been
+// float or double.
+func ResizeBilinearGrad(scope *Scope, grads tf.Output, original_image tf.Output, optional ...ResizeBilinearGradAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResizeBilinearGrad",
+		Input: []tf.Input{
+			grads, original_image,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MaxAttr is an optional argument to Max.
+type MaxAttr func(optionalAttr)
+
+// MaxKeepDims sets the optional keep_dims attribute to value.
+//
+// value: If true, retain reduced dimensions with length 1.
+// If not specified, defaults to false
+func MaxKeepDims(value bool) MaxAttr {
+	return func(m optionalAttr) {
+		m["keep_dims"] = value
+	}
+}
+
+// Computes the maximum of elements across dimensions of a tensor.
+//
+// Reduces `input` along the dimensions given in `axis`. Unless
+// `keep_dims` is true, the rank of the tensor is reduced by 1 for each entry in
+// `axis`. If `keep_dims` is true, the reduced dimensions are
+// retained with length 1.
+//
+// Arguments:
+//	input: The tensor to reduce.
+//	axis: The dimensions to reduce. Must be in the range
+// `[-rank(input), rank(input))`.
+//
+// Returns The reduced tensor.
+func Max(scope *Scope, input tf.Output, axis tf.Output, optional ...MaxAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Max",
+		Input: []tf.Input{
+			input, axis,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Get the value of the tensor specified by its handle.
+//
+// Arguments:
+//	handle: The handle for a tensor stored in the session state.
+//	dtype: The type of the output value.
+//
+// Returns The tensor for the given handle.
+func GetSessionTensor(scope *Scope, handle tf.Output, dtype tf.DataType) (value tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	opspec := tf.OpSpec{
+		Type: "GetSessionTensor",
+		Input: []tf.Input{
+			handle,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Outputs a tensor containing the reduction across all input tensors.
+//
+// Outputs a tensor containing the reduction across all input tensors passed to ops
+// within the same `shared_name.
+//
+// The graph should be constructed so if one op runs with shared_name value `c`,
+// then `num_devices` ops will run with shared_name value `c`.  Failure to do so
+// will cause the graph execution to fail to complete.
+//
+// input: the input to the reduction
+// data: the value of the reduction across all `num_devices` devices.
+// reduction: the reduction operation to perform.
+// num_devices: The number of devices participating in this reduction.
+// shared_name: Identifier that shared between ops of the same reduction.
+func NcclAllReduce(scope *Scope, input tf.Output, reduction string, num_devices int64, shared_name string) (data tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"reduction": reduction, "num_devices": num_devices, "shared_name": shared_name}
+	opspec := tf.OpSpec{
+		Type: "NcclAllReduce",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MinAttr is an optional argument to Min.
+type MinAttr func(optionalAttr)
+
+// MinKeepDims sets the optional keep_dims attribute to value.
+//
+// value: If true, retain reduced dimensions with length 1.
+// If not specified, defaults to false
+func MinKeepDims(value bool) MinAttr {
+	return func(m optionalAttr) {
+		m["keep_dims"] = value
+	}
+}
+
+// Computes the minimum of elements across dimensions of a tensor.
+//
+// Reduces `input` along the dimensions given in `axis`. Unless
+// `keep_dims` is true, the rank of the tensor is reduced by 1 for each entry in
+// `axis`. If `keep_dims` is true, the reduced dimensions are
+// retained with length 1.
+//
+// Arguments:
+//	input: The tensor to reduce.
+//	axis: The dimensions to reduce. Must be in the range
+// `[-rank(input), rank(input))`.
+//
+// Returns The reduced tensor.
+func Min(scope *Scope, input tf.Output, axis tf.Output, optional ...MinAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Min",
+		Input: []tf.Input{
+			input, axis,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SampleDistortedBoundingBoxV2Attr is an optional argument to SampleDistortedBoundingBoxV2.
+type SampleDistortedBoundingBoxV2Attr func(optionalAttr)
+
+// SampleDistortedBoundingBoxV2Seed sets the optional seed attribute to value.
+//
+// value: If either `seed` or `seed2` are set to non-zero, the random number
+// generator is seeded by the given `seed`.  Otherwise, it is seeded by a random
+// seed.
+// If not specified, defaults to 0
+func SampleDistortedBoundingBoxV2Seed(value int64) SampleDistortedBoundingBoxV2Attr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// SampleDistortedBoundingBoxV2Seed2 sets the optional seed2 attribute to value.
+//
+// value: A second seed to avoid seed collision.
+// If not specified, defaults to 0
+func SampleDistortedBoundingBoxV2Seed2(value int64) SampleDistortedBoundingBoxV2Attr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// SampleDistortedBoundingBoxV2AspectRatioRange sets the optional aspect_ratio_range attribute to value.
+//
+// value: The cropped area of the image must have an aspect ratio =
+// width / height within this range.
+// If not specified, defaults to <f:0.75 f:1.33 >
+func SampleDistortedBoundingBoxV2AspectRatioRange(value []float32) SampleDistortedBoundingBoxV2Attr {
+	return func(m optionalAttr) {
+		m["aspect_ratio_range"] = value
+	}
+}
+
+// SampleDistortedBoundingBoxV2AreaRange sets the optional area_range attribute to value.
+//
+// value: The cropped area of the image must contain a fraction of the
+// supplied image within this range.
+// If not specified, defaults to <f:0.05 f:1 >
+func SampleDistortedBoundingBoxV2AreaRange(value []float32) SampleDistortedBoundingBoxV2Attr {
+	return func(m optionalAttr) {
+		m["area_range"] = value
+	}
+}
+
+// SampleDistortedBoundingBoxV2MaxAttempts sets the optional max_attempts attribute to value.
+//
+// value: Number of attempts at generating a cropped region of the image
+// of the specified constraints. After `max_attempts` failures, return the entire
+// image.
+// If not specified, defaults to 100
+func SampleDistortedBoundingBoxV2MaxAttempts(value int64) SampleDistortedBoundingBoxV2Attr {
+	return func(m optionalAttr) {
+		m["max_attempts"] = value
+	}
+}
+
+// SampleDistortedBoundingBoxV2UseImageIfNoBoundingBoxes sets the optional use_image_if_no_bounding_boxes attribute to value.
+//
+// value: Controls behavior if no bounding boxes supplied.
+// If true, assume an implicit bounding box covering the whole input. If false,
+// raise an error.
+// If not specified, defaults to false
+func SampleDistortedBoundingBoxV2UseImageIfNoBoundingBoxes(value bool) SampleDistortedBoundingBoxV2Attr {
+	return func(m optionalAttr) {
+		m["use_image_if_no_bounding_boxes"] = value
+	}
+}
+
+// Generate a single randomly distorted bounding box for an image.
+//
+// Bounding box annotations are often supplied in addition to ground-truth labels
+// in image recognition or object localization tasks. A common technique for
+// training such a system is to randomly distort an image while preserving
+// its content, i.e. *data augmentation*. This Op outputs a randomly distorted
+// localization of an object, i.e. bounding box, given an `image_size`,
+// `bounding_boxes` and a series of constraints.
+//
+// The output of this Op is a single bounding box that may be used to crop the
+// original image. The output is returned as 3 tensors: `begin`, `size` and
+// `bboxes`. The first 2 tensors can be fed directly into `tf.slice` to crop the
+// image. The latter may be supplied to `tf.image.draw_bounding_boxes` to visualize
+// what the bounding box looks like.
+//
+// Bounding boxes are supplied and returned as `[y_min, x_min, y_max, x_max]`. The
+// bounding box coordinates are floats in `[0.0, 1.0]` relative to the width and
+// height of the underlying image.
+//
+// For example,
+//
+// ```python
+//     # Generate a single distorted bounding box.
+//     begin, size, bbox_for_draw = tf.image.sample_distorted_bounding_box(
+//         tf.shape(image),
+//         bounding_boxes=bounding_boxes)
+//
+//     # Draw the bounding box in an image summary.
+//     image_with_box = tf.image.draw_bounding_boxes(tf.expand_dims(image, 0),
+//                                                   bbox_for_draw)
+//     tf.summary.image('images_with_box', image_with_box)
+//
+//     # Employ the bounding box to distort the image.
+//     distorted_image = tf.slice(image, begin, size)
+// ```
+//
+// Note that if no bounding box information is available, setting
+// `use_image_if_no_bounding_boxes = true` will assume there is a single implicit
+// bounding box covering the whole image. If `use_image_if_no_bounding_boxes` is
+// false and no bounding boxes are supplied, an error is raised.
+//
+// Arguments:
+//	image_size: 1-D, containing `[height, width, channels]`.
+//	bounding_boxes: 3-D with shape `[batch, N, 4]` describing the N bounding boxes
+// associated with the image.
+//	min_object_covered: The cropped area of the image must contain at least this
+// fraction of any bounding box supplied. The value of this parameter should be
+// non-negative. In the case of 0, the cropped area does not need to overlap
+// any of the bounding boxes supplied.
+//
+// Returns:
+//	begin: 1-D, containing `[offset_height, offset_width, 0]`. Provide as input to
+// `tf.slice`.
+//	size: 1-D, containing `[target_height, target_width, -1]`. Provide as input to
+// `tf.slice`.
+//	bboxes: 3-D with shape `[1, 1, 4]` containing the distorted bounding box.
+// Provide as input to `tf.image.draw_bounding_boxes`.
+func SampleDistortedBoundingBoxV2(scope *Scope, image_size tf.Output, bounding_boxes tf.Output, min_object_covered tf.Output, optional ...SampleDistortedBoundingBoxV2Attr) (begin tf.Output, size tf.Output, bboxes tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SampleDistortedBoundingBoxV2",
+		Input: []tf.Input{
+			image_size, bounding_boxes, min_object_covered,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// EigAttr is an optional argument to Eig.
+type EigAttr func(optionalAttr)
+
+// EigComputeV sets the optional compute_v attribute to value.
+//
+// value: If `True` then eigenvectors will be computed and returned in `v`.
+// Otherwise, only the eigenvalues will be computed.
+// If not specified, defaults to true
+func EigComputeV(value bool) EigAttr {
+	return func(m optionalAttr) {
+		m["compute_v"] = value
+	}
+}
+
+// Computes the eigen decomposition of one or more square matrices.
+//
+// Computes the eigenvalues and (optionally) right eigenvectors of each inner matrix in
+// `input` such that `input[..., :, :] = v[..., :, :] * diag(e[..., :])`. The eigenvalues
+// are sorted in non-decreasing order.
+//
+// ```python
+// # a is a tensor.
+// # e is a tensor of eigenvalues.
+// # v is a tensor of eigenvectors.
+// e, v = eig(a)
+// e = eig(a, compute_v=False)
+// ```
+//
+// Arguments:
+//	input: `Tensor` input of shape `[N, N]`.
+//
+//
+// Returns:
+//	e: Eigenvalues. Shape is `[N]`.
+//	v: Eigenvectors. Shape is `[N, N]`.
+func Eig(scope *Scope, input tf.Output, Tout tf.DataType, optional ...EigAttr) (e tf.Output, v tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"Tout": Tout}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Eig",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// ProdAttr is an optional argument to Prod.
+type ProdAttr func(optionalAttr)
+
+// ProdKeepDims sets the optional keep_dims attribute to value.
+//
+// value: If true, retain reduced dimensions with length 1.
+// If not specified, defaults to false
+func ProdKeepDims(value bool) ProdAttr {
+	return func(m optionalAttr) {
+		m["keep_dims"] = value
+	}
+}
+
+// Computes the product of elements across dimensions of a tensor.
+//
+// Reduces `input` along the dimensions given in `axis`. Unless
+// `keep_dims` is true, the rank of the tensor is reduced by 1 for each entry in
+// `axis`. If `keep_dims` is true, the reduced dimensions are
+// retained with length 1.
+//
+// Arguments:
+//	input: The tensor to reduce.
+//	axis: The dimensions to reduce. Must be in the range
+// `[-rank(input), rank(input))`.
+//
+// Returns The reduced tensor.
+func Prod(scope *Scope, input tf.Output, axis tf.Output, optional ...ProdAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Prod",
+		Input: []tf.Input{
+			input, axis,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SumAttr is an optional argument to Sum.
+type SumAttr func(optionalAttr)
+
+// SumKeepDims sets the optional keep_dims attribute to value.
+//
+// value: If true, retain reduced dimensions with length 1.
+// If not specified, defaults to false
+func SumKeepDims(value bool) SumAttr {
+	return func(m optionalAttr) {
+		m["keep_dims"] = value
+	}
+}
+
+// Computes the sum of elements across dimensions of a tensor.
+//
+// Reduces `input` along the dimensions given in `axis`. Unless
+// `keep_dims` is true, the rank of the tensor is reduced by 1 for each entry in
+// `axis`. If `keep_dims` is true, the reduced dimensions are
+// retained with length 1.
+//
+// Arguments:
+//	input: The tensor to reduce.
+//	axis: The dimensions to reduce. Must be in the range
+// `[-rank(input), rank(input))`.
+//
+// Returns The reduced tensor.
+func Sum(scope *Scope, input tf.Output, axis tf.Output, optional ...SumAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Sum",
+		Input: []tf.Input{
+			input, axis,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// BoostedTreesQuantileStreamResourceFlushAttr is an optional argument to BoostedTreesQuantileStreamResourceFlush.
+type BoostedTreesQuantileStreamResourceFlushAttr func(optionalAttr)
+
+// BoostedTreesQuantileStreamResourceFlushGenerateQuantiles sets the optional generate_quantiles attribute to value.
+//
+// value: bool; If True, the output will be the num_quantiles for each stream where the ith
+// entry is the ith quantile of the input with an approximation error of epsilon.
+// Duplicate values may be present.
+// If False, the output will be the points in the histogram that we got which roughly
+// translates to 1/epsilon boundaries and without any duplicates.
+// Default to False.
+// If not specified, defaults to false
+func BoostedTreesQuantileStreamResourceFlushGenerateQuantiles(value bool) BoostedTreesQuantileStreamResourceFlushAttr {
+	return func(m optionalAttr) {
+		m["generate_quantiles"] = value
+	}
+}
+
+// Flush the summaries for a quantile stream resource.
+//
+// An op that flushes the summaries for a quantile stream resource.
+//
+// Arguments:
+//	quantile_stream_resource_handle: resource handle referring to a QuantileStreamResource.
+//	num_buckets: int; approximate number of buckets unless using generate_quantiles.
+//
+// Returns the created operation.
+func BoostedTreesQuantileStreamResourceFlush(scope *Scope, quantile_stream_resource_handle tf.Output, num_buckets tf.Output, optional ...BoostedTreesQuantileStreamResourceFlushAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesQuantileStreamResourceFlush",
+		Input: []tf.Input{
+			quantile_stream_resource_handle, num_buckets,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// WholeFileReaderV2Attr is an optional argument to WholeFileReaderV2.
+type WholeFileReaderV2Attr func(optionalAttr)
+
+// WholeFileReaderV2Container sets the optional container attribute to value.
+//
+// value: If non-empty, this reader is placed in the given container.
+// Otherwise, a default container is used.
+// If not specified, defaults to ""
+func WholeFileReaderV2Container(value string) WholeFileReaderV2Attr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// WholeFileReaderV2SharedName sets the optional shared_name attribute to value.
+//
+// value: If non-empty, this reader is named in the given bucket
+// with this shared_name. Otherwise, the node name is used instead.
+// If not specified, defaults to ""
+func WholeFileReaderV2SharedName(value string) WholeFileReaderV2Attr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// A Reader that outputs the entire contents of a file as a value.
+//
+// To use, enqueue filenames in a Queue.  The output of ReaderRead will
+// be a filename (key) and the contents of that file (value).
+//
+// Returns The handle to reference the Reader.
+func WholeFileReaderV2(scope *Scope, optional ...WholeFileReaderV2Attr) (reader_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "WholeFileReaderV2",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ShapeNAttr is an optional argument to ShapeN.
+type ShapeNAttr func(optionalAttr)
+
+// ShapeNOutType sets the optional out_type attribute to value.
+// If not specified, defaults to DT_INT32
+func ShapeNOutType(value tf.DataType) ShapeNAttr {
+	return func(m optionalAttr) {
+		m["out_type"] = value
+	}
+}
+
+// Returns shape of tensors.
+//
+// This operation returns N 1-D integer tensors representing shape of `input[i]s`.
+func ShapeN(scope *Scope, input []tf.Output, optional ...ShapeNAttr) (output []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ShapeN",
+		Input: []tf.Input{
+			tf.OutputList(input),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if output, idx, err = makeOutputList(op, idx, "output"); err != nil {
+		scope.UpdateErr("ShapeN", err)
+		return
+	}
+	return output
+}
+
+// ImageSummaryAttr is an optional argument to ImageSummary.
+type ImageSummaryAttr func(optionalAttr)
+
+// ImageSummaryMaxImages sets the optional max_images attribute to value.
+//
+// value: Max number of batch elements to generate images for.
+// If not specified, defaults to 3
+//
+// REQUIRES: value >= 1
+func ImageSummaryMaxImages(value int64) ImageSummaryAttr {
+	return func(m optionalAttr) {
+		m["max_images"] = value
+	}
+}
+
+// ImageSummaryBadColor sets the optional bad_color attribute to value.
+//
+// value: Color to use for pixels with non-finite values.
+// If not specified, defaults to <dtype:DT_UINT8 tensor_shape:<dim:<size:4 > > int_val:255 int_val:0 int_val:0 int_val:255 >
+func ImageSummaryBadColor(value tf.Tensor) ImageSummaryAttr {
+	return func(m optionalAttr) {
+		m["bad_color"] = value
+	}
+}
+
+// Outputs a `Summary` protocol buffer with images.
+//
+// The summary has up to `max_images` summary values containing images. The
+// images are built from `tensor` which must be 4-D with shape `[batch_size,
+// height, width, channels]` and where `channels` can be:
+//
+// *  1: `tensor` is interpreted as Grayscale.
+// *  3: `tensor` is interpreted as RGB.
+// *  4: `tensor` is interpreted as RGBA.
+//
+// The images have the same number of channels as the input tensor. For float
+// input, the values are normalized one image at a time to fit in the range
+// `[0, 255]`.  `uint8` values are unchanged.  The op uses two different
+// normalization algorithms:
+//
+// *  If the input values are all positive, they are rescaled so the largest one
+//    is 255.
+//
+// *  If any input value is negative, the values are shifted so input value 0.0
+//    is at 127.  They are then rescaled so that either the smallest value is 0,
+//    or the largest one is 255.
+//
+// The `tag` argument is a scalar `Tensor` of type `string`.  It is used to
+// build the `tag` of the summary values:
+//
+// *  If `max_images` is 1, the summary value tag is '*tag*/image'.
+// *  If `max_images` is greater than 1, the summary value tags are
+//    generated sequentially as '*tag*/image/0', '*tag*/image/1', etc.
+//
+// The `bad_color` argument is the color to use in the generated images for
+// non-finite input values.  It is a `uint8` 1-D tensor of length `channels`.
+// Each element must be in the range `[0, 255]` (It represents the value of a
+// pixel in the output image).  Non-finite values in the input tensor are
+// replaced by this tensor in the output image.  The default value is the color
+// red.
+//
+// Arguments:
+//	tag: Scalar. Used to build the `tag` attribute of the summary values.
+//	tensor: 4-D of shape `[batch_size, height, width, channels]` where
+// `channels` is 1, 3, or 4.
+//
+// Returns Scalar. Serialized `Summary` protocol buffer.
+func ImageSummary(scope *Scope, tag tf.Output, tensor tf.Output, optional ...ImageSummaryAttr) (summary tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ImageSummary",
+		Input: []tf.Input{
+			tag, tensor,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// CollectiveBcastSendAttr is an optional argument to CollectiveBcastSend.
+type CollectiveBcastSendAttr func(optionalAttr)
+
+// CollectiveBcastSendCommunicationHint sets the optional communication_hint attribute to value.
+// If not specified, defaults to "auto"
+func CollectiveBcastSendCommunicationHint(value string) CollectiveBcastSendAttr {
+	return func(m optionalAttr) {
+		m["communication_hint"] = value
+	}
+}
+
+// CollectiveBcastSendTimeoutSeconds sets the optional timeout_seconds attribute to value.
+// If not specified, defaults to 0
+func CollectiveBcastSendTimeoutSeconds(value float32) CollectiveBcastSendAttr {
+	return func(m optionalAttr) {
+		m["timeout_seconds"] = value
+	}
+}
+
+// Broadcasts a tensor value to one or more other devices.
+func CollectiveBcastSend(scope *Scope, input tf.Output, group_size int64, group_key int64, instance_key int64, shape tf.Shape, optional ...CollectiveBcastSendAttr) (data tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"group_size": group_size, "group_key": group_key, "instance_key": instance_key, "shape": shape}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CollectiveBcastSend",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// CombinedNonMaxSuppressionAttr is an optional argument to CombinedNonMaxSuppression.
+type CombinedNonMaxSuppressionAttr func(optionalAttr)
+
+// CombinedNonMaxSuppressionPadPerClass sets the optional pad_per_class attribute to value.
+//
+// value: If false, the output nmsed boxes, scores and classes
+// are padded/clipped to `max_total_size`. If true, the
+// output nmsed boxes, scores and classes are padded to be of length
+// `max_size_per_class`*`num_classes`, unless it exceeds `max_total_size` in
+// which case it is clipped to `max_total_size`. Defaults to false.
+// If not specified, defaults to false
+func CombinedNonMaxSuppressionPadPerClass(value bool) CombinedNonMaxSuppressionAttr {
+	return func(m optionalAttr) {
+		m["pad_per_class"] = value
+	}
+}
+
+// CombinedNonMaxSuppressionClipBoxes sets the optional clip_boxes attribute to value.
+//
+// value: If true, assume the box coordinates are between [0, 1] and clip the output boxes
+// if they fall beyond [0, 1]. If false, do not do clipping and output the box
+// coordinates as it is.
+// If not specified, defaults to true
+func CombinedNonMaxSuppressionClipBoxes(value bool) CombinedNonMaxSuppressionAttr {
+	return func(m optionalAttr) {
+		m["clip_boxes"] = value
+	}
+}
+
+// Greedily selects a subset of bounding boxes in descending order of score,
+//
+// This operation performs non_max_suppression on the inputs per batch, across
+// all classes.
+// Prunes away boxes that have high intersection-over-union (IOU) overlap
+// with previously selected boxes.  Bounding boxes are supplied as
+// [y1, x1, y2, x2], where (y1, x1) and (y2, x2) are the coordinates of any
+// diagonal pair of box corners and the coordinates can be provided as normalized
+// (i.e., lying in the interval [0, 1]) or absolute.  Note that this algorithm
+// is agnostic to where the origin is in the coordinate system. Also note that
+// this algorithm is invariant to orthogonal transformations and translations
+// of the coordinate system; thus translating or reflections of the coordinate
+// system result in the same boxes being selected by the algorithm.
+// The output of this operation is the final boxes, scores and classes tensor
+// returned after performing non_max_suppression.
+//
+// Arguments:
+//	boxes: A 4-D float tensor of shape `[batch_size, num_boxes, q, 4]`. If `q` is 1 then
+// same boxes are used for all classes otherwise, if `q` is equal to number of
+// classes, class-specific boxes are used.
+//	scores: A 3-D float tensor of shape `[batch_size, num_boxes, num_classes]`
+// representing a single score corresponding to each box (each row of boxes).
+//	max_output_size_per_class: A scalar integer tensor representing the maximum number of
+// boxes to be selected by non max suppression per class
+//	max_total_size: A scalar representing maximum number of boxes retained over all classes.
+//	iou_threshold: A 0-D float tensor representing the threshold for deciding whether
+// boxes overlap too much with respect to IOU.
+//	score_threshold: A 0-D float tensor representing the threshold for deciding when to remove
+// boxes based on score.
+//
+// Returns:
+//	nmsed_boxes: A [batch_size, max_detections, 4] float32 tensor
+// containing the non-max suppressed boxes.
+//	nmsed_scores: A [batch_size, max_detections] float32 tensor
+// containing the scores for the boxes.
+//	nmsed_classes: A [batch_size, max_detections] float32 tensor
+// containing the classes for the boxes.
+//	valid_detections: A [batch_size] int32 tensor indicating the number of
+// valid detections per batch item. Only the top num_detections[i] entries in
+// nms_boxes[i], nms_scores[i] and nms_class[i] are valid. The rest of the
+// entries are zero paddings.
+func CombinedNonMaxSuppression(scope *Scope, boxes tf.Output, scores tf.Output, max_output_size_per_class tf.Output, max_total_size tf.Output, iou_threshold tf.Output, score_threshold tf.Output, optional ...CombinedNonMaxSuppressionAttr) (nmsed_boxes tf.Output, nmsed_scores tf.Output, nmsed_classes tf.Output, valid_detections tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CombinedNonMaxSuppression",
+		Input: []tf.Input{
+			boxes, scores, max_output_size_per_class, max_total_size, iou_threshold, score_threshold,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3)
+}
+
+// Returns the truth value of x AND y element-wise.
+//
+// *NOTE*: `LogicalAnd` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func LogicalAnd(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "LogicalAnd",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ApproximateEqualAttr is an optional argument to ApproximateEqual.
+type ApproximateEqualAttr func(optionalAttr)
+
+// ApproximateEqualTolerance sets the optional tolerance attribute to value.
+// If not specified, defaults to 1e-05
+func ApproximateEqualTolerance(value float32) ApproximateEqualAttr {
+	return func(m optionalAttr) {
+		m["tolerance"] = value
+	}
+}
+
+// Returns the truth value of abs(x-y) < tolerance element-wise.
+func ApproximateEqual(scope *Scope, x tf.Output, y tf.Output, optional ...ApproximateEqualAttr) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ApproximateEqual",
+		Input: []tf.Input{
+			x, y,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// LowerBoundAttr is an optional argument to LowerBound.
+type LowerBoundAttr func(optionalAttr)
+
+// LowerBoundOutType sets the optional out_type attribute to value.
+// If not specified, defaults to DT_INT32
+func LowerBoundOutType(value tf.DataType) LowerBoundAttr {
+	return func(m optionalAttr) {
+		m["out_type"] = value
+	}
+}
+
+// Applies lower_bound(sorted_search_values, values) along each row.
+//
+// Each set of rows with the same index in (sorted_inputs, values) is treated
+// independently.  The resulting row is the equivalent of calling
+// `np.searchsorted(sorted_inputs, values, side='left')`.
+//
+// The result is not a global index to the entire
+// `Tensor`, but rather just the index in the last dimension.
+//
+// A 2-D example:
+//   sorted_sequence = [[0, 3, 9, 9, 10],
+//                      [1, 2, 3, 4, 5]]
+//   values = [[2, 4, 9],
+//             [0, 2, 6]]
+//
+//   result = LowerBound(sorted_sequence, values)
+//
+//   result == [[1, 2, 2],
+//              [0, 1, 5]]
+//
+// Arguments:
+//	sorted_inputs: 2-D Tensor where each row is ordered.
+//	values: 2-D Tensor with the same numbers of rows as `sorted_search_values`. Contains
+// the values that will be searched for in `sorted_search_values`.
+//
+// Returns A `Tensor` with the same shape as `values`.  It contains the first scalar index
+// into the last dimension where values can be inserted without changing the
+// ordered property.
+func LowerBound(scope *Scope, sorted_inputs tf.Output, values tf.Output, optional ...LowerBoundAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LowerBound",
+		Input: []tf.Input{
+			sorted_inputs, values,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns the truth value of (x > y) element-wise.
+//
+// *NOTE*: `Greater` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+//
+// Example:
+//
+// ```python
+// x = tf.constant([5, 4, 6])
+// y = tf.constant([5, 2, 5])
+// tf.math.greater(x, y) ==> [False, True, True]
+//
+// x = tf.constant([5, 4, 6])
+// y = tf.constant([5])
+// tf.math.greater(x, y) ==> [False, False, True]
+// ```
+func Greater(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Greater",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Compute the polygamma function \\(\psi^{(n)}(x)\\).
+//
+// The polygamma function is defined as:
+//
+//
+// \\(\psi^{(a)}(x) = \frac{d^a}{dx^a} \psi(x)\\)
+//
+// where \\(\psi(x)\\) is the digamma function.
+// The polygamma function is defined only for non-negative integer orders \\a\\.
+func Polygamma(scope *Scope, a tf.Output, x tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Polygamma",
+		Input: []tf.Input{
+			a, x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Shuffle dimensions of x according to a permutation.
+//
+// The output `y` has the same rank as `x`. The shapes of `x` and `y` satisfy:
+//   `y.shape[i] == x.shape[perm[i]] for i in [0, 1, ..., rank(x) - 1]`
+func Transpose(scope *Scope, x tf.Output, perm tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Transpose",
+		Input: []tf.Input{
+			x, perm,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// AssertAttr is an optional argument to Assert.
+type AssertAttr func(optionalAttr)
+
+// AssertSummarize sets the optional summarize attribute to value.
+//
+// value: Print this many entries of each tensor.
+// If not specified, defaults to 3
+func AssertSummarize(value int64) AssertAttr {
+	return func(m optionalAttr) {
+		m["summarize"] = value
+	}
+}
+
+// Asserts that the given condition is true.
+//
+// If `condition` evaluates to false, print the list of tensors in `data`.
+// `summarize` determines how many entries of the tensors to print.
+//
+// Arguments:
+//	condition: The condition to evaluate.
+//	data: The tensors to print out when condition is false.
+//
+// Returns the created operation.
+func Assert(scope *Scope, condition tf.Output, data []tf.Output, optional ...AssertAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Assert",
+		Input: []tf.Input{
+			condition, tf.OutputList(data),
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Computes the gradient of `igamma(a, x)` wrt `a`.
+func IgammaGradA(scope *Scope, a tf.Output, x tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "IgammaGradA",
+		Input: []tf.Input{
+			a, x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Compute the upper regularized incomplete Gamma function `Q(a, x)`.
+//
+// The upper regularized incomplete Gamma function is defined as:
+//
+// \\(Q(a, x) = Gamma(a, x) / Gamma(a) = 1 - P(a, x)\\)
+//
+// where
+//
+// \\(Gamma(a, x) = int_{x}^{\infty} t^{a-1} exp(-t) dt\\)
+//
+// is the upper incomplete Gama function.
+//
+// Note, above `P(a, x)` (`Igamma`) is the lower regularized complete
+// Gamma function.
+func Igammac(scope *Scope, a tf.Output, x tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Igammac",
+		Input: []tf.Input{
+			a, x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns element-wise remainder of division. This emulates C semantics in that
+//
+// the result here is consistent with a truncating divide. E.g. `truncate(x / y) *
+// y + truncate_mod(x, y) = x`.
+//
+// *NOTE*: `TruncateMod` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func TruncateMod(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TruncateMod",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns element-wise remainder of division. This emulates C semantics in that
+//
+// the result here is consistent with a truncating divide. E.g.
+// `tf.truncatediv(x, y) * y + truncate_mod(x, y) = x`.
+//
+// *NOTE*: `Mod` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func Mod(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Mod",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// A substitute for `InterleaveDataset` on a fixed list of `N` datasets.
+//
+// Arguments:
+//	selector_input_dataset: A dataset of scalar `DT_INT64` elements that determines which of the
+// `N` data inputs should produce the next output element.
+//	data_input_datasets: `N` datasets with the same type that will be interleaved according to
+// the values of `selector_input_dataset`.
+//
+//
+func ExperimentalDirectedInterleaveDataset(scope *Scope, selector_input_dataset tf.Output, data_input_datasets []tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalDirectedInterleaveDataset",
+		Input: []tf.Input{
+			selector_input_dataset, tf.OutputList(data_input_datasets),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns the min of x and y (i.e. x < y ? x : y) element-wise.
+//
+// *NOTE*: `Minimum` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func Minimum(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Minimum",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns the max of x and y (i.e. x > y ? x : y) element-wise.
+//
+// *NOTE*: `Maximum` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func Maximum(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Maximum",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns 0 if x == 0, and x * log(y) otherwise, elementwise.
+func Xlogy(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Xlogy",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Increments variable pointed to by 'resource' until it reaches 'limit'.
+//
+// Arguments:
+//	resource: Should be from a scalar `Variable` node.
+//	limit: If incrementing ref would bring it above limit, instead generates an
+// 'OutOfRange' error.
+//
+//
+// Returns A copy of the input before increment. If nothing else modifies the
+// input, the values produced will all be distinct.
+func ResourceCountUpTo(scope *Scope, resource tf.Output, limit int64, T tf.DataType) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"limit": limit, "T": T}
+	opspec := tf.OpSpec{
+		Type: "ResourceCountUpTo",
+		Input: []tf.Input{
+			resource,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// StatefulStandardNormalAttr is an optional argument to StatefulStandardNormal.
+type StatefulStandardNormalAttr func(optionalAttr)
+
+// StatefulStandardNormalDtype sets the optional dtype attribute to value.
+//
+// value: The type of the output.
+// If not specified, defaults to DT_FLOAT
+func StatefulStandardNormalDtype(value tf.DataType) StatefulStandardNormalAttr {
+	return func(m optionalAttr) {
+		m["dtype"] = value
+	}
+}
+
+// Outputs random values from a normal distribution. This op is deprecated in favor of op 'StatefulStandardNormalV2'
+//
+// DEPRECATED at GraphDef version 29: Use StatefulStandardNormalV2 instead
+//
+// The generated values will have mean 0 and standard deviation 1.
+//
+// Arguments:
+//	resource: The handle of the resource variable that stores the state of the RNG.
+//	shape: The shape of the output tensor.
+//
+// Returns A tensor of the specified shape filled with random normal values.
+func StatefulStandardNormal(scope *Scope, resource tf.Output, shape tf.Output, optional ...StatefulStandardNormalAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StatefulStandardNormal",
+		Input: []tf.Input{
+			resource, shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns x / y element-wise for real types.
+//
+// If `x` and `y` are reals, this will return the floating-point division.
+//
+// *NOTE*: `Div` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func RealDiv(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "RealDiv",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns x / y element-wise for integer types.
+//
+// Truncation designates that negative numbers will round fractional quantities
+// toward zero. I.e. -7 / 5 = -1. This matches C semantics but it is different
+// than Python semantics. See `FloorDiv` for a division function that matches
+// Python Semantics.
+//
+// *NOTE*: `TruncateDiv` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func TruncateDiv(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TruncateDiv",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns 0 if the denominator is zero.
+//
+//
+// *NOTE*: `DivNoNan` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func DivNoNan(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "DivNoNan",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Scatter `updates` into an existing tensor according to `indices`.
+//
+// This operation creates a new tensor by applying sparse `updates` to the passed
+// in `tensor`.
+// This operation is very similar to `tf.scatter_nd`, except that the updates are
+// scattered onto an existing tensor (as opposed to a zero-tensor). If the memory
+// for the existing tensor cannot be re-used, a copy is made and updated.
+//
+// If `indices` contains duplicates, then their updates are accumulated (summed).
+//
+// **WARNING**: The order in which updates are applied is nondeterministic, so the
+// output will be nondeterministic if `indices` contains duplicates -- because
+// of some numerical approximation issues, numbers summed in different order
+// may yield different results.
+//
+// `indices` is an integer tensor containing indices into a new tensor of shape
+// `shape`.  The last dimension of `indices` can be at most the rank of `shape`:
+//
+//     indices.shape[-1] <= shape.rank
+//
+// The last dimension of `indices` corresponds to indices into elements
+// (if `indices.shape[-1] = shape.rank`) or slices
+// (if `indices.shape[-1] < shape.rank`) along dimension `indices.shape[-1]` of
+// `shape`.  `updates` is a tensor with shape
+//
+//     indices.shape[:-1] + shape[indices.shape[-1]:]
+//
+// The simplest form of scatter is to insert individual elements in a tensor by
+// index. For example, say we want to insert 4 scattered elements in a rank-1
+// tensor with 8 elements.
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src="https://www.tensorflow.org/images/ScatterNd1.png" alt>
+// </div>
+//
+// In Python, this scatter operation would look like this:
+//
+//     >>> indices = tf.constant([[4], [3], [1], [7]])
+//     >>> updates = tf.constant([9, 10, 11, 12])
+//     >>> tensor = tf.ones([8], dtype=tf.int32)
+//     >>> print(tf.tensor_scatter_nd_update(tensor, indices, updates))
+//     tf.Tensor([ 1 11  1 10  9  1  1 12], shape=(8,), dtype=int32)
+//
+// We can also, insert entire slices of a higher rank tensor all at once. For
+// example, if we wanted to insert two slices in the first dimension of a
+// rank-3 tensor with two matrices of new values.
+//
+// In Python, this scatter operation would look like this:
+//
+//     >>> indices = tf.constant([[0], [2]])
+//     >>> updates = tf.constant([[[5, 5, 5, 5], [6, 6, 6, 6],
+//     ...                         [7, 7, 7, 7], [8, 8, 8, 8]],
+//     ...                        [[5, 5, 5, 5], [6, 6, 6, 6],
+//     ...                         [7, 7, 7, 7], [8, 8, 8, 8]]])
+//     >>> tensor = tf.ones([4, 4, 4], dtype=tf.int32)
+//     >>> print(tf.tensor_scatter_nd_update(tensor, indices, updates).numpy())
+//     [[[5 5 5 5]
+//       [6 6 6 6]
+//       [7 7 7 7]
+//       [8 8 8 8]]
+//      [[1 1 1 1]
+//       [1 1 1 1]
+//       [1 1 1 1]
+//       [1 1 1 1]]
+//      [[5 5 5 5]
+//       [6 6 6 6]
+//       [7 7 7 7]
+//       [8 8 8 8]]
+//      [[1 1 1 1]
+//       [1 1 1 1]
+//       [1 1 1 1]
+//       [1 1 1 1]]]
+//
+// Note that on CPU, if an out of bound index is found, an error is returned.
+// On GPU, if an out of bound index is found, the index is ignored.
+//
+// Arguments:
+//	tensor: Tensor to copy/update.
+//	indices: Index tensor.
+//	updates: Updates to scatter into output.
+//
+// Returns A new tensor with the given shape and updates applied according
+// to the indices.
+func TensorScatterUpdate(scope *Scope, tensor tf.Output, indices tf.Output, updates tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorScatterUpdate",
+		Input: []tf.Input{
+			tensor, indices, updates,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that contains `count` elements from the `input_dataset`.
+//
+// Arguments:
+//
+//	count: A scalar representing the number of elements from the `input_dataset`
+// that should be taken. A value of `-1` indicates that all of `input_dataset`
+// is taken.
+//
+//
+func TakeDataset(scope *Scope, input_dataset tf.Output, count tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "TakeDataset",
+		Input: []tf.Input{
+			input_dataset, count,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns the last element of the input list as well as a list with all but that element.
+//
+// Fails if the list is empty.
+//
+// input_handle: the input list
+// tensor: the withdrawn last element of the list
+// element_dtype: the type of elements in the list
+// element_shape: the shape of the output tensor
+func TensorListPopBack(scope *Scope, input_handle tf.Output, element_shape tf.Output, element_dtype tf.DataType) (output_handle tf.Output, tensor tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"element_dtype": element_dtype}
+	opspec := tf.OpSpec{
+		Type: "TensorListPopBack",
+		Input: []tf.Input{
+			input_handle, element_shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// QueueDequeueManyV2Attr is an optional argument to QueueDequeueManyV2.
+type QueueDequeueManyV2Attr func(optionalAttr)
+
+// QueueDequeueManyV2TimeoutMs sets the optional timeout_ms attribute to value.
+//
+// value: If the queue has fewer than n elements, this operation
+// will block for up to timeout_ms milliseconds.
+// Note: This option is not supported yet.
+// If not specified, defaults to -1
+func QueueDequeueManyV2TimeoutMs(value int64) QueueDequeueManyV2Attr {
+	return func(m optionalAttr) {
+		m["timeout_ms"] = value
+	}
+}
+
+// Dequeues `n` tuples of one or more tensors from the given queue.
+//
+// If the queue is closed and there are fewer than `n` elements, then an
+// OutOfRange error is returned.
+//
+// This operation concatenates queue-element component tensors along the
+// 0th dimension to make a single component tensor.  All of the components
+// in the dequeued tuple will have size `n` in the 0th dimension.
+//
+// This operation has `k` outputs, where `k` is the number of components in
+// the tuples stored in the given queue, and output `i` is the ith
+// component of the dequeued tuple.
+//
+// N.B. If the queue is empty, this operation will block until `n` elements
+// have been dequeued (or 'timeout_ms' elapses, if specified).
+//
+// Arguments:
+//	handle: The handle to a queue.
+//	n: The number of tuples to dequeue.
+//	component_types: The type of each component in a tuple.
+//
+// Returns One or more tensors that were dequeued as a tuple.
+func QueueDequeueManyV2(scope *Scope, handle tf.Output, n tf.Output, component_types []tf.DataType, optional ...QueueDequeueManyV2Attr) (components []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"component_types": component_types}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QueueDequeueManyV2",
+		Input: []tf.Input{
+			handle, n,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if components, idx, err = makeOutputList(op, idx, "components"); err != nil {
+		scope.UpdateErr("QueueDequeueManyV2", err)
+		return
+	}
+	return components
+}
+
+// Returns x * y element-wise. Returns zero if y is zero, even if x if infinite or NaN.
+//
+// *NOTE*: `MulNoNan` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func MulNoNan(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "MulNoNan",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// AsStringAttr is an optional argument to AsString.
+type AsStringAttr func(optionalAttr)
+
+// AsStringPrecision sets the optional precision attribute to value.
+//
+// value: The post-decimal precision to use for floating point numbers.
+// Only used if precision > -1.
+// If not specified, defaults to -1
+func AsStringPrecision(value int64) AsStringAttr {
+	return func(m optionalAttr) {
+		m["precision"] = value
+	}
+}
+
+// AsStringScientific sets the optional scientific attribute to value.
+//
+// value: Use scientific notation for floating point numbers.
+// If not specified, defaults to false
+func AsStringScientific(value bool) AsStringAttr {
+	return func(m optionalAttr) {
+		m["scientific"] = value
+	}
+}
+
+// AsStringShortest sets the optional shortest attribute to value.
+//
+// value: Use shortest representation (either scientific or standard) for
+// floating point numbers.
+// If not specified, defaults to false
+func AsStringShortest(value bool) AsStringAttr {
+	return func(m optionalAttr) {
+		m["shortest"] = value
+	}
+}
+
+// AsStringWidth sets the optional width attribute to value.
+//
+// value: Pad pre-decimal numbers to this width.
+// Applies to both floating point and integer numbers.
+// Only used if width > -1.
+// If not specified, defaults to -1
+func AsStringWidth(value int64) AsStringAttr {
+	return func(m optionalAttr) {
+		m["width"] = value
+	}
+}
+
+// AsStringFill sets the optional fill attribute to value.
+//
+// value: The value to pad if width > -1.  If empty, pads with spaces.
+// Another typical value is '0'.  String cannot be longer than 1 character.
+// If not specified, defaults to ""
+func AsStringFill(value string) AsStringAttr {
+	return func(m optionalAttr) {
+		m["fill"] = value
+	}
+}
+
+// Converts each entry in the given tensor to strings.
+//
+// Supports many numeric types and boolean.
+//
+// For Unicode, see the
+// [https://www.tensorflow.org/tutorials/representation/unicode](Working with Unicode text)
+// tutorial.
+//
+// Examples:
+//
+// >>> tf.strings.as_string([3, 2])
+// <tf.Tensor: shape=(2,), dtype=string, numpy=array([b'3', b'2'], dtype=object)>
+// >>> tf.strings.as_string([3.1415926, 2.71828], precision=2).numpy()
+// array([b'3.14', b'2.72'], dtype=object)
+func AsString(scope *Scope, input tf.Output, optional ...AsStringAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "AsString",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Conv3DBackpropFilterV2Attr is an optional argument to Conv3DBackpropFilterV2.
+type Conv3DBackpropFilterV2Attr func(optionalAttr)
+
+// Conv3DBackpropFilterV2DataFormat sets the optional data_format attribute to value.
+//
+// value: The data format of the input and output data. With the
+// default format "NDHWC", the data is stored in the order of:
+//     [batch, in_depth, in_height, in_width, in_channels].
+// Alternatively, the format could be "NCDHW", the data storage order is:
+//     [batch, in_channels, in_depth, in_height, in_width].
+// If not specified, defaults to "NDHWC"
+func Conv3DBackpropFilterV2DataFormat(value string) Conv3DBackpropFilterV2Attr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// Conv3DBackpropFilterV2Dilations sets the optional dilations attribute to value.
+//
+// value: 1-D tensor of length 5.  The dilation factor for each dimension of
+// `input`. If set to k > 1, there will be k-1 skipped cells between each
+// filter element on that dimension. The dimension order is determined by the
+// value of `data_format`, see above for details. Dilations in the batch and
+// depth dimensions must be 1.
+// If not specified, defaults to <i:1 i:1 i:1 i:1 i:1 >
+func Conv3DBackpropFilterV2Dilations(value []int64) Conv3DBackpropFilterV2Attr {
+	return func(m optionalAttr) {
+		m["dilations"] = value
+	}
+}
+
+// Computes the gradients of 3-D convolution with respect to the filter.
+//
+// Arguments:
+//	input: Shape `[batch, depth, rows, cols, in_channels]`.
+//	filter_sizes: An integer vector representing the tensor shape of `filter`,
+// where `filter` is a 5-D
+// `[filter_depth, filter_height, filter_width, in_channels, out_channels]`
+// tensor.
+//	out_backprop: Backprop signal of shape `[batch, out_depth, out_rows, out_cols,
+// out_channels]`.
+//	strides: 1-D tensor of length 5. The stride of the sliding window for each
+// dimension of `input`. Must have `strides[0] = strides[4] = 1`.
+//	padding: The type of padding algorithm to use.
+func Conv3DBackpropFilterV2(scope *Scope, input tf.Output, filter_sizes tf.Output, out_backprop tf.Output, strides []int64, padding string, optional ...Conv3DBackpropFilterV2Attr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Conv3DBackpropFilterV2",
+		Input: []tf.Input{
+			input, filter_sizes, out_backprop,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns x + y element-wise.
+//
+// *NOTE*: `Add` supports broadcasting. `AddN` does not. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func AddV2(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "AddV2",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// UniformCandidateSamplerAttr is an optional argument to UniformCandidateSampler.
+type UniformCandidateSamplerAttr func(optionalAttr)
+
+// UniformCandidateSamplerSeed sets the optional seed attribute to value.
+//
+// value: If either seed or seed2 are set to be non-zero, the random number
+// generator is seeded by the given seed.  Otherwise, it is seeded by a
+// random seed.
+// If not specified, defaults to 0
+func UniformCandidateSamplerSeed(value int64) UniformCandidateSamplerAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// UniformCandidateSamplerSeed2 sets the optional seed2 attribute to value.
+//
+// value: An second seed to avoid seed collision.
+// If not specified, defaults to 0
+func UniformCandidateSamplerSeed2(value int64) UniformCandidateSamplerAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Generates labels for candidate sampling with a uniform distribution.
+//
+// See explanations of candidate sampling and the data formats at
+// go/candidate-sampling.
+//
+// For each batch, this op picks a single set of sampled candidate labels.
+//
+// The advantages of sampling candidates per-batch are simplicity and the
+// possibility of efficient dense matrix multiplication. The disadvantage is that
+// the sampled candidates must be chosen independently of the context and of the
+// true labels.
+//
+// Arguments:
+//	true_classes: A batch_size * num_true matrix, in which each row contains the
+// IDs of the num_true target_classes in the corresponding original label.
+//	num_true: Number of true labels per context.
+//	num_sampled: Number of candidates to randomly sample.
+//	unique: If unique is true, we sample with rejection, so that all sampled
+// candidates in a batch are unique. This requires some approximation to
+// estimate the post-rejection sampling probabilities.
+//	range_max: The sampler will sample integers from the interval [0, range_max).
+//
+// Returns:
+//	sampled_candidates: A vector of length num_sampled, in which each element is
+// the ID of a sampled candidate.
+//	true_expected_count: A batch_size * num_true matrix, representing
+// the number of times each candidate is expected to occur in a batch
+// of sampled candidates. If unique=true, then this is a probability.
+//	sampled_expected_count: A vector of length num_sampled, for each sampled
+// candidate representing the number of times the candidate is expected
+// to occur in a batch of sampled candidates.  If unique=true, then this is a
+// probability.
+func UniformCandidateSampler(scope *Scope, true_classes tf.Output, num_true int64, num_sampled int64, unique bool, range_max int64, optional ...UniformCandidateSamplerAttr) (sampled_candidates tf.Output, true_expected_count tf.Output, sampled_expected_count tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_true": num_true, "num_sampled": num_sampled, "unique": unique, "range_max": range_max}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "UniformCandidateSampler",
+		Input: []tf.Input{
+			true_classes,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// TryRpcAttr is an optional argument to TryRpc.
+type TryRpcAttr func(optionalAttr)
+
+// TryRpcProtocol sets the optional protocol attribute to value.
+//
+// value: RPC protocol to use.  Empty string means use the default protocol.
+// Options include 'grpc'.
+// If not specified, defaults to ""
+func TryRpcProtocol(value string) TryRpcAttr {
+	return func(m optionalAttr) {
+		m["protocol"] = value
+	}
+}
+
+// TryRpcFailFast sets the optional fail_fast attribute to value.
+//
+// value: `boolean`. If `true` (default), then failures to connect
+// (i.e., the server does not immediately respond) cause an RPC failure.
+// If not specified, defaults to true
+func TryRpcFailFast(value bool) TryRpcAttr {
+	return func(m optionalAttr) {
+		m["fail_fast"] = value
+	}
+}
+
+// TryRpcTimeoutInMs sets the optional timeout_in_ms attribute to value.
+//
+// value: `int`. If `0` (default), then the kernel will run the RPC
+// request and only time out if the RPC deadline passes or the session times out.
+// If this value is greater than `0`, then the op will raise an exception if
+// the RPC takes longer than `timeout_in_ms`.
+// If not specified, defaults to 0
+func TryRpcTimeoutInMs(value int64) TryRpcAttr {
+	return func(m optionalAttr) {
+		m["timeout_in_ms"] = value
+	}
+}
+
+// Perform batches of RPC requests.
+//
+// This op asynchronously performs either a single RPC request, or a batch
+// of requests.  RPC requests are defined by three main parameters:
+//
+//   - `address` (the host+port or BNS address of the request)
+//   - `method` (the method name for the request)
+//   - `request` (the serialized proto string, or vector of strings,
+//      of the RPC request argument).
+//
+// For example, if you have an RPC service running on port localhost:2345,
+// and its interface is configured with the following proto declaration:
+//
+// ```
+// service MyService {
+//   rpc MyMethod(MyRequestProto) returns (MyResponseProto) {
+//   }
+// };
+// ```
+//
+// then call this op with arguments:
+//
+// ```
+// address = "localhost:2345"
+// method = "MyService/MyMethod"
+// ```
+//
+// The `request` tensor is a string tensor representing serialized `MyRequestProto`
+// strings; and the output string tensor `response` will have the same shape
+// and contain (upon successful completion) corresponding serialized
+// `MyResponseProto` strings.
+//
+// For example, to send a single, empty, `MyRequestProto`, call
+// this op with `request = ""`.  To send 5 **parallel** empty requests,
+// call this op with `request = ["", "", "", "", ""]`.
+//
+// More generally, one can create a batch of `MyRequestProto` serialized protos
+// from regular batched tensors using the `encode_proto` op, and convert
+// the response `MyResponseProto` serialized protos to batched tensors
+// using the `decode_proto` op.
+//
+// **NOTE** Working with serialized proto strings is faster than instantiating
+// actual proto objects in memory, so no performance degradation is expected
+// compared to writing custom kernels for this workflow.
+//
+// Unlike the standard `Rpc` op, if the connection fails or the remote worker
+// returns an error status, this op does **not** reraise the exception.
+// Instead, the `status_code` and `status_message` entry for the corresponding RPC
+// call is set with the error returned from the RPC call.  The `response` tensor
+// will contain valid response values for those minibatch entries whose RPCs did
+// not fail; the rest of the entries will have empty strings.
+//
+// Arguments:
+//	address: `0-D` or `1-D`.  The address (i.e. host_name:port) of the RPC server.
+// If this tensor has more than 1 element, then multiple parallel rpc requests
+// are sent.  This argument broadcasts with `method` and `request`.
+//	method: `0-D` or `1-D`.  The method address on the RPC server.
+// If this tensor has more than 1 element, then multiple parallel rpc requests
+// are sent.  This argument broadcasts with `address` and `request`.
+//	request: `0-D` or `1-D`.  Serialized proto strings: the rpc request argument.
+// If this tensor has more than 1 element, then multiple parallel rpc requests
+// are sent.  This argument broadcasts with `address` and `method`.
+//
+// Returns:
+//	response: Same shape as `request`. Serialized proto strings: the rpc responses.
+//	status_code: Same shape as `request`.  Values correspond to tensorflow Status enum codes.
+//	status_message: Same shape as `request`.  Values correspond to Status messages
+// returned from the RPC calls.
+func TryRpc(scope *Scope, address tf.Output, method tf.Output, request tf.Output, optional ...TryRpcAttr) (response tf.Output, status_code tf.Output, status_message tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TryRpc",
+		Input: []tf.Input{
+			address, method, request,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// ResourceGatherAttr is an optional argument to ResourceGather.
+type ResourceGatherAttr func(optionalAttr)
+
+// ResourceGatherBatchDims sets the optional batch_dims attribute to value.
+// If not specified, defaults to 0
+func ResourceGatherBatchDims(value int64) ResourceGatherAttr {
+	return func(m optionalAttr) {
+		m["batch_dims"] = value
+	}
+}
+
+// ResourceGatherValidateIndices sets the optional validate_indices attribute to value.
+// If not specified, defaults to true
+func ResourceGatherValidateIndices(value bool) ResourceGatherAttr {
+	return func(m optionalAttr) {
+		m["validate_indices"] = value
+	}
+}
+
+// Gather slices from the variable pointed to by `resource` according to `indices`.
+//
+// `indices` must be an integer tensor of any dimension (usually 0-D or 1-D).
+// Produces an output tensor with shape `indices.shape + params.shape[1:]` where:
+//
+// ```python
+//     # Scalar indices
+//     output[:, ..., :] = params[indices, :, ... :]
+//
+//     # Vector indices
+//     output[i, :, ..., :] = params[indices[i], :, ... :]
+//
+//     # Higher rank indices
+//     output[i, ..., j, :, ... :] = params[indices[i, ..., j], :, ..., :]
+// ```
+func ResourceGather(scope *Scope, resource tf.Output, indices tf.Output, dtype tf.DataType, optional ...ResourceGatherAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceGather",
+		Input: []tf.Input{
+			resource, indices,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns x + y element-wise.
+//
+// *NOTE*: `Add` supports broadcasting. `AddN` does not. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func Add(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Add",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns element-wise smallest integer not less than x.
+func Ceil(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Ceil",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns element-wise largest integer not greater than x.
+func Floor(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Floor",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the Bessel i0e function of `x` element-wise.
+//
+// Exponentially scaled modified Bessel function of order 0 defined as
+// `bessel_i0e(x) = exp(-abs(x)) bessel_i0(x)`.
+//
+// This function is faster and numerically stabler than `bessel_i0(x)`.
+func BesselI0e(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "BesselI0e",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the trignometric inverse tangent of x element-wise.
+//
+// The `tf.math.atan` operation returns the inverse of `tf.math.tan`, such that
+// if `y = tf.math.tan(x)` then, `x = tf.math.atan(y)`.
+//
+// **Note**: The output of `tf.math.atan` will lie within the invertible range
+// of tan, i.e (-pi/2, pi/2).
+//
+// For example:
+//
+// ```python
+// # Note: [1.047, 0.785] ~= [(pi/3), (pi/4)]
+// x = tf.constant([1.047, 0.785])
+// y = tf.math.tan(x) # [1.731261, 0.99920404]
+//
+// tf.math.atan(y) # [1.047, 0.785] = x
+// ```
+//
+func Atan(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Atan",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes acos of x element-wise.
+func Acos(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Acos",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// FusedBatchNormV2Attr is an optional argument to FusedBatchNormV2.
+type FusedBatchNormV2Attr func(optionalAttr)
+
+// FusedBatchNormV2Epsilon sets the optional epsilon attribute to value.
+//
+// value: A small float number added to the variance of x.
+// If not specified, defaults to 0.0001
+func FusedBatchNormV2Epsilon(value float32) FusedBatchNormV2Attr {
+	return func(m optionalAttr) {
+		m["epsilon"] = value
+	}
+}
+
+// FusedBatchNormV2ExponentialAvgFactor sets the optional exponential_avg_factor attribute to value.
+// If not specified, defaults to 1
+func FusedBatchNormV2ExponentialAvgFactor(value float32) FusedBatchNormV2Attr {
+	return func(m optionalAttr) {
+		m["exponential_avg_factor"] = value
+	}
+}
+
+// FusedBatchNormV2DataFormat sets the optional data_format attribute to value.
+//
+// value: The data format for x and y. Either "NHWC" (default) or "NCHW".
+// If not specified, defaults to "NHWC"
+func FusedBatchNormV2DataFormat(value string) FusedBatchNormV2Attr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// FusedBatchNormV2IsTraining sets the optional is_training attribute to value.
+//
+// value: A bool value to indicate the operation is for training (default)
+// or inference.
+// If not specified, defaults to true
+func FusedBatchNormV2IsTraining(value bool) FusedBatchNormV2Attr {
+	return func(m optionalAttr) {
+		m["is_training"] = value
+	}
+}
+
+// Batch normalization.
+//
+// Note that the size of 4D Tensors are defined by either "NHWC" or "NCHW".
+// The size of 1D Tensors matches the dimension C of the 4D Tensors.
+//
+// Arguments:
+//	x: A 4D Tensor for input data.
+//	scale: A 1D Tensor for scaling factor, to scale the normalized x.
+//	offset: A 1D Tensor for offset, to shift to the normalized x.
+//	mean: A 1D Tensor for population mean. Used for inference only;
+// must be empty for training.
+//	variance: A 1D Tensor for population variance. Used for inference only;
+// must be empty for training.
+//
+// Returns:
+//	y: A 4D Tensor for output data.
+//	batch_mean: A 1D Tensor for the computed batch mean, to be used by TensorFlow
+// to compute the running mean.
+//	batch_variance: A 1D Tensor for the computed batch variance, to be used by
+// TensorFlow to compute the running variance.
+//	reserve_space_1: A 1D Tensor for the computed batch mean, to be reused
+// in the gradient computation.
+//	reserve_space_2: A 1D Tensor for the computed batch variance (inverted variance
+// in the cuDNN case), to be reused in the gradient computation.
+func FusedBatchNormV2(scope *Scope, x tf.Output, scale tf.Output, offset tf.Output, mean tf.Output, variance tf.Output, optional ...FusedBatchNormV2Attr) (y tf.Output, batch_mean tf.Output, batch_variance tf.Output, reserve_space_1 tf.Output, reserve_space_2 tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "FusedBatchNormV2",
+		Input: []tf.Input{
+			x, scale, offset, mean, variance,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3), op.Output(4)
+}
+
+// Computes sine of x element-wise.
+//
+//   Given an input tensor, this function computes sine of every
+//   element in the tensor. Input range is `(-inf, inf)` and
+//   output range is `[-1,1]`.
+//
+//   ```python
+//   x = tf.constant([-float("inf"), -9, -0.5, 1, 1.2, 200, 10, float("inf")])
+//   tf.math.sin(x) ==> [nan -0.4121185 -0.47942555 0.84147096 0.9320391 -0.87329733 -0.54402107 nan]
+//   ```
+func Sin(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Sin",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// PrintAttr is an optional argument to Print.
+type PrintAttr func(optionalAttr)
+
+// PrintMessage sets the optional message attribute to value.
+//
+// value: A string, prefix of the error message.
+// If not specified, defaults to ""
+func PrintMessage(value string) PrintAttr {
+	return func(m optionalAttr) {
+		m["message"] = value
+	}
+}
+
+// PrintFirstN sets the optional first_n attribute to value.
+//
+// value: Only log `first_n` number of times. -1 disables logging.
+// If not specified, defaults to -1
+func PrintFirstN(value int64) PrintAttr {
+	return func(m optionalAttr) {
+		m["first_n"] = value
+	}
+}
+
+// PrintSummarize sets the optional summarize attribute to value.
+//
+// value: Only print this many entries of each tensor.
+// If not specified, defaults to 3
+func PrintSummarize(value int64) PrintAttr {
+	return func(m optionalAttr) {
+		m["summarize"] = value
+	}
+}
+
+// Prints a list of tensors.
+//
+// Passes `input` through to `output` and prints `data` when evaluating.
+//
+// Arguments:
+//	input: The tensor passed to `output`
+//	data: A list of tensors to print out when op is evaluated.
+//
+// Returns = The unmodified `input` tensor
+func Print(scope *Scope, input tf.Output, data []tf.Output, optional ...PrintAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Print",
+		Input: []tf.Input{
+			input, tf.OutputList(data),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the Approximate Minimum Degree (AMD) ordering of `input`.
+//
+// Computes the Approximate Minimum Degree (AMD) ordering for a sparse matrix.
+//
+// The returned permutation may be used to permute the rows and columns of the
+// given sparse matrix. This typically results in permuted sparse matrix's sparse
+// Cholesky (or other decompositions) in having fewer zero fill-in compared to
+// decomposition of the original matrix.
+//
+// The input sparse matrix may have rank 2 or rank 3. The output Tensor,
+// representing would then have rank 1 or 2 respectively, with the same batch
+// shape as the input.
+//
+// Each component of the input sparse matrix must represent a square symmetric
+// matrix; only the lower triangular part of the matrix is read. The values of the
+// sparse matrix does not affect the returned permutation, only the sparsity
+// pattern of the sparse matrix is used. Hence, a single AMD ordering may be
+// reused for the Cholesky decompositions of sparse matrices with the same sparsity
+// pattern but with possibly different values.
+//
+// Each batch component of the output permutation represents a permutation of `N`
+// elements, where the input sparse matrix components each have `N` rows. That is,
+// the component contains each of the integers `{0, .. N-1}` exactly once. The
+// `i`th element represents the row index that the `i`th row maps to.
+//
+// Usage example:
+//
+// ```python
+//     from tensorflow.python.ops.linalg.sparse import sparse_csr_matrix_ops
+//
+//     a_indices = np.array([[0, 0], [1, 1], [2, 1], [2, 2], [3, 3]])
+//     a_values = np.array([1.0, 2.0, 1.0, 3.0, 4.0], np.float32)
+//     a_dense_shape = [4, 4]
+//
+//     with tf.Session() as sess:
+//       # Define (COO format) SparseTensor over Numpy array.
+//       a_st = tf.sparse.SparseTensor(a_indices, a_values, a_dense_shape)
+//
+//       # Convert SparseTensors to CSR SparseMatrix.
+//       a_sm = sparse_csr_matrix_ops.sparse_tensor_to_csr_sparse_matrix(
+//           a_st.indices, a_st.values, a_st.dense_shape)
+//
+//       # Obtain the AMD Ordering for the CSR SparseMatrix.
+//       ordering_amd = sparse_csr_matrix_ops.sparse_matrix_ordering_amd(sparse_matrix)
+//
+//       ordering_amd_value = sess.run(ordering_amd)
+// ```
+//
+// `ordering_amd_value` stores the AMD ordering: `[1 2 3 0]`.
+//
+// input: A `CSRSparseMatrix`.
+//
+// Arguments:
+//	input: A `CSRSparseMatrix`.
+//
+// Returns The Approximate Minimum Degree (AMD) ordering of `input`.
+func SparseMatrixOrderingAMD(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseMatrixOrderingAMD",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes Psi, the derivative of Lgamma (the log of the absolute value of
+//
+// `Gamma(x)`), element-wise.
+func Digamma(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Digamma",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the gradient for the tanh of `x` wrt its input.
+//
+// Specifically, `grad = dy * (1 - y*y)`, where `y = tanh(x)`, and `dy`
+// is the corresponding input gradient.
+func TanhGrad(scope *Scope, y tf.Output, dy tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TanhGrad",
+		Input: []tf.Input{
+			y, dy,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// FusedBatchNormAttr is an optional argument to FusedBatchNorm.
+type FusedBatchNormAttr func(optionalAttr)
+
+// FusedBatchNormEpsilon sets the optional epsilon attribute to value.
+//
+// value: A small float number added to the variance of x.
+// If not specified, defaults to 0.0001
+func FusedBatchNormEpsilon(value float32) FusedBatchNormAttr {
+	return func(m optionalAttr) {
+		m["epsilon"] = value
+	}
+}
+
+// FusedBatchNormExponentialAvgFactor sets the optional exponential_avg_factor attribute to value.
+// If not specified, defaults to 1
+func FusedBatchNormExponentialAvgFactor(value float32) FusedBatchNormAttr {
+	return func(m optionalAttr) {
+		m["exponential_avg_factor"] = value
+	}
+}
+
+// FusedBatchNormDataFormat sets the optional data_format attribute to value.
+//
+// value: The data format for x and y. Either "NHWC" (default) or "NCHW".
+// If not specified, defaults to "NHWC"
+func FusedBatchNormDataFormat(value string) FusedBatchNormAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// FusedBatchNormIsTraining sets the optional is_training attribute to value.
+//
+// value: A bool value to indicate the operation is for training (default)
+// or inference.
+// If not specified, defaults to true
+func FusedBatchNormIsTraining(value bool) FusedBatchNormAttr {
+	return func(m optionalAttr) {
+		m["is_training"] = value
+	}
+}
+
+// Batch normalization.
+//
+// Note that the size of 4D Tensors are defined by either "NHWC" or "NCHW".
+// The size of 1D Tensors matches the dimension C of the 4D Tensors.
+//
+// Arguments:
+//	x: A 4D Tensor for input data.
+//	scale: A 1D Tensor for scaling factor, to scale the normalized x.
+//	offset: A 1D Tensor for offset, to shift to the normalized x.
+//	mean: A 1D Tensor for population mean. Used for inference only;
+// must be empty for training.
+//	variance: A 1D Tensor for population variance. Used for inference only;
+// must be empty for training.
+//
+// Returns:
+//	y: A 4D Tensor for output data.
+//	batch_mean: A 1D Tensor for the computed batch mean, to be used by TensorFlow
+// to compute the running mean.
+//	batch_variance: A 1D Tensor for the computed batch variance, to be used by
+// TensorFlow to compute the running variance.
+//	reserve_space_1: A 1D Tensor for the computed batch mean, to be reused
+// in the gradient computation.
+//	reserve_space_2: A 1D Tensor for the computed batch variance (inverted variance
+// in the cuDNN case), to be reused in the gradient computation.
+func FusedBatchNorm(scope *Scope, x tf.Output, scale tf.Output, offset tf.Output, mean tf.Output, variance tf.Output, optional ...FusedBatchNormAttr) (y tf.Output, batch_mean tf.Output, batch_variance tf.Output, reserve_space_1 tf.Output, reserve_space_2 tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "FusedBatchNorm",
+		Input: []tf.Input{
+			x, scale, offset, mean, variance,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3), op.Output(4)
+}
+
+// SparseMatMulAttr is an optional argument to SparseMatMul.
+type SparseMatMulAttr func(optionalAttr)
+
+// SparseMatMulTransposeA sets the optional transpose_a attribute to value.
+// If not specified, defaults to false
+func SparseMatMulTransposeA(value bool) SparseMatMulAttr {
+	return func(m optionalAttr) {
+		m["transpose_a"] = value
+	}
+}
+
+// SparseMatMulTransposeB sets the optional transpose_b attribute to value.
+// If not specified, defaults to false
+func SparseMatMulTransposeB(value bool) SparseMatMulAttr {
+	return func(m optionalAttr) {
+		m["transpose_b"] = value
+	}
+}
+
+// SparseMatMulAIsSparse sets the optional a_is_sparse attribute to value.
+// If not specified, defaults to false
+func SparseMatMulAIsSparse(value bool) SparseMatMulAttr {
+	return func(m optionalAttr) {
+		m["a_is_sparse"] = value
+	}
+}
+
+// SparseMatMulBIsSparse sets the optional b_is_sparse attribute to value.
+// If not specified, defaults to false
+func SparseMatMulBIsSparse(value bool) SparseMatMulAttr {
+	return func(m optionalAttr) {
+		m["b_is_sparse"] = value
+	}
+}
+
+// Multiply matrix "a" by matrix "b".
+//
+// The inputs must be two-dimensional matrices and the inner dimension of "a" must
+// match the outer dimension of "b". Both "a" and "b" must be `Tensor`s not
+// `SparseTensor`s.  This op is optimized for the case where at least one of "a" or
+// "b" is sparse, in the sense that they have a large proportion of zero values.
+// The breakeven for using this versus a dense matrix multiply on one platform was
+// 30% zero values in the sparse matrix.
+//
+// The gradient computation of this operation will only take advantage of sparsity
+// in the input gradient when that gradient comes from a Relu.
+func SparseMatMul(scope *Scope, a tf.Output, b tf.Output, optional ...SparseMatMulAttr) (product tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseMatMul",
+		Input: []tf.Input{
+			a, b,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Resizes the list.
+//
+//
+// input_handle: the input list
+// size: size of the output list
+//
+func TensorListResize(scope *Scope, input_handle tf.Output, size tf.Output) (output_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorListResize",
+		Input: []tf.Input{
+			input_handle, size,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes inverse hyperbolic tangent of x element-wise.
+//
+//   Given an input tensor, this function computes inverse hyperbolic tangent
+//   for every element in the tensor. Input range is `[-1,1]` and output range is
+//   `[-inf, inf]`. If input is `-1`, output will be `-inf` and if the
+//   input is `1`, output will be `inf`. Values outside the range will have
+//   `nan` as output.
+//
+//   ```python
+//   x = tf.constant([-float("inf"), -1, -0.5, 1, 0, 0.5, 10, float("inf")])
+//   tf.math.atanh(x) ==> [nan -inf -0.54930615 inf  0. 0.54930615 nan nan]
+//   ```
+func Atanh(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Atanh",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes hyperbolic tangent of `x` element-wise.
+//
+//   Given an input tensor, this function computes hyperbolic tangent of every
+//   element in the tensor. Input range is `[-inf, inf]` and
+//   output range is `[-1,1]`.
+//
+//   ```python
+//   x = tf.constant([-float("inf"), -5, -0.5, 1, 1.2, 2, 3, float("inf")])
+//   tf.math.tanh(x) ==> [-1. -0.99990916 -0.46211717 0.7615942 0.8336547 0.9640276 0.9950547 1.]
+//   ```
+func Tanh(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Tanh",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes hyperbolic sine of x element-wise.
+//
+//   Given an input tensor, this function computes hyperbolic sine of every
+//   element in the tensor. Input range is `[-inf,inf]` and output range
+//   is `[-inf,inf]`.
+//
+//   ```python
+//   x = tf.constant([-float("inf"), -9, -0.5, 1, 1.2, 2, 10, float("inf")])
+//   tf.math.sinh(x) ==> [-inf -4.0515420e+03 -5.2109528e-01 1.1752012e+00 1.5094614e+00 3.6268604e+00 1.1013232e+04 inf]
+//   ```
+func Sinh(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Sinh",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResourceApplyProximalAdagradAttr is an optional argument to ResourceApplyProximalAdagrad.
+type ResourceApplyProximalAdagradAttr func(optionalAttr)
+
+// ResourceApplyProximalAdagradUseLocking sets the optional use_locking attribute to value.
+//
+// value: If True, updating of the var and accum tensors will be protected by
+// a lock; otherwise the behavior is undefined, but may exhibit less contention.
+// If not specified, defaults to false
+func ResourceApplyProximalAdagradUseLocking(value bool) ResourceApplyProximalAdagradAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// Update '*var' and '*accum' according to FOBOS with Adagrad learning rate.
+//
+// accum += grad * grad
+// prox_v = var - lr * grad * (1 / sqrt(accum))
+// var = sign(prox_v)/(1+lr*l2) * max{|prox_v|-lr*l1,0}
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	accum: Should be from a Variable().
+//	lr: Scaling factor. Must be a scalar.
+//	l1: L1 regularization. Must be a scalar.
+//	l2: L2 regularization. Must be a scalar.
+//	grad: The gradient.
+//
+// Returns the created operation.
+func ResourceApplyProximalAdagrad(scope *Scope, var_ tf.Output, accum tf.Output, lr tf.Output, l1 tf.Output, l2 tf.Output, grad tf.Output, optional ...ResourceApplyProximalAdagradAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceApplyProximalAdagrad",
+		Input: []tf.Input{
+			var_, accum, lr, l1, l2, grad,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Divides sparse updates into the variable referenced by `resource`.
+//
+// This operation computes
+//
+//     # Scalar indices
+//     ref[indices, ...] /= updates[...]
+//
+//     # Vector indices (for each i)
+//     ref[indices[i], ...] /= updates[i, ...]
+//
+//     # High rank indices (for each i, ..., j)
+//     ref[indices[i, ..., j], ...] /= updates[i, ..., j, ...]
+//
+// Duplicate entries are handled correctly: if multiple `indices` reference
+// the same location, their contributions multiply.
+//
+// Requires `updates.shape = indices.shape + ref.shape[1:]` or `updates.shape = []`.
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src='https://www.tensorflow.org/images/ScatterAdd.png' alt>
+// </div>
+//
+// Arguments:
+//	resource: Should be from a `Variable` node.
+//	indices: A tensor of indices into the first dimension of `ref`.
+//	updates: A tensor of updated values to add to `ref`.
+//
+// Returns the created operation.
+func ResourceScatterDiv(scope *Scope, resource tf.Output, indices tf.Output, updates tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceScatterDiv",
+		Input: []tf.Input{
+			resource, indices, updates,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Computes the trignometric inverse sine of x element-wise.
+//
+// The `tf.math.asin` operation returns the inverse of `tf.math.sin`, such that
+// if `y = tf.math.sin(x)` then, `x = tf.math.asin(y)`.
+//
+// **Note**: The output of `tf.math.asin` will lie within the invertible range
+// of sine, i.e [-pi/2, pi/2].
+//
+// For example:
+//
+// ```python
+// # Note: [1.047, 0.785] ~= [(pi/3), (pi/4)]
+// x = tf.constant([1.047, 0.785])
+// y = tf.math.sin(x) # [0.8659266, 0.7068252]
+//
+// tf.math.asin(y) # [1.047, 0.785] = x
+// ```
+//
+func Asin(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Asin",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes natural logarithm of (1 + x) element-wise.
+//
+// I.e., \\(y = \log_e (1 + x)\\).
+//
+// Example:
+//
+// ```python
+// x = tf.constant([0, 0.5, 1, 5])
+// tf.math.log1p(x) ==> [0., 0.4054651, 0.6931472, 1.7917595]
+// ```
+func Log1p(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Log1p",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Converts the quantized `input` tensor into a lower-precision `output`.
+//
+// Converts the quantized `input` tensor into a lower-precision `output`, using the
+// output range specified with `requested_output_min` and `requested_output_max`.
+//
+// `[input_min, input_max]` are scalar floats that specify the range for the float
+// interpretation of the `input` data. For example, if `input_min` is -1.0f and
+// `input_max` is 1.0f, and we are dealing with `quint16` quantized data, then a 0
+// value in the 16-bit data should be interpreted as -1.0f, and a 65535 means 1.0f.
+//
+// Arguments:
+//
+//	input_min: The float value that the minimum quantized input value represents.
+//	input_max: The float value that the maximum quantized input value represents.
+//	requested_output_min: The float value that the minimum quantized output value represents.
+//	requested_output_max: The float value that the maximum quantized output value represents.
+//	out_type: The type of the output. Should be a lower bit depth than Tinput.
+//
+// Returns:
+//	output
+//	output_min: The requested_output_min value is copied into this output.
+//	output_max: The requested_output_max value is copied into this output.
+func Requantize(scope *Scope, input tf.Output, input_min tf.Output, input_max tf.Output, requested_output_min tf.Output, requested_output_max tf.Output, out_type tf.DataType) (output tf.Output, output_min tf.Output, output_max tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"out_type": out_type}
+	opspec := tf.OpSpec{
+		Type: "Requantize",
+		Input: []tf.Input{
+			input, input_min, input_max, requested_output_min, requested_output_max,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Conv2DBackpropInputAttr is an optional argument to Conv2DBackpropInput.
+type Conv2DBackpropInputAttr func(optionalAttr)
+
+// Conv2DBackpropInputUseCudnnOnGpu sets the optional use_cudnn_on_gpu attribute to value.
+// If not specified, defaults to true
+func Conv2DBackpropInputUseCudnnOnGpu(value bool) Conv2DBackpropInputAttr {
+	return func(m optionalAttr) {
+		m["use_cudnn_on_gpu"] = value
+	}
+}
+
+// Conv2DBackpropInputExplicitPaddings sets the optional explicit_paddings attribute to value.
+//
+// value: If `padding` is `"EXPLICIT"`, the list of explicit padding amounts. For the ith
+// dimension, the amount of padding inserted before and after the dimension is
+// `explicit_paddings[2 * i]` and `explicit_paddings[2 * i + 1]`, respectively. If
+// `padding` is not `"EXPLICIT"`, `explicit_paddings` must be empty.
+// If not specified, defaults to <>
+func Conv2DBackpropInputExplicitPaddings(value []int64) Conv2DBackpropInputAttr {
+	return func(m optionalAttr) {
+		m["explicit_paddings"] = value
+	}
+}
+
+// Conv2DBackpropInputDataFormat sets the optional data_format attribute to value.
+//
+// value: Specify the data format of the input and output data. With the
+// default format "NHWC", the data is stored in the order of:
+//     [batch, in_height, in_width, in_channels].
+// Alternatively, the format could be "NCHW", the data storage order of:
+//     [batch, in_channels, in_height, in_width].
+// If not specified, defaults to "NHWC"
+func Conv2DBackpropInputDataFormat(value string) Conv2DBackpropInputAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// Conv2DBackpropInputDilations sets the optional dilations attribute to value.
+//
+// value: 1-D tensor of length 4.  The dilation factor for each dimension of
+// `input`. If set to k > 1, there will be k-1 skipped cells between each filter
+// element on that dimension. The dimension order is determined by the value of
+// `data_format`, see above for details. Dilations in the batch and depth
+// dimensions must be 1.
+// If not specified, defaults to <i:1 i:1 i:1 i:1 >
+func Conv2DBackpropInputDilations(value []int64) Conv2DBackpropInputAttr {
+	return func(m optionalAttr) {
+		m["dilations"] = value
+	}
+}
+
+// Computes the gradients of convolution with respect to the input.
+//
+// Arguments:
+//	input_sizes: An integer vector representing the shape of `input`,
+// where `input` is a 4-D `[batch, height, width, channels]` tensor.
+//	filter: 4-D with shape
+// `[filter_height, filter_width, in_channels, out_channels]`.
+//	out_backprop: 4-D with shape `[batch, out_height, out_width, out_channels]`.
+// Gradients w.r.t. the output of the convolution.
+//	strides: The stride of the sliding window for each dimension of the input
+// of the convolution. Must be in the same order as the dimension specified with
+// format.
+//	padding: The type of padding algorithm to use.
+//
+// Returns 4-D with shape `[batch, in_height, in_width, in_channels]`.  Gradient
+// w.r.t. the input of the convolution.
+func Conv2DBackpropInput(scope *Scope, input_sizes tf.Output, filter tf.Output, out_backprop tf.Output, strides []int64, padding string, optional ...Conv2DBackpropInputAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Conv2DBackpropInput",
+		Input: []tf.Input{
+			input_sizes, filter, out_backprop,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes `exp(x) - 1` element-wise.
+//
+//   i.e. `exp(x) - 1` or `e^(x) - 1`, where `x` is the input tensor.
+//   `e` denotes Euler's number and is approximately equal to 2.718281.
+//
+//   ```python
+//   x = tf.constant(2.0)
+//   tf.math.expm1(x) ==> 6.389056
+//
+//   x = tf.constant([2.0, 8.0])
+//   tf.math.expm1(x) ==> array([6.389056, 2979.958], dtype=float32)
+//
+//   x = tf.constant(1 + 1j)
+//   tf.math.expm1(x) ==> (0.46869393991588515+2.2873552871788423j)
+//   ```
+func Expm1(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Expm1",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes exponential of x element-wise.  \\(y = e^x\\).
+//
+//   This function computes the exponential of every element in the input tensor.
+//   i.e. `exp(x)` or `e^(x)`, where `x` is the input tensor.
+//   `e` denotes Euler's number and is approximately equal to 2.718281.
+//   Output is positive for any real input.
+//
+//   ```python
+//   x = tf.constant(2.0)
+//   tf.math.exp(x) ==> 7.389056
+//
+//   x = tf.constant([2.0, 8.0])
+//   tf.math.exp(x) ==> array([7.389056, 2980.958], dtype=float32)
+//   ```
+//
+//   For complex numbers, the exponential value is calculated as follows:
+//
+//   ```
+//   e^(x+iy) = e^x * e^iy = e^x * (cos y + i sin y)
+//   ```
+//
+//   Let's consider complex number 1+1j as an example.
+//   e^1 * (cos 1 + i sin 1) = 2.7182818284590 * (0.54030230586+0.8414709848j)
+//
+//   ```python
+//   x = tf.constant(1 + 1j)
+//   tf.math.exp(x) ==> 1.4686939399158851+2.2873552871788423j
+//   ```
+func Exp(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Exp",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes square of x element-wise.
+//
+// I.e., \\(y = x * x = x^2\\).
+func Square(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Square",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the gradient for the inverse of `x` wrt its input.
+//
+// Specifically, `grad = -dy * y*y`, where `y = 1/x`, and `dy`
+// is the corresponding input gradient.
+func ReciprocalGrad(scope *Scope, y tf.Output, dy tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ReciprocalGrad",
+		Input: []tf.Input{
+			y, dy,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the reciprocal of x element-wise.
+//
+// I.e., \\(y = 1 / x\\).
+func Inv(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Inv",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ComplexAbsAttr is an optional argument to ComplexAbs.
+type ComplexAbsAttr func(optionalAttr)
+
+// ComplexAbsTout sets the optional Tout attribute to value.
+// If not specified, defaults to DT_FLOAT
+func ComplexAbsTout(value tf.DataType) ComplexAbsAttr {
+	return func(m optionalAttr) {
+		m["Tout"] = value
+	}
+}
+
+// Computes the complex absolute value of a tensor.
+//
+// Given a tensor `x` of complex numbers, this operation returns a tensor of type
+// `float` or `double` that is the absolute value of each element in `x`. All
+// elements in `x` must be complex numbers of the form \\(a + bj\\). The absolute
+// value is computed as \\( \sqrt{a^2 + b^2}\\).
+func ComplexAbs(scope *Scope, x tf.Output, optional ...ComplexAbsAttr) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ComplexAbs",
+		Input: []tf.Input{
+			x,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the absolute value of a tensor.
+//
+// Given a tensor `x`, this operation returns a tensor containing the absolute
+// value of each element in `x`. For example, if x is an input element and y is
+// an output element, this operation computes \\(y = |x|\\).
+func Abs(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Abs",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Produces a summary of any statistics recorded by the given statistics manager.
+func ExperimentalStatsAggregatorSummary(scope *Scope, iterator tf.Output) (summary tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalStatsAggregatorSummary",
+		Input: []tf.Input{
+			iterator,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MeanAttr is an optional argument to Mean.
+type MeanAttr func(optionalAttr)
+
+// MeanKeepDims sets the optional keep_dims attribute to value.
+//
+// value: If true, retain reduced dimensions with length 1.
+// If not specified, defaults to false
+func MeanKeepDims(value bool) MeanAttr {
+	return func(m optionalAttr) {
+		m["keep_dims"] = value
+	}
+}
+
+// Computes the mean of elements across dimensions of a tensor.
+//
+// Reduces `input` along the dimensions given in `axis`. Unless
+// `keep_dims` is true, the rank of the tensor is reduced by 1 for each entry in
+// `axis`. If `keep_dims` is true, the reduced dimensions are
+// retained with length 1.
+//
+// Arguments:
+//	input: The tensor to reduce.
+//	axis: The dimensions to reduce. Must be in the range
+// `[-rank(input), rank(input))`.
+//
+// Returns The reduced tensor.
+func Mean(scope *Scope, input tf.Output, axis tf.Output, optional ...MeanAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Mean",
+		Input: []tf.Input{
+			input, axis,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RandomStandardNormalAttr is an optional argument to RandomStandardNormal.
+type RandomStandardNormalAttr func(optionalAttr)
+
+// RandomStandardNormalSeed sets the optional seed attribute to value.
+//
+// value: If either `seed` or `seed2` are set to be non-zero, the random number
+// generator is seeded by the given seed.  Otherwise, it is seeded by a
+// random seed.
+// If not specified, defaults to 0
+func RandomStandardNormalSeed(value int64) RandomStandardNormalAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// RandomStandardNormalSeed2 sets the optional seed2 attribute to value.
+//
+// value: A second seed to avoid seed collision.
+// If not specified, defaults to 0
+func RandomStandardNormalSeed2(value int64) RandomStandardNormalAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Outputs random values from a normal distribution.
+//
+// The generated values will have mean 0 and standard deviation 1.
+//
+// Arguments:
+//	shape: The shape of the output tensor.
+//	dtype: The type of the output.
+//
+// Returns A tensor of the specified shape filled with random normal values.
+func RandomStandardNormal(scope *Scope, shape tf.Output, dtype tf.DataType, optional ...RandomStandardNormalAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RandomStandardNormal",
+		Input: []tf.Input{
+			shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the Gauss error function of `x` element-wise.
+func Erf(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Erf",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the maximum along segments of a tensor.
+//
+// Read
+// [the section on segmentation](https://tensorflow.org/api_docs/python/tf/math#Segmentation)
+// for an explanation of segments.
+//
+// Computes a tensor such that
+// \\(output_i = \max_j(data_j)\\) where `max` is over `j` such
+// that `segment_ids[j] == i`.
+//
+// If the max is empty for a given segment ID `i`, `output[i] = 0`.
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src="https://www.tensorflow.org/images/SegmentMax.png" alt>
+// </div>
+//
+// For example:
+//
+// ```
+// c = tf.constant([[1,2,3,4], [4, 3, 2, 1], [5,6,7,8]])
+// tf.segment_max(c, tf.constant([0, 0, 1]))
+// # ==> [[4, 3, 3, 4],
+// #      [5, 6, 7, 8]]
+// ```
+//
+//
+// Arguments:
+//
+//	segment_ids: A 1-D tensor whose size is equal to the size of `data`'s
+// first dimension.  Values should be sorted and can be repeated.
+//
+// Returns Has same shape as data, except for dimension 0 which
+// has size `k`, the number of segments.
+func SegmentMax(scope *Scope, data tf.Output, segment_ids tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SegmentMax",
+		Input: []tf.Input{
+			data, segment_ids,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// CastAttr is an optional argument to Cast.
+type CastAttr func(optionalAttr)
+
+// CastTruncate sets the optional Truncate attribute to value.
+// If not specified, defaults to false
+func CastTruncate(value bool) CastAttr {
+	return func(m optionalAttr) {
+		m["Truncate"] = value
+	}
+}
+
+// Cast x of type SrcT to y of DstT.
+func Cast(scope *Scope, x tf.Output, DstT tf.DataType, optional ...CastAttr) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"DstT": DstT}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Cast",
+		Input: []tf.Input{
+			x,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Generate a sharded filename. The filename is printf formatted as
+//
+//    %s-%05d-of-%05d, basename, shard, num_shards.
+func ShardedFilename(scope *Scope, basename tf.Output, shard tf.Output, num_shards tf.Output) (filename tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ShardedFilename",
+		Input: []tf.Input{
+			basename, shard, num_shards,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Elementwise computes the bitwise OR of `x` and `y`.
+//
+// The result will have those bits set, that are set in `x`, `y` or both. The
+// computation is performed on the underlying representations of `x` and `y`.
+//
+// For example:
+//
+// ```python
+// import tensorflow as tf
+// from tensorflow.python.ops import bitwise_ops
+// dtype_list = [tf.int8, tf.int16, tf.int32, tf.int64,
+//               tf.uint8, tf.uint16, tf.uint32, tf.uint64]
+//
+// for dtype in dtype_list:
+//   lhs = tf.constant([0, 5, 3, 14], dtype=dtype)
+//   rhs = tf.constant([5, 0, 7, 11], dtype=dtype)
+//   exp = tf.constant([5, 5, 7, 15], dtype=tf.float32)
+//
+//   res = bitwise_ops.bitwise_or(lhs, rhs)
+//   tf.assert_equal(tf.cast(res,  tf.float32), exp)  # TRUE
+// ```
+//
+func BitwiseOr(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "BitwiseOr",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SendAttr is an optional argument to Send.
+type SendAttr func(optionalAttr)
+
+// SendClientTerminated sets the optional client_terminated attribute to value.
+//
+// value: If set to true, this indicates that the node was added
+// to the graph as a result of a client-side feed or fetch of Tensor data,
+// in which case the corresponding send or recv is expected to be managed
+// locally by the caller.
+// If not specified, defaults to false
+func SendClientTerminated(value bool) SendAttr {
+	return func(m optionalAttr) {
+		m["client_terminated"] = value
+	}
+}
+
+// Sends the named tensor from send_device to recv_device.
+//
+// Arguments:
+//	tensor: The tensor to send.
+//	tensor_name: The name of the tensor to send.
+//	send_device: The name of the device sending the tensor.
+//	send_device_incarnation: The current incarnation of send_device.
+//	recv_device: The name of the device receiving the tensor.
+//
+// Returns the created operation.
+func Send(scope *Scope, tensor tf.Output, tensor_name string, send_device string, send_device_incarnation int64, recv_device string, optional ...SendAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"tensor_name": tensor_name, "send_device": send_device, "send_device_incarnation": send_device_incarnation, "recv_device": recv_device}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Send",
+		Input: []tf.Input{
+			tensor,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// BatchMatMulV2Attr is an optional argument to BatchMatMulV2.
+type BatchMatMulV2Attr func(optionalAttr)
+
+// BatchMatMulV2AdjX sets the optional adj_x attribute to value.
+//
+// value: If `True`, adjoint the slices of `x`. Defaults to `False`.
+// If not specified, defaults to false
+func BatchMatMulV2AdjX(value bool) BatchMatMulV2Attr {
+	return func(m optionalAttr) {
+		m["adj_x"] = value
+	}
+}
+
+// BatchMatMulV2AdjY sets the optional adj_y attribute to value.
+//
+// value: If `True`, adjoint the slices of `y`. Defaults to `False`.
+// If not specified, defaults to false
+func BatchMatMulV2AdjY(value bool) BatchMatMulV2Attr {
+	return func(m optionalAttr) {
+		m["adj_y"] = value
+	}
+}
+
+// Multiplies slices of two tensors in batches.
+//
+// Multiplies all slices of `Tensor` `x` and `y` (each slice can be
+// viewed as an element of a batch), and arranges the individual results
+// in a single output tensor of the same batch size. Each of the
+// individual slices can optionally be adjointed (to adjoint a matrix
+// means to transpose and conjugate it) before multiplication by setting
+// the `adj_x` or `adj_y` flag to `True`, which are by default `False`.
+//
+// The input tensors `x` and `y` are 2-D or higher with shape `[..., r_x, c_x]`
+// and `[..., r_y, c_y]`.
+//
+// The output tensor is 2-D or higher with shape `[..., r_o, c_o]`, where:
+//
+//     r_o = c_x if adj_x else r_x
+//     c_o = r_y if adj_y else c_y
+//
+// It is computed as:
+//
+//     output[..., :, :] = matrix(x[..., :, :]) * matrix(y[..., :, :])
+//
+// *NOTE*: `BatchMatMulV2` supports broadcasting in the batch dimensions. More
+// about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html).
+//
+//
+// Arguments:
+//	x: 2-D or higher with shape `[..., r_x, c_x]`.
+//	y: 2-D or higher with shape `[..., r_y, c_y]`.
+//
+// Returns 3-D or higher with shape `[..., r_o, c_o]`
+func BatchMatMulV2(scope *Scope, x tf.Output, y tf.Output, optional ...BatchMatMulV2Attr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "BatchMatMulV2",
+		Input: []tf.Input{
+			x, y,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns 0 if x == 0, and x / y otherwise, elementwise.
+func Xdivy(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Xdivy",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Compute the pairwise cross product.
+//
+// `a` and `b` must be the same shape; they can either be simple 3-element vectors,
+// or any shape where the innermost dimension is 3. In the latter case, each pair
+// of corresponding 3-element vectors is cross-multiplied independently.
+//
+// Arguments:
+//	a: A tensor containing 3-element vectors.
+//	b: Another tensor, of same type and shape as `a`.
+//
+// Returns Pairwise cross product of the vectors in `a` and `b`.
+func Cross(scope *Scope, a tf.Output, b tf.Output) (product tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Cross",
+		Input: []tf.Input{
+			a, b,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Sends `input` to all devices that are connected to the output.
+//
+// Sends `input` to all devices that are connected to the output.
+//
+// The graph should be constructed so that all ops connected to the output have a
+// valid device assignment, and the op itself is assigned one of these devices.
+//
+// input: The input to the broadcast.
+// output: The same as input.
+// shape: The shape of the input tensor.
+//
+func NcclBroadcast(scope *Scope, input tf.Output, shape tf.Shape) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"shape": shape}
+	opspec := tf.OpSpec{
+		Type: "NcclBroadcast",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Conv2DAttr is an optional argument to Conv2D.
+type Conv2DAttr func(optionalAttr)
+
+// Conv2DUseCudnnOnGpu sets the optional use_cudnn_on_gpu attribute to value.
+// If not specified, defaults to true
+func Conv2DUseCudnnOnGpu(value bool) Conv2DAttr {
+	return func(m optionalAttr) {
+		m["use_cudnn_on_gpu"] = value
+	}
+}
+
+// Conv2DExplicitPaddings sets the optional explicit_paddings attribute to value.
+//
+// value: If `padding` is `"EXPLICIT"`, the list of explicit padding amounts. For the ith
+// dimension, the amount of padding inserted before and after the dimension is
+// `explicit_paddings[2 * i]` and `explicit_paddings[2 * i + 1]`, respectively. If
+// `padding` is not `"EXPLICIT"`, `explicit_paddings` must be empty.
+// If not specified, defaults to <>
+func Conv2DExplicitPaddings(value []int64) Conv2DAttr {
+	return func(m optionalAttr) {
+		m["explicit_paddings"] = value
+	}
+}
+
+// Conv2DDataFormat sets the optional data_format attribute to value.
+//
+// value: Specify the data format of the input and output data. With the
+// default format "NHWC", the data is stored in the order of:
+//     [batch, height, width, channels].
+// Alternatively, the format could be "NCHW", the data storage order of:
+//     [batch, channels, height, width].
+// If not specified, defaults to "NHWC"
+func Conv2DDataFormat(value string) Conv2DAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// Conv2DDilations sets the optional dilations attribute to value.
+//
+// value: 1-D tensor of length 4.  The dilation factor for each dimension of
+// `input`. If set to k > 1, there will be k-1 skipped cells between each
+// filter element on that dimension. The dimension order is determined by the
+// value of `data_format`, see above for details. Dilations in the batch and
+// depth dimensions must be 1.
+// If not specified, defaults to <i:1 i:1 i:1 i:1 >
+func Conv2DDilations(value []int64) Conv2DAttr {
+	return func(m optionalAttr) {
+		m["dilations"] = value
+	}
+}
+
+// Computes a 2-D convolution given 4-D `input` and `filter` tensors.
+//
+// Given an input tensor of shape `[batch, in_height, in_width, in_channels]`
+// and a filter / kernel tensor of shape
+// `[filter_height, filter_width, in_channels, out_channels]`, this op
+// performs the following:
+//
+// 1. Flattens the filter to a 2-D matrix with shape
+//    `[filter_height * filter_width * in_channels, output_channels]`.
+// 2. Extracts image patches from the input tensor to form a *virtual*
+//    tensor of shape `[batch, out_height, out_width,
+//    filter_height * filter_width * in_channels]`.
+// 3. For each patch, right-multiplies the filter matrix and the image patch
+//    vector.
+//
+// In detail, with the default NHWC format,
+//
+//     output[b, i, j, k] =
+//         sum_{di, dj, q} input[b, strides[1] * i + di, strides[2] * j + dj, q] *
+//                         filter[di, dj, q, k]
+//
+// Must have `strides[0] = strides[3] = 1`.  For the most common case of the same
+// horizontal and vertices strides, `strides = [1, stride, stride, 1]`.
+//
+// Arguments:
+//	input: A 4-D tensor. The dimension order is interpreted according to the value
+// of `data_format`, see below for details.
+//	filter: A 4-D tensor of shape
+// `[filter_height, filter_width, in_channels, out_channels]`
+//	strides: 1-D tensor of length 4.  The stride of the sliding window for each
+// dimension of `input`. The dimension order is determined by the value of
+// `data_format`, see below for details.
+//	padding: The type of padding algorithm to use.
+//
+// Returns A 4-D tensor. The dimension order is determined by the value of
+// `data_format`, see below for details.
+func Conv2D(scope *Scope, input tf.Output, filter tf.Output, strides []int64, padding string, optional ...Conv2DAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Conv2D",
+		Input: []tf.Input{
+			input, filter,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns locations of nonzero / true values in a tensor.
+//
+// This operation returns the coordinates of true elements in `condition`. The
+// coordinates are returned in a 2-D tensor where the first dimension (rows)
+// represents the number of true elements, and the second dimension (columns)
+// represents the coordinates of the true elements. Keep in mind, the shape of
+// the output tensor can vary depending on how many true values there are in
+// `condition`. Indices are output in row-major order.
+//
+// For example:
+//
+// ```
+// # 'input' tensor is [[True, False]
+// #                    [True, False]]
+// # 'input' has two true values, so output has two coordinates.
+// # 'input' has rank of 2, so coordinates have two indices.
+// where(input) ==> [[0, 0],
+//                   [1, 0]]
+//
+// # `condition` tensor is [[[True, False]
+// #                     [True, False]]
+// #                    [[False, True]
+// #                     [False, True]]
+// #                    [[False, False]
+// #                     [False, True]]]
+// # 'input' has 5 true values, so output has 5 coordinates.
+// # 'input' has rank of 3, so coordinates have three indices.
+// where(input) ==> [[0, 0, 0],
+//                   [0, 1, 0],
+//                   [1, 0, 1],
+//                   [1, 1, 1],
+//                   [2, 1, 1]]
+//
+// # `condition` tensor is [[[1.5,  0.0]
+// #                     [-0.5, 0.0]]
+// #                    [[0.0,  0.25]
+// #                     [0.0,  0.75]]
+// #                    [[0.0,  0.0]
+// #                     [0.0,  0.01]]]
+// # 'input' has 5 nonzero values, so output has 5 coordinates.
+// # 'input' has rank of 3, so coordinates have three indices.
+// where(input) ==> [[0, 0, 0],
+//                   [0, 1, 0],
+//                   [1, 0, 1],
+//                   [1, 1, 1],
+//                   [2, 1, 1]]
+//
+// # `condition` tensor is [[[1.5 + 0.0j, 0.0  + 0.0j]
+// #                     [0.0 + 0.5j, 0.0  + 0.0j]]
+// #                    [[0.0 + 0.0j, 0.25 + 1.5j]
+// #                     [0.0 + 0.0j, 0.75 + 0.0j]]
+// #                    [[0.0 + 0.0j, 0.0  + 0.0j]
+// #                     [0.0 + 0.0j, 0.01 + 0.0j]]]
+// # 'input' has 5 nonzero magnitude values, so output has 5 coordinates.
+// # 'input' has rank of 3, so coordinates have three indices.
+// where(input) ==> [[0, 0, 0],
+//                   [0, 1, 0],
+//                   [1, 0, 1],
+//                   [1, 1, 1],
+//                   [2, 1, 1]]
+// ```
+func Where(scope *Scope, condition tf.Output) (index tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Where",
+		Input: []tf.Input{
+			condition,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Deprecated, use python implementation tf.linalg.matrix_exponential.
+//
+// DEPRECATED at GraphDef version 27: Use Python implementation tf.linalg.matrix_exponential instead.
+func MatrixExponential(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "MatrixExponential",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Reduces `input` from `num_devices` using `reduction` to a single device.
+//
+// Reduces `input` from `num_devices` using `reduction` to a single device.
+//
+// The graph should be constructed so that all inputs have a valid device
+// assignment, and the op itself is assigned one of these devices.
+//
+// input: The input to the reduction.
+// data: the value of the reduction across all `num_devices` devices.
+// reduction: the reduction operation to perform.
+func NcclReduce(scope *Scope, input []tf.Output, reduction string) (data tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"reduction": reduction}
+	opspec := tf.OpSpec{
+		Type: "NcclReduce",
+		Input: []tf.Input{
+			tf.OutputList(input),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// QuantizedDepthwiseConv2DWithBiasAndReluAndRequantizeAttr is an optional argument to QuantizedDepthwiseConv2DWithBiasAndReluAndRequantize.
+type QuantizedDepthwiseConv2DWithBiasAndReluAndRequantizeAttr func(optionalAttr)
+
+// QuantizedDepthwiseConv2DWithBiasAndReluAndRequantizeOutType sets the optional out_type attribute to value.
+//
+// value: The type of the output.
+// If not specified, defaults to DT_QUINT8
+func QuantizedDepthwiseConv2DWithBiasAndReluAndRequantizeOutType(value tf.DataType) QuantizedDepthwiseConv2DWithBiasAndReluAndRequantizeAttr {
+	return func(m optionalAttr) {
+		m["out_type"] = value
+	}
+}
+
+// QuantizedDepthwiseConv2DWithBiasAndReluAndRequantizeDilations sets the optional dilations attribute to value.
+//
+// value: List of dilation values.
+// If not specified, defaults to <i:1 i:1 i:1 i:1 >
+func QuantizedDepthwiseConv2DWithBiasAndReluAndRequantizeDilations(value []int64) QuantizedDepthwiseConv2DWithBiasAndReluAndRequantizeAttr {
+	return func(m optionalAttr) {
+		m["dilations"] = value
+	}
+}
+
+// QuantizedDepthwiseConv2DWithBiasAndReluAndRequantizePaddingList sets the optional padding_list attribute to value.
+// If not specified, defaults to <>
+func QuantizedDepthwiseConv2DWithBiasAndReluAndRequantizePaddingList(value []int64) QuantizedDepthwiseConv2DWithBiasAndReluAndRequantizeAttr {
+	return func(m optionalAttr) {
+		m["padding_list"] = value
+	}
+}
+
+// Computes quantized depthwise Conv2D with Bias, Relu and Requantize.
+//
+// Arguments:
+//	input: The original input tensor.
+//	filter: The original filter tensor.
+//	bias: The original bias tensor.
+//	min_input: The float value that the minimum quantized input value represents.
+//	max_input: The float value that the maximum quantized input value represents.
+//	min_filter: The float value that the minimum quantized filter value represents.
+//	max_filter: The float value that the maximum quantized filter value represents.
+//	min_freezed_output: The minimum float value of the output tensor.
+//	max_freezed_output: The maximum float value of the output tensor.
+//	strides: List of stride values.
+//
+//
+// Returns:
+//	output: The output tensor.
+//	min_output: The float value that the minimum quantized output value represents.
+//	max_output: The float value that the maximum quantized output value represents.
+func QuantizedDepthwiseConv2DWithBiasAndReluAndRequantize(scope *Scope, input tf.Output, filter tf.Output, bias tf.Output, min_input tf.Output, max_input tf.Output, min_filter tf.Output, max_filter tf.Output, min_freezed_output tf.Output, max_freezed_output tf.Output, strides []int64, padding string, optional ...QuantizedDepthwiseConv2DWithBiasAndReluAndRequantizeAttr) (output tf.Output, min_output tf.Output, max_output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizedDepthwiseConv2DWithBiasAndReluAndRequantize",
+		Input: []tf.Input{
+			input, filter, bias, min_input, max_input, min_filter, max_filter, min_freezed_output, max_freezed_output,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// QuantizedDepthwiseConv2DWithBiasAndReluAttr is an optional argument to QuantizedDepthwiseConv2DWithBiasAndRelu.
+type QuantizedDepthwiseConv2DWithBiasAndReluAttr func(optionalAttr)
+
+// QuantizedDepthwiseConv2DWithBiasAndReluOutType sets the optional out_type attribute to value.
+//
+// value: The type of the output.
+// If not specified, defaults to DT_QINT32
+func QuantizedDepthwiseConv2DWithBiasAndReluOutType(value tf.DataType) QuantizedDepthwiseConv2DWithBiasAndReluAttr {
+	return func(m optionalAttr) {
+		m["out_type"] = value
+	}
+}
+
+// QuantizedDepthwiseConv2DWithBiasAndReluDilations sets the optional dilations attribute to value.
+//
+// value: List of dilation values.
+// If not specified, defaults to <i:1 i:1 i:1 i:1 >
+func QuantizedDepthwiseConv2DWithBiasAndReluDilations(value []int64) QuantizedDepthwiseConv2DWithBiasAndReluAttr {
+	return func(m optionalAttr) {
+		m["dilations"] = value
+	}
+}
+
+// QuantizedDepthwiseConv2DWithBiasAndReluPaddingList sets the optional padding_list attribute to value.
+// If not specified, defaults to <>
+func QuantizedDepthwiseConv2DWithBiasAndReluPaddingList(value []int64) QuantizedDepthwiseConv2DWithBiasAndReluAttr {
+	return func(m optionalAttr) {
+		m["padding_list"] = value
+	}
+}
+
+// Computes quantized depthwise Conv2D with Bias and Relu.
+//
+// Arguments:
+//	input: The original input tensor.
+//	filter: The original filter tensor.
+//	bias: The original bias tensor.
+//	min_input: The float value that the minimum quantized input value represents.
+//	max_input: The float value that the maximum quantized input value represents.
+//	min_filter: The float value that the minimum quantized filter value represents.
+//	max_filter: The float value that the maximum quantized filter value represents.
+//	strides: List of stride values.
+//
+//
+// Returns:
+//	output: The output tensor.
+//	min_output: The float value that the minimum quantized output value represents.
+//	max_output: The float value that the maximum quantized output value represents.
+func QuantizedDepthwiseConv2DWithBiasAndRelu(scope *Scope, input tf.Output, filter tf.Output, bias tf.Output, min_input tf.Output, max_input tf.Output, min_filter tf.Output, max_filter tf.Output, strides []int64, padding string, optional ...QuantizedDepthwiseConv2DWithBiasAndReluAttr) (output tf.Output, min_output tf.Output, max_output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizedDepthwiseConv2DWithBiasAndRelu",
+		Input: []tf.Input{
+			input, filter, bias, min_input, max_input, min_filter, max_filter,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// MergeV2CheckpointsAttr is an optional argument to MergeV2Checkpoints.
+type MergeV2CheckpointsAttr func(optionalAttr)
+
+// MergeV2CheckpointsDeleteOldDirs sets the optional delete_old_dirs attribute to value.
+//
+// value: see above.
+// If not specified, defaults to true
+func MergeV2CheckpointsDeleteOldDirs(value bool) MergeV2CheckpointsAttr {
+	return func(m optionalAttr) {
+		m["delete_old_dirs"] = value
+	}
+}
+
+// V2 format specific: merges the metadata files of sharded checkpoints.  The
+//
+// result is one logical checkpoint, with one physical metadata file and renamed
+// data files.
+//
+// Intended for "grouping" multiple checkpoints in a sharded checkpoint setup.
+//
+// If delete_old_dirs is true, attempts to delete recursively the dirname of each
+// path in the input checkpoint_prefixes.  This is useful when those paths are non
+// user-facing temporary locations.
+//
+// Arguments:
+//	checkpoint_prefixes: prefixes of V2 checkpoints to merge.
+//	destination_prefix: scalar.  The desired final prefix.  Allowed to be the same
+// as one of the checkpoint_prefixes.
+//
+// Returns the created operation.
+func MergeV2Checkpoints(scope *Scope, checkpoint_prefixes tf.Output, destination_prefix tf.Output, optional ...MergeV2CheckpointsAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MergeV2Checkpoints",
+		Input: []tf.Input{
+			checkpoint_prefixes, destination_prefix,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// QuantizedDepthwiseConv2DWithBiasAttr is an optional argument to QuantizedDepthwiseConv2DWithBias.
+type QuantizedDepthwiseConv2DWithBiasAttr func(optionalAttr)
+
+// QuantizedDepthwiseConv2DWithBiasOutType sets the optional out_type attribute to value.
+//
+// value: The type of the output.
+// If not specified, defaults to DT_QINT32
+func QuantizedDepthwiseConv2DWithBiasOutType(value tf.DataType) QuantizedDepthwiseConv2DWithBiasAttr {
+	return func(m optionalAttr) {
+		m["out_type"] = value
+	}
+}
+
+// QuantizedDepthwiseConv2DWithBiasDilations sets the optional dilations attribute to value.
+//
+// value: List of dilation values.
+// If not specified, defaults to <i:1 i:1 i:1 i:1 >
+func QuantizedDepthwiseConv2DWithBiasDilations(value []int64) QuantizedDepthwiseConv2DWithBiasAttr {
+	return func(m optionalAttr) {
+		m["dilations"] = value
+	}
+}
+
+// Computes quantized depthwise Conv2D with Bias.
+//
+// Arguments:
+//	input: The original input tensor.
+//	filter: The original filter tensor.
+//	bias: The original bias tensor.
+//	min_input: The float value that the minimum quantized input value represents.
+//	max_input: The float value that the maximum quantized input value represents.
+//	min_filter: The float value that the minimum quantized filter value represents.
+//	max_filter: The float value that the maximum quantized filter value represents.
+//	strides: List of stride values.
+//
+//
+// Returns:
+//	output: The output tensor.
+//	min_output: The float value that the minimum quantized output value represents.
+//	max_output: The float value that the maximum quantized output value represents.
+func QuantizedDepthwiseConv2DWithBias(scope *Scope, input tf.Output, filter tf.Output, bias tf.Output, min_input tf.Output, max_input tf.Output, min_filter tf.Output, max_filter tf.Output, strides []int64, padding string, optional ...QuantizedDepthwiseConv2DWithBiasAttr) (output tf.Output, min_output tf.Output, max_output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizedDepthwiseConv2DWithBias",
+		Input: []tf.Input{
+			input, filter, bias, min_input, max_input, min_filter, max_filter,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// QuantizedDepthwiseConv2DAttr is an optional argument to QuantizedDepthwiseConv2D.
+type QuantizedDepthwiseConv2DAttr func(optionalAttr)
+
+// QuantizedDepthwiseConv2DOutType sets the optional out_type attribute to value.
+//
+// value: The type of the output.
+// If not specified, defaults to DT_QINT32
+func QuantizedDepthwiseConv2DOutType(value tf.DataType) QuantizedDepthwiseConv2DAttr {
+	return func(m optionalAttr) {
+		m["out_type"] = value
+	}
+}
+
+// QuantizedDepthwiseConv2DDilations sets the optional dilations attribute to value.
+//
+// value: List of dilation values.
+// If not specified, defaults to <i:1 i:1 i:1 i:1 >
+func QuantizedDepthwiseConv2DDilations(value []int64) QuantizedDepthwiseConv2DAttr {
+	return func(m optionalAttr) {
+		m["dilations"] = value
+	}
+}
+
+// Computes quantized depthwise Conv2D.
+//
+// Arguments:
+//	input: The original input tensor.
+//	filter: The original filter tensor.
+//	min_input: The float value that the minimum quantized input value represents.
+//	max_input: The float value that the maximum quantized input value represents.
+//	min_filter: The float value that the minimum quantized filter value represents.
+//	max_filter: The float value that the maximum quantized filter value represents.
+//	strides: List of stride values.
+//
+//
+// Returns:
+//	output: The output tensor.
+//	min_output: The float value that the minimum quantized output value represents.
+//	max_output: The float value that the maximum quantized output value represents.
+func QuantizedDepthwiseConv2D(scope *Scope, input tf.Output, filter tf.Output, min_input tf.Output, max_input tf.Output, min_filter tf.Output, max_filter tf.Output, strides []int64, padding string, optional ...QuantizedDepthwiseConv2DAttr) (output tf.Output, min_output tf.Output, max_output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizedDepthwiseConv2D",
+		Input: []tf.Input{
+			input, filter, min_input, max_input, min_filter, max_filter,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// DecodeAndCropJpegAttr is an optional argument to DecodeAndCropJpeg.
+type DecodeAndCropJpegAttr func(optionalAttr)
+
+// DecodeAndCropJpegChannels sets the optional channels attribute to value.
+//
+// value: Number of color channels for the decoded image.
+// If not specified, defaults to 0
+func DecodeAndCropJpegChannels(value int64) DecodeAndCropJpegAttr {
+	return func(m optionalAttr) {
+		m["channels"] = value
+	}
+}
+
+// DecodeAndCropJpegRatio sets the optional ratio attribute to value.
+//
+// value: Downscaling ratio.
+// If not specified, defaults to 1
+func DecodeAndCropJpegRatio(value int64) DecodeAndCropJpegAttr {
+	return func(m optionalAttr) {
+		m["ratio"] = value
+	}
+}
+
+// DecodeAndCropJpegFancyUpscaling sets the optional fancy_upscaling attribute to value.
+//
+// value: If true use a slower but nicer upscaling of the
+// chroma planes (yuv420/422 only).
+// If not specified, defaults to true
+func DecodeAndCropJpegFancyUpscaling(value bool) DecodeAndCropJpegAttr {
+	return func(m optionalAttr) {
+		m["fancy_upscaling"] = value
+	}
+}
+
+// DecodeAndCropJpegTryRecoverTruncated sets the optional try_recover_truncated attribute to value.
+//
+// value: If true try to recover an image from truncated input.
+// If not specified, defaults to false
+func DecodeAndCropJpegTryRecoverTruncated(value bool) DecodeAndCropJpegAttr {
+	return func(m optionalAttr) {
+		m["try_recover_truncated"] = value
+	}
+}
+
+// DecodeAndCropJpegAcceptableFraction sets the optional acceptable_fraction attribute to value.
+//
+// value: The minimum required fraction of lines before a truncated
+// input is accepted.
+// If not specified, defaults to 1
+func DecodeAndCropJpegAcceptableFraction(value float32) DecodeAndCropJpegAttr {
+	return func(m optionalAttr) {
+		m["acceptable_fraction"] = value
+	}
+}
+
+// DecodeAndCropJpegDctMethod sets the optional dct_method attribute to value.
+//
+// value: string specifying a hint about the algorithm used for
+// decompression.  Defaults to "" which maps to a system-specific
+// default.  Currently valid values are ["INTEGER_FAST",
+// "INTEGER_ACCURATE"].  The hint may be ignored (e.g., the internal
+// jpeg library changes to a version that does not have that specific
+// option.)
+// If not specified, defaults to ""
+func DecodeAndCropJpegDctMethod(value string) DecodeAndCropJpegAttr {
+	return func(m optionalAttr) {
+		m["dct_method"] = value
+	}
+}
+
+// Decode and Crop a JPEG-encoded image to a uint8 tensor.
+//
+// The attr `channels` indicates the desired number of color channels for the
+// decoded image.
+//
+// Accepted values are:
+//
+// *   0: Use the number of channels in the JPEG-encoded image.
+// *   1: output a grayscale image.
+// *   3: output an RGB image.
+//
+// If needed, the JPEG-encoded image is transformed to match the requested number
+// of color channels.
+//
+// The attr `ratio` allows downscaling the image by an integer factor during
+// decoding.  Allowed values are: 1, 2, 4, and 8.  This is much faster than
+// downscaling the image later.
+//
+//
+// It is equivalent to a combination of decode and crop, but much faster by only
+// decoding partial jpeg image.
+//
+// Arguments:
+//	contents: 0-D.  The JPEG-encoded image.
+//	crop_window: 1-D.  The crop window: [crop_y, crop_x, crop_height, crop_width].
+//
+// Returns 3-D with shape `[height, width, channels]`..
+func DecodeAndCropJpeg(scope *Scope, contents tf.Output, crop_window tf.Output, optional ...DecodeAndCropJpegAttr) (image tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DecodeAndCropJpeg",
+		Input: []tf.Input{
+			contents, crop_window,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// QuantizedConv2DPerChannelAttr is an optional argument to QuantizedConv2DPerChannel.
+type QuantizedConv2DPerChannelAttr func(optionalAttr)
+
+// QuantizedConv2DPerChannelOutType sets the optional out_type attribute to value.
+//
+// value: The quantized type of output tensor that needs to be converted.
+// If not specified, defaults to DT_QINT32
+func QuantizedConv2DPerChannelOutType(value tf.DataType) QuantizedConv2DPerChannelAttr {
+	return func(m optionalAttr) {
+		m["out_type"] = value
+	}
+}
+
+// QuantizedConv2DPerChannelDilations sets the optional dilations attribute to value.
+//
+// value: list of dilation values.
+// If not specified, defaults to <i:1 i:1 i:1 i:1 >
+func QuantizedConv2DPerChannelDilations(value []int64) QuantizedConv2DPerChannelAttr {
+	return func(m optionalAttr) {
+		m["dilations"] = value
+	}
+}
+
+// Computes QuantizedConv2D per channel.
+//
+// Arguments:
+//	input: The original input tensor.
+//	filter: The original filter tensor.
+//	min_input: The minimum value of the input tensor
+//	max_input: The maximum value of the input tensor.
+//	min_filter: The minimum value of the filter tensor.
+//	max_filter: The maximum value of the filter tensor.
+//	strides: list of stride values.
+//
+//
+// Returns:
+//	output: The output tensor.
+//	min_output: The minimum value of the final output tensor.
+//	max_output: The maximum value of the final output tensor.
+func QuantizedConv2DPerChannel(scope *Scope, input tf.Output, filter tf.Output, min_input tf.Output, max_input tf.Output, min_filter tf.Output, max_filter tf.Output, strides []int64, padding string, optional ...QuantizedConv2DPerChannelAttr) (output tf.Output, min_output tf.Output, max_output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizedConv2DPerChannel",
+		Input: []tf.Input{
+			input, filter, min_input, max_input, min_filter, max_filter,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Concatenates quantized tensors along one dimension.
+//
+// Arguments:
+//	concat_dim: 0-D.  The dimension along which to concatenate.  Must be in the
+// range [0, rank(values)).
+//	values: The `N` Tensors to concatenate. Their ranks and types must match,
+// and their sizes must match in all dimensions except `concat_dim`.
+//	input_mins: The minimum scalar values for each of the input tensors.
+//	input_maxes: The maximum scalar values for each of the input tensors.
+//
+// Returns:
+//	output: A `Tensor` with the concatenation of values stacked along the
+// `concat_dim` dimension.  This tensor's shape matches that of `values` except
+// in `concat_dim` where it has the sum of the sizes.
+//	output_min: The float value that the minimum quantized output value represents.
+//	output_max: The float value that the maximum quantized output value represents.
+func QuantizedConcat(scope *Scope, concat_dim tf.Output, values []tf.Output, input_mins []tf.Output, input_maxes []tf.Output) (output tf.Output, output_min tf.Output, output_max tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizedConcat",
+		Input: []tf.Input{
+			concat_dim, tf.OutputList(values), tf.OutputList(input_mins), tf.OutputList(input_maxes),
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Returns the batched diagonal part of a batched tensor.
+//
+// Returns a tensor with the `k[0]`-th to `k[1]`-th diagonals of the batched
+// `input`.
+//
+// Assume `input` has `r` dimensions `[I, J, ..., L, M, N]`.
+// Let `max_diag_len` be the maximum length among all diagonals to be extracted,
+// `max_diag_len = min(M + min(k[1], 0), N + min(-k[0], 0))`
+// Let `num_diags` be the number of diagonals to extract,
+// `num_diags = k[1] - k[0] + 1`.
+//
+// If `num_diags == 1`, the output tensor is of rank `r - 1` with shape
+// `[I, J, ..., L, max_diag_len]` and values:
+//
+// ```
+// diagonal[i, j, ..., l, n]
+//   = input[i, j, ..., l, n+y, n+x] ; if 0 <= n+y < M and 0 <= n+x < N,
+//     padding_value                 ; otherwise.
+// ```
+// where `y = max(-k[1], 0)`, `x = max(k[1], 0)`.
+//
+// Otherwise, the output tensor has rank `r` with dimensions
+// `[I, J, ..., L, num_diags, max_diag_len]` with values:
+//
+// ```
+// diagonal[i, j, ..., l, m, n]
+//   = input[i, j, ..., l, n+y, n+x] ; if 0 <= n+y < M and 0 <= n+x < N,
+//     padding_value                 ; otherwise.
+// ```
+// where `d = k[1] - m`, `y = max(-d, 0)`, and `x = max(d, 0)`.
+//
+// The input must be at least a matrix.
+//
+// For example:
+//
+// ```
+// input = np.array([[[1, 2, 3, 4],  # Input shape: (2, 3, 4)
+//                    [5, 6, 7, 8],
+//                    [9, 8, 7, 6]],
+//                   [[5, 4, 3, 2],
+//                    [1, 2, 3, 4],
+//                    [5, 6, 7, 8]]])
+//
+// # A main diagonal from each batch.
+// tf.matrix_diag_part(input) ==> [[1, 6, 7],  # Output shape: (2, 3)
+//                                 [5, 2, 7]]
+//
+// # A superdiagonal from each batch.
+// tf.matrix_diag_part(input, k = 1)
+//   ==> [[2, 7, 6],  # Output shape: (2, 3)
+//        [4, 3, 8]]
+//
+// # A tridiagonal band from each batch.
+// tf.matrix_diag_part(input, k = (-1, 1))
+//   ==> [[[2, 7, 6],  # Output shape: (2, 3, 3)
+//         [1, 6, 7],
+//         [5, 8, 0]],
+//        [[4, 3, 8],
+//         [5, 2, 7],
+//         [1, 6, 0]]]
+//
+// # Padding value = 9
+// tf.matrix_diag_part(input, k = (1, 3), padding_value = 9)
+//   ==> [[[4, 9, 9],  # Output shape: (2, 3, 3)
+//         [3, 8, 9],
+//         [2, 7, 6]],
+//        [[2, 9, 9],
+//         [3, 4, 9],
+//         [4, 3, 8]]]
+// ```
+//
+// Arguments:
+//	input: Rank `r` tensor where `r >= 2`.
+//	k: Diagonal offset(s). Positive value means superdiagonal, 0 refers to the main
+// diagonal, and negative value means subdiagonals. `k` can be a single integer
+// (for a single diagonal) or a pair of integers specifying the low and high ends
+// of a matrix band. `k[0]` must not be larger than `k[1]`.
+//	padding_value: The value to fill the area outside the specified diagonal band with.
+// Default is 0.
+//
+// Returns The extracted diagonal(s).
+func MatrixDiagPartV2(scope *Scope, input tf.Output, k tf.Output, padding_value tf.Output) (diagonal tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "MatrixDiagPartV2",
+		Input: []tf.Input{
+			input, k, padding_value,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// A container for a multi device iterator resource.
+//
+// Returns:
+//	handle: A handle to a multi device iterator that can be passed to a
+// "MultiDeviceIteratorGetNextFromShard" op. In contrast to MultiDeviceIterator,
+// AnonymousIterator prevents resource sharing by name, and does not keep a
+// reference to the resource container.
+//	deleter: A variant deleter that should be passed into the op that deletes the iterator.
+func AnonymousMultiDeviceIterator(scope *Scope, devices []string, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output, deleter tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"devices": devices, "output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "AnonymousMultiDeviceIterator",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Provides the time since epoch in seconds.
+//
+// Returns the timestamp as a `float64` for seconds since the Unix epoch.
+//
+// Note: the timestamp is computed when the op is executed, not when it is added
+// to the graph.
+func Timestamp(scope *Scope) (ts tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Timestamp",
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns the truth value of (x <= y) element-wise.
+//
+// *NOTE*: `LessEqual` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+//
+// Example:
+//
+// ```python
+// x = tf.constant([5, 4, 6])
+// y = tf.constant([5])
+// tf.math.less_equal(x, y) ==> [True, True, False]
+//
+// x = tf.constant([5, 4, 6])
+// y = tf.constant([5, 6, 6])
+// tf.math.less_equal(x, y) ==> [True, True, True]
+// ```
+func LessEqual(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "LessEqual",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// LoadTPUEmbeddingADAMParametersGradAccumDebugAttr is an optional argument to LoadTPUEmbeddingADAMParametersGradAccumDebug.
+type LoadTPUEmbeddingADAMParametersGradAccumDebugAttr func(optionalAttr)
+
+// LoadTPUEmbeddingADAMParametersGradAccumDebugTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func LoadTPUEmbeddingADAMParametersGradAccumDebugTableId(value int64) LoadTPUEmbeddingADAMParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// LoadTPUEmbeddingADAMParametersGradAccumDebugTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingADAMParametersGradAccumDebugTableName(value string) LoadTPUEmbeddingADAMParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// LoadTPUEmbeddingADAMParametersGradAccumDebugConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingADAMParametersGradAccumDebugConfig(value string) LoadTPUEmbeddingADAMParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Load ADAM embedding parameters with debug support.
+//
+// An op that loads optimization parameters into HBM for embedding. Must be
+// preceded by a ConfigureTPUEmbeddingHost op that sets up the correct
+// embedding table configuration. For example, this op is used to install
+// parameters that are loaded from a checkpoint before a training loop is
+// executed.
+//
+// Arguments:
+//	parameters: Value of parameters used in the ADAM optimization algorithm.
+//	momenta: Value of momenta used in the ADAM optimization algorithm.
+//	velocities: Value of velocities used in the ADAM optimization algorithm.
+//	gradient_accumulators: Value of gradient_accumulators used in the ADAM optimization algorithm.
+//
+//
+//
+// Returns the created operation.
+func LoadTPUEmbeddingADAMParametersGradAccumDebug(scope *Scope, parameters tf.Output, momenta tf.Output, velocities tf.Output, gradient_accumulators tf.Output, num_shards int64, shard_id int64, optional ...LoadTPUEmbeddingADAMParametersGradAccumDebugAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LoadTPUEmbeddingADAMParametersGradAccumDebug",
+		Input: []tf.Input{
+			parameters, momenta, velocities, gradient_accumulators,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// RetrieveTPUEmbeddingRMSPropParametersAttr is an optional argument to RetrieveTPUEmbeddingRMSPropParameters.
+type RetrieveTPUEmbeddingRMSPropParametersAttr func(optionalAttr)
+
+// RetrieveTPUEmbeddingRMSPropParametersTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func RetrieveTPUEmbeddingRMSPropParametersTableId(value int64) RetrieveTPUEmbeddingRMSPropParametersAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingRMSPropParametersTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingRMSPropParametersTableName(value string) RetrieveTPUEmbeddingRMSPropParametersAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingRMSPropParametersConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingRMSPropParametersConfig(value string) RetrieveTPUEmbeddingRMSPropParametersAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Retrieve RMSProp embedding parameters.
+//
+// An op that retrieves optimization parameters from embedding to host
+// memory. Must be preceded by a ConfigureTPUEmbeddingHost op that sets up
+// the correct embedding table configuration. For example, this op is
+// used to retrieve updated parameters before saving a checkpoint.
+//
+// Returns:
+//	parameters: Parameter parameters updated by the RMSProp optimization algorithm.
+//	ms: Parameter ms updated by the RMSProp optimization algorithm.
+//	mom: Parameter mom updated by the RMSProp optimization algorithm.
+func RetrieveTPUEmbeddingRMSPropParameters(scope *Scope, num_shards int64, shard_id int64, optional ...RetrieveTPUEmbeddingRMSPropParametersAttr) (parameters tf.Output, ms tf.Output, mom tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RetrieveTPUEmbeddingRMSPropParameters",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Conv3DBackpropInputV2Attr is an optional argument to Conv3DBackpropInputV2.
+type Conv3DBackpropInputV2Attr func(optionalAttr)
+
+// Conv3DBackpropInputV2DataFormat sets the optional data_format attribute to value.
+//
+// value: The data format of the input and output data. With the
+// default format "NDHWC", the data is stored in the order of:
+//     [batch, in_depth, in_height, in_width, in_channels].
+// Alternatively, the format could be "NCDHW", the data storage order is:
+//     [batch, in_channels, in_depth, in_height, in_width].
+// If not specified, defaults to "NDHWC"
+func Conv3DBackpropInputV2DataFormat(value string) Conv3DBackpropInputV2Attr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// Conv3DBackpropInputV2Dilations sets the optional dilations attribute to value.
+//
+// value: 1-D tensor of length 5.  The dilation factor for each dimension of
+// `input`. If set to k > 1, there will be k-1 skipped cells between each
+// filter element on that dimension. The dimension order is determined by the
+// value of `data_format`, see above for details. Dilations in the batch and
+// depth dimensions must be 1.
+// If not specified, defaults to <i:1 i:1 i:1 i:1 i:1 >
+func Conv3DBackpropInputV2Dilations(value []int64) Conv3DBackpropInputV2Attr {
+	return func(m optionalAttr) {
+		m["dilations"] = value
+	}
+}
+
+// Computes the gradients of 3-D convolution with respect to the input.
+//
+// Arguments:
+//	input_sizes: An integer vector representing the tensor shape of `input`,
+// where `input` is a 5-D
+// `[batch, depth, rows, cols, in_channels]` tensor.
+//	filter: Shape `[depth, rows, cols, in_channels, out_channels]`.
+// `in_channels` must match between `input` and `filter`.
+//	out_backprop: Backprop signal of shape `[batch, out_depth, out_rows, out_cols,
+// out_channels]`.
+//	strides: 1-D tensor of length 5. The stride of the sliding window for each
+// dimension of `input`. Must have `strides[0] = strides[4] = 1`.
+//	padding: The type of padding algorithm to use.
+func Conv3DBackpropInputV2(scope *Scope, input_sizes tf.Output, filter tf.Output, out_backprop tf.Output, strides []int64, padding string, optional ...Conv3DBackpropInputV2Attr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Conv3DBackpropInputV2",
+		Input: []tf.Input{
+			input_sizes, filter, out_backprop,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// LRNAttr is an optional argument to LRN.
+type LRNAttr func(optionalAttr)
+
+// LRNDepthRadius sets the optional depth_radius attribute to value.
+//
+// value: 0-D.  Half-width of the 1-D normalization window.
+// If not specified, defaults to 5
+func LRNDepthRadius(value int64) LRNAttr {
+	return func(m optionalAttr) {
+		m["depth_radius"] = value
+	}
+}
+
+// LRNBias sets the optional bias attribute to value.
+//
+// value: An offset (usually positive to avoid dividing by 0).
+// If not specified, defaults to 1
+func LRNBias(value float32) LRNAttr {
+	return func(m optionalAttr) {
+		m["bias"] = value
+	}
+}
+
+// LRNAlpha sets the optional alpha attribute to value.
+//
+// value: A scale factor, usually positive.
+// If not specified, defaults to 1
+func LRNAlpha(value float32) LRNAttr {
+	return func(m optionalAttr) {
+		m["alpha"] = value
+	}
+}
+
+// LRNBeta sets the optional beta attribute to value.
+//
+// value: An exponent.
+// If not specified, defaults to 0.5
+func LRNBeta(value float32) LRNAttr {
+	return func(m optionalAttr) {
+		m["beta"] = value
+	}
+}
+
+// Local Response Normalization.
+//
+// The 4-D `input` tensor is treated as a 3-D array of 1-D vectors (along the last
+// dimension), and each vector is normalized independently.  Within a given vector,
+// each component is divided by the weighted, squared sum of inputs within
+// `depth_radius`.  In detail,
+//
+//     sqr_sum[a, b, c, d] =
+//         sum(input[a, b, c, d - depth_radius : d + depth_radius + 1] ** 2)
+//     output = input / (bias + alpha * sqr_sum) ** beta
+//
+// For details, see [Krizhevsky et al., ImageNet classification with deep
+// convolutional neural networks (NIPS 2012)](http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks).
+//
+// Arguments:
+//	input: 4-D.
+func LRN(scope *Scope, input tf.Output, optional ...LRNAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LRN",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// QuantizedMatMulWithBiasAndReluAndRequantizeAttr is an optional argument to QuantizedMatMulWithBiasAndReluAndRequantize.
+type QuantizedMatMulWithBiasAndReluAndRequantizeAttr func(optionalAttr)
+
+// QuantizedMatMulWithBiasAndReluAndRequantizeToutput sets the optional Toutput attribute to value.
+// If not specified, defaults to DT_QUINT8
+func QuantizedMatMulWithBiasAndReluAndRequantizeToutput(value tf.DataType) QuantizedMatMulWithBiasAndReluAndRequantizeAttr {
+	return func(m optionalAttr) {
+		m["Toutput"] = value
+	}
+}
+
+// QuantizedMatMulWithBiasAndReluAndRequantizeTransposeA sets the optional transpose_a attribute to value.
+//
+// value: If true, `a` is transposed before multiplication.
+// If not specified, defaults to false
+func QuantizedMatMulWithBiasAndReluAndRequantizeTransposeA(value bool) QuantizedMatMulWithBiasAndReluAndRequantizeAttr {
+	return func(m optionalAttr) {
+		m["transpose_a"] = value
+	}
+}
+
+// QuantizedMatMulWithBiasAndReluAndRequantizeTransposeB sets the optional transpose_b attribute to value.
+//
+// value: If true, `b` is transposed before multiplication.
+// If not specified, defaults to false
+func QuantizedMatMulWithBiasAndReluAndRequantizeTransposeB(value bool) QuantizedMatMulWithBiasAndReluAndRequantizeAttr {
+	return func(m optionalAttr) {
+		m["transpose_b"] = value
+	}
+}
+
+// QuantizedMatMulWithBiasAndReluAndRequantizeInputQuantMode sets the optional input_quant_mode attribute to value.
+//
+// value: Input data quantization mode. Either MIN_FIRST(default) or SCALED.
+// If not specified, defaults to "MIN_FIRST"
+func QuantizedMatMulWithBiasAndReluAndRequantizeInputQuantMode(value string) QuantizedMatMulWithBiasAndReluAndRequantizeAttr {
+	return func(m optionalAttr) {
+		m["input_quant_mode"] = value
+	}
+}
+
+// Perform a quantized matrix multiplication of  `a` by the matrix `b` with bias
+// add and relu and requantize fusion.
+//
+// The inputs must be two-dimensional matrices and 1D bias vector. And the inner
+// dimension of `a` (after being transposed if `transpose_a` is non-zero) must
+// match the outer dimension of `b` (after being transposed if `transposed_b` is
+// non-zero). Then do broadcast add operation with bias values on the matrix
+// multiplication result. The bias size must match inner dimension of `b`.  Then do
+// relu activation to get non-negative result. Then do requantize operation to get
+// final uint8 result.
+//
+// Arguments:
+//	a: A matrix to be multiplied. Must be a two-dimensional tensor of type `quint8`.
+//	b: A matrix to be multiplied and must be a two-dimensional tensor of type `qint8`.
+//	bias: A 1D bias tensor with size matching with inner dimension of `b` (after being
+// transposed if `transposed_b` is non-zero).
+//	min_a: The float value that the lowest quantized `a` value represents.
+//	max_a: The float value that the highest quantized `a` value represents.
+//	min_b: The float value that the lowest quantized `b` value represents.
+//	max_b: The float value that the highest quantized `b` value represents.
+//	min_freezed_output: The float value that the highest quantized output value after requantize.
+//
+//
+// Returns:
+//	out
+//	min_out: The float value that the lowest quantized output value represents.
+//	max_out: The float value that the highest quantized output value represents.
+func QuantizedMatMulWithBiasAndReluAndRequantize(scope *Scope, a tf.Output, b tf.Output, bias tf.Output, min_a tf.Output, max_a tf.Output, min_b tf.Output, max_b tf.Output, min_freezed_output tf.Output, max_freezed_output tf.Output, optional ...QuantizedMatMulWithBiasAndReluAndRequantizeAttr) (out tf.Output, min_out tf.Output, max_out tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizedMatMulWithBiasAndReluAndRequantize",
+		Input: []tf.Input{
+			a, b, bias, min_a, max_a, min_b, max_b, min_freezed_output, max_freezed_output,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// QuantizedMatMulWithBiasAndReluAttr is an optional argument to QuantizedMatMulWithBiasAndRelu.
+type QuantizedMatMulWithBiasAndReluAttr func(optionalAttr)
+
+// QuantizedMatMulWithBiasAndReluToutput sets the optional Toutput attribute to value.
+// If not specified, defaults to DT_QINT32
+func QuantizedMatMulWithBiasAndReluToutput(value tf.DataType) QuantizedMatMulWithBiasAndReluAttr {
+	return func(m optionalAttr) {
+		m["Toutput"] = value
+	}
+}
+
+// QuantizedMatMulWithBiasAndReluTransposeA sets the optional transpose_a attribute to value.
+//
+// value: If true, `a` is transposed before multiplication.
+// If not specified, defaults to false
+func QuantizedMatMulWithBiasAndReluTransposeA(value bool) QuantizedMatMulWithBiasAndReluAttr {
+	return func(m optionalAttr) {
+		m["transpose_a"] = value
+	}
+}
+
+// QuantizedMatMulWithBiasAndReluTransposeB sets the optional transpose_b attribute to value.
+//
+// value: If true, `b` is transposed before multiplication.
+// If not specified, defaults to false
+func QuantizedMatMulWithBiasAndReluTransposeB(value bool) QuantizedMatMulWithBiasAndReluAttr {
+	return func(m optionalAttr) {
+		m["transpose_b"] = value
+	}
+}
+
+// QuantizedMatMulWithBiasAndReluInputQuantMode sets the optional input_quant_mode attribute to value.
+//
+// value: Input data quantization mode. Either MIN_FIRST(default) or SCALED.
+// If not specified, defaults to "MIN_FIRST"
+func QuantizedMatMulWithBiasAndReluInputQuantMode(value string) QuantizedMatMulWithBiasAndReluAttr {
+	return func(m optionalAttr) {
+		m["input_quant_mode"] = value
+	}
+}
+
+// Perform a quantized matrix multiplication of  `a` by the matrix `b` with bias
+// add and relu fusion.
+//
+// The inputs must be two-dimensional matrices and 1D bias vector. And the inner
+// dimension of `a` (after being transposed if `transpose_a` is non-zero) must
+// match the outer dimension of `b` (after being transposed if `transposed_b` is
+// non-zero). Then do broadcast add operation with bias values on the matrix
+// multiplication result. The bias size must match inner dimension of `b`. Then do
+// relu activation to get non-negative result.
+//
+// Arguments:
+//	a: A matrix to be multiplied. Must be a two-dimensional tensor of type `quint8`.
+//	b: A matrix to be multiplied and must be a two-dimensional tensor of type `qint8`.
+//	bias: A 1D bias tensor with size matching with inner dimension of `b` (after being
+// transposed if `transposed_b` is non-zero).
+//	min_a: The float value that the lowest quantized `a` value represents.
+//	max_a: The float value that the highest quantized `a` value represents.
+//	min_b: The float value that the lowest quantized `b` value represents.
+//	max_b: The float value that the highest quantized `b` value represents.
+//
+// Returns:
+//	out
+//	min_out: The float value that the lowest quantized output value represents.
+//	max_out: The float value that the highest quantized output value represents.
+func QuantizedMatMulWithBiasAndRelu(scope *Scope, a tf.Output, b tf.Output, bias tf.Output, min_a tf.Output, max_a tf.Output, min_b tf.Output, max_b tf.Output, optional ...QuantizedMatMulWithBiasAndReluAttr) (out tf.Output, min_out tf.Output, max_out tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizedMatMulWithBiasAndRelu",
+		Input: []tf.Input{
+			a, b, bias, min_a, max_a, min_b, max_b,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// QuantizedMatMulWithBiasAttr is an optional argument to QuantizedMatMulWithBias.
+type QuantizedMatMulWithBiasAttr func(optionalAttr)
+
+// QuantizedMatMulWithBiasToutput sets the optional Toutput attribute to value.
+// If not specified, defaults to DT_QINT32
+func QuantizedMatMulWithBiasToutput(value tf.DataType) QuantizedMatMulWithBiasAttr {
+	return func(m optionalAttr) {
+		m["Toutput"] = value
+	}
+}
+
+// QuantizedMatMulWithBiasTransposeA sets the optional transpose_a attribute to value.
+//
+// value: If true, `a` is transposed before multiplication.
+// If not specified, defaults to false
+func QuantizedMatMulWithBiasTransposeA(value bool) QuantizedMatMulWithBiasAttr {
+	return func(m optionalAttr) {
+		m["transpose_a"] = value
+	}
+}
+
+// QuantizedMatMulWithBiasTransposeB sets the optional transpose_b attribute to value.
+//
+// value: If true, `b` is transposed before multiplication.
+// If not specified, defaults to false
+func QuantizedMatMulWithBiasTransposeB(value bool) QuantizedMatMulWithBiasAttr {
+	return func(m optionalAttr) {
+		m["transpose_b"] = value
+	}
+}
+
+// QuantizedMatMulWithBiasInputQuantMode sets the optional input_quant_mode attribute to value.
+//
+// value: Input data quantization mode. Either MIN_FIRST(default) or SCALED.
+// If not specified, defaults to "MIN_FIRST"
+func QuantizedMatMulWithBiasInputQuantMode(value string) QuantizedMatMulWithBiasAttr {
+	return func(m optionalAttr) {
+		m["input_quant_mode"] = value
+	}
+}
+
+// Performs a quantized matrix multiplication of `a` by the matrix `b` with bias
+// add.
+//
+// The inputs must be two-dimensional matrices and 1D bias vector. And the inner
+// dimension of `a` (after being transposed if `transpose_a` is non-zero) must
+// match the outer dimension of `b` (after being transposed if `transposed_b` is
+// non-zero). Then do broadcast add operation with bias values on the matrix
+// multiplication result. The bias size must match inner dimension of `b`.
+//
+// Arguments:
+//	a: A matrix to be multiplied. Must be a two-dimensional tensor of type `quint8`.
+//	b: A matrix to be multiplied and must be a two-dimensional tensor of type `qint8`.
+//	bias: A 1D bias tensor with size matching inner dimension of `b` (after being
+// transposed if `transposed_b` is non-zero).
+//	min_a: The float value that the lowest quantized `a` value represents.
+//	max_a: The float value that the highest quantized `a` value represents.
+//	min_b: The float value that the lowest quantized `b` value represents.
+//	max_b: The float value that the highest quantized `b` value represents.
+//
+// Returns:
+//	out
+//	min_out: The float value that the lowest quantized output value represents.
+//	max_out: The float value that the highest quantized output value represents.
+func QuantizedMatMulWithBias(scope *Scope, a tf.Output, b tf.Output, bias tf.Output, min_a tf.Output, max_a tf.Output, min_b tf.Output, max_b tf.Output, optional ...QuantizedMatMulWithBiasAttr) (out tf.Output, min_out tf.Output, max_out tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizedMatMulWithBias",
+		Input: []tf.Input{
+			a, b, bias, min_a, max_a, min_b, max_b,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// TensorArrayGatherV2Attr is an optional argument to TensorArrayGatherV2.
+type TensorArrayGatherV2Attr func(optionalAttr)
+
+// TensorArrayGatherV2ElementShape sets the optional element_shape attribute to value.
+// If not specified, defaults to <unknown_rank:true >
+func TensorArrayGatherV2ElementShape(value tf.Shape) TensorArrayGatherV2Attr {
+	return func(m optionalAttr) {
+		m["element_shape"] = value
+	}
+}
+
+// Deprecated. Use TensorArrayGatherV3
+//
+// DEPRECATED at GraphDef version 26: Use TensorArrayGatherV3
+func TensorArrayGatherV2(scope *Scope, handle tf.Output, indices tf.Output, flow_in tf.Output, dtype tf.DataType, optional ...TensorArrayGatherV2Attr) (value tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorArrayGatherV2",
+		Input: []tf.Input{
+			handle, indices, flow_in,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RFFT3DAttr is an optional argument to RFFT3D.
+type RFFT3DAttr func(optionalAttr)
+
+// RFFT3DTcomplex sets the optional Tcomplex attribute to value.
+// If not specified, defaults to DT_COMPLEX64
+func RFFT3DTcomplex(value tf.DataType) RFFT3DAttr {
+	return func(m optionalAttr) {
+		m["Tcomplex"] = value
+	}
+}
+
+// 3D real-valued fast Fourier transform.
+//
+// Computes the 3-dimensional discrete Fourier transform of a real-valued signal
+// over the inner-most 3 dimensions of `input`.
+//
+// Since the DFT of a real signal is Hermitian-symmetric, `RFFT3D` only returns the
+// `fft_length / 2 + 1` unique components of the FFT for the inner-most dimension
+// of `output`: the zero-frequency term, followed by the `fft_length / 2`
+// positive-frequency terms.
+//
+// Along each axis `RFFT3D` is computed on, if `fft_length` is smaller than the
+// corresponding dimension of `input`, the dimension is cropped. If it is larger,
+// the dimension is padded with zeros.
+//
+// Arguments:
+//	input: A float32 tensor.
+//	fft_length: An int32 tensor of shape [3]. The FFT length for each dimension.
+//
+// Returns A complex64 tensor of the same rank as `input`. The inner-most 3
+//   dimensions of `input` are replaced with the their 3D Fourier transform. The
+//   inner-most dimension contains `fft_length / 2 + 1` unique frequency
+//   components.
+//
+// @compatibility(numpy)
+// Equivalent to np.fft.rfftn with 3 dimensions.
+// @end_compatibility
+func RFFT3D(scope *Scope, input tf.Output, fft_length tf.Output, optional ...RFFT3DAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RFFT3D",
+		Input: []tf.Input{
+			input, fft_length,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Reorders a SparseTensor into the canonical, row-major ordering.
+//
+// Note that by convention, all sparse ops preserve the canonical ordering along
+// increasing dimension number. The only time ordering can be violated is during
+// manual manipulation of the indices and values vectors to add entries.
+//
+// Reordering does not affect the shape of the SparseTensor.
+//
+// If the tensor has rank `R` and `N` non-empty values, `input_indices` has
+// shape `[N, R]`, input_values has length `N`, and input_shape has length `R`.
+//
+// Arguments:
+//	input_indices: 2-D.  `N x R` matrix with the indices of non-empty values in a
+// SparseTensor, possibly not in canonical ordering.
+//	input_values: 1-D.  `N` non-empty values corresponding to `input_indices`.
+//	input_shape: 1-D.  Shape of the input SparseTensor.
+//
+// Returns:
+//	output_indices: 2-D.  `N x R` matrix with the same indices as input_indices, but
+// in canonical row-major ordering.
+//	output_values: 1-D.  `N` non-empty values corresponding to `output_indices`.
+func SparseReorder(scope *Scope, input_indices tf.Output, input_values tf.Output, input_shape tf.Output) (output_indices tf.Output, output_values tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseReorder",
+		Input: []tf.Input{
+			input_indices, input_values, input_shape,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Generates fingerprint values.
+//
+// Generates fingerprint values of `data`.
+//
+// Fingerprint op considers the first dimension of `data` as the batch dimension,
+// and `output[i]` contains the fingerprint value generated from contents in
+// `data[i, ...]` for all `i`.
+//
+// Fingerprint op writes fingerprint values as byte arrays. For example, the
+// default method `farmhash64` generates a 64-bit fingerprint value at a time.
+// This 8-byte value is written out as an `uint8` array of size 8, in little-endian
+// order.
+//
+// For example, suppose that `data` has data type `DT_INT32` and shape (2, 3, 4),
+// and that the fingerprint method is `farmhash64`. In this case, the output shape
+// is (2, 8), where 2 is the batch dimension size of `data`, and 8 is the size of
+// each fingerprint value in bytes. `output[0, :]` is generated from 12 integers in
+// `data[0, :, :]` and similarly `output[1, :]` is generated from other 12 integers
+// in `data[1, :, :]`.
+//
+// Note that this op fingerprints the raw underlying buffer, and it does not
+// fingerprint Tensor's metadata such as data type and/or shape. For example, the
+// fingerprint values are invariant under reshapes and bitcasts as long as the
+// batch dimension remain the same:
+//
+// ```
+// Fingerprint(data) == Fingerprint(Reshape(data, ...))
+// Fingerprint(data) == Fingerprint(Bitcast(data, ...))
+// ```
+//
+// For string data, one should expect `Fingerprint(data) !=
+// Fingerprint(ReduceJoin(data))` in general.
+//
+// Arguments:
+//	data: Must have rank 1 or higher.
+//	method: Fingerprint method used by this op. Currently available method is
+// `farmhash::fingerprint64`.
+//
+// Returns A two-dimensional `Tensor` of type `tf.uint8`. The first dimension equals to
+// `data`'s first dimension, and the second dimension size depends on the
+// fingerprint algorithm.
+func Fingerprint(scope *Scope, data tf.Output, method tf.Output) (fingerprint tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Fingerprint",
+		Input: []tf.Input{
+			data, method,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// CopyAttr is an optional argument to Copy.
+type CopyAttr func(optionalAttr)
+
+// CopyTensorName sets the optional tensor_name attribute to value.
+//
+// value: The name of the input tensor.
+// If not specified, defaults to ""
+func CopyTensorName(value string) CopyAttr {
+	return func(m optionalAttr) {
+		m["tensor_name"] = value
+	}
+}
+
+// CopyDebugOpsSpec sets the optional debug_ops_spec attribute to value.
+//
+// value: A list of debug op spec (op, url, gated_grpc) for attached debug
+// ops. Each element of the list has the format
+// <debug_op>;<grpc_url>;<gated_grpc>, wherein gated_grpc is boolean represented
+// as 0/1. E.g., "DebugIdentity;grpc://foo:3333;1",
+// "DebugIdentity;file:///tmp/tfdbg_1;0".
+// If not specified, defaults to <>
+func CopyDebugOpsSpec(value []string) CopyAttr {
+	return func(m optionalAttr) {
+		m["debug_ops_spec"] = value
+	}
+}
+
+// Copy a tensor from CPU-to-CPU or GPU-to-GPU.
+//
+// Performs CPU-to-CPU or GPU-to-GPU deep-copying of tensor, depending on the
+// device on which the tensor is allocated.
+// N.B.: If the all downstream attached debug ops are disabled given the current
+// gRPC gating status, the output will simply forward the input tensor without
+// deep-copying. See the documentation of Debug* ops for more details.
+//
+// Unlike the CopyHost Op, this op does not have HostMemory constraint on its
+// input or output.
+//
+// Arguments:
+//	input: Input tensor.
+func Copy(scope *Scope, input tf.Output, optional ...CopyAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Copy",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Updates specified rows 'i' with values 'v'.
+//
+// Computes `x[i, :] = v; return x`.
+//
+// Originally this function is mutative however for compilation we make this
+// operation create / operate on a copy of `x`.
+//
+// Arguments:
+//	x: A tensor of type `T`.
+//	i: A vector. Indices into the left-most dimension of `x`.
+//	v: A `Tensor` of type T. Same dimension sizes as x except the first dimension, which must be the same as i's size.
+//
+// Returns A `Tensor` of type T. An alias of `x`. The content of `y` is undefined if there are duplicates in `i`.
+func InplaceUpdate(scope *Scope, x tf.Output, i tf.Output, v tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "InplaceUpdate",
+		Input: []tf.Input{
+			x, i, v,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Table initializer that takes two tensors for keys and values respectively.
+//
+// Arguments:
+//	table_handle: Handle to a table which will be initialized.
+//	keys: Keys of type Tkey.
+//	values: Values of type Tval.
+//
+// Returns the created operation.
+func InitializeTableV2(scope *Scope, table_handle tf.Output, keys tf.Output, values tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "InitializeTableV2",
+		Input: []tf.Input{
+			table_handle, keys, values,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// BatchToSpace for N-D tensors of type T.
+//
+// This operation reshapes the "batch" dimension 0 into `M + 1` dimensions of shape
+// `block_shape + [batch]`, interleaves these blocks back into the grid defined by
+// the spatial dimensions `[1, ..., M]`, to obtain a result with the same rank as
+// the input.  The spatial dimensions of this intermediate result are then
+// optionally cropped according to `crops` to produce the output.  This is the
+// reverse of SpaceToBatch.  See below for a precise description.
+//
+// Arguments:
+//	input: N-D with shape `input_shape = [batch] + spatial_shape + remaining_shape`,
+// where spatial_shape has M dimensions.
+//	block_shape: 1-D with shape `[M]`, all values must be >= 1.
+//	crops: 2-D with shape `[M, 2]`, all values must be >= 0.
+//   `crops[i] = [crop_start, crop_end]` specifies the amount to crop from input
+//   dimension `i + 1`, which corresponds to spatial dimension `i`.  It is
+//   required that
+//   `crop_start[i] + crop_end[i] <= block_shape[i] * input_shape[i + 1]`.
+//
+// This operation is equivalent to the following steps:
+//
+// 1. Reshape `input` to `reshaped` of shape:
+//      [block_shape[0], ..., block_shape[M-1],
+//       batch / prod(block_shape),
+//       input_shape[1], ..., input_shape[N-1]]
+//
+// 2. Permute dimensions of `reshaped` to produce `permuted` of shape
+//      [batch / prod(block_shape),
+//
+//       input_shape[1], block_shape[0],
+//       ...,
+//       input_shape[M], block_shape[M-1],
+//
+//       input_shape[M+1], ..., input_shape[N-1]]
+//
+// 3. Reshape `permuted` to produce `reshaped_permuted` of shape
+//      [batch / prod(block_shape),
+//
+//       input_shape[1] * block_shape[0],
+//       ...,
+//       input_shape[M] * block_shape[M-1],
+//
+//       input_shape[M+1],
+//       ...,
+//       input_shape[N-1]]
+//
+// 4. Crop the start and end of dimensions `[1, ..., M]` of
+//    `reshaped_permuted` according to `crops` to produce the output of shape:
+//      [batch / prod(block_shape),
+//
+//       input_shape[1] * block_shape[0] - crops[0,0] - crops[0,1],
+//       ...,
+//       input_shape[M] * block_shape[M-1] - crops[M-1,0] - crops[M-1,1],
+//
+//       input_shape[M+1], ..., input_shape[N-1]]
+//
+// Some examples:
+//
+// (1) For the following input of shape `[4, 1, 1, 1]`, `block_shape = [2, 2]`, and
+//     `crops = [[0, 0], [0, 0]]`:
+//
+// ```
+// [[[[1]]], [[[2]]], [[[3]]], [[[4]]]]
+// ```
+//
+// The output tensor has shape `[1, 2, 2, 1]` and value:
+//
+// ```
+// x = [[[[1], [2]], [[3], [4]]]]
+// ```
+//
+// (2) For the following input of shape `[4, 1, 1, 3]`, `block_shape = [2, 2]`, and
+//     `crops = [[0, 0], [0, 0]]`:
+//
+// ```
+// [[[[1, 2, 3]]], [[[4, 5, 6]]], [[[7, 8, 9]]], [[[10, 11, 12]]]]
+// ```
+//
+// The output tensor has shape `[1, 2, 2, 3]` and value:
+//
+// ```
+// x = [[[[1, 2, 3], [4, 5, 6]],
+//       [[7, 8, 9], [10, 11, 12]]]]
+// ```
+//
+// (3) For the following input of shape `[4, 2, 2, 1]`, `block_shape = [2, 2]`, and
+//     `crops = [[0, 0], [0, 0]]`:
+//
+// ```
+// x = [[[[1], [3]], [[9], [11]]],
+//      [[[2], [4]], [[10], [12]]],
+//      [[[5], [7]], [[13], [15]]],
+//      [[[6], [8]], [[14], [16]]]]
+// ```
+//
+// The output tensor has shape `[1, 4, 4, 1]` and value:
+//
+// ```
+// x = [[[[1],   [2],  [3],  [4]],
+//      [[5],   [6],  [7],  [8]],
+//      [[9],  [10], [11],  [12]],
+//      [[13], [14], [15],  [16]]]]
+// ```
+//
+// (4) For the following input of shape `[8, 1, 3, 1]`, `block_shape = [2, 2]`, and
+//     `crops = [[0, 0], [2, 0]]`:
+//
+// ```
+// x = [[[[0], [1], [3]]], [[[0], [9], [11]]],
+//      [[[0], [2], [4]]], [[[0], [10], [12]]],
+//      [[[0], [5], [7]]], [[[0], [13], [15]]],
+//      [[[0], [6], [8]]], [[[0], [14], [16]]]]
+// ```
+//
+// The output tensor has shape `[2, 2, 4, 1]` and value:
+//
+// ```
+// x = [[[[1],   [2],  [3],  [4]],
+//       [[5],   [6],  [7],  [8]]],
+//      [[[9],  [10], [11],  [12]],
+//       [[13], [14], [15],  [16]]]]
+// ```
+func BatchToSpaceND(scope *Scope, input tf.Output, block_shape tf.Output, crops tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "BatchToSpaceND",
+		Input: []tf.Input{
+			input, block_shape, crops,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// FIFOQueueV2Attr is an optional argument to FIFOQueueV2.
+type FIFOQueueV2Attr func(optionalAttr)
+
+// FIFOQueueV2Shapes sets the optional shapes attribute to value.
+//
+// value: The shape of each component in a value. The length of this attr must
+// be either 0 or the same as the length of component_types. If the length of
+// this attr is 0, the shapes of queue elements are not constrained, and
+// only one element may be dequeued at a time.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func FIFOQueueV2Shapes(value []tf.Shape) FIFOQueueV2Attr {
+	return func(m optionalAttr) {
+		m["shapes"] = value
+	}
+}
+
+// FIFOQueueV2Capacity sets the optional capacity attribute to value.
+//
+// value: The upper bound on the number of elements in this queue.
+// Negative numbers mean no limit.
+// If not specified, defaults to -1
+func FIFOQueueV2Capacity(value int64) FIFOQueueV2Attr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// FIFOQueueV2Container sets the optional container attribute to value.
+//
+// value: If non-empty, this queue is placed in the given container.
+// Otherwise, a default container is used.
+// If not specified, defaults to ""
+func FIFOQueueV2Container(value string) FIFOQueueV2Attr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// FIFOQueueV2SharedName sets the optional shared_name attribute to value.
+//
+// value: If non-empty, this queue will be shared under the given name
+// across multiple sessions.
+// If not specified, defaults to ""
+func FIFOQueueV2SharedName(value string) FIFOQueueV2Attr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// A queue that produces elements in first-in first-out order.
+//
+// Arguments:
+//	component_types: The type of each component in a value.
+//
+// Returns The handle to the queue.
+func FIFOQueueV2(scope *Scope, component_types []tf.DataType, optional ...FIFOQueueV2Attr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"component_types": component_types}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "FIFOQueueV2",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Quantized Batch normalization.
+//
+// This op is deprecated and will be removed in the future. Prefer
+// `tf.nn.batch_normalization`.
+//
+// Arguments:
+//	t: A 4D input Tensor.
+//	t_min: The value represented by the lowest quantized input.
+//	t_max: The value represented by the highest quantized input.
+//	m: A 1D mean Tensor with size matching the last dimension of t.
+// This is the first output from tf.nn.moments,
+// or a saved moving average thereof.
+//	m_min: The value represented by the lowest quantized mean.
+//	m_max: The value represented by the highest quantized mean.
+//	v: A 1D variance Tensor with size matching the last dimension of t.
+// This is the second output from tf.nn.moments,
+// or a saved moving average thereof.
+//	v_min: The value represented by the lowest quantized variance.
+//	v_max: The value represented by the highest quantized variance.
+//	beta: A 1D beta Tensor with size matching the last dimension of t.
+// An offset to be added to the normalized tensor.
+//	beta_min: The value represented by the lowest quantized offset.
+//	beta_max: The value represented by the highest quantized offset.
+//	gamma: A 1D gamma Tensor with size matching the last dimension of t.
+// If "scale_after_normalization" is true, this tensor will be multiplied
+// with the normalized tensor.
+//	gamma_min: The value represented by the lowest quantized gamma.
+//	gamma_max: The value represented by the highest quantized gamma.
+//
+//	variance_epsilon: A small float number to avoid dividing by 0.
+//	scale_after_normalization: A bool indicating whether the resulted tensor
+// needs to be multiplied with gamma.
+func QuantizedBatchNormWithGlobalNormalization(scope *Scope, t tf.Output, t_min tf.Output, t_max tf.Output, m tf.Output, m_min tf.Output, m_max tf.Output, v tf.Output, v_min tf.Output, v_max tf.Output, beta tf.Output, beta_min tf.Output, beta_max tf.Output, gamma tf.Output, gamma_min tf.Output, gamma_max tf.Output, out_type tf.DataType, variance_epsilon float32, scale_after_normalization bool) (result tf.Output, result_min tf.Output, result_max tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"out_type": out_type, "variance_epsilon": variance_epsilon, "scale_after_normalization": scale_after_normalization}
+	opspec := tf.OpSpec{
+		Type: "QuantizedBatchNormWithGlobalNormalization",
+		Input: []tf.Input{
+			t, t_min, t_max, m, m_min, m_max, v, v_min, v_max, beta, beta_min, beta_max, gamma, gamma_min, gamma_max,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// ResourceStridedSliceAssignAttr is an optional argument to ResourceStridedSliceAssign.
+type ResourceStridedSliceAssignAttr func(optionalAttr)
+
+// ResourceStridedSliceAssignBeginMask sets the optional begin_mask attribute to value.
+// If not specified, defaults to 0
+func ResourceStridedSliceAssignBeginMask(value int64) ResourceStridedSliceAssignAttr {
+	return func(m optionalAttr) {
+		m["begin_mask"] = value
+	}
+}
+
+// ResourceStridedSliceAssignEndMask sets the optional end_mask attribute to value.
+// If not specified, defaults to 0
+func ResourceStridedSliceAssignEndMask(value int64) ResourceStridedSliceAssignAttr {
+	return func(m optionalAttr) {
+		m["end_mask"] = value
+	}
+}
+
+// ResourceStridedSliceAssignEllipsisMask sets the optional ellipsis_mask attribute to value.
+// If not specified, defaults to 0
+func ResourceStridedSliceAssignEllipsisMask(value int64) ResourceStridedSliceAssignAttr {
+	return func(m optionalAttr) {
+		m["ellipsis_mask"] = value
+	}
+}
+
+// ResourceStridedSliceAssignNewAxisMask sets the optional new_axis_mask attribute to value.
+// If not specified, defaults to 0
+func ResourceStridedSliceAssignNewAxisMask(value int64) ResourceStridedSliceAssignAttr {
+	return func(m optionalAttr) {
+		m["new_axis_mask"] = value
+	}
+}
+
+// ResourceStridedSliceAssignShrinkAxisMask sets the optional shrink_axis_mask attribute to value.
+// If not specified, defaults to 0
+func ResourceStridedSliceAssignShrinkAxisMask(value int64) ResourceStridedSliceAssignAttr {
+	return func(m optionalAttr) {
+		m["shrink_axis_mask"] = value
+	}
+}
+
+// Assign `value` to the sliced l-value reference of `ref`.
+//
+// The values of `value` are assigned to the positions in the variable
+// `ref` that are selected by the slice parameters. The slice parameters
+// `begin, `end`, `strides`, etc. work exactly as in `StridedSlice`.
+//
+// NOTE this op currently does not support broadcasting and so `value`'s
+// shape must be exactly the shape produced by the slice of `ref`.
+//
+// Returns the created operation.
+func ResourceStridedSliceAssign(scope *Scope, ref tf.Output, begin tf.Output, end tf.Output, strides tf.Output, value tf.Output, optional ...ResourceStridedSliceAssignAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceStridedSliceAssign",
+		Input: []tf.Input{
+			ref, begin, end, strides, value,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// QuantizedRelu6Attr is an optional argument to QuantizedRelu6.
+type QuantizedRelu6Attr func(optionalAttr)
+
+// QuantizedRelu6OutType sets the optional out_type attribute to value.
+// If not specified, defaults to DT_QUINT8
+func QuantizedRelu6OutType(value tf.DataType) QuantizedRelu6Attr {
+	return func(m optionalAttr) {
+		m["out_type"] = value
+	}
+}
+
+// Computes Quantized Rectified Linear 6: `min(max(features, 0), 6)`
+//
+// Arguments:
+//
+//	min_features: The float value that the lowest quantized value represents.
+//	max_features: The float value that the highest quantized value represents.
+//
+// Returns:
+//	activations: Has the same output shape as "features".
+//	min_activations: The float value that the lowest quantized value represents.
+//	max_activations: The float value that the highest quantized value represents.
+func QuantizedRelu6(scope *Scope, features tf.Output, min_features tf.Output, max_features tf.Output, optional ...QuantizedRelu6Attr) (activations tf.Output, min_activations tf.Output, max_activations tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizedRelu6",
+		Input: []tf.Input{
+			features, min_features, max_features,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// DataFormatVecPermuteAttr is an optional argument to DataFormatVecPermute.
+type DataFormatVecPermuteAttr func(optionalAttr)
+
+// DataFormatVecPermuteSrcFormat sets the optional src_format attribute to value.
+//
+// value: source data format.
+// If not specified, defaults to "NHWC"
+func DataFormatVecPermuteSrcFormat(value string) DataFormatVecPermuteAttr {
+	return func(m optionalAttr) {
+		m["src_format"] = value
+	}
+}
+
+// DataFormatVecPermuteDstFormat sets the optional dst_format attribute to value.
+//
+// value: destination data format.
+// If not specified, defaults to "NCHW"
+func DataFormatVecPermuteDstFormat(value string) DataFormatVecPermuteAttr {
+	return func(m optionalAttr) {
+		m["dst_format"] = value
+	}
+}
+
+// Returns the permuted vector/tensor in the destination data format given the
+//
+// one in the source data format.
+//
+// Arguments:
+//	x: Vector of size 4 or Tensor of shape (4, 2) in source data format.
+//
+// Returns Vector of size 4 or Tensor of shape (4, 2) in destination data format.
+func DataFormatVecPermute(scope *Scope, x tf.Output, optional ...DataFormatVecPermuteAttr) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DataFormatVecPermute",
+		Input: []tf.Input{
+			x,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Adds Tensor 'bias' to Tensor 'input' for Quantized types.
+//
+// Broadcasts the values of bias on dimensions 0..N-2 of 'input'.
+//
+// Arguments:
+//
+//	bias: A 1D bias Tensor with size matching the last dimension of 'input'.
+//	min_input: The float value that the lowest quantized input value represents.
+//	max_input: The float value that the highest quantized input value represents.
+//	min_bias: The float value that the lowest quantized bias value represents.
+//	max_bias: The float value that the highest quantized bias value represents.
+//
+//
+// Returns:
+//	output
+//	min_out: The float value that the lowest quantized output value represents.
+//	max_out: The float value that the highest quantized output value represents.
+func QuantizedBiasAdd(scope *Scope, input tf.Output, bias tf.Output, min_input tf.Output, max_input tf.Output, min_bias tf.Output, max_bias tf.Output, out_type tf.DataType) (output tf.Output, min_out tf.Output, max_out tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"out_type": out_type}
+	opspec := tf.OpSpec{
+		Type: "QuantizedBiasAdd",
+		Input: []tf.Input{
+			input, bias, min_input, max_input, min_bias, max_bias,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// MutableDenseHashTableV2Attr is an optional argument to MutableDenseHashTableV2.
+type MutableDenseHashTableV2Attr func(optionalAttr)
+
+// MutableDenseHashTableV2Container sets the optional container attribute to value.
+//
+// value: If non-empty, this table is placed in the given container.
+// Otherwise, a default container is used.
+// If not specified, defaults to ""
+func MutableDenseHashTableV2Container(value string) MutableDenseHashTableV2Attr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// MutableDenseHashTableV2SharedName sets the optional shared_name attribute to value.
+//
+// value: If non-empty, this table is shared under the given name across
+// multiple sessions.
+// If not specified, defaults to ""
+func MutableDenseHashTableV2SharedName(value string) MutableDenseHashTableV2Attr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// MutableDenseHashTableV2UseNodeNameSharing sets the optional use_node_name_sharing attribute to value.
+// If not specified, defaults to false
+func MutableDenseHashTableV2UseNodeNameSharing(value bool) MutableDenseHashTableV2Attr {
+	return func(m optionalAttr) {
+		m["use_node_name_sharing"] = value
+	}
+}
+
+// MutableDenseHashTableV2ValueShape sets the optional value_shape attribute to value.
+//
+// value: The shape of each value.
+// If not specified, defaults to <>
+func MutableDenseHashTableV2ValueShape(value tf.Shape) MutableDenseHashTableV2Attr {
+	return func(m optionalAttr) {
+		m["value_shape"] = value
+	}
+}
+
+// MutableDenseHashTableV2InitialNumBuckets sets the optional initial_num_buckets attribute to value.
+//
+// value: The initial number of hash table buckets. Must be a power
+// to 2.
+// If not specified, defaults to 131072
+func MutableDenseHashTableV2InitialNumBuckets(value int64) MutableDenseHashTableV2Attr {
+	return func(m optionalAttr) {
+		m["initial_num_buckets"] = value
+	}
+}
+
+// MutableDenseHashTableV2MaxLoadFactor sets the optional max_load_factor attribute to value.
+//
+// value: The maximum ratio between number of entries and number of
+// buckets before growing the table. Must be between 0 and 1.
+// If not specified, defaults to 0.8
+func MutableDenseHashTableV2MaxLoadFactor(value float32) MutableDenseHashTableV2Attr {
+	return func(m optionalAttr) {
+		m["max_load_factor"] = value
+	}
+}
+
+// Creates an empty hash table that uses tensors as the backing store.
+//
+// It uses "open addressing" with quadratic reprobing to resolve
+// collisions.
+//
+// This op creates a mutable hash table, specifying the type of its keys and
+// values. Each value must be a scalar. Data can be inserted into the table using
+// the insert operations. It does not support the initialization operation.
+//
+// Arguments:
+//	empty_key: The key used to represent empty key buckets internally. Must not
+// be used in insert or lookup operations.
+//
+//	value_dtype: Type of the table values.
+//
+// Returns Handle to a table.
+func MutableDenseHashTableV2(scope *Scope, empty_key tf.Output, deleted_key tf.Output, value_dtype tf.DataType, optional ...MutableDenseHashTableV2Attr) (table_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"value_dtype": value_dtype}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MutableDenseHashTableV2",
+		Input: []tf.Input{
+			empty_key, deleted_key,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// FractionalAvgPoolGradAttr is an optional argument to FractionalAvgPoolGrad.
+type FractionalAvgPoolGradAttr func(optionalAttr)
+
+// FractionalAvgPoolGradOverlapping sets the optional overlapping attribute to value.
+//
+// value: When set to True, it means when pooling, the values at the boundary
+// of adjacent pooling cells are used by both cells. For example:
+//
+// `index  0  1  2  3  4`
+//
+// `value  20 5  16 3  7`
+//
+// If the pooling sequence is [0, 2, 4], then 16, at index 2 will be used twice.
+// The result would be [41/3, 26/3] for fractional avg pooling.
+// If not specified, defaults to false
+func FractionalAvgPoolGradOverlapping(value bool) FractionalAvgPoolGradAttr {
+	return func(m optionalAttr) {
+		m["overlapping"] = value
+	}
+}
+
+// Computes gradient of the FractionalAvgPool function.
+//
+// Unlike FractionalMaxPoolGrad, we don't need to find arg_max for
+// FractionalAvgPoolGrad, we just need to evenly back-propagate each element of
+// out_backprop to those indices that form the same pooling cell. Therefore, we
+// just need to know the shape of original input tensor, instead of the whole
+// tensor.
+//
+// Arguments:
+//	orig_input_tensor_shape: Original input tensor shape for `fractional_avg_pool`
+//	out_backprop: 4-D with shape `[batch, height, width, channels]`.  Gradients
+// w.r.t. the output of `fractional_avg_pool`.
+//	row_pooling_sequence: row pooling sequence, form pooling region with
+// col_pooling_sequence.
+//	col_pooling_sequence: column pooling sequence, form pooling region with
+// row_pooling sequence.
+//
+// Returns 4-D.  Gradients w.r.t. the input of `fractional_avg_pool`.
+func FractionalAvgPoolGrad(scope *Scope, orig_input_tensor_shape tf.Output, out_backprop tf.Output, row_pooling_sequence tf.Output, col_pooling_sequence tf.Output, optional ...FractionalAvgPoolGradAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "FractionalAvgPoolGrad",
+		Input: []tf.Input{
+			orig_input_tensor_shape, out_backprop, row_pooling_sequence, col_pooling_sequence,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// FractionalMaxPoolGradAttr is an optional argument to FractionalMaxPoolGrad.
+type FractionalMaxPoolGradAttr func(optionalAttr)
+
+// FractionalMaxPoolGradOverlapping sets the optional overlapping attribute to value.
+//
+// value: When set to True, it means when pooling, the values at the boundary
+// of adjacent pooling cells are used by both cells. For example:
+//
+// `index  0  1  2  3  4`
+//
+// `value  20 5  16 3  7`
+//
+// If the pooling sequence is [0, 2, 4], then 16, at index 2 will be used twice.
+// The result would be [20, 16] for fractional max pooling.
+// If not specified, defaults to false
+func FractionalMaxPoolGradOverlapping(value bool) FractionalMaxPoolGradAttr {
+	return func(m optionalAttr) {
+		m["overlapping"] = value
+	}
+}
+
+// Computes gradient of the FractionalMaxPool function.
+//
+// Arguments:
+//	orig_input: Original input for `fractional_max_pool`
+//	orig_output: Original output for `fractional_max_pool`
+//	out_backprop: 4-D with shape `[batch, height, width, channels]`.  Gradients
+// w.r.t. the output of `fractional_max_pool`.
+//	row_pooling_sequence: row pooling sequence, form pooling region with
+// col_pooling_sequence.
+//	col_pooling_sequence: column pooling sequence, form pooling region with
+// row_pooling sequence.
+//
+// Returns 4-D.  Gradients w.r.t. the input of `fractional_max_pool`.
+func FractionalMaxPoolGrad(scope *Scope, orig_input tf.Output, orig_output tf.Output, out_backprop tf.Output, row_pooling_sequence tf.Output, col_pooling_sequence tf.Output, optional ...FractionalMaxPoolGradAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "FractionalMaxPoolGrad",
+		Input: []tf.Input{
+			orig_input, orig_output, out_backprop, row_pooling_sequence, col_pooling_sequence,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// NthElementAttr is an optional argument to NthElement.
+type NthElementAttr func(optionalAttr)
+
+// NthElementReverse sets the optional reverse attribute to value.
+//
+// value: When set to True, find the nth-largest value in the vector and vice
+// versa.
+// If not specified, defaults to false
+func NthElementReverse(value bool) NthElementAttr {
+	return func(m optionalAttr) {
+		m["reverse"] = value
+	}
+}
+
+// Finds values of the `n`-th order statistic for the last dimension.
+//
+// If the input is a vector (rank-1), finds the entries which is the nth-smallest
+// value in the vector and outputs their values as scalar tensor.
+//
+// For matrices (resp. higher rank input), computes the entries which is the
+// nth-smallest value in each row (resp. vector along the last dimension). Thus,
+//
+//     values.shape = input.shape[:-1]
+//
+// Arguments:
+//	input: 1-D or higher with last dimension at least `n+1`.
+//	n: 0-D. Position of sorted vector to select along the last dimension (along
+// each row for matrices). Valid range of n is `[0, input.shape[:-1])`
+//
+// Returns The `n`-th order statistic along each last dimensional slice.
+func NthElement(scope *Scope, input tf.Output, n tf.Output, optional ...NthElementAttr) (values tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "NthElement",
+		Input: []tf.Input{
+			input, n,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Pads a tensor.
+//
+// This operation pads `input` according to the `paddings` and `constant_values`
+// you specify. `paddings` is an integer tensor with shape `[Dn, 2]`, where n is
+// the rank of `input`. For each dimension D of `input`, `paddings[D, 0]` indicates
+// how many padding values to add before the contents of `input` in that dimension,
+// and `paddings[D, 1]` indicates how many padding values to add after the contents
+// of `input` in that dimension. `constant_values` is a scalar tensor of the same
+// type as `input` that indicates the value to use for padding `input`.
+//
+// The padded size of each dimension D of the output is:
+//
+// `paddings(D, 0) + input.dim_size(D) + paddings(D, 1)`
+//
+// For example:
+//
+// ```
+// # 't' is [[1, 1], [2, 2]]
+// # 'paddings' is [[1, 1], [2, 2]]
+// # 'constant_values' is 0
+// # rank of 't' is 2
+// pad(t, paddings) ==> [[0, 0, 0, 0, 0, 0]
+//                       [0, 0, 1, 1, 0, 0]
+//                       [0, 0, 2, 2, 0, 0]
+//                       [0, 0, 0, 0, 0, 0]]
+// ```
+func PadV2(scope *Scope, input tf.Output, paddings tf.Output, constant_values tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "PadV2",
+		Input: []tf.Input{
+			input, paddings, constant_values,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes cos of x element-wise.
+//
+//   Given an input tensor, this function computes cosine of every
+//   element in the tensor. Input range is `(-inf, inf)` and
+//   output range is `[-1,1]`. If input lies outside the boundary, `nan`
+//   is returned.
+//
+//   ```python
+//   x = tf.constant([-float("inf"), -9, -0.5, 1, 1.2, 200, 10000, float("inf")])
+//   tf.math.cos(x) ==> [nan -0.91113025 0.87758255 0.5403023 0.36235774 0.48718765 -0.95215535 nan]
+//   ```
+func Cos(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Cos",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// TopKV2Attr is an optional argument to TopKV2.
+type TopKV2Attr func(optionalAttr)
+
+// TopKV2Sorted sets the optional sorted attribute to value.
+//
+// value: If true the resulting `k` elements will be sorted by the values in
+// descending order.
+// If not specified, defaults to true
+func TopKV2Sorted(value bool) TopKV2Attr {
+	return func(m optionalAttr) {
+		m["sorted"] = value
+	}
+}
+
+// Finds values and indices of the `k` largest elements for the last dimension.
+//
+// If the input is a vector (rank-1), finds the `k` largest entries in the vector
+// and outputs their values and indices as vectors.  Thus `values[j]` is the
+// `j`-th largest entry in `input`, and its index is `indices[j]`.
+//
+// For matrices (resp. higher rank input), computes the top `k` entries in each
+// row (resp. vector along the last dimension).  Thus,
+//
+//     values.shape = indices.shape = input.shape[:-1] + [k]
+//
+// If two elements are equal, the lower-index element appears first.
+//
+// Arguments:
+//	input: 1-D or higher with last dimension at least `k`.
+//	k: 0-D.  Number of top elements to look for along the last dimension (along each
+// row for matrices).
+//
+// Returns:
+//	values: The `k` largest elements along each last dimensional slice.
+//	indices: The indices of `values` within the last dimension of `input`.
+func TopKV2(scope *Scope, input tf.Output, k tf.Output, optional ...TopKV2Attr) (values tf.Output, indices tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TopKV2",
+		Input: []tf.Input{
+			input, k,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// TopKAttr is an optional argument to TopK.
+type TopKAttr func(optionalAttr)
+
+// TopKSorted sets the optional sorted attribute to value.
+//
+// value: If true the resulting `k` elements will be sorted by the values in
+// descending order.
+// If not specified, defaults to true
+func TopKSorted(value bool) TopKAttr {
+	return func(m optionalAttr) {
+		m["sorted"] = value
+	}
+}
+
+// Finds values and indices of the `k` largest elements for the last dimension.
+//
+// DEPRECATED at GraphDef version 7: Use TopKV2 instead
+//
+// If the input is a vector (rank-1), finds the `k` largest entries in the vector
+// and outputs their values and indices as vectors.  Thus `values[j]` is the
+// `j`-th largest entry in `input`, and its index is `indices[j]`.
+//
+// For matrices (resp. higher rank input), computes the top `k` entries in each
+// row (resp. vector along the last dimension).  Thus,
+//
+//     values.shape = indices.shape = input.shape[:-1] + [k]
+//
+// If two elements are equal, the lower-index element appears first.
+//
+// If `k` varies dynamically, use `TopKV2` below.
+//
+// Arguments:
+//	input: 1-D or higher with last dimension at least `k`.
+//	k: Number of top elements to look for along the last dimension (along each
+// row for matrices).
+//
+// Returns:
+//	values: The `k` largest elements along each last dimensional slice.
+//	indices: The indices of `values` within the last dimension of `input`.
+func TopK(scope *Scope, input tf.Output, k int64, optional ...TopKAttr) (values tf.Output, indices tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"k": k}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TopK",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Outputs the single element from the given dataset.
+//
+// Arguments:
+//	dataset: A handle to a dataset that contains a single element.
+//
+//
+//
+// Returns The components of the single element of `input`.
+func DatasetToSingleElement(scope *Scope, dataset tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (components []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "DatasetToSingleElement",
+		Input: []tf.Input{
+			dataset,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if components, idx, err = makeOutputList(op, idx, "components"); err != nil {
+		scope.UpdateErr("DatasetToSingleElement", err)
+		return
+	}
+	return components
+}
+
+// Computes softmax cross entropy cost and gradients to backpropagate.
+//
+// Inputs are the logits, not probabilities.
+//
+// Arguments:
+//	features: batch_size x num_classes matrix
+//	labels: batch_size x num_classes matrix
+// The caller must ensure that each batch of labels represents a valid
+// probability distribution.
+//
+// Returns:
+//	loss: Per example loss (batch_size vector).
+//	backprop: backpropagated gradients (batch_size x num_classes matrix).
+func SoftmaxCrossEntropyWithLogits(scope *Scope, features tf.Output, labels tf.Output) (loss tf.Output, backprop tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SoftmaxCrossEntropyWithLogits",
+		Input: []tf.Input{
+			features, labels,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Computes log softmax activations.
+//
+// For each batch `i` and class `j` we have
+//
+//     logsoftmax[i, j] = logits[i, j] - log(sum(exp(logits[i])))
+//
+// Arguments:
+//	logits: 2-D with shape `[batch_size, num_classes]`.
+//
+// Returns Same shape as `logits`.
+func LogSoftmax(scope *Scope, logits tf.Output) (logsoftmax tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "LogSoftmax",
+		Input: []tf.Input{
+			logits,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes softmax activations.
+//
+// For each batch `i` and class `j` we have
+//
+//     $$softmax[i, j] = exp(logits[i, j]) / sum_j(exp(logits[i, j]))$$
+//
+// Arguments:
+//	logits: 2-D with shape `[batch_size, num_classes]`.
+//
+// Returns Same shape as `logits`.
+func Softmax(scope *Scope, logits tf.Output) (softmax tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Softmax",
+		Input: []tf.Input{
+			logits,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes softsign gradients for a softsign operation.
+//
+// Arguments:
+//	gradients: The backpropagated gradients to the corresponding softsign operation.
+//	features: The features passed as input to the corresponding softsign operation.
+//
+// Returns The gradients: `gradients / (1 + abs(features)) ** 2`.
+func SoftsignGrad(scope *Scope, gradients tf.Output, features tf.Output) (backprops tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SoftsignGrad",
+		Input: []tf.Input{
+			gradients, features,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the LSTM cell backward propagation for 1 timestep.
+//
+// This implementation is to be used in conjunction of LSTMBlockCell.
+//
+// Arguments:
+//	x: The input to the LSTM cell, shape (batch_size, num_inputs).
+//	cs_prev: The previous cell state.
+//	h_prev: The previous h state.
+//	w: The weight matrix.
+//	wci: The weight matrix for input gate peephole connection.
+//	wcf: The weight matrix for forget gate peephole connection.
+//	wco: The weight matrix for output gate peephole connection.
+//	b: The bias vector.
+//	i: The input gate.
+//	cs: The cell state before the tanh.
+//	f: The forget gate.
+//	o: The output gate.
+//	ci: The cell input.
+//	co: The cell after the tanh.
+//	cs_grad: The current gradient of cs.
+//	h_grad: The gradient of h vector.
+//	use_peephole: Whether the cell uses peephole connections.
+//
+// Returns:
+//	cs_prev_grad: The gradient of cs to be back-propped.
+//	dicfo: The derivative wrt to [i, cs, f, o].
+//	wci_grad: The gradient for wci to be back-propped.
+//	wcf_grad: The gradient for wcf to be back-propped.
+//	wco_grad: The gradient for wco to be back-propped.
+func LSTMBlockCellGrad(scope *Scope, x tf.Output, cs_prev tf.Output, h_prev tf.Output, w tf.Output, wci tf.Output, wcf tf.Output, wco tf.Output, b tf.Output, i tf.Output, cs tf.Output, f tf.Output, o tf.Output, ci tf.Output, co tf.Output, cs_grad tf.Output, h_grad tf.Output, use_peephole bool) (cs_prev_grad tf.Output, dicfo tf.Output, wci_grad tf.Output, wcf_grad tf.Output, wco_grad tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"use_peephole": use_peephole}
+	opspec := tf.OpSpec{
+		Type: "LSTMBlockCellGrad",
+		Input: []tf.Input{
+			x, cs_prev, h_prev, w, wci, wcf, wco, b, i, cs, f, o, ci, co, cs_grad, h_grad,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3), op.Output(4)
+}
+
+// Computes gradients for the scaled exponential linear (Selu) operation.
+//
+// Arguments:
+//	gradients: The backpropagated gradients to the corresponding Selu operation.
+//	outputs: The outputs of the corresponding Selu operation.
+//
+// Returns The gradients: `gradients * (outputs + scale * alpha)`
+// if outputs < 0, `scale * gradients` otherwise.
+func SeluGrad(scope *Scope, gradients tf.Output, outputs tf.Output) (backprops tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SeluGrad",
+		Input: []tf.Input{
+			gradients, outputs,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes gradients for the exponential linear (Elu) operation.
+//
+// Arguments:
+//	gradients: The backpropagated gradients to the corresponding Elu operation.
+//	outputs: The outputs of the corresponding Elu operation.
+//
+// Returns The gradients: `gradients * (outputs + 1)` if outputs < 0,
+// `gradients` otherwise.
+func EluGrad(scope *Scope, gradients tf.Output, outputs tf.Output) (backprops tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "EluGrad",
+		Input: []tf.Input{
+			gradients, outputs,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// LeakyReluGradAttr is an optional argument to LeakyReluGrad.
+type LeakyReluGradAttr func(optionalAttr)
+
+// LeakyReluGradAlpha sets the optional alpha attribute to value.
+// If not specified, defaults to 0.2
+func LeakyReluGradAlpha(value float32) LeakyReluGradAttr {
+	return func(m optionalAttr) {
+		m["alpha"] = value
+	}
+}
+
+// Computes rectified linear gradients for a LeakyRelu operation.
+//
+// Arguments:
+//	gradients: The backpropagated gradients to the corresponding LeakyRelu operation.
+//	features: The features passed as input to the corresponding LeakyRelu operation,
+// OR the outputs of that operation (both work equivalently).
+//
+// Returns `gradients * (features > 0) + alpha * gradients * (features <= 0)`.
+func LeakyReluGrad(scope *Scope, gradients tf.Output, features tf.Output, optional ...LeakyReluGradAttr) (backprops tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LeakyReluGrad",
+		Input: []tf.Input{
+			gradients, features,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the gradient of morphological 2-D dilation with respect to the filter.
+//
+// Arguments:
+//	input: 4-D with shape `[batch, in_height, in_width, depth]`.
+//	filter: 3-D with shape `[filter_height, filter_width, depth]`.
+//	out_backprop: 4-D with shape `[batch, out_height, out_width, depth]`.
+//	strides: 1-D of length 4. The stride of the sliding window for each dimension of
+// the input tensor. Must be: `[1, stride_height, stride_width, 1]`.
+//	rates: 1-D of length 4. The input stride for atrous morphological dilation.
+// Must be: `[1, rate_height, rate_width, 1]`.
+//	padding: The type of padding algorithm to use.
+//
+// Returns 3-D with shape `[filter_height, filter_width, depth]`.
+func Dilation2DBackpropFilter(scope *Scope, input tf.Output, filter tf.Output, out_backprop tf.Output, strides []int64, rates []int64, padding string) (filter_backprop tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"strides": strides, "rates": rates, "padding": padding}
+	opspec := tf.OpSpec{
+		Type: "Dilation2DBackpropFilter",
+		Input: []tf.Input{
+			input, filter, out_backprop,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Converts the given variant tensor to an iterator and stores it in the given resource.
+//
+// Arguments:
+//	resource_handle: A handle to an iterator resource.
+//	serialized: A variant tensor storing the state of the iterator contained in the
+// resource.
+//
+// Returns the created operation.
+func DeserializeIterator(scope *Scope, resource_handle tf.Output, serialized tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "DeserializeIterator",
+		Input: []tf.Input{
+			resource_handle, serialized,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Computes the gradient for the rsqrt of `x` wrt its input.
+//
+// Specifically, `grad = dy * -0.5 * y^3`, where `y = rsqrt(x)`, and `dy`
+// is the corresponding input gradient.
+func RsqrtGrad(scope *Scope, y tf.Output, dy tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "RsqrtGrad",
+		Input: []tf.Input{
+			y, dy,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MaxPoolWithArgmaxAttr is an optional argument to MaxPoolWithArgmax.
+type MaxPoolWithArgmaxAttr func(optionalAttr)
+
+// MaxPoolWithArgmaxTargmax sets the optional Targmax attribute to value.
+// If not specified, defaults to DT_INT64
+func MaxPoolWithArgmaxTargmax(value tf.DataType) MaxPoolWithArgmaxAttr {
+	return func(m optionalAttr) {
+		m["Targmax"] = value
+	}
+}
+
+// MaxPoolWithArgmaxIncludeBatchInIndex sets the optional include_batch_in_index attribute to value.
+//
+// value: Whether to include batch dimension in flattened index of `argmax`.
+// If not specified, defaults to false
+func MaxPoolWithArgmaxIncludeBatchInIndex(value bool) MaxPoolWithArgmaxAttr {
+	return func(m optionalAttr) {
+		m["include_batch_in_index"] = value
+	}
+}
+
+// Performs max pooling on the input and outputs both max values and indices.
+//
+// The indices in `argmax` are flattened, so that a maximum value at position
+// `[b, y, x, c]` becomes flattened index:
+// `(y * width + x) * channels + c` if `include_batch_in_index` is False;
+// `((b * height + y) * width + x) * channels + c` if `include_batch_in_index` is True.
+//
+// The indices returned are always in `[0, height) x [0, width)` before flattening,
+// even if padding is involved and the mathematically correct answer is outside
+// (either negative or too large).  This is a bug, but fixing it is difficult to do
+// in a safe backwards compatible way, especially due to flattening.
+//
+// Arguments:
+//	input: 4-D with shape `[batch, height, width, channels]`.  Input to pool over.
+//	ksize: The size of the window for each dimension of the input tensor.
+//	strides: The stride of the sliding window for each dimension of the
+// input tensor.
+//	padding: The type of padding algorithm to use.
+//
+// Returns:
+//	output: The max pooled output tensor.
+//	argmax: 4-D.  The flattened indices of the max values chosen for each output.
+func MaxPoolWithArgmax(scope *Scope, input tf.Output, ksize []int64, strides []int64, padding string, optional ...MaxPoolWithArgmaxAttr) (output tf.Output, argmax tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"ksize": ksize, "strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MaxPoolWithArgmax",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// MaxPoolGradGradAttr is an optional argument to MaxPoolGradGrad.
+type MaxPoolGradGradAttr func(optionalAttr)
+
+// MaxPoolGradGradDataFormat sets the optional data_format attribute to value.
+//
+// value: Specify the data format of the input and output data. With the
+// default format "NHWC", the data is stored in the order of:
+//     [batch, in_height, in_width, in_channels].
+// Alternatively, the format could be "NCHW", the data storage order of:
+//     [batch, in_channels, in_height, in_width].
+// If not specified, defaults to "NHWC"
+func MaxPoolGradGradDataFormat(value string) MaxPoolGradGradAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// Computes second-order gradients of the maxpooling function.
+//
+// Arguments:
+//	orig_input: The original input tensor.
+//	orig_output: The original output tensor.
+//	grad: 4-D.  Gradients of gradients w.r.t. the input of `max_pool`.
+//	ksize: The size of the window for each dimension of the input tensor.
+//	strides: The stride of the sliding window for each dimension of the
+// input tensor.
+//	padding: The type of padding algorithm to use.
+//
+// Returns Gradients of gradients w.r.t. the input to `max_pool`.
+func MaxPoolGradGrad(scope *Scope, orig_input tf.Output, orig_output tf.Output, grad tf.Output, ksize []int64, strides []int64, padding string, optional ...MaxPoolGradGradAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"ksize": ksize, "strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MaxPoolGradGrad",
+		Input: []tf.Input{
+			orig_input, orig_output, grad,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MaxPoolGradV2Attr is an optional argument to MaxPoolGradV2.
+type MaxPoolGradV2Attr func(optionalAttr)
+
+// MaxPoolGradV2DataFormat sets the optional data_format attribute to value.
+//
+// value: Specify the data format of the input and output data. With the
+// default format "NHWC", the data is stored in the order of:
+//     [batch, in_height, in_width, in_channels].
+// Alternatively, the format could be "NCHW", the data storage order of:
+//     [batch, in_channels, in_height, in_width].
+// If not specified, defaults to "NHWC"
+func MaxPoolGradV2DataFormat(value string) MaxPoolGradV2Attr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// Computes gradients of the maxpooling function.
+//
+// Arguments:
+//	orig_input: The original input tensor.
+//	orig_output: The original output tensor.
+//	grad: 4-D.  Gradients w.r.t. the output of `max_pool`.
+//	ksize: The size of the window for each dimension of the input tensor.
+//	strides: The stride of the sliding window for each dimension of the
+// input tensor.
+//	padding: The type of padding algorithm to use.
+//
+// Returns Gradients w.r.t. the input to `max_pool`.
+func MaxPoolGradV2(scope *Scope, orig_input tf.Output, orig_output tf.Output, grad tf.Output, ksize tf.Output, strides tf.Output, padding string, optional ...MaxPoolGradV2Attr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MaxPoolGradV2",
+		Input: []tf.Input{
+			orig_input, orig_output, grad, ksize, strides,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Concats all tensors in the list along the 0th dimension.
+//
+// Requires that all tensors have the same shape except the first dimension.
+//
+// input_handle: The input list.
+// element_shape: The shape of the uninitialized elements in the list. If the first
+//   dimension is not -1, it is assumed that all list elements have the same
+//   leading dim.
+// leading_dims: The list of leading dims of uninitialized list elements. Used if
+//   the leading dim of input_handle.element_shape or the element_shape input arg
+//   is not already set.
+// tensor: The concated result.
+// lengths: Output tensor containing sizes of the 0th dimension of tensors in the list, used for computing the gradient.
+//
+func TensorListConcatV2(scope *Scope, input_handle tf.Output, element_shape tf.Output, leading_dims tf.Output, element_dtype tf.DataType) (tensor tf.Output, lengths tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"element_dtype": element_dtype}
+	opspec := tf.OpSpec{
+		Type: "TensorListConcatV2",
+		Input: []tf.Input{
+			input_handle, element_shape, leading_dims,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// MaxPoolV2Attr is an optional argument to MaxPoolV2.
+type MaxPoolV2Attr func(optionalAttr)
+
+// MaxPoolV2DataFormat sets the optional data_format attribute to value.
+//
+// value: Specify the data format of the input and output data. With the
+// default format "NHWC", the data is stored in the order of:
+//     [batch, in_height, in_width, in_channels].
+// Alternatively, the format could be "NCHW", the data storage order of:
+//     [batch, in_channels, in_height, in_width].
+// If not specified, defaults to "NHWC"
+func MaxPoolV2DataFormat(value string) MaxPoolV2Attr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// Performs max pooling on the input.
+//
+// Arguments:
+//	input: 4-D input to pool over.
+//	ksize: The size of the window for each dimension of the input tensor.
+//	strides: The stride of the sliding window for each dimension of the
+// input tensor.
+//	padding: The type of padding algorithm to use.
+//
+// Returns The max pooled output tensor.
+func MaxPoolV2(scope *Scope, input tf.Output, ksize tf.Output, strides tf.Output, padding string, optional ...MaxPoolV2Attr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MaxPoolV2",
+		Input: []tf.Input{
+			input, ksize, strides,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SparseReduceSumAttr is an optional argument to SparseReduceSum.
+type SparseReduceSumAttr func(optionalAttr)
+
+// SparseReduceSumKeepDims sets the optional keep_dims attribute to value.
+//
+// value: If true, retain reduced dimensions with length 1.
+// If not specified, defaults to false
+func SparseReduceSumKeepDims(value bool) SparseReduceSumAttr {
+	return func(m optionalAttr) {
+		m["keep_dims"] = value
+	}
+}
+
+// Computes the sum of elements across dimensions of a SparseTensor.
+//
+// This Op takes a SparseTensor and is the sparse counterpart to
+// `tf.reduce_sum()`.  In particular, this Op also returns a dense `Tensor`
+// instead of a sparse one.
+//
+// Reduces `sp_input` along the dimensions given in `reduction_axes`.  Unless
+// `keep_dims` is true, the rank of the tensor is reduced by 1 for each entry in
+// `reduction_axes`. If `keep_dims` is true, the reduced dimensions are retained
+// with length 1.
+//
+// If `reduction_axes` has no entries, all dimensions are reduced, and a tensor
+// with a single element is returned.  Additionally, the axes can be negative,
+// which are interpreted according to the indexing rules in Python.
+//
+// Arguments:
+//	input_indices: 2-D.  `N x R` matrix with the indices of non-empty values in a
+// SparseTensor, possibly not in canonical ordering.
+//	input_values: 1-D.  `N` non-empty values corresponding to `input_indices`.
+//	input_shape: 1-D.  Shape of the input SparseTensor.
+//	reduction_axes: 1-D.  Length-`K` vector containing the reduction axes.
+//
+// Returns `R-K`-D.  The reduced Tensor.
+func SparseReduceSum(scope *Scope, input_indices tf.Output, input_values tf.Output, input_shape tf.Output, reduction_axes tf.Output, optional ...SparseReduceSumAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseReduceSum",
+		Input: []tf.Input{
+			input_indices, input_values, input_shape, reduction_axes,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Compute the Hurwitz zeta function \\(\zeta(x, q)\\).
+//
+// The Hurwitz zeta function is defined as:
+//
+//
+// \\(\zeta(x, q) = \sum_{n=0}^{\infty} (q + n)^{-x}\\)
+func Zeta(scope *Scope, x tf.Output, q tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Zeta",
+		Input: []tf.Input{
+			x, q,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns which elements of x are Inf.
+//
+// @compatibility(numpy)
+// Equivalent to np.isinf
+// @end_compatibility
+//
+// Example:
+//
+// ```python
+// x = tf.constant([5.0, np.inf, 6.8, np.inf])
+// tf.math.is_inf(x) ==> [False, True, False, True]
+// ```
+func IsInf(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "IsInf",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MaxPool3DAttr is an optional argument to MaxPool3D.
+type MaxPool3DAttr func(optionalAttr)
+
+// MaxPool3DDataFormat sets the optional data_format attribute to value.
+//
+// value: The data format of the input and output data. With the
+// default format "NDHWC", the data is stored in the order of:
+//     [batch, in_depth, in_height, in_width, in_channels].
+// Alternatively, the format could be "NCDHW", the data storage order is:
+//     [batch, in_channels, in_depth, in_height, in_width].
+// If not specified, defaults to "NDHWC"
+func MaxPool3DDataFormat(value string) MaxPool3DAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// Performs 3D max pooling on the input.
+//
+// Arguments:
+//	input: Shape `[batch, depth, rows, cols, channels]` tensor to pool over.
+//	ksize: 1-D tensor of length 5. The size of the window for each dimension of
+// the input tensor. Must have `ksize[0] = ksize[4] = 1`.
+//	strides: 1-D tensor of length 5. The stride of the sliding window for each
+// dimension of `input`. Must have `strides[0] = strides[4] = 1`.
+//	padding: The type of padding algorithm to use.
+//
+// Returns The max pooled output tensor.
+func MaxPool3D(scope *Scope, input tf.Output, ksize []int64, strides []int64, padding string, optional ...MaxPool3DAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"ksize": ksize, "strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MaxPool3D",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Deprecated. Use TensorArrayCloseV3
+//
+// DEPRECATED at GraphDef version 26: Use TensorArrayCloseV3
+//
+// Returns the created operation.
+func TensorArrayCloseV2(scope *Scope, handle tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorArrayCloseV2",
+		Input: []tf.Input{
+			handle,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// AvgPool3DGradAttr is an optional argument to AvgPool3DGrad.
+type AvgPool3DGradAttr func(optionalAttr)
+
+// AvgPool3DGradDataFormat sets the optional data_format attribute to value.
+//
+// value: The data format of the input and output data. With the
+// default format "NDHWC", the data is stored in the order of:
+//     [batch, in_depth, in_height, in_width, in_channels].
+// Alternatively, the format could be "NCDHW", the data storage order is:
+//     [batch, in_channels, in_depth, in_height, in_width].
+// If not specified, defaults to "NDHWC"
+func AvgPool3DGradDataFormat(value string) AvgPool3DGradAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// Computes gradients of average pooling function.
+//
+// Arguments:
+//	orig_input_shape: The original input dimensions.
+//	grad: Output backprop of shape `[batch, depth, rows, cols, channels]`.
+//	ksize: 1-D tensor of length 5. The size of the window for each dimension of
+// the input tensor. Must have `ksize[0] = ksize[4] = 1`.
+//	strides: 1-D tensor of length 5. The stride of the sliding window for each
+// dimension of `input`. Must have `strides[0] = strides[4] = 1`.
+//	padding: The type of padding algorithm to use.
+//
+// Returns The backprop for input.
+func AvgPool3DGrad(scope *Scope, orig_input_shape tf.Output, grad tf.Output, ksize []int64, strides []int64, padding string, optional ...AvgPool3DGradAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"ksize": ksize, "strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "AvgPool3DGrad",
+		Input: []tf.Input{
+			orig_input_shape, grad,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Conv3DBackpropFilterAttr is an optional argument to Conv3DBackpropFilter.
+type Conv3DBackpropFilterAttr func(optionalAttr)
+
+// Conv3DBackpropFilterDilations sets the optional dilations attribute to value.
+// If not specified, defaults to <i:1 i:1 i:1 i:1 i:1 >
+func Conv3DBackpropFilterDilations(value []int64) Conv3DBackpropFilterAttr {
+	return func(m optionalAttr) {
+		m["dilations"] = value
+	}
+}
+
+// Computes the gradients of 3-D convolution with respect to the filter.
+//
+// DEPRECATED at GraphDef version 10: Use Conv3DBackpropFilterV2
+//
+// Arguments:
+//	input: Shape `[batch, depth, rows, cols, in_channels]`.
+//	filter: Shape `[depth, rows, cols, in_channels, out_channels]`.
+// `in_channels` must match between `input` and `filter`.
+//	out_backprop: Backprop signal of shape `[batch, out_depth, out_rows, out_cols,
+// out_channels]`.
+//	strides: 1-D tensor of length 5. The stride of the sliding window for each
+// dimension of `input`. Must have `strides[0] = strides[4] = 1`.
+//	padding: The type of padding algorithm to use.
+func Conv3DBackpropFilter(scope *Scope, input tf.Output, filter tf.Output, out_backprop tf.Output, strides []int64, padding string, optional ...Conv3DBackpropFilterAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Conv3DBackpropFilter",
+		Input: []tf.Input{
+			input, filter, out_backprop,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Conv3DAttr is an optional argument to Conv3D.
+type Conv3DAttr func(optionalAttr)
+
+// Conv3DDataFormat sets the optional data_format attribute to value.
+//
+// value: The data format of the input and output data. With the
+// default format "NDHWC", the data is stored in the order of:
+//     [batch, in_depth, in_height, in_width, in_channels].
+// Alternatively, the format could be "NCDHW", the data storage order is:
+//     [batch, in_channels, in_depth, in_height, in_width].
+// If not specified, defaults to "NDHWC"
+func Conv3DDataFormat(value string) Conv3DAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// Conv3DDilations sets the optional dilations attribute to value.
+//
+// value: 1-D tensor of length 5.  The dilation factor for each dimension of
+// `input`. If set to k > 1, there will be k-1 skipped cells between each
+// filter element on that dimension. The dimension order is determined by the
+// value of `data_format`, see above for details. Dilations in the batch and
+// depth dimensions must be 1.
+// If not specified, defaults to <i:1 i:1 i:1 i:1 i:1 >
+func Conv3DDilations(value []int64) Conv3DAttr {
+	return func(m optionalAttr) {
+		m["dilations"] = value
+	}
+}
+
+// Computes a 3-D convolution given 5-D `input` and `filter` tensors.
+//
+// In signal processing, cross-correlation is a measure of similarity of
+// two waveforms as a function of a time-lag applied to one of them. This
+// is also known as a sliding dot product or sliding inner-product.
+//
+// Our Conv3D implements a form of cross-correlation.
+//
+// Arguments:
+//	input: Shape `[batch, in_depth, in_height, in_width, in_channels]`.
+//	filter: Shape `[filter_depth, filter_height, filter_width, in_channels,
+// out_channels]`. `in_channels` must match between `input` and `filter`.
+//	strides: 1-D tensor of length 5. The stride of the sliding window for each
+// dimension of `input`. Must have `strides[0] = strides[4] = 1`.
+//	padding: The type of padding algorithm to use.
+func Conv3D(scope *Scope, input tf.Output, filter tf.Output, strides []int64, padding string, optional ...Conv3DAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Conv3D",
+		Input: []tf.Input{
+			input, filter,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// UniqueWithCountsAttr is an optional argument to UniqueWithCounts.
+type UniqueWithCountsAttr func(optionalAttr)
+
+// UniqueWithCountsOutIdx sets the optional out_idx attribute to value.
+// If not specified, defaults to DT_INT32
+func UniqueWithCountsOutIdx(value tf.DataType) UniqueWithCountsAttr {
+	return func(m optionalAttr) {
+		m["out_idx"] = value
+	}
+}
+
+// Finds unique elements in a 1-D tensor.
+//
+// This operation returns a tensor `y` containing all of the unique elements of `x`
+// sorted in the same order that they occur in `x`. This operation also returns a
+// tensor `idx` the same size as `x` that contains the index of each value of `x`
+// in the unique output `y`. Finally, it returns a third tensor `count` that
+// contains the count of each element of `y` in `x`. In other words:
+//
+// `y[idx[i]] = x[i] for i in [0, 1,...,rank(x) - 1]`
+//
+// For example:
+//
+// ```
+// # tensor 'x' is [1, 1, 2, 4, 4, 4, 7, 8, 8]
+// y, idx, count = unique_with_counts(x)
+// y ==> [1, 2, 4, 7, 8]
+// idx ==> [0, 0, 1, 2, 2, 2, 3, 4, 4]
+// count ==> [2, 1, 3, 1, 2]
+// ```
+//
+// Arguments:
+//	x: 1-D.
+//
+// Returns:
+//	y: 1-D.
+//	idx: 1-D.
+//	count: 1-D.
+func UniqueWithCounts(scope *Scope, x tf.Output, optional ...UniqueWithCountsAttr) (y tf.Output, idx tf.Output, count tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "UniqueWithCounts",
+		Input: []tf.Input{
+			x,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// ResizeBicubicGradAttr is an optional argument to ResizeBicubicGrad.
+type ResizeBicubicGradAttr func(optionalAttr)
+
+// ResizeBicubicGradAlignCorners sets the optional align_corners attribute to value.
+//
+// value: If true, the centers of the 4 corner pixels of the input and grad tensors are
+// aligned. Defaults to false.
+// If not specified, defaults to false
+func ResizeBicubicGradAlignCorners(value bool) ResizeBicubicGradAttr {
+	return func(m optionalAttr) {
+		m["align_corners"] = value
+	}
+}
+
+// ResizeBicubicGradHalfPixelCenters sets the optional half_pixel_centers attribute to value.
+// If not specified, defaults to false
+func ResizeBicubicGradHalfPixelCenters(value bool) ResizeBicubicGradAttr {
+	return func(m optionalAttr) {
+		m["half_pixel_centers"] = value
+	}
+}
+
+// Computes the gradient of bicubic interpolation.
+//
+// Arguments:
+//	grads: 4-D with shape `[batch, height, width, channels]`.
+//	original_image: 4-D with shape `[batch, orig_height, orig_width, channels]`,
+// The image tensor that was resized.
+//
+// Returns 4-D with shape `[batch, orig_height, orig_width, channels]`.
+// Gradients with respect to the input image. Input image must have been
+// float or double.
+func ResizeBicubicGrad(scope *Scope, grads tf.Output, original_image tf.Output, optional ...ResizeBicubicGradAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResizeBicubicGrad",
+		Input: []tf.Input{
+			grads, original_image,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns a list which has the passed-in `Tensor` as last element and the other elements of the given list in `input_handle`.
+//
+// tensor: The tensor to put on the list.
+// input_handle: The old list.
+// output_handle: A list with the elements of the old list followed by tensor.
+// element_dtype: the type of elements in the list.
+// element_shape: a shape compatible with that of elements in the list.
+func TensorListPushBack(scope *Scope, input_handle tf.Output, tensor tf.Output) (output_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorListPushBack",
+		Input: []tf.Input{
+			input_handle, tensor,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns which elements of x are NaN.
+//
+// @compatibility(numpy)
+// Equivalent to np.isnan
+// @end_compatibility
+//
+// Example:
+//
+// ```python
+// x = tf.constant([5.0, np.nan, 6.8, np.nan, np.inf])
+// tf.math.is_nan(x) ==> [False, True, False, True, False]
+// ```
+func IsNan(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "IsNan",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Adds a value to the current value of a variable.
+//
+// Any ReadVariableOp with a control dependency on this op is guaranteed to
+// see the incremented value or a subsequent newer one.
+//
+// Arguments:
+//	resource: handle to the resource in which to store the variable.
+//	value: the value by which the variable will be incremented.
+//
+// Returns the created operation.
+func AssignAddVariableOp(scope *Scope, resource tf.Output, value tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "AssignAddVariableOp",
+		Input: []tf.Input{
+			resource, value,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// DepthwiseConv2dNativeBackpropInputAttr is an optional argument to DepthwiseConv2dNativeBackpropInput.
+type DepthwiseConv2dNativeBackpropInputAttr func(optionalAttr)
+
+// DepthwiseConv2dNativeBackpropInputExplicitPaddings sets the optional explicit_paddings attribute to value.
+// If not specified, defaults to <>
+func DepthwiseConv2dNativeBackpropInputExplicitPaddings(value []int64) DepthwiseConv2dNativeBackpropInputAttr {
+	return func(m optionalAttr) {
+		m["explicit_paddings"] = value
+	}
+}
+
+// DepthwiseConv2dNativeBackpropInputDataFormat sets the optional data_format attribute to value.
+//
+// value: Specify the data format of the input and output data. With the
+// default format "NHWC", the data is stored in the order of:
+//     [batch, height, width, channels].
+// Alternatively, the format could be "NCHW", the data storage order of:
+//     [batch, channels, height, width].
+// If not specified, defaults to "NHWC"
+func DepthwiseConv2dNativeBackpropInputDataFormat(value string) DepthwiseConv2dNativeBackpropInputAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// DepthwiseConv2dNativeBackpropInputDilations sets the optional dilations attribute to value.
+//
+// value: 1-D tensor of length 4.  The dilation factor for each dimension of
+// `input`. If set to k > 1, there will be k-1 skipped cells between each filter
+// element on that dimension. The dimension order is determined by the value of
+// `data_format`, see above for details. Dilations in the batch and depth
+// dimensions must be 1.
+// If not specified, defaults to <i:1 i:1 i:1 i:1 >
+func DepthwiseConv2dNativeBackpropInputDilations(value []int64) DepthwiseConv2dNativeBackpropInputAttr {
+	return func(m optionalAttr) {
+		m["dilations"] = value
+	}
+}
+
+// Computes the gradients of depthwise convolution with respect to the input.
+//
+// Arguments:
+//	input_sizes: An integer vector representing the shape of `input`, based
+// on `data_format`.  For example, if `data_format` is 'NHWC' then
+//  `input` is a 4-D `[batch, height, width, channels]` tensor.
+//	filter: 4-D with shape
+// `[filter_height, filter_width, in_channels, depthwise_multiplier]`.
+//	out_backprop: 4-D with shape  based on `data_format`.
+// For example, if `data_format` is 'NHWC' then
+// out_backprop shape is `[batch, out_height, out_width, out_channels]`.
+// Gradients w.r.t. the output of the convolution.
+//	strides: The stride of the sliding window for each dimension of the input
+// of the convolution.
+//	padding: The type of padding algorithm to use.
+//
+// Returns 4-D with shape according to `data_format`.  For example, if
+// `data_format` is 'NHWC', output shape is `[batch, in_height,
+// in_width, in_channels]`.  Gradient w.r.t. the input of the
+// convolution.
+func DepthwiseConv2dNativeBackpropInput(scope *Scope, input_sizes tf.Output, filter tf.Output, out_backprop tf.Output, strides []int64, padding string, optional ...DepthwiseConv2dNativeBackpropInputAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DepthwiseConv2dNativeBackpropInput",
+		Input: []tf.Input{
+			input_sizes, filter, out_backprop,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Updates the table to associates keys with values.
+//
+// The tensor `keys` must be of the same type as the keys of the table.
+// The tensor `values` must be of the type of the table values.
+//
+// Arguments:
+//	table_handle: Handle to the table.
+//	keys: Any shape.  Keys to look up.
+//	values: Values to associate with keys.
+//
+// Returns the created operation.
+func LookupTableInsertV2(scope *Scope, table_handle tf.Output, keys tf.Output, values tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "LookupTableInsertV2",
+		Input: []tf.Input{
+			table_handle, keys, values,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Component-wise multiplies a SparseTensor by a dense Tensor.
+//
+// The output locations corresponding to the implicitly zero elements in the sparse
+// tensor will be zero (i.e., will not take up storage space), regardless of the
+// contents of the dense tensor (even if it's +/-INF and that INF*0 == NaN).
+//
+// *Limitation*: this Op only broadcasts the dense side to the sparse side, but not
+// the other direction.
+//
+// Arguments:
+//	sp_indices: 2-D.  `N x R` matrix with the indices of non-empty values in a
+// SparseTensor, possibly not in canonical ordering.
+//	sp_values: 1-D.  `N` non-empty values corresponding to `sp_indices`.
+//	sp_shape: 1-D.  Shape of the input SparseTensor.
+//	dense: `R`-D.  The dense Tensor operand.
+//
+// Returns 1-D.  The `N` values that are operated on.
+func SparseDenseCwiseMul(scope *Scope, sp_indices tf.Output, sp_values tf.Output, sp_shape tf.Output, dense tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseDenseCwiseMul",
+		Input: []tf.Input{
+			sp_indices, sp_values, sp_shape, dense,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Performs a padding as a preprocess during a convolution.
+//
+// Similar to FusedResizeAndPadConv2d, this op allows for an optimized
+// implementation where the spatial padding transformation stage is fused with the
+// im2col lookup, but in this case without the bilinear filtering required for
+// resizing. Fusing the padding prevents the need to write out the intermediate
+// results as whole tensors, reducing memory pressure, and we can get some latency
+// gains by merging the transformation calculations.
+// The data_format attribute for Conv2D isn't supported by this op, and 'NHWC'
+// order is used instead.
+// Internally this op uses a single per-graph scratch buffer, which means that it
+// will block if multiple versions are being run in parallel. This is because this
+// operator is primarily an optimization to minimize memory usage.
+//
+// Arguments:
+//	input: 4-D with shape `[batch, in_height, in_width, in_channels]`.
+//	paddings: A two-column matrix specifying the padding sizes. The number of
+// rows must be the same as the rank of `input`.
+//	filter: 4-D with shape
+// `[filter_height, filter_width, in_channels, out_channels]`.
+//
+//	strides: 1-D of length 4.  The stride of the sliding window for each dimension
+// of `input`. Must be in the same order as the dimension specified with format.
+//	padding: The type of padding algorithm to use.
+func FusedPadConv2D(scope *Scope, input tf.Output, paddings tf.Output, filter tf.Output, mode string, strides []int64, padding string) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"mode": mode, "strides": strides, "padding": padding}
+	opspec := tf.OpSpec{
+		Type: "FusedPadConv2D",
+		Input: []tf.Input{
+			input, paddings, filter,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Adjust the hue of one or more images.
+//
+// `images` is a tensor of at least 3 dimensions.  The last dimension is
+// interpreted as channels, and must be three.
+//
+// The input image is considered in the RGB colorspace. Conceptually, the RGB
+// colors are first mapped into HSV. A delta is then applied all the hue values,
+// and then remapped back to RGB colorspace.
+//
+// Arguments:
+//	images: Images to adjust.  At least 3-D.
+//	delta: A float delta to add to the hue.
+//
+// Returns The hue-adjusted image or images.
+func AdjustHue(scope *Scope, images tf.Output, delta tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "AdjustHue",
+		Input: []tf.Input{
+			images, delta,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// List of the given size with empty elements.
+//
+// element_shape: the shape of the future elements of the list
+// num_elements: the number of elements to reserve
+// handle: the output list
+// element_dtype: the desired type of elements in the list.
+func TensorListReserve(scope *Scope, element_shape tf.Output, num_elements tf.Output, element_dtype tf.DataType) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"element_dtype": element_dtype}
+	opspec := tf.OpSpec{
+		Type: "TensorListReserve",
+		Input: []tf.Input{
+			element_shape, num_elements,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Clips tensor values to a specified min and max.
+//
+// Given a tensor `t`, this operation returns a tensor of the same type and
+// shape as `t` with its values clipped to `clip_value_min` and `clip_value_max`.
+// Any values less than `clip_value_min` are set to `clip_value_min`. Any values
+// greater than `clip_value_max` are set to `clip_value_max`.
+//
+// Arguments:
+//	t: A `Tensor`.
+//	clip_value_min: A 0-D (scalar) `Tensor`, or a `Tensor` with the same shape
+// as `t`. The minimum value to clip by.
+//	clip_value_max: A 0-D (scalar) `Tensor`, or a `Tensor` with the same shape
+// as `t`. The maximum value to clip by.
+//
+// Returns A clipped `Tensor` with the same shape as input 't'.
+func ClipByValue(scope *Scope, t tf.Output, clip_value_min tf.Output, clip_value_max tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ClipByValue",
+		Input: []tf.Input{
+			t, clip_value_min, clip_value_max,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Conv2DBackpropFilterAttr is an optional argument to Conv2DBackpropFilter.
+type Conv2DBackpropFilterAttr func(optionalAttr)
+
+// Conv2DBackpropFilterUseCudnnOnGpu sets the optional use_cudnn_on_gpu attribute to value.
+// If not specified, defaults to true
+func Conv2DBackpropFilterUseCudnnOnGpu(value bool) Conv2DBackpropFilterAttr {
+	return func(m optionalAttr) {
+		m["use_cudnn_on_gpu"] = value
+	}
+}
+
+// Conv2DBackpropFilterExplicitPaddings sets the optional explicit_paddings attribute to value.
+//
+// value: If `padding` is `"EXPLICIT"`, the list of explicit padding amounts. For the ith
+// dimension, the amount of padding inserted before and after the dimension is
+// `explicit_paddings[2 * i]` and `explicit_paddings[2 * i + 1]`, respectively. If
+// `padding` is not `"EXPLICIT"`, `explicit_paddings` must be empty.
+// If not specified, defaults to <>
+func Conv2DBackpropFilterExplicitPaddings(value []int64) Conv2DBackpropFilterAttr {
+	return func(m optionalAttr) {
+		m["explicit_paddings"] = value
+	}
+}
+
+// Conv2DBackpropFilterDataFormat sets the optional data_format attribute to value.
+//
+// value: Specify the data format of the input and output data. With the
+// default format "NHWC", the data is stored in the order of:
+//     [batch, in_height, in_width, in_channels].
+// Alternatively, the format could be "NCHW", the data storage order of:
+//     [batch, in_channels, in_height, in_width].
+// If not specified, defaults to "NHWC"
+func Conv2DBackpropFilterDataFormat(value string) Conv2DBackpropFilterAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// Conv2DBackpropFilterDilations sets the optional dilations attribute to value.
+//
+// value: 1-D tensor of length 4.  The dilation factor for each dimension of
+// `input`. If set to k > 1, there will be k-1 skipped cells between each filter
+// element on that dimension. The dimension order is determined by the value of
+// `data_format`, see above for details. Dilations in the batch and depth
+// dimensions must be 1.
+// If not specified, defaults to <i:1 i:1 i:1 i:1 >
+func Conv2DBackpropFilterDilations(value []int64) Conv2DBackpropFilterAttr {
+	return func(m optionalAttr) {
+		m["dilations"] = value
+	}
+}
+
+// Computes the gradients of convolution with respect to the filter.
+//
+// Arguments:
+//	input: 4-D with shape `[batch, in_height, in_width, in_channels]`.
+//	filter_sizes: An integer vector representing the tensor shape of `filter`,
+// where `filter` is a 4-D
+// `[filter_height, filter_width, in_channels, out_channels]` tensor.
+//	out_backprop: 4-D with shape `[batch, out_height, out_width, out_channels]`.
+// Gradients w.r.t. the output of the convolution.
+//	strides: The stride of the sliding window for each dimension of the input
+// of the convolution. Must be in the same order as the dimension specified with
+// format.
+//	padding: The type of padding algorithm to use.
+//
+// Returns 4-D with shape
+// `[filter_height, filter_width, in_channels, out_channels]`.  Gradient w.r.t.
+// the `filter` input of the convolution.
+func Conv2DBackpropFilter(scope *Scope, input tf.Output, filter_sizes tf.Output, out_backprop tf.Output, strides []int64, padding string, optional ...Conv2DBackpropFilterAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Conv2DBackpropFilter",
+		Input: []tf.Input{
+			input, filter_sizes, out_backprop,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns the truth value of x OR y element-wise.
+//
+// *NOTE*: `LogicalOr` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func LogicalOr(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "LogicalOr",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Adds `bias` to `value`.
+//
+// This is a deprecated version of BiasAdd and will be soon removed.
+//
+// This is a special case of `tf.add` where `bias` is restricted to be 1-D.
+// Broadcasting is supported, so `value` may have any number of dimensions.
+//
+// Arguments:
+//	value: Any number of dimensions.
+//	bias: 1-D with size the last dimension of `value`.
+//
+// Returns Broadcasted sum of `value` and `bias`.
+func BiasAddV1(scope *Scope, value tf.Output, bias tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "BiasAddV1",
+		Input: []tf.Input{
+			value, bias,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// BiasAddAttr is an optional argument to BiasAdd.
+type BiasAddAttr func(optionalAttr)
+
+// BiasAddDataFormat sets the optional data_format attribute to value.
+//
+// value: Specify the data format of the input and output data. With the
+// default format "NHWC", the bias tensor will be added to the last dimension
+// of the value tensor.
+// Alternatively, the format could be "NCHW", the data storage order of:
+//     [batch, in_channels, in_height, in_width].
+// The tensor will be added to "in_channels", the third-to-the-last
+//     dimension.
+// If not specified, defaults to "NHWC"
+func BiasAddDataFormat(value string) BiasAddAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// Adds `bias` to `value`.
+//
+// This is a special case of `tf.add` where `bias` is restricted to be 1-D.
+// Broadcasting is supported, so `value` may have any number of dimensions.
+//
+// Arguments:
+//	value: Any number of dimensions.
+//	bias: 1-D with size the last dimension of `value`.
+//
+// Returns Broadcasted sum of `value` and `bias`.
+func BiasAdd(scope *Scope, value tf.Output, bias tf.Output, optional ...BiasAddAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "BiasAdd",
+		Input: []tf.Input{
+			value, bias,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the gradient for the inverse of `x` wrt its input.
+//
+// Specifically, `grad = -dy * y*y`, where `y = 1/x`, and `dy`
+// is the corresponding input gradient.
+func InvGrad(scope *Scope, y tf.Output, dy tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "InvGrad",
+		Input: []tf.Input{
+			y, dy,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Adds up a SparseTensor and a dense Tensor, using these special rules:
+//
+// (1) Broadcasts the dense side to have the same shape as the sparse side, if
+//     eligible;
+// (2) Then, only the dense values pointed to by the indices of the SparseTensor
+//     participate in the cwise addition.
+//
+// By these rules, the result is a logical SparseTensor with exactly the same
+// indices and shape, but possibly with different non-zero values.  The output of
+// this Op is the resultant non-zero values.
+//
+// Arguments:
+//	sp_indices: 2-D.  `N x R` matrix with the indices of non-empty values in a
+// SparseTensor, possibly not in canonical ordering.
+//	sp_values: 1-D.  `N` non-empty values corresponding to `sp_indices`.
+//	sp_shape: 1-D.  Shape of the input SparseTensor.
+//	dense: `R`-D.  The dense Tensor operand.
+//
+// Returns 1-D.  The `N` values that are operated on.
+func SparseDenseCwiseAdd(scope *Scope, sp_indices tf.Output, sp_values tf.Output, sp_shape tf.Output, dense tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseDenseCwiseAdd",
+		Input: []tf.Input{
+			sp_indices, sp_values, sp_shape, dense,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// EncodeBase64Attr is an optional argument to EncodeBase64.
+type EncodeBase64Attr func(optionalAttr)
+
+// EncodeBase64Pad sets the optional pad attribute to value.
+//
+// value: Bool whether padding is applied at the ends.
+// If not specified, defaults to false
+func EncodeBase64Pad(value bool) EncodeBase64Attr {
+	return func(m optionalAttr) {
+		m["pad"] = value
+	}
+}
+
+// Encode strings into web-safe base64 format.
+//
+// Refer to the following article for more information on base64 format:
+// en.wikipedia.org/wiki/Base64. Base64 strings may have padding with '=' at the
+// end so that the encoded has length multiple of 4. See Padding section of the
+// link above.
+//
+// Web-safe means that the encoder uses - and _ instead of + and /.
+//
+// Arguments:
+//	input: Strings to be encoded.
+//
+// Returns Input strings encoded in base64.
+func EncodeBase64(scope *Scope, input tf.Output, optional ...EncodeBase64Attr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "EncodeBase64",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// FakeQuantWithMinMaxArgsAttr is an optional argument to FakeQuantWithMinMaxArgs.
+type FakeQuantWithMinMaxArgsAttr func(optionalAttr)
+
+// FakeQuantWithMinMaxArgsMin sets the optional min attribute to value.
+// If not specified, defaults to -6
+func FakeQuantWithMinMaxArgsMin(value float32) FakeQuantWithMinMaxArgsAttr {
+	return func(m optionalAttr) {
+		m["min"] = value
+	}
+}
+
+// FakeQuantWithMinMaxArgsMax sets the optional max attribute to value.
+// If not specified, defaults to 6
+func FakeQuantWithMinMaxArgsMax(value float32) FakeQuantWithMinMaxArgsAttr {
+	return func(m optionalAttr) {
+		m["max"] = value
+	}
+}
+
+// FakeQuantWithMinMaxArgsNumBits sets the optional num_bits attribute to value.
+// If not specified, defaults to 8
+func FakeQuantWithMinMaxArgsNumBits(value int64) FakeQuantWithMinMaxArgsAttr {
+	return func(m optionalAttr) {
+		m["num_bits"] = value
+	}
+}
+
+// FakeQuantWithMinMaxArgsNarrowRange sets the optional narrow_range attribute to value.
+// If not specified, defaults to false
+func FakeQuantWithMinMaxArgsNarrowRange(value bool) FakeQuantWithMinMaxArgsAttr {
+	return func(m optionalAttr) {
+		m["narrow_range"] = value
+	}
+}
+
+// Fake-quantize the 'inputs' tensor, type float to 'outputs' tensor of same type.
+//
+// Attributes
+//
+// *   `[min; max]` define the clamping range for the `inputs` data.
+// *   `inputs` values are quantized into the quantization range (
+// `[0; 2^num_bits - 1]` when `narrow_range` is false and `[1; 2^num_bits - 1]`
+// when it is true) and then de-quantized and output as floats in `[min; max]`
+// interval.
+// *   `num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
+//
+// Before quantization, `min` and `max` values are adjusted with the following
+// logic.
+// It is suggested to have `min <= 0 <= max`. If `0` is not in the range of values,
+// the behavior can be unexpected:
+//
+// *   If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
+// *   If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
+// *   If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
+// `min_adj = scale * round(min / scale)` and `max_adj = max + min_adj - min`.
+//
+// Quantization is called fake since the output is still in floating point.
+func FakeQuantWithMinMaxArgs(scope *Scope, inputs tf.Output, optional ...FakeQuantWithMinMaxArgsAttr) (outputs tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "FakeQuantWithMinMaxArgs",
+		Input: []tf.Input{
+			inputs,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Batch normalization.
+//
+// DEPRECATED at GraphDef version 9: Use tf.nn.batch_normalization()
+//
+// This op is deprecated. Prefer `tf.nn.batch_normalization`.
+//
+// Arguments:
+//	t: A 4D input Tensor.
+//	m: A 1D mean Tensor with size matching the last dimension of t.
+// This is the first output from tf.nn.moments,
+// or a saved moving average thereof.
+//	v: A 1D variance Tensor with size matching the last dimension of t.
+// This is the second output from tf.nn.moments,
+// or a saved moving average thereof.
+//	beta: A 1D beta Tensor with size matching the last dimension of t.
+// An offset to be added to the normalized tensor.
+//	gamma: A 1D gamma Tensor with size matching the last dimension of t.
+// If "scale_after_normalization" is true, this tensor will be multiplied
+// with the normalized tensor.
+//	variance_epsilon: A small float number to avoid dividing by 0.
+//	scale_after_normalization: A bool indicating whether the resulted tensor
+// needs to be multiplied with gamma.
+func BatchNormWithGlobalNormalization(scope *Scope, t tf.Output, m tf.Output, v tf.Output, beta tf.Output, gamma tf.Output, variance_epsilon float32, scale_after_normalization bool) (result tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"variance_epsilon": variance_epsilon, "scale_after_normalization": scale_after_normalization}
+	opspec := tf.OpSpec{
+		Type: "BatchNormWithGlobalNormalization",
+		Input: []tf.Input{
+			t, m, v, beta, gamma,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// QuantizedConv2DAttr is an optional argument to QuantizedConv2D.
+type QuantizedConv2DAttr func(optionalAttr)
+
+// QuantizedConv2DOutType sets the optional out_type attribute to value.
+// If not specified, defaults to DT_QINT32
+func QuantizedConv2DOutType(value tf.DataType) QuantizedConv2DAttr {
+	return func(m optionalAttr) {
+		m["out_type"] = value
+	}
+}
+
+// QuantizedConv2DDilations sets the optional dilations attribute to value.
+//
+// value: 1-D tensor of length 4.  The dilation factor for each dimension of
+// `input`. If set to k > 1, there will be k-1 skipped cells between each
+// filter element on that dimension. The dimension order is determined by the
+// value of `data_format`, see above for details. Dilations in the batch and
+// depth dimensions must be 1.
+// If not specified, defaults to <i:1 i:1 i:1 i:1 >
+func QuantizedConv2DDilations(value []int64) QuantizedConv2DAttr {
+	return func(m optionalAttr) {
+		m["dilations"] = value
+	}
+}
+
+// Computes a 2D convolution given quantized 4D input and filter tensors.
+//
+// The inputs are quantized tensors where the lowest value represents the real
+// number of the associated minimum, and the highest represents the maximum.
+// This means that you can only interpret the quantized output in the same way, by
+// taking the returned minimum and maximum values into account.
+//
+// Arguments:
+//
+//	filter: filter's input_depth dimension must match input's depth dimensions.
+//	min_input: The float value that the lowest quantized input value represents.
+//	max_input: The float value that the highest quantized input value represents.
+//	min_filter: The float value that the lowest quantized filter value represents.
+//	max_filter: The float value that the highest quantized filter value represents.
+//	strides: The stride of the sliding window for each dimension of the input
+// tensor.
+//	padding: The type of padding algorithm to use.
+//
+// Returns:
+//	output
+//	min_output: The float value that the lowest quantized output value represents.
+//	max_output: The float value that the highest quantized output value represents.
+func QuantizedConv2D(scope *Scope, input tf.Output, filter tf.Output, min_input tf.Output, max_input tf.Output, min_filter tf.Output, max_filter tf.Output, strides []int64, padding string, optional ...QuantizedConv2DAttr) (output tf.Output, min_output tf.Output, max_output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizedConv2D",
+		Input: []tf.Input{
+			input, filter, min_input, max_input, min_filter, max_filter,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Computes rectified linear 6 gradients for a Relu6 operation.
+//
+// Arguments:
+//	gradients: The backpropagated gradients to the corresponding Relu6 operation.
+//	features: The features passed as input to the corresponding Relu6 operation, or
+// its output; using either one produces the same result.
+//
+// Returns The gradients:
+// `gradients * (features > 0) * (features < 6)`.
+func Relu6Grad(scope *Scope, gradients tf.Output, features tf.Output) (backprops tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Relu6Grad",
+		Input: []tf.Input{
+			gradients, features,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// StringSplitAttr is an optional argument to StringSplit.
+type StringSplitAttr func(optionalAttr)
+
+// StringSplitSkipEmpty sets the optional skip_empty attribute to value.
+//
+// value: A `bool`. If `True`, skip the empty strings from the result.
+// If not specified, defaults to true
+func StringSplitSkipEmpty(value bool) StringSplitAttr {
+	return func(m optionalAttr) {
+		m["skip_empty"] = value
+	}
+}
+
+// Split elements of `input` based on `delimiter` into a `SparseTensor`.
+//
+// Let N be the size of source (typically N will be the batch size). Split each
+// element of `input` based on `delimiter` and return a `SparseTensor`
+// containing the splitted tokens. Empty tokens are ignored.
+//
+// `delimiter` can be empty, or a string of split characters. If `delimiter` is an
+//  empty string, each element of `input` is split into individual single-byte
+//  character strings, including splitting of UTF-8 multibyte sequences. Otherwise
+//  every character of `delimiter` is a potential split point.
+//
+// For example:
+//   N = 2, input[0] is 'hello world' and input[1] is 'a b c', then the output
+//   will be
+//
+//   indices = [0, 0;
+//              0, 1;
+//              1, 0;
+//              1, 1;
+//              1, 2]
+//   shape = [2, 3]
+//   values = ['hello', 'world', 'a', 'b', 'c']
+//
+// Arguments:
+//	input: 1-D. Strings to split.
+//	delimiter: 0-D. Delimiter characters (bytes), or empty string.
+//
+// Returns:
+//	indices: A dense matrix of int64 representing the indices of the sparse tensor.
+//	values: A vector of strings corresponding to the splited values.
+//	shape: a length-2 vector of int64 representing the shape of the sparse
+// tensor, where the first value is N and the second value is the maximum number
+// of tokens in a single input entry.
+func StringSplit(scope *Scope, input tf.Output, delimiter tf.Output, optional ...StringSplitAttr) (indices tf.Output, values tf.Output, shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StringSplit",
+		Input: []tf.Input{
+			input, delimiter,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Assigns sparse updates to the variable referenced by `resource`.
+//
+// This operation computes
+//
+//     # Scalar indices
+//     ref[indices, ...] = updates[...]
+//
+//     # Vector indices (for each i)
+//     ref[indices[i], ...] = updates[i, ...]
+//
+//     # High rank indices (for each i, ..., j)
+//     ref[indices[i, ..., j], ...] = updates[i, ..., j, ...]
+//
+// Arguments:
+//	resource: Should be from a `Variable` node.
+//	indices: A tensor of indices into the first dimension of `ref`.
+//	updates: A tensor of updated values to add to `ref`.
+//
+// Returns the created operation.
+func ResourceScatterUpdate(scope *Scope, resource tf.Output, indices tf.Output, updates tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceScatterUpdate",
+		Input: []tf.Input{
+			resource, indices, updates,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// EditDistanceAttr is an optional argument to EditDistance.
+type EditDistanceAttr func(optionalAttr)
+
+// EditDistanceNormalize sets the optional normalize attribute to value.
+//
+// value: boolean (if true, edit distances are normalized by length of truth).
+//
+// The output is:
+// If not specified, defaults to true
+func EditDistanceNormalize(value bool) EditDistanceAttr {
+	return func(m optionalAttr) {
+		m["normalize"] = value
+	}
+}
+
+// Computes the (possibly normalized) Levenshtein Edit Distance.
+//
+// The inputs are variable-length sequences provided by SparseTensors
+//   (hypothesis_indices, hypothesis_values, hypothesis_shape)
+// and
+//   (truth_indices, truth_values, truth_shape).
+//
+// The inputs are:
+//
+// Arguments:
+//	hypothesis_indices: The indices of the hypothesis list SparseTensor.
+// This is an N x R int64 matrix.
+//	hypothesis_values: The values of the hypothesis list SparseTensor.
+// This is an N-length vector.
+//	hypothesis_shape: The shape of the hypothesis list SparseTensor.
+// This is an R-length vector.
+//	truth_indices: The indices of the truth list SparseTensor.
+// This is an M x R int64 matrix.
+//	truth_values: The values of the truth list SparseTensor.
+// This is an M-length vector.
+//	truth_shape: truth indices, vector.
+//
+// Returns A dense float tensor with rank R - 1.
+//
+// For the example input:
+//
+//     // hypothesis represents a 2x1 matrix with variable-length values:
+//     //   (0,0) = ["a"]
+//     //   (1,0) = ["b"]
+//     hypothesis_indices = [[0, 0, 0],
+//                           [1, 0, 0]]
+//     hypothesis_values = ["a", "b"]
+//     hypothesis_shape = [2, 1, 1]
+//
+//     // truth represents a 2x2 matrix with variable-length values:
+//     //   (0,0) = []
+//     //   (0,1) = ["a"]
+//     //   (1,0) = ["b", "c"]
+//     //   (1,1) = ["a"]
+//     truth_indices = [[0, 1, 0],
+//                      [1, 0, 0],
+//                      [1, 0, 1],
+//                      [1, 1, 0]]
+//     truth_values = ["a", "b", "c", "a"]
+//     truth_shape = [2, 2, 2]
+//     normalize = true
+//
+// The output will be:
+//
+//     // output is a 2x2 matrix with edit distances normalized by truth lengths.
+//     output = [[inf, 1.0],  // (0,0): no truth, (0,1): no hypothesis
+//               [0.5, 1.0]]  // (1,0): addition, (1,1): no hypothesis
+func EditDistance(scope *Scope, hypothesis_indices tf.Output, hypothesis_values tf.Output, hypothesis_shape tf.Output, truth_indices tf.Output, truth_values tf.Output, truth_shape tf.Output, optional ...EditDistanceAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "EditDistance",
+		Input: []tf.Input{
+			hypothesis_indices, hypothesis_values, hypothesis_shape, truth_indices, truth_values, truth_shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Concatenates a list of `N` tensors along the first dimension.
+//
+// The input tensors are all required to have size 1 in the first dimension.
+//
+// For example:
+//
+// ```
+// # 'x' is [[1, 4]]
+// # 'y' is [[2, 5]]
+// # 'z' is [[3, 6]]
+// parallel_concat([x, y, z]) => [[1, 4], [2, 5], [3, 6]]  # Pack along first dim.
+// ```
+//
+// The difference between concat and parallel_concat is that concat requires all
+// of the inputs be computed before the operation will begin but doesn't require
+// that the input shapes be known during graph construction.  Parallel concat
+// will copy pieces of the input into the output as they become available, in
+// some situations this can provide a performance benefit.
+//
+// Arguments:
+//	values: Tensors to be concatenated. All must have size 1 in the first dimension
+// and same shape.
+//	shape: the final shape of the result; should be equal to the shapes of any input
+// but with the number of input values in the first dimension.
+//
+// Returns The concatenated tensor.
+func ParallelConcat(scope *Scope, values []tf.Output, shape tf.Shape) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"shape": shape}
+	opspec := tf.OpSpec{
+		Type: "ParallelConcat",
+		Input: []tf.Input{
+			tf.OutputList(values),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// AvgPoolGradAttr is an optional argument to AvgPoolGrad.
+type AvgPoolGradAttr func(optionalAttr)
+
+// AvgPoolGradDataFormat sets the optional data_format attribute to value.
+//
+// value: Specify the data format of the input and output data. With the
+// default format "NHWC", the data is stored in the order of:
+//     [batch, in_height, in_width, in_channels].
+// Alternatively, the format could be "NCHW", the data storage order of:
+//     [batch, in_channels, in_height, in_width].
+// If not specified, defaults to "NHWC"
+func AvgPoolGradDataFormat(value string) AvgPoolGradAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// Computes gradients of the average pooling function.
+//
+// Arguments:
+//	orig_input_shape: 1-D.  Shape of the original input to `avg_pool`.
+//	grad: 4-D with shape `[batch, height, width, channels]`.  Gradients w.r.t.
+// the output of `avg_pool`.
+//	ksize: The size of the sliding window for each dimension of the input.
+//	strides: The stride of the sliding window for each dimension of the input.
+//	padding: The type of padding algorithm to use.
+//
+// Returns 4-D.  Gradients w.r.t. the input of `avg_pool`.
+func AvgPoolGrad(scope *Scope, orig_input_shape tf.Output, grad tf.Output, ksize []int64, strides []int64, padding string, optional ...AvgPoolGradAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"ksize": ksize, "strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "AvgPoolGrad",
+		Input: []tf.Input{
+			orig_input_shape, grad,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// LoadAndRemapMatrixAttr is an optional argument to LoadAndRemapMatrix.
+type LoadAndRemapMatrixAttr func(optionalAttr)
+
+// LoadAndRemapMatrixMaxRowsInMemory sets the optional max_rows_in_memory attribute to value.
+//
+// value: The maximum number of rows to load from the checkpoint at
+// once. If less than or equal to 0, the entire matrix will be loaded into
+// memory. Setting this arg trades increased disk reads for lower memory usage.
+// If not specified, defaults to -1
+func LoadAndRemapMatrixMaxRowsInMemory(value int64) LoadAndRemapMatrixAttr {
+	return func(m optionalAttr) {
+		m["max_rows_in_memory"] = value
+	}
+}
+
+// Loads a 2-D (matrix) `Tensor` with name `old_tensor_name` from the checkpoint
+//
+// at `ckpt_path` and potentially reorders its rows and columns using the
+// specified remappings.
+//
+// Most users should use one of the wrapper initializers (such as
+// `tf.contrib.framework.load_and_remap_matrix_initializer`) instead of this
+// function directly.
+//
+// The remappings are 1-D tensors with the following properties:
+//
+// * `row_remapping` must have exactly `num_rows` entries. Row `i` of the output
+//   matrix will be initialized from the row corresponding to index
+//   `row_remapping[i]` in the old `Tensor` from the checkpoint.
+// * `col_remapping` must have either 0 entries (indicating that no column
+//   reordering is needed) or `num_cols` entries. If specified, column `j` of the
+//   output matrix will be initialized from the column corresponding to index
+//   `col_remapping[j]` in the old `Tensor` from the checkpoint.
+// * A value of -1 in either of the remappings signifies a "missing" entry. In that
+//   case, values from the `initializing_values` tensor will be used to fill that
+//   missing row or column. If `row_remapping` has `r` missing entries and
+//   `col_remapping` has `c` missing entries, then the following condition must be
+//   true:
+//
+// `(r * num_cols) + (c * num_rows) - (r * c) == len(initializing_values)`
+//
+// The remapping tensors can be generated using the GenerateVocabRemapping op.
+//
+// As an example, with row_remapping = [1, 0, -1], col_remapping = [0, 2, -1],
+// initializing_values = [0.5, -0.5, 0.25, -0.25, 42], and w(i, j) representing
+// the value from row i, column j of the old tensor in the checkpoint, the output
+// matrix will look like the following:
+//
+// [[w(1, 0),  w(1, 2),  0.5],
+//  [w(0, 0),  w(0, 2), -0.5],
+//  [0.25,    -0.25,      42]]
+//
+// Arguments:
+//	ckpt_path: Path to the TensorFlow checkpoint (version 2, `TensorBundle`) from
+// which the old matrix `Tensor` will be loaded.
+//	old_tensor_name: Name of the 2-D `Tensor` to load from checkpoint.
+//	row_remapping: An int `Tensor` of row remappings (generally created by
+// `generate_vocab_remapping`).  Even if no row remapping is needed, this must
+// still be an index-valued Tensor (e.g. [0, 1, 2, ...]), or a shifted
+// index-valued `Tensor` (e.g. [8, 9, 10, ...], for partitioned `Variables`).
+//	col_remapping: An int `Tensor` of column remappings (generally created by
+// `generate_vocab_remapping`).  May be a size-0 `Tensor` if only row remapping
+// is to be done (e.g. column ordering is the same).
+//	initializing_values: A float `Tensor` containing  values to fill in for cells
+// in the output matrix that are not loaded from the checkpoint. Length must be
+// exactly the same as the number of missing / new cells.
+//	num_rows: Number of rows (length of the 1st dimension) in the output matrix.
+//	num_cols: Number of columns (length of the 2nd dimension) in the output matrix.
+//
+// Returns Output matrix containing existing values loaded from the
+// checkpoint, and with any missing values filled in from initializing_values.
+func LoadAndRemapMatrix(scope *Scope, ckpt_path tf.Output, old_tensor_name tf.Output, row_remapping tf.Output, col_remapping tf.Output, initializing_values tf.Output, num_rows int64, num_cols int64, optional ...LoadAndRemapMatrixAttr) (output_matrix tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_rows": num_rows, "num_cols": num_cols}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LoadAndRemapMatrix",
+		Input: []tf.Input{
+			ckpt_path, old_tensor_name, row_remapping, col_remapping, initializing_values,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Does nothing. Only useful as a placeholder for control edges.
+//
+// Returns the created operation.
+func NoOp(scope *Scope) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "NoOp",
+	}
+	return scope.AddOperation(opspec)
+}
+
+// ResourceSparseApplyRMSPropAttr is an optional argument to ResourceSparseApplyRMSProp.
+type ResourceSparseApplyRMSPropAttr func(optionalAttr)
+
+// ResourceSparseApplyRMSPropUseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var, ms, and mom tensors is protected
+// by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceSparseApplyRMSPropUseLocking(value bool) ResourceSparseApplyRMSPropAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// Update '*var' according to the RMSProp algorithm.
+//
+// Note that in dense implementation of this algorithm, ms and mom will
+// update even if the grad is zero, but in this sparse implementation, ms
+// and mom will not update in iterations during which the grad is zero.
+//
+// mean_square = decay * mean_square + (1-decay) * gradient ** 2
+// Delta = learning_rate * gradient / sqrt(mean_square + epsilon)
+//
+// ms <- rho * ms_{t-1} + (1-rho) * grad * grad
+// mom <- momentum * mom_{t-1} + lr * grad / sqrt(ms + epsilon)
+// var <- var - mom
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	ms: Should be from a Variable().
+//	mom: Should be from a Variable().
+//	lr: Scaling factor. Must be a scalar.
+//	rho: Decay rate. Must be a scalar.
+//
+//	epsilon: Ridge term. Must be a scalar.
+//	grad: The gradient.
+//	indices: A vector of indices into the first dimension of var, ms and mom.
+//
+// Returns the created operation.
+func ResourceSparseApplyRMSProp(scope *Scope, var_ tf.Output, ms tf.Output, mom tf.Output, lr tf.Output, rho tf.Output, momentum tf.Output, epsilon tf.Output, grad tf.Output, indices tf.Output, optional ...ResourceSparseApplyRMSPropAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceSparseApplyRMSProp",
+		Input: []tf.Input{
+			var_, ms, mom, lr, rho, momentum, epsilon, grad, indices,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// StringToNumberAttr is an optional argument to StringToNumber.
+type StringToNumberAttr func(optionalAttr)
+
+// StringToNumberOutType sets the optional out_type attribute to value.
+//
+// value: The numeric type to interpret each string in `string_tensor` as.
+// If not specified, defaults to DT_FLOAT
+func StringToNumberOutType(value tf.DataType) StringToNumberAttr {
+	return func(m optionalAttr) {
+		m["out_type"] = value
+	}
+}
+
+// Converts each string in the input Tensor to the specified numeric type.
+//
+// (Note that int32 overflow results in an error while float overflow
+// results in a rounded value.)
+//
+// Example:
+//
+// >>> strings = ["5.0", "3.0", "7.0"]
+// >>> tf.strings.to_number(strings)
+// <tf.Tensor: shape=(3,), dtype=float32, numpy=array([5., 3., 7.], dtype=float32)>
+//
+//
+// Returns A Tensor of the same shape as the input `string_tensor`.
+func StringToNumber(scope *Scope, string_tensor tf.Output, optional ...StringToNumberAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StringToNumber",
+		Input: []tf.Input{
+			string_tensor,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Transforms a Tensor into a serialized TensorProto proto.
+//
+// Arguments:
+//	tensor: A Tensor of type `T`.
+//
+// Returns A serialized TensorProto proto of the input tensor.
+func SerializeTensor(scope *Scope, tensor tf.Output) (serialized tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SerializeTensor",
+		Input: []tf.Input{
+			tensor,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Transforms a serialized tensorflow.TensorProto proto into a Tensor.
+//
+// Arguments:
+//	serialized: A scalar string containing a serialized TensorProto proto.
+//	out_type: The type of the serialized tensor.  The provided type must match the
+// type of the serialized tensor and no implicit conversion will take place.
+//
+// Returns A Tensor of type `out_type`.
+func ParseTensor(scope *Scope, serialized tf.Output, out_type tf.DataType) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"out_type": out_type}
+	opspec := tf.OpSpec{
+		Type: "ParseTensor",
+		Input: []tf.Input{
+			serialized,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns element-wise integer closest to x.
+//
+// If the result is midway between two representable values,
+// the even representable is chosen.
+// For example:
+//
+// ```
+// rint(-1.5) ==> -2.0
+// rint(0.5000001) ==> 1.0
+// rint([-1.7, -1.5, -0.2, 0.2, 1.5, 1.7, 2.0]) ==> [-2., -2., -0., 0., 2., 2., 2.]
+// ```
+func Rint(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Rint",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Reverses specific dimensions of a tensor.
+//
+// Given a `tensor`, and a `bool` tensor `dims` representing the dimensions
+// of `tensor`, this operation reverses each dimension i of `tensor` where
+// `dims[i]` is `True`.
+//
+// `tensor` can have up to 8 dimensions. The number of dimensions
+// of `tensor` must equal the number of elements in `dims`. In other words:
+//
+// `rank(tensor) = size(dims)`
+//
+// For example:
+//
+// ```
+// # tensor 't' is [[[[ 0,  1,  2,  3],
+// #                  [ 4,  5,  6,  7],
+// #                  [ 8,  9, 10, 11]],
+// #                 [[12, 13, 14, 15],
+// #                  [16, 17, 18, 19],
+// #                  [20, 21, 22, 23]]]]
+// # tensor 't' shape is [1, 2, 3, 4]
+//
+// # 'dims' is [False, False, False, True]
+// reverse(t, dims) ==> [[[[ 3,  2,  1,  0],
+//                         [ 7,  6,  5,  4],
+//                         [ 11, 10, 9, 8]],
+//                        [[15, 14, 13, 12],
+//                         [19, 18, 17, 16],
+//                         [23, 22, 21, 20]]]]
+//
+// # 'dims' is [False, True, False, False]
+// reverse(t, dims) ==> [[[[12, 13, 14, 15],
+//                         [16, 17, 18, 19],
+//                         [20, 21, 22, 23]
+//                        [[ 0,  1,  2,  3],
+//                         [ 4,  5,  6,  7],
+//                         [ 8,  9, 10, 11]]]]
+//
+// # 'dims' is [False, False, True, False]
+// reverse(t, dims) ==> [[[[8, 9, 10, 11],
+//                         [4, 5, 6, 7],
+//                         [0, 1, 2, 3]]
+//                        [[20, 21, 22, 23],
+//                         [16, 17, 18, 19],
+//                         [12, 13, 14, 15]]]]
+// ```
+//
+// Arguments:
+//	tensor: Up to 8-D.
+//	dims: 1-D. The dimensions to reverse.
+//
+// Returns The same shape as `tensor`.
+func Reverse(scope *Scope, tensor tf.Output, dims tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Reverse",
+		Input: []tf.Input{
+			tensor, dims,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Wraps an arbitrary MLIR computation expressed as a module with a main() function.
+//
+// This operation does not have an associated kernel and is not intended to be
+// executed in a regular TensorFlow session. Instead it is intended to be used for
+// testing or for special case where a user intends to pass custom MLIR computation
+// through a TensorFlow graph with the intent of having custom tooling processing
+// it downstream (when targeting a different environment, like TensorFlow lite for
+// example).
+// The MLIR module is expected to have a main() function that will be used as an
+// entry point. The inputs to the operations will be passed as argument to the
+// main() function and the returned values of the main function mapped to the
+// outputs.
+// Example usage:
+//
+// ```
+// import tensorflow as tf
+// from tensorflow.compiler.mlir.tensorflow.gen_mlir_passthrough_op import mlir_passthrough_op
+//
+// mlir_module = '''python
+// func @main(%arg0 : tensor<10xf32>, %arg1 : tensor<10xf32>) -> tensor<10x10xf32> {
+//    %add = "magic.op"(%arg0, %arg1) : (tensor<10xf32>, tensor<10xf32>) -> tensor<10x10xf32>
+//    return %ret : tensor<10x10xf32>
+// }
+// '''
+//
+// @tf.function
+// def foo(x, y):
+//   return mlir_passthrough_op([x, y], mlir_module, Toutputs=[tf.float32])
+//
+// graph_def = foo.get_concrete_function(tf.TensorSpec([10], tf.float32), tf.TensorSpec([10], tf.float32)).graph.as_graph_def()
+// ```
+func MlirPassthroughOp(scope *Scope, inputs []tf.Output, mlir_module string, Toutputs []tf.DataType) (outputs []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"mlir_module": mlir_module, "Toutputs": Toutputs}
+	opspec := tf.OpSpec{
+		Type: "MlirPassthroughOp",
+		Input: []tf.Input{
+			tf.OutputList(inputs),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if outputs, idx, err = makeOutputList(op, idx, "outputs"); err != nil {
+		scope.UpdateErr("MlirPassthroughOp", err)
+		return
+	}
+	return outputs
+}
+
+// StringLowerAttr is an optional argument to StringLower.
+type StringLowerAttr func(optionalAttr)
+
+// StringLowerEncoding sets the optional encoding attribute to value.
+// If not specified, defaults to ""
+func StringLowerEncoding(value string) StringLowerAttr {
+	return func(m optionalAttr) {
+		m["encoding"] = value
+	}
+}
+
+// Converts all uppercase characters into their respective lowercase replacements.
+//
+// Example:
+//
+// >>> tf.strings.lower("CamelCase string and ALL CAPS")
+// <tf.Tensor: shape=(), dtype=string, numpy=b'camelcase string and all caps'>
+//
+func StringLower(scope *Scope, input tf.Output, optional ...StringLowerAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StringLower",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ParseSequenceExampleV2Attr is an optional argument to ParseSequenceExampleV2.
+type ParseSequenceExampleV2Attr func(optionalAttr)
+
+// ParseSequenceExampleV2NcontextSparse sets the optional Ncontext_sparse attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func ParseSequenceExampleV2NcontextSparse(value int64) ParseSequenceExampleV2Attr {
+	return func(m optionalAttr) {
+		m["Ncontext_sparse"] = value
+	}
+}
+
+// ParseSequenceExampleV2ContextSparseTypes sets the optional context_sparse_types attribute to value.
+//
+// value: A list of Ncontext_sparse types; the data types of data in
+// each context Feature given in context_sparse_keys.
+// Currently the ParseSingleSequenceExample supports DT_FLOAT (FloatList),
+// DT_INT64 (Int64List), and DT_STRING (BytesList).
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseSequenceExampleV2ContextSparseTypes(value []tf.DataType) ParseSequenceExampleV2Attr {
+	return func(m optionalAttr) {
+		m["context_sparse_types"] = value
+	}
+}
+
+// ParseSequenceExampleV2ContextRaggedValueTypes sets the optional context_ragged_value_types attribute to value.
+//
+// value: RaggedTensor.value dtypes for the ragged context features.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseSequenceExampleV2ContextRaggedValueTypes(value []tf.DataType) ParseSequenceExampleV2Attr {
+	return func(m optionalAttr) {
+		m["context_ragged_value_types"] = value
+	}
+}
+
+// ParseSequenceExampleV2ContextRaggedSplitTypes sets the optional context_ragged_split_types attribute to value.
+//
+// value: RaggedTensor.row_split dtypes for the ragged context features.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseSequenceExampleV2ContextRaggedSplitTypes(value []tf.DataType) ParseSequenceExampleV2Attr {
+	return func(m optionalAttr) {
+		m["context_ragged_split_types"] = value
+	}
+}
+
+// ParseSequenceExampleV2ContextDenseShapes sets the optional context_dense_shapes attribute to value.
+//
+// value: A list of Ncontext_dense shapes; the shapes of data in
+// each context Feature given in context_dense_keys.
+// The number of elements in the Feature corresponding to context_dense_key[j]
+// must always equal context_dense_shapes[j].NumEntries().
+// The shape of context_dense_values[j] will match context_dense_shapes[j].
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseSequenceExampleV2ContextDenseShapes(value []tf.Shape) ParseSequenceExampleV2Attr {
+	return func(m optionalAttr) {
+		m["context_dense_shapes"] = value
+	}
+}
+
+// ParseSequenceExampleV2NfeatureListSparse sets the optional Nfeature_list_sparse attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func ParseSequenceExampleV2NfeatureListSparse(value int64) ParseSequenceExampleV2Attr {
+	return func(m optionalAttr) {
+		m["Nfeature_list_sparse"] = value
+	}
+}
+
+// ParseSequenceExampleV2NfeatureListDense sets the optional Nfeature_list_dense attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func ParseSequenceExampleV2NfeatureListDense(value int64) ParseSequenceExampleV2Attr {
+	return func(m optionalAttr) {
+		m["Nfeature_list_dense"] = value
+	}
+}
+
+// ParseSequenceExampleV2FeatureListDenseTypes sets the optional feature_list_dense_types attribute to value.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseSequenceExampleV2FeatureListDenseTypes(value []tf.DataType) ParseSequenceExampleV2Attr {
+	return func(m optionalAttr) {
+		m["feature_list_dense_types"] = value
+	}
+}
+
+// ParseSequenceExampleV2FeatureListSparseTypes sets the optional feature_list_sparse_types attribute to value.
+//
+// value: A list of Nfeature_list_sparse types; the data types
+// of data in each FeatureList given in feature_list_sparse_keys.
+// Currently the ParseSingleSequenceExample supports DT_FLOAT (FloatList),
+// DT_INT64 (Int64List), and DT_STRING (BytesList).
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseSequenceExampleV2FeatureListSparseTypes(value []tf.DataType) ParseSequenceExampleV2Attr {
+	return func(m optionalAttr) {
+		m["feature_list_sparse_types"] = value
+	}
+}
+
+// ParseSequenceExampleV2FeatureListRaggedValueTypes sets the optional feature_list_ragged_value_types attribute to value.
+//
+// value: RaggedTensor.value dtypes for the ragged FeatureList features.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseSequenceExampleV2FeatureListRaggedValueTypes(value []tf.DataType) ParseSequenceExampleV2Attr {
+	return func(m optionalAttr) {
+		m["feature_list_ragged_value_types"] = value
+	}
+}
+
+// ParseSequenceExampleV2FeatureListRaggedSplitTypes sets the optional feature_list_ragged_split_types attribute to value.
+//
+// value: RaggedTensor.row_split dtypes for the ragged FeatureList features.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseSequenceExampleV2FeatureListRaggedSplitTypes(value []tf.DataType) ParseSequenceExampleV2Attr {
+	return func(m optionalAttr) {
+		m["feature_list_ragged_split_types"] = value
+	}
+}
+
+// ParseSequenceExampleV2FeatureListDenseShapes sets the optional feature_list_dense_shapes attribute to value.
+//
+// value: A list of Nfeature_list_dense shapes; the shapes of
+// data in each FeatureList given in feature_list_dense_keys.
+// The shape of each Feature in the FeatureList corresponding to
+// feature_list_dense_key[j] must always equal
+// feature_list_dense_shapes[j].NumEntries().
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseSequenceExampleV2FeatureListDenseShapes(value []tf.Shape) ParseSequenceExampleV2Attr {
+	return func(m optionalAttr) {
+		m["feature_list_dense_shapes"] = value
+	}
+}
+
+// Transforms a vector of tf.io.SequenceExample protos (as strings) into
+// typed tensors.
+//
+// Arguments:
+//	serialized: A scalar or vector containing binary serialized SequenceExample protos.
+//	debug_name: A scalar or vector containing the names of the serialized protos.
+// May contain, for example, table key (descriptive) name for the
+// corresponding serialized proto.  This is purely useful for debugging
+// purposes, and the presence of values here has no effect on the output.
+// May also be an empty vector if no name is available.
+//	context_sparse_keys: The keys expected in the Examples' features associated with context_sparse
+// values.
+//	context_dense_keys: The keys expected in the SequenceExamples' context features associated with
+// dense values.
+//	context_ragged_keys: The keys expected in the Examples' features associated with context_ragged
+// values.
+//	feature_list_sparse_keys: The keys expected in the FeatureLists associated with sparse values.
+//	feature_list_dense_keys: The keys expected in the SequenceExamples' feature_lists associated
+// with lists of dense values.
+//	feature_list_ragged_keys: The keys expected in the FeatureLists associated with ragged values.
+//	feature_list_dense_missing_assumed_empty: A vector corresponding 1:1 with feature_list_dense_keys, indicating which
+// features may be missing from the SequenceExamples.  If the associated
+// FeatureList is missing, it is treated as empty.
+//	context_dense_defaults: A list of Ncontext_dense Tensors (some may be empty).
+// context_dense_defaults[j] provides default values
+// when the SequenceExample's context map lacks context_dense_key[j].
+// If an empty Tensor is provided for context_dense_defaults[j],
+// then the Feature context_dense_keys[j] is required.
+// The input type is inferred from context_dense_defaults[j], even when it's
+// empty.  If context_dense_defaults[j] is not empty, its shape must match
+// context_dense_shapes[j].
+func ParseSequenceExampleV2(scope *Scope, serialized tf.Output, debug_name tf.Output, context_sparse_keys tf.Output, context_dense_keys tf.Output, context_ragged_keys tf.Output, feature_list_sparse_keys tf.Output, feature_list_dense_keys tf.Output, feature_list_ragged_keys tf.Output, feature_list_dense_missing_assumed_empty tf.Output, context_dense_defaults []tf.Output, optional ...ParseSequenceExampleV2Attr) (context_sparse_indices []tf.Output, context_sparse_values []tf.Output, context_sparse_shapes []tf.Output, context_dense_values []tf.Output, context_ragged_values []tf.Output, context_ragged_row_splits []tf.Output, feature_list_sparse_indices []tf.Output, feature_list_sparse_values []tf.Output, feature_list_sparse_shapes []tf.Output, feature_list_dense_values []tf.Output, feature_list_dense_lengths []tf.Output, feature_list_ragged_values []tf.Output, feature_list_ragged_outer_splits []tf.Output, feature_list_ragged_inner_splits []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ParseSequenceExampleV2",
+		Input: []tf.Input{
+			serialized, debug_name, context_sparse_keys, context_dense_keys, context_ragged_keys, feature_list_sparse_keys, feature_list_dense_keys, feature_list_ragged_keys, feature_list_dense_missing_assumed_empty, tf.OutputList(context_dense_defaults),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if context_sparse_indices, idx, err = makeOutputList(op, idx, "context_sparse_indices"); err != nil {
+		scope.UpdateErr("ParseSequenceExampleV2", err)
+		return
+	}
+	if context_sparse_values, idx, err = makeOutputList(op, idx, "context_sparse_values"); err != nil {
+		scope.UpdateErr("ParseSequenceExampleV2", err)
+		return
+	}
+	if context_sparse_shapes, idx, err = makeOutputList(op, idx, "context_sparse_shapes"); err != nil {
+		scope.UpdateErr("ParseSequenceExampleV2", err)
+		return
+	}
+	if context_dense_values, idx, err = makeOutputList(op, idx, "context_dense_values"); err != nil {
+		scope.UpdateErr("ParseSequenceExampleV2", err)
+		return
+	}
+	if context_ragged_values, idx, err = makeOutputList(op, idx, "context_ragged_values"); err != nil {
+		scope.UpdateErr("ParseSequenceExampleV2", err)
+		return
+	}
+	if context_ragged_row_splits, idx, err = makeOutputList(op, idx, "context_ragged_row_splits"); err != nil {
+		scope.UpdateErr("ParseSequenceExampleV2", err)
+		return
+	}
+	if feature_list_sparse_indices, idx, err = makeOutputList(op, idx, "feature_list_sparse_indices"); err != nil {
+		scope.UpdateErr("ParseSequenceExampleV2", err)
+		return
+	}
+	if feature_list_sparse_values, idx, err = makeOutputList(op, idx, "feature_list_sparse_values"); err != nil {
+		scope.UpdateErr("ParseSequenceExampleV2", err)
+		return
+	}
+	if feature_list_sparse_shapes, idx, err = makeOutputList(op, idx, "feature_list_sparse_shapes"); err != nil {
+		scope.UpdateErr("ParseSequenceExampleV2", err)
+		return
+	}
+	if feature_list_dense_values, idx, err = makeOutputList(op, idx, "feature_list_dense_values"); err != nil {
+		scope.UpdateErr("ParseSequenceExampleV2", err)
+		return
+	}
+	if feature_list_dense_lengths, idx, err = makeOutputList(op, idx, "feature_list_dense_lengths"); err != nil {
+		scope.UpdateErr("ParseSequenceExampleV2", err)
+		return
+	}
+	if feature_list_ragged_values, idx, err = makeOutputList(op, idx, "feature_list_ragged_values"); err != nil {
+		scope.UpdateErr("ParseSequenceExampleV2", err)
+		return
+	}
+	if feature_list_ragged_outer_splits, idx, err = makeOutputList(op, idx, "feature_list_ragged_outer_splits"); err != nil {
+		scope.UpdateErr("ParseSequenceExampleV2", err)
+		return
+	}
+	if feature_list_ragged_inner_splits, idx, err = makeOutputList(op, idx, "feature_list_ragged_inner_splits"); err != nil {
+		scope.UpdateErr("ParseSequenceExampleV2", err)
+		return
+	}
+	return context_sparse_indices, context_sparse_values, context_sparse_shapes, context_dense_values, context_ragged_values, context_ragged_row_splits, feature_list_sparse_indices, feature_list_sparse_values, feature_list_sparse_shapes, feature_list_dense_values, feature_list_dense_lengths, feature_list_ragged_values, feature_list_ragged_outer_splits, feature_list_ragged_inner_splits
+}
+
+// Gives a guarantee to the TF runtime that the input tensor is a constant.
+//
+// The runtime is then free to make optimizations based on this.
+//
+// Only accepts value typed tensors as inputs and rejects resource variable handles
+// as input.
+//
+// Returns the input tensor without modification.
+func GuaranteeConst(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "GuaranteeConst",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Transforms a tf.Example proto (as a string) into typed tensors.
+//
+// Arguments:
+//	serialized: A vector containing a batch of binary serialized Example protos.
+//	dense_defaults: A list of Tensors (some may be empty), whose length matches
+// the length of `dense_keys`. dense_defaults[j] provides default values
+// when the example's feature_map lacks dense_key[j].  If an empty Tensor is
+// provided for dense_defaults[j], then the Feature dense_keys[j] is required.
+// The input type is inferred from dense_defaults[j], even when it's empty.
+// If dense_defaults[j] is not empty, and dense_shapes[j] is fully defined,
+// then the shape of dense_defaults[j] must match that of dense_shapes[j].
+// If dense_shapes[j] has an undefined major dimension (variable strides dense
+// feature), dense_defaults[j] must contain a single element:
+// the padding element.
+//	num_sparse: The number of sparse features to be parsed from the example. This
+// must match the lengths of `sparse_keys` and `sparse_types`.
+//	sparse_keys: A list of `num_sparse` strings.
+// The keys expected in the Examples' features associated with sparse values.
+//	dense_keys: The keys expected in the Examples' features associated with dense
+// values.
+//	sparse_types: A list of `num_sparse` types; the data types of data in each
+// Feature given in sparse_keys.
+// Currently the ParseSingleExample op supports DT_FLOAT (FloatList),
+// DT_INT64 (Int64List), and DT_STRING (BytesList).
+//	dense_shapes: The shapes of data in each Feature given in dense_keys.
+// The length of this list must match the length of `dense_keys`.  The
+// number of elements in the Feature corresponding to dense_key[j] must
+// always equal dense_shapes[j].NumEntries().  If dense_shapes[j] ==
+// (D0, D1, ..., DN) then the shape of output Tensor dense_values[j]
+// will be (D0, D1, ..., DN): In the case dense_shapes[j] = (-1, D1,
+// ..., DN), the shape of the output Tensor dense_values[j] will be (M,
+// D1, .., DN), where M is the number of blocks of elements of length
+// D1 * .... * DN, in the input.
+func ParseSingleExample(scope *Scope, serialized tf.Output, dense_defaults []tf.Output, num_sparse int64, sparse_keys []string, dense_keys []string, sparse_types []tf.DataType, dense_shapes []tf.Shape) (sparse_indices []tf.Output, sparse_values []tf.Output, sparse_shapes []tf.Output, dense_values []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_sparse": num_sparse, "sparse_keys": sparse_keys, "dense_keys": dense_keys, "sparse_types": sparse_types, "dense_shapes": dense_shapes}
+	opspec := tf.OpSpec{
+		Type: "ParseSingleExample",
+		Input: []tf.Input{
+			serialized, tf.OutputList(dense_defaults),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if sparse_indices, idx, err = makeOutputList(op, idx, "sparse_indices"); err != nil {
+		scope.UpdateErr("ParseSingleExample", err)
+		return
+	}
+	if sparse_values, idx, err = makeOutputList(op, idx, "sparse_values"); err != nil {
+		scope.UpdateErr("ParseSingleExample", err)
+		return
+	}
+	if sparse_shapes, idx, err = makeOutputList(op, idx, "sparse_shapes"); err != nil {
+		scope.UpdateErr("ParseSingleExample", err)
+		return
+	}
+	if dense_values, idx, err = makeOutputList(op, idx, "dense_values"); err != nil {
+		scope.UpdateErr("ParseSingleExample", err)
+		return
+	}
+	return sparse_indices, sparse_values, sparse_shapes, dense_values
+}
+
+// Scatter `updates` into a new tensor according to `indices`.
+//
+// Creates a new tensor by applying sparse `updates` to individual values or
+// slices within a tensor (initially zero for numeric, empty for string) of
+// the given `shape` according to indices.  This operator is the inverse of the
+// `tf.gather_nd` operator which extracts values or slices from a given tensor.
+//
+// This operation is similar to tensor_scatter_add, except that the tensor is
+// zero-initialized. Calling `tf.scatter_nd(indices, values, shape)` is identical
+// to `tensor_scatter_add(tf.zeros(shape, values.dtype), indices, values)`
+//
+// If `indices` contains duplicates, then their updates are accumulated (summed).
+//
+// **WARNING**: The order in which updates are applied is nondeterministic, so the
+// output will be nondeterministic if `indices` contains duplicates -- because
+// of some numerical approximation issues, numbers summed in different order
+// may yield different results.
+//
+// `indices` is an integer tensor containing indices into a new tensor of shape
+// `shape`.  The last dimension of `indices` can be at most the rank of `shape`:
+//
+//     indices.shape[-1] <= shape.rank
+//
+// The last dimension of `indices` corresponds to indices into elements
+// (if `indices.shape[-1] = shape.rank`) or slices
+// (if `indices.shape[-1] < shape.rank`) along dimension `indices.shape[-1]` of
+// `shape`.  `updates` is a tensor with shape
+//
+//     indices.shape[:-1] + shape[indices.shape[-1]:]
+//
+// The simplest form of scatter is to insert individual elements in a tensor by
+// index. For example, say we want to insert 4 scattered elements in a rank-1
+// tensor with 8 elements.
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src="https://www.tensorflow.org/images/ScatterNd1.png" alt>
+// </div>
+//
+// In Python, this scatter operation would look like this:
+//
+// ```python
+//     indices = tf.constant([[4], [3], [1], [7]])
+//     updates = tf.constant([9, 10, 11, 12])
+//     shape = tf.constant([8])
+//     scatter = tf.scatter_nd(indices, updates, shape)
+//     print(scatter)
+// ```
+//
+// The resulting tensor would look like this:
+//
+//     [0, 11, 0, 10, 9, 0, 0, 12]
+//
+// We can also, insert entire slices of a higher rank tensor all at once. For
+// example, if we wanted to insert two slices in the first dimension of a
+// rank-3 tensor with two matrices of new values.
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src="https://www.tensorflow.org/images/ScatterNd2.png" alt>
+// </div>
+//
+// In Python, this scatter operation would look like this:
+//
+// ```python
+//     indices = tf.constant([[0], [2]])
+//     updates = tf.constant([[[5, 5, 5, 5], [6, 6, 6, 6],
+//                             [7, 7, 7, 7], [8, 8, 8, 8]],
+//                            [[5, 5, 5, 5], [6, 6, 6, 6],
+//                             [7, 7, 7, 7], [8, 8, 8, 8]]])
+//     shape = tf.constant([4, 4, 4])
+//     scatter = tf.scatter_nd(indices, updates, shape)
+//     print(scatter)
+// ```
+//
+// The resulting tensor would look like this:
+//
+//     [[[5, 5, 5, 5], [6, 6, 6, 6], [7, 7, 7, 7], [8, 8, 8, 8]],
+//      [[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]],
+//      [[5, 5, 5, 5], [6, 6, 6, 6], [7, 7, 7, 7], [8, 8, 8, 8]],
+//      [[0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0], [0, 0, 0, 0]]]
+//
+// Note that on CPU, if an out of bound index is found, an error is returned.
+// On GPU, if an out of bound index is found, the index is ignored.
+//
+// Arguments:
+//	indices: Index tensor.
+//	updates: Updates to scatter into output.
+//	shape: 1-D. The shape of the resulting tensor.
+//
+// Returns A new tensor with the given shape and updates applied according
+// to the indices.
+func ScatterNd(scope *Scope, indices tf.Output, updates tf.Output, shape tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ScatterNd",
+		Input: []tf.Input{
+			indices, updates, shape,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// UniqueAttr is an optional argument to Unique.
+type UniqueAttr func(optionalAttr)
+
+// UniqueOutIdx sets the optional out_idx attribute to value.
+// If not specified, defaults to DT_INT32
+func UniqueOutIdx(value tf.DataType) UniqueAttr {
+	return func(m optionalAttr) {
+		m["out_idx"] = value
+	}
+}
+
+// Finds unique elements in a 1-D tensor.
+//
+// This operation returns a tensor `y` containing all of the unique elements of `x`
+// sorted in the same order that they occur in `x`; `x` does not need to be sorted.
+// This operation also returns a tensor `idx` the same size as `x` that contains
+// the index of each value of `x` in the unique output `y`. In other words:
+//
+// `y[idx[i]] = x[i] for i in [0, 1,...,rank(x) - 1]`
+//
+// Examples:
+//
+// ```
+// # tensor 'x' is [1, 1, 2, 4, 4, 4, 7, 8, 8]
+// y, idx = unique(x)
+// y ==> [1, 2, 4, 7, 8]
+// idx ==> [0, 0, 1, 2, 2, 2, 3, 4, 4]
+// ```
+//
+// ```
+// # tensor 'x' is [4, 5, 1, 2, 3, 3, 4, 5]
+// y, idx = unique(x)
+// y ==> [4, 5, 1, 2, 3]
+// idx ==> [0, 1, 2, 3, 4, 4, 0, 1]
+// ```
+//
+// Arguments:
+//	x: 1-D.
+//
+// Returns:
+//	y: 1-D.
+//	idx: 1-D.
+func Unique(scope *Scope, x tf.Output, optional ...UniqueAttr) (y tf.Output, idx tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Unique",
+		Input: []tf.Input{
+			x,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Converts a `RaggedTensor` into a `SparseTensor` with the same values.
+//
+// input=ragged.from_nested_row_splits(rt_dense_values, rt_nested_splits)
+// output=SparseTensor(indices=sparse_indices, values=sparse_values,
+//                     dense_shape=sparse_dense_shape)
+//
+// Arguments:
+//	rt_nested_splits: The `row_splits` for the `RaggedTensor`.
+//	rt_dense_values: The `flat_values` for the `RaggedTensor`.
+//
+// Returns:
+//	sparse_indices: The indices for the `SparseTensor`.
+//	sparse_values: The values of the `SparseTensor`.
+//	sparse_dense_shape: `sparse_dense_shape` is a tight bounding box of the input `RaggedTensor`.
+func RaggedTensorToSparse(scope *Scope, rt_nested_splits []tf.Output, rt_dense_values tf.Output) (sparse_indices tf.Output, sparse_values tf.Output, sparse_dense_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "RaggedTensorToSparse",
+		Input: []tf.Input{
+			tf.OutputList(rt_nested_splits), rt_dense_values,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Returns the name of the device on which `resource` has been placed.
+func ExperimentalIteratorGetDevice(scope *Scope, resource tf.Output) (device tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalIteratorGetDevice",
+		Input: []tf.Input{
+			resource,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Records the bytes size of each element of `input_dataset` in a StatsAggregator.
+func ExperimentalBytesProducedStatsDataset(scope *Scope, input_dataset tf.Output, tag tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalBytesProducedStatsDataset",
+		Input: []tf.Input{
+			input_dataset, tag,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes exponential linear: `exp(features) - 1` if < 0, `features` otherwise.
+//
+// See [Fast and Accurate Deep Network Learning by Exponential Linear Units (ELUs)
+// ](http://arxiv.org/abs/1511.07289)
+func Elu(scope *Scope, features tf.Output) (activations tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Elu",
+		Input: []tf.Input{
+			features,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// AddSparseToTensorsMapAttr is an optional argument to AddSparseToTensorsMap.
+type AddSparseToTensorsMapAttr func(optionalAttr)
+
+// AddSparseToTensorsMapContainer sets the optional container attribute to value.
+//
+// value: The container name for the `SparseTensorsMap` created by this op.
+// If not specified, defaults to ""
+func AddSparseToTensorsMapContainer(value string) AddSparseToTensorsMapAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// AddSparseToTensorsMapSharedName sets the optional shared_name attribute to value.
+//
+// value: The shared name for the `SparseTensorsMap` created by this op.
+// If blank, the new Operation's unique name is used.
+// If not specified, defaults to ""
+func AddSparseToTensorsMapSharedName(value string) AddSparseToTensorsMapAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Add a `SparseTensor` to a `SparseTensorsMap` return its handle.
+//
+// A `SparseTensor` is represented by three tensors: `sparse_indices`,
+// `sparse_values`, and `sparse_shape`.
+//
+// This operator takes the given `SparseTensor` and adds it to a container
+// object (a `SparseTensorsMap`).  A unique key within this container is generated
+// in the form of an `int64`, and this is the value that is returned.
+//
+// The `SparseTensor` can then be read out as part of a minibatch by passing
+// the key as a vector element to `TakeManySparseFromTensorsMap`.  To ensure
+// the correct `SparseTensorsMap` is accessed, ensure that the same
+// `container` and `shared_name` are passed to that Op.  If no `shared_name`
+// is provided here, instead use the *name* of the Operation created by calling
+// `AddSparseToTensorsMap` as the `shared_name` passed to
+// `TakeManySparseFromTensorsMap`.  Ensure the Operations are colocated.
+//
+// Arguments:
+//	sparse_indices: 2-D.  The `indices` of the `SparseTensor`.
+//	sparse_values: 1-D.  The `values` of the `SparseTensor`.
+//	sparse_shape: 1-D.  The `shape` of the `SparseTensor`.
+//
+// Returns 0-D.  The handle of the `SparseTensor` now stored in the
+// `SparseTensorsMap`.
+func AddSparseToTensorsMap(scope *Scope, sparse_indices tf.Output, sparse_values tf.Output, sparse_shape tf.Output, optional ...AddSparseToTensorsMapAttr) (sparse_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "AddSparseToTensorsMap",
+		Input: []tf.Input{
+			sparse_indices, sparse_values, sparse_shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Transforms a vector of tf.Example protos (as strings) into typed tensors.
+//
+// Arguments:
+//	serialized: A scalar or vector containing binary serialized Example protos.
+//	names: A tensor containing the names of the serialized protos.
+// Corresponds 1:1 with the `serialized` tensor.
+// May contain, for example, table key (descriptive) names for the
+// corresponding serialized protos.  These are purely useful for debugging
+// purposes, and the presence of values here has no effect on the output.
+// May also be an empty vector if no names are available.
+// If non-empty, this tensor must have the same shape as "serialized".
+//	sparse_keys: Vector of strings.
+// The keys expected in the Examples' features associated with sparse values.
+//	dense_keys: Vector of strings.
+// The keys expected in the Examples' features associated with dense values.
+//	ragged_keys: Vector of strings.
+// The keys expected in the Examples' features associated with ragged values.
+//	dense_defaults: A list of Tensors (some may be empty).  Corresponds 1:1 with `dense_keys`.
+// dense_defaults[j] provides default values
+// when the example's feature_map lacks dense_key[j].  If an empty Tensor is
+// provided for dense_defaults[j], then the Feature dense_keys[j] is required.
+// The input type is inferred from dense_defaults[j], even when it's empty.
+// If dense_defaults[j] is not empty, and dense_shapes[j] is fully defined,
+// then the shape of dense_defaults[j] must match that of dense_shapes[j].
+// If dense_shapes[j] has an undefined major dimension (variable strides dense
+// feature), dense_defaults[j] must contain a single element:
+// the padding element.
+//	num_sparse: The number of sparse keys.
+//	sparse_types: A list of `num_sparse` types; the data types of data in each Feature
+// given in sparse_keys.
+// Currently the ParseExample supports DT_FLOAT (FloatList),
+// DT_INT64 (Int64List), and DT_STRING (BytesList).
+//	ragged_value_types: A list of `num_ragged` types; the data types of data in each Feature
+// given in ragged_keys (where `num_ragged = sparse_keys.size()`).
+// Currently the ParseExample supports DT_FLOAT (FloatList),
+// DT_INT64 (Int64List), and DT_STRING (BytesList).
+//	ragged_split_types: A list of `num_ragged` types; the data types of row_splits in each Feature
+// given in ragged_keys (where `num_ragged = sparse_keys.size()`).
+// May be DT_INT32 or DT_INT64.
+//	dense_shapes: A list of `num_dense` shapes; the shapes of data in each Feature
+// given in dense_keys (where `num_dense = dense_keys.size()`).
+// The number of elements in the Feature corresponding to dense_key[j]
+// must always equal dense_shapes[j].NumEntries().
+// If dense_shapes[j] == (D0, D1, ..., DN) then the shape of output
+// Tensor dense_values[j] will be (|serialized|, D0, D1, ..., DN):
+// The dense outputs are just the inputs row-stacked by batch.
+// This works for dense_shapes[j] = (-1, D1, ..., DN).  In this case
+// the shape of the output Tensor dense_values[j] will be
+// (|serialized|, M, D1, .., DN), where M is the maximum number of blocks
+// of elements of length D1 * .... * DN, across all minibatch entries
+// in the input.  Any minibatch entry with less than M blocks of elements of
+// length D1 * ... * DN will be padded with the corresponding default_value
+// scalar element along the second dimension.
+func ParseExampleV2(scope *Scope, serialized tf.Output, names tf.Output, sparse_keys tf.Output, dense_keys tf.Output, ragged_keys tf.Output, dense_defaults []tf.Output, num_sparse int64, sparse_types []tf.DataType, ragged_value_types []tf.DataType, ragged_split_types []tf.DataType, dense_shapes []tf.Shape) (sparse_indices []tf.Output, sparse_values []tf.Output, sparse_shapes []tf.Output, dense_values []tf.Output, ragged_values []tf.Output, ragged_row_splits []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_sparse": num_sparse, "sparse_types": sparse_types, "ragged_value_types": ragged_value_types, "ragged_split_types": ragged_split_types, "dense_shapes": dense_shapes}
+	opspec := tf.OpSpec{
+		Type: "ParseExampleV2",
+		Input: []tf.Input{
+			serialized, names, sparse_keys, dense_keys, ragged_keys, tf.OutputList(dense_defaults),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if sparse_indices, idx, err = makeOutputList(op, idx, "sparse_indices"); err != nil {
+		scope.UpdateErr("ParseExampleV2", err)
+		return
+	}
+	if sparse_values, idx, err = makeOutputList(op, idx, "sparse_values"); err != nil {
+		scope.UpdateErr("ParseExampleV2", err)
+		return
+	}
+	if sparse_shapes, idx, err = makeOutputList(op, idx, "sparse_shapes"); err != nil {
+		scope.UpdateErr("ParseExampleV2", err)
+		return
+	}
+	if dense_values, idx, err = makeOutputList(op, idx, "dense_values"); err != nil {
+		scope.UpdateErr("ParseExampleV2", err)
+		return
+	}
+	if ragged_values, idx, err = makeOutputList(op, idx, "ragged_values"); err != nil {
+		scope.UpdateErr("ParseExampleV2", err)
+		return
+	}
+	if ragged_row_splits, idx, err = makeOutputList(op, idx, "ragged_row_splits"); err != nil {
+		scope.UpdateErr("ParseExampleV2", err)
+		return
+	}
+	return sparse_indices, sparse_values, sparse_shapes, dense_values, ragged_values, ragged_row_splits
+}
+
+// Saves input tensors slices to disk.
+//
+// This is like `Save` except that tensors can be listed in the saved file as being
+// a slice of a larger tensor.  `shapes_and_slices` specifies the shape of the
+// larger tensor and the slice that this tensor covers. `shapes_and_slices` must
+// have as many elements as `tensor_names`.
+//
+// Elements of the `shapes_and_slices` input must either be:
+//
+// *  The empty string, in which case the corresponding tensor is
+//    saved normally.
+// *  A string of the form `dim0 dim1 ... dimN-1 slice-spec` where the
+//    `dimI` are the dimensions of the larger tensor and `slice-spec`
+//    specifies what part is covered by the tensor to save.
+//
+// `slice-spec` itself is a `:`-separated list: `slice0:slice1:...:sliceN-1`
+// where each `sliceI` is either:
+//
+// *  The string `-` meaning that the slice covers all indices of this dimension
+// *  `start,length` where `start` and `length` are integers.  In that
+//    case the slice covers `length` indices starting at `start`.
+//
+// See also `Save`.
+//
+// Arguments:
+//	filename: Must have a single element. The name of the file to which we write the
+// tensor.
+//	tensor_names: Shape `[N]`. The names of the tensors to be saved.
+//	shapes_and_slices: Shape `[N]`.  The shapes and slice specifications to use when
+// saving the tensors.
+//	data: `N` tensors to save.
+//
+// Returns the created operation.
+func SaveSlices(scope *Scope, filename tf.Output, tensor_names tf.Output, shapes_and_slices tf.Output, data []tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SaveSlices",
+		Input: []tf.Input{
+			filename, tensor_names, shapes_and_slices, tf.OutputList(data),
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// AvgPool3DAttr is an optional argument to AvgPool3D.
+type AvgPool3DAttr func(optionalAttr)
+
+// AvgPool3DDataFormat sets the optional data_format attribute to value.
+//
+// value: The data format of the input and output data. With the
+// default format "NDHWC", the data is stored in the order of:
+//     [batch, in_depth, in_height, in_width, in_channels].
+// Alternatively, the format could be "NCDHW", the data storage order is:
+//     [batch, in_channels, in_depth, in_height, in_width].
+// If not specified, defaults to "NDHWC"
+func AvgPool3DDataFormat(value string) AvgPool3DAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// Performs 3D average pooling on the input.
+//
+// Arguments:
+//	input: Shape `[batch, depth, rows, cols, channels]` tensor to pool over.
+//	ksize: 1-D tensor of length 5. The size of the window for each dimension of
+// the input tensor. Must have `ksize[0] = ksize[4] = 1`.
+//	strides: 1-D tensor of length 5. The stride of the sliding window for each
+// dimension of `input`. Must have `strides[0] = strides[4] = 1`.
+//	padding: The type of padding algorithm to use.
+//
+// Returns The average pooled output tensor.
+func AvgPool3D(scope *Scope, input tf.Output, ksize []int64, strides []int64, padding string, optional ...AvgPool3DAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"ksize": ksize, "strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "AvgPool3D",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// FusedBatchNormGradV3Attr is an optional argument to FusedBatchNormGradV3.
+type FusedBatchNormGradV3Attr func(optionalAttr)
+
+// FusedBatchNormGradV3Epsilon sets the optional epsilon attribute to value.
+//
+// value: A small float number added to the variance of x.
+// If not specified, defaults to 0.0001
+func FusedBatchNormGradV3Epsilon(value float32) FusedBatchNormGradV3Attr {
+	return func(m optionalAttr) {
+		m["epsilon"] = value
+	}
+}
+
+// FusedBatchNormGradV3DataFormat sets the optional data_format attribute to value.
+//
+// value: The data format for y_backprop, x, x_backprop.
+// Either "NHWC" (default) or "NCHW".
+// If not specified, defaults to "NHWC"
+func FusedBatchNormGradV3DataFormat(value string) FusedBatchNormGradV3Attr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// FusedBatchNormGradV3IsTraining sets the optional is_training attribute to value.
+//
+// value: A bool value to indicate the operation is for training (default)
+// or inference.
+// If not specified, defaults to true
+func FusedBatchNormGradV3IsTraining(value bool) FusedBatchNormGradV3Attr {
+	return func(m optionalAttr) {
+		m["is_training"] = value
+	}
+}
+
+// Gradient for batch normalization.
+//
+// Note that the size of 4D Tensors are defined by either "NHWC" or "NCHW".
+// The size of 1D Tensors matches the dimension C of the 4D Tensors.
+//
+// Arguments:
+//	y_backprop: A 4D Tensor for the gradient with respect to y.
+//	x: A 4D Tensor for input data.
+//	scale: A 1D Tensor for scaling factor, to scale the normalized x.
+//	reserve_space_1: When is_training is True, a 1D Tensor for the computed batch
+// mean to be reused in gradient computation. When is_training is
+// False, a 1D Tensor for the population mean to be reused in both
+// 1st and 2nd order gradient computation.
+//	reserve_space_2: When is_training is True, a 1D Tensor for the computed batch
+// variance (inverted variance in the cuDNN case) to be reused in
+// gradient computation. When is_training is False, a 1D Tensor
+// for the population variance to be reused in both 1st and 2nd
+// order gradient computation.
+//	reserve_space_3: When is_training is True, a 1D Tensor for some intermediate results to be reused
+// in gradient computation. When is_training is False, a dummy empty Tensor will be
+// created.
+//
+// Returns:
+//	x_backprop: A 4D Tensor for the gradient with respect to x.
+//	scale_backprop: A 1D Tensor for the gradient with respect to scale.
+//	offset_backprop: A 1D Tensor for the gradient with respect to offset.
+//	reserve_space_4: Unused placeholder to match the mean input in FusedBatchNorm.
+//	reserve_space_5: Unused placeholder to match the variance input
+// in FusedBatchNorm.
+func FusedBatchNormGradV3(scope *Scope, y_backprop tf.Output, x tf.Output, scale tf.Output, reserve_space_1 tf.Output, reserve_space_2 tf.Output, reserve_space_3 tf.Output, optional ...FusedBatchNormGradV3Attr) (x_backprop tf.Output, scale_backprop tf.Output, offset_backprop tf.Output, reserve_space_4 tf.Output, reserve_space_5 tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "FusedBatchNormGradV3",
+		Input: []tf.Input{
+			y_backprop, x, scale, reserve_space_1, reserve_space_2, reserve_space_3,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3), op.Output(4)
+}
+
+// Returns the number of records this Reader has produced.
+//
+// This is the same as the number of ReaderRead executions that have
+// succeeded.
+//
+// Arguments:
+//	reader_handle: Handle to a Reader.
+func ReaderNumRecordsProducedV2(scope *Scope, reader_handle tf.Output) (records_produced tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ReaderNumRecordsProducedV2",
+		Input: []tf.Input{
+			reader_handle,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// DecodeRawAttr is an optional argument to DecodeRaw.
+type DecodeRawAttr func(optionalAttr)
+
+// DecodeRawLittleEndian sets the optional little_endian attribute to value.
+//
+// value: Whether the input `bytes` are in little-endian order.
+// Ignored for `out_type` values that are stored in a single byte like
+// `uint8`.
+// If not specified, defaults to true
+func DecodeRawLittleEndian(value bool) DecodeRawAttr {
+	return func(m optionalAttr) {
+		m["little_endian"] = value
+	}
+}
+
+// Reinterpret the bytes of a string as a vector of numbers.
+//
+// Arguments:
+//	bytes: All the elements must have the same length.
+//
+//
+// Returns A Tensor with one more dimension than the input `bytes`.  The
+// added dimension will have size equal to the length of the elements
+// of `bytes` divided by the number of bytes to represent `out_type`.
+func DecodeRaw(scope *Scope, bytes tf.Output, out_type tf.DataType, optional ...DecodeRawAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"out_type": out_type}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DecodeRaw",
+		Input: []tf.Input{
+			bytes,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Gather ragged slices from `params` axis `0` according to `indices`.
+//
+// Outputs a `RaggedTensor` output composed from `output_dense_values` and
+// `output_nested_splits`, such that:
+//
+// ```python
+// output.shape = indices.shape + params.shape[1:]
+// output.ragged_rank = indices.shape.ndims + params.ragged_rank
+// output[i...j, d0...dn] = params[indices[i...j], d0...dn]
+// ```
+//
+// where
+//
+// * `params =
+//    ragged.from_nested_row_splits(params_dense_values, params_nested_splits)`
+//    provides the values that should be gathered.
+// * `indices` ia a dense tensor with dtype `int32` or `int64`, indicating which
+//    values should be gathered.
+// * `output =
+//    ragged.from_nested_row_splits(output_dense_values, output_nested_splits)`
+//    is the output tensor.
+//
+// (Note: This c++ op is used to implement the higher-level python
+// `tf.ragged.gather` op, which also supports ragged indices.)
+//
+//
+// Arguments:
+//	params_nested_splits: The `nested_row_splits` tensors that define the row-partitioning for the
+// `params` RaggedTensor input.
+//	params_dense_values: The `flat_values` for the `params` RaggedTensor. There was a terminology change
+// at the python level from dense_values to flat_values, so dense_values is the
+// deprecated name.
+//	indices: Indices in the outermost dimension of `params` of the values that should be
+// gathered.
+//	OUTPUT_RAGGED_RANK: The ragged rank of the output RaggedTensor. `output_nested_splits` will contain
+// this number of `row_splits` tensors. This value should equal
+// `indices.shape.ndims + params.ragged_rank - 1`.
+//
+// Returns:
+//	output_nested_splits: The `nested_row_splits` tensors that define the row-partitioning for the
+// returned RaggedTensor.
+//	output_dense_values: The `flat_values` for the returned RaggedTensor.
+func RaggedGather(scope *Scope, params_nested_splits []tf.Output, params_dense_values tf.Output, indices tf.Output, OUTPUT_RAGGED_RANK int64) (output_nested_splits []tf.Output, output_dense_values tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"OUTPUT_RAGGED_RANK": OUTPUT_RAGGED_RANK}
+	opspec := tf.OpSpec{
+		Type: "RaggedGather",
+		Input: []tf.Input{
+			tf.OutputList(params_nested_splits), params_dense_values, indices,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if output_nested_splits, idx, err = makeOutputList(op, idx, "output_nested_splits"); err != nil {
+		scope.UpdateErr("RaggedGather", err)
+		return
+	}
+	output_dense_values = op.Output(idx)
+	return output_nested_splits, output_dense_values
+}
+
+// QuantizeV2Attr is an optional argument to QuantizeV2.
+type QuantizeV2Attr func(optionalAttr)
+
+// QuantizeV2Mode sets the optional mode attribute to value.
+// If not specified, defaults to "MIN_COMBINED"
+func QuantizeV2Mode(value string) QuantizeV2Attr {
+	return func(m optionalAttr) {
+		m["mode"] = value
+	}
+}
+
+// QuantizeV2RoundMode sets the optional round_mode attribute to value.
+// If not specified, defaults to "HALF_AWAY_FROM_ZERO"
+func QuantizeV2RoundMode(value string) QuantizeV2Attr {
+	return func(m optionalAttr) {
+		m["round_mode"] = value
+	}
+}
+
+// QuantizeV2NarrowRange sets the optional narrow_range attribute to value.
+// If not specified, defaults to false
+func QuantizeV2NarrowRange(value bool) QuantizeV2Attr {
+	return func(m optionalAttr) {
+		m["narrow_range"] = value
+	}
+}
+
+// QuantizeV2Axis sets the optional axis attribute to value.
+// If not specified, defaults to -1
+func QuantizeV2Axis(value int64) QuantizeV2Attr {
+	return func(m optionalAttr) {
+		m["axis"] = value
+	}
+}
+
+// QuantizeV2EnsureMinimumRange sets the optional ensure_minimum_range attribute to value.
+// If not specified, defaults to 0.01
+func QuantizeV2EnsureMinimumRange(value float32) QuantizeV2Attr {
+	return func(m optionalAttr) {
+		m["ensure_minimum_range"] = value
+	}
+}
+
+// Quantize the 'input' tensor of type float to 'output' tensor of type 'T'.
+//
+// [min_range, max_range] are scalar floats that specify the range for
+// the 'input' data. The 'mode' attribute controls exactly which calculations are
+// used to convert the float values to their quantized equivalents.  The
+// 'round_mode' attribute controls which rounding tie-breaking algorithm is used
+// when rounding float values to their quantized equivalents.
+//
+// In 'MIN_COMBINED' mode, each value of the tensor will undergo the following:
+//
+// ```
+// out[i] = (in[i] - min_range) * range(T) / (max_range - min_range)
+// if T == qint8: out[i] -= (range(T) + 1) / 2.0
+// ```
+//
+// here `range(T) = numeric_limits<T>::max() - numeric_limits<T>::min()`
+//
+// *MIN_COMBINED Mode Example*
+//
+// Assume the input is type float and has a possible range of [0.0, 6.0] and the
+// output type is quint8 ([0, 255]). The min_range and max_range values should be
+// specified as 0.0 and 6.0. Quantizing from float to quint8 will multiply each
+// value of the input by 255/6 and cast to quint8.
+//
+// If the output type was qint8 ([-128, 127]), the operation will additionally
+// subtract each value by 128 prior to casting, so that the range of values aligns
+// with the range of qint8.
+//
+// If the mode is 'MIN_FIRST', then this approach is used:
+//
+// ```
+// num_discrete_values = 1 << (# of bits in T)
+// range_adjust = num_discrete_values / (num_discrete_values - 1)
+// range = (range_max - range_min) * range_adjust
+// range_scale = num_discrete_values / range
+// quantized = round(input * range_scale) - round(range_min * range_scale) +
+//   numeric_limits<T>::min()
+// quantized = max(quantized, numeric_limits<T>::min())
+// quantized = min(quantized, numeric_limits<T>::max())
+// ```
+//
+// The biggest difference between this and MIN_COMBINED is that the minimum range
+// is rounded first, before it's subtracted from the rounded value. With
+// MIN_COMBINED, a small bias is introduced where repeated iterations of quantizing
+// and dequantizing will introduce a larger and larger error.
+//
+// *SCALED mode Example*
+//
+// `SCALED` mode matches the quantization approach used in
+// `QuantizeAndDequantize{V2|V3}`.
+//
+// If the mode is `SCALED`, the quantization is performed by multiplying each
+// input value by a scaling_factor.
+// The scaling_factor is determined from `min_range` and `max_range` to be as large
+// as possible such that the range from `min_range` to `max_range` is representable
+// within values of type T.
+//
+// ```c++
+//
+//   const int min_T = std::numeric_limits<T>::min();
+//   const int max_T = std::numeric_limits<T>::max();
+//   const float max_float = std::numeric_limits<float>::max();
+//
+//   const float scale_factor_from_min_side =
+//       (min_T * min_range > 0) ? min_T / min_range : max_float;
+//   const float scale_factor_from_max_side =
+//       (max_T * max_range > 0) ? max_T / max_range : max_float;
+//
+//   const float scale_factor = std::min(scale_factor_from_min_side,
+//                                       scale_factor_from_max_side);
+// ```
+//
+// We next use the scale_factor to adjust min_range and max_range as follows:
+//
+// ```c++
+//       min_range = min_T / scale_factor;
+//       max_range = max_T / scale_factor;
+// ```
+//
+//
+// e.g. if T = qint8, and initially min_range = -10, and max_range = 9, we would
+// compare -128/-10.0 = 12.8 to 127/9.0 = 14.11, and set scaling_factor = 12.8
+// In this case, min_range would remain -10, but max_range would be adjusted to
+// 127 / 12.8 = 9.921875
+//
+// So we will quantize input values in the range (-10, 9.921875) to (-128, 127).
+//
+// The input tensor can now be quantized by clipping values to the range
+// `min_range` to `max_range`, then multiplying by scale_factor as follows:
+//
+// ```c++
+// result = round(min(max_range, max(min_range, input)) * scale_factor)
+// ```
+//
+// The adjusted `min_range` and `max_range` are returned as outputs 2 and 3 of
+// this operation. These outputs should be used as the range for any further
+// calculations.
+//
+//
+// *narrow_range (bool) attribute*
+//
+// If true, we do not use the minimum quantized value.
+// i.e. for int8 the quantized output, it would be restricted to the range
+// -127..127 instead of the full -128..127 range.
+// This is provided for compatibility with certain inference backends.
+// (Only applies to SCALED mode)
+//
+//
+// *axis (int) attribute*
+//
+// An optional `axis` attribute can specify a dimension index of the input tensor,
+// such that quantization ranges will be calculated and applied separately for each
+// slice of the tensor along that dimension. This is useful for per-channel
+// quantization.
+//
+// If axis is specified, min_range and max_range
+//
+// if `axis`=None, per-tensor quantization is performed as normal.
+//
+//
+// *ensure_minimum_range (float) attribute*
+//
+// Ensures the minimum quantization range is at least this value.
+// The legacy default value for this is 0.01, but it is strongly suggested to
+// set it to 0 for new uses.
+//
+//
+// Arguments:
+//
+//	min_range: The minimum value of the quantization range. This value may be adjusted by the
+// op depending on other parameters. The adjusted value is written to `output_min`.
+// If the `axis` attribute is specified, this must be a 1-D tensor whose size
+// matches the `axis` dimension of the input and output tensors.
+//	max_range: The maximum value of the quantization range. This value may be adjusted by the
+// op depending on other parameters. The adjusted value is written to `output_max`.
+// If the `axis` attribute is specified, this must be a 1-D tensor whose size
+// matches the `axis` dimension of the input and output tensors.
+//
+//
+// Returns:
+//	output: The quantized data produced from the float input.
+//	output_min: The final quantization range minimum, used to clip input values before scaling
+// and rounding them to quantized values.
+// If the `axis` attribute is specified, this will be a 1-D tensor whose size
+// matches the `axis` dimension of the input and output tensors.
+//	output_max: The final quantization range maximum, used to clip input values before scaling
+// and rounding them to quantized values.
+// If the `axis` attribute is specified, this will be a 1-D tensor whose size
+// matches the `axis` dimension of the input and output tensors.
+func QuantizeV2(scope *Scope, input tf.Output, min_range tf.Output, max_range tf.Output, T tf.DataType, optional ...QuantizeV2Attr) (output tf.Output, output_min tf.Output, output_max tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"T": T}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizeV2",
+		Input: []tf.Input{
+			input, min_range, max_range,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Returns the truth value of (x >= y) element-wise.
+//
+// *NOTE*: `GreaterEqual` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+//
+// Example:
+//
+// ```python
+// x = tf.constant([5, 4, 6, 7])
+// y = tf.constant([5, 2, 5, 10])
+// tf.math.greater_equal(x, y) ==> [True, True, True, False]
+//
+// x = tf.constant([5, 4, 6, 7])
+// y = tf.constant([5])
+// tf.math.greater_equal(x, y) ==> [True, False, True, True]
+// ```
+func GreaterEqual(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "GreaterEqual",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// BatchAttr is an optional argument to Batch.
+type BatchAttr func(optionalAttr)
+
+// BatchMaxEnqueuedBatches sets the optional max_enqueued_batches attribute to value.
+// If not specified, defaults to 10
+func BatchMaxEnqueuedBatches(value int64) BatchAttr {
+	return func(m optionalAttr) {
+		m["max_enqueued_batches"] = value
+	}
+}
+
+// BatchAllowedBatchSizes sets the optional allowed_batch_sizes attribute to value.
+// If not specified, defaults to <>
+func BatchAllowedBatchSizes(value []int64) BatchAttr {
+	return func(m optionalAttr) {
+		m["allowed_batch_sizes"] = value
+	}
+}
+
+// BatchContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func BatchContainer(value string) BatchAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// BatchSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func BatchSharedName(value string) BatchAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// BatchBatchingQueue sets the optional batching_queue attribute to value.
+// If not specified, defaults to ""
+func BatchBatchingQueue(value string) BatchAttr {
+	return func(m optionalAttr) {
+		m["batching_queue"] = value
+	}
+}
+
+// Batches all input tensors nondeterministically.
+//
+// When many instances of this Op are being run concurrently with the same
+// container/shared_name in the same device, some will output zero-shaped Tensors
+// and others will output Tensors of size up to max_batch_size.
+//
+// All Tensors in in_tensors are batched together (so, for example, labels and
+// features should be batched with a single instance of this operation.
+//
+// Each invocation of batch emits an `id` scalar which will be used to identify
+// this particular invocation when doing unbatch or its gradient.
+//
+// Each op which emits a non-empty batch will also emit a non-empty batch_index
+// Tensor, which, is a [K, 3] matrix where each row contains the invocation's id,
+// start, and length of elements of each set of Tensors present in batched_tensors.
+//
+// Batched tensors are concatenated along the first dimension, and all tensors in
+// in_tensors must have the first dimension of the same size.
+//
+// in_tensors: The tensors to be batched.
+// num_batch_threads: Number of scheduling threads for processing batches of work.
+//  Determines the number of batches processed in parallel.
+// max_batch_size: Batch sizes will never be bigger than this.
+// batch_timeout_micros: Maximum number of microseconds to wait before outputting
+//  an incomplete batch.
+// allowed_batch_sizes: Optional list of allowed batch sizes. If left empty, does
+//  nothing. Otherwise, supplies a list of batch sizes, causing the op to pad
+//  batches up to one of those sizes. The entries must increase monotonically, and
+//  the final entry must equal max_batch_size.
+// grad_timeout_micros: The timeout to use for the gradient. See Unbatch.
+// batched_tensors: Either empty tensors or a batch of concatenated Tensors.
+// batch_index: If out_tensors is non-empty, has information to invert it.
+// container: Controls the scope of sharing of this batch.
+// id: always contains a scalar with a unique ID for this invocation of Batch.
+// shared_name: Concurrently running instances of batch in the same device with the
+//  same container and shared_name will batch their elements together. If left
+//  empty, the op name will be used as the shared name.
+// T: the types of tensors to be batched.
+func Batch(scope *Scope, in_tensors []tf.Output, num_batch_threads int64, max_batch_size int64, batch_timeout_micros int64, grad_timeout_micros int64, optional ...BatchAttr) (batched_tensors []tf.Output, batch_index tf.Output, id tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_batch_threads": num_batch_threads, "max_batch_size": max_batch_size, "batch_timeout_micros": batch_timeout_micros, "grad_timeout_micros": grad_timeout_micros}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Batch",
+		Input: []tf.Input{
+			tf.OutputList(in_tensors),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if batched_tensors, idx, err = makeOutputList(op, idx, "batched_tensors"); err != nil {
+		scope.UpdateErr("Batch", err)
+		return
+	}
+	batch_index = op.Output(idx)
+	id = op.Output(idx)
+	return batched_tensors, batch_index, id
+}
+
+// UnicodeDecodeAttr is an optional argument to UnicodeDecode.
+type UnicodeDecodeAttr func(optionalAttr)
+
+// UnicodeDecodeErrors sets the optional errors attribute to value.
+//
+// value: Error handling policy when there is invalid formatting found in the input.
+// The value of 'strict' will cause the operation to produce a InvalidArgument
+// error on any invalid input formatting. A value of 'replace' (the default) will
+// cause the operation to replace any invalid formatting in the input with the
+// `replacement_char` codepoint. A value of 'ignore' will cause the operation to
+// skip any invalid formatting in the input and produce no corresponding output
+// character.
+// If not specified, defaults to "replace"
+func UnicodeDecodeErrors(value string) UnicodeDecodeAttr {
+	return func(m optionalAttr) {
+		m["errors"] = value
+	}
+}
+
+// UnicodeDecodeReplacementChar sets the optional replacement_char attribute to value.
+//
+// value: The replacement character codepoint to be used in place of any invalid
+// formatting in the input when `errors='replace'`. Any valid unicode codepoint may
+// be used. The default value is the default unicode replacement character is
+// 0xFFFD or U+65533.)
+// If not specified, defaults to 65533
+func UnicodeDecodeReplacementChar(value int64) UnicodeDecodeAttr {
+	return func(m optionalAttr) {
+		m["replacement_char"] = value
+	}
+}
+
+// UnicodeDecodeReplaceControlCharacters sets the optional replace_control_characters attribute to value.
+//
+// value: Whether to replace the C0 control characters (00-1F) with the
+// `replacement_char`. Default is false.
+// If not specified, defaults to false
+func UnicodeDecodeReplaceControlCharacters(value bool) UnicodeDecodeAttr {
+	return func(m optionalAttr) {
+		m["replace_control_characters"] = value
+	}
+}
+
+// UnicodeDecodeTsplits sets the optional Tsplits attribute to value.
+// If not specified, defaults to DT_INT64
+func UnicodeDecodeTsplits(value tf.DataType) UnicodeDecodeAttr {
+	return func(m optionalAttr) {
+		m["Tsplits"] = value
+	}
+}
+
+// Decodes each string in `input` into a sequence of Unicode code points.
+//
+// The character codepoints for all strings are returned using a single vector
+// `char_values`, with strings expanded to characters in row-major order.
+//
+// The `row_splits` tensor indicates where the codepoints for
+// each input string begin and end within the `char_values` tensor.
+// In particular, the values for the `i`th
+// string (in row-major order) are stored in the slice
+// `[row_splits[i]:row_splits[i+1]]`. Thus:
+//
+// * `char_values[row_splits[i]+j]` is the Unicode codepoint for the `j`th
+//   character in the `i`th string (in row-major order).
+// * `row_splits[i+1] - row_splits[i]` is the number of characters in the `i`th
+//   string (in row-major order).
+//
+// Arguments:
+//	input: The text to be decoded. Can have any shape. Note that the output is flattened
+// to a vector of char values.
+//	input_encoding: Text encoding of the input strings. This is any of the encodings supported
+// by ICU ucnv algorithmic converters. Examples: `"UTF-16", "US ASCII", "UTF-8"`.
+//
+// Returns:
+//	row_splits: A 1D int32 tensor containing the row splits.
+//	char_values: A 1D int32 Tensor containing the decoded codepoints.
+func UnicodeDecode(scope *Scope, input tf.Output, input_encoding string, optional ...UnicodeDecodeAttr) (row_splits tf.Output, char_values tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"input_encoding": input_encoding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "UnicodeDecode",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Create a dense tensor from a ragged tensor, possibly altering its shape.
+//
+// The `ragged_to_dense` op creates a dense tensor from a list of row partition
+// tensors, a value vector, and default values. If the shape is unspecified, the
+// minimal shape required to contain all the elements in the ragged tensor (the
+// natural shape) will be used. If some dimensions are left unspecified, then the
+// size of the natural shape is used in that dimension.
+//
+// The default_value will be broadcast to the output shape. After that, the values
+// from the ragged tensor overwrite the default values. Note that the default_value
+// must have less dimensions than the value.
+//
+// The row partition tensors are in the order of the dimensions.
+// At present, the types can be:
+// * "ROW_SPLITS": the row_splits tensor from the ragged tensor.
+// * "VALUE_ROWIDS": the value_rowids tensor from the ragged tensor.
+// * "FIRST_DIM_SIZE": if value_rowids is used for the first dimension, then it
+//   is preceded by "FIRST_DIM_SIZE".
+//
+// Arguments:
+//	shape: The desired shape of the the output tensor. If left unspecified (empty),
+// the minimal shape required to contain all the elements in the ragged tensor
+// (the natural shape) will be used. If some dimensions are left unspecified, then
+// the size of the natural shape is used in that dimension.
+//
+// Note that dense dimensions cannot be modified by the shape argument. Trying to
+// change the size of a dense dimension will cause the op to fail.
+// Examples:
+// natural shape: [4, 5, 6]
+// shape: -1
+// output shape: [4, 5, 6]
+//
+// natural shape: [4, 5, 6]
+// shape: [3, -1, 2]
+// output shape: [3, 5, 2]
+//
+// natural shape: [4, 5, 6]
+// shape: [3, 7, 2]
+// output shape: [3, 7, 2]
+//
+//	values: A 1D tensor representing the values of the ragged tensor.
+//	default_value: The default_value when the shape is larger than the ragged tensor. The
+// default_value is broadcast until it is the shape of the output tensor, and
+// then overwritten by values in the ragged tensor. The default value must be
+// compatible with this broadcast operation, and must have fewer dimensions than
+// the value tensor.
+//
+//	row_partition_types: The types of the row partition tensors. At present, these can be:
+// * "ROW_SPLITS": the row_splits tensor from the ragged tensor.
+// * "VALUE_ROWIDS": the value_rowids tensor from the ragged tensor.
+// * "FIRST_DIM_SIZE": if value_rowids is used for the first dimension, then it
+//   is preceeded by "FIRST_DIM_SIZE".
+// The tensors are in the order of the dimensions.
+//
+// Returns The resulting dense tensor.
+func RaggedTensorToTensor(scope *Scope, shape tf.Output, values tf.Output, default_value tf.Output, row_partition_tensors []tf.Output, row_partition_types []string) (result tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"row_partition_types": row_partition_types}
+	opspec := tf.OpSpec{
+		Type: "RaggedTensorToTensor",
+		Input: []tf.Input{
+			shape, values, default_value, tf.OutputList(row_partition_tensors),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// BatchMatMulAttr is an optional argument to BatchMatMul.
+type BatchMatMulAttr func(optionalAttr)
+
+// BatchMatMulAdjX sets the optional adj_x attribute to value.
+//
+// value: If `True`, adjoint the slices of `x`. Defaults to `False`.
+// If not specified, defaults to false
+func BatchMatMulAdjX(value bool) BatchMatMulAttr {
+	return func(m optionalAttr) {
+		m["adj_x"] = value
+	}
+}
+
+// BatchMatMulAdjY sets the optional adj_y attribute to value.
+//
+// value: If `True`, adjoint the slices of `y`. Defaults to `False`.
+// If not specified, defaults to false
+func BatchMatMulAdjY(value bool) BatchMatMulAttr {
+	return func(m optionalAttr) {
+		m["adj_y"] = value
+	}
+}
+
+// Multiplies slices of two tensors in batches.
+//
+// Multiplies all slices of `Tensor` `x` and `y` (each slice can be
+// viewed as an element of a batch), and arranges the individual results
+// in a single output tensor of the same batch size. Each of the
+// individual slices can optionally be adjointed (to adjoint a matrix
+// means to transpose and conjugate it) before multiplication by setting
+// the `adj_x` or `adj_y` flag to `True`, which are by default `False`.
+//
+// The input tensors `x` and `y` are 2-D or higher with shape `[..., r_x, c_x]`
+// and `[..., r_y, c_y]`.
+//
+// The output tensor is 2-D or higher with shape `[..., r_o, c_o]`, where:
+//
+//     r_o = c_x if adj_x else r_x
+//     c_o = r_y if adj_y else c_y
+//
+// It is computed as:
+//
+//     output[..., :, :] = matrix(x[..., :, :]) * matrix(y[..., :, :])
+//
+// Arguments:
+//	x: 2-D or higher with shape `[..., r_x, c_x]`.
+//	y: 2-D or higher with shape `[..., r_y, c_y]`.
+//
+// Returns 3-D or higher with shape `[..., r_o, c_o]`
+func BatchMatMul(scope *Scope, x tf.Output, y tf.Output, optional ...BatchMatMulAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "BatchMatMul",
+		Input: []tf.Input{
+			x, y,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RaggedTensorFromVariantAttr is an optional argument to RaggedTensorFromVariant.
+type RaggedTensorFromVariantAttr func(optionalAttr)
+
+// RaggedTensorFromVariantTsplits sets the optional Tsplits attribute to value.
+// If not specified, defaults to DT_INT64
+func RaggedTensorFromVariantTsplits(value tf.DataType) RaggedTensorFromVariantAttr {
+	return func(m optionalAttr) {
+		m["Tsplits"] = value
+	}
+}
+
+// Decodes a `variant` Tensor into a `RaggedTensor`.
+//
+// Decodes the given `variant` Tensor and returns a `RaggedTensor`. The input
+// could be a scalar, meaning it encodes a single `RaggedTensor` with ragged_rank
+// `output_ragged_rank`. It could also have an arbitrary rank, in which case each
+// element is decoded into a `RaggedTensor` with ragged_rank `input_ragged_rank`
+// and these are then stacked according to the input shape to output a single
+// `RaggedTensor` with ragged_rank `output_ragged_rank`. Each `variant` element in
+// the input Tensor is decoded by retrieving from the element a 1-D `variant`
+// Tensor with `input_ragged_rank + 1` Tensors, corresponding to the splits and
+// values of the decoded `RaggedTensor`. If `input_ragged_rank` is -1, then it is
+// inferred as `output_ragged_rank` - `rank(encoded_ragged)`. See
+// `RaggedTensorToVariant` for the corresponding encoding logic.
+//
+//
+// Arguments:
+//	encoded_ragged: A `variant` Tensor containing encoded `RaggedTensor`s.
+//	input_ragged_rank: The ragged rank of each encoded `RaggedTensor` component in the input. If set to
+// -1, this is inferred as `output_ragged_rank` - `rank(encoded_ragged)`
+//	output_ragged_rank: The expected ragged rank of the output `RaggedTensor`. The following must hold:
+// `output_ragged_rank = rank(encoded_ragged) + input_ragged_rank`.
+//
+//
+// Returns:
+//	output_nested_splits: A list of one or more Tensors representing the splits of the output
+// `RaggedTensor`.
+//	output_dense_values: A Tensor representing the values of the output `RaggedTensor`.
+func RaggedTensorFromVariant(scope *Scope, encoded_ragged tf.Output, input_ragged_rank int64, output_ragged_rank int64, Tvalues tf.DataType, optional ...RaggedTensorFromVariantAttr) (output_nested_splits []tf.Output, output_dense_values tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"input_ragged_rank": input_ragged_rank, "output_ragged_rank": output_ragged_rank, "Tvalues": Tvalues}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RaggedTensorFromVariant",
+		Input: []tf.Input{
+			encoded_ragged,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if output_nested_splits, idx, err = makeOutputList(op, idx, "output_nested_splits"); err != nil {
+		scope.UpdateErr("RaggedTensorFromVariant", err)
+		return
+	}
+	output_dense_values = op.Output(idx)
+	return output_nested_splits, output_dense_values
+}
+
+// RandomPoissonV2Attr is an optional argument to RandomPoissonV2.
+type RandomPoissonV2Attr func(optionalAttr)
+
+// RandomPoissonV2Seed sets the optional seed attribute to value.
+//
+// value: If either `seed` or `seed2` are set to be non-zero, the random number
+// generator is seeded by the given seed.  Otherwise, it is seeded by a
+// random seed.
+// If not specified, defaults to 0
+func RandomPoissonV2Seed(value int64) RandomPoissonV2Attr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// RandomPoissonV2Seed2 sets the optional seed2 attribute to value.
+//
+// value: A second seed to avoid seed collision.
+// If not specified, defaults to 0
+func RandomPoissonV2Seed2(value int64) RandomPoissonV2Attr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// RandomPoissonV2Dtype sets the optional dtype attribute to value.
+// If not specified, defaults to DT_INT64
+func RandomPoissonV2Dtype(value tf.DataType) RandomPoissonV2Attr {
+	return func(m optionalAttr) {
+		m["dtype"] = value
+	}
+}
+
+// Outputs random values from the Poisson distribution(s) described by rate.
+//
+// This op uses two algorithms, depending on rate. If rate >= 10, then
+// the algorithm by Hormann is used to acquire samples via
+// transformation-rejection.
+// See http://www.sciencedirect.com/science/article/pii/0167668793909974.
+//
+// Otherwise, Knuth's algorithm is used to acquire samples via multiplying uniform
+// random variables.
+// See Donald E. Knuth (1969). Seminumerical Algorithms. The Art of Computer
+// Programming, Volume 2. Addison Wesley
+//
+// Arguments:
+//	shape: 1-D integer tensor. Shape of independent samples to draw from each
+// distribution described by the shape parameters given in rate.
+//	rate: A tensor in which each scalar is a "rate" parameter describing the
+// associated poisson distribution.
+//
+// Returns A tensor with shape `shape + shape(rate)`. Each slice
+// `[:, ..., :, i0, i1, ...iN]` contains the samples drawn for
+// `rate[i0, i1, ...iN]`.
+func RandomPoissonV2(scope *Scope, shape tf.Output, rate tf.Output, optional ...RandomPoissonV2Attr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RandomPoissonV2",
+		Input: []tf.Input{
+			shape, rate,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that takes a Bernoulli sample of the contents of another dataset.
+//
+// There is no transformation in the `tf.data` Python API for creating this dataset.
+// Instead, it is created as a result of the `filter_with_random_uniform_fusion`
+// static optimization. Whether this optimization is performed is determined by the
+// `experimental_optimization.filter_with_random_uniform_fusion` option of
+// `tf.data.Options`.
+//
+// Arguments:
+//
+//	rate: A scalar representing the sample rate. Each element of `input_dataset` is
+// retained with this probability, independent of all other elements.
+//	seed: A scalar representing seed of random number generator.
+//	seed2: A scalar representing seed2 of random number generator.
+//
+//
+func SamplingDataset(scope *Scope, input_dataset tf.Output, rate tf.Output, seed tf.Output, seed2 tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "SamplingDataset",
+		Input: []tf.Input{
+			input_dataset, rate, seed, seed2,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Reads and outputs the entire contents of the input filename.
+func ReadFile(scope *Scope, filename tf.Output) (contents tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ReadFile",
+		Input: []tf.Input{
+			filename,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes requantization range per channel.
+//
+// Arguments:
+//	input: The original input tensor.
+//	input_min: The minimum value of the input tensor
+//	input_max: The maximum value of the input tensor.
+//	clip_value_max: The maximum value of the output that needs to be clipped.
+// Example: set this to 6 for Relu6.
+//
+// Returns:
+//	output_min: The minimum value of the final output tensor
+//	output_max: The maximum value of the final output tensor.
+func RequantizationRangePerChannel(scope *Scope, input tf.Output, input_min tf.Output, input_max tf.Output, clip_value_max float32) (output_min tf.Output, output_max tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"clip_value_max": clip_value_max}
+	opspec := tf.OpSpec{
+		Type: "RequantizationRangePerChannel",
+		Input: []tf.Input{
+			input, input_min, input_max,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// TruncatedNormalAttr is an optional argument to TruncatedNormal.
+type TruncatedNormalAttr func(optionalAttr)
+
+// TruncatedNormalSeed sets the optional seed attribute to value.
+//
+// value: If either `seed` or `seed2` are set to be non-zero, the random number
+// generator is seeded by the given seed.  Otherwise, it is seeded by a
+// random seed.
+// If not specified, defaults to 0
+func TruncatedNormalSeed(value int64) TruncatedNormalAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// TruncatedNormalSeed2 sets the optional seed2 attribute to value.
+//
+// value: A second seed to avoid seed collision.
+// If not specified, defaults to 0
+func TruncatedNormalSeed2(value int64) TruncatedNormalAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Outputs random values from a truncated normal distribution.
+//
+// The generated values follow a normal distribution with mean 0 and standard
+// deviation 1, except that values whose magnitude is more than 2 standard
+// deviations from the mean are dropped and re-picked.
+//
+// Arguments:
+//	shape: The shape of the output tensor.
+//	dtype: The type of the output.
+//
+// Returns A tensor of the specified shape filled with random truncated normal
+// values.
+func TruncatedNormal(scope *Scope, shape tf.Output, dtype tf.DataType, optional ...TruncatedNormalAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TruncatedNormal",
+		Input: []tf.Input{
+			shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ParameterizedTruncatedNormalAttr is an optional argument to ParameterizedTruncatedNormal.
+type ParameterizedTruncatedNormalAttr func(optionalAttr)
+
+// ParameterizedTruncatedNormalSeed sets the optional seed attribute to value.
+//
+// value: If either `seed` or `seed2` are set to be non-zero, the random number
+// generator is seeded by the given seed.  Otherwise, it is seeded by a
+// random seed.
+// If not specified, defaults to 0
+func ParameterizedTruncatedNormalSeed(value int64) ParameterizedTruncatedNormalAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// ParameterizedTruncatedNormalSeed2 sets the optional seed2 attribute to value.
+//
+// value: A second seed to avoid seed collision.
+// If not specified, defaults to 0
+func ParameterizedTruncatedNormalSeed2(value int64) ParameterizedTruncatedNormalAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Outputs random values from a normal distribution. The parameters may each be a
+//
+// scalar which applies to the entire output, or a vector of length shape[0] which
+// stores the parameters for each batch.
+//
+// Arguments:
+//	shape: The shape of the output tensor. Batches are indexed by the 0th dimension.
+//	means: The mean parameter of each batch.
+//	stdevs: The standard deviation parameter of each batch. Must be greater than 0.
+//	minvals: The minimum cutoff. May be -infinity.
+//	maxvals: The maximum cutoff. May be +infinity, and must be more than the minval
+// for each batch.
+//
+// Returns A matrix of shape num_batches x samples_per_batch, filled with random
+// truncated normal values using the parameters for each row.
+func ParameterizedTruncatedNormal(scope *Scope, shape tf.Output, means tf.Output, stdevs tf.Output, minvals tf.Output, maxvals tf.Output, optional ...ParameterizedTruncatedNormalAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ParameterizedTruncatedNormal",
+		Input: []tf.Input{
+			shape, means, stdevs, minvals, maxvals,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// QuantizedMatMulAttr is an optional argument to QuantizedMatMul.
+type QuantizedMatMulAttr func(optionalAttr)
+
+// QuantizedMatMulToutput sets the optional Toutput attribute to value.
+// If not specified, defaults to DT_QINT32
+func QuantizedMatMulToutput(value tf.DataType) QuantizedMatMulAttr {
+	return func(m optionalAttr) {
+		m["Toutput"] = value
+	}
+}
+
+// QuantizedMatMulTransposeA sets the optional transpose_a attribute to value.
+//
+// value: If true, `a` is transposed before multiplication.
+// If not specified, defaults to false
+func QuantizedMatMulTransposeA(value bool) QuantizedMatMulAttr {
+	return func(m optionalAttr) {
+		m["transpose_a"] = value
+	}
+}
+
+// QuantizedMatMulTransposeB sets the optional transpose_b attribute to value.
+//
+// value: If true, `b` is transposed before multiplication.
+// If not specified, defaults to false
+func QuantizedMatMulTransposeB(value bool) QuantizedMatMulAttr {
+	return func(m optionalAttr) {
+		m["transpose_b"] = value
+	}
+}
+
+// QuantizedMatMulTactivation sets the optional Tactivation attribute to value.
+//
+// value: The type of output produced by activation function
+// following this operation.
+// If not specified, defaults to DT_QUINT8
+func QuantizedMatMulTactivation(value tf.DataType) QuantizedMatMulAttr {
+	return func(m optionalAttr) {
+		m["Tactivation"] = value
+	}
+}
+
+// Perform a quantized matrix multiplication of  `a` by the matrix `b`.
+//
+// The inputs must be two-dimensional matrices and the inner dimension of
+// `a` (after being transposed if `transpose_a` is non-zero) must match the
+// outer dimension of `b` (after being transposed if `transposed_b` is
+// non-zero).
+//
+// Arguments:
+//	a: Must be a two-dimensional tensor.
+//	b: Must be a two-dimensional tensor.
+//	min_a: The float value that the lowest quantized `a` value represents.
+//	max_a: The float value that the highest quantized `a` value represents.
+//	min_b: The float value that the lowest quantized `b` value represents.
+//	max_b: The float value that the highest quantized `b` value represents.
+//
+// Returns:
+//	out
+//	min_out: The float value that the lowest quantized output value represents.
+//	max_out: The float value that the highest quantized output value represents.
+func QuantizedMatMul(scope *Scope, a tf.Output, b tf.Output, min_a tf.Output, max_a tf.Output, min_b tf.Output, max_b tf.Output, optional ...QuantizedMatMulAttr) (out tf.Output, min_out tf.Output, max_out tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizedMatMul",
+		Input: []tf.Input{
+			a, b, min_a, max_a, min_b, max_b,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Extract `patches` from `images` and put them in the "depth" output dimension.
+//
+// Arguments:
+//	images: 4-D Tensor with shape `[batch, in_rows, in_cols, depth]`.
+//	ksizes: The size of the sliding window for each dimension of `images`.
+//	strides: How far the centers of two consecutive patches are in
+// the images. Must be: `[1, stride_rows, stride_cols, 1]`.
+//	rates: Must be: `[1, rate_rows, rate_cols, 1]`. This is the
+// input stride, specifying how far two consecutive patch samples are in the
+// input. Equivalent to extracting patches with
+// `patch_sizes_eff = patch_sizes + (patch_sizes - 1) * (rates - 1)`, followed by
+// subsampling them spatially by a factor of `rates`. This is equivalent to
+// `rate` in dilated (a.k.a. Atrous) convolutions.
+//	padding: The type of padding algorithm to use.
+//
+// Returns 4-D Tensor with shape `[batch, out_rows, out_cols, ksize_rows *
+// ksize_cols * depth]` containing image patches with size
+// `ksize_rows x ksize_cols x depth` vectorized in the "depth" dimension. Note
+// `out_rows` and `out_cols` are the dimensions of the output patches.
+func ExtractImagePatches(scope *Scope, images tf.Output, ksizes []int64, strides []int64, rates []int64, padding string) (patches tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"ksizes": ksizes, "strides": strides, "rates": rates, "padding": padding}
+	opspec := tf.OpSpec{
+		Type: "ExtractImagePatches",
+		Input: []tf.Input{
+			images,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Forwards the value of an available tensor from `inputs` to `output`.
+//
+// `Merge` waits for at least one of the tensors in `inputs` to become available.
+// It is usually combined with `Switch` to implement branching.
+//
+// `Merge` forwards the first tensor to become available to `output`, and sets
+// `value_index` to its index in `inputs`.
+//
+// Arguments:
+//	inputs: The input tensors, exactly one of which will become available.
+//
+// Returns:
+//	output: Will be set to the available input tensor.
+//	value_index: The index of the chosen input tensor in `inputs`.
+func Merge(scope *Scope, inputs []tf.Output) (output tf.Output, value_index tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Merge",
+		Input: []tf.Input{
+			tf.OutputList(inputs),
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// PaddedBatchDatasetV2Attr is an optional argument to PaddedBatchDatasetV2.
+type PaddedBatchDatasetV2Attr func(optionalAttr)
+
+// PaddedBatchDatasetV2ParallelCopy sets the optional parallel_copy attribute to value.
+// If not specified, defaults to false
+func PaddedBatchDatasetV2ParallelCopy(value bool) PaddedBatchDatasetV2Attr {
+	return func(m optionalAttr) {
+		m["parallel_copy"] = value
+	}
+}
+
+// Creates a dataset that batches and pads `batch_size` elements from the input.
+//
+// Arguments:
+//
+//	batch_size: A scalar representing the number of elements to accumulate in a
+// batch.
+//	padded_shapes: A list of int64 tensors representing the desired padded shapes
+// of the corresponding output components. These shapes may be partially
+// specified, using `-1` to indicate that a particular dimension should be
+// padded to the maximum size of all batch elements.
+//	padding_values: A list of scalars containing the padding value to use for
+// each of the outputs.
+//	drop_remainder: A scalar representing whether the last batch should be dropped in case its size
+// is smaller than desired.
+//
+func PaddedBatchDatasetV2(scope *Scope, input_dataset tf.Output, batch_size tf.Output, padded_shapes []tf.Output, padding_values []tf.Output, drop_remainder tf.Output, output_shapes []tf.Shape, optional ...PaddedBatchDatasetV2Attr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_shapes": output_shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "PaddedBatchDatasetV2",
+		Input: []tf.Input{
+			input_dataset, batch_size, tf.OutputList(padded_shapes), tf.OutputList(padding_values), drop_remainder,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// BlockLSTMV2Attr is an optional argument to BlockLSTMV2.
+type BlockLSTMV2Attr func(optionalAttr)
+
+// BlockLSTMV2CellClip sets the optional cell_clip attribute to value.
+//
+// value: Value to clip the 'cs' value to.
+// If not specified, defaults to 0
+func BlockLSTMV2CellClip(value float32) BlockLSTMV2Attr {
+	return func(m optionalAttr) {
+		m["cell_clip"] = value
+	}
+}
+
+// BlockLSTMV2UsePeephole sets the optional use_peephole attribute to value.
+//
+// value: Whether to use peephole weights.
+// If not specified, defaults to false
+func BlockLSTMV2UsePeephole(value bool) BlockLSTMV2Attr {
+	return func(m optionalAttr) {
+		m["use_peephole"] = value
+	}
+}
+
+// Computes the LSTM cell forward propagation for all the time steps.
+//
+// This is equivalent to applying LSTMBlockCell in a loop, like so:
+//
+// ```python
+// for x1 in unpack(x):
+//   i1, cs1, f1, o1, ci1, co1, h1 = LSTMBlock(
+//     x1, cs_prev, h_prev, w, wci, wcf, wco, b)
+//   cs_prev = cs1
+//   h_prev = h1
+//   i.append(i1)
+//   cs.append(cs1)
+//   f.append(f1)
+//   o.append(o1)
+//   ci.append(ci1)
+//   co.append(co1)
+//   h.append(h1)
+// return pack(i), pack(cs), pack(f), pack(o), pack(ci), pack(ch), pack(h)
+//
+// Note that unlike LSTMBlockCell (and BlockLSTM) which uses ICFO gate layout,
+// this op uses IFCO. So in order for the following snippet to be equivalent
+// all gate-related outputs should be reordered.
+// ```
+//
+// Arguments:
+//	seq_len_max: Maximum time length actually used by this input. Outputs are padded
+// with zeros beyond this length.
+//	x: The sequence input to the LSTM, shape (timelen, batch_size, num_inputs).
+//	cs_prev: Value of the initial cell state.
+//	h_prev: Initial output of cell (to be used for peephole).
+//	w: The weight matrix.
+//	wci: The weight matrix for input gate peephole connection.
+//	wcf: The weight matrix for forget gate peephole connection.
+//	wco: The weight matrix for output gate peephole connection.
+//	b: The bias vector.
+//
+// Returns:
+//	i: The input gate over the whole time sequence.
+//	cs: The cell state before the tanh over the whole time sequence.
+//	f: The forget gate over the whole time sequence.
+//	o: The output gate over the whole time sequence.
+//	ci: The cell input over the whole time sequence.
+//	co: The cell after the tanh over the whole time sequence.
+//	h: The output h vector over the whole time sequence.
+func BlockLSTMV2(scope *Scope, seq_len_max tf.Output, x tf.Output, cs_prev tf.Output, h_prev tf.Output, w tf.Output, wci tf.Output, wcf tf.Output, wco tf.Output, b tf.Output, optional ...BlockLSTMV2Attr) (i tf.Output, cs tf.Output, f tf.Output, o tf.Output, ci tf.Output, co tf.Output, h tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "BlockLSTMV2",
+		Input: []tf.Input{
+			seq_len_max, x, cs_prev, h_prev, w, wci, wcf, wco, b,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3), op.Output(4), op.Output(5), op.Output(6)
+}
+
+// Return a tensor with the same shape and contents as the input tensor or value.
+func Identity(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Identity",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Outputs a `Summary` protocol buffer with scalar values.
+//
+// The input `tags` and `values` must have the same shape.  The generated summary
+// has a summary value for each tag-value pair in `tags` and `values`.
+//
+// Arguments:
+//	tags: Tags for the summary.
+//	values: Same shape as `tags.  Values for the summary.
+//
+// Returns Scalar.  Serialized `Summary` protocol buffer.
+func ScalarSummary(scope *Scope, tags tf.Output, values tf.Output) (summary tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ScalarSummary",
+		Input: []tf.Input{
+			tags, values,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResourceSparseApplyProximalAdagradAttr is an optional argument to ResourceSparseApplyProximalAdagrad.
+type ResourceSparseApplyProximalAdagradAttr func(optionalAttr)
+
+// ResourceSparseApplyProximalAdagradUseLocking sets the optional use_locking attribute to value.
+//
+// value: If True, updating of the var and accum tensors will be protected by
+// a lock; otherwise the behavior is undefined, but may exhibit less contention.
+// If not specified, defaults to false
+func ResourceSparseApplyProximalAdagradUseLocking(value bool) ResourceSparseApplyProximalAdagradAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// Sparse update entries in '*var' and '*accum' according to FOBOS algorithm.
+//
+// That is for rows we have grad for, we update var and accum as follows:
+// accum += grad * grad
+// prox_v = var
+// prox_v -= lr * grad * (1 / sqrt(accum))
+// var = sign(prox_v)/(1+lr*l2) * max{|prox_v|-lr*l1,0}
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	accum: Should be from a Variable().
+//	lr: Learning rate. Must be a scalar.
+//	l1: L1 regularization. Must be a scalar.
+//	l2: L2 regularization. Must be a scalar.
+//	grad: The gradient.
+//	indices: A vector of indices into the first dimension of var and accum.
+//
+// Returns the created operation.
+func ResourceSparseApplyProximalAdagrad(scope *Scope, var_ tf.Output, accum tf.Output, lr tf.Output, l1 tf.Output, l2 tf.Output, grad tf.Output, indices tf.Output, optional ...ResourceSparseApplyProximalAdagradAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceSparseApplyProximalAdagrad",
+		Input: []tf.Input{
+			var_, accum, lr, l1, l2, grad, indices,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Computes numerical negative value element-wise.
+//
+// I.e., \\(y = -x\\).
+func Neg(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Neg",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Calculates the gradient of the SparseMatrixSoftmax op.
+//
+// Arguments:
+//	softmax: A CSRSparseMatrix.
+//	grad_softmax: The gradient of `softmax`.
+//
+//
+// Returns The output gradient.
+func SparseMatrixSoftmaxGrad(scope *Scope, softmax tf.Output, grad_softmax tf.Output, type_ tf.DataType) (gradient tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"type": type_}
+	opspec := tf.OpSpec{
+		Type: "SparseMatrixSoftmaxGrad",
+		Input: []tf.Input{
+			softmax, grad_softmax,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the GRU cell back-propagation for 1 time step.
+//
+// Args
+//     x: Input to the GRU cell.
+//     h_prev: State input from the previous GRU cell.
+//     w_ru: Weight matrix for the reset and update gate.
+//     w_c: Weight matrix for the cell connection gate.
+//     b_ru: Bias vector for the reset and update gate.
+//     b_c: Bias vector for the cell connection gate.
+//     r: Output of the reset gate.
+//     u: Output of the update gate.
+//     c: Output of the cell connection gate.
+//     d_h: Gradients of the h_new wrt to objective function.
+//
+// Returns
+//     d_x: Gradients of the x wrt to objective function.
+//     d_h_prev: Gradients of the h wrt to objective function.
+//     d_c_bar Gradients of the c_bar wrt to objective function.
+//     d_r_bar_u_bar Gradients of the r_bar & u_bar wrt to objective function.
+//
+// This kernel op implements the following mathematical equations:
+//
+// Note on notation of the variables:
+//
+// Concatenation of a and b is represented by a_b
+// Element-wise dot product of a and b is represented by ab
+// Element-wise dot product is represented by \circ
+// Matrix multiplication is represented by *
+//
+// Additional notes for clarity:
+//
+// `w_ru` can be segmented into 4 different matrices.
+// ```
+// w_ru = [w_r_x w_u_x
+//         w_r_h_prev w_u_h_prev]
+// ```
+// Similarly, `w_c` can be segmented into 2 different matrices.
+// ```
+// w_c = [w_c_x w_c_h_prevr]
+// ```
+// Same goes for biases.
+// ```
+// b_ru = [b_ru_x b_ru_h]
+// b_c = [b_c_x b_c_h]
+// ```
+// Another note on notation:
+// ```
+// d_x = d_x_component_1 + d_x_component_2
+//
+// where d_x_component_1 = d_r_bar * w_r_x^T + d_u_bar * w_r_x^T
+// and d_x_component_2 = d_c_bar * w_c_x^T
+//
+// d_h_prev = d_h_prev_component_1 + d_h_prevr \circ r + d_h \circ u
+// where d_h_prev_componenet_1 = d_r_bar * w_r_h_prev^T + d_u_bar * w_r_h_prev^T
+// ```
+//
+// Mathematics behind the Gradients below:
+// ```
+// d_c_bar = d_h \circ (1-u) \circ (1-c \circ c)
+// d_u_bar = d_h \circ (h-c) \circ u \circ (1-u)
+//
+// d_r_bar_u_bar = [d_r_bar d_u_bar]
+//
+// [d_x_component_1 d_h_prev_component_1] = d_r_bar_u_bar * w_ru^T
+//
+// [d_x_component_2 d_h_prevr] = d_c_bar * w_c^T
+//
+// d_x = d_x_component_1 + d_x_component_2
+//
+// d_h_prev = d_h_prev_component_1 + d_h_prevr \circ r + u
+// ```
+// Below calculation is performed in the python wrapper for the Gradients
+// (not in the gradient kernel.)
+// ```
+// d_w_ru = x_h_prevr^T * d_c_bar
+//
+// d_w_c = x_h_prev^T * d_r_bar_u_bar
+//
+// d_b_ru = sum of d_r_bar_u_bar along axis = 0
+//
+// d_b_c = sum of d_c_bar along axis = 0
+// ```
+func GRUBlockCellGrad(scope *Scope, x tf.Output, h_prev tf.Output, w_ru tf.Output, w_c tf.Output, b_ru tf.Output, b_c tf.Output, r tf.Output, u tf.Output, c tf.Output, d_h tf.Output) (d_x tf.Output, d_h_prev tf.Output, d_c_bar tf.Output, d_r_bar_u_bar tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "GRUBlockCellGrad",
+		Input: []tf.Input{
+			x, h_prev, w_ru, w_c, b_ru, b_c, r, u, c, d_h,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3)
+}
+
+// TextLineReaderV2Attr is an optional argument to TextLineReaderV2.
+type TextLineReaderV2Attr func(optionalAttr)
+
+// TextLineReaderV2SkipHeaderLines sets the optional skip_header_lines attribute to value.
+//
+// value: Number of lines to skip from the beginning of every file.
+// If not specified, defaults to 0
+func TextLineReaderV2SkipHeaderLines(value int64) TextLineReaderV2Attr {
+	return func(m optionalAttr) {
+		m["skip_header_lines"] = value
+	}
+}
+
+// TextLineReaderV2Container sets the optional container attribute to value.
+//
+// value: If non-empty, this reader is placed in the given container.
+// Otherwise, a default container is used.
+// If not specified, defaults to ""
+func TextLineReaderV2Container(value string) TextLineReaderV2Attr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// TextLineReaderV2SharedName sets the optional shared_name attribute to value.
+//
+// value: If non-empty, this reader is named in the given bucket
+// with this shared_name. Otherwise, the node name is used instead.
+// If not specified, defaults to ""
+func TextLineReaderV2SharedName(value string) TextLineReaderV2Attr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// A Reader that outputs the lines of a file delimited by '\n'.
+//
+// Returns The handle to reference the Reader.
+func TextLineReaderV2(scope *Scope, optional ...TextLineReaderV2Attr) (reader_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TextLineReaderV2",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Encode audio data using the WAV file format.
+//
+// This operation will generate a string suitable to be saved out to create a .wav
+// audio file. It will be encoded in the 16-bit PCM format. It takes in float
+// values in the range -1.0f to 1.0f, and any outside that value will be clamped to
+// that range.
+//
+// `audio` is a 2-D float Tensor of shape `[length, channels]`.
+// `sample_rate` is a scalar Tensor holding the rate to use (e.g. 44100).
+//
+// Arguments:
+//	audio: 2-D with shape `[length, channels]`.
+//	sample_rate: Scalar containing the sample frequency.
+//
+// Returns 0-D. WAV-encoded file contents.
+func EncodeWav(scope *Scope, audio tf.Output, sample_rate tf.Output) (contents tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "EncodeWav",
+		Input: []tf.Input{
+			audio, sample_rate,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// EuclideanNormAttr is an optional argument to EuclideanNorm.
+type EuclideanNormAttr func(optionalAttr)
+
+// EuclideanNormKeepDims sets the optional keep_dims attribute to value.
+//
+// value: If true, retain reduced dimensions with length 1.
+// If not specified, defaults to false
+func EuclideanNormKeepDims(value bool) EuclideanNormAttr {
+	return func(m optionalAttr) {
+		m["keep_dims"] = value
+	}
+}
+
+// Computes the euclidean norm of elements across dimensions of a tensor.
+//
+// Reduces `input` along the dimensions given in `axis`. Unless
+// `keep_dims` is true, the rank of the tensor is reduced by 1 for each entry in
+// `axis`. If `keep_dims` is true, the reduced dimensions are
+// retained with length 1.
+//
+// Arguments:
+//	input: The tensor to reduce.
+//	axis: The dimensions to reduce. Must be in the range
+// `[-rank(input), rank(input))`.
+//
+// Returns The reduced tensor.
+func EuclideanNorm(scope *Scope, input tf.Output, axis tf.Output, optional ...EuclideanNormAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "EuclideanNorm",
+		Input: []tf.Input{
+			input, axis,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// IRFFT3DAttr is an optional argument to IRFFT3D.
+type IRFFT3DAttr func(optionalAttr)
+
+// IRFFT3DTreal sets the optional Treal attribute to value.
+// If not specified, defaults to DT_FLOAT
+func IRFFT3DTreal(value tf.DataType) IRFFT3DAttr {
+	return func(m optionalAttr) {
+		m["Treal"] = value
+	}
+}
+
+// Inverse 3D real-valued fast Fourier transform.
+//
+// Computes the inverse 3-dimensional discrete Fourier transform of a real-valued
+// signal over the inner-most 3 dimensions of `input`.
+//
+// The inner-most 3 dimensions of `input` are assumed to be the result of `RFFT3D`:
+// The inner-most dimension contains the `fft_length / 2 + 1` unique components of
+// the DFT of a real-valued signal. If `fft_length` is not provided, it is computed
+// from the size of the inner-most 3 dimensions of `input`. If the FFT length used
+// to compute `input` is odd, it should be provided since it cannot be inferred
+// properly.
+//
+// Along each axis `IRFFT3D` is computed on, if `fft_length` (or
+// `fft_length / 2 + 1` for the inner-most dimension) is smaller than the
+// corresponding dimension of `input`, the dimension is cropped. If it is larger,
+// the dimension is padded with zeros.
+//
+// Arguments:
+//	input: A complex tensor.
+//	fft_length: An int32 tensor of shape [3]. The FFT length for each dimension.
+//
+// Returns A float32 tensor of the same rank as `input`. The inner-most 3
+//   dimensions of `input` are replaced with the `fft_length` samples of their
+//   inverse 3D real Fourier transform.
+//
+// @compatibility(numpy)
+// Equivalent to np.irfftn with 3 dimensions.
+// @end_compatibility
+func IRFFT3D(scope *Scope, input tf.Output, fft_length tf.Output, optional ...IRFFT3DAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "IRFFT3D",
+		Input: []tf.Input{
+			input, fft_length,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns the element-wise min of two SparseTensors.
+//
+// Assumes the two SparseTensors have the same shape, i.e., no broadcasting.
+//
+// Arguments:
+//	a_indices: 2-D.  `N x R` matrix with the indices of non-empty values in a
+// SparseTensor, in the canonical lexicographic ordering.
+//	a_values: 1-D.  `N` non-empty values corresponding to `a_indices`.
+//	a_shape: 1-D.  Shape of the input SparseTensor.
+//	b_indices: counterpart to `a_indices` for the other operand.
+//	b_values: counterpart to `a_values` for the other operand; must be of the same dtype.
+//	b_shape: counterpart to `a_shape` for the other operand; the two shapes must be equal.
+//
+// Returns:
+//	output_indices: 2-D.  The indices of the output SparseTensor.
+//	output_values: 1-D.  The values of the output SparseTensor.
+func SparseSparseMinimum(scope *Scope, a_indices tf.Output, a_values tf.Output, a_shape tf.Output, b_indices tf.Output, b_values tf.Output, b_shape tf.Output) (output_indices tf.Output, output_values tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseSparseMinimum",
+		Input: []tf.Input{
+			a_indices, a_values, a_shape, b_indices, b_values, b_shape,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// StatefulStandardNormalV2Attr is an optional argument to StatefulStandardNormalV2.
+type StatefulStandardNormalV2Attr func(optionalAttr)
+
+// StatefulStandardNormalV2Dtype sets the optional dtype attribute to value.
+//
+// value: The type of the output.
+// If not specified, defaults to DT_FLOAT
+func StatefulStandardNormalV2Dtype(value tf.DataType) StatefulStandardNormalV2Attr {
+	return func(m optionalAttr) {
+		m["dtype"] = value
+	}
+}
+
+// Outputs random values from a normal distribution.
+//
+// The generated values will have mean 0 and standard deviation 1.
+//
+// Arguments:
+//	resource: The handle of the resource variable that stores the state of the RNG.
+//	algorithm: The RNG algorithm.
+//	shape: The shape of the output tensor.
+//
+// Returns A tensor of the specified shape filled with random normal values.
+func StatefulStandardNormalV2(scope *Scope, resource tf.Output, algorithm tf.Output, shape tf.Output, optional ...StatefulStandardNormalV2Attr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StatefulStandardNormalV2",
+		Input: []tf.Input{
+			resource, algorithm, shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResourceSparseApplyFtrlAttr is an optional argument to ResourceSparseApplyFtrl.
+type ResourceSparseApplyFtrlAttr func(optionalAttr)
+
+// ResourceSparseApplyFtrlUseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var and accum tensors will be protected
+// by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceSparseApplyFtrlUseLocking(value bool) ResourceSparseApplyFtrlAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// ResourceSparseApplyFtrlMultiplyLinearByLr sets the optional multiply_linear_by_lr attribute to value.
+// If not specified, defaults to false
+func ResourceSparseApplyFtrlMultiplyLinearByLr(value bool) ResourceSparseApplyFtrlAttr {
+	return func(m optionalAttr) {
+		m["multiply_linear_by_lr"] = value
+	}
+}
+
+// Update relevant entries in '*var' according to the Ftrl-proximal scheme.
+//
+// That is for rows we have grad for, we update var, accum and linear as follows:
+// accum_new = accum + grad * grad
+// linear += grad - (accum_new^(-lr_power) - accum^(-lr_power)) / lr * var
+// quadratic = 1.0 / (accum_new^(lr_power) * lr) + 2 * l2
+// var = (sign(linear) * l1 - linear) / quadratic if |linear| > l1 else 0.0
+// accum = accum_new
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	accum: Should be from a Variable().
+//	linear: Should be from a Variable().
+//	grad: The gradient.
+//	indices: A vector of indices into the first dimension of var and accum.
+//	lr: Scaling factor. Must be a scalar.
+//	l1: L1 regularization. Must be a scalar.
+//	l2: L2 regularization. Must be a scalar.
+//	lr_power: Scaling factor. Must be a scalar.
+//
+// Returns the created operation.
+func ResourceSparseApplyFtrl(scope *Scope, var_ tf.Output, accum tf.Output, linear tf.Output, grad tf.Output, indices tf.Output, lr tf.Output, l1 tf.Output, l2 tf.Output, lr_power tf.Output, optional ...ResourceSparseApplyFtrlAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceSparseApplyFtrl",
+		Input: []tf.Input{
+			var_, accum, linear, grad, indices, lr, l1, l2, lr_power,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Debugging/model interpretability outputs for each example.
+//
+// It traverses all the trees and computes debug metrics for individual examples,
+// such as getting split feature ids and logits after each split along the decision
+// path used to compute directional feature contributions.
+//
+// Arguments:
+//
+//	bucketized_features: A list of rank 1 Tensors containing bucket id for each
+// feature.
+//	logits_dimension: scalar, dimension of the logits, to be used for constructing the protos in
+// examples_debug_outputs_serialized.
+//
+// Returns Output rank 1 Tensor containing a proto serialized as a string for each example.
+func BoostedTreesExampleDebugOutputs(scope *Scope, tree_ensemble_handle tf.Output, bucketized_features []tf.Output, logits_dimension int64) (examples_debug_outputs_serialized tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"logits_dimension": logits_dimension}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesExampleDebugOutputs",
+		Input: []tf.Input{
+			tree_ensemble_handle, tf.OutputList(bucketized_features),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// StatefulUniformAttr is an optional argument to StatefulUniform.
+type StatefulUniformAttr func(optionalAttr)
+
+// StatefulUniformDtype sets the optional dtype attribute to value.
+//
+// value: The type of the output.
+// If not specified, defaults to DT_FLOAT
+func StatefulUniformDtype(value tf.DataType) StatefulUniformAttr {
+	return func(m optionalAttr) {
+		m["dtype"] = value
+	}
+}
+
+// Outputs random values from a uniform distribution.
+//
+// The generated values follow a uniform distribution in the range `[0, 1)`. The
+// lower bound 0 is included in the range, while the upper bound 1 is excluded.
+//
+// Arguments:
+//	resource: The handle of the resource variable that stores the state of the RNG.
+//	algorithm: The RNG algorithm.
+//	shape: The shape of the output tensor.
+//
+// Returns Random values with specified shape.
+func StatefulUniform(scope *Scope, resource tf.Output, algorithm tf.Output, shape tf.Output, optional ...StatefulUniformAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StatefulUniform",
+		Input: []tf.Input{
+			resource, algorithm, shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates ngrams from ragged string data.
+//
+// This op accepts a ragged tensor with 1 ragged dimension containing only
+// strings and outputs a ragged tensor with 1 ragged dimension containing ngrams
+// of that string, joined along the innermost axis.
+//
+// Arguments:
+//	data: The values tensor of the ragged string tensor to make ngrams out of. Must be a
+// 1D string tensor.
+//	data_splits: The splits tensor of the ragged string tensor to make ngrams out of.
+//	separator: The string to append between elements of the token. Use "" for no separator.
+//	ngram_widths: The sizes of the ngrams to create.
+//	left_pad: The string to use to pad the left side of the ngram sequence. Only used if
+// pad_width != 0.
+//	right_pad: The string to use to pad the right side of the ngram sequence. Only used if
+// pad_width != 0.
+//	pad_width: The number of padding elements to add to each side of each
+// sequence. Note that padding will never be greater than 'ngram_widths'-1
+// regardless of this value. If `pad_width=-1`, then add `max(ngram_widths)-1`
+// elements.
+//
+//
+// Returns:
+//	ngrams: The values tensor of the output ngrams ragged tensor.
+//	ngrams_splits: The splits tensor of the output ngrams ragged tensor.
+func StringNGrams(scope *Scope, data tf.Output, data_splits tf.Output, separator string, ngram_widths []int64, left_pad string, right_pad string, pad_width int64, preserve_short_sequences bool) (ngrams tf.Output, ngrams_splits tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"separator": separator, "ngram_widths": ngram_widths, "left_pad": left_pad, "right_pad": right_pad, "pad_width": pad_width, "preserve_short_sequences": preserve_short_sequences}
+	opspec := tf.OpSpec{
+		Type: "StringNGrams",
+		Input: []tf.Input{
+			data, data_splits,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Reduces sparse updates into the variable referenced by `resource` using the `min` operation.
+//
+// This operation computes
+//
+//     # Scalar indices
+//     ref[indices, ...] = min(ref[indices, ...], updates[...])
+//
+//     # Vector indices (for each i)
+//     ref[indices[i], ...] = min(ref[indices[i], ...], updates[i, ...])
+//
+//     # High rank indices (for each i, ..., j)
+//     ref[indices[i, ..., j], ...] = min(ref[indices[i, ..., j], ...], updates[i, ..., j, ...])
+//
+// Duplicate entries are handled correctly: if multiple `indices` reference
+// the same location, their contributions are combined.
+//
+// Requires `updates.shape = indices.shape + ref.shape[1:]` or `updates.shape = []`.
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src='https://www.tensorflow.org/images/ScatterAdd.png' alt>
+// </div>
+//
+// Arguments:
+//	resource: Should be from a `Variable` node.
+//	indices: A tensor of indices into the first dimension of `ref`.
+//	updates: A tensor of updated values to add to `ref`.
+//
+// Returns the created operation.
+func ResourceScatterMin(scope *Scope, resource tf.Output, indices tf.Output, updates tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceScatterMin",
+		Input: []tf.Input{
+			resource, indices, updates,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Multiplies sparse updates into the variable referenced by `resource`.
+//
+// This operation computes
+//
+//     # Scalar indices
+//     ref[indices, ...] *= updates[...]
+//
+//     # Vector indices (for each i)
+//     ref[indices[i], ...] *= updates[i, ...]
+//
+//     # High rank indices (for each i, ..., j)
+//     ref[indices[i, ..., j], ...] *= updates[i, ..., j, ...]
+//
+// Duplicate entries are handled correctly: if multiple `indices` reference
+// the same location, their contributions multiply.
+//
+// Requires `updates.shape = indices.shape + ref.shape[1:]` or `updates.shape = []`.
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src='https://www.tensorflow.org/images/ScatterAdd.png' alt>
+// </div>
+//
+// Arguments:
+//	resource: Should be from a `Variable` node.
+//	indices: A tensor of indices into the first dimension of `ref`.
+//	updates: A tensor of updated values to add to `ref`.
+//
+// Returns the created operation.
+func ResourceScatterMul(scope *Scope, resource tf.Output, indices tf.Output, updates tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceScatterMul",
+		Input: []tf.Input{
+			resource, indices, updates,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Compresses a dataset element.
+func CompressElement(scope *Scope, components []tf.Output) (compressed tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "CompressElement",
+		Input: []tf.Input{
+			tf.OutputList(components),
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MatMulAttr is an optional argument to MatMul.
+type MatMulAttr func(optionalAttr)
+
+// MatMulTransposeA sets the optional transpose_a attribute to value.
+//
+// value: If true, "a" is transposed before multiplication.
+// If not specified, defaults to false
+func MatMulTransposeA(value bool) MatMulAttr {
+	return func(m optionalAttr) {
+		m["transpose_a"] = value
+	}
+}
+
+// MatMulTransposeB sets the optional transpose_b attribute to value.
+//
+// value: If true, "b" is transposed before multiplication.
+// If not specified, defaults to false
+func MatMulTransposeB(value bool) MatMulAttr {
+	return func(m optionalAttr) {
+		m["transpose_b"] = value
+	}
+}
+
+// Multiply the matrix "a" by the matrix "b".
+//
+// The inputs must be two-dimensional matrices and the inner dimension of
+// "a" (after being transposed if transpose_a is true) must match the
+// outer dimension of "b" (after being transposed if transposed_b is
+// true).
+//
+// *Note*: The default kernel implementation for MatMul on GPUs uses
+// cublas.
+func MatMul(scope *Scope, a tf.Output, b tf.Output, optional ...MatMulAttr) (product tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MatMul",
+		Input: []tf.Input{
+			a, b,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SparseReduceSumSparseAttr is an optional argument to SparseReduceSumSparse.
+type SparseReduceSumSparseAttr func(optionalAttr)
+
+// SparseReduceSumSparseKeepDims sets the optional keep_dims attribute to value.
+//
+// value: If true, retain reduced dimensions with length 1.
+// If not specified, defaults to false
+func SparseReduceSumSparseKeepDims(value bool) SparseReduceSumSparseAttr {
+	return func(m optionalAttr) {
+		m["keep_dims"] = value
+	}
+}
+
+// Computes the sum of elements across dimensions of a SparseTensor.
+//
+// This Op takes a SparseTensor and is the sparse counterpart to
+// `tf.reduce_sum()`.  In contrast to SparseReduceSum, this Op returns a
+// SparseTensor.
+//
+// Reduces `sp_input` along the dimensions given in `reduction_axes`.  Unless
+// `keep_dims` is true, the rank of the tensor is reduced by 1 for each entry in
+// `reduction_axes`. If `keep_dims` is true, the reduced dimensions are retained
+// with length 1.
+//
+// If `reduction_axes` has no entries, all dimensions are reduced, and a tensor
+// with a single element is returned.  Additionally, the axes can be negative,
+// which are interpreted according to the indexing rules in Python.
+//
+// Arguments:
+//	input_indices: 2-D.  `N x R` matrix with the indices of non-empty values in a
+// SparseTensor, possibly not in canonical ordering.
+//	input_values: 1-D.  `N` non-empty values corresponding to `input_indices`.
+//	input_shape: 1-D.  Shape of the input SparseTensor.
+//	reduction_axes: 1-D.  Length-`K` vector containing the reduction axes.
+func SparseReduceSumSparse(scope *Scope, input_indices tf.Output, input_values tf.Output, input_shape tf.Output, reduction_axes tf.Output, optional ...SparseReduceSumSparseAttr) (output_indices tf.Output, output_values tf.Output, output_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseReduceSumSparse",
+		Input: []tf.Input{
+			input_indices, input_values, input_shape, reduction_axes,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Computes rectified linear: `max(features, 0)`.
+//
+// See: https://en.wikipedia.org/wiki/Rectifier_(neural_networks)
+// Example usage:
+// >>> tf.nn.relu([-2., 0., -0., 3.]).numpy()
+// array([ 0.,  0., -0.,  3.], dtype=float32)
+func Relu(scope *Scope, features tf.Output) (activations tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Relu",
+		Input: []tf.Input{
+			features,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Get the number of nodes in a tree
+//
+// Arguments:
+//	tree_handle: Handle to the tree resource.
+//
+// Returns The size of the tree.
+func TensorForestTreeSize(scope *Scope, tree_handle tf.Output) (tree_size tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorForestTreeSize",
+		Input: []tf.Input{
+			tree_handle,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Subtracts sparse updates from the variable referenced by `resource`.
+//
+// This operation computes
+//
+//     # Scalar indices
+//     ref[indices, ...] -= updates[...]
+//
+//     # Vector indices (for each i)
+//     ref[indices[i], ...] -= updates[i, ...]
+//
+//     # High rank indices (for each i, ..., j)
+//     ref[indices[i, ..., j], ...] -= updates[i, ..., j, ...]
+//
+// Duplicate entries are handled correctly: if multiple `indices` reference
+// the same location, their contributions add.
+//
+// Requires `updates.shape = indices.shape + ref.shape[1:]` or `updates.shape = []`.
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src='https://www.tensorflow.org/images/ScatterAdd.png' alt>
+// </div>
+//
+// Arguments:
+//	resource: Should be from a `Variable` node.
+//	indices: A tensor of indices into the first dimension of `ref`.
+//	updates: A tensor of updated values to add to `ref`.
+//
+// Returns the created operation.
+func ResourceScatterSub(scope *Scope, resource tf.Output, indices tf.Output, updates tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceScatterSub",
+		Input: []tf.Input{
+			resource, indices, updates,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Returns the cardinality of `input_dataset`.
+//
+// Returns the cardinality of `input_dataset`.
+//
+// Arguments:
+//	input_dataset: A variant tensor representing the dataset to return cardinality for.
+//
+// Returns The cardinality of `input_dataset`. Named constants are used to represent
+// infinite and unknown cardinality.
+func DatasetCardinality(scope *Scope, input_dataset tf.Output) (cardinality tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "DatasetCardinality",
+		Input: []tf.Input{
+			input_dataset,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that emits each dim-0 slice of `components` once.
+func TensorSliceDataset(scope *Scope, components []tf.Output, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "TensorSliceDataset",
+		Input: []tf.Input{
+			tf.OutputList(components),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RetrieveTPUEmbeddingMDLAdagradLightParametersAttr is an optional argument to RetrieveTPUEmbeddingMDLAdagradLightParameters.
+type RetrieveTPUEmbeddingMDLAdagradLightParametersAttr func(optionalAttr)
+
+// RetrieveTPUEmbeddingMDLAdagradLightParametersTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func RetrieveTPUEmbeddingMDLAdagradLightParametersTableId(value int64) RetrieveTPUEmbeddingMDLAdagradLightParametersAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingMDLAdagradLightParametersTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingMDLAdagradLightParametersTableName(value string) RetrieveTPUEmbeddingMDLAdagradLightParametersAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingMDLAdagradLightParametersConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingMDLAdagradLightParametersConfig(value string) RetrieveTPUEmbeddingMDLAdagradLightParametersAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Retrieve MDL Adagrad Light embedding parameters.
+//
+// An op that retrieves optimization parameters from embedding to host
+// memory. Must be preceded by a ConfigureTPUEmbeddingHost op that sets up
+// the correct embedding table configuration. For example, this op is
+// used to retrieve updated parameters before saving a checkpoint.
+//
+// Returns:
+//	parameters: Parameter parameters updated by the MDL Adagrad Light optimization algorithm.
+//	accumulators: Parameter accumulators updated by the MDL Adagrad Light optimization algorithm.
+//	weights: Parameter weights updated by the MDL Adagrad Light optimization algorithm.
+//	benefits: Parameter benefits updated by the MDL Adagrad Light optimization algorithm.
+func RetrieveTPUEmbeddingMDLAdagradLightParameters(scope *Scope, num_shards int64, shard_id int64, optional ...RetrieveTPUEmbeddingMDLAdagradLightParametersAttr) (parameters tf.Output, accumulators tf.Output, weights tf.Output, benefits tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RetrieveTPUEmbeddingMDLAdagradLightParameters",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3)
+}
+
+// Adds sparse updates to the variable referenced by `resource`.
+//
+// This operation computes
+//
+//     # Scalar indices
+//     ref[indices, ...] += updates[...]
+//
+//     # Vector indices (for each i)
+//     ref[indices[i], ...] += updates[i, ...]
+//
+//     # High rank indices (for each i, ..., j)
+//     ref[indices[i, ..., j], ...] += updates[i, ..., j, ...]
+//
+// Duplicate entries are handled correctly: if multiple `indices` reference
+// the same location, their contributions add.
+//
+// Requires `updates.shape = indices.shape + ref.shape[1:]` or `updates.shape = []`.
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src='https://www.tensorflow.org/images/ScatterAdd.png' alt>
+// </div>
+//
+// Arguments:
+//	resource: Should be from a `Variable` node.
+//	indices: A tensor of indices into the first dimension of `ref`.
+//	updates: A tensor of updated values to add to `ref`.
+//
+// Returns the created operation.
+func ResourceScatterAdd(scope *Scope, resource tf.Output, indices tf.Output, updates tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceScatterAdd",
+		Input: []tf.Input{
+			resource, indices, updates,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// This op consumes a lock created by `MutexLock`.
+//
+// This op exists to consume a tensor created by `MutexLock` (other than
+// direct control dependencies).  It should be the only that consumes the tensor,
+// and will raise an error if it is not.  Its only purpose is to keep the
+// mutex lock tensor alive until it is consumed by this op.
+//
+// **NOTE**: This operation must run on the same device as its input.  This may
+// be enforced via the `colocate_with` mechanism.
+//
+// Arguments:
+//	mutex_lock: A tensor returned by `MutexLock`.
+//
+// Returns the created operation.
+func ConsumeMutexLock(scope *Scope, mutex_lock tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ConsumeMutexLock",
+		Input: []tf.Input{
+			mutex_lock,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Adjust the contrast of one or more images.
+//
+// `images` is a tensor of at least 3 dimensions.  The last 3 dimensions are
+// interpreted as `[height, width, channels]`.  The other dimensions only
+// represent a collection of images, such as `[batch, height, width, channels].`
+//
+// Contrast is adjusted independently for each channel of each image.
+//
+// For each channel, the Op first computes the mean of the image pixels in the
+// channel and then adjusts each component of each pixel to
+// `(x - mean) * contrast_factor + mean`.
+//
+// Arguments:
+//	images: Images to adjust.  At least 3-D.
+//	contrast_factor: A float multiplier for adjusting contrast.
+//
+// Returns The contrast-adjusted image or images.
+func AdjustContrastv2(scope *Scope, images tf.Output, contrast_factor tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "AdjustContrastv2",
+		Input: []tf.Input{
+			images, contrast_factor,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Produces the average pool of the input tensor for quantized types.
+//
+// Arguments:
+//	input: 4-D with shape `[batch, height, width, channels]`.
+//	min_input: The float value that the lowest quantized input value represents.
+//	max_input: The float value that the highest quantized input value represents.
+//	ksize: The size of the window for each dimension of the input tensor.
+// The length must be 4 to match the number of dimensions of the input.
+//	strides: The stride of the sliding window for each dimension of the input
+// tensor.  The length must be 4 to match the number of dimensions of the input.
+//	padding: The type of padding algorithm to use.
+//
+// Returns:
+//	output
+//	min_output: The float value that the lowest quantized output value represents.
+//	max_output: The float value that the highest quantized output value represents.
+func QuantizedAvgPool(scope *Scope, input tf.Output, min_input tf.Output, max_input tf.Output, ksize []int64, strides []int64, padding string) (output tf.Output, min_output tf.Output, max_output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"ksize": ksize, "strides": strides, "padding": padding}
+	opspec := tf.OpSpec{
+		Type: "QuantizedAvgPool",
+		Input: []tf.Input{
+			input, min_input, max_input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// VariableShapeAttr is an optional argument to VariableShape.
+type VariableShapeAttr func(optionalAttr)
+
+// VariableShapeOutType sets the optional out_type attribute to value.
+// If not specified, defaults to DT_INT32
+func VariableShapeOutType(value tf.DataType) VariableShapeAttr {
+	return func(m optionalAttr) {
+		m["out_type"] = value
+	}
+}
+
+// Returns the shape of the variable pointed to by `resource`.
+//
+// This operation returns a 1-D integer tensor representing the shape of `input`.
+//
+// For example:
+//
+// ```
+// # 't' is [[[1, 1, 1], [2, 2, 2]], [[3, 3, 3], [4, 4, 4]]]
+// shape(t) ==> [2, 2, 3]
+// ```
+func VariableShape(scope *Scope, input tf.Output, optional ...VariableShapeAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "VariableShape",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the minimum along segments of a tensor.
+//
+// Read
+// [the section on segmentation](https://tensorflow.org/api_docs/python/tf/math#Segmentation)
+// for an explanation of segments.
+//
+// This operator is similar to the unsorted segment sum operator found
+// [(here)](../../../api_docs/python/math_ops.md#UnsortedSegmentSum).
+// Instead of computing the sum over segments, it computes the minimum such that:
+//
+// \\(output_i = \min_{j...} data_[j...]\\) where min is over tuples `j...` such
+// that `segment_ids[j...] == i`.
+//
+// If the minimum is empty for a given segment ID `i`, it outputs the largest
+// possible value for the specific numeric type,
+// `output[i] = numeric_limits<T>::max()`.
+//
+// For example:
+//
+// ``` python
+// c = tf.constant([[1,2,3,4], [5,6,7,8], [4,3,2,1]])
+// tf.unsorted_segment_min(c, tf.constant([0, 1, 0]), num_segments=2)
+// # ==> [[ 1,  2, 2, 1],
+// #       [5,  6, 7, 8]]
+// ```
+//
+// If the given segment ID `i` is negative, then the corresponding value is
+// dropped, and will not be included in the result.
+//
+// Arguments:
+//
+//	segment_ids: A tensor whose shape is a prefix of `data.shape`.
+//
+//
+// Returns Has same shape as data, except for the first `segment_ids.rank`
+// dimensions, which are replaced with a single dimension which has size
+// `num_segments`.
+func UnsortedSegmentMin(scope *Scope, data tf.Output, segment_ids tf.Output, num_segments tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "UnsortedSegmentMin",
+		Input: []tf.Input{
+			data, segment_ids, num_segments,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResourceScatterNdSubAttr is an optional argument to ResourceScatterNdSub.
+type ResourceScatterNdSubAttr func(optionalAttr)
+
+// ResourceScatterNdSubUseLocking sets the optional use_locking attribute to value.
+//
+// value: An optional bool. Defaults to True. If True, the assignment will
+// be protected by a lock; otherwise the behavior is undefined,
+// but may exhibit less contention.
+// If not specified, defaults to true
+func ResourceScatterNdSubUseLocking(value bool) ResourceScatterNdSubAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// Applies sparse subtraction to individual values or slices in a Variable.
+//
+// `ref` is a `Tensor` with rank `P` and `indices` is a `Tensor` of rank `Q`.
+//
+// `indices` must be integer tensor, containing indices into `ref`.
+// It must be shape `[d_0, ..., d_{Q-2}, K]` where `0 < K <= P`.
+//
+// The innermost dimension of `indices` (with length `K`) corresponds to
+// indices into elements (if `K = P`) or slices (if `K < P`) along the `K`th
+// dimension of `ref`.
+//
+// `updates` is `Tensor` of rank `Q-1+P-K` with shape:
+//
+// ```
+// [d_0, ..., d_{Q-2}, ref.shape[K], ..., ref.shape[P-1]]
+// ```
+//
+// For example, say we want to subtract 4 scattered elements from a rank-1 tensor
+// with 8 elements. In Python, that subtraction would look like this:
+//
+// ```python
+// ref = tf.Variable([1, 2, 3, 4, 5, 6, 7, 8], use_resource=True)
+// indices = tf.constant([[4], [3], [1], [7]])
+// updates = tf.constant([9, 10, 11, 12])
+// sub = tf.scatter_nd_sub(ref, indices, updates)
+// with tf.Session() as sess:
+//   print sess.run(sub)
+// ```
+//
+// The resulting update to ref would look like this:
+//
+//     [1, -9, 3, -6, -4, 6, 7, -4]
+//
+// See `tf.scatter_nd` for more details about how to make updates to
+// slices.
+//
+// Arguments:
+//	ref: A resource handle. Must be from a VarHandleOp.
+//	indices: A Tensor. Must be one of the following types: int32, int64.
+// A tensor of indices into ref.
+//	updates: A Tensor. Must have the same type as ref. A tensor of
+// values to add to ref.
+//
+// Returns the created operation.
+func ResourceScatterNdSub(scope *Scope, ref tf.Output, indices tf.Output, updates tf.Output, optional ...ResourceScatterNdSubAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceScatterNdSub",
+		Input: []tf.Input{
+			ref, indices, updates,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// DataFormatDimMapAttr is an optional argument to DataFormatDimMap.
+type DataFormatDimMapAttr func(optionalAttr)
+
+// DataFormatDimMapSrcFormat sets the optional src_format attribute to value.
+//
+// value: source data format.
+// If not specified, defaults to "NHWC"
+func DataFormatDimMapSrcFormat(value string) DataFormatDimMapAttr {
+	return func(m optionalAttr) {
+		m["src_format"] = value
+	}
+}
+
+// DataFormatDimMapDstFormat sets the optional dst_format attribute to value.
+//
+// value: destination data format.
+// If not specified, defaults to "NCHW"
+func DataFormatDimMapDstFormat(value string) DataFormatDimMapAttr {
+	return func(m optionalAttr) {
+		m["dst_format"] = value
+	}
+}
+
+// Returns the dimension index in the destination data format given the one in
+//
+// the source data format.
+//
+// Arguments:
+//	x: A Tensor with each element as a dimension index in source data format.
+// Must be in the range [-4, 4).
+//
+// Returns A Tensor with each element as a dimension index in destination data format.
+func DataFormatDimMap(scope *Scope, x tf.Output, optional ...DataFormatDimMapAttr) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DataFormatDimMap",
+		Input: []tf.Input{
+			x,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Assigns a new value to a variable.
+//
+// Any ReadVariableOp with a control dependency on this op is guaranteed to return
+// this value or a subsequent newer value of the variable.
+//
+// Arguments:
+//	resource: handle to the resource in which to store the variable.
+//	value: the value to set the new tensor to use.
+//
+// Returns the created operation.
+func AssignVariableOp(scope *Scope, resource tf.Output, value tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "AssignVariableOp",
+		Input: []tf.Input{
+			resource, value,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// UpperBoundAttr is an optional argument to UpperBound.
+type UpperBoundAttr func(optionalAttr)
+
+// UpperBoundOutType sets the optional out_type attribute to value.
+// If not specified, defaults to DT_INT32
+func UpperBoundOutType(value tf.DataType) UpperBoundAttr {
+	return func(m optionalAttr) {
+		m["out_type"] = value
+	}
+}
+
+// Applies upper_bound(sorted_search_values, values) along each row.
+//
+// Each set of rows with the same index in (sorted_inputs, values) is treated
+// independently.  The resulting row is the equivalent of calling
+// `np.searchsorted(sorted_inputs, values, side='right')`.
+//
+// The result is not a global index to the entire
+// `Tensor`, but rather just the index in the last dimension.
+//
+// A 2-D example:
+//   sorted_sequence = [[0, 3, 9, 9, 10],
+//                      [1, 2, 3, 4, 5]]
+//   values = [[2, 4, 9],
+//             [0, 2, 6]]
+//
+//   result = UpperBound(sorted_sequence, values)
+//
+//   result == [[1, 2, 4],
+//              [0, 2, 5]]
+//
+// Arguments:
+//	sorted_inputs: 2-D Tensor where each row is ordered.
+//	values: 2-D Tensor with the same numbers of rows as `sorted_search_values`. Contains
+// the values that will be searched for in `sorted_search_values`.
+//
+// Returns A `Tensor` with the same shape as `values`.  It contains the last scalar index
+// into the last dimension where values can be inserted without changing the
+// ordered property.
+func UpperBound(scope *Scope, sorted_inputs tf.Output, values tf.Output, optional ...UpperBoundAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "UpperBound",
+		Input: []tf.Input{
+			sorted_inputs, values,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResourceApplyFtrlV2Attr is an optional argument to ResourceApplyFtrlV2.
+type ResourceApplyFtrlV2Attr func(optionalAttr)
+
+// ResourceApplyFtrlV2UseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var and accum tensors will be protected
+// by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceApplyFtrlV2UseLocking(value bool) ResourceApplyFtrlV2Attr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// ResourceApplyFtrlV2MultiplyLinearByLr sets the optional multiply_linear_by_lr attribute to value.
+// If not specified, defaults to false
+func ResourceApplyFtrlV2MultiplyLinearByLr(value bool) ResourceApplyFtrlV2Attr {
+	return func(m optionalAttr) {
+		m["multiply_linear_by_lr"] = value
+	}
+}
+
+// Update '*var' according to the Ftrl-proximal scheme.
+//
+// grad_with_shrinkage = grad + 2 * l2_shrinkage * var
+// accum_new = accum + grad_with_shrinkage * grad_with_shrinkage
+// linear += grad_with_shrinkage +
+//     (accum_new^(-lr_power) - accum^(-lr_power)) / lr * var
+// quadratic = 1.0 / (accum_new^(lr_power) * lr) + 2 * l2
+// var = (sign(linear) * l1 - linear) / quadratic if |linear| > l1 else 0.0
+// accum = accum_new
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	accum: Should be from a Variable().
+//	linear: Should be from a Variable().
+//	grad: The gradient.
+//	lr: Scaling factor. Must be a scalar.
+//	l1: L1 regularization. Must be a scalar.
+//	l2: L2 shrinkage regularization. Must be a scalar.
+//
+//	lr_power: Scaling factor. Must be a scalar.
+//
+// Returns the created operation.
+func ResourceApplyFtrlV2(scope *Scope, var_ tf.Output, accum tf.Output, linear tf.Output, grad tf.Output, lr tf.Output, l1 tf.Output, l2 tf.Output, l2_shrinkage tf.Output, lr_power tf.Output, optional ...ResourceApplyFtrlV2Attr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceApplyFtrlV2",
+		Input: []tf.Input{
+			var_, accum, linear, grad, lr, l1, l2, l2_shrinkage, lr_power,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Deprecated. Use TensorArraySplitV3
+//
+// DEPRECATED at GraphDef version 26: Use TensorArraySplitV3
+func TensorArraySplitV2(scope *Scope, handle tf.Output, value tf.Output, lengths tf.Output, flow_in tf.Output) (flow_out tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorArraySplitV2",
+		Input: []tf.Input{
+			handle, value, lengths, flow_in,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ComputeAccidentalHitsAttr is an optional argument to ComputeAccidentalHits.
+type ComputeAccidentalHitsAttr func(optionalAttr)
+
+// ComputeAccidentalHitsSeed sets the optional seed attribute to value.
+//
+// value: If either seed or seed2 are set to be non-zero, the random number
+// generator is seeded by the given seed.  Otherwise, it is seeded by a
+// random seed.
+// If not specified, defaults to 0
+func ComputeAccidentalHitsSeed(value int64) ComputeAccidentalHitsAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// ComputeAccidentalHitsSeed2 sets the optional seed2 attribute to value.
+//
+// value: An second seed to avoid seed collision.
+// If not specified, defaults to 0
+func ComputeAccidentalHitsSeed2(value int64) ComputeAccidentalHitsAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Computes the ids of the positions in sampled_candidates that match true_labels.
+//
+// When doing log-odds NCE, the result of this op should be passed through a
+// SparseToDense op, then added to the logits of the sampled candidates. This has
+// the effect of 'removing' the sampled labels that match the true labels by
+// making the classifier sure that they are sampled labels.
+//
+// Arguments:
+//	true_classes: The true_classes output of UnpackSparseLabels.
+//	sampled_candidates: The sampled_candidates output of CandidateSampler.
+//	num_true: Number of true labels per context.
+//
+// Returns:
+//	indices: A vector of indices corresponding to rows of true_candidates.
+//	ids: A vector of IDs of positions in sampled_candidates that match a true_label
+// for the row with the corresponding index in indices.
+//	weights: A vector of the same length as indices and ids, in which each element
+// is -FLOAT_MAX.
+func ComputeAccidentalHits(scope *Scope, true_classes tf.Output, sampled_candidates tf.Output, num_true int64, optional ...ComputeAccidentalHitsAttr) (indices tf.Output, ids tf.Output, weights tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_true": num_true}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ComputeAccidentalHits",
+		Input: []tf.Input{
+			true_classes, sampled_candidates,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// VarHandleOpAttr is an optional argument to VarHandleOp.
+type VarHandleOpAttr func(optionalAttr)
+
+// VarHandleOpContainer sets the optional container attribute to value.
+//
+// value: the container this variable is placed in.
+// If not specified, defaults to ""
+func VarHandleOpContainer(value string) VarHandleOpAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// VarHandleOpSharedName sets the optional shared_name attribute to value.
+//
+// value: the name by which this variable is referred to.
+// If not specified, defaults to ""
+func VarHandleOpSharedName(value string) VarHandleOpAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// VarHandleOpAllowedDevices sets the optional allowed_devices attribute to value.
+//
+// value: The allowed devices containing the resource variable. Set when the output
+// ResourceHandle represents a per-replica/partitioned resource variable.
+// If not specified, defaults to <>
+func VarHandleOpAllowedDevices(value []string) VarHandleOpAttr {
+	return func(m optionalAttr) {
+		m["allowed_devices"] = value
+	}
+}
+
+// Creates a handle to a Variable resource.
+//
+// Arguments:
+//	dtype: the type of this variable. Must agree with the dtypes
+// of all ops using this variable.
+//	shape: The (possibly partially specified) shape of this variable.
+func VarHandleOp(scope *Scope, dtype tf.DataType, shape tf.Shape, optional ...VarHandleOpAttr) (resource tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype, "shape": shape}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "VarHandleOp",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns (x - y)(x - y) element-wise.
+//
+// *NOTE*: `SquaredDifference` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func SquaredDifference(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SquaredDifference",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that emits the records from one or more TFRecord files.
+//
+// Arguments:
+//	filenames: A scalar or vector containing the name(s) of the file(s) to be
+// read.
+//	compression_type: A scalar containing either (i) the empty string (no
+// compression), (ii) "ZLIB", or (iii) "GZIP".
+//	buffer_size: A scalar representing the number of bytes to buffer. A value of
+// 0 means no buffering will be performed.
+func TFRecordDataset(scope *Scope, filenames tf.Output, compression_type tf.Output, buffer_size tf.Output) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TFRecordDataset",
+		Input: []tf.Input{
+			filenames, compression_type, buffer_size,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RpcAttr is an optional argument to Rpc.
+type RpcAttr func(optionalAttr)
+
+// RpcProtocol sets the optional protocol attribute to value.
+//
+// value: RPC protocol to use.  Empty string means use the default protocol.
+// Options include 'grpc'.
+// If not specified, defaults to ""
+func RpcProtocol(value string) RpcAttr {
+	return func(m optionalAttr) {
+		m["protocol"] = value
+	}
+}
+
+// RpcFailFast sets the optional fail_fast attribute to value.
+//
+// value: `boolean`. If `true` (default), then failures to connect
+// (i.e., the server does not immediately respond) cause an RPC failure.
+// If not specified, defaults to true
+func RpcFailFast(value bool) RpcAttr {
+	return func(m optionalAttr) {
+		m["fail_fast"] = value
+	}
+}
+
+// RpcTimeoutInMs sets the optional timeout_in_ms attribute to value.
+//
+// value: `int`. If `0` (default), then the kernel will run the RPC
+// request and only time out if the RPC deadline passes or the session times out.
+// If this value is greater than `0`, then the op will raise an exception if
+// the RPC takes longer than `timeout_in_ms`.
+// If not specified, defaults to 0
+func RpcTimeoutInMs(value int64) RpcAttr {
+	return func(m optionalAttr) {
+		m["timeout_in_ms"] = value
+	}
+}
+
+// Perform batches of RPC requests.
+//
+// This op asynchronously performs either a single RPC request, or a batch
+// of requests.  RPC requests are defined by three main parameters:
+//
+//   - `address` (the host+port or BNS address of the request)
+//   - `method` (the RPC method name for the request)
+//   - `request` (the serialized proto string, or vector of strings,
+//      of the RPC request argument).
+//
+// For example, if you have an RPC service running on port localhost:2345,
+// and its interface is configured with the following proto declaration:
+//
+// ```
+// service MyService {
+//   rpc MyMethod(MyRequestProto) returns (MyResponseProto) {
+//   }
+// };
+// ```
+//
+// then call this op with arguments:
+//
+// ```
+// address = "localhost:2345"
+// method = "MyService/MyMethod"
+// ```
+//
+// The `request` tensor is a string tensor representing serialized `MyRequestProto`
+// strings; and the output string tensor `response` will have the same shape
+// and contain (upon successful completion) corresponding serialized
+// `MyResponseProto` strings.
+//
+// For example, to send a single, empty, `MyRequestProto`, call
+// this op with `request = ""`.  To send 5 **parallel** empty requests,
+// call this op with `request = ["", "", "", "", ""]`.
+//
+// More generally, one can create a batch of `MyRequestProto` serialized protos
+// from regular batched tensors using the `encode_proto` op, and convert
+// the response `MyResponseProto` serialized protos to batched tensors
+// using the `decode_proto` op.
+//
+// **NOTE** Working with serialized proto strings is faster than instantiating
+// actual proto objects in memory, so no performance degradation is expected
+// compared to writing custom kernels for this workflow.
+//
+// If the connection fails or the remote worker returns an error
+// status, the op reraises this exception locally.
+//
+// See the `TryRpc` op if you prefer to handle RPC failures manually in the graph.
+//
+// Arguments:
+//	address: `0-D` or `1-D`.  The address (i.e. host_name:port) of the RPC server.
+// If this tensor has more than 1 element, then multiple parallel rpc requests
+// are sent.  This argument broadcasts with `method` and `request`.
+//	method: `0-D` or `1-D`.  The method address on the RPC server.
+// If this tensor has more than 1 element, then multiple parallel rpc requests
+// are sent.  This argument broadcasts with `address` and `request`.
+//	request: `0-D` or `1-D`.  Serialized proto strings: the rpc request argument.
+// If this tensor has more than 1 element, then multiple parallel rpc requests
+// are sent.  This argument broadcasts with `address` and `method`.
+//
+// Returns Same shape as `request`. Serialized proto strings: the rpc responses.
+func Rpc(scope *Scope, address tf.Output, method tf.Output, request tf.Output, optional ...RpcAttr) (response tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Rpc",
+		Input: []tf.Input{
+			address, method, request,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// BoostedTreesQuantileStreamResourceHandleOpAttr is an optional argument to BoostedTreesQuantileStreamResourceHandleOp.
+type BoostedTreesQuantileStreamResourceHandleOpAttr func(optionalAttr)
+
+// BoostedTreesQuantileStreamResourceHandleOpContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func BoostedTreesQuantileStreamResourceHandleOpContainer(value string) BoostedTreesQuantileStreamResourceHandleOpAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// BoostedTreesQuantileStreamResourceHandleOpSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func BoostedTreesQuantileStreamResourceHandleOpSharedName(value string) BoostedTreesQuantileStreamResourceHandleOpAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Creates a handle to a BoostedTreesQuantileStreamResource.
+func BoostedTreesQuantileStreamResourceHandleOp(scope *Scope, optional ...BoostedTreesQuantileStreamResourceHandleOpAttr) (resource tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesQuantileStreamResourceHandleOp",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResourceSparseApplyAdagradAttr is an optional argument to ResourceSparseApplyAdagrad.
+type ResourceSparseApplyAdagradAttr func(optionalAttr)
+
+// ResourceSparseApplyAdagradUseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var and accum tensors will be protected
+// by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceSparseApplyAdagradUseLocking(value bool) ResourceSparseApplyAdagradAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// ResourceSparseApplyAdagradUpdateSlots sets the optional update_slots attribute to value.
+// If not specified, defaults to true
+func ResourceSparseApplyAdagradUpdateSlots(value bool) ResourceSparseApplyAdagradAttr {
+	return func(m optionalAttr) {
+		m["update_slots"] = value
+	}
+}
+
+// Update relevant entries in '*var' and '*accum' according to the adagrad scheme.
+//
+// That is for rows we have grad for, we update var and accum as follows:
+// accum += grad * grad
+// var -= lr * grad * (1 / sqrt(accum))
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	accum: Should be from a Variable().
+//	lr: Learning rate. Must be a scalar.
+//	grad: The gradient.
+//	indices: A vector of indices into the first dimension of var and accum.
+//
+// Returns the created operation.
+func ResourceSparseApplyAdagrad(scope *Scope, var_ tf.Output, accum tf.Output, lr tf.Output, grad tf.Output, indices tf.Output, optional ...ResourceSparseApplyAdagradAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceSparseApplyAdagrad",
+		Input: []tf.Input{
+			var_, accum, lr, grad, indices,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// EagerPyFuncAttr is an optional argument to EagerPyFunc.
+type EagerPyFuncAttr func(optionalAttr)
+
+// EagerPyFuncIsAsync sets the optional is_async attribute to value.
+// If not specified, defaults to false
+func EagerPyFuncIsAsync(value bool) EagerPyFuncAttr {
+	return func(m optionalAttr) {
+		m["is_async"] = value
+	}
+}
+
+// Eagerly executes a python function to compute func(input)->output. The
+//
+// semantics of the input, output, and attributes are the same as those for
+// PyFunc.
+func EagerPyFunc(scope *Scope, input []tf.Output, token string, Tout []tf.DataType, optional ...EagerPyFuncAttr) (output []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"token": token, "Tout": Tout}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "EagerPyFunc",
+		Input: []tf.Input{
+			tf.OutputList(input),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if output, idx, err = makeOutputList(op, idx, "output"); err != nil {
+		scope.UpdateErr("EagerPyFunc", err)
+		return
+	}
+	return output
+}
+
+// SdcaOptimizerV2Attr is an optional argument to SdcaOptimizerV2.
+type SdcaOptimizerV2Attr func(optionalAttr)
+
+// SdcaOptimizerV2Adaptive sets the optional adaptive attribute to value.
+//
+// value: Whether to use Adaptive SDCA for the inner loop.
+// If not specified, defaults to true
+func SdcaOptimizerV2Adaptive(value bool) SdcaOptimizerV2Attr {
+	return func(m optionalAttr) {
+		m["adaptive"] = value
+	}
+}
+
+// Distributed version of Stochastic Dual Coordinate Ascent (SDCA) optimizer for
+//
+// linear models with L1 + L2 regularization. As global optimization objective is
+// strongly-convex, the optimizer optimizes the dual objective at each step. The
+// optimizer applies each update one example at a time. Examples are sampled
+// uniformly, and the optimizer is learning rate free and enjoys linear convergence
+// rate.
+//
+// [Proximal Stochastic Dual Coordinate Ascent](http://arxiv.org/pdf/1211.2717v1.pdf).<br>
+// Shai Shalev-Shwartz, Tong Zhang. 2012
+//
+// $$Loss Objective = \sum f_{i} (wx_{i}) + (l2 / 2) * |w|^2 + l1 * |w|$$
+//
+// [Adding vs. Averaging in Distributed Primal-Dual Optimization](http://arxiv.org/abs/1502.03508).<br>
+// Chenxin Ma, Virginia Smith, Martin Jaggi, Michael I. Jordan,
+// Peter Richtarik, Martin Takac. 2015
+//
+// [Stochastic Dual Coordinate Ascent with Adaptive Probabilities](https://arxiv.org/abs/1502.08053).<br>
+// Dominik Csiba, Zheng Qu, Peter Richtarik. 2015
+//
+// Arguments:
+//	sparse_example_indices: a list of vectors which contain example indices.
+//	sparse_feature_indices: a list of vectors which contain feature indices.
+//	sparse_feature_values: a list of vectors which contains feature value
+// associated with each feature group.
+//	dense_features: a list of matrices which contains the dense feature values.
+//	example_weights: a vector which contains the weight associated with each
+// example.
+//	example_labels: a vector which contains the label/target associated with each
+// example.
+//	sparse_indices: a list of vectors where each value is the indices which has
+// corresponding weights in sparse_weights. This field maybe omitted for the
+// dense approach.
+//	sparse_weights: a list of vectors where each value is the weight associated with
+// a sparse feature group.
+//	dense_weights: a list of vectors where the values are the weights associated
+// with a dense feature group.
+//	example_state_data: a list of vectors containing the example state data.
+//	loss_type: Type of the primal loss. Currently SdcaSolver supports logistic,
+// squared and hinge losses.
+//	l1: Symmetric l1 regularization strength.
+//	l2: Symmetric l2 regularization strength.
+//	num_loss_partitions: Number of partitions of the global loss function.
+//	num_inner_iterations: Number of iterations per mini-batch.
+//
+// Returns:
+//	out_example_state_data: a list of vectors containing the updated example state
+// data.
+//	out_delta_sparse_weights: a list of vectors where each value is the delta
+// weights associated with a sparse feature group.
+//	out_delta_dense_weights: a list of vectors where the values are the delta
+// weights associated with a dense feature group.
+func SdcaOptimizerV2(scope *Scope, sparse_example_indices []tf.Output, sparse_feature_indices []tf.Output, sparse_feature_values []tf.Output, dense_features []tf.Output, example_weights tf.Output, example_labels tf.Output, sparse_indices []tf.Output, sparse_weights []tf.Output, dense_weights []tf.Output, example_state_data tf.Output, loss_type string, l1 float32, l2 float32, num_loss_partitions int64, num_inner_iterations int64, optional ...SdcaOptimizerV2Attr) (out_example_state_data tf.Output, out_delta_sparse_weights []tf.Output, out_delta_dense_weights []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"loss_type": loss_type, "l1": l1, "l2": l2, "num_loss_partitions": num_loss_partitions, "num_inner_iterations": num_inner_iterations}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SdcaOptimizerV2",
+		Input: []tf.Input{
+			tf.OutputList(sparse_example_indices), tf.OutputList(sparse_feature_indices), tf.OutputList(sparse_feature_values), tf.OutputList(dense_features), example_weights, example_labels, tf.OutputList(sparse_indices), tf.OutputList(sparse_weights), tf.OutputList(dense_weights), example_state_data,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	out_example_state_data = op.Output(idx)
+	if out_delta_sparse_weights, idx, err = makeOutputList(op, idx, "out_delta_sparse_weights"); err != nil {
+		scope.UpdateErr("SdcaOptimizerV2", err)
+		return
+	}
+	if out_delta_dense_weights, idx, err = makeOutputList(op, idx, "out_delta_dense_weights"); err != nil {
+		scope.UpdateErr("SdcaOptimizerV2", err)
+		return
+	}
+	return out_example_state_data, out_delta_sparse_weights, out_delta_dense_weights
+}
+
+// MaxPool3DGradGradAttr is an optional argument to MaxPool3DGradGrad.
+type MaxPool3DGradGradAttr func(optionalAttr)
+
+// MaxPool3DGradGradDataFormat sets the optional data_format attribute to value.
+//
+// value: The data format of the input and output data. With the
+// default format "NDHWC", the data is stored in the order of:
+//     [batch, in_depth, in_height, in_width, in_channels].
+// Alternatively, the format could be "NCDHW", the data storage order is:
+//     [batch, in_channels, in_depth, in_height, in_width].
+// If not specified, defaults to "NDHWC"
+func MaxPool3DGradGradDataFormat(value string) MaxPool3DGradGradAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// Computes second-order gradients of the maxpooling function.
+//
+// Arguments:
+//	orig_input: The original input tensor.
+//	orig_output: The original output tensor.
+//	grad: Output backprop of shape `[batch, depth, rows, cols, channels]`.
+//	ksize: 1-D tensor of length 5. The size of the window for each dimension of
+// the input tensor. Must have `ksize[0] = ksize[4] = 1`.
+//	strides: 1-D tensor of length 5. The stride of the sliding window for each
+// dimension of `input`. Must have `strides[0] = strides[4] = 1`.
+//	padding: The type of padding algorithm to use.
+//
+// Returns Gradients of gradients w.r.t. the input to `max_pool`.
+func MaxPool3DGradGrad(scope *Scope, orig_input tf.Output, orig_output tf.Output, grad tf.Output, ksize []int64, strides []int64, padding string, optional ...MaxPool3DGradGradAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"ksize": ksize, "strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MaxPool3DGradGrad",
+		Input: []tf.Input{
+			orig_input, orig_output, grad,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that contains the elements of `input_dataset` ignoring errors.
+func IgnoreErrorsDataset(scope *Scope, input_dataset tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "IgnoreErrorsDataset",
+		Input: []tf.Input{
+			input_dataset,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Deprecated. Use TensorArrayGradV3
+//
+// DEPRECATED at GraphDef version 26: Use TensorArrayWriteV3
+func TensorArrayWriteV2(scope *Scope, handle tf.Output, index tf.Output, value tf.Output, flow_in tf.Output) (flow_out tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorArrayWriteV2",
+		Input: []tf.Input{
+			handle, index, value, flow_in,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// DenseToSparseSetOperationAttr is an optional argument to DenseToSparseSetOperation.
+type DenseToSparseSetOperationAttr func(optionalAttr)
+
+// DenseToSparseSetOperationValidateIndices sets the optional validate_indices attribute to value.
+// If not specified, defaults to true
+func DenseToSparseSetOperationValidateIndices(value bool) DenseToSparseSetOperationAttr {
+	return func(m optionalAttr) {
+		m["validate_indices"] = value
+	}
+}
+
+// Applies set operation along last dimension of `Tensor` and `SparseTensor`.
+//
+// See SetOperationOp::SetOperationFromContext for values of `set_operation`.
+//
+// Input `set2` is a `SparseTensor` represented by `set2_indices`, `set2_values`,
+// and `set2_shape`. For `set2` ranked `n`, 1st `n-1` dimensions must be the same
+// as `set1`. Dimension `n` contains values in a set, duplicates are allowed but
+// ignored.
+//
+// If `validate_indices` is `True`, this op validates the order and range of `set2`
+// indices.
+//
+// Output `result` is a `SparseTensor` represented by `result_indices`,
+// `result_values`, and `result_shape`. For `set1` and `set2` ranked `n`, this
+// has rank `n` and the same 1st `n-1` dimensions as `set1` and `set2`. The `nth`
+// dimension contains the result of `set_operation` applied to the corresponding
+// `[0...n-1]` dimension of `set`.
+//
+// Arguments:
+//	set1: `Tensor` with rank `n`. 1st `n-1` dimensions must be the same as `set2`.
+// Dimension `n` contains values in a set, duplicates are allowed but ignored.
+//	set2_indices: 2D `Tensor`, indices of a `SparseTensor`. Must be in row-major
+// order.
+//	set2_values: 1D `Tensor`, values of a `SparseTensor`. Must be in row-major
+// order.
+//	set2_shape: 1D `Tensor`, shape of a `SparseTensor`. `set2_shape[0...n-1]` must
+// be the same as the 1st `n-1` dimensions of `set1`, `result_shape[n]` is the
+// max set size across `n-1` dimensions.
+//
+//
+// Returns:
+//	result_indices: 2D indices of a `SparseTensor`.
+//	result_values: 1D values of a `SparseTensor`.
+//	result_shape: 1D `Tensor` shape of a `SparseTensor`. `result_shape[0...n-1]` is
+// the same as the 1st `n-1` dimensions of `set1` and `set2`, `result_shape[n]`
+// is the max result set size across all `0...n-1` dimensions.
+func DenseToSparseSetOperation(scope *Scope, set1 tf.Output, set2_indices tf.Output, set2_values tf.Output, set2_shape tf.Output, set_operation string, optional ...DenseToSparseSetOperationAttr) (result_indices tf.Output, result_values tf.Output, result_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"set_operation": set_operation}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DenseToSparseSetOperation",
+		Input: []tf.Input{
+			set1, set2_indices, set2_values, set2_shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// CollectiveBcastRecvAttr is an optional argument to CollectiveBcastRecv.
+type CollectiveBcastRecvAttr func(optionalAttr)
+
+// CollectiveBcastRecvCommunicationHint sets the optional communication_hint attribute to value.
+// If not specified, defaults to "auto"
+func CollectiveBcastRecvCommunicationHint(value string) CollectiveBcastRecvAttr {
+	return func(m optionalAttr) {
+		m["communication_hint"] = value
+	}
+}
+
+// CollectiveBcastRecvTimeoutSeconds sets the optional timeout_seconds attribute to value.
+// If not specified, defaults to 0
+func CollectiveBcastRecvTimeoutSeconds(value float32) CollectiveBcastRecvAttr {
+	return func(m optionalAttr) {
+		m["timeout_seconds"] = value
+	}
+}
+
+// Receives a tensor value broadcast from another device.
+func CollectiveBcastRecv(scope *Scope, T tf.DataType, group_size int64, group_key int64, instance_key int64, shape tf.Shape, optional ...CollectiveBcastRecvAttr) (data tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"T": T, "group_size": group_size, "group_key": group_key, "instance_key": instance_key, "shape": shape}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CollectiveBcastRecv",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Scatter the data from the input value into specific TensorArray elements.
+//
+// `indices` must be a vector, its length must match the first dim of `value`.
+//
+// Arguments:
+//	handle: The handle to a TensorArray.
+//	indices: The locations at which to write the tensor elements.
+//	value: The concatenated tensor to write to the TensorArray.
+//	flow_in: A float scalar that enforces proper chaining of operations.
+//
+// Returns A float scalar that enforces proper chaining of operations.
+func TensorArrayScatterV3(scope *Scope, handle tf.Output, indices tf.Output, value tf.Output, flow_in tf.Output) (flow_out tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorArrayScatterV3",
+		Input: []tf.Input{
+			handle, indices, value, flow_in,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the matrix square root of one or more square matrices:
+//
+// matmul(sqrtm(A), sqrtm(A)) = A
+//
+// The input matrix should be invertible. If the input matrix is real, it should
+// have no eigenvalues which are real and negative (pairs of complex conjugate
+// eigenvalues are allowed).
+//
+// The matrix square root is computed by first reducing the matrix to
+// quasi-triangular form with the real Schur decomposition. The square root
+// of the quasi-triangular matrix is then computed directly. Details of
+// the algorithm can be found in: Nicholas J. Higham, "Computing real
+// square roots of a real matrix", Linear Algebra Appl., 1987.
+//
+// The input is a tensor of shape `[..., M, M]` whose inner-most 2 dimensions
+// form square matrices. The output is a tensor of the same shape as the input
+// containing the matrix square root for all input submatrices `[..., :, :]`.
+//
+// Arguments:
+//	input: Shape is `[..., M, M]`.
+//
+// Returns Shape is `[..., M, M]`.
+//
+// @compatibility(scipy)
+// Equivalent to scipy.linalg.sqrtm
+// @end_compatibility
+func MatrixSquareRoot(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "MatrixSquareRoot",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MutexV2Attr is an optional argument to MutexV2.
+type MutexV2Attr func(optionalAttr)
+
+// MutexV2Container sets the optional container attribute to value.
+//
+// value: If non-empty, this variable is placed in the given container.
+// Otherwise, a default container is used.
+// If not specified, defaults to ""
+func MutexV2Container(value string) MutexV2Attr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// MutexV2SharedName sets the optional shared_name attribute to value.
+//
+// value: If non-empty, this variable is named in the given bucket
+// with this shared_name. Otherwise, the node name is used instead.
+// If not specified, defaults to ""
+func MutexV2SharedName(value string) MutexV2Attr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Creates a Mutex resource that can be locked by `MutexLock`.
+//
+// Returns The mutex resource.
+func MutexV2(scope *Scope, optional ...MutexV2Attr) (resource tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MutexV2",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns the truth value of (x < y) element-wise.
+//
+// *NOTE*: `Less` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+//
+// Example:
+//
+// ```python
+// x = tf.constant([5, 4, 6])
+// y = tf.constant([5])
+// tf.math.less(x, y) ==> [False, True, False]
+//
+// x = tf.constant([5, 4, 6])
+// y = tf.constant([5, 6, 7])
+// tf.math.less(x, y) ==> [False, True, True]
+// ```
+func Less(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Less",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MaxPoolGradGradWithArgmaxAttr is an optional argument to MaxPoolGradGradWithArgmax.
+type MaxPoolGradGradWithArgmaxAttr func(optionalAttr)
+
+// MaxPoolGradGradWithArgmaxIncludeBatchInIndex sets the optional include_batch_in_index attribute to value.
+//
+// value: Whether to include batch dimension in flattened index of `argmax`.
+// If not specified, defaults to false
+func MaxPoolGradGradWithArgmaxIncludeBatchInIndex(value bool) MaxPoolGradGradWithArgmaxAttr {
+	return func(m optionalAttr) {
+		m["include_batch_in_index"] = value
+	}
+}
+
+// Computes second-order gradients of the maxpooling function.
+//
+// Arguments:
+//	input: The original input.
+//	grad: 4-D with shape `[batch, height, width, channels]`.  Gradients w.r.t. the
+// input of `max_pool`.
+//	argmax: The indices of the maximum values chosen for each output of `max_pool`.
+//	ksize: The size of the window for each dimension of the input tensor.
+//	strides: The stride of the sliding window for each dimension of the
+// input tensor.
+//	padding: The type of padding algorithm to use.
+//
+// Returns Gradients of gradients w.r.t. the input of `max_pool`.
+func MaxPoolGradGradWithArgmax(scope *Scope, input tf.Output, grad tf.Output, argmax tf.Output, ksize []int64, strides []int64, padding string, optional ...MaxPoolGradGradWithArgmaxAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"ksize": ksize, "strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MaxPoolGradGradWithArgmax",
+		Input: []tf.Input{
+			input, grad, argmax,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// StridedSliceAttr is an optional argument to StridedSlice.
+type StridedSliceAttr func(optionalAttr)
+
+// StridedSliceBeginMask sets the optional begin_mask attribute to value.
+//
+// value: a bitmask where a bit i being 1 means to ignore the begin
+// value and instead use the largest interval possible. At runtime
+// begin[i] will be replaced with `[0, n-1)` if `stride[i] > 0` or
+// `[-1, n-1]` if `stride[i] < 0`
+// If not specified, defaults to 0
+func StridedSliceBeginMask(value int64) StridedSliceAttr {
+	return func(m optionalAttr) {
+		m["begin_mask"] = value
+	}
+}
+
+// StridedSliceEndMask sets the optional end_mask attribute to value.
+//
+// value: analogous to `begin_mask`
+// If not specified, defaults to 0
+func StridedSliceEndMask(value int64) StridedSliceAttr {
+	return func(m optionalAttr) {
+		m["end_mask"] = value
+	}
+}
+
+// StridedSliceEllipsisMask sets the optional ellipsis_mask attribute to value.
+//
+// value: a bitmask where bit `i` being 1 means the `i`th
+// position is actually an ellipsis. One bit at most can be 1.
+// If `ellipsis_mask == 0`, then an implicit ellipsis mask of `1 << (m+1)`
+// is provided. This means that `foo[3:5] == foo[3:5, ...]`. An ellipsis
+// implicitly creates as many range specifications as necessary to fully
+// specify the sliced range for every dimension. For example for a 4-dimensional
+// tensor `foo` the slice `foo[2, ..., 5:8]` implies `foo[2, :, :, 5:8]`.
+// If not specified, defaults to 0
+func StridedSliceEllipsisMask(value int64) StridedSliceAttr {
+	return func(m optionalAttr) {
+		m["ellipsis_mask"] = value
+	}
+}
+
+// StridedSliceNewAxisMask sets the optional new_axis_mask attribute to value.
+//
+// value: a bitmask where bit `i` being 1 means the `i`th
+// specification creates a new shape 1 dimension. For example
+// `foo[:4, tf.newaxis, :2]` would produce a shape `(4, 1, 2)` tensor.
+// If not specified, defaults to 0
+func StridedSliceNewAxisMask(value int64) StridedSliceAttr {
+	return func(m optionalAttr) {
+		m["new_axis_mask"] = value
+	}
+}
+
+// StridedSliceShrinkAxisMask sets the optional shrink_axis_mask attribute to value.
+//
+// value: a bitmask where bit `i` implies that the `i`th
+// specification should shrink the dimensionality. begin and end
+// must imply a slice of size 1 in the dimension. For example in
+// python one might do `foo[:, 3, :]` which would result in
+// `shrink_axis_mask` being 2.
+// If not specified, defaults to 0
+func StridedSliceShrinkAxisMask(value int64) StridedSliceAttr {
+	return func(m optionalAttr) {
+		m["shrink_axis_mask"] = value
+	}
+}
+
+// Return a strided slice from `input`.
+//
+// Note, most python users will want to use the Python `Tensor.__getitem__`
+// or `Variable.__getitem__` rather than this op directly.
+//
+// The goal of this op is to produce a new tensor with a subset of
+// the elements from the `n` dimensional `input` tensor. The subset is chosen using
+// a sequence of `m` sparse range specifications encoded into the arguments
+// of this function. Note, in some cases
+// `m` could be equal to `n`, but this need not be the case. Each
+// range specification entry can be one of the following:
+//
+// - An ellipsis (...). Ellipses are used to imply zero or more
+//   dimensions of full-dimension selection and are produced using
+//   `ellipsis_mask`. For example, `foo[...]` is the identity slice.
+//
+// - A new axis. This is used to insert a new shape=1 dimension and is
+//   produced using `new_axis_mask`. For example, `foo[:, ...]` where
+//   `foo` is shape `(3, 4)` produces a `(1, 3, 4)` tensor.
+//
+//
+// - A range `begin:end:stride`. This is used to specify how much to choose from
+//   a given dimension. `stride` can be any integer but 0.  `begin` is an integer
+//   which represents the index of the first value to select while `end` represents
+//   the index of the last value to select. The number of values selected in each
+//   dimension is `end - begin` if `stride > 0` and `begin - end` if `stride < 0`.
+//   `begin` and `end` can be negative where `-1` is the last element, `-2` is
+//   the second to last. `begin_mask` controls whether to replace the explicitly
+//   given `begin` with an implicit effective value of `0` if `stride > 0` and
+//   `-1` if `stride < 0`. `end_mask` is analogous but produces the number
+//   required to create the largest open interval. For example, given a shape
+//   `(3,)` tensor `foo[:]`, the effective `begin` and `end` are `0` and `3`. Do
+//   not assume this is equivalent to `foo[0:-1]` which has an effective `begin`
+//   and `end` of `0` and `2`. Another example is `foo[-2::-1]` which reverses the
+//   first dimension of a tensor while dropping the last two (in the original
+//   order elements). For example `foo = [1,2,3,4]; foo[-2::-1]` is `[4,3]`.
+//
+// - A single index. This is used to keep only elements that have a given
+//   index. For example (`foo[2, :]` on a shape `(5,6)` tensor produces a
+//   shape `(6,)` tensor. This is encoded in `begin` and `end` and
+//   `shrink_axis_mask`.
+//
+// Each conceptual range specification is encoded in the op's argument. This
+// encoding is best understand by considering a non-trivial example. In
+// particular,
+// `foo[1, 2:4, None, ..., :-3:-1, :]` will be encoded as
+//
+// ```
+// begin = [1, 2, x, x, 0, x] # x denotes don't care (usually 0)
+// end = [2, 4, x, x, -3, x]
+// strides = [1, 1, x, x, -1, 1]
+// begin_mask = 1<<4 | 1 << 5 = 48
+// end_mask = 1<<5 = 32
+// ellipsis_mask = 1<<3 = 8
+// new_axis_mask = 1<<2 4
+// shrink_axis_mask = 1<<0
+// ```
+//
+// In this case if `foo.shape` is (5, 5, 5, 5, 5, 5) the final shape of
+// the slice becomes (2, 1, 5, 5, 2, 5).
+// Let us walk step by step through each argument specification.
+//
+// 1.  The first argument in the example slice is turned into `begin = 1` and
+// `end = begin + 1 = 2`. To disambiguate from the original spec `2:4` we
+// also set the appropriate bit in `shrink_axis_mask`.
+//
+// 2. `2:4` is contributes 2, 4, 1 to begin, end, and stride. All masks have
+// zero bits contributed.
+//
+// 3. None is a synonym for `tf.newaxis`. This means insert a dimension of size 1
+// dimension in the final shape. Dummy values are contributed to begin,
+// end and stride, while the new_axis_mask bit is set.
+//
+// 4. `...` grab the full ranges from as many dimensions as needed to
+// fully specify a slice for every dimension of the input shape.
+//
+// 5. `:-3:-1` shows the use of negative indices. A negative index `i` associated
+// with a dimension that has shape `s` is converted to a positive index
+// `s + i`. So `-1` becomes `s-1` (i.e. the last element). This conversion
+// is done internally so begin, end and strides receive x, -3, and -1.
+// The appropriate begin_mask bit is set to indicate the start range is the
+// full range (ignoring the x).
+//
+// 6. `:` indicates that the entire contents of the corresponding dimension
+// is selected. This is equivalent to `::` or `0::1`. begin, end, and strides
+// receive 0, 0, and 1, respectively. The appropriate bits in `begin_mask` and
+// `end_mask` are also set.
+//
+// *Requirements*:
+//   `0 != strides[i] for i in [0, m)`
+//   `ellipsis_mask must be a power of two (only one ellipsis)`
+//
+// Arguments:
+//
+//	begin: `begin[k]` specifies the offset into the `k`th range specification.
+// The exact dimension this corresponds to will be determined by context.
+// Out-of-bounds values will be silently clamped. If the `k`th bit of
+// `begin_mask` then `begin[k]` is ignored and the full range of the
+// appropriate dimension is used instead. Negative values causes indexing
+// to start from the highest element e.g. If `foo==[1,2,3]` then `foo[-1]==3`.
+//	end: `end[i]` is like `begin` with the exception that `end_mask` is
+// used to determine full ranges.
+//	strides: `strides[i]` specifies the increment in the `i`th specification
+// after extracting a given element. Negative indices will reverse
+// the original order. Out or range values are
+// clamped to `[0,dim[i]) if slice[i]>0` or `[-1,dim[i]-1] if slice[i] < 0`
+func StridedSlice(scope *Scope, input tf.Output, begin tf.Output, end tf.Output, strides tf.Output, optional ...StridedSliceAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StridedSlice",
+		Input: []tf.Input{
+			input, begin, end, strides,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RetrieveTPUEmbeddingRMSPropParametersGradAccumDebugAttr is an optional argument to RetrieveTPUEmbeddingRMSPropParametersGradAccumDebug.
+type RetrieveTPUEmbeddingRMSPropParametersGradAccumDebugAttr func(optionalAttr)
+
+// RetrieveTPUEmbeddingRMSPropParametersGradAccumDebugTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func RetrieveTPUEmbeddingRMSPropParametersGradAccumDebugTableId(value int64) RetrieveTPUEmbeddingRMSPropParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingRMSPropParametersGradAccumDebugTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingRMSPropParametersGradAccumDebugTableName(value string) RetrieveTPUEmbeddingRMSPropParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingRMSPropParametersGradAccumDebugConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingRMSPropParametersGradAccumDebugConfig(value string) RetrieveTPUEmbeddingRMSPropParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Retrieve RMSProp embedding parameters with debug support.
+//
+// An op that retrieves optimization parameters from embedding to host
+// memory. Must be preceded by a ConfigureTPUEmbeddingHost op that sets up
+// the correct embedding table configuration. For example, this op is
+// used to retrieve updated parameters before saving a checkpoint.
+//
+// Returns:
+//	parameters: Parameter parameters updated by the RMSProp optimization algorithm.
+//	ms: Parameter ms updated by the RMSProp optimization algorithm.
+//	mom: Parameter mom updated by the RMSProp optimization algorithm.
+//	gradient_accumulators: Parameter gradient_accumulators updated by the RMSProp optimization algorithm.
+func RetrieveTPUEmbeddingRMSPropParametersGradAccumDebug(scope *Scope, num_shards int64, shard_id int64, optional ...RetrieveTPUEmbeddingRMSPropParametersGradAccumDebugAttr) (parameters tf.Output, ms tf.Output, mom tf.Output, gradient_accumulators tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RetrieveTPUEmbeddingRMSPropParametersGradAccumDebug",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3)
+}
+
+// DenseToDenseSetOperationAttr is an optional argument to DenseToDenseSetOperation.
+type DenseToDenseSetOperationAttr func(optionalAttr)
+
+// DenseToDenseSetOperationValidateIndices sets the optional validate_indices attribute to value.
+// If not specified, defaults to true
+func DenseToDenseSetOperationValidateIndices(value bool) DenseToDenseSetOperationAttr {
+	return func(m optionalAttr) {
+		m["validate_indices"] = value
+	}
+}
+
+// Applies set operation along last dimension of 2 `Tensor` inputs.
+//
+// See SetOperationOp::SetOperationFromContext for values of `set_operation`.
+//
+// Output `result` is a `SparseTensor` represented by `result_indices`,
+// `result_values`, and `result_shape`. For `set1` and `set2` ranked `n`, this
+// has rank `n` and the same 1st `n-1` dimensions as `set1` and `set2`. The `nth`
+// dimension contains the result of `set_operation` applied to the corresponding
+// `[0...n-1]` dimension of `set`.
+//
+// Arguments:
+//	set1: `Tensor` with rank `n`. 1st `n-1` dimensions must be the same as `set2`.
+// Dimension `n` contains values in a set, duplicates are allowed but ignored.
+//	set2: `Tensor` with rank `n`. 1st `n-1` dimensions must be the same as `set1`.
+// Dimension `n` contains values in a set, duplicates are allowed but ignored.
+//
+//
+// Returns:
+//	result_indices: 2D indices of a `SparseTensor`.
+//	result_values: 1D values of a `SparseTensor`.
+//	result_shape: 1D `Tensor` shape of a `SparseTensor`. `result_shape[0...n-1]` is
+// the same as the 1st `n-1` dimensions of `set1` and `set2`, `result_shape[n]`
+// is the max result set size across all `0...n-1` dimensions.
+func DenseToDenseSetOperation(scope *Scope, set1 tf.Output, set2 tf.Output, set_operation string, optional ...DenseToDenseSetOperationAttr) (result_indices tf.Output, result_values tf.Output, result_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"set_operation": set_operation}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DenseToDenseSetOperation",
+		Input: []tf.Input{
+			set1, set2,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Returns the set of files matching one or more glob patterns.
+//
+// Note that this routine only supports wildcard characters in the
+// basename portion of the pattern, not in the directory portion.
+// Note also that the order of filenames returned is deterministic.
+//
+// Arguments:
+//	pattern: Shell wildcard pattern(s). Scalar or vector of type string.
+//
+// Returns A vector of matching filenames.
+func MatchingFiles(scope *Scope, pattern tf.Output) (filenames tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "MatchingFiles",
+		Input: []tf.Input{
+			pattern,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns element-wise remainder of division. When `x < 0` xor `y < 0` is
+//
+// true, this follows Python semantics in that the result here is consistent
+// with a flooring divide. E.g. `floor(x / y) * y + mod(x, y) = x`.
+//
+// *NOTE*: `FloorMod` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func FloorMod(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "FloorMod",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Slice a `SparseTensor` based on the `start` and `size`.
+//
+// For example, if the input is
+//
+//     input_tensor = shape = [2, 7]
+//     [    a   d e  ]
+//     [b c          ]
+//
+// Graphically the output tensors are:
+//
+//     sparse_slice([0, 0], [2, 4]) = shape = [2, 4]
+//     [    a  ]
+//     [b c    ]
+//
+//     sparse_slice([0, 4], [2, 3]) = shape = [2, 3]
+//     [ d e  ]
+//     [      ]
+//
+// Arguments:
+//	indices: 2-D tensor represents the indices of the sparse tensor.
+//	values: 1-D tensor represents the values of the sparse tensor.
+//	shape: 1-D. tensor represents the shape of the sparse tensor.
+//	start: 1-D. tensor represents the start of the slice.
+//	size: 1-D. tensor represents the size of the slice.
+// output indices: A list of 1-D tensors represents the indices of the output
+// sparse tensors.
+//
+// Returns:
+//	output_indices
+//	output_values: A list of 1-D tensors represents the values of the output sparse
+// tensors.
+//	output_shape: A list of 1-D tensors represents the shape of the output sparse
+// tensors.
+func SparseSlice(scope *Scope, indices tf.Output, values tf.Output, shape tf.Output, start tf.Output, size tf.Output) (output_indices tf.Output, output_values tf.Output, output_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseSlice",
+		Input: []tf.Input{
+			indices, values, shape, start, size,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// SparseMatrixSparseMatMulAttr is an optional argument to SparseMatrixSparseMatMul.
+type SparseMatrixSparseMatMulAttr func(optionalAttr)
+
+// SparseMatrixSparseMatMulTransposeA sets the optional transpose_a attribute to value.
+//
+// value: Indicates whether `a` should be transposed.
+// If not specified, defaults to false
+func SparseMatrixSparseMatMulTransposeA(value bool) SparseMatrixSparseMatMulAttr {
+	return func(m optionalAttr) {
+		m["transpose_a"] = value
+	}
+}
+
+// SparseMatrixSparseMatMulTransposeB sets the optional transpose_b attribute to value.
+//
+// value: Indicates whether `b` should be transposed.
+// If not specified, defaults to false
+func SparseMatrixSparseMatMulTransposeB(value bool) SparseMatrixSparseMatMulAttr {
+	return func(m optionalAttr) {
+		m["transpose_b"] = value
+	}
+}
+
+// SparseMatrixSparseMatMulAdjointA sets the optional adjoint_a attribute to value.
+//
+// value: Indicates whether `a` should be conjugate-transposed.
+// If not specified, defaults to false
+func SparseMatrixSparseMatMulAdjointA(value bool) SparseMatrixSparseMatMulAttr {
+	return func(m optionalAttr) {
+		m["adjoint_a"] = value
+	}
+}
+
+// SparseMatrixSparseMatMulAdjointB sets the optional adjoint_b attribute to value.
+//
+// value: Indicates whether `b` should be conjugate-transposed.
+// If not specified, defaults to false
+func SparseMatrixSparseMatMulAdjointB(value bool) SparseMatrixSparseMatMulAttr {
+	return func(m optionalAttr) {
+		m["adjoint_b"] = value
+	}
+}
+
+// Sparse-matrix-multiplies two CSR matrices `a` and `b`.
+//
+// Performs a matrix multiplication of a sparse matrix `a` with a sparse matrix
+// `b`; returns a sparse matrix `a * b`, unless either `a` or `b` is transposed or
+// adjointed.
+//
+// Each matrix may be transposed or adjointed (conjugated and transposed)
+// according to the Boolean parameters `transpose_a`, `adjoint_a`, `transpose_b`
+// and `adjoint_b`. At most one of `transpose_a` or `adjoint_a` may be True.
+// Similarly, at most one of `transpose_b` or `adjoint_b` may be True.
+//
+// The inputs must have compatible shapes. That is, the inner dimension of `a`
+// must be equal to the outer dimension of `b`. This requirement is adjusted
+// according to whether either `a` or `b` is transposed or adjointed.
+//
+// The `type` parameter denotes the type of the matrix elements. Both `a` and `b`
+// must have the same type. The supported types are: `float32`, `float64`,
+// `complex64` and `complex128`.
+//
+// Both `a` and `b` must have the same rank. Broadcasting is not supported. If they
+// have rank 3, each batch of 2D CSRSparseMatrices within `a` and `b` must have the
+// same dense shape.
+//
+// The sparse matrix product may have numeric (non-structural) zeros.
+// TODO(anudhyan): Consider adding a boolean attribute to control whether to prune
+// zeros.
+//
+// Usage example:
+//
+// ```python
+//     from tensorflow.python.ops.linalg.sparse import sparse_csr_matrix_ops
+//
+//     a_indices = np.array([[0, 0], [2, 3], [2, 4], [3, 0]])
+//     a_values = np.array([1.0, 5.0, -1.0, -2.0], np.float32)
+//     a_dense_shape = [4, 5]
+//
+//     b_indices = np.array([[0, 0], [3, 0], [3, 1]])
+//     b_values = np.array([2.0, 7.0, 8.0], np.float32)
+//     b_dense_shape = [5, 3]
+//
+//     with tf.Session() as sess:
+//       # Define (COO format) Sparse Tensors over Numpy arrays
+//       a_st = tf.sparse.SparseTensor(a_indices, a_values, a_dense_shape)
+//       b_st = tf.sparse.SparseTensor(b_indices, b_values, b_dense_shape)
+//
+//       # Convert SparseTensors to CSR SparseMatrix
+//       a_sm = sparse_csr_matrix_ops.sparse_tensor_to_csr_sparse_matrix(
+//           a_st.indices, a_st.values, a_st.dense_shape)
+//       b_sm = sparse_csr_matrix_ops.sparse_tensor_to_csr_sparse_matrix(
+//           b_st.indices, b_st.values, b_st.dense_shape)
+//
+//       # Compute the CSR SparseMatrix matrix multiplication
+//       c_sm = sparse_csr_matrix_ops.sparse_matrix_sparse_mat_mul(
+//           a=a_sm, b=b_sm, type=tf.float32)
+//
+//       # Convert the CSR SparseMatrix product to a dense Tensor
+//       c_sm_dense = sparse_csr_matrix_ops.csr_sparse_matrix_to_dense(
+//           c_sm, tf.float32)
+//       # Evaluate the dense Tensor value
+//       c_sm_dense_value = sess.run(c_sm_dense)
+// ```
+//
+// `c_sm_dense_value` stores the dense matrix product:
+//
+// ```
+//     [[  2.   0.   0.]
+//      [  0.   0.   0.]
+//      [ 35.  40.   0.]
+//      [ -4.   0.   0.]]
+// ```
+//
+// a: A `CSRSparseMatrix`.
+// b: A `CSRSparseMatrix` with the same type and rank as `a`.
+// type: The type of both `a` and `b`.
+// transpose_a: If True, `a` transposed before multiplication.
+// transpose_b: If True, `b` transposed before multiplication.
+// adjoint_a: If True, `a` adjointed before multiplication.
+// adjoint_b: If True, `b` adjointed before multiplication.
+//
+// Arguments:
+//	a: A CSRSparseMatrix.
+//	b: A CSRSparseMatrix.
+//
+//
+// Returns A CSRSparseMatrix.
+func SparseMatrixSparseMatMul(scope *Scope, a tf.Output, b tf.Output, type_ tf.DataType, optional ...SparseMatrixSparseMatMulAttr) (c tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"type": type_}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseMatrixSparseMatMul",
+		Input: []tf.Input{
+			a, b,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// CopyHostAttr is an optional argument to CopyHost.
+type CopyHostAttr func(optionalAttr)
+
+// CopyHostTensorName sets the optional tensor_name attribute to value.
+//
+// value: The name of the input tensor.
+// If not specified, defaults to ""
+func CopyHostTensorName(value string) CopyHostAttr {
+	return func(m optionalAttr) {
+		m["tensor_name"] = value
+	}
+}
+
+// CopyHostDebugOpsSpec sets the optional debug_ops_spec attribute to value.
+//
+// value: A list of debug op spec (op, url, gated_grpc) for attached debug
+// ops. Each element of the list has the format
+// <debug_op>;<grpc_url>;<gated_grpc>, wherein gated_grpc is boolean represented
+// as 0/1. E.g., "DebugIdentity;grpc://foo:3333;1",
+// "DebugIdentity;file:///tmp/tfdbg_1;0".
+// If not specified, defaults to <>
+func CopyHostDebugOpsSpec(value []string) CopyHostAttr {
+	return func(m optionalAttr) {
+		m["debug_ops_spec"] = value
+	}
+}
+
+// Copy a tensor to host.
+//
+// Performs CPU-to-CPU deep-copying of tensor.
+// N.B.: If the all downstream attached debug ops are disabled given the current
+// gRPC gating status, the output will simply forward the input tensor without
+// deep-copying. See the documentation of Debug* ops for more details.
+//
+// Unlike the Copy Op, this op has HostMemory constraint on its input or output.
+//
+// Arguments:
+//	input: Input tensor.
+func CopyHost(scope *Scope, input tf.Output, optional ...CopyHostAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CopyHost",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Sparse addition of two CSR matrices, C = alpha * A + beta * B.
+//
+// The gradients of SparseMatrixAdd outputs with respect to alpha and beta are not
+// currently defined (TensorFlow will return zeros for these entries).
+//
+// Arguments:
+//	a: A CSRSparseMatrix.
+//	b: A CSRSparseMatrix.
+//	alpha: A constant scalar.
+//	beta: A constant scalar.
+//
+// Returns A CSRSparseMatrix.
+func SparseMatrixAdd(scope *Scope, a tf.Output, b tf.Output, alpha tf.Output, beta tf.Output) (c tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseMatrixAdd",
+		Input: []tf.Input{
+			a, b, alpha, beta,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SparseMatrixMatMulAttr is an optional argument to SparseMatrixMatMul.
+type SparseMatrixMatMulAttr func(optionalAttr)
+
+// SparseMatrixMatMulTransposeA sets the optional transpose_a attribute to value.
+//
+// value: Indicates whether `a` should be transposed.
+// If not specified, defaults to false
+func SparseMatrixMatMulTransposeA(value bool) SparseMatrixMatMulAttr {
+	return func(m optionalAttr) {
+		m["transpose_a"] = value
+	}
+}
+
+// SparseMatrixMatMulTransposeB sets the optional transpose_b attribute to value.
+//
+// value: Indicates whether `b` should be transposed.
+// If not specified, defaults to false
+func SparseMatrixMatMulTransposeB(value bool) SparseMatrixMatMulAttr {
+	return func(m optionalAttr) {
+		m["transpose_b"] = value
+	}
+}
+
+// SparseMatrixMatMulAdjointA sets the optional adjoint_a attribute to value.
+//
+// value: Indicates whether `a` should be conjugate-transposed.
+// If not specified, defaults to false
+func SparseMatrixMatMulAdjointA(value bool) SparseMatrixMatMulAttr {
+	return func(m optionalAttr) {
+		m["adjoint_a"] = value
+	}
+}
+
+// SparseMatrixMatMulAdjointB sets the optional adjoint_b attribute to value.
+//
+// value: Indicates whether `b` should be conjugate-transposed.
+// If not specified, defaults to false
+func SparseMatrixMatMulAdjointB(value bool) SparseMatrixMatMulAttr {
+	return func(m optionalAttr) {
+		m["adjoint_b"] = value
+	}
+}
+
+// SparseMatrixMatMulTransposeOutput sets the optional transpose_output attribute to value.
+//
+// value: Transposes the product of `a` and `b`.
+// If not specified, defaults to false
+func SparseMatrixMatMulTransposeOutput(value bool) SparseMatrixMatMulAttr {
+	return func(m optionalAttr) {
+		m["transpose_output"] = value
+	}
+}
+
+// SparseMatrixMatMulConjugateOutput sets the optional conjugate_output attribute to value.
+//
+// value: Conjugates the product of `a` and `b`.
+// If not specified, defaults to false
+func SparseMatrixMatMulConjugateOutput(value bool) SparseMatrixMatMulAttr {
+	return func(m optionalAttr) {
+		m["conjugate_output"] = value
+	}
+}
+
+// Matrix-multiplies a sparse matrix with a dense matrix.
+//
+// Returns a dense matrix.
+// For inputs A and B, where A is CSR and B is dense; this op returns a dense C;
+//
+// If transpose_output is false, returns:
+// ```
+//   C = A . B
+// ```
+//
+// If transpose_output is `true`, returns:
+// ```
+//   C = transpose(A . B) = transpose(B) . transpose(A)
+// ```
+// where the transposition is performed along the two innermost (matrix)
+// dimensions.
+//
+// If conjugate_output is `true`, returns:
+// ```
+//   C = conjugate(A . B) = conjugate(A) . conjugate(B)
+// ```
+//
+// If both conjugate_output and transpose_output are `true`, returns:
+// ```
+//   C = conjugate(transpose(A . B)) = conjugate(transpose(B)) .
+//                                     conjugate(transpose(A))
+// ```
+//
+// Arguments:
+//	a: A CSRSparseMatrix.
+//	b: A dense tensor.
+//
+// Returns A dense output tensor.
+func SparseMatrixMatMul(scope *Scope, a tf.Output, b tf.Output, optional ...SparseMatrixMatMulAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseMatrixMatMul",
+		Input: []tf.Input{
+			a, b,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Reads out the CSR components at batch `index`.
+//
+// This op is meant only for debugging / testing, and its interface is not expected
+// to be stable.
+//
+// Arguments:
+//	csr_sparse_matrix: A batched CSRSparseMatrix.
+//	index: The index in `csr_sparse_matrix`'s batch.
+//
+//
+// Returns:
+//	row_ptrs: An array containing CSR matrix row pointers.
+//	col_inds: An array containing CSR matrix column indices.
+//	values: An array containing CSR matrix nonzero values.
+func CSRSparseMatrixComponents(scope *Scope, csr_sparse_matrix tf.Output, index tf.Output, type_ tf.DataType) (row_ptrs tf.Output, col_inds tf.Output, values tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"type": type_}
+	opspec := tf.OpSpec{
+		Type: "CSRSparseMatrixComponents",
+		Input: []tf.Input{
+			csr_sparse_matrix, index,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// StringSplitV2Attr is an optional argument to StringSplitV2.
+type StringSplitV2Attr func(optionalAttr)
+
+// StringSplitV2Maxsplit sets the optional maxsplit attribute to value.
+//
+// value: An `int`. If `maxsplit > 0`, limit of the split of the result.
+// If not specified, defaults to -1
+func StringSplitV2Maxsplit(value int64) StringSplitV2Attr {
+	return func(m optionalAttr) {
+		m["maxsplit"] = value
+	}
+}
+
+// Split elements of `source` based on `sep` into a `SparseTensor`.
+//
+// Let N be the size of source (typically N will be the batch size). Split each
+// element of `source` based on `sep` and return a `SparseTensor`
+// containing the split tokens. Empty tokens are ignored.
+//
+// For example, N = 2, source[0] is 'hello world' and source[1] is 'a b c',
+// then the output will be
+// ```
+// st.indices = [0, 0;
+//               0, 1;
+//               1, 0;
+//               1, 1;
+//               1, 2]
+// st.shape = [2, 3]
+// st.values = ['hello', 'world', 'a', 'b', 'c']
+// ```
+//
+// If `sep` is given, consecutive delimiters are not grouped together and are
+// deemed to delimit empty strings. For example, source of `"1<>2<><>3"` and
+// sep of `"<>"` returns `["1", "2", "", "3"]`. If `sep` is None or an empty
+// string, consecutive whitespace are regarded as a single separator, and the
+// result will contain no empty strings at the startor end if the string has
+// leading or trailing whitespace.
+//
+// Note that the above mentioned behavior matches python's str.split.
+//
+// Arguments:
+//	input: `1-D` string `Tensor`, the strings to split.
+//	sep: `0-D` string `Tensor`, the delimiter character.
+func StringSplitV2(scope *Scope, input tf.Output, sep tf.Output, optional ...StringSplitV2Attr) (indices tf.Output, values tf.Output, shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StringSplitV2",
+		Input: []tf.Input{
+			input, sep,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Compute the lower regularized incomplete Gamma function `P(a, x)`.
+//
+// The lower regularized incomplete Gamma function is defined as:
+//
+//
+// \\(P(a, x) = gamma(a, x) / Gamma(a) = 1 - Q(a, x)\\)
+//
+// where
+//
+// \\(gamma(a, x) = \\int_{0}^{x} t^{a-1} exp(-t) dt\\)
+//
+// is the lower incomplete Gamma function.
+//
+// Note, above `Q(a, x)` (`Igammac`) is the upper regularized complete
+// Gamma function.
+func Igamma(scope *Scope, a tf.Output, x tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Igamma",
+		Input: []tf.Input{
+			a, x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Convert a (possibly batched) CSRSparseMatrix to dense.
+//
+// Arguments:
+//	sparse_input: A batched CSRSparseMatrix.
+//
+//
+// Returns A dense tensor.
+func CSRSparseMatrixToDense(scope *Scope, sparse_input tf.Output, type_ tf.DataType) (dense_output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"type": type_}
+	opspec := tf.OpSpec{
+		Type: "CSRSparseMatrixToDense",
+		Input: []tf.Input{
+			sparse_input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// IteratorFromStringHandleAttr is an optional argument to IteratorFromStringHandle.
+type IteratorFromStringHandleAttr func(optionalAttr)
+
+// IteratorFromStringHandleOutputTypes sets the optional output_types attribute to value.
+//
+// value: If specified, defines the type of each tuple component in an
+// element produced by the resulting iterator.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func IteratorFromStringHandleOutputTypes(value []tf.DataType) IteratorFromStringHandleAttr {
+	return func(m optionalAttr) {
+		m["output_types"] = value
+	}
+}
+
+// IteratorFromStringHandleOutputShapes sets the optional output_shapes attribute to value.
+//
+// value: If specified, defines the shape of each tuple component in an
+// element produced by the resulting iterator.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func IteratorFromStringHandleOutputShapes(value []tf.Shape) IteratorFromStringHandleAttr {
+	return func(m optionalAttr) {
+		m["output_shapes"] = value
+	}
+}
+
+// Converts the given string representing a handle to an iterator to a resource.
+//
+// Arguments:
+//	string_handle: A string representation of the given handle.
+//
+// Returns A handle to an iterator resource.
+func IteratorFromStringHandle(scope *Scope, string_handle tf.Output, optional ...IteratorFromStringHandleAttr) (resource_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "IteratorFromStringHandle",
+		Input: []tf.Input{
+			string_handle,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Add all input tensors element wise.
+//
+//   Inputs must be of same size and shape.
+//
+//   ```python
+//   x = [9, 7, 10]
+//   tf.math.add_n(x) ==> 26
+//   ```
+func AddN(scope *Scope, inputs []tf.Output) (sum tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "AddN",
+		Input: []tf.Input{
+			tf.OutputList(inputs),
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Converts a (possibly batched) CSRSparesMatrix to a SparseTensor.
+//
+// Arguments:
+//	sparse_matrix: A (possibly batched) CSRSparseMatrix.
+//
+//
+// Returns:
+//	indices: SparseTensor indices.
+//	values: SparseTensor values.
+//	dense_shape: SparseTensor dense shape.
+func CSRSparseMatrixToSparseTensor(scope *Scope, sparse_matrix tf.Output, type_ tf.DataType) (indices tf.Output, values tf.Output, dense_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"type": type_}
+	opspec := tf.OpSpec{
+		Type: "CSRSparseMatrixToSparseTensor",
+		Input: []tf.Input{
+			sparse_matrix,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// SizeAttr is an optional argument to Size.
+type SizeAttr func(optionalAttr)
+
+// SizeOutType sets the optional out_type attribute to value.
+// If not specified, defaults to DT_INT32
+func SizeOutType(value tf.DataType) SizeAttr {
+	return func(m optionalAttr) {
+		m["out_type"] = value
+	}
+}
+
+// Returns the size of a tensor.
+//
+// This operation returns an integer representing the number of elements in
+// `input`.
+//
+// For example:
+//
+// ```
+// # 't' is [[[1, 1,, 1], [2, 2, 2]], [[3, 3, 3], [4, 4, 4]]]]
+// size(t) ==> 12
+// ```
+func Size(scope *Scope, input tf.Output, optional ...SizeAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Size",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// TPUReplicatedInputAttr is an optional argument to TPUReplicatedInput.
+type TPUReplicatedInputAttr func(optionalAttr)
+
+// TPUReplicatedInputIsMirroredVariable sets the optional is_mirrored_variable attribute to value.
+// If not specified, defaults to false
+func TPUReplicatedInputIsMirroredVariable(value bool) TPUReplicatedInputAttr {
+	return func(m optionalAttr) {
+		m["is_mirrored_variable"] = value
+	}
+}
+
+// TPUReplicatedInputIndex sets the optional index attribute to value.
+// If not specified, defaults to -1
+func TPUReplicatedInputIndex(value int64) TPUReplicatedInputAttr {
+	return func(m optionalAttr) {
+		m["index"] = value
+	}
+}
+
+// TPUReplicatedInputIsPacked sets the optional is_packed attribute to value.
+// If not specified, defaults to false
+func TPUReplicatedInputIsPacked(value bool) TPUReplicatedInputAttr {
+	return func(m optionalAttr) {
+		m["is_packed"] = value
+	}
+}
+
+// Connects N inputs to an N-way replicated TPU computation.
+//
+// This operation holds a replicated input to a `tpu.replicate()` computation subgraph.
+// Each replicated input has the same shape and type alongside the output.
+//
+// For example:
+// ```
+// %a = "tf.opA"()
+// %b = "tf.opB"()
+// %replicated_input = "tf.TPUReplicatedInput"(%a, %b)
+// %computation = "tf.Computation"(%replicated_input)
+// ```
+// The above computation has a replicated input of two replicas.
+func TPUReplicatedInput(scope *Scope, inputs []tf.Output, optional ...TPUReplicatedInputAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TPUReplicatedInput",
+		Input: []tf.Input{
+			tf.OutputList(inputs),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a tensor filled with a scalar value.
+//
+// This operation creates a tensor of shape `dims` and fills it with `value`.
+//
+// For example:
+//
+// ```
+// # Output tensor has shape [2, 3].
+// fill([2, 3], 9) ==> [[9, 9, 9]
+//                      [9, 9, 9]]
+// ```
+//
+// `tf.fill` differs from `tf.constant` in a few ways:
+//
+// *   `tf.fill` only supports scalar contents, whereas `tf.constant` supports
+//     Tensor values.
+// *   `tf.fill` creates an Op in the computation graph that constructs the actual
+//     Tensor value at runtime. This is in contrast to `tf.constant` which embeds
+//     the entire Tensor into the graph with a `Const` node.
+// *   Because `tf.fill` evaluates at graph runtime, it supports dynamic shapes
+//     based on other runtime Tensors, unlike `tf.constant`.
+//
+// Arguments:
+//	dims: 1-D. Represents the shape of the output tensor.
+//	value: 0-D (scalar). Value to fill the returned tensor.
+//
+// @compatibility(numpy)
+// Equivalent to np.full
+// @end_compatibility
+func Fill(scope *Scope, dims tf.Output, value tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Fill",
+		Input: []tf.Input{
+			dims, value,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Converts a dense tensor to a (possibly batched) CSRSparseMatrix.
+//
+// Arguments:
+//	dense_input: A Dense tensor.
+//	indices: Indices of nonzero elements.
+//
+// Returns A (possibly batched) CSRSparseMatrix.
+func DenseToCSRSparseMatrix(scope *Scope, dense_input tf.Output, indices tf.Output) (sparse_output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "DenseToCSRSparseMatrix",
+		Input: []tf.Input{
+			dense_input, indices,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Fills empty rows in the input 2-D `SparseTensor` with a default value.
+//
+// The input `SparseTensor` is represented via the tuple of inputs
+// (`indices`, `values`, `dense_shape`).  The output `SparseTensor` has the
+// same `dense_shape` but with indices `output_indices` and values
+// `output_values`.
+//
+// This op inserts a single entry for every row that doesn't have any values.
+// The index is created as `[row, 0, ..., 0]` and the inserted value
+// is `default_value`.
+//
+// For example, suppose `sp_input` has shape `[5, 6]` and non-empty values:
+//
+//     [0, 1]: a
+//     [0, 3]: b
+//     [2, 0]: c
+//     [3, 1]: d
+//
+// Rows 1 and 4 are empty, so the output will be of shape `[5, 6]` with values:
+//
+//     [0, 1]: a
+//     [0, 3]: b
+//     [1, 0]: default_value
+//     [2, 0]: c
+//     [3, 1]: d
+//     [4, 0]: default_value
+//
+// The output `SparseTensor` will be in row-major order and will have the
+// same shape as the input.
+//
+// This op also returns an indicator vector shaped `[dense_shape[0]]` such that
+//
+//     empty_row_indicator[i] = True iff row i was an empty row.
+//
+// And a reverse index map vector shaped `[indices.shape[0]]` that is used during
+// backpropagation,
+//
+//     reverse_index_map[j] = out_j s.t. indices[j, :] == output_indices[out_j, :]
+//
+// Arguments:
+//	indices: 2-D. the indices of the sparse tensor.
+//	values: 1-D. the values of the sparse tensor.
+//	dense_shape: 1-D. the shape of the sparse tensor.
+//	default_value: 0-D. default value to insert into location `[row, 0, ..., 0]`
+//   for rows missing from the input sparse tensor.
+// output indices: 2-D. the indices of the filled sparse tensor.
+//
+// Returns:
+//	output_indices
+//	output_values: 1-D. the values of the filled sparse tensor.
+//	empty_row_indicator: 1-D. whether the dense row was missing in the
+// input sparse tensor.
+//	reverse_index_map: 1-D. a map from the input indices to the output indices.
+func SparseFillEmptyRows(scope *Scope, indices tf.Output, values tf.Output, dense_shape tf.Output, default_value tf.Output) (output_indices tf.Output, output_values tf.Output, empty_row_indicator tf.Output, reverse_index_map tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseFillEmptyRows",
+		Input: []tf.Input{
+			indices, values, dense_shape, default_value,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3)
+}
+
+// Makes the summary of quantiles for the batch.
+//
+// An op that takes a list of tensors (one tensor per feature) and outputs the
+// quantile summaries for each tensor.
+//
+// Arguments:
+//	float_values: float; List of Rank 1 Tensors each containing values for a single feature.
+//	example_weights: float; Rank 1 Tensor with weights per instance.
+//	epsilon: float; The required maximum approximation error.
+//
+// Returns float; List of Rank 2 Tensors each containing the quantile summary
+// (value, weight, min_rank, max_rank) of a single feature.
+func BoostedTreesMakeQuantileSummaries(scope *Scope, float_values []tf.Output, example_weights tf.Output, epsilon tf.Output) (summaries []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesMakeQuantileSummaries",
+		Input: []tf.Input{
+			tf.OutputList(float_values), example_weights, epsilon,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if summaries, idx, err = makeOutputList(op, idx, "summaries"); err != nil {
+		scope.UpdateErr("BoostedTreesMakeQuantileSummaries", err)
+		return
+	}
+	return summaries
+}
+
+// TakeManySparseFromTensorsMapAttr is an optional argument to TakeManySparseFromTensorsMap.
+type TakeManySparseFromTensorsMapAttr func(optionalAttr)
+
+// TakeManySparseFromTensorsMapContainer sets the optional container attribute to value.
+//
+// value: The container name for the `SparseTensorsMap` read by this op.
+// If not specified, defaults to ""
+func TakeManySparseFromTensorsMapContainer(value string) TakeManySparseFromTensorsMapAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// TakeManySparseFromTensorsMapSharedName sets the optional shared_name attribute to value.
+//
+// value: The shared name for the `SparseTensorsMap` read by this op.
+// It should not be blank; rather the `shared_name` or unique Operation name
+// of the Op that created the original `SparseTensorsMap` should be used.
+// If not specified, defaults to ""
+func TakeManySparseFromTensorsMapSharedName(value string) TakeManySparseFromTensorsMapAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Read `SparseTensors` from a `SparseTensorsMap` and concatenate them.
+//
+// The input `sparse_handles` must be an `int64` matrix of shape `[N, 1]` where
+// `N` is the minibatch size and the rows correspond to the output handles of
+// `AddSparseToTensorsMap` or `AddManySparseToTensorsMap`.  The ranks of the
+// original `SparseTensor` objects that went into the given input ops must all
+// match.  When the final `SparseTensor` is created, it has rank one
+// higher than the ranks of the incoming `SparseTensor` objects
+// (they have been concatenated along a new row dimension on the left).
+//
+// The output `SparseTensor` object's shape values for all dimensions but the
+// first are the max across the input `SparseTensor` objects' shape values
+// for the corresponding dimensions.  Its first shape value is `N`, the minibatch
+// size.
+//
+// The input `SparseTensor` objects' indices are assumed ordered in
+// standard lexicographic order.  If this is not the case, after this
+// step run `SparseReorder` to restore index ordering.
+//
+// For example, if the handles represent an input, which is a `[2, 3]` matrix
+// representing two original `SparseTensor` objects:
+//
+// ```
+//     index = [ 0]
+//             [10]
+//             [20]
+//     values = [1, 2, 3]
+//     shape = [50]
+// ```
+//
+// and
+//
+// ```
+//     index = [ 2]
+//             [10]
+//     values = [4, 5]
+//     shape = [30]
+// ```
+//
+// then the final `SparseTensor` will be:
+//
+// ```
+//     index = [0  0]
+//             [0 10]
+//             [0 20]
+//             [1  2]
+//             [1 10]
+//     values = [1, 2, 3, 4, 5]
+//     shape = [2 50]
+// ```
+//
+// Arguments:
+//	sparse_handles: 1-D, The `N` serialized `SparseTensor` objects.
+// Shape: `[N]`.
+//	dtype: The `dtype` of the `SparseTensor` objects stored in the
+// `SparseTensorsMap`.
+//
+// Returns:
+//	sparse_indices: 2-D.  The `indices` of the minibatch `SparseTensor`.
+//	sparse_values: 1-D.  The `values` of the minibatch `SparseTensor`.
+//	sparse_shape: 1-D.  The `shape` of the minibatch `SparseTensor`.
+func TakeManySparseFromTensorsMap(scope *Scope, sparse_handles tf.Output, dtype tf.DataType, optional ...TakeManySparseFromTensorsMapAttr) (sparse_indices tf.Output, sparse_values tf.Output, sparse_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TakeManySparseFromTensorsMap",
+		Input: []tf.Input{
+			sparse_handles,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// LoadTPUEmbeddingCenteredRMSPropParametersAttr is an optional argument to LoadTPUEmbeddingCenteredRMSPropParameters.
+type LoadTPUEmbeddingCenteredRMSPropParametersAttr func(optionalAttr)
+
+// LoadTPUEmbeddingCenteredRMSPropParametersTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func LoadTPUEmbeddingCenteredRMSPropParametersTableId(value int64) LoadTPUEmbeddingCenteredRMSPropParametersAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// LoadTPUEmbeddingCenteredRMSPropParametersTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingCenteredRMSPropParametersTableName(value string) LoadTPUEmbeddingCenteredRMSPropParametersAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// LoadTPUEmbeddingCenteredRMSPropParametersConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingCenteredRMSPropParametersConfig(value string) LoadTPUEmbeddingCenteredRMSPropParametersAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Load centered RMSProp embedding parameters.
+//
+// An op that loads optimization parameters into HBM for embedding. Must be
+// preceded by a ConfigureTPUEmbeddingHost op that sets up the correct
+// embedding table configuration. For example, this op is used to install
+// parameters that are loaded from a checkpoint before a training loop is
+// executed.
+//
+// Arguments:
+//	parameters: Value of parameters used in the centered RMSProp optimization algorithm.
+//	ms: Value of ms used in the centered RMSProp optimization algorithm.
+//	mom: Value of mom used in the centered RMSProp optimization algorithm.
+//	mg: Value of mg used in the centered RMSProp optimization algorithm.
+//
+//
+//
+// Returns the created operation.
+func LoadTPUEmbeddingCenteredRMSPropParameters(scope *Scope, parameters tf.Output, ms tf.Output, mom tf.Output, mg tf.Output, num_shards int64, shard_id int64, optional ...LoadTPUEmbeddingCenteredRMSPropParametersAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LoadTPUEmbeddingCenteredRMSPropParameters",
+		Input: []tf.Input{
+			parameters, ms, mom, mg,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// RandomPoissonAttr is an optional argument to RandomPoisson.
+type RandomPoissonAttr func(optionalAttr)
+
+// RandomPoissonSeed sets the optional seed attribute to value.
+// If not specified, defaults to 0
+func RandomPoissonSeed(value int64) RandomPoissonAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// RandomPoissonSeed2 sets the optional seed2 attribute to value.
+// If not specified, defaults to 0
+func RandomPoissonSeed2(value int64) RandomPoissonAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Use RandomPoissonV2 instead.
+//
+// DEPRECATED at GraphDef version 25: Replaced by RandomPoissonV2
+func RandomPoisson(scope *Scope, shape tf.Output, rate tf.Output, optional ...RandomPoissonAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RandomPoisson",
+		Input: []tf.Input{
+			shape, rate,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Compute the regularized incomplete beta integral \\(I_x(a, b)\\).
+//
+// The regularized incomplete beta integral is defined as:
+//
+//
+// \\(I_x(a, b) = \frac{B(x; a, b)}{B(a, b)}\\)
+//
+// where
+//
+//
+// \\(B(x; a, b) = \int_0^x t^{a-1} (1 - t)^{b-1} dt\\)
+//
+//
+// is the incomplete beta function and \\(B(a, b)\\) is the *complete*
+// beta function.
+func Betainc(scope *Scope, a tf.Output, b tf.Output, x tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Betainc",
+		Input: []tf.Input{
+			a, b, x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Reduces sparse updates into the variable referenced by `resource` using the `max` operation.
+//
+// This operation computes
+//
+//     # Scalar indices
+//     ref[indices, ...] = max(ref[indices, ...], updates[...])
+//
+//     # Vector indices (for each i)
+//     ref[indices[i], ...] = max(ref[indices[i], ...], updates[i, ...])
+//
+//     # High rank indices (for each i, ..., j)
+//     ref[indices[i, ..., j], ...] = max(ref[indices[i, ..., j], ...], updates[i, ..., j, ...])
+//
+// Duplicate entries are handled correctly: if multiple `indices` reference
+// the same location, their contributions are combined.
+//
+// Requires `updates.shape = indices.shape + ref.shape[1:]` or `updates.shape = []`.
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src='https://www.tensorflow.org/images/ScatterAdd.png' alt>
+// </div>
+//
+// Arguments:
+//	resource: Should be from a `Variable` node.
+//	indices: A tensor of indices into the first dimension of `ref`.
+//	updates: A tensor of updated values to add to `ref`.
+//
+// Returns the created operation.
+func ResourceScatterMax(scope *Scope, resource tf.Output, indices tf.Output, updates tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceScatterMax",
+		Input: []tf.Input{
+			resource, indices, updates,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// AddManySparseToTensorsMapAttr is an optional argument to AddManySparseToTensorsMap.
+type AddManySparseToTensorsMapAttr func(optionalAttr)
+
+// AddManySparseToTensorsMapContainer sets the optional container attribute to value.
+//
+// value: The container name for the `SparseTensorsMap` created by this op.
+// If not specified, defaults to ""
+func AddManySparseToTensorsMapContainer(value string) AddManySparseToTensorsMapAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// AddManySparseToTensorsMapSharedName sets the optional shared_name attribute to value.
+//
+// value: The shared name for the `SparseTensorsMap` created by this op.
+// If blank, the new Operation's unique name is used.
+// If not specified, defaults to ""
+func AddManySparseToTensorsMapSharedName(value string) AddManySparseToTensorsMapAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Add an `N`-minibatch `SparseTensor` to a `SparseTensorsMap`, return `N` handles.
+//
+// A `SparseTensor` of rank `R` is represented by three tensors: `sparse_indices`,
+// `sparse_values`, and `sparse_shape`, where
+//
+// ```sparse_indices.shape[1] == sparse_shape.shape[0] == R```
+//
+// An `N`-minibatch of `SparseTensor` objects is represented as a `SparseTensor`
+// having a first `sparse_indices` column taking values between `[0, N)`, where
+// the minibatch size `N == sparse_shape[0]`.
+//
+// The input `SparseTensor` must have rank `R` greater than 1, and the first
+// dimension is treated as the minibatch dimension.  Elements of the `SparseTensor`
+// must be sorted in increasing order of this first dimension.  The stored
+// `SparseTensor` objects pointed to by each row of the output `sparse_handles`
+// will have rank `R-1`.
+//
+// The `SparseTensor` values can then be read out as part of a minibatch by passing
+// the given keys as vector elements to `TakeManySparseFromTensorsMap`.  To ensure
+// the correct `SparseTensorsMap` is accessed, ensure that the same
+// `container` and `shared_name` are passed to that Op.  If no `shared_name`
+// is provided here, instead use the *name* of the Operation created by calling
+// `AddManySparseToTensorsMap` as the `shared_name` passed to
+// `TakeManySparseFromTensorsMap`.  Ensure the Operations are colocated.
+//
+// Arguments:
+//	sparse_indices: 2-D.  The `indices` of the minibatch `SparseTensor`.
+// `sparse_indices[:, 0]` must be ordered values in `[0, N)`.
+//	sparse_values: 1-D.  The `values` of the minibatch `SparseTensor`.
+//	sparse_shape: 1-D.  The `shape` of the minibatch `SparseTensor`.
+// The minibatch size `N == sparse_shape[0]`.
+//
+// Returns 1-D.  The handles of the `SparseTensor` now stored in the
+// `SparseTensorsMap`.  Shape: `[N]`.
+func AddManySparseToTensorsMap(scope *Scope, sparse_indices tf.Output, sparse_values tf.Output, sparse_shape tf.Output, optional ...AddManySparseToTensorsMapAttr) (sparse_handles tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "AddManySparseToTensorsMap",
+		Input: []tf.Input{
+			sparse_indices, sparse_values, sparse_shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes tan of x element-wise.
+//
+//   Given an input tensor, this function computes tangent of every
+//   element in the tensor. Input range is `(-inf, inf)` and
+//   output range is `(-inf, inf)`. If input lies outside the boundary, `nan`
+//   is returned.
+//
+//   ```python
+//   x = tf.constant([-float("inf"), -9, -0.5, 1, 1.2, 200, 10000, float("inf")])
+//   tf.math.tan(x) ==> [nan 0.45231566 -0.5463025 1.5574077 2.572152 -1.7925274 0.32097113 nan]
+//   ```
+func Tan(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Tan",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// BiasAddGradAttr is an optional argument to BiasAddGrad.
+type BiasAddGradAttr func(optionalAttr)
+
+// BiasAddGradDataFormat sets the optional data_format attribute to value.
+//
+// value: Specify the data format of the input and output data. With the
+// default format "NHWC", the bias tensor will be added to the last dimension
+// of the value tensor.
+// Alternatively, the format could be "NCHW", the data storage order of:
+//     [batch, in_channels, in_height, in_width].
+// The tensor will be added to "in_channels", the third-to-the-last
+//     dimension.
+// If not specified, defaults to "NHWC"
+func BiasAddGradDataFormat(value string) BiasAddGradAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// The backward operation for "BiasAdd" on the "bias" tensor.
+//
+// It accumulates all the values from out_backprop into the feature dimension.
+// For NHWC data format, the feature dimension is the last. For NCHW data format,
+// the feature dimension is the third-to-last.
+//
+// Arguments:
+//	out_backprop: Any number of dimensions.
+//
+// Returns 1-D with size the feature dimension of `out_backprop`.
+func BiasAddGrad(scope *Scope, out_backprop tf.Output, optional ...BiasAddGradAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "BiasAddGrad",
+		Input: []tf.Input{
+			out_backprop,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Advance the counter of a counter-based RNG.
+//
+// The state of the RNG after
+// `rng_skip(n)` will be the same as that after `stateful_uniform([n])`
+// (or any other distribution). The actual increment added to the
+// counter is an unspecified implementation detail.
+//
+// Arguments:
+//	resource: The handle of the resource variable that stores the state of the RNG.
+//	algorithm: The RNG algorithm.
+//	delta: The amount of advancement.
+//
+// Returns the created operation.
+func RngSkip(scope *Scope, resource tf.Output, algorithm tf.Output, delta tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "RngSkip",
+		Input: []tf.Input{
+			resource, algorithm, delta,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Generates values in an interval.
+//
+// A sequence of `num` evenly-spaced values are generated beginning at `start`.
+// If `num > 1`, the values in the sequence increase by `stop - start / num - 1`,
+// so that the last one is exactly `stop`.
+//
+// For example:
+//
+// ```
+// tf.linspace(10.0, 12.0, 3, name="linspace") => [ 10.0  11.0  12.0]
+// ```
+//
+// Arguments:
+//	start: 0-D tensor. First entry in the range.
+//	stop: 0-D tensor. Last entry in the range.
+//	num: 0-D tensor. Number of values to generate.
+//
+// Returns 1-D. The generated values.
+func LinSpace(scope *Scope, start tf.Output, stop tf.Output, num tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "LinSpace",
+		Input: []tf.Input{
+			start, stop, num,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MultinomialAttr is an optional argument to Multinomial.
+type MultinomialAttr func(optionalAttr)
+
+// MultinomialSeed sets the optional seed attribute to value.
+//
+// value: If either seed or seed2 is set to be non-zero, the internal random number
+// generator is seeded by the given seed.  Otherwise, a random seed is used.
+// If not specified, defaults to 0
+func MultinomialSeed(value int64) MultinomialAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// MultinomialSeed2 sets the optional seed2 attribute to value.
+//
+// value: A second seed to avoid seed collision.
+// If not specified, defaults to 0
+func MultinomialSeed2(value int64) MultinomialAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// MultinomialOutputDtype sets the optional output_dtype attribute to value.
+// If not specified, defaults to DT_INT64
+func MultinomialOutputDtype(value tf.DataType) MultinomialAttr {
+	return func(m optionalAttr) {
+		m["output_dtype"] = value
+	}
+}
+
+// Draws samples from a multinomial distribution.
+//
+// Arguments:
+//	logits: 2-D Tensor with shape `[batch_size, num_classes]`.  Each slice `[i, :]`
+// represents the unnormalized log probabilities for all classes.
+//	num_samples: 0-D.  Number of independent samples to draw for each row slice.
+//
+// Returns 2-D Tensor with shape `[batch_size, num_samples]`.  Each slice `[i, :]`
+// contains the drawn class labels with range `[0, num_classes)`.
+func Multinomial(scope *Scope, logits tf.Output, num_samples tf.Output, optional ...MultinomialAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Multinomial",
+		Input: []tf.Input{
+			logits, num_samples,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// NonDeterministicIntsAttr is an optional argument to NonDeterministicInts.
+type NonDeterministicIntsAttr func(optionalAttr)
+
+// NonDeterministicIntsDtype sets the optional dtype attribute to value.
+//
+// value: The type of the output.
+// If not specified, defaults to DT_INT64
+func NonDeterministicIntsDtype(value tf.DataType) NonDeterministicIntsAttr {
+	return func(m optionalAttr) {
+		m["dtype"] = value
+	}
+}
+
+// Non-deterministically generates some integers.
+//
+// This op may use some OS-provided source of non-determinism (e.g. an RNG), so each execution will give different results.
+//
+// Arguments:
+//	shape: The shape of the output tensor.
+//
+// Returns Non-deterministic integer values with specified shape.
+func NonDeterministicInts(scope *Scope, shape tf.Output, optional ...NonDeterministicIntsAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "NonDeterministicInts",
+		Input: []tf.Input{
+			shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that caches elements from `input_dataset`.
+//
+// A CacheDataset will iterate over the input_dataset, and store tensors. If the
+// cache already exists, the cache will be used. If the cache is inappropriate
+// (e.g. cannot be opened, contains tensors of the wrong shape / size), an error
+// will the returned when used.
+//
+// Arguments:
+//
+//	filename: A path on the filesystem where we should cache the dataset. Note: this
+// will be a directory.
+//
+//
+func CacheDataset(scope *Scope, input_dataset tf.Output, filename tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "CacheDataset",
+		Input: []tf.Input{
+			input_dataset, filename,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ThreadPoolHandleAttr is an optional argument to ThreadPoolHandle.
+type ThreadPoolHandleAttr func(optionalAttr)
+
+// ThreadPoolHandleMaxIntraOpParallelism sets the optional max_intra_op_parallelism attribute to value.
+//
+// value: The maximum degree of parallelism to use within operations that execute on this
+// threadpool.
+// If not specified, defaults to 1
+func ThreadPoolHandleMaxIntraOpParallelism(value int64) ThreadPoolHandleAttr {
+	return func(m optionalAttr) {
+		m["max_intra_op_parallelism"] = value
+	}
+}
+
+// ThreadPoolHandleContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func ThreadPoolHandleContainer(value string) ThreadPoolHandleAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// ThreadPoolHandleSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func ThreadPoolHandleSharedName(value string) ThreadPoolHandleAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Creates a dataset that uses a custom thread pool to compute `input_dataset`.
+//
+// Arguments:
+//	num_threads: The number of threads in the thread pool.
+//	display_name: A human-readable name for the threads that may be visible in some
+// visualizations.
+// threadpool.
+//
+// Returns A resource that can be consumed by one or more ExperimentalThreadPoolDataset
+// ops.
+func ThreadPoolHandle(scope *Scope, num_threads int64, display_name string, optional ...ThreadPoolHandleAttr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_threads": num_threads, "display_name": display_name}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ThreadPoolHandle",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SparseReduceMaxSparseAttr is an optional argument to SparseReduceMaxSparse.
+type SparseReduceMaxSparseAttr func(optionalAttr)
+
+// SparseReduceMaxSparseKeepDims sets the optional keep_dims attribute to value.
+//
+// value: If true, retain reduced dimensions with length 1.
+// If not specified, defaults to false
+func SparseReduceMaxSparseKeepDims(value bool) SparseReduceMaxSparseAttr {
+	return func(m optionalAttr) {
+		m["keep_dims"] = value
+	}
+}
+
+// Computes the max of elements across dimensions of a SparseTensor.
+//
+// This Op takes a SparseTensor and is the sparse counterpart to
+// `tf.reduce_max()`.  In contrast to SparseReduceMax, this Op returns a
+// SparseTensor.
+//
+// Reduces `sp_input` along the dimensions given in `reduction_axes`.  Unless
+// `keep_dims` is true, the rank of the tensor is reduced by 1 for each entry in
+// `reduction_axes`. If `keep_dims` is true, the reduced dimensions are retained
+// with length 1.
+//
+// If `reduction_axes` has no entries, all dimensions are reduced, and a tensor
+// with a single element is returned.  Additionally, the axes can be negative,
+// which are interpreted according to the indexing rules in Python.
+//
+// Arguments:
+//	input_indices: 2-D.  `N x R` matrix with the indices of non-empty values in a
+// SparseTensor, possibly not in canonical ordering.
+//	input_values: 1-D.  `N` non-empty values corresponding to `input_indices`.
+//	input_shape: 1-D.  Shape of the input SparseTensor.
+//	reduction_axes: 1-D.  Length-`K` vector containing the reduction axes.
+func SparseReduceMaxSparse(scope *Scope, input_indices tf.Output, input_values tf.Output, input_shape tf.Output, reduction_axes tf.Output, optional ...SparseReduceMaxSparseAttr) (output_indices tf.Output, output_values tf.Output, output_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseReduceMaxSparse",
+		Input: []tf.Input{
+			input_indices, input_values, input_shape, reduction_axes,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Computes the maximum along segments of a tensor.
+//
+// Read
+// [the section on segmentation](https://tensorflow.org/api_docs/python/tf/math#Segmentation)
+// for an explanation of segments.
+//
+// This operator is similar to the unsorted segment sum operator found
+// [(here)](../../../api_docs/python/math_ops.md#UnsortedSegmentSum).
+// Instead of computing the sum over segments, it computes the maximum such that:
+//
+// \\(output_i = \max_{j...} data[j...]\\) where max is over tuples `j...` such
+// that `segment_ids[j...] == i`.
+//
+// If the maximum is empty for a given segment ID `i`, it outputs the smallest
+// possible value for the specific numeric type,
+// `output[i] = numeric_limits<T>::lowest()`.
+//
+// If the given segment ID `i` is negative, then the corresponding value is
+// dropped, and will not be included in the result.
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src="https://www.tensorflow.org/images/UnsortedSegmentMax.png" alt>
+// </div>
+//
+// For example:
+//
+// ``` python
+// c = tf.constant([[1,2,3,4], [5,6,7,8], [4,3,2,1]])
+// tf.unsorted_segment_max(c, tf.constant([0, 1, 0]), num_segments=2)
+// # ==> [[ 4,  3, 3, 4],
+// #       [5,  6, 7, 8]]
+// ```
+//
+//
+// Arguments:
+//
+//	segment_ids: A tensor whose shape is a prefix of `data.shape`.
+//
+//
+// Returns Has same shape as data, except for the first `segment_ids.rank`
+// dimensions, which are replaced with a single dimension which has size
+// `num_segments`.
+func UnsortedSegmentMax(scope *Scope, data tf.Output, segment_ids tf.Output, num_segments tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "UnsortedSegmentMax",
+		Input: []tf.Input{
+			data, segment_ids, num_segments,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// StringUpperAttr is an optional argument to StringUpper.
+type StringUpperAttr func(optionalAttr)
+
+// StringUpperEncoding sets the optional encoding attribute to value.
+// If not specified, defaults to ""
+func StringUpperEncoding(value string) StringUpperAttr {
+	return func(m optionalAttr) {
+		m["encoding"] = value
+	}
+}
+
+// Converts all lowercase characters into their respective uppercase replacements.
+//
+// Example:
+//
+// >>> tf.strings.upper("CamelCase string and ALL CAPS")
+// <tf.Tensor: shape=(), dtype=string, numpy=b'CAMELCASE STRING AND ALL CAPS'>
+//
+func StringUpper(scope *Scope, input tf.Output, optional ...StringUpperAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StringUpper",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Set a summary_writer_interface to record statistics using given stats_aggregator.
+//
+// Returns the created operation.
+func StatsAggregatorSetSummaryWriter(scope *Scope, stats_aggregator tf.Output, summary tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "StatsAggregatorSetSummaryWriter",
+		Input: []tf.Input{
+			stats_aggregator, summary,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// FusedBatchNormGradAttr is an optional argument to FusedBatchNormGrad.
+type FusedBatchNormGradAttr func(optionalAttr)
+
+// FusedBatchNormGradEpsilon sets the optional epsilon attribute to value.
+//
+// value: A small float number added to the variance of x.
+// If not specified, defaults to 0.0001
+func FusedBatchNormGradEpsilon(value float32) FusedBatchNormGradAttr {
+	return func(m optionalAttr) {
+		m["epsilon"] = value
+	}
+}
+
+// FusedBatchNormGradDataFormat sets the optional data_format attribute to value.
+//
+// value: The data format for y_backprop, x, x_backprop.
+// Either "NHWC" (default) or "NCHW".
+// If not specified, defaults to "NHWC"
+func FusedBatchNormGradDataFormat(value string) FusedBatchNormGradAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// FusedBatchNormGradIsTraining sets the optional is_training attribute to value.
+//
+// value: A bool value to indicate the operation is for training (default)
+// or inference.
+// If not specified, defaults to true
+func FusedBatchNormGradIsTraining(value bool) FusedBatchNormGradAttr {
+	return func(m optionalAttr) {
+		m["is_training"] = value
+	}
+}
+
+// Gradient for batch normalization.
+//
+// Note that the size of 4D Tensors are defined by either "NHWC" or "NCHW".
+// The size of 1D Tensors matches the dimension C of the 4D Tensors.
+//
+// Arguments:
+//	y_backprop: A 4D Tensor for the gradient with respect to y.
+//	x: A 4D Tensor for input data.
+//	scale: A 1D Tensor for scaling factor, to scale the normalized x.
+//	reserve_space_1: When is_training is True, a 1D Tensor for the computed batch
+// mean to be reused in gradient computation. When is_training is
+// False, a 1D Tensor for the population mean to be reused in both
+// 1st and 2nd order gradient computation.
+//	reserve_space_2: When is_training is True, a 1D Tensor for the computed batch
+// variance (inverted variance in the cuDNN case) to be reused in
+// gradient computation. When is_training is False, a 1D Tensor
+// for the population variance to be reused in both 1st and 2nd
+// order gradient computation.
+//
+// Returns:
+//	x_backprop: A 4D Tensor for the gradient with respect to x.
+//	scale_backprop: A 1D Tensor for the gradient with respect to scale.
+//	offset_backprop: A 1D Tensor for the gradient with respect to offset.
+//	reserve_space_3: Unused placeholder to match the mean input in FusedBatchNorm.
+//	reserve_space_4: Unused placeholder to match the variance input
+// in FusedBatchNorm.
+func FusedBatchNormGrad(scope *Scope, y_backprop tf.Output, x tf.Output, scale tf.Output, reserve_space_1 tf.Output, reserve_space_2 tf.Output, optional ...FusedBatchNormGradAttr) (x_backprop tf.Output, scale_backprop tf.Output, offset_backprop tf.Output, reserve_space_3 tf.Output, reserve_space_4 tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "FusedBatchNormGrad",
+		Input: []tf.Input{
+			y_backprop, x, scale, reserve_space_1, reserve_space_2,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3), op.Output(4)
+}
+
+// Subtracts a value from the current value of a variable.
+//
+// Any ReadVariableOp with a control dependency on this op is guaranteed to
+// see the decremented value or a subsequent newer one.
+//
+// Arguments:
+//	resource: handle to the resource in which to store the variable.
+//	value: the value by which the variable will be incremented.
+//
+// Returns the created operation.
+func AssignSubVariableOp(scope *Scope, resource tf.Output, value tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "AssignSubVariableOp",
+		Input: []tf.Input{
+			resource, value,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// SparseReduceMaxAttr is an optional argument to SparseReduceMax.
+type SparseReduceMaxAttr func(optionalAttr)
+
+// SparseReduceMaxKeepDims sets the optional keep_dims attribute to value.
+//
+// value: If true, retain reduced dimensions with length 1.
+// If not specified, defaults to false
+func SparseReduceMaxKeepDims(value bool) SparseReduceMaxAttr {
+	return func(m optionalAttr) {
+		m["keep_dims"] = value
+	}
+}
+
+// Computes the max of elements across dimensions of a SparseTensor.
+//
+// This Op takes a SparseTensor and is the sparse counterpart to
+// `tf.reduce_max()`.  In particular, this Op also returns a dense `Tensor`
+// instead of a sparse one.
+//
+// Reduces `sp_input` along the dimensions given in `reduction_axes`.  Unless
+// `keep_dims` is true, the rank of the tensor is reduced by 1 for each entry in
+// `reduction_axes`. If `keep_dims` is true, the reduced dimensions are retained
+// with length 1.
+//
+// If `reduction_axes` has no entries, all dimensions are reduced, and a tensor
+// with a single element is returned.  Additionally, the axes can be negative,
+// which are interpreted according to the indexing rules in Python.
+//
+// Arguments:
+//	input_indices: 2-D.  `N x R` matrix with the indices of non-empty values in a
+// SparseTensor, possibly not in canonical ordering.
+//	input_values: 1-D.  `N` non-empty values corresponding to `input_indices`.
+//	input_shape: 1-D.  Shape of the input SparseTensor.
+//	reduction_axes: 1-D.  Length-`K` vector containing the reduction axes.
+//
+// Returns `R-K`-D.  The reduced Tensor.
+func SparseReduceMax(scope *Scope, input_indices tf.Output, input_values tf.Output, input_shape tf.Output, reduction_axes tf.Output, optional ...SparseReduceMaxAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseReduceMax",
+		Input: []tf.Input{
+			input_indices, input_values, input_shape, reduction_axes,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Generates sparse cross from a list of sparse and dense tensors.
+//
+// The op takes two lists, one of 2D `SparseTensor` and one of 2D `Tensor`, each
+// representing features of one feature column. It outputs a 2D `SparseTensor` with
+// the batchwise crosses of these features.
+//
+// For example, if the inputs are
+//
+//     inputs[0]: SparseTensor with shape = [2, 2]
+//     [0, 0]: "a"
+//     [1, 0]: "b"
+//     [1, 1]: "c"
+//
+//     inputs[1]: SparseTensor with shape = [2, 1]
+//     [0, 0]: "d"
+//     [1, 0]: "e"
+//
+//     inputs[2]: Tensor [["f"], ["g"]]
+//
+// then the output will be
+//
+//     shape = [2, 2]
+//     [0, 0]: "a_X_d_X_f"
+//     [1, 0]: "b_X_e_X_g"
+//     [1, 1]: "c_X_e_X_g"
+//
+// if hashed_output=true then the output will be
+//
+//     shape = [2, 2]
+//     [0, 0]: FingerprintCat64(
+//                 Fingerprint64("f"), FingerprintCat64(
+//                     Fingerprint64("d"), Fingerprint64("a")))
+//     [1, 0]: FingerprintCat64(
+//                 Fingerprint64("g"), FingerprintCat64(
+//                     Fingerprint64("e"), Fingerprint64("b")))
+//     [1, 1]: FingerprintCat64(
+//                 Fingerprint64("g"), FingerprintCat64(
+//                     Fingerprint64("e"), Fingerprint64("c")))
+//
+// Arguments:
+//	indices: 2-D.  Indices of each input `SparseTensor`.
+//	values: 1-D.   values of each `SparseTensor`.
+//	shapes: 1-D.   Shapes of each `SparseTensor`.
+//	dense_inputs: 2-D.    Columns represented by dense `Tensor`.
+//	num_buckets: It is used if hashed_output is true.
+// output = hashed_value%num_buckets if num_buckets > 0 else hashed_value.
+//	strong_hash: boolean, if true, siphash with salt will be used instead of farmhash.
+//	salt: Specify the salt that will be used by the siphash function.
+//
+// Returns:
+//	output_indices: 2-D.  Indices of the concatenated `SparseTensor`.
+//	output_values: 1-D.  Non-empty values of the concatenated or hashed
+// `SparseTensor`.
+//	output_shape: 1-D.  Shape of the concatenated `SparseTensor`.
+func SparseCrossHashed(scope *Scope, indices []tf.Output, values []tf.Output, shapes []tf.Output, dense_inputs []tf.Output, num_buckets tf.Output, strong_hash tf.Output, salt tf.Output) (output_indices tf.Output, output_values tf.Output, output_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseCrossHashed",
+		Input: []tf.Input{
+			tf.OutputList(indices), tf.OutputList(values), tf.OutputList(shapes), tf.OutputList(dense_inputs), num_buckets, strong_hash, salt,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// QuantizedInstanceNormAttr is an optional argument to QuantizedInstanceNorm.
+type QuantizedInstanceNormAttr func(optionalAttr)
+
+// QuantizedInstanceNormOutputRangeGiven sets the optional output_range_given attribute to value.
+//
+// value: If True, `given_y_min` and `given_y_min`
+// and `given_y_max` are used as the output range. Otherwise,
+// the implementation computes the output range.
+// If not specified, defaults to false
+func QuantizedInstanceNormOutputRangeGiven(value bool) QuantizedInstanceNormAttr {
+	return func(m optionalAttr) {
+		m["output_range_given"] = value
+	}
+}
+
+// QuantizedInstanceNormGivenYMin sets the optional given_y_min attribute to value.
+//
+// value: Output in `y_min` if `output_range_given` is True.
+// If not specified, defaults to 0
+func QuantizedInstanceNormGivenYMin(value float32) QuantizedInstanceNormAttr {
+	return func(m optionalAttr) {
+		m["given_y_min"] = value
+	}
+}
+
+// QuantizedInstanceNormGivenYMax sets the optional given_y_max attribute to value.
+//
+// value: Output in `y_max` if `output_range_given` is True.
+// If not specified, defaults to 0
+func QuantizedInstanceNormGivenYMax(value float32) QuantizedInstanceNormAttr {
+	return func(m optionalAttr) {
+		m["given_y_max"] = value
+	}
+}
+
+// QuantizedInstanceNormVarianceEpsilon sets the optional variance_epsilon attribute to value.
+//
+// value: A small float number to avoid dividing by 0.
+// If not specified, defaults to 1e-05
+func QuantizedInstanceNormVarianceEpsilon(value float32) QuantizedInstanceNormAttr {
+	return func(m optionalAttr) {
+		m["variance_epsilon"] = value
+	}
+}
+
+// QuantizedInstanceNormMinSeparation sets the optional min_separation attribute to value.
+//
+// value: Minimum value of `y_max - y_min`
+// If not specified, defaults to 0.001
+func QuantizedInstanceNormMinSeparation(value float32) QuantizedInstanceNormAttr {
+	return func(m optionalAttr) {
+		m["min_separation"] = value
+	}
+}
+
+// Quantized Instance normalization.
+//
+// Arguments:
+//	x: A 4D input Tensor.
+//	x_min: The value represented by the lowest quantized input.
+//	x_max: The value represented by the highest quantized input.
+//
+// Returns:
+//	y: A 4D Tensor.
+//	y_min: The value represented by the lowest quantized output.
+//	y_max: The value represented by the highest quantized output.
+func QuantizedInstanceNorm(scope *Scope, x tf.Output, x_min tf.Output, x_max tf.Output, optional ...QuantizedInstanceNormAttr) (y tf.Output, y_min tf.Output, y_max tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizedInstanceNorm",
+		Input: []tf.Input{
+			x, x_min, x_max,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// FusedBatchNormV3Attr is an optional argument to FusedBatchNormV3.
+type FusedBatchNormV3Attr func(optionalAttr)
+
+// FusedBatchNormV3Epsilon sets the optional epsilon attribute to value.
+//
+// value: A small float number added to the variance of x.
+// If not specified, defaults to 0.0001
+func FusedBatchNormV3Epsilon(value float32) FusedBatchNormV3Attr {
+	return func(m optionalAttr) {
+		m["epsilon"] = value
+	}
+}
+
+// FusedBatchNormV3ExponentialAvgFactor sets the optional exponential_avg_factor attribute to value.
+// If not specified, defaults to 1
+func FusedBatchNormV3ExponentialAvgFactor(value float32) FusedBatchNormV3Attr {
+	return func(m optionalAttr) {
+		m["exponential_avg_factor"] = value
+	}
+}
+
+// FusedBatchNormV3DataFormat sets the optional data_format attribute to value.
+//
+// value: The data format for x and y. Either "NHWC" (default) or "NCHW".
+// If not specified, defaults to "NHWC"
+func FusedBatchNormV3DataFormat(value string) FusedBatchNormV3Attr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// FusedBatchNormV3IsTraining sets the optional is_training attribute to value.
+//
+// value: A bool value to indicate the operation is for training (default)
+// or inference.
+// If not specified, defaults to true
+func FusedBatchNormV3IsTraining(value bool) FusedBatchNormV3Attr {
+	return func(m optionalAttr) {
+		m["is_training"] = value
+	}
+}
+
+// Batch normalization.
+//
+// Note that the size of 4D Tensors are defined by either "NHWC" or "NCHW".
+// The size of 1D Tensors matches the dimension C of the 4D Tensors.
+//
+// Arguments:
+//	x: A 4D Tensor for input data.
+//	scale: A 1D Tensor for scaling factor, to scale the normalized x.
+//	offset: A 1D Tensor for offset, to shift to the normalized x.
+//	mean: A 1D Tensor for population mean. Used for inference only;
+// must be empty for training.
+//	variance: A 1D Tensor for population variance. Used for inference only;
+// must be empty for training.
+//
+// Returns:
+//	y: A 4D Tensor for output data.
+//	batch_mean: A 1D Tensor for the computed batch mean, to be used by TensorFlow
+// to compute the running mean.
+//	batch_variance: A 1D Tensor for the computed batch variance, to be used by
+// TensorFlow to compute the running variance.
+//	reserve_space_1: A 1D Tensor for the computed batch mean, to be reused
+// in the gradient computation.
+//	reserve_space_2: A 1D Tensor for the computed batch variance (inverted variance
+// in the cuDNN case), to be reused in the gradient computation.
+//	reserve_space_3: A 1D Tensor for some intermediate results, to be reused in the gradient
+// computation for better efficiency.
+func FusedBatchNormV3(scope *Scope, x tf.Output, scale tf.Output, offset tf.Output, mean tf.Output, variance tf.Output, optional ...FusedBatchNormV3Attr) (y tf.Output, batch_mean tf.Output, batch_variance tf.Output, reserve_space_1 tf.Output, reserve_space_2 tf.Output, reserve_space_3 tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "FusedBatchNormV3",
+		Input: []tf.Input{
+			x, scale, offset, mean, variance,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3), op.Output(4), op.Output(5)
+}
+
+// Computes reciprocal of square root of x element-wise.
+//
+// I.e., \\(y = 1 / \sqrt{x}\\).
+func Rsqrt(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Rsqrt",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// The gradient operator for the SparseSlice op.
+//
+// This op takes in the upstream gradient w.r.t. non-empty values of
+// the sliced `SparseTensor`, and outputs the gradients w.r.t.
+// the non-empty values of input `SparseTensor`.
+//
+// Arguments:
+//	backprop_val_grad: 1-D. The gradient with respect to
+// the non-empty values of the sliced `SparseTensor`.
+//	input_indices: 2-D.  The `indices` of the input `SparseTensor`.
+//	input_start: 1-D. tensor represents the start of the slice.
+//	output_indices: 2-D.  The `indices` of the sliced `SparseTensor`.
+//
+// Returns 1-D. The gradient with respect to the non-empty values of input `SparseTensor`.
+func SparseSliceGrad(scope *Scope, backprop_val_grad tf.Output, input_indices tf.Output, input_start tf.Output, output_indices tf.Output) (val_grad tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseSliceGrad",
+		Input: []tf.Input{
+			backprop_val_grad, input_indices, input_start, output_indices,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Generates sparse cross from a list of sparse and dense tensors.
+//
+// The op takes two lists, one of 2D `SparseTensor` and one of 2D `Tensor`, each
+// representing features of one feature column. It outputs a 2D `SparseTensor` with
+// the batchwise crosses of these features.
+//
+// For example, if the inputs are
+//
+//     inputs[0]: SparseTensor with shape = [2, 2]
+//     [0, 0]: "a"
+//     [1, 0]: "b"
+//     [1, 1]: "c"
+//
+//     inputs[1]: SparseTensor with shape = [2, 1]
+//     [0, 0]: "d"
+//     [1, 0]: "e"
+//
+//     inputs[2]: Tensor [["f"], ["g"]]
+//
+// then the output will be
+//
+//     shape = [2, 2]
+//     [0, 0]: "a_X_d_X_f"
+//     [1, 0]: "b_X_e_X_g"
+//     [1, 1]: "c_X_e_X_g"
+//
+// if hashed_output=true then the output will be
+//
+//     shape = [2, 2]
+//     [0, 0]: FingerprintCat64(
+//                 Fingerprint64("f"), FingerprintCat64(
+//                     Fingerprint64("d"), Fingerprint64("a")))
+//     [1, 0]: FingerprintCat64(
+//                 Fingerprint64("g"), FingerprintCat64(
+//                     Fingerprint64("e"), Fingerprint64("b")))
+//     [1, 1]: FingerprintCat64(
+//                 Fingerprint64("g"), FingerprintCat64(
+//                     Fingerprint64("e"), Fingerprint64("c")))
+//
+// Arguments:
+//	indices: 2-D.  Indices of each input `SparseTensor`.
+//	values: 1-D.   values of each `SparseTensor`.
+//	shapes: 1-D.   Shapes of each `SparseTensor`.
+//	dense_inputs: 2-D.    Columns represented by dense `Tensor`.
+//	sep: string used when joining a list of string inputs, can be used as separator later.
+//
+// Returns:
+//	output_indices: 2-D.  Indices of the concatenated `SparseTensor`.
+//	output_values: 1-D.  Non-empty values of the concatenated or hashed
+// `SparseTensor`.
+//	output_shape: 1-D.  Shape of the concatenated `SparseTensor`.
+func SparseCrossV2(scope *Scope, indices []tf.Output, values []tf.Output, shapes []tf.Output, dense_inputs []tf.Output, sep tf.Output) (output_indices tf.Output, output_values tf.Output, output_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseCrossV2",
+		Input: []tf.Input{
+			tf.OutputList(indices), tf.OutputList(values), tf.OutputList(shapes), tf.OutputList(dense_inputs), sep,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Pads a tensor with mirrored values.
+//
+// This operation pads a `input` with mirrored values according to the `paddings`
+// you specify. `paddings` is an integer tensor with shape `[n, 2]`, where n is
+// the rank of `input`. For each dimension D of `input`, `paddings[D, 0]` indicates
+// how many values to add before the contents of `input` in that dimension, and
+// `paddings[D, 1]` indicates how many values to add after the contents of `input`
+// in that dimension. Both `paddings[D, 0]` and `paddings[D, 1]` must be no greater
+// than `input.dim_size(D)` (or `input.dim_size(D) - 1`) if `copy_border` is true
+// (if false, respectively).
+//
+// The padded size of each dimension D of the output is:
+//
+// `paddings(D, 0) + input.dim_size(D) + paddings(D, 1)`
+//
+// For example:
+//
+// ```
+// # 't' is [[1, 2, 3], [4, 5, 6]].
+// # 'paddings' is [[1, 1]], [2, 2]].
+// # 'mode' is SYMMETRIC.
+// # rank of 't' is 2.
+// pad(t, paddings) ==> [[2, 1, 1, 2, 3, 3, 2]
+//                       [2, 1, 1, 2, 3, 3, 2]
+//                       [5, 4, 4, 5, 6, 6, 5]
+//                       [5, 4, 4, 5, 6, 6, 5]]
+// ```
+//
+// Arguments:
+//	input: The input tensor to be padded.
+//	paddings: A two-column matrix specifying the padding sizes. The number of
+// rows must be the same as the rank of `input`.
+//	mode: Either `REFLECT` or `SYMMETRIC`. In reflect mode the padded regions
+// do not include the borders, while in symmetric mode the padded regions
+// do include the borders. For example, if `input` is `[1, 2, 3]` and `paddings`
+// is `[0, 2]`, then the output is `[1, 2, 3, 2, 1]` in reflect mode, and
+// it is `[1, 2, 3, 3, 2]` in symmetric mode.
+//
+// Returns The padded tensor.
+func MirrorPad(scope *Scope, input tf.Output, paddings tf.Output, mode string) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"mode": mode}
+	opspec := tf.OpSpec{
+		Type: "MirrorPad",
+		Input: []tf.Input{
+			input, paddings,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// TensorArrayV3Attr is an optional argument to TensorArrayV3.
+type TensorArrayV3Attr func(optionalAttr)
+
+// TensorArrayV3ElementShape sets the optional element_shape attribute to value.
+//
+// value: The expected shape of an element, if known. Used to
+// validate the shapes of TensorArray elements. If this shape is not
+// fully specified, gathering zero-size TensorArrays is an error.
+// If not specified, defaults to <unknown_rank:true >
+func TensorArrayV3ElementShape(value tf.Shape) TensorArrayV3Attr {
+	return func(m optionalAttr) {
+		m["element_shape"] = value
+	}
+}
+
+// TensorArrayV3DynamicSize sets the optional dynamic_size attribute to value.
+//
+// value: A boolean that determines whether writes to the TensorArray
+// are allowed to grow the size.  By default, this is not allowed.
+// If not specified, defaults to false
+func TensorArrayV3DynamicSize(value bool) TensorArrayV3Attr {
+	return func(m optionalAttr) {
+		m["dynamic_size"] = value
+	}
+}
+
+// TensorArrayV3ClearAfterRead sets the optional clear_after_read attribute to value.
+//
+// value: If true (default), Tensors in the TensorArray are cleared
+// after being read.  This disables multiple read semantics but allows early
+// release of memory.
+// If not specified, defaults to true
+func TensorArrayV3ClearAfterRead(value bool) TensorArrayV3Attr {
+	return func(m optionalAttr) {
+		m["clear_after_read"] = value
+	}
+}
+
+// TensorArrayV3IdenticalElementShapes sets the optional identical_element_shapes attribute to value.
+//
+// value: If true (default is false), then all
+// elements in the TensorArray will be expected to have have identical shapes.
+// This allows certain behaviors, like dynamically checking for
+// consistent shapes on write, and being able to fill in properly
+// shaped zero tensors on stack -- even if the element_shape attribute
+// is not fully defined.
+// If not specified, defaults to false
+func TensorArrayV3IdenticalElementShapes(value bool) TensorArrayV3Attr {
+	return func(m optionalAttr) {
+		m["identical_element_shapes"] = value
+	}
+}
+
+// TensorArrayV3TensorArrayName sets the optional tensor_array_name attribute to value.
+//
+// value: Overrides the name used for the temporary tensor_array
+// resource. Default value is the name of the 'TensorArray' op (which
+// is guaranteed unique).
+// If not specified, defaults to ""
+func TensorArrayV3TensorArrayName(value string) TensorArrayV3Attr {
+	return func(m optionalAttr) {
+		m["tensor_array_name"] = value
+	}
+}
+
+// An array of Tensors of given size.
+//
+// Write data via Write and read via Read or Pack.
+//
+// Arguments:
+//	size: The size of the array.
+//	dtype: The type of the elements on the tensor_array.
+//
+// Returns:
+//	handle: The handle to the TensorArray.
+//	flow: A scalar used to control gradient flow.
+func TensorArrayV3(scope *Scope, size tf.Output, dtype tf.DataType, optional ...TensorArrayV3Attr) (handle tf.Output, flow tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorArrayV3",
+		Input: []tf.Input{
+			size,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// MatrixSolveLsAttr is an optional argument to MatrixSolveLs.
+type MatrixSolveLsAttr func(optionalAttr)
+
+// MatrixSolveLsFast sets the optional fast attribute to value.
+// If not specified, defaults to true
+func MatrixSolveLsFast(value bool) MatrixSolveLsAttr {
+	return func(m optionalAttr) {
+		m["fast"] = value
+	}
+}
+
+// Solves one or more linear least-squares problems.
+//
+// `matrix` is a tensor of shape `[..., M, N]` whose inner-most 2 dimensions
+// form real or complex matrices of size `[M, N]`. `Rhs` is a tensor of the same
+// type as `matrix` and shape `[..., M, K]`.
+// The output is a tensor shape `[..., N, K]` where each output matrix solves
+// each of the equations
+// `matrix[..., :, :]` * `output[..., :, :]` = `rhs[..., :, :]`
+// in the least squares sense.
+//
+// We use the following notation for (complex) matrix and right-hand sides
+// in the batch:
+//
+// `matrix`=\\(A \in \mathbb{C}^{m \times n}\\),
+// `rhs`=\\(B  \in \mathbb{C}^{m \times k}\\),
+// `output`=\\(X  \in \mathbb{C}^{n \times k}\\),
+// `l2_regularizer`=\\(\lambda \in \mathbb{R}\\).
+//
+// If `fast` is `True`, then the solution is computed by solving the normal
+// equations using Cholesky decomposition. Specifically, if \\(m \ge n\\) then
+// \\(X = (A^H A + \lambda I)^{-1} A^H B\\), which solves the least-squares
+// problem \\(X = \mathrm{argmin}_{Z \in \Re^{n \times k} } ||A Z - B||_F^2 + \lambda ||Z||_F^2\\).
+// If \\(m \lt n\\) then `output` is computed as
+// \\(X = A^H (A A^H + \lambda I)^{-1} B\\), which (for \\(\lambda = 0\\)) is the
+// minimum-norm solution to the under-determined linear system, i.e.
+// \\(X = \mathrm{argmin}_{Z \in \mathbb{C}^{n \times k} } ||Z||_F^2 \\),
+// subject to \\(A Z = B\\). Notice that the fast path is only numerically stable
+// when \\(A\\) is numerically full rank and has a condition number
+// \\(\mathrm{cond}(A) \lt \frac{1}{\sqrt{\epsilon_{mach} } }\\) or \\(\lambda\\) is
+// sufficiently large.
+//
+// If `fast` is `False` an algorithm based on the numerically robust complete
+// orthogonal decomposition is used. This computes the minimum-norm
+// least-squares solution, even when \\(A\\) is rank deficient. This path is
+// typically 6-7 times slower than the fast path. If `fast` is `False` then
+// `l2_regularizer` is ignored.
+//
+// Arguments:
+//	matrix: Shape is `[..., M, N]`.
+//	rhs: Shape is `[..., M, K]`.
+//	l2_regularizer: Scalar tensor.
+//
+// @compatibility(numpy)
+// Equivalent to np.linalg.lstsq
+// @end_compatibility
+//
+// Returns Shape is `[..., N, K]`.
+func MatrixSolveLs(scope *Scope, matrix tf.Output, rhs tf.Output, l2_regularizer tf.Output, optional ...MatrixSolveLsAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MatrixSolveLs",
+		Input: []tf.Input{
+			matrix, rhs, l2_regularizer,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Generates sparse cross from a list of sparse and dense tensors.
+//
+// The op takes two lists, one of 2D `SparseTensor` and one of 2D `Tensor`, each
+// representing features of one feature column. It outputs a 2D `SparseTensor` with
+// the batchwise crosses of these features.
+//
+// For example, if the inputs are
+//
+//     inputs[0]: SparseTensor with shape = [2, 2]
+//     [0, 0]: "a"
+//     [1, 0]: "b"
+//     [1, 1]: "c"
+//
+//     inputs[1]: SparseTensor with shape = [2, 1]
+//     [0, 0]: "d"
+//     [1, 0]: "e"
+//
+//     inputs[2]: Tensor [["f"], ["g"]]
+//
+// then the output will be
+//
+//     shape = [2, 2]
+//     [0, 0]: "a_X_d_X_f"
+//     [1, 0]: "b_X_e_X_g"
+//     [1, 1]: "c_X_e_X_g"
+//
+// if hashed_output=true then the output will be
+//
+//     shape = [2, 2]
+//     [0, 0]: FingerprintCat64(
+//                 Fingerprint64("f"), FingerprintCat64(
+//                     Fingerprint64("d"), Fingerprint64("a")))
+//     [1, 0]: FingerprintCat64(
+//                 Fingerprint64("g"), FingerprintCat64(
+//                     Fingerprint64("e"), Fingerprint64("b")))
+//     [1, 1]: FingerprintCat64(
+//                 Fingerprint64("g"), FingerprintCat64(
+//                     Fingerprint64("e"), Fingerprint64("c")))
+//
+// Arguments:
+//	indices: 2-D.  Indices of each input `SparseTensor`.
+//	values: 1-D.   values of each `SparseTensor`.
+//	shapes: 1-D.   Shapes of each `SparseTensor`.
+//	dense_inputs: 2-D.    Columns represented by dense `Tensor`.
+//	hashed_output: If true, returns the hash of the cross instead of the string.
+// This will allow us avoiding string manipulations.
+//	num_buckets: It is used if hashed_output is true.
+// output = hashed_value%num_buckets if num_buckets > 0 else hashed_value.
+//	hash_key: Specify the hash_key that will be used by the `FingerprintCat64`
+// function to combine the crosses fingerprints.
+//
+//
+//
+// Returns:
+//	output_indices: 2-D.  Indices of the concatenated `SparseTensor`.
+//	output_values: 1-D.  Non-empty values of the concatenated or hashed
+// `SparseTensor`.
+//	output_shape: 1-D.  Shape of the concatenated `SparseTensor`.
+func SparseCross(scope *Scope, indices []tf.Output, values []tf.Output, shapes []tf.Output, dense_inputs []tf.Output, hashed_output bool, num_buckets int64, hash_key int64, out_type tf.DataType, internal_type tf.DataType) (output_indices tf.Output, output_values tf.Output, output_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"hashed_output": hashed_output, "num_buckets": num_buckets, "hash_key": hash_key, "out_type": out_type, "internal_type": internal_type}
+	opspec := tf.OpSpec{
+		Type: "SparseCross",
+		Input: []tf.Input{
+			tf.OutputList(indices), tf.OutputList(values), tf.OutputList(shapes), tf.OutputList(dense_inputs),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Generate a glob pattern matching all sharded file names.
+func ShardedFilespec(scope *Scope, basename tf.Output, num_shards tf.Output) (filename tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ShardedFilespec",
+		Input: []tf.Input{
+			basename, num_shards,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RetrieveTPUEmbeddingProximalAdagradParametersAttr is an optional argument to RetrieveTPUEmbeddingProximalAdagradParameters.
+type RetrieveTPUEmbeddingProximalAdagradParametersAttr func(optionalAttr)
+
+// RetrieveTPUEmbeddingProximalAdagradParametersTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func RetrieveTPUEmbeddingProximalAdagradParametersTableId(value int64) RetrieveTPUEmbeddingProximalAdagradParametersAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingProximalAdagradParametersTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingProximalAdagradParametersTableName(value string) RetrieveTPUEmbeddingProximalAdagradParametersAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingProximalAdagradParametersConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingProximalAdagradParametersConfig(value string) RetrieveTPUEmbeddingProximalAdagradParametersAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Retrieve proximal Adagrad embedding parameters.
+//
+// An op that retrieves optimization parameters from embedding to host
+// memory. Must be preceded by a ConfigureTPUEmbeddingHost op that sets up
+// the correct embedding table configuration. For example, this op is
+// used to retrieve updated parameters before saving a checkpoint.
+//
+// Returns:
+//	parameters: Parameter parameters updated by the proximal Adagrad optimization algorithm.
+//	accumulators: Parameter accumulators updated by the proximal Adagrad optimization algorithm.
+func RetrieveTPUEmbeddingProximalAdagradParameters(scope *Scope, num_shards int64, shard_id int64, optional ...RetrieveTPUEmbeddingProximalAdagradParametersAttr) (parameters tf.Output, accumulators tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RetrieveTPUEmbeddingProximalAdagradParameters",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// ReduceJoinAttr is an optional argument to ReduceJoin.
+type ReduceJoinAttr func(optionalAttr)
+
+// ReduceJoinKeepDims sets the optional keep_dims attribute to value.
+//
+// value: If `True`, retain reduced dimensions with length `1`.
+// If not specified, defaults to false
+func ReduceJoinKeepDims(value bool) ReduceJoinAttr {
+	return func(m optionalAttr) {
+		m["keep_dims"] = value
+	}
+}
+
+// ReduceJoinSeparator sets the optional separator attribute to value.
+//
+// value: The separator to use when joining.
+// If not specified, defaults to ""
+func ReduceJoinSeparator(value string) ReduceJoinAttr {
+	return func(m optionalAttr) {
+		m["separator"] = value
+	}
+}
+
+// Joins a string Tensor across the given dimensions.
+//
+// Computes the string join across dimensions in the given string Tensor of shape
+// `[\\(d_0, d_1, ..., d_{n-1}\\)]`.  Returns a new Tensor created by joining the input
+// strings with the given separator (default: empty string).  Negative indices are
+// counted backwards from the end, with `-1` being equivalent to `n - 1`.  If
+// indices are not specified, joins across all dimensions beginning from `n - 1`
+// through `0`.
+//
+// For example:
+//
+// ```python
+// # tensor `a` is [["a", "b"], ["c", "d"]]
+// tf.reduce_join(a, 0) ==> ["ac", "bd"]
+// tf.reduce_join(a, 1) ==> ["ab", "cd"]
+// tf.reduce_join(a, -2) = tf.reduce_join(a, 0) ==> ["ac", "bd"]
+// tf.reduce_join(a, -1) = tf.reduce_join(a, 1) ==> ["ab", "cd"]
+// tf.reduce_join(a, 0, keep_dims=True) ==> [["ac", "bd"]]
+// tf.reduce_join(a, 1, keep_dims=True) ==> [["ab"], ["cd"]]
+// tf.reduce_join(a, 0, separator=".") ==> ["a.c", "b.d"]
+// tf.reduce_join(a, [0, 1]) ==> "acbd"
+// tf.reduce_join(a, [1, 0]) ==> "abcd"
+// tf.reduce_join(a, []) ==> [["a", "b"], ["c", "d"]]
+// tf.reduce_join(a) = tf.reduce_join(a, [1, 0]) ==> "abcd"
+// ```
+//
+// Arguments:
+//	inputs: The input to be joined.  All reduced indices must have non-zero size.
+//	reduction_indices: The dimensions to reduce over.  Dimensions are reduced in the
+// order specified.  Omitting `reduction_indices` is equivalent to passing
+// `[n-1, n-2, ..., 0]`.  Negative indices from `-n` to `-1` are supported.
+//
+// Returns Has shape equal to that of the input with reduced dimensions removed or
+// set to `1` depending on `keep_dims`.
+func ReduceJoin(scope *Scope, inputs tf.Output, reduction_indices tf.Output, optional ...ReduceJoinAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ReduceJoin",
+		Input: []tf.Input{
+			inputs, reduction_indices,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Inverse 2D fast Fourier transform.
+//
+// Computes the inverse 2-dimensional discrete Fourier transform over the
+// inner-most 2 dimensions of `input`.
+//
+// Arguments:
+//	input: A complex tensor.
+//
+// Returns A complex tensor of the same shape as `input`. The inner-most 2
+//   dimensions of `input` are replaced with their inverse 2D Fourier transform.
+//
+// @compatibility(numpy)
+// Equivalent to np.fft.ifft2
+// @end_compatibility
+func IFFT2D(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "IFFT2D",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Concatenates a list of `SparseTensor` along the specified dimension.
+//
+// Concatenation is with respect to the dense versions of these sparse tensors.
+// It is assumed that each input is a `SparseTensor` whose elements are ordered
+// along increasing dimension number.
+//
+// All inputs' shapes must match, except for the concat dimension.  The
+// `indices`, `values`, and `shapes` lists must have the same length.
+//
+// The output shape is identical to the inputs', except along the concat
+// dimension, where it is the sum of the inputs' sizes along that dimension.
+//
+// The output elements will be resorted to preserve the sort order along
+// increasing dimension number.
+//
+// This op runs in `O(M log M)` time, where `M` is the total number of non-empty
+// values across all inputs. This is due to the need for an internal sort in
+// order to concatenate efficiently across an arbitrary dimension.
+//
+// For example, if `concat_dim = 1` and the inputs are
+//
+//     sp_inputs[0]: shape = [2, 3]
+//     [0, 2]: "a"
+//     [1, 0]: "b"
+//     [1, 1]: "c"
+//
+//     sp_inputs[1]: shape = [2, 4]
+//     [0, 1]: "d"
+//     [0, 2]: "e"
+//
+// then the output will be
+//
+//     shape = [2, 7]
+//     [0, 2]: "a"
+//     [0, 4]: "d"
+//     [0, 5]: "e"
+//     [1, 0]: "b"
+//     [1, 1]: "c"
+//
+// Graphically this is equivalent to doing
+//
+//     [    a] concat [  d e  ] = [    a   d e  ]
+//     [b c  ]        [       ]   [b c          ]
+//
+// Arguments:
+//	indices: 2-D.  Indices of each input `SparseTensor`.
+//	values: 1-D.  Non-empty values of each `SparseTensor`.
+//	shapes: 1-D.  Shapes of each `SparseTensor`.
+//	concat_dim: Dimension to concatenate along. Must be in range [-rank, rank),
+// where rank is the number of dimensions in each input `SparseTensor`.
+//
+// Returns:
+//	output_indices: 2-D.  Indices of the concatenated `SparseTensor`.
+//	output_values: 1-D.  Non-empty values of the concatenated `SparseTensor`.
+//	output_shape: 1-D.  Shape of the concatenated `SparseTensor`.
+func SparseConcat(scope *Scope, indices []tf.Output, values []tf.Output, shapes []tf.Output, concat_dim int64) (output_indices tf.Output, output_values tf.Output, output_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"concat_dim": concat_dim}
+	opspec := tf.OpSpec{
+		Type: "SparseConcat",
+		Input: []tf.Input{
+			tf.OutputList(indices), tf.OutputList(values), tf.OutputList(shapes),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// DestroyResourceOpAttr is an optional argument to DestroyResourceOp.
+type DestroyResourceOpAttr func(optionalAttr)
+
+// DestroyResourceOpIgnoreLookupError sets the optional ignore_lookup_error attribute to value.
+//
+// value: whether to ignore the error when the resource
+// doesn't exist.
+// If not specified, defaults to true
+func DestroyResourceOpIgnoreLookupError(value bool) DestroyResourceOpAttr {
+	return func(m optionalAttr) {
+		m["ignore_lookup_error"] = value
+	}
+}
+
+// Deletes the resource specified by the handle.
+//
+// All subsequent operations using the resource will result in a NotFound
+// error status.
+//
+// Arguments:
+//	resource: handle to the resource to delete.
+//
+// Returns the created operation.
+func DestroyResourceOp(scope *Scope, resource tf.Output, optional ...DestroyResourceOpAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DestroyResourceOp",
+		Input: []tf.Input{
+			resource,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// HistogramFixedWidthAttr is an optional argument to HistogramFixedWidth.
+type HistogramFixedWidthAttr func(optionalAttr)
+
+// HistogramFixedWidthDtype sets the optional dtype attribute to value.
+// If not specified, defaults to DT_INT32
+func HistogramFixedWidthDtype(value tf.DataType) HistogramFixedWidthAttr {
+	return func(m optionalAttr) {
+		m["dtype"] = value
+	}
+}
+
+// Return histogram of values.
+//
+// Given the tensor `values`, this operation returns a rank 1 histogram counting
+// the number of entries in `values` that fall into every bin.  The bins are
+// equal width and determined by the arguments `value_range` and `nbins`.
+//
+// ```python
+// # Bins will be:  (-inf, 1), [1, 2), [2, 3), [3, 4), [4, inf)
+// nbins = 5
+// value_range = [0.0, 5.0]
+// new_values = [-1.0, 0.0, 1.5, 2.0, 5.0, 15]
+//
+// with tf.get_default_session() as sess:
+//   hist = tf.histogram_fixed_width(new_values, value_range, nbins=5)
+//   variables.global_variables_initializer().run()
+//   sess.run(hist) => [2, 1, 1, 0, 2]
+// ```
+//
+// Arguments:
+//	values: Numeric `Tensor`.
+//	value_range: Shape [2] `Tensor` of same `dtype` as `values`.
+// values <= value_range[0] will be mapped to hist[0],
+// values >= value_range[1] will be mapped to hist[-1].
+//	nbins: Scalar `int32 Tensor`.  Number of histogram bins.
+//
+// Returns A 1-D `Tensor` holding histogram of values.
+func HistogramFixedWidth(scope *Scope, values tf.Output, value_range tf.Output, nbins tf.Output, optional ...HistogramFixedWidthAttr) (out tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "HistogramFixedWidth",
+		Input: []tf.Input{
+			values, value_range, nbins,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that uses a custom thread pool to compute `input_dataset`.
+//
+// Arguments:
+//
+//	thread_pool: A resource produced by the ThreadPoolHandle op.
+//
+//
+func ThreadPoolDataset(scope *Scope, input_dataset tf.Output, thread_pool tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "ThreadPoolDataset",
+		Input: []tf.Input{
+			input_dataset, thread_pool,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Bitcasts a tensor from one type to another without copying data.
+//
+// Given a tensor `input`, this operation returns a tensor that has the same buffer
+// data as `input` with datatype `type`.
+//
+// If the input datatype `T` is larger than the output datatype `type` then the
+// shape changes from [...] to [..., sizeof(`T`)/sizeof(`type`)].
+//
+// If `T` is smaller than `type`, the operator requires that the rightmost
+// dimension be equal to sizeof(`type`)/sizeof(`T`). The shape then goes from
+// [..., sizeof(`type`)/sizeof(`T`)] to [...].
+//
+// tf.bitcast() and tf.cast() work differently when real dtype is casted as a complex dtype
+// (e.g. tf.complex64 or tf.complex128) as tf.cast() make imaginary part 0 while tf.bitcast()
+// gives module error.
+// For example,
+//
+// Example 1:
+//
+// >>> a = [1., 2., 3.]
+// >>> equality_bitcast = tf.bitcast(a, tf.complex128)
+// Traceback (most recent call last):
+// ...
+// InvalidArgumentError: Cannot bitcast from 1 to 18 [Op:Bitcast]
+// >>> equality_cast = tf.cast(a, tf.complex128)
+// >>> print(equality_cast)
+// tf.Tensor([1.+0.j 2.+0.j 3.+0.j], shape=(3,), dtype=complex128)
+//
+// Example 2:
+//
+// >>> tf.bitcast(tf.constant(0xffffffff, dtype=tf.uint32), tf.uint8)
+// <tf.Tensor: shape=(4,), dtype=uint8, numpy=array([255, 255, 255, 255], dtype=uint8)>
+//
+// Example 3:
+//
+// >>> x = [1., 2., 3.]
+// >>> y = [0., 2., 3.]
+// >>> equality= tf.equal(x,y)
+// >>> equality_cast = tf.cast(equality,tf.float32)
+// >>> equality_bitcast = tf.bitcast(equality_cast,tf.uint8)
+// >>> print(equality)
+// tf.Tensor([False True True], shape=(3,), dtype=bool)
+// >>> print(equality_cast)
+// tf.Tensor([0. 1. 1.], shape=(3,), dtype=float32)
+// >>> print(equality_bitcast)
+// tf.Tensor(
+//     [[  0   0   0   0]
+//      [  0   0 128  63]
+//      [  0   0 128  63]], shape=(3, 4), dtype=uint8)
+//
+// *NOTE*: Bitcast is implemented as a low-level cast, so machines with different
+// endian orderings will give different results.
+func Bitcast(scope *Scope, input tf.Output, type_ tf.DataType) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"type": type_}
+	opspec := tf.OpSpec{
+		Type: "Bitcast",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResourceApplyAdagradDAAttr is an optional argument to ResourceApplyAdagradDA.
+type ResourceApplyAdagradDAAttr func(optionalAttr)
+
+// ResourceApplyAdagradDAUseLocking sets the optional use_locking attribute to value.
+//
+// value: If True, updating of the var and accum tensors will be protected by
+// a lock; otherwise the behavior is undefined, but may exhibit less contention.
+// If not specified, defaults to false
+func ResourceApplyAdagradDAUseLocking(value bool) ResourceApplyAdagradDAAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// Update '*var' according to the proximal adagrad scheme.
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	gradient_accumulator: Should be from a Variable().
+//	gradient_squared_accumulator: Should be from a Variable().
+//	grad: The gradient.
+//	lr: Scaling factor. Must be a scalar.
+//	l1: L1 regularization. Must be a scalar.
+//	l2: L2 regularization. Must be a scalar.
+//	global_step: Training step number. Must be a scalar.
+//
+// Returns the created operation.
+func ResourceApplyAdagradDA(scope *Scope, var_ tf.Output, gradient_accumulator tf.Output, gradient_squared_accumulator tf.Output, grad tf.Output, lr tf.Output, l1 tf.Output, l2 tf.Output, global_step tf.Output, optional ...ResourceApplyAdagradDAAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceApplyAdagradDA",
+		Input: []tf.Input{
+			var_, gradient_accumulator, gradient_squared_accumulator, grad, lr, l1, l2, global_step,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// SparseToDenseAttr is an optional argument to SparseToDense.
+type SparseToDenseAttr func(optionalAttr)
+
+// SparseToDenseValidateIndices sets the optional validate_indices attribute to value.
+//
+// value: If true, indices are checked to make sure they are sorted in
+// lexicographic order and that there are no repeats.
+// If not specified, defaults to true
+func SparseToDenseValidateIndices(value bool) SparseToDenseAttr {
+	return func(m optionalAttr) {
+		m["validate_indices"] = value
+	}
+}
+
+// Converts a sparse representation into a dense tensor.
+//
+// Builds an array `dense` with shape `output_shape` such that
+//
+// ```
+// # If sparse_indices is scalar
+// dense[i] = (i == sparse_indices ? sparse_values : default_value)
+//
+// # If sparse_indices is a vector, then for each i
+// dense[sparse_indices[i]] = sparse_values[i]
+//
+// # If sparse_indices is an n by d matrix, then for each i in [0, n)
+// dense[sparse_indices[i][0], ..., sparse_indices[i][d-1]] = sparse_values[i]
+// ```
+//
+// All other values in `dense` are set to `default_value`.  If `sparse_values` is a
+// scalar, all sparse indices are set to this single value.
+//
+// Indices should be sorted in lexicographic order, and indices must not
+// contain any repeats. If `validate_indices` is true, these properties
+// are checked during execution.
+//
+// Arguments:
+//	sparse_indices: 0-D, 1-D, or 2-D.  `sparse_indices[i]` contains the complete
+// index where `sparse_values[i]` will be placed.
+//	output_shape: 1-D.  Shape of the dense output tensor.
+//	sparse_values: 1-D.  Values corresponding to each row of `sparse_indices`,
+// or a scalar value to be used for all sparse indices.
+//	default_value: Scalar value to set for indices not specified in
+// `sparse_indices`.
+//
+// Returns Dense output tensor of shape `output_shape`.
+func SparseToDense(scope *Scope, sparse_indices tf.Output, output_shape tf.Output, sparse_values tf.Output, default_value tf.Output, optional ...SparseToDenseAttr) (dense tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseToDense",
+		Input: []tf.Input{
+			sparse_indices, output_shape, sparse_values, default_value,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// OrderedMapClearAttr is an optional argument to OrderedMapClear.
+type OrderedMapClearAttr func(optionalAttr)
+
+// OrderedMapClearCapacity sets the optional capacity attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func OrderedMapClearCapacity(value int64) OrderedMapClearAttr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// OrderedMapClearMemoryLimit sets the optional memory_limit attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func OrderedMapClearMemoryLimit(value int64) OrderedMapClearAttr {
+	return func(m optionalAttr) {
+		m["memory_limit"] = value
+	}
+}
+
+// OrderedMapClearContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func OrderedMapClearContainer(value string) OrderedMapClearAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// OrderedMapClearSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func OrderedMapClearSharedName(value string) OrderedMapClearAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Op removes all elements in the underlying container.
+//
+// Returns the created operation.
+func OrderedMapClear(scope *Scope, dtypes []tf.DataType, optional ...OrderedMapClearAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "OrderedMapClear",
+
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// MaxPoolAttr is an optional argument to MaxPool.
+type MaxPoolAttr func(optionalAttr)
+
+// MaxPoolDataFormat sets the optional data_format attribute to value.
+//
+// value: Specify the data format of the input and output data. With the
+// default format "NHWC", the data is stored in the order of:
+//     [batch, in_height, in_width, in_channels].
+// Alternatively, the format could be "NCHW", the data storage order of:
+//     [batch, in_channels, in_height, in_width].
+// If not specified, defaults to "NHWC"
+func MaxPoolDataFormat(value string) MaxPoolAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// Performs max pooling on the input.
+//
+// Arguments:
+//	input: 4-D input to pool over.
+//	ksize: The size of the window for each dimension of the input tensor.
+//	strides: The stride of the sliding window for each dimension of the
+// input tensor.
+//	padding: The type of padding algorithm to use.
+//
+// Returns The max pooled output tensor.
+func MaxPool(scope *Scope, input tf.Output, ksize []int64, strides []int64, padding string, optional ...MaxPoolAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"ksize": ksize, "strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MaxPool",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// BlockLSTMAttr is an optional argument to BlockLSTM.
+type BlockLSTMAttr func(optionalAttr)
+
+// BlockLSTMForgetBias sets the optional forget_bias attribute to value.
+//
+// value: The forget gate bias.
+// If not specified, defaults to 1
+func BlockLSTMForgetBias(value float32) BlockLSTMAttr {
+	return func(m optionalAttr) {
+		m["forget_bias"] = value
+	}
+}
+
+// BlockLSTMCellClip sets the optional cell_clip attribute to value.
+//
+// value: Value to clip the 'cs' value to.
+// If not specified, defaults to 3
+func BlockLSTMCellClip(value float32) BlockLSTMAttr {
+	return func(m optionalAttr) {
+		m["cell_clip"] = value
+	}
+}
+
+// BlockLSTMUsePeephole sets the optional use_peephole attribute to value.
+//
+// value: Whether to use peephole weights.
+// If not specified, defaults to false
+func BlockLSTMUsePeephole(value bool) BlockLSTMAttr {
+	return func(m optionalAttr) {
+		m["use_peephole"] = value
+	}
+}
+
+// Computes the LSTM cell forward propagation for all the time steps.
+//
+// This is equivalent to applying LSTMBlockCell in a loop, like so:
+//
+// ```python
+// for x1 in unpack(x):
+//   i1, cs1, f1, o1, ci1, co1, h1 = LSTMBlock(
+//     x1, cs_prev, h_prev, w, wci, wcf, wco, b)
+//   cs_prev = cs1
+//   h_prev = h1
+//   i.append(i1)
+//   cs.append(cs1)
+//   f.append(f1)
+//   o.append(o1)
+//   ci.append(ci1)
+//   co.append(co1)
+//   h.append(h1)
+// return pack(i), pack(cs), pack(f), pack(o), pack(ci), pack(ch), pack(h)
+// ```
+//
+// Arguments:
+//	seq_len_max: Maximum time length actually used by this input. Outputs are padded
+// with zeros beyond this length.
+//	x: The sequence input to the LSTM, shape (timelen, batch_size, num_inputs).
+//	cs_prev: Value of the initial cell state.
+//	h_prev: Initial output of cell (to be used for peephole).
+//	w: The weight matrix.
+//	wci: The weight matrix for input gate peephole connection.
+//	wcf: The weight matrix for forget gate peephole connection.
+//	wco: The weight matrix for output gate peephole connection.
+//	b: The bias vector.
+//
+// Returns:
+//	i: The input gate over the whole time sequence.
+//	cs: The cell state before the tanh over the whole time sequence.
+//	f: The forget gate over the whole time sequence.
+//	o: The output gate over the whole time sequence.
+//	ci: The cell input over the whole time sequence.
+//	co: The cell after the tanh over the whole time sequence.
+//	h: The output h vector over the whole time sequence.
+func BlockLSTM(scope *Scope, seq_len_max tf.Output, x tf.Output, cs_prev tf.Output, h_prev tf.Output, w tf.Output, wci tf.Output, wcf tf.Output, wco tf.Output, b tf.Output, optional ...BlockLSTMAttr) (i tf.Output, cs tf.Output, f tf.Output, o tf.Output, ci tf.Output, co tf.Output, h tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "BlockLSTM",
+		Input: []tf.Input{
+			seq_len_max, x, cs_prev, h_prev, w, wci, wcf, wco, b,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3), op.Output(4), op.Output(5), op.Output(6)
+}
+
+// Computes the GRU cell forward propagation for 1 time step.
+//
+// Args
+//     x: Input to the GRU cell.
+//     h_prev: State input from the previous GRU cell.
+//     w_ru: Weight matrix for the reset and update gate.
+//     w_c: Weight matrix for the cell connection gate.
+//     b_ru: Bias vector for the reset and update gate.
+//     b_c: Bias vector for the cell connection gate.
+//
+// Returns
+//     r: Output of the reset gate.
+//     u: Output of the update gate.
+//     c: Output of the cell connection gate.
+//     h: Current state of the GRU cell.
+//
+// Note on notation of the variables:
+//
+// Concatenation of a and b is represented by a_b
+// Element-wise dot product of a and b is represented by ab
+// Element-wise dot product is represented by \circ
+// Matrix multiplication is represented by *
+//
+// Biases are initialized with :
+// `b_ru` - constant_initializer(1.0)
+// `b_c` - constant_initializer(0.0)
+//
+// This kernel op implements the following mathematical equations:
+//
+// ```
+// x_h_prev = [x, h_prev]
+//
+// [r_bar u_bar] = x_h_prev * w_ru + b_ru
+//
+// r = sigmoid(r_bar)
+// u = sigmoid(u_bar)
+//
+// h_prevr = h_prev \circ r
+//
+// x_h_prevr = [x h_prevr]
+//
+// c_bar = x_h_prevr * w_c + b_c
+// c = tanh(c_bar)
+//
+// h = (1-u) \circ c + u \circ h_prev
+// ```
+func GRUBlockCell(scope *Scope, x tf.Output, h_prev tf.Output, w_ru tf.Output, w_c tf.Output, b_ru tf.Output, b_c tf.Output) (r tf.Output, u tf.Output, c tf.Output, h tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "GRUBlockCell",
+		Input: []tf.Input{
+			x, h_prev, w_ru, w_c, b_ru, b_c,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3)
+}
+
+// Deserialize and concatenate `SparseTensors` from a serialized minibatch.
+//
+// The input `serialized_sparse` must be a string matrix of shape `[N x 3]` where
+// `N` is the minibatch size and the rows correspond to packed outputs of
+// `SerializeSparse`.  The ranks of the original `SparseTensor` objects
+// must all match.  When the final `SparseTensor` is created, it has rank one
+// higher than the ranks of the incoming `SparseTensor` objects
+// (they have been concatenated along a new row dimension).
+//
+// The output `SparseTensor` object's shape values for all dimensions but the
+// first are the max across the input `SparseTensor` objects' shape values
+// for the corresponding dimensions.  Its first shape value is `N`, the minibatch
+// size.
+//
+// The input `SparseTensor` objects' indices are assumed ordered in
+// standard lexicographic order.  If this is not the case, after this
+// step run `SparseReorder` to restore index ordering.
+//
+// For example, if the serialized input is a `[2 x 3]` matrix representing two
+// original `SparseTensor` objects:
+//
+//     index = [ 0]
+//             [10]
+//             [20]
+//     values = [1, 2, 3]
+//     shape = [50]
+//
+// and
+//
+//     index = [ 2]
+//             [10]
+//     values = [4, 5]
+//     shape = [30]
+//
+// then the final deserialized `SparseTensor` will be:
+//
+//     index = [0  0]
+//             [0 10]
+//             [0 20]
+//             [1  2]
+//             [1 10]
+//     values = [1, 2, 3, 4, 5]
+//     shape = [2 50]
+//
+// Arguments:
+//	serialized_sparse: 2-D, The `N` serialized `SparseTensor` objects.
+// Must have 3 columns.
+//	dtype: The `dtype` of the serialized `SparseTensor` objects.
+func DeserializeManySparse(scope *Scope, serialized_sparse tf.Output, dtype tf.DataType) (sparse_indices tf.Output, sparse_values tf.Output, sparse_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	opspec := tf.OpSpec{
+		Type: "DeserializeManySparse",
+		Input: []tf.Input{
+			serialized_sparse,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// MapStageAttr is an optional argument to MapStage.
+type MapStageAttr func(optionalAttr)
+
+// MapStageCapacity sets the optional capacity attribute to value.
+//
+// value: Maximum number of elements in the Staging Area. If > 0, inserts
+// on the container will block when the capacity is reached.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func MapStageCapacity(value int64) MapStageAttr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// MapStageMemoryLimit sets the optional memory_limit attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func MapStageMemoryLimit(value int64) MapStageAttr {
+	return func(m optionalAttr) {
+		m["memory_limit"] = value
+	}
+}
+
+// MapStageContainer sets the optional container attribute to value.
+//
+// value: If non-empty, this queue is placed in the given container. Otherwise,
+// a default container is used.
+// If not specified, defaults to ""
+func MapStageContainer(value string) MapStageAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// MapStageSharedName sets the optional shared_name attribute to value.
+//
+// value: It is necessary to match this name to the matching Unstage Op.
+// If not specified, defaults to ""
+func MapStageSharedName(value string) MapStageAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Stage (key, values) in the underlying container which behaves like a hashtable.
+//
+// Arguments:
+//	key: int64
+//
+//	values: a list of tensors
+// dtypes A list of data types that inserted values should adhere to.
+//
+//
+// Returns the created operation.
+func MapStage(scope *Scope, key tf.Output, indices tf.Output, values []tf.Output, dtypes []tf.DataType, optional ...MapStageAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MapStage",
+		Input: []tf.Input{
+			key, indices, tf.OutputList(values),
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// FusedBatchNormGradV2Attr is an optional argument to FusedBatchNormGradV2.
+type FusedBatchNormGradV2Attr func(optionalAttr)
+
+// FusedBatchNormGradV2Epsilon sets the optional epsilon attribute to value.
+//
+// value: A small float number added to the variance of x.
+// If not specified, defaults to 0.0001
+func FusedBatchNormGradV2Epsilon(value float32) FusedBatchNormGradV2Attr {
+	return func(m optionalAttr) {
+		m["epsilon"] = value
+	}
+}
+
+// FusedBatchNormGradV2DataFormat sets the optional data_format attribute to value.
+//
+// value: The data format for y_backprop, x, x_backprop.
+// Either "NHWC" (default) or "NCHW".
+// If not specified, defaults to "NHWC"
+func FusedBatchNormGradV2DataFormat(value string) FusedBatchNormGradV2Attr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// FusedBatchNormGradV2IsTraining sets the optional is_training attribute to value.
+//
+// value: A bool value to indicate the operation is for training (default)
+// or inference.
+// If not specified, defaults to true
+func FusedBatchNormGradV2IsTraining(value bool) FusedBatchNormGradV2Attr {
+	return func(m optionalAttr) {
+		m["is_training"] = value
+	}
+}
+
+// Gradient for batch normalization.
+//
+// Note that the size of 4D Tensors are defined by either "NHWC" or "NCHW".
+// The size of 1D Tensors matches the dimension C of the 4D Tensors.
+//
+// Arguments:
+//	y_backprop: A 4D Tensor for the gradient with respect to y.
+//	x: A 4D Tensor for input data.
+//	scale: A 1D Tensor for scaling factor, to scale the normalized x.
+//	reserve_space_1: When is_training is True, a 1D Tensor for the computed batch
+// mean to be reused in gradient computation. When is_training is
+// False, a 1D Tensor for the population mean to be reused in both
+// 1st and 2nd order gradient computation.
+//	reserve_space_2: When is_training is True, a 1D Tensor for the computed batch
+// variance (inverted variance in the cuDNN case) to be reused in
+// gradient computation. When is_training is False, a 1D Tensor
+// for the population variance to be reused in both 1st and 2nd
+// order gradient computation.
+//
+// Returns:
+//	x_backprop: A 4D Tensor for the gradient with respect to x.
+//	scale_backprop: A 1D Tensor for the gradient with respect to scale.
+//	offset_backprop: A 1D Tensor for the gradient with respect to offset.
+//	reserve_space_3: Unused placeholder to match the mean input in FusedBatchNorm.
+//	reserve_space_4: Unused placeholder to match the variance input
+// in FusedBatchNorm.
+func FusedBatchNormGradV2(scope *Scope, y_backprop tf.Output, x tf.Output, scale tf.Output, reserve_space_1 tf.Output, reserve_space_2 tf.Output, optional ...FusedBatchNormGradV2Attr) (x_backprop tf.Output, scale_backprop tf.Output, offset_backprop tf.Output, reserve_space_3 tf.Output, reserve_space_4 tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "FusedBatchNormGradV2",
+		Input: []tf.Input{
+			y_backprop, x, scale, reserve_space_1, reserve_space_2,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3), op.Output(4)
+}
+
+// SparseTensorDenseMatMulAttr is an optional argument to SparseTensorDenseMatMul.
+type SparseTensorDenseMatMulAttr func(optionalAttr)
+
+// SparseTensorDenseMatMulAdjointA sets the optional adjoint_a attribute to value.
+//
+// value: Use the adjoint of A in the matrix multiply.  If A is complex, this
+// is transpose(conj(A)).  Otherwise it's transpose(A).
+// If not specified, defaults to false
+func SparseTensorDenseMatMulAdjointA(value bool) SparseTensorDenseMatMulAttr {
+	return func(m optionalAttr) {
+		m["adjoint_a"] = value
+	}
+}
+
+// SparseTensorDenseMatMulAdjointB sets the optional adjoint_b attribute to value.
+//
+// value: Use the adjoint of B in the matrix multiply.  If B is complex, this
+// is transpose(conj(B)).  Otherwise it's transpose(B).
+// If not specified, defaults to false
+func SparseTensorDenseMatMulAdjointB(value bool) SparseTensorDenseMatMulAttr {
+	return func(m optionalAttr) {
+		m["adjoint_b"] = value
+	}
+}
+
+// Multiply SparseTensor (of rank 2) "A" by dense matrix "B".
+//
+// No validity checking is performed on the indices of A.  However, the following
+// input format is recommended for optimal behavior:
+//
+// if adjoint_a == false:
+//   A should be sorted in lexicographically increasing order.  Use SparseReorder
+//   if you're not sure.
+// if adjoint_a == true:
+//   A should be sorted in order of increasing dimension 1 (i.e., "column major"
+//   order instead of "row major" order).
+//
+// Arguments:
+//	a_indices: 2-D.  The `indices` of the `SparseTensor`, size `[nnz, 2]` Matrix.
+//	a_values: 1-D.  The `values` of the `SparseTensor`, size `[nnz]` Vector.
+//	a_shape: 1-D.  The `shape` of the `SparseTensor`, size `[2]` Vector.
+//	b: 2-D.  A dense Matrix.
+func SparseTensorDenseMatMul(scope *Scope, a_indices tf.Output, a_values tf.Output, a_shape tf.Output, b tf.Output, optional ...SparseTensorDenseMatMulAttr) (product tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseTensorDenseMatMul",
+		Input: []tf.Input{
+			a_indices, a_values, a_shape, b,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Sets the index-th position of the list to contain the given tensor.
+//
+// input_handle: the list
+// index: the position in the list to which the tensor will be assigned
+// item: the element to be assigned to that position
+// output_handle: the new list, with the element in the proper position
+//
+func TensorListSetItem(scope *Scope, input_handle tf.Output, index tf.Output, item tf.Output) (output_handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorListSetItem",
+		Input: []tf.Input{
+			input_handle, index, item,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Writes the given dataset to the given file using the TFRecord format.
+//
+// Arguments:
+//	input_dataset: A variant tensor representing the dataset to write.
+//	filename: A scalar string tensor representing the filename to use.
+//	compression_type: A scalar string tensor containing either (i) the empty string (no
+// compression), (ii) "ZLIB", or (iii) "GZIP".
+//
+// Returns the created operation.
+func DatasetToTFRecord(scope *Scope, input_dataset tf.Output, filename tf.Output, compression_type tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "DatasetToTFRecord",
+		Input: []tf.Input{
+			input_dataset, filename, compression_type,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// ResourceSparseApplyCenteredRMSPropAttr is an optional argument to ResourceSparseApplyCenteredRMSProp.
+type ResourceSparseApplyCenteredRMSPropAttr func(optionalAttr)
+
+// ResourceSparseApplyCenteredRMSPropUseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var, mg, ms, and mom tensors is
+// protected by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceSparseApplyCenteredRMSPropUseLocking(value bool) ResourceSparseApplyCenteredRMSPropAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// Update '*var' according to the centered RMSProp algorithm.
+//
+// The centered RMSProp algorithm uses an estimate of the centered second moment
+// (i.e., the variance) for normalization, as opposed to regular RMSProp, which
+// uses the (uncentered) second moment. This often helps with training, but is
+// slightly more expensive in terms of computation and memory.
+//
+// Note that in dense implementation of this algorithm, mg, ms, and mom will
+// update even if the grad is zero, but in this sparse implementation, mg, ms,
+// and mom will not update in iterations during which the grad is zero.
+//
+// mean_square = decay * mean_square + (1-decay) * gradient ** 2
+// mean_grad = decay * mean_grad + (1-decay) * gradient
+// Delta = learning_rate * gradient / sqrt(mean_square + epsilon - mean_grad ** 2)
+//
+// ms <- rho * ms_{t-1} + (1-rho) * grad * grad
+// mom <- momentum * mom_{t-1} + lr * grad / sqrt(ms + epsilon)
+// var <- var - mom
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	mg: Should be from a Variable().
+//	ms: Should be from a Variable().
+//	mom: Should be from a Variable().
+//	lr: Scaling factor. Must be a scalar.
+//	rho: Decay rate. Must be a scalar.
+//
+//	epsilon: Ridge term. Must be a scalar.
+//	grad: The gradient.
+//	indices: A vector of indices into the first dimension of var, ms and mom.
+//
+// Returns the created operation.
+func ResourceSparseApplyCenteredRMSProp(scope *Scope, var_ tf.Output, mg tf.Output, ms tf.Output, mom tf.Output, lr tf.Output, rho tf.Output, momentum tf.Output, epsilon tf.Output, grad tf.Output, indices tf.Output, optional ...ResourceSparseApplyCenteredRMSPropAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceSparseApplyCenteredRMSProp",
+		Input: []tf.Input{
+			var_, mg, ms, mom, lr, rho, momentum, epsilon, grad, indices,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Elementwise computes the bitwise XOR of `x` and `y`.
+//
+// The result will have those bits set, that are different in `x` and `y`. The
+// computation is performed on the underlying representations of `x` and `y`.
+//
+// For example:
+//
+// ```python
+// import tensorflow as tf
+// from tensorflow.python.ops import bitwise_ops
+// dtype_list = [tf.int8, tf.int16, tf.int32, tf.int64,
+//               tf.uint8, tf.uint16, tf.uint32, tf.uint64]
+//
+// for dtype in dtype_list:
+//   lhs = tf.constant([0, 5, 3, 14], dtype=dtype)
+//   rhs = tf.constant([5, 0, 7, 11], dtype=dtype)
+//   exp = tf.constant([5, 5, 4, 5],  dtype=tf.float32)
+//
+//   res = bitwise_ops.bitwise_xor(lhs, rhs)
+//   tf.assert_equal(tf.cast(res, tf.float32), exp) # TRUE
+// ```
+//
+func BitwiseXor(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "BitwiseXor",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Adds two `SparseTensor` objects to produce another `SparseTensor`.
+//
+// The input `SparseTensor` objects' indices are assumed ordered in standard
+// lexicographic order.  If this is not the case, before this step run
+// `SparseReorder` to restore index ordering.
+//
+// By default, if two values sum to zero at some index, the output `SparseTensor`
+// would still include that particular location in its index, storing a zero in the
+// corresponding value slot.  To override this, callers can specify `thresh`,
+// indicating that if the sum has a magnitude strictly smaller than `thresh`, its
+// corresponding value and index would then not be included.  In particular,
+// `thresh == 0` (default) means everything is kept and actual thresholding happens
+// only for a positive value.
+//
+// In the following shapes, `nnz` is the count after taking `thresh` into account.
+//
+// Arguments:
+//	a_indices: 2-D.  The `indices` of the first `SparseTensor`, size `[nnz, ndims]` Matrix.
+//	a_values: 1-D.  The `values` of the first `SparseTensor`, size `[nnz]` Vector.
+//	a_shape: 1-D.  The `shape` of the first `SparseTensor`, size `[ndims]` Vector.
+//	b_indices: 2-D.  The `indices` of the second `SparseTensor`, size `[nnz, ndims]` Matrix.
+//	b_values: 1-D.  The `values` of the second `SparseTensor`, size `[nnz]` Vector.
+//	b_shape: 1-D.  The `shape` of the second `SparseTensor`, size `[ndims]` Vector.
+//	thresh: 0-D.  The magnitude threshold that determines if an output value/index
+// pair takes space.
+func SparseAdd(scope *Scope, a_indices tf.Output, a_values tf.Output, a_shape tf.Output, b_indices tf.Output, b_values tf.Output, b_shape tf.Output, thresh tf.Output) (sum_indices tf.Output, sum_values tf.Output, sum_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseAdd",
+		Input: []tf.Input{
+			a_indices, a_values, a_shape, b_indices, b_values, b_shape, thresh,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// RandomShuffleAttr is an optional argument to RandomShuffle.
+type RandomShuffleAttr func(optionalAttr)
+
+// RandomShuffleSeed sets the optional seed attribute to value.
+//
+// value: If either `seed` or `seed2` are set to be non-zero, the random number
+// generator is seeded by the given seed.  Otherwise, it is seeded by a
+// random seed.
+// If not specified, defaults to 0
+func RandomShuffleSeed(value int64) RandomShuffleAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// RandomShuffleSeed2 sets the optional seed2 attribute to value.
+//
+// value: A second seed to avoid seed collision.
+// If not specified, defaults to 0
+func RandomShuffleSeed2(value int64) RandomShuffleAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Randomly shuffles a tensor along its first dimension.
+//
+//   The tensor is shuffled along dimension 0, such that each `value[j]` is mapped
+//   to one and only one `output[i]`. For example, a mapping that might occur for a
+//   3x2 tensor is:
+//
+// ```
+// [[1, 2],       [[5, 6],
+//  [3, 4],  ==>   [1, 2],
+//  [5, 6]]        [3, 4]]
+// ```
+//
+// Arguments:
+//	value: The tensor to be shuffled.
+//
+// Returns A tensor of same shape and type as `value`, shuffled along its first
+// dimension.
+func RandomShuffle(scope *Scope, value tf.Output, optional ...RandomShuffleAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RandomShuffle",
+		Input: []tf.Input{
+			value,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Selects elements from `x` or `y`, depending on `condition`.
+//
+// The `x`, and `y` tensors must all have the same shape, and the
+// output will also have that shape.
+//
+// The `condition` tensor must be a scalar if `x` and `y` are scalars.
+// If `x` and `y` are vectors or higher rank, then `condition` must be either a
+// scalar, a vector with size matching the first dimension of `x`, or must have
+// the same shape as `x`.
+//
+// The `condition` tensor acts as a mask that chooses, based on the value at each
+// element, whether the corresponding element / row in the output should be
+// taken from `x` (if true) or `y` (if false).
+//
+// If `condition` is a vector and `x` and `y` are higher rank matrices, then
+// it chooses which row (outer dimension) to copy from `x` and `y`.
+// If `condition` has the same shape as `x` and `y`, then it chooses which
+// element to copy from `x` and `y`.
+//
+// For example:
+//
+// ```python
+// # 'condition' tensor is [[True,  False]
+// #                        [False, True]]
+// # 't' is [[1, 2],
+// #         [3, 4]]
+// # 'e' is [[5, 6],
+// #         [7, 8]]
+// select(condition, t, e)  # => [[1, 6], [7, 4]]
+//
+//
+// # 'condition' tensor is [True, False]
+// # 't' is [[1, 2],
+// #         [3, 4]]
+// # 'e' is [[5, 6],
+// #         [7, 8]]
+// select(condition, t, e) ==> [[1, 2],
+//                              [7, 8]]
+//
+// ```
+//
+// Arguments:
+//
+//	x: = A `Tensor` which may have the same shape as `condition`.
+// If `condition` is rank 1, `x` may have higher rank,
+// but its first dimension must match the size of `condition`.
+//	y: = A `Tensor` with the same type and shape as `x`.
+//
+// Returns = A `Tensor` with the same type and shape as `x` and `y`.
+func Select(scope *Scope, condition tf.Output, x tf.Output, y tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Select",
+		Input: []tf.Input{
+			condition, x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// The gradient operator for the SparseAdd op.
+//
+// The SparseAdd op calculates A + B, where A, B, and the sum are all represented
+// as `SparseTensor` objects.  This op takes in the upstream gradient w.r.t.
+// non-empty values of the sum, and outputs the gradients w.r.t. the non-empty
+// values of A and B.
+//
+// Arguments:
+//	backprop_val_grad: 1-D with shape `[nnz(sum)]`.  The gradient with respect to
+// the non-empty values of the sum.
+//	a_indices: 2-D.  The `indices` of the `SparseTensor` A, size `[nnz(A), ndims]`.
+//	b_indices: 2-D.  The `indices` of the `SparseTensor` B, size `[nnz(B), ndims]`.
+//	sum_indices: 2-D.  The `indices` of the sum `SparseTensor`, size
+// `[nnz(sum), ndims]`.
+//
+// Returns:
+//	a_val_grad: 1-D with shape `[nnz(A)]`. The gradient with respect to the
+// non-empty values of A.
+//	b_val_grad: 1-D with shape `[nnz(B)]`. The gradient with respect to the
+// non-empty values of B.
+func SparseAddGrad(scope *Scope, backprop_val_grad tf.Output, a_indices tf.Output, b_indices tf.Output, sum_indices tf.Output) (a_val_grad tf.Output, b_val_grad tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseAddGrad",
+		Input: []tf.Input{
+			backprop_val_grad, a_indices, b_indices, sum_indices,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Removes keys and its associated values from a table.
+//
+// The tensor `keys` must of the same type as the keys of the table. Keys not
+// already in the table are silently ignored.
+//
+// Arguments:
+//	table_handle: Handle to the table.
+//	keys: Any shape.  Keys of the elements to remove.
+//
+// Returns the created operation.
+func LookupTableRemoveV2(scope *Scope, table_handle tf.Output, keys tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "LookupTableRemoveV2",
+		Input: []tf.Input{
+			table_handle, keys,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// NotEqualAttr is an optional argument to NotEqual.
+type NotEqualAttr func(optionalAttr)
+
+// NotEqualIncompatibleShapeError sets the optional incompatible_shape_error attribute to value.
+// If not specified, defaults to true
+func NotEqualIncompatibleShapeError(value bool) NotEqualAttr {
+	return func(m optionalAttr) {
+		m["incompatible_shape_error"] = value
+	}
+}
+
+// Returns the truth value of (x != y) element-wise.
+//
+// *NOTE*: `NotEqual` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func NotEqual(scope *Scope, x tf.Output, y tf.Output, optional ...NotEqualAttr) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "NotEqual",
+		Input: []tf.Input{
+			x, y,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the complementary error function of `x` element-wise.
+func Erfc(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Erfc",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RandomUniformIntAttr is an optional argument to RandomUniformInt.
+type RandomUniformIntAttr func(optionalAttr)
+
+// RandomUniformIntSeed sets the optional seed attribute to value.
+//
+// value: If either `seed` or `seed2` are set to be non-zero, the random number
+// generator is seeded by the given seed.  Otherwise, it is seeded by a
+// random seed.
+// If not specified, defaults to 0
+func RandomUniformIntSeed(value int64) RandomUniformIntAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// RandomUniformIntSeed2 sets the optional seed2 attribute to value.
+//
+// value: A second seed to avoid seed collision.
+// If not specified, defaults to 0
+func RandomUniformIntSeed2(value int64) RandomUniformIntAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Outputs random integers from a uniform distribution.
+//
+// The generated values are uniform integers in the range `[minval, maxval)`.
+// The lower bound `minval` is included in the range, while the upper bound
+// `maxval` is excluded.
+//
+// The random integers are slightly biased unless `maxval - minval` is an exact
+// power of two.  The bias is small for values of `maxval - minval` significantly
+// smaller than the range of the output (either `2^32` or `2^64`).
+//
+// Arguments:
+//	shape: The shape of the output tensor.
+//	minval: 0-D.  Inclusive lower bound on the generated integers.
+//	maxval: 0-D.  Exclusive upper bound on the generated integers.
+//
+// Returns A tensor of the specified shape filled with uniform random integers.
+func RandomUniformInt(scope *Scope, shape tf.Output, minval tf.Output, maxval tf.Output, optional ...RandomUniformIntAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RandomUniformInt",
+		Input: []tf.Input{
+			shape, minval, maxval,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// StridedSliceGradAttr is an optional argument to StridedSliceGrad.
+type StridedSliceGradAttr func(optionalAttr)
+
+// StridedSliceGradBeginMask sets the optional begin_mask attribute to value.
+// If not specified, defaults to 0
+func StridedSliceGradBeginMask(value int64) StridedSliceGradAttr {
+	return func(m optionalAttr) {
+		m["begin_mask"] = value
+	}
+}
+
+// StridedSliceGradEndMask sets the optional end_mask attribute to value.
+// If not specified, defaults to 0
+func StridedSliceGradEndMask(value int64) StridedSliceGradAttr {
+	return func(m optionalAttr) {
+		m["end_mask"] = value
+	}
+}
+
+// StridedSliceGradEllipsisMask sets the optional ellipsis_mask attribute to value.
+// If not specified, defaults to 0
+func StridedSliceGradEllipsisMask(value int64) StridedSliceGradAttr {
+	return func(m optionalAttr) {
+		m["ellipsis_mask"] = value
+	}
+}
+
+// StridedSliceGradNewAxisMask sets the optional new_axis_mask attribute to value.
+// If not specified, defaults to 0
+func StridedSliceGradNewAxisMask(value int64) StridedSliceGradAttr {
+	return func(m optionalAttr) {
+		m["new_axis_mask"] = value
+	}
+}
+
+// StridedSliceGradShrinkAxisMask sets the optional shrink_axis_mask attribute to value.
+// If not specified, defaults to 0
+func StridedSliceGradShrinkAxisMask(value int64) StridedSliceGradAttr {
+	return func(m optionalAttr) {
+		m["shrink_axis_mask"] = value
+	}
+}
+
+// Returns the gradient of `StridedSlice`.
+//
+// Since `StridedSlice` cuts out pieces of its `input` which is size
+// `shape`, its gradient will have the same shape (which is passed here
+// as `shape`). The gradient will be zero in any element that the slice
+// does not select.
+//
+// Arguments are the same as StridedSliceGrad with the exception that
+// `dy` is the input gradient to be propagated and `shape` is the
+// shape of `StridedSlice`'s `input`.
+func StridedSliceGrad(scope *Scope, shape tf.Output, begin tf.Output, end tf.Output, strides tf.Output, dy tf.Output, optional ...StridedSliceGradAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StridedSliceGrad",
+		Input: []tf.Input{
+			shape, begin, end, strides, dy,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResourceApplyFtrlAttr is an optional argument to ResourceApplyFtrl.
+type ResourceApplyFtrlAttr func(optionalAttr)
+
+// ResourceApplyFtrlUseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var and accum tensors will be protected
+// by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceApplyFtrlUseLocking(value bool) ResourceApplyFtrlAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// ResourceApplyFtrlMultiplyLinearByLr sets the optional multiply_linear_by_lr attribute to value.
+// If not specified, defaults to false
+func ResourceApplyFtrlMultiplyLinearByLr(value bool) ResourceApplyFtrlAttr {
+	return func(m optionalAttr) {
+		m["multiply_linear_by_lr"] = value
+	}
+}
+
+// Update '*var' according to the Ftrl-proximal scheme.
+//
+// accum_new = accum + grad * grad
+// linear += grad - (accum_new^(-lr_power) - accum^(-lr_power)) / lr * var
+// quadratic = 1.0 / (accum_new^(lr_power) * lr) + 2 * l2
+// var = (sign(linear) * l1 - linear) / quadratic if |linear| > l1 else 0.0
+// accum = accum_new
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	accum: Should be from a Variable().
+//	linear: Should be from a Variable().
+//	grad: The gradient.
+//	lr: Scaling factor. Must be a scalar.
+//	l1: L1 regularization. Must be a scalar.
+//	l2: L2 regularization. Must be a scalar.
+//	lr_power: Scaling factor. Must be a scalar.
+//
+// Returns the created operation.
+func ResourceApplyFtrl(scope *Scope, var_ tf.Output, accum tf.Output, linear tf.Output, grad tf.Output, lr tf.Output, l1 tf.Output, l2 tf.Output, lr_power tf.Output, optional ...ResourceApplyFtrlAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceApplyFtrl",
+		Input: []tf.Input{
+			var_, accum, linear, grad, lr, l1, l2, lr_power,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Creates a dataset that contains the unique elements of `input_dataset`.
+func ExperimentalUniqueDataset(scope *Scope, input_dataset tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalUniqueDataset",
+		Input: []tf.Input{
+			input_dataset,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// StringFormatAttr is an optional argument to StringFormat.
+type StringFormatAttr func(optionalAttr)
+
+// StringFormatTemplate sets the optional template attribute to value.
+//
+// value: A string, the template to format tensor summaries into.
+// If not specified, defaults to "%s"
+func StringFormatTemplate(value string) StringFormatAttr {
+	return func(m optionalAttr) {
+		m["template"] = value
+	}
+}
+
+// StringFormatPlaceholder sets the optional placeholder attribute to value.
+//
+// value: A string, at each placeholder in the template a subsequent tensor summary will be inserted.
+// If not specified, defaults to "%s"
+func StringFormatPlaceholder(value string) StringFormatAttr {
+	return func(m optionalAttr) {
+		m["placeholder"] = value
+	}
+}
+
+// StringFormatSummarize sets the optional summarize attribute to value.
+//
+// value: When formatting the tensor summaries print the first and last summarize entries of each tensor dimension.
+// If not specified, defaults to 3
+func StringFormatSummarize(value int64) StringFormatAttr {
+	return func(m optionalAttr) {
+		m["summarize"] = value
+	}
+}
+
+// Formats a string template using a list of tensors.
+//
+// Formats a string template using a list of tensors, pretty-printing tensor summaries.
+//
+// Arguments:
+//	inputs: The list of tensors to format into the placeholder string.
+//
+// Returns = The resulting string scalar.
+func StringFormat(scope *Scope, inputs []tf.Output, optional ...StringFormatAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StringFormat",
+		Input: []tf.Input{
+			tf.OutputList(inputs),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Converts a SparseTensor to a (possibly batched) CSRSparseMatrix.
+//
+// Arguments:
+//	indices: SparseTensor indices.
+//	values: SparseTensor values.
+//	dense_shape: SparseTensor dense shape.
+//
+// Returns A (possibly batched) CSRSparseMatrix.
+func SparseTensorToCSRSparseMatrix(scope *Scope, indices tf.Output, values tf.Output, dense_shape tf.Output) (sparse_matrix tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseTensorToCSRSparseMatrix",
+		Input: []tf.Input{
+			indices, values, dense_shape,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes fingerprints of the input strings.
+//
+// Arguments:
+//	input: vector of strings to compute fingerprints on.
+//
+// Returns a (N,2) shaped matrix where N is the number of elements in the input
+// vector. Each row contains the low and high parts of the fingerprint.
+func SdcaFprint(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SdcaFprint",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ParseSequenceExampleAttr is an optional argument to ParseSequenceExample.
+type ParseSequenceExampleAttr func(optionalAttr)
+
+// ParseSequenceExampleNcontextSparse sets the optional Ncontext_sparse attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func ParseSequenceExampleNcontextSparse(value int64) ParseSequenceExampleAttr {
+	return func(m optionalAttr) {
+		m["Ncontext_sparse"] = value
+	}
+}
+
+// ParseSequenceExampleNcontextDense sets the optional Ncontext_dense attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func ParseSequenceExampleNcontextDense(value int64) ParseSequenceExampleAttr {
+	return func(m optionalAttr) {
+		m["Ncontext_dense"] = value
+	}
+}
+
+// ParseSequenceExampleNfeatureListSparse sets the optional Nfeature_list_sparse attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func ParseSequenceExampleNfeatureListSparse(value int64) ParseSequenceExampleAttr {
+	return func(m optionalAttr) {
+		m["Nfeature_list_sparse"] = value
+	}
+}
+
+// ParseSequenceExampleNfeatureListDense sets the optional Nfeature_list_dense attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func ParseSequenceExampleNfeatureListDense(value int64) ParseSequenceExampleAttr {
+	return func(m optionalAttr) {
+		m["Nfeature_list_dense"] = value
+	}
+}
+
+// ParseSequenceExampleContextSparseTypes sets the optional context_sparse_types attribute to value.
+//
+// value: A list of Ncontext_sparse types; the data types of data in
+// each context Feature given in context_sparse_keys.
+// Currently the ParseSingleSequenceExample supports DT_FLOAT (FloatList),
+// DT_INT64 (Int64List), and DT_STRING (BytesList).
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseSequenceExampleContextSparseTypes(value []tf.DataType) ParseSequenceExampleAttr {
+	return func(m optionalAttr) {
+		m["context_sparse_types"] = value
+	}
+}
+
+// ParseSequenceExampleFeatureListDenseTypes sets the optional feature_list_dense_types attribute to value.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseSequenceExampleFeatureListDenseTypes(value []tf.DataType) ParseSequenceExampleAttr {
+	return func(m optionalAttr) {
+		m["feature_list_dense_types"] = value
+	}
+}
+
+// ParseSequenceExampleContextDenseShapes sets the optional context_dense_shapes attribute to value.
+//
+// value: A list of Ncontext_dense shapes; the shapes of data in
+// each context Feature given in context_dense_keys.
+// The number of elements in the Feature corresponding to context_dense_key[j]
+// must always equal context_dense_shapes[j].NumEntries().
+// The shape of context_dense_values[j] will match context_dense_shapes[j].
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseSequenceExampleContextDenseShapes(value []tf.Shape) ParseSequenceExampleAttr {
+	return func(m optionalAttr) {
+		m["context_dense_shapes"] = value
+	}
+}
+
+// ParseSequenceExampleFeatureListSparseTypes sets the optional feature_list_sparse_types attribute to value.
+//
+// value: A list of Nfeature_list_sparse types; the data types
+// of data in each FeatureList given in feature_list_sparse_keys.
+// Currently the ParseSingleSequenceExample supports DT_FLOAT (FloatList),
+// DT_INT64 (Int64List), and DT_STRING (BytesList).
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseSequenceExampleFeatureListSparseTypes(value []tf.DataType) ParseSequenceExampleAttr {
+	return func(m optionalAttr) {
+		m["feature_list_sparse_types"] = value
+	}
+}
+
+// ParseSequenceExampleFeatureListDenseShapes sets the optional feature_list_dense_shapes attribute to value.
+//
+// value: A list of Nfeature_list_dense shapes; the shapes of
+// data in each FeatureList given in feature_list_dense_keys.
+// The shape of each Feature in the FeatureList corresponding to
+// feature_list_dense_key[j] must always equal
+// feature_list_dense_shapes[j].NumEntries().
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func ParseSequenceExampleFeatureListDenseShapes(value []tf.Shape) ParseSequenceExampleAttr {
+	return func(m optionalAttr) {
+		m["feature_list_dense_shapes"] = value
+	}
+}
+
+// Transforms a vector of brain.SequenceExample protos (as strings) into typed tensors.
+//
+// Arguments:
+//	serialized: A vector containing binary serialized SequenceExample protos.
+//	debug_name: A vector containing the names of the serialized protos.
+// May contain, for example, table key (descriptive) name for the
+// corresponding serialized proto.  This is purely useful for debugging
+// purposes, and the presence of values here has no effect on the output.
+// May also be an empty vector if no name is available.
+//	context_dense_defaults: A list of Ncontext_dense Tensors (some may be empty).
+// context_dense_defaults[j] provides default values
+// when the SequenceExample's context map lacks context_dense_key[j].
+// If an empty Tensor is provided for context_dense_defaults[j],
+// then the Feature context_dense_keys[j] is required.
+// The input type is inferred from context_dense_defaults[j], even when it's
+// empty.  If context_dense_defaults[j] is not empty, its shape must match
+// context_dense_shapes[j].
+//	feature_list_dense_missing_assumed_empty: A vector listing the
+// FeatureList keys which may be missing from the SequenceExamples.  If the
+// associated FeatureList is missing, it is treated as empty.  By default,
+// any FeatureList not listed in this vector must exist in the SequenceExamples.
+//	context_sparse_keys: A list of Ncontext_sparse string Tensors (scalars).
+// The keys expected in the Examples' features associated with context_sparse
+// values.
+//	context_dense_keys: A list of Ncontext_dense string Tensors (scalars).
+// The keys expected in the SequenceExamples' context features associated with
+// dense values.
+//	feature_list_sparse_keys: A list of Nfeature_list_sparse string Tensors
+// (scalars).  The keys expected in the FeatureLists associated with sparse
+// values.
+//	feature_list_dense_keys: A list of Nfeature_list_dense string Tensors (scalars).
+// The keys expected in the SequenceExamples' feature_lists associated
+// with lists of dense values.
+func ParseSequenceExample(scope *Scope, serialized tf.Output, debug_name tf.Output, context_dense_defaults []tf.Output, feature_list_dense_missing_assumed_empty []string, context_sparse_keys []string, context_dense_keys []string, feature_list_sparse_keys []string, feature_list_dense_keys []string, optional ...ParseSequenceExampleAttr) (context_sparse_indices []tf.Output, context_sparse_values []tf.Output, context_sparse_shapes []tf.Output, context_dense_values []tf.Output, feature_list_sparse_indices []tf.Output, feature_list_sparse_values []tf.Output, feature_list_sparse_shapes []tf.Output, feature_list_dense_values []tf.Output, feature_list_dense_lengths []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"feature_list_dense_missing_assumed_empty": feature_list_dense_missing_assumed_empty, "context_sparse_keys": context_sparse_keys, "context_dense_keys": context_dense_keys, "feature_list_sparse_keys": feature_list_sparse_keys, "feature_list_dense_keys": feature_list_dense_keys}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ParseSequenceExample",
+		Input: []tf.Input{
+			serialized, debug_name, tf.OutputList(context_dense_defaults),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if context_sparse_indices, idx, err = makeOutputList(op, idx, "context_sparse_indices"); err != nil {
+		scope.UpdateErr("ParseSequenceExample", err)
+		return
+	}
+	if context_sparse_values, idx, err = makeOutputList(op, idx, "context_sparse_values"); err != nil {
+		scope.UpdateErr("ParseSequenceExample", err)
+		return
+	}
+	if context_sparse_shapes, idx, err = makeOutputList(op, idx, "context_sparse_shapes"); err != nil {
+		scope.UpdateErr("ParseSequenceExample", err)
+		return
+	}
+	if context_dense_values, idx, err = makeOutputList(op, idx, "context_dense_values"); err != nil {
+		scope.UpdateErr("ParseSequenceExample", err)
+		return
+	}
+	if feature_list_sparse_indices, idx, err = makeOutputList(op, idx, "feature_list_sparse_indices"); err != nil {
+		scope.UpdateErr("ParseSequenceExample", err)
+		return
+	}
+	if feature_list_sparse_values, idx, err = makeOutputList(op, idx, "feature_list_sparse_values"); err != nil {
+		scope.UpdateErr("ParseSequenceExample", err)
+		return
+	}
+	if feature_list_sparse_shapes, idx, err = makeOutputList(op, idx, "feature_list_sparse_shapes"); err != nil {
+		scope.UpdateErr("ParseSequenceExample", err)
+		return
+	}
+	if feature_list_dense_values, idx, err = makeOutputList(op, idx, "feature_list_dense_values"); err != nil {
+		scope.UpdateErr("ParseSequenceExample", err)
+		return
+	}
+	if feature_list_dense_lengths, idx, err = makeOutputList(op, idx, "feature_list_dense_lengths"); err != nil {
+		scope.UpdateErr("ParseSequenceExample", err)
+		return
+	}
+	return context_sparse_indices, context_sparse_values, context_sparse_shapes, context_dense_values, feature_list_sparse_indices, feature_list_sparse_values, feature_list_sparse_shapes, feature_list_dense_values, feature_list_dense_lengths
+}
+
+// Returns true if queue is closed.
+//
+// This operation returns true if the queue is closed and false if the queue
+// is open.
+//
+// Arguments:
+//	handle: The handle to a queue.
+func QueueIsClosedV2(scope *Scope, handle tf.Output) (is_closed tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "QueueIsClosedV2",
+		Input: []tf.Input{
+			handle,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Inverse 3D fast Fourier transform.
+//
+// Computes the inverse 3-dimensional discrete Fourier transform over the
+// inner-most 3 dimensions of `input`.
+//
+// Arguments:
+//	input: A complex tensor.
+//
+// Returns A complex tensor of the same shape as `input`. The inner-most 3
+//   dimensions of `input` are replaced with their inverse 3D Fourier transform.
+//
+// @compatibility(numpy)
+// Equivalent to np.fft.ifftn with 3 dimensions.
+// @end_compatibility
+func IFFT3D(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "IFFT3D",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// QueueDequeueUpToV2Attr is an optional argument to QueueDequeueUpToV2.
+type QueueDequeueUpToV2Attr func(optionalAttr)
+
+// QueueDequeueUpToV2TimeoutMs sets the optional timeout_ms attribute to value.
+//
+// value: If the queue has fewer than n elements, this operation
+// will block for up to timeout_ms milliseconds.
+// Note: This option is not supported yet.
+// If not specified, defaults to -1
+func QueueDequeueUpToV2TimeoutMs(value int64) QueueDequeueUpToV2Attr {
+	return func(m optionalAttr) {
+		m["timeout_ms"] = value
+	}
+}
+
+// Dequeues `n` tuples of one or more tensors from the given queue.
+//
+// This operation is not supported by all queues.  If a queue does not support
+// DequeueUpTo, then an Unimplemented error is returned.
+//
+// If the queue is closed and there are more than 0 but less than `n`
+// elements remaining, then instead of returning an OutOfRange error like
+// QueueDequeueMany, less than `n` elements are returned immediately.  If
+// the queue is closed and there are 0 elements left in the queue, then
+// an OutOfRange error is returned just like in QueueDequeueMany.
+// Otherwise the behavior is identical to QueueDequeueMany:
+//
+// This operation concatenates queue-element component tensors along the
+// 0th dimension to make a single component tensor.  All of the components
+// in the dequeued tuple will have size n in the 0th dimension.
+//
+// This operation has `k` outputs, where `k` is the number of components in
+// the tuples stored in the given queue, and output `i` is the ith
+// component of the dequeued tuple.
+//
+// Arguments:
+//	handle: The handle to a queue.
+//	n: The number of tuples to dequeue.
+//	component_types: The type of each component in a tuple.
+//
+// Returns One or more tensors that were dequeued as a tuple.
+func QueueDequeueUpToV2(scope *Scope, handle tf.Output, n tf.Output, component_types []tf.DataType, optional ...QueueDequeueUpToV2Attr) (components []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"component_types": component_types}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QueueDequeueUpToV2",
+		Input: []tf.Input{
+			handle, n,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if components, idx, err = makeOutputList(op, idx, "components"); err != nil {
+		scope.UpdateErr("QueueDequeueUpToV2", err)
+		return
+	}
+	return components
+}
+
+// Says whether the targets are in the top `K` predictions.
+//
+// This outputs a `batch_size` bool array, an entry `out[i]` is `true` if the
+// prediction for the target class is among the top `k` predictions among
+// all predictions for example `i`. Note that the behavior of `InTopK` differs
+// from the `TopK` op in its handling of ties; if multiple classes have the
+// same prediction value and straddle the top-`k` boundary, all of those
+// classes are considered to be in the top `k`.
+//
+// More formally, let
+//
+//   \\(predictions_i\\) be the predictions for all classes for example `i`,
+//   \\(targets_i\\) be the target class for example `i`,
+//   \\(out_i\\) be the output for example `i`,
+//
+// $$out_i = predictions_{i, targets_i} \in TopKIncludingTies(predictions_i)$$
+//
+// Arguments:
+//	predictions: A `batch_size` x `classes` tensor.
+//	targets: A `batch_size` vector of class ids.
+//	k: Number of top elements to look at for computing precision.
+//
+// Returns Computed Precision at `k` as a `bool Tensor`.
+func InTopK(scope *Scope, predictions tf.Output, targets tf.Output, k int64) (precision tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"k": k}
+	opspec := tf.OpSpec{
+		Type: "InTopK",
+		Input: []tf.Input{
+			predictions, targets,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns x - y element-wise.
+//
+// *NOTE*: `Subtract` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func Sub(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Sub",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// FusedResizeAndPadConv2DAttr is an optional argument to FusedResizeAndPadConv2D.
+type FusedResizeAndPadConv2DAttr func(optionalAttr)
+
+// FusedResizeAndPadConv2DResizeAlignCorners sets the optional resize_align_corners attribute to value.
+//
+// value: If true, the centers of the 4 corner pixels of the input and output tensors are
+// aligned, preserving the values at the corner pixels. Defaults to false.
+// If not specified, defaults to false
+func FusedResizeAndPadConv2DResizeAlignCorners(value bool) FusedResizeAndPadConv2DAttr {
+	return func(m optionalAttr) {
+		m["resize_align_corners"] = value
+	}
+}
+
+// Performs a resize and padding as a preprocess during a convolution.
+//
+// It's often possible to do spatial transformations more efficiently as part of
+// the packing stage of a convolution, so this op allows for an optimized
+// implementation where these stages are fused together. This prevents the need to
+// write out the intermediate results as whole tensors, reducing memory pressure,
+// and we can get some latency gains by merging the transformation calculations.
+// The data_format attribute for Conv2D isn't supported by this op, and defaults to
+// 'NHWC' order.
+// Internally this op uses a single per-graph scratch buffer, which means that it
+// will block if multiple versions are being run in parallel. This is because this
+// operator is primarily an optimization to minimize memory usage.
+//
+// Arguments:
+//	input: 4-D with shape `[batch, in_height, in_width, in_channels]`.
+//	size: A 1-D int32 Tensor of 2 elements: `new_height, new_width`.  The
+// new size for the images.
+//	paddings: A two-column matrix specifying the padding sizes. The number of
+// rows must be the same as the rank of `input`.
+//	filter: 4-D with shape
+// `[filter_height, filter_width, in_channels, out_channels]`.
+//
+//	strides: 1-D of length 4.  The stride of the sliding window for each dimension
+// of `input`. Must be in the same order as the dimension specified with format.
+//	padding: The type of padding algorithm to use.
+func FusedResizeAndPadConv2D(scope *Scope, input tf.Output, size tf.Output, paddings tf.Output, filter tf.Output, mode string, strides []int64, padding string, optional ...FusedResizeAndPadConv2DAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"mode": mode, "strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "FusedResizeAndPadConv2D",
+		Input: []tf.Input{
+			input, size, paddings, filter,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the product along segments of a tensor.
+//
+// Read
+// [the section on segmentation](https://tensorflow.org/api_docs/python/tf/math#Segmentation)
+// for an explanation of segments.
+//
+// This operator is similar to the unsorted segment sum operator found
+// [(here)](../../../api_docs/python/math_ops.md#UnsortedSegmentSum).
+// Instead of computing the sum over segments, it computes the product of all
+// entries belonging to a segment such that:
+//
+// \\(output_i = \prod_{j...} data[j...]\\) where the product is over tuples
+// `j...` such that `segment_ids[j...] == i`.
+//
+// For example:
+//
+// ``` python
+// c = tf.constant([[1,2,3,4], [5,6,7,8], [4,3,2,1]])
+// tf.unsorted_segment_prod(c, tf.constant([0, 1, 0]), num_segments=2)
+// # ==> [[ 4,  6, 6, 4],
+// #       [5,  6, 7, 8]]
+// ```
+//
+// If there is no entry for a given segment ID `i`, it outputs 1.
+//
+// If the given segment ID `i` is negative, then the corresponding value is
+// dropped, and will not be included in the result.
+//
+// Arguments:
+//
+//	segment_ids: A tensor whose shape is a prefix of `data.shape`.
+//
+//
+// Returns Has same shape as data, except for the first `segment_ids.rank`
+// dimensions, which are replaced with a single dimension which has size
+// `num_segments`.
+func UnsortedSegmentProd(scope *Scope, data tf.Output, segment_ids tf.Output, num_segments tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "UnsortedSegmentProd",
+		Input: []tf.Input{
+			data, segment_ids, num_segments,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// FakeQuantWithMinMaxVarsPerChannelAttr is an optional argument to FakeQuantWithMinMaxVarsPerChannel.
+type FakeQuantWithMinMaxVarsPerChannelAttr func(optionalAttr)
+
+// FakeQuantWithMinMaxVarsPerChannelNumBits sets the optional num_bits attribute to value.
+// If not specified, defaults to 8
+func FakeQuantWithMinMaxVarsPerChannelNumBits(value int64) FakeQuantWithMinMaxVarsPerChannelAttr {
+	return func(m optionalAttr) {
+		m["num_bits"] = value
+	}
+}
+
+// FakeQuantWithMinMaxVarsPerChannelNarrowRange sets the optional narrow_range attribute to value.
+// If not specified, defaults to false
+func FakeQuantWithMinMaxVarsPerChannelNarrowRange(value bool) FakeQuantWithMinMaxVarsPerChannelAttr {
+	return func(m optionalAttr) {
+		m["narrow_range"] = value
+	}
+}
+
+// Fake-quantize the 'inputs' tensor of type float via per-channel floats
+//
+// Fake-quantize the `inputs` tensor of type float per-channel and one of the
+// shapes: `[d]`, `[b, d]` `[b, h, w, d]` via per-channel floats `min` and `max`
+// of shape `[d]` to `outputs` tensor of same shape as `inputs`.
+//
+// Attributes
+//
+// *   `[min; max]` define the clamping range for the `inputs` data.
+// *   `inputs` values are quantized into the quantization range (
+// `[0; 2^num_bits - 1]` when `narrow_range` is false and `[1; 2^num_bits - 1]`
+// when it is true) and then de-quantized and output as floats in `[min; max]`
+// interval.
+// *   `num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
+//
+// Before quantization, `min` and `max` values are adjusted with the following
+// logic.
+// It is suggested to have `min <= 0 <= max`. If `0` is not in the range of values,
+// the behavior can be unexpected:
+//
+// *   If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
+// *   If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
+// *   If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
+// `min_adj = scale * round(min / scale)` and `max_adj = max + min_adj - min`.
+//
+// This operation has a gradient and thus allows for training `min` and `max`
+// values.
+func FakeQuantWithMinMaxVarsPerChannel(scope *Scope, inputs tf.Output, min tf.Output, max tf.Output, optional ...FakeQuantWithMinMaxVarsPerChannelAttr) (outputs tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "FakeQuantWithMinMaxVarsPerChannel",
+		Input: []tf.Input{
+			inputs, min, max,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Checks whether a resource handle-based variable has been initialized.
+//
+// Arguments:
+//	resource: the input resource handle.
+//
+// Returns a scalar boolean which is true if the variable has been
+// initialized.
+func VarIsInitializedOp(scope *Scope, resource tf.Output) (is_initialized tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "VarIsInitializedOp",
+		Input: []tf.Input{
+			resource,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// AngleAttr is an optional argument to Angle.
+type AngleAttr func(optionalAttr)
+
+// AngleTout sets the optional Tout attribute to value.
+// If not specified, defaults to DT_FLOAT
+func AngleTout(value tf.DataType) AngleAttr {
+	return func(m optionalAttr) {
+		m["Tout"] = value
+	}
+}
+
+// Returns the argument of a complex number.
+//
+// Given a tensor `input` of complex numbers, this operation returns a tensor of
+// type `float` that is the argument of each element in `input`. All elements in
+// `input` must be complex numbers of the form \\(a + bj\\), where *a*
+// is the real part and *b* is the imaginary part.
+//
+// The argument returned by this operation is of the form \\(atan2(b, a)\\).
+//
+// For example:
+//
+// ```
+// # tensor 'input' is [-2.25 + 4.75j, 3.25 + 5.75j]
+// tf.angle(input) ==> [2.0132, 1.056]
+// ```
+//
+// @compatibility(numpy)
+// Equivalent to np.angle.
+// @end_compatibility
+func Angle(scope *Scope, input tf.Output, optional ...AngleAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Angle",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes natural logarithm of x element-wise.
+//
+// I.e., \\(y = \log_e x\\).
+//
+// Example:
+//
+// ```python
+// x = tf.constant([0, 0.5, 1, 5])
+// tf.math.log(x) ==> [-inf, -0.6931472,  0. ,  1.609438]
+// ```
+func Log(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Log",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// IRFFT2DAttr is an optional argument to IRFFT2D.
+type IRFFT2DAttr func(optionalAttr)
+
+// IRFFT2DTreal sets the optional Treal attribute to value.
+// If not specified, defaults to DT_FLOAT
+func IRFFT2DTreal(value tf.DataType) IRFFT2DAttr {
+	return func(m optionalAttr) {
+		m["Treal"] = value
+	}
+}
+
+// Inverse 2D real-valued fast Fourier transform.
+//
+// Computes the inverse 2-dimensional discrete Fourier transform of a real-valued
+// signal over the inner-most 2 dimensions of `input`.
+//
+// The inner-most 2 dimensions of `input` are assumed to be the result of `RFFT2D`:
+// The inner-most dimension contains the `fft_length / 2 + 1` unique components of
+// the DFT of a real-valued signal. If `fft_length` is not provided, it is computed
+// from the size of the inner-most 2 dimensions of `input`. If the FFT length used
+// to compute `input` is odd, it should be provided since it cannot be inferred
+// properly.
+//
+// Along each axis `IRFFT2D` is computed on, if `fft_length` (or
+// `fft_length / 2 + 1` for the inner-most dimension) is smaller than the
+// corresponding dimension of `input`, the dimension is cropped. If it is larger,
+// the dimension is padded with zeros.
+//
+// Arguments:
+//	input: A complex tensor.
+//	fft_length: An int32 tensor of shape [2]. The FFT length for each dimension.
+//
+// Returns A float32 tensor of the same rank as `input`. The inner-most 2
+//   dimensions of `input` are replaced with the `fft_length` samples of their
+//   inverse 2D Fourier transform.
+//
+// @compatibility(numpy)
+// Equivalent to np.fft.irfft2
+// @end_compatibility
+func IRFFT2D(scope *Scope, input tf.Output, fft_length tf.Output, optional ...IRFFT2DAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "IRFFT2D",
+		Input: []tf.Input{
+			input, fft_length,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResizeBicubicAttr is an optional argument to ResizeBicubic.
+type ResizeBicubicAttr func(optionalAttr)
+
+// ResizeBicubicAlignCorners sets the optional align_corners attribute to value.
+//
+// value: If true, the centers of the 4 corner pixels of the input and output tensors are
+// aligned, preserving the values at the corner pixels. Defaults to false.
+// If not specified, defaults to false
+func ResizeBicubicAlignCorners(value bool) ResizeBicubicAttr {
+	return func(m optionalAttr) {
+		m["align_corners"] = value
+	}
+}
+
+// ResizeBicubicHalfPixelCenters sets the optional half_pixel_centers attribute to value.
+// If not specified, defaults to false
+func ResizeBicubicHalfPixelCenters(value bool) ResizeBicubicAttr {
+	return func(m optionalAttr) {
+		m["half_pixel_centers"] = value
+	}
+}
+
+// Resize `images` to `size` using bicubic interpolation.
+//
+// Input images can be of different types but output images are always float.
+//
+// Arguments:
+//	images: 4-D with shape `[batch, height, width, channels]`.
+//	size: = A 1-D int32 Tensor of 2 elements: `new_height, new_width`.  The
+// new size for the images.
+//
+// Returns 4-D with shape
+// `[batch, new_height, new_width, channels]`.
+func ResizeBicubic(scope *Scope, images tf.Output, size tf.Output, optional ...ResizeBicubicAttr) (resized_images tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResizeBicubic",
+		Input: []tf.Input{
+			images, size,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the mean along sparse segments of a tensor.
+//
+// Like `SparseSegmentMean`, but allows missing ids in `segment_ids`. If an id is
+// missing, the `output` tensor at that position will be zeroed.
+//
+// Read
+// [the section on segmentation](https://tensorflow.org/api_docs/python/tf/math#Segmentation)
+// for an explanation of segments.
+//
+// Arguments:
+//
+//	indices: A 1-D tensor. Has same rank as `segment_ids`.
+//	segment_ids: A 1-D tensor. Values should be sorted and can be repeated.
+//	num_segments: Should equal the number of distinct segment IDs.
+//
+// Returns Has same shape as data, except for dimension 0 which has size
+// `num_segments`.
+func SparseSegmentMeanWithNumSegments(scope *Scope, data tf.Output, indices tf.Output, segment_ids tf.Output, num_segments tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseSegmentMeanWithNumSegments",
+		Input: []tf.Input{
+			data, indices, segment_ids, num_segments,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RFFT2DAttr is an optional argument to RFFT2D.
+type RFFT2DAttr func(optionalAttr)
+
+// RFFT2DTcomplex sets the optional Tcomplex attribute to value.
+// If not specified, defaults to DT_COMPLEX64
+func RFFT2DTcomplex(value tf.DataType) RFFT2DAttr {
+	return func(m optionalAttr) {
+		m["Tcomplex"] = value
+	}
+}
+
+// 2D real-valued fast Fourier transform.
+//
+// Computes the 2-dimensional discrete Fourier transform of a real-valued signal
+// over the inner-most 2 dimensions of `input`.
+//
+// Since the DFT of a real signal is Hermitian-symmetric, `RFFT2D` only returns the
+// `fft_length / 2 + 1` unique components of the FFT for the inner-most dimension
+// of `output`: the zero-frequency term, followed by the `fft_length / 2`
+// positive-frequency terms.
+//
+// Along each axis `RFFT2D` is computed on, if `fft_length` is smaller than the
+// corresponding dimension of `input`, the dimension is cropped. If it is larger,
+// the dimension is padded with zeros.
+//
+// Arguments:
+//	input: A float32 tensor.
+//	fft_length: An int32 tensor of shape [2]. The FFT length for each dimension.
+//
+// Returns A complex64 tensor of the same rank as `input`. The inner-most 2
+//   dimensions of `input` are replaced with their 2D Fourier transform. The
+//   inner-most dimension contains `fft_length / 2 + 1` unique frequency
+//   components.
+//
+// @compatibility(numpy)
+// Equivalent to np.fft.rfft2
+// @end_compatibility
+func RFFT2D(scope *Scope, input tf.Output, fft_length tf.Output, optional ...RFFT2DAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RFFT2D",
+		Input: []tf.Input{
+			input, fft_length,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RFFTAttr is an optional argument to RFFT.
+type RFFTAttr func(optionalAttr)
+
+// RFFTTcomplex sets the optional Tcomplex attribute to value.
+// If not specified, defaults to DT_COMPLEX64
+func RFFTTcomplex(value tf.DataType) RFFTAttr {
+	return func(m optionalAttr) {
+		m["Tcomplex"] = value
+	}
+}
+
+// Real-valued fast Fourier transform.
+//
+// Computes the 1-dimensional discrete Fourier transform of a real-valued signal
+// over the inner-most dimension of `input`.
+//
+// Since the DFT of a real signal is Hermitian-symmetric, `RFFT` only returns the
+// `fft_length / 2 + 1` unique components of the FFT: the zero-frequency term,
+// followed by the `fft_length / 2` positive-frequency terms.
+//
+// Along the axis `RFFT` is computed on, if `fft_length` is smaller than the
+// corresponding dimension of `input`, the dimension is cropped. If it is larger,
+// the dimension is padded with zeros.
+//
+// Arguments:
+//	input: A float32 tensor.
+//	fft_length: An int32 tensor of shape [1]. The FFT length.
+//
+// Returns A complex64 tensor of the same rank as `input`. The inner-most
+//   dimension of `input` is replaced with the `fft_length / 2 + 1` unique
+//   frequency components of its 1D Fourier transform.
+//
+// @compatibility(numpy)
+// Equivalent to np.fft.rfft
+// @end_compatibility
+func RFFT(scope *Scope, input tf.Output, fft_length tf.Output, optional ...RFFTAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RFFT",
+		Input: []tf.Input{
+			input, fft_length,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// JPEG encode input image with provided compression quality.
+//
+// `image` is a 3-D uint8 Tensor of shape `[height, width, channels]`.
+// `quality` is an int32 jpeg compression quality value between 0 and 100.
+//
+//
+// Arguments:
+//	images: Images to adjust.  At least 3-D.
+//	quality: An int quality to encode to.
+//
+// Returns 0-D. JPEG-encoded image.
+func EncodeJpegVariableQuality(scope *Scope, images tf.Output, quality tf.Output) (contents tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "EncodeJpegVariableQuality",
+		Input: []tf.Input{
+			images, quality,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// 3D fast Fourier transform.
+//
+// Computes the 3-dimensional discrete Fourier transform over the inner-most 3
+// dimensions of `input`.
+//
+// Arguments:
+//	input: A complex tensor.
+//
+// Returns A complex tensor of the same shape as `input`. The inner-most 3
+//   dimensions of `input` are replaced with their 3D Fourier transform.
+//
+// @compatibility(numpy)
+// Equivalent to np.fft.fftn with 3 dimensions.
+// @end_compatibility
+func FFT3D(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "FFT3D",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that passes a sliding window over `input_dataset`.
+//
+// Arguments:
+//
+//	window_size: A scalar representing the number of elements in the
+// sliding window.
+//	window_shift: A scalar representing the steps moving the sliding window
+// forward in one iteration. It must be positive.
+//	window_stride: A scalar representing the stride of the input elements of the sliding window.
+// It must be positive.
+//
+//
+func SlidingWindowDataset(scope *Scope, input_dataset tf.Output, window_size tf.Output, window_shift tf.Output, window_stride tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "SlidingWindowDataset",
+		Input: []tf.Input{
+			input_dataset, window_size, window_shift, window_stride,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Locks a mutex resource.  The output is the lock.  So long as the lock tensor
+//
+// is alive, any other request to use `MutexLock` with this mutex will wait.
+//
+// This is particularly useful for creating a critical section when used in
+// conjunction with `MutexLockIdentity`:
+//
+// ```python
+//
+// mutex = mutex_v2(
+//   shared_name=handle_name, container=container, name=name)
+//
+// def execute_in_critical_section(fn, *args, **kwargs):
+//   lock = gen_resource_variable_ops.mutex_lock(mutex)
+//
+//   with ops.control_dependencies([lock]):
+//     r = fn(*args, **kwargs)
+//
+//   with ops.control_dependencies(nest.flatten(r)):
+//     with ops.colocate_with(mutex):
+//       ensure_lock_exists = mutex_lock_identity(lock)
+//
+//     # Make sure that if any element of r is accessed, all of
+//     # them are executed together.
+//     r = nest.map_structure(tf.identity, r)
+//
+//   with ops.control_dependencies([ensure_lock_exists]):
+//     return nest.map_structure(tf.identity, r)
+// ```
+//
+// While `fn` is running in the critical section, no other functions which wish to
+// use this critical section may run.
+//
+// Often the use case is that two executions of the same graph, in parallel,
+// wish to run `fn`; and we wish to ensure that only one of them executes
+// at a time.  This is especially important if `fn` modifies one or more
+// variables at a time.
+//
+// It is also useful if two separate functions must share a resource, but we
+// wish to ensure the usage is exclusive.
+//
+// Arguments:
+//	mutex: The mutex resource to lock.
+//
+// Returns A tensor that keeps a shared pointer to a lock on the mutex;
+// when the Tensor is destroyed, the use count on the shared pointer is decreased
+// by 1.  When it reaches 0, the lock is released.
+func MutexLock(scope *Scope, mutex tf.Output) (mutex_lock tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "MutexLock",
+		Input: []tf.Input{
+			mutex,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// MaxPoolGradWithArgmaxAttr is an optional argument to MaxPoolGradWithArgmax.
+type MaxPoolGradWithArgmaxAttr func(optionalAttr)
+
+// MaxPoolGradWithArgmaxIncludeBatchInIndex sets the optional include_batch_in_index attribute to value.
+//
+// value: Whether to include batch dimension in flattened index of `argmax`.
+// If not specified, defaults to false
+func MaxPoolGradWithArgmaxIncludeBatchInIndex(value bool) MaxPoolGradWithArgmaxAttr {
+	return func(m optionalAttr) {
+		m["include_batch_in_index"] = value
+	}
+}
+
+// Computes gradients of the maxpooling function.
+//
+// Arguments:
+//	input: The original input.
+//	grad: 4-D with shape `[batch, height, width, channels]`.  Gradients w.r.t. the
+// output of `max_pool`.
+//	argmax: The indices of the maximum values chosen for each output of `max_pool`.
+//	ksize: The size of the window for each dimension of the input tensor.
+//	strides: The stride of the sliding window for each dimension of the
+// input tensor.
+//	padding: The type of padding algorithm to use.
+//
+// Returns Gradients w.r.t. the input of `max_pool`.
+func MaxPoolGradWithArgmax(scope *Scope, input tf.Output, grad tf.Output, argmax tf.Output, ksize []int64, strides []int64, padding string, optional ...MaxPoolGradWithArgmaxAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"ksize": ksize, "strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MaxPoolGradWithArgmax",
+		Input: []tf.Input{
+			input, grad, argmax,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// 2D fast Fourier transform.
+//
+// Computes the 2-dimensional discrete Fourier transform over the inner-most
+// 2 dimensions of `input`.
+//
+// Arguments:
+//	input: A complex tensor.
+//
+// Returns A complex tensor of the same shape as `input`. The inner-most 2
+//   dimensions of `input` are replaced with their 2D Fourier transform.
+//
+// @compatibility(numpy)
+// Equivalent to np.fft.fft2
+// @end_compatibility
+func FFT2D(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "FFT2D",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SdcaOptimizerAttr is an optional argument to SdcaOptimizer.
+type SdcaOptimizerAttr func(optionalAttr)
+
+// SdcaOptimizerAdaptative sets the optional adaptative attribute to value.
+//
+// value: Whether to use Adaptive SDCA for the inner loop.
+// If not specified, defaults to true
+func SdcaOptimizerAdaptative(value bool) SdcaOptimizerAttr {
+	return func(m optionalAttr) {
+		m["adaptative"] = value
+	}
+}
+
+// Distributed version of Stochastic Dual Coordinate Ascent (SDCA) optimizer for
+//
+// linear models with L1 + L2 regularization. As global optimization objective is
+// strongly-convex, the optimizer optimizes the dual objective at each step. The
+// optimizer applies each update one example at a time. Examples are sampled
+// uniformly, and the optimizer is learning rate free and enjoys linear convergence
+// rate.
+//
+// [Proximal Stochastic Dual Coordinate Ascent](http://arxiv.org/pdf/1211.2717v1.pdf).<br>
+// Shai Shalev-Shwartz, Tong Zhang. 2012
+//
+// $$Loss Objective = \sum f_{i} (wx_{i}) + (l2 / 2) * |w|^2 + l1 * |w|$$
+//
+// [Adding vs. Averaging in Distributed Primal-Dual Optimization](http://arxiv.org/abs/1502.03508).<br>
+// Chenxin Ma, Virginia Smith, Martin Jaggi, Michael I. Jordan,
+// Peter Richtarik, Martin Takac. 2015
+//
+// [Stochastic Dual Coordinate Ascent with Adaptive Probabilities](https://arxiv.org/abs/1502.08053).<br>
+// Dominik Csiba, Zheng Qu, Peter Richtarik. 2015
+//
+// Arguments:
+//	sparse_example_indices: a list of vectors which contain example indices.
+//	sparse_feature_indices: a list of vectors which contain feature indices.
+//	sparse_feature_values: a list of vectors which contains feature value
+// associated with each feature group.
+//	dense_features: a list of matrices which contains the dense feature values.
+//	example_weights: a vector which contains the weight associated with each
+// example.
+//	example_labels: a vector which contains the label/target associated with each
+// example.
+//	sparse_indices: a list of vectors where each value is the indices which has
+// corresponding weights in sparse_weights. This field maybe omitted for the
+// dense approach.
+//	sparse_weights: a list of vectors where each value is the weight associated with
+// a sparse feature group.
+//	dense_weights: a list of vectors where the values are the weights associated
+// with a dense feature group.
+//	example_state_data: a list of vectors containing the example state data.
+//	loss_type: Type of the primal loss. Currently SdcaSolver supports logistic,
+// squared and hinge losses.
+//	l1: Symmetric l1 regularization strength.
+//	l2: Symmetric l2 regularization strength.
+//	num_loss_partitions: Number of partitions of the global loss function.
+//	num_inner_iterations: Number of iterations per mini-batch.
+//
+// Returns:
+//	out_example_state_data: a list of vectors containing the updated example state
+// data.
+//	out_delta_sparse_weights: a list of vectors where each value is the delta
+// weights associated with a sparse feature group.
+//	out_delta_dense_weights: a list of vectors where the values are the delta
+// weights associated with a dense feature group.
+func SdcaOptimizer(scope *Scope, sparse_example_indices []tf.Output, sparse_feature_indices []tf.Output, sparse_feature_values []tf.Output, dense_features []tf.Output, example_weights tf.Output, example_labels tf.Output, sparse_indices []tf.Output, sparse_weights []tf.Output, dense_weights []tf.Output, example_state_data tf.Output, loss_type string, l1 float32, l2 float32, num_loss_partitions int64, num_inner_iterations int64, optional ...SdcaOptimizerAttr) (out_example_state_data tf.Output, out_delta_sparse_weights []tf.Output, out_delta_dense_weights []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"loss_type": loss_type, "l1": l1, "l2": l2, "num_loss_partitions": num_loss_partitions, "num_inner_iterations": num_inner_iterations}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SdcaOptimizer",
+		Input: []tf.Input{
+			tf.OutputList(sparse_example_indices), tf.OutputList(sparse_feature_indices), tf.OutputList(sparse_feature_values), tf.OutputList(dense_features), example_weights, example_labels, tf.OutputList(sparse_indices), tf.OutputList(sparse_weights), tf.OutputList(dense_weights), example_state_data,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	out_example_state_data = op.Output(idx)
+	if out_delta_sparse_weights, idx, err = makeOutputList(op, idx, "out_delta_sparse_weights"); err != nil {
+		scope.UpdateErr("SdcaOptimizer", err)
+		return
+	}
+	if out_delta_dense_weights, idx, err = makeOutputList(op, idx, "out_delta_dense_weights"); err != nil {
+		scope.UpdateErr("SdcaOptimizer", err)
+		return
+	}
+	return out_example_state_data, out_delta_sparse_weights, out_delta_dense_weights
+}
+
+// Inverse fast Fourier transform.
+//
+// Computes the inverse 1-dimensional discrete Fourier transform over the
+// inner-most dimension of `input`.
+//
+// Arguments:
+//	input: A complex tensor.
+//
+// Returns A complex tensor of the same shape as `input`. The inner-most
+//   dimension of `input` is replaced with its inverse 1D Fourier transform.
+//
+// @compatibility(numpy)
+// Equivalent to np.fft.ifft
+// @end_compatibility
+func IFFT(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "IFFT",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// CollectiveGatherAttr is an optional argument to CollectiveGather.
+type CollectiveGatherAttr func(optionalAttr)
+
+// CollectiveGatherCommunicationHint sets the optional communication_hint attribute to value.
+// If not specified, defaults to "auto"
+func CollectiveGatherCommunicationHint(value string) CollectiveGatherAttr {
+	return func(m optionalAttr) {
+		m["communication_hint"] = value
+	}
+}
+
+// CollectiveGatherTimeoutSeconds sets the optional timeout_seconds attribute to value.
+// If not specified, defaults to 0
+func CollectiveGatherTimeoutSeconds(value float32) CollectiveGatherAttr {
+	return func(m optionalAttr) {
+		m["timeout_seconds"] = value
+	}
+}
+
+// Mutually accumulates multiple tensors of identical type and shape.
+func CollectiveGather(scope *Scope, input tf.Output, group_size int64, group_key int64, instance_key int64, shape tf.Shape, optional ...CollectiveGatherAttr) (data tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"group_size": group_size, "group_key": group_key, "instance_key": instance_key, "shape": shape}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CollectiveGather",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// L2 Loss.
+//
+// Computes half the L2 norm of a tensor without the `sqrt`:
+//
+//     output = sum(t ** 2) / 2
+//
+// Arguments:
+//	t: Typically 2-D, but may have any dimensions.
+//
+// Returns 0-D.
+func L2Loss(scope *Scope, t tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "L2Loss",
+		Input: []tf.Input{
+			t,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// An op that receives embedding activations on the TPU.
+//
+// The TPU system performs the embedding lookups and aggregations specified by
+// the arguments to TPUEmbeddingEnqueue(Integer/Sparse/SparseTensor)Batch. The
+// results of these aggregations are visible to the Tensorflow Graph as the
+// outputs of a RecvTPUEmbeddingActivations op. This op returns a list containing
+// one Tensor of activations per table specified in the model. There can be at
+// most one RecvTPUEmbeddingActivations op in the TPU graph.
+//
+// Arguments:
+//	num_outputs: The number of output activation tensors, equal to the number of
+// embedding tables in the model.
+//	config: Serialized TPUEmbeddingConfiguration proto.
+//
+// Returns A TensorList of embedding activations containing one Tensor per
+// embedding table in the model.
+func RecvTPUEmbeddingActivations(scope *Scope, num_outputs int64, config string) (outputs []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_outputs": num_outputs, "config": config}
+	opspec := tf.OpSpec{
+		Type: "RecvTPUEmbeddingActivations",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if outputs, idx, err = makeOutputList(op, idx, "outputs"); err != nil {
+		scope.UpdateErr("RecvTPUEmbeddingActivations", err)
+		return
+	}
+	return outputs
+}
+
+// Computes the derivative of a Gamma random sample w.r.t. `alpha`.
+func RandomGammaGrad(scope *Scope, alpha tf.Output, sample tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "RandomGammaGrad",
+		Input: []tf.Input{
+			alpha, sample,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// LRNGradAttr is an optional argument to LRNGrad.
+type LRNGradAttr func(optionalAttr)
+
+// LRNGradDepthRadius sets the optional depth_radius attribute to value.
+//
+// value: A depth radius.
+// If not specified, defaults to 5
+func LRNGradDepthRadius(value int64) LRNGradAttr {
+	return func(m optionalAttr) {
+		m["depth_radius"] = value
+	}
+}
+
+// LRNGradBias sets the optional bias attribute to value.
+//
+// value: An offset (usually > 0 to avoid dividing by 0).
+// If not specified, defaults to 1
+func LRNGradBias(value float32) LRNGradAttr {
+	return func(m optionalAttr) {
+		m["bias"] = value
+	}
+}
+
+// LRNGradAlpha sets the optional alpha attribute to value.
+//
+// value: A scale factor, usually positive.
+// If not specified, defaults to 1
+func LRNGradAlpha(value float32) LRNGradAttr {
+	return func(m optionalAttr) {
+		m["alpha"] = value
+	}
+}
+
+// LRNGradBeta sets the optional beta attribute to value.
+//
+// value: An exponent.
+// If not specified, defaults to 0.5
+func LRNGradBeta(value float32) LRNGradAttr {
+	return func(m optionalAttr) {
+		m["beta"] = value
+	}
+}
+
+// Gradients for Local Response Normalization.
+//
+// Arguments:
+//	input_grads: 4-D with shape `[batch, height, width, channels]`.
+//	input_image: 4-D with shape `[batch, height, width, channels]`.
+//	output_image: 4-D with shape `[batch, height, width, channels]`.
+//
+// Returns The gradients for LRN.
+func LRNGrad(scope *Scope, input_grads tf.Output, input_image tf.Output, output_image tf.Output, optional ...LRNGradAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LRNGrad",
+		Input: []tf.Input{
+			input_grads, input_image, output_image,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// PrelinearizeAttr is an optional argument to Prelinearize.
+type PrelinearizeAttr func(optionalAttr)
+
+// PrelinearizeShape sets the optional shape attribute to value.
+//
+// value: The shape of the tensor.
+// If not specified, defaults to <>
+func PrelinearizeShape(value tf.Shape) PrelinearizeAttr {
+	return func(m optionalAttr) {
+		m["shape"] = value
+	}
+}
+
+// PrelinearizeLayout sets the optional layout attribute to value.
+//
+// value: A vector holding the requested layout in minor-to-major sequence. If a layout
+// attribute is passed but its values are all -1 the layout will be computed by
+// the infeed operation.
+// If not specified, defaults to <>
+func PrelinearizeLayout(value []int64) PrelinearizeAttr {
+	return func(m optionalAttr) {
+		m["layout"] = value
+	}
+}
+
+// An op which linearizes one Tensor value to an opaque variant tensor.
+//
+// Arguments:
+//	input: A tensor that will be linearized.
+func Prelinearize(scope *Scope, input tf.Output, optional ...PrelinearizeAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Prelinearize",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the sparse Cholesky decomposition of `input`.
+//
+// Computes the Sparse Cholesky decomposition of a sparse matrix, with the given
+// fill-in reducing permutation.
+//
+// The input sparse matrix and the fill-in reducing permutation `permutation` must
+// have compatible shapes. If the sparse matrix has rank 3; with the batch
+// dimension `B`, then the `permutation` must be of rank 2; with the same batch
+// dimension `B`. There is no support for broadcasting.
+//
+// Furthermore, each component vector of `permutation` must be of length `N`,
+// containing each of the integers {0, 1, ..., N - 1} exactly once, where `N` is
+// the number of rows of each component of the sparse matrix.
+//
+// Each component of the input sparse matrix must represent a symmetric positive
+// definite (SPD) matrix; although only the lower triangular part of the matrix is
+// read. If any individual component is not SPD, then an InvalidArgument error is
+// thrown.
+//
+// The returned sparse matrix has the same dense shape as the input sparse matrix.
+// For each component `A` of the input sparse matrix, the corresponding output
+// sparse matrix represents `L`, the lower triangular Cholesky factor satisfying
+// the following identity:
+//
+// ```
+//   A = L * Lt
+// ```
+//
+// where Lt denotes the transpose of L (or its conjugate transpose, if `type` is
+// `complex64` or `complex128`).
+//
+// The `type` parameter denotes the type of the matrix elements. The supported
+// types are: `float32`, `float64`, `complex64` and `complex128`.
+//
+// Usage example:
+//
+// ```python
+//     from tensorflow.python.ops.linalg.sparse import sparse_csr_matrix_ops
+//
+//     a_indices = np.array([[0, 0], [1, 1], [2, 1], [2, 2], [3, 3]])
+//     a_values = np.array([1.0, 2.0, 1.0, 3.0, 4.0], np.float32)
+//     a_dense_shape = [4, 4]
+//
+//     with tf.Session() as sess:
+//       # Define (COO format) SparseTensor over Numpy array.
+//       a_st = tf.sparse.SparseTensor(a_indices, a_values, a_dense_shape)
+//
+//       # Convert SparseTensors to CSR SparseMatrix.
+//       a_sm = sparse_csr_matrix_ops.sparse_tensor_to_csr_sparse_matrix(
+//           a_st.indices, a_st.values, a_st.dense_shape)
+//
+//       # Obtain the Sparse Cholesky factor using AMD Ordering for reducing zero
+//       # fill-in (number of structural non-zeros in the sparse Cholesky factor).
+//       ordering_amd = sparse_csr_matrix_ops.sparse_matrix_ordering_amd(sparse_matrix)
+//       cholesky_sparse_matrices = (
+//           sparse_csr_matrix_ops.sparse_matrix_sparse_cholesky(
+//               sparse_matrix, ordering_amd, type=tf.float32))
+//
+//       # Convert the CSRSparseMatrix Cholesky factor to a dense Tensor
+//       dense_cholesky = sparse_csr_matrix_ops.csr_sparse_matrix_to_dense(
+//           cholesky_sparse_matrices, tf.float32)
+//
+//       # Evaluate the dense Tensor value.
+//       dense_cholesky_value = sess.run(dense_cholesky)
+// ```
+//
+// `dense_cholesky_value` stores the dense Cholesky factor:
+//
+// ```
+//     [[  1.  0.    0.    0.]
+//      [  0.  1.41  0.    0.]
+//      [  0.  0.70  1.58  0.]
+//      [  0.  0.    0.    2.]]
+// ```
+//
+//
+// input: A `CSRSparseMatrix`.
+// permutation: A `Tensor`.
+// type: The type of `input`.
+//
+// Arguments:
+//	input: A `CSRSparseMatrix`.
+//	permutation: A fill-in reducing permutation matrix.
+//
+//
+// Returns The sparse Cholesky decompsition of `input`.
+func SparseMatrixSparseCholesky(scope *Scope, input tf.Output, permutation tf.Output, type_ tf.DataType) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"type": type_}
+	opspec := tf.OpSpec{
+		Type: "SparseMatrixSparseCholesky",
+		Input: []tf.Input{
+			input, permutation,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// StatefulTruncatedNormalAttr is an optional argument to StatefulTruncatedNormal.
+type StatefulTruncatedNormalAttr func(optionalAttr)
+
+// StatefulTruncatedNormalDtype sets the optional dtype attribute to value.
+//
+// value: The type of the output.
+// If not specified, defaults to DT_FLOAT
+func StatefulTruncatedNormalDtype(value tf.DataType) StatefulTruncatedNormalAttr {
+	return func(m optionalAttr) {
+		m["dtype"] = value
+	}
+}
+
+// Outputs random values from a truncated normal distribution.
+//
+// The generated values follow a normal distribution with mean 0 and standard
+// deviation 1, except that values whose magnitude is more than 2 standard
+// deviations from the mean are dropped and re-picked.
+//
+// Arguments:
+//	resource: The handle of the resource variable that stores the state of the RNG.
+//	algorithm: The RNG algorithm.
+//	shape: The shape of the output tensor.
+//
+// Returns Random values with specified shape.
+func StatefulTruncatedNormal(scope *Scope, resource tf.Output, algorithm tf.Output, shape tf.Output, optional ...StatefulTruncatedNormalAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StatefulTruncatedNormal",
+		Input: []tf.Input{
+			resource, algorithm, shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns the rank of a tensor.
+//
+// This operation returns an integer representing the rank of `input`.
+//
+// For example:
+//
+// ```
+// # 't' is [[[1, 1, 1], [2, 2, 2]], [[3, 3, 3], [4, 4, 4]]]
+// # shape of tensor 't' is [2, 2, 3]
+// rank(t) ==> 3
+// ```
+//
+// **Note**: The rank of a tensor is not the same as the rank of a matrix. The rank
+// of a tensor is the number of indices required to uniquely select each element
+// of the tensor. Rank is also known as "order", "degree", or "ndims."
+func Rank(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Rank",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// EncodeJpegAttr is an optional argument to EncodeJpeg.
+type EncodeJpegAttr func(optionalAttr)
+
+// EncodeJpegFormat sets the optional format attribute to value.
+//
+// value: Per pixel image format.
+// If not specified, defaults to ""
+func EncodeJpegFormat(value string) EncodeJpegAttr {
+	return func(m optionalAttr) {
+		m["format"] = value
+	}
+}
+
+// EncodeJpegQuality sets the optional quality attribute to value.
+//
+// value: Quality of the compression from 0 to 100 (higher is better and slower).
+// If not specified, defaults to 95
+func EncodeJpegQuality(value int64) EncodeJpegAttr {
+	return func(m optionalAttr) {
+		m["quality"] = value
+	}
+}
+
+// EncodeJpegProgressive sets the optional progressive attribute to value.
+//
+// value: If True, create a JPEG that loads progressively (coarse to fine).
+// If not specified, defaults to false
+func EncodeJpegProgressive(value bool) EncodeJpegAttr {
+	return func(m optionalAttr) {
+		m["progressive"] = value
+	}
+}
+
+// EncodeJpegOptimizeSize sets the optional optimize_size attribute to value.
+//
+// value: If True, spend CPU/RAM to reduce size with no quality change.
+// If not specified, defaults to false
+func EncodeJpegOptimizeSize(value bool) EncodeJpegAttr {
+	return func(m optionalAttr) {
+		m["optimize_size"] = value
+	}
+}
+
+// EncodeJpegChromaDownsampling sets the optional chroma_downsampling attribute to value.
+//
+// value: See http://en.wikipedia.org/wiki/Chroma_subsampling.
+// If not specified, defaults to true
+func EncodeJpegChromaDownsampling(value bool) EncodeJpegAttr {
+	return func(m optionalAttr) {
+		m["chroma_downsampling"] = value
+	}
+}
+
+// EncodeJpegDensityUnit sets the optional density_unit attribute to value.
+//
+// value: Unit used to specify `x_density` and `y_density`:
+// pixels per inch (`'in'`) or centimeter (`'cm'`).
+// If not specified, defaults to "in"
+func EncodeJpegDensityUnit(value string) EncodeJpegAttr {
+	return func(m optionalAttr) {
+		m["density_unit"] = value
+	}
+}
+
+// EncodeJpegXDensity sets the optional x_density attribute to value.
+//
+// value: Horizontal pixels per density unit.
+// If not specified, defaults to 300
+func EncodeJpegXDensity(value int64) EncodeJpegAttr {
+	return func(m optionalAttr) {
+		m["x_density"] = value
+	}
+}
+
+// EncodeJpegYDensity sets the optional y_density attribute to value.
+//
+// value: Vertical pixels per density unit.
+// If not specified, defaults to 300
+func EncodeJpegYDensity(value int64) EncodeJpegAttr {
+	return func(m optionalAttr) {
+		m["y_density"] = value
+	}
+}
+
+// EncodeJpegXmpMetadata sets the optional xmp_metadata attribute to value.
+//
+// value: If not empty, embed this XMP metadata in the image header.
+// If not specified, defaults to ""
+func EncodeJpegXmpMetadata(value string) EncodeJpegAttr {
+	return func(m optionalAttr) {
+		m["xmp_metadata"] = value
+	}
+}
+
+// JPEG-encode an image.
+//
+// `image` is a 3-D uint8 Tensor of shape `[height, width, channels]`.
+//
+// The attr `format` can be used to override the color format of the encoded
+// output.  Values can be:
+//
+// *   `''`: Use a default format based on the number of channels in the image.
+// *   `grayscale`: Output a grayscale JPEG image.  The `channels` dimension
+//     of `image` must be 1.
+// *   `rgb`: Output an RGB JPEG image. The `channels` dimension
+//     of `image` must be 3.
+//
+// If `format` is not specified or is the empty string, a default format is picked
+// in function of the number of channels in `image`:
+//
+// *   1: Output a grayscale image.
+// *   3: Output an RGB image.
+//
+// Arguments:
+//	image: 3-D with shape `[height, width, channels]`.
+//
+// Returns 0-D. JPEG-encoded image.
+func EncodeJpeg(scope *Scope, image tf.Output, optional ...EncodeJpegAttr) (contents tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "EncodeJpeg",
+		Input: []tf.Input{
+			image,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Checks whether a quantile stream has been initialized.
+//
+// An Op that checks if quantile stream resource is initialized.
+//
+// Arguments:
+//	quantile_stream_resource_handle: resource; The reference to quantile stream resource handle.
+//
+// Returns bool; True if the resource is initialized, False otherwise.
+func IsBoostedTreesQuantileStreamResourceInitialized(scope *Scope, quantile_stream_resource_handle tf.Output) (is_initialized tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "IsBoostedTreesQuantileStreamResourceInitialized",
+		Input: []tf.Input{
+			quantile_stream_resource_handle,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Fast Fourier transform.
+//
+// Computes the 1-dimensional discrete Fourier transform over the inner-most
+// dimension of `input`.
+//
+// Arguments:
+//	input: A complex tensor.
+//
+// Returns A complex tensor of the same shape as `input`. The inner-most
+//   dimension of `input` is replaced with its 1D Fourier transform.
+//
+// @compatibility(numpy)
+// Equivalent to np.fft.fft
+// @end_compatibility
+func FFT(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "FFT",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// UniqueV2Attr is an optional argument to UniqueV2.
+type UniqueV2Attr func(optionalAttr)
+
+// UniqueV2OutIdx sets the optional out_idx attribute to value.
+// If not specified, defaults to DT_INT32
+func UniqueV2OutIdx(value tf.DataType) UniqueV2Attr {
+	return func(m optionalAttr) {
+		m["out_idx"] = value
+	}
+}
+
+// Finds unique elements along an axis of a tensor.
+//
+// This operation either returns a tensor `y` containing unique elements
+// along the `axis` of a tensor. The returned unique elements is sorted
+// in the same order as they occur along `axis` in `x`.
+// This operation also returns a tensor `idx` that is the same size as
+// the number of the elements in `x` along the `axis` dimension. It
+// contains the index in the unique output `y`.
+// In other words, for an `1-D` tensor `x` with `axis = None:
+//
+// `y[idx[i]] = x[i] for i in [0, 1,...,rank(x) - 1]`
+//
+// For example:
+//
+// ```
+// # tensor 'x' is [1, 1, 2, 4, 4, 4, 7, 8, 8]
+// y, idx = unique(x)
+// y ==> [1, 2, 4, 7, 8]
+// idx ==> [0, 0, 1, 2, 2, 2, 3, 4, 4]
+// ```
+//
+// For an `2-D` tensor `x` with `axis = 0`:
+//
+// ```
+// # tensor 'x' is [[1, 0, 0],
+// #                [1, 0, 0],
+// #                [2, 0, 0]]
+// y, idx = unique(x, axis=0)
+// y ==> [[1, 0, 0],
+//        [2, 0, 0]]
+// idx ==> [0, 0, 1]
+// ```
+//
+// For an `2-D` tensor `x` with `axis = 1`:
+//
+// ```
+// # tensor 'x' is [[1, 0, 0],
+// #                [1, 0, 0],
+// #                [2, 0, 0]]
+// y, idx = unique(x, axis=1)
+// y ==> [[1, 0],
+//        [1, 0],
+//        [2, 0]]
+// idx ==> [0, 1, 1]
+// ```
+//
+// Arguments:
+//	x: A `Tensor`.
+//	axis: A `Tensor` of type `int32` (default: None). The axis of the Tensor to
+// find the unique elements.
+//
+// Returns:
+//	y: A `Tensor`. Unique elements along the `axis` of `Tensor` x.
+//	idx: A 1-D Tensor. Has the same type as x that contains the index of each
+// value of x in the output y.
+func UniqueV2(scope *Scope, x tf.Output, axis tf.Output, optional ...UniqueV2Attr) (y tf.Output, idx tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "UniqueV2",
+		Input: []tf.Input{
+			x, axis,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// DecodePaddedRawAttr is an optional argument to DecodePaddedRaw.
+type DecodePaddedRawAttr func(optionalAttr)
+
+// DecodePaddedRawLittleEndian sets the optional little_endian attribute to value.
+//
+// value: Whether the input `input_bytes` is in little-endian order. Ignored for
+// `out_type` values that are stored in a single byte, like `uint8`
+// If not specified, defaults to true
+func DecodePaddedRawLittleEndian(value bool) DecodePaddedRawAttr {
+	return func(m optionalAttr) {
+		m["little_endian"] = value
+	}
+}
+
+// Reinterpret the bytes of a string as a vector of numbers.
+//
+// Arguments:
+//	input_bytes: Tensor of string to be decoded.
+//	fixed_length: Length in bytes for each element of the decoded output. Must be a multiple
+// of the size of the output type.
+//
+//
+// Returns A Tensor with one more dimension than the input `bytes`. The added dimension
+// will have size equal to the length of the elements of `bytes` divided by the
+// number of bytes to represent `out_type`.
+func DecodePaddedRaw(scope *Scope, input_bytes tf.Output, fixed_length tf.Output, out_type tf.DataType, optional ...DecodePaddedRawAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"out_type": out_type}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DecodePaddedRaw",
+		Input: []tf.Input{
+			input_bytes, fixed_length,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RetrieveTPUEmbeddingADAMParametersAttr is an optional argument to RetrieveTPUEmbeddingADAMParameters.
+type RetrieveTPUEmbeddingADAMParametersAttr func(optionalAttr)
+
+// RetrieveTPUEmbeddingADAMParametersTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func RetrieveTPUEmbeddingADAMParametersTableId(value int64) RetrieveTPUEmbeddingADAMParametersAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingADAMParametersTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingADAMParametersTableName(value string) RetrieveTPUEmbeddingADAMParametersAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingADAMParametersConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingADAMParametersConfig(value string) RetrieveTPUEmbeddingADAMParametersAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Retrieve ADAM embedding parameters.
+//
+// An op that retrieves optimization parameters from embedding to host
+// memory. Must be preceded by a ConfigureTPUEmbeddingHost op that sets up
+// the correct embedding table configuration. For example, this op is
+// used to retrieve updated parameters before saving a checkpoint.
+//
+// Returns:
+//	parameters: Parameter parameters updated by the ADAM optimization algorithm.
+//	momenta: Parameter momenta updated by the ADAM optimization algorithm.
+//	velocities: Parameter velocities updated by the ADAM optimization algorithm.
+func RetrieveTPUEmbeddingADAMParameters(scope *Scope, num_shards int64, shard_id int64, optional ...RetrieveTPUEmbeddingADAMParametersAttr) (parameters tf.Output, momenta tf.Output, velocities tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RetrieveTPUEmbeddingADAMParameters",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// StatelessRandomBinomialAttr is an optional argument to StatelessRandomBinomial.
+type StatelessRandomBinomialAttr func(optionalAttr)
+
+// StatelessRandomBinomialDtype sets the optional dtype attribute to value.
+//
+// value: The type of the output.
+// If not specified, defaults to DT_INT64
+func StatelessRandomBinomialDtype(value tf.DataType) StatelessRandomBinomialAttr {
+	return func(m optionalAttr) {
+		m["dtype"] = value
+	}
+}
+
+// Outputs deterministic pseudorandom random numbers from a binomial distribution.
+//
+// Outputs random values from a binomial distribution.
+//
+// The outputs are a deterministic function of `shape`, `seed`, `counts`, and `probs`.
+//
+// Arguments:
+//	shape: The shape of the output tensor.
+//	seed: 2 seeds (shape [2]).
+//	counts: The counts of the binomial distribution. Must be broadcastable with `probs`,
+// and broadcastable with the rightmost dimensions of `shape`.
+//	probs: The probability of success for the binomial distribution. Must be broadcastable
+// with `counts` and broadcastable with the rightmost dimensions of `shape`.
+//
+// Returns Random values with specified shape.
+func StatelessRandomBinomial(scope *Scope, shape tf.Output, seed tf.Output, counts tf.Output, probs tf.Output, optional ...StatelessRandomBinomialAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StatelessRandomBinomial",
+		Input: []tf.Input{
+			shape, seed, counts, probs,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RetrieveTPUEmbeddingAdagradParametersGradAccumDebugAttr is an optional argument to RetrieveTPUEmbeddingAdagradParametersGradAccumDebug.
+type RetrieveTPUEmbeddingAdagradParametersGradAccumDebugAttr func(optionalAttr)
+
+// RetrieveTPUEmbeddingAdagradParametersGradAccumDebugTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func RetrieveTPUEmbeddingAdagradParametersGradAccumDebugTableId(value int64) RetrieveTPUEmbeddingAdagradParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingAdagradParametersGradAccumDebugTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingAdagradParametersGradAccumDebugTableName(value string) RetrieveTPUEmbeddingAdagradParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingAdagradParametersGradAccumDebugConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingAdagradParametersGradAccumDebugConfig(value string) RetrieveTPUEmbeddingAdagradParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Retrieve Adagrad embedding parameters with debug support.
+//
+// An op that retrieves optimization parameters from embedding to host
+// memory. Must be preceded by a ConfigureTPUEmbeddingHost op that sets up
+// the correct embedding table configuration. For example, this op is
+// used to retrieve updated parameters before saving a checkpoint.
+//
+// Returns:
+//	parameters: Parameter parameters updated by the Adagrad optimization algorithm.
+//	accumulators: Parameter accumulators updated by the Adagrad optimization algorithm.
+//	gradient_accumulators: Parameter gradient_accumulators updated by the Adagrad optimization algorithm.
+func RetrieveTPUEmbeddingAdagradParametersGradAccumDebug(scope *Scope, num_shards int64, shard_id int64, optional ...RetrieveTPUEmbeddingAdagradParametersGradAccumDebugAttr) (parameters tf.Output, accumulators tf.Output, gradient_accumulators tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RetrieveTPUEmbeddingAdagradParametersGradAccumDebug",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// StatelessMultinomialAttr is an optional argument to StatelessMultinomial.
+type StatelessMultinomialAttr func(optionalAttr)
+
+// StatelessMultinomialOutputDtype sets the optional output_dtype attribute to value.
+// If not specified, defaults to DT_INT64
+func StatelessMultinomialOutputDtype(value tf.DataType) StatelessMultinomialAttr {
+	return func(m optionalAttr) {
+		m["output_dtype"] = value
+	}
+}
+
+// Draws samples from a multinomial distribution.
+//
+// Arguments:
+//	logits: 2-D Tensor with shape `[batch_size, num_classes]`.  Each slice `[i, :]`
+// represents the unnormalized log probabilities for all classes.
+//	num_samples: 0-D.  Number of independent samples to draw for each row slice.
+//	seed: 2 seeds (shape [2]).
+//
+// Returns 2-D Tensor with shape `[batch_size, num_samples]`.  Each slice `[i, :]`
+// contains the drawn class labels with range `[0, num_classes)`.
+func StatelessMultinomial(scope *Scope, logits tf.Output, num_samples tf.Output, seed tf.Output, optional ...StatelessMultinomialAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StatelessMultinomial",
+		Input: []tf.Input{
+			logits, num_samples, seed,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns a copy of the input tensor.
+func Snapshot(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Snapshot",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Adds up a `SparseTensor` and a dense `Tensor`, producing a dense `Tensor`.
+//
+// This Op does not require `a_indices` be sorted in standard lexicographic order.
+//
+// Arguments:
+//	a_indices: 2-D.  The `indices` of the `SparseTensor`, with shape `[nnz, ndims]`.
+//	a_values: 1-D.  The `values` of the `SparseTensor`, with shape `[nnz]`.
+//	a_shape: 1-D.  The `shape` of the `SparseTensor`, with shape `[ndims]`.
+//	b: `ndims`-D Tensor.  With shape `a_shape`.
+func SparseTensorDenseAdd(scope *Scope, a_indices tf.Output, a_values tf.Output, a_shape tf.Output, b tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseTensorDenseAdd",
+		Input: []tf.Input{
+			a_indices, a_values, a_shape, b,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RaggedBincountAttr is an optional argument to RaggedBincount.
+type RaggedBincountAttr func(optionalAttr)
+
+// RaggedBincountBinaryOutput sets the optional binary_output attribute to value.
+//
+// value: bool; Whether the kernel should count the appearance or number of occurrences.
+// If not specified, defaults to false
+func RaggedBincountBinaryOutput(value bool) RaggedBincountAttr {
+	return func(m optionalAttr) {
+		m["binary_output"] = value
+	}
+}
+
+// Counts the number of occurrences of each value in an integer array.
+//
+// Outputs a vector with length `size` and the same dtype as `weights`. If
+// `weights` are empty, then index `i` stores the number of times the value `i` is
+// counted in `arr`. If `weights` are non-empty, then index `i` stores the sum of
+// the value in `weights` at each index where the corresponding value in `arr` is
+// `i`.
+//
+// Values in `arr` outside of the range [0, size) are ignored.
+//
+// Arguments:
+//	splits: 1D int64 `Tensor`.
+//	values: 2D int `Tensor`.
+//	size: non-negative int scalar `Tensor`.
+//	weights: is an int32, int64, float32, or float64 `Tensor` with the same
+// shape as `input`, or a length-0 `Tensor`, in which case it acts as all weights
+// equal to 1.
+//
+// Returns 1D `Tensor` with length equal to `size` or 2D `Tensor` with [batch_size, `size`].
+// The counts or summed weights for each value in the range [0, size).
+func RaggedBincount(scope *Scope, splits tf.Output, values tf.Output, size tf.Output, weights tf.Output, optional ...RaggedBincountAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RaggedBincount",
+		Input: []tf.Input{
+			splits, values, size, weights,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// StatelessRandomNormalAttr is an optional argument to StatelessRandomNormal.
+type StatelessRandomNormalAttr func(optionalAttr)
+
+// StatelessRandomNormalDtype sets the optional dtype attribute to value.
+//
+// value: The type of the output.
+// If not specified, defaults to DT_FLOAT
+func StatelessRandomNormalDtype(value tf.DataType) StatelessRandomNormalAttr {
+	return func(m optionalAttr) {
+		m["dtype"] = value
+	}
+}
+
+// Outputs deterministic pseudorandom values from a normal distribution.
+//
+// The generated values will have mean 0 and standard deviation 1.
+//
+// The outputs are a deterministic function of `shape` and `seed`.
+//
+// Arguments:
+//	shape: The shape of the output tensor.
+//	seed: 2 seeds (shape [2]).
+//
+// Returns Random values with specified shape.
+func StatelessRandomNormal(scope *Scope, shape tf.Output, seed tf.Output, optional ...StatelessRandomNormalAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StatelessRandomNormal",
+		Input: []tf.Input{
+			shape, seed,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the sum along sparse segments of a tensor divided by the sqrt of N.
+//
+// N is the size of the segment being reduced.
+//
+// See `tf.sparse.segment_sum` for usage examples.
+//
+//
+// Arguments:
+//
+//	indices: A 1-D tensor. Has same rank as `segment_ids`.
+//	segment_ids: A 1-D tensor. Values should be sorted and can be repeated.
+//
+// Returns Has same shape as data, except for dimension 0 which
+// has size `k`, the number of segments.
+func SparseSegmentSqrtN(scope *Scope, data tf.Output, indices tf.Output, segment_ids tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseSegmentSqrtN",
+		Input: []tf.Input{
+			data, indices, segment_ids,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// UnicodeDecodeWithOffsetsAttr is an optional argument to UnicodeDecodeWithOffsets.
+type UnicodeDecodeWithOffsetsAttr func(optionalAttr)
+
+// UnicodeDecodeWithOffsetsErrors sets the optional errors attribute to value.
+//
+// value: Error handling policy when there is invalid formatting found in the input.
+// The value of 'strict' will cause the operation to produce a InvalidArgument
+// error on any invalid input formatting. A value of 'replace' (the default) will
+// cause the operation to replace any invalid formatting in the input with the
+// `replacement_char` codepoint. A value of 'ignore' will cause the operation to
+// skip any invalid formatting in the input and produce no corresponding output
+// character.
+// If not specified, defaults to "replace"
+func UnicodeDecodeWithOffsetsErrors(value string) UnicodeDecodeWithOffsetsAttr {
+	return func(m optionalAttr) {
+		m["errors"] = value
+	}
+}
+
+// UnicodeDecodeWithOffsetsReplacementChar sets the optional replacement_char attribute to value.
+//
+// value: The replacement character codepoint to be used in place of any invalid
+// formatting in the input when `errors='replace'`. Any valid unicode codepoint may
+// be used. The default value is the default unicode replacement character is
+// 0xFFFD or U+65533.)
+// If not specified, defaults to 65533
+func UnicodeDecodeWithOffsetsReplacementChar(value int64) UnicodeDecodeWithOffsetsAttr {
+	return func(m optionalAttr) {
+		m["replacement_char"] = value
+	}
+}
+
+// UnicodeDecodeWithOffsetsReplaceControlCharacters sets the optional replace_control_characters attribute to value.
+//
+// value: Whether to replace the C0 control characters (00-1F) with the
+// `replacement_char`. Default is false.
+// If not specified, defaults to false
+func UnicodeDecodeWithOffsetsReplaceControlCharacters(value bool) UnicodeDecodeWithOffsetsAttr {
+	return func(m optionalAttr) {
+		m["replace_control_characters"] = value
+	}
+}
+
+// UnicodeDecodeWithOffsetsTsplits sets the optional Tsplits attribute to value.
+// If not specified, defaults to DT_INT64
+func UnicodeDecodeWithOffsetsTsplits(value tf.DataType) UnicodeDecodeWithOffsetsAttr {
+	return func(m optionalAttr) {
+		m["Tsplits"] = value
+	}
+}
+
+// Decodes each string in `input` into a sequence of Unicode code points.
+//
+// The character codepoints for all strings are returned using a single vector
+// `char_values`, with strings expanded to characters in row-major order.
+// Similarly, the character start byte offsets are returned using a single vector
+// `char_to_byte_starts`, with strings expanded in row-major order.
+//
+// The `row_splits` tensor indicates where the codepoints and start offsets for
+// each input string begin and end within the `char_values` and
+// `char_to_byte_starts` tensors.  In particular, the values for the `i`th
+// string (in row-major order) are stored in the slice
+// `[row_splits[i]:row_splits[i+1]]`. Thus:
+//
+// * `char_values[row_splits[i]+j]` is the Unicode codepoint for the `j`th
+//   character in the `i`th string (in row-major order).
+// * `char_to_bytes_starts[row_splits[i]+j]` is the start byte offset for the `j`th
+//   character in the `i`th string (in row-major order).
+// * `row_splits[i+1] - row_splits[i]` is the number of characters in the `i`th
+//   string (in row-major order).
+//
+// Arguments:
+//	input: The text to be decoded. Can have any shape. Note that the output is flattened
+// to a vector of char values.
+//	input_encoding: Text encoding of the input strings. This is any of the encodings supported
+// by ICU ucnv algorithmic converters. Examples: `"UTF-16", "US ASCII", "UTF-8"`.
+//
+// Returns:
+//	row_splits: A 1D int32 tensor containing the row splits.
+//	char_values: A 1D int32 Tensor containing the decoded codepoints.
+//	char_to_byte_starts: A 1D int32 Tensor containing the byte index in the input string where each
+// character in `char_values` starts.
+func UnicodeDecodeWithOffsets(scope *Scope, input tf.Output, input_encoding string, optional ...UnicodeDecodeWithOffsetsAttr) (row_splits tf.Output, char_values tf.Output, char_to_byte_starts tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"input_encoding": input_encoding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "UnicodeDecodeWithOffsets",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+//   This op is used as a placeholder in If branch functions. It doesn't provide a
+//   valid output when run, so must either be removed (e.g. replaced with a
+//   function input) or guaranteed not to be used (e.g. if mirroring an
+//   intermediate output needed for the gradient computation of the other branch).
+//
+// Arguments:
+//	dtype: The type of the output.
+//	shape:     The purported shape of the output. This is only used for shape inference;
+//     the output will not necessarily have this shape. Can be a partial shape.
+//
+// Returns     \"Fake\" output value. This should not be consumed by another op.
+func FakeParam(scope *Scope, dtype tf.DataType, shape tf.Shape) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype, "shape": shape}
+	opspec := tf.OpSpec{
+		Type: "FakeParam",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// UnicodeTranscodeAttr is an optional argument to UnicodeTranscode.
+type UnicodeTranscodeAttr func(optionalAttr)
+
+// UnicodeTranscodeErrors sets the optional errors attribute to value.
+//
+// value: Error handling policy when there is invalid formatting found in the input.
+// The value of 'strict' will cause the operation to produce a InvalidArgument
+// error on any invalid input formatting. A value of 'replace' (the default) will
+// cause the operation to replace any invalid formatting in the input with the
+// `replacement_char` codepoint. A value of 'ignore' will cause the operation to
+// skip any invalid formatting in the input and produce no corresponding output
+// character.
+// If not specified, defaults to "replace"
+func UnicodeTranscodeErrors(value string) UnicodeTranscodeAttr {
+	return func(m optionalAttr) {
+		m["errors"] = value
+	}
+}
+
+// UnicodeTranscodeReplacementChar sets the optional replacement_char attribute to value.
+//
+// value: The replacement character codepoint to be used in place of any invalid
+// formatting in the input when `errors='replace'`. Any valid unicode codepoint may
+// be used. The default value is the default unicode replacement character is
+// 0xFFFD or U+65533.)
+//
+// Note that for UTF-8, passing a replacement character expressible in 1 byte, such
+// as ' ', will preserve string alignment to the source since invalid bytes will be
+// replaced with a 1-byte replacement. For UTF-16-BE and UTF-16-LE, any 1 or 2 byte
+// replacement character will preserve byte alignment to the source.
+// If not specified, defaults to 65533
+func UnicodeTranscodeReplacementChar(value int64) UnicodeTranscodeAttr {
+	return func(m optionalAttr) {
+		m["replacement_char"] = value
+	}
+}
+
+// UnicodeTranscodeReplaceControlCharacters sets the optional replace_control_characters attribute to value.
+//
+// value: Whether to replace the C0 control characters (00-1F) with the
+// `replacement_char`. Default is false.
+// If not specified, defaults to false
+func UnicodeTranscodeReplaceControlCharacters(value bool) UnicodeTranscodeAttr {
+	return func(m optionalAttr) {
+		m["replace_control_characters"] = value
+	}
+}
+
+// Transcode the input text from a source encoding to a destination encoding.
+//
+// The input is a string tensor of any shape. The output is a string tensor of
+// the same shape containing the transcoded strings. Output strings are always
+// valid unicode. If the input contains invalid encoding positions, the
+// `errors` attribute sets the policy for how to deal with them. If the default
+// error-handling policy is used, invalid formatting will be substituted in the
+// output by the `replacement_char`. If the errors policy is to `ignore`, any
+// invalid encoding positions in the input are skipped and not included in the
+// output. If it set to `strict` then any invalid formatting will result in an
+// InvalidArgument error.
+//
+// This operation can be used with `output_encoding = input_encoding` to enforce
+// correct formatting for inputs even if they are already in the desired encoding.
+//
+// If the input is prefixed by a Byte Order Mark needed to determine encoding
+// (e.g. if the encoding is UTF-16 and the BOM indicates big-endian), then that
+// BOM will be consumed and not emitted into the output. If the input encoding
+// is marked with an explicit endianness (e.g. UTF-16-BE), then the BOM is
+// interpreted as a non-breaking-space and is preserved in the output (including
+// always for UTF-8).
+//
+// The end result is that if the input is marked as an explicit endianness the
+// transcoding is faithful to all codepoints in the source. If it is not marked
+// with an explicit endianness, the BOM is not considered part of the string itself
+// but as metadata, and so is not preserved in the output.
+//
+// Examples:
+//
+// >>> tf.strings.unicode_transcode(["Hello", "TensorFlow", "2.x"], "UTF-8", "UTF-16-BE")
+// <tf.Tensor: shape=(3,), dtype=string, numpy=
+// array([b'\x00H\x00e\x00l\x00l\x00o',
+//        b'\x00T\x00e\x00n\x00s\x00o\x00r\x00F\x00l\x00o\x00w',
+//        b'\x002\x00.\x00x'], dtype=object)>
+// >>> tf.strings.unicode_transcode(["A", "B", "C"], "US ASCII", "UTF-8").numpy()
+// array([b'A', b'B', b'C'], dtype=object)
+//
+// Arguments:
+//	input: The text to be processed. Can have any shape.
+//	input_encoding: Text encoding of the input strings. This is any of the encodings supported
+// by ICU ucnv algorithmic converters. Examples: `"UTF-16", "US ASCII", "UTF-8"`.
+//	output_encoding: The unicode encoding to use in the output. Must be one of
+// `"UTF-8", "UTF-16-BE", "UTF-32-BE"`. Multi-byte encodings will be big-endian.
+//
+// Returns A string tensor containing unicode text encoded using `output_encoding`.
+func UnicodeTranscode(scope *Scope, input tf.Output, input_encoding string, output_encoding string, optional ...UnicodeTranscodeAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"input_encoding": input_encoding, "output_encoding": output_encoding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "UnicodeTranscode",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns x // y element-wise.
+//
+// *NOTE*: `FloorDiv` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func FloorDiv(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "FloorDiv",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// An Op to permute tensors across replicated TPU instances.
+//
+// Each instance supplies its own input.
+//
+// For example, suppose there are 4 TPU instances: `[A, B, C, D]`. Passing
+// source_target_pairs=`[[0,1],[1,2],[2,3],[3,0]]` gets the outputs:
+// `[D, A, B, C]`.
+//
+// Arguments:
+//	input: The local input to be permuted. Currently only supports float and
+// bfloat16.
+//	source_target_pairs: A tensor with shape [num_pairs, 2].
+//
+// Returns The permuted input.
+func CollectivePermute(scope *Scope, input tf.Output, source_target_pairs tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "CollectivePermute",
+		Input: []tf.Input{
+			input, source_target_pairs,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// QuantizedReluXAttr is an optional argument to QuantizedReluX.
+type QuantizedReluXAttr func(optionalAttr)
+
+// QuantizedReluXOutType sets the optional out_type attribute to value.
+// If not specified, defaults to DT_QUINT8
+func QuantizedReluXOutType(value tf.DataType) QuantizedReluXAttr {
+	return func(m optionalAttr) {
+		m["out_type"] = value
+	}
+}
+
+// Computes Quantized Rectified Linear X: `min(max(features, 0), max_value)`
+//
+// Arguments:
+//
+//
+//	min_features: The float value that the lowest quantized value represents.
+//	max_features: The float value that the highest quantized value represents.
+//
+// Returns:
+//	activations: Has the same output shape as "features".
+//	min_activations: The float value that the lowest quantized value represents.
+//	max_activations: The float value that the highest quantized value represents.
+func QuantizedReluX(scope *Scope, features tf.Output, max_value tf.Output, min_features tf.Output, max_features tf.Output, optional ...QuantizedReluXAttr) (activations tf.Output, min_activations tf.Output, max_activations tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizedReluX",
+		Input: []tf.Input{
+			features, max_value, min_features, max_features,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Forwards `data` to the output port determined by `pred`.
+//
+// If `pred` is true, the `data` input is forwarded to `output_true`. Otherwise,
+// the data goes to `output_false`.
+//
+// See also `RefSwitch` and `Merge`.
+//
+// Arguments:
+//	data: The tensor to be forwarded to the appropriate output.
+//	pred: A scalar that specifies which output port will receive data.
+//
+// Returns:
+//	output_false: If `pred` is false, data will be forwarded to this output.
+//	output_true: If `pred` is true, data will be forwarded to this output.
+func Switch(scope *Scope, data tf.Output, pred tf.Output) (output_false tf.Output, output_true tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Switch",
+		Input: []tf.Input{
+			data, pred,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// RetrieveTPUEmbeddingFTRLParametersGradAccumDebugAttr is an optional argument to RetrieveTPUEmbeddingFTRLParametersGradAccumDebug.
+type RetrieveTPUEmbeddingFTRLParametersGradAccumDebugAttr func(optionalAttr)
+
+// RetrieveTPUEmbeddingFTRLParametersGradAccumDebugTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func RetrieveTPUEmbeddingFTRLParametersGradAccumDebugTableId(value int64) RetrieveTPUEmbeddingFTRLParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingFTRLParametersGradAccumDebugTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingFTRLParametersGradAccumDebugTableName(value string) RetrieveTPUEmbeddingFTRLParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingFTRLParametersGradAccumDebugConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingFTRLParametersGradAccumDebugConfig(value string) RetrieveTPUEmbeddingFTRLParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Retrieve FTRL embedding parameters with debug support.
+//
+// An op that retrieves optimization parameters from embedding to host
+// memory. Must be preceded by a ConfigureTPUEmbeddingHost op that sets up
+// the correct embedding table configuration. For example, this op is
+// used to retrieve updated parameters before saving a checkpoint.
+//
+// Returns:
+//	parameters: Parameter parameters updated by the FTRL optimization algorithm.
+//	accumulators: Parameter accumulators updated by the FTRL optimization algorithm.
+//	linears: Parameter linears updated by the FTRL optimization algorithm.
+//	gradient_accumulators: Parameter gradient_accumulators updated by the FTRL optimization algorithm.
+func RetrieveTPUEmbeddingFTRLParametersGradAccumDebug(scope *Scope, num_shards int64, shard_id int64, optional ...RetrieveTPUEmbeddingFTRLParametersGradAccumDebugAttr) (parameters tf.Output, accumulators tf.Output, linears tf.Output, gradient_accumulators tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RetrieveTPUEmbeddingFTRLParametersGradAccumDebug",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3)
+}
+
+// UnicodeEncodeAttr is an optional argument to UnicodeEncode.
+type UnicodeEncodeAttr func(optionalAttr)
+
+// UnicodeEncodeErrors sets the optional errors attribute to value.
+//
+// value: Error handling policy when there is invalid formatting found in the input.
+// The value of 'strict' will cause the operation to produce a InvalidArgument
+// error on any invalid input formatting. A value of 'replace' (the default) will
+// cause the operation to replace any invalid formatting in the input with the
+// `replacement_char` codepoint. A value of 'ignore' will cause the operation to
+// skip any invalid formatting in the input and produce no corresponding output
+// character.
+// If not specified, defaults to "replace"
+func UnicodeEncodeErrors(value string) UnicodeEncodeAttr {
+	return func(m optionalAttr) {
+		m["errors"] = value
+	}
+}
+
+// UnicodeEncodeReplacementChar sets the optional replacement_char attribute to value.
+//
+// value: The replacement character codepoint to be used in place of any invalid
+// formatting in the input when `errors='replace'`. Any valid unicode codepoint may
+// be used. The default value is the default unicode replacement character is
+// 0xFFFD (U+65533).
+// If not specified, defaults to 65533
+func UnicodeEncodeReplacementChar(value int64) UnicodeEncodeAttr {
+	return func(m optionalAttr) {
+		m["replacement_char"] = value
+	}
+}
+
+// Encode a tensor of ints into unicode strings.
+//
+// Returns a vector of strings, where `output[i]` is constructed by encoding the
+// Unicode codepoints in `input_values[input_splits[i]:input_splits[i+1]]`
+// using `output_encoding`.
+//
+// ---
+//
+// Example:
+//
+// ```
+// input_values = [72, 101, 108, 108, 111, 87, 111, 114, 108, 100]
+// input_splits = [0, 5, 10]
+// output_encoding = 'UTF-8'
+//
+// output = ['Hello', 'World']
+// ```
+//
+// Arguments:
+//	input_values: A 1D tensor containing the unicode codepoints that should be encoded.
+//	input_splits: A 1D tensor specifying how the unicode codepoints should be split into strings.
+// In particular, `output[i]` is constructed by encoding the codepoints in the
+// slice `input_values[input_splits[i]:input_splits[i+1]]`.
+//	output_encoding: Unicode encoding of the output strings. Valid encodings are: `"UTF-8",
+// "UTF-16-BE", and "UTF-32-BE"`.
+//
+// Returns The 1-D Tensor of strings encoded from the provided unicode codepoints.
+func UnicodeEncode(scope *Scope, input_values tf.Output, input_splits tf.Output, output_encoding string, optional ...UnicodeEncodeAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_encoding": output_encoding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "UnicodeEncode",
+		Input: []tf.Input{
+			input_values, input_splits,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// PrelinearizeTupleAttr is an optional argument to PrelinearizeTuple.
+type PrelinearizeTupleAttr func(optionalAttr)
+
+// PrelinearizeTupleLayouts sets the optional layouts attribute to value.
+//
+// value: A vector holding the requested layout in minor-to-major sequence for all the
+// tuple shapes in the order the shapes appear in the "shapes" input. The layout
+// elements for a sub-shape can be set to -1 in which case the corresponding layout
+// will be computed by the infeed operation.
+// If not specified, defaults to <>
+func PrelinearizeTupleLayouts(value []int64) PrelinearizeTupleAttr {
+	return func(m optionalAttr) {
+		m["layouts"] = value
+	}
+}
+
+// An op which linearizes multiple Tensor values to an opaque variant tensor.
+//
+// Arguments:
+//	inputs: A list of tensors that will be provided using the infeed mechanism.
+//	shapes: The shapes of each tensor in `inputs`.
+func PrelinearizeTuple(scope *Scope, inputs []tf.Output, shapes []tf.Shape, optional ...PrelinearizeTupleAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"shapes": shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "PrelinearizeTuple",
+		Input: []tf.Input{
+			tf.OutputList(inputs),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the LSTM cell backward propagation for the entire time sequence.
+//
+// This implementation is to be used in conjunction of LSTMBlock.
+//
+// Arguments:
+//	seq_len_max: Maximum time length actually used by this input. Outputs are padded
+// with zeros beyond this length.
+//	x: The sequence input to the LSTM, shape (timelen, batch_size, num_inputs).
+//	cs_prev: Value of the initial cell state.
+//	h_prev: Initial output of cell (to be used for peephole).
+//	w: The weight matrix.
+//	wci: The weight matrix for input gate peephole connection.
+//	wcf: The weight matrix for forget gate peephole connection.
+//	wco: The weight matrix for output gate peephole connection.
+//	b: The bias vector.
+//	i: The input gate over the whole time sequence.
+//	cs: The cell state before the tanh over the whole time sequence.
+//	f: The forget gate over the whole time sequence.
+//	o: The output gate over the whole time sequence.
+//	ci: The cell input over the whole time sequence.
+//	co: The cell after the tanh over the whole time sequence.
+//	h: The output h vector over the whole time sequence.
+//	cs_grad: The current gradient of cs.
+//	h_grad: The gradient of h vector.
+//	use_peephole: Whether to use peephole weights.
+//
+// Returns:
+//	x_grad: The gradient of x to be back-propped.
+//	cs_prev_grad: The gradient of cs_prev to be back-propped.
+//	h_prev_grad: The gradient of h_prev to be back-propped.
+//	w_grad: The gradient for w to be back-propped.
+//	wci_grad: The gradient for wci to be back-propped.
+//	wcf_grad: The gradient for wcf to be back-propped.
+//	wco_grad: The gradient for wco to be back-propped.
+//	b_grad: The gradient for w to be back-propped.
+func BlockLSTMGrad(scope *Scope, seq_len_max tf.Output, x tf.Output, cs_prev tf.Output, h_prev tf.Output, w tf.Output, wci tf.Output, wcf tf.Output, wco tf.Output, b tf.Output, i tf.Output, cs tf.Output, f tf.Output, o tf.Output, ci tf.Output, co tf.Output, h tf.Output, cs_grad tf.Output, h_grad tf.Output, use_peephole bool) (x_grad tf.Output, cs_prev_grad tf.Output, h_prev_grad tf.Output, w_grad tf.Output, wci_grad tf.Output, wcf_grad tf.Output, wco_grad tf.Output, b_grad tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"use_peephole": use_peephole}
+	opspec := tf.OpSpec{
+		Type: "BlockLSTMGrad",
+		Input: []tf.Input{
+			seq_len_max, x, cs_prev, h_prev, w, wci, wcf, wco, b, i, cs, f, o, ci, co, h, cs_grad, h_grad,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3), op.Output(4), op.Output(5), op.Output(6), op.Output(7)
+}
+
+// OutfeedDequeueTupleAttr is an optional argument to OutfeedDequeueTuple.
+type OutfeedDequeueTupleAttr func(optionalAttr)
+
+// OutfeedDequeueTupleDeviceOrdinal sets the optional device_ordinal attribute to value.
+//
+// value: The TPU device to use. This should be -1 when the Op
+// is running on a TPU device, and >= 0 when the Op is running on the CPU
+// device.
+// If not specified, defaults to -1
+func OutfeedDequeueTupleDeviceOrdinal(value int64) OutfeedDequeueTupleAttr {
+	return func(m optionalAttr) {
+		m["device_ordinal"] = value
+	}
+}
+
+// Retrieve multiple values from the computation outfeed.
+//
+// This operation will block indefinitely until data is available. Output `i`
+// corresponds to XLA tuple element `i`.
+//
+// Arguments:
+//	dtypes: The element types of each element in `outputs`.
+//	shapes: The shapes of each tensor in `outputs`.
+//
+// Returns A list of tensors that will be read from the outfeed.
+func OutfeedDequeueTuple(scope *Scope, dtypes []tf.DataType, shapes []tf.Shape, optional ...OutfeedDequeueTupleAttr) (outputs []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes, "shapes": shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "OutfeedDequeueTuple",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if outputs, idx, err = makeOutputList(op, idx, "outputs"); err != nil {
+		scope.UpdateErr("OutfeedDequeueTuple", err)
+		return
+	}
+	return outputs
+}
+
+// DecodeCompressedAttr is an optional argument to DecodeCompressed.
+type DecodeCompressedAttr func(optionalAttr)
+
+// DecodeCompressedCompressionType sets the optional compression_type attribute to value.
+//
+// value: A scalar containing either (i) the empty string (no
+// compression), (ii) "ZLIB", or (iii) "GZIP".
+// If not specified, defaults to ""
+func DecodeCompressedCompressionType(value string) DecodeCompressedAttr {
+	return func(m optionalAttr) {
+		m["compression_type"] = value
+	}
+}
+
+// Decompress strings.
+//
+// This op decompresses each element of the `bytes` input `Tensor`, which
+// is assumed to be compressed using the given `compression_type`.
+//
+// The `output` is a string `Tensor` of the same shape as `bytes`,
+// each element containing the decompressed data from the corresponding
+// element in `bytes`.
+//
+// Arguments:
+//	bytes: A Tensor of string which is compressed.
+//
+// Returns A Tensor with the same shape as input `bytes`, uncompressed
+// from bytes.
+func DecodeCompressed(scope *Scope, bytes tf.Output, optional ...DecodeCompressedAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DecodeCompressed",
+		Input: []tf.Input{
+			bytes,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Converts each string in the input Tensor to its hash mod by a number of buckets.
+//
+// The hash function is deterministic on the content of the string within the
+// process. The hash function is a keyed hash function, where attribute `key`
+// defines the key of the hash function. `key` is an array of 2 elements.
+//
+// A strong hash is important when inputs may be malicious, e.g. URLs with
+// additional components. Adversaries could try to make their inputs hash to the
+// same bucket for a denial-of-service attack or to skew the results. A strong
+// hash can be used to make it difficult to find inputs with a skewed hash value
+// distribution over buckets. This requires that the hash function is
+// seeded by a high-entropy (random) "key" unknown to the adversary.
+//
+// The additional robustness comes at a cost of roughly 4x higher compute
+// time than `tf.string_to_hash_bucket_fast`.
+//
+// Examples:
+//
+// >>> tf.strings.to_hash_bucket_strong(["Hello", "TF"], 3, [1, 2]).numpy()
+// array([2, 0])
+//
+// Arguments:
+//	input: The strings to assign a hash bucket.
+//	num_buckets: The number of buckets.
+//	key: The key used to seed the hash function, passed as a list of two uint64
+// elements.
+//
+// Returns A Tensor of the same shape as the input `string_tensor`.
+func StringToHashBucketStrong(scope *Scope, input tf.Output, num_buckets int64, key []int64) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_buckets": num_buckets, "key": key}
+	opspec := tf.OpSpec{
+		Type: "StringToHashBucketStrong",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Deserialize bucket boundaries and ready flag into current QuantileAccumulator.
+//
+// An op that deserializes bucket boundaries and are boundaries ready flag into current QuantileAccumulator.
+//
+// Arguments:
+//	quantile_stream_resource_handle: resource handle referring to a QuantileStreamResource.
+//	bucket_boundaries: float; List of Rank 1 Tensors each containing the bucket boundaries for a feature.
+//
+// Returns the created operation.
+func BoostedTreesQuantileStreamResourceDeserialize(scope *Scope, quantile_stream_resource_handle tf.Output, bucket_boundaries []tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesQuantileStreamResourceDeserialize",
+		Input: []tf.Input{
+			quantile_stream_resource_handle, tf.OutputList(bucket_boundaries),
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// ResourceApplyAdadeltaAttr is an optional argument to ResourceApplyAdadelta.
+type ResourceApplyAdadeltaAttr func(optionalAttr)
+
+// ResourceApplyAdadeltaUseLocking sets the optional use_locking attribute to value.
+//
+// value: If True, updating of the var, accum and update_accum tensors will be protected by
+// a lock; otherwise the behavior is undefined, but may exhibit less contention.
+// If not specified, defaults to false
+func ResourceApplyAdadeltaUseLocking(value bool) ResourceApplyAdadeltaAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// Update '*var' according to the adadelta scheme.
+//
+// accum = rho() * accum + (1 - rho()) * grad.square();
+// update = (update_accum + epsilon).sqrt() * (accum + epsilon()).rsqrt() * grad;
+// update_accum = rho() * update_accum + (1 - rho()) * update.square();
+// var -= update;
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	accum: Should be from a Variable().
+//	accum_update: Should be from a Variable().
+//	lr: Scaling factor. Must be a scalar.
+//	rho: Decay factor. Must be a scalar.
+//	epsilon: Constant factor. Must be a scalar.
+//	grad: The gradient.
+//
+// Returns the created operation.
+func ResourceApplyAdadelta(scope *Scope, var_ tf.Output, accum tf.Output, accum_update tf.Output, lr tf.Output, rho tf.Output, epsilon tf.Output, grad tf.Output, optional ...ResourceApplyAdadeltaAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceApplyAdadelta",
+		Input: []tf.Input{
+			var_, accum, accum_update, lr, rho, epsilon, grad,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Converts each string in the input Tensor to its hash mod by a number of buckets.
+//
+// The hash function is deterministic on the content of the string within the
+// process and will never change. However, it is not suitable for cryptography.
+// This function may be used when CPU time is scarce and inputs are trusted or
+// unimportant. There is a risk of adversaries constructing inputs that all hash
+// to the same bucket. To prevent this problem, use a strong hash function with
+// `tf.string_to_hash_bucket_strong`.
+//
+// Examples:
+//
+// >>> tf.strings.to_hash_bucket_fast(["Hello", "TensorFlow", "2.x"], 3).numpy()
+// array([0, 2, 2])
+//
+// Arguments:
+//	input: The strings to assign a hash bucket.
+//	num_buckets: The number of buckets.
+//
+// Returns A Tensor of the same shape as the input `string_tensor`.
+func StringToHashBucketFast(scope *Scope, input tf.Output, num_buckets int64) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_buckets": num_buckets}
+	opspec := tf.OpSpec{
+		Type: "StringToHashBucketFast",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// StringJoinAttr is an optional argument to StringJoin.
+type StringJoinAttr func(optionalAttr)
+
+// StringJoinSeparator sets the optional separator attribute to value.
+//
+// value: string, an optional join separator.
+// If not specified, defaults to ""
+func StringJoinSeparator(value string) StringJoinAttr {
+	return func(m optionalAttr) {
+		m["separator"] = value
+	}
+}
+
+// Joins the strings in the given list of string tensors into one tensor;
+//
+// with the given separator (default is an empty separator).
+//
+// Examples:
+//
+// >>> s = ["hello", "world", "tensorflow"]
+// >>> tf.strings.join(s, " ")
+// <tf.Tensor: shape=(), dtype=string, numpy=b'hello world tensorflow'>
+//
+// Arguments:
+//	inputs: A list of string tensors.  The tensors must all have the same shape,
+// or be scalars.  Scalars may be mixed in; these will be broadcast to the shape
+// of non-scalar inputs.
+func StringJoin(scope *Scope, inputs []tf.Output, optional ...StringJoinAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StringJoin",
+		Input: []tf.Input{
+			tf.OutputList(inputs),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Replaces the contents of the table with the specified keys and values.
+//
+// The tensor `keys` must be of the same type as the keys of the table.
+// The tensor `values` must be of the type of the table values.
+//
+// Arguments:
+//	table_handle: Handle to the table.
+//	keys: Any shape.  Keys to look up.
+//	values: Values to associate with keys.
+//
+// Returns the created operation.
+func LookupTableImportV2(scope *Scope, table_handle tf.Output, keys tf.Output, values tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "LookupTableImportV2",
+		Input: []tf.Input{
+			table_handle, keys, values,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// LoadTPUEmbeddingMomentumParametersAttr is an optional argument to LoadTPUEmbeddingMomentumParameters.
+type LoadTPUEmbeddingMomentumParametersAttr func(optionalAttr)
+
+// LoadTPUEmbeddingMomentumParametersTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func LoadTPUEmbeddingMomentumParametersTableId(value int64) LoadTPUEmbeddingMomentumParametersAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// LoadTPUEmbeddingMomentumParametersTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingMomentumParametersTableName(value string) LoadTPUEmbeddingMomentumParametersAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// LoadTPUEmbeddingMomentumParametersConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingMomentumParametersConfig(value string) LoadTPUEmbeddingMomentumParametersAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Load Momentum embedding parameters.
+//
+// An op that loads optimization parameters into HBM for embedding. Must be
+// preceded by a ConfigureTPUEmbeddingHost op that sets up the correct
+// embedding table configuration. For example, this op is used to install
+// parameters that are loaded from a checkpoint before a training loop is
+// executed.
+//
+// Arguments:
+//	parameters: Value of parameters used in the Momentum optimization algorithm.
+//	momenta: Value of momenta used in the Momentum optimization algorithm.
+//
+//
+//
+// Returns the created operation.
+func LoadTPUEmbeddingMomentumParameters(scope *Scope, parameters tf.Output, momenta tf.Output, num_shards int64, shard_id int64, optional ...LoadTPUEmbeddingMomentumParametersAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LoadTPUEmbeddingMomentumParameters",
+		Input: []tf.Input{
+			parameters, momenta,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// SkipgramAttr is an optional argument to Skipgram.
+type SkipgramAttr func(optionalAttr)
+
+// SkipgramWindowSize sets the optional window_size attribute to value.
+//
+// value: The number of words to predict to the left and right of the target.
+// If not specified, defaults to 5
+func SkipgramWindowSize(value int64) SkipgramAttr {
+	return func(m optionalAttr) {
+		m["window_size"] = value
+	}
+}
+
+// SkipgramMinCount sets the optional min_count attribute to value.
+//
+// value: The minimum number of word occurrences for it to be included in the
+// vocabulary.
+// If not specified, defaults to 5
+func SkipgramMinCount(value int64) SkipgramAttr {
+	return func(m optionalAttr) {
+		m["min_count"] = value
+	}
+}
+
+// SkipgramSubsample sets the optional subsample attribute to value.
+//
+// value: Threshold for word occurrence. Words that appear with higher
+// frequency will be randomly down-sampled. Set to 0 to disable.
+// If not specified, defaults to 0.001
+func SkipgramSubsample(value float32) SkipgramAttr {
+	return func(m optionalAttr) {
+		m["subsample"] = value
+	}
+}
+
+// Parses a text file and creates a batch of examples.
+//
+// DEPRECATED at GraphDef version 19: Moving word2vec into tensorflow_models/tutorials and deprecating its ops here as a result
+//
+// Arguments:
+//	filename: The corpus's text file name.
+//	batch_size: The size of produced batch.
+//
+// Returns:
+//	vocab_word: A vector of words in the corpus.
+//	vocab_freq: Frequencies of words. Sorted in the non-ascending order.
+//	words_per_epoch: Number of words per epoch in the data file.
+//	current_epoch: The current epoch number.
+//	total_words_processed: The total number of words processed so far.
+//	examples: A vector of word ids.
+//	labels: A vector of word ids.
+func Skipgram(scope *Scope, filename string, batch_size int64, optional ...SkipgramAttr) (vocab_word tf.Output, vocab_freq tf.Output, words_per_epoch tf.Output, current_epoch tf.Output, total_words_processed tf.Output, examples tf.Output, labels tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"filename": filename, "batch_size": batch_size}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Skipgram",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3), op.Output(4), op.Output(5), op.Output(6)
+}
+
+// StaticRegexReplaceAttr is an optional argument to StaticRegexReplace.
+type StaticRegexReplaceAttr func(optionalAttr)
+
+// StaticRegexReplaceReplaceGlobal sets the optional replace_global attribute to value.
+//
+// value: If True, the replacement is global, otherwise the replacement
+// is done only on the first match.
+// If not specified, defaults to true
+func StaticRegexReplaceReplaceGlobal(value bool) StaticRegexReplaceAttr {
+	return func(m optionalAttr) {
+		m["replace_global"] = value
+	}
+}
+
+// Replaces the match of pattern in input with rewrite.
+//
+// It follows the re2 syntax (https://github.com/google/re2/wiki/Syntax)
+//
+// Arguments:
+//	input: The text to be processed.
+//	pattern: The regular expression to match the input.
+//	rewrite: The rewrite to be applied to the matched expression.
+//
+// Returns The text after applying pattern and rewrite.
+func StaticRegexReplace(scope *Scope, input tf.Output, pattern string, rewrite string, optional ...StaticRegexReplaceAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"pattern": pattern, "rewrite": rewrite}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StaticRegexReplace",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns which elements of x are finite.
+//
+// @compatibility(numpy)
+// Equivalent to np.isfinite
+// @end_compatibility
+//
+// Example:
+//
+// ```python
+// x = tf.constant([5.0, 4.8, 6.8, np.inf, np.nan])
+// tf.math.is_finite(x) ==> [True, True, True, False, False]
+// ```
+func IsFinite(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "IsFinite",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns a tensor of zeros with the same shape and type as x.
+//
+// Arguments:
+//	x: a tensor of type T.
+//
+// Returns a tensor of the same shape and type as x but filled with zeros.
+func ZerosLike(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ZerosLike",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RetrieveTPUEmbeddingAdadeltaParametersAttr is an optional argument to RetrieveTPUEmbeddingAdadeltaParameters.
+type RetrieveTPUEmbeddingAdadeltaParametersAttr func(optionalAttr)
+
+// RetrieveTPUEmbeddingAdadeltaParametersTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func RetrieveTPUEmbeddingAdadeltaParametersTableId(value int64) RetrieveTPUEmbeddingAdadeltaParametersAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingAdadeltaParametersTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingAdadeltaParametersTableName(value string) RetrieveTPUEmbeddingAdadeltaParametersAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingAdadeltaParametersConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingAdadeltaParametersConfig(value string) RetrieveTPUEmbeddingAdadeltaParametersAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Retrieve Adadelta embedding parameters.
+//
+// An op that retrieves optimization parameters from embedding to host
+// memory. Must be preceded by a ConfigureTPUEmbeddingHost op that sets up
+// the correct embedding table configuration. For example, this op is
+// used to retrieve updated parameters before saving a checkpoint.
+//
+// Returns:
+//	parameters: Parameter parameters updated by the Adadelta optimization algorithm.
+//	accumulators: Parameter accumulators updated by the Adadelta optimization algorithm.
+//	updates: Parameter updates updated by the Adadelta optimization algorithm.
+func RetrieveTPUEmbeddingAdadeltaParameters(scope *Scope, num_shards int64, shard_id int64, optional ...RetrieveTPUEmbeddingAdadeltaParametersAttr) (parameters tf.Output, accumulators tf.Output, updates tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RetrieveTPUEmbeddingAdadeltaParameters",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// RegexReplaceAttr is an optional argument to RegexReplace.
+type RegexReplaceAttr func(optionalAttr)
+
+// RegexReplaceReplaceGlobal sets the optional replace_global attribute to value.
+//
+// value: If True, the replacement is global (that is, all matches of the `pattern` regular
+// expression in each input string are rewritten), otherwise the `rewrite`
+// substitution is only made for the first `pattern` match.
+// If not specified, defaults to true
+func RegexReplaceReplaceGlobal(value bool) RegexReplaceAttr {
+	return func(m optionalAttr) {
+		m["replace_global"] = value
+	}
+}
+
+// Replaces matches of the `pattern` regular expression in `input` with the
+// replacement string provided in `rewrite`.
+//
+// It follows the re2 syntax (https://github.com/google/re2/wiki/Syntax)
+//
+// Arguments:
+//	input: The text to be processed.
+//	pattern: The regular expression to be matched in the `input` strings.
+//	rewrite: The rewrite string to be substituted for the `pattern` expression where it is
+// matched in the `input` strings.
+//
+// Returns The text after applying pattern match and rewrite substitution.
+func RegexReplace(scope *Scope, input tf.Output, pattern tf.Output, rewrite tf.Output, optional ...RegexReplaceAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RegexReplace",
+		Input: []tf.Input{
+			input, pattern, rewrite,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ExperimentalRebatchDatasetAttr is an optional argument to ExperimentalRebatchDataset.
+type ExperimentalRebatchDatasetAttr func(optionalAttr)
+
+// ExperimentalRebatchDatasetUseFallback sets the optional use_fallback attribute to value.
+// If not specified, defaults to true
+func ExperimentalRebatchDatasetUseFallback(value bool) ExperimentalRebatchDatasetAttr {
+	return func(m optionalAttr) {
+		m["use_fallback"] = value
+	}
+}
+
+// Creates a dataset that changes the batch size.
+//
+// Creates a dataset that changes the batch size of the dataset to current batch
+// size // num_replicas.
+//
+// Arguments:
+//	input_dataset: A variant tensor representing the input dataset.
+//	num_replicas: A scalar representing the number of replicas to distribute this batch across. As
+// a result of this transformation the current batch size would end up being
+// divided  by this parameter.
+//
+//
+func ExperimentalRebatchDataset(scope *Scope, input_dataset tf.Output, num_replicas tf.Output, output_types []tf.DataType, output_shapes []tf.Shape, optional ...ExperimentalRebatchDatasetAttr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ExperimentalRebatchDataset",
+		Input: []tf.Input{
+			input_dataset, num_replicas,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Concatenates tensors along one dimension.
+//
+// Arguments:
+//	concat_dim: 0-D.  The dimension along which to concatenate.  Must be in the
+// range [0, rank(values)).
+//	values: The `N` Tensors to concatenate. Their ranks and types must match,
+// and their sizes must match in all dimensions except `concat_dim`.
+//
+// Returns A `Tensor` with the concatenation of values stacked along the
+// `concat_dim` dimension.  This tensor's shape matches that of `values` except
+// in `concat_dim` where it has the sum of the sizes.
+func Concat(scope *Scope, concat_dim tf.Output, values []tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Concat",
+		Input: []tf.Input{
+			concat_dim, tf.OutputList(values),
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResourceApplyPowerSignAttr is an optional argument to ResourceApplyPowerSign.
+type ResourceApplyPowerSignAttr func(optionalAttr)
+
+// ResourceApplyPowerSignUseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var and m tensors is
+// protected by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceApplyPowerSignUseLocking(value bool) ResourceApplyPowerSignAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// Update '*var' according to the AddSign update.
+//
+// m_t <- beta1 * m_{t-1} + (1 - beta1) * g
+// update <- exp(logbase * sign_decay * sign(g) * sign(m_t)) * g
+// variable <- variable - lr_t * update
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	m: Should be from a Variable().
+//	lr: Scaling factor. Must be a scalar.
+//	logbase: Must be a scalar.
+//	sign_decay: Must be a scalar.
+//	beta: Must be a scalar.
+//	grad: The gradient.
+//
+// Returns the created operation.
+func ResourceApplyPowerSign(scope *Scope, var_ tf.Output, m tf.Output, lr tf.Output, logbase tf.Output, sign_decay tf.Output, beta tf.Output, grad tf.Output, optional ...ResourceApplyPowerSignAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceApplyPowerSign",
+		Input: []tf.Input{
+			var_, m, lr, logbase, sign_decay, beta, grad,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Converts a tensor to a scalar predicate.
+//
+// Converts a tensor to a scalar predicate with the following rules:
+//
+// - For 0D tensors, truthiness is determined by comparing against a "zero"
+//   value. For numerical types it is the obvious zero. For strings it is the
+//   empty string.
+//
+// - For >0D tensors, truthiness is determined by looking at the number of
+//   elements. If has zero elements, then the result is false. Otherwise the
+//   result is true.
+//
+// This matches the behavior of If and While for determining if a tensor counts
+// as true/false for a branch condition.
+func ToBool(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ToBool",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// GenerateBoundingBoxProposalsAttr is an optional argument to GenerateBoundingBoxProposals.
+type GenerateBoundingBoxProposalsAttr func(optionalAttr)
+
+// GenerateBoundingBoxProposalsPostNmsTopn sets the optional post_nms_topn attribute to value.
+//
+// value: An integer. Maximum number of rois in the output.
+// If not specified, defaults to 300
+func GenerateBoundingBoxProposalsPostNmsTopn(value int64) GenerateBoundingBoxProposalsAttr {
+	return func(m optionalAttr) {
+		m["post_nms_topn"] = value
+	}
+}
+
+// This op produces Region of Interests from given bounding boxes(bbox_deltas) encoded wrt anchors according to eq.2 in arXiv:1506.01497
+//
+//       The op selects top `pre_nms_topn` scoring boxes, decodes them with respect to anchors,
+//       applies non-maximal suppression on overlapping boxes with higher than
+//       `nms_threshold` intersection-over-union (iou) value, discarding boxes where shorter
+//       side is less than `min_size`.
+//       Inputs:
+//       `scores`: A 4D tensor of shape [Batch, Height, Width, Num Anchors] containing the scores per anchor at given position
+//       `bbox_deltas`: is a tensor of shape [Batch, Height, Width, 4 x Num Anchors] boxes encoded to each anchor
+//       `anchors`: A 1D tensor of shape [4 x Num Anchors], representing the anchors.
+//       Outputs:
+//       `rois`: output RoIs, a 3D tensor of shape [Batch, post_nms_topn, 4], padded by 0 if less than post_nms_topn candidates found.
+//       `roi_probabilities`: probability scores of each roi in 'rois', a 2D tensor of shape [Batch,post_nms_topn], padded with 0 if needed, sorted by scores.
+//
+// Arguments:
+//	scores: A 4-D float tensor of shape `[num_images, height, width, num_achors]` containing scores of the boxes for given anchors, can be unsorted.
+//	bbox_deltas: A 4-D float tensor of shape `[num_images, height, width, 4 x num_anchors]`. encoding boxes with respec to each anchor.
+// Coordinates are given in the form [dy, dx, dh, dw].
+//	image_info: A 2-D float tensor of shape `[num_images, 5]` containing image information Height, Width, Scale.
+//	anchors: A 2-D float tensor of shape `[num_anchors, 4]` describing the anchor boxes. Boxes are formatted in the form [y1, x1, y2, x2].
+//	nms_threshold: A scalar float tensor for non-maximal-suppression threshold.
+//	pre_nms_topn: A scalar int tensor for the number of top scoring boxes to be used as input.
+//	min_size: A scalar float tensor. Any box that has a smaller size than min_size will be discarded.
+//
+// Returns:
+//	rois: A 3-D float tensor of shape `[num_images,post_nms_topn,4]` representing the selected
+// region of interest boxes. Sorted in descending order in scores.
+//	roi_probabilities: A 2-D float tensor of shape `[num_images, post_nms_topn]` representing the score of the
+// region of interest box in `rois` tensor at the same index.
+func GenerateBoundingBoxProposals(scope *Scope, scores tf.Output, bbox_deltas tf.Output, image_info tf.Output, anchors tf.Output, nms_threshold tf.Output, pre_nms_topn tf.Output, min_size tf.Output, optional ...GenerateBoundingBoxProposalsAttr) (rois tf.Output, roi_probabilities tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "GenerateBoundingBoxProposals",
+		Input: []tf.Input{
+			scores, bbox_deltas, image_info, anchors, nms_threshold, pre_nms_topn, min_size,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// InitializeTableFromTextFileV2Attr is an optional argument to InitializeTableFromTextFileV2.
+type InitializeTableFromTextFileV2Attr func(optionalAttr)
+
+// InitializeTableFromTextFileV2VocabSize sets the optional vocab_size attribute to value.
+//
+// value: Number of elements of the file, use -1 if unknown.
+// If not specified, defaults to -1
+//
+// REQUIRES: value >= -1
+func InitializeTableFromTextFileV2VocabSize(value int64) InitializeTableFromTextFileV2Attr {
+	return func(m optionalAttr) {
+		m["vocab_size"] = value
+	}
+}
+
+// InitializeTableFromTextFileV2Delimiter sets the optional delimiter attribute to value.
+//
+// value: Delimiter to separate fields in a line.
+// If not specified, defaults to "\t"
+func InitializeTableFromTextFileV2Delimiter(value string) InitializeTableFromTextFileV2Attr {
+	return func(m optionalAttr) {
+		m["delimiter"] = value
+	}
+}
+
+// Initializes a table from a text file.
+//
+// It inserts one key-value pair into the table for each line of the file.
+// The key and value is extracted from the whole line content, elements from the
+// split line based on `delimiter` or the line number (starting from zero).
+// Where to extract the key and value from a line is specified by `key_index` and
+// `value_index`.
+//
+// - A value of -1 means use the line number(starting from zero), expects `int64`.
+// - A value of -2 means use the whole line content, expects `string`.
+// - A value >= 0 means use the index (starting at zero) of the split line based
+//   on `delimiter`.
+//
+// Arguments:
+//	table_handle: Handle to a table which will be initialized.
+//	filename: Filename of a vocabulary text file.
+//	key_index: Column index in a line to get the table `key` values from.
+//	value_index: Column index that represents information of a line to get the table
+// `value` values from.
+//
+// Returns the created operation.
+func InitializeTableFromTextFileV2(scope *Scope, table_handle tf.Output, filename tf.Output, key_index int64, value_index int64, optional ...InitializeTableFromTextFileV2Attr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"key_index": key_index, "value_index": value_index}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "InitializeTableFromTextFileV2",
+		Input: []tf.Input{
+			table_handle, filename,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Returns an element-wise indication of the sign of a number.
+//
+// `y = sign(x) = -1` if `x < 0`; 0 if `x == 0`; 1 if `x > 0`.
+//
+// For complex numbers, `y = sign(x) = x / |x|` if `x != 0`, otherwise `y = 0`.
+//
+// Example usage:
+// >>> tf.math.sign([0., 2., -3.])
+// <tf.Tensor: shape=(3,), dtype=float32, numpy=array([ 0.,  1., -1.], dtype=float32)>
+func Sign(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Sign",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResourceApplyAddSignAttr is an optional argument to ResourceApplyAddSign.
+type ResourceApplyAddSignAttr func(optionalAttr)
+
+// ResourceApplyAddSignUseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var and m tensors is
+// protected by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceApplyAddSignUseLocking(value bool) ResourceApplyAddSignAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// Update '*var' according to the AddSign update.
+//
+// m_t <- beta1 * m_{t-1} + (1 - beta1) * g
+// update <- (alpha + sign_decay * sign(g) *sign(m)) * g
+// variable <- variable - lr_t * update
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	m: Should be from a Variable().
+//	lr: Scaling factor. Must be a scalar.
+//	alpha: Must be a scalar.
+//	sign_decay: Must be a scalar.
+//	beta: Must be a scalar.
+//	grad: The gradient.
+//
+// Returns the created operation.
+func ResourceApplyAddSign(scope *Scope, var_ tf.Output, m tf.Output, lr tf.Output, alpha tf.Output, sign_decay tf.Output, beta tf.Output, grad tf.Output, optional ...ResourceApplyAddSignAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceApplyAddSign",
+		Input: []tf.Input{
+			var_, m, lr, alpha, sign_decay, beta, grad,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Returns the number of work units this Reader has finished processing.
+//
+// Arguments:
+//	reader_handle: Handle to a Reader.
+func ReaderNumWorkUnitsCompletedV2(scope *Scope, reader_handle tf.Output) (units_completed tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ReaderNumWorkUnitsCompletedV2",
+		Input: []tf.Input{
+			reader_handle,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// FractionalMaxPoolAttr is an optional argument to FractionalMaxPool.
+type FractionalMaxPoolAttr func(optionalAttr)
+
+// FractionalMaxPoolPseudoRandom sets the optional pseudo_random attribute to value.
+//
+// value: When set to True, generates the pooling sequence in a
+// pseudorandom fashion, otherwise, in a random fashion. Check paper [Benjamin
+// Graham, Fractional Max-Pooling](http://arxiv.org/abs/1412.6071) for
+// difference between pseudorandom and random.
+// If not specified, defaults to false
+func FractionalMaxPoolPseudoRandom(value bool) FractionalMaxPoolAttr {
+	return func(m optionalAttr) {
+		m["pseudo_random"] = value
+	}
+}
+
+// FractionalMaxPoolOverlapping sets the optional overlapping attribute to value.
+//
+// value: When set to True, it means when pooling, the values at the boundary
+// of adjacent pooling cells are used by both cells. For example:
+//
+// `index  0  1  2  3  4`
+//
+// `value  20 5  16 3  7`
+//
+// If the pooling sequence is [0, 2, 4], then 16, at index 2 will be used twice.
+// The result would be [20, 16] for fractional max pooling.
+// If not specified, defaults to false
+func FractionalMaxPoolOverlapping(value bool) FractionalMaxPoolAttr {
+	return func(m optionalAttr) {
+		m["overlapping"] = value
+	}
+}
+
+// FractionalMaxPoolDeterministic sets the optional deterministic attribute to value.
+//
+// value: When set to True, a fixed pooling region will be used when
+// iterating over a FractionalMaxPool node in the computation graph. Mainly used
+// in unit test to make FractionalMaxPool deterministic.
+// If not specified, defaults to false
+func FractionalMaxPoolDeterministic(value bool) FractionalMaxPoolAttr {
+	return func(m optionalAttr) {
+		m["deterministic"] = value
+	}
+}
+
+// FractionalMaxPoolSeed sets the optional seed attribute to value.
+//
+// value: If either seed or seed2 are set to be non-zero, the random number
+// generator is seeded by the given seed.  Otherwise, it is seeded by a
+// random seed.
+// If not specified, defaults to 0
+func FractionalMaxPoolSeed(value int64) FractionalMaxPoolAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// FractionalMaxPoolSeed2 sets the optional seed2 attribute to value.
+//
+// value: An second seed to avoid seed collision.
+// If not specified, defaults to 0
+func FractionalMaxPoolSeed2(value int64) FractionalMaxPoolAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Performs fractional max pooling on the input.
+//
+// Fractional max pooling is slightly different than regular max pooling.  In
+// regular max pooling, you downsize an input set by taking the maximum value of
+// smaller N x N subsections of the set (often 2x2), and try to reduce the set by
+// a factor of N, where N is an integer.  Fractional max pooling, as you might
+// expect from the word "fractional", means that the overall reduction ratio N
+// does not have to be an integer.
+//
+// The sizes of the pooling regions are generated randomly but are fairly uniform.
+// For example, let's look at the height dimension, and the constraints on the
+// list of rows that will be pool boundaries.
+//
+// First we define the following:
+//
+// 1.  input_row_length : the number of rows from the input set
+// 2.  output_row_length : which will be smaller than the input
+// 3.  alpha = input_row_length / output_row_length : our reduction ratio
+// 4.  K = floor(alpha)
+// 5.  row_pooling_sequence : this is the result list of pool boundary rows
+//
+// Then, row_pooling_sequence should satisfy:
+//
+// 1.  a[0] = 0 : the first value of the sequence is 0
+// 2.  a[end] = input_row_length : the last value of the sequence is the size
+// 3.  K <= (a[i+1] - a[i]) <= K+1 : all intervals are K or K+1 size
+// 4.  length(row_pooling_sequence) = output_row_length+1
+//
+// For more details on fractional max pooling, see this paper:
+// [Benjamin Graham, Fractional Max-Pooling](http://arxiv.org/abs/1412.6071)
+//
+// Arguments:
+//	value: 4-D with shape `[batch, height, width, channels]`.
+//	pooling_ratio: Pooling ratio for each dimension of `value`, currently only
+// supports row and col dimension and should be >= 1.0. For example, a valid
+// pooling ratio looks like [1.0, 1.44, 1.73, 1.0]. The first and last elements
+// must be 1.0 because we don't allow pooling on batch and channels
+// dimensions. 1.44 and 1.73 are pooling ratio on height and width dimensions
+// respectively.
+//
+// Returns:
+//	output: output tensor after fractional max pooling.
+//	row_pooling_sequence: row pooling sequence, needed to calculate gradient.
+//	col_pooling_sequence: column pooling sequence, needed to calculate gradient.
+func FractionalMaxPool(scope *Scope, value tf.Output, pooling_ratio []float32, optional ...FractionalMaxPoolAttr) (output tf.Output, row_pooling_sequence tf.Output, col_pooling_sequence tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"pooling_ratio": pooling_ratio}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "FractionalMaxPool",
+		Input: []tf.Input{
+			value,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Computes the reciprocal of x element-wise.
+//
+// I.e., \\(y = 1 / x\\).
+func Reciprocal(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Reciprocal",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// LoadTPUEmbeddingAdagradParametersGradAccumDebugAttr is an optional argument to LoadTPUEmbeddingAdagradParametersGradAccumDebug.
+type LoadTPUEmbeddingAdagradParametersGradAccumDebugAttr func(optionalAttr)
+
+// LoadTPUEmbeddingAdagradParametersGradAccumDebugTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func LoadTPUEmbeddingAdagradParametersGradAccumDebugTableId(value int64) LoadTPUEmbeddingAdagradParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// LoadTPUEmbeddingAdagradParametersGradAccumDebugTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingAdagradParametersGradAccumDebugTableName(value string) LoadTPUEmbeddingAdagradParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// LoadTPUEmbeddingAdagradParametersGradAccumDebugConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingAdagradParametersGradAccumDebugConfig(value string) LoadTPUEmbeddingAdagradParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Load Adagrad embedding parameters with debug support.
+//
+// An op that loads optimization parameters into HBM for embedding. Must be
+// preceded by a ConfigureTPUEmbeddingHost op that sets up the correct
+// embedding table configuration. For example, this op is used to install
+// parameters that are loaded from a checkpoint before a training loop is
+// executed.
+//
+// Arguments:
+//	parameters: Value of parameters used in the Adagrad optimization algorithm.
+//	accumulators: Value of accumulators used in the Adagrad optimization algorithm.
+//	gradient_accumulators: Value of gradient_accumulators used in the Adagrad optimization algorithm.
+//
+//
+//
+// Returns the created operation.
+func LoadTPUEmbeddingAdagradParametersGradAccumDebug(scope *Scope, parameters tf.Output, accumulators tf.Output, gradient_accumulators tf.Output, num_shards int64, shard_id int64, optional ...LoadTPUEmbeddingAdagradParametersGradAccumDebugAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LoadTPUEmbeddingAdagradParametersGradAccumDebug",
+		Input: []tf.Input{
+			parameters, accumulators, gradient_accumulators,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Strip leading and trailing whitespaces from the Tensor.
+//
+// Arguments:
+//	input: A string `Tensor` of any shape.
+//
+// Returns A string `Tensor` of the same shape as the input.
+//
+// Examples:
+//
+// >>> tf.strings.strip(["\nTensorFlow", "     The python library    "]).numpy()
+// array([b'TensorFlow', b'The python library'], dtype=object)
+func StringStrip(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "StringStrip",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the minimum along segments of a tensor.
+//
+// Read
+// [the section on segmentation](https://tensorflow.org/api_docs/python/tf/math#Segmentation)
+// for an explanation of segments.
+//
+// Computes a tensor such that
+// \\(output_i = \min_j(data_j)\\) where `min` is over `j` such
+// that `segment_ids[j] == i`.
+//
+// If the min is empty for a given segment ID `i`, `output[i] = 0`.
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src="https://www.tensorflow.org/images/SegmentMin.png" alt>
+// </div>
+//
+// For example:
+//
+// ```
+// c = tf.constant([[1,2,3,4], [4, 3, 2, 1], [5,6,7,8]])
+// tf.segment_min(c, tf.constant([0, 0, 1]))
+// # ==> [[1, 2, 2, 1],
+// #      [5, 6, 7, 8]]
+// ```
+//
+// Arguments:
+//
+//	segment_ids: A 1-D tensor whose size is equal to the size of `data`'s
+// first dimension.  Values should be sorted and can be repeated.
+//
+// Returns Has same shape as data, except for dimension 0 which
+// has size `k`, the number of segments.
+func SegmentMin(scope *Scope, data tf.Output, segment_ids tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SegmentMin",
+		Input: []tf.Input{
+			data, segment_ids,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Execute a sub graph on a remote processor.
+//
+// The graph specifications(such as graph itself, input tensors and output names)
+// are stored as a serialized protocol buffer of RemoteFusedGraphExecuteInfo
+// as serialized_remote_fused_graph_execute_info.
+// The specifications will be passed to a dedicated registered
+// remote fused graph executor.  The executor will send the graph specifications
+// to a remote processor and execute that graph.  The execution results
+// will be passed to consumer nodes as outputs of this node.
+//
+// Arguments:
+//	inputs: Arbitrary number of tensors with arbitrary data types
+//
+//	serialized_remote_fused_graph_execute_info: Serialized protocol buffer
+// of RemoteFusedGraphExecuteInfo which contains graph specifications.
+//
+// Returns Arbitrary number of tensors with arbitrary data types
+func RemoteFusedGraphExecute(scope *Scope, inputs []tf.Output, Toutputs []tf.DataType, serialized_remote_fused_graph_execute_info string) (outputs []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"Toutputs": Toutputs, "serialized_remote_fused_graph_execute_info": serialized_remote_fused_graph_execute_info}
+	opspec := tf.OpSpec{
+		Type: "RemoteFusedGraphExecute",
+		Input: []tf.Input{
+			tf.OutputList(inputs),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if outputs, idx, err = makeOutputList(op, idx, "outputs"); err != nil {
+		scope.UpdateErr("RemoteFusedGraphExecute", err)
+		return
+	}
+	return outputs
+}
+
+// LoadTPUEmbeddingMDLAdagradLightParametersAttr is an optional argument to LoadTPUEmbeddingMDLAdagradLightParameters.
+type LoadTPUEmbeddingMDLAdagradLightParametersAttr func(optionalAttr)
+
+// LoadTPUEmbeddingMDLAdagradLightParametersTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func LoadTPUEmbeddingMDLAdagradLightParametersTableId(value int64) LoadTPUEmbeddingMDLAdagradLightParametersAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// LoadTPUEmbeddingMDLAdagradLightParametersTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingMDLAdagradLightParametersTableName(value string) LoadTPUEmbeddingMDLAdagradLightParametersAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// LoadTPUEmbeddingMDLAdagradLightParametersConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingMDLAdagradLightParametersConfig(value string) LoadTPUEmbeddingMDLAdagradLightParametersAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Load MDL Adagrad Light embedding parameters.
+//
+// An op that loads optimization parameters into HBM for embedding. Must be
+// preceded by a ConfigureTPUEmbeddingHost op that sets up the correct
+// embedding table configuration. For example, this op is used to install
+// parameters that are loaded from a checkpoint before a training loop is
+// executed.
+//
+// Arguments:
+//	parameters: Value of parameters used in the MDL Adagrad Light optimization algorithm.
+//	accumulators: Value of accumulators used in the MDL Adagrad Light optimization algorithm.
+//	weights: Value of weights used in the MDL Adagrad Light optimization algorithm.
+//	benefits: Value of benefits used in the MDL Adagrad Light optimization algorithm.
+//
+//
+//
+// Returns the created operation.
+func LoadTPUEmbeddingMDLAdagradLightParameters(scope *Scope, parameters tf.Output, accumulators tf.Output, weights tf.Output, benefits tf.Output, num_shards int64, shard_id int64, optional ...LoadTPUEmbeddingMDLAdagradLightParametersAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LoadTPUEmbeddingMDLAdagradLightParameters",
+		Input: []tf.Input{
+			parameters, accumulators, weights, benefits,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// MapPeekAttr is an optional argument to MapPeek.
+type MapPeekAttr func(optionalAttr)
+
+// MapPeekCapacity sets the optional capacity attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func MapPeekCapacity(value int64) MapPeekAttr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// MapPeekMemoryLimit sets the optional memory_limit attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func MapPeekMemoryLimit(value int64) MapPeekAttr {
+	return func(m optionalAttr) {
+		m["memory_limit"] = value
+	}
+}
+
+// MapPeekContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func MapPeekContainer(value string) MapPeekAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// MapPeekSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func MapPeekSharedName(value string) MapPeekAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Op peeks at the values at the specified key.  If the
+//
+// underlying container does not contain this key
+// this op will block until it does.
+func MapPeek(scope *Scope, key tf.Output, indices tf.Output, dtypes []tf.DataType, optional ...MapPeekAttr) (values []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MapPeek",
+		Input: []tf.Input{
+			key, indices,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if values, idx, err = makeOutputList(op, idx, "values"); err != nil {
+		scope.UpdateErr("MapPeek", err)
+		return
+	}
+	return values
+}
+
+// RetrieveTPUEmbeddingCenteredRMSPropParametersAttr is an optional argument to RetrieveTPUEmbeddingCenteredRMSPropParameters.
+type RetrieveTPUEmbeddingCenteredRMSPropParametersAttr func(optionalAttr)
+
+// RetrieveTPUEmbeddingCenteredRMSPropParametersTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func RetrieveTPUEmbeddingCenteredRMSPropParametersTableId(value int64) RetrieveTPUEmbeddingCenteredRMSPropParametersAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingCenteredRMSPropParametersTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingCenteredRMSPropParametersTableName(value string) RetrieveTPUEmbeddingCenteredRMSPropParametersAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingCenteredRMSPropParametersConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingCenteredRMSPropParametersConfig(value string) RetrieveTPUEmbeddingCenteredRMSPropParametersAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Retrieve centered RMSProp embedding parameters.
+//
+// An op that retrieves optimization parameters from embedding to host
+// memory. Must be preceded by a ConfigureTPUEmbeddingHost op that sets up
+// the correct embedding table configuration. For example, this op is
+// used to retrieve updated parameters before saving a checkpoint.
+//
+// Returns:
+//	parameters: Parameter parameters updated by the centered RMSProp optimization algorithm.
+//	ms: Parameter ms updated by the centered RMSProp optimization algorithm.
+//	mom: Parameter mom updated by the centered RMSProp optimization algorithm.
+//	mg: Parameter mg updated by the centered RMSProp optimization algorithm.
+func RetrieveTPUEmbeddingCenteredRMSPropParameters(scope *Scope, num_shards int64, shard_id int64, optional ...RetrieveTPUEmbeddingCenteredRMSPropParametersAttr) (parameters tf.Output, ms tf.Output, mom tf.Output, mg tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RetrieveTPUEmbeddingCenteredRMSPropParameters",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3)
+}
+
+// Transforms a vector of brain.Example protos (as strings) into typed tensors.
+//
+// Arguments:
+//	serialized: A vector containing a batch of binary serialized Example protos.
+//	names: A vector containing the names of the serialized protos.
+// May contain, for example, table key (descriptive) names for the
+// corresponding serialized protos.  These are purely useful for debugging
+// purposes, and the presence of values here has no effect on the output.
+// May also be an empty vector if no names are available.
+// If non-empty, this vector must be the same length as "serialized".
+//	sparse_keys: A list of Nsparse string Tensors (scalars).
+// The keys expected in the Examples' features associated with sparse values.
+//	dense_keys: A list of Ndense string Tensors (scalars).
+// The keys expected in the Examples' features associated with dense values.
+//	dense_defaults: A list of Ndense Tensors (some may be empty).
+// dense_defaults[j] provides default values
+// when the example's feature_map lacks dense_key[j].  If an empty Tensor is
+// provided for dense_defaults[j], then the Feature dense_keys[j] is required.
+// The input type is inferred from dense_defaults[j], even when it's empty.
+// If dense_defaults[j] is not empty, and dense_shapes[j] is fully defined,
+// then the shape of dense_defaults[j] must match that of dense_shapes[j].
+// If dense_shapes[j] has an undefined major dimension (variable strides dense
+// feature), dense_defaults[j] must contain a single element:
+// the padding element.
+//	sparse_types: A list of Nsparse types; the data types of data in each Feature
+// given in sparse_keys.
+// Currently the ParseExample supports DT_FLOAT (FloatList),
+// DT_INT64 (Int64List), and DT_STRING (BytesList).
+//	dense_shapes: A list of Ndense shapes; the shapes of data in each Feature
+// given in dense_keys.
+// The number of elements in the Feature corresponding to dense_key[j]
+// must always equal dense_shapes[j].NumEntries().
+// If dense_shapes[j] == (D0, D1, ..., DN) then the shape of output
+// Tensor dense_values[j] will be (|serialized|, D0, D1, ..., DN):
+// The dense outputs are just the inputs row-stacked by batch.
+// This works for dense_shapes[j] = (-1, D1, ..., DN).  In this case
+// the shape of the output Tensor dense_values[j] will be
+// (|serialized|, M, D1, .., DN), where M is the maximum number of blocks
+// of elements of length D1 * .... * DN, across all minibatch entries
+// in the input.  Any minibatch entry with less than M blocks of elements of
+// length D1 * ... * DN will be padded with the corresponding default_value
+// scalar element along the second dimension.
+func ParseExample(scope *Scope, serialized tf.Output, names tf.Output, sparse_keys []tf.Output, dense_keys []tf.Output, dense_defaults []tf.Output, sparse_types []tf.DataType, dense_shapes []tf.Shape) (sparse_indices []tf.Output, sparse_values []tf.Output, sparse_shapes []tf.Output, dense_values []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"sparse_types": sparse_types, "dense_shapes": dense_shapes}
+	opspec := tf.OpSpec{
+		Type: "ParseExample",
+		Input: []tf.Input{
+			serialized, names, tf.OutputList(sparse_keys), tf.OutputList(dense_keys), tf.OutputList(dense_defaults),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if sparse_indices, idx, err = makeOutputList(op, idx, "sparse_indices"); err != nil {
+		scope.UpdateErr("ParseExample", err)
+		return
+	}
+	if sparse_values, idx, err = makeOutputList(op, idx, "sparse_values"); err != nil {
+		scope.UpdateErr("ParseExample", err)
+		return
+	}
+	if sparse_shapes, idx, err = makeOutputList(op, idx, "sparse_shapes"); err != nil {
+		scope.UpdateErr("ParseExample", err)
+		return
+	}
+	if dense_values, idx, err = makeOutputList(op, idx, "dense_values"); err != nil {
+		scope.UpdateErr("ParseExample", err)
+		return
+	}
+	return sparse_indices, sparse_values, sparse_shapes, dense_values
+}
+
+// DatasetToGraphAttr is an optional argument to DatasetToGraph.
+type DatasetToGraphAttr func(optionalAttr)
+
+// DatasetToGraphStatefulWhitelist sets the optional stateful_whitelist attribute to value.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func DatasetToGraphStatefulWhitelist(value []string) DatasetToGraphAttr {
+	return func(m optionalAttr) {
+		m["stateful_whitelist"] = value
+	}
+}
+
+// DatasetToGraphAllowStateful sets the optional allow_stateful attribute to value.
+// If not specified, defaults to false
+func DatasetToGraphAllowStateful(value bool) DatasetToGraphAttr {
+	return func(m optionalAttr) {
+		m["allow_stateful"] = value
+	}
+}
+
+// DatasetToGraphStripDeviceAssignment sets the optional strip_device_assignment attribute to value.
+// If not specified, defaults to false
+func DatasetToGraphStripDeviceAssignment(value bool) DatasetToGraphAttr {
+	return func(m optionalAttr) {
+		m["strip_device_assignment"] = value
+	}
+}
+
+// Returns a serialized GraphDef representing `input_dataset`.
+//
+// Returns a graph representation for `input_dataset`.
+//
+// Arguments:
+//	input_dataset: A variant tensor representing the dataset to return the graph representation for.
+//
+// Returns The graph representation of the dataset (as serialized GraphDef).
+func DatasetToGraph(scope *Scope, input_dataset tf.Output, optional ...DatasetToGraphAttr) (graph tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DatasetToGraph",
+		Input: []tf.Input{
+			input_dataset,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResourceSparseApplyAdadeltaAttr is an optional argument to ResourceSparseApplyAdadelta.
+type ResourceSparseApplyAdadeltaAttr func(optionalAttr)
+
+// ResourceSparseApplyAdadeltaUseLocking sets the optional use_locking attribute to value.
+//
+// value: If True, updating of the var and accum tensors will be protected by
+// a lock; otherwise the behavior is undefined, but may exhibit less contention.
+// If not specified, defaults to false
+func ResourceSparseApplyAdadeltaUseLocking(value bool) ResourceSparseApplyAdadeltaAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// var: Should be from a Variable().
+//
+// Arguments:
+//
+//	accum: Should be from a Variable().
+//	accum_update: : Should be from a Variable().
+//	lr: Learning rate. Must be a scalar.
+//	rho: Decay factor. Must be a scalar.
+//	epsilon: Constant factor. Must be a scalar.
+//	grad: The gradient.
+//	indices: A vector of indices into the first dimension of var and accum.
+//
+// Returns the created operation.
+func ResourceSparseApplyAdadelta(scope *Scope, var_ tf.Output, accum tf.Output, accum_update tf.Output, lr tf.Output, rho tf.Output, epsilon tf.Output, grad tf.Output, indices tf.Output, optional ...ResourceSparseApplyAdadeltaAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceSparseApplyAdadelta",
+		Input: []tf.Input{
+			var_, accum, accum_update, lr, rho, epsilon, grad, indices,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Computes sigmoid of `x` element-wise.
+//
+// Specifically, `y = 1 / (1 + exp(-x))`.
+func Sigmoid(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Sigmoid",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RetrieveTPUEmbeddingADAMParametersGradAccumDebugAttr is an optional argument to RetrieveTPUEmbeddingADAMParametersGradAccumDebug.
+type RetrieveTPUEmbeddingADAMParametersGradAccumDebugAttr func(optionalAttr)
+
+// RetrieveTPUEmbeddingADAMParametersGradAccumDebugTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func RetrieveTPUEmbeddingADAMParametersGradAccumDebugTableId(value int64) RetrieveTPUEmbeddingADAMParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingADAMParametersGradAccumDebugTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingADAMParametersGradAccumDebugTableName(value string) RetrieveTPUEmbeddingADAMParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingADAMParametersGradAccumDebugConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingADAMParametersGradAccumDebugConfig(value string) RetrieveTPUEmbeddingADAMParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Retrieve ADAM embedding parameters with debug support.
+//
+// An op that retrieves optimization parameters from embedding to host
+// memory. Must be preceded by a ConfigureTPUEmbeddingHost op that sets up
+// the correct embedding table configuration. For example, this op is
+// used to retrieve updated parameters before saving a checkpoint.
+//
+// Returns:
+//	parameters: Parameter parameters updated by the ADAM optimization algorithm.
+//	momenta: Parameter momenta updated by the ADAM optimization algorithm.
+//	velocities: Parameter velocities updated by the ADAM optimization algorithm.
+//	gradient_accumulators: Parameter gradient_accumulators updated by the ADAM optimization algorithm.
+func RetrieveTPUEmbeddingADAMParametersGradAccumDebug(scope *Scope, num_shards int64, shard_id int64, optional ...RetrieveTPUEmbeddingADAMParametersGradAccumDebugAttr) (parameters tf.Output, momenta tf.Output, velocities tf.Output, gradient_accumulators tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RetrieveTPUEmbeddingADAMParametersGradAccumDebug",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3)
+}
+
+// ResourceApplyAdamAttr is an optional argument to ResourceApplyAdam.
+type ResourceApplyAdamAttr func(optionalAttr)
+
+// ResourceApplyAdamUseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var, m, and v tensors will be protected
+// by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceApplyAdamUseLocking(value bool) ResourceApplyAdamAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// ResourceApplyAdamUseNesterov sets the optional use_nesterov attribute to value.
+//
+// value: If `True`, uses the nesterov update.
+// If not specified, defaults to false
+func ResourceApplyAdamUseNesterov(value bool) ResourceApplyAdamAttr {
+	return func(m optionalAttr) {
+		m["use_nesterov"] = value
+	}
+}
+
+// Update '*var' according to the Adam algorithm.
+//
+// $$\text{lr}_t := \mathrm{learning_rate} * \sqrt{1 - \beta_2^t} / (1 - \beta_1^t)$$
+// $$m_t := \beta_1 * m_{t-1} + (1 - \beta_1) * g$$
+// $$v_t := \beta_2 * v_{t-1} + (1 - \beta_2) * g * g$$
+// $$\text{variable} := \text{variable} - \text{lr}_t * m_t / (\sqrt{v_t} + \epsilon)$$
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	m: Should be from a Variable().
+//	v: Should be from a Variable().
+//	beta1_power: Must be a scalar.
+//	beta2_power: Must be a scalar.
+//	lr: Scaling factor. Must be a scalar.
+//	beta1: Momentum factor. Must be a scalar.
+//	beta2: Momentum factor. Must be a scalar.
+//	epsilon: Ridge term. Must be a scalar.
+//	grad: The gradient.
+//
+// Returns the created operation.
+func ResourceApplyAdam(scope *Scope, var_ tf.Output, m tf.Output, v tf.Output, beta1_power tf.Output, beta2_power tf.Output, lr tf.Output, beta1 tf.Output, beta2 tf.Output, epsilon tf.Output, grad tf.Output, optional ...ResourceApplyAdamAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceApplyAdam",
+		Input: []tf.Input{
+			var_, m, v, beta1_power, beta2_power, lr, beta1, beta2, epsilon, grad,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// CumsumAttr is an optional argument to Cumsum.
+type CumsumAttr func(optionalAttr)
+
+// CumsumExclusive sets the optional exclusive attribute to value.
+//
+// value: If `True`, perform exclusive cumsum.
+// If not specified, defaults to false
+func CumsumExclusive(value bool) CumsumAttr {
+	return func(m optionalAttr) {
+		m["exclusive"] = value
+	}
+}
+
+// CumsumReverse sets the optional reverse attribute to value.
+//
+// value: A `bool` (default: False).
+// If not specified, defaults to false
+func CumsumReverse(value bool) CumsumAttr {
+	return func(m optionalAttr) {
+		m["reverse"] = value
+	}
+}
+
+// Compute the cumulative sum of the tensor `x` along `axis`.
+//
+// By default, this op performs an inclusive cumsum, which means that the first
+// element of the input is identical to the first element of the output:
+//
+// ```python
+// tf.cumsum([a, b, c])  # => [a, a + b, a + b + c]
+// ```
+//
+// By setting the `exclusive` kwarg to `True`, an exclusive cumsum is
+// performed instead:
+//
+// ```python
+// tf.cumsum([a, b, c], exclusive=True)  # => [0, a, a + b]
+// ```
+//
+// By setting the `reverse` kwarg to `True`, the cumsum is performed in the
+// opposite direction:
+//
+// ```python
+// tf.cumsum([a, b, c], reverse=True)  # => [a + b + c, b + c, c]
+// ```
+//
+// This is more efficient than using separate `tf.reverse` ops.
+//
+// The `reverse` and `exclusive` kwargs can also be combined:
+//
+// ```python
+// tf.cumsum([a, b, c], exclusive=True, reverse=True)  # => [b + c, c, 0]
+// ```
+//
+// Arguments:
+//	x: A `Tensor`. Must be one of the following types: `float32`, `float64`,
+// `int64`, `int32`, `uint8`, `uint16`, `int16`, `int8`, `complex64`,
+// `complex128`, `qint8`, `quint8`, `qint32`, `half`.
+//	axis: A `Tensor` of type `int32` (default: 0). Must be in the range
+// `[-rank(x), rank(x))`.
+func Cumsum(scope *Scope, x tf.Output, axis tf.Output, optional ...CumsumAttr) (out tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Cumsum",
+		Input: []tf.Input{
+			x, axis,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Performs gradient updates of embedding tables.
+//
+// Arguments:
+//	inputs: A TensorList of gradients with which to update embedding tables.
+// This argument has the same length and shapes as the return value of
+// RecvTPUEmbeddingActivations, but contains gradients of the model's loss
+// with respect to the embedding activations. The embedding tables are updated
+// from these gradients via the optimizer specified in the TPU embedding
+// configuration given to tpu.initialize_system.
+//	learning_rates: A TensorList of float32 scalars, one for each dynamic learning
+// rate tag: see the comments in
+// //third_party/tensorflow/core/protobuf/tpu/optimization_parameters.proto.
+// Multiple tables can share the same dynamic learning rate tag as specified
+// in the configuration. If the learning rates for all tables are constant,
+// this list should be empty.
+//	config: Serialized TPUEmbeddingConfiguration proto.
+//
+// Returns the created operation.
+func SendTPUEmbeddingGradients(scope *Scope, inputs []tf.Output, learning_rates []tf.Output, config string) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"config": config}
+	opspec := tf.OpSpec{
+		Type: "SendTPUEmbeddingGradients",
+		Input: []tf.Input{
+			tf.OutputList(inputs), tf.OutputList(learning_rates),
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// ResourceApplyKerasMomentumAttr is an optional argument to ResourceApplyKerasMomentum.
+type ResourceApplyKerasMomentumAttr func(optionalAttr)
+
+// ResourceApplyKerasMomentumUseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var and accum tensors will be protected
+// by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceApplyKerasMomentumUseLocking(value bool) ResourceApplyKerasMomentumAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// ResourceApplyKerasMomentumUseNesterov sets the optional use_nesterov attribute to value.
+//
+// value: If `True`, the tensor passed to compute grad will be
+// var + momentum * accum, so in the end, the var you get is actually
+// var + momentum * accum.
+// If not specified, defaults to false
+func ResourceApplyKerasMomentumUseNesterov(value bool) ResourceApplyKerasMomentumAttr {
+	return func(m optionalAttr) {
+		m["use_nesterov"] = value
+	}
+}
+
+// Update '*var' according to the momentum scheme.
+//
+// Set use_nesterov = True if you want to use Nesterov momentum.
+//
+// accum = accum * momentum - lr * grad
+// var += accum
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	accum: Should be from a Variable().
+//	lr: Scaling factor. Must be a scalar.
+//	grad: The gradient.
+//	momentum: Momentum. Must be a scalar.
+//
+// Returns the created operation.
+func ResourceApplyKerasMomentum(scope *Scope, var_ tf.Output, accum tf.Output, lr tf.Output, grad tf.Output, momentum tf.Output, optional ...ResourceApplyKerasMomentumAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceApplyKerasMomentum",
+		Input: []tf.Input{
+			var_, accum, lr, grad, momentum,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// TensorArrayConcatV2Attr is an optional argument to TensorArrayConcatV2.
+type TensorArrayConcatV2Attr func(optionalAttr)
+
+// TensorArrayConcatV2ElementShapeExcept0 sets the optional element_shape_except0 attribute to value.
+// If not specified, defaults to <unknown_rank:true >
+func TensorArrayConcatV2ElementShapeExcept0(value tf.Shape) TensorArrayConcatV2Attr {
+	return func(m optionalAttr) {
+		m["element_shape_except0"] = value
+	}
+}
+
+// Deprecated. Use TensorArrayConcatV3
+func TensorArrayConcatV2(scope *Scope, handle tf.Output, flow_in tf.Output, dtype tf.DataType, optional ...TensorArrayConcatV2Attr) (value tf.Output, lengths tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorArrayConcatV2",
+		Input: []tf.Input{
+			handle, flow_in,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// MatrixSolveAttr is an optional argument to MatrixSolve.
+type MatrixSolveAttr func(optionalAttr)
+
+// MatrixSolveAdjoint sets the optional adjoint attribute to value.
+//
+// value: Boolean indicating whether to solve with `matrix` or its (block-wise)
+// adjoint.
+// If not specified, defaults to false
+func MatrixSolveAdjoint(value bool) MatrixSolveAttr {
+	return func(m optionalAttr) {
+		m["adjoint"] = value
+	}
+}
+
+// Solves systems of linear equations.
+//
+// `Matrix` is a tensor of shape `[..., M, M]` whose inner-most 2 dimensions
+// form square matrices. `Rhs` is a tensor of shape `[..., M, K]`. The `output` is
+// a tensor shape `[..., M, K]`.  If `adjoint` is `False` then each output matrix
+// satisfies `matrix[..., :, :] * output[..., :, :] = rhs[..., :, :]`.
+// If `adjoint` is `True` then each output matrix satisfies
+// `adjoint(matrix[..., :, :]) * output[..., :, :] = rhs[..., :, :]`.
+//
+// Arguments:
+//	matrix: Shape is `[..., M, M]`.
+//	rhs: Shape is `[..., M, K]`.
+//
+// Returns Shape is `[..., M, K]`.
+func MatrixSolve(scope *Scope, matrix tf.Output, rhs tf.Output, optional ...MatrixSolveAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MatrixSolve",
+		Input: []tf.Input{
+			matrix, rhs,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Writes contents to the file at input filename. Creates file and recursively
+//
+// creates directory if not existing.
+//
+// Arguments:
+//	filename: scalar. The name of the file to which we write the contents.
+//	contents: scalar. The content to be written to the output file.
+//
+// Returns the created operation.
+func WriteFile(scope *Scope, filename tf.Output, contents tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "WriteFile",
+		Input: []tf.Input{
+			filename, contents,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// ResourceSparseApplyMomentumAttr is an optional argument to ResourceSparseApplyMomentum.
+type ResourceSparseApplyMomentumAttr func(optionalAttr)
+
+// ResourceSparseApplyMomentumUseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var and accum tensors will be protected
+// by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceSparseApplyMomentumUseLocking(value bool) ResourceSparseApplyMomentumAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// ResourceSparseApplyMomentumUseNesterov sets the optional use_nesterov attribute to value.
+//
+// value: If `True`, the tensor passed to compute grad will be
+// var - lr * momentum * accum, so in the end, the var you get is actually
+// var - lr * momentum * accum.
+// If not specified, defaults to false
+func ResourceSparseApplyMomentumUseNesterov(value bool) ResourceSparseApplyMomentumAttr {
+	return func(m optionalAttr) {
+		m["use_nesterov"] = value
+	}
+}
+
+// Update relevant entries in '*var' and '*accum' according to the momentum scheme.
+//
+// Set use_nesterov = True if you want to use Nesterov momentum.
+//
+// That is for rows we have grad for, we update var and accum as follows:
+//
+// accum = accum * momentum + grad
+// var -= lr * accum
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	accum: Should be from a Variable().
+//	lr: Learning rate. Must be a scalar.
+//	grad: The gradient.
+//	indices: A vector of indices into the first dimension of var and accum.
+//	momentum: Momentum. Must be a scalar.
+//
+// Returns the created operation.
+func ResourceSparseApplyMomentum(scope *Scope, var_ tf.Output, accum tf.Output, lr tf.Output, grad tf.Output, indices tf.Output, momentum tf.Output, optional ...ResourceSparseApplyMomentumAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceSparseApplyMomentum",
+		Input: []tf.Input{
+			var_, accum, lr, grad, indices, momentum,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// RecvAttr is an optional argument to Recv.
+type RecvAttr func(optionalAttr)
+
+// RecvClientTerminated sets the optional client_terminated attribute to value.
+//
+// value: If set to true, this indicates that the node was added
+// to the graph as a result of a client-side feed or fetch of Tensor data,
+// in which case the corresponding send or recv is expected to be managed
+// locally by the caller.
+// If not specified, defaults to false
+func RecvClientTerminated(value bool) RecvAttr {
+	return func(m optionalAttr) {
+		m["client_terminated"] = value
+	}
+}
+
+// Receives the named tensor from send_device on recv_device.
+//
+// Arguments:
+//
+//	tensor_name: The name of the tensor to receive.
+//	send_device: The name of the device sending the tensor.
+//	send_device_incarnation: The current incarnation of send_device.
+//	recv_device: The name of the device receiving the tensor.
+//
+// Returns The tensor to receive.
+func Recv(scope *Scope, tensor_type tf.DataType, tensor_name string, send_device string, send_device_incarnation int64, recv_device string, optional ...RecvAttr) (tensor tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"tensor_type": tensor_type, "tensor_name": tensor_name, "send_device": send_device, "send_device_incarnation": send_device_incarnation, "recv_device": recv_device}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Recv",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// OrderedMapStageAttr is an optional argument to OrderedMapStage.
+type OrderedMapStageAttr func(optionalAttr)
+
+// OrderedMapStageCapacity sets the optional capacity attribute to value.
+//
+// value: Maximum number of elements in the Staging Area. If > 0, inserts
+// on the container will block when the capacity is reached.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func OrderedMapStageCapacity(value int64) OrderedMapStageAttr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// OrderedMapStageMemoryLimit sets the optional memory_limit attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func OrderedMapStageMemoryLimit(value int64) OrderedMapStageAttr {
+	return func(m optionalAttr) {
+		m["memory_limit"] = value
+	}
+}
+
+// OrderedMapStageContainer sets the optional container attribute to value.
+//
+// value: If non-empty, this queue is placed in the given container. Otherwise,
+// a default container is used.
+// If not specified, defaults to ""
+func OrderedMapStageContainer(value string) OrderedMapStageAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// OrderedMapStageSharedName sets the optional shared_name attribute to value.
+//
+// value: It is necessary to match this name to the matching Unstage Op.
+// If not specified, defaults to ""
+func OrderedMapStageSharedName(value string) OrderedMapStageAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Stage (key, values) in the underlying container which behaves like a ordered
+//
+// associative container.   Elements are ordered by key.
+//
+// Arguments:
+//	key: int64
+//
+//	values: a list of tensors
+// dtypes A list of data types that inserted values should adhere to.
+//
+//
+// Returns the created operation.
+func OrderedMapStage(scope *Scope, key tf.Output, indices tf.Output, values []tf.Output, dtypes []tf.DataType, optional ...OrderedMapStageAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "OrderedMapStage",
+		Input: []tf.Input{
+			key, indices, tf.OutputList(values),
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// TPUReplicateMetadataAttr is an optional argument to TPUReplicateMetadata.
+type TPUReplicateMetadataAttr func(optionalAttr)
+
+// TPUReplicateMetadataNumCoresPerReplica sets the optional num_cores_per_replica attribute to value.
+//
+// value: Number of cores per replica. Used for model parallelism.
+// If not specified, defaults to 1
+func TPUReplicateMetadataNumCoresPerReplica(value int64) TPUReplicateMetadataAttr {
+	return func(m optionalAttr) {
+		m["num_cores_per_replica"] = value
+	}
+}
+
+// TPUReplicateMetadataTopology sets the optional topology attribute to value.
+//
+// value: TopologyProto indicating the topology of the TPU pod slice.
+// If not specified, defaults to ""
+func TPUReplicateMetadataTopology(value string) TPUReplicateMetadataAttr {
+	return func(m optionalAttr) {
+		m["topology"] = value
+	}
+}
+
+// TPUReplicateMetadataUseTpu sets the optional use_tpu attribute to value.
+//
+// value: Whether to place the computation on the TPU.
+// If not specified, defaults to true
+func TPUReplicateMetadataUseTpu(value bool) TPUReplicateMetadataAttr {
+	return func(m optionalAttr) {
+		m["use_tpu"] = value
+	}
+}
+
+// TPUReplicateMetadataDeviceAssignment sets the optional device_assignment attribute to value.
+//
+// value: The assignment of devices for the computation.
+// If not specified, defaults to <>
+func TPUReplicateMetadataDeviceAssignment(value []int64) TPUReplicateMetadataAttr {
+	return func(m optionalAttr) {
+		m["device_assignment"] = value
+	}
+}
+
+// TPUReplicateMetadataComputationShape sets the optional computation_shape attribute to value.
+//
+// value: DEPRECATED. Use num_cores_per_replica instead.
+// If not specified, defaults to <>
+func TPUReplicateMetadataComputationShape(value []int64) TPUReplicateMetadataAttr {
+	return func(m optionalAttr) {
+		m["computation_shape"] = value
+	}
+}
+
+// TPUReplicateMetadataHostComputeCore sets the optional host_compute_core attribute to value.
+// If not specified, defaults to <>
+func TPUReplicateMetadataHostComputeCore(value []string) TPUReplicateMetadataAttr {
+	return func(m optionalAttr) {
+		m["host_compute_core"] = value
+	}
+}
+
+// TPUReplicateMetadataPaddingMap sets the optional padding_map attribute to value.
+// If not specified, defaults to <>
+func TPUReplicateMetadataPaddingMap(value []string) TPUReplicateMetadataAttr {
+	return func(m optionalAttr) {
+		m["padding_map"] = value
+	}
+}
+
+// TPUReplicateMetadataStepMarkerLocation sets the optional step_marker_location attribute to value.
+// If not specified, defaults to "STEP_MARK_AT_ENTRY"
+func TPUReplicateMetadataStepMarkerLocation(value string) TPUReplicateMetadataAttr {
+	return func(m optionalAttr) {
+		m["step_marker_location"] = value
+	}
+}
+
+// TPUReplicateMetadataAllowSoftPlacement sets the optional allow_soft_placement attribute to value.
+// If not specified, defaults to false
+func TPUReplicateMetadataAllowSoftPlacement(value bool) TPUReplicateMetadataAttr {
+	return func(m optionalAttr) {
+		m["allow_soft_placement"] = value
+	}
+}
+
+// Metadata indicating how the TPU computation should be replicated.
+//
+// This operation holds the metadata common to operations of a `tpu.replicate()` computation subgraph.
+//
+// Arguments:
+//	num_replicas: Number of replicas of the computation
+//
+// Returns the created operation.
+func TPUReplicateMetadata(scope *Scope, num_replicas int64, optional ...TPUReplicateMetadataAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_replicas": num_replicas}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TPUReplicateMetadata",
+
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// TensorListConcatAttr is an optional argument to TensorListConcat.
+type TensorListConcatAttr func(optionalAttr)
+
+// TensorListConcatElementShape sets the optional element_shape attribute to value.
+// If not specified, defaults to <unknown_rank:true >
+func TensorListConcatElementShape(value tf.Shape) TensorListConcatAttr {
+	return func(m optionalAttr) {
+		m["element_shape"] = value
+	}
+}
+
+// Concats all tensors in the list along the 0th dimension.
+//
+// Requires that all tensors have the same shape except the first dimension.
+//
+// input_handle: The input list.
+// tensor: The concated result.
+// lengths: Output tensor containing sizes of the 0th dimension of tensors in the list, used for computing the gradient.
+//
+func TensorListConcat(scope *Scope, input_handle tf.Output, element_dtype tf.DataType, optional ...TensorListConcatAttr) (tensor tf.Output, lengths tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"element_dtype": element_dtype}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "TensorListConcat",
+		Input: []tf.Input{
+			input_handle,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// LoadTPUEmbeddingRMSPropParametersGradAccumDebugAttr is an optional argument to LoadTPUEmbeddingRMSPropParametersGradAccumDebug.
+type LoadTPUEmbeddingRMSPropParametersGradAccumDebugAttr func(optionalAttr)
+
+// LoadTPUEmbeddingRMSPropParametersGradAccumDebugTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func LoadTPUEmbeddingRMSPropParametersGradAccumDebugTableId(value int64) LoadTPUEmbeddingRMSPropParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// LoadTPUEmbeddingRMSPropParametersGradAccumDebugTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingRMSPropParametersGradAccumDebugTableName(value string) LoadTPUEmbeddingRMSPropParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// LoadTPUEmbeddingRMSPropParametersGradAccumDebugConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingRMSPropParametersGradAccumDebugConfig(value string) LoadTPUEmbeddingRMSPropParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Load RMSProp embedding parameters with debug support.
+//
+// An op that loads optimization parameters into HBM for embedding. Must be
+// preceded by a ConfigureTPUEmbeddingHost op that sets up the correct
+// embedding table configuration. For example, this op is used to install
+// parameters that are loaded from a checkpoint before a training loop is
+// executed.
+//
+// Arguments:
+//	parameters: Value of parameters used in the RMSProp optimization algorithm.
+//	ms: Value of ms used in the RMSProp optimization algorithm.
+//	mom: Value of mom used in the RMSProp optimization algorithm.
+//	gradient_accumulators: Value of gradient_accumulators used in the RMSProp optimization algorithm.
+//
+//
+//
+// Returns the created operation.
+func LoadTPUEmbeddingRMSPropParametersGradAccumDebug(scope *Scope, parameters tf.Output, ms tf.Output, mom tf.Output, gradient_accumulators tf.Output, num_shards int64, shard_id int64, optional ...LoadTPUEmbeddingRMSPropParametersGradAccumDebugAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LoadTPUEmbeddingRMSPropParametersGradAccumDebug",
+		Input: []tf.Input{
+			parameters, ms, mom, gradient_accumulators,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Concatenates tensors along one dimension.
+//
+// Arguments:
+//	values: List of `N` Tensors to concatenate. Their ranks and types must match,
+// and their sizes must match in all dimensions except `concat_dim`.
+//	axis: 0-D.  The dimension along which to concatenate.  Must be in the
+// range [-rank(values), rank(values)).
+//
+// Returns A `Tensor` with the concatenation of values stacked along the
+// `concat_dim` dimension.  This tensor's shape matches that of `values` except
+// in `concat_dim` where it has the sum of the sizes.
+func ConcatV2(scope *Scope, values []tf.Output, axis tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ConcatV2",
+		Input: []tf.Input{
+			tf.OutputList(values), axis,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// LoadTPUEmbeddingFTRLParametersGradAccumDebugAttr is an optional argument to LoadTPUEmbeddingFTRLParametersGradAccumDebug.
+type LoadTPUEmbeddingFTRLParametersGradAccumDebugAttr func(optionalAttr)
+
+// LoadTPUEmbeddingFTRLParametersGradAccumDebugTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func LoadTPUEmbeddingFTRLParametersGradAccumDebugTableId(value int64) LoadTPUEmbeddingFTRLParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// LoadTPUEmbeddingFTRLParametersGradAccumDebugTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingFTRLParametersGradAccumDebugTableName(value string) LoadTPUEmbeddingFTRLParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// LoadTPUEmbeddingFTRLParametersGradAccumDebugConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingFTRLParametersGradAccumDebugConfig(value string) LoadTPUEmbeddingFTRLParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Load FTRL embedding parameters with debug support.
+//
+// An op that loads optimization parameters into HBM for embedding. Must be
+// preceded by a ConfigureTPUEmbeddingHost op that sets up the correct
+// embedding table configuration. For example, this op is used to install
+// parameters that are loaded from a checkpoint before a training loop is
+// executed.
+//
+// Arguments:
+//	parameters: Value of parameters used in the FTRL optimization algorithm.
+//	accumulators: Value of accumulators used in the FTRL optimization algorithm.
+//	linears: Value of linears used in the FTRL optimization algorithm.
+//	gradient_accumulators: Value of gradient_accumulators used in the FTRL optimization algorithm.
+//
+//
+//
+// Returns the created operation.
+func LoadTPUEmbeddingFTRLParametersGradAccumDebug(scope *Scope, parameters tf.Output, accumulators tf.Output, linears tf.Output, gradient_accumulators tf.Output, num_shards int64, shard_id int64, optional ...LoadTPUEmbeddingFTRLParametersGradAccumDebugAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LoadTPUEmbeddingFTRLParametersGradAccumDebug",
+		Input: []tf.Input{
+			parameters, accumulators, linears, gradient_accumulators,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// LoadTPUEmbeddingAdadeltaParametersAttr is an optional argument to LoadTPUEmbeddingAdadeltaParameters.
+type LoadTPUEmbeddingAdadeltaParametersAttr func(optionalAttr)
+
+// LoadTPUEmbeddingAdadeltaParametersTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func LoadTPUEmbeddingAdadeltaParametersTableId(value int64) LoadTPUEmbeddingAdadeltaParametersAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// LoadTPUEmbeddingAdadeltaParametersTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingAdadeltaParametersTableName(value string) LoadTPUEmbeddingAdadeltaParametersAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// LoadTPUEmbeddingAdadeltaParametersConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingAdadeltaParametersConfig(value string) LoadTPUEmbeddingAdadeltaParametersAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Load Adadelta embedding parameters.
+//
+// An op that loads optimization parameters into HBM for embedding. Must be
+// preceded by a ConfigureTPUEmbeddingHost op that sets up the correct
+// embedding table configuration. For example, this op is used to install
+// parameters that are loaded from a checkpoint before a training loop is
+// executed.
+//
+// Arguments:
+//	parameters: Value of parameters used in the Adadelta optimization algorithm.
+//	accumulators: Value of accumulators used in the Adadelta optimization algorithm.
+//	updates: Value of updates used in the Adadelta optimization algorithm.
+//
+//
+//
+// Returns the created operation.
+func LoadTPUEmbeddingAdadeltaParameters(scope *Scope, parameters tf.Output, accumulators tf.Output, updates tf.Output, num_shards int64, shard_id int64, optional ...LoadTPUEmbeddingAdadeltaParametersAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LoadTPUEmbeddingAdadeltaParameters",
+		Input: []tf.Input{
+			parameters, accumulators, updates,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// ResourceSparseApplyFtrlV2Attr is an optional argument to ResourceSparseApplyFtrlV2.
+type ResourceSparseApplyFtrlV2Attr func(optionalAttr)
+
+// ResourceSparseApplyFtrlV2UseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var and accum tensors will be protected
+// by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceSparseApplyFtrlV2UseLocking(value bool) ResourceSparseApplyFtrlV2Attr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// ResourceSparseApplyFtrlV2MultiplyLinearByLr sets the optional multiply_linear_by_lr attribute to value.
+// If not specified, defaults to false
+func ResourceSparseApplyFtrlV2MultiplyLinearByLr(value bool) ResourceSparseApplyFtrlV2Attr {
+	return func(m optionalAttr) {
+		m["multiply_linear_by_lr"] = value
+	}
+}
+
+// Update relevant entries in '*var' according to the Ftrl-proximal scheme.
+//
+// That is for rows we have grad for, we update var, accum and linear as follows:
+// grad_with_shrinkage = grad + 2 * l2_shrinkage * var
+// accum_new = accum + grad_with_shrinkage * grad_with_shrinkage
+// linear += grad_with_shrinkage +
+//     (accum_new^(-lr_power) - accum^(-lr_power)) / lr * var
+// quadratic = 1.0 / (accum_new^(lr_power) * lr) + 2 * l2
+// var = (sign(linear) * l1 - linear) / quadratic if |linear| > l1 else 0.0
+// accum = accum_new
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	accum: Should be from a Variable().
+//	linear: Should be from a Variable().
+//	grad: The gradient.
+//	indices: A vector of indices into the first dimension of var and accum.
+//	lr: Scaling factor. Must be a scalar.
+//	l1: L1 regularization. Must be a scalar.
+//	l2: L2 shrinkage regularization. Must be a scalar.
+//
+//	lr_power: Scaling factor. Must be a scalar.
+//
+// Returns the created operation.
+func ResourceSparseApplyFtrlV2(scope *Scope, var_ tf.Output, accum tf.Output, linear tf.Output, grad tf.Output, indices tf.Output, lr tf.Output, l1 tf.Output, l2 tf.Output, l2_shrinkage tf.Output, lr_power tf.Output, optional ...ResourceSparseApplyFtrlV2Attr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceSparseApplyFtrlV2",
+		Input: []tf.Input{
+			var_, accum, linear, grad, indices, lr, l1, l2, l2_shrinkage, lr_power,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// ResourceSparseApplyAdagradV2Attr is an optional argument to ResourceSparseApplyAdagradV2.
+type ResourceSparseApplyAdagradV2Attr func(optionalAttr)
+
+// ResourceSparseApplyAdagradV2UseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var and accum tensors will be protected
+// by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceSparseApplyAdagradV2UseLocking(value bool) ResourceSparseApplyAdagradV2Attr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// ResourceSparseApplyAdagradV2UpdateSlots sets the optional update_slots attribute to value.
+// If not specified, defaults to true
+func ResourceSparseApplyAdagradV2UpdateSlots(value bool) ResourceSparseApplyAdagradV2Attr {
+	return func(m optionalAttr) {
+		m["update_slots"] = value
+	}
+}
+
+// Update relevant entries in '*var' and '*accum' according to the adagrad scheme.
+//
+// That is for rows we have grad for, we update var and accum as follows:
+// accum += grad * grad
+// var -= lr * grad * (1 / sqrt(accum))
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	accum: Should be from a Variable().
+//	lr: Learning rate. Must be a scalar.
+//	epsilon: Constant factor. Must be a scalar.
+//	grad: The gradient.
+//	indices: A vector of indices into the first dimension of var and accum.
+//
+// Returns the created operation.
+func ResourceSparseApplyAdagradV2(scope *Scope, var_ tf.Output, accum tf.Output, lr tf.Output, epsilon tf.Output, grad tf.Output, indices tf.Output, optional ...ResourceSparseApplyAdagradV2Attr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceSparseApplyAdagradV2",
+		Input: []tf.Input{
+			var_, accum, lr, epsilon, grad, indices,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Restore a Reader to its initial clean state.
+//
+// Arguments:
+//	reader_handle: Handle to a Reader.
+//
+// Returns the created operation.
+func ReaderResetV2(scope *Scope, reader_handle tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ReaderResetV2",
+		Input: []tf.Input{
+			reader_handle,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// FakeQuantWithMinMaxVarsPerChannelGradientAttr is an optional argument to FakeQuantWithMinMaxVarsPerChannelGradient.
+type FakeQuantWithMinMaxVarsPerChannelGradientAttr func(optionalAttr)
+
+// FakeQuantWithMinMaxVarsPerChannelGradientNumBits sets the optional num_bits attribute to value.
+//
+// value: The bitwidth of the quantization; between 2 and 16, inclusive.
+// If not specified, defaults to 8
+func FakeQuantWithMinMaxVarsPerChannelGradientNumBits(value int64) FakeQuantWithMinMaxVarsPerChannelGradientAttr {
+	return func(m optionalAttr) {
+		m["num_bits"] = value
+	}
+}
+
+// FakeQuantWithMinMaxVarsPerChannelGradientNarrowRange sets the optional narrow_range attribute to value.
+//
+// value: Whether to quantize into 2^num_bits - 1 distinct values.
+// If not specified, defaults to false
+func FakeQuantWithMinMaxVarsPerChannelGradientNarrowRange(value bool) FakeQuantWithMinMaxVarsPerChannelGradientAttr {
+	return func(m optionalAttr) {
+		m["narrow_range"] = value
+	}
+}
+
+// Compute gradients for a FakeQuantWithMinMaxVarsPerChannel operation.
+//
+// Arguments:
+//	gradients: Backpropagated gradients above the FakeQuantWithMinMaxVars operation,
+// shape one of: `[d]`, `[b, d]`,  `[b, h, w, d]`.
+//	inputs: Values passed as inputs to the FakeQuantWithMinMaxVars operation, shape
+//   same as `gradients`.
+// min, max: Quantization interval, floats of shape `[d]`.
+//
+//
+//
+// Returns:
+//	backprops_wrt_input: Backpropagated gradients w.r.t. inputs, shape same as
+// `inputs`:
+//   `gradients * (inputs >= min && inputs <= max)`.
+//	backprop_wrt_min: Backpropagated gradients w.r.t. min parameter, shape `[d]`:
+// `sum_per_d(gradients * (inputs < min))`.
+//	backprop_wrt_max: Backpropagated gradients w.r.t. max parameter, shape `[d]`:
+// `sum_per_d(gradients * (inputs > max))`.
+func FakeQuantWithMinMaxVarsPerChannelGradient(scope *Scope, gradients tf.Output, inputs tf.Output, min tf.Output, max tf.Output, optional ...FakeQuantWithMinMaxVarsPerChannelGradientAttr) (backprops_wrt_input tf.Output, backprop_wrt_min tf.Output, backprop_wrt_max tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "FakeQuantWithMinMaxVarsPerChannelGradient",
+		Input: []tf.Input{
+			gradients, inputs, min, max,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// PrintV2Attr is an optional argument to PrintV2.
+type PrintV2Attr func(optionalAttr)
+
+// PrintV2OutputStream sets the optional output_stream attribute to value.
+//
+// value: A string specifying the output stream or logging level to print to.
+// If not specified, defaults to "stderr"
+func PrintV2OutputStream(value string) PrintV2Attr {
+	return func(m optionalAttr) {
+		m["output_stream"] = value
+	}
+}
+
+// PrintV2End sets the optional end attribute to value.
+// If not specified, defaults to "\n"
+func PrintV2End(value string) PrintV2Attr {
+	return func(m optionalAttr) {
+		m["end"] = value
+	}
+}
+
+// Prints a string scalar.
+//
+// Prints a string scalar to the desired output_stream.
+//
+// Arguments:
+//	input: The string scalar to print.
+//
+// Returns the created operation.
+func PrintV2(scope *Scope, input tf.Output, optional ...PrintV2Attr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "PrintV2",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Inserts a dimension of 1 into a tensor's shape.
+//
+// Given a tensor `input`, this operation inserts a dimension of 1 at the
+// dimension index `axis` of `input`'s shape. The dimension index `axis` starts at
+// zero; if you specify a negative number for `axis` it is counted backward from
+// the end.
+//
+// This operation is useful if you want to add a batch dimension to a single
+// element. For example, if you have a single image of shape `[height, width,
+// channels]`, you can make it a batch of 1 image with `expand_dims(image, 0)`,
+// which will make the shape `[1, height, width, channels]`.
+//
+// Other examples:
+//
+// ```
+// # 't' is a tensor of shape [2]
+// shape(expand_dims(t, 0)) ==> [1, 2]
+// shape(expand_dims(t, 1)) ==> [2, 1]
+// shape(expand_dims(t, -1)) ==> [2, 1]
+//
+// # 't2' is a tensor of shape [2, 3, 5]
+// shape(expand_dims(t2, 0)) ==> [1, 2, 3, 5]
+// shape(expand_dims(t2, 2)) ==> [2, 3, 1, 5]
+// shape(expand_dims(t2, 3)) ==> [2, 3, 5, 1]
+// ```
+//
+// This operation requires that:
+//
+// `-1-input.dims() <= dim <= input.dims()`
+//
+// This operation is related to `squeeze()`, which removes dimensions of
+// size 1.
+//
+// Arguments:
+//
+//	axis: 0-D (scalar). Specifies the dimension index at which to
+// expand the shape of `input`. Must be in the range
+// `[-rank(input) - 1, rank(input)]`.
+//
+// Returns Contains the same data as `input`, but its shape has an additional
+// dimension of size 1 added.
+func ExpandDims(scope *Scope, input tf.Output, axis tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ExpandDims",
+		Input: []tf.Input{
+			input, axis,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResourceSparseApplyProximalGradientDescentAttr is an optional argument to ResourceSparseApplyProximalGradientDescent.
+type ResourceSparseApplyProximalGradientDescentAttr func(optionalAttr)
+
+// ResourceSparseApplyProximalGradientDescentUseLocking sets the optional use_locking attribute to value.
+//
+// value: If True, the subtraction will be protected by a lock;
+// otherwise the behavior is undefined, but may exhibit less contention.
+// If not specified, defaults to false
+func ResourceSparseApplyProximalGradientDescentUseLocking(value bool) ResourceSparseApplyProximalGradientDescentAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// Sparse update '*var' as FOBOS algorithm with fixed learning rate.
+//
+// That is for rows we have grad for, we update var as follows:
+// prox_v = var - alpha * grad
+// var = sign(prox_v)/(1+alpha*l2) * max{|prox_v|-alpha*l1,0}
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	alpha: Scaling factor. Must be a scalar.
+//	l1: L1 regularization. Must be a scalar.
+//	l2: L2 regularization. Must be a scalar.
+//	grad: The gradient.
+//	indices: A vector of indices into the first dimension of var and accum.
+//
+// Returns the created operation.
+func ResourceSparseApplyProximalGradientDescent(scope *Scope, var_ tf.Output, alpha tf.Output, l1 tf.Output, l2 tf.Output, grad tf.Output, indices tf.Output, optional ...ResourceSparseApplyProximalGradientDescentAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceSparseApplyProximalGradientDescent",
+		Input: []tf.Input{
+			var_, alpha, l1, l2, grad, indices,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// SparseMatrixTransposeAttr is an optional argument to SparseMatrixTranspose.
+type SparseMatrixTransposeAttr func(optionalAttr)
+
+// SparseMatrixTransposeConjugate sets the optional conjugate attribute to value.
+//
+// value: Indicates whether `input` should be conjugated.
+// If not specified, defaults to false
+func SparseMatrixTransposeConjugate(value bool) SparseMatrixTransposeAttr {
+	return func(m optionalAttr) {
+		m["conjugate"] = value
+	}
+}
+
+// Transposes the inner (matrix) dimensions of a CSRSparseMatrix.
+//
+// Transposes the inner (matrix) dimensions of a SparseMatrix and optionally
+// conjugates its values.
+//
+// Arguments:
+//	input: A CSRSparseMatrix.
+//
+//
+// Returns A CSRSparseMatrix.
+func SparseMatrixTranspose(scope *Scope, input tf.Output, type_ tf.DataType, optional ...SparseMatrixTransposeAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"type": type_}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseMatrixTranspose",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// LoadTPUEmbeddingProximalAdagradParametersAttr is an optional argument to LoadTPUEmbeddingProximalAdagradParameters.
+type LoadTPUEmbeddingProximalAdagradParametersAttr func(optionalAttr)
+
+// LoadTPUEmbeddingProximalAdagradParametersTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func LoadTPUEmbeddingProximalAdagradParametersTableId(value int64) LoadTPUEmbeddingProximalAdagradParametersAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// LoadTPUEmbeddingProximalAdagradParametersTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingProximalAdagradParametersTableName(value string) LoadTPUEmbeddingProximalAdagradParametersAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// LoadTPUEmbeddingProximalAdagradParametersConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingProximalAdagradParametersConfig(value string) LoadTPUEmbeddingProximalAdagradParametersAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Load proximal Adagrad embedding parameters.
+//
+// An op that loads optimization parameters into HBM for embedding. Must be
+// preceded by a ConfigureTPUEmbeddingHost op that sets up the correct
+// embedding table configuration. For example, this op is used to install
+// parameters that are loaded from a checkpoint before a training loop is
+// executed.
+//
+// Arguments:
+//	parameters: Value of parameters used in the proximal Adagrad optimization algorithm.
+//	accumulators: Value of accumulators used in the proximal Adagrad optimization algorithm.
+//
+//
+//
+// Returns the created operation.
+func LoadTPUEmbeddingProximalAdagradParameters(scope *Scope, parameters tf.Output, accumulators tf.Output, num_shards int64, shard_id int64, optional ...LoadTPUEmbeddingProximalAdagradParametersAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LoadTPUEmbeddingProximalAdagradParameters",
+		Input: []tf.Input{
+			parameters, accumulators,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// ResourceSparseApplyAdagradDAAttr is an optional argument to ResourceSparseApplyAdagradDA.
+type ResourceSparseApplyAdagradDAAttr func(optionalAttr)
+
+// ResourceSparseApplyAdagradDAUseLocking sets the optional use_locking attribute to value.
+//
+// value: If True, updating of the var and accum tensors will be protected by
+// a lock; otherwise the behavior is undefined, but may exhibit less contention.
+// If not specified, defaults to false
+func ResourceSparseApplyAdagradDAUseLocking(value bool) ResourceSparseApplyAdagradDAAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// Update entries in '*var' and '*accum' according to the proximal adagrad scheme.
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	gradient_accumulator: Should be from a Variable().
+//	gradient_squared_accumulator: Should be from a Variable().
+//	grad: The gradient.
+//	indices: A vector of indices into the first dimension of var and accum.
+//	lr: Learning rate. Must be a scalar.
+//	l1: L1 regularization. Must be a scalar.
+//	l2: L2 regularization. Must be a scalar.
+//	global_step: Training step number. Must be a scalar.
+//
+// Returns the created operation.
+func ResourceSparseApplyAdagradDA(scope *Scope, var_ tf.Output, gradient_accumulator tf.Output, gradient_squared_accumulator tf.Output, grad tf.Output, indices tf.Output, lr tf.Output, l1 tf.Output, l2 tf.Output, global_step tf.Output, optional ...ResourceSparseApplyAdagradDAAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceSparseApplyAdagradDA",
+		Input: []tf.Input{
+			var_, gradient_accumulator, gradient_squared_accumulator, grad, indices, lr, l1, l2, global_step,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Applies softmax to a batched N-D `SparseTensor`.
+//
+// The inputs represent an N-D SparseTensor  with logical shape `[..., B, C]`
+// (where `N >= 2`), and with indices sorted in the canonical lexicographic order.
+//
+// This op is equivalent to applying the normal `tf.nn.softmax()` to each innermost
+// logical submatrix with shape `[B, C]`, but with the catch that *the implicitly
+// zero elements do not participate*.  Specifically, the algorithm is equivalent
+// to the following:
+//
+//   (1) Applies `tf.nn.softmax()` to a densified view of each innermost submatrix
+//       with shape `[B, C]`, along the size-C dimension;
+//   (2) Masks out the original implicitly-zero locations;
+//   (3) Renormalizes the remaining elements.
+//
+// Hence, the `SparseTensor` result has exactly the same non-zero indices and
+// shape.
+//
+// Arguments:
+//	sp_indices: 2-D.  `NNZ x R` matrix with the indices of non-empty values in a
+// SparseTensor, in canonical ordering.
+//	sp_values: 1-D.  `NNZ` non-empty values corresponding to `sp_indices`.
+//	sp_shape: 1-D.  Shape of the input SparseTensor.
+//
+// Returns 1-D.  The `NNZ` values for the result `SparseTensor`.
+func SparseSoftmax(scope *Scope, sp_indices tf.Output, sp_values tf.Output, sp_shape tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseSoftmax",
+		Input: []tf.Input{
+			sp_indices, sp_values, sp_shape,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes hyperbolic cosine of x element-wise.
+//
+//   Given an input tensor, this function computes hyperbolic cosine of every
+//   element in the tensor. Input range is `[-inf, inf]` and output range
+//   is `[1, inf]`.
+//
+//   ```python
+//   x = tf.constant([-float("inf"), -9, -0.5, 1, 1.2, 2, 10, float("inf")])
+//   tf.math.cosh(x) ==> [inf 4.0515420e+03 1.1276259e+00 1.5430807e+00 1.8106556e+00 3.7621956e+00 1.1013233e+04 inf]
+//   ```
+func Cosh(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Cosh",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// CollectiveReduceAttr is an optional argument to CollectiveReduce.
+type CollectiveReduceAttr func(optionalAttr)
+
+// CollectiveReduceWaitFor sets the optional wait_for attribute to value.
+// If not specified, defaults to <>
+func CollectiveReduceWaitFor(value []int64) CollectiveReduceAttr {
+	return func(m optionalAttr) {
+		m["wait_for"] = value
+	}
+}
+
+// CollectiveReduceCommunicationHint sets the optional communication_hint attribute to value.
+// If not specified, defaults to "auto"
+func CollectiveReduceCommunicationHint(value string) CollectiveReduceAttr {
+	return func(m optionalAttr) {
+		m["communication_hint"] = value
+	}
+}
+
+// CollectiveReduceTimeoutSeconds sets the optional timeout_seconds attribute to value.
+// If not specified, defaults to 0
+func CollectiveReduceTimeoutSeconds(value float32) CollectiveReduceAttr {
+	return func(m optionalAttr) {
+		m["timeout_seconds"] = value
+	}
+}
+
+// Mutually reduces multiple tensors of identical type and shape.
+func CollectiveReduce(scope *Scope, input tf.Output, group_size int64, group_key int64, instance_key int64, merge_op string, final_op string, subdiv_offsets []int64, optional ...CollectiveReduceAttr) (data tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"group_size": group_size, "group_key": group_key, "instance_key": instance_key, "merge_op": merge_op, "final_op": final_op, "subdiv_offsets": subdiv_offsets}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CollectiveReduce",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResourceApplyAdaMaxAttr is an optional argument to ResourceApplyAdaMax.
+type ResourceApplyAdaMaxAttr func(optionalAttr)
+
+// ResourceApplyAdaMaxUseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var, m, and v tensors will be protected
+// by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceApplyAdaMaxUseLocking(value bool) ResourceApplyAdaMaxAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// Update '*var' according to the AdaMax algorithm.
+//
+// m_t <- beta1 * m_{t-1} + (1 - beta1) * g
+// v_t <- max(beta2 * v_{t-1}, abs(g))
+// variable <- variable - learning_rate / (1 - beta1^t) * m_t / (v_t + epsilon)
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	m: Should be from a Variable().
+//	v: Should be from a Variable().
+//	beta1_power: Must be a scalar.
+//	lr: Scaling factor. Must be a scalar.
+//	beta1: Momentum factor. Must be a scalar.
+//	beta2: Momentum factor. Must be a scalar.
+//	epsilon: Ridge term. Must be a scalar.
+//	grad: The gradient.
+//
+// Returns the created operation.
+func ResourceApplyAdaMax(scope *Scope, var_ tf.Output, m tf.Output, v tf.Output, beta1_power tf.Output, lr tf.Output, beta1 tf.Output, beta2 tf.Output, epsilon tf.Output, grad tf.Output, optional ...ResourceApplyAdaMaxAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceApplyAdaMax",
+		Input: []tf.Input{
+			var_, m, v, beta1_power, lr, beta1, beta2, epsilon, grad,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Computes the LSTM cell backward propagation for the entire time sequence.
+//
+// This implementation is to be used in conjunction of BlockLSTMV2.
+//
+// Arguments:
+//	seq_len_max: Maximum time length actually used by this input. Outputs are padded
+// with zeros beyond this length.
+//	x: The sequence input to the LSTM, shape (timelen, batch_size, num_inputs).
+//	cs_prev: Value of the initial cell state.
+//	h_prev: Initial output of cell (to be used for peephole).
+//	w: The weight matrix.
+//	wci: The weight matrix for input gate peephole connection.
+//	wcf: The weight matrix for forget gate peephole connection.
+//	wco: The weight matrix for output gate peephole connection.
+//	b: The bias vector.
+//	i: The input gate over the whole time sequence.
+//	cs: The cell state before the tanh over the whole time sequence.
+//	f: The forget gate over the whole time sequence.
+//	o: The output gate over the whole time sequence.
+//	ci: The cell input over the whole time sequence.
+//	co: The cell after the tanh over the whole time sequence.
+//	h: The output h vector over the whole time sequence.
+//	cs_grad: The current gradient of cs.
+//	h_grad: The gradient of h vector.
+//	use_peephole: Whether to use peephole weights.
+//
+// Returns:
+//	x_grad: The gradient of x to be back-propped.
+//	cs_prev_grad: The gradient of cs_prev to be back-propped.
+//	h_prev_grad: The gradient of h_prev to be back-propped.
+//	w_grad: The gradient for w to be back-propped.
+//	wci_grad: The gradient for wci to be back-propped.
+//	wcf_grad: The gradient for wcf to be back-propped.
+//	wco_grad: The gradient for wco to be back-propped.
+//	b_grad: The gradient for w to be back-propped.
+func BlockLSTMGradV2(scope *Scope, seq_len_max tf.Output, x tf.Output, cs_prev tf.Output, h_prev tf.Output, w tf.Output, wci tf.Output, wcf tf.Output, wco tf.Output, b tf.Output, i tf.Output, cs tf.Output, f tf.Output, o tf.Output, ci tf.Output, co tf.Output, h tf.Output, cs_grad tf.Output, h_grad tf.Output, use_peephole bool) (x_grad tf.Output, cs_prev_grad tf.Output, h_prev_grad tf.Output, w_grad tf.Output, wci_grad tf.Output, wcf_grad tf.Output, wco_grad tf.Output, b_grad tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"use_peephole": use_peephole}
+	opspec := tf.OpSpec{
+		Type: "BlockLSTMGradV2",
+		Input: []tf.Input{
+			seq_len_max, x, cs_prev, h_prev, w, wci, wcf, wco, b, i, cs, f, o, ci, co, h, cs_grad, h_grad,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3), op.Output(4), op.Output(5), op.Output(6), op.Output(7)
+}
+
+// Returns the element-wise max of two SparseTensors.
+//
+// Assumes the two SparseTensors have the same shape, i.e., no broadcasting.
+//
+// Arguments:
+//	a_indices: 2-D.  `N x R` matrix with the indices of non-empty values in a
+// SparseTensor, in the canonical lexicographic ordering.
+//	a_values: 1-D.  `N` non-empty values corresponding to `a_indices`.
+//	a_shape: 1-D.  Shape of the input SparseTensor.
+//	b_indices: counterpart to `a_indices` for the other operand.
+//	b_values: counterpart to `a_values` for the other operand; must be of the same dtype.
+//	b_shape: counterpart to `a_shape` for the other operand; the two shapes must be equal.
+//
+// Returns:
+//	output_indices: 2-D.  The indices of the output SparseTensor.
+//	output_values: 1-D.  The values of the output SparseTensor.
+func SparseSparseMaximum(scope *Scope, a_indices tf.Output, a_values tf.Output, a_shape tf.Output, b_indices tf.Output, b_values tf.Output, b_shape tf.Output) (output_indices tf.Output, output_values tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseSparseMaximum",
+		Input: []tf.Input{
+			a_indices, a_values, a_shape, b_indices, b_values, b_shape,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Computes the Bessel i1e function of `x` element-wise.
+//
+// Exponentially scaled modified Bessel function of order 0 defined as
+// `bessel_i1e(x) = exp(-abs(x)) bessel_i1(x)`.
+//
+// This function is faster and numerically stabler than `bessel_i1(x)`.
+func BesselI1e(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "BesselI1e",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// LSTMBlockCellAttr is an optional argument to LSTMBlockCell.
+type LSTMBlockCellAttr func(optionalAttr)
+
+// LSTMBlockCellForgetBias sets the optional forget_bias attribute to value.
+//
+// value: The forget gate bias.
+// If not specified, defaults to 1
+func LSTMBlockCellForgetBias(value float32) LSTMBlockCellAttr {
+	return func(m optionalAttr) {
+		m["forget_bias"] = value
+	}
+}
+
+// LSTMBlockCellCellClip sets the optional cell_clip attribute to value.
+//
+// value: Value to clip the 'cs' value to.
+// If not specified, defaults to 3
+func LSTMBlockCellCellClip(value float32) LSTMBlockCellAttr {
+	return func(m optionalAttr) {
+		m["cell_clip"] = value
+	}
+}
+
+// LSTMBlockCellUsePeephole sets the optional use_peephole attribute to value.
+//
+// value: Whether to use peephole weights.
+// If not specified, defaults to false
+func LSTMBlockCellUsePeephole(value bool) LSTMBlockCellAttr {
+	return func(m optionalAttr) {
+		m["use_peephole"] = value
+	}
+}
+
+// Computes the LSTM cell forward propagation for 1 time step.
+//
+// This implementation uses 1 weight matrix and 1 bias vector, and there's an
+// optional peephole connection.
+//
+// This kernel op implements the following mathematical equations:
+//
+// ```python
+// xh = [x, h_prev]
+// [i, f, ci, o] = xh * w + b
+// f = f + forget_bias
+//
+// if not use_peephole:
+//   wci = wcf = wco = 0
+//
+// i = sigmoid(cs_prev * wci + i)
+// f = sigmoid(cs_prev * wcf + f)
+// ci = tanh(ci)
+//
+// cs = ci .* i + cs_prev .* f
+// cs = clip(cs, cell_clip)
+//
+// o = sigmoid(cs * wco + o)
+// co = tanh(cs)
+// h = co .* o
+// ```
+//
+// Arguments:
+//	x: The input to the LSTM cell, shape (batch_size, num_inputs).
+//	cs_prev: Value of the cell state at previous time step.
+//	h_prev: Output of the previous cell at previous time step.
+//	w: The weight matrix.
+//	wci: The weight matrix for input gate peephole connection.
+//	wcf: The weight matrix for forget gate peephole connection.
+//	wco: The weight matrix for output gate peephole connection.
+//	b: The bias vector.
+//
+// Returns:
+//	i: The input gate.
+//	cs: The cell state before the tanh.
+//	f: The forget gate.
+//	o: The output gate.
+//	ci: The cell input.
+//	co: The cell after the tanh.
+//	h: The output h vector.
+func LSTMBlockCell(scope *Scope, x tf.Output, cs_prev tf.Output, h_prev tf.Output, w tf.Output, wci tf.Output, wcf tf.Output, wco tf.Output, b tf.Output, optional ...LSTMBlockCellAttr) (i tf.Output, cs tf.Output, f tf.Output, o tf.Output, ci tf.Output, co tf.Output, h tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LSTMBlockCell",
+		Input: []tf.Input{
+			x, cs_prev, h_prev, w, wci, wcf, wco, b,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3), op.Output(4), op.Output(5), op.Output(6)
+}
+
+// A TPU core selector Op.
+//
+// This Op produces a set of TPU cores (for warm-up) or a single TPU core
+// (for regular inference) to execute the TPU program on. The output is
+// consumed by TPUPartitionedCall.
+//
+// Returns A vector 1 or more TPU cores.
+func TPUOrdinalSelector(scope *Scope) (device_ordinals tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TPUOrdinalSelector",
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SubstrAttr is an optional argument to Substr.
+type SubstrAttr func(optionalAttr)
+
+// SubstrUnit sets the optional unit attribute to value.
+//
+// value: The unit that is used to create the substring.  One of: `"BYTE"` (for
+// defining position and length by bytes) or `"UTF8_CHAR"` (for the UTF-8
+// encoded Unicode code points).  The default is `"BYTE"`. Results are undefined if
+// `unit=UTF8_CHAR` and the `input` strings do not contain structurally valid
+// UTF-8.
+// If not specified, defaults to "BYTE"
+func SubstrUnit(value string) SubstrAttr {
+	return func(m optionalAttr) {
+		m["unit"] = value
+	}
+}
+
+// Return substrings from `Tensor` of strings.
+//
+// For each string in the input `Tensor`, creates a substring starting at index
+// `pos` with a total length of `len`.
+//
+// If `len` defines a substring that would extend beyond the length of the input
+// string, or if `len` is negative, then as many characters as possible are used.
+//
+// A negative `pos` indicates distance within the string backwards from the end.
+//
+// If `pos` specifies an index which is out of range for any of the input strings,
+// then an `InvalidArgumentError` is thrown.
+//
+// `pos` and `len` must have the same shape, otherwise a `ValueError` is thrown on
+// Op creation.
+//
+// *NOTE*: `Substr` supports broadcasting up to two dimensions. More about
+// broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+//
+// ---
+//
+// Examples
+//
+// Using scalar `pos` and `len`:
+//
+// ```python
+// input = [b'Hello', b'World']
+// position = 1
+// length = 3
+//
+// output = [b'ell', b'orl']
+// ```
+//
+// Using `pos` and `len` with same shape as `input`:
+//
+// ```python
+// input = [[b'ten', b'eleven', b'twelve'],
+//          [b'thirteen', b'fourteen', b'fifteen'],
+//          [b'sixteen', b'seventeen', b'eighteen']]
+// position = [[1, 2, 3],
+//             [1, 2, 3],
+//             [1, 2, 3]]
+// length =   [[2, 3, 4],
+//             [4, 3, 2],
+//             [5, 5, 5]]
+//
+// output = [[b'en', b'eve', b'lve'],
+//           [b'hirt', b'urt', b'te'],
+//           [b'ixtee', b'vente', b'hteen']]
+// ```
+//
+// Broadcasting `pos` and `len` onto `input`:
+//
+// ```
+// input = [[b'ten', b'eleven', b'twelve'],
+//          [b'thirteen', b'fourteen', b'fifteen'],
+//          [b'sixteen', b'seventeen', b'eighteen'],
+//          [b'nineteen', b'twenty', b'twentyone']]
+// position = [1, 2, 3]
+// length =   [1, 2, 3]
+//
+// output = [[b'e', b'ev', b'lve'],
+//           [b'h', b'ur', b'tee'],
+//           [b'i', b've', b'hte'],
+//           [b'i', b'en', b'nty']]
+// ```
+//
+// Broadcasting `input` onto `pos` and `len`:
+//
+// ```
+// input = b'thirteen'
+// position = [1, 5, 7]
+// length =   [3, 2, 1]
+//
+// output = [b'hir', b'ee', b'n']
+// ```
+//
+// Raises:
+//
+//   * `ValueError`: If the first argument cannot be converted to a
+//      Tensor of `dtype string`.
+//   * `InvalidArgumentError`: If indices are out of range.
+//   * `ValueError`: If `pos` and `len` are not the same shape.
+//
+//
+// Arguments:
+//	input: Tensor of strings
+//	pos: Scalar defining the position of first character in each substring
+//	len: Scalar defining the number of characters to include in each substring
+//
+// Returns Tensor of substrings
+func Substr(scope *Scope, input tf.Output, pos tf.Output, len tf.Output, optional ...SubstrAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Substr",
+		Input: []tf.Input{
+			input, pos, len,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Convert JSON-encoded Example records to binary protocol buffer strings.
+//
+// This op translates a tensor containing Example records, encoded using
+// the [standard JSON
+// mapping](https://developers.google.com/protocol-buffers/docs/proto3#json),
+// into a tensor containing the same records encoded as binary protocol
+// buffers. The resulting tensor can then be fed to any of the other
+// Example-parsing ops.
+//
+// Arguments:
+//	json_examples: Each string is a JSON object serialized according to the JSON
+// mapping of the Example proto.
+//
+// Returns Each string is a binary Example protocol buffer corresponding
+// to the respective element of `json_examples`.
+func DecodeJSONExample(scope *Scope, json_examples tf.Output) (binary_examples tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "DecodeJSONExample",
+		Input: []tf.Input{
+			json_examples,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Elementwise computes the bitwise AND of `x` and `y`.
+//
+// The result will have those bits set, that are set in both `x` and `y`. The
+// computation is performed on the underlying representations of `x` and `y`.
+//
+// For example:
+//
+// ```python
+// import tensorflow as tf
+// from tensorflow.python.ops import bitwise_ops
+// dtype_list = [tf.int8, tf.int16, tf.int32, tf.int64,
+//               tf.uint8, tf.uint16, tf.uint32, tf.uint64]
+//
+// for dtype in dtype_list:
+//   lhs = tf.constant([0, 5, 3, 14], dtype=dtype)
+//   rhs = tf.constant([5, 0, 7, 11], dtype=dtype)
+//   exp = tf.constant([0, 0, 3, 10], dtype=tf.float32)
+//
+//   res = bitwise_ops.bitwise_and(lhs, rhs)
+//   tf.assert_equal(tf.cast(res, tf.float32), exp) # TRUE
+// ```
+//
+func BitwiseAnd(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "BitwiseAnd",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// DecodeCSVAttr is an optional argument to DecodeCSV.
+type DecodeCSVAttr func(optionalAttr)
+
+// DecodeCSVFieldDelim sets the optional field_delim attribute to value.
+//
+// value: char delimiter to separate fields in a record.
+// If not specified, defaults to ","
+func DecodeCSVFieldDelim(value string) DecodeCSVAttr {
+	return func(m optionalAttr) {
+		m["field_delim"] = value
+	}
+}
+
+// DecodeCSVUseQuoteDelim sets the optional use_quote_delim attribute to value.
+//
+// value: If false, treats double quotation marks as regular
+// characters inside of the string fields (ignoring RFC 4180, Section 2,
+// Bullet 5).
+// If not specified, defaults to true
+func DecodeCSVUseQuoteDelim(value bool) DecodeCSVAttr {
+	return func(m optionalAttr) {
+		m["use_quote_delim"] = value
+	}
+}
+
+// DecodeCSVNaValue sets the optional na_value attribute to value.
+//
+// value: Additional string to recognize as NA/NaN.
+// If not specified, defaults to ""
+func DecodeCSVNaValue(value string) DecodeCSVAttr {
+	return func(m optionalAttr) {
+		m["na_value"] = value
+	}
+}
+
+// DecodeCSVSelectCols sets the optional select_cols attribute to value.
+// If not specified, defaults to <>
+func DecodeCSVSelectCols(value []int64) DecodeCSVAttr {
+	return func(m optionalAttr) {
+		m["select_cols"] = value
+	}
+}
+
+// Convert CSV records to tensors. Each column maps to one tensor.
+//
+// RFC 4180 format is expected for the CSV records.
+// (https://tools.ietf.org/html/rfc4180)
+// Note that we allow leading and trailing spaces with int or float field.
+//
+// Arguments:
+//	records: Each string is a record/row in the csv and all records should have
+// the same format.
+//	record_defaults: One tensor per column of the input record, with either a
+// scalar default value for that column or an empty vector if the column is
+// required.
+//
+// Returns Each tensor will have the same shape as records.
+func DecodeCSV(scope *Scope, records tf.Output, record_defaults []tf.Output, optional ...DecodeCSVAttr) (output []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DecodeCSV",
+		Input: []tf.Input{
+			records, tf.OutputList(record_defaults),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if output, idx, err = makeOutputList(op, idx, "output"); err != nil {
+		scope.UpdateErr("DecodeCSV", err)
+		return
+	}
+	return output
+}
+
+// SerializeIteratorAttr is an optional argument to SerializeIterator.
+type SerializeIteratorAttr func(optionalAttr)
+
+// SerializeIteratorExternalStatePolicy sets the optional external_state_policy attribute to value.
+// If not specified, defaults to 0
+func SerializeIteratorExternalStatePolicy(value int64) SerializeIteratorAttr {
+	return func(m optionalAttr) {
+		m["external_state_policy"] = value
+	}
+}
+
+// Converts the given `resource_handle` representing an iterator to a variant tensor.
+//
+// Arguments:
+//	resource_handle: A handle to an iterator resource.
+//
+// Returns A variant tensor storing the state of the iterator contained in the
+// resource.
+func SerializeIterator(scope *Scope, resource_handle tf.Output, optional ...SerializeIteratorAttr) (serialized tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SerializeIterator",
+		Input: []tf.Input{
+			resource_handle,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResourceApplyCenteredRMSPropAttr is an optional argument to ResourceApplyCenteredRMSProp.
+type ResourceApplyCenteredRMSPropAttr func(optionalAttr)
+
+// ResourceApplyCenteredRMSPropUseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var, mg, ms, and mom tensors is
+// protected by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceApplyCenteredRMSPropUseLocking(value bool) ResourceApplyCenteredRMSPropAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// Update '*var' according to the centered RMSProp algorithm.
+//
+// The centered RMSProp algorithm uses an estimate of the centered second moment
+// (i.e., the variance) for normalization, as opposed to regular RMSProp, which
+// uses the (uncentered) second moment. This often helps with training, but is
+// slightly more expensive in terms of computation and memory.
+//
+// Note that in dense implementation of this algorithm, mg, ms, and mom will
+// update even if the grad is zero, but in this sparse implementation, mg, ms,
+// and mom will not update in iterations during which the grad is zero.
+//
+// mean_square = decay * mean_square + (1-decay) * gradient ** 2
+// mean_grad = decay * mean_grad + (1-decay) * gradient
+//
+// Delta = learning_rate * gradient / sqrt(mean_square + epsilon - mean_grad ** 2)
+//
+// mg <- rho * mg_{t-1} + (1-rho) * grad
+// ms <- rho * ms_{t-1} + (1-rho) * grad * grad
+// mom <- momentum * mom_{t-1} + lr * grad / sqrt(ms - mg * mg + epsilon)
+// var <- var - mom
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	mg: Should be from a Variable().
+//	ms: Should be from a Variable().
+//	mom: Should be from a Variable().
+//	lr: Scaling factor. Must be a scalar.
+//	rho: Decay rate. Must be a scalar.
+//
+//	epsilon: Ridge term. Must be a scalar.
+//	grad: The gradient.
+//
+// Returns the created operation.
+func ResourceApplyCenteredRMSProp(scope *Scope, var_ tf.Output, mg tf.Output, ms tf.Output, mom tf.Output, lr tf.Output, rho tf.Output, momentum tf.Output, epsilon tf.Output, grad tf.Output, optional ...ResourceApplyCenteredRMSPropAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceApplyCenteredRMSProp",
+		Input: []tf.Input{
+			var_, mg, ms, mom, lr, rho, momentum, epsilon, grad,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// UnsortedSegmentJoinAttr is an optional argument to UnsortedSegmentJoin.
+type UnsortedSegmentJoinAttr func(optionalAttr)
+
+// UnsortedSegmentJoinSeparator sets the optional separator attribute to value.
+//
+// value: The separator to use when joining.
+// If not specified, defaults to ""
+func UnsortedSegmentJoinSeparator(value string) UnsortedSegmentJoinAttr {
+	return func(m optionalAttr) {
+		m["separator"] = value
+	}
+}
+
+// Joins the elements of `inputs` based on `segment_ids`.
+//
+// Computes the string join along segments of a tensor.
+// Given `segment_ids` with rank `N` and `data` with rank `N+M`:
+//
+//     `output[i, k1...kM] = strings.join([data[j1...jN, k1...kM])`
+//
+// where the join is over all [j1...jN] such that segment_ids[j1...jN] = i.
+// Strings are joined in row-major order.
+//
+// For example:
+//
+// ```python
+// inputs = [['Y', 'q', 'c'], ['Y', '6', '6'], ['p', 'G', 'a']]
+// output_array = string_ops.unsorted_segment_join(inputs=inputs,
+//                                                 segment_ids=[1, 0, 1],
+//                                                 num_segments=2,
+//                                                 separator=':'))
+// # output_array ==> [['Y', '6', '6'], ['Y:p', 'q:G', 'c:a']]
+//
+//
+// inputs = ['this', 'is', 'a', 'test']
+// output_array = string_ops.unsorted_segment_join(inputs=inputs,
+//                                                 segment_ids=[0, 0, 0, 0],
+//                                                 num_segments=1,
+//                                                 separator=':'))
+// # output_array ==> ['this:is:a:test']
+// ```
+//
+// Arguments:
+//	inputs: The input to be joined.
+//	segment_ids: A tensor whose shape is a prefix of data.shape.  Negative segment ids are not
+// supported.
+//	num_segments: A scalar.
+func UnsortedSegmentJoin(scope *Scope, inputs tf.Output, segment_ids tf.Output, num_segments tf.Output, optional ...UnsortedSegmentJoinAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "UnsortedSegmentJoin",
+		Input: []tf.Input{
+			inputs, segment_ids, num_segments,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// LuAttr is an optional argument to Lu.
+type LuAttr func(optionalAttr)
+
+// LuOutputIdxType sets the optional output_idx_type attribute to value.
+// If not specified, defaults to DT_INT32
+func LuOutputIdxType(value tf.DataType) LuAttr {
+	return func(m optionalAttr) {
+		m["output_idx_type"] = value
+	}
+}
+
+// Computes the LU decomposition of one or more square matrices.
+//
+// The input is a tensor of shape `[..., M, M]` whose inner-most 2 dimensions
+// form square matrices.
+//
+// The input has to be invertible.
+//
+// The output consists of two tensors LU and P containing the LU decomposition
+// of all input submatrices `[..., :, :]`. LU encodes the lower triangular and
+// upper triangular factors.
+//
+// For each input submatrix of shape `[M, M]`, L is a lower triangular matrix of
+// shape `[M, M]` with unit diagonal whose entries correspond to the strictly lower
+// triangular part of LU. U is a upper triangular matrix of shape `[M, M]` whose
+// entries correspond to the upper triangular part, including the diagonal, of LU.
+//
+// P represents a permutation matrix encoded as a list of indices each between `0`
+// and `M-1`, inclusive. If P_mat denotes the permutation matrix corresponding to
+// P, then the L, U and P satisfies P_mat * input = L * U.
+//
+// Arguments:
+//	input: A tensor of shape `[..., M, M]` whose inner-most 2 dimensions form matrices of
+// size `[M, M]`.
+//
+// Returns:
+//	lu: A tensor of shape `[..., M, M]` whose strictly lower triangular part denotes the
+// lower triangular factor `L` with unit diagonal, and whose upper triangular part
+// denotes the upper triangular factor `U`.
+//	p: Permutation of the rows encoded as a list of indices in `0..M-1`. Shape is
+// `[..., M]`.
+// @compatibility(scipy)
+// Similar to `scipy.linalg.lu`, except the triangular factors `L` and `U` are
+// packed into a single tensor, the permutation is applied to `input` instead of
+// the right hand side and the permutation `P` is returned as a list of indices
+// instead of a permutation matrix.
+// @end_compatibility
+func Lu(scope *Scope, input tf.Output, optional ...LuAttr) (lu tf.Output, p tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Lu",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Outputs deterministic pseudorandom random numbers from a Poisson distribution.
+//
+// Outputs random values from a Poisson distribution.
+//
+// The outputs are a deterministic function of `shape`, `seed`, and `lam`.
+//
+// Arguments:
+//	shape: The shape of the output tensor.
+//	seed: 2 seeds (shape [2]).
+//	lam: The rate of the Poisson distribution. Shape must match the rightmost dimensions
+// of `shape`.
+//	dtype: The type of the output.
+//
+// Returns Random values with specified shape.
+func StatelessRandomPoisson(scope *Scope, shape tf.Output, seed tf.Output, lam tf.Output, dtype tf.DataType) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	opspec := tf.OpSpec{
+		Type: "StatelessRandomPoisson",
+		Input: []tf.Input{
+			shape, seed, lam,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns the truth value of `NOT x` element-wise.
+//
+// Arguments:
+//	x: A `Tensor` of type `bool`.
+//
+// Returns A `Tensor` of type `bool` with the same shape as `x`. The logical negation of `x`.
+func LogicalNot(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "LogicalNot",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ImageProjectiveTransformV2Attr is an optional argument to ImageProjectiveTransformV2.
+type ImageProjectiveTransformV2Attr func(optionalAttr)
+
+// ImageProjectiveTransformV2FillMode sets the optional fill_mode attribute to value.
+//
+// value: Fill mode, "REFLECT", "WRAP", or "CONSTANT".
+// If not specified, defaults to "CONSTANT"
+func ImageProjectiveTransformV2FillMode(value string) ImageProjectiveTransformV2Attr {
+	return func(m optionalAttr) {
+		m["fill_mode"] = value
+	}
+}
+
+// Applies the given transform to each of the images.
+//
+// If one row of `transforms` is `[a0, a1, a2, b0, b1, b2, c0, c1]`, then it maps
+// the *output* point `(x, y)` to a transformed *input* point
+// `(x', y') = ((a0 x + a1 y + a2) / k, (b0 x + b1 y + b2) / k)`, where
+// `k = c0 x + c1 y + 1`. If the transformed point lays outside of the input
+// image, the output pixel is set to 0.
+//
+// Arguments:
+//	images: 4-D with shape `[batch, height, width, channels]`.
+//	transforms: 2-D Tensor, `[batch, 8]` or `[1, 8]` matrix, where each row corresponds to a 3 x 3
+// projective transformation matrix, with the last entry assumed to be 1. If there
+// is one row, the same transformation will be applied to all images.
+//	output_shape: 1-D Tensor [new_height, new_width].
+//	interpolation: Interpolation method, "NEAREST" or "BILINEAR".
+//
+// Returns 4-D with shape
+// `[batch, new_height, new_width, channels]`.
+func ImageProjectiveTransformV2(scope *Scope, images tf.Output, transforms tf.Output, output_shape tf.Output, interpolation string, optional ...ImageProjectiveTransformV2Attr) (transformed_images tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"interpolation": interpolation}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ImageProjectiveTransformV2",
+		Input: []tf.Input{
+			images, transforms, output_shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes rectified linear gradients for a Relu operation.
+//
+// Arguments:
+//	gradients: The backpropagated gradients to the corresponding Relu operation.
+//	features: The features passed as input to the corresponding Relu operation, OR
+// the outputs of that operation (both work equivalently).
+//
+// Returns `gradients * (features > 0)`.
+func ReluGrad(scope *Scope, gradients tf.Output, features tf.Output) (backprops tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ReluGrad",
+		Input: []tf.Input{
+			gradients, features,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResourceApplyMomentumAttr is an optional argument to ResourceApplyMomentum.
+type ResourceApplyMomentumAttr func(optionalAttr)
+
+// ResourceApplyMomentumUseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var and accum tensors will be protected
+// by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceApplyMomentumUseLocking(value bool) ResourceApplyMomentumAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// ResourceApplyMomentumUseNesterov sets the optional use_nesterov attribute to value.
+//
+// value: If `True`, the tensor passed to compute grad will be
+// var - lr * momentum * accum, so in the end, the var you get is actually
+// var - lr * momentum * accum.
+// If not specified, defaults to false
+func ResourceApplyMomentumUseNesterov(value bool) ResourceApplyMomentumAttr {
+	return func(m optionalAttr) {
+		m["use_nesterov"] = value
+	}
+}
+
+// Update '*var' according to the momentum scheme.
+//
+// Set use_nesterov = True if you want to use Nesterov momentum.
+//
+// accum = accum * momentum + grad
+// var -= lr * accum
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	accum: Should be from a Variable().
+//	lr: Scaling factor. Must be a scalar.
+//	grad: The gradient.
+//	momentum: Momentum. Must be a scalar.
+//
+// Returns the created operation.
+func ResourceApplyMomentum(scope *Scope, var_ tf.Output, accum tf.Output, lr tf.Output, grad tf.Output, momentum tf.Output, optional ...ResourceApplyMomentumAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceApplyMomentum",
+		Input: []tf.Input{
+			var_, accum, lr, grad, momentum,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// IRFFTAttr is an optional argument to IRFFT.
+type IRFFTAttr func(optionalAttr)
+
+// IRFFTTreal sets the optional Treal attribute to value.
+// If not specified, defaults to DT_FLOAT
+func IRFFTTreal(value tf.DataType) IRFFTAttr {
+	return func(m optionalAttr) {
+		m["Treal"] = value
+	}
+}
+
+// Inverse real-valued fast Fourier transform.
+//
+// Computes the inverse 1-dimensional discrete Fourier transform of a real-valued
+// signal over the inner-most dimension of `input`.
+//
+// The inner-most dimension of `input` is assumed to be the result of `RFFT`: the
+// `fft_length / 2 + 1` unique components of the DFT of a real-valued signal. If
+// `fft_length` is not provided, it is computed from the size of the inner-most
+// dimension of `input` (`fft_length = 2 * (inner - 1)`). If the FFT length used to
+// compute `input` is odd, it should be provided since it cannot be inferred
+// properly.
+//
+// Along the axis `IRFFT` is computed on, if `fft_length / 2 + 1` is smaller
+// than the corresponding dimension of `input`, the dimension is cropped. If it is
+// larger, the dimension is padded with zeros.
+//
+// Arguments:
+//	input: A complex tensor.
+//	fft_length: An int32 tensor of shape [1]. The FFT length.
+//
+// Returns A float32 tensor of the same rank as `input`. The inner-most
+//   dimension of `input` is replaced with the `fft_length` samples of its inverse
+//   1D Fourier transform.
+//
+// @compatibility(numpy)
+// Equivalent to np.fft.irfft
+// @end_compatibility
+func IRFFT(scope *Scope, input tf.Output, fft_length tf.Output, optional ...IRFFTAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "IRFFT",
+		Input: []tf.Input{
+			input, fft_length,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// EnqueueTPUEmbeddingSparseBatchAttr is an optional argument to EnqueueTPUEmbeddingSparseBatch.
+type EnqueueTPUEmbeddingSparseBatchAttr func(optionalAttr)
+
+// EnqueueTPUEmbeddingSparseBatchDeviceOrdinal sets the optional device_ordinal attribute to value.
+//
+// value: The TPU device to use. Should be >= 0 and less than the number
+// of TPU cores in the task on which the node is placed.
+// If not specified, defaults to -1
+func EnqueueTPUEmbeddingSparseBatchDeviceOrdinal(value int64) EnqueueTPUEmbeddingSparseBatchAttr {
+	return func(m optionalAttr) {
+		m["device_ordinal"] = value
+	}
+}
+
+// EnqueueTPUEmbeddingSparseBatchCombiners sets the optional combiners attribute to value.
+//
+// value: A list of string scalars, one for each embedding table that specify
+// how to normalize the embedding activations after weighted summation.
+// Supported combiners are 'mean', 'sum', or 'sqrtn'. It is invalid to have
+// the sum of the weights be 0 for 'mean' or the sum of the squared weights be
+// 0 for 'sqrtn'. If combiners isn't passed, the default is to use 'sum' for
+// all tables.
+// If not specified, defaults to <>
+func EnqueueTPUEmbeddingSparseBatchCombiners(value []string) EnqueueTPUEmbeddingSparseBatchAttr {
+	return func(m optionalAttr) {
+		m["combiners"] = value
+	}
+}
+
+// An op that enqueues TPUEmbedding input indices from a SparseTensor.
+//
+// This Op eases the porting of code that uses embedding_lookup_sparse(),
+// although some Python preprocessing of the SparseTensor arguments to
+// embedding_lookup_sparse() is required to produce the arguments to this Op,
+// since only a single EnqueueTPUEmbeddingSparseBatch Op is allowed per training
+// step.
+//
+// The tensors at corresponding positions in the three input lists
+// must have the same shape, i.e. rank 1 with dim_size() equal to the total
+// number of lookups into the table described by the corresponding table_id.
+//
+// Arguments:
+//	sample_indices: A list of rank 1 Tensors specifying the training example and
+// feature to which the corresponding embedding_indices and aggregation_weights
+// values belong. sample_indices[i] must equal b * nf + f, where nf is the
+// number of features from the corresponding table, f is in [0, nf), and
+// b is in [0, batch size).
+//	embedding_indices: A list of rank 1 Tensors, indices into the embedding tables.
+//	aggregation_weights: A list of rank 1 Tensors containing per sample -- i.e. per
+// (training example, feature) -- aggregation weights.
+//	mode_override: A string input that overrides the mode specified in the
+// TPUEmbeddingConfiguration. Supported values are {'unspecified', 'inference',
+// 'training', 'backward_pass_only'}. When set to 'unspecified', the mode set
+// in TPUEmbeddingConfiguration is used, otherwise mode_override is used.
+//
+// Returns the created operation.
+func EnqueueTPUEmbeddingSparseBatch(scope *Scope, sample_indices []tf.Output, embedding_indices []tf.Output, aggregation_weights []tf.Output, mode_override tf.Output, optional ...EnqueueTPUEmbeddingSparseBatchAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "EnqueueTPUEmbeddingSparseBatch",
+		Input: []tf.Input{
+			tf.OutputList(sample_indices), tf.OutputList(embedding_indices), tf.OutputList(aggregation_weights), mode_override,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// ResourceScatterNdUpdateAttr is an optional argument to ResourceScatterNdUpdate.
+type ResourceScatterNdUpdateAttr func(optionalAttr)
+
+// ResourceScatterNdUpdateUseLocking sets the optional use_locking attribute to value.
+//
+// value: An optional bool. Defaults to True. If True, the assignment will
+// be protected by a lock; otherwise the behavior is undefined,
+// but may exhibit less contention.
+// If not specified, defaults to true
+func ResourceScatterNdUpdateUseLocking(value bool) ResourceScatterNdUpdateAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// Applies sparse `updates` to individual values or slices within a given
+//
+// variable according to `indices`.
+//
+// `ref` is a `Tensor` with rank `P` and `indices` is a `Tensor` of rank `Q`.
+//
+// `indices` must be integer tensor, containing indices into `ref`.
+// It must be shape `[d_0, ..., d_{Q-2}, K]` where `0 < K <= P`.
+//
+// The innermost dimension of `indices` (with length `K`) corresponds to
+// indices into elements (if `K = P`) or slices (if `K < P`) along the `K`th
+// dimension of `ref`.
+//
+// `updates` is `Tensor` of rank `Q-1+P-K` with shape:
+//
+// ```
+// [d_0, ..., d_{Q-2}, ref.shape[K], ..., ref.shape[P-1]].
+// ```
+//
+// For example, say we want to update 4 scattered elements to a rank-1 tensor to
+// 8 elements. In Python, that update would look like this:
+//
+// ```python
+//     ref = tf.Variable([1, 2, 3, 4, 5, 6, 7, 8])
+//     indices = tf.constant([[4], [3], [1] ,[7]])
+//     updates = tf.constant([9, 10, 11, 12])
+//     update = tf.scatter_nd_update(ref, indices, updates)
+//     with tf.Session() as sess:
+//       print sess.run(update)
+// ```
+//
+// The resulting update to ref would look like this:
+//
+//     [1, 11, 3, 10, 9, 6, 7, 12]
+//
+// See `tf.scatter_nd` for more details about how to make updates to
+// slices.
+//
+// Arguments:
+//	ref: A resource handle. Must be from a VarHandleOp.
+//	indices: A Tensor. Must be one of the following types: int32, int64.
+// A tensor of indices into ref.
+//	updates: A Tensor. Must have the same type as ref. A tensor of updated
+// values to add to ref.
+//
+// Returns the created operation.
+func ResourceScatterNdUpdate(scope *Scope, ref tf.Output, indices tf.Output, updates tf.Output, optional ...ResourceScatterNdUpdateAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceScatterNdUpdate",
+		Input: []tf.Input{
+			ref, indices, updates,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Computes square root of x element-wise.
+//
+// I.e., \\(y = \sqrt{x} = x^{1/2}\\).
+func Sqrt(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Sqrt",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Gradient op for `MirrorPad` op. This op folds a mirror-padded tensor.
+//
+// This operation folds the padded areas of `input` by `MirrorPad` according to the
+// `paddings` you specify. `paddings` must be the same as `paddings` argument
+// given to the corresponding `MirrorPad` op.
+//
+// The folded size of each dimension D of the output is:
+//
+// `input.dim_size(D) - paddings(D, 0) - paddings(D, 1)`
+//
+// For example:
+//
+// ```
+// # 't' is [[1, 2, 3], [4, 5, 6], [7, 8, 9]].
+// # 'paddings' is [[0, 1]], [0, 1]].
+// # 'mode' is SYMMETRIC.
+// # rank of 't' is 2.
+// pad(t, paddings) ==> [[ 1,  5]
+//                       [11, 28]]
+// ```
+//
+// Arguments:
+//	input: The input tensor to be folded.
+//	paddings: A two-column matrix specifying the padding sizes. The number of
+// rows must be the same as the rank of `input`.
+//	mode: The mode used in the `MirrorPad` op.
+//
+// Returns The folded tensor.
+func MirrorPadGrad(scope *Scope, input tf.Output, paddings tf.Output, mode string) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"mode": mode}
+	opspec := tf.OpSpec{
+		Type: "MirrorPadGrad",
+		Input: []tf.Input{
+			input, paddings,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Produces the max pool of the input tensor for quantized types.
+//
+// Arguments:
+//	input: The 4D (batch x rows x cols x depth) Tensor to MaxReduce over.
+//	min_input: The float value that the lowest quantized input value represents.
+//	max_input: The float value that the highest quantized input value represents.
+//	ksize: The size of the window for each dimension of the input tensor.
+// The length must be 4 to match the number of dimensions of the input.
+//	strides: The stride of the sliding window for each dimension of the input
+// tensor. The length must be 4 to match the number of dimensions of the input.
+//	padding: The type of padding algorithm to use.
+//
+// Returns:
+//	output
+//	min_output: The float value that the lowest quantized output value represents.
+//	max_output: The float value that the highest quantized output value represents.
+func QuantizedMaxPool(scope *Scope, input tf.Output, min_input tf.Output, max_input tf.Output, ksize []int64, strides []int64, padding string) (output tf.Output, min_output tf.Output, max_output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"ksize": ksize, "strides": strides, "padding": padding}
+	opspec := tf.OpSpec{
+		Type: "QuantizedMaxPool",
+		Input: []tf.Input{
+			input, min_input, max_input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// ResourceApplyAdagradAttr is an optional argument to ResourceApplyAdagrad.
+type ResourceApplyAdagradAttr func(optionalAttr)
+
+// ResourceApplyAdagradUseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var and accum tensors will be protected
+// by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceApplyAdagradUseLocking(value bool) ResourceApplyAdagradAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// ResourceApplyAdagradUpdateSlots sets the optional update_slots attribute to value.
+// If not specified, defaults to true
+func ResourceApplyAdagradUpdateSlots(value bool) ResourceApplyAdagradAttr {
+	return func(m optionalAttr) {
+		m["update_slots"] = value
+	}
+}
+
+// Update '*var' according to the adagrad scheme.
+//
+// accum += grad * grad
+// var -= lr * grad * (1 / sqrt(accum))
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	accum: Should be from a Variable().
+//	lr: Scaling factor. Must be a scalar.
+//	grad: The gradient.
+//
+// Returns the created operation.
+func ResourceApplyAdagrad(scope *Scope, var_ tf.Output, accum tf.Output, lr tf.Output, grad tf.Output, optional ...ResourceApplyAdagradAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceApplyAdagrad",
+		Input: []tf.Input{
+			var_, accum, lr, grad,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// The gradient of SparseFillEmptyRows.
+//
+// Takes vectors reverse_index_map, shaped `[N]`, and grad_values,
+// shaped `[N_full]`, where `N_full >= N` and copies data into either
+// `d_values` or `d_default_value`.  Here `d_values` is shaped `[N]` and
+// `d_default_value` is a scalar.
+//
+//   d_values[j] = grad_values[reverse_index_map[j]]
+//   d_default_value = sum_{k : 0 .. N_full - 1} (
+//      grad_values[k] * 1{k not in reverse_index_map})
+//
+// Arguments:
+//	reverse_index_map: 1-D.  The reverse index map from SparseFillEmptyRows.
+//	grad_values: 1-D.  The gradients from backprop.
+//
+// Returns:
+//	d_values: 1-D.  The backprop into values.
+//	d_default_value: 0-D.  The backprop into default_value.
+func SparseFillEmptyRowsGrad(scope *Scope, reverse_index_map tf.Output, grad_values tf.Output) (d_values tf.Output, d_default_value tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseFillEmptyRowsGrad",
+		Input: []tf.Input{
+			reverse_index_map, grad_values,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// MaxPool3DGradAttr is an optional argument to MaxPool3DGrad.
+type MaxPool3DGradAttr func(optionalAttr)
+
+// MaxPool3DGradDataFormat sets the optional data_format attribute to value.
+//
+// value: The data format of the input and output data. With the
+// default format "NDHWC", the data is stored in the order of:
+//     [batch, in_depth, in_height, in_width, in_channels].
+// Alternatively, the format could be "NCDHW", the data storage order is:
+//     [batch, in_channels, in_depth, in_height, in_width].
+// If not specified, defaults to "NDHWC"
+func MaxPool3DGradDataFormat(value string) MaxPool3DGradAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// Computes gradients of 3D max pooling function.
+//
+// Arguments:
+//	orig_input: The original input tensor.
+//	orig_output: The original output tensor.
+//	grad: Output backprop of shape `[batch, depth, rows, cols, channels]`.
+//	ksize: 1-D tensor of length 5. The size of the window for each dimension of
+// the input tensor. Must have `ksize[0] = ksize[4] = 1`.
+//	strides: 1-D tensor of length 5. The stride of the sliding window for each
+// dimension of `input`. Must have `strides[0] = strides[4] = 1`.
+//	padding: The type of padding algorithm to use.
+func MaxPool3DGrad(scope *Scope, orig_input tf.Output, orig_output tf.Output, grad tf.Output, ksize []int64, strides []int64, padding string, optional ...MaxPool3DGradAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"ksize": ksize, "strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MaxPool3DGrad",
+		Input: []tf.Input{
+			orig_input, orig_output, grad,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResourceApplyRMSPropAttr is an optional argument to ResourceApplyRMSProp.
+type ResourceApplyRMSPropAttr func(optionalAttr)
+
+// ResourceApplyRMSPropUseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var, ms, and mom tensors is protected
+// by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceApplyRMSPropUseLocking(value bool) ResourceApplyRMSPropAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// Update '*var' according to the RMSProp algorithm.
+//
+// Note that in dense implementation of this algorithm, ms and mom will
+// update even if the grad is zero, but in this sparse implementation, ms
+// and mom will not update in iterations during which the grad is zero.
+//
+// mean_square = decay * mean_square + (1-decay) * gradient ** 2
+// Delta = learning_rate * gradient / sqrt(mean_square + epsilon)
+//
+// ms <- rho * ms_{t-1} + (1-rho) * grad * grad
+// mom <- momentum * mom_{t-1} + lr * grad / sqrt(ms + epsilon)
+// var <- var - mom
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	ms: Should be from a Variable().
+//	mom: Should be from a Variable().
+//	lr: Scaling factor. Must be a scalar.
+//	rho: Decay rate. Must be a scalar.
+//
+//	epsilon: Ridge term. Must be a scalar.
+//	grad: The gradient.
+//
+// Returns the created operation.
+func ResourceApplyRMSProp(scope *Scope, var_ tf.Output, ms tf.Output, mom tf.Output, lr tf.Output, rho tf.Output, momentum tf.Output, epsilon tf.Output, grad tf.Output, optional ...ResourceApplyRMSPropAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceApplyRMSProp",
+		Input: []tf.Input{
+			var_, ms, mom, lr, rho, momentum, epsilon, grad,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Reshapes a SparseTensor to represent values in a new dense shape.
+//
+// This operation has the same semantics as reshape on the represented dense
+// tensor.  The `input_indices` are recomputed based on the requested `new_shape`.
+//
+// If one component of `new_shape` is the special value -1, the size of that
+// dimension is computed so that the total dense size remains constant.  At
+// most one component of `new_shape` can be -1.  The number of dense elements
+// implied by `new_shape` must be the same as the number of dense elements
+// originally implied by `input_shape`.
+//
+// Reshaping does not affect the order of values in the SparseTensor.
+//
+// If the input tensor has rank `R_in` and `N` non-empty values, and `new_shape`
+// has length `R_out`, then `input_indices` has shape `[N, R_in]`,
+// `input_shape` has length `R_in`, `output_indices` has shape `[N, R_out]`, and
+// `output_shape` has length `R_out`.
+//
+// Arguments:
+//	input_indices: 2-D.  `N x R_in` matrix with the indices of non-empty values in a
+// SparseTensor.
+//	input_shape: 1-D.  `R_in` vector with the input SparseTensor's dense shape.
+//	new_shape: 1-D.  `R_out` vector with the requested new dense shape.
+//
+// Returns:
+//	output_indices: 2-D.  `N x R_out` matrix with the updated indices of non-empty
+// values in the output SparseTensor.
+//	output_shape: 1-D.  `R_out` vector with the full dense shape of the output
+// SparseTensor.  This is the same as `new_shape` but with any -1 dimensions
+// filled in.
+func SparseReshape(scope *Scope, input_indices tf.Output, input_shape tf.Output, new_shape tf.Output) (output_indices tf.Output, output_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseReshape",
+		Input: []tf.Input{
+			input_indices, input_shape, new_shape,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Elementwise computes the bitwise left-shift of `x` and `y`.
+//
+// If `y` is negative, or greater than or equal to the width of `x` in bits the
+// result is implementation defined.
+//
+// Example:
+//
+// ```python
+// import tensorflow as tf
+// from tensorflow.python.ops import bitwise_ops
+// import numpy as np
+// dtype_list = [tf.int8, tf.int16, tf.int32, tf.int64]
+//
+// for dtype in dtype_list:
+//   lhs = tf.constant([-1, -5, -3, -14], dtype=dtype)
+//   rhs = tf.constant([5, 0, 7, 11], dtype=dtype)
+//
+//   left_shift_result = bitwise_ops.left_shift(lhs, rhs)
+//
+//   print(left_shift_result)
+//
+// # This will print:
+// # tf.Tensor([ -32   -5 -128    0], shape=(4,), dtype=int8)
+// # tf.Tensor([   -32     -5   -384 -28672], shape=(4,), dtype=int16)
+// # tf.Tensor([   -32     -5   -384 -28672], shape=(4,), dtype=int32)
+// # tf.Tensor([   -32     -5   -384 -28672], shape=(4,), dtype=int64)
+//
+// lhs = np.array([-2, 64, 101, 32], dtype=np.int8)
+// rhs = np.array([-1, -5, -3, -14], dtype=np.int8)
+// bitwise_ops.left_shift(lhs, rhs)
+// # <tf.Tensor: shape=(4,), dtype=int8, numpy=array([ -2,  64, 101,  32], dtype=int8)>
+// ```
+//
+func LeftShift(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "LeftShift",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Generates a feature cross from a list of tensors, and returns it as a
+// RaggedTensor.  See `tf.ragged.cross` for more details.
+//
+// Arguments:
+//	ragged_values: The values tensor for each RaggedTensor input.
+//	ragged_row_splits: The row_splits tensor for each RaggedTensor input.
+//	sparse_indices: The indices tensor for each SparseTensor input.
+//	sparse_values: The values tensor for each SparseTensor input.
+//	sparse_shape: The dense_shape tensor for each SparseTensor input.
+//	dense_inputs: The tf.Tensor inputs.
+//	input_order: String specifying the tensor type for each input.  The `i`th character in
+// this string specifies the type of the `i`th input, and is one of: 'R' (ragged),
+// 'D' (dense), or 'S' (sparse).  This attr is used to ensure that the crossed
+// values are combined in the order of the inputs from the call to tf.ragged.cross.
+//
+//
+//
+//
+//
+//
+// Returns:
+//	output_values: The `values` for the returned `RaggedTensor`.
+//	output_row_splits: The `row_splits` for the returned `RaggedTensor`.
+func RaggedCross(scope *Scope, ragged_values []tf.Output, ragged_row_splits []tf.Output, sparse_indices []tf.Output, sparse_values []tf.Output, sparse_shape []tf.Output, dense_inputs []tf.Output, input_order string, hashed_output bool, num_buckets int64, hash_key int64, out_values_type tf.DataType, out_row_splits_type tf.DataType) (output_values tf.Output, output_row_splits tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"input_order": input_order, "hashed_output": hashed_output, "num_buckets": num_buckets, "hash_key": hash_key, "out_values_type": out_values_type, "out_row_splits_type": out_row_splits_type}
+	opspec := tf.OpSpec{
+		Type: "RaggedCross",
+		Input: []tf.Input{
+			tf.OutputList(ragged_values), tf.OutputList(ragged_row_splits), tf.OutputList(sparse_indices), tf.OutputList(sparse_values), tf.OutputList(sparse_shape), tf.OutputList(dense_inputs),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Output a fact about factorials.
+func Fact(scope *Scope) (fact tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Fact",
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes softmax cross entropy cost and gradients to backpropagate.
+//
+// Unlike `SoftmaxCrossEntropyWithLogits`, this operation does not accept
+// a matrix of label probabilities, but rather a single label per row
+// of features.  This label is considered to have probability 1.0 for the
+// given row.
+//
+// Inputs are the logits, not probabilities.
+//
+// Arguments:
+//	features: batch_size x num_classes matrix
+//	labels: batch_size vector with values in [0, num_classes).
+// This is the label for the given minibatch entry.
+//
+// Returns:
+//	loss: Per example loss (batch_size vector).
+//	backprop: backpropagated gradients (batch_size x num_classes matrix).
+func SparseSoftmaxCrossEntropyWithLogits(scope *Scope, features tf.Output, labels tf.Output) (loss tf.Output, backprop tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseSoftmaxCrossEntropyWithLogits",
+		Input: []tf.Input{
+			features, labels,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Worker heartbeat op.
+//
+// Heartbeats may be sent periodically to indicate the coordinator is still active,
+// to retrieve the current worker status and to expedite shutdown when necessary.
+//
+// Arguments:
+//	request: A string tensor containing a serialized WorkerHeartbeatRequest
+//
+// Returns A string tensor containing a serialized WorkerHeartbeatResponse
+func WorkerHeartbeat(scope *Scope, request tf.Output) (response tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "WorkerHeartbeat",
+		Input: []tf.Input{
+			request,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResourceApplyProximalGradientDescentAttr is an optional argument to ResourceApplyProximalGradientDescent.
+type ResourceApplyProximalGradientDescentAttr func(optionalAttr)
+
+// ResourceApplyProximalGradientDescentUseLocking sets the optional use_locking attribute to value.
+//
+// value: If True, the subtraction will be protected by a lock;
+// otherwise the behavior is undefined, but may exhibit less contention.
+// If not specified, defaults to false
+func ResourceApplyProximalGradientDescentUseLocking(value bool) ResourceApplyProximalGradientDescentAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// Update '*var' as FOBOS algorithm with fixed learning rate.
+//
+// prox_v = var - alpha * delta
+// var = sign(prox_v)/(1+alpha*l2) * max{|prox_v|-alpha*l1,0}
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	alpha: Scaling factor. Must be a scalar.
+//	l1: L1 regularization. Must be a scalar.
+//	l2: L2 regularization. Must be a scalar.
+//	delta: The change.
+//
+// Returns the created operation.
+func ResourceApplyProximalGradientDescent(scope *Scope, var_ tf.Output, alpha tf.Output, l1 tf.Output, l2 tf.Output, delta tf.Output, optional ...ResourceApplyProximalGradientDescentAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceApplyProximalGradientDescent",
+		Input: []tf.Input{
+			var_, alpha, l1, l2, delta,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// RandomUniformAttr is an optional argument to RandomUniform.
+type RandomUniformAttr func(optionalAttr)
+
+// RandomUniformSeed sets the optional seed attribute to value.
+//
+// value: If either `seed` or `seed2` are set to be non-zero, the random number
+// generator is seeded by the given seed.  Otherwise, it is seeded by a
+// random seed.
+// If not specified, defaults to 0
+func RandomUniformSeed(value int64) RandomUniformAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// RandomUniformSeed2 sets the optional seed2 attribute to value.
+//
+// value: A second seed to avoid seed collision.
+// If not specified, defaults to 0
+func RandomUniformSeed2(value int64) RandomUniformAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Outputs random values from a uniform distribution.
+//
+// The generated values follow a uniform distribution in the range `[0, 1)`. The
+// lower bound 0 is included in the range, while the upper bound 1 is excluded.
+//
+// Arguments:
+//	shape: The shape of the output tensor.
+//	dtype: The type of the output.
+//
+// Returns A tensor of the specified shape filled with uniform random values.
+func RandomUniform(scope *Scope, shape tf.Output, dtype tf.DataType, optional ...RandomUniformAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RandomUniform",
+		Input: []tf.Input{
+			shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RetrieveTPUEmbeddingAdadeltaParametersGradAccumDebugAttr is an optional argument to RetrieveTPUEmbeddingAdadeltaParametersGradAccumDebug.
+type RetrieveTPUEmbeddingAdadeltaParametersGradAccumDebugAttr func(optionalAttr)
+
+// RetrieveTPUEmbeddingAdadeltaParametersGradAccumDebugTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func RetrieveTPUEmbeddingAdadeltaParametersGradAccumDebugTableId(value int64) RetrieveTPUEmbeddingAdadeltaParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingAdadeltaParametersGradAccumDebugTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingAdadeltaParametersGradAccumDebugTableName(value string) RetrieveTPUEmbeddingAdadeltaParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingAdadeltaParametersGradAccumDebugConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingAdadeltaParametersGradAccumDebugConfig(value string) RetrieveTPUEmbeddingAdadeltaParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Retrieve Adadelta embedding parameters with debug support.
+//
+// An op that retrieves optimization parameters from embedding to host
+// memory. Must be preceded by a ConfigureTPUEmbeddingHost op that sets up
+// the correct embedding table configuration. For example, this op is
+// used to retrieve updated parameters before saving a checkpoint.
+//
+// Returns:
+//	parameters: Parameter parameters updated by the Adadelta optimization algorithm.
+//	accumulators: Parameter accumulators updated by the Adadelta optimization algorithm.
+//	updates: Parameter updates updated by the Adadelta optimization algorithm.
+//	gradient_accumulators: Parameter gradient_accumulators updated by the Adadelta optimization algorithm.
+func RetrieveTPUEmbeddingAdadeltaParametersGradAccumDebug(scope *Scope, num_shards int64, shard_id int64, optional ...RetrieveTPUEmbeddingAdadeltaParametersGradAccumDebugAttr) (parameters tf.Output, accumulators tf.Output, updates tf.Output, gradient_accumulators tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RetrieveTPUEmbeddingAdadeltaParametersGradAccumDebug",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3)
+}
+
+// Retrieves the tree ensemble resource stamp token, number of trees and growing statistics.
+//
+// Arguments:
+//	tree_ensemble_handle: Handle to the tree ensemble.
+//
+// Returns:
+//	stamp_token: Stamp token of the tree ensemble resource.
+//	num_trees: The number of trees in the tree ensemble resource.
+//	num_finalized_trees: The number of trees that were finished successfully.
+//	num_attempted_layers: The number of layers we attempted to build (but not necessarily succeeded).
+//	last_layer_nodes_range: Rank size 2 tensor that contains start and end ids of the nodes in the latest
+// layer.
+func BoostedTreesGetEnsembleStates(scope *Scope, tree_ensemble_handle tf.Output) (stamp_token tf.Output, num_trees tf.Output, num_finalized_trees tf.Output, num_attempted_layers tf.Output, last_layer_nodes_range tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesGetEnsembleStates",
+		Input: []tf.Input{
+			tree_ensemble_handle,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3), op.Output(4)
+}
+
+// ResourceScatterNdAddAttr is an optional argument to ResourceScatterNdAdd.
+type ResourceScatterNdAddAttr func(optionalAttr)
+
+// ResourceScatterNdAddUseLocking sets the optional use_locking attribute to value.
+//
+// value: An optional bool. Defaults to True. If True, the assignment will
+// be protected by a lock; otherwise the behavior is undefined,
+// but may exhibit less contention.
+// If not specified, defaults to true
+func ResourceScatterNdAddUseLocking(value bool) ResourceScatterNdAddAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// Applies sparse addition to individual values or slices in a Variable.
+//
+// `ref` is a `Tensor` with rank `P` and `indices` is a `Tensor` of rank `Q`.
+//
+// `indices` must be integer tensor, containing indices into `ref`.
+// It must be shape `[d_0, ..., d_{Q-2}, K]` where `0 < K <= P`.
+//
+// The innermost dimension of `indices` (with length `K`) corresponds to
+// indices into elements (if `K = P`) or slices (if `K < P`) along the `K`th
+// dimension of `ref`.
+//
+// `updates` is `Tensor` of rank `Q-1+P-K` with shape:
+//
+// ```
+// [d_0, ..., d_{Q-2}, ref.shape[K], ..., ref.shape[P-1]]
+// ```
+//
+// For example, say we want to add 4 scattered elements to a rank-1 tensor to
+// 8 elements. In Python, that addition would look like this:
+//
+// ```python
+// ref = tf.Variable([1, 2, 3, 4, 5, 6, 7, 8], use_resource=True)
+// indices = tf.constant([[4], [3], [1], [7]])
+// updates = tf.constant([9, 10, 11, 12])
+// add = tf.scatter_nd_add(ref, indices, updates)
+// with tf.Session() as sess:
+//   print sess.run(add)
+// ```
+//
+// The resulting update to ref would look like this:
+//
+//     [1, 13, 3, 14, 14, 6, 7, 20]
+//
+// See `tf.scatter_nd` for more details about how to make updates to
+// slices.
+//
+// Arguments:
+//	ref: A resource handle. Must be from a VarHandleOp.
+//	indices: A Tensor. Must be one of the following types: int32, int64.
+// A tensor of indices into ref.
+//	updates: A Tensor. Must have the same type as ref. A tensor of
+// values to add to ref.
+//
+// Returns the created operation.
+func ResourceScatterNdAdd(scope *Scope, ref tf.Output, indices tf.Output, updates tf.Output, optional ...ResourceScatterNdAddAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceScatterNdAdd",
+		Input: []tf.Input{
+			ref, indices, updates,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Counts the number of occurrences of each value in an integer array.
+//
+// Outputs a vector with length `size` and the same dtype as `weights`. If
+// `weights` are empty, then index `i` stores the number of times the value `i` is
+// counted in `arr`. If `weights` are non-empty, then index `i` stores the sum of
+// the value in `weights` at each index where the corresponding value in `arr` is
+// `i`.
+//
+// Values in `arr` outside of the range [0, size) are ignored.
+//
+// Arguments:
+//	arr: int32 `Tensor`.
+//	size: non-negative int32 scalar `Tensor`.
+//	weights: is an int32, int64, float32, or float64 `Tensor` with the same
+// shape as `arr`, or a length-0 `Tensor`, in which case it acts as all weights
+// equal to 1.
+//
+// Returns 1D `Tensor` with length equal to `size`. The counts or summed weights for
+// each value in the range [0, size).
+func Bincount(scope *Scope, arr tf.Output, size tf.Output, weights tf.Output) (bins tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Bincount",
+		Input: []tf.Input{
+			arr, size, weights,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Gradients for batch normalization.
+//
+// DEPRECATED at GraphDef version 9: Use tf.nn.batch_normalization()
+//
+// This op is deprecated. See `tf.nn.batch_normalization`.
+//
+// Arguments:
+//	t: A 4D input Tensor.
+//	m: A 1D mean Tensor with size matching the last dimension of t.
+// This is the first output from tf.nn.moments,
+// or a saved moving average thereof.
+//	v: A 1D variance Tensor with size matching the last dimension of t.
+// This is the second output from tf.nn.moments,
+// or a saved moving average thereof.
+//	gamma: A 1D gamma Tensor with size matching the last dimension of t.
+// If "scale_after_normalization" is true, this Tensor will be multiplied
+// with the normalized Tensor.
+//	backprop: 4D backprop Tensor.
+//	variance_epsilon: A small float number to avoid dividing by 0.
+//	scale_after_normalization: A bool indicating whether the resulted tensor
+// needs to be multiplied with gamma.
+//
+// Returns:
+//	dx: 4D backprop tensor for input.
+//	dm: 1D backprop tensor for mean.
+//	dv: 1D backprop tensor for variance.
+//	db: 1D backprop tensor for beta.
+//	dg: 1D backprop tensor for gamma.
+func BatchNormWithGlobalNormalizationGrad(scope *Scope, t tf.Output, m tf.Output, v tf.Output, gamma tf.Output, backprop tf.Output, variance_epsilon float32, scale_after_normalization bool) (dx tf.Output, dm tf.Output, dv tf.Output, db tf.Output, dg tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"variance_epsilon": variance_epsilon, "scale_after_normalization": scale_after_normalization}
+	opspec := tf.OpSpec{
+		Type: "BatchNormWithGlobalNormalizationGrad",
+		Input: []tf.Input{
+			t, m, v, gamma, backprop,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3), op.Output(4)
+}
+
+// CropAndResizeGradImageAttr is an optional argument to CropAndResizeGradImage.
+type CropAndResizeGradImageAttr func(optionalAttr)
+
+// CropAndResizeGradImageMethod sets the optional method attribute to value.
+//
+// value: A string specifying the interpolation method. Only 'bilinear' is
+// supported for now.
+// If not specified, defaults to "bilinear"
+func CropAndResizeGradImageMethod(value string) CropAndResizeGradImageAttr {
+	return func(m optionalAttr) {
+		m["method"] = value
+	}
+}
+
+// Computes the gradient of the crop_and_resize op wrt the input image tensor.
+//
+// Arguments:
+//	grads: A 4-D tensor of shape `[num_boxes, crop_height, crop_width, depth]`.
+//	boxes: A 2-D tensor of shape `[num_boxes, 4]`. The `i`-th row of the tensor
+// specifies the coordinates of a box in the `box_ind[i]` image and is specified
+// in normalized coordinates `[y1, x1, y2, x2]`. A normalized coordinate value of
+// `y` is mapped to the image coordinate at `y * (image_height - 1)`, so as the
+// `[0, 1]` interval of normalized image height is mapped to
+// `[0, image_height - 1] in image height coordinates. We do allow y1 > y2, in
+// which case the sampled crop is an up-down flipped version of the original
+// image. The width dimension is treated similarly. Normalized coordinates
+// outside the `[0, 1]` range are allowed, in which case we use
+// `extrapolation_value` to extrapolate the input image values.
+//	box_ind: A 1-D tensor of shape `[num_boxes]` with int32 values in `[0, batch)`.
+// The value of `box_ind[i]` specifies the image that the `i`-th box refers to.
+//	image_size: A 1-D tensor with value `[batch, image_height, image_width, depth]`
+// containing the original image size. Both `image_height` and `image_width` need
+// to be positive.
+//
+//
+// Returns A 4-D tensor of shape `[batch, image_height, image_width, depth]`.
+func CropAndResizeGradImage(scope *Scope, grads tf.Output, boxes tf.Output, box_ind tf.Output, image_size tf.Output, T tf.DataType, optional ...CropAndResizeGradImageAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"T": T}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CropAndResizeGradImage",
+		Input: []tf.Input{
+			grads, boxes, box_ind, image_size,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// OutfeedDequeueAttr is an optional argument to OutfeedDequeue.
+type OutfeedDequeueAttr func(optionalAttr)
+
+// OutfeedDequeueDeviceOrdinal sets the optional device_ordinal attribute to value.
+//
+// value: The TPU device to use. This should be -1 when the Op
+// is running on a TPU device, and >= 0 when the Op is running on the CPU
+// device.
+// If not specified, defaults to -1
+func OutfeedDequeueDeviceOrdinal(value int64) OutfeedDequeueAttr {
+	return func(m optionalAttr) {
+		m["device_ordinal"] = value
+	}
+}
+
+// Retrieves a single tensor from the computation outfeed.
+//
+// This operation will block indefinitely until data is available.
+//
+// Arguments:
+//	dtype: The type of elements in the tensor.
+//	shape: The shape of the tensor.
+//
+// Returns A tensor that will be read from the device outfeed.
+func OutfeedDequeue(scope *Scope, dtype tf.DataType, shape tf.Shape, optional ...OutfeedDequeueAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype, "shape": shape}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "OutfeedDequeue",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// An Op to sum inputs across replicated TPU instances.
+//
+// Each instance supplies its own input.
+//
+// For example, suppose there are 8 TPU instances: `[A, B, C, D, E, F, G, H]`.
+// Passing group_assignment=`[[0,2,4,6],[1,3,5,7]]` sets `A, C, E, G` as group 0,
+// and `B, D, F, H` as group 1. Thus we get the outputs:
+// `[A+C+E+G, B+D+F+H, A+C+E+G, B+D+F+H, A+C+E+G, B+D+F+H, A+C+E+G, B+D+F+H]`.
+//
+// Arguments:
+//	input: The local input to the sum.
+//	group_assignment: An int32 tensor with shape
+// [num_groups, num_replicas_per_group]. `group_assignment[i]` represents the
+// replica ids in the ith subgroup.
+//
+// Returns The sum of all the distributed inputs.
+func CrossReplicaSum(scope *Scope, input tf.Output, group_assignment tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "CrossReplicaSum",
+		Input: []tf.Input{
+			input, group_assignment,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// EnqueueTPUEmbeddingRaggedTensorBatchAttr is an optional argument to EnqueueTPUEmbeddingRaggedTensorBatch.
+type EnqueueTPUEmbeddingRaggedTensorBatchAttr func(optionalAttr)
+
+// EnqueueTPUEmbeddingRaggedTensorBatchDeviceOrdinal sets the optional device_ordinal attribute to value.
+//
+// value: The TPU device to use. Should be >= 0 and less than the number
+// of TPU cores in the task on which the node is placed.
+// If not specified, defaults to -1
+func EnqueueTPUEmbeddingRaggedTensorBatchDeviceOrdinal(value int64) EnqueueTPUEmbeddingRaggedTensorBatchAttr {
+	return func(m optionalAttr) {
+		m["device_ordinal"] = value
+	}
+}
+
+// EnqueueTPUEmbeddingRaggedTensorBatchCombiners sets the optional combiners attribute to value.
+//
+// value: A list of string scalars, one for each embedding table that specify
+// how to normalize the embedding activations after weighted summation.
+// Supported combiners are 'mean', 'sum', or 'sqrtn'. It is invalid to have
+// the sum of the weights be 0 for 'mean' or the sum of the squared weights be
+// 0 for 'sqrtn'. If combiners isn't passed, the default is to use 'sum' for
+// all tables.
+// If not specified, defaults to <>
+func EnqueueTPUEmbeddingRaggedTensorBatchCombiners(value []string) EnqueueTPUEmbeddingRaggedTensorBatchAttr {
+	return func(m optionalAttr) {
+		m["combiners"] = value
+	}
+}
+
+// EnqueueTPUEmbeddingRaggedTensorBatchMaxSequenceLengths sets the optional max_sequence_lengths attribute to value.
+// If not specified, defaults to <>
+func EnqueueTPUEmbeddingRaggedTensorBatchMaxSequenceLengths(value []int64) EnqueueTPUEmbeddingRaggedTensorBatchAttr {
+	return func(m optionalAttr) {
+		m["max_sequence_lengths"] = value
+	}
+}
+
+// Eases the porting of code that uses tf.nn.embedding_lookup().
+//
+// sample_splits[i], embedding_indices[i] and aggregation_weights[i] correspond
+// to the ith feature. table_ids[i] indicates which embedding table to look up ith
+// feature.
+//
+// The tensors at corresponding positions in two of the input lists,
+// embedding_indices and aggregation_weights, must have the same shape, i.e. rank 1
+// with dim_size() equal to the total number of lookups into the table described by
+// the corresponding feature.
+//
+// Arguments:
+//	sample_splits: A list of rank 1 Tensors specifying the break points for splitting
+// embedding_indices and aggregation_weights into rows.
+// It corresponds to ids.row_splits in embedding_lookup(), when ids is a
+// RaggedTensor.
+//	embedding_indices: A list of rank 1 Tensors, indices into the embedding tables.
+// It corresponds to ids.values in embedding_lookup(), when ids is a RaggedTensor.
+//	aggregation_weights: A list of rank 1 Tensors containing per training example
+// aggregation weights. It corresponds to the values field of a RaggedTensor
+// with the same row_splits as ids in embedding_lookup(), when ids is a
+// RaggedTensor.
+//	mode_override: A string input that overrides the mode specified in the
+// TPUEmbeddingConfiguration. Supported values are {'unspecified', 'inference',
+// 'training', 'backward_pass_only'}. When set to 'unspecified', the mode set
+// in TPUEmbeddingConfiguration is used, otherwise mode_override is used.
+//	table_ids: A list of integers specifying the identifier of the embedding table
+// (offset of TableDescriptor in the TPUEmbeddingConfiguration) to lookup the
+// corresponding input. The ith input is looked up using table_ids[i]. The size
+// of the table_ids list must be equal to that of sample_indices,
+// embedding_indices and aggregation_weights.
+//
+// Returns the created operation.
+func EnqueueTPUEmbeddingRaggedTensorBatch(scope *Scope, sample_splits []tf.Output, embedding_indices []tf.Output, aggregation_weights []tf.Output, mode_override tf.Output, table_ids []int64, optional ...EnqueueTPUEmbeddingRaggedTensorBatchAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"table_ids": table_ids}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "EnqueueTPUEmbeddingRaggedTensorBatch",
+		Input: []tf.Input{
+			tf.OutputList(sample_splits), tf.OutputList(embedding_indices), tf.OutputList(aggregation_weights), mode_override,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// FakeQuantWithMinMaxVarsAttr is an optional argument to FakeQuantWithMinMaxVars.
+type FakeQuantWithMinMaxVarsAttr func(optionalAttr)
+
+// FakeQuantWithMinMaxVarsNumBits sets the optional num_bits attribute to value.
+// If not specified, defaults to 8
+func FakeQuantWithMinMaxVarsNumBits(value int64) FakeQuantWithMinMaxVarsAttr {
+	return func(m optionalAttr) {
+		m["num_bits"] = value
+	}
+}
+
+// FakeQuantWithMinMaxVarsNarrowRange sets the optional narrow_range attribute to value.
+// If not specified, defaults to false
+func FakeQuantWithMinMaxVarsNarrowRange(value bool) FakeQuantWithMinMaxVarsAttr {
+	return func(m optionalAttr) {
+		m["narrow_range"] = value
+	}
+}
+
+// Fake-quantize the 'inputs' tensor of type float via global float scalars
+//
+// Fake-quantize the `inputs` tensor of type float via global float scalars
+// `min` and `max` to `outputs` tensor of same shape as `inputs`.
+//
+// Attributes
+//
+// *   `[min; max]` define the clamping range for the `inputs` data.
+// *   `inputs` values are quantized into the quantization range (
+// `[0; 2^num_bits - 1]` when `narrow_range` is false and `[1; 2^num_bits - 1]`
+// when it is true) and then de-quantized and output as floats in `[min; max]`
+// interval.
+// *   `num_bits` is the bitwidth of the quantization; between 2 and 16, inclusive.
+//
+// Before quantization, `min` and `max` values are adjusted with the following
+// logic.
+// It is suggested to have `min <= 0 <= max`. If `0` is not in the range of values,
+// the behavior can be unexpected:
+//
+// *   If `0 < min < max`: `min_adj = 0` and `max_adj = max - min`.
+// *   If `min < max < 0`: `min_adj = min - max` and `max_adj = 0`.
+// *   If `min <= 0 <= max`: `scale = (max - min) / (2^num_bits - 1) `,
+// `min_adj = scale * round(min / scale)` and `max_adj = max + min_adj - min`.
+//
+// This operation has a gradient and thus allows for training `min` and `max`
+// values.
+func FakeQuantWithMinMaxVars(scope *Scope, inputs tf.Output, min tf.Output, max tf.Output, optional ...FakeQuantWithMinMaxVarsAttr) (outputs tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "FakeQuantWithMinMaxVars",
+		Input: []tf.Input{
+			inputs, min, max,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Enqueue multiple Tensor values on the computation outfeed.
+//
+// Arguments:
+//	inputs: A list of tensors that will be inserted into the outfeed queue as an
+// XLA tuple.
+//
+// Returns the created operation.
+func OutfeedEnqueueTuple(scope *Scope, inputs []tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "OutfeedEnqueueTuple",
+		Input: []tf.Input{
+			tf.OutputList(inputs),
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Returns the number of nonzeroes of `sparse_matrix`.
+//
+// Arguments:
+//	sparse_matrix: A CSRSparseMatrix.
+//
+// Returns The number of nonzeroes of `sparse_matrix`.
+func SparseMatrixNNZ(scope *Scope, sparse_matrix tf.Output) (nnz tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseMatrixNNZ",
+		Input: []tf.Input{
+			sparse_matrix,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// DecodeJpegAttr is an optional argument to DecodeJpeg.
+type DecodeJpegAttr func(optionalAttr)
+
+// DecodeJpegChannels sets the optional channels attribute to value.
+//
+// value: Number of color channels for the decoded image.
+// If not specified, defaults to 0
+func DecodeJpegChannels(value int64) DecodeJpegAttr {
+	return func(m optionalAttr) {
+		m["channels"] = value
+	}
+}
+
+// DecodeJpegRatio sets the optional ratio attribute to value.
+//
+// value: Downscaling ratio.
+// If not specified, defaults to 1
+func DecodeJpegRatio(value int64) DecodeJpegAttr {
+	return func(m optionalAttr) {
+		m["ratio"] = value
+	}
+}
+
+// DecodeJpegFancyUpscaling sets the optional fancy_upscaling attribute to value.
+//
+// value: If true use a slower but nicer upscaling of the
+// chroma planes (yuv420/422 only).
+// If not specified, defaults to true
+func DecodeJpegFancyUpscaling(value bool) DecodeJpegAttr {
+	return func(m optionalAttr) {
+		m["fancy_upscaling"] = value
+	}
+}
+
+// DecodeJpegTryRecoverTruncated sets the optional try_recover_truncated attribute to value.
+//
+// value: If true try to recover an image from truncated input.
+// If not specified, defaults to false
+func DecodeJpegTryRecoverTruncated(value bool) DecodeJpegAttr {
+	return func(m optionalAttr) {
+		m["try_recover_truncated"] = value
+	}
+}
+
+// DecodeJpegAcceptableFraction sets the optional acceptable_fraction attribute to value.
+//
+// value: The minimum required fraction of lines before a truncated
+// input is accepted.
+// If not specified, defaults to 1
+func DecodeJpegAcceptableFraction(value float32) DecodeJpegAttr {
+	return func(m optionalAttr) {
+		m["acceptable_fraction"] = value
+	}
+}
+
+// DecodeJpegDctMethod sets the optional dct_method attribute to value.
+//
+// value: string specifying a hint about the algorithm used for
+// decompression.  Defaults to "" which maps to a system-specific
+// default.  Currently valid values are ["INTEGER_FAST",
+// "INTEGER_ACCURATE"].  The hint may be ignored (e.g., the internal
+// jpeg library changes to a version that does not have that specific
+// option.)
+// If not specified, defaults to ""
+func DecodeJpegDctMethod(value string) DecodeJpegAttr {
+	return func(m optionalAttr) {
+		m["dct_method"] = value
+	}
+}
+
+// Decode a JPEG-encoded image to a uint8 tensor.
+//
+// The attr `channels` indicates the desired number of color channels for the
+// decoded image.
+//
+// Accepted values are:
+//
+// *   0: Use the number of channels in the JPEG-encoded image.
+// *   1: output a grayscale image.
+// *   3: output an RGB image.
+//
+// If needed, the JPEG-encoded image is transformed to match the requested number
+// of color channels.
+//
+// The attr `ratio` allows downscaling the image by an integer factor during
+// decoding.  Allowed values are: 1, 2, 4, and 8.  This is much faster than
+// downscaling the image later.
+//
+//
+// This op also supports decoding PNGs and non-animated GIFs since the interface is
+// the same, though it is cleaner to use `tf.io.decode_image`.
+//
+// Arguments:
+//	contents: 0-D.  The JPEG-encoded image.
+//
+// Returns 3-D with shape `[height, width, channels]`..
+func DecodeJpeg(scope *Scope, contents tf.Output, optional ...DecodeJpegAttr) (image tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DecodeJpeg",
+		Input: []tf.Input{
+			contents,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// LoadTPUEmbeddingADAMParametersAttr is an optional argument to LoadTPUEmbeddingADAMParameters.
+type LoadTPUEmbeddingADAMParametersAttr func(optionalAttr)
+
+// LoadTPUEmbeddingADAMParametersTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func LoadTPUEmbeddingADAMParametersTableId(value int64) LoadTPUEmbeddingADAMParametersAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// LoadTPUEmbeddingADAMParametersTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingADAMParametersTableName(value string) LoadTPUEmbeddingADAMParametersAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// LoadTPUEmbeddingADAMParametersConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingADAMParametersConfig(value string) LoadTPUEmbeddingADAMParametersAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Load ADAM embedding parameters.
+//
+// An op that loads optimization parameters into HBM for embedding. Must be
+// preceded by a ConfigureTPUEmbeddingHost op that sets up the correct
+// embedding table configuration. For example, this op is used to install
+// parameters that are loaded from a checkpoint before a training loop is
+// executed.
+//
+// Arguments:
+//	parameters: Value of parameters used in the ADAM optimization algorithm.
+//	momenta: Value of momenta used in the ADAM optimization algorithm.
+//	velocities: Value of velocities used in the ADAM optimization algorithm.
+//
+//
+//
+// Returns the created operation.
+func LoadTPUEmbeddingADAMParameters(scope *Scope, parameters tf.Output, momenta tf.Output, velocities tf.Output, num_shards int64, shard_id int64, optional ...LoadTPUEmbeddingADAMParametersAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LoadTPUEmbeddingADAMParameters",
+		Input: []tf.Input{
+			parameters, momenta, velocities,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Records the latency of producing `input_dataset` elements in a StatsAggregator.
+func LatencyStatsDataset(scope *Scope, input_dataset tf.Output, tag tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "LatencyStatsDataset",
+		Input: []tf.Input{
+			input_dataset, tag,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes the power of one value to another.
+//
+// Given a tensor `x` and a tensor `y`, this operation computes \\(x^y\\) for
+// corresponding elements in `x` and `y`. For example:
+//
+// ```
+// # tensor 'x' is [[2, 2]], [3, 3]]
+// # tensor 'y' is [[8, 16], [2, 3]]
+// tf.pow(x, y) ==> [[256, 65536], [9, 27]]
+// ```
+func Pow(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Pow",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Element-wise multiplication of a sparse matrix with a dense tensor.
+//
+// Returns a sparse matrix.
+//
+// The dense tensor `b` may be either a scalar; otherwise `a` must be a rank-3
+// `SparseMatrix`; in this case `b` must be shaped `[batch_size, 1, 1]` and the
+// multiply operation broadcasts.
+//
+// **NOTE** even if `b` is zero, the sparsity structure of the output does not
+// change.
+//
+// Arguments:
+//	a: A CSRSparseMatrix.
+//	b: A dense tensor.
+//
+// Returns A dense output tensor.
+func SparseMatrixMul(scope *Scope, a tf.Output, b tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseMatrixMul",
+		Input: []tf.Input{
+			a, b,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns the element-wise sum of a list of tensors.
+//
+// `tf.accumulate_n_v2` performs the same operation as `tf.add_n`, but does not
+// wait for all of its inputs to be ready before beginning to sum. This can
+// save memory if inputs are ready at different times, since minimum temporary
+// storage is proportional to the output size rather than the inputs size.
+//
+// Unlike the original `accumulate_n`, `accumulate_n_v2` is differentiable.
+//
+// Returns a `Tensor` of same shape and type as the elements of `inputs`.
+//
+// Arguments:
+//	inputs: A list of `Tensor` objects, each with same shape and type.
+//	shape: Shape of elements of `inputs`.
+func AccumulateNV2(scope *Scope, inputs []tf.Output, shape tf.Shape) (sum tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"shape": shape}
+	opspec := tf.OpSpec{
+		Type: "AccumulateNV2",
+		Input: []tf.Input{
+			tf.OutputList(inputs),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// An op enabling differentiation of TPU Embeddings.
+//
+// This op simply returns its first input, which is assumed to have been sliced
+// from the Tensors returned by TPUEmbeddingDequeueActivations. The presence of
+// this op, and its first argument being a trainable Variable, enables automatic
+// differentiation of graphs containing embeddings via the TPU Embedding Python
+// libraries.
+//
+// Arguments:
+//	embedding_variable: A trainable variable, enabling optimizers to find this op.
+//	sliced_activations: The embedding activations Tensor to return.
+//	table_id: The id of the table in the embedding layer configuration from which
+// these activations were computed.
+//	lookup_id: Identifier of the set of embedding indices which produced these
+// activations.
+func TPUEmbeddingActivations(scope *Scope, embedding_variable tf.Output, sliced_activations tf.Output, table_id int64, lookup_id int64) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"table_id": table_id, "lookup_id": lookup_id}
+	opspec := tf.OpSpec{
+		Type: "TPUEmbeddingActivations",
+		Input: []tf.Input{
+			embedding_variable, sliced_activations,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// QuantizeAndDequantizeV3Attr is an optional argument to QuantizeAndDequantizeV3.
+type QuantizeAndDequantizeV3Attr func(optionalAttr)
+
+// QuantizeAndDequantizeV3SignedInput sets the optional signed_input attribute to value.
+// If not specified, defaults to true
+func QuantizeAndDequantizeV3SignedInput(value bool) QuantizeAndDequantizeV3Attr {
+	return func(m optionalAttr) {
+		m["signed_input"] = value
+	}
+}
+
+// QuantizeAndDequantizeV3RangeGiven sets the optional range_given attribute to value.
+// If not specified, defaults to true
+func QuantizeAndDequantizeV3RangeGiven(value bool) QuantizeAndDequantizeV3Attr {
+	return func(m optionalAttr) {
+		m["range_given"] = value
+	}
+}
+
+// QuantizeAndDequantizeV3NarrowRange sets the optional narrow_range attribute to value.
+// If not specified, defaults to false
+func QuantizeAndDequantizeV3NarrowRange(value bool) QuantizeAndDequantizeV3Attr {
+	return func(m optionalAttr) {
+		m["narrow_range"] = value
+	}
+}
+
+// QuantizeAndDequantizeV3Axis sets the optional axis attribute to value.
+// If not specified, defaults to -1
+func QuantizeAndDequantizeV3Axis(value int64) QuantizeAndDequantizeV3Attr {
+	return func(m optionalAttr) {
+		m["axis"] = value
+	}
+}
+
+// Quantizes then dequantizes a tensor.
+//
+// This is almost identical to QuantizeAndDequantizeV2, except that num_bits is a
+// tensor, so its value can change during training.
+func QuantizeAndDequantizeV3(scope *Scope, input tf.Output, input_min tf.Output, input_max tf.Output, num_bits tf.Output, optional ...QuantizeAndDequantizeV3Attr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "QuantizeAndDequantizeV3",
+		Input: []tf.Input{
+			input, input_min, input_max, num_bits,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns x * y element-wise.
+//
+// *NOTE*: `Multiply` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func Mul(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Mul",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Computes softplus gradients for a softplus operation.
+//
+// Arguments:
+//	gradients: The backpropagated gradients to the corresponding softplus operation.
+//	features: The features passed as input to the corresponding softplus operation.
+//
+// Returns The gradients: `gradients / (1 + exp(-features))`.
+func SoftplusGrad(scope *Scope, gradients tf.Output, features tf.Output) (backprops tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SoftplusGrad",
+		Input: []tf.Input{
+			gradients, features,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns the item in the list with the given index.
+//
+// input_handle: the list
+// index: the position in the list from which an element will be retrieved
+// item: the element at that position
+//
+//
+func TensorListGetItem(scope *Scope, input_handle tf.Output, index tf.Output, element_shape tf.Output, element_dtype tf.DataType) (item tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"element_dtype": element_dtype}
+	opspec := tf.OpSpec{
+		Type: "TensorListGetItem",
+		Input: []tf.Input{
+			input_handle, index, element_shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RetrieveTPUEmbeddingProximalAdagradParametersGradAccumDebugAttr is an optional argument to RetrieveTPUEmbeddingProximalAdagradParametersGradAccumDebug.
+type RetrieveTPUEmbeddingProximalAdagradParametersGradAccumDebugAttr func(optionalAttr)
+
+// RetrieveTPUEmbeddingProximalAdagradParametersGradAccumDebugTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func RetrieveTPUEmbeddingProximalAdagradParametersGradAccumDebugTableId(value int64) RetrieveTPUEmbeddingProximalAdagradParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingProximalAdagradParametersGradAccumDebugTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingProximalAdagradParametersGradAccumDebugTableName(value string) RetrieveTPUEmbeddingProximalAdagradParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingProximalAdagradParametersGradAccumDebugConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingProximalAdagradParametersGradAccumDebugConfig(value string) RetrieveTPUEmbeddingProximalAdagradParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Retrieve proximal Adagrad embedding parameters with debug support.
+//
+// An op that retrieves optimization parameters from embedding to host
+// memory. Must be preceded by a ConfigureTPUEmbeddingHost op that sets up
+// the correct embedding table configuration. For example, this op is
+// used to retrieve updated parameters before saving a checkpoint.
+//
+// Returns:
+//	parameters: Parameter parameters updated by the proximal Adagrad optimization algorithm.
+//	accumulators: Parameter accumulators updated by the proximal Adagrad optimization algorithm.
+//	gradient_accumulators: Parameter gradient_accumulators updated by the proximal Adagrad optimization algorithm.
+func RetrieveTPUEmbeddingProximalAdagradParametersGradAccumDebug(scope *Scope, num_shards int64, shard_id int64, optional ...RetrieveTPUEmbeddingProximalAdagradParametersGradAccumDebugAttr) (parameters tf.Output, accumulators tf.Output, gradient_accumulators tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RetrieveTPUEmbeddingProximalAdagradParametersGradAccumDebug",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Returns the value stored in an Optional variant or raises an error if none exists.
+func OptionalGetValue(scope *Scope, optional tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (components []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "OptionalGetValue",
+		Input: []tf.Input{
+			optional,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if components, idx, err = makeOutputList(op, idx, "components"); err != nil {
+		scope.UpdateErr("OptionalGetValue", err)
+		return
+	}
+	return components
+}
+
+// Determine the script codes of a given tensor of Unicode integer code points.
+//
+// This operation converts Unicode code points to script codes corresponding to
+// each code point. Script codes correspond to International Components for
+// Unicode (ICU) UScriptCode values. See http://icu-project.org/apiref/icu4c/uscript_8h.html.
+// Returns -1 (USCRIPT_INVALID_CODE) for invalid codepoints. Output shape will
+// match input shape.
+//
+// Examples:
+//
+// >>> tf.strings.unicode_script([1, 31, 38])
+// <tf.Tensor: shape=(3,), dtype=int32, numpy=array([0, 0, 0], dtype=int32)>
+//
+// Arguments:
+//	input: A Tensor of int32 Unicode code points.
+//
+// Returns A Tensor of int32 script codes corresponding to each input code point.
+func UnicodeScript(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "UnicodeScript",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// CropAndResizeAttr is an optional argument to CropAndResize.
+type CropAndResizeAttr func(optionalAttr)
+
+// CropAndResizeMethod sets the optional method attribute to value.
+//
+// value: A string specifying the sampling method for resizing. It can be either
+// `"bilinear"` or `"nearest"` and default to `"bilinear"`. Currently two sampling
+// methods are supported: Bilinear and Nearest Neighbor.
+// If not specified, defaults to "bilinear"
+func CropAndResizeMethod(value string) CropAndResizeAttr {
+	return func(m optionalAttr) {
+		m["method"] = value
+	}
+}
+
+// CropAndResizeExtrapolationValue sets the optional extrapolation_value attribute to value.
+//
+// value: Value used for extrapolation, when applicable.
+// If not specified, defaults to 0
+func CropAndResizeExtrapolationValue(value float32) CropAndResizeAttr {
+	return func(m optionalAttr) {
+		m["extrapolation_value"] = value
+	}
+}
+
+// Extracts crops from the input image tensor and resizes them.
+//
+// Extracts crops from the input image tensor and resizes them using bilinear
+// sampling or nearest neighbor sampling (possibly with aspect ratio change) to a
+// common output size specified by `crop_size`. This is more general than the
+// `crop_to_bounding_box` op which extracts a fixed size slice from the input image
+// and does not allow resizing or aspect ratio change.
+//
+// Returns a tensor with `crops` from the input `image` at positions defined at the
+// bounding box locations in `boxes`. The cropped boxes are all resized (with
+// bilinear or nearest neighbor interpolation) to a fixed
+// `size = [crop_height, crop_width]`. The result is a 4-D tensor
+// `[num_boxes, crop_height, crop_width, depth]`. The resizing is corner aligned.
+// In particular, if `boxes = [[0, 0, 1, 1]]`, the method will give identical
+// results to using `tf.image.resize_bilinear()` or
+// `tf.image.resize_nearest_neighbor()`(depends on the `method` argument) with
+// `align_corners=True`.
+//
+// Arguments:
+//	image: A 4-D tensor of shape `[batch, image_height, image_width, depth]`.
+// Both `image_height` and `image_width` need to be positive.
+//	boxes: A 2-D tensor of shape `[num_boxes, 4]`. The `i`-th row of the tensor
+// specifies the coordinates of a box in the `box_ind[i]` image and is specified
+// in normalized coordinates `[y1, x1, y2, x2]`. A normalized coordinate value of
+// `y` is mapped to the image coordinate at `y * (image_height - 1)`, so as the
+// `[0, 1]` interval of normalized image height is mapped to
+// `[0, image_height - 1]` in image height coordinates. We do allow `y1` > `y2`, in
+// which case the sampled crop is an up-down flipped version of the original
+// image. The width dimension is treated similarly. Normalized coordinates
+// outside the `[0, 1]` range are allowed, in which case we use
+// `extrapolation_value` to extrapolate the input image values.
+//	box_ind: A 1-D tensor of shape `[num_boxes]` with int32 values in `[0, batch)`.
+// The value of `box_ind[i]` specifies the image that the `i`-th box refers to.
+//	crop_size: A 1-D tensor of 2 elements, `size = [crop_height, crop_width]`. All
+// cropped image patches are resized to this size. The aspect ratio of the image
+// content is not preserved. Both `crop_height` and `crop_width` need to be
+// positive.
+//
+// Returns A 4-D tensor of shape `[num_boxes, crop_height, crop_width, depth]`.
+func CropAndResize(scope *Scope, image tf.Output, boxes tf.Output, box_ind tf.Output, crop_size tf.Output, optional ...CropAndResizeAttr) (crops tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CropAndResize",
+		Input: []tf.Input{
+			image, boxes, box_ind, crop_size,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// DepthwiseConv2dNativeBackpropFilterAttr is an optional argument to DepthwiseConv2dNativeBackpropFilter.
+type DepthwiseConv2dNativeBackpropFilterAttr func(optionalAttr)
+
+// DepthwiseConv2dNativeBackpropFilterExplicitPaddings sets the optional explicit_paddings attribute to value.
+// If not specified, defaults to <>
+func DepthwiseConv2dNativeBackpropFilterExplicitPaddings(value []int64) DepthwiseConv2dNativeBackpropFilterAttr {
+	return func(m optionalAttr) {
+		m["explicit_paddings"] = value
+	}
+}
+
+// DepthwiseConv2dNativeBackpropFilterDataFormat sets the optional data_format attribute to value.
+//
+// value: Specify the data format of the input and output data. With the
+// default format "NHWC", the data is stored in the order of:
+//     [batch, height, width, channels].
+// Alternatively, the format could be "NCHW", the data storage order of:
+//     [batch, channels, height, width].
+// If not specified, defaults to "NHWC"
+func DepthwiseConv2dNativeBackpropFilterDataFormat(value string) DepthwiseConv2dNativeBackpropFilterAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// DepthwiseConv2dNativeBackpropFilterDilations sets the optional dilations attribute to value.
+//
+// value: 1-D tensor of length 4.  The dilation factor for each dimension of
+// `input`. If set to k > 1, there will be k-1 skipped cells between each filter
+// element on that dimension. The dimension order is determined by the value of
+// `data_format`, see above for details. Dilations in the batch and depth
+// dimensions must be 1.
+// If not specified, defaults to <i:1 i:1 i:1 i:1 >
+func DepthwiseConv2dNativeBackpropFilterDilations(value []int64) DepthwiseConv2dNativeBackpropFilterAttr {
+	return func(m optionalAttr) {
+		m["dilations"] = value
+	}
+}
+
+// Computes the gradients of depthwise convolution with respect to the filter.
+//
+// Arguments:
+//	input: 4-D with shape based on `data_format`.  For example, if
+// `data_format` is 'NHWC' then `input` is a 4-D `[batch, in_height,
+// in_width, in_channels]` tensor.
+//	filter_sizes: An integer vector representing the tensor shape of `filter`,
+// where `filter` is a 4-D
+// `[filter_height, filter_width, in_channels, depthwise_multiplier]` tensor.
+//	out_backprop: 4-D with shape  based on `data_format`.
+// For example, if `data_format` is 'NHWC' then
+// out_backprop shape is `[batch, out_height, out_width, out_channels]`.
+// Gradients w.r.t. the output of the convolution.
+//	strides: The stride of the sliding window for each dimension of the input
+// of the convolution.
+//	padding: The type of padding algorithm to use.
+//
+// Returns 4-D with shape
+// `[filter_height, filter_width, in_channels, out_channels]`.  Gradient w.r.t.
+// the `filter` input of the convolution.
+func DepthwiseConv2dNativeBackpropFilter(scope *Scope, input tf.Output, filter_sizes tf.Output, out_backprop tf.Output, strides []int64, padding string, optional ...DepthwiseConv2dNativeBackpropFilterAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DepthwiseConv2dNativeBackpropFilter",
+		Input: []tf.Input{
+			input, filter_sizes, out_backprop,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that zips together `input_datasets`.
+//
+// The elements of the resulting dataset are created by zipping corresponding
+// elements from each of the input datasets.
+//
+// The size of the resulting dataset will match the size of the smallest input
+// dataset, and no error will be raised if input datasets have different sizes.
+//
+// Arguments:
+//	input_datasets: List of `N` variant Tensors representing datasets to be zipped together.
+//
+//
+func ZipDataset(scope *Scope, input_datasets []tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "ZipDataset",
+		Input: []tf.Input{
+			tf.OutputList(input_datasets),
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Rounds the values of a tensor to the nearest integer, element-wise.
+//
+// Rounds half to even.  Also known as bankers rounding. If you want to round
+// according to the current system rounding mode use std::cint.
+func Round(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Round",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a tree ensemble model and returns a handle to it.
+//
+// Arguments:
+//	tree_ensemble_handle: Handle to the tree ensemble resource to be created.
+//	stamp_token: Token to use as the initial value of the resource stamp.
+//	tree_ensemble_serialized: Serialized proto of the tree ensemble.
+//
+// Returns the created operation.
+func BoostedTreesCreateEnsemble(scope *Scope, tree_ensemble_handle tf.Output, stamp_token tf.Output, tree_ensemble_serialized tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesCreateEnsemble",
+		Input: []tf.Input{
+			tree_ensemble_handle, stamp_token, tree_ensemble_serialized,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Calculates the softmax of a CSRSparseMatrix.
+//
+// Calculate the softmax of the innermost dimensions of a SparseMatrix.
+//
+// Missing values are treated as `-inf` (i.e., logits of zero probability); and
+// the output has the same sparsity structure as the input (though missing values
+// in the output may now be treated as having probability zero).
+//
+// Arguments:
+//	logits: A CSRSparseMatrix.
+//
+//
+// Returns A CSRSparseMatrix.
+func SparseMatrixSoftmax(scope *Scope, logits tf.Output, type_ tf.DataType) (softmax tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"type": type_}
+	opspec := tf.OpSpec{
+		Type: "SparseMatrixSoftmax",
+		Input: []tf.Input{
+			logits,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RestoreAttr is an optional argument to Restore.
+type RestoreAttr func(optionalAttr)
+
+// RestorePreferredShard sets the optional preferred_shard attribute to value.
+//
+// value: Index of file to open first if multiple files match
+// `file_pattern`.
+// If not specified, defaults to -1
+func RestorePreferredShard(value int64) RestoreAttr {
+	return func(m optionalAttr) {
+		m["preferred_shard"] = value
+	}
+}
+
+// Restores a tensor from checkpoint files.
+//
+// Reads a tensor stored in one or several files. If there are several files (for
+// instance because a tensor was saved as slices), `file_pattern` may contain
+// wildcard symbols (`*` and `?`) in the filename portion only, not in the
+// directory portion.
+//
+// If a `file_pattern` matches several files, `preferred_shard` can be used to hint
+// in which file the requested tensor is likely to be found. This op will first
+// open the file at index `preferred_shard` in the list of matching files and try
+// to restore tensors from that file.  Only if some tensors or tensor slices are
+// not found in that first file, then the Op opens all the files. Setting
+// `preferred_shard` to match the value passed as the `shard` input
+// of a matching `Save` Op may speed up Restore.  This attribute only affects
+// performance, not correctness.  The default value -1 means files are processed in
+// order.
+//
+// See also `RestoreSlice`.
+//
+// Arguments:
+//	file_pattern: Must have a single element. The pattern of the files from
+// which we read the tensor.
+//	tensor_name: Must have a single element. The name of the tensor to be
+// restored.
+//	dt: The type of the tensor to be restored.
+//
+// Returns The restored tensor.
+func Restore(scope *Scope, file_pattern tf.Output, tensor_name tf.Output, dt tf.DataType, optional ...RestoreAttr) (tensor tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dt": dt}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Restore",
+		Input: []tf.Input{
+			file_pattern, tensor_name,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns the next record (key, value pair) produced by a Reader.
+//
+// Will dequeue from the input queue if necessary (e.g. when the
+// Reader needs to start reading from a new file since it has finished
+// with the previous file).
+//
+// Arguments:
+//	reader_handle: Handle to a Reader.
+//	queue_handle: Handle to a Queue, with string work items.
+//
+// Returns:
+//	key: A scalar.
+//	value: A scalar.
+func ReaderReadV2(scope *Scope, reader_handle tf.Output, queue_handle tf.Output) (key tf.Output, value tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ReaderReadV2",
+		Input: []tf.Input{
+			reader_handle, queue_handle,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// CumprodAttr is an optional argument to Cumprod.
+type CumprodAttr func(optionalAttr)
+
+// CumprodExclusive sets the optional exclusive attribute to value.
+//
+// value: If `True`, perform exclusive cumprod.
+// If not specified, defaults to false
+func CumprodExclusive(value bool) CumprodAttr {
+	return func(m optionalAttr) {
+		m["exclusive"] = value
+	}
+}
+
+// CumprodReverse sets the optional reverse attribute to value.
+//
+// value: A `bool` (default: False).
+// If not specified, defaults to false
+func CumprodReverse(value bool) CumprodAttr {
+	return func(m optionalAttr) {
+		m["reverse"] = value
+	}
+}
+
+// Compute the cumulative product of the tensor `x` along `axis`.
+//
+// By default, this op performs an inclusive cumprod, which means that the first
+// element of the input is identical to the first element of the output:
+//
+// ```python
+// tf.cumprod([a, b, c])  # => [a, a * b, a * b * c]
+// ```
+//
+// By setting the `exclusive` kwarg to `True`, an exclusive cumprod is
+// performed instead:
+//
+// ```python
+// tf.cumprod([a, b, c], exclusive=True)  # => [1, a, a * b]
+// ```
+//
+// By setting the `reverse` kwarg to `True`, the cumprod is performed in the
+// opposite direction:
+//
+// ```python
+// tf.cumprod([a, b, c], reverse=True)  # => [a * b * c, b * c, c]
+// ```
+//
+// This is more efficient than using separate `tf.reverse` ops.
+//
+// The `reverse` and `exclusive` kwargs can also be combined:
+//
+// ```python
+// tf.cumprod([a, b, c], exclusive=True, reverse=True)  # => [b * c, c, 1]
+// ```
+//
+// Arguments:
+//	x: A `Tensor`. Must be one of the following types: `float32`, `float64`,
+// `int64`, `int32`, `uint8`, `uint16`, `int16`, `int8`, `complex64`,
+// `complex128`, `qint8`, `quint8`, `qint32`, `half`.
+//	axis: A `Tensor` of type `int32` (default: 0). Must be in the range
+// `[-rank(x), rank(x))`.
+func Cumprod(scope *Scope, x tf.Output, axis tf.Output, optional ...CumprodAttr) (out tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Cumprod",
+		Input: []tf.Input{
+			x, axis,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// LoadTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugAttr is an optional argument to LoadTPUEmbeddingStochasticGradientDescentParametersGradAccumDebug.
+type LoadTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugAttr func(optionalAttr)
+
+// LoadTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func LoadTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugTableId(value int64) LoadTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// LoadTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugTableName(value string) LoadTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// LoadTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugConfig(value string) LoadTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Load SGD embedding parameters.
+//
+// An op that loads optimization parameters into HBM for embedding. Must be
+// preceded by a ConfigureTPUEmbeddingHost op that sets up the correct
+// embedding table configuration. For example, this op is used to install
+// parameters that are loaded from a checkpoint before a training loop is
+// executed.
+//
+// Arguments:
+//	parameters: Value of parameters used in the stochastic gradient descent optimization algorithm.
+//	gradient_accumulators: Value of gradient_accumulators used in the Adadelta optimization algorithm.
+//
+//
+//
+// Returns the created operation.
+func LoadTPUEmbeddingStochasticGradientDescentParametersGradAccumDebug(scope *Scope, parameters tf.Output, gradient_accumulators tf.Output, num_shards int64, shard_id int64, optional ...LoadTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LoadTPUEmbeddingStochasticGradientDescentParametersGradAccumDebug",
+		Input: []tf.Input{
+			parameters, gradient_accumulators,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Creates a dataset that batches input elements into a SparseTensor.
+//
+// Arguments:
+//	input_dataset: A handle to an input dataset. Must have a single component.
+//	batch_size: A scalar representing the number of elements to accumulate in a
+// batch.
+//	row_shape: A vector representing the dense shape of each row in the produced
+// SparseTensor. The shape may be partially specified, using `-1` to indicate
+// that a particular dimension should use the maximum size of all batch elements.
+//
+//
+func DenseToSparseBatchDataset(scope *Scope, input_dataset tf.Output, batch_size tf.Output, row_shape tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "DenseToSparseBatchDataset",
+		Input: []tf.Input{
+			input_dataset, batch_size, row_shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// LoadTPUEmbeddingAdadeltaParametersGradAccumDebugAttr is an optional argument to LoadTPUEmbeddingAdadeltaParametersGradAccumDebug.
+type LoadTPUEmbeddingAdadeltaParametersGradAccumDebugAttr func(optionalAttr)
+
+// LoadTPUEmbeddingAdadeltaParametersGradAccumDebugTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func LoadTPUEmbeddingAdadeltaParametersGradAccumDebugTableId(value int64) LoadTPUEmbeddingAdadeltaParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// LoadTPUEmbeddingAdadeltaParametersGradAccumDebugTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingAdadeltaParametersGradAccumDebugTableName(value string) LoadTPUEmbeddingAdadeltaParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// LoadTPUEmbeddingAdadeltaParametersGradAccumDebugConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingAdadeltaParametersGradAccumDebugConfig(value string) LoadTPUEmbeddingAdadeltaParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Load Adadelta parameters with debug support.
+//
+// An op that loads optimization parameters into HBM for embedding. Must be
+// preceded by a ConfigureTPUEmbeddingHost op that sets up the correct
+// embedding table configuration. For example, this op is used to install
+// parameters that are loaded from a checkpoint before a training loop is
+// executed.
+//
+// Arguments:
+//	parameters: Value of parameters used in the Adadelta optimization algorithm.
+//	accumulators: Value of accumulators used in the Adadelta optimization algorithm.
+//	updates: Value of updates used in the Adadelta optimization algorithm.
+//	gradient_accumulators: Value of gradient_accumulators used in the Adadelta optimization algorithm.
+//
+//
+//
+// Returns the created operation.
+func LoadTPUEmbeddingAdadeltaParametersGradAccumDebug(scope *Scope, parameters tf.Output, accumulators tf.Output, updates tf.Output, gradient_accumulators tf.Output, num_shards int64, shard_id int64, optional ...LoadTPUEmbeddingAdadeltaParametersGradAccumDebugAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LoadTPUEmbeddingAdadeltaParametersGradAccumDebug",
+		Input: []tf.Input{
+			parameters, accumulators, updates, gradient_accumulators,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Returns x / y element-wise.
+//
+// *NOTE*: `Div` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+func Div(scope *Scope, x tf.Output, y tf.Output) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Div",
+		Input: []tf.Input{
+			x, y,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Enqueue a Tensor on the computation outfeed.
+//
+// Arguments:
+//	input: A tensor that will be inserted into the outfeed queue.
+//
+// Returns the created operation.
+func OutfeedEnqueue(scope *Scope, input tf.Output) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "OutfeedEnqueue",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	return scope.AddOperation(opspec)
+}
+
+// LoadTPUEmbeddingProximalAdagradParametersGradAccumDebugAttr is an optional argument to LoadTPUEmbeddingProximalAdagradParametersGradAccumDebug.
+type LoadTPUEmbeddingProximalAdagradParametersGradAccumDebugAttr func(optionalAttr)
+
+// LoadTPUEmbeddingProximalAdagradParametersGradAccumDebugTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func LoadTPUEmbeddingProximalAdagradParametersGradAccumDebugTableId(value int64) LoadTPUEmbeddingProximalAdagradParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// LoadTPUEmbeddingProximalAdagradParametersGradAccumDebugTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingProximalAdagradParametersGradAccumDebugTableName(value string) LoadTPUEmbeddingProximalAdagradParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// LoadTPUEmbeddingProximalAdagradParametersGradAccumDebugConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingProximalAdagradParametersGradAccumDebugConfig(value string) LoadTPUEmbeddingProximalAdagradParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Load proximal Adagrad embedding parameters with debug support.
+//
+// An op that loads optimization parameters into HBM for embedding. Must be
+// preceded by a ConfigureTPUEmbeddingHost op that sets up the correct
+// embedding table configuration. For example, this op is used to install
+// parameters that are loaded from a checkpoint before a training loop is
+// executed.
+//
+// Arguments:
+//	parameters: Value of parameters used in the proximal Adagrad optimization algorithm.
+//	accumulators: Value of accumulators used in the proximal Adagrad optimization algorithm.
+//	gradient_accumulators: Value of gradient_accumulators used in the proximal Adagrad optimization algorithm.
+//
+//
+//
+// Returns the created operation.
+func LoadTPUEmbeddingProximalAdagradParametersGradAccumDebug(scope *Scope, parameters tf.Output, accumulators tf.Output, gradient_accumulators tf.Output, num_shards int64, shard_id int64, optional ...LoadTPUEmbeddingProximalAdagradParametersGradAccumDebugAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LoadTPUEmbeddingProximalAdagradParametersGradAccumDebug",
+		Input: []tf.Input{
+			parameters, accumulators, gradient_accumulators,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Serializes the tree ensemble to a proto.
+//
+// Arguments:
+//	tree_ensemble_handle: Handle to the tree ensemble.
+//
+// Returns:
+//	stamp_token: Stamp token of the tree ensemble resource.
+//	tree_ensemble_serialized: Serialized proto of the ensemble.
+func BoostedTreesSerializeEnsemble(scope *Scope, tree_ensemble_handle tf.Output) (stamp_token tf.Output, tree_ensemble_serialized tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "BoostedTreesSerializeEnsemble",
+		Input: []tf.Input{
+			tree_ensemble_handle,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// Computes inverse hyperbolic cosine of x element-wise.
+//
+// Given an input tensor, the function computes inverse hyperbolic cosine of every element.
+// Input range is `[1, inf]`. It returns `nan` if the input lies outside the range.
+//
+// ```python
+// x = tf.constant([-2, -0.5, 1, 1.2, 200, 10000, float("inf")])
+// tf.math.acosh(x) ==> [nan nan 0. 0.62236255 5.9914584 9.903487 inf]
+// ```
+func Acosh(scope *Scope, x tf.Output) (y tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Acosh",
+		Input: []tf.Input{
+			x,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Outputs deterministic pseudorandom random numbers from a gamma distribution.
+//
+// Outputs random values from a gamma distribution.
+//
+// The outputs are a deterministic function of `shape`, `seed`, and `alpha`.
+//
+// Arguments:
+//	shape: The shape of the output tensor.
+//	seed: 2 seeds (shape [2]).
+//	alpha: The concentration of the gamma distribution. Shape must match the rightmost
+// dimensions of `shape`.
+//
+// Returns Random values with specified shape.
+func StatelessRandomGammaV2(scope *Scope, shape tf.Output, seed tf.Output, alpha tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "StatelessRandomGammaV2",
+		Input: []tf.Input{
+			shape, seed, alpha,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates a dataset that executes a SQL query and emits rows of the result set.
+//
+// Arguments:
+//	driver_name: The database type. Currently, the only supported type is 'sqlite'.
+//	data_source_name: A connection string to connect to the database.
+//	query: A SQL query to execute.
+//
+//
+func SqlDataset(scope *Scope, driver_name tf.Output, data_source_name tf.Output, query tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "SqlDataset",
+		Input: []tf.Input{
+			driver_name, data_source_name, query,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Outputs deterministic pseudorandom random integers from a uniform distribution.
+//
+// The generated values follow a uniform distribution in the range `[minval, maxval)`.
+//
+// The outputs are a deterministic function of `shape`, `seed`, `minval`, and `maxval`.
+//
+// Arguments:
+//	shape: The shape of the output tensor.
+//	seed: 2 seeds (shape [2]).
+//	minval: Minimum value (inclusive, scalar).
+//	maxval: Maximum value (exclusive, scalar).
+//
+// Returns Random values with specified shape.
+func StatelessRandomUniformInt(scope *Scope, shape tf.Output, seed tf.Output, minval tf.Output, maxval tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "StatelessRandomUniformInt",
+		Input: []tf.Input{
+			shape, seed, minval, maxval,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Returns a batched diagonal tensor with a given batched diagonal values.
+//
+// Given a `diagonal`, this operation returns a tensor with the `diagonal` and
+// everything else padded with zeros. The diagonal is computed as follows:
+//
+// Assume `diagonal` has `k` dimensions `[I, J, K, ..., N]`, then the output is a
+// tensor of rank `k+1` with dimensions [I, J, K, ..., N, N]` where:
+//
+// `output[i, j, k, ..., m, n] = 1{m=n} * diagonal[i, j, k, ..., n]`.
+//
+// For example:
+//
+// ```
+// # 'diagonal' is [[1, 2, 3, 4], [5, 6, 7, 8]]
+//
+// and diagonal.shape = (2, 4)
+//
+// tf.matrix_diag(diagonal) ==> [[[1, 0, 0, 0]
+//                                      [0, 2, 0, 0]
+//                                      [0, 0, 3, 0]
+//                                      [0, 0, 0, 4]],
+//                                     [[5, 0, 0, 0]
+//                                      [0, 6, 0, 0]
+//                                      [0, 0, 7, 0]
+//                                      [0, 0, 0, 8]]]
+//
+// which has shape (2, 4, 4)
+// ```
+//
+// Arguments:
+//	diagonal: Rank `k`, where `k >= 1`.
+//
+// Returns Rank `k+1`, with `output.shape = diagonal.shape + [diagonal.shape[-1]]`.
+func MatrixDiag(scope *Scope, diagonal tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "MatrixDiag",
+		Input: []tf.Input{
+			diagonal,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// StatelessTruncatedNormalAttr is an optional argument to StatelessTruncatedNormal.
+type StatelessTruncatedNormalAttr func(optionalAttr)
+
+// StatelessTruncatedNormalDtype sets the optional dtype attribute to value.
+//
+// value: The type of the output.
+// If not specified, defaults to DT_FLOAT
+func StatelessTruncatedNormalDtype(value tf.DataType) StatelessTruncatedNormalAttr {
+	return func(m optionalAttr) {
+		m["dtype"] = value
+	}
+}
+
+// Outputs deterministic pseudorandom values from a truncated normal distribution.
+//
+// The generated values follow a normal distribution with mean 0 and standard
+// deviation 1, except that values whose magnitude is more than 2 standard
+// deviations from the mean are dropped and re-picked.
+//
+// The outputs are a deterministic function of `shape` and `seed`.
+//
+// Arguments:
+//	shape: The shape of the output tensor.
+//	seed: 2 seeds (shape [2]).
+//
+// Returns Random values with specified shape.
+func StatelessTruncatedNormal(scope *Scope, shape tf.Output, seed tf.Output, optional ...StatelessTruncatedNormalAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StatelessTruncatedNormal",
+		Input: []tf.Input{
+			shape, seed,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RetrieveTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugAttr is an optional argument to RetrieveTPUEmbeddingStochasticGradientDescentParametersGradAccumDebug.
+type RetrieveTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugAttr func(optionalAttr)
+
+// RetrieveTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func RetrieveTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugTableId(value int64) RetrieveTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugTableName(value string) RetrieveTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugConfig(value string) RetrieveTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Retrieve SGD embedding parameters with debug support.
+//
+// An op that retrieves optimization parameters from embedding to host
+// memory. Must be preceded by a ConfigureTPUEmbeddingHost op that sets up
+// the correct embedding table configuration. For example, this op is
+// used to retrieve updated parameters before saving a checkpoint.
+//
+// Returns:
+//	parameters: Parameter parameters updated by the stochastic gradient descent optimization algorithm.
+//	gradient_accumulators: Parameter gradient_accumulators updated by the Adadelta optimization algorithm.
+func RetrieveTPUEmbeddingStochasticGradientDescentParametersGradAccumDebug(scope *Scope, num_shards int64, shard_id int64, optional ...RetrieveTPUEmbeddingStochasticGradientDescentParametersGradAccumDebugAttr) (parameters tf.Output, gradient_accumulators tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RetrieveTPUEmbeddingStochasticGradientDescentParametersGradAccumDebug",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// StatelessRandomUniformAttr is an optional argument to StatelessRandomUniform.
+type StatelessRandomUniformAttr func(optionalAttr)
+
+// StatelessRandomUniformDtype sets the optional dtype attribute to value.
+//
+// value: The type of the output.
+// If not specified, defaults to DT_FLOAT
+func StatelessRandomUniformDtype(value tf.DataType) StatelessRandomUniformAttr {
+	return func(m optionalAttr) {
+		m["dtype"] = value
+	}
+}
+
+// Outputs deterministic pseudorandom random values from a uniform distribution.
+//
+// The generated values follow a uniform distribution in the range `[0, 1)`. The
+// lower bound 0 is included in the range, while the upper bound 1 is excluded.
+//
+// The outputs are a deterministic function of `shape` and `seed`.
+//
+// Arguments:
+//	shape: The shape of the output tensor.
+//	seed: 2 seeds (shape [2]).
+//
+// Returns Random values with specified shape.
+func StatelessRandomUniform(scope *Scope, shape tf.Output, seed tf.Output, optional ...StatelessRandomUniformAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StatelessRandomUniform",
+		Input: []tf.Input{
+			shape, seed,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// StatelessRandomUniformFullIntAttr is an optional argument to StatelessRandomUniformFullInt.
+type StatelessRandomUniformFullIntAttr func(optionalAttr)
+
+// StatelessRandomUniformFullIntDtype sets the optional dtype attribute to value.
+//
+// value: The type of the output.
+// If not specified, defaults to DT_UINT64
+func StatelessRandomUniformFullIntDtype(value tf.DataType) StatelessRandomUniformFullIntAttr {
+	return func(m optionalAttr) {
+		m["dtype"] = value
+	}
+}
+
+// Outputs deterministic pseudorandom random integers from a uniform distribution.
+//
+// The generated values are uniform integers covering the whole range of `dtype`.
+//
+// The outputs are a deterministic function of `shape` and `seed`.
+//
+// Arguments:
+//	shape: The shape of the output tensor.
+//	seed: 2 seeds (shape [2]).
+//
+// Returns Random values with specified shape.
+func StatelessRandomUniformFullInt(scope *Scope, shape tf.Output, seed tf.Output, optional ...StatelessRandomUniformFullIntAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StatelessRandomUniformFullInt",
+		Input: []tf.Input{
+			shape, seed,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RetrieveTPUEmbeddingMomentumParametersGradAccumDebugAttr is an optional argument to RetrieveTPUEmbeddingMomentumParametersGradAccumDebug.
+type RetrieveTPUEmbeddingMomentumParametersGradAccumDebugAttr func(optionalAttr)
+
+// RetrieveTPUEmbeddingMomentumParametersGradAccumDebugTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func RetrieveTPUEmbeddingMomentumParametersGradAccumDebugTableId(value int64) RetrieveTPUEmbeddingMomentumParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingMomentumParametersGradAccumDebugTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingMomentumParametersGradAccumDebugTableName(value string) RetrieveTPUEmbeddingMomentumParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingMomentumParametersGradAccumDebugConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingMomentumParametersGradAccumDebugConfig(value string) RetrieveTPUEmbeddingMomentumParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Retrieve Momentum embedding parameters with debug support.
+//
+// An op that retrieves optimization parameters from embedding to host
+// memory. Must be preceded by a ConfigureTPUEmbeddingHost op that sets up
+// the correct embedding table configuration. For example, this op is
+// used to retrieve updated parameters before saving a checkpoint.
+//
+// Returns:
+//	parameters: Parameter parameters updated by the Momentum optimization algorithm.
+//	momenta: Parameter momenta updated by the Momentum optimization algorithm.
+//	gradient_accumulators: Parameter gradient_accumulators updated by the Momentum optimization algorithm.
+func RetrieveTPUEmbeddingMomentumParametersGradAccumDebug(scope *Scope, num_shards int64, shard_id int64, optional ...RetrieveTPUEmbeddingMomentumParametersGradAccumDebugAttr) (parameters tf.Output, momenta tf.Output, gradient_accumulators tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RetrieveTPUEmbeddingMomentumParametersGradAccumDebug",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// EqualAttr is an optional argument to Equal.
+type EqualAttr func(optionalAttr)
+
+// EqualIncompatibleShapeError sets the optional incompatible_shape_error attribute to value.
+// If not specified, defaults to true
+func EqualIncompatibleShapeError(value bool) EqualAttr {
+	return func(m optionalAttr) {
+		m["incompatible_shape_error"] = value
+	}
+}
+
+// Returns the truth value of (x == y) element-wise.
+//
+// *NOTE*: `Equal` supports broadcasting. More about broadcasting
+// [here](http://docs.scipy.org/doc/numpy/user/basics.broadcasting.html)
+//
+// ```python
+// x = tf.constant([2, 4])
+// y = tf.constant(2)
+// tf.math.equal(x, y) ==> array([True, False])
+//
+// x = tf.constant([2, 4])
+// y = tf.constant([2, 4])
+// tf.math.equal(x, y) ==> array([True,  True])
+// ```
+func Equal(scope *Scope, x tf.Output, y tf.Output, optional ...EqualAttr) (z tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Equal",
+		Input: []tf.Input{
+			x, y,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SparseToSparseSetOperationAttr is an optional argument to SparseToSparseSetOperation.
+type SparseToSparseSetOperationAttr func(optionalAttr)
+
+// SparseToSparseSetOperationValidateIndices sets the optional validate_indices attribute to value.
+// If not specified, defaults to true
+func SparseToSparseSetOperationValidateIndices(value bool) SparseToSparseSetOperationAttr {
+	return func(m optionalAttr) {
+		m["validate_indices"] = value
+	}
+}
+
+// Applies set operation along last dimension of 2 `SparseTensor` inputs.
+//
+// See SetOperationOp::SetOperationFromContext for values of `set_operation`.
+//
+// If `validate_indices` is `True`, `SparseToSparseSetOperation` validates the
+// order and range of `set1` and `set2` indices.
+//
+// Input `set1` is a `SparseTensor` represented by `set1_indices`, `set1_values`,
+// and `set1_shape`. For `set1` ranked `n`, 1st `n-1` dimensions must be the same
+// as `set2`. Dimension `n` contains values in a set, duplicates are allowed but
+// ignored.
+//
+// Input `set2` is a `SparseTensor` represented by `set2_indices`, `set2_values`,
+// and `set2_shape`. For `set2` ranked `n`, 1st `n-1` dimensions must be the same
+// as `set1`. Dimension `n` contains values in a set, duplicates are allowed but
+// ignored.
+//
+// If `validate_indices` is `True`, this op validates the order and range of `set1`
+// and `set2` indices.
+//
+// Output `result` is a `SparseTensor` represented by `result_indices`,
+// `result_values`, and `result_shape`. For `set1` and `set2` ranked `n`, this
+// has rank `n` and the same 1st `n-1` dimensions as `set1` and `set2`. The `nth`
+// dimension contains the result of `set_operation` applied to the corresponding
+// `[0...n-1]` dimension of `set`.
+//
+// Arguments:
+//	set1_indices: 2D `Tensor`, indices of a `SparseTensor`. Must be in row-major
+// order.
+//	set1_values: 1D `Tensor`, values of a `SparseTensor`. Must be in row-major
+// order.
+//	set1_shape: 1D `Tensor`, shape of a `SparseTensor`. `set1_shape[0...n-1]` must
+// be the same as `set2_shape[0...n-1]`, `set1_shape[n]` is the
+// max set size across `0...n-1` dimensions.
+//	set2_indices: 2D `Tensor`, indices of a `SparseTensor`. Must be in row-major
+// order.
+//	set2_values: 1D `Tensor`, values of a `SparseTensor`. Must be in row-major
+// order.
+//	set2_shape: 1D `Tensor`, shape of a `SparseTensor`. `set2_shape[0...n-1]` must
+// be the same as `set1_shape[0...n-1]`, `set2_shape[n]` is the
+// max set size across `0...n-1` dimensions.
+//
+//
+// Returns:
+//	result_indices: 2D indices of a `SparseTensor`.
+//	result_values: 1D values of a `SparseTensor`.
+//	result_shape: 1D `Tensor` shape of a `SparseTensor`. `result_shape[0...n-1]` is
+// the same as the 1st `n-1` dimensions of `set1` and `set2`, `result_shape[n]`
+// is the max result set size across all `0...n-1` dimensions.
+func SparseToSparseSetOperation(scope *Scope, set1_indices tf.Output, set1_values tf.Output, set1_shape tf.Output, set2_indices tf.Output, set2_values tf.Output, set2_shape tf.Output, set_operation string, optional ...SparseToSparseSetOperationAttr) (result_indices tf.Output, result_values tf.Output, result_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"set_operation": set_operation}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseToSparseSetOperation",
+		Input: []tf.Input{
+			set1_indices, set1_values, set1_shape, set2_indices, set2_values, set2_shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// InfeedEnqueueTupleAttr is an optional argument to InfeedEnqueueTuple.
+type InfeedEnqueueTupleAttr func(optionalAttr)
+
+// InfeedEnqueueTupleLayouts sets the optional layouts attribute to value.
+//
+// value: A vector holding the requested layout in minor-to-major sequence for
+// all the tuple shapes, in the order the shapes appear in the "shapes" input.
+// The layout elements for a sub-shape can be set to -1, in which case the
+// corresponding layout will be computed by the infeed operation.
+// If not specified, defaults to <>
+func InfeedEnqueueTupleLayouts(value []int64) InfeedEnqueueTupleAttr {
+	return func(m optionalAttr) {
+		m["layouts"] = value
+	}
+}
+
+// InfeedEnqueueTupleDeviceOrdinal sets the optional device_ordinal attribute to value.
+//
+// value: The TPU device to use. This should be -1 when the Op
+// is running on a TPU device, and >= 0 when the Op is running on the CPU
+// device.
+// If not specified, defaults to -1
+func InfeedEnqueueTupleDeviceOrdinal(value int64) InfeedEnqueueTupleAttr {
+	return func(m optionalAttr) {
+		m["device_ordinal"] = value
+	}
+}
+
+// Feeds multiple Tensor values into the computation as an XLA tuple.
+//
+// Arguments:
+//	inputs: A list of tensors that will be provided using the infeed mechanism.
+//	shapes: The shapes of each tensor in `inputs`.
+//
+// Returns the created operation.
+func InfeedEnqueueTuple(scope *Scope, inputs []tf.Output, shapes []tf.Shape, optional ...InfeedEnqueueTupleAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"shapes": shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "InfeedEnqueueTuple",
+		Input: []tf.Input{
+			tf.OutputList(inputs),
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// LoadTPUEmbeddingRMSPropParametersAttr is an optional argument to LoadTPUEmbeddingRMSPropParameters.
+type LoadTPUEmbeddingRMSPropParametersAttr func(optionalAttr)
+
+// LoadTPUEmbeddingRMSPropParametersTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func LoadTPUEmbeddingRMSPropParametersTableId(value int64) LoadTPUEmbeddingRMSPropParametersAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// LoadTPUEmbeddingRMSPropParametersTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingRMSPropParametersTableName(value string) LoadTPUEmbeddingRMSPropParametersAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// LoadTPUEmbeddingRMSPropParametersConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingRMSPropParametersConfig(value string) LoadTPUEmbeddingRMSPropParametersAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Load RMSProp embedding parameters.
+//
+// An op that loads optimization parameters into HBM for embedding. Must be
+// preceded by a ConfigureTPUEmbeddingHost op that sets up the correct
+// embedding table configuration. For example, this op is used to install
+// parameters that are loaded from a checkpoint before a training loop is
+// executed.
+//
+// Arguments:
+//	parameters: Value of parameters used in the RMSProp optimization algorithm.
+//	ms: Value of ms used in the RMSProp optimization algorithm.
+//	mom: Value of mom used in the RMSProp optimization algorithm.
+//
+//
+//
+// Returns the created operation.
+func LoadTPUEmbeddingRMSPropParameters(scope *Scope, parameters tf.Output, ms tf.Output, mom tf.Output, num_shards int64, shard_id int64, optional ...LoadTPUEmbeddingRMSPropParametersAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LoadTPUEmbeddingRMSPropParameters",
+		Input: []tf.Input{
+			parameters, ms, mom,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Creates a Dataset that returns pseudorandom numbers.
+//
+// Creates a Dataset that returns a stream of uniformly distributed
+// pseudorandom 64-bit signed integers.
+//
+// In the TensorFlow Python API, you can instantiate this dataset via the
+// class `tf.data.experimental.RandomDataset`.
+//
+// Instances of this dataset are also created as a result of the
+// `hoist_random_uniform` static optimization. Whether this optimization is
+// performed is determined by the `experimental_optimization.hoist_random_uniform`
+// option of `tf.data.Options`.
+//
+// Arguments:
+//	seed: A scalar seed for the random number generator. If either seed or
+// seed2 is set to be non-zero, the random number generator is seeded
+// by the given seed.  Otherwise, a random seed is used.
+//	seed2: A second scalar seed to avoid seed collision.
+//
+//
+func RandomDataset(scope *Scope, seed tf.Output, seed2 tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "RandomDataset",
+		Input: []tf.Input{
+			seed, seed2,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// FractionalAvgPoolAttr is an optional argument to FractionalAvgPool.
+type FractionalAvgPoolAttr func(optionalAttr)
+
+// FractionalAvgPoolPseudoRandom sets the optional pseudo_random attribute to value.
+//
+// value: When set to True, generates the pooling sequence in a
+// pseudorandom fashion, otherwise, in a random fashion. Check paper [Benjamin
+// Graham, Fractional Max-Pooling](http://arxiv.org/abs/1412.6071) for
+// difference between pseudorandom and random.
+// If not specified, defaults to false
+func FractionalAvgPoolPseudoRandom(value bool) FractionalAvgPoolAttr {
+	return func(m optionalAttr) {
+		m["pseudo_random"] = value
+	}
+}
+
+// FractionalAvgPoolOverlapping sets the optional overlapping attribute to value.
+//
+// value: When set to True, it means when pooling, the values at the boundary
+// of adjacent pooling cells are used by both cells. For example:
+//
+// `index  0  1  2  3  4`
+//
+// `value  20 5  16 3  7`
+//
+// If the pooling sequence is [0, 2, 4], then 16, at index 2 will be used twice.
+// The result would be [41/3, 26/3] for fractional avg pooling.
+// If not specified, defaults to false
+func FractionalAvgPoolOverlapping(value bool) FractionalAvgPoolAttr {
+	return func(m optionalAttr) {
+		m["overlapping"] = value
+	}
+}
+
+// FractionalAvgPoolDeterministic sets the optional deterministic attribute to value.
+//
+// value: When set to True, a fixed pooling region will be used when
+// iterating over a FractionalAvgPool node in the computation graph. Mainly used
+// in unit test to make FractionalAvgPool deterministic.
+// If not specified, defaults to false
+func FractionalAvgPoolDeterministic(value bool) FractionalAvgPoolAttr {
+	return func(m optionalAttr) {
+		m["deterministic"] = value
+	}
+}
+
+// FractionalAvgPoolSeed sets the optional seed attribute to value.
+//
+// value: If either seed or seed2 are set to be non-zero, the random number
+// generator is seeded by the given seed.  Otherwise, it is seeded by a
+// random seed.
+// If not specified, defaults to 0
+func FractionalAvgPoolSeed(value int64) FractionalAvgPoolAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// FractionalAvgPoolSeed2 sets the optional seed2 attribute to value.
+//
+// value: An second seed to avoid seed collision.
+// If not specified, defaults to 0
+func FractionalAvgPoolSeed2(value int64) FractionalAvgPoolAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// Performs fractional average pooling on the input.
+//
+// Fractional average pooling is similar to Fractional max pooling in the pooling
+// region generation step. The only difference is that after pooling regions are
+// generated, a mean operation is performed instead of a max operation in each
+// pooling region.
+//
+// Arguments:
+//	value: 4-D with shape `[batch, height, width, channels]`.
+//	pooling_ratio: Pooling ratio for each dimension of `value`, currently only
+// supports row and col dimension and should be >= 1.0. For example, a valid
+// pooling ratio looks like [1.0, 1.44, 1.73, 1.0]. The first and last elements
+// must be 1.0 because we don't allow pooling on batch and channels
+// dimensions. 1.44 and 1.73 are pooling ratio on height and width dimensions
+// respectively.
+//
+// Returns:
+//	output: output tensor after fractional avg pooling.
+//	row_pooling_sequence: row pooling sequence, needed to calculate gradient.
+//	col_pooling_sequence: column pooling sequence, needed to calculate gradient.
+func FractionalAvgPool(scope *Scope, value tf.Output, pooling_ratio []float32, optional ...FractionalAvgPoolAttr) (output tf.Output, row_pooling_sequence tf.Output, col_pooling_sequence tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"pooling_ratio": pooling_ratio}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "FractionalAvgPool",
+		Input: []tf.Input{
+			value,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// StatefulUniformFullIntAttr is an optional argument to StatefulUniformFullInt.
+type StatefulUniformFullIntAttr func(optionalAttr)
+
+// StatefulUniformFullIntDtype sets the optional dtype attribute to value.
+//
+// value: The type of the output.
+// If not specified, defaults to DT_UINT64
+func StatefulUniformFullIntDtype(value tf.DataType) StatefulUniformFullIntAttr {
+	return func(m optionalAttr) {
+		m["dtype"] = value
+	}
+}
+
+// Outputs random integers from a uniform distribution.
+//
+// The generated values are uniform integers covering the whole range of `dtype`.
+//
+// Arguments:
+//	resource: The handle of the resource variable that stores the state of the RNG.
+//	algorithm: The RNG algorithm.
+//	shape: The shape of the output tensor.
+//
+// Returns Random values with specified shape.
+func StatefulUniformFullInt(scope *Scope, resource tf.Output, algorithm tf.Output, shape tf.Output, optional ...StatefulUniformFullIntAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "StatefulUniformFullInt",
+		Input: []tf.Input{
+			resource, algorithm, shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// LoadTPUEmbeddingStochasticGradientDescentParametersAttr is an optional argument to LoadTPUEmbeddingStochasticGradientDescentParameters.
+type LoadTPUEmbeddingStochasticGradientDescentParametersAttr func(optionalAttr)
+
+// LoadTPUEmbeddingStochasticGradientDescentParametersTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func LoadTPUEmbeddingStochasticGradientDescentParametersTableId(value int64) LoadTPUEmbeddingStochasticGradientDescentParametersAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// LoadTPUEmbeddingStochasticGradientDescentParametersTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingStochasticGradientDescentParametersTableName(value string) LoadTPUEmbeddingStochasticGradientDescentParametersAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// LoadTPUEmbeddingStochasticGradientDescentParametersConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingStochasticGradientDescentParametersConfig(value string) LoadTPUEmbeddingStochasticGradientDescentParametersAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Load SGD embedding parameters.
+//
+// An op that loads optimization parameters into HBM for embedding. Must be
+// preceded by a ConfigureTPUEmbeddingHost op that sets up the correct
+// embedding table configuration. For example, this op is used to install
+// parameters that are loaded from a checkpoint before a training loop is
+// executed.
+//
+// Arguments:
+//	parameters: Value of parameters used in the stochastic gradient descent optimization algorithm.
+//
+//
+//
+// Returns the created operation.
+func LoadTPUEmbeddingStochasticGradientDescentParameters(scope *Scope, parameters tf.Output, num_shards int64, shard_id int64, optional ...LoadTPUEmbeddingStochasticGradientDescentParametersAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LoadTPUEmbeddingStochasticGradientDescentParameters",
+		Input: []tf.Input{
+			parameters,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// RequantizePerChannelAttr is an optional argument to RequantizePerChannel.
+type RequantizePerChannelAttr func(optionalAttr)
+
+// RequantizePerChannelOutType sets the optional out_type attribute to value.
+//
+// value: The quantized type of output tensor that needs to be converted.
+// If not specified, defaults to DT_QUINT8
+func RequantizePerChannelOutType(value tf.DataType) RequantizePerChannelAttr {
+	return func(m optionalAttr) {
+		m["out_type"] = value
+	}
+}
+
+// Requantizes input with min and max values known per channel.
+//
+// Arguments:
+//	input: The original input tensor.
+//	input_min: The minimum value of the input tensor
+//	input_max: The maximum value of the input tensor.
+//	requested_output_min: The minimum value of the output tensor requested.
+//	requested_output_max: The maximum value of the output tensor requested.
+//
+// Returns:
+//	output: Output tensor.
+//	output_min: The minimum value of the final output tensor
+//	output_max: The maximum value of the final output tensor.
+func RequantizePerChannel(scope *Scope, input tf.Output, input_min tf.Output, input_max tf.Output, requested_output_min tf.Output, requested_output_max tf.Output, optional ...RequantizePerChannelAttr) (output tf.Output, output_min tf.Output, output_max tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RequantizePerChannel",
+		Input: []tf.Input{
+			input, input_min, input_max, requested_output_min, requested_output_max,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// LeakyReluAttr is an optional argument to LeakyRelu.
+type LeakyReluAttr func(optionalAttr)
+
+// LeakyReluAlpha sets the optional alpha attribute to value.
+// If not specified, defaults to 0.2
+func LeakyReluAlpha(value float32) LeakyReluAttr {
+	return func(m optionalAttr) {
+		m["alpha"] = value
+	}
+}
+
+// Computes rectified linear: `max(features, features * alpha)`.
+func LeakyRelu(scope *Scope, features tf.Output, optional ...LeakyReluAttr) (activations tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LeakyRelu",
+		Input: []tf.Input{
+			features,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Component-wise divides a SparseTensor by a dense Tensor.
+//
+// *Limitation*: this Op only broadcasts the dense side to the sparse side, but not
+// the other direction.
+//
+// Arguments:
+//	sp_indices: 2-D.  `N x R` matrix with the indices of non-empty values in a
+// SparseTensor, possibly not in canonical ordering.
+//	sp_values: 1-D.  `N` non-empty values corresponding to `sp_indices`.
+//	sp_shape: 1-D.  Shape of the input SparseTensor.
+//	dense: `R`-D.  The dense Tensor operand.
+//
+// Returns 1-D.  The `N` values that are operated on.
+func SparseDenseCwiseDiv(scope *Scope, sp_indices tf.Output, sp_values tf.Output, sp_shape tf.Output, dense tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SparseDenseCwiseDiv",
+		Input: []tf.Input{
+			sp_indices, sp_values, sp_shape, dense,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// EnqueueTPUEmbeddingIntegerBatchAttr is an optional argument to EnqueueTPUEmbeddingIntegerBatch.
+type EnqueueTPUEmbeddingIntegerBatchAttr func(optionalAttr)
+
+// EnqueueTPUEmbeddingIntegerBatchDeviceOrdinal sets the optional device_ordinal attribute to value.
+//
+// value: The TPU device to use. Should be >= 0 and less than the number
+// of TPU cores in the task on which the node is placed.
+// If not specified, defaults to -1
+func EnqueueTPUEmbeddingIntegerBatchDeviceOrdinal(value int64) EnqueueTPUEmbeddingIntegerBatchAttr {
+	return func(m optionalAttr) {
+		m["device_ordinal"] = value
+	}
+}
+
+// An op that enqueues a list of input batch tensors to TPUEmbedding.
+//
+// Arguments:
+//	batch: A list of 1D tensors, one for each embedding table, containing the
+// indices into the tables.
+//	mode_override: A string input that overrides the mode specified in the
+// TPUEmbeddingConfiguration. Supported values are {'unspecified', 'inference',
+// 'training', 'backward_pass_only'}. When set to 'unspecified', the mode set
+// in TPUEmbeddingConfiguration is used, otherwise mode_override is used.
+//
+// Returns the created operation.
+func EnqueueTPUEmbeddingIntegerBatch(scope *Scope, batch []tf.Output, mode_override tf.Output, optional ...EnqueueTPUEmbeddingIntegerBatchAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "EnqueueTPUEmbeddingIntegerBatch",
+		Input: []tf.Input{
+			tf.OutputList(batch), mode_override,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// MapClearAttr is an optional argument to MapClear.
+type MapClearAttr func(optionalAttr)
+
+// MapClearCapacity sets the optional capacity attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func MapClearCapacity(value int64) MapClearAttr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// MapClearMemoryLimit sets the optional memory_limit attribute to value.
+// If not specified, defaults to 0
+//
+// REQUIRES: value >= 0
+func MapClearMemoryLimit(value int64) MapClearAttr {
+	return func(m optionalAttr) {
+		m["memory_limit"] = value
+	}
+}
+
+// MapClearContainer sets the optional container attribute to value.
+// If not specified, defaults to ""
+func MapClearContainer(value string) MapClearAttr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// MapClearSharedName sets the optional shared_name attribute to value.
+// If not specified, defaults to ""
+func MapClearSharedName(value string) MapClearAttr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// Op removes all elements in the underlying container.
+//
+// Returns the created operation.
+func MapClear(scope *Scope, dtypes []tf.DataType, optional ...MapClearAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MapClear",
+
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Deserialize `SparseTensor` objects.
+//
+// The input `serialized_sparse` must have the shape `[?, ?, ..., ?, 3]` where
+// the last dimension stores serialized `SparseTensor` objects and the other N
+// dimensions (N >= 0) correspond to a batch. The ranks of the original
+// `SparseTensor` objects must all match. When the final `SparseTensor` is
+// created, its rank is the rank of the incoming `SparseTensor` objects plus N;
+// the sparse tensors have been concatenated along new dimensions, one for each
+// batch.
+//
+// The output `SparseTensor` object's shape values for the original dimensions
+// are the max across the input `SparseTensor` objects' shape values for the
+// corresponding dimensions. The new dimensions match the size of the batch.
+//
+// The input `SparseTensor` objects' indices are assumed ordered in
+// standard lexicographic order.  If this is not the case, after this
+// step run `SparseReorder` to restore index ordering.
+//
+// For example, if the serialized input is a `[2 x 3]` matrix representing two
+// original `SparseTensor` objects:
+//
+//     index = [ 0]
+//             [10]
+//             [20]
+//     values = [1, 2, 3]
+//     shape = [50]
+//
+// and
+//
+//     index = [ 2]
+//             [10]
+//     values = [4, 5]
+//     shape = [30]
+//
+// then the final deserialized `SparseTensor` will be:
+//
+//     index = [0  0]
+//             [0 10]
+//             [0 20]
+//             [1  2]
+//             [1 10]
+//     values = [1, 2, 3, 4, 5]
+//     shape = [2 50]
+//
+// Arguments:
+//	serialized_sparse: The serialized `SparseTensor` objects. The last dimension
+// must have 3 columns.
+//	dtype: The `dtype` of the serialized `SparseTensor` objects.
+func DeserializeSparse(scope *Scope, serialized_sparse tf.Output, dtype tf.DataType) (sparse_indices tf.Output, sparse_values tf.Output, sparse_shape tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	opspec := tf.OpSpec{
+		Type: "DeserializeSparse",
+		Input: []tf.Input{
+			serialized_sparse,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Decode web-safe base64-encoded strings.
+//
+// Input may or may not have padding at the end. See EncodeBase64 for padding.
+// Web-safe means that input must use - and _ instead of + and /.
+//
+// Arguments:
+//	input: Base64 strings to decode.
+//
+// Returns Decoded strings.
+func DecodeBase64(scope *Scope, input tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "DecodeBase64",
+		Input: []tf.Input{
+			input,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// LoadTPUEmbeddingAdagradParametersAttr is an optional argument to LoadTPUEmbeddingAdagradParameters.
+type LoadTPUEmbeddingAdagradParametersAttr func(optionalAttr)
+
+// LoadTPUEmbeddingAdagradParametersTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func LoadTPUEmbeddingAdagradParametersTableId(value int64) LoadTPUEmbeddingAdagradParametersAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// LoadTPUEmbeddingAdagradParametersTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingAdagradParametersTableName(value string) LoadTPUEmbeddingAdagradParametersAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// LoadTPUEmbeddingAdagradParametersConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingAdagradParametersConfig(value string) LoadTPUEmbeddingAdagradParametersAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Load Adagrad embedding parameters.
+//
+// An op that loads optimization parameters into HBM for embedding. Must be
+// preceded by a ConfigureTPUEmbeddingHost op that sets up the correct
+// embedding table configuration. For example, this op is used to install
+// parameters that are loaded from a checkpoint before a training loop is
+// executed.
+//
+// Arguments:
+//	parameters: Value of parameters used in the Adagrad optimization algorithm.
+//	accumulators: Value of accumulators used in the Adagrad optimization algorithm.
+//
+//
+//
+// Returns the created operation.
+func LoadTPUEmbeddingAdagradParameters(scope *Scope, parameters tf.Output, accumulators tf.Output, num_shards int64, shard_id int64, optional ...LoadTPUEmbeddingAdagradParametersAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LoadTPUEmbeddingAdagradParameters",
+		Input: []tf.Input{
+			parameters, accumulators,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Returns the gradient of `Tile`.
+//
+// DEPRECATED at GraphDef version 3: TileGrad has been replaced with reduce_sum
+//
+// Since `Tile` takes an input and repeats the input `multiples` times
+// along each dimension, `TileGrad` takes in `multiples` and aggregates
+// each repeated tile of `input` into `output`.
+func TileGrad(scope *Scope, input tf.Output, multiples tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TileGrad",
+		Input: []tf.Input{
+			input, multiples,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// AudioSummaryAttr is an optional argument to AudioSummary.
+type AudioSummaryAttr func(optionalAttr)
+
+// AudioSummaryMaxOutputs sets the optional max_outputs attribute to value.
+//
+// value: Max number of batch elements to generate audio for.
+// If not specified, defaults to 3
+//
+// REQUIRES: value >= 1
+func AudioSummaryMaxOutputs(value int64) AudioSummaryAttr {
+	return func(m optionalAttr) {
+		m["max_outputs"] = value
+	}
+}
+
+// Outputs a `Summary` protocol buffer with audio.
+//
+// DEPRECATED at GraphDef version 15: Use AudioSummaryV2.
+//
+// The summary has up to `max_outputs` summary values containing audio. The
+// audio is built from `tensor` which must be 3-D with shape `[batch_size,
+// frames, channels]` or 2-D with shape `[batch_size, frames]`. The values are
+// assumed to be in the range of `[-1.0, 1.0]` with a sample rate of `sample_rate`.
+//
+// The `tag` argument is a scalar `Tensor` of type `string`.  It is used to
+// build the `tag` of the summary values:
+//
+// *  If `max_outputs` is 1, the summary value tag is '*tag*/audio'.
+// *  If `max_outputs` is greater than 1, the summary value tags are
+//    generated sequentially as '*tag*/audio/0', '*tag*/audio/1', etc.
+//
+// Arguments:
+//	tag: Scalar. Used to build the `tag` attribute of the summary values.
+//	tensor: 2-D of shape `[batch_size, frames]`.
+//	sample_rate: The sample rate of the signal in hertz.
+//
+// Returns Scalar. Serialized `Summary` protocol buffer.
+func AudioSummary(scope *Scope, tag tf.Output, tensor tf.Output, sample_rate float32, optional ...AudioSummaryAttr) (summary tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"sample_rate": sample_rate}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "AudioSummary",
+		Input: []tf.Input{
+			tag, tensor,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// LoadTPUEmbeddingFTRLParametersAttr is an optional argument to LoadTPUEmbeddingFTRLParameters.
+type LoadTPUEmbeddingFTRLParametersAttr func(optionalAttr)
+
+// LoadTPUEmbeddingFTRLParametersTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func LoadTPUEmbeddingFTRLParametersTableId(value int64) LoadTPUEmbeddingFTRLParametersAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// LoadTPUEmbeddingFTRLParametersTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingFTRLParametersTableName(value string) LoadTPUEmbeddingFTRLParametersAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// LoadTPUEmbeddingFTRLParametersConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingFTRLParametersConfig(value string) LoadTPUEmbeddingFTRLParametersAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Load FTRL embedding parameters.
+//
+// An op that loads optimization parameters into HBM for embedding. Must be
+// preceded by a ConfigureTPUEmbeddingHost op that sets up the correct
+// embedding table configuration. For example, this op is used to install
+// parameters that are loaded from a checkpoint before a training loop is
+// executed.
+//
+// Arguments:
+//	parameters: Value of parameters used in the FTRL optimization algorithm.
+//	accumulators: Value of accumulators used in the FTRL optimization algorithm.
+//	linears: Value of linears used in the FTRL optimization algorithm.
+//
+//
+//
+// Returns the created operation.
+func LoadTPUEmbeddingFTRLParameters(scope *Scope, parameters tf.Output, accumulators tf.Output, linears tf.Output, num_shards int64, shard_id int64, optional ...LoadTPUEmbeddingFTRLParametersAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LoadTPUEmbeddingFTRLParameters",
+		Input: []tf.Input{
+			parameters, accumulators, linears,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Conv3DBackpropInputAttr is an optional argument to Conv3DBackpropInput.
+type Conv3DBackpropInputAttr func(optionalAttr)
+
+// Conv3DBackpropInputDilations sets the optional dilations attribute to value.
+// If not specified, defaults to <i:1 i:1 i:1 i:1 i:1 >
+func Conv3DBackpropInputDilations(value []int64) Conv3DBackpropInputAttr {
+	return func(m optionalAttr) {
+		m["dilations"] = value
+	}
+}
+
+// Computes the gradients of 3-D convolution with respect to the input.
+//
+// DEPRECATED at GraphDef version 10: Use Conv3DBackpropInputV2
+//
+// Arguments:
+//	input: Shape `[batch, depth, rows, cols, in_channels]`.
+//	filter: Shape `[depth, rows, cols, in_channels, out_channels]`.
+// `in_channels` must match between `input` and `filter`.
+//	out_backprop: Backprop signal of shape `[batch, out_depth, out_rows, out_cols,
+// out_channels]`.
+//	strides: 1-D tensor of length 5. The stride of the sliding window for each
+// dimension of `input`. Must have `strides[0] = strides[4] = 1`.
+//	padding: The type of padding algorithm to use.
+func Conv3DBackpropInput(scope *Scope, input tf.Output, filter tf.Output, out_backprop tf.Output, strides []int64, padding string, optional ...Conv3DBackpropInputAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "Conv3DBackpropInput",
+		Input: []tf.Input{
+			input, filter, out_backprop,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// DepthwiseConv2dNativeAttr is an optional argument to DepthwiseConv2dNative.
+type DepthwiseConv2dNativeAttr func(optionalAttr)
+
+// DepthwiseConv2dNativeExplicitPaddings sets the optional explicit_paddings attribute to value.
+// If not specified, defaults to <>
+func DepthwiseConv2dNativeExplicitPaddings(value []int64) DepthwiseConv2dNativeAttr {
+	return func(m optionalAttr) {
+		m["explicit_paddings"] = value
+	}
+}
+
+// DepthwiseConv2dNativeDataFormat sets the optional data_format attribute to value.
+//
+// value: Specify the data format of the input and output data. With the
+// default format "NHWC", the data is stored in the order of:
+//     [batch, height, width, channels].
+// Alternatively, the format could be "NCHW", the data storage order of:
+//     [batch, channels, height, width].
+// If not specified, defaults to "NHWC"
+func DepthwiseConv2dNativeDataFormat(value string) DepthwiseConv2dNativeAttr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// DepthwiseConv2dNativeDilations sets the optional dilations attribute to value.
+//
+// value: 1-D tensor of length 4.  The dilation factor for each dimension of
+// `input`. If set to k > 1, there will be k-1 skipped cells between each filter
+// element on that dimension. The dimension order is determined by the value of
+// `data_format`, see above for details. Dilations in the batch and depth
+// dimensions must be 1.
+// If not specified, defaults to <i:1 i:1 i:1 i:1 >
+func DepthwiseConv2dNativeDilations(value []int64) DepthwiseConv2dNativeAttr {
+	return func(m optionalAttr) {
+		m["dilations"] = value
+	}
+}
+
+// Computes a 2-D depthwise convolution given 4-D `input` and `filter` tensors.
+//
+// Given an input tensor of shape `[batch, in_height, in_width, in_channels]`
+// and a filter / kernel tensor of shape
+// `[filter_height, filter_width, in_channels, channel_multiplier]`, containing
+// `in_channels` convolutional filters of depth 1, `depthwise_conv2d` applies
+// a different filter to each input channel (expanding from 1 channel to
+// `channel_multiplier` channels for each), then concatenates the results
+// together. Thus, the output has `in_channels * channel_multiplier` channels.
+//
+// ```
+// for k in 0..in_channels-1
+//   for q in 0..channel_multiplier-1
+//     output[b, i, j, k * channel_multiplier + q] =
+//       sum_{di, dj} input[b, strides[1] * i + di, strides[2] * j + dj, k] *
+//                         filter[di, dj, k, q]
+// ```
+//
+// Must have `strides[0] = strides[3] = 1`.  For the most common case of the same
+// horizontal and vertices strides, `strides = [1, stride, stride, 1]`.
+//
+// Arguments:
+//
+//
+//	strides: 1-D of length 4.  The stride of the sliding window for each dimension
+// of `input`.
+//	padding: The type of padding algorithm to use.
+func DepthwiseConv2dNative(scope *Scope, input tf.Output, filter tf.Output, strides []int64, padding string, optional ...DepthwiseConv2dNativeAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"strides": strides, "padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "DepthwiseConv2dNative",
+		Input: []tf.Input{
+			input, filter,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates an all-zeros CSRSparseMatrix with shape `dense_shape`.
+//
+// Arguments:
+//	dense_shape: The desired matrix shape.
+//
+//
+// Returns An empty CSR matrix with shape `dense_shape`.
+func SparseMatrixZeros(scope *Scope, dense_shape tf.Output, type_ tf.DataType) (sparse_matrix tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"type": type_}
+	opspec := tf.OpSpec{
+		Type: "SparseMatrixZeros",
+		Input: []tf.Input{
+			dense_shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// PaddingFIFOQueueV2Attr is an optional argument to PaddingFIFOQueueV2.
+type PaddingFIFOQueueV2Attr func(optionalAttr)
+
+// PaddingFIFOQueueV2Shapes sets the optional shapes attribute to value.
+//
+// value: The shape of each component in a value. The length of this attr must
+// be either 0 or the same as the length of component_types.
+// Shapes of fixed rank but variable size are allowed by setting
+// any shape dimension to -1.  In this case, the inputs' shape may vary along
+// the given dimension, and DequeueMany will pad the given dimension with
+// zeros up to the maximum shape of all elements in the given batch.
+// If the length of this attr is 0, different queue elements may have
+// different ranks and shapes, but only one element may be dequeued at a time.
+// If not specified, defaults to <>
+//
+// REQUIRES: len(value) >= 0
+func PaddingFIFOQueueV2Shapes(value []tf.Shape) PaddingFIFOQueueV2Attr {
+	return func(m optionalAttr) {
+		m["shapes"] = value
+	}
+}
+
+// PaddingFIFOQueueV2Capacity sets the optional capacity attribute to value.
+//
+// value: The upper bound on the number of elements in this queue.
+// Negative numbers mean no limit.
+// If not specified, defaults to -1
+func PaddingFIFOQueueV2Capacity(value int64) PaddingFIFOQueueV2Attr {
+	return func(m optionalAttr) {
+		m["capacity"] = value
+	}
+}
+
+// PaddingFIFOQueueV2Container sets the optional container attribute to value.
+//
+// value: If non-empty, this queue is placed in the given container.
+// Otherwise, a default container is used.
+// If not specified, defaults to ""
+func PaddingFIFOQueueV2Container(value string) PaddingFIFOQueueV2Attr {
+	return func(m optionalAttr) {
+		m["container"] = value
+	}
+}
+
+// PaddingFIFOQueueV2SharedName sets the optional shared_name attribute to value.
+//
+// value: If non-empty, this queue will be shared under the given name
+// across multiple sessions.
+// If not specified, defaults to ""
+func PaddingFIFOQueueV2SharedName(value string) PaddingFIFOQueueV2Attr {
+	return func(m optionalAttr) {
+		m["shared_name"] = value
+	}
+}
+
+// A queue that produces elements in first-in first-out order.
+//
+// Variable-size shapes are allowed by setting the corresponding shape dimensions
+// to 0 in the shape attr.  In this case DequeueMany will pad up to the maximum
+// size of any given element in the minibatch.  See below for details.
+//
+// Arguments:
+//	component_types: The type of each component in a value.
+//
+// Returns The handle to the queue.
+func PaddingFIFOQueueV2(scope *Scope, component_types []tf.DataType, optional ...PaddingFIFOQueueV2Attr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"component_types": component_types}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "PaddingFIFOQueueV2",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// LoadTPUEmbeddingMomentumParametersGradAccumDebugAttr is an optional argument to LoadTPUEmbeddingMomentumParametersGradAccumDebug.
+type LoadTPUEmbeddingMomentumParametersGradAccumDebugAttr func(optionalAttr)
+
+// LoadTPUEmbeddingMomentumParametersGradAccumDebugTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func LoadTPUEmbeddingMomentumParametersGradAccumDebugTableId(value int64) LoadTPUEmbeddingMomentumParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// LoadTPUEmbeddingMomentumParametersGradAccumDebugTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingMomentumParametersGradAccumDebugTableName(value string) LoadTPUEmbeddingMomentumParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// LoadTPUEmbeddingMomentumParametersGradAccumDebugConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func LoadTPUEmbeddingMomentumParametersGradAccumDebugConfig(value string) LoadTPUEmbeddingMomentumParametersGradAccumDebugAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Load Momentum embedding parameters with debug support.
+//
+// An op that loads optimization parameters into HBM for embedding. Must be
+// preceded by a ConfigureTPUEmbeddingHost op that sets up the correct
+// embedding table configuration. For example, this op is used to install
+// parameters that are loaded from a checkpoint before a training loop is
+// executed.
+//
+// Arguments:
+//	parameters: Value of parameters used in the Momentum optimization algorithm.
+//	momenta: Value of momenta used in the Momentum optimization algorithm.
+//	gradient_accumulators: Value of gradient_accumulators used in the Momentum optimization algorithm.
+//
+//
+//
+// Returns the created operation.
+func LoadTPUEmbeddingMomentumParametersGradAccumDebug(scope *Scope, parameters tf.Output, momenta tf.Output, gradient_accumulators tf.Output, num_shards int64, shard_id int64, optional ...LoadTPUEmbeddingMomentumParametersGradAccumDebugAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "LoadTPUEmbeddingMomentumParametersGradAccumDebug",
+		Input: []tf.Input{
+			parameters, momenta, gradient_accumulators,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Constructs a tensor by tiling a given tensor.
+//
+// This operation creates a new tensor by replicating `input` `multiples` times.
+// The output tensor's i'th dimension has `input.dims(i) * multiples[i]` elements,
+// and the values of `input` are replicated `multiples[i]` times along the 'i'th
+// dimension. For example, tiling `[a b c d]` by `[2]` produces
+// `[a b c d a b c d]`.
+//
+// >>> a = tf.constant([[1,2,3],[4,5,6]], tf.int32)
+// >>> b = tf.constant([1,2], tf.int32)
+// >>> tf.tile(a, b)
+// <tf.Tensor: shape=(2, 6), dtype=int32, numpy=
+// array([[1, 2, 3, 1, 2, 3],
+//        [4, 5, 6, 4, 5, 6]], dtype=int32)>
+// >>> c = tf.constant([2,1], tf.int32)
+// >>> tf.tile(a, c)
+// <tf.Tensor: shape=(4, 3), dtype=int32, numpy=
+// array([[1, 2, 3],
+//        [4, 5, 6],
+//        [1, 2, 3],
+//        [4, 5, 6]], dtype=int32)>
+// >>> d = tf.constant([2,2], tf.int32)
+// >>> tf.tile(a, d)
+// <tf.Tensor: shape=(4, 6), dtype=int32, numpy=
+// array([[1, 2, 3, 1, 2, 3],
+//        [4, 5, 6, 4, 5, 6],
+//        [1, 2, 3, 1, 2, 3],
+//        [4, 5, 6, 4, 5, 6]], dtype=int32)>
+//
+// Arguments:
+//	input: 1-D or higher.
+//	multiples: 1-D. Length must be the same as the number of dimensions in `input`
+func Tile(scope *Scope, input tf.Output, multiples tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "Tile",
+		Input: []tf.Input{
+			input, multiples,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SerializeSparseAttr is an optional argument to SerializeSparse.
+type SerializeSparseAttr func(optionalAttr)
+
+// SerializeSparseOutType sets the optional out_type attribute to value.
+//
+// value: The `dtype` to use for serialization; the supported types are `string`
+// (default) and `variant`.
+// If not specified, defaults to DT_STRING
+func SerializeSparseOutType(value tf.DataType) SerializeSparseAttr {
+	return func(m optionalAttr) {
+		m["out_type"] = value
+	}
+}
+
+// Serialize a `SparseTensor` into a `[3]` `Tensor` object.
+//
+// Arguments:
+//	sparse_indices: 2-D.  The `indices` of the `SparseTensor`.
+//	sparse_values: 1-D.  The `values` of the `SparseTensor`.
+//	sparse_shape: 1-D.  The `shape` of the `SparseTensor`.
+func SerializeSparse(scope *Scope, sparse_indices tf.Output, sparse_values tf.Output, sparse_shape tf.Output, optional ...SerializeSparseAttr) (serialized_sparse tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SerializeSparse",
+		Input: []tf.Input{
+			sparse_indices, sparse_values, sparse_shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Extracts the average gradient in the given ConditionalAccumulator.
+//
+// The op blocks until sufficient (i.e., more than num_required)
+// gradients have been accumulated.  If the accumulator has already
+// aggregated more than num_required gradients, it returns the average of
+// the accumulated gradients.  Also automatically increments the recorded
+// global_step in the accumulator by 1, and resets the aggregate to 0.
+//
+// Arguments:
+//	handle: The handle to an accumulator.
+//	num_required: Number of gradients required before we return an aggregate.
+//	dtype: The data type of accumulated gradients. Needs to correspond to the type
+// of the accumulator.
+//
+// Returns The average of the accumulated gradients.
+func ResourceAccumulatorTakeGradient(scope *Scope, handle tf.Output, num_required tf.Output, dtype tf.DataType) (average tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtype": dtype}
+	opspec := tf.OpSpec{
+		Type: "ResourceAccumulatorTakeGradient",
+		Input: []tf.Input{
+			handle, num_required,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// InfeedEnqueueAttr is an optional argument to InfeedEnqueue.
+type InfeedEnqueueAttr func(optionalAttr)
+
+// InfeedEnqueueShape sets the optional shape attribute to value.
+//
+// value: The shape of the tensor.
+// If not specified, defaults to <>
+func InfeedEnqueueShape(value tf.Shape) InfeedEnqueueAttr {
+	return func(m optionalAttr) {
+		m["shape"] = value
+	}
+}
+
+// InfeedEnqueueLayout sets the optional layout attribute to value.
+//
+// value: A vector holding the requested layout in minor-to-major sequence.
+// If a layout attribute is passed, but its values are all -1, the layout will
+// be computed by the infeed operation.
+// If not specified, defaults to <>
+func InfeedEnqueueLayout(value []int64) InfeedEnqueueAttr {
+	return func(m optionalAttr) {
+		m["layout"] = value
+	}
+}
+
+// InfeedEnqueueDeviceOrdinal sets the optional device_ordinal attribute to value.
+//
+// value: The TPU device to use. This should be -1 when the Op
+// is running on a TPU device, and >= 0 when the Op is running on the CPU
+// device.
+// If not specified, defaults to -1
+func InfeedEnqueueDeviceOrdinal(value int64) InfeedEnqueueAttr {
+	return func(m optionalAttr) {
+		m["device_ordinal"] = value
+	}
+}
+
+// An op which feeds a single Tensor value into the computation.
+//
+// Arguments:
+//	input: A tensor that will be provided using the infeed mechanism.
+//
+// Returns the created operation.
+func InfeedEnqueue(scope *Scope, input tf.Output, optional ...InfeedEnqueueAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "InfeedEnqueue",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Computes the mean along segments of a tensor.
+//
+// Read
+// [the section on segmentation](https://tensorflow.org/api_docs/python/tf/math#Segmentation)
+// for an explanation of segments.
+//
+// Computes a tensor such that
+// \\(output_i = \frac{\sum_j data_j}{N}\\) where `mean` is
+// over `j` such that `segment_ids[j] == i` and `N` is the total number of
+// values summed.
+//
+// If the mean is empty for a given segment ID `i`, `output[i] = 0`.
+//
+// <div style="width:70%; margin:auto; margin-bottom:10px; margin-top:20px;">
+// <img style="width:100%" src="https://www.tensorflow.org/images/SegmentMean.png" alt>
+// </div>
+//
+// For example:
+//
+// ```
+// c = tf.constant([[1.0,2,3,4], [4, 3, 2, 1], [5,6,7,8]])
+// tf.segment_mean(c, tf.constant([0, 0, 1]))
+// # ==> [[2.5, 2.5, 2.5, 2.5],
+// #      [5, 6, 7, 8]]
+// ```
+//
+//
+// Arguments:
+//
+//	segment_ids: A 1-D tensor whose size is equal to the size of `data`'s
+// first dimension.  Values should be sorted and can be repeated.
+//
+// Returns Has same shape as data, except for dimension 0 which
+// has size `k`, the number of segments.
+func SegmentMean(scope *Scope, data tf.Output, segment_ids tf.Output) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "SegmentMean",
+		Input: []tf.Input{
+			data, segment_ids,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// CTCLossV2Attr is an optional argument to CTCLossV2.
+type CTCLossV2Attr func(optionalAttr)
+
+// CTCLossV2PreprocessCollapseRepeated sets the optional preprocess_collapse_repeated attribute to value.
+//
+// value: Scalar, if true then repeated labels are
+// collapsed prior to the CTC calculation.
+// If not specified, defaults to false
+func CTCLossV2PreprocessCollapseRepeated(value bool) CTCLossV2Attr {
+	return func(m optionalAttr) {
+		m["preprocess_collapse_repeated"] = value
+	}
+}
+
+// CTCLossV2CtcMergeRepeated sets the optional ctc_merge_repeated attribute to value.
+//
+// value: Scalar.  If set to false, *during* CTC calculation
+// repeated non-blank labels will not be merged and are interpreted as
+// individual labels.  This is a simplified version of CTC.
+// If not specified, defaults to true
+func CTCLossV2CtcMergeRepeated(value bool) CTCLossV2Attr {
+	return func(m optionalAttr) {
+		m["ctc_merge_repeated"] = value
+	}
+}
+
+// CTCLossV2IgnoreLongerOutputsThanInputs sets the optional ignore_longer_outputs_than_inputs attribute to value.
+//
+// value: Scalar. If set to true, during CTC
+// calculation, items that have longer output sequences than input sequences
+// are skipped: they don't contribute to the loss term and have zero-gradient.
+// If not specified, defaults to false
+func CTCLossV2IgnoreLongerOutputsThanInputs(value bool) CTCLossV2Attr {
+	return func(m optionalAttr) {
+		m["ignore_longer_outputs_than_inputs"] = value
+	}
+}
+
+// Calculates the CTC Loss (log probability) for each batch entry.  Also calculates
+//
+// the gradient.  This class performs the softmax operation for you, so inputs
+// should be e.g. linear projections of outputs by an LSTM.
+//
+// Arguments:
+//	inputs: 3-D, shape: `(max_time x batch_size x num_classes)`, the logits. Default blank
+// label is 0 rather num_classes - 1.
+//	labels_indices: The indices of a `SparseTensor<int32, 2>`.
+// `labels_indices(i, :) == [b, t]` means `labels_values(i)` stores the id for
+// `(batch b, time t)`.
+//	labels_values: The values (labels) associated with the given batch and time.
+//	sequence_length: A vector containing sequence lengths (batch).
+//
+// Returns:
+//	loss: A vector (batch) containing log-probabilities.
+//	gradient: The gradient of `loss`.  3-D, shape:
+// `(max_time x batch_size x num_classes)`.
+func CTCLossV2(scope *Scope, inputs tf.Output, labels_indices tf.Output, labels_values tf.Output, sequence_length tf.Output, optional ...CTCLossV2Attr) (loss tf.Output, gradient tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CTCLossV2",
+		Input: []tf.Input{
+			inputs, labels_indices, labels_values, sequence_length,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// ResourceSparseApplyKerasMomentumAttr is an optional argument to ResourceSparseApplyKerasMomentum.
+type ResourceSparseApplyKerasMomentumAttr func(optionalAttr)
+
+// ResourceSparseApplyKerasMomentumUseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var and accum tensors will be protected
+// by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceSparseApplyKerasMomentumUseLocking(value bool) ResourceSparseApplyKerasMomentumAttr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// ResourceSparseApplyKerasMomentumUseNesterov sets the optional use_nesterov attribute to value.
+//
+// value: If `True`, the tensor passed to compute grad will be
+// var + momentum * accum, so in the end, the var you get is actually
+// var + momentum * accum.
+// If not specified, defaults to false
+func ResourceSparseApplyKerasMomentumUseNesterov(value bool) ResourceSparseApplyKerasMomentumAttr {
+	return func(m optionalAttr) {
+		m["use_nesterov"] = value
+	}
+}
+
+// Update relevant entries in '*var' and '*accum' according to the momentum scheme.
+//
+// Set use_nesterov = True if you want to use Nesterov momentum.
+//
+// That is for rows we have grad for, we update var and accum as follows:
+//
+// accum = accum * momentum - lr * grad
+// var += accum
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	accum: Should be from a Variable().
+//	lr: Learning rate. Must be a scalar.
+//	grad: The gradient.
+//	indices: A vector of indices into the first dimension of var and accum.
+//	momentum: Momentum. Must be a scalar.
+//
+// Returns the created operation.
+func ResourceSparseApplyKerasMomentum(scope *Scope, var_ tf.Output, accum tf.Output, lr tf.Output, grad tf.Output, indices tf.Output, momentum tf.Output, optional ...ResourceSparseApplyKerasMomentumAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceSparseApplyKerasMomentum",
+		Input: []tf.Input{
+			var_, accum, lr, grad, indices, momentum,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// MaxPoolGradGradV2Attr is an optional argument to MaxPoolGradGradV2.
+type MaxPoolGradGradV2Attr func(optionalAttr)
+
+// MaxPoolGradGradV2DataFormat sets the optional data_format attribute to value.
+//
+// value: Specify the data format of the input and output data. With the
+// default format "NHWC", the data is stored in the order of:
+//     [batch, in_height, in_width, in_channels].
+// Alternatively, the format could be "NCHW", the data storage order of:
+//     [batch, in_channels, in_height, in_width].
+// If not specified, defaults to "NHWC"
+func MaxPoolGradGradV2DataFormat(value string) MaxPoolGradGradV2Attr {
+	return func(m optionalAttr) {
+		m["data_format"] = value
+	}
+}
+
+// Computes second-order gradients of the maxpooling function.
+//
+// Arguments:
+//	orig_input: The original input tensor.
+//	orig_output: The original output tensor.
+//	grad: 4-D.  Gradients of gradients w.r.t. the input of `max_pool`.
+//	ksize: The size of the window for each dimension of the input tensor.
+//	strides: The stride of the sliding window for each dimension of the
+// input tensor.
+//	padding: The type of padding algorithm to use.
+//
+// Returns Gradients of gradients w.r.t. the input to `max_pool`.
+func MaxPoolGradGradV2(scope *Scope, orig_input tf.Output, orig_output tf.Output, grad tf.Output, ksize tf.Output, strides tf.Output, padding string, optional ...MaxPoolGradGradV2Attr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"padding": padding}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "MaxPoolGradGradV2",
+		Input: []tf.Input{
+			orig_input, orig_output, grad, ksize, strides,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RetrieveTPUEmbeddingMomentumParametersAttr is an optional argument to RetrieveTPUEmbeddingMomentumParameters.
+type RetrieveTPUEmbeddingMomentumParametersAttr func(optionalAttr)
+
+// RetrieveTPUEmbeddingMomentumParametersTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func RetrieveTPUEmbeddingMomentumParametersTableId(value int64) RetrieveTPUEmbeddingMomentumParametersAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingMomentumParametersTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingMomentumParametersTableName(value string) RetrieveTPUEmbeddingMomentumParametersAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingMomentumParametersConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingMomentumParametersConfig(value string) RetrieveTPUEmbeddingMomentumParametersAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Retrieve Momentum embedding parameters.
+//
+// An op that retrieves optimization parameters from embedding to host
+// memory. Must be preceded by a ConfigureTPUEmbeddingHost op that sets up
+// the correct embedding table configuration. For example, this op is
+// used to retrieve updated parameters before saving a checkpoint.
+//
+// Returns:
+//	parameters: Parameter parameters updated by the Momentum optimization algorithm.
+//	momenta: Parameter momenta updated by the Momentum optimization algorithm.
+func RetrieveTPUEmbeddingMomentumParameters(scope *Scope, num_shards int64, shard_id int64, optional ...RetrieveTPUEmbeddingMomentumParametersAttr) (parameters tf.Output, momenta tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RetrieveTPUEmbeddingMomentumParameters",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1)
+}
+
+// ConfigureDistributedTPUAttr is an optional argument to ConfigureDistributedTPU.
+type ConfigureDistributedTPUAttr func(optionalAttr)
+
+// ConfigureDistributedTPUEmbeddingConfig sets the optional embedding_config attribute to value.
+//
+// value: Reserved. Do not use.
+// If not specified, defaults to ""
+func ConfigureDistributedTPUEmbeddingConfig(value string) ConfigureDistributedTPUAttr {
+	return func(m optionalAttr) {
+		m["embedding_config"] = value
+	}
+}
+
+// ConfigureDistributedTPUTpuEmbeddingConfig sets the optional tpu_embedding_config attribute to value.
+//
+// value: Serialized tensorflow.tpu.TPUEmbeddingConfiguration that
+// describes the embedding lookups of the program.
+// If not specified, defaults to ""
+func ConfigureDistributedTPUTpuEmbeddingConfig(value string) ConfigureDistributedTPUAttr {
+	return func(m optionalAttr) {
+		m["tpu_embedding_config"] = value
+	}
+}
+
+// ConfigureDistributedTPUIsGlobalInit sets the optional is_global_init attribute to value.
+//
+// value: Reserved. Do not use.
+// If not specified, defaults to false
+func ConfigureDistributedTPUIsGlobalInit(value bool) ConfigureDistributedTPUAttr {
+	return func(m optionalAttr) {
+		m["is_global_init"] = value
+	}
+}
+
+// ConfigureDistributedTPUEnableWholeMeshCompilations sets the optional enable_whole_mesh_compilations attribute to value.
+// If not specified, defaults to false
+func ConfigureDistributedTPUEnableWholeMeshCompilations(value bool) ConfigureDistributedTPUAttr {
+	return func(m optionalAttr) {
+		m["enable_whole_mesh_compilations"] = value
+	}
+}
+
+// ConfigureDistributedTPUCompilationFailureClosesChips sets the optional compilation_failure_closes_chips attribute to value.
+// If not specified, defaults to true
+func ConfigureDistributedTPUCompilationFailureClosesChips(value bool) ConfigureDistributedTPUAttr {
+	return func(m optionalAttr) {
+		m["compilation_failure_closes_chips"] = value
+	}
+}
+
+// Sets up the centralized structures for a distributed TPU system.
+//
+// Returns A serialized tensorflow.tpu.TopologyProto that describes the TPU
+// topology.
+func ConfigureDistributedTPU(scope *Scope, optional ...ConfigureDistributedTPUAttr) (topology tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ConfigureDistributedTPU",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+//   Combines (nests of) input elements into a dataset of (nests of) windows.
+//
+//   A "window" is a finite dataset of flat elements of size `size` (or possibly
+//   fewer if there are not enough input elements to fill the window and
+//   `drop_remainder` evaluates to false).
+//
+//   The `shift` argument determines the number of input elements by which
+//   the window moves on each iteration.  The first element in the `k`th window
+//   will be element
+//
+//   ```
+//   1 + (k-1) * shift
+//   ```
+//
+//   of the input dataset. In particular, the first element of the first window
+//   will always be the first element of the input dataset.
+//
+//   If the `stride` parameter is greater than 1, then each window will skip
+//   `(stride - 1)` input elements between each element that appears in the
+//   window. Output windows will still contain `size` elements regardless of
+//   the value of `stride`.
+//
+//   The `stride` argument determines the stride of the input elements, and the
+//   `shift` argument determines the shift of the window.
+//
+//   For example, letting `{...}` to represent a Dataset:
+//
+//   - `tf.data.Dataset.range(7).window(2)` produces
+//     `{{0, 1}, {2, 3}, {4, 5}, {6}}`
+//   - `tf.data.Dataset.range(7).window(3, 2, 1, True)` produces
+//     `{{0, 1, 2}, {2, 3, 4}, {4, 5, 6}}`
+//   - `tf.data.Dataset.range(7).window(3, 1, 2, True)` produces
+//     `{{0, 2, 4}, {1, 3, 5}, {2, 4, 6}}`
+//
+//   Note that when the `window` transformation is applied to a dataset of
+//   nested elements, it produces a dataset of nested windows.
+//
+//   For example:
+//
+//   - `tf.data.Dataset.from_tensor_slices((range(4), range(4))).window(2)`
+//     produces `{({0, 1}, {0, 1}), ({2, 3}, {2, 3})}`
+//   - `tf.data.Dataset.from_tensor_slices({"a": range(4)}).window(2)`
+//     produces `{{"a": {0, 1}}, {"a": {2, 3}}}`
+//
+// Arguments:
+//
+//	size: An integer scalar, representing the number of elements
+// of the input dataset to combine into a window. Must be positive.
+//	shift: An integer scalar, representing the number of input elements
+// by which the window moves in each iteration.  Defaults to `size`.
+// Must be positive.
+//	stride: An integer scalar, representing the stride of the input elements
+// in the sliding window. Must be positive. The default value of 1 means
+// "retain every input element".
+//	drop_remainder: A Boolean scalar, representing whether the last window should be
+// dropped if its size is smaller than `window_size`.
+//
+//
+func WindowDataset(scope *Scope, input_dataset tf.Output, size tf.Output, shift tf.Output, stride tf.Output, drop_remainder tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "WindowDataset",
+		Input: []tf.Input{
+			input_dataset, size, shift, stride, drop_remainder,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// SetSizeAttr is an optional argument to SetSize.
+type SetSizeAttr func(optionalAttr)
+
+// SetSizeValidateIndices sets the optional validate_indices attribute to value.
+// If not specified, defaults to true
+func SetSizeValidateIndices(value bool) SetSizeAttr {
+	return func(m optionalAttr) {
+		m["validate_indices"] = value
+	}
+}
+
+// Number of unique elements along last dimension of input `set`.
+//
+// Input `set` is a `SparseTensor` represented by `set_indices`, `set_values`,
+// and `set_shape`. The last dimension contains values in a set, duplicates are
+// allowed but ignored.
+//
+// If `validate_indices` is `True`, this op validates the order and range of `set`
+// indices.
+//
+// Arguments:
+//	set_indices: 2D `Tensor`, indices of a `SparseTensor`.
+//	set_values: 1D `Tensor`, values of a `SparseTensor`.
+//	set_shape: 1D `Tensor`, shape of a `SparseTensor`.
+//
+// Returns For `set` ranked `n`, this is a `Tensor` with rank `n-1`, and the same 1st
+// `n-1` dimensions as `set`. Each value is the number of unique elements in
+// the corresponding `[0...n-1]` dimension of `set`.
+func SetSize(scope *Scope, set_indices tf.Output, set_values tf.Output, set_shape tf.Output, optional ...SetSizeAttr) (size tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SetSize",
+		Input: []tf.Input{
+			set_indices, set_values, set_shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// AutoShardDatasetAttr is an optional argument to AutoShardDataset.
+type AutoShardDatasetAttr func(optionalAttr)
+
+// AutoShardDatasetAutoShardPolicy sets the optional auto_shard_policy attribute to value.
+// If not specified, defaults to 0
+func AutoShardDatasetAutoShardPolicy(value int64) AutoShardDatasetAttr {
+	return func(m optionalAttr) {
+		m["auto_shard_policy"] = value
+	}
+}
+
+// Creates a dataset that shards the input dataset.
+//
+// Creates a dataset that shards the input dataset by num_workers, returning a
+// sharded dataset for the index-th worker. This attempts to automatically shard
+// a dataset by examining the Dataset graph and inserting a shard op before the
+// inputs to a reader Dataset (e.g. CSVDataset, TFRecordDataset).
+//
+// This dataset will throw a NotFound error if we cannot shard the dataset
+// automatically.
+//
+// Arguments:
+//	input_dataset: A variant tensor representing the input dataset.
+//	num_workers: A scalar representing the number of workers to distribute this dataset across.
+//	index: A scalar representing the index of the current worker out of num_workers.
+//
+//
+func AutoShardDataset(scope *Scope, input_dataset tf.Output, num_workers tf.Output, index tf.Output, output_types []tf.DataType, output_shapes []tf.Shape, optional ...AutoShardDatasetAttr) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "AutoShardDataset",
+		Input: []tf.Input{
+			input_dataset, num_workers, index,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// InfeedEnqueuePrelinearizedBufferAttr is an optional argument to InfeedEnqueuePrelinearizedBuffer.
+type InfeedEnqueuePrelinearizedBufferAttr func(optionalAttr)
+
+// InfeedEnqueuePrelinearizedBufferDeviceOrdinal sets the optional device_ordinal attribute to value.
+//
+// value: The TPU device to use. This should be -1 when the Op is running on a TPU device
+// and = 0 when the Op is running on the CPU device.
+// If not specified, defaults to -1
+func InfeedEnqueuePrelinearizedBufferDeviceOrdinal(value int64) InfeedEnqueuePrelinearizedBufferAttr {
+	return func(m optionalAttr) {
+		m["device_ordinal"] = value
+	}
+}
+
+// An op which enqueues prelinearized buffer into TPU infeed.
+//
+// Arguments:
+//	input: A variant tensor representing linearized output.
+//
+// Returns the created operation.
+func InfeedEnqueuePrelinearizedBuffer(scope *Scope, input tf.Output, optional ...InfeedEnqueuePrelinearizedBufferAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "InfeedEnqueuePrelinearizedBuffer",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// RetrieveTPUEmbeddingFTRLParametersAttr is an optional argument to RetrieveTPUEmbeddingFTRLParameters.
+type RetrieveTPUEmbeddingFTRLParametersAttr func(optionalAttr)
+
+// RetrieveTPUEmbeddingFTRLParametersTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func RetrieveTPUEmbeddingFTRLParametersTableId(value int64) RetrieveTPUEmbeddingFTRLParametersAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingFTRLParametersTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingFTRLParametersTableName(value string) RetrieveTPUEmbeddingFTRLParametersAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingFTRLParametersConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingFTRLParametersConfig(value string) RetrieveTPUEmbeddingFTRLParametersAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Retrieve FTRL embedding parameters.
+//
+// An op that retrieves optimization parameters from embedding to host
+// memory. Must be preceded by a ConfigureTPUEmbeddingHost op that sets up
+// the correct embedding table configuration. For example, this op is
+// used to retrieve updated parameters before saving a checkpoint.
+//
+// Returns:
+//	parameters: Parameter parameters updated by the FTRL optimization algorithm.
+//	accumulators: Parameter accumulators updated by the FTRL optimization algorithm.
+//	linears: Parameter linears updated by the FTRL optimization algorithm.
+func RetrieveTPUEmbeddingFTRLParameters(scope *Scope, num_shards int64, shard_id int64, optional ...RetrieveTPUEmbeddingFTRLParametersAttr) (parameters tf.Output, accumulators tf.Output, linears tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RetrieveTPUEmbeddingFTRLParameters",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2)
+}
+
+// Returns the result of a TPU compilation.
+//
+// This operation returns the result of a TPU compilation as a serialized
+// CompilationResultProto, which holds a status and an error message if an error
+// occurred during compilation.
+func TPUCompilationResult(scope *Scope) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "TPUCompilationResult",
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// ResourceApplyAdagradV2Attr is an optional argument to ResourceApplyAdagradV2.
+type ResourceApplyAdagradV2Attr func(optionalAttr)
+
+// ResourceApplyAdagradV2UseLocking sets the optional use_locking attribute to value.
+//
+// value: If `True`, updating of the var and accum tensors will be protected
+// by a lock; otherwise the behavior is undefined, but may exhibit less
+// contention.
+// If not specified, defaults to false
+func ResourceApplyAdagradV2UseLocking(value bool) ResourceApplyAdagradV2Attr {
+	return func(m optionalAttr) {
+		m["use_locking"] = value
+	}
+}
+
+// ResourceApplyAdagradV2UpdateSlots sets the optional update_slots attribute to value.
+// If not specified, defaults to true
+func ResourceApplyAdagradV2UpdateSlots(value bool) ResourceApplyAdagradV2Attr {
+	return func(m optionalAttr) {
+		m["update_slots"] = value
+	}
+}
+
+// Update '*var' according to the adagrad scheme.
+//
+// accum += grad * grad
+// var -= lr * grad * (1 / (sqrt(accum) + epsilon))
+//
+// Arguments:
+//	var_: Should be from a Variable().
+//	accum: Should be from a Variable().
+//	lr: Scaling factor. Must be a scalar.
+//	epsilon: Constant factor. Must be a scalar.
+//	grad: The gradient.
+//
+// Returns the created operation.
+func ResourceApplyAdagradV2(scope *Scope, var_ tf.Output, accum tf.Output, lr tf.Output, epsilon tf.Output, grad tf.Output, optional ...ResourceApplyAdagradV2Attr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceApplyAdagradV2",
+		Input: []tf.Input{
+			var_, accum, lr, epsilon, grad,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Shuts down a running distributed TPU system.
+//
+// The op returns an error if no system is running.
+//
+// Returns the created operation.
+func ShutdownDistributedTPU(scope *Scope) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ShutdownDistributedTPU",
+	}
+	return scope.AddOperation(opspec)
+}
+
+// SerializeManySparseAttr is an optional argument to SerializeManySparse.
+type SerializeManySparseAttr func(optionalAttr)
+
+// SerializeManySparseOutType sets the optional out_type attribute to value.
+//
+// value: The `dtype` to use for serialization; the supported types are `string`
+// (default) and `variant`.
+// If not specified, defaults to DT_STRING
+func SerializeManySparseOutType(value tf.DataType) SerializeManySparseAttr {
+	return func(m optionalAttr) {
+		m["out_type"] = value
+	}
+}
+
+// Serialize an `N`-minibatch `SparseTensor` into an `[N, 3]` `Tensor` object.
+//
+// The `SparseTensor` must have rank `R` greater than 1, and the first dimension
+// is treated as the minibatch dimension.  Elements of the `SparseTensor`
+// must be sorted in increasing order of this first dimension.  The serialized
+// `SparseTensor` objects going into each row of `serialized_sparse` will have
+// rank `R-1`.
+//
+// The minibatch size `N` is extracted from `sparse_shape[0]`.
+//
+// Arguments:
+//	sparse_indices: 2-D.  The `indices` of the minibatch `SparseTensor`.
+//	sparse_values: 1-D.  The `values` of the minibatch `SparseTensor`.
+//	sparse_shape: 1-D.  The `shape` of the minibatch `SparseTensor`.
+func SerializeManySparse(scope *Scope, sparse_indices tf.Output, sparse_values tf.Output, sparse_shape tf.Output, optional ...SerializeManySparseAttr) (serialized_sparse tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "SerializeManySparse",
+		Input: []tf.Input{
+			sparse_indices, sparse_values, sparse_shape,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Says whether the targets are in the top `K` predictions.
+//
+// This outputs a `batch_size` bool array, an entry `out[i]` is `true` if the
+// prediction for the target class is among the top `k` predictions among
+// all predictions for example `i`. Note that the behavior of `InTopK` differs
+// from the `TopK` op in its handling of ties; if multiple classes have the
+// same prediction value and straddle the top-`k` boundary, all of those
+// classes are considered to be in the top `k`.
+//
+// More formally, let
+//
+//   \\(predictions_i\\) be the predictions for all classes for example `i`,
+//   \\(targets_i\\) be the target class for example `i`,
+//   \\(out_i\\) be the output for example `i`,
+//
+// $$out_i = predictions_{i, targets_i} \in TopKIncludingTies(predictions_i)$$
+//
+// Arguments:
+//	predictions: A `batch_size` x `classes` tensor.
+//	targets: A `batch_size` vector of class ids.
+//	k: Number of top elements to look at for computing precision.
+//
+// Returns Computed precision at `k` as a `bool Tensor`.
+func InTopKV2(scope *Scope, predictions tf.Output, targets tf.Output, k tf.Output) (precision tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "InTopKV2",
+		Input: []tf.Input{
+			predictions, targets, k,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Creates an Optional variant with no value.
+func OptionalNone(scope *Scope) (optional tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "OptionalNone",
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// RetrieveTPUEmbeddingStochasticGradientDescentParametersAttr is an optional argument to RetrieveTPUEmbeddingStochasticGradientDescentParameters.
+type RetrieveTPUEmbeddingStochasticGradientDescentParametersAttr func(optionalAttr)
+
+// RetrieveTPUEmbeddingStochasticGradientDescentParametersTableId sets the optional table_id attribute to value.
+// If not specified, defaults to -1
+func RetrieveTPUEmbeddingStochasticGradientDescentParametersTableId(value int64) RetrieveTPUEmbeddingStochasticGradientDescentParametersAttr {
+	return func(m optionalAttr) {
+		m["table_id"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingStochasticGradientDescentParametersTableName sets the optional table_name attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingStochasticGradientDescentParametersTableName(value string) RetrieveTPUEmbeddingStochasticGradientDescentParametersAttr {
+	return func(m optionalAttr) {
+		m["table_name"] = value
+	}
+}
+
+// RetrieveTPUEmbeddingStochasticGradientDescentParametersConfig sets the optional config attribute to value.
+// If not specified, defaults to ""
+func RetrieveTPUEmbeddingStochasticGradientDescentParametersConfig(value string) RetrieveTPUEmbeddingStochasticGradientDescentParametersAttr {
+	return func(m optionalAttr) {
+		m["config"] = value
+	}
+}
+
+// Retrieve SGD embedding parameters.
+//
+// An op that retrieves optimization parameters from embedding to host
+// memory. Must be preceded by a ConfigureTPUEmbeddingHost op that sets up
+// the correct embedding table configuration. For example, this op is
+// used to retrieve updated parameters before saving a checkpoint.
+//
+// Returns Parameter parameters updated by the stochastic gradient descent optimization algorithm.
+func RetrieveTPUEmbeddingStochasticGradientDescentParameters(scope *Scope, num_shards int64, shard_id int64, optional ...RetrieveTPUEmbeddingStochasticGradientDescentParametersAttr) (parameters tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_shards": num_shards, "shard_id": shard_id}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "RetrieveTPUEmbeddingStochasticGradientDescentParameters",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// CudnnRNNAttr is an optional argument to CudnnRNN.
+type CudnnRNNAttr func(optionalAttr)
+
+// CudnnRNNRnnMode sets the optional rnn_mode attribute to value.
+// If not specified, defaults to "lstm"
+func CudnnRNNRnnMode(value string) CudnnRNNAttr {
+	return func(m optionalAttr) {
+		m["rnn_mode"] = value
+	}
+}
+
+// CudnnRNNInputMode sets the optional input_mode attribute to value.
+// If not specified, defaults to "linear_input"
+func CudnnRNNInputMode(value string) CudnnRNNAttr {
+	return func(m optionalAttr) {
+		m["input_mode"] = value
+	}
+}
+
+// CudnnRNNDirection sets the optional direction attribute to value.
+// If not specified, defaults to "unidirectional"
+func CudnnRNNDirection(value string) CudnnRNNAttr {
+	return func(m optionalAttr) {
+		m["direction"] = value
+	}
+}
+
+// CudnnRNNDropout sets the optional dropout attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNDropout(value float32) CudnnRNNAttr {
+	return func(m optionalAttr) {
+		m["dropout"] = value
+	}
+}
+
+// CudnnRNNSeed sets the optional seed attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNSeed(value int64) CudnnRNNAttr {
+	return func(m optionalAttr) {
+		m["seed"] = value
+	}
+}
+
+// CudnnRNNSeed2 sets the optional seed2 attribute to value.
+// If not specified, defaults to 0
+func CudnnRNNSeed2(value int64) CudnnRNNAttr {
+	return func(m optionalAttr) {
+		m["seed2"] = value
+	}
+}
+
+// CudnnRNNIsTraining sets the optional is_training attribute to value.
+// If not specified, defaults to true
+func CudnnRNNIsTraining(value bool) CudnnRNNAttr {
+	return func(m optionalAttr) {
+		m["is_training"] = value
+	}
+}
+
+// A RNN backed by cuDNN.
+//
+// Computes the RNN from the input and initial states, with respect to the params
+// buffer.
+//
+// rnn_mode: Indicates the type of the RNN model.
+// input_mode: Indicate whether there is a linear projection between the input and
+//   the actual computation before the first layer. 'skip_input' is only allowed
+//   when input_size == num_units; 'auto_select' implies 'skip_input' when
+//   input_size == num_units; otherwise, it implies 'linear_input'.
+// direction: Indicates whether a bidirectional model will be used. Should be
+//   "unidirectional" or "bidirectional".
+// dropout: Dropout probability. When set to 0., dropout is disabled.
+// seed: The 1st part of a seed to initialize dropout.
+// seed2: The 2nd part of a seed to initialize dropout.
+// input: A 3-D tensor with the shape of [seq_length, batch_size, input_size].
+// input_h: A 3-D tensor with the shape of [num_layer * dir, batch_size,
+//     num_units].
+// input_c: For LSTM, a 3-D tensor with the shape of
+//     [num_layer * dir, batch, num_units]. For other models, it is ignored.
+// params: A 1-D tensor that contains the weights and biases in an opaque layout.
+//     The size must be created through CudnnRNNParamsSize, and initialized
+//     separately. Note that they might not be compatible across different
+//     generations. So it is a good idea to save and restore
+// output: A 3-D tensor with the shape of [seq_length, batch_size,
+//     dir * num_units].
+// output_h: The same shape has input_h.
+// output_c: The same shape as input_c for LSTM. An empty tensor for other models.
+// is_training: Indicates whether this operation is used for inference or
+//   training.
+// reserve_space: An opaque tensor that can be used in backprop calculation. It
+//   is only produced if is_training is false.
+func CudnnRNN(scope *Scope, input tf.Output, input_h tf.Output, input_c tf.Output, params tf.Output, optional ...CudnnRNNAttr) (output tf.Output, output_h tf.Output, output_c tf.Output, reserve_space tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "CudnnRNN",
+		Input: []tf.Input{
+			input, input_h, input_c, params,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0), op.Output(1), op.Output(2), op.Output(3)
+}
+
+// Creates a dataset that batches `batch_size` elements from `input_dataset`.
+//
+// Arguments:
+//
+//	batch_size: A scalar representing the number of elements to accumulate in a
+// batch.
+//
+//
+func BatchDataset(scope *Scope, input_dataset tf.Output, batch_size tf.Output, output_types []tf.DataType, output_shapes []tf.Shape) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"output_types": output_types, "output_shapes": output_shapes}
+	opspec := tf.OpSpec{
+		Type: "BatchDataset",
+		Input: []tf.Input{
+			input_dataset, batch_size,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// EnqueueTPUEmbeddingSparseTensorBatchAttr is an optional argument to EnqueueTPUEmbeddingSparseTensorBatch.
+type EnqueueTPUEmbeddingSparseTensorBatchAttr func(optionalAttr)
+
+// EnqueueTPUEmbeddingSparseTensorBatchDeviceOrdinal sets the optional device_ordinal attribute to value.
+//
+// value: The TPU device to use. Should be >= 0 and less than the number
+// of TPU cores in the task on which the node is placed.
+// If not specified, defaults to -1
+func EnqueueTPUEmbeddingSparseTensorBatchDeviceOrdinal(value int64) EnqueueTPUEmbeddingSparseTensorBatchAttr {
+	return func(m optionalAttr) {
+		m["device_ordinal"] = value
+	}
+}
+
+// EnqueueTPUEmbeddingSparseTensorBatchCombiners sets the optional combiners attribute to value.
+//
+// value: A list of string scalars, one for each embedding table that specify
+// how to normalize the embedding activations after weighted summation.
+// Supported combiners are 'mean', 'sum', or 'sqrtn'. It is invalid to have
+// the sum of the weights be 0 for 'mean' or the sum of the squared weights be
+// 0 for 'sqrtn'. If combiners isn't passed, the default is to use 'sum' for
+// all tables.
+// If not specified, defaults to <>
+func EnqueueTPUEmbeddingSparseTensorBatchCombiners(value []string) EnqueueTPUEmbeddingSparseTensorBatchAttr {
+	return func(m optionalAttr) {
+		m["combiners"] = value
+	}
+}
+
+// EnqueueTPUEmbeddingSparseTensorBatchMaxSequenceLengths sets the optional max_sequence_lengths attribute to value.
+// If not specified, defaults to <>
+func EnqueueTPUEmbeddingSparseTensorBatchMaxSequenceLengths(value []int64) EnqueueTPUEmbeddingSparseTensorBatchAttr {
+	return func(m optionalAttr) {
+		m["max_sequence_lengths"] = value
+	}
+}
+
+// Eases the porting of code that uses tf.nn.embedding_lookup_sparse().
+//
+// sample_indices[i], embedding_indices[i] and aggregation_weights[i] correspond
+// to the ith feature. table_ids[i] indicates which embedding table to look up ith
+// feature.
+//
+// The tensors at corresponding positions in the three input lists (sample_indices,
+// embedding_indices and aggregation_weights) must have the same shape, i.e. rank 1
+// with dim_size() equal to the total number of lookups into the table described by
+// the corresponding feature.
+//
+// Arguments:
+//	sample_indices: A list of rank 1 Tensors specifying the training example to
+// which the corresponding embedding_indices and aggregation_weights values
+// belong. It corresponds to sp_ids.indices[:,0] in  embedding_lookup_sparse().
+//	embedding_indices: A list of rank 1 Tensors, indices into the embedding tables.
+// It corresponds to sp_ids.values in embedding_lookup_sparse().
+//	aggregation_weights: A list of rank 1 Tensors containing per training example
+// aggregation weights. It corresponds to sp_weights.values in
+// embedding_lookup_sparse().
+//	mode_override: A string input that overrides the mode specified in the
+// TPUEmbeddingConfiguration. Supported values are {'unspecified', 'inference',
+// 'training', 'backward_pass_only'}. When set to 'unspecified', the mode set
+// in TPUEmbeddingConfiguration is used, otherwise mode_override is used.
+//	table_ids: A list of integers specifying the identifier of the embedding table
+// (offset of TableDescriptor in the TPUEmbeddingConfiguration) to lookup the
+// corresponding input. The ith input is looked up using table_ids[i]. The size
+// of the table_ids list must be equal to that of sample_indices,
+// embedding_indices and aggregation_weights.
+//
+// Returns the created operation.
+func EnqueueTPUEmbeddingSparseTensorBatch(scope *Scope, sample_indices []tf.Output, embedding_indices []tf.Output, aggregation_weights []tf.Output, mode_override tf.Output, table_ids []int64, optional ...EnqueueTPUEmbeddingSparseTensorBatchAttr) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"table_ids": table_ids}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "EnqueueTPUEmbeddingSparseTensorBatch",
+		Input: []tf.Input{
+			tf.OutputList(sample_indices), tf.OutputList(embedding_indices), tf.OutputList(aggregation_weights), mode_override,
+		},
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// ReverseSequenceAttr is an optional argument to ReverseSequence.
+type ReverseSequenceAttr func(optionalAttr)
+
+// ReverseSequenceBatchDim sets the optional batch_dim attribute to value.
+//
+// value: The dimension along which reversal is performed.
+// If not specified, defaults to 0
+func ReverseSequenceBatchDim(value int64) ReverseSequenceAttr {
+	return func(m optionalAttr) {
+		m["batch_dim"] = value
+	}
+}
+
+// Reverses variable length slices.
+//
+// This op first slices `input` along the dimension `batch_dim`, and for each
+// slice `i`, reverses the first `seq_lengths[i]` elements along
+// the dimension `seq_dim`.
+//
+// The elements of `seq_lengths` must obey `seq_lengths[i] <= input.dims[seq_dim]`,
+// and `seq_lengths` must be a vector of length `input.dims[batch_dim]`.
+//
+// The output slice `i` along dimension `batch_dim` is then given by input
+// slice `i`, with the first `seq_lengths[i]` slices along dimension
+// `seq_dim` reversed.
+//
+// For example:
+//
+// ```
+// # Given this:
+// batch_dim = 0
+// seq_dim = 1
+// input.dims = (4, 8, ...)
+// seq_lengths = [7, 2, 3, 5]
+//
+// # then slices of input are reversed on seq_dim, but only up to seq_lengths:
+// output[0, 0:7, :, ...] = input[0, 7:0:-1, :, ...]
+// output[1, 0:2, :, ...] = input[1, 2:0:-1, :, ...]
+// output[2, 0:3, :, ...] = input[2, 3:0:-1, :, ...]
+// output[3, 0:5, :, ...] = input[3, 5:0:-1, :, ...]
+//
+// # while entries past seq_lens are copied through:
+// output[0, 7:, :, ...] = input[0, 7:, :, ...]
+// output[1, 2:, :, ...] = input[1, 2:, :, ...]
+// output[2, 3:, :, ...] = input[2, 3:, :, ...]
+// output[3, 2:, :, ...] = input[3, 2:, :, ...]
+// ```
+//
+// In contrast, if:
+//
+// ```
+// # Given this:
+// batch_dim = 2
+// seq_dim = 0
+// input.dims = (8, ?, 4, ...)
+// seq_lengths = [7, 2, 3, 5]
+//
+// # then slices of input are reversed on seq_dim, but only up to seq_lengths:
+// output[0:7, :, 0, :, ...] = input[7:0:-1, :, 0, :, ...]
+// output[0:2, :, 1, :, ...] = input[2:0:-1, :, 1, :, ...]
+// output[0:3, :, 2, :, ...] = input[3:0:-1, :, 2, :, ...]
+// output[0:5, :, 3, :, ...] = input[5:0:-1, :, 3, :, ...]
+//
+// # while entries past seq_lens are copied through:
+// output[7:, :, 0, :, ...] = input[7:, :, 0, :, ...]
+// output[2:, :, 1, :, ...] = input[2:, :, 1, :, ...]
+// output[3:, :, 2, :, ...] = input[3:, :, 2, :, ...]
+// output[2:, :, 3, :, ...] = input[2:, :, 3, :, ...]
+// ```
+//
+// Arguments:
+//	input: The input to reverse.
+//	seq_lengths: 1-D with length `input.dims(batch_dim)` and
+// `max(seq_lengths) <= input.dims(seq_dim)`
+//	seq_dim: The dimension which is partially reversed.
+//
+// Returns The partially reversed input. It has the same shape as `input`.
+func ReverseSequence(scope *Scope, input tf.Output, seq_lengths tf.Output, seq_dim int64, optional ...ReverseSequenceAttr) (output tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"seq_dim": seq_dim}
+	for _, a := range optional {
+		a(attrs)
+	}
+	opspec := tf.OpSpec{
+		Type: "ReverseSequence",
+		Input: []tf.Input{
+			input, seq_lengths,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Fetches multiple values from infeed as an XLA tuple.
+//
+// Arguments:
+//	dtypes: The element types of each element in `outputs`.
+//	shapes: The shapes of each tensor in `outputs`.
+//
+// Returns A list of tensors that will be provided using the infeed mechanism.
+func InfeedDequeueTuple(scope *Scope, dtypes []tf.DataType, shapes []tf.Shape) (outputs []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"dtypes": dtypes, "shapes": shapes}
+	opspec := tf.OpSpec{
+		Type: "InfeedDequeueTuple",
+
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if outputs, idx, err = makeOutputList(op, idx, "outputs"); err != nil {
+		scope.UpdateErr("InfeedDequeueTuple", err)
+		return
+	}
+	return outputs
+}
+
+// Creates and returns an empty tensor list.
+//
+// All list elements must be tensors of dtype element_dtype and shape compatible
+// with element_shape.
+//
+// handle: an empty tensor list.
+// element_dtype: the type of elements in the list.
+// element_shape: a shape compatible with that of elements in the list.
+func EmptyTensorList(scope *Scope, element_shape tf.Output, max_num_elements tf.Output, element_dtype tf.DataType) (handle tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"element_dtype": element_dtype}
+	opspec := tf.OpSpec{
+		Type: "EmptyTensorList",
+		Input: []tf.Input{
+			element_shape, max_num_elements,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Sets up TPUEmbedding in a distributed TPU system.
+//
+// Arguments:
+//	config: Serialized tensorflow.tpu.TPUEmbeddingConfiguration that
+// describes the embedding lookups of the program.
+//
+// Returns the created operation.
+func ConfigureTPUEmbedding(scope *Scope, config string) (o *tf.Operation) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"config": config}
+	opspec := tf.OpSpec{
+		Type: "ConfigureTPUEmbedding",
+
+		Attrs: attrs,
+	}
+	return scope.AddOperation(opspec)
+}
+
+// Returns the number of gradients aggregated in the given accumulators.
+//
+// Arguments:
+//	handle: The handle to an accumulator.
+//
+// Returns The number of gradients aggregated in the given accumulator.
+func ResourceAccumulatorNumAccumulated(scope *Scope, handle tf.Output) (num_accumulated tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	opspec := tf.OpSpec{
+		Type: "ResourceAccumulatorNumAccumulated",
+		Input: []tf.Input{
+			handle,
+		},
+	}
+	op := scope.AddOperation(opspec)
+	return op.Output(0)
+}
+
+// Connects N outputs from an N-way replicated TPU computation.
+//
+// This operation holds a replicated output from a `tpu.replicate()` computation subgraph.
+// Each replicated output has the same shape and type alongside the input.
+//
+// For example:
+// ```
+// %computation = "tf.Computation"()
+// %replicated_output:2 = "tf.TPUReplicatedOutput"(%computation)
+// ```
+// The above computation has a replicated output of two replicas.
+func TPUReplicatedOutput(scope *Scope, input tf.Output, num_replicas int64) (outputs []tf.Output) {
+	if scope.Err() != nil {
+		return
+	}
+	attrs := map[string]interface{}{"num_replicas": num_replicas}
+	opspec := tf.OpSpec{
+		Type: "TPUReplicatedOutput",
+		Input: []tf.Input{
+			input,
+		},
+		Attrs: attrs,
+	}
+	op := scope.AddOperation(opspec)
+	if scope.Err() != nil {
+		return
+	}
+	var idx int
+	var err error
+	if outputs, idx, err = makeOutputList(op, idx, "outputs"); err != nil {
+		scope.UpdateErr("TPUReplicatedOutput", err)
+		return
+	}
+	return outputs
+}
diff --git a/tensorflow/python/BUILD b/tensorflow/python/BUILD
index 682d15712b7..e71900b430f 100644
--- a/tensorflow/python/BUILD
+++ b/tensorflow/python/BUILD
@@ -32,7 +32,7 @@ load("//tensorflow:tensorflow.bzl", "tf_py_test")
 # buildifier: disable=same-origin-load
 load("//tensorflow:tensorflow.bzl", "tf_py_build_info_genrule")
 load("//tensorflow/core/platform:build_config.bzl", "pyx_library", "tf_additional_all_protos", "tf_additional_lib_deps", "tf_proto_library", "tf_proto_library_py", "tf_protos_grappler")  # @unused
-load("//tensorflow/core/platform:build_config_root.bzl", "if_static", "tf_additional_plugin_deps", "tf_additional_xla_deps_py")
+load("//tensorflow/core/platform:build_config_root.bzl", "if_static", "tf_additional_plugin_deps", "tf_additional_profiler_deps", "tf_additional_xla_deps_py")
 load("//tensorflow/python:build_defs.bzl", "tf_gen_op_wrapper_private_py")
 load(
     "//third_party/ngraph:build_defs.bzl",
@@ -223,6 +223,7 @@ py_library(
         "//tensorflow/python/ops/linalg",
         "//tensorflow/python/ops/linalg/sparse",
         "//tensorflow/python/ops/losses",
+        "//tensorflow/python/ops/numpy_ops:numpy",
         "//tensorflow/python/ops/parallel_for",
         "//tensorflow/python/ops/ragged",
         "//tensorflow/python/ops/signal",
@@ -284,6 +285,7 @@ py_library(
     deps = [
         ":_pywrap_util_port",
         ":lib",
+        ":platform_build_info",
         ":pywrap_tfe",
         ":util",
         "//tensorflow/core:protos_all_py",
@@ -348,6 +350,24 @@ tf_py_test(
     ],
 )
 
+tf_py_test(
+    name = "sysconfig_test",
+    size = "small",
+    srcs = ["platform/sysconfig_test.py"],
+    data = [
+        "platform/sysconfig.py",
+    ],
+    python_version = "PY3",
+    tags = [
+        "no_pip",
+        "no_windows",
+    ],
+    deps = [
+        ":platform",
+        ":platform_test",
+    ],
+)
+
 tf_py_test(
     name = "flags_test",
     size = "small",
@@ -6012,7 +6032,8 @@ pywrap_tensorflow_macro(
         "//tensorflow/core/util/tensor_bundle",
         "//tensorflow/compiler/mlir/python:mlir",
     ] + (tf_additional_lib_deps() +
-         tf_additional_plugin_deps()) + if_ngraph([
+         tf_additional_plugin_deps() +
+         tf_additional_profiler_deps()) + if_ngraph([
         "@ngraph_tf//:ngraph_tf",
     ]) + if_xla_available([
         "//tensorflow/compiler/aot:tfcompile_lib",
diff --git a/tensorflow/python/autograph/pyct/inspect_utils_test.py b/tensorflow/python/autograph/pyct/inspect_utils_test.py
index 25c0e161b32..890f9e31f44 100644
--- a/tensorflow/python/autograph/pyct/inspect_utils_test.py
+++ b/tensorflow/python/autograph/pyct/inspect_utils_test.py
@@ -586,20 +586,26 @@ class InspectUtilsTest(test.TestCase):
     self.assertTrue(inspect_utils.isconstructor(AbcSubclass))
 
   def test_getfutureimports_functions(self):
-    self.assertEqual(
-        inspect_utils.getfutureimports(basic_definitions.function_with_print),
-        ('absolute_import', 'division', 'print_function', 'with_statement'))
+    imps = inspect_utils.getfutureimports(basic_definitions.function_with_print)
+    self.assertIn('absolute_import', imps)
+    self.assertIn('division', imps)
+    self.assertIn('print_function', imps)
+    self.assertNotIn('generators', imps)
 
   def test_getfutureimports_lambdas(self):
-    self.assertEqual(
-        inspect_utils.getfutureimports(basic_definitions.simple_lambda),
-        ('absolute_import', 'division', 'print_function', 'with_statement'))
+    imps = inspect_utils.getfutureimports(basic_definitions.simple_lambda)
+    self.assertIn('absolute_import', imps)
+    self.assertIn('division', imps)
+    self.assertIn('print_function', imps)
+    self.assertNotIn('generators', imps)
 
   def test_getfutureimports_methods(self):
-    self.assertEqual(
-        inspect_utils.getfutureimports(
-            basic_definitions.SimpleClass.method_with_print),
-        ('absolute_import', 'division', 'print_function', 'with_statement'))
+    imps = inspect_utils.getfutureimports(
+        basic_definitions.SimpleClass.method_with_print)
+    self.assertIn('absolute_import', imps)
+    self.assertIn('division', imps)
+    self.assertIn('print_function', imps)
+    self.assertNotIn('generators', imps)
 
 
 if __name__ == '__main__':
diff --git a/tensorflow/python/autograph/pyct/origin_info_test.py b/tensorflow/python/autograph/pyct/origin_info_test.py
index 823dacfe2ed..a806b87a74d 100644
--- a/tensorflow/python/autograph/pyct/origin_info_test.py
+++ b/tensorflow/python/autograph/pyct/origin_info_test.py
@@ -18,6 +18,7 @@ from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function
 
+import inspect
 import sys
 import textwrap
 
@@ -67,10 +68,11 @@ class OriginInfoTest(test.TestCase):
     module_path = tf_inspect.getsourcefile(test_fn)
 
     # Origin line numbers below should match those in basic_definitions.py
+    fn_start = inspect.getsourcelines(test_fn)[1]
 
     definition_loc = origin_info.LineLocation('test_filename', 1)
     self.assertIn(definition_loc, source_map)
-    self.assertEqual(source_map[definition_loc].loc.lineno, 23)
+    self.assertEqual(source_map[definition_loc].loc.lineno, fn_start)
     self.assertEqual(source_map[definition_loc].loc.filename, module_path)
     self.assertEqual(source_map[definition_loc].function_name,
                      'simple_function')
@@ -81,10 +83,11 @@ class OriginInfoTest(test.TestCase):
     module_path = tf_inspect.getsourcefile(test_fn)
 
     # Origin line numbers below should match those in basic_definitions.py
+    fn_start = inspect.getsourcelines(test_fn)[1]
 
     call_loc = origin_info.LineLocation('test_filename', 3)
     self.assertIn(call_loc, source_map)
-    self.assertEqual(source_map[call_loc].loc.lineno, 55)
+    self.assertEqual(source_map[call_loc].loc.lineno, fn_start + 2)
     self.assertEqual(source_map[call_loc].loc.filename, module_path)
     self.assertEqual(source_map[call_loc].function_name,
                      'function_with_multiline_call')
@@ -92,7 +95,7 @@ class OriginInfoTest(test.TestCase):
 
     second_arg_loc = origin_info.LineLocation('test_filename', 5)
     self.assertIn(second_arg_loc, source_map)
-    self.assertEqual(source_map[second_arg_loc].loc.lineno, 57)
+    self.assertEqual(source_map[second_arg_loc].loc.lineno, fn_start + 4)
     self.assertEqual(source_map[second_arg_loc].loc.filename, module_path)
     self.assertEqual(source_map[second_arg_loc].function_name,
                      'function_with_multiline_call')
@@ -150,42 +153,43 @@ class OriginInfoTest(test.TestCase):
     origin_info.resolve_entity(node, source, test_fn)
 
     # The line numbers below should match those in basic_definitions.py
+    fn_start = inspect.getsourcelines(test_fn)[1]
 
     def_origin = anno.getanno(node, anno.Basic.ORIGIN)
-    self.assertEqual(def_origin.loc.lineno, 23)
+    self.assertEqual(def_origin.loc.lineno, fn_start)
     self.assertEqual(def_origin.loc.col_offset, 0)
     self.assertEqual(def_origin.source_code_line, 'def simple_function(x):')
     self.assertIsNone(def_origin.comment)
 
     docstring_origin = anno.getanno(node.body[0], anno.Basic.ORIGIN)
-    self.assertEqual(docstring_origin.loc.lineno, 24)
+    self.assertEqual(docstring_origin.loc.lineno, fn_start + 1)
     self.assertEqual(docstring_origin.loc.col_offset, 2)
     self.assertEqual(docstring_origin.source_code_line, '  """Docstring."""')
     self.assertIsNone(docstring_origin.comment)
 
     ret_origin = anno.getanno(node.body[1], anno.Basic.ORIGIN)
-    self.assertEqual(ret_origin.loc.lineno, 25)
+    self.assertEqual(ret_origin.loc.lineno, fn_start + 2)
     self.assertEqual(ret_origin.loc.col_offset, 2)
     self.assertEqual(ret_origin.source_code_line, '  return x  # comment')
     self.assertEqual(ret_origin.comment, 'comment')
 
   def test_resolve_entity_nested_function(self):
-
     test_fn = basic_definitions.nested_functions
     node, source = parser.parse_entity(
         test_fn, inspect_utils.getfutureimports(test_fn))
     origin_info.resolve_entity(node, source, test_fn)
 
     # The line numbers below should match those in basic_definitions.py
+    fn_start = inspect.getsourcelines(test_fn)[1]
 
     inner_def_origin = anno.getanno(node.body[1], anno.Basic.ORIGIN)
-    self.assertEqual(inner_def_origin.loc.lineno, 31)
+    self.assertEqual(inner_def_origin.loc.lineno, fn_start + 3)
     self.assertEqual(inner_def_origin.loc.col_offset, 2)
     self.assertEqual(inner_def_origin.source_code_line, '  def inner_fn(y):')
     self.assertIsNone(inner_def_origin.comment)
 
     inner_ret_origin = anno.getanno(node.body[1].body[0], anno.Basic.ORIGIN)
-    self.assertEqual(inner_ret_origin.loc.lineno, 32)
+    self.assertEqual(inner_ret_origin.loc.lineno, fn_start + 4)
     self.assertEqual(inner_ret_origin.loc.col_offset, 4)
     self.assertEqual(inner_ret_origin.source_code_line, '    return y')
     self.assertIsNone(inner_ret_origin.comment)
@@ -193,58 +197,59 @@ class OriginInfoTest(test.TestCase):
   def test_resolve_entity_indented_block(self):
 
     test_fn = basic_definitions.SimpleClass.simple_method
-    node, source = parser.parse_entity(
-        test_fn, inspect_utils.getfutureimports(test_fn))
+    node, source = parser.parse_entity(test_fn,
+                                       inspect_utils.getfutureimports(test_fn))
     origin_info.resolve_entity(node, source, test_fn)
 
     # The line numbers below should match those in basic_definitions.py
+    fn_start = inspect.getsourcelines(test_fn)[1]
 
     def_origin = anno.getanno(node, anno.Basic.ORIGIN)
-    self.assertEqual(def_origin.loc.lineno, 46)
+    self.assertEqual(def_origin.loc.lineno, fn_start)
     self.assertEqual(def_origin.loc.col_offset, 2)
     self.assertEqual(def_origin.source_code_line, 'def simple_method(self):')
     self.assertIsNone(def_origin.comment)
 
     ret_origin = anno.getanno(node.body[0], anno.Basic.ORIGIN)
-    self.assertEqual(ret_origin.loc.lineno, 47)
+    self.assertEqual(ret_origin.loc.lineno, fn_start + 1)
     self.assertEqual(ret_origin.loc.col_offset, 4)
     self.assertEqual(ret_origin.source_code_line, '  return self')
     self.assertIsNone(ret_origin.comment)
 
   def test_resolve_entity_decorated_function(self):
-
     test_fn = basic_definitions.decorated_function
-    node, source = parser.parse_entity(
-        test_fn, inspect_utils.getfutureimports(test_fn))
+    node, source = parser.parse_entity(test_fn,
+                                       inspect_utils.getfutureimports(test_fn))
     origin_info.resolve_entity(node, source, test_fn)
 
     # The line numbers below should match those in basic_definitions.py
+    fn_start = inspect.getsourcelines(test_fn)[1]
 
     def_origin = anno.getanno(node, anno.Basic.ORIGIN)
     if sys.version_info >= (3, 8):
-      self.assertEqual(def_origin.loc.lineno, 67)
-      self.assertEqual(
-          def_origin.source_code_line, 'def decorated_function(x):')
+      self.assertEqual(def_origin.loc.lineno, fn_start + 2)
+      self.assertEqual(def_origin.source_code_line,
+                       'def decorated_function(x):')
     else:
-      self.assertEqual(def_origin.loc.lineno, 65)
+      self.assertEqual(def_origin.loc.lineno, fn_start)
       self.assertEqual(def_origin.source_code_line, '@basic_decorator')
     self.assertEqual(def_origin.loc.col_offset, 0)
     self.assertIsNone(def_origin.comment)
 
     if_origin = anno.getanno(node.body[0], anno.Basic.ORIGIN)
-    self.assertEqual(if_origin.loc.lineno, 68)
+    self.assertEqual(if_origin.loc.lineno, fn_start + 3)
     self.assertEqual(if_origin.loc.col_offset, 2)
     self.assertEqual(if_origin.source_code_line, '  if x > 0:')
     self.assertIsNone(if_origin.comment)
 
     ret1_origin = anno.getanno(node.body[0].body[0], anno.Basic.ORIGIN)
-    self.assertEqual(ret1_origin.loc.lineno, 69)
+    self.assertEqual(ret1_origin.loc.lineno, fn_start + 4)
     self.assertEqual(ret1_origin.loc.col_offset, 4)
     self.assertEqual(ret1_origin.source_code_line, '    return 1')
     self.assertIsNone(ret1_origin.comment)
 
     ret2_origin = anno.getanno(node.body[1], anno.Basic.ORIGIN)
-    self.assertEqual(ret2_origin.loc.lineno, 70)
+    self.assertEqual(ret2_origin.loc.lineno, fn_start + 5)
     self.assertEqual(ret2_origin.loc.col_offset, 2)
     self.assertEqual(ret2_origin.source_code_line, '  return 2')
     self.assertIsNone(ret2_origin.comment)
diff --git a/tensorflow/python/compat/compat.py b/tensorflow/python/compat/compat.py
index 2a625496569..30deba7b99f 100644
--- a/tensorflow/python/compat/compat.py
+++ b/tensorflow/python/compat/compat.py
@@ -33,7 +33,7 @@ from tensorflow.python.util.tf_export import tf_export
 # This value changes every day with an automatic CL. It can be modified in code
 # via `forward_compatibility_horizon()` or with the environment variable
 # TF_FORWARD_COMPATIBILITY_DELTA_DAYS, which is added to the compatibility date.
-_FORWARD_COMPATIBILITY_HORIZON = datetime.date(2020, 6, 2)
+_FORWARD_COMPATIBILITY_HORIZON = datetime.date(2020, 6, 7)
 _FORWARD_COMPATIBILITY_DELTA_DAYS_VAR_NAME = "TF_FORWARD_COMPATIBILITY_DELTA_DAYS"
 _FORWARD_COMPATIBILITY_DATE_NUMBER = None
 
diff --git a/tensorflow/python/data/experimental/ops/cardinality.py b/tensorflow/python/data/experimental/ops/cardinality.py
index f1b8908fa68..3505d1fc87e 100644
--- a/tensorflow/python/data/experimental/ops/cardinality.py
+++ b/tensorflow/python/data/experimental/ops/cardinality.py
@@ -20,6 +20,7 @@ from __future__ import print_function
 from tensorflow.python.data.ops import dataset_ops
 from tensorflow.python.framework import dtypes
 from tensorflow.python.framework import ops
+from tensorflow.python.ops import gen_dataset_ops
 from tensorflow.python.ops import gen_experimental_dataset_ops as ged_ops
 from tensorflow.python.util.tf_export import tf_export
 
@@ -64,7 +65,7 @@ def cardinality(dataset):
     constant `INFINITE_CARDINALITY` and `UNKNOWN_CARDINALITY` respectively.
   """
 
-  return ged_ops.dataset_cardinality(dataset._variant_tensor)  # pylint: disable=protected-access
+  return gen_dataset_ops.dataset_cardinality(dataset._variant_tensor)  # pylint: disable=protected-access
 
 
 @tf_export("data.experimental.assert_cardinality")
diff --git a/tensorflow/python/data/kernel_tests/shuffle_test.py b/tensorflow/python/data/kernel_tests/shuffle_test.py
index eaa4afb93a8..7a6521ecbdc 100644
--- a/tensorflow/python/data/kernel_tests/shuffle_test.py
+++ b/tensorflow/python/data/kernel_tests/shuffle_test.py
@@ -23,8 +23,6 @@ import functools
 from absl.testing import parameterized
 import numpy as np
 
-from tensorflow.python import tf2
-from tensorflow.python.compat import compat
 from tensorflow.python.data.kernel_tests import test_base
 from tensorflow.python.data.ops import dataset_ops
 from tensorflow.python.eager import function
@@ -331,17 +329,11 @@ class ShuffleTest(test_base.DatasetTestBase, parameterized.TestCase):
     with self.assertRaises(errors.OutOfRangeError):
       self.evaluate(get_next())
 
-  # We skip v2 eager since the v2 eager shuffle dataset is not serializable due
-  # to its use of an external seed generator resource.
   @combinations.generate(
       combinations.times(
-          test_base.graph_only_combinations() +
-          combinations.combine(mode=["eager"]),
+          test_base.default_test_combinations(),
           combinations.combine(reshuffle=[True, False])))
   def testRerandomizeOnReplicate(self, reshuffle):
-    if tf2.enabled() and not compat.forward_compatible(2020, 5, 22):
-      self.skipTest("Functionality currently not supported.")
-
     random_seed.set_random_seed(None)
     # When no seeds are fixed, each instantiation of the shuffle dataset should
     # produce elements in a different order.
diff --git a/tensorflow/python/data/ops/dataset_ops.py b/tensorflow/python/data/ops/dataset_ops.py
index 372c19855af..53ccd9edaae 100644
--- a/tensorflow/python/data/ops/dataset_ops.py
+++ b/tensorflow/python/data/ops/dataset_ops.py
@@ -30,7 +30,6 @@ from six.moves import queue as Queue  # pylint: disable=redefined-builtin
 
 from tensorflow.core.framework import graph_pb2
 from tensorflow.python import tf2
-from tensorflow.python.compat import compat
 from tensorflow.python.data.experimental.ops import distribute_options
 from tensorflow.python.data.experimental.ops import optimization_options
 from tensorflow.python.data.experimental.ops import stats_options
@@ -2157,7 +2156,7 @@ name=None))
       named constants `tf.data.INFINITE_CARDINALITY` and
       `tf.data.UNKNOWN_CARDINALITY` respectively.
     """
-    return ged_ops.dataset_cardinality(self._variant_tensor)
+    return gen_dataset_ops.dataset_cardinality(self._variant_tensor)
 
 
 @tf_export(v1=["data.Dataset"])
@@ -3609,54 +3608,6 @@ class RangeDataset(DatasetSource):
     return self._structure
 
 
-# This can be deleted after the forward compatibility window for switching
-# to using dummy resource expires on 5/20.
-class _MemoryCacheDeleter(object):
-  """An object which cleans up an anonymous memory cache resource.
-
-  An alternative to defining a __del__ method on an object. Even if the parent
-  object is part of a reference cycle, the cycle will be collectable.
-  """
-
-  def __init__(self, handle, device, deleter):
-    self._deleter = deleter
-    self._handle = handle
-    self._device = device
-    self._eager_mode = context.executing_eagerly()
-
-  def __del__(self):
-    with ops.device(self._device):
-      # Make sure the resource is deleted in the same mode as it was created in.
-      if self._eager_mode:
-        with context.eager_mode():
-          gen_dataset_ops.delete_memory_cache(
-              handle=self._handle, deleter=self._deleter)
-      else:
-        with context.graph_mode():
-          gen_dataset_ops.delete_memory_cache(
-              handle=self._handle, deleter=self._deleter)
-
-
-# This can be deleted after the forward compatibility window for switching
-# to using dummy resource expires on 5/20.
-class _MemoryCache(object):
-  """Represents a memory cache resource."""
-
-  def __init__(self):
-    super(_MemoryCache, self).__init__()
-    if compat.forward_compatible(2020, 5, 20):
-      self._handle = gen_dataset_ops.dummy_memory_cache()
-    else:
-      self._device = context.context().device_name
-      self._handle, self._deleter = gen_dataset_ops.anonymous_memory_cache()
-      self._resource_deleter = _MemoryCacheDeleter(
-          handle=self._handle, device=self._device, deleter=self._deleter)
-
-  @property
-  def handle(self):
-    return self._handle
-
-
 class CacheDataset(UnaryUnchangedStructureDataset):
   """A `Dataset` that caches elements of its input."""
 
@@ -3666,11 +3617,10 @@ class CacheDataset(UnaryUnchangedStructureDataset):
     self._filename = ops.convert_to_tensor(
         filename, dtype=dtypes.string, name="filename")
     if tf2.enabled() and (context.executing_eagerly() or ops.inside_function()):
-      self._cache = _MemoryCache()
       variant_tensor = gen_dataset_ops.cache_dataset_v2(
           input_dataset._variant_tensor,  # pylint: disable=protected-access
           filename=self._filename,
-          cache=self._cache.handle,
+          cache=gen_dataset_ops.dummy_memory_cache(),
           **self._flat_structure)
     else:
       variant_tensor = gen_dataset_ops.cache_dataset(
@@ -3680,56 +3630,6 @@ class CacheDataset(UnaryUnchangedStructureDataset):
     super(CacheDataset, self).__init__(input_dataset, variant_tensor)
 
 
-# This can be deleted after the forward compatibility window for switching
-# to using dummy resource expires on 5/22.
-class _SeedGeneratorDeleter(object):
-  """An object which cleans up an anonymous seed generator resource.
-
-  An alternative to defining a __del__ method on an object. Even if the parent
-  object is part of a reference cycle, the cycle will be collectable.
-  """
-
-  def __init__(self, handle, device, deleter):
-    self._deleter = deleter
-    self._handle = handle
-    self._device = device
-    self._eager_mode = context.executing_eagerly()
-
-  def __del__(self):
-    with ops.device(self._device):
-      # Make sure the resource is deleted in the same mode as it was created in.
-      if self._eager_mode:
-        with context.eager_mode():
-          gen_dataset_ops.delete_seed_generator(
-              handle=self._handle, deleter=self._deleter)
-      else:
-        with context.graph_mode():
-          gen_dataset_ops.delete_seed_generator(
-              handle=self._handle, deleter=self._deleter)
-
-
-# This can be deleted after the forward compatibility window for switching
-# to using dummy resource expires on 5/22.
-class _SeedGenerator(object):
-  """Represents a fixed seed generator resource."""
-
-  def __init__(self, seed, seed2, reshuffle):
-    super(_SeedGenerator, self).__init__()
-    if compat.forward_compatible(2020, 5, 22):
-      self._handle = gen_dataset_ops.dummy_seed_generator()
-    else:
-      self._device = context.context().device_name
-      self._handle, self._deleter = (
-          gen_dataset_ops.anonymous_seed_generator(
-              seed=seed, seed2=seed2, reshuffle=reshuffle))
-      self._resource_deleter = _SeedGeneratorDeleter(
-          handle=self._handle, device=self._device, deleter=self._deleter)
-
-  @property
-  def handle(self):
-    return self._handle
-
-
 class ShuffleDataset(UnaryUnchangedStructureDataset):
   """A `Dataset` that randomly shuffles the elements of its input."""
 
@@ -3767,23 +3667,14 @@ class ShuffleDataset(UnaryUnchangedStructureDataset):
 
     if (tf2.enabled() and
         (context.executing_eagerly() or ops.inside_function())):
-      self._seed_generator = _SeedGenerator(self._seed, self._seed2,
-                                            self._reshuffle_each_iteration)
-      if compat.forward_compatible(2020, 5, 22):
-        variant_tensor = gen_dataset_ops.shuffle_dataset_v3(
-            input_dataset._variant_tensor,  # pylint: disable=protected-access
-            buffer_size=self._buffer_size,
-            seed=self._seed,
-            seed2=self._seed2,
-            seed_generator=self._seed_generator.handle,
-            reshuffle_each_iteration=self._reshuffle_each_iteration,
-            **self._flat_structure)
-      else:
-        variant_tensor = gen_dataset_ops.shuffle_dataset_v2(
-            input_dataset._variant_tensor,  # pylint: disable=protected-access
-            buffer_size=self._buffer_size,
-            seed_generator=self._seed_generator.handle,
-            **self._flat_structure)
+      variant_tensor = gen_dataset_ops.shuffle_dataset_v3(
+          input_dataset._variant_tensor,  # pylint: disable=protected-access
+          buffer_size=self._buffer_size,
+          seed=self._seed,
+          seed2=self._seed2,
+          seed_generator=gen_dataset_ops.dummy_seed_generator(),
+          reshuffle_each_iteration=self._reshuffle_each_iteration,
+          **self._flat_structure)
     else:
       variant_tensor = gen_dataset_ops.shuffle_dataset(
           input_dataset._variant_tensor,  # pylint: disable=protected-access
diff --git a/tensorflow/python/debug/lib/debug_v2_ops_test.py b/tensorflow/python/debug/lib/debug_v2_ops_test.py
index 07721920f63..10de01f4f2e 100644
--- a/tensorflow/python/debug/lib/debug_v2_ops_test.py
+++ b/tensorflow/python/debug/lib/debug_v2_ops_test.py
@@ -23,6 +23,7 @@ import os
 import numpy as np
 
 from tensorflow.core.protobuf import debug_event_pb2
+from tensorflow.python.compat import compat
 from tensorflow.python.debug.lib import debug_events_reader
 from tensorflow.python.debug.lib import debug_events_writer
 from tensorflow.python.debug.lib import dumping_callback_test_lib
@@ -40,6 +41,12 @@ from tensorflow.python.platform import googletest
 
 
 class DebugIdentityV2OpTest(dumping_callback_test_lib.DumpingCallbackTestBase):
+  """Tests for DebugIdentityV2Op: when DebugEventsWriter is initialized.
+
+  DebugEventsWriter being initialized prior to DebugIdentityV2 ops being invoked
+  for the first time is the typical case (e.g., tfdbg2 running on a local
+  machine with only local devices.)
+  """
 
   def setUp(self):
     super(DebugIdentityV2OpTest, self).setUp()
@@ -57,8 +64,6 @@ class DebugIdentityV2OpTest(dumping_callback_test_lib.DumpingCallbackTestBase):
 
     @def_function.function
     def write_debug_trace(x):
-      # DebugIdentityV2 is a stateful op. It ought to be included by auto
-      # control dependency.
       square = math_ops.square(x)
       gen_debug_ops.debug_identity_v2(
           square,
@@ -223,6 +228,64 @@ class DebugIdentityV2OpTest(dumping_callback_test_lib.DumpingCallbackTestBase):
         with self.assertRaises(StopIteration):
           next(graph_trace_iter)
 
+
+class DebugIdentityV2OpUninitializedWriterTest(
+    dumping_callback_test_lib.DumpingCallbackTestBase):
+  """Tests for DebugIdentityV2Op: when DebugEventsWriter is not initialized.
+
+  This case can occur when DebugIdentityV2Ops are running on a remote
+  TensorFlow server (e.g., a TPU worker).
+  """
+
+  @test_util.run_in_graph_and_eager_modes
+  def testInvokingDebugIdentityV2OpBeforeCreatingDebugEventsWriterWorks(self):
+    if not compat.forward_compatible(2020, 6, 24):
+      self.skipTest("Functionality currently not supported.")
+    circular_buffer_size = 3
+
+    @def_function.function
+    def write_debug_trace(x):
+      # DebugIdentityV2 is a stateful op. It ought to be included by auto
+      # control dependency.
+      square = math_ops.square(x)
+      gen_debug_ops.debug_identity_v2(
+          square,
+          tfdbg_context_id="deadbeaf",
+          op_name="Square",
+          output_slot=0,
+          tensor_debug_mode=debug_event_pb2.TensorDebugMode.FULL_TENSOR,
+          debug_urls=["file://%s" % self.dump_root],
+          circular_buffer_size=circular_buffer_size)
+      return square
+
+    # The DebugIdentityV2 ops are invokes *before* a DebugEventsWriter at the
+    # same dump root is created.
+    for i in range(circular_buffer_size * 2):
+      self.assertAllClose(
+          write_debug_trace(np.array([i]).astype(np.float32)), [i**2.0])
+    writer = debug_events_writer.DebugEventsWriter(self.dump_root,
+                                                   circular_buffer_size)
+    writer.FlushNonExecutionFiles()
+    writer.FlushExecutionFiles()
+
+    with debug_events_reader.DebugEventsReader(self.dump_root) as reader:
+      graph_trace_iter = reader.graph_execution_traces_iterator()
+      graph_execution_traces = []
+      while True:
+        try:
+          graph_execution_traces.append(
+              next(graph_trace_iter).debug_event.graph_execution_trace)
+        except StopIteration:
+          break
+      self.assertLen(graph_execution_traces, circular_buffer_size)
+      for i in range(circular_buffer_size):
+        self.assertAllClose(
+            tensor_util.MakeNdarray(graph_execution_traces[i].tensor_proto),
+            [(i + circular_buffer_size)**2.0])
+
+
+class DebugNumericSummaryV2Test(test_util.TensorFlowTestCase):
+
   @test_util.run_in_graph_and_eager_modes
   def testDebugNumericSummaryV2OpReduceInfNanThreeSlots(self):
 
diff --git a/tensorflow/python/debug/lib/dumping_callback.py b/tensorflow/python/debug/lib/dumping_callback.py
index f012faf5f3c..7e61631fb1c 100644
--- a/tensorflow/python/debug/lib/dumping_callback.py
+++ b/tensorflow/python/debug/lib/dumping_callback.py
@@ -30,6 +30,7 @@ from six.moves import xrange  # pylint: disable=redefined-builtin
 from tensorflow.core.framework import tensor_pb2
 from tensorflow.core.protobuf import debug_event_pb2
 from tensorflow.core.protobuf import graph_debug_info_pb2
+from tensorflow.python.compat import compat as tf_compat
 from tensorflow.python.debug.lib import debug_events_writer
 from tensorflow.python.debug.lib import op_callbacks_common
 from tensorflow.python.debug.lib import source_utils
@@ -366,17 +367,31 @@ class _DumpingCallback(object):
         with self._symbolic_tensor_counter_lock:
           debug_identity_name = ("DebugIdentityV2_%d" %
                                  self._symbolic_tensor_counter)
-        debug_tensor = gen_debug_ops.debug_identity_v2(
-            # Use an empty (shape=[0]) float32 tensor for the NO_TENSOR mode
-            # as a low-overhead placeholder, since no actual tensor value is
-            # traced.
-            constant_op.constant([], dtype=dtypes.float32),
-            tfdbg_context_id=tfdbg_context_id,
-            op_name=op_name,
-            output_slot=output_slot,
-            tensor_debug_mode=self._tensor_debug_mode,
-            debug_urls=debug_urls,
-            name=debug_identity_name)
+        if tf_compat.forward_compatible(2020, 6, 24):
+          debug_tensor = gen_debug_ops.debug_identity_v2(
+              # Use an empty (shape=[0]) float32 tensor for the NO_TENSOR mode
+              # as a low-overhead placeholder, since no actual tensor value is
+              # traced.
+              constant_op.constant([], dtype=dtypes.float32),
+              tfdbg_context_id=tfdbg_context_id,
+              op_name=op_name,
+              output_slot=output_slot,
+              tensor_debug_mode=self._tensor_debug_mode,
+              debug_urls=debug_urls,
+              circular_buffer_size=self._circular_buffer_size,
+              name=debug_identity_name)
+        else:
+          debug_tensor = gen_debug_ops.debug_identity_v2(
+              # Use an empty (shape=[0]) float32 tensor for the NO_TENSOR mode
+              # as a low-overhead placeholder, since no actual tensor value is
+              # traced.
+              constant_op.constant([], dtype=dtypes.float32),
+              tfdbg_context_id=tfdbg_context_id,
+              op_name=op_name,
+              output_slot=output_slot,
+              tensor_debug_mode=self._tensor_debug_mode,
+              debug_urls=debug_urls,
+              name=debug_identity_name)
         if is_v1_graph_mode:
           instrumented_tensors.append(self._process_v1_graph_mode_tensor(
               op_type, tensor, debug_tensor, tensor_debug_mode))
@@ -400,17 +415,31 @@ class _DumpingCallback(object):
           if is_v1_graph_mode:
             instrumented_tensors.append(tensor)
           continue
-        debug_tensor = gen_debug_ops.debug_identity_v2(
-            gen_debug_ops.debug_numeric_summary_v2(
-                tensor,
-                tensor_id=tensor_ids[output_slot],
-                tensor_debug_mode=self._tensor_debug_mode,
-                output_dtype=dtypes.float64),
-            tfdbg_context_id=tfdbg_context_id,
-            op_name=op_name,
-            output_slot=output_slot,
-            tensor_debug_mode=self._tensor_debug_mode,
-            debug_urls=debug_urls)
+        if tf_compat.forward_compatible(2020, 6, 24):
+          debug_tensor = gen_debug_ops.debug_identity_v2(
+              gen_debug_ops.debug_numeric_summary_v2(
+                  tensor,
+                  tensor_id=tensor_ids[output_slot],
+                  tensor_debug_mode=self._tensor_debug_mode,
+                  output_dtype=dtypes.float64),
+              tfdbg_context_id=tfdbg_context_id,
+              op_name=op_name,
+              output_slot=output_slot,
+              tensor_debug_mode=self._tensor_debug_mode,
+              debug_urls=debug_urls,
+              circular_buffer_size=self._circular_buffer_size)
+        else:
+          debug_tensor = gen_debug_ops.debug_identity_v2(
+              gen_debug_ops.debug_numeric_summary_v2(
+                  tensor,
+                  tensor_id=tensor_ids[output_slot],
+                  tensor_debug_mode=self._tensor_debug_mode,
+                  output_dtype=dtypes.float64),
+              tfdbg_context_id=tfdbg_context_id,
+              op_name=op_name,
+              output_slot=output_slot,
+              tensor_debug_mode=self._tensor_debug_mode,
+              debug_urls=debug_urls)
         if is_v1_graph_mode:
           instrumented_tensors.append(self._process_v1_graph_mode_tensor(
               op_type, tensor, debug_tensor, tensor_debug_mode))
@@ -424,13 +453,23 @@ class _DumpingCallback(object):
           if is_v1_graph_mode:
             instrumented_tensors.append(tensor)
           continue
-        debug_tensor = gen_debug_ops.debug_identity_v2(
-            tensor,
-            tfdbg_context_id=tfdbg_context_id,
-            op_name=op_name,
-            output_slot=output_slot,
-            tensor_debug_mode=self._tensor_debug_mode,
-            debug_urls=debug_urls)
+        if tf_compat.forward_compatible(2020, 6, 24):
+          debug_tensor = gen_debug_ops.debug_identity_v2(
+              tensor,
+              tfdbg_context_id=tfdbg_context_id,
+              op_name=op_name,
+              output_slot=output_slot,
+              tensor_debug_mode=self._tensor_debug_mode,
+              debug_urls=debug_urls,
+              circular_buffer_size=self._circular_buffer_size)
+        else:
+          debug_tensor = gen_debug_ops.debug_identity_v2(
+              tensor,
+              tfdbg_context_id=tfdbg_context_id,
+              op_name=op_name,
+              output_slot=output_slot,
+              tensor_debug_mode=self._tensor_debug_mode,
+              debug_urls=debug_urls)
         if is_v1_graph_mode:
           instrumented_tensors.append(self._process_v1_graph_mode_tensor(
               op_type, tensor, debug_tensor, tensor_debug_mode))
diff --git a/tensorflow/python/distribute/BUILD b/tensorflow/python/distribute/BUILD
index 26027d46c98..977452cdad1 100644
--- a/tensorflow/python/distribute/BUILD
+++ b/tensorflow/python/distribute/BUILD
@@ -1791,3 +1791,19 @@ cuda_py_test(
         "@absl_py//absl/testing:parameterized",
     ],
 )
+
+distribute_py_test(
+    name = "tf_function_test",
+    srcs = ["tf_function_test.py"],
+    main = "tf_function_test.py",
+    tags = [
+        "multi_and_single_gpu",
+    ],
+    deps = [
+        ":combinations",
+        ":strategy_combinations",
+        "//tensorflow/python:array_ops",
+        "//tensorflow/python/eager:def_function",
+        "//tensorflow/python/eager:test",
+    ],
+)
diff --git a/tensorflow/python/distribute/distribute_lib.py b/tensorflow/python/distribute/distribute_lib.py
index b77163cb97a..5ea738765b7 100644
--- a/tensorflow/python/distribute/distribute_lib.py
+++ b/tensorflow/python/distribute/distribute_lib.py
@@ -620,10 +620,8 @@ class StrategyBase(object):
     if not hasattr(extended, "_retrace_functions_for_each_device"):
       # pylint: disable=protected-access
       # `extended._retrace_functions_for_each_device` dictates
-      # 1) whether all the ops created inside function will have devices
-      #    inherited from outer stack, and
-      # 2) whether the same function will be retraced when it is called on
-      #    different devices.
+      # whether the same function will be retraced when it is called on
+      # different devices.
       try:
         extended._retrace_functions_for_each_device = (
             len(extended.worker_devices) > 1)
diff --git a/tensorflow/python/distribute/parallel_device/pywrap_parallel_device.cc b/tensorflow/python/distribute/parallel_device/pywrap_parallel_device.cc
index 62488cb31e7..dd97d906f36 100644
--- a/tensorflow/python/distribute/parallel_device/pywrap_parallel_device.cc
+++ b/tensorflow/python/distribute/parallel_device/pywrap_parallel_device.cc
@@ -52,7 +52,7 @@ PYBIND11_MODULE(_pywrap_parallel_device, m) {
           tensorflow::Safe_PyObjectPtr device_capsule(
               PyCapsule_New(device, "TFE_CustomDevice", &CallDelete_Device));
           void* device_info;
-          tensorflow::eager::AllocateParallelDevice(
+          tensorflow::parallel_device::AllocateParallelDevice(
               name, underlying_devices_c.data(), underlying_devices_c.size(),
               device, &device_info);
           if (PyErr_Occurred()) throw py::error_already_set();
diff --git a/tensorflow/python/distribute/tf_function_test.py b/tensorflow/python/distribute/tf_function_test.py
new file mode 100644
index 00000000000..5dc82cfd81b
--- /dev/null
+++ b/tensorflow/python/distribute/tf_function_test.py
@@ -0,0 +1,131 @@
+# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Tests for tf.function + distribution strategies."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from absl.testing import parameterized
+
+from tensorflow.python.compat import v2_compat
+from tensorflow.python.distribute import combinations
+from tensorflow.python.distribute import device_util
+from tensorflow.python.distribute import strategy_combinations
+from tensorflow.python.distribute import values
+from tensorflow.python.eager import def_function
+from tensorflow.python.eager import test
+from tensorflow.python.framework import dtypes
+from tensorflow.python.framework import ops
+from tensorflow.python.ops import array_ops
+from tensorflow.python.ops import math_ops
+from tensorflow.python.ops import variables
+
+
+class TFFunctionTest(test.TestCase, parameterized.TestCase):
+
+  def setup(self):
+    # Clear the state for every test.
+    def_function.run_functions_eagerly(False)
+
+  @combinations.generate(
+      combinations.combine(
+          distribution=strategy_combinations.all_strategies,
+          mode=["eager"],
+          run_functions_eagerly=[True, False]
+      ))
+  def testDefaultDeviceInsideFunctionWithScope(
+      self, distribution, run_functions_eagerly):
+
+    def_function.run_functions_eagerly(run_functions_eagerly)
+    expected_device = (device_util.canonicalize("cpu:0")
+                       if run_functions_eagerly else "")
+    with distribution.scope():
+      with ops.device_v2("cpu:0"):
+        @def_function.function
+        def add():
+          one = array_ops.ones([])
+          self.assertEqual(expected_device, one.device)
+          return one + 1
+
+        add()
+
+  @combinations.generate(
+      combinations.combine(
+          distribution=strategy_combinations.all_strategies,
+          mode=["eager"],
+          run_functions_eagerly=[True, False]
+      ))
+  def testDefaultDeviceInsideNestedFunctionWithScope(
+      self, distribution, run_functions_eagerly):
+
+    def_function.run_functions_eagerly(run_functions_eagerly)
+    expected_device = (device_util.canonicalize("cpu:0")
+                       if run_functions_eagerly else "")
+    with distribution.scope():
+      @def_function.function
+      def foo():
+        with ops.device("cpu:0"):
+
+          @def_function.function
+          def bar():
+            one = array_ops.ones([])
+            self.assertEqual(expected_device, one.device)
+            return one + 1
+
+          bar()
+
+      foo()
+
+  @combinations.generate(
+      combinations.combine(
+          distribution=strategy_combinations.all_strategies,
+          mode=["eager"],
+          run_functions_eagerly=[True, False]
+      ))
+  def testReadVariableInsideFunction(self, distribution, run_functions_eagerly):
+
+    # Get devices on which variables will be placed. Default strategy does not
+    # define this, so assume cpu:0 in that case.
+    try:
+      devices = distribution.extended.parameter_devices
+    except RuntimeError:
+      devices = ["cpu:0"]
+
+    with distribution.scope():
+      v = variables.Variable(0.)
+      if isinstance(v, values.DistributedVariable):
+        for i in range(len(devices)):
+          # NOTE: Assigning manually to component variables so we can test
+          # different values on different devices. Using .assign on the
+          # mirrored variable itself will lead to a synchronization which
+          # will prohibit testing different values.
+          replica_variable = v._values[i]
+          replica_variable.assign(math_ops.cast(i, dtypes.float32))
+
+    @def_function.function
+    def read():
+      return v.read_value()
+
+    for i, d in enumerate(devices):
+      with ops.device(d):
+        # Verify that the value from each device is read, when in that device
+        # scope.
+        self.assertEqual(math_ops.cast(i, dtypes.float32), read())
+
+
+if __name__ == "__main__":
+  v2_compat.enable_v2_behavior()
+  test.main()
diff --git a/tensorflow/python/distribute/tpu_strategy.py b/tensorflow/python/distribute/tpu_strategy.py
index a8ffa618064..278d30afb06 100644
--- a/tensorflow/python/distribute/tpu_strategy.py
+++ b/tensorflow/python/distribute/tpu_strategy.py
@@ -59,6 +59,8 @@ from tensorflow.python.tpu.ops import tpu_ops
 from tensorflow.python.util import nest
 from tensorflow.python.util.tf_export import tf_export
 
+_XLA_OP_BY_OP_INPUTS_LIMIT = 200
+
 
 @contextlib.contextmanager
 def maybe_init_scope():
@@ -676,8 +678,18 @@ class TPUExtended(distribute_lib.StrategyExtendedV1):
       return cross_device_ops_lib.reduce_non_distributed_value(
           reduce_op, value, destinations, self._num_replicas_in_sync)
 
+    # Currently XLA op by op mode has a limit for the number of inputs for a
+    # single op, thus we break one `add_n` op into a group of `add_n` ops to
+    # work around the constraint.
     # TODO(cjfj): Detect when it is possible to use `cross_replica_sum`.
-    output = math_ops.add_n(value.values)
+    if len(value.values) <= _XLA_OP_BY_OP_INPUTS_LIMIT:
+      output = math_ops.add_n(value.values)
+    else:
+      output = array_ops.zeros_like(
+          value.values[0], dtype=value.values[0].dtype)
+      for i in range(0, len(value.values), _XLA_OP_BY_OP_INPUTS_LIMIT):
+        output += math_ops.add_n(value.values[i:i + _XLA_OP_BY_OP_INPUTS_LIMIT])
+
     if reduce_op == reduce_util.ReduceOp.MEAN:
       output *= (1. / len(value.values))
 
diff --git a/tensorflow/python/distribute/values.py b/tensorflow/python/distribute/values.py
index 9c830f7081c..41ab1b40ecc 100644
--- a/tensorflow/python/distribute/values.py
+++ b/tensorflow/python/distribute/values.py
@@ -183,6 +183,17 @@ class DistributedDelegate(DistributedValues):
                                              "_distributed_container"):
       return super(DistributedDelegate, self).__getattr__(name)
 
+    # This allows copy.copy(DistributedDelegate). When copying an object,
+    # copy.copy doesn't invoke its __init__ method, instead it makes a new
+    # empty object, then copies the attributes over. copy.copy looks for
+    # attributes like "__getstate__" in case the object implements its custom
+    # copying. Since DistributedDelegate doesn't have those attributes defined,
+    # __getattr__ will be invoked, which tries to access "_values" attributes,
+    # but that doesn't exist either because this is an empty object, and again
+    # __getattr__ is invoked, leading to an infinite recursion.
+    if name == "_values":
+      raise AttributeError()
+
     # TODO(priyag): This needs to be made robust against pitfalls from mix use
     # __getattr__ and @property. See b/120402273.
     return getattr(self._get(), name)
diff --git a/tensorflow/python/distribute/values_test.py b/tensorflow/python/distribute/values_test.py
index 8016bfe9265..e91d89abc7d 100644
--- a/tensorflow/python/distribute/values_test.py
+++ b/tensorflow/python/distribute/values_test.py
@@ -19,6 +19,7 @@ from __future__ import division
 from __future__ import print_function
 
 import collections
+import copy
 import itertools
 import os
 
@@ -336,6 +337,20 @@ class DistributedDelegateTest(test.TestCase):
     with self.assertRaises(TypeError):
       _ = v[2]
 
+  @test_util.run_in_graph_and_eager_modes
+  def testCopy(self):
+
+    class Foo(object):
+
+      def __init__(self, x):
+        self.x = x
+
+    v = values.DistributedDelegate((Foo(7), Foo(8)))
+    v_shallow_copy = copy.copy(v)
+    self.assertEqual(v.x, v_shallow_copy.x)
+    v_deep_copy = copy.deepcopy(v)
+    self.assertEqual(v.x, v_deep_copy.x)
+
 
 def _device_str(d):
   return "/device:GPU:" + str(d)
diff --git a/tensorflow/python/eager/BUILD b/tensorflow/python/eager/BUILD
index a44d8a493c1..014cb97f72f 100644
--- a/tensorflow/python/eager/BUILD
+++ b/tensorflow/python/eager/BUILD
@@ -803,6 +803,7 @@ cuda_py_test(
     name = "def_function_test",
     srcs = ["def_function_test.py"],
     python_version = "PY3",
+    tfrt_enabled = True,
     deps = [
         ":def_function",
         "//tensorflow/python:client_testlib",
diff --git a/tensorflow/python/eager/benchmarks_test.py b/tensorflow/python/eager/benchmarks_test.py
index 3add04a81ec..315a4cfd056 100644
--- a/tensorflow/python/eager/benchmarks_test.py
+++ b/tensorflow/python/eager/benchmarks_test.py
@@ -463,6 +463,15 @@ class MicroBenchmarks(benchmarks_test_base.MicroBenchmarksBase):
     func = lambda: f(m, m, transpose_b=transpose_b)
     self._run(func, num_iters, execution_mode=execution_mode)
 
+  def _benchmark_defun_args_matmul(self, m, num_iters, execution_mode=None):
+
+    @def_function.function
+    def defun_matmul(m):
+      return math_ops.matmul(m, m)
+
+    func = lambda: defun_matmul(m)
+    self._run(func, num_iters, execution_mode=execution_mode)
+
   def _benchmark_nested_defun_matmul(self, m, transpose_b, num_iters):
     inner = function.defun(math_ops.matmul)
 
@@ -630,6 +639,14 @@ class MicroBenchmarks(benchmarks_test_base.MicroBenchmarksBase):
       self._benchmark_defun_matmul(
           m, transpose_b=False, num_iters=self._num_iters_2_by_2)
 
+  @test_util.disable_tfrt("Graph is not supported yet. b/156187905")
+  def benchmark_defun_args_matmul_2_by_2_GPU(self):
+    if not context.num_gpus():
+      return
+    with context.device(GPU):
+      m = self._m_2_by_2.gpu()
+      self._benchmark_defun_args_matmul(m, num_iters=self._num_iters_2_by_2)
+
   @test_util.disable_tfrt("async not supported")
   def benchmark_defun_matmul_2_by_2_GPU_async(self):
     if not context.num_gpus():
@@ -690,7 +707,6 @@ class MicroBenchmarks(benchmarks_test_base.MicroBenchmarksBase):
       self._benchmark_tfe_py_execute_matmul(
           m, transpose_b=True, num_iters=self._num_iters_100_by_784)
 
-  @test_util.disable_tfrt("Graph is not supported yet. b/156187905")
   def benchmark_defun_matmul_100_by_784_CPU(self):
     with context.device(CPU):
       m = self._m_100_by_784.cpu()
diff --git a/tensorflow/python/eager/def_function_test.py b/tensorflow/python/eager/def_function_test.py
index d5daa3acc99..0549da2c256 100644
--- a/tensorflow/python/eager/def_function_test.py
+++ b/tensorflow/python/eager/def_function_test.py
@@ -68,6 +68,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
 
     self.assertAllEqual(fn(constant_op.constant(4.0)), 8.0)
 
+  @test_util.disable_tfrt('Variable argument is not supported')
   def testFailIfVariablesAreCreatedMoreThanOnce(self):
 
     @def_function.function
@@ -77,6 +78,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
     with self.assertRaises(ValueError):
       fn(1.0)
 
+  @test_util.disable_tfrt('Variable argument is not supported')
   def testFailIfVariablesAreCreatedMoreThanOnceNoWeakRef(self):
     state = []
 
@@ -96,6 +98,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
 
     self.assertAllEqual(f(range(5)), 1.0)
 
+  @test_util.disable_tfrt('Variable argument is not supported')
   def testCorrectVariableCreation(self):
 
     state = []
@@ -109,6 +112,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
     self.assertAllEqual(fn(constant_op.constant(1.0)), 2.0)
     self.assertAllEqual(fn(constant_op.constant(3.0)), 6.0)
 
+  @test_util.disable_tfrt('Variable argument is not supported')
   def testFunctionInitializer(self):
 
     state = []
@@ -121,6 +125,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
 
     self.assertAllEqual(fn(constant_op.constant(1.0)), 2.0)
 
+  @test_util.disable_tfrt('Variable argument is not supported')
   def testFunctionMultipleVariableInitializer(self):
 
     state = []
@@ -134,6 +139,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
 
     self.assertAllEqual(fn(constant_op.constant(1.0)), [2.0, 5.0])
 
+  @test_util.disable_tfrt('Variable argument is not supported')
   def testFunctionInitializationFunction(self):
 
     state = []
@@ -151,6 +157,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
     init_fn()
     self.assertEqual(state[0].numpy(), 2.0)
 
+  @test_util.disable_tfrt('Variable argument is not supported')
   def testVariableInitializerNotConstant(self):
 
     state = []
@@ -180,6 +187,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
       self.assertAllEqual(sess.run(state[0]), 2.0)
       self.assertAllEqual(self.evaluate(result), 6.0)
 
+  @test_util.disable_tfrt('Variable argument is not supported')
   def testLegacyGraphModeVariablesNonTrivialInitializer(self):
     with ops.Graph().as_default(), self.test_session() as sess:
       state = []
@@ -199,6 +207,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
       self.assertAllEqual(sess.run(state[0]), 6.0)
       self.assertAllEqual(self.evaluate(result), 18.0)
 
+  @test_util.disable_tfrt('Variable argument is not supported')
   def testLegacyGraphModeInputDependentInitializerFails(self):
     with ops.Graph().as_default():
       state = []
@@ -213,6 +222,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
           lift_to_graph.UnliftableError, r'transitively.* mul .* x'):
         fn(constant_op.constant(3.0))
 
+  @test_util.disable_tfrt('Variable argument is not supported')
   def testMethod(self):
 
     class MyModel(object):
@@ -241,6 +251,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
         def_function.function(functools.partial(lambda x, y: x + y, 1.))(
             constant_op.constant(2.)))
 
+  @test_util.disable_tfrt('Partial is not supported')
   def test_functools_partial_new_default(self):
     def f(x=3, y=7):
       return x + y
@@ -249,6 +260,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
     self.assertEqual(func().numpy(), 9)
     self.assertEqual(func(y=8).numpy(), 11)
 
+  @test_util.disable_tfrt('Partial is not supported')
   def test_functools_partial_keywords(self):
     def f(x, y):
       return x + y
@@ -257,6 +269,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
         functools.partial(f, x=array_ops.zeros([1]), y=array_ops.zeros([1])))
     self.assertAllEqual(func(), [0.0])
 
+  @test_util.disable_tfrt('Partial is not supported')
   def test_functools_partial_single_positional(self):
     def f(x, y):
       return x + y
@@ -265,6 +278,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
         functools.partial(f, constant_op.constant(1)))
     self.assertAllEqual(func(5), 6)
 
+  @test_util.disable_tfrt('Partial is not supported')
   def test_complicated_partial_with_defaults(self):
 
     def identity(*args):
@@ -312,6 +326,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
                      (tensor_spec.TensorSpec(
                          None, dtypes.float32, name='x'),))
 
+  @test_util.disable_tfrt('Variable argument is not supported')
   @test_util.run_in_graph_and_eager_modes
   def test_variable_naming(self):
     class HasVars(module.Module):
@@ -382,6 +397,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
                                  'defined in another function or code block'):
       f(array_ops.zeros(shape=(8, 42, 3)))
 
+  @test_util.disable_tfrt('Control flow is not supported')
   def testRuntimeErrorNotSticky(self):
 
     @def_function.function
@@ -486,6 +502,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
                  constant_op.constant(3.),
                  constant_op.constant(4.)))
 
+  @test_util.disable_tfrt('Variable argument is not supported')
   def testVariableCreatorScope(self):
     created_variables = []
     captured_variables = []
@@ -505,6 +522,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
       f()
     self.assertEqual(created_variables, captured_variables)
 
+  @test_util.disable_tfrt('Variable argument is not supported')
   def testVarAlreadyInitializedNoClobbering(self):
     v_holder = []
 
@@ -522,6 +540,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
     add_var.get_concrete_function(constant_op.constant(2.))
     self.assertAllClose([13., 14.], add_var(constant_op.constant(2.)))
 
+  @test_util.disable_tfrt('Variable argument is not supported')
   def testSameVariableTwice(self):
     v = variables.Variable(1.0)
 
@@ -531,6 +550,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
 
     self.assertAllEqual(add(v, v), 2.0)
 
+  @test_util.disable_tfrt('Variable argument is not supported')
   def testVariableUpdate(self):
     v1 = variables.Variable(1.0)
     v2 = variables.Variable(2.0)
@@ -566,6 +586,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
 
     self.assertIs(func_a, func_b)
 
+  @test_util.disable_tfrt('Nested function is not supported')
   def testInitializationInNestedCall(self):
     v_holder = []
 
@@ -588,6 +609,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
     v_holder[1].assign(11.)
     self.assertAllClose([14., 15.], wrapper(constant_op.constant(2.)))
 
+  @test_util.disable_tfrt('Variable argument is not supported')
   @test_util.run_gpu_only
   def testDeviceAnnotationRespected(self):
     a = []
@@ -607,6 +629,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
     create_variable()
     self.assertRegexpMatches(a[0].device, 'CPU')
 
+  @test_util.disable_tfrt('Variable argument is not supported')
   @test_util.run_gpu_only
   def testDeviceAnnotationForInitializerRespected(self):
     a = []
@@ -681,6 +704,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
       self.assertEqual(self.evaluate(cloned(x)),
                        self.evaluate(cloned_py_function(x)))
 
+  @test_util.disable_tfrt('Variable argument is not supported')
   def testLiftPlaceholderInitializedVariable(self):
     with ops.Graph().as_default():
       var_list = []
@@ -834,6 +858,7 @@ class DefFunctionTest(test.TestCase, parameterized.TestCase):
     self.assertLen(logs.output, 1)
     self.assertIn('Tracing is expensive', logs.output[0])
 
+  @test_util.disable_tfrt('Nested function is not supported')
   def test_frequent_retracing_warning_nested(self):
     if sys.version_info[0] < 3:
       self.skipTest('self.assertLogs() call is not available in Python 2.')
diff --git a/tensorflow/python/eager/remote_cluster_test.py b/tensorflow/python/eager/remote_cluster_test.py
index 025899e271e..efe4e08cbf7 100644
--- a/tensorflow/python/eager/remote_cluster_test.py
+++ b/tensorflow/python/eager/remote_cluster_test.py
@@ -530,8 +530,7 @@ class DynamicClusterTest(test.TestCase, parameterized.TestCase):
     threads.append(threading.Thread(target=update_server_def_fn))
     for t in threads:
       t.start()
-    for t in threads:
-      t.join()
+    self._coord.join(threads)
     for result in results:
       np.testing.assert_array_equal([[2, 2], [2, 2]], result)
 
diff --git a/tensorflow/python/feature_column/BUILD b/tensorflow/python/feature_column/BUILD
index 786c26c009a..8f62fc2d1be 100644
--- a/tensorflow/python/feature_column/BUILD
+++ b/tensorflow/python/feature_column/BUILD
@@ -23,6 +23,7 @@ py_library(
     srcs_version = "PY2AND3",
     deps = [
         ":utils",
+        "@six_archive//:six",
         "//tensorflow/python:array_ops",
         "//tensorflow/python:check_ops",
         "//tensorflow/python:control_flow_ops",
@@ -48,7 +49,13 @@ py_library(
         "//tensorflow/python:variables",
         "//tensorflow/python/eager:context",
         "//tensorflow/python/keras/engine",
-        "@six_archive//:six",
+        # TODO(scottzhu): Remove metrics after we cleanup the keras internal cyclar dependency.
+        #     //third_party/tensorflow/python/feature_column:feature_column
+        #     //third_party/tensorflow/python/keras/engine:engine
+        # .-> //third_party/tensorflow/python/keras/distribute:distribute
+        # |   //third_party/tensorflow/python/keras:metrics
+        # `-- //third_party/tensorflow/python/keras/distribute:distribute
+        "//tensorflow/python/keras:metrics",
     ],
 )
 
@@ -72,7 +79,6 @@ py_library(
         "//tensorflow/python:init_ops",
         "//tensorflow/python:lookup_ops",
         "//tensorflow/python:math_ops",
-        "//tensorflow/python:nn_ops",
         "//tensorflow/python:parsing_ops",
         "//tensorflow/python:platform",
         "//tensorflow/python:sparse_ops",
@@ -85,11 +91,7 @@ py_library(
         "//tensorflow/python:variable_scope",
         "//tensorflow/python:variables",
         "//tensorflow/python/eager:context",
-        "//tensorflow/python/keras:backend",
         "//tensorflow/python/keras:initializers",
-        "//tensorflow/python/keras/engine",
-        "//tensorflow/python/keras/engine:base_layer",
-        "//tensorflow/python/keras/layers",
         "//tensorflow/python/keras/utils:generic_utils",
         "//tensorflow/python/training/tracking",
         "//tensorflow/python/training/tracking:data_structures",
@@ -236,11 +238,8 @@ py_test(
     deps = [
         ":feature_column_v2",
         "//tensorflow/python:client_testlib",
-        "//tensorflow/python:framework_ops",
         "//tensorflow/python:parsing_ops",
-        "//tensorflow/python:training",
         "//tensorflow/python:util",
-        "//tensorflow/python/keras/layers",
     ],
 )
 
@@ -254,37 +253,3 @@ tf_py_test(
         "@absl_py//absl/testing:parameterized",
     ],
 )
-
-tf_py_test(
-    name = "keras_integration_test",
-    size = "medium",
-    srcs = ["keras_integration_test.py"],
-    python_version = "PY3",
-    shard_count = 4,
-    tags = [
-        "nomac",  # TODO(mihaimaruseac): b/127695564
-        "notsan",
-    ],
-    deps = [
-        ":feature_column_py",
-        "//tensorflow/python:client_testlib",
-        "//tensorflow/python/keras",
-        "//third_party/py/numpy",
-        "@absl_py//absl/testing:parameterized",
-    ],
-)
-
-tf_py_test(
-    name = "save_test",
-    size = "medium",
-    srcs = ["save_test.py"],
-    python_version = "PY3",
-    deps = [
-        ":feature_column_v2",
-        "//tensorflow/python:client_testlib",
-        "//tensorflow/python/keras",
-        "//tensorflow/python/keras:combinations",
-        "//third_party/py/numpy",
-        "@absl_py//absl/testing:parameterized",
-    ],
-)
diff --git a/tensorflow/python/feature_column/save_test.py b/tensorflow/python/feature_column/save_test.py
deleted file mode 100644
index 05370911295..00000000000
--- a/tensorflow/python/feature_column/save_test.py
+++ /dev/null
@@ -1,133 +0,0 @@
-# Copyright 2019 The TensorFlow Authors. All Rights Reserved.
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-# ==============================================================================
-"""Tests for Keras model saving code."""
-
-from __future__ import absolute_import
-from __future__ import division
-from __future__ import print_function
-
-from absl.testing import parameterized
-import numpy as np
-
-from tensorflow.python import keras
-from tensorflow.python.eager import context
-from tensorflow.python.feature_column import feature_column_lib
-from tensorflow.python.framework import sparse_tensor
-from tensorflow.python.keras import combinations
-from tensorflow.python.keras.saving import model_config
-from tensorflow.python.ops import lookup_ops
-from tensorflow.python.platform import test
-
-
-class TestSaveModel(test.TestCase, parameterized.TestCase):
-
-  @combinations.generate(combinations.combine(mode=['graph', 'eager']))
-  def test_saving_with_dense_features(self):
-    cols = [
-        feature_column_lib.numeric_column('a'),
-        feature_column_lib.indicator_column(
-            feature_column_lib.categorical_column_with_vocabulary_list(
-                'b', ['one', 'two']))
-    ]
-    input_layers = {
-        'a': keras.layers.Input(shape=(1,), name='a'),
-        'b': keras.layers.Input(shape=(1,), name='b', dtype='string')
-    }
-
-    fc_layer = feature_column_lib.DenseFeatures(cols)(input_layers)
-    output = keras.layers.Dense(10)(fc_layer)
-
-    model = keras.models.Model(input_layers, output)
-
-    model.compile(
-        loss=keras.losses.MSE,
-        optimizer='rmsprop',
-        metrics=[keras.metrics.categorical_accuracy])
-
-    config = model.to_json()
-    loaded_model = model_config.model_from_json(config)
-
-    inputs_a = np.arange(10).reshape(10, 1)
-    inputs_b = np.arange(10).reshape(10, 1).astype('str')
-
-    with self.cached_session():
-      # Initialize tables for V1 lookup.
-      if not context.executing_eagerly():
-        self.evaluate(lookup_ops.tables_initializer())
-
-      self.assertLen(loaded_model.predict({'a': inputs_a, 'b': inputs_b}), 10)
-
-  @combinations.generate(combinations.combine(mode=['graph', 'eager']))
-  def test_saving_with_sequence_features(self):
-    cols = [
-        feature_column_lib.sequence_numeric_column('a'),
-        feature_column_lib.indicator_column(
-            feature_column_lib.sequence_categorical_column_with_vocabulary_list(
-                'b', ['one', 'two']))
-    ]
-    input_layers = {
-        'a':
-            keras.layers.Input(shape=(None, 1), sparse=True, name='a'),
-        'b':
-            keras.layers.Input(
-                shape=(None, 1), sparse=True, name='b', dtype='string')
-    }
-
-    fc_layer, _ = feature_column_lib.SequenceFeatures(cols)(input_layers)
-    # TODO(tibell): Figure out the right dtype and apply masking.
-    # sequence_length_mask = array_ops.sequence_mask(sequence_length)
-    # x = keras.layers.GRU(32)(fc_layer, mask=sequence_length_mask)
-    x = keras.layers.GRU(32)(fc_layer)
-    output = keras.layers.Dense(10)(x)
-
-    model = keras.models.Model(input_layers, output)
-
-    model.compile(
-        loss=keras.losses.MSE,
-        optimizer='rmsprop',
-        metrics=[keras.metrics.categorical_accuracy])
-
-    config = model.to_json()
-    loaded_model = model_config.model_from_json(config)
-
-    batch_size = 10
-    timesteps = 1
-
-    values_a = np.arange(10, dtype=np.float32)
-    indices_a = np.zeros((10, 3), dtype=np.int64)
-    indices_a[:, 0] = np.arange(10)
-    inputs_a = sparse_tensor.SparseTensor(indices_a, values_a,
-                                          (batch_size, timesteps, 1))
-
-    values_b = np.zeros(10, dtype=np.str)
-    indices_b = np.zeros((10, 3), dtype=np.int64)
-    indices_b[:, 0] = np.arange(10)
-    inputs_b = sparse_tensor.SparseTensor(indices_b, values_b,
-                                          (batch_size, timesteps, 1))
-
-    with self.cached_session():
-      # Initialize tables for V1 lookup.
-      if not context.executing_eagerly():
-        self.evaluate(lookup_ops.tables_initializer())
-
-      self.assertLen(
-          loaded_model.predict({
-              'a': inputs_a,
-              'b': inputs_b
-          }, steps=1), batch_size)
-
-
-if __name__ == '__main__':
-  test.main()
diff --git a/tensorflow/python/framework/func_graph.py b/tensorflow/python/framework/func_graph.py
index 6b358a3c51a..b0f8821b17f 100644
--- a/tensorflow/python/framework/func_graph.py
+++ b/tensorflow/python/framework/func_graph.py
@@ -365,35 +365,33 @@ class FuncGraph(ops.Graph):
     @tf_contextlib.contextmanager
     def inner_cm():
       """Context manager for copying distribute.Strategy scope information."""
-      graph = ops.get_default_graph()
       # pylint: disable=protected-access
       # TODO(b/112906995, nareshmodi): distribution strategy depends on
       # inheriting this stack from the default graph even in eager mode. Maybe
       # it should be part of the eager context? This would also allow us to
       # remove a get_default_graph() call from the function cache lookup.
+      graph = ops.get_default_graph()
       old_strategy_stack = self._distribution_strategy_stack
       self._distribution_strategy_stack = list(
           graph._distribution_strategy_stack)
-      uses_distribution_strategy = (
-          self._distribution_strategy_stack and
-          self._distribution_strategy_stack[-1].strategy.extended
-          ._retrace_functions_for_each_device)
+
       # We ignore device placements from any outer scopes while tracing the
       # function when possible, to avoid hard-coding them in the function
       # graph. "Default" placements come from the PartitionedCallOp's placement,
       # so that the same trace of the Python function may be placed on several
       # different devices and saved functions may be placed on new devices when
       # restored.
+      # However, we need to preserve the outer device stack in the following
+      # cases in non eager context:
+      # 1. device stack is callable
+      # 2. When using distribution strategy with legacy graph mode.
       old_device_stack = self._device_function_stack
-      if context.executing_eagerly():
-        if uses_distribution_strategy:
-          self._device_function_stack = self._device_function_stack.copy()
-          self._add_device_to_stack(context.context().device_name)
-      else:
-        if (uses_distribution_strategy or
-            device_stack_has_callable(graph._device_function_stack)):
-          # Hard-code devices from device functions in the function body
-          self._device_function_stack = graph._device_function_stack.copy()
+      if (not context.executing_eagerly() and
+          (device_stack_has_callable(graph._device_function_stack) or
+           (self._distribution_strategy_stack and
+            not ops.executing_eagerly_outside_functions()))):
+        # Hard-code devices from device functions in the function body
+        self._device_function_stack = graph._device_function_stack.copy()
 
       old_creator_stack = self._variable_creator_stack
       self._variable_creator_stack = graph._variable_creator_stack
diff --git a/tensorflow/python/keras/BUILD b/tensorflow/python/keras/BUILD
index 67fa0c18ebd..4e88494a374 100755
--- a/tensorflow/python/keras/BUILD
+++ b/tensorflow/python/keras/BUILD
@@ -86,6 +86,7 @@ py_library(
         "//tensorflow/python/distribute:distribute_coordinator",
         "//tensorflow/python/distribute:distribute_lib",
         "//tensorflow/python/distribute:multi_worker_util",
+        "//tensorflow/python/keras/engine:keras_tensor",
     ],
 )
 
diff --git a/tensorflow/python/keras/backend.py b/tensorflow/python/keras/backend.py
index b29b519477c..10d36bc09da 100644
--- a/tensorflow/python/keras/backend.py
+++ b/tensorflow/python/keras/backend.py
@@ -49,8 +49,10 @@ from tensorflow.python.framework import func_graph
 from tensorflow.python.framework import ops
 from tensorflow.python.framework import sparse_tensor
 from tensorflow.python.framework import tensor_shape
+from tensorflow.python.framework import tensor_spec
 from tensorflow.python.framework import tensor_util
 from tensorflow.python.keras import backend_config
+from tensorflow.python.keras.engine import keras_tensor
 from tensorflow.python.ops import array_ops
 from tensorflow.python.ops import clip_ops
 from tensorflow.python.ops import control_flow_ops
@@ -1070,6 +1072,8 @@ def is_keras_tensor(x):
   True
 
   """
+  if keras_tensor.keras_tensors_enabled():
+    return isinstance(x, keras_tensor.KerasTensor)
   if not isinstance(x,
                     (ops.Tensor, variables_module.Variable,
                      sparse_tensor.SparseTensor, ragged_tensor.RaggedTensor)):
@@ -1120,35 +1124,56 @@ def placeholder(shape=None,
     raise ValueError(
         'Cannot set both sparse and ragged to True when creating a placeholder.'
     )
-
   if dtype is None:
     dtype = floatx()
   if not shape:
     if ndim:
       shape = (None,) * ndim
-  with get_graph().as_default():
+  if keras_tensor.keras_tensors_enabled():
+    spec = tensor_spec.TensorSpec(
+        shape=shape, dtype=dtype, name=name)
     if sparse:
-      x = array_ops.sparse_placeholder(dtype, shape=shape, name=name)
+      spec = sparse_tensor.SparseTensorSpec(
+          shape=shape, dtype=dtype)
     elif ragged:
       ragged_rank = 0
       for i in range(1, len(shape)):
-        if shape[i] is None:
+        # Hacky because could be tensorshape or tuple maybe?
+        # Or just tensorshape?
+        if shape[i] is None or (
+            hasattr(shape[i], 'value') and
+            shape[i].value is None):
           ragged_rank = i
-      type_spec = ragged_tensor.RaggedTensorSpec(
+      spec = ragged_tensor.RaggedTensorSpec(
           shape=shape, dtype=dtype, ragged_rank=ragged_rank)
-      def tensor_spec_to_placeholder(tensorspec):
-        return array_ops.placeholder(tensorspec.dtype, tensorspec.shape)
-      x = nest.map_structure(tensor_spec_to_placeholder, type_spec,
-                             expand_composites=True)
-    else:
-      x = array_ops.placeholder(dtype, shape=shape, name=name)
+
+    x = keras_tensor.KerasTensor(spec, name=name)
+  else:
+    with get_graph().as_default():
+      if sparse:
+        x = array_ops.sparse_placeholder(dtype, shape=shape, name=name)
+      elif ragged:
+        ragged_rank = 0
+        for i in range(1, len(shape)):
+          if shape[i] is None:
+            ragged_rank = i
+        type_spec = ragged_tensor.RaggedTensorSpec(
+            shape=shape, dtype=dtype, ragged_rank=ragged_rank)
+        def tensor_spec_to_placeholder(tensorspec):
+          return array_ops.placeholder(tensorspec.dtype, tensorspec.shape)
+        x = nest.map_structure(tensor_spec_to_placeholder, type_spec,
+                               expand_composites=True)
+      else:
+        x = array_ops.placeholder(dtype, shape=shape, name=name)
 
   if context.executing_eagerly():
     # Add keras_history connectivity information to the placeholder
     # when the placeholder is built in a top-level eager context
     # (intended to be used with keras.backend.function)
     from tensorflow.python.keras.engine import input_layer  # pylint: disable=g-import-not-at-top
-    return input_layer.Input(tensor=x)
+    x = input_layer.Input(tensor=x)
+    if keras_tensor.keras_tensors_enabled():
+      x._is_backend_placeholder = True
 
   return x
 
@@ -1163,6 +1188,8 @@ def is_placeholder(x):
       Boolean.
   """
   try:
+    if keras_tensor.keras_tensors_enabled():
+      return hasattr(x, '_is_backend_placeholder')
     if isinstance(x, composite_tensor.CompositeTensor):
       flat_components = nest.flatten(x, expand_composites=True)
       return py_any(is_placeholder(c) for c in flat_components)
@@ -1644,15 +1671,41 @@ def update_sub(x, decrement):
 
 @keras_export('keras.backend.moving_average_update')
 def moving_average_update(x, value, momentum):
-  """Compute the moving average of a variable.
+  """Compute the exponential moving average of a value.
+
+  The moving average 'x' is updated with 'value' following:
+
+  ```
+  x = x * momentum + value * (1 - momentum)
+  ```
+
+  For example:
+
+  >>> x = tf.Variable(0.0)
+  >>> momentum=0.9
+  >>> moving_average_update(x, value = 2.0, momentum=momentum).numpy()
+  >>> x.numpy()
+  0.2
+
+  The result will be biased towards the initial value of the variable.
+
+  If the variable was initialized to zero, you can divide by
+  `1 - momentum ** num_updates` to debias it (Section 3 of
+  [Kingma et al., 2015](https://arxiv.org/abs/1412.6980)):
+
+  >>> num_updates = 1.0
+  >>> x_zdb = x/(1 - momentum**num_updates)
+  >>> x_zdb.numpy()
+  2.0
 
   Arguments:
-      x: A Variable.
-      value: A tensor with the same shape as `variable`.
+      x: A Variable, the moving average.
+      value: A tensor with the same shape as `x`, the new value to be
+        averaged in.
       momentum: The moving average momentum.
 
   Returns:
-      An Operation to update the variable.
+      The updated variable.
   """
   zero_debias = not tf2.enabled()
   return moving_averages.assign_moving_average(
@@ -3817,10 +3870,10 @@ def function(inputs, outputs, updates=None, name=None, **kwargs):
   """
   if ops.executing_eagerly_outside_functions():
     if kwargs:
-      raise ValueError('Session keyword arguments are not support during '
+      raise ValueError('Session keyword arguments are not supported during '
                        'eager execution. You passed: %s' % (kwargs,))
     if updates:
-      raise ValueError('`updates` argument is not support during '
+      raise ValueError('`updates` argument is not supported during '
                        'eager execution. You passed: %s' % (updates,))
     from tensorflow.python.keras import models  # pylint: disable=g-import-not-at-top
     from tensorflow.python.keras.utils import tf_utils  # pylint: disable=g-import-not-at-top
diff --git a/tensorflow/python/keras/callbacks.py b/tensorflow/python/keras/callbacks.py
index 3d585190867..1bca5419774 100644
--- a/tensorflow/python/keras/callbacks.py
+++ b/tensorflow/python/keras/callbacks.py
@@ -2000,7 +2000,10 @@ class TensorBoard(Callback, version_utils.TensorBoardVersionSelector):
     for layer in self.model.layers:
       if isinstance(layer, embeddings.Embedding):
         embedding = config.embeddings.add()
-        embedding.tensor_name = layer.name + '/.ATTRIBUTES/VARIABLE_VALUE'
+        # Embeddings are always the first layer, so this naming should be
+        # consistent in any keras models checkpoints.
+        name = 'layer_with_weights-0/embeddings/.ATTRIBUTES/VARIABLE_VALUE'
+        embedding.tensor_name = name
 
         if self.embeddings_metadata is not None:
           if isinstance(self.embeddings_metadata, str):
diff --git a/tensorflow/python/keras/callbacks_test.py b/tensorflow/python/keras/callbacks_test.py
index 2f1256ee3ee..28f85304688 100644
--- a/tensorflow/python/keras/callbacks_test.py
+++ b/tensorflow/python/keras/callbacks_test.py
@@ -1975,12 +1975,12 @@ class TestTensorBoardV2(keras_parameterized.TestCase):
         callbacks=[tb_cbk])
 
     with open(os.path.join(self.logdir, 'projector_config.pbtxt')) as f:
-      self.assertEqual(
-          f.readlines(), [
-              'embeddings {\n',
-              '  tensor_name: "test_embedding/.ATTRIBUTES/VARIABLE_VALUE"\n',
-              '  metadata_path: "metadata.tsv"\n',
-              '}\n'])
+      self.assertEqual(f.readlines(), [
+          'embeddings {\n',
+          ('  tensor_name: '
+           '"layer_with_weights-0/embeddings/.ATTRIBUTES/VARIABLE_VALUE"\n'),
+          '  metadata_path: "metadata.tsv"\n', '}\n'
+      ])
 
   def test_custom_summary(self):
     if not context.executing_eagerly():
diff --git a/tensorflow/python/keras/combinations.py b/tensorflow/python/keras/combinations.py
index 1d8e6a91930..08fb9c64000 100644
--- a/tensorflow/python/keras/combinations.py
+++ b/tensorflow/python/keras/combinations.py
@@ -61,6 +61,10 @@ def keras_model_type_combinations():
   return combinations.combine(model_type=KERAS_MODEL_TYPES)
 
 
+def keras_tensor_combinations():
+  return combinations.combine(use_keras_tensors=['True', 'False'])
+
+
 class KerasModeCombination(test_combinations.TestCombination):
   """Combination for Keras test mode.
 
@@ -100,11 +104,32 @@ class KerasModelTypeCombination(test_combinations.TestCombination):
     return [test_combinations.OptionalParameter('model_type')]
 
 
+class KerasTensorCombination(test_combinations.TestCombination):
+  """Combination for whether KerasTensors are being used or not.
+
+  It by default includes `True` and `False`:
+  running Keras's functional API with KerasTensors
+  as the inputs, and without.
+  """
+
+  def context_managers(self, kwargs):
+    use_keras_tensors = kwargs.pop('use_keras_tensors', None)
+
+    if use_keras_tensors is not None:
+      return [testing_utils.use_keras_tensors_scope(use_keras_tensors)]
+    else:
+      return []
+
+  def parameter_modifiers(self):
+    return [test_combinations.OptionalParameter('use_keras_tensors')]
+
+
 _defaults = combinations.generate.keywords['test_combinations']
 generate = functools.partial(
     combinations.generate,
     test_combinations=_defaults +
-    (KerasModeCombination(), KerasModelTypeCombination()))
+    (KerasModeCombination(), KerasModelTypeCombination(),
+     KerasTensorCombination()))
 combine = test_combinations.combine
 times = test_combinations.times
 NamedObject = test_combinations.NamedObject
diff --git a/tensorflow/python/keras/distribute/multi_worker_callback_tf2_test.py b/tensorflow/python/keras/distribute/multi_worker_callback_tf2_test.py
index 8daa46f6ea3..9c7756e3f5c 100644
--- a/tensorflow/python/keras/distribute/multi_worker_callback_tf2_test.py
+++ b/tensorflow/python/keras/distribute/multi_worker_callback_tf2_test.py
@@ -195,6 +195,7 @@ class KerasCallbackMultiProcessTest(parameterized.TestCase, test.TestCase):
         if 'Interrupting!' not in str(e):
           raise
 
+      multi_process_runner.barrier().wait()
       backup_filepath = os.path.join(bar_dir, 'checkpoint')
       test_obj.assertTrue(file_io.file_exists(backup_filepath))
       test_obj.assertTrue(file_io.file_exists(saving_filepath))
@@ -208,6 +209,7 @@ class KerasCallbackMultiProcessTest(parameterized.TestCase, test.TestCase):
               callbacks.BackupAndRestore(backup_dir=bar_dir),
               AssertCallback()
           ])
+      multi_process_runner.barrier().wait()
       test_obj.assertFalse(file_io.file_exists(backup_filepath))
       test_obj.assertTrue(file_io.file_exists(saving_filepath))
 
diff --git a/tensorflow/python/keras/engine/BUILD b/tensorflow/python/keras/engine/BUILD
index 231ab7661f0..c64e38122e9 100644
--- a/tensorflow/python/keras/engine/BUILD
+++ b/tensorflow/python/keras/engine/BUILD
@@ -23,7 +23,6 @@ py_library(
         "compile_utils.py",
         "functional.py",
         "input_layer.py",
-        "node.py",
         "partial_batch_padding_handler.py",
         "saving.py",
         "sequential.py",
@@ -41,6 +40,8 @@ py_library(
         ":base_preprocessing_layer",
         ":data_adapter",
         ":input_spec",
+        ":keras_tensor",
+        ":node",
         "//tensorflow/python:composite_tensor_utils",
         "//tensorflow/python:py_checkpoint_reader",
         "//tensorflow/python/data",
@@ -110,31 +111,56 @@ py_library(
     srcs_version = "PY2AND3",
     deps = [
         ":base_layer_utils",
-        "//tensorflow/core:protos_all_py",
+        ":input_spec",
+        ":node",
+        "//third_party/py/numpy",
+        "@six_archive//:six",
+        "//tensorflow/python:array_ops",
+        "//tensorflow/python:auto_control_deps",
         "//tensorflow/python:constant_op",
+        "//tensorflow/python:dtypes",
+        "//tensorflow/python:errors",
+        "//tensorflow/python:framework_ops",
+        "//tensorflow/python:func_graph",
+        "//tensorflow/python:math_ops",
+        "//tensorflow/python:platform",
+        "//tensorflow/python:resource_variable_ops",
+        "//tensorflow/python:sparse_tensor",
+        "//tensorflow/python:tensor_spec",
+        "//tensorflow/python:tensor_util",
         "//tensorflow/python:tf2",
-        "//tensorflow/python/data",
-        "//tensorflow/python/distribute:distribute_coordinator",
+        "//tensorflow/python:tf_export",
+        "//tensorflow/python:util",
+        "//tensorflow/python:variables",
+        "//tensorflow/python/autograph/core",
+        "//tensorflow/python/autograph/impl",
         "//tensorflow/python/distribute:distribute_lib",
-        "//tensorflow/python/distribute:input_lib",
-        "//tensorflow/python/distribute:reduce_util",
         "//tensorflow/python/distribute:sharded_variable",
+        "//tensorflow/python/eager:context",
+        "//tensorflow/python/eager:execute",
+        "//tensorflow/python/eager:function",
         "//tensorflow/python/eager:monitoring",
         "//tensorflow/python/keras:backend",
         "//tensorflow/python/keras:constraints",
+        "//tensorflow/python/keras:initializers",
+        # TODO(keras-team): Fix the cyclar deps between layer and metrics.
+        # "//tensorflow/python/keras:metrics",
         "//tensorflow/python/keras:regularizers",
-        "//tensorflow/python/keras/distribute",
         "//tensorflow/python/keras/mixed_precision/experimental:autocast_variable",
         "//tensorflow/python/keras/mixed_precision/experimental:loss_scale_optimizer",
         "//tensorflow/python/keras/mixed_precision/experimental:policy",
+        "//tensorflow/python/keras/saving",
         "//tensorflow/python/keras/utils:generic_utils",
         "//tensorflow/python/keras/utils:layer_utils",
         "//tensorflow/python/keras/utils:tf_utils",
         "//tensorflow/python/keras/utils:version_utils",
         "//tensorflow/python/module",
+        "//tensorflow/python/ops/ragged:ragged_tensor",
+        "//tensorflow/python/training/tracking",
+        "//tensorflow/python/training/tracking:base",
         "//tensorflow/python/training/tracking:data_structures",
+        "//tensorflow/python/training/tracking:layer_utils",
         "//tensorflow/tools/docs:doc_controls",
-        "@six_archive//:six",
     ],
 )
 
@@ -164,6 +190,18 @@ py_library(
     ],
 )
 
+py_library(
+    name = "keras_tensor",
+    srcs = ["keras_tensor.py"],
+    srcs_version = "PY2AND3",
+    deps = [
+        "//tensorflow/python:dtypes",
+        "//tensorflow/python:lib",
+        "//tensorflow/python:tensor_spec",
+        "@six_archive//:six",
+    ],
+)
+
 py_library(
     name = "base_preprocessing_layer",
     srcs = [
@@ -181,6 +219,22 @@ py_library(
     ],
 )
 
+py_library(
+    name = "node",
+    srcs = ["node.py"],
+    srcs_version = "PY2AND3",
+    deps = [
+        ":base_layer_utils",
+        "//tensorflow/python:framework_ops",
+        "//tensorflow/python:platform",
+        "//tensorflow/python:tensor_util",
+        "//tensorflow/python:util",
+        "//tensorflow/python/keras:backend",
+        "//tensorflow/python/keras/utils:tf_utils",
+        "//third_party/py/numpy",
+    ],
+)
+
 tf_py_test(
     name = "base_layer_utils_test",
     srcs = ["base_layer_utils_test.py"],
@@ -426,6 +480,24 @@ tf_py_test(
     ],
 )
 
+tf_py_test(
+    name = "feature_columns_integration_test",
+    size = "medium",
+    srcs = ["feature_columns_integration_test.py"],
+    python_version = "PY3",
+    tags = [
+        "nomac",  # TODO(mihaimaruseac): b/127695564
+        "notsan",
+    ],
+    deps = [
+        "//tensorflow/python:client_testlib",
+        "//tensorflow/python/feature_column:feature_column_py",
+        "//tensorflow/python/keras",
+        "//third_party/py/numpy",
+        "@absl_py//absl/testing:parameterized",
+    ],
+)
+
 tf_py_test(
     name = "training_eager_test",
     size = "medium",
diff --git a/tensorflow/python/keras/engine/base_layer.py b/tensorflow/python/keras/engine/base_layer.py
index 9a79c4fd1de..561355f5f57 100644
--- a/tensorflow/python/keras/engine/base_layer.py
+++ b/tensorflow/python/keras/engine/base_layer.py
@@ -55,6 +55,7 @@ from tensorflow.python.keras import initializers
 from tensorflow.python.keras import regularizers
 from tensorflow.python.keras.engine import base_layer_utils
 from tensorflow.python.keras.engine import input_spec
+from tensorflow.python.keras.engine import keras_tensor
 from tensorflow.python.keras.engine import node as node_module
 from tensorflow.python.keras.mixed_precision.experimental import autocast_variable
 from tensorflow.python.keras.mixed_precision.experimental import loss_scale_optimizer
@@ -316,7 +317,11 @@ class Layer(module.Module, version_utils.LayerVersionSelector):
     self._saved_model_inputs_spec = None
     # Provides information about which inputs are compatible with the layer.
     self._input_spec = None
-    self.supports_masking = False
+
+    # `Layer.compute_mask` will be called at the end of `Layer.__call__` if
+    # `Layer.compute_mask` is overridden, or if the `Layer` subclass sets
+    # `self.supports_masking=True`.
+    self._supports_masking = not generic_utils.is_default(self.compute_mask)
 
     self._init_set_name(name)
     self._activity_regularizer = regularizers.get(
@@ -393,11 +398,6 @@ class Layer(module.Module, version_utils.LayerVersionSelector):
     # might want to turn it off, like Sequential model.
     self._auto_track_sub_layers = True
 
-    # Will compute masking if `compute_mask` is overridden or `supports_masking`
-    # is set.
-    self._compute_mask_overridden = (not getattr(self.compute_mask,
-                                                 '_is_default', False))
-
   @trackable.no_automatic_dependency_tracking
   @generic_utils.default
   def build(self, input_shape):
@@ -701,7 +701,7 @@ class Layer(module.Module, version_utils.LayerVersionSelector):
       # with the shape the Layer will be called on (these users will have to
       # implement `compute_output_shape` themselves).
       self._maybe_build(input_shape)
-      with func_graph.FuncGraph('graph').as_default():
+      with func_graph.FuncGraph(str(self.name) + '_scratch_graph').as_default():
         input_shape = tf_utils.convert_shapes(input_shape, to_tuples=False)
         def _make_placeholder_like(shape):
           ph = backend.placeholder(shape=shape, dtype=self.dtype)
@@ -760,6 +760,76 @@ class Layer(module.Module, version_utils.LayerVersionSelector):
         lambda s: tensor_spec.TensorSpec(dtype=dtype, shape=s),
         output_shape)
 
+  def _keras_tensor_symbolic_call(self, inputs, input_masks, args, kwargs):
+    if self.dynamic:
+      # We will use static shape inference to return symbolic tensors
+      # matching the specifications of the layer outputs.
+      # Since `self.dynamic` is True, we will never attempt to
+      # run the underlying TF graph (which is disconnected).
+      # TODO(fchollet): consider py_func as an alternative, which
+      # would enable us to run the underlying graph if needed.
+      input_signature = nest.map_structure(
+          lambda x: tensor_spec.TensorSpec(shape=x.shape, dtype=x.dtype),
+          inputs)
+      output_signature = self.compute_output_signature(input_signature)
+      return nest.map_structure(keras_tensor.KerasTensor, output_signature)
+    else:
+      return self._infer_output_signature(inputs, args, kwargs, input_masks)
+
+  def _infer_output_signature(self, inputs, args, kwargs, input_masks):
+    """TODO(kaftan): Docstring."""
+
+    call_fn = self.call
+    # Wrapping `call` function in autograph to allow for dynamic control
+    # flow and control dependencies in call. We are limiting this to
+    # subclassed layers as autograph is strictly needed only for
+    # subclassed layers and models.
+    # tf_convert will respect the value of autograph setting in the
+    # enclosing tf.function, if any.
+    if (base_layer_utils.is_subclassed(self) and
+        not base_layer_utils.from_saved_model(self)):
+      call_fn = autograph.tf_convert(self.call, ag_ctx.control_status_ctx())
+
+    # We enter a scratch graph and build placeholder inputs inside of it that
+    # match the input args.
+    # We then call the layer inside of the scratch graph to identify the
+    # output signatures, then we build KerasTensors corresponding to those
+    # outputs.
+    scratch_graph = func_graph.FuncGraph(str(self.name) + '_scratch_graph')
+    with scratch_graph.as_default():
+      inputs = nest.map_structure(
+          keras_tensor.keras_tensor_to_placeholder, inputs)
+      args = nest.map_structure(
+          keras_tensor.keras_tensor_to_placeholder, args)
+      kwargs = nest.map_structure(
+          keras_tensor.keras_tensor_to_placeholder, kwargs)
+      input_masks = nest.map_structure(
+          keras_tensor.keras_tensor_to_placeholder, input_masks)
+
+      inputs = self._maybe_cast_inputs(inputs)
+
+      # try:
+      with backend.name_scope(self._name_scope()):
+        with ops.enable_auto_cast_variables(self._compute_dtype_object):
+          # Build layer if applicable (if the `build` method has been
+          # overridden).
+          # TODO(kaftan): do we maybe_build here, or have we already done it?
+          self._maybe_build(inputs)
+          outputs = call_fn(inputs, *args, **kwargs)
+
+        self._handle_activity_regularization(inputs, outputs)
+      self._set_mask_metadata(inputs, outputs, input_masks,
+                              build_graph=False)
+
+    outputs = nest.map_structure(keras_tensor.keras_tensor_from_tensor, outputs)
+    if hasattr(self, '_set_inputs') and not self.inputs:
+      # TODO(kaftan): figure out if we ned to do this at all
+      # Subclassed network: explicitly set metadata normally set by
+      # a call to self._set_inputs().
+      self._set_inputs(inputs, outputs)
+    del scratch_graph
+    return outputs
+
   @generic_utils.default
   def compute_mask(self, inputs, mask=None):  # pylint: disable=unused-argument
     """Computes an output mask tensor.
@@ -772,7 +842,7 @@ class Layer(module.Module, version_utils.LayerVersionSelector):
         None or a tensor (or list of tensors,
             one per output tensor of the layer).
     """
-    if not self.supports_masking:
+    if not self._supports_masking:
       if any(m is not None for m in nest.flatten(mask)):
         raise TypeError('Layer ' + self.name + ' does not support masking, '
                         'but was passed an input_mask: ' + str(mask))
@@ -811,7 +881,7 @@ class Layer(module.Module, version_utils.LayerVersionSelector):
       raise RuntimeError(
           'You must call `super().__init__()` in the layer constructor.')
 
-    # 'inputs` (the first arg in the method spec) is special cased in
+    # `inputs` (the first arg in the method spec) is special cased in
     # layer call due to historical reasons.
     # This special casing currently takes the form of:
     # - 'inputs' must be explicitly passed. A layer cannot have zero arguments,
@@ -820,35 +890,98 @@ class Layer(module.Module, version_utils.LayerVersionSelector):
     # - implicit masks / mask metadata are only collected from 'inputs`
     # - Layers are built using shape info from 'inputs' only
     # - input_spec compatibility is only checked against `inputs`
-    # - checking if a layer has ragged tensor support is only done against
-    #   `inputs`
     # - mixed precision casting (autocast) is only applied to `inputs`,
     #   not to any other argument.
-    # - configuring the Functional API SavedModel saving spec for deciding what
-    #   should be serialized during SavedModel saving
+    # - setting the SavedModel saving spec.
     inputs, args, kwargs = self._split_out_first_arg(args, kwargs)
-
-    call_context = base_layer_utils.call_context()
-    in_call = call_context.in_call
     input_list = nest.flatten(inputs)
 
-    # We will attempt to build a TF graph if & only if all inputs are symbolic.
-    # This is always the case in graph mode. It can also be the case in eager
-    # mode when all inputs can be traced back to `keras.Input()` (when building
-    # models using the functional API).
-    # TODO(kaftan): make this not special case inputs. Instead
-    # build a functional api model if *any* *arg or **kwarg is symbolic,
-    # even if part of the data structure in that arg is not symbolic.
-    build_graph = tf_utils.are_all_symbolic_tensors(input_list)
+    # Functional Model construction mode is invoked when `Layer`s are called on
+    # symbolic `KerasTensor`s, i.e.:
+    # >> inputs = tf.keras.Input(10)
+    # >> outputs = MyLayer()(inputs)  # Functional construction mode.
+    # >> model = tf.keras.Model(inputs, outputs)
+    if _in_functional_construction_mode(inputs, args, kwargs, input_list):
+      return self._functional_construction_call(inputs, args, kwargs,
+                                                input_list)
+
+    # Maintains info about the `Layer.call` stack.
+    call_context = base_layer_utils.call_context()
 
     # Accept NumPy and scalar inputs by converting to Tensors.
     if any(isinstance(x, (np.ndarray, float, int)) for x in input_list):
+      inputs = nest.map_structure(_convert_numpy_or_python_types, inputs)
+      input_list = nest.flatten(inputs)
+
+    # Handle `mask` propagation from previous layer to current layer. Masks can
+    # be propagated explicitly via the `mask` argument, or implicitly via
+    # setting the `_keras_mask` attribute on the inputs to a Layer. Masks passed
+    # explicitly take priority.
+    input_masks, mask_is_implicit = self._get_input_masks(
+        inputs, input_list, args, kwargs)
+    if self._expects_mask_arg and mask_is_implicit:
+      kwargs['mask'] = input_masks
+
+    # Training mode for `Layer.call` is set via (in order of priority):
+    # (1) The `training` argument passed to this `Layer.call`.
+    # (2) The training mode of an outer `Layer.call`.
+    # (3) The default mode set by `tf.keras.backed.set_learning_phase` (if set).
+    training_mode = self._set_training_mode(args, kwargs, call_context)
+
+    # Losses are cleared for all sublayers on the outermost `Layer.call`.
+    # Losses are not cleared on inner `Layer.call`s, because sublayers can be
+    # called multiple times.
+    if not call_context.in_call:
+      self._clear_losses()
+
+    eager = context.executing_eagerly()
+    with call_context.enter(
+        layer=self,
+        inputs=inputs,
+        build_graph=not eager,
+        training=training_mode):
+
+      if self._autocast:
+        inputs = self._maybe_cast_inputs(inputs, input_list)
+
+      if eager:
+        call_fn = self.call
+        name_scope = self._name
+      else:
+        input_spec.assert_input_compatibility(self.input_spec, inputs,
+                                              self.name)
+        name_scope = self._name_scope()  # Avoid autoincrementing.
+        call_fn = self._autographed_call()
+
+      with ops.name_scope_v2(name_scope):
+        if not self.built:
+          self._maybe_build(inputs)
+
+        with ops.enable_auto_cast_variables(self._compute_dtype_object):
+          outputs = call_fn(inputs, *args, **kwargs)
+
+        if self._activity_regularizer:
+          self._handle_activity_regularization(inputs, outputs)
+        if self._supports_masking:
+          self._set_mask_metadata(inputs, outputs, input_masks, not eager)
+        if self._saved_model_inputs_spec is None:
+          self._set_save_spec(inputs)
+
+        return outputs
+
+  def _functional_construction_call(self, inputs, args, kwargs, input_list):
+    call_context = base_layer_utils.call_context()
+
+    # Accept NumPy and scalar inputs by converting to Tensors.
+    if any(isinstance(x, (np.ndarray, float, int)) for x in input_list):
+
       def _convert_non_tensor(x):
         # Don't call `ops.convert_to_tensor_v2` on all `inputs` because
         # `SparseTensors` can't be converted to `Tensor`.
         if isinstance(x, (np.ndarray, float, int)):
           return ops.convert_to_tensor_v2(x)
         return x
+
       inputs = nest.map_structure(_convert_non_tensor, inputs)
       input_list = nest.flatten(inputs)
 
@@ -888,125 +1021,168 @@ class Layer(module.Module, version_utils.LayerVersionSelector):
           training_value = math_ops.cast(training_value, dtypes.bool)
         else:
           training_value = bool(training_value)
-        args, kwargs = self._set_call_arg_value(
-            'training', training_value, args, kwargs)
+        args, kwargs = self._set_call_arg_value('training', training_value,
+                                                args, kwargs)
         training_arg_passed_by_framework = True
 
+    if keras_tensor.keras_tensors_enabled():
+      with call_context.enter(
+          layer=self, inputs=inputs, build_graph=True, training=training_value):
+        # Check input assumptions set after layer building, e.g. input shape.
+        outputs = self._keras_tensor_symbolic_call(
+            inputs, input_masks, args, kwargs)
+
+        if outputs is None:
+          raise ValueError('A layer\'s `call` method should return a '
+                           'Tensor or a list of Tensors, not None '
+                           '(layer: ' + self.name + ').')
+        if training_arg_passed_by_framework:
+          args, kwargs = self._set_call_arg_value(
+              'training', None, args, kwargs, pop_kwarg_if_none=True)
+        if mask_arg_passed_by_framework:
+          kwargs.pop('mask')
+        # Node connectivity does not special-case the first argument.
+        outputs = self._set_connectivity_metadata((inputs,) + args, kwargs,
+                                                  outputs)
+        return outputs
+
     # Only create Keras history if at least one tensor originates from a
     # `keras.Input`. Otherwise this Layer may be being used outside the Keras
     # framework.
     # TODO(kaftan): make this not special case inputs
-    if build_graph and base_layer_utils.needs_keras_history(inputs):
+    if base_layer_utils.needs_keras_history(inputs):
       base_layer_utils.create_keras_history(inputs)
 
-    with call_context.enter(self, inputs, build_graph, training_value):
-      # Check input assumptions set after layer building, e.g. input shape.
-      if build_graph:
-        # Losses are cleared for all Layers when the outermost layer is called.
-        # Losses are not cleared each time an inner layer is called, bc inner
-        # Layers can be reused in a Model.
-        if not in_call and base_layer_utils.is_in_tf_function():
-          self._clear_losses()
+    with call_context.enter(
+        layer=self, inputs=inputs, build_graph=True, training=training_value):
+      # Symbolic execution on symbolic tensors. We will attempt to build
+      # the corresponding TF subgraph inside `backend.get_graph()`
+      # TODO(reedwm): We should assert input compatibility after the inputs
+      # are casted, not before.
+      input_spec.assert_input_compatibility(self.input_spec, inputs, self.name)
+      graph = backend.get_graph()
+      # Use `self._name_scope()` to avoid auto-incrementing the name.
+      with graph.as_default(), backend.name_scope(self._name_scope()):
+        # Build layer if applicable (if the `build` method has been
+        # overridden).
+        self._maybe_build(inputs)
+        cast_inputs = self._maybe_cast_inputs(inputs, input_list)
 
-        # Symbolic execution on symbolic tensors. We will attempt to build
-        # the corresponding TF subgraph inside `backend.get_graph()`
-        # TODO(reedwm): We should assert input compatibility after the inputs
-        # are casted, not before.
-        input_spec.assert_input_compatibility(self.input_spec, inputs,
-                                              self.name)
-        graph = backend.get_graph()
-        # Use `self._name_scope()` to avoid auto-incrementing the name.
-        with graph.as_default(), backend.name_scope(self._name_scope()):
-          # Build layer if applicable (if the `build` method has been
-          # overridden).
-          self._maybe_build(inputs)
-          cast_inputs = self._maybe_cast_inputs(inputs, input_list)
-
-          if not self.dynamic:
-            # Wrapping `call` function in autograph to allow for dynamic control
-            # flow and control dependencies in call. We are limiting this to
-            # subclassed layers as autograph is strictly needed only for
-            # subclassed layers and models.
-            # tf_convert will respect the value of autograph setting in the
-            # enclosing tf.function, if any.
-            if (base_layer_utils.is_subclassed(self) and
-                not base_layer_utils.from_saved_model(self)):
-              call_fn = autograph.tf_convert(
-                  self.call, ag_ctx.control_status_ctx())
-            else:
-              call_fn = self.call
-
-            try:
-              with ops.enable_auto_cast_variables(self._compute_dtype_object):
-                # Add auto_control_deps in V2 when they are not already added by
-                # a `tf.function`.
-                if (ops.executing_eagerly_outside_functions() and
-                    not base_layer_utils.is_in_eager_or_tf_function()):
-                  with auto_control_deps.AutomaticControlDependencies() as acd:
-                    outputs = call_fn(cast_inputs, *args, **kwargs)
-                    # Wrap Tensors in `outputs` in `tf.identity` to avoid
-                    # circular dependencies.
-                    outputs = base_layer_utils.mark_as_return(outputs, acd)
-                else:
-                  outputs = call_fn(cast_inputs, *args, **kwargs)
-
-            except errors.OperatorNotAllowedInGraphError as e:
-              raise TypeError('You are attempting to use Python control '
-                              'flow in a layer that was not declared to be '
-                              'dynamic. Pass `dynamic=True` to the class '
-                              'constructor.\nEncountered error:\n"""\n' +
-                              str(e) + '\n"""')
+        if not self.dynamic:
+          # Wrapping `call` function in autograph to allow for dynamic control
+          # flow and control dependencies in call. We are limiting this to
+          # subclassed layers as autograph is strictly needed only for
+          # subclassed layers and models.
+          # tf_convert will respect the value of autograph setting in the
+          # enclosing tf.function, if any.
+          if (base_layer_utils.is_subclassed(self) and
+              not base_layer_utils.from_saved_model(self)):
+            call_fn = autograph.tf_convert(self.call,
+                                           ag_ctx.control_status_ctx())
           else:
-            # We will use static shape inference to return symbolic tensors
-            # matching the specifications of the layer outputs.
-            # Since `self.dynamic` is True, we will never attempt to
-            # run the underlying TF graph (which is disconnected).
-            # TODO(fchollet): consider py_func as an alternative, which
-            # would enable us to run the underlying graph if needed.
-            outputs = self._symbolic_call(inputs)
+            call_fn = self.call
 
-          if outputs is None:
-            raise ValueError('A layer\'s `call` method should return a '
-                             'Tensor or a list of Tensors, not None '
-                             '(layer: ' + self.name + ').')
-          # TODO(kaftan): This should be 'any' and check all args
-          if base_layer_utils.have_all_keras_metadata(inputs):
-            if training_arg_passed_by_framework:
-              args, kwargs = self._set_call_arg_value(
-                  'training', None, args, kwargs, pop_kwarg_if_none=True)
-            if mask_arg_passed_by_framework:
-              kwargs.pop('mask')
-            # Node connectivity does not special-case the first argument.
-            outputs = self._set_connectivity_metadata((inputs,) + args, kwargs,
-                                                      outputs)
-          self._handle_activity_regularization(inputs, outputs)
-          self._set_mask_metadata(inputs, outputs, input_masks, build_graph)
-          if hasattr(self, '_set_inputs') and not self.inputs:
-            # Subclassed network: explicitly set metadata normally set by
-            # a call to self._set_inputs().
-            self._set_inputs(cast_inputs, outputs)
-      else:
-        # Eager execution on data tensors.
+          try:
+            with ops.enable_auto_cast_variables(self._compute_dtype_object):
+              # Add auto_control_deps in V2 when they are not already added by
+              # a `tf.function`.
+              if (ops.executing_eagerly_outside_functions() and
+                  not base_layer_utils.is_in_eager_or_tf_function()):
+                with auto_control_deps.AutomaticControlDependencies() as acd:
+                  outputs = call_fn(cast_inputs, *args, **kwargs)
+                  # Wrap Tensors in `outputs` in `tf.identity` to avoid
+                  # circular dependencies.
+                  outputs = base_layer_utils.mark_as_return(outputs, acd)
+              else:
+                outputs = call_fn(cast_inputs, *args, **kwargs)
 
-        # Losses are cleared for all Layers when the outermost layer is called.
-        # Losses are not cleared each time an inner layer is called, bc inner
-        # Layers can be reused in a Model.
-        if not in_call:
-          self._clear_losses()
+          except errors.OperatorNotAllowedInGraphError as e:
+            raise TypeError('You are attempting to use Python control '
+                            'flow in a layer that was not declared to be '
+                            'dynamic. Pass `dynamic=True` to the class '
+                            'constructor.\nEncountered error:\n"""\n' + str(e) +
+                            '\n"""')
+        else:
+          # We will use static shape inference to return symbolic tensors
+          # matching the specifications of the layer outputs.
+          # Since `self.dynamic` is True, we will never attempt to
+          # run the underlying TF graph (which is disconnected).
+          # TODO(fchollet): consider py_func as an alternative, which
+          # would enable us to run the underlying graph if needed.
+          outputs = self._symbolic_call(inputs)
 
-        # In Eager mode, `ops.name_scope_v2` does not autoincrement the name.
-        with ops.name_scope_v2(self.name):
-          self._maybe_build(inputs)
-          cast_inputs = self._maybe_cast_inputs(inputs, input_list)
-          with ops.enable_auto_cast_variables(self._compute_dtype_object):
-            outputs = self.call(cast_inputs, *args, **kwargs)
-          self._handle_activity_regularization(inputs, outputs)
-          self._set_mask_metadata(inputs, outputs, input_masks, build_graph)
-          if self._saved_model_inputs_spec is None:
-            self._set_save_spec(cast_inputs)
+        if outputs is None:
+          raise ValueError('A layer\'s `call` method should return a '
+                           'Tensor or a list of Tensors, not None '
+                           '(layer: ' + self.name + ').')
+        # TODO(kaftan): This should be 'any' and check all args
+        if base_layer_utils.have_all_keras_metadata(inputs):
+          if training_arg_passed_by_framework:
+            args, kwargs = self._set_call_arg_value(
+                'training', None, args, kwargs, pop_kwarg_if_none=True)
+          if mask_arg_passed_by_framework:
+            kwargs.pop('mask')
+          # Node connectivity does not special-case the first argument.
+          outputs = self._set_connectivity_metadata((inputs,) + args, kwargs,
+                                                    outputs)
+        self._handle_activity_regularization(inputs, outputs)
+        self._set_mask_metadata(inputs, outputs, input_masks, True)
+        if hasattr(self, '_set_inputs') and not self.inputs:
+          # Subclassed network: explicitly set metadata normally set by
+          # a call to self._set_inputs().
+          self._set_inputs(cast_inputs, outputs)
 
     return outputs
 
+  def _set_training_mode(self, args, kwargs, call_context):
+    training_mode = None
+    if self._expects_training_arg:
+      # (1) `training` was passed to this `Layer.call`.
+      if self._call_arg_was_passed('training', args, kwargs):
+        training_mode = self._get_call_arg_value('training', args, kwargs)
+      if training_mode is None:
+        call_ctx_training = call_context.training
+        # (2) `training` mode is inferred from an outer `Layer.call`.
+        if call_ctx_training is not None:
+          training_mode = call_ctx_training
+        # (3) User set `tf.keras.backend.set_learning_phase`.
+        elif backend.global_learning_phase_is_set():
+          training_mode = backend.learning_phase()
+          # Ensure value is a `bool` or `tf.bool`.
+          if isinstance(training_mode, bool):
+            pass
+          elif tensor_util.is_tensor(training_mode):
+            training_mode = math_ops.cast(training_mode, dtypes.bool)
+          else:
+            training_mode = bool(training_mode)
+
+        # For case (2) or (3), `training` arg is passed by framework.
+        if training_mode is not None:
+          kwargs['training'] = training_mode
+    else:
+      if 'training' in kwargs:
+        # `training` was passed to this `Layer` but is not needed for
+        # `Layer.call`. It will set the default mode for inner `Layer.call`s.
+        training_mode = kwargs.pop('training')
+      else:
+        # Grab the current `training` mode from any outer `Layer.call`.
+        training_mode = call_context.training
+
+    return training_mode
+
+  def _autographed_call(self):
+    # Wrapping `call` function in autograph to allow for dynamic control
+    # flow and control dependencies in call. We are limiting this to
+    # subclassed layers as autograph is strictly needed only for
+    # subclassed layers and models.
+    # tf_convert will respect the value of autograph setting in the
+    # enclosing tf.function, if any.
+    if (base_layer_utils.is_subclassed(self) and
+        not base_layer_utils.from_saved_model(self)):
+      return autograph.tf_convert(self.call, ag_ctx.control_status_ctx())
+    else:
+      return self.call
+
   @property
   def dtype(self):
     """Dtype used by the weights of the layer, set in the constructor."""
@@ -1017,6 +1193,15 @@ class Layer(module.Module, version_utils.LayerVersionSelector):
     """Name of the layer (string), set in the constructor."""
     return self._name
 
+  @property
+  def supports_masking(self):
+    """Whether this layer supports computing a mask using `compute_mask`."""
+    return self._supports_masking
+
+  @supports_masking.setter
+  def supports_masking(self, value):
+    self._supports_masking = value
+
   @property
   def dynamic(self):
     """Whether the layer is dynamic (eager-only); set in the constructor."""
@@ -1144,6 +1329,10 @@ class Layer(module.Module, version_utils.LayerVersionSelector):
   @property
   @doc_controls.do_not_doc_inheritable
   def updates(self):
+    if (keras_tensor.keras_tensors_enabled()
+        and ops.executing_eagerly_outside_functions()):
+      return []
+
     collected_updates = []
     all_layers = self._flatten_layers()
     with backend.get_graph().as_default():
@@ -1306,10 +1495,12 @@ class Layer(module.Module, version_utils.LayerVersionSelector):
         continue
       if loss is None:
         continue
-      if not tensor_util.is_tensor(loss):
+      if not tensor_util.is_tensor(loss) and not isinstance(
+          loss, keras_tensor.KerasTensor):
         loss = ops.convert_to_tensor_v2(loss, dtype=backend.floatx())
       # TF Functions should take the eager path.
-      if (tf_utils.is_symbolic_tensor(loss) and
+      if ((tf_utils.is_symbolic_tensor(loss) or
+           isinstance(loss, keras_tensor.KerasTensor)) and
           not base_layer_utils.is_in_tf_function()):
         symbolic_losses.append(loss)
       elif tensor_util.is_tensor(loss):
@@ -1325,7 +1516,7 @@ class Layer(module.Module, version_utils.LayerVersionSelector):
 
     self._eager_losses.extend(eager_losses)
 
-    if in_call_context:
+    if in_call_context and not keras_tensor.keras_tensors_enabled():
       for symbolic_loss in symbolic_losses:
         self._losses.append(symbolic_loss)
     else:
@@ -1427,7 +1618,10 @@ class Layer(module.Module, version_utils.LayerVersionSelector):
       raise TypeError('Unknown keyword arguments: ', str(kwargs.keys()))
 
     from_metric_obj = hasattr(value, '_metric_obj')
-    is_symbolic = tf_utils.is_symbolic_tensor(value)
+    if keras_tensor.keras_tensors_enabled():
+      is_symbolic = isinstance(value, keras_tensor.KerasTensor)
+    else:
+      is_symbolic = tf_utils.is_symbolic_tensor(value)
     in_call_context = base_layer_utils.call_context().in_call
 
     if name is None and not from_metric_obj:
@@ -2124,7 +2318,7 @@ class Layer(module.Module, version_utils.LayerVersionSelector):
     """
     return self._dtype_policy.compute_dtype
 
-  def _maybe_cast_inputs(self, inputs, input_list):
+  def _maybe_cast_inputs(self, inputs, input_list=None):
     """Maybe casts the inputs to the compute dtype.
 
     If self._compute_dtype is floating-point, and self_autocast is True,
@@ -2137,6 +2331,9 @@ class Layer(module.Module, version_utils.LayerVersionSelector):
     Returns:
       `inputs`, but tensors may have been casted to self._compute_dtype
     """
+    if not input_list:
+      input_list = nest.flatten(inputs)
+
     compute_dtype_object = self._compute_dtype_object
     should_autocast = (
         self._autocast and compute_dtype_object and
@@ -2270,7 +2467,7 @@ class Layer(module.Module, version_utils.LayerVersionSelector):
     # Many `Layer`s don't need to call `compute_mask`.
     # This method is optimized to do as little work as needed for the common
     # case.
-    if not self.supports_masking and not self._compute_mask_overridden:
+    if not self._supports_masking:
       return
 
     flat_outputs = nest.flatten(outputs)
@@ -2305,8 +2502,7 @@ class Layer(module.Module, version_utils.LayerVersionSelector):
         output._keras_mask._keras_history_checked = True
 
   def _get_input_masks(self, inputs, input_list, args, kwargs):
-    if (not self._expects_mask_arg and not self.supports_masking and
-        not self._compute_mask_overridden):
+    if not self._supports_masking and not self._expects_mask_arg:
       # Input masks only need to be retrieved if they are needed for `call`
       # or `compute_mask`.
       input_masks = None
@@ -3001,6 +3197,28 @@ class AddMetric(Layer):
     return config
 
 
+def _in_functional_construction_mode(inputs, args, kwargs, input_list):  # pylint: disable=unused-argument
+  """Check the arguments to see if we are constructing a functional model."""
+  if keras_tensor.keras_tensors_enabled():
+    # We are constructing a functional model if any of the inputs
+    # are KerasTensors
+    return any(
+        isinstance(tensor, keras_tensor.KerasTensor)
+        for tensor in nest.flatten([inputs, args, kwargs]))
+  else:
+    if context.executing_eagerly():
+      return all(tf_utils.is_symbolic_tensor(t) for t in input_list)
+    else:
+      return (base_layer_utils.is_in_keras_graph() or
+              all(hasattr(t, '_keras_history') for t in input_list))
+
+
+def _convert_numpy_or_python_types(x):
+  if isinstance(x, (np.ndarray, float, int)):
+    return ops.convert_to_tensor_v2(x)
+  return x
+
+
 # Avoid breaking users who directly import this symbol from this file.
 # TODO(fchollet): remove this.
 InputSpec = input_spec.InputSpec  # pylint:disable=invalid-name
diff --git a/tensorflow/python/keras/engine/base_layer_test.py b/tensorflow/python/keras/engine/base_layer_test.py
index 13fb2b28bb7..b861d7e4b5b 100644
--- a/tensorflow/python/keras/engine/base_layer_test.py
+++ b/tensorflow/python/keras/engine/base_layer_test.py
@@ -48,7 +48,6 @@ from tensorflow.python.keras.optimizer_v2 import rmsprop
 from tensorflow.python.keras.utils import tf_utils
 from tensorflow.python.layers import core as legacy_core
 from tensorflow.python.ops import array_ops
-from tensorflow.python.ops import control_flow_ops
 from tensorflow.python.ops import math_ops
 from tensorflow.python.ops import state_ops
 from tensorflow.python.ops import summary_ops_v2
@@ -86,7 +85,9 @@ class InvalidLayer(base_layer.Layer):
 
 class BaseLayerTest(keras_parameterized.TestCase):
 
-  @combinations.generate(combinations.keras_model_type_combinations())
+  @combinations.generate(combinations.times(
+      combinations.keras_model_type_combinations(),
+      combinations.keras_tensor_combinations()))
   def test_dynamic_layer(self):
     model = testing_utils.get_model_from_layers([DynamicLayer(dynamic=True)],
                                                 input_shape=(3,))
@@ -95,16 +96,30 @@ class BaseLayerTest(keras_parameterized.TestCase):
     self.assertEqual(model.run_eagerly, True)
     model.train_on_batch(np.random.random((2, 3)), np.random.random((2, 3)))
 
-  @combinations.generate(combinations.keras_model_type_combinations())
+  @combinations.generate(combinations.times(
+      combinations.keras_model_type_combinations(),
+      combinations.keras_tensor_combinations()))
   def test_dynamic_layer_error(self):
-    with self.assertRaisesRegexp(TypeError,
-                                 'attempting to use Python control flow'):
+    # Functional Models hit the `dyanamic=True` error during construction.
+    # Subclass Models should just throw the original autograph error during
+    # execution.
+    raised_error = False
+    try:
       model = testing_utils.get_model_from_layers([DynamicLayer()],
                                                   input_shape=(3,))
       model.compile(rmsprop.RMSprop(0.001), loss='mse')
       model.train_on_batch(np.random.random((2, 3)), np.random.random((2, 3)))
+    except errors_impl.OperatorNotAllowedInGraphError as e:
+      if 'iterating over `tf.Tensor` is not allowed' in str(e):
+        raised_error = True
+    except TypeError as e:
+      if 'attempting to use Python control flow' in str(e):
+        raised_error = True
+    self.assertTrue(raised_error)
 
-  @combinations.generate(combinations.keras_model_type_combinations())
+  @combinations.generate(combinations.times(
+      combinations.keras_model_type_combinations(),
+      combinations.keras_tensor_combinations()))
   def test_dynamic_layer_error_running_in_graph_mode(self):
     with ops.get_default_graph().as_default():
       model = testing_utils.get_model_from_layers([DynamicLayer(dynamic=True)],
@@ -145,11 +160,6 @@ class BaseLayerTest(keras_parameterized.TestCase):
     self.assertEqual(layer.build_counter, 1)
     self.assertEqual(layer.build_shape.as_list(), [None, 10])
 
-  def test_eager_switch_case_input(self):
-    task = input_layer.Input(shape=(), dtype=dtypes.int32)
-    control_flow_ops.switch_case(
-        task[0], [lambda: constant_op.constant(1.0) for _ in range(10)])
-
   def test_dynamic_layer_with_deferred_sequential_model(self):
     model = sequential.Sequential([DynamicLayer(dynamic=True), layers.Dense(3)])
     self.assertEqual(model.dynamic, True)
@@ -274,6 +284,7 @@ class BaseLayerTest(keras_parameterized.TestCase):
   @combinations.generate(
       combinations.times(
           combinations.keras_model_type_combinations(),
+          combinations.keras_tensor_combinations(),
           combinations.combine(mode=['graph', 'eager'])))
   def test_build_with_numpy_data(self):
     model_layers = [
@@ -374,7 +385,8 @@ class BaseLayerTest(keras_parameterized.TestCase):
   # b/124459427: can't test with `run_eagerly=True` for now.
   @combinations.generate(
       combinations.times(combinations.keras_mode_combinations(),
-                         combinations.keras_model_type_combinations()))
+                         combinations.keras_model_type_combinations(),
+                         combinations.keras_tensor_combinations()))
   def test_training_arg_in_defun(self):
     layer = self._get_layer_with_training_arg()
     model = testing_utils.get_model_from_layers([layer], input_shape=(1,))
@@ -399,7 +411,8 @@ class BaseLayerTest(keras_parameterized.TestCase):
 
   @combinations.generate(
       combinations.times(combinations.keras_mode_combinations(),
-                         combinations.keras_model_type_combinations()))
+                         combinations.keras_model_type_combinations(),
+                         combinations.keras_tensor_combinations()))
   def test_raw_variable_assignment(self):
 
     class RawVariableLayer(base_layer.Layer):
diff --git a/tensorflow/python/keras/engine/base_layer_utils_test.py b/tensorflow/python/keras/engine/base_layer_utils_test.py
index d27a4cd3297..c039befda7f 100644
--- a/tensorflow/python/keras/engine/base_layer_utils_test.py
+++ b/tensorflow/python/keras/engine/base_layer_utils_test.py
@@ -20,12 +20,17 @@ import numpy as np
 
 from tensorflow.python import keras
 from tensorflow.python.framework import dtypes
+from tensorflow.python.framework import sparse_tensor
 from tensorflow.python.keras import backend
 from tensorflow.python.keras import combinations
 from tensorflow.python.keras import keras_parameterized
+from tensorflow.python.keras import testing_utils
 from tensorflow.python.keras.engine import base_layer_utils
 from tensorflow.python.ops import lookup_ops
 from tensorflow.python.ops import math_ops
+from tensorflow.python.ops import sparse_ops
+from tensorflow.python.ops.ragged import ragged_factory_ops
+from tensorflow.python.ops.ragged import ragged_tensor
 from tensorflow.python.platform import test
 
 
@@ -70,7 +75,7 @@ class TrackableWeightHandlerTest(keras_parameterized.TestCase):
     _ = backend.batch_get_value(table_handler.get_tensors())
 
 
-@combinations.generate(combinations.combine(mode=['graph', 'eager']))
+@combinations.generate(combinations.combine(mode=['eager']))
 class OpLayerTest(keras_parameterized.TestCase):
 
   def test_tensor_op_layer(self):
@@ -96,6 +101,35 @@ class OpLayerTest(keras_parameterized.TestCase):
       float_values = math_ops.cast(int_values, dtypes.float32)
       _ = keras.Model(int_values, float_values)
 
+  def test_ragged_op_layer_keras_tensors(self):
+    with testing_utils.use_keras_tensors_scope(True):
+      int_values = keras.Input(shape=(None,), dtype=dtypes.int32, ragged=True)
+      float_values = math_ops.cast(int_values, dtypes.float32)
+      model = keras.Model(int_values, float_values)
+      model.compile(loss='mse')
+
+      input_data = ragged_factory_ops.constant(
+          [[1, 2], [3, 4]], dtype=np.int32)
+      expected = [[1.0, 2.0], [3.0, 4.0]]
+      output = model.predict(input_data)
+      self.assertIsInstance(output, ragged_tensor.RaggedTensor)
+      self.assertAllClose(expected, output)
+
+  def test_sparse_op_layer_keras_tensors(self):
+    with testing_utils.use_keras_tensors_scope(True):
+      int_values = keras.Input(shape=(None,), dtype=dtypes.int32, sparse=True)
+      float_values = math_ops.cast(int_values, dtypes.float32)
+      _ = keras.Model(int_values, float_values)
+      model = keras.Model(int_values, float_values)
+      model.compile(loss='mse')
+
+      input_data = sparse_ops.from_dense(
+          np.array([[1, 2], [3, 4]], dtype=np.int32))
+      expected = [[1.0, 2.0], [3.0, 4.0]]
+      output = model.predict(input_data)
+      self.assertIsInstance(output, sparse_tensor.SparseTensor)
+      self.assertAllClose(expected, sparse_ops.sparse_tensor_to_dense(output))
+
 
 if __name__ == '__main__':
   test.main()
diff --git a/tensorflow/python/keras/engine/compile_utils.py b/tensorflow/python/keras/engine/compile_utils.py
index aa5578dd03c..3858639f024 100644
--- a/tensorflow/python/keras/engine/compile_utils.py
+++ b/tensorflow/python/keras/engine/compile_utils.py
@@ -200,8 +200,7 @@ class LossesContainer(Container):
         continue
 
       y_t, y_p, sw = match_dtype_and_rank(y_t, y_p, sw)
-      sw = apply_mask(y_p, sw)
-
+      sw = apply_mask(y_p, sw, get_mask(y_p))
       loss_value = loss_obj(y_t, y_p, sample_weight=sw)
 
       loss_metric_value = loss_value
@@ -401,12 +400,13 @@ class MetricsContainer(Container):
         continue
 
       y_t, y_p, sw = match_dtype_and_rank(y_t, y_p, sw)
-      sw = apply_mask(y_p, sw)
+      mask = get_mask(y_p)
+      sw = apply_mask(y_p, sw, mask)
 
       for metric_obj in metric_objs:
         if metric_obj is None:
           continue
-        metric_obj.update_state(y_t, y_p)
+        metric_obj.update_state(y_t, y_p, sample_weight=mask)
 
       for weighted_metric_obj in weighted_metric_objs:
         if weighted_metric_obj is None:
@@ -461,6 +461,9 @@ class MetricsContainer(Container):
         else:
           metric_obj = metrics_mod.categorical_crossentropy
 
+    if isinstance(metric_obj, losses_mod.Loss):
+      metric_obj._allow_sum_over_batch_size = True  # pylint: disable=protected-access
+
     if not isinstance(metric_obj, metrics_mod.Metric):
       if isinstance(metric, six.string_types):
         metric_name = metric
@@ -620,10 +623,13 @@ def match_dtype_and_rank(y_t, y_p, sw):
   return y_t, y_p, sw
 
 
-def apply_mask(y_p, sw):
+def get_mask(y_p):
+  """Returns Keras mask from tensor."""
+  return getattr(y_p, '_keras_mask', None)
+
+
+def apply_mask(y_p, sw, mask):
   """Applies any mask on predictions to sample weights."""
-  # Handle Keras mask on outputs.
-  mask = getattr(y_p, '_keras_mask', None)
   if mask is not None:
     mask = math_ops.cast(mask, y_p.dtype)
     if sw is not None:
diff --git a/tensorflow/python/keras/engine/compile_utils_test.py b/tensorflow/python/keras/engine/compile_utils_test.py
index 189d385de07..39127270539 100644
--- a/tensorflow/python/keras/engine/compile_utils_test.py
+++ b/tensorflow/python/keras/engine/compile_utils_test.py
@@ -18,11 +18,13 @@ from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function
 
+from tensorflow.python.distribute import one_device_strategy
 from tensorflow.python.framework import constant_op
 from tensorflow.python.framework import dtypes
 from tensorflow.python.framework import ops
 from tensorflow.python.keras import backend as K
 from tensorflow.python.keras import keras_parameterized
+from tensorflow.python.keras import losses as losses_mod
 from tensorflow.python.keras import metrics as metrics_mod
 from tensorflow.python.keras.engine import compile_utils
 from tensorflow.python.ops import array_ops
@@ -289,6 +291,53 @@ class LossesContainerTest(keras_parameterized.TestCase):
     total_loss = loss_container(y_t, y_p)
     self.assertEqual(total_loss.dtype, dtypes.float64)
 
+  def test_loss_masking(self):
+    loss_container = compile_utils.LossesContainer('mae')
+    y_p = constant_op.constant([[[1], [1]], [[0], [0]]], dtype=dtypes.float32)
+    y_t = constant_op.constant([[[1], [1]], [[1], [1]]], dtype=dtypes.float32)
+    y_p._keras_mask = constant_op.constant([[1, 0], [1, 0]],
+                                           dtype=dtypes.float32)
+
+    total_loss = loss_container(y_t, y_p)
+    self.assertAlmostEqual(total_loss.numpy(), .25)  # sum over batch size
+
+    self.assertLen(loss_container.metrics, 1)
+    loss_metric = loss_container.metrics[0]
+    self.assertEqual(loss_metric.name, 'loss')
+    self.assertAlmostEqual(loss_metric.result().numpy(), .25)
+
+  def test_loss_sample_weight(self):
+    loss_container = compile_utils.LossesContainer('mae')
+    y_p = constant_op.constant([[[1], [1]], [[0], [0]]], dtype=dtypes.float32)
+    y_t = constant_op.constant([[[1], [1]], [[1], [1]]], dtype=dtypes.float32)
+    sw = constant_op.constant([[.2, .3], [.5, 0]], dtype=dtypes.float32)
+
+    total_loss = loss_container(y_t, y_p, sample_weight=sw)
+    # (0 * .2 + 0 * .3 + 1 * .5 + 1 * 0) / 4
+    self.assertAlmostEqual(total_loss.numpy(), .125)
+
+    self.assertLen(loss_container.metrics, 1)
+    loss_metric = loss_container.metrics[0]
+    self.assertEqual(loss_metric.name, 'loss')
+    self.assertAlmostEqual(loss_metric.result().numpy(), .125)
+
+  def test_loss_masking_sample_weight(self):
+    loss_container = compile_utils.LossesContainer('mae')
+    y_p = constant_op.constant([[[1], [1]], [[0], [0]]], dtype=dtypes.float32)
+    y_t = constant_op.constant([[[1], [1]], [[1], [1]]], dtype=dtypes.float32)
+    sw = constant_op.constant([[.2, .3], [.5, 0]], dtype=dtypes.float32)
+    y_p._keras_mask = constant_op.constant([[1, 0], [1, 0]],
+                                           dtype=dtypes.float32)
+
+    total_loss = loss_container(y_t, y_p, sample_weight=sw)
+    # (0 * .2 + 1 * .5) / 4
+    self.assertAlmostEqual(total_loss.numpy(), .125)  # sum over batch size
+
+    self.assertLen(loss_container.metrics, 1)
+    loss_metric = loss_container.metrics[0]
+    self.assertEqual(loss_metric.name, 'loss')
+    self.assertAlmostEqual(loss_metric.result().numpy(), .125)
+
 
 class MetricsContainerTest(keras_parameterized.TestCase):
 
@@ -566,6 +615,76 @@ class MetricsContainerTest(keras_parameterized.TestCase):
     self.assertEqual(mse_metric.name, 'output3_mse')
     self.assertEqual(mse_metric.result().numpy(), 4.)
 
+  def test_metrics_masking(self):
+    metrics_container = compile_utils.MetricsContainer(
+        metrics=['mae'], weighted_metrics=['mse'])
+    y_p = constant_op.constant([[[1], [1]], [[0], [0]]], dtype=dtypes.float32)
+    y_t = constant_op.constant([[[1], [1]], [[1], [1]]], dtype=dtypes.float32)
+    y_p._keras_mask = constant_op.constant([[1, 1], [0, 0]],
+                                           dtype=dtypes.float32)
+
+    metrics_container.update_state(y_t, y_p)
+    self.assertLen(metrics_container.metrics, 2)
+
+    mae_metric = metrics_container.metrics[0]
+    self.assertEqual(mae_metric.name, 'mae')
+    self.assertAlmostEqual(mae_metric.result().numpy(), 0)
+
+    weighted_mae_metric = metrics_container.metrics[1]
+    self.assertEqual(weighted_mae_metric.name, 'mse')
+    self.assertAlmostEqual(weighted_mae_metric.result().numpy(), 0)
+
+  def test_metrics_sample_weight(self):
+    metrics_container = compile_utils.MetricsContainer(
+        metrics=['mae'], weighted_metrics=['mse'])
+    y_p = constant_op.constant([[[1], [1]], [[0], [1]]], dtype=dtypes.float32)
+    y_t = constant_op.constant([[[1], [1]], [[1], [1]]], dtype=dtypes.float32)
+    sw = constant_op.constant([[.2, .3], [.5, 0]], dtype=dtypes.float32)
+
+    metrics_container.update_state(y_t, y_p, sample_weight=sw)
+    self.assertLen(metrics_container.metrics, 2)
+
+    mae_metric = metrics_container.metrics[0]
+    self.assertEqual(mae_metric.name, 'mae')
+    self.assertAlmostEqual(mae_metric.result().numpy(), .25)  # 1 / 4
+
+    weighted_mae_metric = metrics_container.metrics[1]
+    self.assertEqual(weighted_mae_metric.name, 'mse')
+    self.assertAlmostEqual(weighted_mae_metric.result().numpy(), .5)  # .5 / 1
+
+  def test_metrics_masking_sample_weight(self):
+    metrics_container = compile_utils.MetricsContainer(
+        metrics=['mae'], weighted_metrics=['mse'])
+    y_p = constant_op.constant([[[1], [1]], [[0], [1]]], dtype=dtypes.float32)
+    y_t = constant_op.constant([[[1], [1]], [[1], [1]]], dtype=dtypes.float32)
+    sw = constant_op.constant([[.3, .2], [.2, .3]], dtype=dtypes.float32)
+    y_p._keras_mask = constant_op.constant([[1, 0], [1, 0]],
+                                           dtype=dtypes.float32)
+
+    metrics_container.update_state(y_t, y_p, sample_weight=sw)
+    self.assertLen(metrics_container.metrics, 2)
+
+    mae_metric = metrics_container.metrics[0]
+    self.assertEqual(mae_metric.name, 'mae')
+    self.assertAlmostEqual(mae_metric.result().numpy(), .5)  # 1 / .5
+
+    weighted_mae_metric = metrics_container.metrics[1]
+    self.assertEqual(weighted_mae_metric.name, 'mse')
+    self.assertAlmostEqual(weighted_mae_metric.result().numpy(), .2 / .5)
+
+  def test_loss_class_as_metric_with_distribution(self):
+    distribution = one_device_strategy.OneDeviceStrategy('/device:CPU:0')
+    with distribution.scope():
+      metric_container = compile_utils.MetricsContainer(
+          losses_mod.MeanSquaredError())
+      y_t, y_p = array_ops.ones((10, 5)), array_ops.zeros((10, 5))
+      metric_container.update_state(y_t, y_p)
+
+      self.assertLen(metric_container.metrics, 1)
+      metric = metric_container.metrics[0]
+      self.assertEqual(metric.name, 'mean_squared_error')
+      self.assertEqual(metric.result().numpy(), 1.)
+
 
 if __name__ == '__main__':
   ops.enable_eager_execution()
diff --git a/tensorflow/python/keras/engine/data_adapter.py b/tensorflow/python/keras/engine/data_adapter.py
index ecfcfda589b..bf0bbb7d994 100644
--- a/tensorflow/python/keras/engine/data_adapter.py
+++ b/tensorflow/python/keras/engine/data_adapter.py
@@ -1365,8 +1365,10 @@ def expand_1d(data):
   return nest.map_structure(_expand_single_1d_tensor, data)
 
 
-def train_validation_split(arrays, validation_split, shuffle=True):
-  """Split arrays into random train and validation subsets.
+def train_validation_split(arrays, validation_split):
+  """Split arrays into train and validation subsets in deterministic order.
+
+  The last part of data will become validation data.
 
   Arguments:
     arrays: Tensors to split. Allowed inputs are arbitrarily nested structures
@@ -1374,10 +1376,6 @@ def train_validation_split(arrays, validation_split, shuffle=True):
     validation_split: Float between 0 and 1. The proportion of the dataset to
       include in the validation split. The rest of the dataset will be included
       in the training split.
-    shuffle: Bool. Whether to shuffle the data before performing a split. If
-      `False`, the last `validation_split` fraction of that training data will
-      become the validation split.
-
   Returns:
     `(train_arrays, validation_arrays)`
   """
@@ -1406,12 +1404,7 @@ def train_validation_split(arrays, validation_split, shuffle=True):
 
   # Assumes all arrays have the same batch shape or are `None`.
   batch_dim = int(first_non_none.shape[0])
-  indices = ops.convert_to_tensor_v2(range(batch_dim))
-  if shuffle:
-    indices = random_ops.random_shuffle(indices)
   split_at = int(math.floor(batch_dim * (1. - validation_split)))
-  train_indices = indices[:split_at]
-  val_indices = indices[split_at:]
 
   if split_at == 0 or split_at == batch_dim:
     raise ValueError(
@@ -1421,16 +1414,15 @@ def train_validation_split(arrays, validation_split, shuffle=True):
         "different value for the `validation_split` argument." .format(
             batch_dim=batch_dim, validation_split=validation_split))
 
-  def _split(t, indices):
+  def _split(t, start, end):
     if t is None:
       return t
-    t = ops.convert_to_tensor_v2(t)
-    return array_ops.gather_v2(t, indices)
+    return t[start:end]
 
   train_arrays = nest.map_structure(
-      functools.partial(_split, indices=train_indices), arrays)
+      functools.partial(_split, start=0, end=split_at), arrays)
   val_arrays = nest.map_structure(
-      functools.partial(_split, indices=val_indices), arrays)
+      functools.partial(_split, start=split_at, end=batch_dim), arrays)
 
   return train_arrays, val_arrays
 
diff --git a/tensorflow/python/keras/engine/data_adapter_test.py b/tensorflow/python/keras/engine/data_adapter_test.py
index 37b204c87ff..3f4e6d0cb83 100644
--- a/tensorflow/python/keras/engine/data_adapter_test.py
+++ b/tensorflow/python/keras/engine/data_adapter_test.py
@@ -985,7 +985,7 @@ class DataHandlerTest(keras_parameterized.TestCase):
 class TestValidationSplit(keras_parameterized.TestCase):
 
   @parameterized.named_parameters(('numpy_arrays', True), ('tensors', False))
-  def test_validation_split_shuffled(self, use_numpy):
+  def test_validation_split_unshuffled(self, use_numpy):
     if use_numpy:
       x = np.array([0, 1, 2, 3, 4])
       y = np.array([0, 2, 4, 6, 8])
@@ -998,48 +998,13 @@ class TestValidationSplit(keras_parameterized.TestCase):
     (train_x, train_y, train_sw), (val_x, val_y, val_sw) = (
         data_adapter.train_validation_split((x, y, sw), validation_split=0.2))
 
-    self.assertEqual(int(train_x.shape[0]), 4)
-    self.assertEqual(int(train_y.shape[0]), 4)
-    self.assertEqual(int(train_sw.shape[0]), 4)
-    for i in range(4):
-      # Check that all arrays were shuffled in identical order.
-      self.assertEqual(2 * train_x[i].numpy(), train_y[i].numpy())
-      self.assertEqual(2 * train_y[i].numpy(), train_sw[i].numpy())
-
-    self.assertEqual(int(val_x.shape[0]), 1)
-    self.assertEqual(int(val_y.shape[0]), 1)
-    self.assertEqual(int(val_sw.shape[0]), 1)
-    for i in range(1):
-      # Check that all arrays were shuffled in identical order.
-      self.assertEqual(2 * train_x[i].numpy(), train_y[i].numpy())
-      self.assertEqual(2 * train_y[i].numpy(), train_sw[i].numpy())
-
-    # Check that arrays contain expected values.
-    self.assertEqual(
-        sorted(array_ops.concat([train_x, val_x], axis=0).numpy().tolist()),
-        sorted(ops.convert_to_tensor_v2(x).numpy().tolist()))
-    self.assertEqual(
-        sorted(array_ops.concat([train_y, val_y], axis=0).numpy().tolist()),
-        sorted(ops.convert_to_tensor_v2(y).numpy().tolist()))
-    self.assertEqual(
-        sorted(array_ops.concat([train_sw, val_sw], axis=0).numpy().tolist()),
-        sorted(ops.convert_to_tensor_v2(sw).numpy().tolist()))
-
-  @parameterized.named_parameters(('numpy_arrays', True), ('tensors', False))
-  def test_validation_split_unshuffled(self, use_numpy):
     if use_numpy:
-      x = np.array([0, 1, 2, 3, 4])
-      y = np.array([0, 2, 4, 6, 8])
-      sw = np.array([0, 4, 8, 12, 16])
-    else:
-      x = ops.convert_to_tensor_v2([0, 1, 2, 3, 4])
-      y = ops.convert_to_tensor_v2([0, 2, 4, 6, 8])
-      sw = ops.convert_to_tensor_v2([0, 4, 8, 12, 16])
-
-    (train_x, train_y, train_sw), (val_x, val_y, val_sw) = (
-        data_adapter.train_validation_split((x, y, sw),
-                                            validation_split=0.2,
-                                            shuffle=False))
+      train_x = ops.convert_to_tensor_v2(train_x)
+      train_y = ops.convert_to_tensor_v2(train_y)
+      train_sw = ops.convert_to_tensor_v2(train_sw)
+      val_x = ops.convert_to_tensor_v2(val_x)
+      val_y = ops.convert_to_tensor_v2(val_y)
+      val_sw = ops.convert_to_tensor_v2(val_sw)
 
     self.assertEqual(train_x.numpy().tolist(), [0, 1, 2, 3])
     self.assertEqual(train_y.numpy().tolist(), [0, 2, 4, 6])
diff --git a/tensorflow/python/feature_column/keras_integration_test.py b/tensorflow/python/keras/engine/feature_columns_integration_test.py
similarity index 65%
rename from tensorflow/python/feature_column/keras_integration_test.py
rename to tensorflow/python/keras/engine/feature_columns_integration_test.py
index 456c0204350..04d708feb5e 100644
--- a/tensorflow/python/feature_column/keras_integration_test.py
+++ b/tensorflow/python/keras/engine/feature_columns_integration_test.py
@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 # ==============================================================================
-"""Tests specific to Feature Columns and Keras integration."""
+"""Tests specific to Feature Columns integration."""
 
 from __future__ import absolute_import
 from __future__ import division
@@ -21,17 +21,11 @@ from __future__ import print_function
 import numpy as np
 
 from tensorflow.python import keras
-from tensorflow.python import tf2
 from tensorflow.python.data.ops import dataset_ops
 from tensorflow.python.feature_column import feature_column_lib as fc
-from tensorflow.python.feature_column import feature_column_v2
 from tensorflow.python.keras import keras_parameterized
 from tensorflow.python.keras import metrics as metrics_module
 from tensorflow.python.keras import testing_utils
-from tensorflow.python.keras.feature_column import dense_features_v2
-from tensorflow.python.keras.optimizer_v2 import gradient_descent
-from tensorflow.python.keras.premade import linear
-from tensorflow.python.keras.premade import wide_deep
 from tensorflow.python.keras.utils import np_utils
 from tensorflow.python.platform import test
 
@@ -304,108 +298,6 @@ class FeatureColumnsIntegrationTest(keras_parameterized.TestCase):
                   loss=keras.losses.BinaryCrossentropy())
     model.fit(dataset)
 
-  def test_serialization_dense_features(self):
-    dense_feature = fc.DenseFeatures([fc.numeric_column('a')])
-    config = keras.layers.serialize(dense_feature)
-    self.assertEqual(config['class_name'], 'DenseFeatures')
-
-    revived = keras.layers.deserialize(config)
-    if tf2.enabled():
-      self.assertIsInstance(revived, dense_features_v2.DenseFeatures)
-    else:
-      self.assertIsInstance(revived, fc.DenseFeatures)
-      self.assertNotIsInstance(revived, dense_features_v2.DenseFeatures)
-
-  # This test is an example for a regression on categorical inputs, i.e.,
-  # the output is 0.4, 0.6, 0.9 when input is 'alpha', 'beta', 'gamma'
-  # separately.
-  @keras_parameterized.run_all_keras_modes(always_skip_v1=True)
-  def test_linear_model_with_feature_column(self):
-    vocab_list = ['alpha', 'beta', 'gamma']
-    vocab_val = [0.4, 0.6, 0.9]
-    data = np.random.choice(vocab_list, size=256)
-    y = np.zeros_like(data, dtype=np.float32)
-    for vocab, val in zip(vocab_list, vocab_val):
-      indices = np.where(data == vocab)
-      y[indices] = val + np.random.uniform(
-          low=-0.01, high=0.01, size=indices[0].shape)
-    cat_column = feature_column_v2.categorical_column_with_vocabulary_list(
-        key='symbol', vocabulary_list=vocab_list)
-    ind_column = feature_column_v2.indicator_column(cat_column)
-    dense_feature_layer = dense_features_v2.DenseFeatures([ind_column])
-    linear_model = linear.LinearModel(
-        use_bias=False, kernel_initializer='zeros')
-    combined = keras.Sequential([dense_feature_layer, linear_model])
-    opt = gradient_descent.SGD(learning_rate=0.1)
-    combined.compile(opt, 'mse', [])
-    combined.fit(x={'symbol': data}, y=y, batch_size=32, epochs=10)
-    self.assertAllClose([[0.4], [0.6], [0.9]],
-                        combined.layers[1].dense_layers[0].kernel.numpy(),
-                        atol=0.01)
-
-  # This test is an example for cases where linear and dnn model accepts
-  # same raw input and same transformed inputs, i.e., the raw input is
-  # categorical, and both linear and dnn model accept one hot encoding.
-  @keras_parameterized.run_all_keras_modes(always_skip_v1=True)
-  def test_wide_deep_model_with_single_feature_column(self):
-    vocab_list = ['alpha', 'beta', 'gamma']
-    vocab_val = [0.4, 0.6, 0.9]
-    data = np.random.choice(vocab_list, size=256)
-    y = np.zeros_like(data, dtype=np.float32)
-    for vocab, val in zip(vocab_list, vocab_val):
-      indices = np.where(data == vocab)
-      y[indices] = val + np.random.uniform(
-          low=-0.01, high=0.01, size=indices[0].shape)
-    cat_column = feature_column_v2.categorical_column_with_vocabulary_list(
-        key='symbol', vocabulary_list=vocab_list)
-    ind_column = feature_column_v2.indicator_column(cat_column)
-    dense_feature_layer = dense_features_v2.DenseFeatures([ind_column])
-    linear_model = linear.LinearModel(
-        use_bias=False, kernel_initializer='zeros')
-    dnn_model = keras.Sequential([keras.layers.Dense(units=1)])
-    wide_deep_model = wide_deep.WideDeepModel(linear_model, dnn_model)
-    combined = keras.Sequential([dense_feature_layer, wide_deep_model])
-    opt = gradient_descent.SGD(learning_rate=0.1)
-    combined.compile(
-        opt,
-        'mse', [],
-        run_eagerly=testing_utils.should_run_eagerly())
-    combined.fit(x={'symbol': data}, y=y, batch_size=32, epochs=10)
-
-  # This test is an example for cases where linear and dnn model accepts
-  # same raw input but different transformed inputs, i.e,. the raw input is
-  # categorical, and linear model accepts one hot encoding, while dnn model
-  # accepts embedding encoding.
-  @keras_parameterized.run_all_keras_modes(always_skip_v1=True)
-  def test_wide_deep_model_with_two_feature_columns(self):
-    vocab_list = ['alpha', 'beta', 'gamma']
-    vocab_val = [0.4, 0.6, 0.9]
-    data = np.random.choice(vocab_list, size=256)
-    y = np.zeros_like(data, dtype=np.float32)
-    for vocab, val in zip(vocab_list, vocab_val):
-      indices = np.where(data == vocab)
-      y[indices] = val + np.random.uniform(
-          low=-0.01, high=0.01, size=indices[0].shape)
-    cat_column = feature_column_v2.categorical_column_with_vocabulary_list(
-        key='symbol', vocabulary_list=vocab_list)
-    ind_column = feature_column_v2.indicator_column(cat_column)
-    emb_column = feature_column_v2.embedding_column(cat_column, dimension=5)
-    linear_feature_layer = dense_features_v2.DenseFeatures([ind_column])
-    linear_model = linear.LinearModel(
-        use_bias=False, kernel_initializer='zeros')
-    combined_linear = keras.Sequential(
-        [linear_feature_layer, linear_model])
-    dnn_model = keras.Sequential([keras.layers.Dense(units=1)])
-    dnn_feature_layer = dense_features_v2.DenseFeatures([emb_column])
-    combined_dnn = keras.Sequential([dnn_feature_layer, dnn_model])
-    wide_deep_model = wide_deep.WideDeepModel(combined_linear, combined_dnn)
-    opt = gradient_descent.SGD(learning_rate=0.1)
-    wide_deep_model.compile(
-        opt,
-        'mse', [],
-        run_eagerly=testing_utils.should_run_eagerly())
-    wide_deep_model.fit(x={'symbol': data}, y=y, batch_size=32, epochs=10)
-
 
 if __name__ == '__main__':
   test.main()
diff --git a/tensorflow/python/keras/engine/functional.py b/tensorflow/python/keras/engine/functional.py
index bdc800d4d1b..741cc831f02 100644
--- a/tensorflow/python/keras/engine/functional.py
+++ b/tensorflow/python/keras/engine/functional.py
@@ -32,6 +32,7 @@ from tensorflow.python.keras import backend
 from tensorflow.python.keras.engine import base_layer
 from tensorflow.python.keras.engine import base_layer_utils
 from tensorflow.python.keras.engine import input_layer as input_layer_module
+from tensorflow.python.keras.engine import keras_tensor
 from tensorflow.python.keras.engine import node as node_module
 from tensorflow.python.keras.engine import training as training_lib
 from tensorflow.python.keras.engine import training_utils
@@ -994,7 +995,8 @@ def _map_subgraph_network(inputs, outputs):
   Returns:
     A tuple of List{Node] and List[Layer].
   """
-  base_layer_utils.create_keras_history(outputs)
+  if not keras_tensor.keras_tensors_enabled():
+    base_layer_utils.create_keras_history(outputs)
   # Keep only nodes and layers in the topology between inputs and outputs.
   _, nodes_by_depth, layers, _ = _map_graph_network(inputs, outputs)
   return nest.flatten([nodes for nodes in nodes_by_depth.values()]), layers
diff --git a/tensorflow/python/keras/engine/functional_test.py b/tensorflow/python/keras/engine/functional_test.py
index 90fc9f2697f..b877e81af15 100644
--- a/tensorflow/python/keras/engine/functional_test.py
+++ b/tensorflow/python/keras/engine/functional_test.py
@@ -21,6 +21,7 @@ from __future__ import print_function
 import numpy as np
 
 from tensorflow.python.eager import context
+from tensorflow.python.eager import def_function
 from tensorflow.python.framework import constant_op
 from tensorflow.python.framework import dtypes
 from tensorflow.python.framework import ops
@@ -964,6 +965,45 @@ class NetworkConstructionTest(keras_parameterized.TestCase):
     # Check that second input was correctly added to first.
     self.assertEqual(history.history['loss'][0], 0.0)
 
+  @combinations.generate(combinations.times(
+      combinations.keras_mode_combinations(),
+      combinations.keras_tensor_combinations()))
+  def test_call_kwarg_derived_from_keras_layer_and_first_arg_is_constant(self):
+
+    class MaybeAdd(layers.Layer):
+
+      def call(self, x1, x2=None):
+        if x2 is not None:
+          return x1 + x2
+        return x1
+
+    input2 = input_layer_lib.Input(10)
+    outputs = MaybeAdd()(3., x2=input2)
+    model = training_lib.Model([input2], outputs)
+    model.compile(
+        'sgd',
+        'mse',
+        run_eagerly=testing_utils.should_run_eagerly())
+    history = model.fit(
+        x=7 * np.ones((10, 10)),
+        y=10 * np.ones((10, 10)),
+        batch_size=2)
+    # Check that second input was correctly added to first.
+    self.assertEqual(history.history['loss'][0], 0.0)
+
+    model = training_lib.Model.from_config(
+        model.get_config(), custom_objects={'MaybeAdd': MaybeAdd})
+    model.compile(
+        'sgd',
+        'mse',
+        run_eagerly=testing_utils.should_run_eagerly())
+    history = model.fit(
+        x=7 * np.ones((10, 10)),
+        y=10 * np.ones((10, 10)),
+        batch_size=2)
+    # Check that second input was correctly added to first.
+    self.assertEqual(history.history['loss'][0], 0.0)
+
   @combinations.generate(combinations.keras_mode_combinations())
   def test_composite_call_kwarg_derived_from_keras_layer(self):
 
@@ -1005,6 +1045,58 @@ class NetworkConstructionTest(keras_parameterized.TestCase):
     # Check that second input was correctly added to first.
     self.assertEqual(history.history['loss'][0], 0.0)
 
+  @combinations.generate(combinations.times(
+      combinations.keras_mode_combinations(mode='eager'),
+      combinations.keras_tensor_combinations()))
+  def test_call_some_not_all_nested_in_first_arg_derived_from_keras_layer(self):
+    # This functionality is unsupported in v1 graphs
+
+    class AddAll(layers.Layer):
+
+      def call(self, x1_x2, x3):
+        x1, x2 = x1_x2
+        out = x1 + x2
+        if x3 is not None:
+          for t in x3.values():
+            out += t
+        return out
+
+    input1 = input_layer_lib.Input(10)
+    input2 = input_layer_lib.Input(10)
+    input3 = input_layer_lib.Input(10)
+
+    outputs = AddAll()(
+        [input1, 4 * array_ops.ones((1, 10))],
+        x3={
+            'a': input2,
+            'b': input3,
+            'c': 5 * array_ops.ones((1, 10))
+        })
+    model = training_lib.Model([input1, input2, input3], outputs)
+    model.compile(
+        'sgd',
+        'mse',
+        run_eagerly=testing_utils.should_run_eagerly())
+    history = model.fit(
+        x=[np.ones((10, 10)), 2 * np.ones((10, 10)), 3 * np.ones((10, 10))],
+        y=15 * np.ones((10, 10)),
+        batch_size=2)
+    # Check that all inputs were correctly added.
+    self.assertEqual(history.history['loss'][0], 0.0)
+
+    model = training_lib.Model.from_config(
+        model.get_config(), custom_objects={'AddAll': AddAll})
+    model.compile(
+        'sgd',
+        'mse',
+        run_eagerly=testing_utils.should_run_eagerly())
+    history = model.fit(
+        x=[np.ones((10, 10)), 2 * np.ones((10, 10)), 3 * np.ones((10, 10))],
+        y=15 * np.ones((10, 10)),
+        batch_size=2)
+    # Check that all inputs were correctly added.
+    self.assertEqual(history.history['loss'][0], 0.0)
+
   @combinations.generate(combinations.keras_mode_combinations())
   def test_call_nested_arg_derived_from_keras_layer(self):
 
@@ -1168,6 +1260,28 @@ class NetworkConstructionTest(keras_parameterized.TestCase):
     self.assertIn('trackable', model._unconditional_dependency_names)
     self.assertEqual(model.trackable, model._lookup_dependency('trackable'))
 
+  @combinations.generate(combinations.combine(mode=['graph', 'eager']))
+  def test_model_construction_in_tf_function(self):
+
+    d = {'model': None}
+
+    @def_function.function
+    def fn(x):
+      if d['model'] is None:
+        # Check that Functional can be built in a `tf.function`.
+        inputs = input_layer_lib.Input(10)
+        outputs = layers.Dense(1)(inputs)
+        model = functional.Functional(inputs, outputs)
+        d['model'] = model
+      else:
+        model = d['model']
+
+      return model(x)
+
+    x = array_ops.ones((10, 10))
+    y = fn(x)
+    self.assertEqual(y.shape.as_list(), [10, 1])
+
 
 class DeferredModeTest(keras_parameterized.TestCase):
 
diff --git a/tensorflow/python/keras/engine/input_layer.py b/tensorflow/python/keras/engine/input_layer.py
index 8075cc3fd0f..b7720754d0c 100644
--- a/tensorflow/python/keras/engine/input_layer.py
+++ b/tensorflow/python/keras/engine/input_layer.py
@@ -26,6 +26,7 @@ from tensorflow.python.framework import tensor_spec
 from tensorflow.python.keras import backend
 from tensorflow.python.keras.distribute import distributed_training_utils
 from tensorflow.python.keras.engine import base_layer
+from tensorflow.python.keras.engine import keras_tensor
 from tensorflow.python.keras.engine import node as node_module
 from tensorflow.python.keras.saving.saved_model import layer_serialization
 from tensorflow.python.keras.utils import tf_utils
@@ -116,6 +117,10 @@ class InputLayer(base_layer.Layer):
     if kwargs:
       raise ValueError('Unrecognized keyword arguments:', kwargs.keys())
 
+    if sparse and ragged:
+      raise ValueError(
+          'Cannot set both sparse and ragged to True in a Keras input.')
+
     if not name:
       prefix = 'input'
       name = prefix + '_' + str(backend.get_uid(prefix))
@@ -157,7 +162,14 @@ class InputLayer(base_layer.Layer):
       self.is_placeholder = True
       self._batch_input_shape = batch_input_shape
     else:
-      if not tf_utils.is_symbolic_tensor(input_tensor):
+      raise_eager_tensor_error = False
+      if keras_tensor.keras_tensors_enabled():
+        if not isinstance(input_tensor, keras_tensor.keras_tensors_enabled()):
+          raise_eager_tensor_error = True
+      else:
+        if not tf_utils.is_symbolic_tensor(input_tensor):
+          raise_eager_tensor_error = True
+      if raise_eager_tensor_error:
         raise ValueError('You should not pass an EagerTensor to `Input`. '
                          'For example, instead of creating an '
                          'InputLayer, you should instantiate your model and '
@@ -173,7 +185,8 @@ class InputLayer(base_layer.Layer):
     node_module.Node(layer=self, outputs=input_tensor)
 
     # Store type spec
-    if isinstance(input_tensor, composite_tensor.CompositeTensor):
+    if isinstance(input_tensor, (
+        composite_tensor.CompositeTensor, keras_tensor.KerasTensor)):
       self._type_spec = input_tensor._type_spec  # pylint: disable=protected-access
     else:
       self._type_spec = tensor_spec.TensorSpec(
diff --git a/tensorflow/python/keras/engine/keras_tensor.py b/tensorflow/python/keras/engine/keras_tensor.py
new file mode 100644
index 00000000000..4ea01da8db2
--- /dev/null
+++ b/tensorflow/python/keras/engine/keras_tensor.py
@@ -0,0 +1,229 @@
+# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Keras Input Tensor used to track functional API Topology."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from tensorflow.python.framework import ops
+from tensorflow.python.framework import type_spec as type_spec_module
+from tensorflow.python.ops import array_ops
+from tensorflow.python.util import nest
+from tensorflow.python.util import object_identity
+
+_KERAS_TENSORS_ENABLED = False
+
+
+def enable_keras_tensors():
+  """Enable using KerasTensors in Keras's functional API."""
+  global _KERAS_TENSORS_ENABLED
+  _KERAS_TENSORS_ENABLED = True
+
+
+def disable_keras_tensors():
+  """Disable using KerasTensors in Keras's functional API."""
+  global _KERAS_TENSORS_ENABLED
+  _KERAS_TENSORS_ENABLED = False
+
+
+def keras_tensors_enabled():
+  """Return a bool specifying if KerasTensors are enabled."""
+  return _KERAS_TENSORS_ENABLED and ops.executing_eagerly_outside_functions()
+
+
+class KerasTensor(object):
+  """A representation of a Keras in/output during Functional API construction.
+
+  `KerasTensor`s are an alternative representation for Keras `Inputs`
+  and for intermediate outputs of layers during Functional API construction of
+  models. They are a lightweight data structure comprised of only the
+  `tf.TypeSpec` of the Tensor that will be consumed/produced in the
+  corresponding position of the model.
+
+  They implement just small subset of `tf.Tensor`'s attributes and
+  methods, and also overload
+  the same operators as `tf.Tensor` and automatically turn them into
+  Keras layers in the model.
+
+  `KerasTensor`s are still internal-only and are a work in progress, but they
+  have several advantages over using a graph `tf.Tensor` to represent
+  symbolic values in functional models.
+  - Unlike symbolic tensors, they do not need to refer to a graph. This means
+    Keras does not need to maintain a never-deleted global background graph
+    containing all layers ever called during functional model construction when
+    constructing Functional Models with KerasTensors. These memory savings
+    can be significant.
+
+  - Triggering Keras functional model construction is simpler
+    when it just has to check whether something is a KerasTensor, rather
+    than trying to infer if a tensor was meant to be a symbolic keras
+    representation or just a value produced during function tracing.
+
+  - Autolambda layers (converting tf ops on symbolic Keras tensors to lambda
+    Keras layers in the model) use TF's internal dispatching mechanism, instead
+    of trying to manually walk a graph and extract nodes from it.
+    The dispatching mechanism is simpler, works more reliably, and is less
+    likely to run into issues with composite tensors or strange tf ops/nodes.
+
+    (And when it fails, it's by design: because dispatch is explicitly not
+    supported on the op & it's more obvious that dispatch doesn't support the
+    setting).
+
+  - Because they support arbitrary typespecs, models/layers that use
+    KerasTensors are generally more friendly to composite tensors of different
+    types than using symbolic graph tensors (which must have a TensorSpec and
+    can't have arbitrary typespecs)
+
+  To experiment with using KerasTensors instead of symbolic graph `tf.Tensors`,
+  import keras_tensor directly and call `keras_tensor.enable_keras_tensors()`
+  """
+
+  def __init__(self, type_spec, name=None):
+    """Construct a KerasTensor from a type_spec and an optional name."""
+    if not isinstance(type_spec, type_spec_module.TypeSpec):
+      raise ValueError('KerasTensors must be constructed with a `tf.TypeSpec`.')
+
+    self._type_spec = type_spec
+    if name is None and hasattr(type_spec, 'name'):
+      name = type_spec.name
+    self._name = name
+
+  @property
+  def type_spec(self):
+    """Returns the `TypeSpec` that represents this Tensor."""
+    return self._type_spec
+
+  @property
+  def shape(self):
+    """Returns the `TensorShape` that represents the shape of the tensor."""
+    # TODO(kaftan): This is only valid for normal/sparse/ragged tensors.
+    # may need to raise an error when it's not valid for a type_spec,
+    # but some keras code (e.g. build-related stuff) will likely fail when
+    # it can't access shape or dtype
+    return self._type_spec._shape  # pylint: disable=protected-access
+
+  def get_shape(self):
+    return self.shape
+
+  @property
+  def dtype(self):
+    """Returns the `dtype` of elements in the tensor."""
+    # TODO(kaftan): This is only valid for normal/sparse/ragged tensors.
+    # may need to raise an error when it's not valid for a type_spec,
+    # but some keras code (e.g. build-related stuff) will likely fail when
+    # it can't access shape or dtype
+    return self._type_spec._dtype  # pylint: disable=protected-access
+
+  def ref(self):
+    """Returns a hashable reference object to this KerasTensor.
+
+    The primary use case for this API is to put KerasTensors in a
+    set/dictionary. We can't put tensors in a set/dictionary as
+    `tensor.__hash__()` is not available and tensor equality (`==`) is supposed
+    to produce a tensor representing if the two inputs are equal.
+
+    See the documentation of `tf.Tensor.ref()` for more info.
+    """
+    return object_identity.Reference(self)
+
+  def __iter__(self):
+    shape = None
+    if self.shape.ndims is not None:
+      shape = [dim.value for dim in self.shape.dims]
+
+    if shape is None:
+      raise TypeError('Cannot iterate over a KerasTensor with unknown shape.')
+    if not shape:
+      raise TypeError('Cannot iterate over a scalar.')
+    if shape[0] is None:
+      raise TypeError(
+          'Cannot iterate over a KerasTensor with unknown first dimension.')
+    return _KerasTensorIterator(self, shape[0])
+
+  @property
+  def name(self):
+    """Returns the (optionally provided) name of the described tensor."""
+    return self._name
+
+  @classmethod
+  def _overload_all_operators(cls):  # pylint: disable=invalid-name
+    """Register overloads for all operators."""
+    for operator in ops.Tensor.OVERLOADABLE_OPERATORS:
+      cls._overload_operator(operator)
+
+  @classmethod
+  def _overload_operator(cls, operator):  # pylint: disable=invalid-name
+    """Overload an operator with the same overloading as `ops.Tensor`.
+
+    We pull the operator out of ops.Tensor dynamically to avoid ordering issues.
+
+    Args:
+      operator: string. The operator name.
+    """
+    tensor_oper = getattr(ops.Tensor, operator)
+
+    # Compatibility with Python 2:
+    # Python 2 unbound methods have type checks for the first arg,
+    # so we need to extract the underlying function
+    tensor_oper = getattr(tensor_oper, '__func__', tensor_oper)
+
+    setattr(cls, operator, tensor_oper)
+
+
+KerasTensor._overload_all_operators()  # pylint: disable=protected-access
+
+
+class _KerasTensorIterator(object):
+  """Iterates over the leading dim of a KerasTensor. Performs 0 error checks."""
+
+  def __init__(self, tensor, dim0):
+    self._tensor = tensor
+    self._index = 0
+    self._limit = dim0
+
+  def __iter__(self):
+    return self
+
+  def __next__(self):
+    if self._index == self._limit:
+      raise StopIteration
+    result = self._tensor[self._index]
+    self._index += 1
+    return result
+
+  next = __next__  # python2.x compatibility.
+
+
+def keras_tensor_to_placeholder(x):
+  """TODO(kaftan): Docstring."""
+  if isinstance(x, KerasTensor):
+    def tensor_spec_to_placeholder(tensorspec):
+      return array_ops.placeholder(tensorspec.dtype, tensorspec.shape)
+    ph = nest.map_structure(tensor_spec_to_placeholder, x.type_spec,
+                            expand_composites=True)
+    return ph
+  else:
+    return x
+
+
+def keras_tensor_from_tensor(x):
+  name = getattr(x, 'name', None)
+  out = KerasTensor(type_spec_module.type_spec_from_value(x), name=name)
+  if hasattr(x, '_keras_mask'):
+    out._keras_mask = KerasTensor(  # pylint: disable=protected-access
+        type_spec_module.type_spec_from_value(x._keras_mask))  # pylint: disable=protected-access
+
+  return out
diff --git a/tensorflow/python/keras/engine/node.py b/tensorflow/python/keras/engine/node.py
index e8d98389bd7..c75741da8ec 100644
--- a/tensorflow/python/keras/engine/node.py
+++ b/tensorflow/python/keras/engine/node.py
@@ -27,6 +27,7 @@ from tensorflow.python.framework import ops
 from tensorflow.python.framework import tensor_util
 from tensorflow.python.keras import backend
 from tensorflow.python.keras.engine import base_layer_utils
+from tensorflow.python.keras.engine import keras_tensor
 from tensorflow.python.keras.utils import tf_utils
 from tensorflow.python.platform import tf_logging as logging
 from tensorflow.python.util import nest
@@ -80,11 +81,13 @@ class Node(object):
     self._single_positional_tensor_passed = (not self.call_kwargs and len(
         self.call_args) == 1 and tensor_util.is_tensor(self.call_args[0]))
 
-    # Create TensorFlowOpLayers if needed.
-    for obj in self._flat_arguments:
-      if (isinstance(obj, ops.Tensor) and
-          base_layer_utils.needs_keras_history(obj, ignore_call_context=True)):
-        base_layer_utils.create_keras_history(obj)
+    if not keras_tensor.keras_tensors_enabled():
+      # Create TensorFlowOpLayers if needed.
+      for obj in self._flat_arguments:
+        if (isinstance(obj, ops.Tensor) and
+            base_layer_utils.needs_keras_history(
+                obj, ignore_call_context=True)):
+          base_layer_utils.create_keras_history(obj)
 
     self._keras_inputs = []
     self._keras_inputs_ids_and_indices = []
diff --git a/tensorflow/python/keras/engine/training.py b/tensorflow/python/keras/engine/training.py
index adaba76e820..29ff31d56db 100644
--- a/tensorflow/python/keras/engine/training.py
+++ b/tensorflow/python/keras/engine/training.py
@@ -1030,9 +1030,8 @@ class Model(base_layer.Layer, version_utils.ModelVersionSelector):
       # Create the validation data using the training data. Only supported for
       # `Tensor` and `NumPy` input.
       (x, y, sample_weight), validation_data = (
-          data_adapter.train_validation_split((x, y, sample_weight),
-                                              validation_split=validation_split,
-                                              shuffle=False))
+          data_adapter.train_validation_split(
+              (x, y, sample_weight), validation_split=validation_split))
 
     with self.distribute_strategy.scope(), \
          training_utils.RespectCompiledTrainableState(self):
@@ -1232,15 +1231,21 @@ class Model(base_layer.Layer, version_utils.ModelVersionSelector):
     Computation is done in batches (see the `batch_size` arg.)
 
     Arguments:
-        x: Input data. It could be: - A Numpy array (or array-like), or a list
-          of arrays (in case the model has multiple inputs). - A TensorFlow
-          tensor, or a list of tensors (in case the model has multiple inputs).
-          - A dict mapping input names to the corresponding array/tensors, if
-          the model has named inputs. - A `tf.data` dataset. - A generator or
-          `keras.utils.Sequence` instance. A more detailed description of
-          unpacking behavior for iterator types (Dataset, generator, Sequence)
-          is given in the `Unpacking behavior for iterator-like inputs` section
-          of `Model.fit`.
+        x: Input data. It could be:
+          - A Numpy array (or array-like), or a list of arrays
+            (in case the model has multiple inputs).
+          - A TensorFlow tensor, or a list of tensors
+            (in case the model has multiple inputs).
+          - A dict mapping input names to the corresponding array/tensors,
+            if the model has named inputs.
+          - A `tf.data` dataset. Should return a tuple
+            of either `(inputs, targets)` or
+            `(inputs, targets, sample_weights)`.
+          - A generator or `keras.utils.Sequence` returning `(inputs, targets)`
+            or `(inputs, targets, sample_weights)`.
+          A more detailed description of unpacking behavior for iterator types
+          (Dataset, generator, Sequence) is given in the `Unpacking behavior
+          for iterator-like inputs` section of `Model.fit`.
         y: Target data. Like the input data `x`, it could be either Numpy
           array(s) or TensorFlow tensor(s). It should be consistent with `x`
           (you cannot have Numpy inputs and tensor targets, or inversely). If
diff --git a/tensorflow/python/keras/engine/training_test.py b/tensorflow/python/keras/engine/training_test.py
index 111833ba8b5..bc63c3acec6 100644
--- a/tensorflow/python/keras/engine/training_test.py
+++ b/tensorflow/python/keras/engine/training_test.py
@@ -690,7 +690,7 @@ class TrainingTest(keras_parameterized.TestCase):
         metrics=['accuracy'],
         run_eagerly=testing_utils.should_run_eagerly())
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   def test_that_trainable_disables_updates(self):
     val_a = np.random.random((10, 4))
     val_out = np.random.random((10, 4))
@@ -1584,7 +1584,7 @@ class TestExceptionsAndWarnings(keras_parameterized.TestCase):
           run_eagerly=testing_utils.should_run_eagerly())
 
   @keras_parameterized.run_with_all_model_types
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   def test_sparse_op_with_op_layer(self):
     inputs = layers_module.Input(shape=(2,), sparse=True, name='sparse_tensor')
     output = sparse_ops.sparse_minimum(inputs, inputs)
@@ -2817,7 +2817,7 @@ class TestTrainingWithMetrics(keras_parameterized.TestCase):
     scores = model.train_on_batch(x, y, sample_weight=w)
     self.assertArrayNear(scores, [0.3328, 0.8], 0.001)
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   def test_add_metric_with_tensor_on_model(self):
     x = layers_module.Input(shape=(1,))
     y = layers_module.Dense(1, kernel_initializer='ones')(x)
@@ -2932,7 +2932,8 @@ class TestTrainingWithMetrics(keras_parameterized.TestCase):
     self.assertEqual(history.history['metric_1'][-1], 5)
     self.assertAlmostEqual(history.history['val_metric_1'][-1], 5, 0)
 
-  @keras_parameterized.run_all_keras_modes(always_skip_v1=True)
+  @keras_parameterized.run_all_keras_modes(always_skip_v1=True,
+                                           skip_keras_tensors=True)
   def test_model_metrics_list(self):
 
     class LayerWithAddMetric(layers_module.Layer):
diff --git a/tensorflow/python/keras/integration_test/BUILD b/tensorflow/python/keras/integration_test/BUILD
index 80d8fb86345..faf8894f813 100644
--- a/tensorflow/python/keras/integration_test/BUILD
+++ b/tensorflow/python/keras/integration_test/BUILD
@@ -75,6 +75,7 @@ cuda_py_test(
     name = "gradient_checkpoint_test",
     srcs = ["gradient_checkpoint_test.py"],
     python_version = "PY3",
+    tags = ["no_rocm"],
     deps = [
         "//tensorflow:tensorflow_py",
         "//tensorflow/python:extra_py_tests_deps",
diff --git a/tensorflow/python/keras/integration_test/gradient_checkpoint_test.py b/tensorflow/python/keras/integration_test/gradient_checkpoint_test.py
index 100f3ca2022..9d9e0a062b3 100644
--- a/tensorflow/python/keras/integration_test/gradient_checkpoint_test.py
+++ b/tensorflow/python/keras/integration_test/gradient_checkpoint_test.py
@@ -75,7 +75,7 @@ def _limit_gpu_memory():
   if gpus:
     tf.config.experimental.set_virtual_device_configuration(
         gpus[0],
-        [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=1152)])
+        [tf.config.experimental.VirtualDeviceConfiguration(memory_limit=1024)])
     return True
   return False
 
diff --git a/tensorflow/python/keras/keras_parameterized.py b/tensorflow/python/keras/keras_parameterized.py
index 850cf915665..a69452e247f 100644
--- a/tensorflow/python/keras/keras_parameterized.py
+++ b/tensorflow/python/keras/keras_parameterized.py
@@ -303,7 +303,8 @@ def _test_sequential_model_type(f, test_or_class, *args, **kwargs):
 def run_all_keras_modes(test_or_class=None,
                         config=None,
                         always_skip_v1=False,
-                        always_skip_eager=False):
+                        always_skip_eager=False,
+                        **kwargs):
   """Execute the decorated test with all keras execution modes.
 
   This decorator is intended to be applied either to individual test methods in
@@ -361,6 +362,9 @@ def run_all_keras_modes(test_or_class=None,
       when Tensorflow v2 behavior is not enabled.
     always_skip_eager: If True, does not execute the decorated test
       with eager execution modes.
+    **kwargs: Additional kwargs for configuring tests for
+     in-progress Keras behaviors/ refactorings that we haven't fully
+     rolled out yet
 
   Returns:
     Returns a decorator that will run the decorated test method multiple times.
@@ -369,8 +373,14 @@ def run_all_keras_modes(test_or_class=None,
     ImportError: If abseil parameterized is not installed or not included as
       a target dependency.
   """
+  skip_keras_tensors = kwargs.pop('skip_keras_tensors', False)
+  if kwargs:
+    raise ValueError('Unrecognized keyword args: {}'.format(kwargs))
 
   params = [('_v2_function', 'v2_function')]
+  if not skip_keras_tensors:
+    params.append(('_v2_function_use_keras_tensors',
+                   'v2_function_use_keras_tensors'))
   if not always_skip_eager:
     params.append(('_v2_eager', 'v2_eager'))
   if not (always_skip_v1 or tf2.enabled()):
@@ -390,6 +400,8 @@ def run_all_keras_modes(test_or_class=None,
         _v2_eager_test(f, self, *args, **kwargs)
       elif run_mode == 'v2_function':
         _v2_function_test(f, self, *args, **kwargs)
+      elif run_mode == 'v2_function_use_keras_tensors':
+        _v2_function_and_kerastensors_test(f, self, *args, **kwargs)
       else:
         return ValueError('Unknown run mode %s' % run_mode)
 
@@ -417,6 +429,13 @@ def _v2_function_test(f, test_or_class, *args, **kwargs):
       f(test_or_class, *args, **kwargs)
 
 
+def _v2_function_and_kerastensors_test(f, test_or_class, *args, **kwargs):
+  with context.eager_mode():
+    with testing_utils.run_eagerly_scope(False):
+      with testing_utils.use_keras_tensors_scope(True):
+        f(test_or_class, *args, **kwargs)
+
+
 def _test_or_class_decorator(test_or_class, single_method_decorator):
   """Decorate a test or class with a decorator intended for one method.
 
diff --git a/tensorflow/python/keras/keras_parameterized_test.py b/tensorflow/python/keras/keras_parameterized_test.py
index 4d5e93da929..9bddc6608ff 100644
--- a/tensorflow/python/keras/keras_parameterized_test.py
+++ b/tensorflow/python/keras/keras_parameterized_test.py
@@ -27,6 +27,7 @@ from tensorflow.python import tf2
 from tensorflow.python.eager import context
 from tensorflow.python.keras import keras_parameterized
 from tensorflow.python.keras import testing_utils
+from tensorflow.python.keras.engine import keras_tensor
 from tensorflow.python.platform import googletest
 
 
@@ -206,7 +207,7 @@ class KerasParameterizedTest(keras_parameterized.TestCase):
       def runTest(self):
         pass
 
-      @keras_parameterized.run_all_keras_modes
+      @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
       def testBody(self):
         mode = "eager" if context.executing_eagerly() else "graph"
         should_run_eagerly = testing_utils.should_run_eagerly()
@@ -242,6 +243,54 @@ class KerasParameterizedTest(keras_parameterized.TestCase):
       ts.run(res)
       self.assertLen(l, 4)
 
+  def test_run_all_keras_modes_include_keras_tensors(self):
+    l = []
+
+    class ExampleTest(keras_parameterized.TestCase):
+
+      def runTest(self):
+        pass
+
+      @keras_parameterized.run_all_keras_modes()
+      def testBody(self):
+        mode = "eager" if context.executing_eagerly() else "graph"
+        should_run_eagerly = testing_utils.should_run_eagerly()
+        l.append((mode, should_run_eagerly,
+                  keras_tensor.keras_tensors_enabled()))
+
+    e = ExampleTest()
+    if not tf2.enabled():
+      e.testBody_v1_session()
+    e.testBody_v2_eager()
+    e.testBody_v2_function()
+    e.testBody_v2_function_use_keras_tensors()
+
+    if not tf2.enabled():
+      self.assertLen(l, 4)
+      self.assertAllEqual(l, [
+          ("graph", False, False),
+          ("eager", True, False),
+          ("eager", False, False),
+          ("eager", False, True),
+      ])
+
+      ts = unittest.makeSuite(ExampleTest)
+      res = unittest.TestResult()
+      ts.run(res)
+      self.assertLen(l, 8)
+    else:
+      self.assertLen(l, 3)
+      self.assertAllEqual(l, [
+          ("eager", True, False),
+          ("eager", False, False),
+          ("eager", False, True),
+      ])
+
+      ts = unittest.makeSuite(ExampleTest)
+      res = unittest.TestResult()
+      ts.run(res)
+      self.assertLen(l, 6)
+
   def test_run_all_keras_modes_extra_params(self):
     l = []
 
@@ -250,7 +299,7 @@ class KerasParameterizedTest(keras_parameterized.TestCase):
       def runTest(self):
         pass
 
-      @keras_parameterized.run_all_keras_modes
+      @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
       @parameterized.named_parameters(
           [dict(testcase_name="_0", with_brackets=True),
            dict(testcase_name="_1", with_brackets=False)])
@@ -300,7 +349,8 @@ class KerasParameterizedTest(keras_parameterized.TestCase):
       def runTest(self):
         pass
 
-      @keras_parameterized.run_all_keras_modes(always_skip_v1=True)
+      @keras_parameterized.run_all_keras_modes(always_skip_v1=True,
+                                               skip_keras_tensors=True)
       def testBody(self):
         mode = "eager" if context.executing_eagerly() else "graph"
         should_run_eagerly = testing_utils.should_run_eagerly()
@@ -330,7 +380,7 @@ class KerasParameterizedTest(keras_parameterized.TestCase):
         pass
 
       @keras_parameterized.run_with_all_model_types
-      @keras_parameterized.run_all_keras_modes
+      @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
       def testBody(self):
         mode = "eager" if context.executing_eagerly() else "graph"
         should_run_eagerly = testing_utils.should_run_eagerly()
@@ -382,7 +432,7 @@ class KerasParameterizedTest(keras_parameterized.TestCase):
       def runTest(self):
         pass
 
-      @keras_parameterized.run_all_keras_modes
+      @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
       @keras_parameterized.run_with_all_model_types
       def testBody(self):
         mode = "eager" if context.executing_eagerly() else "graph"
@@ -431,7 +481,7 @@ class KerasParameterizedTest(keras_parameterized.TestCase):
     l = []
 
     @keras_parameterized.run_with_all_model_types
-    @keras_parameterized.run_all_keras_modes
+    @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
     class ExampleTest(keras_parameterized.TestCase):
 
       def runTest(self):
@@ -491,7 +541,7 @@ class KerasParameterizedTest(keras_parameterized.TestCase):
       def runTest(self):
         pass
 
-      @keras_parameterized.run_all_keras_modes
+      @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
       @parameterized.named_parameters(dict(testcase_name="_arg",
                                            arg=True))
       def testBody(self, arg):
@@ -537,7 +587,7 @@ class KerasParameterizedTest(keras_parameterized.TestCase):
 
     self.assertLen(l, len(expected_combinations) * 2)
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   @parameterized.named_parameters(dict(testcase_name="argument",
                                        arg=True))
   def test_run_all_keras_modes_extra_params_2(self, arg):
diff --git a/tensorflow/python/keras/layers/BUILD b/tensorflow/python/keras/layers/BUILD
index 0b664d01b6a..91bace936d3 100644
--- a/tensorflow/python/keras/layers/BUILD
+++ b/tensorflow/python/keras/layers/BUILD
@@ -139,8 +139,10 @@ py_library(
         "//tensorflow/python/keras:base_layer",
         "//tensorflow/python/keras:constraints",
         "//tensorflow/python/keras:initializers",
+        "//tensorflow/python/keras:losses",
         "//tensorflow/python/keras:regularizers",
         "//tensorflow/python/keras/engine:input_spec",
+        "//tensorflow/python/keras/engine:keras_tensor",
         "//tensorflow/python/keras/layers/ops:core",
         "//tensorflow/python/keras/utils:engine_utils",
         "//tensorflow/python/keras/utils:generic_utils",
diff --git a/tensorflow/python/keras/layers/convolutional.py b/tensorflow/python/keras/layers/convolutional.py
index 7623f72925e..9a96fc9ecd1 100644
--- a/tensorflow/python/keras/layers/convolutional.py
+++ b/tensorflow/python/keras/layers/convolutional.py
@@ -199,7 +199,7 @@ class Conv(Layer):
     self.input_spec = InputSpec(min_ndim=self.rank + 2,
                                 axes={channel_axis: input_channel})
 
-    self._build_conv_op_input_shape = input_shape
+    self._build_conv_op_data_shape = input_shape[-(self.rank + 1):]
     self._build_input_channel = input_channel
     self._padding_op = self._get_padding_op()
     self._conv_op_data_format = conv_utils.convert_data_format(
@@ -224,11 +224,11 @@ class Conv(Layer):
           padding=self._padding_op,
           data_format=self._conv_op_data_format,
           num_spatial_dims=self.rank)
-      self._build_conv_op_input_shape = inputs.get_shape()
+      self._build_conv_op_data_shape = inputs.shape[-(self.rank + 1):]
 
     # Apply causal padding to inputs for Conv1D.
     if self.padding == 'causal' and self.__class__.__name__ == 'Conv1D':
-      inputs = array_ops.pad(inputs, self._compute_causal_padding())
+      inputs = array_ops.pad(inputs, self._compute_causal_padding(inputs))
 
     outputs = self._convolution_op(inputs, self.kernel)
 
@@ -324,13 +324,17 @@ class Conv(Layer):
     base_config = super(Conv, self).get_config()
     return dict(list(base_config.items()) + list(config.items()))
 
-  def _compute_causal_padding(self):
+  def _compute_causal_padding(self, inputs):
     """Calculates padding for 'causal' option for 1-d conv layers."""
     left_pad = self.dilation_rate[0] * (self.kernel_size[0] - 1)
-    if self.data_format == 'channels_last':
-      causal_padding = [[0, 0], [left_pad, 0], [0, 0]]
+    if getattr(inputs.shape, 'ndims', None) is None:
+      batch_rank = 1
     else:
-      causal_padding = [[0, 0], [0, 0], [left_pad, 0]]
+      batch_rank = len(inputs.shape) - 2
+    if self.data_format == 'channels_last':
+      causal_padding = [[0, 0]] * batch_rank + [[left_pad, 0], [0, 0]]
+    else:
+      causal_padding = [[0, 0]] * batch_rank + [[0, 0], [left_pad, 0]]
     return causal_padding
 
   def _get_channel_axis(self):
@@ -369,11 +373,12 @@ class Conv(Layer):
     Returns:
       `True` or `False` to indicate whether to recreate the conv_op.
     """
-    call_input_shape = inputs.shape
-    # If the most specific compatible shape between _build_input_shape and
-    # call_input_shape is not _build_input_shape then we must re-build.
-    return self._build_conv_op_input_shape.most_specific_compatible_shape(
-        call_input_shape) != self._build_conv_op_input_shape
+    call_data_shape = inputs.shape[-(self.rank + 1):]
+    # If the most specific compatible shape between _build_data_shape and
+    # call_data_shape is not _build_data_shape then we must re-build.
+    return (self._build_conv_op_data_shape
+            != self._build_conv_op_data_shape.most_specific_compatible_shape(
+                call_data_shape))
 
 
 @keras_export('keras.layers.Conv1D', 'keras.layers.Convolution1D')
@@ -404,6 +409,16 @@ class Conv1D(Conv):
   >>> print(y.shape)
   (4, 8, 32)
 
+  >>> # With extended batch shape [4, 7] (e.g. weather data where batch
+  >>> # dimensions correspond to spatial location and the third dimension
+  >>> # corresponds to time.)
+  >>> input_shape = (4, 7, 10, 128)
+  >>> x = tf.random.normal(input_shape)
+  >>> y = tf.keras.layers.Conv1D(
+  ... 32, 3, activation='relu', input_shape=input_shape[2:])(x)
+  >>> print(y.shape)
+  (4, 7, 8, 32)
+
   Arguments:
     filters: Integer, the dimensionality of the output space
       (i.e. the number of output filters in the convolution).
@@ -451,10 +466,10 @@ class Conv1D(Conv):
       see `keras.constraints`).
 
   Input shape:
-    3D tensor with shape: `(batch_size, steps, input_dim)`
+    3+D tensor with shape: `batch_shape + (steps, input_dim)`
 
   Output shape:
-    3D tensor with shape: `(batch_size, new_steps, filters)`
+    3+D tensor with shape: `batch_shape + (new_steps, filters)`
       `steps` value might have changed due to padding or strides.
 
   Returns:
@@ -462,7 +477,7 @@ class Conv1D(Conv):
     `activation(conv1d(inputs, kernel) + bias)`.
 
   Raises:
-    ValueError: when both `strides` > 1 and `dilation_rate` > 1.
+    ValueError: when both `strides > 1` and `dilation_rate > 1`.
   """
 
   def __init__(self,
@@ -702,6 +717,15 @@ class Conv3D(Conv):
   >>> print(y.shape)
   (4, 26, 26, 26, 2)
 
+  >>> # With extended batch shape [4, 7], e.g. a batch of 4 videos of 3D frames,
+  >>> # with 7 frames per video.
+  >>> input_shape = (4, 7, 28, 28, 28, 1)
+  >>> x = tf.random.normal(input_shape)
+  >>> y = tf.keras.layers.Conv3D(
+  ... 2, 3, activation='relu', input_shape=input_shape[2:])(x)
+  >>> print(y.shape)
+  (4, 7, 26, 26, 26, 2)
+
   Arguments:
     filters: Integer, the dimensionality of the output space
       (i.e. the number of output filters in the convolution).
@@ -721,9 +745,9 @@ class Conv3D(Conv):
       one of `channels_last` (default) or `channels_first`.
       The ordering of the dimensions in the inputs.
       `channels_last` corresponds to inputs with shape
-      `(batch_size, spatial_dim1, spatial_dim2, spatial_dim3, channels)`
+      `batch_shape + (spatial_dim1, spatial_dim2, spatial_dim3, channels)`
       while `channels_first` corresponds to inputs with shape
-      `(batch_size, channels, spatial_dim1, spatial_dim2, spatial_dim3)`.
+      `batch_shape + (channels, spatial_dim1, spatial_dim2, spatial_dim3)`.
       It defaults to the `image_data_format` value found in your
       Keras config file at `~/.keras/keras.json`.
       If you never set it, then it will be "channels_last".
@@ -760,30 +784,30 @@ class Conv3D(Conv):
       see `keras.constraints`).
 
   Input shape:
-    5D tensor with shape:
-    `(batch_size, channels, conv_dim1, conv_dim2, conv_dim3)` if
+    5+D tensor with shape:
+    `batch_shape + (channels, conv_dim1, conv_dim2, conv_dim3)` if
       data_format='channels_first'
-    or 5D tensor with shape:
-    `(batch_size, conv_dim1, conv_dim2, conv_dim3, channels)` if
+    or 5+D tensor with shape:
+    `batch_shape + (conv_dim1, conv_dim2, conv_dim3, channels)` if
       data_format='channels_last'.
 
   Output shape:
-    5D tensor with shape:
-    `(batch_size, filters, new_conv_dim1, new_conv_dim2, new_conv_dim3)` if
+    5+D tensor with shape:
+    `batch_shape + (filters, new_conv_dim1, new_conv_dim2, new_conv_dim3)` if
       data_format='channels_first'
-    or 5D tensor with shape:
-    `(batch_size, new_conv_dim1, new_conv_dim2, new_conv_dim3, filters)` if
+    or 5+D tensor with shape:
+    `batch_shape + (new_conv_dim1, new_conv_dim2, new_conv_dim3, filters)` if
       data_format='channels_last'.
     `new_conv_dim1`, `new_conv_dim2` and `new_conv_dim3` values might have
       changed due to padding.
 
   Returns:
-    A tensor of rank 5 representing
+    A tensor of rank 5+ representing
     `activation(conv3d(inputs, kernel) + bias)`.
 
   Raises:
     ValueError: if `padding` is "causal".
-    ValueError: when both `strides` > 1 and `dilation_rate` > 1.
+    ValueError: when both `strides > 1` and `dilation_rate > 1`.
   """
 
   def __init__(self,
diff --git a/tensorflow/python/keras/layers/convolutional_test.py b/tensorflow/python/keras/layers/convolutional_test.py
index 7d54184aa9c..3bb2ff59d1f 100644
--- a/tensorflow/python/keras/layers/convolutional_test.py
+++ b/tensorflow/python/keras/layers/convolutional_test.py
@@ -49,6 +49,21 @@ class Conv1DTest(keras_parameterized.TestCase):
           input_shape=(num_samples, length, stack_size),
           expected_output_shape=expected_output_shape)
 
+  def _run_test_extra_batch_dim(self, kwargs, expected_output_shape):
+    batch_shape = (2, 11)
+    stack_size = 3
+    length = 7
+
+    with self.cached_session(use_gpu=True):
+      if expected_output_shape is not None:
+        expected_output_shape = (None,) + expected_output_shape
+
+      testing_utils.layer_test(
+          keras.layers.Conv1D,
+          kwargs=kwargs,
+          input_shape=batch_shape + (length, stack_size),
+          expected_output_shape=expected_output_shape)
+
   @parameterized.named_parameters(
       ('padding_valid', {
           'padding': 'valid'
@@ -85,6 +100,7 @@ class Conv1DTest(keras_parameterized.TestCase):
     kwargs['kernel_size'] = 3
     if not requires_gpu or test.is_gpu_available(cuda_only=True):
       self._run_test(kwargs, expected_output_shape)
+      self._run_test_extra_batch_dim(kwargs, expected_output_shape)
 
   def test_conv1d_regularizers(self):
     kwargs = {
@@ -222,6 +238,19 @@ class Conv2DTest(keras_parameterized.TestCase):
       self._run_test(kwargs, expected_output_shape)
       self._run_test_extra_batch_dim(kwargs, expected_output_shape)
 
+  def test_conv2d_op_not_recreated_on_different_batch_shape(self):
+    layer = keras.layers.Conv2D(2, 3)
+    layer(np.ones((1, 28, 28, 2)))
+    # pylint: disable=protected-access
+    old_conv_op = layer._convolution_op
+    # Expand batch to rank-2 shape (5, 5)
+    layer(np.ones((5, 5, 28, 28, 2)))
+    self.assertEqual(old_conv_op, layer._convolution_op)
+    layer(np.ones((1, 30, 30, 2)))
+    # 'HW' changed, so the conv object is rebuilt
+    self.assertNotEqual(old_conv_op, layer._convolution_op)
+    # pylint: enable=protected-access
+
   def test_conv2d_regularizers(self):
     kwargs = {
         'filters': 3,
@@ -281,6 +310,27 @@ class Conv3DTest(keras_parameterized.TestCase):
           expected_output_shape=expected_output_shape,
           validate_training=validate_training)
 
+  def _run_test_extra_batch_dim(self,
+                                kwargs,
+                                expected_output_shape,
+                                validate_training=True):
+    batch_shape = (2, 11)
+    stack_size = 3
+    num_row = 7
+    num_col = 6
+    depth = 5
+
+    with self.cached_session(use_gpu=True):
+      if expected_output_shape is not None:
+        expected_output_shape = (None,) + expected_output_shape
+
+      testing_utils.layer_test(
+          keras.layers.Conv3D,
+          kwargs=kwargs,
+          input_shape=batch_shape + (depth, num_row, num_col, stack_size),
+          expected_output_shape=expected_output_shape,
+          validate_training=validate_training)
+
   @parameterized.named_parameters(
       ('padding_valid', {
           'padding': 'valid'
@@ -313,6 +363,8 @@ class Conv3DTest(keras_parameterized.TestCase):
     test_training = 'groups' not in kwargs or not test_util.is_xla_enabled()
     if not requires_gpu or test.is_gpu_available(cuda_only=True):
       self._run_test(kwargs, expected_output_shape, test_training)
+      self._run_test_extra_batch_dim(kwargs, expected_output_shape,
+                                     test_training)
 
   def test_conv3d_regularizers(self):
     kwargs = {
@@ -394,7 +446,7 @@ class GroupedConvTest(keras_parameterized.TestCase):
       ('Conv2D', keras.layers.Conv2D, (32, 12, 12, 32)),
       ('Conv3D', keras.layers.Conv3D, (32, 12, 12, 12, 32)),
   )
-  def test_group_conv(self, layer_cls, input_shape):
+  def disable_test_group_conv(self, layer_cls, input_shape):
     if test.is_gpu_available(cuda_only=True):
       with test_util.use_gpu():
         inputs = random_ops.random_uniform(shape=input_shape)
diff --git a/tensorflow/python/keras/layers/core.py b/tensorflow/python/keras/layers/core.py
index 61820afdf2a..abfb025db30 100644
--- a/tensorflow/python/keras/layers/core.py
+++ b/tensorflow/python/keras/layers/core.py
@@ -39,6 +39,7 @@ from tensorflow.python.keras import backend as K
 from tensorflow.python.keras import constraints
 from tensorflow.python.keras import initializers
 from tensorflow.python.keras import regularizers
+from tensorflow.python.keras.engine import keras_tensor
 from tensorflow.python.keras.engine.base_layer import Layer
 from tensorflow.python.keras.engine.input_spec import InputSpec
 from tensorflow.python.keras.layers.ops import core as core_ops
@@ -52,8 +53,12 @@ from tensorflow.python.ops import nn
 from tensorflow.python.ops import variable_scope
 from tensorflow.python.platform import tf_logging
 from tensorflow.python.training.tracking import base as trackable
+from tensorflow.python.util import dispatch
 from tensorflow.python.util import nest
+from tensorflow.python.util import tf_decorator
 from tensorflow.python.util import tf_inspect
+from tensorflow.python.util.tf_export import get_canonical_name_for_symbol
+from tensorflow.python.util.tf_export import get_symbol_from_name
 from tensorflow.python.util.tf_export import keras_export
 
 
@@ -1025,7 +1030,7 @@ class Lambda(Layer):
   def _parse_function_from_config(
       cls, config, custom_objects, func_attr_name, module_attr_name,
       func_type_attr_name):
-    globs = globals()
+    globs = globals().copy()
     module = config.pop(module_attr_name, None)
     if module in sys.modules:
       globs.update(sys.modules[module].__dict__)
@@ -1255,3 +1260,155 @@ class ActivityRegularization(Layer):
     config = {'l1': self.l1, 'l2': self.l2}
     base_config = super(ActivityRegularization, self).get_config()
     return dict(list(base_config.items()) + list(config.items()))
+
+
+class TFOpLambda(Layer):
+  """Wraps TF API symbols in a `Layer` object.
+
+  It is inserted by the Functional API construction whenever users call
+  a supported TF symbol on KerasTensors.
+
+  Like Lambda layers, this layer tries to raise warnings when it detects users
+  explicitly use variables in the call. (To let them know
+  that the layer will not capture the variables).
+
+  This is useful in the case where users do something like:
+  x = keras.Input(...)
+  y = tf.Variable(...)
+  out = x * tf_variable
+  """
+
+  @trackable.no_automatic_dependency_tracking
+  def __init__(self, function, **kwargs):
+    self.function = function
+    self.symbol = (
+        get_canonical_name_for_symbol(self.function,
+                                      add_prefix_to_v1_names=True) or
+        get_canonical_name_for_symbol(self.function,
+                                      api_name='keras',
+                                      add_prefix_to_v1_names=True))
+    kwargs['autocast'] = False
+
+    # Decorate the function to produce this layer's call method
+    def _call_wrapper(*args, **kwargs):
+      return self._call_wrapper(*args, **kwargs)
+    self.call = tf_decorator.make_decorator(function, _call_wrapper)
+
+    super(TFOpLambda, self).__init__(**kwargs)
+
+    # Warning on every invocation will be quite irksome in Eager mode.
+    self._already_warned = False
+
+    self._expects_training_arg = False
+    self._expects_mask_arg = False
+
+  def _call_wrapper(self, *args, **kwargs):
+    created_variables = []
+    def _variable_creator(next_creator, **creator_kwargs):
+      var = next_creator(**creator_kwargs)
+      created_variables.append(var)
+      return var
+
+    with backprop.GradientTape(watch_accessed_variables=True) as tape, \
+        variable_scope.variable_creator_scope(_variable_creator):
+      # We explicitly drop `name` arguments here,
+      # to guard against the case where an op explicitly has a
+      # `name` passed (which is susceptible to producing
+      # multiple ops w/ the same name when the layer is reused)
+      kwargs.pop('name', None)
+      result = self.function(*args, **kwargs)
+    self._check_variables(created_variables, tape.watched_variables())
+    return result
+
+  def _check_variables(self, created_variables, accessed_variables):
+    if not created_variables and not accessed_variables:
+      # In the common case that a Lambda layer does not touch a Variable, we
+      # don't want to incur the runtime cost of assembling any state used for
+      # checking only to immediately discard it.
+      return
+
+    tracked_weights = set(v.ref() for v in self.weights)
+    untracked_new_vars = [
+        v for v in created_variables if v.ref() not in tracked_weights
+    ]
+    if untracked_new_vars:
+      variable_str = '\n'.join('  {}'.format(i) for i in untracked_new_vars)
+      error_str = textwrap.dedent(
+          '''
+          The following Variables were created within a Lambda layer ({name})
+          but are not tracked by said layer:
+          {variable_str}
+          The layer cannot safely ensure proper Variable reuse across multiple
+          calls, and consquently this behavior is disallowed for safety. Lambda
+          layers are not well suited to stateful computation; instead, writing a
+          subclassed Layer is the recommend way to define layers with
+          Variables.'''
+      ).format(name=self.name, variable_str=variable_str)
+      raise ValueError(error_str)
+
+    untracked_used_vars = [
+        v for v in accessed_variables if v.ref() not in tracked_weights
+    ]
+    if untracked_used_vars and not self._already_warned:
+      variable_str = '\n'.join('  {}'.format(i) for i in untracked_used_vars)
+      self._warn(textwrap.dedent(
+          '''
+          The following Variables were used a Lambda layer's call ({name}), but
+          are not present in its tracked objects:
+          {variable_str}
+          It is possible that this is intended behavior, but it is more likely
+          an omission. This is a strong indication that this layer should be
+          formulated as a subclassed Layer rather than a Lambda layer.'''
+      ).format(name=self.name, variable_str=variable_str))
+      self._already_warned = True
+
+  def _warn(self, msg):
+    # This method will be overridden in a unit test to raise an error, because
+    # self.assertWarns is not universally implemented.
+    return tf_logging.warn(msg)
+
+  def get_config(self):
+    if not self.symbol:
+      raise ValueError('This Keras op layer was generated from %s, a method '
+                       'that is not an exposed in the TensorFlow API. This '
+                       'may have happened if the method was explicitly '
+                       'decorated to add dispatching support, and it was used '
+                       'during Functional model construction. '
+                       'To ensure cross-version compatibility of Keras models '
+                       'that use op layers, only op layers produced from '
+                       'exported TF API symbols can be serialized.'
+                       % self.function)
+    config = {
+        'function': self.symbol
+    }
+
+    base_config = super(TFOpLambda, self).get_config()
+    return dict(list(base_config.items()) + list(config.items()))
+
+  @classmethod
+  def from_config(cls, config, custom_objects=None):
+    config = config.copy()
+    symbol_name = config['function']
+    function = get_symbol_from_name(symbol_name)
+    if not function:
+      raise ValueError(
+          'TF symbol `tf.%s` could not be found.' % symbol_name)
+
+    config['function'] = function
+
+    return cls(**config)
+
+
+class KerasOpDispatcher(dispatch.GlobalOpDispatcher):
+  """A global dispatcher that allows building a functional model with TF Ops."""
+
+  def handle(self, op, args, kwargs):
+    """Handle the specified operation with the specified arguments."""
+    if any(
+        isinstance(x, keras_tensor.KerasTensor)
+        for x in nest.flatten([args, kwargs])):
+      return TFOpLambda(op)(*args, **kwargs)
+    else:
+      return self.NOT_SUPPORTED
+
+KerasOpDispatcher().register()
diff --git a/tensorflow/python/keras/layers/core_test.py b/tensorflow/python/keras/layers/core_test.py
index 70ad63c17eb..15cd8157c0c 100644
--- a/tensorflow/python/keras/layers/core_test.py
+++ b/tensorflow/python/keras/layers/core_test.py
@@ -29,6 +29,7 @@ from tensorflow.python.framework import ops
 from tensorflow.python.framework import tensor_spec
 from tensorflow.python.keras import keras_parameterized
 from tensorflow.python.keras import testing_utils
+from tensorflow.python.keras.layers import core
 from tensorflow.python.keras.mixed_precision.experimental import policy
 from tensorflow.python.ops import array_ops
 from tensorflow.python.ops import math_ops
@@ -274,6 +275,12 @@ class LambdaLayerTest(keras_parameterized.TestCase):
     expected_out = ragged_factory_ops.constant([[2.0], [3.0, 4.0]])
     self.assertAllClose(out, expected_out)
 
+  def test_lambda_deserialization_does_not_pollute_core(self):
+    layer = keras.layers.Lambda(lambda x: x + 1)
+    config = layer.get_config()
+    keras.layers.Lambda.from_config(config)
+    self.assertNotIn(self.__class__.__name__, dir(core))
+
 
 class TestStatefulLambda(keras_parameterized.TestCase):
 
diff --git a/tensorflow/python/keras/layers/legacy_rnn/rnn_cell_impl.py b/tensorflow/python/keras/layers/legacy_rnn/rnn_cell_impl.py
index 1d03780d51b..43bcd799f8b 100644
--- a/tensorflow/python/keras/layers/legacy_rnn/rnn_cell_impl.py
+++ b/tensorflow/python/keras/layers/legacy_rnn/rnn_cell_impl.py
@@ -245,8 +245,8 @@ class RNNCell(base_layer.Layer):
 
   def _rnn_get_variable(self, getter, *args, **kwargs):
     variable = getter(*args, **kwargs)
-    if context.executing_eagerly():
-      trainable = variable._trainable  # pylint: disable=protected-access
+    if ops.executing_eagerly_outside_functions():
+      trainable = variable.trainable
     else:
       trainable = (
           variable in tf_variables.trainable_variables() or
diff --git a/tensorflow/python/keras/layers/merge.py b/tensorflow/python/keras/layers/merge.py
index 73646a638ea..211dd95645f 100644
--- a/tensorflow/python/keras/layers/merge.py
+++ b/tensorflow/python/keras/layers/merge.py
@@ -523,7 +523,11 @@ class Concatenate(_Merge):
 
   @tf_utils.shape_type_conversion
   def compute_output_shape(self, input_shape):
-    if not isinstance(input_shape, (tuple, list)):
+    if ((not isinstance(input_shape, (tuple, list))) or
+        (not isinstance(input_shape[0], (tuple, list)))):
+      # The tf_utils.shape_type_conversion decorator turns tensorshapes
+      # into tuples, so we need to verify that `input_shape` is a list/tuple,
+      # *and* that the individual elements are themselves shape tuples.
       raise ValueError('A `Concatenate` layer should be called '
                        'on a list of inputs.')
     input_shapes = input_shape
diff --git a/tensorflow/python/keras/layers/preprocessing/category_crossing.py b/tensorflow/python/keras/layers/preprocessing/category_crossing.py
index 84e5332bea5..fa0237595ac 100644
--- a/tensorflow/python/keras/layers/preprocessing/category_crossing.py
+++ b/tensorflow/python/keras/layers/preprocessing/category_crossing.py
@@ -19,8 +19,10 @@ from __future__ import division
 from __future__ import print_function
 
 import itertools
+import numpy as np
 
 from tensorflow.python.framework import dtypes
+from tensorflow.python.framework import ops
 from tensorflow.python.framework import sparse_tensor
 from tensorflow.python.framework import tensor_shape
 from tensorflow.python.framework import tensor_spec
@@ -40,8 +42,8 @@ class CategoryCrossing(Layer):
   output (similar to Cartesian product). The output dtype is string.
 
   Usage:
-  >>> inp_1 = tf.constant([['a'], ['b'], ['c']])
-  >>> inp_2 = tf.constant([['d'], ['e'], ['f']])
+  >>> inp_1 = ['a', 'b', 'c']
+  >>> inp_2 = ['d', 'e', 'f']
   >>> layer = tf.keras.layers.experimental.preprocessing.CategoryCrossing()
   >>> layer([inp_1, inp_2])
   <tf.Tensor: shape=(3, 1), dtype=string, numpy=
@@ -50,8 +52,8 @@ class CategoryCrossing(Layer):
            [b'c_X_f']], dtype=object)>
 
 
-  >>> inp_1 = tf.constant([['a'], ['b'], ['c']])
-  >>> inp_2 = tf.constant([['d'], ['e'], ['f']])
+  >>> inp_1 = ['a', 'b', 'c']
+  >>> inp_2 = ['d', 'e', 'f']
   >>> layer = tf.keras.layers.experimental.preprocessing.CategoryCrossing(
   ...    separator='-')
   >>> layer([inp_1, inp_2])
@@ -137,7 +139,15 @@ class CategoryCrossing(Layer):
       return sparse_ops.sparse_tensor_to_dense(
           sparse_ops.sparse_cross(partial_inputs, separator=self.separator))
 
+  def _preprocess_input(self, inp):
+    if isinstance(inp, (list, tuple, np.ndarray)):
+      inp = ops.convert_to_tensor(inp)
+      if inp.shape.rank == 1:
+        inp = array_ops.expand_dims(inp, axis=-1)
+    return inp
+
   def call(self, inputs):
+    inputs = [self._preprocess_input(inp) for inp in inputs]
     depth_tuple = self._depth_tuple if self.depth else (len(inputs),)
     ragged_out = sparse_out = False
     if any(ragged_tensor.is_ragged(inp) for inp in inputs):
diff --git a/tensorflow/python/keras/layers/preprocessing/category_crossing_test.py b/tensorflow/python/keras/layers/preprocessing/category_crossing_test.py
index f076c9ea865..83e78c4dd46 100644
--- a/tensorflow/python/keras/layers/preprocessing/category_crossing_test.py
+++ b/tensorflow/python/keras/layers/preprocessing/category_crossing_test.py
@@ -179,6 +179,18 @@ class CategoryCrossingTest(keras_parameterized.TestCase):
     output = layer([inputs_0, inputs_1])
     self.assertAllEqual([[b'1_X_1', b'1_X_3', b'2_X_1', b'2_X_3']], output)
 
+  def test_crossing_with_list_inputs(self):
+    layer = category_crossing.CategoryCrossing()
+    inputs_0 = [[1, 2]]
+    inputs_1 = [[1, 3]]
+    output = layer([inputs_0, inputs_1])
+    self.assertAllEqual([[b'1_X_1', b'1_X_3', b'2_X_1', b'2_X_3']], output)
+
+    inputs_0 = [1, 2]
+    inputs_1 = [1, 3]
+    output = layer([inputs_0, inputs_1])
+    self.assertAllEqual([[b'1_X_1'], [b'2_X_3']], output)
+
   def test_crossing_dense_inputs_depth_int(self):
     layer = category_crossing.CategoryCrossing(depth=1)
     inputs_0 = constant_op.constant([['a'], ['b'], ['c']])
diff --git a/tensorflow/python/keras/layers/preprocessing/category_encoding.py b/tensorflow/python/keras/layers/preprocessing/category_encoding.py
index a0b7d275e35..74f5a3a7ed8 100644
--- a/tensorflow/python/keras/layers/preprocessing/category_encoding.py
+++ b/tensorflow/python/keras/layers/preprocessing/category_encoding.py
@@ -63,7 +63,7 @@ class CategoryEncoding(base_preprocessing_layer.CombinerPreprocessingLayer):
   Examples:
 
   >>> layer = tf.keras.layers.experimental.preprocessing.CategoryEncoding(
-  ...           max_tokens=4)
+  ...           max_tokens=4, output_mode="count")
   >>> layer([[0, 1], [0, 0], [1, 2], [3, 1]])
   <tf.Tensor: shape=(4, 4), dtype=float32, numpy=
     array([[1., 1., 0., 0.],
@@ -75,7 +75,7 @@ class CategoryEncoding(base_preprocessing_layer.CombinerPreprocessingLayer):
   Examples with weighted inputs:
 
   >>> layer = tf.keras.layers.experimental.preprocessing.CategoryEncoding(
-  ...           max_tokens=4)
+  ...           max_tokens=4, output_mode="count")
   >>> count_weights = np.array([[.1, .2], [.1, .1], [.2, .3], [.4, .2]])
   >>> layer([[0, 1], [0, 0], [1, 2], [3, 1]], count_weights=count_weights)
   <tf.Tensor: shape=(4, 4), dtype=float64, numpy=
@@ -88,7 +88,8 @@ class CategoryEncoding(base_preprocessing_layer.CombinerPreprocessingLayer):
   Attributes:
     max_tokens: The maximum size of the vocabulary for this layer. If None,
       there is no cap on the size of the vocabulary.
-    output_mode: Optional specification for the output of the layer. Values can
+    output_mode: Specification for the output of the layer.
+      Defaults to "binary". Values can
       be "binary", "count" or "tf-idf", configuring the layer as follows:
         "binary": Outputs a single int array per batch, of either vocab_size or
           max_tokens size, containing 1s in all elements where the token mapped
@@ -109,7 +110,7 @@ class CategoryEncoding(base_preprocessing_layer.CombinerPreprocessingLayer):
 
   def __init__(self,
                max_tokens=None,
-               output_mode=COUNT,
+               output_mode=BINARY,
                sparse=False,
                **kwargs):
     # 'output_mode' must be one of (COUNT, BINARY, TFIDF)
diff --git a/tensorflow/python/keras/layers/recurrent_v2.py b/tensorflow/python/keras/layers/recurrent_v2.py
index adefb689a1f..14ccbebbce5 100644
--- a/tensorflow/python/keras/layers/recurrent_v2.py
+++ b/tensorflow/python/keras/layers/recurrent_v2.py
@@ -603,7 +603,7 @@ def gpu_gru(inputs, init_h, kernel, recurrent_kernel, bias, mask, time_major,
   # (6 * units)
   bias = array_ops.split(K.flatten(bias), 6)
 
-  if build_info.is_cuda_build:
+  if build_info.build_info['is_cuda_build']:
     # Note that the gate order for CuDNN is different from the canonical format.
     # canonical format is [z, r, h], whereas CuDNN is [r, z, h]. The swap need
     # to be done for kernel, recurrent_kernel, input_bias, recurrent_bias.
@@ -1365,7 +1365,7 @@ def gpu_lstm(inputs, init_h, init_c, kernel, recurrent_kernel, bias, mask,
   # so that mathematically it is same as the canonical LSTM implementation.
   full_bias = array_ops.concat((array_ops.zeros_like(bias), bias), 0)
 
-  if build_info.is_rocm_build:
+  if build_info.build_info['is_rocm_build']:
     # ROCm MIOpen's weight sequence for LSTM is different from both canonical
     # and Cudnn format
     # MIOpen: [i, f, o, c] Cudnn/Canonical: [i, f, c, o]
diff --git a/tensorflow/python/keras/layers/tensorflow_op_layer_test.py b/tensorflow/python/keras/layers/tensorflow_op_layer_test.py
index 1a328995a80..0bd479c6923 100644
--- a/tensorflow/python/keras/layers/tensorflow_op_layer_test.py
+++ b/tensorflow/python/keras/layers/tensorflow_op_layer_test.py
@@ -50,7 +50,6 @@ def _single_identity_op_at_end():
   inputs = keras.Input(shape=(10,))
   x = keras.layers.Dense(10)(inputs)
   outputs = array_ops.identity(x)
-  assert 'Identity' in outputs.name
   return keras.Model(inputs, outputs)
 
 
@@ -186,7 +185,7 @@ def _reuse_ancillary_layer():
   return model
 
 
-@keras_parameterized.run_all_keras_modes
+@keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
 class AutoLambdaTest(keras_parameterized.TestCase):
 
   @parameterized.named_parameters(
@@ -313,11 +312,6 @@ class AutoLambdaTest(keras_parameterized.TestCase):
     e = 3  # Fudge factor to prevent flakiness.
     self.assertLess(size_500, (10 * e) * size_50)
 
-  def test_no_mask_tracking(self):
-    x = keras.backend.placeholder((10, 10))
-    y = keras.layers.Masking(0.)(x)
-    self.assertTrue(y._keras_mask._keras_history_checked)
-
   def test_built(self):
     inputs = keras.Input(shape=(10,))
     outputs = gen_nn_ops.relu(inputs)
diff --git a/tensorflow/python/keras/losses.py b/tensorflow/python/keras/losses.py
index 54afe2e26ce..89e6e1e6d24 100644
--- a/tensorflow/python/keras/losses.py
+++ b/tensorflow/python/keras/losses.py
@@ -1414,7 +1414,8 @@ def categorical_hinge(y_true, y_pred):
   y_true = math_ops.cast(y_true, y_pred.dtype)
   pos = math_ops.reduce_sum(y_true * y_pred, axis=-1)
   neg = math_ops.reduce_max((1. - y_true) * y_pred, axis=-1)
-  return math_ops.maximum(0., neg - pos + 1.)
+  zero = math_ops.cast(0., y_pred.dtype)
+  return math_ops.maximum(neg - pos + 1., zero)
 
 
 @keras_export('keras.losses.huber', v1=[])
diff --git a/tensorflow/python/keras/models_test.py b/tensorflow/python/keras/models_test.py
index d7ef7d73208..4db01d66edb 100644
--- a/tensorflow/python/keras/models_test.py
+++ b/tensorflow/python/keras/models_test.py
@@ -79,7 +79,7 @@ def _get_model(input_shape=(4,)):
 
 class TestModelCloning(keras_parameterized.TestCase):
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   @parameterized.named_parameters([
       {'testcase_name': 'has_input_layer',
        'input_shape': (4,),
@@ -142,7 +142,7 @@ class TestModelCloning(keras_parameterized.TestCase):
       self.assertIsInstance(new_model._layers[0], keras.layers.InputLayer)
       self.assertTrue(new_model._is_graph_network)
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   @parameterized.named_parameters([
       {'testcase_name': 'clone_weights', 'share_weights': False},
       {'testcase_name': 'share_weights', 'share_weights': True},
diff --git a/tensorflow/python/keras/optimizer_v2/BUILD b/tensorflow/python/keras/optimizer_v2/BUILD
index 9e844b41332..be793378538 100644
--- a/tensorflow/python/keras/optimizer_v2/BUILD
+++ b/tensorflow/python/keras/optimizer_v2/BUILD
@@ -105,6 +105,7 @@ cuda_py_test(
     size = "medium",
     srcs = ["adam_test.py"],
     shard_count = 4,
+    tags = ["no_rocm"],
     deps = [
         ":optimizer_v2",
         "//tensorflow/python:client_testlib",
diff --git a/tensorflow/python/keras/premade/linear_test.py b/tensorflow/python/keras/premade/linear_test.py
index 7c560de44dd..6fa1767a60a 100644
--- a/tensorflow/python/keras/premade/linear_test.py
+++ b/tensorflow/python/keras/premade/linear_test.py
@@ -22,6 +22,7 @@ import numpy as np
 
 from tensorflow.python.eager import backprop
 from tensorflow.python.eager import context
+from tensorflow.python.feature_column import feature_column_v2 as fc
 from tensorflow.python.framework import constant_op
 from tensorflow.python.framework import dtypes
 from tensorflow.python.framework import sparse_tensor
@@ -29,7 +30,9 @@ from tensorflow.python.keras import backend
 from tensorflow.python.keras import keras_parameterized
 from tensorflow.python.keras import losses
 from tensorflow.python.keras.engine import input_layer
+from tensorflow.python.keras.engine import sequential
 from tensorflow.python.keras.engine import training
+from tensorflow.python.keras.feature_column import dense_features_v2
 from tensorflow.python.keras.layers import core
 from tensorflow.python.keras.optimizer_v2 import gradient_descent
 from tensorflow.python.keras.premade import linear
@@ -37,7 +40,8 @@ from tensorflow.python.ops import variables
 from tensorflow.python.platform import test
 
 
-@keras_parameterized.run_all_keras_modes(always_skip_v1=True)
+@keras_parameterized.run_all_keras_modes(always_skip_v1=True,
+                                         skip_keras_tensors=True)
 class LinearModelTest(keras_parameterized.TestCase):
 
   def test_linear_model_with_single_input(self):
@@ -126,6 +130,33 @@ class LinearModelTest(keras_parameterized.TestCase):
         grads_and_vars = zip(grads, model.trainable_variables)
         opt.apply_gradients(grads_and_vars)
 
+  # This test is an example for a regression on categorical inputs, i.e.,
+  # the output is 0.4, 0.6, 0.9 when input is 'alpha', 'beta', 'gamma'
+  # separately.
+  def test_linear_model_with_feature_column(self):
+    with context.eager_mode():
+      vocab_list = ['alpha', 'beta', 'gamma']
+      vocab_val = [0.4, 0.6, 0.9]
+      data = np.random.choice(vocab_list, size=256)
+      y = np.zeros_like(data, dtype=np.float32)
+      for vocab, val in zip(vocab_list, vocab_val):
+        indices = np.where(data == vocab)
+        y[indices] = val + np.random.uniform(
+            low=-0.01, high=0.01, size=indices[0].shape)
+      cat_column = fc.categorical_column_with_vocabulary_list(
+          key='symbol', vocabulary_list=vocab_list)
+      ind_column = fc.indicator_column(cat_column)
+      dense_feature_layer = dense_features_v2.DenseFeatures([ind_column])
+      linear_model = linear.LinearModel(
+          use_bias=False, kernel_initializer='zeros')
+      combined = sequential.Sequential([dense_feature_layer, linear_model])
+      opt = gradient_descent.SGD(learning_rate=0.1)
+      combined.compile(opt, 'mse', [])
+      combined.fit(x={'symbol': data}, y=y, batch_size=32, epochs=10)
+      self.assertAllClose([[0.4], [0.6], [0.9]],
+                          combined.layers[1].dense_layers[0].kernel.numpy(),
+                          atol=0.01)
+
   def test_config(self):
     linear_model = linear.LinearModel(units=3, use_bias=True)
     config = linear_model.get_config()
diff --git a/tensorflow/python/keras/premade/wide_deep_test.py b/tensorflow/python/keras/premade/wide_deep_test.py
index 5388d1129ab..eae28c31df8 100644
--- a/tensorflow/python/keras/premade/wide_deep_test.py
+++ b/tensorflow/python/keras/premade/wide_deep_test.py
@@ -21,11 +21,13 @@ from __future__ import print_function
 import numpy as np
 
 from tensorflow.python.eager import context
+from tensorflow.python.feature_column import feature_column_v2 as fc
 from tensorflow.python.keras import keras_parameterized
 from tensorflow.python.keras import testing_utils
 from tensorflow.python.keras.engine import input_layer
 from tensorflow.python.keras.engine import sequential
 from tensorflow.python.keras.engine import training
+from tensorflow.python.keras.feature_column import dense_features_v2
 from tensorflow.python.keras.layers import core
 from tensorflow.python.keras.optimizer_v2 import gradient_descent
 from tensorflow.python.keras.premade import linear
@@ -35,7 +37,8 @@ from tensorflow.python.ops import variables
 from tensorflow.python.platform import test
 
 
-@keras_parameterized.run_all_keras_modes(always_skip_v1=True)
+@keras_parameterized.run_all_keras_modes(always_skip_v1=True,
+                                         skip_keras_tensors=True)
 class WideDeepModelTest(keras_parameterized.TestCase):
 
   def test_wide_deep_model(self):
@@ -186,6 +189,67 @@ class WideDeepModelTest(keras_parameterized.TestCase):
         run_eagerly=testing_utils.should_run_eagerly())
     wide_deep_model.fit(inputs, output, epochs=50)
 
+  # This test is an example for cases where linear and dnn model accepts
+  # same raw input and same transformed inputs, i.e., the raw input is
+  # categorical, and both linear and dnn model accept one hot encoding.
+  def test_wide_deep_model_with_single_feature_column(self):
+    vocab_list = ['alpha', 'beta', 'gamma']
+    vocab_val = [0.4, 0.6, 0.9]
+    data = np.random.choice(vocab_list, size=256)
+    y = np.zeros_like(data, dtype=np.float32)
+    for vocab, val in zip(vocab_list, vocab_val):
+      indices = np.where(data == vocab)
+      y[indices] = val + np.random.uniform(
+          low=-0.01, high=0.01, size=indices[0].shape)
+    cat_column = fc.categorical_column_with_vocabulary_list(
+        key='symbol', vocabulary_list=vocab_list)
+    ind_column = fc.indicator_column(cat_column)
+    dense_feature_layer = dense_features_v2.DenseFeatures([ind_column])
+    linear_model = linear.LinearModel(
+        use_bias=False, kernel_initializer='zeros')
+    dnn_model = sequential.Sequential([core.Dense(units=1)])
+    wide_deep_model = wide_deep.WideDeepModel(linear_model, dnn_model)
+    combined = sequential.Sequential([dense_feature_layer, wide_deep_model])
+    opt = gradient_descent.SGD(learning_rate=0.1)
+    combined.compile(
+        opt,
+        'mse', [],
+        run_eagerly=testing_utils.should_run_eagerly())
+    combined.fit(x={'symbol': data}, y=y, batch_size=32, epochs=10)
+
+  # This test is an example for cases where linear and dnn model accepts
+  # same raw input but different transformed inputs, i.e,. the raw input is
+  # categorical, and linear model accepts one hot encoding, while dnn model
+  # accepts embedding encoding.
+  def test_wide_deep_model_with_two_feature_columns(self):
+    vocab_list = ['alpha', 'beta', 'gamma']
+    vocab_val = [0.4, 0.6, 0.9]
+    data = np.random.choice(vocab_list, size=256)
+    y = np.zeros_like(data, dtype=np.float32)
+    for vocab, val in zip(vocab_list, vocab_val):
+      indices = np.where(data == vocab)
+      y[indices] = val + np.random.uniform(
+          low=-0.01, high=0.01, size=indices[0].shape)
+    cat_column = fc.categorical_column_with_vocabulary_list(
+        key='symbol', vocabulary_list=vocab_list)
+    ind_column = fc.indicator_column(cat_column)
+    emb_column = fc.embedding_column(cat_column, dimension=5)
+    linear_feature_layer = dense_features_v2.DenseFeatures([ind_column])
+    linear_model = linear.LinearModel(
+        use_bias=False, kernel_initializer='zeros')
+    combined_linear = sequential.Sequential(
+        [linear_feature_layer, linear_model])
+    dnn_model = sequential.Sequential([core.Dense(units=1)])
+    dnn_feature_layer = dense_features_v2.DenseFeatures([emb_column])
+    combined_dnn = sequential.Sequential([dnn_feature_layer, dnn_model])
+    wide_deep_model = wide_deep.WideDeepModel(combined_linear, combined_dnn)
+    opt = gradient_descent.SGD(learning_rate=0.1)
+    wide_deep_model.compile(
+        opt,
+        'mse', [],
+        run_eagerly=testing_utils.should_run_eagerly())
+    wide_deep_model.fit(x={'symbol': data}, y=y, batch_size=32, epochs=10)
+
   def test_config(self):
     linear_model = linear.LinearModel(units=1)
     dnn_model = sequential.Sequential([core.Dense(units=1, input_dim=3)])
diff --git a/tensorflow/python/keras/protobuf/projector_config.proto b/tensorflow/python/keras/protobuf/projector_config.proto
index b7105b949d0..8861dc9b4b0 100644
--- a/tensorflow/python/keras/protobuf/projector_config.proto
+++ b/tensorflow/python/keras/protobuf/projector_config.proto
@@ -13,6 +13,10 @@ See the License for the specific language governing permissions and
 limitations under the License.
 ==============================================================================*/
 
+// This file is a copy of the TensorBoard ProjectorConfig proto.
+// Keep this file in sync with the source proto definition at
+// https://github.com/tensorflow/tensorboard/blob/master/tensorboard/plugins/projector/projector_config.proto
+
 syntax = "proto3";
 
 package third_party.tensorflow.python.keras.protobuf;
diff --git a/tensorflow/python/keras/regularizers_test.py b/tensorflow/python/keras/regularizers_test.py
index b10218ba114..c2c2e6c4a01 100644
--- a/tensorflow/python/keras/regularizers_test.py
+++ b/tensorflow/python/keras/regularizers_test.py
@@ -83,7 +83,7 @@ class KerasRegularizersTest(keras_parameterized.TestCase,
     self.assertEqual(len(model.losses), 1)
     model.fit(x_train, y_train, batch_size=10, epochs=1, verbose=0)
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   @parameterized.named_parameters([
       ('l1', regularizers.l1()),
       ('l2', regularizers.l2()),
@@ -126,7 +126,7 @@ class KerasRegularizersTest(keras_parameterized.TestCase,
         model.get_config(), custom_objects={'my_regularizer': my_regularizer})
     self.assertEqual(model2.layers[1].kernel_regularizer, my_regularizer)
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   @parameterized.named_parameters([
       ('l1', regularizers.l1()),
       ('l2', regularizers.l2()),
@@ -144,7 +144,7 @@ class KerasRegularizersTest(keras_parameterized.TestCase,
         run_eagerly=testing_utils.should_run_eagerly())
     self.assertLen(model.losses, 5)
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   @parameterized.named_parameters([
       ('l1', regularizers.l1()),
       ('l2', regularizers.l2()),
@@ -166,7 +166,7 @@ class KerasRegularizersTest(keras_parameterized.TestCase,
         run_eagerly=testing_utils.should_run_eagerly())
     self.assertLen(model.losses, 6)
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   @parameterized.named_parameters([
       ('l1', regularizers.l1()),
       ('l2', regularizers.l2()),
diff --git a/tensorflow/python/keras/saving/BUILD b/tensorflow/python/keras/saving/BUILD
index e25a733e0c9..258c1dbfaec 100644
--- a/tensorflow/python/keras/saving/BUILD
+++ b/tensorflow/python/keras/saving/BUILD
@@ -109,6 +109,7 @@ tf_py_test(
     python_version = "PY3",
     deps = [
         "//tensorflow/python:client_testlib",
+        "//tensorflow/python/feature_column:feature_column_v2",
         "//tensorflow/python/keras",
         "//tensorflow/python/keras:combinations",
         "//third_party/py/numpy",
@@ -146,6 +147,7 @@ tf_py_test(
     ],
     deps = [
         "//tensorflow/python:client_testlib",
+        "//tensorflow/python/compat:v2_compat",
         "//tensorflow/python/distribute:mirrored_strategy",
         "//tensorflow/python/keras",
         "//tensorflow/python/keras:combinations",
diff --git a/tensorflow/python/keras/saving/save_test.py b/tensorflow/python/keras/saving/save_test.py
index f0a0d82ade9..5c5846fe738 100644
--- a/tensorflow/python/keras/saving/save_test.py
+++ b/tensorflow/python/keras/saving/save_test.py
@@ -25,14 +25,19 @@ from absl.testing import parameterized
 import numpy as np
 
 from tensorflow.python import keras
+from tensorflow.python.eager import context
+from tensorflow.python.feature_column import feature_column_lib
+from tensorflow.python.framework import sparse_tensor
 from tensorflow.python.framework import test_util
 from tensorflow.python.keras import combinations
 from tensorflow.python.keras import losses
 from tensorflow.python.keras import testing_utils
 from tensorflow.python.keras.engine import sequential
 from tensorflow.python.keras.layers import core
+from tensorflow.python.keras.saving import model_config
 from tensorflow.python.keras.saving import save
 from tensorflow.python.keras.utils import generic_utils
+from tensorflow.python.ops import lookup_ops
 from tensorflow.python.platform import test
 from tensorflow.python.saved_model import loader_impl
 
@@ -100,6 +105,101 @@ class TestSaveModel(test.TestCase, parameterized.TestCase):
       save.save_model(self.model, path, save_format='tf')
       save.load_model(path)
 
+  @combinations.generate(combinations.combine(mode=['graph', 'eager']))
+  def test_saving_with_dense_features(self):
+    cols = [
+        feature_column_lib.numeric_column('a'),
+        feature_column_lib.indicator_column(
+            feature_column_lib.categorical_column_with_vocabulary_list(
+                'b', ['one', 'two']))
+    ]
+    input_layers = {
+        'a': keras.layers.Input(shape=(1,), name='a'),
+        'b': keras.layers.Input(shape=(1,), name='b', dtype='string')
+    }
+
+    fc_layer = feature_column_lib.DenseFeatures(cols)(input_layers)
+    output = keras.layers.Dense(10)(fc_layer)
+
+    model = keras.models.Model(input_layers, output)
+
+    model.compile(
+        loss=keras.losses.MSE,
+        optimizer='rmsprop',
+        metrics=[keras.metrics.categorical_accuracy])
+
+    config = model.to_json()
+    loaded_model = model_config.model_from_json(config)
+
+    inputs_a = np.arange(10).reshape(10, 1)
+    inputs_b = np.arange(10).reshape(10, 1).astype('str')
+
+    with self.cached_session():
+      # Initialize tables for V1 lookup.
+      if not context.executing_eagerly():
+        self.evaluate(lookup_ops.tables_initializer())
+
+      self.assertLen(loaded_model.predict({'a': inputs_a, 'b': inputs_b}), 10)
+
+  @combinations.generate(combinations.combine(mode=['graph', 'eager']))
+  def test_saving_with_sequence_features(self):
+    cols = [
+        feature_column_lib.sequence_numeric_column('a'),
+        feature_column_lib.indicator_column(
+            feature_column_lib.sequence_categorical_column_with_vocabulary_list(
+                'b', ['one', 'two']))
+    ]
+    input_layers = {
+        'a':
+            keras.layers.Input(shape=(None, 1), sparse=True, name='a'),
+        'b':
+            keras.layers.Input(
+                shape=(None, 1), sparse=True, name='b', dtype='string')
+    }
+
+    fc_layer, _ = feature_column_lib.SequenceFeatures(cols)(input_layers)
+    # TODO(tibell): Figure out the right dtype and apply masking.
+    # sequence_length_mask = array_ops.sequence_mask(sequence_length)
+    # x = keras.layers.GRU(32)(fc_layer, mask=sequence_length_mask)
+    x = keras.layers.GRU(32)(fc_layer)
+    output = keras.layers.Dense(10)(x)
+
+    model = keras.models.Model(input_layers, output)
+
+    model.compile(
+        loss=keras.losses.MSE,
+        optimizer='rmsprop',
+        metrics=[keras.metrics.categorical_accuracy])
+
+    config = model.to_json()
+    loaded_model = model_config.model_from_json(config)
+
+    batch_size = 10
+    timesteps = 1
+
+    values_a = np.arange(10, dtype=np.float32)
+    indices_a = np.zeros((10, 3), dtype=np.int64)
+    indices_a[:, 0] = np.arange(10)
+    inputs_a = sparse_tensor.SparseTensor(indices_a, values_a,
+                                          (batch_size, timesteps, 1))
+
+    values_b = np.zeros(10, dtype=np.str)
+    indices_b = np.zeros((10, 3), dtype=np.int64)
+    indices_b[:, 0] = np.arange(10)
+    inputs_b = sparse_tensor.SparseTensor(indices_b, values_b,
+                                          (batch_size, timesteps, 1))
+
+    with self.cached_session():
+      # Initialize tables for V1 lookup.
+      if not context.executing_eagerly():
+        self.evaluate(lookup_ops.tables_initializer())
+
+      self.assertLen(
+          loaded_model.predict({
+              'a': inputs_a,
+              'b': inputs_b
+          }, steps=1), batch_size)
+
   @combinations.generate(combinations.combine(mode=['graph', 'eager']))
   def test_saving_h5_for_rnn_layers(self):
     # See https://github.com/tensorflow/tensorflow/issues/35731 for details.
diff --git a/tensorflow/python/keras/testing_utils.py b/tensorflow/python/keras/testing_utils.py
index b41abbdf1f5..1928588fea1 100644
--- a/tensorflow/python/keras/testing_utils.py
+++ b/tensorflow/python/keras/testing_utils.py
@@ -33,6 +33,7 @@ from tensorflow.python.keras import backend
 from tensorflow.python.keras import layers
 from tensorflow.python.keras import models
 from tensorflow.python.keras.engine import base_layer_utils
+from tensorflow.python.keras.engine import keras_tensor
 from tensorflow.python.keras.optimizer_v2 import adadelta as adadelta_v2
 from tensorflow.python.keras.optimizer_v2 import adagrad as adagrad_v2
 from tensorflow.python.keras.optimizer_v2 import adam as adam_v2
@@ -331,6 +332,29 @@ def run_eagerly_scope(value):
     _thread_local_data.run_eagerly = previous_value
 
 
+@tf_contextlib.contextmanager
+def use_keras_tensors_scope(value):
+  """Provides a scope within which we use KerasTensors in the func. API or not.
+
+  The boolean gets restored to its original value upon exiting the scope.
+
+  Arguments:
+     value: Bool specifying if we should build functional models
+      using KerasTensors in the active test.
+     Should be True or False.
+
+  Yields:
+    The provided value.
+  """
+  previous_value = keras_tensor._KERAS_TENSORS_ENABLED  # pylint: disable=protected-access
+  try:
+    keras_tensor._KERAS_TENSORS_ENABLED = value  # pylint: disable=protected-access
+    yield value
+  finally:
+    # Restore KerasTensor usage to initial value.
+    keras_tensor._KERAS_TENSORS_ENABLED = previous_value  # pylint: disable=protected-access
+
+
 def should_run_eagerly():
   """Returns whether the models we are testing should be run eagerly."""
   if _thread_local_data.run_eagerly is None:
diff --git a/tensorflow/python/keras/tests/add_loss_correctness_test.py b/tensorflow/python/keras/tests/add_loss_correctness_test.py
index a19eec75ffb..a7708b0999d 100644
--- a/tensorflow/python/keras/tests/add_loss_correctness_test.py
+++ b/tensorflow/python/keras/tests/add_loss_correctness_test.py
@@ -69,7 +69,7 @@ class TestAddLossCorrectness(keras_parameterized.TestCase):
     self.y = np.array([[0.5], [2.], [3.5]], dtype='float32')
     self.w = np.array([[1.25], [0.5], [1.25]], dtype='float32')
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   def test_loss_on_model_fit(self):
     inputs = Input(shape=(1,))
     targets = Input(shape=(1,))
@@ -85,7 +85,8 @@ class TestAddLossCorrectness(keras_parameterized.TestCase):
     self.assertAllClose(history.history['loss'], [2., 1.8, 1.6, 1.4, 1.2], 1e-3)
 
   @keras_parameterized.run_with_all_model_types(exclude_models=['sequential'])
-  @keras_parameterized.run_all_keras_modes(always_skip_v1=True)
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True,
+                                           always_skip_v1=True)
   def test_loss_callable_on_model_fit(self):
     model = testing_utils.get_model_from_layers([testing_utils.Bias()],
                                                 input_shape=(1,))
@@ -144,7 +145,7 @@ class TestAddLossCorrectness(keras_parameterized.TestCase):
       loss = [train_step(self.x, self.y) for _ in range(5)]
       self.assertAllClose(loss, [0., -0.05, -0.1, -0.15, -0.2], 1e-3)
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   def test_loss_with_sample_weight_on_model_fit(self):
     inputs = Input(shape=(1,))
     targets = Input(shape=(1,))
@@ -181,7 +182,7 @@ class TestAddLossCorrectness(keras_parameterized.TestCase):
       loss = [train_step(self.x, self.y, self.w) for _ in range(5)]
       self.assertAllClose(loss, [2., 1.8, 1.6, 1.4, 1.2], 1e-3)
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   def test_loss_with_sample_weight_in_model_call(self):
 
     class MyModel(Model):
@@ -209,7 +210,7 @@ class TestAddLossCorrectness(keras_parameterized.TestCase):
     eval_out = model.evaluate([self.x, self.y, self.w])
     self.assertAlmostEqual(eval_out, 1.0, 3)
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   def test_loss_with_sample_weight_in_layer_call(self):
 
     class MyLayer(layers.Layer):
@@ -244,7 +245,7 @@ class TestAddLossCorrectness(keras_parameterized.TestCase):
     output = model.test_on_batch([self.x, self.y, self.w])
     self.assertAlmostEqual(output, 1.0, 3)
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   def test_loss_on_layer(self):
 
     class MyLayer(layers.Layer):
@@ -265,7 +266,7 @@ class TestAddLossCorrectness(keras_parameterized.TestCase):
     loss = model.train_on_batch(np.ones((2, 3)), np.ones((2, 3)))
     self.assertEqual(loss, 2 * 3)
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   @keras_parameterized.run_with_all_model_types
   def test_activity_regularizer(self):
     loss = {}
@@ -299,7 +300,7 @@ class TestAddLossCorrectness(keras_parameterized.TestCase):
       loss[reg] = model.evaluate(x, y)
     self.assertLess(loss[None], loss['l2'])
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   @keras_parameterized.run_with_all_model_types
   def test_activity_regularizer_loss_value(self):
     layer = layers.Dense(
@@ -318,7 +319,7 @@ class TestAddLossCorrectness(keras_parameterized.TestCase):
     loss = model.test_on_batch(x)
     self.assertAlmostEqual(0.01, loss, places=4)
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   def test_activity_regularizer_batch_independent(self):
     inputs = layers.Input(shape=(10,))
     x = layers.Dense(10, activation='relu', activity_regularizer='l2')(inputs)
@@ -334,7 +335,7 @@ class TestAddLossCorrectness(keras_parameterized.TestCase):
     loss_big_batch = model.test_on_batch(np.ones((20, 10), 'float32'))
     self.assertAlmostEqual(loss_small_batch, loss_big_batch, places=4)
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   def test_with_shared_layer(self):
 
     class LayerWithLoss(layers.Layer):
@@ -351,7 +352,7 @@ class TestAddLossCorrectness(keras_parameterized.TestCase):
     self.assertEqual(len(m2.losses), 2)
     self.assertAllClose(m2.losses, [6, 12])
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   def test_with_shared_nested_layer(self):
 
     class LayerWithLoss(layers.Layer):
@@ -377,7 +378,7 @@ class TestAddLossCorrectness(keras_parameterized.TestCase):
     self.assertEqual(len(m2.losses), 2)
     self.assertAllClose(m2.losses, [6, 12])
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   def test_clear_losses(self):
 
     class LayerWithSharedNestedLossLayer(layers.Layer):
@@ -428,7 +429,7 @@ class TestAddLossCorrectness(keras_parameterized.TestCase):
       self.assertEqual(len(model.get_losses_for(x4)), 2)
       self.assertEqual(len(model.get_losses_for(None)), 1)
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   def test_invalid_constant_input(self):
     with context.eager_mode():
       inputs = Input(shape=(1,))
@@ -439,7 +440,7 @@ class TestAddLossCorrectness(keras_parameterized.TestCase):
           'Expected a symbolic Tensors or a callable for the loss value'):
         model.add_loss(1.)
 
-  @keras_parameterized.run_all_keras_modes
+  @keras_parameterized.run_all_keras_modes(skip_keras_tensors=True)
   def test_invalid_variable_input(self):
     with context.eager_mode():
       inputs = Input(shape=(1,))
diff --git a/tensorflow/python/keras/tests/op_callbacks_test.py b/tensorflow/python/keras/tests/op_callbacks_test.py
index 53a3983601a..a8abc07e3d4 100644
--- a/tensorflow/python/keras/tests/op_callbacks_test.py
+++ b/tensorflow/python/keras/tests/op_callbacks_test.py
@@ -43,6 +43,7 @@ _STATELESS_IF_OP = b"StatelessIf"
 _SWITCH_OP = b"Switch"
 _VAR_HANDLE_OP = b"VarHandleOp"
 _WHILE_OP = b"While"
+_CASE_OP = b"Case"
 
 
 class _NumpyFunctionCallback(object):
@@ -80,8 +81,9 @@ class _NumpyFunctionCallback(object):
       for output in outputs:
         if compat.as_bytes(op_type) in (_ENTER_OP, _EXIT_OP, _IF_OP, _MERGE_OP,
                                         _NEXT_ITERATION_OP, _STATELESS_IF_OP,
-                                        _SWITCH_OP, _WHILE_OP, _IDENTITY_OP,
-                                        _VAR_HANDLE_OP, _PLACEHOLDER_OP):
+                                        _SWITCH_OP, _WHILE_OP, _CASE_OP,
+                                        _IDENTITY_OP, _VAR_HANDLE_OP,
+                                        _PLACEHOLDER_OP):
           # TODO(cais): Overriding the output of StatelessIf, If and While ops
           # currently fails with error. Investigate (b/139668453).
           # Avoid instrumenting Identity ops as well, as they are inserted
diff --git a/tensorflow/python/keras/utils/composite_tensor_support_test.py b/tensorflow/python/keras/utils/composite_tensor_support_test.py
index 5b3c3df42b8..f31558ddba8 100644
--- a/tensorflow/python/keras/utils/composite_tensor_support_test.py
+++ b/tensorflow/python/keras/utils/composite_tensor_support_test.py
@@ -506,7 +506,8 @@ class RaggedTensorInputTest(keras_parameterized.TestCase,
     model_input = input_layer.Input(
         shape=(None, None), ragged=True, name=input_name, dtype=dtypes.int32,
         batch_size=2)
-    self.assertIsInstance(model_input, ragged_tensor.RaggedTensor)
+    self.assertIsInstance(model_input._type_spec,
+                          ragged_tensor.RaggedTensorSpec)
     self.assertEqual(model_input.shape.as_list(), [2, None, None])
     layers = [ToDense(default_value=-1)]
     model = get_model_from_layers_with_input(layers, model_input=model_input)
@@ -602,7 +603,8 @@ class RaggedTensorInputValidationTest(keras_parameterized.TestCase,
 
 
 @keras_parameterized.run_with_all_model_types()
-@keras_parameterized.run_all_keras_modes(always_skip_v1=True)
+@keras_parameterized.run_all_keras_modes(always_skip_v1=True,
+                                         skip_keras_tensors=True)
 class CompositeTensorModelPredictTest(keras_parameterized.TestCase):
 
   def _normalize_shape(self, shape):
diff --git a/tensorflow/python/keras/utils/generic_utils.py b/tensorflow/python/keras/utils/generic_utils.py
index 27015cbc8f2..27cab6b65da 100644
--- a/tensorflow/python/keras/utils/generic_utils.py
+++ b/tensorflow/python/keras/utils/generic_utils.py
@@ -296,6 +296,15 @@ def class_and_config_for_serialized_keras_object(
     raise ValueError('Unknown ' + printable_module_name + ': ' + class_name)
 
   cls_config = config['config']
+  # Check if `cls_config` is a list. If it is a list, return the class and the
+  # associated class configs for recursively deserialization. This case will
+  # happen on the old version of sequential model (e.g. `keras_version` ==
+  # "2.0.6"), which is serialized in a different structure, for example
+  # "{'class_name': 'Sequential',
+  #   'config': [{'class_name': 'Embedding', 'config': ...}, {}, ...]}".
+  if isinstance(cls_config, list):
+    return (cls, cls_config)
+
   deserialized_objects = {}
   for key, item in cls_config.items():
     if isinstance(item, dict) and '__passive_serialization__' in item:
diff --git a/tensorflow/python/keras/utils/generic_utils_test.py b/tensorflow/python/keras/utils/generic_utils_test.py
index ddaa60c3c24..1f0c1e2cd89 100644
--- a/tensorflow/python/keras/utils/generic_utils_test.py
+++ b/tensorflow/python/keras/utils/generic_utils_test.py
@@ -18,6 +18,8 @@ from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function
 
+import numpy as np
+
 from tensorflow.python import keras
 from tensorflow.python.platform import test
 
@@ -298,6 +300,51 @@ class SerializeKerasObjectTest(test.TestCase):
     self.assertEqual(new_layer.units, 3)
     self.assertIs(new_layer.units.fn, serializable_fn)
 
+  def test_serializable_with_old_config(self):
+    # model config generated by tf-1.2.1
+    old_model_config = {
+        'class_name':
+            'Sequential',
+        'config': [{
+            'class_name': 'Dense',
+            'config': {
+                'name': 'dense_1',
+                'trainable': True,
+                'batch_input_shape': [None, 784],
+                'dtype': 'float32',
+                'units': 32,
+                'activation': 'linear',
+                'use_bias': True,
+                'kernel_initializer': {
+                    'class_name': 'Ones',
+                    'config': {
+                        'dtype': 'float32'
+                    }
+                },
+                'bias_initializer': {
+                    'class_name': 'Zeros',
+                    'config': {
+                        'dtype': 'float32'
+                    }
+                },
+                'kernel_regularizer': None,
+                'bias_regularizer': None,
+                'activity_regularizer': None,
+                'kernel_constraint': None,
+                'bias_constraint': None
+            }
+        }]
+    }
+    old_model = keras.utils.generic_utils.deserialize_keras_object(
+        old_model_config, module_objects={'Sequential': keras.Sequential})
+    new_model = keras.Sequential([
+        keras.layers.Dense(32, input_dim=784, kernel_initializer='Ones'),
+    ])
+    input_data = np.random.normal(2, 1, (5, 784))
+    output = old_model.predict(input_data)
+    expected_output = new_model.predict(input_data)
+    self.assertAllEqual(output, expected_output)
+
 
 class SliceArraysTest(test.TestCase):
 
diff --git a/tensorflow/python/kernel_tests/clip_ops_test.py b/tensorflow/python/kernel_tests/clip_ops_test.py
index 433ec689375..f6523b9b094 100644
--- a/tensorflow/python/kernel_tests/clip_ops_test.py
+++ b/tensorflow/python/kernel_tests/clip_ops_test.py
@@ -34,7 +34,8 @@ from tensorflow.python.platform import test
 
 class ClipTest(test.TestCase):
 
-  def DISABLED_testClipByValueGradient(self):
+  @test_util.run_deprecated_v1
+  def testClipByValueGradient(self):
     inputs = constant_op.constant([1.0, 2.0, 3.0, 4.0], dtype=dtypes.float32)
     outputs_1 = clip_ops.clip_by_value(inputs, 0.5, 3.5)
     min_val = constant_op.constant([0.5, 0.5, 0.5, 0.5], dtype=dtypes.float32)
@@ -61,10 +62,15 @@ class ClipTest(test.TestCase):
     self.assertAllClose(np_ans, tf_ans)
 
   # [Tensor, Scalar, Scalar]
-  def DISABLED_testClipByValue0Type(self):
+  def testClipByValue0Type(self):
     for dtype in [
-        dtypes.float16, dtypes.float32, dtypes.float64, dtypes.int8,
-        dtypes.int16, dtypes.int32, dtypes.int64, dtypes.uint8, dtypes.uint16
+        dtypes.float16,
+        dtypes.float32,
+        dtypes.float64,
+        dtypes.int16,
+        dtypes.int32,
+        dtypes.int64,
+        dtypes.uint8,
     ]:
       with self.cached_session(use_gpu=True):
         x = constant_op.constant([1, 2, 3, 4, 5, 6], shape=[2, 3], dtype=dtype)
@@ -77,10 +83,15 @@ class ClipTest(test.TestCase):
       self.assertAllClose(np_ans, tf_ans)
 
   # [Tensor, Tensor, Scalar]
-  def DISABLED_testClipByValue1Type(self):
+  def testClipByValue1Type(self):
     for dtype in [
-        dtypes.float16, dtypes.float32, dtypes.float64, dtypes.int8,
-        dtypes.int16, dtypes.int32, dtypes.int64, dtypes.uint8, dtypes.uint16
+        dtypes.float16,
+        dtypes.float32,
+        dtypes.float64,
+        dtypes.int16,
+        dtypes.int32,
+        dtypes.int64,
+        dtypes.uint8,
     ]:
       with self.cached_session(use_gpu=True):
         x = constant_op.constant([1, 2, 3, 4, 5, 6], shape=[2, 3], dtype=dtype)
@@ -94,10 +105,15 @@ class ClipTest(test.TestCase):
       self.assertAllClose(np_ans, tf_ans)
 
   # [Tensor, Scalar, Tensor]
-  def DISABLED_testClipByValue2Type(self):
+  def testClipByValue2Type(self):
     for dtype in [
-        dtypes.float16, dtypes.float32, dtypes.float64, dtypes.int8,
-        dtypes.int16, dtypes.int32, dtypes.int64, dtypes.uint8, dtypes.uint16
+        dtypes.float16,
+        dtypes.float32,
+        dtypes.float64,
+        dtypes.int16,
+        dtypes.int32,
+        dtypes.int64,
+        dtypes.uint8,
     ]:
       with self.cached_session(use_gpu=True):
         x = constant_op.constant([1, 2, 3, 4, 5, 6], shape=[2, 3], dtype=dtype)
@@ -111,10 +127,15 @@ class ClipTest(test.TestCase):
       self.assertAllClose(np_ans, tf_ans)
 
   # [Tensor, Tensor, Tensor]
-  def DISABLED_testClipByValue3Type(self):
+  def testClipByValue3Type(self):
     for dtype in [
-        dtypes.float16, dtypes.float32, dtypes.float64, dtypes.int8,
-        dtypes.int16, dtypes.int32, dtypes.int64, dtypes.uint8, dtypes.uint16
+        dtypes.float16,
+        dtypes.float32,
+        dtypes.float64,
+        dtypes.int16,
+        dtypes.int32,
+        dtypes.int64,
+        dtypes.uint8,
     ]:
       with self.cached_session(use_gpu=True):
         x = constant_op.constant([1, 2, 3, 4, 5, 6], shape=[2, 3], dtype=dtype)
diff --git a/tensorflow/python/kernel_tests/conv1d_test.py b/tensorflow/python/kernel_tests/conv1d_test.py
index 5ac8c11130b..78b184a43ba 100644
--- a/tensorflow/python/kernel_tests/conv1d_test.py
+++ b/tensorflow/python/kernel_tests/conv1d_test.py
@@ -55,6 +55,32 @@ class Conv1DTest(test.TestCase):
             self.assertEqual(len(output), 2)
             self.assertAllClose(output, [2 * 1 + 1 * 2, 2 * 3 + 1 * 4])
 
+  def testExpandedBatch(self):
+    """Test that argument passing to conv1d is handled properly."""
+    # double datatype is currently not supported for convolution ops
+    # on the ROCm platform
+    x = constant_op.constant([1, 2, 3, 4], dtype=dtypes.float32)
+    x = array_ops.expand_dims(x, 0)  # Add batch dimension
+    x = array_ops.expand_dims(x, 2)  # And depth dimension
+    x = array_ops.stack([x, x])  # Make batch shape [2, 1]
+    filters = constant_op.constant([2, 1], dtype=dtypes.float32)
+    filters = array_ops.expand_dims(filters, 1)  # in_channels
+    filters = array_ops.expand_dims(filters, 2)  # out_channels
+    # Filters is 2x1x1
+    for stride in [1, 2]:
+      with self.cached_session(use_gpu=test.is_gpu_available()):
+        c = nn_ops.conv1d(x, filters, stride, padding="VALID")
+        reduced = array_ops.squeeze(c)  # Sequeeze out dims 1 and 3.
+        output = self.evaluate(reduced)
+        if stride == 1:
+          self.assertAllClose(output,
+                              [[2 * 1 + 1 * 2, 2 * 2 + 1 * 3, 2 * 3 + 1 * 4],
+                               [2 * 1 + 1 * 2, 2 * 2 + 1 * 3, 2 * 3 + 1 * 4]])
+        else:
+          self.assertAllClose(
+              output,
+              [[2 * 1 + 1 * 2, 2 * 3 + 1 * 4], [2 * 1 + 1 * 2, 2 * 3 + 1 * 4]])
+
   def testConv1DTranspose(self):
     with self.cached_session():
       stride = 2
diff --git a/tensorflow/python/kernel_tests/conv_ops_3d_test.py b/tensorflow/python/kernel_tests/conv_ops_3d_test.py
index ce46f1d2782..ff4da3afc9f 100644
--- a/tensorflow/python/kernel_tests/conv_ops_3d_test.py
+++ b/tensorflow/python/kernel_tests/conv_ops_3d_test.py
@@ -66,12 +66,8 @@ class Conv3DTest(test.TestCase):
 
   def _SetupValuesForDevice(self, tensor_in_sizes, filter_in_sizes, stride,
                             padding, data_format, dtype, use_gpu):
-    total_size_tensor = 1
-    total_size_filter = 1
-    for s in tensor_in_sizes:
-      total_size_tensor *= s
-    for s in filter_in_sizes:
-      total_size_filter *= s
+    total_size_tensor = np.prod(tensor_in_sizes)
+    total_size_filter = np.prod(filter_in_sizes)
 
     # Initializes the input tensor with array containing numbers from 0 to 1.
     # We keep the input tensor values fairly small to avoid overflowing float16
@@ -126,12 +122,8 @@ class Conv3DTest(test.TestCase):
   def _ComputeReferenceDilatedConv(self, tensor_in_sizes, filter_in_sizes,
                                    stride, dilation, padding, data_format,
                                    use_gpu):
-    total_size_tensor = 1
-    total_size_filter = 1
-    for s in tensor_in_sizes:
-      total_size_tensor *= s
-    for s in filter_in_sizes:
-      total_size_filter *= s
+    total_size_tensor = np.prod(tensor_in_sizes)
+    total_size_filter = np.prod(filter_in_sizes)
 
     # Initializes the input tensor with array containing incrementing
     # numbers from 1.
@@ -196,6 +188,69 @@ class Conv3DTest(test.TestCase):
             self.assertAllClose(
                 e_value.flatten(), c_value.flatten(), atol=tolerance, rtol=1e-6)
 
+  def _CreateNumpyTensor(self, sizes):
+    return np.asarray([f * 1.0
+                       for f in range(1,
+                                      np.prod(sizes) + 1)]).reshape(sizes)
+
+  @test_util.run_in_graph_and_eager_modes
+  def testConv3DExpandedBatch(self):
+    tensor_in_sizes_batch = [10, 2, 3, 1, 3]
+    tensor_in_sizes_expanded_batch = [2, 5, 2, 3, 1, 3]
+    filter_in_sizes = [1, 1, 1, 3, 3]
+    filter_in = self._CreateNumpyTensor(filter_in_sizes)
+    x1 = self._CreateNumpyTensor(tensor_in_sizes_batch)
+    x2 = x1.reshape(tensor_in_sizes_expanded_batch)
+    conv1 = nn_ops.conv3d_v2(
+        x1, filter_in, strides=[1, 1, 1, 1, 1], padding="VALID")
+    conv2 = nn_ops.conv3d_v2(
+        x2, filter_in, strides=[1, 1, 1, 1, 1], padding="VALID")
+    self.assertEqual(conv1.shape, tensor_in_sizes_batch)
+    self.assertEqual(conv2.shape, tensor_in_sizes_expanded_batch)
+    self.assertAllEqual(conv1, self.evaluate(conv2).reshape(conv1.shape))
+
+  @test_util.run_in_graph_and_eager_modes
+  def testConvolutionClass3DExpandedBatch(self):
+    tensor_in_sizes_batch = [10, 2, 3, 1, 3]
+    tensor_in_sizes_expanded_batch = [2, 5, 2, 3, 1, 3]
+    filter_in_sizes = [1, 1, 1, 3, 3]
+    filter_in = self._CreateNumpyTensor(filter_in_sizes)
+    x1 = self._CreateNumpyTensor(tensor_in_sizes_batch)
+    x2 = x1.reshape(tensor_in_sizes_expanded_batch)
+    convolver1 = nn_ops.Convolution(
+        input_shape=x1.shape,
+        filter_shape=filter_in.shape,
+        strides=[1, 1, 1],
+        padding="VALID")
+    self.assertEqual(convolver1.num_batch_dims, 1)
+    convolver2 = nn_ops.Convolution(
+        input_shape=x2.shape,
+        filter_shape=filter_in.shape,
+        strides=[1, 1, 1],
+        padding="VALID")
+    self.assertEqual(convolver2.num_batch_dims, 2)
+    conv1 = convolver1(x1, filter_in)
+    conv2 = convolver2(x2, filter_in)
+    self.assertEqual(conv1.shape, tensor_in_sizes_batch)
+    self.assertEqual(conv2.shape, tensor_in_sizes_expanded_batch)
+    self.assertAllEqual(conv1, self.evaluate(conv2).reshape(conv1.shape))
+
+  @test_util.run_in_graph_and_eager_modes
+  def testConvolutionWith2SpatialDimensionsAndExpandedBatch(self):
+    tensor_in_sizes_batch = [10, 2, 3, 1, 3]
+    tensor_in_sizes_expanded_batch = [2, 5, 2, 3, 1, 3]
+    filter_in_sizes = [1, 1, 1, 3, 3]
+    filter_in = self._CreateNumpyTensor(filter_in_sizes)
+    x1 = self._CreateNumpyTensor(tensor_in_sizes_batch)
+    x2 = x1.reshape(tensor_in_sizes_expanded_batch)
+    conv1 = nn_ops.convolution(
+        x1, filter_in, strides=[1, 1, 1], padding="VALID")
+    conv2 = nn_ops.convolution(
+        x2, filter_in, strides=[1, 1, 1], padding="VALID")
+    self.assertEqual(conv1.shape, tensor_in_sizes_batch)
+    self.assertEqual(conv2.shape, tensor_in_sizes_expanded_batch)
+    self.assertAllEqual(conv1, self.evaluate(conv2).reshape(conv1.shape))
+
   def testConv3D1x1x1Filter(self):
     expected_output = [
         0.18518519, 0.22222222, 0.25925926, 0.40740741, 0.5, 0.59259259,
diff --git a/tensorflow/python/kernel_tests/gather_op_test.py b/tensorflow/python/kernel_tests/gather_op_test.py
index b966110963c..c5b42dd60a7 100644
--- a/tensorflow/python/kernel_tests/gather_op_test.py
+++ b/tensorflow/python/kernel_tests/gather_op_test.py
@@ -25,6 +25,7 @@ from tensorflow.python.eager import backprop
 from tensorflow.python.eager import context
 from tensorflow.python.framework import constant_op
 from tensorflow.python.framework import dtypes
+from tensorflow.python.framework import errors
 from tensorflow.python.framework import ops
 from tensorflow.python.framework import test_util
 from tensorflow.python.ops import array_ops
@@ -212,6 +213,12 @@ class GatherTest(test.TestCase, parameterized.TestCase):
     gather_t = array_ops.gather(params, indices, axis=axis)
     self.assertEqual(None, gather_t.shape)
 
+  def testBadIndicesType(self):
+    with self.assertRaisesRegex(
+        (TypeError, errors.InvalidArgumentError),
+        "float.* not in.* list of allowed values: int32, int64"):
+      self.evaluate(array_ops.gather([0], 0.))
+
   @test_util.disable_xla(
       "Assertion inside an op is not supported in XLA. Instead XLA clamps the "
       "index to be in bounds and returns the indexed value there (Don't rely "
diff --git a/tensorflow/python/kernel_tests/softmax_op_test.py b/tensorflow/python/kernel_tests/softmax_op_test.py
index 82fe328dd10..4debe180690 100644
--- a/tensorflow/python/kernel_tests/softmax_op_test.py
+++ b/tensorflow/python/kernel_tests/softmax_op_test.py
@@ -27,6 +27,7 @@ from tensorflow.python.framework import dtypes
 from tensorflow.python.framework import errors_impl
 from tensorflow.python.framework import test_util
 from tensorflow.python.ops import array_ops
+from tensorflow.python.ops import math_ops
 from tensorflow.python.ops import nn_ops
 from tensorflow.python.platform import test
 from tensorflow.python.platform import tf_logging as logging
@@ -55,13 +56,21 @@ class SoftmaxTest(test.TestCase):
       res = res.astype(np.float16)
     return res
 
-  def _testSoftmax(self, np_features, dim=-1, log=False, use_gpu=False):
+  def _testSoftmax(self,
+                   np_features,
+                   dim=-1,
+                   log=False,
+                   dtype=None,
+                   use_gpu=False):
     # A previous version of the code checked the op name rather than the op type
     # to distinguish between log and non-log.  Use an arbitrary name to catch
     # this bug in future.
     name = "arbitrary"
     np_softmax = self._npSoftmax(np_features, dim=dim, log=log)
     with self.cached_session(use_gpu=use_gpu):
+      if dtype is not None:
+        np_features = math_ops.cast(np_features, dtype=dtype)
+
       if log:
         tf_softmax = nn_ops.log_softmax(np_features, axis=dim, name=name)
       else:
@@ -69,15 +78,15 @@ class SoftmaxTest(test.TestCase):
       out = self.evaluate(tf_softmax)
     self.assertAllCloseAccordingToType(np_softmax, out)
     self.assertShapeEqual(np_softmax, tf_softmax)
-    if not log:
+    if not log and dtype is None:
       # Bonus check: the softmaxes should add to one in dimension dim.
       sum_along_dim = np.sum(out, axis=dim)
       self.assertAllCloseAccordingToType(
           np.ones(sum_along_dim.shape), sum_along_dim)
 
-  def _testAll(self, features):
-    self._testSoftmax(features, use_gpu=True)
-    self._testSoftmax(features, log=True, use_gpu=True)
+  def _testAll(self, features, dtype=None):
+    self._testSoftmax(features, dtype=dtype, use_gpu=True)
+    self._testSoftmax(features, dtype=dtype, log=True, use_gpu=True)
     self._testOverflow(use_gpu=True)
 
   def testNpSoftmax(self):
@@ -158,6 +167,22 @@ class SoftmaxTest(test.TestCase):
         np.array([[1., 1., 1., 1.], [1., 2., 3., 4.]]).astype(np.float64))
     self._testOverflow()
 
+  @unittest.skipUnless(test.is_built_with_gpu_support(),
+                       "Test only applicable when running on GPUs")
+  def testDoubleGPU(self):
+    if test.is_gpu_available(cuda_only=True):
+      rows = [2**x + np.random.randint(0, 16) for x in range(1, 4)]
+      cols = [2**x + np.random.randint(0, 16) for x in range(1, 4)]
+      for row, col in zip(rows, cols):
+        logging.info("Testing softmax float dtype in shape [%d, %d]", row, col)
+        data = np.random.rand(row, col)
+        self._testAll(data.astype(np.float64))
+
+  def testBfloat16(self):
+    self._testAll(
+        np.array([[1., 1., 1., 1.], [1., 2., 3., 4.]]).astype(np.float32),
+        dtype=dtypes.bfloat16)
+
   def test1DTensorAsInput(self):
     self._testSoftmax(
         np.array([3., 2., 3., 9.]).astype(np.float64), use_gpu=False)
diff --git a/tensorflow/python/ops/array_grad.py b/tensorflow/python/ops/array_grad.py
index e1ed4443017..c9a634230e0 100644
--- a/tensorflow/python/ops/array_grad.py
+++ b/tensorflow/python/ops/array_grad.py
@@ -641,7 +641,8 @@ def _GatherV2Grad(op, grad):
   # For axis 0 gathers, build an appropriately shaped IndexedSlices.
   if axis_static == 0:
     if context.executing_eagerly():
-      params_tail_shape = params_shape.cpu()[1:]
+      with ops.device("/cpu:0"):
+        params_tail_shape = array_ops.identity(params_shape)[1:]
     else:
       params_tail_shape = params_shape[1:]
     values_shape = array_ops.concat([indices_size, params_tail_shape], 0)
diff --git a/tensorflow/python/ops/gradients_test.py b/tensorflow/python/ops/gradients_test.py
index 9a9ce72a557..fc5f38aedba 100644
--- a/tensorflow/python/ops/gradients_test.py
+++ b/tensorflow/python/ops/gradients_test.py
@@ -431,6 +431,63 @@ class GradientsTest(test_util.TensorFlowTestCase, parameterized.TestCase):
           ValueError, "Unknown value for unconnected_gradients: 'nonsense'"):
         gradients.gradients([y], [x], unconnected_gradients="nonsense")
 
+  @parameterized.parameters(unconnected_gradients.UnconnectedGradients.ZERO,
+                            unconnected_gradients.UnconnectedGradients.NONE)
+  def testUnconnectedOpWithMultipleOutputs(self, unconnected_gradients_val):
+    with ops.Graph().as_default():
+      #  a    b
+      #  |    |
+      # IdentityN
+      #  |    |
+      #  c    d
+      #  |
+      # Identity
+      #  |
+      #  e
+      a = constant_op.constant(1.0)
+      b = constant_op.constant(1.0)
+      c, d = array_ops.identity_n([a, b])
+      e = array_ops.identity(c)
+      # The aggregated grads for the IdentityN node would look like
+      # [Tensor, None]. We expect this None to be converted to zeros.
+      output = gradients.gradients(
+          e, d, unconnected_gradients=unconnected_gradients_val)
+      if (unconnected_gradients_val ==
+          unconnected_gradients.UnconnectedGradients.ZERO):
+        self.assertIsNotNone(output[0])
+      else:
+        self.assertIsNone(output[0])
+
+  @parameterized.parameters(unconnected_gradients.UnconnectedGradients.ZERO,
+                            unconnected_gradients.UnconnectedGradients.NONE)
+  def testUnconnectedOpWithMultipleOutputsStopGradient(
+      self, unconnected_gradients_val):
+    with ops.Graph().as_default():
+      #  a    b
+      #  |    |
+      # IdentityN
+      #  |    |
+      #  c    d
+      #  |    |
+      #  SG   |
+      #  |    |
+      #   \  /
+      #    +
+      #    e
+      a = constant_op.constant(1.0)
+      b = constant_op.constant(1.0)
+      c, d = array_ops.identity_n([a, b])
+      e = array_ops.stop_gradient(c) + d
+      # The aggregated grads for the IdentityN node would look like
+      # [None, Tensor]. We expect this None to be converted to zeros.
+      output = gradients.gradients(
+          e, c, unconnected_gradients=unconnected_gradients_val)
+      if (unconnected_gradients_val ==
+          unconnected_gradients.UnconnectedGradients.ZERO):
+        self.assertIsNotNone(output[0])
+      else:
+        self.assertIsNone(output[0])
+
 
 class FunctionGradientsTest(test_util.TensorFlowTestCase):
 
diff --git a/tensorflow/python/ops/gradients_util.py b/tensorflow/python/ops/gradients_util.py
index 96903353dac..56fe2d0529d 100644
--- a/tensorflow/python/ops/gradients_util.py
+++ b/tensorflow/python/ops/gradients_util.py
@@ -781,18 +781,22 @@ def _SetGrad(grads, t, grad):
     op_grads[t.value_index] = grad
 
 
+def _ZerosLike(t):
+  t_dtype = default_gradient.get_zeros_dtype(t)
+  if t.dtype == dtypes.resource:
+    return array_ops.zeros(
+        resource_variable_ops.variable_shape(t), dtype=t_dtype)
+  else:
+    return array_ops.zeros_like(t, dtype=t_dtype)
+
+
 def _GetGrad(grads, t, unconnected_gradients):
   """Gets gradient for tensor "t"."""
   op = t.op
   op_grads = grads.get(op)
   if not op_grads:
     if unconnected_gradients == UnconnectedGradients.ZERO:
-      t_dtype = default_gradient.get_zeros_dtype(t)
-      if t.dtype == dtypes.resource:
-        return array_ops.zeros(
-            resource_variable_ops.variable_shape(t), dtype=t_dtype)
-      else:
-        return array_ops.zeros_like(t, dtype=t_dtype)
+      return _ZerosLike(t)
     elif unconnected_gradients == UnconnectedGradients.NONE:
       return None
     else:
@@ -800,6 +804,10 @@ def _GetGrad(grads, t, unconnected_gradients):
           "Unknown value for unconnected_gradients: %r" % unconnected_gradients)
 
   t_grad = op_grads[t.value_index]
+  # This can happen if some other output of `t.op` has non-None grad.
+  if unconnected_gradients == UnconnectedGradients.ZERO and t_grad is None:
+    return _ZerosLike(t)
+
   assert not isinstance(
       t_grad, list), ("gradients list should have been aggregated by now.")
   return t_grad
diff --git a/tensorflow/python/ops/image_ops_impl.py b/tensorflow/python/ops/image_ops_impl.py
index 0532b24edca..a934639d524 100644
--- a/tensorflow/python/ops/image_ops_impl.py
+++ b/tensorflow/python/ops/image_ops_impl.py
@@ -218,7 +218,8 @@ def _CheckAtLeast3DImage(image, require_static=True):
     else:
       image_shape = image.get_shape().with_rank_at_least(3)
   except ValueError:
-    raise ValueError("'image' must be at least three-dimensional.")
+    raise ValueError("'image' (shape %s) must be at least three-dimensional." %
+                     image.shape)
   if require_static and not image_shape.is_fully_defined():
     raise ValueError('\'image\' must be fully defined.')
   if any(x == 0 for x in image_shape[-3:]):
@@ -283,7 +284,8 @@ def _CheckGrayscaleImage(image, require_static=True):
     else:
       image_shape = image.get_shape().with_rank_at_least(2)
   except ValueError:
-    raise ValueError('A grayscale image must be at least two-dimensional.')
+    raise ValueError('A grayscale image (shape %s) must be at least '
+                     'two-dimensional.' % image.shape)
   if require_static and not image_shape.is_fully_defined():
     raise ValueError('\'image\' must be fully defined.')
   if image_shape.is_fully_defined():
@@ -383,6 +385,7 @@ def random_flip_left_right(image, seed=None):
   [[[2], [1]], [[4], [3]]]
 
   Randomly flip multiple images.
+
   >>> images = np.array(
   ... [
   ...     [[[1], [2]], [[3], [4]]],
@@ -449,7 +452,8 @@ def _random_flip(image, flip_index, seed, scope_name):
       flipped_input = array_ops.reverse(image, [flip_index + 1])
       return flips * flipped_input + (1 - flips) * image
     else:
-      raise ValueError('\'image\' must have either 3 or 4 dimensions.')
+      raise ValueError(
+          '\'image\' (shape %s) must have either 3 or 4 dimensions.' % shape)
 
 
 @tf_export('image.flip_left_right')
@@ -550,7 +554,8 @@ def _flip(image, flip_index, scope_name):
     elif shape.ndims == 4:
       return array_ops.reverse(image, [flip_index + 1])
     else:
-      raise ValueError('\'image\' must have either 3 or 4 dimensions.')
+      raise ValueError(
+          '\'image\' (shape %s)must have either 3 or 4 dimensions.' % shape)
 
 
 @tf_export('image.rot90')
@@ -599,7 +604,8 @@ def rot90(image, k=1, name=None):
     elif shape.ndims == 4:
       return _rot90_4D(image, k, scope)
     else:
-      raise ValueError('\'image\' must have either 3 or 4 dimensions.')
+      raise ValueError(
+          '\'image\' (shape %s) must have either 3 or 4 dimensions.' % shape)
 
 
 def _rot90_3D(image, k, name_scope):
@@ -721,7 +727,8 @@ def transpose(image, name=None):
     elif shape.ndims == 4:
       return array_ops.transpose(image, [0, 2, 1, 3], name=name)
     else:
-      raise ValueError('\'image\' must have either 3 or 4 dimensions.')
+      raise ValueError(
+          '\'image\' (shape %s) must have either 3 or 4 dimensions.' % shape)
 
 
 @tf_export('image.central_crop')
@@ -928,7 +935,9 @@ def pad_to_bounding_box(image, offset_height, offset_width, target_height,
       image = array_ops.expand_dims(image, 0)
       image.set_shape([None] * 4)
     elif image_shape.ndims != 4:
-      raise ValueError('\'image\' must have either 3 or 4 dimensions.')
+      raise ValueError(
+          '\'image\' (shape %s) must have either 3 or 4 dimensions.' %
+          image_shape)
 
     assert_ops = _CheckAtLeast3DImage(image, require_static=False)
     batch, height, width, depth = _ImageDimensions(image, rank=4)
@@ -1012,7 +1021,9 @@ def crop_to_bounding_box(image, offset_height, offset_width, target_height,
       image = array_ops.expand_dims(image, 0)
       image.set_shape([None] * 4)
     elif image_shape.ndims != 4:
-      raise ValueError('\'image\' must have either 3 or 4 dimensions.')
+      raise ValueError(
+          '\'image\' (shape %s) must have either 3 or 4 dimensions.' %
+          image_shape)
 
     assert_ops = _CheckAtLeast3DImage(image, require_static=False)
 
@@ -1092,7 +1103,9 @@ def resize_image_with_crop_or_pad(image, target_height, target_width):
       image = array_ops.expand_dims(image, 0)
       image.set_shape([None] * 4)
     elif image_shape.ndims != 4:
-      raise ValueError('\'image\' must have either 3 or 4 dimensions.')
+      raise ValueError(
+          '\'image\' (shape %s) must have either 3 or 4 dimensions.' %
+          image_shape)
 
     assert_ops = _CheckAtLeast3DImage(image, require_static=False)
     assert_ops += _assert(target_width > 0, ValueError,
@@ -1548,7 +1561,9 @@ def _resize_image_with_pad_common(image, target_height, target_width,
       image = array_ops.expand_dims(image, 0)
       image.set_shape([None] * 4)
     elif image_shape.ndims != 4:
-      raise ValueError('\'image\' must have either 3 or 4 dimensions.')
+      raise ValueError(
+          '\'image\' (shape %s) must have either 3 or 4 dimensions.' %
+          image_shape)
 
     assert_ops = _CheckAtLeast3DImage(image, require_static=False)
     assert_ops += _assert(target_width > 0, ValueError,
diff --git a/tensorflow/python/ops/image_ops_test.py b/tensorflow/python/ops/image_ops_test.py
index 2ed077a862c..3530885fe07 100644
--- a/tensorflow/python/ops/image_ops_test.py
+++ b/tensorflow/python/ops/image_ops_test.py
@@ -239,7 +239,7 @@ class GrayscaleToRGBTest(test_util.TensorFlowTestCase):
       x_tf = constant_op.constant(x_np, shape=x_np.shape)
 
       # this is the error message we expect the function to raise
-      err_msg = "A grayscale image must be at least two-dimensional"
+      err_msg = "must be at least two-dimensional"
       with self.assertRaisesRegexp(ValueError, err_msg):
         image_ops.grayscale_to_rgb(x_tf)
 
@@ -1682,7 +1682,7 @@ class CropToBoundingBoxTest(test_util.TensorFlowTestCase):
     for x_shape in ([3, 5], [1, 3, 5, 1, 1]):
       self._assertRaises(x, x_shape, offset_height, offset_width, target_height,
                          target_width,
-                         "'image' must have either 3 or 4 dimensions.")
+                         "must have either 3 or 4 dimensions.")
 
   @test_util.run_deprecated_v1
   def testZeroLengthInput(self):
@@ -2022,7 +2022,7 @@ class PadToBoundingBoxTest(test_util.TensorFlowTestCase):
     for x_shape in ([3, 5], [1, 3, 5, 1, 1]):
       self._assertRaises(x, x_shape, offset_height, offset_width, target_height,
                          target_width,
-                         "'image' must have either 3 or 4 dimensions.")
+                         "must have either 3 or 4 dimensions.")
 
   @test_util.run_deprecated_v1
   def testZeroLengthInput(self):
@@ -3734,11 +3734,11 @@ class ResizeImageWithCropOrPadTest(test_util.TensorFlowTestCase):
 
     for x_shape in ([3, 5],):
       self._assertRaises(x, x_shape, target_height, target_width,
-                         "'image' must have either 3 or 4 dimensions.")
+                         "must have either 3 or 4 dimensions.")
 
     for x_shape in ([1, 3, 5, 1, 1],):
       self._assertRaises(x, x_shape, target_height, target_width,
-                         "'image' must have either 3 or 4 dimensions.")
+                         "must have either 3 or 4 dimensions.")
 
   @test_util.run_deprecated_v1
   def testZeroLengthInput(self):
diff --git a/tensorflow/python/ops/nn_ops.py b/tensorflow/python/ops/nn_ops.py
index 589f9984be1..a61ae753121 100644
--- a/tensorflow/python/ops/nn_ops.py
+++ b/tensorflow/python/ops/nn_ops.py
@@ -228,7 +228,7 @@ class _NonAtrousConvolution(object):
                          % data_format)
       self.strides = strides
       self.data_format = data_format
-      self.conv_op = gen_nn_ops.conv3d
+      self.conv_op = _conv3d_expanded_batch
 
   # Note that we need this adapter since argument names for conv1d don't match
   # those for gen_nn_ops.conv2d and gen_nn_ops.conv3d.
@@ -241,7 +241,6 @@ class _NonAtrousConvolution(object):
         padding=padding,
         data_format=data_format,
         name=name)
-
   # pylint: enable=redefined-builtin
 
   def __call__(self, inp, filter):  # pylint: disable=redefined-builtin
@@ -728,10 +727,7 @@ def _with_space_to_batch_base_paddings(filter_shape, num_spatial_dims,
   # Spatial dimensions of the filters and the upsampled filters in which we
   # introduce (rate - 1) zeros between consecutive filter values.
   filter_spatial_shape = filter_shape[:num_spatial_dims]
-  dilated_filter_spatial_shape = (
-      filter_spatial_shape + (filter_spatial_shape - 1) *
-      (rate_or_const_rate - 1))
-  pad_extra_shape = dilated_filter_spatial_shape - 1
+  pad_extra_shape = (filter_spatial_shape - 1) * rate_or_const_rate
 
   # When full_padding_shape is odd, we pad more at end, following the same
   # convention as conv2d.
@@ -1104,7 +1100,7 @@ def convolution_internal(
     scope = "convolution"
 
   with ops.name_scope(name, scope, [input, filters]) as name:
-    conv_ops = {1: conv1d, 2: _conv2d_expanded_batch, 3: gen_nn_ops.conv3d}
+    conv_ops = {1: conv1d, 2: _conv2d_expanded_batch, 3: _conv3d_expanded_batch}
 
     if device_context.enclosing_tpu_context() is not None or all(
         i == 1 for i in dilations):
@@ -1783,40 +1779,42 @@ def conv1d(
     name=None,
     input=None,  # pylint: disable=redefined-builtin
     dilations=None):
-  r"""Computes a 1-D convolution given 3-D input and filter tensors.
+  r"""Computes a 1-D convolution of input with rank `>=3` and a `3-D` filter.
 
   Given an input tensor of shape
-    [batch, in_width, in_channels]
-  if data_format is "NWC", or
-    [batch, in_channels, in_width]
-  if data_format is "NCW",
+    `batch_shape + [in_width, in_channels]`
+  if `data_format` is `"NWC"`, or
+    `batch_shape + [in_channels, in_width]`
+  if `data_format` is `"NCW"`,
   and a filter / kernel tensor of shape
-  [filter_width, in_channels, out_channels], this op reshapes
-  the arguments to pass them to conv2d to perform the equivalent
+  `[filter_width, in_channels, out_channels]`, this op reshapes
+  the arguments to pass them to `conv2d` to perform the equivalent
   convolution operation.
 
   Internally, this op reshapes the input tensors and invokes `tf.nn.conv2d`.
   For example, if `data_format` does not start with "NC", a tensor of shape
-    [batch, in_width, in_channels]
+    `batch_shape + [in_width, in_channels]`
   is reshaped to
-    [batch, 1, in_width, in_channels],
+    `batch_shape + [1, in_width, in_channels]`,
   and the filter is reshaped to
-    [1, filter_width, in_channels, out_channels].
+    `[1, filter_width, in_channels, out_channels]`.
   The result is then reshaped back to
-    [batch, out_width, out_channels]
+    `batch_shape + [out_width, out_channels]`
   \(where out_width is a function of the stride and padding as in conv2d\) and
   returned to the caller.
 
   Args:
-    value: A 3D `Tensor`.  Must be of type `float16`, `float32`, or `float64`.
-    filters: A 3D `Tensor`.  Must have the same type as `value`.
+    value: A Tensor of rank at least 3. Must be of type `float16`, `float32`, or
+      `float64`.
+    filters: A Tensor of rank at least 3.  Must have the same type as `value`.
     stride: An int or list of `ints` that has length `1` or `3`.  The number of
       entries by which the filter is moved right at each step.
     padding: 'SAME' or 'VALID'
     use_cudnn_on_gpu: An optional `bool`.  Defaults to `True`.
     data_format: An optional `string` from `"NWC", "NCW"`.  Defaults to `"NWC"`,
-      the data is stored in the order of [batch, in_width, in_channels].  The
-      `"NCW"` format stores data as [batch, in_channels, in_width].
+      the data is stored in the order of `batch_shape + [in_width,
+      in_channels]`.  The `"NCW"` format stores data as `batch_shape +
+      [in_channels, in_width]`.
     name: A name for the operation (optional).
     input: Alias for value.
     dilations: An int or list of `ints` that has length `1` or `3` which
@@ -1832,14 +1830,14 @@ def conv1d(
   """
   value = deprecation.deprecated_argument_lookup("input", input, "value", value)
   with ops.name_scope(name, "conv1d", [value, filters]) as name:
-    # Reshape the input tensor to [batch, 1, in_width, in_channels]
+    # Reshape the input tensor to batch_shape + [1, in_width, in_channels]
     if data_format is None or data_format == "NHWC" or data_format == "NWC":
       data_format = "NHWC"
-      spatial_start_dim = 1
+      spatial_start_dim = -3
       channel_index = 2
     elif data_format == "NCHW" or data_format == "NCW":
       data_format = "NCHW"
-      spatial_start_dim = 2
+      spatial_start_dim = -2
       channel_index = 1
     else:
       raise ValueError("data_format must be \"NWC\" or \"NCW\".")
@@ -1848,15 +1846,30 @@ def conv1d(
 
     value = array_ops.expand_dims(value, spatial_start_dim)
     filters = array_ops.expand_dims(filters, 0)
-    result = gen_nn_ops.conv2d(
-        value,
-        filters,
-        strides,
-        padding,
-        use_cudnn_on_gpu=use_cudnn_on_gpu,
-        data_format=data_format,
-        dilations=dilations,
-        name=name)
+    if value.shape.ndims in (4, 3, 2, 1, 0, None):
+      result = gen_nn_ops.conv2d(
+          value,
+          filters,
+          strides,
+          padding,
+          use_cudnn_on_gpu=use_cudnn_on_gpu,
+          data_format=data_format,
+          dilations=dilations,
+          name=name)
+    else:
+      result = squeeze_batch_dims(
+          value,
+          functools.partial(
+              gen_nn_ops.conv2d,
+              filter=filters,
+              strides=strides,
+              padding=padding,
+              use_cudnn_on_gpu=use_cudnn_on_gpu,
+              data_format=data_format,
+              dilations=dilations,
+          ),
+          inner_rank=3,
+          name=name)
     return array_ops.squeeze(result, [spatial_start_dim])
 
 
@@ -1873,36 +1886,38 @@ def conv1d_v2(
   r"""Computes a 1-D convolution given 3-D input and filter tensors.
 
   Given an input tensor of shape
-    [batch, in_width, in_channels]
-  if data_format is "NWC", or
-    [batch, in_channels, in_width]
-  if data_format is "NCW",
+    `batch_shape + [in_width, in_channels]`
+  if `data_format` is `"NWC"`, or
+    `batch_shape + [in_channels, in_width]`
+  if `data_format` is `"NCW"`,
   and a filter / kernel tensor of shape
-  [filter_width, in_channels, out_channels], this op reshapes
-  the arguments to pass them to conv2d to perform the equivalent
+  `[filter_width, in_channels, out_channels]`, this op reshapes
+  the arguments to pass them to `conv2d` to perform the equivalent
   convolution operation.
 
   Internally, this op reshapes the input tensors and invokes `tf.nn.conv2d`.
-  For example, if `data_format` does not start with "NC", a tensor of shape
-    [batch, in_width, in_channels]
+  For example, if `data_format` does not start with `"NC"`, a tensor of shape
+    `batch_shape + [in_width, in_channels]`
   is reshaped to
-    [batch, 1, in_width, in_channels],
+    `batch_shape + [1, in_width, in_channels]`,
   and the filter is reshaped to
-    [1, filter_width, in_channels, out_channels].
+    `[1, filter_width, in_channels, out_channels]`.
   The result is then reshaped back to
-    [batch, out_width, out_channels]
+    `batch_shape + [out_width, out_channels]`
   \(where out_width is a function of the stride and padding as in conv2d\) and
   returned to the caller.
 
   Args:
-    input: A 3D `Tensor`.  Must be of type `float16`, `float32`, or `float64`.
-    filters: A 3D `Tensor`.  Must have the same type as `input`.
+    input: A Tensor of rank at least 3. Must be of type `float16`, `float32`, or
+      `float64`.
+    filters: A Tensor of rank at least 3.  Must have the same type as `input`.
     stride: An int or list of `ints` that has length `1` or `3`.  The number of
       entries by which the filter is moved right at each step.
     padding: 'SAME' or 'VALID'
     data_format: An optional `string` from `"NWC", "NCW"`.  Defaults to `"NWC"`,
-      the data is stored in the order of [batch, in_width, in_channels].  The
-      `"NCW"` format stores data as [batch, in_channels, in_width].
+      the data is stored in the order of
+      `batch_shape + [in_width, in_channels]`.  The `"NCW"` format stores data
+      as `batch_shape + [in_channels, in_width]`.
     dilations: An int or list of `ints` that has length `1` or `3` which
       defaults to 1. The dilation factor for each dimension of input. If set to
       k > 1, there will be k-1 skipped cells between each filter element on that
@@ -1947,9 +1962,9 @@ def conv1d_transpose(
     input: A 3-D `Tensor` of type `float` and shape
       `[batch, in_width, in_channels]` for `NWC` data format or
       `[batch, in_channels, in_width]` for `NCW` data format.
-    filters: A 3-D `Tensor` with the same type as `value` and shape
+    filters: A 3-D `Tensor` with the same type as `input` and shape
       `[filter_width, output_channels, in_channels]`.  `filter`'s
-      `in_channels` dimension must match that of `value`.
+      `in_channels` dimension must match that of `input`.
     output_shape: A 1-D `Tensor`, containing three elements, representing the
       output shape of the deconvolution op.
     strides: An int or list of `ints` that has length `1` or `3`.  The number of
@@ -1964,7 +1979,7 @@ def conv1d_transpose(
     name: Optional name for the returned tensor.
 
   Returns:
-    A `Tensor` with the same type as `value`.
+    A `Tensor` with the same type as `input`.
 
   Raises:
     ValueError: If input/output depth does not match `filter`'s shape, if
@@ -2071,9 +2086,9 @@ def conv2d_v2(input,  # pylint: disable=redefined-builtin
   Args:
     input: A `Tensor`. Must be one of the following types:
       `half`, `bfloat16`, `float32`, `float64`.
-      A 4+-D tensor. The dimension order is interpreted according to the value
-      of `data_format`; with the all-but-inner-3 dimensions acting as batch
-      dimensions.  See below for details.
+      A Tensor of rank at least 4. The dimension order is interpreted according
+      to the value of `data_format`; with the all-but-inner-3 dimensions acting
+      as batch dimensions. See below for details.
     filters: A `Tensor`. Must have the same type as `input`.
       A 4-D tensor of shape
       `[filter_height, filter_width, in_channels, out_channels]`
@@ -2214,6 +2229,8 @@ def conv2d(  # pylint: disable=redefined-builtin,dangerous-default-value
     ndims = getattr(shape, "ndims", -1)
     if ndims == -1: ndims = len(shape)
   if ndims in (4, 3, 2, 1, 0, None):
+    # We avoid calling squeeze_batch_dims to reduce extra python function
+    # call slowdown in eager mode.  This branch doesn't require reshapes.
     return gen_nn_ops.conv2d(
         input,
         filter=filter,
@@ -2535,6 +2552,8 @@ def _conv2d_expanded_batch(
     ndims = getattr(shape, "ndims", -1)
     if ndims == -1: ndims = len(shape)
   if ndims in (4, 3, 2, 1, 0, None):
+    # We avoid calling squeeze_batch_dims to reduce extra python function
+    # call slowdown in eager mode.  This branch doesn't require reshapes.
     return gen_nn_ops.conv2d(
         input,
         filter=filters,
@@ -2931,6 +2950,46 @@ def depthwise_conv2d_native_backprop_filter(  # pylint: disable=redefined-builti
       name=name)
 
 
+def _conv3d_expanded_batch(
+    input,  # pylint: disable=redefined-builtin
+    filter,  # pylint: disable=redefined-builtin
+    strides,
+    padding,
+    data_format,
+    dilations=None,
+    name=None):
+  """Helper function for `conv3d`; handles expanded batches."""
+  # Try really hard to avoid modifying the legacy name sceops - return early.
+  shape = getattr(input, "shape", None)
+  if shape is not None:
+    ndims = getattr(shape, "ndims", -1)
+    if ndims == -1:
+      ndims = len(shape)
+  if ndims in (5, 4, 3, 2, 1, 0, None):
+    # We avoid calling squeeze_batch_dims to reduce extra python function
+    # call slowdown in eager mode.  This branch doesn't require reshapes.
+    return gen_nn_ops.conv3d(
+        input,
+        filter,
+        strides,
+        padding,
+        data_format=data_format,
+        dilations=dilations,
+        name=name)
+  else:
+    return squeeze_batch_dims(
+        input,
+        functools.partial(
+            gen_nn_ops.conv3d,
+            filter=filter,
+            strides=strides,
+            padding=padding,
+            data_format=data_format,
+            dilations=dilations),
+        inner_rank=4,
+        name=name)
+
+
 @tf_export("nn.conv3d", v1=[])
 @dispatch.add_dispatch_support
 def conv3d_v2(input,  # pylint: disable=redefined-builtin,missing-docstring
@@ -2942,13 +3001,8 @@ def conv3d_v2(input,  # pylint: disable=redefined-builtin,missing-docstring
               name=None):
   if dilations is None:
     dilations = [1, 1, 1, 1, 1]
-  return gen_nn_ops.conv3d(input,
-                           filters,
-                           strides,
-                           padding,
-                           data_format=data_format,
-                           dilations=dilations,
-                           name=name)
+  return _conv3d_expanded_batch(input, filters, strides, padding, data_format,
+                                dilations, name)
 
 
 @tf_export(v1=["nn.conv3d"])
@@ -3064,9 +3118,9 @@ def conv3d_transpose_v2(input,  # pylint: disable=redefined-builtin
     input: A 5-D `Tensor` of type `float` and shape `[batch, depth, height,
       width, in_channels]` for `NDHWC` data format or `[batch, in_channels,
       depth, height, width]` for `NCDHW` data format.
-    filters: A 5-D `Tensor` with the same type as `value` and shape `[depth,
+    filters: A 5-D `Tensor` with the same type as `input` and shape `[depth,
       height, width, output_channels, in_channels]`.  `filter`'s `in_channels`
-      dimension must match that of `value`.
+      dimension must match that of `input`.
     output_shape: A 1-D `Tensor` representing the output shape of the
       deconvolution op.
     strides: An int or list of `ints` that has length `1`, `3` or `5`.  The
@@ -3088,7 +3142,7 @@ def conv3d_transpose_v2(input,  # pylint: disable=redefined-builtin
     name: Optional name for the returned tensor.
 
   Returns:
-    A `Tensor` with the same type as `value`.
+    A `Tensor` with the same type as `input`.
 
   References:
     Deconvolutional Networks:
diff --git a/tensorflow/python/ops/numpy_ops/BUILD b/tensorflow/python/ops/numpy_ops/BUILD
index 5b4dae352d6..5879bc9f062 100644
--- a/tensorflow/python/ops/numpy_ops/BUILD
+++ b/tensorflow/python/ops/numpy_ops/BUILD
@@ -1,4 +1,4 @@
-# TF numpy API
+load("//tensorflow:tensorflow.bzl", "cuda_py_test")
 
 package(
     default_visibility = [
@@ -8,9 +8,110 @@ package(
 )
 
 py_library(
-    name = "numpy_ops",
+    name = "numpy",
     srcs = [
         "__init__.py",
+        "np_array_ops.py",
+        "np_arrays.py",
+        "np_dtypes.py",
+        "np_math_ops.py",
+        "np_random.py",
+        "np_utils.py",
     ],
     srcs_version = "PY2AND3",
+    visibility = ["//visibility:public"],
+    deps = [
+        "//tensorflow/python:array_ops",
+        "//tensorflow/python:bitwise_ops",
+        "//tensorflow/python:constant_op",
+        "//tensorflow/python:control_flow_ops",
+        "//tensorflow/python:dtypes",
+        "//tensorflow/python:errors",
+        "//tensorflow/python:framework_ops",
+        "//tensorflow/python:indexed_slices",
+        "//tensorflow/python:linalg_ops",
+        "//tensorflow/python:manip_ops",
+        "//tensorflow/python:math_ops",
+        "//tensorflow/python:sort_ops",
+        "//tensorflow/python:tensor_util",
+        "//tensorflow/python:util",
+        "//third_party/py/numpy",
+    ],
+)
+
+cuda_py_test(
+    name = "np_arrays_test",
+    srcs = ["np_arrays_test.py"],
+    deps = [
+        ":numpy",
+        "//tensorflow/python:dtypes",
+        "//tensorflow/python:framework_ops",
+        "//tensorflow/python:math_ops",
+        "//tensorflow/python:platform",
+        "//third_party/py/numpy",
+        "@absl_py//absl/testing:parameterized",
+    ],
+)
+
+cuda_py_test(
+    name = "np_array_ops_test",
+    srcs = ["np_array_ops_test.py"],
+    deps = [
+        ":numpy",
+        "//tensorflow/python:platform",
+        "//third_party/py/numpy",
+        "@six_archive//:six",
+    ],
+)
+
+cuda_py_test(
+    name = "np_backprop_test",
+    srcs = ["np_backprop_test.py"],
+    deps = [
+        ":numpy",
+        "//tensorflow/python:platform",
+        "@absl_py//absl/testing:parameterized",
+    ],
+)
+
+cuda_py_test(
+    name = "np_logic_test",
+    srcs = ["np_logic_test.py"],
+    deps = [
+        ":numpy",
+        "//third_party/py/numpy",
+    ],
+)
+
+cuda_py_test(
+    name = "np_math_ops_test",
+    srcs = ["np_math_ops_test.py"],
+    deps = [
+        ":numpy",
+        "//tensorflow/python:platform",
+        "//third_party/py/numpy",
+        "@absl_py//absl/testing:parameterized",
+    ],
+)
+
+cuda_py_test(
+    name = "np_random_test",
+    srcs = ["np_random_test.py"],
+    deps = [
+        ":numpy",
+        "//tensorflow/python:platform",
+        "//third_party/py/numpy",
+        "@absl_py//absl/testing:parameterized",
+        "@six_archive//:six",
+    ],
+)
+
+cuda_py_test(
+    name = "np_utils_test",
+    srcs = ["np_utils_test.py"],
+    deps = [
+        ":numpy",
+        "//tensorflow/python:platform",
+        "@absl_py//absl/testing:parameterized",
+    ],
 )
diff --git a/tensorflow/python/ops/numpy_ops/__init__.py b/tensorflow/python/ops/numpy_ops/__init__.py
index d78a4c3a6fb..383206a83fd 100644
--- a/tensorflow/python/ops/numpy_ops/__init__.py
+++ b/tensorflow/python/ops/numpy_ops/__init__.py
@@ -13,7 +13,32 @@
 # limitations under the License.
 # ==============================================================================
 """Tensorflow numpy API."""
+# pylint: disable=g-direct-tensorflow-import
 
 from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function
+
+from tensorflow.python.ops.numpy_ops import np_random as random
+
+# pylint: disable=wildcard-import
+
+from tensorflow.python.ops.numpy_ops.np_array_ops import *
+# TODO(wangpeng): Move ShardedNdArray, convert_to_tensor, tensor_to_ndarray out
+# of here.
+from tensorflow.python.ops.numpy_ops.np_arrays import convert_to_tensor
+from tensorflow.python.ops.numpy_ops.np_arrays import ndarray
+from tensorflow.python.ops.numpy_ops.np_arrays import ShardedNdArray
+from tensorflow.python.ops.numpy_ops.np_arrays import tensor_to_ndarray
+from tensorflow.python.ops.numpy_ops.np_dtypes import *
+from tensorflow.python.ops.numpy_ops.np_math_ops import *
+from tensorflow.python.ops.numpy_ops.np_utils import finfo
+from tensorflow.python.ops.numpy_ops.np_utils import promote_types
+from tensorflow.python.ops.numpy_ops.np_utils import result_type
+# pylint: enable=wildcard-import
+
+# pylint: disable=redefined-builtin,undefined-variable
+max = amax
+min = amin
+round = around
+# pylint: enable=redefined-builtin,undefined-variable
diff --git a/tensorflow/python/ops/numpy_ops/np_array_ops.py b/tensorflow/python/ops/numpy_ops/np_array_ops.py
new file mode 100644
index 00000000000..a7b03fbb1ee
--- /dev/null
+++ b/tensorflow/python/ops/numpy_ops/np_array_ops.py
@@ -0,0 +1,1621 @@
+# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Common array methods."""
+# pylint: disable=g-direct-tensorflow-import
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import functools
+import math
+import numpy as np
+import six
+
+from tensorflow.python.framework import constant_op
+from tensorflow.python.framework import dtypes
+from tensorflow.python.framework import ops
+from tensorflow.python.ops import array_ops
+from tensorflow.python.ops import clip_ops
+from tensorflow.python.ops import control_flow_ops
+from tensorflow.python.ops import linalg_ops
+from tensorflow.python.ops import manip_ops
+from tensorflow.python.ops import math_ops
+from tensorflow.python.ops import sort_ops
+from tensorflow.python.ops.numpy_ops import np_arrays
+from tensorflow.python.ops.numpy_ops import np_dtypes
+from tensorflow.python.ops.numpy_ops import np_utils
+from tensorflow.python.util import nest
+
+
+def empty(shape, dtype=float):  # pylint: disable=redefined-outer-name
+  """Returns an empty array with the specified shape and dtype.
+
+  Args:
+    shape: A fully defined shape. Could be - NumPy array or a python scalar,
+      list or tuple of integers, - TensorFlow tensor/ndarray of integer type and
+      rank <=1.
+    dtype: Optional, defaults to float. The type of the resulting ndarray. Could
+      be a python type, a NumPy type or a TensorFlow `DType`.
+
+  Returns:
+    An ndarray.
+  """
+  return zeros(shape, dtype)
+
+
+def empty_like(a, dtype=None):
+  """Returns an empty array with the shape and possibly type of the input array.
+
+  Args:
+    a: array_like. Could be an ndarray, a Tensor or any object that can be
+      converted to a Tensor using `tf.convert_to_tensor`.
+    dtype: Optional, defaults to dtype of the input array. The type of the
+      resulting ndarray. Could be a python type, a NumPy type or a TensorFlow
+      `DType`.
+
+  Returns:
+    An ndarray.
+  """
+  return zeros_like(a, dtype)
+
+
+def zeros(shape, dtype=float):  # pylint: disable=redefined-outer-name
+  """Returns an ndarray with the given shape and type filled with zeros.
+
+  Args:
+    shape: A fully defined shape. Could be - NumPy array or a python scalar,
+      list or tuple of integers, - TensorFlow tensor/ndarray of integer type and
+      rank <=1.
+    dtype: Optional, defaults to float. The type of the resulting ndarray. Could
+      be a python type, a NumPy type or a TensorFlow `DType`.
+
+  Returns:
+    An ndarray.
+  """
+  if dtype:
+    dtype = np_utils.result_type(dtype)
+  if isinstance(shape, np_arrays.ndarray):
+    shape = shape.data
+  return np_arrays.tensor_to_ndarray(array_ops.zeros(shape, dtype=dtype))
+
+
+def zeros_like(a, dtype=None):
+  """Returns an array of zeros with the shape and type of the input array.
+
+  Args:
+    a: array_like. Could be an ndarray, a Tensor or any object that can be
+      converted to a Tensor using `tf.convert_to_tensor`.
+    dtype: Optional, defaults to dtype of the input array. The type of the
+      resulting ndarray. Could be a python type, a NumPy type or a TensorFlow
+      `DType`.
+
+  Returns:
+    An ndarray.
+  """
+  if isinstance(a, np_arrays.ndarray):
+    a = a.data
+  if dtype is None:
+    # We need to let np_utils.result_type decide the dtype, not tf.zeros_like
+    dtype = np_utils.result_type(a)
+  else:
+    # TF and numpy has different interpretations of Python types such as
+    # `float`, so we let `np_utils.result_type` decide.
+    dtype = np_utils.result_type(dtype)
+  dtype = dtypes.as_dtype(dtype)  # Work around b/149877262
+  return np_arrays.tensor_to_ndarray(array_ops.zeros_like(a, dtype))
+
+
+def ones(shape, dtype=float):  # pylint: disable=redefined-outer-name
+  """Returns an ndarray with the given shape and type filled with ones.
+
+  Args:
+    shape: A fully defined shape. Could be - NumPy array or a python scalar,
+      list or tuple of integers, - TensorFlow tensor/ndarray of integer type and
+      rank <=1.
+    dtype: Optional, defaults to float. The type of the resulting ndarray. Could
+      be a python type, a NumPy type or a TensorFlow `DType`.
+
+  Returns:
+    An ndarray.
+  """
+  if dtype:
+    dtype = np_utils.result_type(dtype)
+  if isinstance(shape, np_arrays.ndarray):
+    shape = shape.data
+  return np_arrays.tensor_to_ndarray(array_ops.ones(shape, dtype=dtype))
+
+
+def ones_like(a, dtype=None):
+  """Returns an array of ones with the shape and type of the input array.
+
+  Args:
+    a: array_like. Could be an ndarray, a Tensor or any object that can be
+      converted to a Tensor using `tf.convert_to_tensor`.
+    dtype: Optional, defaults to dtype of the input array. The type of the
+      resulting ndarray. Could be a python type, a NumPy type or a TensorFlow
+      `DType`.
+
+  Returns:
+    An ndarray.
+  """
+  if isinstance(a, np_arrays.ndarray):
+    a = a.data
+  if dtype is None:
+    dtype = np_utils.result_type(a)
+  else:
+    dtype = np_utils.result_type(dtype)
+  return np_arrays.tensor_to_ndarray(array_ops.ones_like(a, dtype))
+
+
+@np_utils.np_doc(np.eye)
+def eye(N, M=None, k=0, dtype=float):  # pylint: disable=invalid-name,missing-docstring
+  if dtype:
+    dtype = np_utils.result_type(dtype)
+  if not M:
+    M = N
+  # Making sure N, M and k are `int`
+  N = int(N)
+  M = int(M)
+  k = int(k)
+  if k >= M or -k >= N:
+    # tf.linalg.diag will raise an error in this case
+    return zeros([N, M], dtype=dtype)
+  if k == 0:
+    return np_arrays.tensor_to_ndarray(linalg_ops.eye(N, M, dtype=dtype))
+  # We need the precise length, otherwise tf.linalg.diag will raise an error
+  diag_len = min(N, M)
+  if k > 0:
+    if N >= M:
+      diag_len -= k
+    elif N + k > M:
+      diag_len = M - k
+  elif k <= 0:
+    if M >= N:
+      diag_len += k
+    elif M - k > N:
+      diag_len = N + k
+  diagonal_ = array_ops.ones([diag_len], dtype=dtype)
+  return np_arrays.tensor_to_ndarray(
+      array_ops.matrix_diag(diagonal=diagonal_, num_rows=N, num_cols=M, k=k))
+
+
+def identity(n, dtype=float):
+  """Returns a square array with ones on the main diagonal and zeros elsewhere.
+
+  Args:
+    n: number of rows/cols.
+    dtype: Optional, defaults to float. The type of the resulting ndarray. Could
+      be a python type, a NumPy type or a TensorFlow `DType`.
+
+  Returns:
+    An ndarray of shape (n, n) and requested type.
+  """
+  return eye(N=n, M=n, dtype=dtype)
+
+
+def full(shape, fill_value, dtype=None):  # pylint: disable=redefined-outer-name
+  """Returns an array with given shape and dtype filled with `fill_value`.
+
+  Args:
+    shape: A valid shape object. Could be a native python object or an object of
+      type ndarray, numpy.ndarray or tf.TensorShape.
+    fill_value: array_like. Could be an ndarray, a Tensor or any object that can
+      be converted to a Tensor using `tf.convert_to_tensor`.
+    dtype: Optional, defaults to dtype of the `fill_value`. The type of the
+      resulting ndarray. Could be a python type, a NumPy type or a TensorFlow
+      `DType`.
+
+  Returns:
+    An ndarray.
+
+  Raises:
+    ValueError: if `fill_value` can not be broadcast to shape `shape`.
+  """
+  fill_value = asarray(fill_value, dtype=dtype)
+  if np_utils.isscalar(shape):
+    shape = array_ops.reshape(shape, [1])
+  return np_arrays.tensor_to_ndarray(
+      array_ops.broadcast_to(fill_value.data, shape))
+
+
+# Using doc only here since np full_like signature doesn't seem to have the
+# shape argument (even though it exists in the documentation online).
+@np_utils.np_doc_only(np.full_like)
+def full_like(a, fill_value, dtype=None, order='K', subok=True, shape=None):  # pylint: disable=missing-docstring,redefined-outer-name
+  """order, subok and shape arguments mustn't be changed."""
+  if order != 'K':
+    raise ValueError('Non-standard orders are not supported.')
+  if not subok:
+    raise ValueError('subok being False is not supported.')
+  if shape:
+    raise ValueError('Overriding the shape is not supported.')
+
+  a = asarray(a).data
+  dtype = dtype or np_utils.result_type(a)
+  fill_value = asarray(fill_value, dtype=dtype)
+  return np_arrays.tensor_to_ndarray(
+      array_ops.broadcast_to(fill_value.data, array_ops.shape(a)))
+
+
+# TODO(wangpeng): investigate whether we can make `copy` default to False.
+# TODO(wangpeng): np_utils.np_doc can't handle np.array because np.array is a
+#   builtin function. Make np_utils.np_doc support builtin functions.
+def array(val, dtype=None, copy=True, ndmin=0):  # pylint: disable=redefined-outer-name
+  """Creates an ndarray with the contents of val.
+
+  Args:
+    val: array_like. Could be an ndarray, a Tensor or any object that can be
+      converted to a Tensor using `tf.convert_to_tensor`.
+    dtype: Optional, defaults to dtype of the `val`. The type of the resulting
+      ndarray. Could be a python type, a NumPy type or a TensorFlow `DType`.
+    copy: Determines whether to create a copy of the backing buffer. Since
+      Tensors are immutable, a copy is made only if val is placed on a different
+      device than the current one. Even if `copy` is False, a new Tensor may
+      need to be built to satisfy `dtype` and `ndim`. This is used only if `val`
+      is an ndarray or a Tensor.
+    ndmin: The minimum rank of the returned array.
+
+  Returns:
+    An ndarray.
+  """
+  if dtype:
+    dtype = np_utils.result_type(dtype)
+  if isinstance(val, np_arrays.ndarray):
+    result_t = val.data
+  else:
+    result_t = val
+
+  if copy and isinstance(result_t, ops.Tensor):
+    # Note: In eager mode, a copy of `result_t` is made only if it is not on
+    # the context device.
+    result_t = array_ops.identity(result_t)
+
+  if not isinstance(result_t, ops.Tensor):
+    if not dtype:
+      dtype = np_utils.result_type(result_t)
+    # We can't call `convert_to_tensor(result_t, dtype=dtype)` here because
+    # convert_to_tensor doesn't allow incompatible arguments such as (5.5, int)
+    # while np.array allows them. We need to convert-then-cast.
+    def maybe_data(x):
+      if isinstance(x, np_arrays.ndarray):
+        return x.data
+      return x
+
+    # Handles lists of ndarrays
+    result_t = nest.map_structure(maybe_data, result_t)
+    result_t = np_arrays.convert_to_tensor(result_t)
+    result_t = math_ops.cast(result_t, dtype=dtype)
+  elif dtype:
+    result_t = math_ops.cast(result_t, dtype)
+  ndims = array_ops.rank(result_t)
+
+  def true_fn():
+    old_shape = array_ops.shape(result_t)
+    new_shape = array_ops.concat(
+        [array_ops.ones(ndmin - ndims, dtypes.int32), old_shape], axis=0)
+    return array_ops.reshape(result_t, new_shape)
+
+  result_t = np_utils.cond(
+      np_utils.greater(ndmin, ndims), true_fn, lambda: result_t)
+  return np_arrays.tensor_to_ndarray(result_t)
+
+
+@np_utils.np_doc(np.asarray)
+def asarray(a, dtype=None):
+  if dtype:
+    dtype = np_utils.result_type(dtype)
+  if isinstance(a, np_arrays.ndarray) and (not dtype or dtype == a.dtype):
+    return a
+  return array(a, dtype, copy=False)
+
+
+@np_utils.np_doc(np.asanyarray)
+def asanyarray(a, dtype=None):
+  return asarray(a, dtype)
+
+
+@np_utils.np_doc(np.ascontiguousarray)
+def ascontiguousarray(a, dtype=None):
+  return array(a, dtype, ndmin=1)
+
+
+# Numerical ranges.
+def arange(start, stop=None, step=1, dtype=None):
+  """Returns `step`-separated values in the range [start, stop).
+
+  Args:
+    start: Start of the interval. Included in the range.
+    stop: End of the interval. If not specified, `start` is treated as 0 and
+      `start` value is used as `stop`. If specified, it is not included in the
+      range if `step` is integer. When `step` is floating point, it may or may
+      not be included.
+    step: The difference between 2 consecutive values in the output range. It is
+      recommended to use `linspace` instead of using non-integer values for
+      `step`.
+    dtype: Optional. Type of the resulting ndarray. Could be a python type, a
+      NumPy type or a TensorFlow `DType`. If not provided, the largest type of
+      `start`, `stop`, `step` is used.
+
+  Raises:
+    ValueError: If step is zero.
+  """
+  if not step:
+    raise ValueError('step must be non-zero.')
+  if dtype:
+    dtype = np_utils.result_type(dtype)
+  else:
+    if stop is None:
+      dtype = np_utils.result_type(start, step)
+    else:
+      dtype = np_utils.result_type(start, step, stop)
+  if step > 0 and ((stop is not None and start > stop) or
+                   (stop is None and start < 0)):
+    return array([], dtype=dtype)
+  if step < 0 and ((stop is not None and start < stop) or
+                   (stop is None and start > 0)):
+    return array([], dtype=dtype)
+  # TODO(srbs): There are some bugs when start or stop is float type and dtype
+  # is integer type.
+  return np_arrays.tensor_to_ndarray(
+      math_ops.cast(math_ops.range(start, limit=stop, delta=step), dtype=dtype))
+
+
+@np_utils.np_doc(np.geomspace)
+def geomspace(start, stop, num=50, endpoint=True, dtype=float):  # pylint: disable=missing-docstring
+  if dtype:
+    dtype = np_utils.result_type(dtype)
+  if num < 0:
+    raise ValueError('Number of samples {} must be non-negative.'.format(num))
+  if not num:
+    return empty([0])
+  step = 1.
+  if endpoint:
+    if num > 1:
+      step = math_ops.pow((stop / start), 1 / (num - 1))
+  else:
+    step = math_ops.pow((stop / start), 1 / num)
+  result = math_ops.cast(math_ops.range(num), step.dtype)
+  result = math_ops.pow(step, result)
+  result = math_ops.multiply(result, start)
+  if dtype:
+    result = math_ops.cast(result, dtype=dtype)
+  return np_arrays.tensor_to_ndarray(result)
+
+
+# Building matrices.
+@np_utils.np_doc(np.diag)
+def diag(v, k=0):  # pylint: disable=missing-docstring
+  """Raises an error if input is not 1- or 2-d."""
+  v = asarray(v).data
+  v_rank = array_ops.rank(v)
+
+  v.shape.with_rank_at_most(2)
+
+  # TODO(nareshmodi): Consider a np_utils.Assert version that will fail during
+  # tracing time if the shape is known.
+  control_flow_ops.Assert(
+      np_utils.logical_or(math_ops.equal(v_rank, 1), math_ops.equal(v_rank, 2)),
+      [v_rank])
+
+  def _diag(v, k):
+    return np_utils.cond(
+        math_ops.equal(array_ops.size(v), 0),
+        lambda: array_ops.zeros([abs(k), abs(k)], dtype=v.dtype),
+        lambda: array_ops.matrix_diag(v, k=k))
+
+  def _diag_part(v, k):
+    v_shape = array_ops.shape(v)
+    v, k = np_utils.cond(
+        np_utils.logical_or(
+            np_utils.less_equal(k, -1 * np_utils.getitem(v_shape, 0)),
+            np_utils.greater_equal(k, np_utils.getitem(v_shape, 1)),
+        ), lambda: (array_ops.zeros([0, 0], dtype=v.dtype), 0), lambda: (v, k))
+    result = array_ops.matrix_diag_part(v, k=k)
+    return result
+
+  result = np_utils.cond(
+      math_ops.equal(v_rank, 1), lambda: _diag(v, k), lambda: _diag_part(v, k))
+  return np_utils.tensor_to_ndarray(result)
+
+
+@np_utils.np_doc(np.diagonal)
+def diagonal(a, offset=0, axis1=0, axis2=1):  # pylint: disable=missing-docstring
+  a = asarray(a).data
+
+  maybe_rank = a.shape.rank
+  if maybe_rank is not None and offset == 0 and (
+      axis1 == maybe_rank - 2 or axis1 == -2) and (axis2 == maybe_rank - 1 or
+                                                   axis2 == -1):
+    return np_utils.tensor_to_ndarray(array_ops.matrix_diag_part(a))
+
+  a = moveaxis(np_utils.tensor_to_ndarray(a), (axis1, axis2), (-2, -1)).data
+
+  a_shape = array_ops.shape(a)
+
+  def _zeros():  # pylint: disable=missing-docstring
+    return (array_ops.zeros(
+        array_ops.concat([a_shape[:-1], [0]], 0), dtype=a.dtype), 0)
+
+  # All zeros since diag_part doesn't handle all possible k (aka offset).
+  # Written this way since cond will run shape inference on both branches,
+  # and diag_part shape inference will fail when offset is out of bounds.
+  a, offset = np_utils.cond(
+      np_utils.logical_or(
+          np_utils.less_equal(offset, -1 * np_utils.getitem(a_shape, -2)),
+          np_utils.greater_equal(offset, np_utils.getitem(a_shape, -1)),
+      ), _zeros, lambda: (a, offset))
+
+  a = np_utils.tensor_to_ndarray(array_ops.matrix_diag_part(a, k=offset))
+  return a
+
+
+def diagflat(v, k=0):
+  """Returns a 2-d array with flattened `v` as diagonal.
+
+  Args:
+    v: array_like of any rank. Gets flattened when setting as diagonal. Could be
+      an ndarray, a Tensor or any object that can be converted to a Tensor using
+      `tf.convert_to_tensor`.
+    k: Position of the diagonal. Defaults to 0, the main diagonal. Positive
+      values refer to diagonals shifted right, negative values refer to
+      diagonals shifted left.
+
+  Returns:
+    2-d ndarray.
+  """
+  v = asarray(v)
+  return diag(array_ops.reshape(v.data, [-1]), k)
+
+
+def _promote_dtype(*arrays):
+  dtype = np_utils.result_type(*arrays)
+  return [asarray(a, dtype=dtype) for a in arrays]
+
+
+def all(a, axis=None, keepdims=None):  # pylint: disable=redefined-builtin
+  """Whether all array elements or those along an axis evaluate to true.
+
+  Casts the array to bool type if it is not already and uses `tf.reduce_all` to
+  compute the result.
+
+  Args:
+    a: array_like. Could be an ndarray, a Tensor or any object that can be
+      converted to a Tensor using `tf.convert_to_tensor`.
+    axis: Optional. Could be an int or a tuple of integers. If not specified,
+      the reduction is performed over all array indices.
+    keepdims: If true, retains reduced dimensions with length 1.
+
+  Returns:
+    An ndarray. Note that unlike NumPy this does not return a scalar bool if
+    `axis` is None.
+  """
+  a = asarray(a, dtype=bool)
+  return np_utils.tensor_to_ndarray(
+      math_ops.reduce_all(input_tensor=a.data, axis=axis, keepdims=keepdims))
+
+
+def any(a, axis=None, keepdims=None):  # pylint: disable=redefined-builtin
+  """Whether any element in the entire array or in an axis evaluates to true.
+
+  Casts the array to bool type if it is not already and uses `tf.reduce_any` to
+  compute the result.
+
+  Args:
+    a: array_like. Could be an ndarray, a Tensor or any object that can be
+      converted to a Tensor using `tf.convert_to_tensor`.
+    axis: Optional. Could be an int or a tuple of integers. If not specified,
+      the reduction is performed over all array indices.
+    keepdims: If true, retains reduced dimensions with length 1.
+
+  Returns:
+    An ndarray. Note that unlike NumPy this does not return a scalar bool if
+    `axis` is None.
+  """
+  a = asarray(a, dtype=bool)
+  return np_utils.tensor_to_ndarray(
+      math_ops.reduce_any(input_tensor=a.data, axis=axis, keepdims=keepdims))
+
+
+def compress(condition, a, axis=None):
+  """Compresses `a` by selecting values along `axis` with `condition` true.
+
+  Uses `tf.boolean_mask`.
+
+  Args:
+    condition: 1-d array of bools. If `condition` is shorter than the array axis
+      (or the flattened array if axis is None), it is padded with False.
+    a: array_like. Could be an ndarray, a Tensor or any object that can be
+      converted to a Tensor using `tf.convert_to_tensor`.
+    axis: Optional. Axis along which to select elements. If None, `condition` is
+      applied on flattened array.
+
+  Returns:
+    An ndarray.
+
+  Raises:
+    ValueError: if `condition` is not of rank 1.
+  """
+  condition = asarray(condition, dtype=bool)
+  a = asarray(a)
+
+  if condition.ndim != 1:
+    raise ValueError('condition must be a 1-d array.')
+  # `np.compress` treats scalars as 1-d arrays.
+  if a.ndim == 0:
+    a = ravel(a)
+
+  if axis is None:
+    a = ravel(a)
+    axis = 0
+
+  if axis < 0:
+    axis += a.ndim
+
+  assert axis >= 0 and axis < a.ndim
+
+  # `tf.boolean_mask` requires the first dimensions of array and condition to
+  # match. `np.compress` pads condition with False when it is shorter.
+  condition_t = condition.data
+  a_t = a.data
+  if condition.shape[0] < a.shape[axis]:
+    padding = array_ops.fill([a.shape[axis] - condition.shape[0]], False)
+    condition_t = array_ops.concat([condition_t, padding], axis=0)
+  return np_utils.tensor_to_ndarray(
+      array_ops.boolean_mask(tensor=a_t, mask=condition_t, axis=axis))
+
+
+def copy(a):
+  """Returns a copy of the array."""
+  return array(a, copy=True)
+
+
+def _maybe_promote_to_int(a):
+  if dtypes.as_dtype(a.dtype).is_integer:
+    # If a is an integer type and its precision is less than that of `int`,
+    # the output type will be `int`.
+    output_type = np.promote_types(a.dtype, int)
+    if output_type != a.dtype:
+      a = asarray(a, dtype=output_type)
+
+  return a
+
+
+@np_utils.np_doc(np.cumprod)
+def cumprod(a, axis=None, dtype=None):  # pylint: disable=missing-docstring
+  a = asarray(a, dtype=dtype)
+
+  if dtype is None:
+    a = _maybe_promote_to_int(a)
+
+  # If axis is None, the input is flattened.
+  if axis is None:
+    a = ravel(a)
+    axis = 0
+  elif axis < 0:
+    axis += array_ops.rank(a.data)
+  return np_utils.tensor_to_ndarray(math_ops.cumprod(a.data, axis))
+
+
+@np_utils.np_doc(np.cumsum)
+def cumsum(a, axis=None, dtype=None):  # pylint: disable=missing-docstring
+  a = asarray(a, dtype=dtype)
+
+  if dtype is None:
+    a = _maybe_promote_to_int(a)
+
+  # If axis is None, the input is flattened.
+  if axis is None:
+    a = ravel(a)
+    axis = 0
+  elif axis < 0:
+    axis += array_ops.rank(a.data)
+  return np_utils.tensor_to_ndarray(math_ops.cumsum(a.data, axis))
+
+
+def imag(a):
+  """Returns imaginary parts of all elements in `a`.
+
+  Uses `tf.imag`.
+
+  Args:
+    a: array_like. Could be an ndarray, a Tensor or any object that can be
+      converted to a Tensor using `tf.convert_to_tensor`.
+
+  Returns:
+    An ndarray with the same shape as `a`.
+  """
+  a = asarray(a)
+  # TODO(srbs): np.imag returns a scalar if a is a scalar, whereas we always
+  # return an ndarray.
+  return np_utils.tensor_to_ndarray(math_ops.imag(a.data))
+
+
+_TO_INT_ = 0
+_TO_FLOAT = 1
+
+
+def _reduce(tf_fn,
+            a,
+            axis=None,
+            dtype=None,
+            keepdims=None,
+            promote_int=_TO_INT_,
+            tf_bool_fn=None,
+            preserve_bool=False):
+  """A general reduction function.
+
+  Args:
+    tf_fn: the TF reduction function.
+    a: the array to be reduced.
+    axis: (optional) the axis along which to do the reduction. If None, all
+      dimensions are reduced.
+    dtype: (optional) the dtype of the result.
+    keepdims: (optional) whether to keep the reduced dimension(s).
+    promote_int: how to promote integer and bool inputs. There are three
+      choices. (1) `_TO_INT_` always promotes them to np.int_ or np.uint; (2)
+      `_TO_FLOAT` always promotes them to a float type (determined by
+      dtypes.default_float_type); (3) None: don't promote.
+    tf_bool_fn: (optional) the TF reduction function for bool inputs. It will
+      only be used if `dtype` is explicitly set to `np.bool_` or if `a`'s dtype
+      is `np.bool_` and `preserve_bool` is True.
+    preserve_bool: a flag to control whether to use `tf_bool_fn` if `a`'s dtype
+      is `np.bool_` (some reductions such as np.sum convert bools to integers,
+      while others such as np.max preserve bools.
+
+  Returns:
+    An ndarray.
+  """
+  if dtype:
+    dtype = np_utils.result_type(dtype)
+  if keepdims is None:
+    keepdims = False
+  a = asarray(a, dtype=dtype)
+  if ((dtype == np.bool_ or preserve_bool and a.dtype == np.bool_) and
+      tf_bool_fn is not None):
+    return np_utils.tensor_to_ndarray(
+        tf_bool_fn(input_tensor=a.data, axis=axis, keepdims=keepdims))
+  if dtype is None:
+    dtype = a.dtype
+    if np.issubdtype(dtype, np.integer) or dtype == np.bool_:
+      if promote_int == _TO_INT_:
+        # If a is an integer/bool type and whose bit width is less than np.int_,
+        # numpy up-casts it to np.int_ based on the documentation at
+        # https://numpy.org/doc/1.18/reference/generated/numpy.sum.html
+        if dtype == np.bool_:
+          is_signed = True
+          width = 8  # We can use any number here that is less than 64
+        else:
+          is_signed = np.issubdtype(dtype, np.signedinteger)
+          width = np.iinfo(dtype).bits
+        # Numpy int_ and uint are defined as 'long' and 'unsigned long', so
+        # should have the same bit width.
+        if width < np.iinfo(np.int_).bits:
+          if is_signed:
+            dtype = np.int_
+          else:
+            dtype = np.uint
+          a = a.astype(dtype)
+      elif promote_int == _TO_FLOAT:
+        a = a.astype(np_dtypes.default_float_type())
+
+  return np_utils.tensor_to_ndarray(
+      tf_fn(input_tensor=a.data, axis=axis, keepdims=keepdims))
+
+
+@np_utils.np_doc(np.sum)
+def sum(a, axis=None, dtype=None, keepdims=None):  # pylint: disable=redefined-builtin
+  return _reduce(
+      math_ops.reduce_sum,
+      a,
+      axis=axis,
+      dtype=dtype,
+      keepdims=keepdims,
+      tf_bool_fn=math_ops.reduce_any)
+
+
+@np_utils.np_doc(np.prod)
+def prod(a, axis=None, dtype=None, keepdims=None):
+  return _reduce(
+      math_ops.reduce_prod,
+      a,
+      axis=axis,
+      dtype=dtype,
+      keepdims=keepdims,
+      tf_bool_fn=math_ops.reduce_all)
+
+
+@np_utils.np_doc(np.mean)
+def mean(a, axis=None, dtype=None, keepdims=None):
+  return _reduce(
+      math_ops.reduce_mean,
+      a,
+      axis=axis,
+      dtype=dtype,
+      keepdims=keepdims,
+      promote_int=_TO_FLOAT)
+
+
+@np_utils.np_doc(np.amax)
+def amax(a, axis=None, keepdims=None):
+  return _reduce(
+      math_ops.reduce_max,
+      a,
+      axis=axis,
+      dtype=None,
+      keepdims=keepdims,
+      promote_int=None,
+      tf_bool_fn=math_ops.reduce_any,
+      preserve_bool=True)
+
+
+@np_utils.np_doc(np.amin)
+def amin(a, axis=None, keepdims=None):
+  return _reduce(
+      math_ops.reduce_min,
+      a,
+      axis=axis,
+      dtype=None,
+      keepdims=keepdims,
+      promote_int=None,
+      tf_bool_fn=math_ops.reduce_all,
+      preserve_bool=True)
+
+
+# TODO(wangpeng): Remove this workaround once b/157232284 is fixed
+def _reduce_variance_complex(input_tensor, axis, keepdims):
+  f = functools.partial(math_ops.reduce_variance, axis=axis, keepdims=keepdims)
+  return f(math_ops.real(input_tensor)) + f(math_ops.imag(input_tensor))
+
+
+# TODO(wangpeng): Remove this workaround once b/157232284 is fixed
+def _reduce_std_complex(input_tensor, axis, keepdims):
+  y = _reduce_variance_complex(
+      input_tensor=input_tensor, axis=axis, keepdims=keepdims)
+  return math_ops.sqrt(y)
+
+
+@np_utils.np_doc(np.var)
+def var(a, axis=None, keepdims=None):  # pylint: disable=missing-function-docstring
+
+  def f(input_tensor, axis, keepdims):
+    if input_tensor.dtype in (dtypes.complex64, dtypes.complex128):
+      # A workaround for b/157232284
+      fn = _reduce_variance_complex
+    else:
+      fn = math_ops.reduce_variance
+    return fn(input_tensor=input_tensor, axis=axis, keepdims=keepdims)
+
+  return _reduce(
+      f, a, axis=axis, dtype=None, keepdims=keepdims, promote_int=_TO_FLOAT)
+
+
+@np_utils.np_doc(np.std)
+def std(a, axis=None, keepdims=None):  # pylint: disable=missing-function-docstring
+
+  def f(input_tensor, axis, keepdims):
+    if input_tensor.dtype in (dtypes.complex64, dtypes.complex128):
+      # A workaround for b/157232284
+      fn = _reduce_std_complex
+    else:
+      fn = math_ops.reduce_std
+    return fn(input_tensor=input_tensor, axis=axis, keepdims=keepdims)
+
+  return _reduce(
+      f, a, axis=axis, dtype=None, keepdims=keepdims, promote_int=_TO_FLOAT)
+
+
+@np_utils.np_doc(np.ravel)
+def ravel(a):  # pylint: disable=missing-docstring
+  a = asarray(a)
+  if a.ndim == 1:
+    return a
+  return np_utils.tensor_to_ndarray(array_ops.reshape(a.data, [-1]))
+
+
+setattr(np_arrays.ndarray, 'ravel', ravel)
+
+
+def real(val):
+  """Returns real parts of all elements in `a`.
+
+  Uses `tf.real`.
+
+  Args:
+    val: array_like. Could be an ndarray, a Tensor or any object that can be
+      converted to a Tensor using `tf.convert_to_tensor`.
+
+  Returns:
+    An ndarray with the same shape as `a`.
+  """
+  val = asarray(val)
+  # TODO(srbs): np.real returns a scalar if val is a scalar, whereas we always
+  # return an ndarray.
+  return np_utils.tensor_to_ndarray(math_ops.real(val.data))
+
+
+@np_utils.np_doc(np.repeat)
+def repeat(a, repeats, axis=None):  # pylint: disable=missing-docstring
+  a = asarray(a).data
+  original_shape = a._shape_as_list()  # pylint: disable=protected-access
+  # Best effort recovery of the shape.
+  if original_shape is not None and None not in original_shape:
+    if not original_shape:
+      original_shape = (repeats,)
+    else:
+      repeats_np = np.ravel(np.array(repeats))
+      if repeats_np.size == 1:
+        repeats_np = repeats_np.item()
+        if axis is None:
+          original_shape = (repeats_np * np.prod(original_shape),)
+        else:
+          original_shape[axis] = repeats_np * original_shape[axis]
+      else:
+        if axis is None:
+          original_shape = (repeats_np.sum(),)
+        else:
+          original_shape[axis] = repeats_np.sum()
+
+  repeats = asarray(repeats).data
+  result = array_ops.repeat(a, repeats, axis)
+  result.set_shape(original_shape)
+
+  return np_utils.tensor_to_ndarray(result)
+
+
+@np_utils.np_doc(np.around)
+def around(a, decimals=0):  # pylint: disable=missing-docstring
+  a = asarray(a)
+  dtype = a.dtype
+  factor = math.pow(10, decimals)
+  if np.issubdtype(dtype, np.inexact):
+    factor = math_ops.cast(factor, dtype)
+  else:
+    # Use float as the working dtype when a.dtype is exact (e.g. integer),
+    # because `decimals` can be negative.
+    float_dtype = np_dtypes.default_float_type()
+    a = a.astype(float_dtype).data
+    factor = math_ops.cast(factor, float_dtype)
+  a = math_ops.multiply(a, factor)
+  a = math_ops.round(a)
+  a = math_ops.divide(a, factor)
+  return np_utils.tensor_to_ndarray(a).astype(dtype)
+
+
+round_ = around
+setattr(np_arrays.ndarray, '__round__', around)
+
+
+@np_utils.np_doc(np.reshape)
+def reshape(a, newshape, order='C'):
+  """order argument can only b 'C' or 'F'."""
+  if order not in {'C', 'F'}:
+    raise ValueError('Unsupported order argument {}'.format(order))
+
+  a = asarray(a)
+  if isinstance(newshape, np_arrays.ndarray):
+    newshape = newshape.data
+  if isinstance(newshape, int):
+    newshape = [newshape]
+
+  if order == 'F':
+    r = array_ops.transpose(
+        array_ops.reshape(array_ops.transpose(a.data), newshape[::-1]))
+  else:
+    r = array_ops.reshape(a.data, newshape)
+
+  return np_utils.tensor_to_ndarray(r)
+
+
+def _reshape_method_wrapper(a, *newshape, **kwargs):
+  order = kwargs.pop('order', 'C')
+  if kwargs:
+    raise ValueError('Unsupported arguments: {}'.format(kwargs.keys()))
+
+  if len(newshape) == 1 and not isinstance(newshape[0], int):
+    newshape = newshape[0]
+
+  return reshape(a, newshape, order=order)
+
+
+def expand_dims(a, axis):
+  """Expand the shape of an array.
+
+  Args:
+    a: array_like. Could be an ndarray, a Tensor or any object that can be
+      converted to a Tensor using `tf.convert_to_tensor`.
+    axis: int. axis on which to expand the shape.
+
+  Returns:
+    An ndarray with the contents and dtype of `a` and shape expanded on axis.
+  """
+  a = asarray(a)
+  return np_utils.tensor_to_ndarray(array_ops.expand_dims(a.data, axis=axis))
+
+
+def squeeze(a, axis=None):
+  """Removes single-element axes from the array.
+
+  Args:
+    a: array_like. Could be an ndarray, a Tensor or any object that can be
+      converted to a Tensor using `tf.convert_to_tensor`.
+    axis: scalar or list/tuple of ints.
+  TODO(srbs): tf.squeeze throws error when axis is a Tensor eager execution is
+    enabled. So we cannot allow axis to be array_like here. Fix.
+
+  Returns:
+    An ndarray.
+  """
+  a = asarray(a)
+  return np_utils.tensor_to_ndarray(array_ops.squeeze(a, axis))
+
+
+def transpose(a, axes=None):
+  """Permutes dimensions of the array.
+
+  Args:
+    a: array_like. Could be an ndarray, a Tensor or any object that can be
+      converted to a Tensor using `tf.convert_to_tensor`.
+    axes: array_like. A list of ints with length rank(a) or None specifying the
+      order of permutation. The i'th dimension of the output array corresponds
+      to axes[i]'th dimension of the `a`. If None, the axes are reversed.
+
+  Returns:
+    An ndarray.
+  """
+  a = asarray(a)
+  if axes is not None:
+    axes = asarray(axes)
+  return np_utils.tensor_to_ndarray(array_ops.transpose(a=a.data, perm=axes))
+
+
+@np_utils.np_doc(np.swapaxes)
+def swapaxes(a, axis1, axis2):  # pylint: disable=missing-docstring
+  a = asarray(a)
+
+  a_rank = array_ops.rank(a)
+  if axis1 < 0:
+    axis1 += a_rank
+  if axis2 < 0:
+    axis2 += a_rank
+
+  perm = math_ops.range(a_rank)
+  perm = array_ops.tensor_scatter_update(perm, [[axis1], [axis2]],
+                                         [axis2, axis1])
+  a = array_ops.transpose(a, perm)
+
+  return np_utils.tensor_to_ndarray(a)
+
+
+@np_utils.np_doc(np.moveaxis)
+def moveaxis(a, source, destination):  # pylint: disable=missing-docstring
+  """Raises ValueError if source, destination not in (-ndim(a), ndim(a))."""
+  if not source and not destination:
+    return a
+
+  a = asarray(a).data
+
+  if isinstance(source, int):
+    source = (source,)
+  if isinstance(destination, int):
+    destination = (destination,)
+
+  a_rank = np_utils._maybe_static(array_ops.rank(a))  # pylint: disable=protected-access
+
+  def _correct_axis(axis, rank):
+    if axis < 0:
+      return axis + rank
+    return axis
+
+  source = tuple(_correct_axis(axis, a_rank) for axis in source)
+  destination = tuple(_correct_axis(axis, a_rank) for axis in destination)
+
+  if a.shape.rank is not None:
+    perm = [i for i in range(a_rank) if i not in source]
+    for dest, src in sorted(zip(destination, source)):
+      assert dest <= len(perm)
+      perm.insert(dest, src)
+  else:
+    r = math_ops.range(a_rank)
+
+    def _remove_indices(a, b):
+      """Remove indices (`b`) from `a`."""
+      items = array_ops.unstack(sort_ops.sort(array_ops.stack(b)), num=len(b))
+
+      i = 0
+      result = []
+
+      for item in items:
+        result.append(a[i:item])
+        i = item + 1
+
+      result.append(a[i:])
+
+      return array_ops.concat(result, 0)
+
+    minus_sources = _remove_indices(r, source)
+    minus_dest = _remove_indices(r, destination)
+
+    perm = array_ops.scatter_nd(
+        array_ops.expand_dims(minus_dest, 1), minus_sources, [a_rank])
+    perm = array_ops.tensor_scatter_update(
+        perm, array_ops.expand_dims(destination, 1), source)
+  a = array_ops.transpose(a, perm)
+
+  return np_utils.tensor_to_ndarray(a)
+
+
+def _setitem(arr, index, value):
+  """Sets the `value` at `index` in the array `arr`.
+
+  This works by replacing the slice at `index` in the tensor with `value`.
+  Since tensors are immutable, this builds a new tensor using the `tf.concat`
+  op. Currently, only 0-d and 1-d indices are supported.
+
+  Note that this may break gradients e.g.
+
+  a = tf_np.array([1, 2, 3])
+  old_a_t = a.data
+
+  with tf.GradientTape(persistent=True) as g:
+    g.watch(a.data)
+    b = a * 2
+    a[0] = 5
+  g.gradient(b.data, [a.data])  # [None]
+  g.gradient(b.data, [old_a_t])  # [[2., 2., 2.]]
+
+  Here `d_b / d_a` is `[None]` since a.data no longer points to the same
+  tensor.
+
+  Args:
+    arr: array_like.
+    index: scalar or 1-d integer array.
+    value: value to set at index.
+
+  Returns:
+    ndarray
+
+  Raises:
+    ValueError: if `index` is not a scalar or 1-d array.
+  """
+  # TODO(srbs): Figure out a solution to the gradient problem.
+  arr = asarray(arr)
+  index = asarray(index)
+  if index.ndim == 0:
+    index = ravel(index)
+  elif index.ndim > 1:
+    raise ValueError('index must be a scalar or a 1-d array.')
+  value = asarray(value, dtype=arr.dtype)
+  if arr.shape[len(index):] != value.shape:
+    value = full(arr.shape[len(index):], value)
+  prefix_t = arr.data[:index.data[0]]
+  postfix_t = arr.data[index.data[0] + 1:]
+  if len(index) == 1:
+    arr._data = array_ops.concat(  # pylint: disable=protected-access
+        [prefix_t, array_ops.expand_dims(value.data, 0), postfix_t], 0)
+  else:
+    subarray = arr[index.data[0]]
+    _setitem(subarray, index[1:], value)
+    arr._data = array_ops.concat(  # pylint: disable=protected-access
+        [prefix_t, array_ops.expand_dims(subarray.data, 0), postfix_t], 0)
+
+
+setattr(np_arrays.ndarray, 'transpose', transpose)
+setattr(np_arrays.ndarray, 'reshape', _reshape_method_wrapper)
+setattr(np_arrays.ndarray, '__setitem__', _setitem)
+
+
+def pad(ary, pad_width, mode, constant_values=0):
+  """Pads an array.
+
+  Args:
+    ary: array_like of rank N. Input array.
+    pad_width: {sequence, array_like, int}. Number of values padded to the edges
+      of each axis. ((before_1, after_1), ... (before_N, after_N)) unique pad
+      widths for each axis. ((before, after),) yields same before and after pad
+      for each axis. (pad,) or int is a shortcut for before = after = pad width
+      for all axes.
+    mode: string. One of the following string values: 'constant' Pads with a
+      constant value. 'reflect' Pads with the reflection of the vector mirrored
+      on the first and last values of the vector along each axis. 'symmetric'
+      Pads with the reflection of the vector mirrored along the edge of the
+      array.
+      **NOTE**: The supported list of `mode` does not match that of numpy's.
+    constant_values: scalar with same dtype as `array`. Used in 'constant' mode
+      as the pad value.  Default is 0.
+
+  Returns:
+    An ndarray padded array of rank equal to `array` with shape increased
+    according to `pad_width`.
+
+  Raises:
+    ValueError if `mode` is not supported.
+  """
+  if not (mode == 'constant' or mode == 'reflect' or mode == 'symmetric'):
+    raise ValueError('Unsupported padding mode: ' + mode)
+  mode = mode.upper()
+  ary = asarray(ary)
+  pad_width = asarray(pad_width, dtype=dtypes.int32)
+  return np_utils.tensor_to_ndarray(
+      array_ops.pad(
+          tensor=ary.data,
+          paddings=pad_width.data,
+          mode=mode,
+          constant_values=constant_values))
+
+
+@np_utils.np_doc(np.take)
+def take(a, indices, axis=None, out=None, mode='clip'):
+  """out argument is not supported, and default mode is clip."""
+  if out is not None:
+    raise ValueError('out argument is not supported in take.')
+
+  if mode not in {'raise', 'clip', 'wrap'}:
+    raise ValueError("Invalid mode '{}' for take".format(mode))
+
+  a = asarray(a).data
+  indices = asarray(indices).data
+
+  if axis is None:
+    a = array_ops.reshape(a, [-1])
+    axis = 0
+
+  axis_size = array_ops.shape(a, out_type=indices.dtype)[axis]
+  if mode == 'clip':
+    indices = clip_ops.clip_by_value(indices, 0, axis_size - 1)
+  elif mode == 'wrap':
+    indices = math_ops.floormod(indices, axis_size)
+  else:
+    raise ValueError("The 'raise' mode to take is not supported.")
+
+  return np_utils.tensor_to_ndarray(array_ops.gather(a, indices, axis=axis))
+
+
+@np_utils.np_doc_only(np.where)
+def where(condition, x=None, y=None):
+  """Raises ValueError if exactly one of x or y is not None."""
+  condition = asarray(condition, dtype=np.bool_)
+  if x is None and y is None:
+    return nonzero(condition)
+  elif x is not None and y is not None:
+    x, y = _promote_dtype(x, y)
+    return np_utils.tensor_to_ndarray(
+        array_ops.where_v2(condition.data, x.data, y.data))
+  raise ValueError('Both x and y must be ndarrays, or both must be None.')
+
+
+@np_utils.np_doc(np.select)
+def select(condlist, choicelist, default=0):  # pylint: disable=missing-docstring
+  if len(condlist) != len(choicelist):
+    msg = 'condlist must have length equal to choicelist ({} vs {})'
+    raise ValueError(msg.format(len(condlist), len(choicelist)))
+  if not condlist:
+    raise ValueError('condlist must be non-empty')
+  choices = _promote_dtype(default, *choicelist)
+  choicelist = choices[1:]
+  output = choices[0]
+  # The traversal is in reverse order so we can return the first value in
+  # choicelist where condlist is True.
+  for cond, choice in zip(condlist[::-1], choicelist[::-1]):
+    output = where(cond, choice, output)
+  return output
+
+
+def shape(a):
+  """Return the shape of an array.
+
+  Args:
+    a: array_like. Input array.
+
+  Returns:
+    Tuple of ints.
+  """
+  a = asarray(a)
+  return a.shape
+
+
+def ndim(a):
+  a = asarray(a)
+  return a.ndim
+
+
+def isscalar(a):
+  return ndim(a) == 0
+
+
+def _boundaries_to_sizes(a, boundaries, axis):
+  """Converting boundaries of splits to sizes of splits.
+
+  Args:
+    a: the array to be split.
+    boundaries: the boundaries, as in np.split.
+    axis: the axis along which to split.
+
+  Returns:
+    A list of sizes of the splits, as in tf.split.
+  """
+  if axis >= len(a.shape):
+    raise ValueError('axis %s is out of bound for shape %s' % (axis, a.shape))
+  total_size = a.shape[axis]
+  sizes = []
+  sizes_sum = 0
+  prev = 0
+  for i, b in enumerate(boundaries):
+    size = b - prev
+    if size < 0:
+      raise ValueError('The %s-th boundary %s is smaller than the previous '
+                       'boundary %s' % (i, b, prev))
+    size = min(size, max(0, total_size - sizes_sum))
+    sizes.append(size)
+    sizes_sum += size
+    prev = b
+  sizes.append(max(0, total_size - sizes_sum))
+  return sizes
+
+
+@np_utils.np_doc(np.split)
+def split(ary, indices_or_sections, axis=0):
+  ary = asarray(ary)
+  if not isinstance(indices_or_sections, six.integer_types):
+    indices_or_sections = _boundaries_to_sizes(ary, indices_or_sections, axis)
+  result = array_ops.split(ary.data, indices_or_sections, axis=axis)
+  return [np_utils.tensor_to_ndarray(a) for a in result]
+
+
+def _split_on_axis(np_fun, axis):
+
+  @np_utils.np_doc(np_fun)
+  def f(ary, indices_or_sections):
+    return split(ary, indices_or_sections, axis=axis)
+
+  return f
+
+
+vsplit = _split_on_axis(np.vsplit, axis=0)
+hsplit = _split_on_axis(np.hsplit, axis=1)
+dsplit = _split_on_axis(np.dsplit, axis=2)
+
+
+@np_utils.np_doc(np.broadcast_to)
+def broadcast_to(array, shape):  # pylint: disable=redefined-outer-name
+  return full(shape, array)
+
+
+@np_utils.np_doc(np.stack)
+def stack(arrays, axis=0):
+  arrays = _promote_dtype(*arrays)  # pylint: disable=protected-access
+  unwrapped_arrays = [
+      a.data if isinstance(a, np_arrays.ndarray) else a for a in arrays
+  ]
+  return asarray(array_ops.stack(unwrapped_arrays, axis))
+
+
+@np_utils.np_doc(np.hstack)
+def hstack(tup):
+  arrays = [atleast_1d(a) for a in tup]
+  arrays = _promote_dtype(*arrays)  # pylint: disable=protected-access
+  unwrapped_arrays = [
+      a.data if isinstance(a, np_arrays.ndarray) else a for a in arrays
+  ]
+  rank = array_ops.rank(unwrapped_arrays[0])
+  return np_utils.cond(
+      math_ops.equal(rank,
+                     1), lambda: array_ops.concat(unwrapped_arrays, axis=0),
+      lambda: array_ops.concat(unwrapped_arrays, axis=1))
+
+
+@np_utils.np_doc(np.vstack)
+def vstack(tup):
+  arrays = [atleast_2d(a) for a in tup]
+  arrays = _promote_dtype(*arrays)  # pylint: disable=protected-access
+  unwrapped_arrays = [
+      a.data if isinstance(a, np_arrays.ndarray) else a for a in arrays
+  ]
+  return array_ops.concat(unwrapped_arrays, axis=0)
+
+
+@np_utils.np_doc(np.dstack)
+def dstack(tup):
+  arrays = [atleast_3d(a) for a in tup]
+  arrays = _promote_dtype(*arrays)  # pylint: disable=protected-access
+  unwrapped_arrays = [
+      a.data if isinstance(a, np_arrays.ndarray) else a for a in arrays
+  ]
+  return array_ops.concat(unwrapped_arrays, axis=2)
+
+
+def _pad_left_to(n, old_shape):
+  old_shape = asarray(old_shape, dtype=np.int32).data
+  new_shape = array_ops.pad(
+      old_shape, [[math_ops.maximum(n - array_ops.size(old_shape), 0), 0]],
+      constant_values=1)
+  return asarray(new_shape)
+
+
+def _atleast_nd(n, new_shape, *arys):
+  """Reshape arrays to be at least `n`-dimensional.
+
+  Args:
+    n: The minimal rank.
+    new_shape: a function that takes `n` and the old shape and returns the
+      desired new shape.
+    *arys: ndarray(s) to be reshaped.
+
+  Returns:
+    The reshaped array(s).
+  """
+
+  def f(x):
+    # pylint: disable=g-long-lambda
+    x = asarray(x)
+    return asarray(
+        np_utils.cond(
+            np_utils.greater(n, array_ops.rank(x)),
+            lambda: reshape(x, new_shape(n, array_ops.shape(x.data))).data,
+            lambda: x.data))
+
+  arys = list(map(f, arys))
+  if len(arys) == 1:
+    return arys[0]
+  else:
+    return arys
+
+
+@np_utils.np_doc(np.atleast_1d)
+def atleast_1d(*arys):
+  return _atleast_nd(1, _pad_left_to, *arys)
+
+
+@np_utils.np_doc(np.atleast_2d)
+def atleast_2d(*arys):
+  return _atleast_nd(2, _pad_left_to, *arys)
+
+
+@np_utils.np_doc(np.atleast_3d)
+def atleast_3d(*arys):  # pylint: disable=missing-docstring
+
+  def new_shape(_, old_shape):
+    # pylint: disable=g-long-lambda
+    ndim_ = array_ops.size(old_shape)
+    return np_utils.cond(
+        math_ops.equal(ndim_, 0),
+        lambda: constant_op.constant([1, 1, 1], dtype=dtypes.int32),
+        lambda: np_utils.cond(
+            math_ops.equal(ndim_, 1), lambda: array_ops.pad(
+                old_shape, [[1, 1]], constant_values=1), lambda: array_ops.pad(
+                    old_shape, [[0, 1]], constant_values=1)))
+
+  return _atleast_nd(3, new_shape, *arys)
+
+
+@np_utils.np_doc(np.nonzero)
+def nonzero(a):
+  a = atleast_1d(a).data
+  if a.shape.rank is None:
+    raise ValueError("The rank of `a` is unknown, so we can't decide how many "
+                     'arrays to return.')
+  return nest.map_structure(
+      np_arrays.tensor_to_ndarray,
+      array_ops.unstack(
+          array_ops.where_v2(math_ops.cast(a, dtypes.bool)),
+          a.shape.rank,
+          axis=1))
+
+
+@np_utils.np_doc(np.diag_indices)
+def diag_indices(n, ndim=2):  # pylint: disable=missing-docstring,redefined-outer-name
+  if n < 0:
+    raise ValueError(
+        'n argument to diag_indices must be nonnegative, got {}'.format(n))
+  if ndim < 0:
+    raise ValueError(
+        'ndim argument to diag_indices must be nonnegative, got {}'.format(
+            ndim))
+
+  return (math_ops.range(n),) * ndim
+
+
+@np_utils.np_doc(np.tri)
+def tri(N, M=None, k=0, dtype=None):  # pylint: disable=invalid-name,missing-docstring
+  M = M if M is not None else N
+  if dtype is not None:
+    dtype = np_utils.result_type(dtype)
+  else:
+    dtype = np_dtypes.default_float_type()
+
+  if k < 0:
+    lower = -k - 1
+    if lower > N:
+      r = array_ops.zeros([N, M], dtype)
+    else:
+      # Keep as tf bool, since we create an upper triangular matrix and invert
+      # it.
+      o = array_ops.ones([N, M], dtype=dtypes.bool)
+      r = math_ops.cast(
+          math_ops.logical_not(array_ops.matrix_band_part(o, lower, -1)), dtype)
+  else:
+    o = array_ops.ones([N, M], dtype)
+    if k > M:
+      r = o
+    else:
+      r = array_ops.matrix_band_part(o, -1, k)
+  return np_utils.tensor_to_ndarray(r)
+
+
+@np_utils.np_doc(np.tril)
+def tril(m, k=0):  # pylint: disable=missing-docstring
+  m = asarray(m).data
+  m_shape = m.shape.as_list()
+
+  if len(m_shape) < 2:
+    raise ValueError('Argument to tril must have rank at least 2')
+
+  if m_shape[-1] is None or m_shape[-2] is None:
+    raise ValueError('Currently, the last two dimensions of the input array '
+                     'need to be known.')
+
+  z = constant_op.constant(0, m.dtype)
+
+  mask = tri(*m_shape[-2:], k=k, dtype=bool)
+  return np_utils.tensor_to_ndarray(
+      array_ops.where_v2(
+          array_ops.broadcast_to(mask, array_ops.shape(m)), m, z))
+
+
+@np_utils.np_doc(np.triu)
+def triu(m, k=0):  # pylint: disable=missing-docstring
+  m = asarray(m).data
+  m_shape = m.shape.as_list()
+
+  if len(m_shape) < 2:
+    raise ValueError('Argument to triu must have rank at least 2')
+
+  if m_shape[-1] is None or m_shape[-2] is None:
+    raise ValueError('Currently, the last two dimensions of the input array '
+                     'need to be known.')
+
+  z = constant_op.constant(0, m.dtype)
+
+  mask = tri(*m_shape[-2:], k=k - 1, dtype=bool)
+  return np_utils.tensor_to_ndarray(
+      array_ops.where_v2(
+          array_ops.broadcast_to(mask, array_ops.shape(m)), z, m))
+
+
+@np_utils.np_doc(np.flip)
+def flip(m, axis=None):  # pylint: disable=missing-docstring
+  m = asarray(m).data
+
+  if axis is None:
+    return np_utils.tensor_to_ndarray(
+        array_ops.reverse(m, math_ops.range(array_ops.rank(m))))
+
+  axis = np_utils._canonicalize_axis(axis, array_ops.rank(m))  # pylint: disable=protected-access
+
+  return np_utils.tensor_to_ndarray(array_ops.reverse(m, [axis]))
+
+
+@np_utils.np_doc(np.flipud)
+def flipud(m):  # pylint: disable=missing-docstring
+  return flip(m, 0)
+
+
+@np_utils.np_doc(np.fliplr)
+def fliplr(m):  # pylint: disable=missing-docstring
+  return flip(m, 1)
+
+
+@np_utils.np_doc(np.roll)
+def roll(a, shift, axis=None):  # pylint: disable=missing-docstring
+  a = asarray(a).data
+
+  if axis is not None:
+    return np_utils.tensor_to_ndarray(manip_ops.roll(a, shift, axis))
+
+  # If axis is None, the roll happens as a 1-d tensor.
+  original_shape = array_ops.shape(a)
+  a = manip_ops.roll(array_ops.reshape(a, [-1]), shift, 0)
+  return np_utils.tensor_to_ndarray(array_ops.reshape(a, original_shape))
+
+
+@np_utils.np_doc(np.rot90)
+def rot90(m, k=1, axes=(0, 1)):  # pylint: disable=missing-docstring
+  m_rank = array_ops.rank(m)
+  ax1, ax2 = np_utils._canonicalize_axes(axes, m_rank)  # pylint: disable=protected-access
+
+  k = k % 4
+  if k == 0:
+    return m
+  elif k == 2:
+    return flip(flip(m, ax1), ax2)
+  else:
+    perm = math_ops.range(m_rank)
+    perm = array_ops.tensor_scatter_update(perm, [[ax1], [ax2]], [ax2, ax1])
+
+    if k == 1:
+      return transpose(flip(m, ax2), perm)
+    else:
+      return flip(transpose(m, perm), ax2)
+
+
+@np_utils.np_doc(np.vander)
+def vander(x, N=None, increasing=False):  # pylint: disable=missing-docstring,invalid-name
+  x = asarray(x).data
+
+  x_shape = array_ops.shape(x)
+  N = N or x_shape[0]
+
+  N_temp = np_utils.get_static_value(N)  # pylint: disable=invalid-name
+  if N_temp is not None:
+    N = N_temp
+    if N < 0:
+      raise ValueError('N must be nonnegative')
+  else:
+    control_flow_ops.Assert(N >= 0, [N])
+
+  rank = array_ops.rank(x)
+  rank_temp = np_utils.get_static_value(rank)
+  if rank_temp is not None:
+    rank = rank_temp
+    if rank != 1:
+      raise ValueError('x must be a one-dimensional array')
+  else:
+    control_flow_ops.Assert(math_ops.equal(rank, 1), [rank])
+
+  if increasing:
+    start = 0
+    limit = N
+    delta = 1
+  else:
+    start = N - 1
+    limit = -1
+    delta = -1
+
+  x = array_ops.expand_dims(x, -1)
+  return np_utils.tensor_to_ndarray(
+      math_ops.pow(
+          x, math_ops.cast(math_ops.range(start, limit, delta), dtype=x.dtype)))
+
+
+@np_utils.np_doc(np.ix_)
+def ix_(*args):  # pylint: disable=missing-docstring
+  n = len(args)
+  output = []
+  for i, a in enumerate(args):
+    a = asarray(a).data
+    a_rank = array_ops.rank(a)
+    a_rank_temp = np_utils.get_static_value(a_rank)
+    if a_rank_temp is not None:
+      a_rank = a_rank_temp
+      if a_rank != 1:
+        raise ValueError('Arguments must be 1-d, got arg {} of rank {}'.format(
+            i, a_rank))
+    else:
+      control_flow_ops.Assert(math_ops.equal(a_rank, 1), [a_rank])
+
+    new_shape = [1] * n
+    new_shape[i] = -1
+    dtype = a.dtype
+    if dtype == dtypes.bool:
+      output.append(
+          np_utils.tensor_to_ndarray(
+              array_ops.reshape(nonzero(a)[0].data, new_shape)))
+    elif dtype.is_integer:
+      output.append(np_utils.tensor_to_ndarray(array_ops.reshape(a, new_shape)))
+    else:
+      raise ValueError(
+          'Only integer and bool dtypes are supported, got {}'.format(dtype))
+
+  return output
diff --git a/tensorflow/python/ops/numpy_ops/np_array_ops_test.py b/tensorflow/python/ops/numpy_ops/np_array_ops_test.py
new file mode 100644
index 00000000000..8194c1b2897
--- /dev/null
+++ b/tensorflow/python/ops/numpy_ops/np_array_ops_test.py
@@ -0,0 +1,1154 @@
+# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Tests for tf numpy array methods."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import itertools
+import sys
+import numpy as np
+from six.moves import range
+from six.moves import zip
+
+from tensorflow.python.framework import constant_op
+from tensorflow.python.framework import dtypes
+from tensorflow.python.framework import indexed_slices
+from tensorflow.python.framework import ops
+from tensorflow.python.framework import tensor_shape
+from tensorflow.python.framework import test_util
+from tensorflow.python.ops import array_ops
+from tensorflow.python.ops.numpy_ops import np_array_ops
+from tensorflow.python.ops.numpy_ops import np_arrays
+from tensorflow.python.platform import test
+
+
+class ArrayCreationTest(test.TestCase):
+
+  def setUp(self):
+    super(ArrayCreationTest, self).setUp()
+    python_shapes = [
+        0, 1, 2, (), (1,), (2,), (1, 2, 3), [], [1], [2], [1, 2, 3]
+    ]
+    self.shape_transforms = [
+        lambda x: x, lambda x: np.array(x, dtype=int),
+        lambda x: np_array_ops.array(x, dtype=int), tensor_shape.TensorShape
+    ]
+
+    self.all_shapes = []
+    for fn in self.shape_transforms:
+      self.all_shapes.extend([fn(s) for s in python_shapes])
+
+    if sys.version_info.major == 3:
+      # There is a bug of np.empty (and alike) in Python 3 causing a crash when
+      # the `shape` argument is an np_arrays.ndarray scalar (or tf.Tensor
+      # scalar).
+      def not_ndarray_scalar(s):
+        return not (isinstance(s, np_arrays.ndarray) and s.ndim == 0)
+
+      self.all_shapes = list(filter(not_ndarray_scalar, self.all_shapes))
+
+    self.all_types = [
+        int, float, np.int16, np.int32, np.int64, np.float16, np.float32,
+        np.float64
+    ]
+
+    source_array_data = [
+        1,
+        5.5,
+        7,
+        (),
+        (8, 10.),
+        ((), ()),
+        ((1, 4), (2, 8)),
+        [],
+        [7],
+        [8, 10.],
+        [[], []],
+        [[1, 4], [2, 8]],
+        ([], []),
+        ([1, 4], [2, 8]),
+        [(), ()],
+        [(1, 4), (2, 8)],
+    ]
+
+    self.array_transforms = [
+        lambda x: x,
+        ops.convert_to_tensor,
+        np.array,
+        np_array_ops.array,
+    ]
+    self.all_arrays = []
+    for fn in self.array_transforms:
+      self.all_arrays.extend([fn(s) for s in source_array_data])
+
+  def testEmpty(self):
+    for s in self.all_shapes:
+      actual = np_array_ops.empty(s)
+      expected = np.empty(s)
+      msg = 'shape: {}'.format(s)
+      self.match_shape(actual, expected, msg)
+      self.match_dtype(actual, expected, msg)
+
+    for s, t in itertools.product(self.all_shapes, self.all_types):
+      actual = np_array_ops.empty(s, t)
+      expected = np.empty(s, t)
+      msg = 'shape: {}, dtype: {}'.format(s, t)
+      self.match_shape(actual, expected, msg)
+      self.match_dtype(actual, expected, msg)
+
+  def testEmptyLike(self):
+    for a in self.all_arrays:
+      actual = np_array_ops.empty_like(a)
+      expected = np.empty_like(a)
+      msg = 'array: {}'.format(a)
+      self.match_shape(actual, expected, msg)
+      self.match_dtype(actual, expected, msg)
+
+    for a, t in itertools.product(self.all_arrays, self.all_types):
+      actual = np_array_ops.empty_like(a, t)
+      expected = np.empty_like(a, t)
+      msg = 'array: {} type: {}'.format(a, t)
+      self.match_shape(actual, expected, msg)
+      self.match_dtype(actual, expected, msg)
+
+  def testZeros(self):
+    for s in self.all_shapes:
+      actual = np_array_ops.zeros(s)
+      expected = np.zeros(s)
+      msg = 'shape: {}'.format(s)
+      self.match(actual, expected, msg)
+
+    for s, t in itertools.product(self.all_shapes, self.all_types):
+      actual = np_array_ops.zeros(s, t)
+      expected = np.zeros(s, t)
+      msg = 'shape: {}, dtype: {}'.format(s, t)
+      self.match(actual, expected, msg)
+
+  def testZerosLike(self):
+    for a in self.all_arrays:
+      actual = np_array_ops.zeros_like(a)
+      expected = np.zeros_like(a)
+      msg = 'array: {}'.format(a)
+      self.match(actual, expected, msg)
+
+    for a, t in itertools.product(self.all_arrays, self.all_types):
+      actual = np_array_ops.zeros_like(a, t)
+      expected = np.zeros_like(a, t)
+      msg = 'array: {} type: {}'.format(a, t)
+      self.match(actual, expected, msg)
+
+  def testOnes(self):
+    for s in self.all_shapes:
+      actual = np_array_ops.ones(s)
+      expected = np.ones(s)
+      msg = 'shape: {}'.format(s)
+      self.match(actual, expected, msg)
+
+    for s, t in itertools.product(self.all_shapes, self.all_types):
+      actual = np_array_ops.ones(s, t)
+      expected = np.ones(s, t)
+      msg = 'shape: {}, dtype: {}'.format(s, t)
+      self.match(actual, expected, msg)
+
+  def testOnesLike(self):
+    for a in self.all_arrays:
+      actual = np_array_ops.ones_like(a)
+      expected = np.ones_like(a)
+      msg = 'array: {}'.format(a)
+      self.match(actual, expected, msg)
+
+    for a, t in itertools.product(self.all_arrays, self.all_types):
+      actual = np_array_ops.ones_like(a, t)
+      expected = np.ones_like(a, t)
+      msg = 'array: {} type: {}'.format(a, t)
+      self.match(actual, expected, msg)
+
+  def testEye(self):
+    n_max = 3
+    m_max = 3
+
+    for n in range(1, n_max + 1):
+      self.match(np_array_ops.eye(n), np.eye(n))
+      for k in range(-n, n + 1):
+        self.match(np_array_ops.eye(n, k=k), np.eye(n, k=k))
+      for m in range(1, m_max + 1):
+        self.match(np_array_ops.eye(n, m), np.eye(n, m))
+        for k in range(-n, m):
+          self.match(np_array_ops.eye(n, k=k), np.eye(n, k=k))
+          self.match(np_array_ops.eye(n, m, k), np.eye(n, m, k))
+
+    for dtype in self.all_types:
+      for n in range(1, n_max + 1):
+        self.match(np_array_ops.eye(n, dtype=dtype), np.eye(n, dtype=dtype))
+        for k in range(-n, n + 1):
+          self.match(
+              np_array_ops.eye(n, k=k, dtype=dtype),
+              np.eye(n, k=k, dtype=dtype))
+        for m in range(1, m_max + 1):
+          self.match(
+              np_array_ops.eye(n, m, dtype=dtype), np.eye(n, m, dtype=dtype))
+          for k in range(-n, m):
+            self.match(
+                np_array_ops.eye(n, k=k, dtype=dtype),
+                np.eye(n, k=k, dtype=dtype))
+            self.match(
+                np_array_ops.eye(n, m, k, dtype=dtype),
+                np.eye(n, m, k, dtype=dtype))
+
+  def testIdentity(self):
+    n_max = 3
+
+    for n in range(1, n_max + 1):
+      self.match(np_array_ops.identity(n), np.identity(n))
+
+    for dtype in self.all_types:
+      for n in range(1, n_max + 1):
+        self.match(
+            np_array_ops.identity(n, dtype=dtype), np.identity(n, dtype=dtype))
+
+  def testFull(self):
+    # List of 2-tuples of fill value and shape.
+    data = [
+        (5, ()),
+        (5, (7,)),
+        (5., (7,)),
+        ([5, 8], (2,)),
+        ([5, 8], (3, 2)),
+        ([[5], [8]], (2, 3)),
+        ([[5], [8]], (3, 2, 5)),
+        ([[5.], [8.]], (3, 2, 5)),
+        ([[3, 4], [5, 6], [7, 8]], (3, 3, 2)),
+    ]
+    for f, s in data:
+      for fn1, fn2 in itertools.product(self.array_transforms,
+                                        self.shape_transforms):
+        fill_value = fn1(f)
+        shape = fn2(s)
+        self.match(
+            np_array_ops.full(shape, fill_value), np.full(shape, fill_value))
+        for dtype in self.all_types:
+          self.match(
+              np_array_ops.full(shape, fill_value, dtype=dtype),
+              np.full(shape, fill_value, dtype=dtype))
+
+  def testFullLike(self):
+    # List of 2-tuples of fill value and shape.
+    data = [
+        (5, ()),
+        (5, (7,)),
+        (5., (7,)),
+        ([5, 8], (2,)),
+        ([5, 8], (3, 2)),
+        ([[5], [8]], (2, 3)),
+        ([[5], [8]], (3, 2, 5)),
+        ([[5.], [8.]], (3, 2, 5)),
+    ]
+    zeros_builders = [np_array_ops.zeros, np.zeros]
+    for f, s in data:
+      for fn1, fn2, arr_dtype in itertools.product(self.array_transforms,
+                                                   zeros_builders,
+                                                   self.all_types):
+        fill_value = fn1(f)
+        arr = fn2(s, arr_dtype)
+        self.match(
+            np_array_ops.full_like(arr, fill_value),
+            np.full_like(arr, fill_value))
+        for dtype in self.all_types:
+          self.match(
+              np_array_ops.full_like(arr, fill_value, dtype=dtype),
+              np.full_like(arr, fill_value, dtype=dtype))
+
+  def testArray(self):
+    ndmins = [0, 1, 2, 5]
+    for a, dtype, ndmin, copy in itertools.product(self.all_arrays,
+                                                   self.all_types, ndmins,
+                                                   [True, False]):
+      self.match(
+          np_array_ops.array(a, dtype=dtype, ndmin=ndmin, copy=copy),
+          np.array(a, dtype=dtype, ndmin=ndmin, copy=copy))
+
+    zeros_list = np_array_ops.zeros(5)
+
+    # TODO(srbs): Test that copy=True when context.device is different from
+    # tensor device copies the tensor.
+
+    # Backing tensor is the same if copy=False, other attributes being None.
+    self.assertIs(
+        np_array_ops.array(zeros_list, copy=False).data, zeros_list.data)
+    self.assertIs(
+        np_array_ops.array(zeros_list.data, copy=False).data, zeros_list.data)
+
+    # Backing tensor is different if ndmin is not satisfied.
+    self.assertIsNot(
+        np_array_ops.array(zeros_list, copy=False, ndmin=2).data,
+        zeros_list.data)
+    self.assertIsNot(
+        np_array_ops.array(zeros_list.data, copy=False, ndmin=2).data,
+        zeros_list.data)
+    self.assertIs(
+        np_array_ops.array(zeros_list, copy=False, ndmin=1).data,
+        zeros_list.data)
+    self.assertIs(
+        np_array_ops.array(zeros_list.data, copy=False, ndmin=1).data,
+        zeros_list.data)
+
+    # Backing tensor is different if dtype is not satisfied.
+    self.assertIsNot(
+        np_array_ops.array(zeros_list, copy=False, dtype=int).data,
+        zeros_list.data)
+    self.assertIsNot(
+        np_array_ops.array(zeros_list.data, copy=False, dtype=int).data,
+        zeros_list.data)
+    self.assertIs(
+        np_array_ops.array(zeros_list, copy=False, dtype=float).data,
+        zeros_list.data)
+    self.assertIs(
+        np_array_ops.array(zeros_list.data, copy=False, dtype=float).data,
+        zeros_list.data)
+
+  def testAsArray(self):
+    for a, dtype in itertools.product(self.all_arrays, self.all_types):
+      self.match(
+          np_array_ops.asarray(a, dtype=dtype), np.asarray(a, dtype=dtype))
+
+    zeros_list = np_array_ops.zeros(5)
+    # Same instance is returned if no dtype is specified and input is ndarray.
+    self.assertIs(np_array_ops.asarray(zeros_list), zeros_list)
+    # Different instance is returned if dtype is specified and input is ndarray.
+    self.assertIsNot(np_array_ops.asarray(zeros_list, dtype=int), zeros_list)
+
+  def testAsAnyArray(self):
+    for a, dtype in itertools.product(self.all_arrays, self.all_types):
+      self.match(
+          np_array_ops.asanyarray(a, dtype=dtype),
+          np.asanyarray(a, dtype=dtype))
+    zeros_list = np_array_ops.zeros(5)
+    # Same instance is returned if no dtype is specified and input is ndarray.
+    self.assertIs(np_array_ops.asanyarray(zeros_list), zeros_list)
+    # Different instance is returned if dtype is specified and input is ndarray.
+    self.assertIsNot(np_array_ops.asanyarray(zeros_list, dtype=int), zeros_list)
+
+  def testAsContiguousArray(self):
+    for a, dtype in itertools.product(self.all_arrays, self.all_types):
+      self.match(
+          np_array_ops.ascontiguousarray(a, dtype=dtype),
+          np.ascontiguousarray(a, dtype=dtype))
+
+  def testARange(self):
+    int_values = np.arange(-3, 3).tolist()
+    float_values = np.arange(-3.5, 3.5).tolist()
+    all_values = int_values + float_values
+    for dtype in self.all_types:
+      for start in all_values:
+        msg = 'dtype:{} start:{}'.format(dtype, start)
+        self.match(np_array_ops.arange(start), np.arange(start), msg=msg)
+        self.match(
+            np_array_ops.arange(start, dtype=dtype),
+            np.arange(start, dtype=dtype),
+            msg=msg)
+        for stop in all_values:
+          msg = 'dtype:{} start:{} stop:{}'.format(dtype, start, stop)
+          self.match(
+              np_array_ops.arange(start, stop), np.arange(start, stop), msg=msg)
+          # TODO(srbs): Investigate and remove check.
+          # There are some bugs when start or stop is float and dtype is int.
+          if not isinstance(start, float) and not isinstance(stop, float):
+            self.match(
+                np_array_ops.arange(start, stop, dtype=dtype),
+                np.arange(start, stop, dtype=dtype),
+                msg=msg)
+          # Note: We intentionally do not test with float values for step
+          # because numpy.arange itself returns inconsistent results. e.g.
+          # np.arange(0.5, 3, step=0.5, dtype=int) returns
+          # array([0, 1, 2, 3, 4])
+          for step in int_values:
+            msg = 'dtype:{} start:{} stop:{} step:{}'.format(
+                dtype, start, stop, step)
+            if not step:
+              with self.assertRaises(ValueError):
+                self.match(
+                    np_array_ops.arange(start, stop, step),
+                    np.arange(start, stop, step),
+                    msg=msg)
+                if not isinstance(start, float) and not isinstance(stop, float):
+                  self.match(
+                      np_array_ops.arange(start, stop, step, dtype=dtype),
+                      np.arange(start, stop, step, dtype=dtype),
+                      msg=msg)
+            else:
+              self.match(
+                  np_array_ops.arange(start, stop, step),
+                  np.arange(start, stop, step),
+                  msg=msg)
+              if not isinstance(start, float) and not isinstance(stop, float):
+                self.match(
+                    np_array_ops.arange(start, stop, step, dtype=dtype),
+                    np.arange(start, stop, step, dtype=dtype),
+                    msg=msg)
+
+  def testGeomSpace(self):
+
+    def run_test(start, stop, **kwargs):
+      arg1 = start
+      arg2 = stop
+      if test_util.is_gpu_available():
+        decimal = 3
+      else:
+        decimal = 4
+      self.match(
+          np_array_ops.geomspace(arg1, arg2, **kwargs),
+          np.geomspace(arg1, arg2, **kwargs),
+          msg='geomspace({}, {})'.format(arg1, arg2),
+          almost=True,
+          decimal=decimal)
+
+    run_test(1, 1000, num=5)
+    run_test(1, 1000, num=5, endpoint=False)
+    run_test(-1, -1000, num=5)
+    run_test(-1, -1000, num=5, endpoint=False)
+
+  def testDiag(self):
+    array_transforms = [
+        lambda x: x,  # Identity,
+        ops.convert_to_tensor,
+        np.array,
+        lambda x: np.array(x, dtype=np.float32),
+        lambda x: np.array(x, dtype=np.float64),
+        np_array_ops.array,
+        lambda x: np_array_ops.array(x, dtype=np.float32),
+        lambda x: np_array_ops.array(x, dtype=np.float64)
+    ]
+
+    def run_test(arr):
+      for fn in array_transforms:
+        arr = fn(arr)
+        self.match(
+            np_array_ops.diag(arr), np.diag(arr), msg='diag({})'.format(arr))
+        for k in range(-3, 3):
+          self.match(
+              np_array_ops.diag(arr, k),
+              np.diag(arr, k),
+              msg='diag({}, k={})'.format(arr, k))
+
+    # 2-d arrays.
+    run_test(np.arange(9).reshape((3, 3)).tolist())
+    run_test(np.arange(6).reshape((2, 3)).tolist())
+    run_test(np.arange(6).reshape((3, 2)).tolist())
+    run_test(np.arange(3).reshape((1, 3)).tolist())
+    run_test(np.arange(3).reshape((3, 1)).tolist())
+    run_test([[5]])
+    run_test([[]])
+    run_test([[], []])
+
+    # 1-d arrays.
+    run_test([])
+    run_test([1])
+    run_test([1, 2])
+
+  def testDiagFlat(self):
+    array_transforms = [
+        lambda x: x,  # Identity,
+        ops.convert_to_tensor,
+        np.array,
+        lambda x: np.array(x, dtype=np.float32),
+        lambda x: np.array(x, dtype=np.float64),
+        np_array_ops.array,
+        lambda x: np_array_ops.array(x, dtype=np.float32),
+        lambda x: np_array_ops.array(x, dtype=np.float64)
+    ]
+
+    def run_test(arr):
+      for fn in array_transforms:
+        arr = fn(arr)
+        self.match(
+            np_array_ops.diagflat(arr),
+            np.diagflat(arr),
+            msg='diagflat({})'.format(arr))
+        for k in range(-3, 3):
+          self.match(
+              np_array_ops.diagflat(arr, k),
+              np.diagflat(arr, k),
+              msg='diagflat({}, k={})'.format(arr, k))
+
+    # 1-d arrays.
+    run_test([])
+    run_test([1])
+    run_test([1, 2])
+    # 2-d arrays.
+    run_test([[]])
+    run_test([[5]])
+    run_test([[], []])
+    run_test(np.arange(4).reshape((2, 2)).tolist())
+    run_test(np.arange(2).reshape((2, 1)).tolist())
+    run_test(np.arange(2).reshape((1, 2)).tolist())
+    # 3-d arrays
+    run_test(np.arange(8).reshape((2, 2, 2)).tolist())
+
+  def match_shape(self, actual, expected, msg=None):
+    if msg:
+      msg = 'Shape match failed for: {}. Expected: {} Actual: {}'.format(
+          msg, expected.shape, actual.shape)
+    self.assertEqual(actual.shape, expected.shape, msg=msg)
+    if msg:
+      msg = 'Shape: {} is not a tuple for {}'.format(actual.shape, msg)
+    self.assertIsInstance(actual.shape, tuple, msg=msg)
+
+  def match_dtype(self, actual, expected, msg=None):
+    if msg:
+      msg = 'Dtype match failed for: {}. Expected: {} Actual: {}.'.format(
+          msg, expected.dtype, actual.dtype)
+    self.assertEqual(actual.dtype, expected.dtype, msg=msg)
+
+  def match(self, actual, expected, msg=None, almost=False, decimal=7):
+    msg_ = 'Expected: {} Actual: {}'.format(expected, actual)
+    if msg:
+      msg = '{} {}'.format(msg_, msg)
+    else:
+      msg = msg_
+    self.assertIsInstance(actual, np_arrays.ndarray)
+    self.match_dtype(actual, expected, msg)
+    self.match_shape(actual, expected, msg)
+    if not almost:
+      if not actual.shape:
+        self.assertEqual(actual.tolist(), expected.tolist())
+      else:
+        self.assertSequenceEqual(actual.tolist(), expected.tolist())
+    else:
+      np.testing.assert_almost_equal(
+          actual.tolist(), expected.tolist(), decimal=decimal)
+
+  def testIndexedSlices(self):
+    dtype = dtypes.int64
+    iss = indexed_slices.IndexedSlices(
+        values=np_array_ops.ones([2, 3], dtype=dtype),
+        indices=constant_op.constant([1, 9]),
+        dense_shape=[10, 3])
+    a = np_array_ops.array(iss, copy=False)
+    expected = array_ops.scatter_nd([[1], [9]],
+                                    array_ops.ones([2, 3], dtype=dtype),
+                                    [10, 3])
+    self.assertAllEqual(expected, a)
+
+
+class ArrayMethodsTest(test.TestCase):
+
+  def setUp(self):
+    super(ArrayMethodsTest, self).setUp()
+    self.array_transforms = [
+        lambda x: x,
+        ops.convert_to_tensor,
+        np.array,
+        np_array_ops.array,
+    ]
+
+  def testAllAny(self):
+
+    def run_test(arr, *args, **kwargs):
+      for fn in self.array_transforms:
+        arr = fn(arr)
+        self.match(
+            np_array_ops.all(arr, *args, **kwargs),
+            np.all(arr, *args, **kwargs))
+        self.match(
+            np_array_ops.any(arr, *args, **kwargs),
+            np.any(arr, *args, **kwargs))
+
+    run_test(0)
+    run_test(1)
+    run_test([])
+    run_test([[True, False], [True, True]])
+    run_test([[True, False], [True, True]], axis=0)
+    run_test([[True, False], [True, True]], axis=0, keepdims=True)
+    run_test([[True, False], [True, True]], axis=1)
+    run_test([[True, False], [True, True]], axis=1, keepdims=True)
+    run_test([[True, False], [True, True]], axis=(0, 1))
+    run_test([[True, False], [True, True]], axis=(0, 1), keepdims=True)
+    run_test([5.2, 3.5], axis=0)
+    run_test([1, 0], axis=0)
+
+  def testCompress(self):
+
+    def run_test(condition, arr, *args, **kwargs):
+      for fn1 in self.array_transforms:
+        for fn2 in self.array_transforms:
+          arg1 = fn1(condition)
+          arg2 = fn2(arr)
+          self.match(
+              np_array_ops.compress(arg1, arg2, *args, **kwargs),
+              np.compress(
+                  np.asarray(arg1).astype(np.bool), arg2, *args, **kwargs))
+
+    run_test([True], 5)
+    run_test([False], 5)
+    run_test([], 5)
+    run_test([True, False, True], [1, 2, 3])
+    run_test([True, False], [1, 2, 3])
+    run_test([False, True], [[1, 2], [3, 4]])
+    run_test([1, 0, 1], [1, 2, 3])
+    run_test([1, 0], [1, 2, 3])
+    run_test([0, 1], [[1, 2], [3, 4]])
+    run_test([True], [[1, 2], [3, 4]])
+    run_test([False, True], [[1, 2], [3, 4]], axis=1)
+    run_test([False, True], [[1, 2], [3, 4]], axis=0)
+    run_test([False, True], [[1, 2], [3, 4]], axis=-1)
+    run_test([False, True], [[1, 2], [3, 4]], axis=-2)
+
+  def testCopy(self):
+
+    def run_test(arr, *args, **kwargs):
+      for fn in self.array_transforms:
+        arg = fn(arr)
+        self.match(
+            np_array_ops.copy(arg, *args, **kwargs),
+            np.copy(arg, *args, **kwargs))
+
+    run_test([])
+    run_test([1, 2, 3])
+    run_test([1., 2., 3.])
+    run_test([True])
+    run_test(np.arange(9).reshape((3, 3)).tolist())
+
+  def testCumProdAndSum(self):
+
+    def run_test(arr, *args, **kwargs):
+      for fn in self.array_transforms:
+        arg = fn(arr)
+        self.match(
+            np_array_ops.cumprod(arg, *args, **kwargs),
+            np.cumprod(arg, *args, **kwargs))
+        self.match(
+            np_array_ops.cumsum(arg, *args, **kwargs),
+            np.cumsum(arg, *args, **kwargs))
+
+    run_test([])
+    run_test([1, 2, 3])
+    run_test([1, 2, 3], dtype=float)
+    run_test([1, 2, 3], dtype=np.float32)
+    run_test([1, 2, 3], dtype=np.float64)
+    run_test([1., 2., 3.])
+    run_test([1., 2., 3.], dtype=int)
+    run_test([1., 2., 3.], dtype=np.int32)
+    run_test([1., 2., 3.], dtype=np.int64)
+    run_test([[1, 2], [3, 4]], axis=1)
+    run_test([[1, 2], [3, 4]], axis=0)
+    run_test([[1, 2], [3, 4]], axis=-1)
+    run_test([[1, 2], [3, 4]], axis=-2)
+
+  def testImag(self):
+
+    def run_test(arr, *args, **kwargs):
+      for fn in self.array_transforms:
+        arg = fn(arr)
+        self.match(
+            np_array_ops.imag(arg, *args, **kwargs),
+            # np.imag may return a scalar so we convert to a np.ndarray.
+            np.array(np.imag(arg, *args, **kwargs)))
+
+    run_test(1)
+    run_test(5.5)
+    run_test(5 + 3j)
+    run_test(3j)
+    run_test([])
+    run_test([1, 2, 3])
+    run_test([1 + 5j, 2 + 3j])
+    run_test([[1 + 5j, 2 + 3j], [1 + 7j, 2 + 8j]])
+
+  def testAMaxAMin(self):
+
+    def run_test(arr, *args, **kwargs):
+      axis = kwargs.pop('axis', None)
+      for fn1 in self.array_transforms:
+        for fn2 in self.array_transforms:
+          arr_arg = fn1(arr)
+          axis_arg = fn2(axis) if axis is not None else None
+          self.match(
+              np_array_ops.amax(arr_arg, axis=axis_arg, *args, **kwargs),
+              np.amax(arr_arg, axis=axis, *args, **kwargs))
+          self.match(
+              np_array_ops.amin(arr_arg, axis=axis_arg, *args, **kwargs),
+              np.amin(arr_arg, axis=axis, *args, **kwargs))
+
+    run_test([1, 2, 3])
+    run_test([1., 2., 3.])
+    run_test([[1, 2], [3, 4]], axis=1)
+    run_test([[1, 2], [3, 4]], axis=0)
+    run_test([[1, 2], [3, 4]], axis=-1)
+    run_test([[1, 2], [3, 4]], axis=-2)
+    run_test([[1, 2], [3, 4]], axis=(0, 1))
+    run_test(np.arange(8).reshape((2, 2, 2)).tolist(), axis=(0, 2))
+    run_test(
+        np.arange(8).reshape((2, 2, 2)).tolist(), axis=(0, 2), keepdims=True)
+    run_test(np.arange(8).reshape((2, 2, 2)).tolist(), axis=(2, 0))
+    run_test(
+        np.arange(8).reshape((2, 2, 2)).tolist(), axis=(2, 0), keepdims=True)
+
+  def testMean(self):
+
+    def run_test(arr, *args, **kwargs):
+      axis = kwargs.pop('axis', None)
+      for fn1 in self.array_transforms:
+        for fn2 in self.array_transforms:
+          arr_arg = fn1(arr)
+          axis_arg = fn2(axis) if axis is not None else None
+          self.match(
+              np_array_ops.mean(arr_arg, axis=axis_arg, *args, **kwargs),
+              np.mean(arr_arg, axis=axis, *args, **kwargs))
+
+    run_test([1, 2, 1])
+    run_test([1., 2., 1.])
+    run_test([1., 2., 1.], dtype=int)
+    run_test([[1, 2], [3, 4]], axis=1)
+    run_test([[1, 2], [3, 4]], axis=0)
+    run_test([[1, 2], [3, 4]], axis=-1)
+    run_test([[1, 2], [3, 4]], axis=-2)
+    run_test([[1, 2], [3, 4]], axis=(0, 1))
+    run_test(np.arange(8).reshape((2, 2, 2)).tolist(), axis=(0, 2))
+    run_test(
+        np.arange(8).reshape((2, 2, 2)).tolist(), axis=(0, 2), keepdims=True)
+    run_test(np.arange(8).reshape((2, 2, 2)).tolist(), axis=(2, 0))
+    run_test(
+        np.arange(8).reshape((2, 2, 2)).tolist(), axis=(2, 0), keepdims=True)
+
+  def testProd(self):
+
+    def run_test(arr, *args, **kwargs):
+      for fn in self.array_transforms:
+        arg = fn(arr)
+        self.match(
+            np_array_ops.prod(arg, *args, **kwargs),
+            np.prod(arg, *args, **kwargs))
+
+    run_test([1, 2, 3])
+    run_test([1., 2., 3.])
+    run_test(np.array([1, 2, 3], dtype=np.int16))
+    run_test([[1, 2], [3, 4]], axis=1)
+    run_test([[1, 2], [3, 4]], axis=0)
+    run_test([[1, 2], [3, 4]], axis=-1)
+    run_test([[1, 2], [3, 4]], axis=-2)
+    run_test([[1, 2], [3, 4]], axis=(0, 1))
+    run_test(np.arange(8).reshape((2, 2, 2)).tolist(), axis=(0, 2))
+    run_test(
+        np.arange(8).reshape((2, 2, 2)).tolist(), axis=(0, 2), keepdims=True)
+    run_test(np.arange(8).reshape((2, 2, 2)).tolist(), axis=(2, 0))
+    run_test(
+        np.arange(8).reshape((2, 2, 2)).tolist(), axis=(2, 0), keepdims=True)
+
+  def _testReduce(self, math_fun, np_fun, name):
+    axis_transforms = [
+        lambda x: x,  # Identity,
+        ops.convert_to_tensor,
+        np.array,
+        np_array_ops.array,
+        lambda x: np_array_ops.array(x, dtype=np.float32),
+        lambda x: np_array_ops.array(x, dtype=np.float64),
+    ]
+
+    def run_test(a, **kwargs):
+      axis = kwargs.pop('axis', None)
+      for fn1 in self.array_transforms:
+        for fn2 in axis_transforms:
+          arg1 = fn1(a)
+          axis_arg = fn2(axis) if axis is not None else None
+          self.match(
+              math_fun(arg1, axis=axis_arg, **kwargs),
+              np_fun(arg1, axis=axis, **kwargs),
+              msg='{}({}, axis={}, keepdims={})'.format(name, arg1, axis,
+                                                        kwargs.get('keepdims')))
+
+    run_test(5)
+    run_test([2, 3])
+    run_test([[2, -3], [-6, 7]])
+    run_test([[2, -3], [-6, 7]], axis=0)
+    run_test([[2, -3], [-6, 7]], axis=0, keepdims=True)
+    run_test([[2, -3], [-6, 7]], axis=1)
+    run_test([[2, -3], [-6, 7]], axis=1, keepdims=True)
+    run_test([[2, -3], [-6, 7]], axis=(0, 1))
+    run_test([[2, -3], [-6, 7]], axis=(1, 0))
+
+  def testSum(self):
+    self._testReduce(np_array_ops.sum, np.sum, 'sum')
+
+  def testAmax(self):
+    self._testReduce(np_array_ops.amax, np.amax, 'amax')
+
+  def testRavel(self):
+
+    def run_test(arr, *args, **kwargs):
+      for fn in self.array_transforms:
+        arg = fn(arr)
+        self.match(
+            np_array_ops.ravel(arg, *args, **kwargs),
+            np.ravel(arg, *args, **kwargs))
+
+    run_test(5)
+    run_test(5.)
+    run_test([])
+    run_test([[]])
+    run_test([[], []])
+    run_test([1, 2, 3])
+    run_test([1., 2., 3.])
+    run_test([[1, 2], [3, 4]])
+    run_test(np.arange(8).reshape((2, 2, 2)).tolist())
+
+  def testReal(self):
+
+    def run_test(arr, *args, **kwargs):
+      for fn in self.array_transforms:
+        arg = fn(arr)
+        self.match(
+            np_array_ops.real(arg, *args, **kwargs),
+            np.array(np.real(arg, *args, **kwargs)))
+
+    run_test(1)
+    run_test(5.5)
+    run_test(5 + 3j)
+    run_test(3j)
+    run_test([])
+    run_test([1, 2, 3])
+    run_test([1 + 5j, 2 + 3j])
+    run_test([[1 + 5j, 2 + 3j], [1 + 7j, 2 + 8j]])
+
+  def testRepeat(self):
+
+    def run_test(arr, repeats, *args, **kwargs):
+      for fn1 in self.array_transforms:
+        for fn2 in self.array_transforms:
+          arr_arg = fn1(arr)
+          repeats_arg = fn2(repeats)
+          self.match(
+              np_array_ops.repeat(arr_arg, repeats_arg, *args, **kwargs),
+              np.repeat(arr_arg, repeats_arg, *args, **kwargs))
+
+    run_test(1, 2)
+    run_test([1, 2], 2)
+    run_test([1, 2], [2])
+    run_test([1, 2], [1, 2])
+    run_test([[1, 2], [3, 4]], 3, axis=0)
+    run_test([[1, 2], [3, 4]], 3, axis=1)
+    run_test([[1, 2], [3, 4]], [3], axis=0)
+    run_test([[1, 2], [3, 4]], [3], axis=1)
+    run_test([[1, 2], [3, 4]], [3, 2], axis=0)
+    run_test([[1, 2], [3, 4]], [3, 2], axis=1)
+    run_test([[1, 2], [3, 4]], [3, 2], axis=-1)
+    run_test([[1, 2], [3, 4]], [3, 2], axis=-2)
+
+  def testAround(self):
+
+    def run_test(arr, *args, **kwargs):
+      for fn in self.array_transforms:
+        arg = fn(arr)
+        self.match(
+            np_array_ops.around(arg, *args, **kwargs),
+            np.around(arg, *args, **kwargs))
+
+    run_test(5.5)
+    run_test(5.567, decimals=2)
+    run_test([])
+    run_test([1.27, 2.49, 2.75], decimals=1)
+    run_test([23.6, 45.1], decimals=-1)
+
+  def testReshape(self):
+
+    def run_test(arr, newshape, *args, **kwargs):
+      for fn1 in self.array_transforms:
+        for fn2 in self.array_transforms:
+          arr_arg = fn1(arr)
+          newshape_arg = fn2(newshape)
+          self.match(
+              np_array_ops.reshape(arr_arg, newshape_arg, *args, **kwargs),
+              np.reshape(arr_arg, newshape, *args, **kwargs))
+
+    run_test(5, [-1])
+    run_test([], [-1])
+    run_test([1, 2, 3], [1, 3])
+    run_test([1, 2, 3], [3, 1])
+    run_test([1, 2, 3, 4], [2, 2])
+    run_test([1, 2, 3, 4], [2, 1, 2])
+
+  def testExpandDims(self):
+
+    def run_test(arr, axis):
+      self.match(np_array_ops.expand_dims(arr, axis), np.expand_dims(arr, axis))
+
+    run_test([1, 2, 3], 0)
+    run_test([1, 2, 3], 1)
+
+  def testSqueeze(self):
+
+    def run_test(arr, *args, **kwargs):
+      for fn in self.array_transforms:
+        arg = fn(arr)
+        # Note: np.squeeze ignores the axis arg for non-ndarray objects.
+        # This looks like a bug: https://github.com/numpy/numpy/issues/8201
+        # So we convert the arg to np.ndarray before passing to np.squeeze.
+        self.match(
+            np_array_ops.squeeze(arg, *args, **kwargs),
+            np.squeeze(np.array(arg), *args, **kwargs))
+
+    run_test(5)
+    run_test([])
+    run_test([5])
+    run_test([[1, 2, 3]])
+    run_test([[[1], [2], [3]]])
+    run_test([[[1], [2], [3]]], axis=0)
+    run_test([[[1], [2], [3]]], axis=2)
+    run_test([[[1], [2], [3]]], axis=(0, 2))
+    run_test([[[1], [2], [3]]], axis=-1)
+    run_test([[[1], [2], [3]]], axis=-3)
+
+  def testTranspose(self):
+
+    def run_test(arr, axes=None):
+      for fn1 in self.array_transforms:
+        for fn2 in self.array_transforms:
+          arr_arg = fn1(arr)
+          axes_arg = fn2(axes) if axes is not None else None
+          self.match(
+              np_array_ops.transpose(arr_arg, axes_arg),
+              np.transpose(arr_arg, axes))
+
+    run_test(5)
+    run_test([])
+    run_test([5])
+    run_test([5, 6, 7])
+    run_test(np.arange(30).reshape(2, 3, 5).tolist())
+    run_test(np.arange(30).reshape(2, 3, 5).tolist(), [0, 1, 2])
+    run_test(np.arange(30).reshape(2, 3, 5).tolist(), [0, 2, 1])
+    run_test(np.arange(30).reshape(2, 3, 5).tolist(), [1, 0, 2])
+    run_test(np.arange(30).reshape(2, 3, 5).tolist(), [1, 2, 0])
+    run_test(np.arange(30).reshape(2, 3, 5).tolist(), [2, 0, 1])
+    run_test(np.arange(30).reshape(2, 3, 5).tolist(), [2, 1, 0])
+
+  def testSetItem(self):
+
+    def run_test(arr, index, value):
+      for fn in self.array_transforms:
+        value_arg = fn(value)
+        tf_array = np_array_ops.array(arr)
+        np_array = np.array(arr)
+        tf_array[index] = value_arg
+        # TODO(srbs): "setting an array element with a sequence" is thrown
+        # if we do not wrap value_arg in a numpy array. Investigate how this can
+        # be avoided.
+        np_array[index] = np.array(value_arg)
+        self.match(tf_array, np_array)
+
+    run_test([1, 2, 3], 1, 5)
+    run_test([[1, 2], [3, 4]], 0, [6, 7])
+    run_test([[1, 2], [3, 4]], 1, [6, 7])
+    run_test([[1, 2], [3, 4]], (0, 1), 6)
+    run_test([[1, 2], [3, 4]], 0, 6)  # Value needs to broadcast.
+
+  def match_shape(self, actual, expected, msg=None):
+    if msg:
+      msg = 'Shape match failed for: {}. Expected: {} Actual: {}'.format(
+          msg, expected.shape, actual.shape)
+    self.assertEqual(actual.shape, expected.shape, msg=msg)
+    if msg:
+      msg = 'Shape: {} is not a tuple for {}'.format(actual.shape, msg)
+    self.assertIsInstance(actual.shape, tuple, msg=msg)
+
+  def match_dtype(self, actual, expected, msg=None):
+    if msg:
+      msg = 'Dtype match failed for: {}. Expected: {} Actual: {}.'.format(
+          msg, expected.dtype, actual.dtype)
+    self.assertEqual(actual.dtype, expected.dtype, msg=msg)
+
+  def match(self, actual, expected, msg=None, check_dtype=True):
+    msg_ = 'Expected: {} Actual: {}'.format(expected, actual)
+    if msg:
+      msg = '{} {}'.format(msg_, msg)
+    else:
+      msg = msg_
+    self.assertIsInstance(actual, np_arrays.ndarray)
+    if check_dtype:
+      self.match_dtype(actual, expected, msg)
+    self.match_shape(actual, expected, msg)
+    if not actual.shape:
+      self.assertAllClose(actual.tolist(), expected.tolist())
+    else:
+      self.assertAllClose(actual.tolist(), expected.tolist())
+
+  def testPad(self):
+    t = [[1, 2, 3], [4, 5, 6]]
+    paddings = [[
+        1,
+        1,
+    ], [2, 2]]
+    self.assertAllEqual(
+        np_array_ops.pad(t, paddings, 'constant'),
+        [[0, 0, 0, 0, 0, 0, 0], [0, 0, 1, 2, 3, 0, 0], [0, 0, 4, 5, 6, 0, 0],
+         [0, 0, 0, 0, 0, 0, 0]])
+
+    self.assertAllEqual(
+        np_array_ops.pad(t, paddings, 'reflect'),
+        [[6, 5, 4, 5, 6, 5, 4], [3, 2, 1, 2, 3, 2, 1], [6, 5, 4, 5, 6, 5, 4],
+         [3, 2, 1, 2, 3, 2, 1]])
+
+    self.assertAllEqual(
+        np_array_ops.pad(t, paddings, 'symmetric'),
+        [[2, 1, 1, 2, 3, 3, 2], [2, 1, 1, 2, 3, 3, 2], [5, 4, 4, 5, 6, 6, 5],
+         [5, 4, 4, 5, 6, 6, 5]])
+
+  def testTake(self):
+    a = [4, 3, 5, 7, 6, 8]
+    indices = [0, 1, 4]
+    self.assertAllEqual([4, 3, 6], np_array_ops.take(a, indices))
+    indices = [[0, 1], [2, 3]]
+    self.assertAllEqual([[4, 3], [5, 7]], np_array_ops.take(a, indices))
+    a = [[4, 3, 5], [7, 6, 8]]
+    self.assertAllEqual([[4, 3], [5, 7]], np_array_ops.take(a, indices))
+    a = np.random.rand(2, 16, 3)
+    axis = 1
+    self.assertAllEqual(
+        np.take(a, indices, axis=axis),
+        np_array_ops.take(a, indices, axis=axis))
+
+  def testWhere(self):
+    self.assertAllEqual([[1.0, 1.0], [1.0, 1.0]],
+                        np_array_ops.where([True], [1.0, 1.0],
+                                           [[0, 0], [0, 0]]))
+
+  def testShape(self):
+    self.assertAllEqual((1, 2), np_array_ops.shape([[0, 0]]))
+
+  def testSwapaxes(self):
+    x = [[1, 2, 3]]
+    self.assertAllEqual([[1], [2], [3]], np_array_ops.swapaxes(x, 0, 1))
+    self.assertAllEqual([[1], [2], [3]], np_array_ops.swapaxes(x, -2, -1))
+    x = [[[0, 1], [2, 3]], [[4, 5], [6, 7]]]
+    self.assertAllEqual([[[0, 4], [2, 6]], [[1, 5], [3, 7]]],
+                        np_array_ops.swapaxes(x, 0, 2))
+    self.assertAllEqual([[[0, 4], [2, 6]], [[1, 5], [3, 7]]],
+                        np_array_ops.swapaxes(x, -3, -1))
+
+  def testMoveaxis(self):
+
+    def _test(*args):
+      expected = np.moveaxis(*args)
+      raw_ans = np_array_ops.moveaxis(*args)
+
+      self.assertAllEqual(expected, raw_ans)
+
+    a = np.random.rand(1, 2, 3, 4, 5, 6)
+
+    # Basic
+    _test(a, (0, 2), (3, 5))
+    _test(a, (0, 2), (-1, -3))
+    _test(a, (-6, -4), (3, 5))
+    _test(a, (-6, -4), (-1, -3))
+    _test(a, 0, 4)
+    _test(a, -6, -2)
+    _test(a, tuple(range(6)), tuple(range(6)))
+    _test(a, tuple(range(6)), tuple(reversed(range(6))))
+    _test(a, (), ())
+
+  def testNdim(self):
+    self.assertAllEqual(0, np_array_ops.ndim(0.5))
+    self.assertAllEqual(1, np_array_ops.ndim([1, 2]))
+
+  def testIsscalar(self):
+    self.assertTrue(np_array_ops.isscalar(0.5))
+    self.assertTrue(np_array_ops.isscalar(5))
+    self.assertTrue(np_array_ops.isscalar(False))
+    self.assertFalse(np_array_ops.isscalar([1, 2]))
+
+  def assertListEqual(self, a, b):
+    self.assertAllEqual(len(a), len(b))
+    for x, y in zip(a, b):
+      self.assertAllEqual(x, y)
+
+  def testSplit(self):
+    x = np_array_ops.arange(9)
+    y = np_array_ops.split(x, 3)
+    self.assertListEqual([([0, 1, 2]), ([3, 4, 5]), ([6, 7, 8])], y)
+
+    x = np_array_ops.arange(8)
+    y = np_array_ops.split(x, [3, 5, 6, 10])
+    self.assertListEqual([([0, 1, 2]), ([3, 4]), ([5]), ([6, 7]), ([])], y)
+
+
+class ArrayManipulationTest(test.TestCase):
+
+  def setUp(self):
+    super(ArrayManipulationTest, self).setUp()
+    self.array_transforms = [
+        lambda x: x,
+        ops.convert_to_tensor,
+        np.array,
+        np_array_ops.array,
+    ]
+
+  def testBroadcastTo(self):
+
+    def run_test(arr, shape):
+      for fn in self.array_transforms:
+        arg1 = fn(arr)
+        self.match(
+            np_array_ops.broadcast_to(arg1, shape),
+            np.broadcast_to(arg1, shape))
+
+    run_test(1, 2)
+    run_test(1, (2, 2))
+    run_test([1, 2], (2, 2))
+    run_test([[1], [2]], (2, 2))
+    run_test([[1, 2]], (3, 2))
+    run_test([[[1, 2]], [[3, 4]], [[5, 6]]], (3, 4, 2))
+
+  def testIx_(self):
+    possible_arys = [[True, True], [True, False], [False, False],
+                     list(range(5)), np_array_ops.empty(0, dtype=np.int64)]
+    for r in range(len(possible_arys)):
+      for arys in itertools.combinations_with_replacement(possible_arys, r):
+        tnp_ans = np_array_ops.ix_(*arys)
+        onp_ans = np.ix_(*arys)
+        for t, o in zip(tnp_ans, onp_ans):
+          self.match(t, o)
+
+  def match_shape(self, actual, expected, msg=None):
+    if msg:
+      msg = 'Shape match failed for: {}. Expected: {} Actual: {}'.format(
+          msg, expected.shape, actual.shape)
+    self.assertEqual(actual.shape, expected.shape, msg=msg)
+    if msg:
+      msg = 'Shape: {} is not a tuple for {}'.format(actual.shape, msg)
+    self.assertIsInstance(actual.shape, tuple, msg=msg)
+
+  def match_dtype(self, actual, expected, msg=None):
+    if msg:
+      msg = 'Dtype match failed for: {}. Expected: {} Actual: {}.'.format(
+          msg, expected.dtype, actual.dtype)
+    self.assertEqual(actual.dtype, expected.dtype, msg=msg)
+
+  def match(self, actual, expected, msg=None):
+    msg_ = 'Expected: {} Actual: {}'.format(expected, actual)
+    if msg:
+      msg = '{} {}'.format(msg_, msg)
+    else:
+      msg = msg_
+    self.assertIsInstance(actual, np_arrays.ndarray)
+    self.match_dtype(actual, expected, msg)
+    self.match_shape(actual, expected, msg)
+    if not actual.shape:
+      self.assertEqual(actual.tolist(), expected.tolist())
+    else:
+      self.assertSequenceEqual(actual.tolist(), expected.tolist())
+
+
+if __name__ == '__main__':
+  ops.enable_eager_execution()
+  test.main()
diff --git a/tensorflow/python/ops/numpy_ops/np_arrays.py b/tensorflow/python/ops/numpy_ops/np_arrays.py
new file mode 100644
index 00000000000..bd58e486f50
--- /dev/null
+++ b/tensorflow/python/ops/numpy_ops/np_arrays.py
@@ -0,0 +1,292 @@
+# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""ndarray class."""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import numpy as np
+import six
+
+from tensorflow.python.framework import dtypes
+from tensorflow.python.framework import ops
+from tensorflow.python.ops import array_ops
+from tensorflow.python.ops import math_ops
+from tensorflow.python.ops.numpy_ops import np_dtypes
+
+
+def convert_to_tensor(value, dtype=None):
+  # A safer version of `tf.convert_to_tensor` to work around b/149876037.
+  # TODO(wangpeng): Remove this function once the bug is fixed.
+  if (dtype is None and isinstance(value, six.integer_types) and
+      value >= 2**63):
+    dtype = dtypes.uint64
+  elif (dtype is None and isinstance(value, float)):
+    dtype = np_dtypes.default_float_type()
+  return ops.convert_to_tensor(value, dtype=dtype)
+
+
+class ndarray(object):  # pylint: disable=invalid-name
+  """Equivalent of numpy.ndarray backed by TensorFlow tensors.
+
+  This does not support all features of NumPy ndarrays e.g. strides and
+  memory order since, unlike NumPy, the backing storage is not a raw memory
+  buffer.
+
+  TODO(srbs): Clearly specify which attributes and methods are not supported
+  or if there are any differences in behavior.
+  """
+
+  def __init__(self, shape, dtype=float, buffer=None):  # pylint: disable=redefined-builtin
+    """Initializes an ndarray.
+
+    This is a low level interface for building ndarrays and should be avoided.
+    Users should instead use methods in array_creation.py.
+
+    This class provides a numpy.ndarray like interface for a TF Tensor with a
+    fully-defined shape. Note that, unlike the backing buffer of np.ndarray,
+    Tensors are immutable. So, operations like `__setitem__` are performed by
+    replacing the Tensor. This restricts the ability to implement NumPy `view`
+    semantics.
+
+    Compared to numpy.ndarray, this does not support `offset`, `strides`
+    and `order` arguments.
+
+    Args:
+      shape: The shape of the array. Must be a scalar, an iterable of integers
+        or a `TensorShape` object.
+      dtype: Optional. The dtype of the array. Must be a python type, a numpy
+        type or a tensorflow `DType` object.
+      buffer: Optional. The backing buffer of the array. Must have shape
+        `shape`. Must be a `ndarray`, `np.ndarray` or a `Tensor`.
+
+    Raises:
+      ValueError: If `buffer` is specified and its shape does not match
+       `shape`.
+    """
+    if dtype and not isinstance(dtype, dtypes.DType):
+      dtype = dtypes.as_dtype(np.dtype(dtype))
+    if buffer is None:
+      buffer = array_ops.zeros(shape, dtype=dtype)
+    else:
+      if isinstance(buffer, ndarray):
+        buffer = buffer.data
+      elif isinstance(buffer, np.ndarray):
+        # If `buffer` is a np.ndarray, the Tensor will share the underlying
+        # storage of the array.
+        buffer = convert_to_tensor(value=buffer, dtype=dtype)
+      elif not isinstance(buffer, ops.Tensor):
+        raise ValueError('Unexpected type for `buffer` {}. Must be an ndarray,'
+                         ' Tensor or np.ndarray.'.format(type(buffer)))
+
+      if shape is not None and tuple(shape) != buffer._shape_tuple():  # pylint: disable=protected-access
+        # TODO(srbs): NumPy allows this. Investigate if/how to support this.
+        raise ValueError('shape arg must match buffer.shape.')
+
+    assert isinstance(buffer, ops.Tensor)
+    if dtype and dtype != buffer.dtype:
+      buffer = array_ops.bitcast(buffer, dtype)
+    self._data = buffer
+    self.base = None
+
+  @property
+  def data(self):
+    """Tensor object containing the array data.
+
+    This has a few key differences from the Python buffer object used in
+    NumPy arrays.
+    1. Tensors are immutable. So operations requiring in-place edit, e.g.
+       __setitem__, are performed by replacing the underlying buffer with a new
+       one.
+    2. Tensors do not provide access to their raw buffer.
+
+    Returns:
+      A Tensor.
+    """
+    return self._data
+
+  @property
+  def shape(self):
+    """Returns a tuple of array dimensions."""
+    return self.data._shape_tuple()  # pylint: disable=protected-access
+
+  @property
+  def dtype(self):
+    return np.dtype(self.data.dtype.as_numpy_dtype)
+
+  @property
+  def ndim(self):
+    return self.data.shape.ndims
+
+  @property
+  def size(self):
+    """Returns the number of elements in the array."""
+    return np.prod(self.shape)
+
+  @property
+  def T(self):  # pylint: disable=invalid-name
+    return self.transpose()
+
+  def __len__(self):
+    if self.shape:
+      return self.shape[0]
+    else:
+      raise TypeError('len() of unsized object.')
+
+  def astype(self, dtype):
+    if self.dtype == dtype:
+      return self
+    else:
+      return tensor_to_ndarray(math_ops.cast(self.data, dtype))
+
+  # Unary operations
+  def __neg__(self):
+    return tensor_to_ndarray(-self.data)  # pylint: disable=invalid-unary-operand-type
+
+  def __pos__(self):
+    return self
+
+  __hash__ = None
+
+  def __int__(self):
+    return int(self.data)
+
+  def __float__(self):
+    return float(self.data)
+
+  def __nonzero__(self):
+    return bool(self.data)
+
+  def __bool__(self):
+    return self.__nonzero__()
+
+  def __getitem__(self, slice_spec):
+    # TODO(srbs): Need to support better indexing.
+    result_t = self.data.__getitem__(slice_spec)
+    return tensor_to_ndarray(result_t)
+
+  def __iter__(self):
+    for i in range(self.shape[0]):
+      result_t = self.data[i]
+      yield tensor_to_ndarray(result_t)
+    return
+
+  def __array__(self, dtype=None):
+    """Returns a NumPy ndarray.
+
+    This allows instances of this class to be directly used in NumPy routines.
+    However, doing that may force a copy to CPU.
+
+    Args:
+      dtype: A NumPy compatible type.
+
+    Returns:
+      A NumPy ndarray.
+    """
+    return np.asarray(self.data, dtype)
+
+  __array_priority__ = 110
+
+  def __index__(self):
+    """Returns a python scalar.
+
+    This allows using an instance of this class as an array index.
+    Note that only arrays of integer types with size 1 can be used as array
+    indices.
+
+    Returns:
+      A Python scalar.
+
+    Raises:
+      TypeError: If the array is not of an integer type.
+      ValueError: If the array does not have size 1.
+    """
+    # TODO(wangpeng): Handle graph mode
+    return np.asscalar(self.data.numpy())
+
+  def tolist(self):
+    return self.data.numpy().tolist()
+
+  def __str__(self):
+    return 'ndarray<{}>'.format(self.data.__str__())
+
+  def __repr__(self):
+    return 'ndarray<{}>'.format(self.data.__repr__())
+
+
+def tensor_to_ndarray(tensor):
+  return ndarray(tensor._shape_tuple(), dtype=tensor.dtype, buffer=tensor)  # pylint: disable=protected-access
+
+
+def ndarray_to_tensor(arr, dtype=None, name=None, as_ref=False):
+  if as_ref:
+    raise ValueError('as_ref is not supported.')
+  if dtype and dtypes.as_dtype(arr.dtype) != dtype:
+    return math_ops.cast(arr.data, dtype)
+  result_t = arr.data
+  if name:
+    result_t = array_ops.identity(result_t, name=name)
+  return result_t
+
+
+ops.register_tensor_conversion_function(ndarray, ndarray_to_tensor)
+
+
+# Don't use a namedtuple since nest considers that a tuple and unflattens and
+# flattens it.
+class ShardedNdArray(object):
+  """Wrapper over ndarray that can contain tensors on multiple devices.
+
+    This is returned by extensions.pmap, and contains the individual tensors on
+    different devices.
+  """
+
+  def __init__(self, tensors):
+    """Initializes the ShardedNdArray.
+
+    Note that the tensors should be ordered in the way the pmap producing these
+    tensors is run.
+
+    Args:
+      tensors: list or tuple of eager tensors, one for each device.
+    """
+
+    if not isinstance(tensors, (list, tuple)) or not tensors:
+      raise ValueError(
+          'Unable to create a ShardedNdArray without a list of tensors.')
+    self.tensors = tensors
+    self.n_devices = len(tensors)
+
+  def __getitem__(self, i):
+    return self.tensors[i]
+
+  @property
+  def shape(self):
+    return (self.n_devices,) + self.tensors[0]._shape_tuple()  # pylint: disable=protected-access
+
+  @property
+  def dtype(self):
+    return self.tensors[0].dtype
+
+
+def convert_sharded_tensor_to_eager_tensor(value, *args, **kwargs):
+  del args, kwargs
+  # TODO(nareshmodi): Consider a collective op to gather the tensors from the
+  # various devices for performance reasons.
+  return array_ops.stack(value.tensors)
+
+
+ops.register_tensor_conversion_function(ShardedNdArray,
+                                        convert_sharded_tensor_to_eager_tensor)
diff --git a/tensorflow/python/ops/numpy_ops/np_arrays_test.py b/tensorflow/python/ops/numpy_ops/np_arrays_test.py
new file mode 100644
index 00000000000..feced98438d
--- /dev/null
+++ b/tensorflow/python/ops/numpy_ops/np_arrays_test.py
@@ -0,0 +1,189 @@
+# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Tests for ndarray."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import collections
+
+import numpy as np
+
+from tensorflow.python.framework import dtypes
+from tensorflow.python.framework import ops
+from tensorflow.python.ops import array_ops
+from tensorflow.python.ops.numpy_ops import np_arrays
+# Required for operator overloads
+from tensorflow.python.ops.numpy_ops import np_math_ops  # pylint: disable=unused-import
+from tensorflow.python.platform import test
+
+t2a = np_arrays.tensor_to_ndarray
+
+
+class ArrayTest(test.TestCase):
+
+  def testDtype(self):
+    a = t2a(array_ops.zeros(shape=[1, 2], dtype=dtypes.int64))
+    self.assertIs(a.dtype.type, np.int64)
+    self.assertAllEqual(0, a.dtype.type(0))
+
+  def testAstype(self):
+    a = t2a(ops.convert_to_tensor(value=1.1,
+                                  dtype=dtypes.float32)).astype(np.int32)
+    self.assertIs(a.dtype.type, np.int32)
+    self.assertAllEqual(1, a)
+    a = t2a(ops.convert_to_tensor(value=[0.0, 1.1],
+                                  dtype=dtypes.float32)).astype(np.bool_)
+    self.assertIs(a.dtype.type, np.bool_)
+    self.assertAllEqual([False, True], a)
+
+  def testNeg(self):
+    a = t2a(ops.convert_to_tensor(value=[1.0, 2.0]))
+    self.assertAllEqual([-1.0, -2.0], -a)
+
+  def _testBinOp(self, a, b, out, f, types=None):
+    a = t2a(ops.convert_to_tensor(value=a, dtype=np.int32))
+    b = t2a(ops.convert_to_tensor(value=b, dtype=np.int32))
+    if not isinstance(out, np_arrays.ndarray):
+      out = t2a(ops.convert_to_tensor(value=out, dtype=np.int32))
+    if types is None:
+      types = [[np.int32, np.int32, np.int32], [np.int64, np.int32, np.int64],
+               [np.int32, np.int64, np.int64],
+               [np.float32, np.int32, np.float64],
+               [np.int32, np.float32, np.float64],
+               [np.float32, np.float32, np.float32],
+               [np.float64, np.float32, np.float64],
+               [np.float32, np.float64, np.float64]]
+    for a_type, b_type, out_type in types:
+      o = f(a.astype(a_type), b.astype(b_type))
+      self.assertIs(o.dtype.type, out_type)
+      out = out.astype(out_type)
+      if np.issubdtype(out_type, np.inexact):
+        self.assertAllClose(out, o)
+      else:
+        self.assertAllEqual(out, o)
+
+  def testAdd(self):
+    self._testBinOp([1, 2], [3, 4], [4, 6], lambda a, b: a.__add__(b))
+
+  def testRadd(self):
+    self._testBinOp([1, 2], [3, 4], [4, 6], lambda a, b: b.__radd__(a))
+
+  def testSub(self):
+    self._testBinOp([1, 2], [3, 5], [-2, -3], lambda a, b: a.__sub__(b))
+
+  def testRsub(self):
+    self._testBinOp([1, 2], [3, 5], [-2, -3], lambda a, b: b.__rsub__(a))
+
+  def testMul(self):
+    self._testBinOp([1, 2], [3, 4], [3, 8], lambda a, b: a.__mul__(b))
+
+  def testRmul(self):
+    self._testBinOp([1, 2], [3, 4], [3, 8], lambda a, b: b.__rmul__(a))
+
+  def testPow(self):
+    self._testBinOp([4, 5], [3, 2], [64, 25], lambda a, b: a.__pow__(b))
+
+  def testRpow(self):
+    self._testBinOp([4, 5], [3, 2], [64, 25], lambda a, b: b.__rpow__(a))
+
+  _truediv_types = [[np.int32, np.int32, np.float64],
+                    [np.int64, np.int32, np.float64],
+                    [np.int32, np.int64, np.float64],
+                    [np.float32, np.int32, np.float64],
+                    [np.int32, np.float32, np.float64],
+                    [np.float32, np.float32, np.float32],
+                    [np.float64, np.float32, np.float64],
+                    [np.float32, np.float64, np.float64]]
+
+  def testTruediv(self):
+    self._testBinOp([3, 5], [2, 4],
+                    t2a(ops.convert_to_tensor(value=[1.5, 1.25])),
+                    lambda a, b: a.__truediv__(b),
+                    types=self._truediv_types)
+
+  def testRtruediv(self):
+    self._testBinOp([3, 5], [2, 4],
+                    t2a(ops.convert_to_tensor(value=[1.5, 1.25])),
+                    lambda a, b: b.__rtruediv__(a),
+                    types=self._truediv_types)
+
+  def _testCmp(self, a, b, out, f):
+    a = t2a(ops.convert_to_tensor(value=a, dtype=np.int32))
+    b = t2a(ops.convert_to_tensor(value=b, dtype=np.int32))
+    types = [[np.int32, np.int32], [np.int64, np.int32], [np.int32, np.int64],
+             [np.float32, np.int32], [np.int32, np.float32],
+             [np.float32, np.float32], [np.float64, np.float32],
+             [np.float32, np.float64]]
+    for a_type, b_type in types:
+      o = f(a.astype(a_type), b.astype(b_type))
+      self.assertAllEqual(out, o)
+
+  def testLt(self):
+    self._testCmp([1, 2, 3], [3, 2, 1], [True, False, False],
+                  lambda a, b: a.__lt__(b))
+
+  def testLe(self):
+    self._testCmp([1, 2, 3], [3, 2, 1], [True, True, False],
+                  lambda a, b: a.__le__(b))
+
+  def testGt(self):
+    self._testCmp([1, 2, 3], [3, 2, 1], [False, False, True],
+                  lambda a, b: a.__gt__(b))
+
+  def testGe(self):
+    self._testCmp([1, 2, 3], [3, 2, 1], [False, True, True],
+                  lambda a, b: a.__ge__(b))
+
+  def testEq(self):
+    self._testCmp([1, 2, 3], [3, 2, 1], [False, True, False],
+                  lambda a, b: a.__eq__(b))
+
+  def testNe(self):
+    self._testCmp([1, 2, 3], [3, 2, 1], [True, False, True],
+                  lambda a, b: a.__ne__(b))
+
+  def testInt(self):
+    v = 10
+    u = int(t2a(ops.convert_to_tensor(value=v)))
+    self.assertIsInstance(u, int)
+    self.assertAllEqual(v, u)
+
+  def testFloat(self):
+    v = 21.32
+    u = float(t2a(ops.convert_to_tensor(value=v)))
+    self.assertIsInstance(u, float)
+    self.assertAllClose(v, u)
+
+  def testBool(self):
+    b = bool(t2a(ops.convert_to_tensor(value=10)))
+    self.assertIsInstance(b, bool)
+    self.assertTrue(b)
+    self.assertFalse(bool(t2a(ops.convert_to_tensor(value=0))))
+    self.assertTrue(bool(t2a(ops.convert_to_tensor(value=0.1))))
+    self.assertFalse(bool(t2a(ops.convert_to_tensor(value=0.0))))
+
+  def testHash(self):
+    a = t2a(ops.convert_to_tensor(value=10))
+    self.assertNotIsInstance(a, collections.Hashable)
+    with self.assertRaisesWithPredicateMatch(TypeError, r'unhashable type'):
+      hash(a)
+
+
+if __name__ == '__main__':
+  # TODO(wangpeng): Test in graph mode as well.
+  ops.enable_eager_execution()
+  test.main()
diff --git a/tensorflow/python/ops/numpy_ops/np_backprop_test.py b/tensorflow/python/ops/numpy_ops/np_backprop_test.py
new file mode 100644
index 00000000000..65c532153a4
--- /dev/null
+++ b/tensorflow/python/ops/numpy_ops/np_backprop_test.py
@@ -0,0 +1,69 @@
+# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Tests for backpropgration on tf-numpy functions."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from tensorflow.python.eager import backprop
+from tensorflow.python.framework import ops
+from tensorflow.python.ops.numpy_ops import np_array_ops
+# Required for operator overloads
+from tensorflow.python.ops.numpy_ops import np_math_ops  # pylint: disable=unused-import
+from tensorflow.python.platform import test
+
+
+class BackpropTest(test.TestCase):
+
+  def test_setitem(self):
+    # Single integer index.
+    a = np_array_ops.array([1., 2., 3.])
+    b = np_array_ops.array(5.)
+    c = np_array_ops.array(10.)
+
+    tensors = [arr.data for arr in [a, b, c]]
+    with backprop.GradientTape() as g:
+      g.watch(tensors)
+      a[1] = b + c
+      loss = np_array_ops.sum(a)
+
+    gradients = g.gradient(loss.data, tensors)
+    self.assertSequenceEqual(
+        np_array_ops.array(gradients[0]).tolist(), [1., 0., 1.])
+    self.assertEqual(np_array_ops.array(gradients[1]).tolist(), 1.)
+    self.assertEqual(np_array_ops.array(gradients[2]).tolist(), 1.)
+
+    # Tuple index.
+    a = np_array_ops.array([[[1., 2.], [3., 4.]], [[5., 6.],
+                                                   [7., 8.]]])  # 2x2x2 array.
+    b = np_array_ops.array([10., 11.])
+
+    tensors = [arr.data for arr in [a, b]]
+    with backprop.GradientTape() as g:
+      g.watch(tensors)
+      a[(1, 0)] = b
+      loss = np_array_ops.sum(a)
+
+    gradients = g.gradient(loss.data, tensors)
+    self.assertSequenceEqual(
+        np_array_ops.array(gradients[0]).tolist(),
+        [[[1., 1.], [1., 1.]], [[0., 0.], [1., 1.]]])
+    self.assertEqual(np_array_ops.array(gradients[1]).tolist(), [1., 1.])
+
+
+if __name__ == '__main__':
+  ops.enable_eager_execution()
+  test.main()
diff --git a/tensorflow/python/ops/numpy_ops/np_dtypes.py b/tensorflow/python/ops/numpy_ops/np_dtypes.py
new file mode 100644
index 00000000000..cefd09f800d
--- /dev/null
+++ b/tensorflow/python/ops/numpy_ops/np_dtypes.py
@@ -0,0 +1,96 @@
+# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Dtypes and dtype utilities."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import numpy as np
+
+# We use numpy's dtypes instead of TF's, because the user expects to use them
+# with numpy facilities such as `np.dtype(np.int64)` and
+# `if x.dtype.type is np.int64`.
+# pylint: disable=unused-import
+# pylint: disable=g-bad-import-order
+from numpy import bool_
+from numpy import int_
+from numpy import int16
+from numpy import int32
+from numpy import int64
+from numpy import int8
+from numpy import uint16
+from numpy import uint32
+from numpy import uint64
+from numpy import uint8
+from numpy import float_
+from numpy import float16
+from numpy import float32
+from numpy import float64
+from numpy import complex_
+from numpy import complex64
+from numpy import complex128
+
+from numpy import inexact
+
+from numpy import iinfo
+from numpy import issubdtype
+
+from numpy import inf
+
+# TODO(wangpeng): Make bfloat16 a numpy dtype instead of using TF's
+from tensorflow.python.framework.dtypes import bfloat16
+# pylint: enable=g-bad-import-order
+# pylint: enable=unused-import
+
+_to_float32 = {
+    np.dtype('float64'): np.dtype('float32'),
+    np.dtype('complex128'): np.dtype('complex64'),
+}
+
+_allow_float64 = True
+
+
+def is_allow_float64():
+  return _allow_float64
+
+
+def set_allow_float64(b):
+  global _allow_float64
+  _allow_float64 = b
+
+
+def canonicalize_dtype(dtype):
+  if not is_allow_float64():
+    return _to_float32.get(dtype, dtype)
+  else:
+    return dtype
+
+
+def _result_type(*arrays_and_dtypes):
+  dtype = np.result_type(*arrays_and_dtypes)
+  return canonicalize_dtype(dtype)
+
+
+def default_float_type():
+  """Gets the default float type.
+
+  Returns:
+    If `is_allow_float64()` is true, returns float64; otherwise returns float32.
+  """
+  if is_allow_float64():
+    return float64
+  else:
+    return float32
diff --git a/tensorflow/python/ops/numpy_ops/np_logic_test.py b/tensorflow/python/ops/numpy_ops/np_logic_test.py
new file mode 100644
index 00000000000..85826873356
--- /dev/null
+++ b/tensorflow/python/ops/numpy_ops/np_logic_test.py
@@ -0,0 +1,110 @@
+# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Tests for tf numpy random number methods."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import numpy as np
+
+from tensorflow.python.framework import dtypes
+from tensorflow.python.framework import ops
+from tensorflow.python.ops.numpy_ops import np_array_ops
+from tensorflow.python.ops.numpy_ops import np_arrays
+from tensorflow.python.ops.numpy_ops import np_math_ops
+from tensorflow.python.platform import test
+
+
+class LogicTest(test.TestCase):
+
+  def setUp(self):
+    super(LogicTest, self).setUp()
+    self.array_transforms = [
+        lambda x: x,  # Identity,
+        ops.convert_to_tensor,
+        np.array,
+        lambda x: np.array(x, dtype=np.int32),
+        lambda x: np.array(x, dtype=np.int64),
+        lambda x: np.array(x, dtype=np.float32),
+        lambda x: np.array(x, dtype=np.float64),
+        np_array_ops.array,
+        lambda x: np_array_ops.array(x, dtype=dtypes.int32),
+        lambda x: np_array_ops.array(x, dtype=dtypes.int64),
+        lambda x: np_array_ops.array(x, dtype=dtypes.float32),
+        lambda x: np_array_ops.array(x, dtype=dtypes.float64),
+    ]
+
+  def testEqual(self):
+
+    def run_test(x1, x2=None):
+      if x2 is None:
+        x2 = x1
+      for fn1 in self.array_transforms:
+        for fn2 in self.array_transforms:
+          arg1 = fn1(x1)
+          arg2 = fn2(x2)
+          self.match(
+              np_math_ops.equal(arg1, arg2),
+              np.equal(
+                  make_numpy_compatible(arg1), make_numpy_compatible(arg2)))
+
+    run_test(1)
+    run_test(1, 2)
+    run_test([1, 2])
+    run_test([1, 2, 3], [2])
+    run_test([[1, 2], [3, 4]], [1, 2])
+    run_test([[1, 2], [1, 4]], [1, 2])
+    run_test([1, 2], [[1, 2], [1, 4]])
+    run_test([[1, 2], [3, 4]], [[1, 2], [3, 4]])
+    run_test([[1, 2], [3, 4]], [[1, 3], [3, 4]])
+
+  def match_shape(self, actual, expected, msg=None):
+    if msg:
+      msg = 'Shape match failed for: {}. Expected: {} Actual: {}'.format(
+          msg, expected.shape, actual.shape)
+    self.assertEqual(actual.shape, expected.shape, msg=msg)
+    if msg:
+      msg = 'Shape: {} is not a tuple for {}'.format(actual.shape, msg)
+    self.assertIsInstance(actual.shape, tuple, msg=msg)
+
+  def match_dtype(self, actual, expected, msg=None):
+    if msg:
+      msg = 'Dtype match failed for: {}. Expected: {} Actual: {}.'.format(
+          msg, expected.dtype, actual.dtype)
+    self.assertEqual(actual.dtype, expected.dtype, msg=msg)
+
+  def match(self, actual, expected, msg=None):
+    msg_ = 'Expected: {} Actual: {}'.format(expected, actual)
+    if msg:
+      msg = '{} {}'.format(msg_, msg)
+    else:
+      msg = msg_
+    self.assertIsInstance(actual, np_arrays.ndarray)
+    self.match_dtype(actual, expected, msg)
+    self.match_shape(actual, expected, msg)
+    if not actual.shape:
+      self.assertEqual(actual.tolist(), expected.tolist())
+    else:
+      self.assertSequenceEqual(actual.tolist(), expected.tolist())
+
+
+def make_numpy_compatible(s):
+  return s if not isinstance(s, np_arrays.ndarray) else s.data.numpy()
+
+
+if __name__ == '__main__':
+  ops.enable_eager_execution()
+  test.main()
diff --git a/tensorflow/python/ops/numpy_ops/np_math_ops.py b/tensorflow/python/ops/numpy_ops/np_math_ops.py
new file mode 100644
index 00000000000..dac60b63bfe
--- /dev/null
+++ b/tensorflow/python/ops/numpy_ops/np_math_ops.py
@@ -0,0 +1,1255 @@
+# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Mathematical operations."""
+# pylint: disable=g-direct-tensorflow-import
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import sys
+
+import numpy as np
+import six
+
+from tensorflow.python.framework import constant_op
+from tensorflow.python.framework import dtypes
+from tensorflow.python.framework import errors
+from tensorflow.python.framework import ops
+from tensorflow.python.ops import array_ops
+from tensorflow.python.ops import bitwise_ops
+from tensorflow.python.ops import clip_ops
+from tensorflow.python.ops import control_flow_ops
+from tensorflow.python.ops import math_ops
+from tensorflow.python.ops import sort_ops
+from tensorflow.python.ops.numpy_ops import np_array_ops
+from tensorflow.python.ops.numpy_ops import np_arrays
+from tensorflow.python.ops.numpy_ops import np_dtypes
+from tensorflow.python.ops.numpy_ops import np_utils
+
+
+@np_utils.np_doc_only(np.dot)
+def dot(a, b):  # pylint: disable=missing-docstring
+
+  def f(a, b):  # pylint: disable=missing-docstring
+    return np_utils.cond(
+        np_utils.logical_or(
+            math_ops.equal(array_ops.rank(a), 0),
+            math_ops.equal(array_ops.rank(b), 0)),
+        lambda: a * b,
+        lambda: np_utils.cond(  # pylint: disable=g-long-lambda
+            math_ops.equal(array_ops.rank(b), 1), lambda: math_ops.tensordot(
+                a, b, axes=[[-1], [-1]]), lambda: math_ops.tensordot(
+                    a, b, axes=[[-1], [-2]])))
+
+  return _bin_op(f, a, b)
+
+
+# TODO(wangpeng): Make element-wise ops `ufunc`s
+def _bin_op(tf_fun, a, b, promote=True):
+  if promote:
+    a, b = np_array_ops._promote_dtype(a, b)  # pylint: disable=protected-access
+  else:
+    a = np_array_ops.array(a)
+    b = np_array_ops.array(b)
+  return np_utils.tensor_to_ndarray(tf_fun(a.data, b.data))
+
+
+@np_utils.np_doc(np.add)
+def add(x1, x2):
+
+  def add_or_or(x1, x2):
+    if x1.dtype == dtypes.bool:
+      assert x2.dtype == dtypes.bool
+      return math_ops.logical_or(x1, x2)
+    return math_ops.add(x1, x2)
+
+  return _bin_op(add_or_or, x1, x2)
+
+
+@np_utils.np_doc(np.subtract)
+def subtract(x1, x2):
+  return _bin_op(math_ops.subtract, x1, x2)
+
+
+@np_utils.np_doc(np.multiply)
+def multiply(x1, x2):
+
+  def mul_or_and(x1, x2):
+    if x1.dtype == dtypes.bool:
+      assert x2.dtype == dtypes.bool
+      return math_ops.logical_and(x1, x2)
+    return math_ops.multiply(x1, x2)
+
+  return _bin_op(mul_or_and, x1, x2)
+
+
+@np_utils.np_doc(np.true_divide)
+def true_divide(x1, x2):  # pylint: disable=missing-function-docstring
+
+  def _avoid_float64(x1, x2):
+    if x1.dtype == x2.dtype and x1.dtype in (dtypes.int32, dtypes.int64):
+      x1 = math_ops.cast(x1, dtype=dtypes.float32)
+      x2 = math_ops.cast(x2, dtype=dtypes.float32)
+    return x1, x2
+
+  def f(x1, x2):
+    if x1.dtype == dtypes.bool:
+      assert x2.dtype == dtypes.bool
+      float_ = np_dtypes.default_float_type()
+      x1 = math_ops.cast(x1, float_)
+      x2 = math_ops.cast(x2, float_)
+    if not np_dtypes.is_allow_float64():
+      # math_ops.truediv in Python3 produces float64 when both inputs are int32
+      # or int64. We want to avoid that when is_allow_float64() is False.
+      x1, x2 = _avoid_float64(x1, x2)
+    return math_ops.truediv(x1, x2)
+
+  return _bin_op(f, x1, x2)
+
+
+divide = true_divide
+
+
+@np_utils.np_doc(np.floor_divide)
+def floor_divide(x1, x2):  # pylint: disable=missing-function-docstring
+
+  def f(x1, x2):
+    if x1.dtype == dtypes.bool:
+      assert x2.dtype == dtypes.bool
+      x1 = math_ops.cast(x1, dtypes.int8)
+      x2 = math_ops.cast(x2, dtypes.int8)
+    return math_ops.floordiv(x1, x2)
+
+  return _bin_op(f, x1, x2)
+
+
+@np_utils.np_doc(np.mod)
+def mod(x1, x2):  # pylint: disable=missing-function-docstring
+
+  def f(x1, x2):
+    if x1.dtype == dtypes.bool:
+      assert x2.dtype == dtypes.bool
+      x1 = math_ops.cast(x1, dtypes.int8)
+      x2 = math_ops.cast(x2, dtypes.int8)
+    return math_ops.mod(x1, x2)
+
+  return _bin_op(f, x1, x2)
+
+
+remainder = mod
+
+
+@np_utils.np_doc(np.divmod)
+def divmod(x1, x2):  # pylint: disable=redefined-builtin
+  return floor_divide(x1, x2), mod(x1, x2)
+
+
+@np_utils.np_doc(np.maximum)
+def maximum(x1, x2):
+
+  def max_or_or(x1, x2):
+    if x1.dtype == dtypes.bool:
+      assert x2.dtype == dtypes.bool
+      return math_ops.logical_or(x1, x2)
+    return math_ops.maximum(x1, x2)
+
+  return _bin_op(max_or_or, x1, x2)
+
+
+@np_utils.np_doc(np.minimum)
+def minimum(x1, x2):
+
+  def min_or_and(x1, x2):
+    if x1.dtype == dtypes.bool:
+      assert x2.dtype == dtypes.bool
+      return math_ops.logical_and(x1, x2)
+    return math_ops.minimum(x1, x2)
+
+  return _bin_op(min_or_and, x1, x2)
+
+
+@np_utils.np_doc(np.clip)
+def clip(a, a_min, a_max):  # pylint: disable=missing-docstring
+  if a_min is None and a_max is None:
+    raise ValueError('Not more than one of `a_min` and `a_max` may be `None`.')
+  if a_min is None:
+    return minimum(a, a_max)
+  elif a_max is None:
+    return maximum(a, a_min)
+  else:
+    a, a_min, a_max = np_array_ops._promote_dtype(a, a_min, a_max)  # pylint: disable=protected-access
+    return np_utils.tensor_to_ndarray(
+        clip_ops.clip_by_value(
+            *np_utils.tf_broadcast(a.data, a_min.data, a_max.data)))
+
+
+@np_utils.np_doc(np.matmul)
+def matmul(x1, x2):  # pylint: disable=missing-docstring
+
+  def f(x1, x2):
+    try:
+      return np_utils.cond(
+          math_ops.equal(array_ops.rank(x2), 1),
+          lambda: math_ops.tensordot(x1, x2, axes=1),
+          lambda: np_utils.cond(
+              math_ops.equal(array_ops.rank(x1), 1),  # pylint: disable=g-long-lambda
+              lambda: math_ops.tensordot(  # pylint: disable=g-long-lambda
+                  x1, x2, axes=[[0], [-2]]),
+              lambda: math_ops.matmul(x1, x2)))
+    except errors.InvalidArgumentError as err:
+      six.reraise(ValueError, ValueError(str(err)), sys.exc_info()[2])
+
+  return _bin_op(f, x1, x2)
+
+
+@np_utils.np_doc(np.tensordot)
+def tensordot(a, b, axes=2):
+  return _bin_op(lambda a, b: math_ops.tensordot(a, b, axes=axes), a, b)
+
+
+@np_utils.np_doc_only(np.inner)
+def inner(a, b):  # pylint: disable=missing-function-docstring
+
+  def f(a, b):
+    return np_utils.cond(
+        np_utils.logical_or(
+            math_ops.equal(array_ops.rank(a), 0),
+            math_ops.equal(array_ops.rank(b), 0)), lambda: a * b,
+        lambda: math_ops.tensordot(a, b, axes=[[-1], [-1]]))
+
+  return _bin_op(f, a, b)
+
+
+@np_utils.np_doc(np.cross)
+def cross(a, b, axisa=-1, axisb=-1, axisc=-1, axis=None):  # pylint: disable=missing-docstring
+
+  def f(a, b):  # pylint: disable=missing-docstring
+    # We can't assign to captured variable `axisa`, so make a new variable
+    axis_a = axisa
+    axis_b = axisb
+    axis_c = axisc
+    if axis is not None:
+      axis_a = axis
+      axis_b = axis
+      axis_c = axis
+    if axis_a < 0:
+      axis_a = np_utils.add(axis_a, array_ops.rank(a))
+    if axis_b < 0:
+      axis_b = np_utils.add(axis_b, array_ops.rank(b))
+
+    def maybe_move_axis_to_last(a, axis):
+
+      def move_axis_to_last(a, axis):
+        return array_ops.transpose(
+            a,
+            array_ops.concat([
+                math_ops.range(axis),
+                math_ops.range(axis + 1, array_ops.rank(a)), [axis]
+            ],
+                             axis=0))
+
+      return np_utils.cond(axis == np_utils.subtract(array_ops.rank(a), 1),
+                           lambda: a, lambda: move_axis_to_last(a, axis))
+
+    a = maybe_move_axis_to_last(a, axis_a)
+    b = maybe_move_axis_to_last(b, axis_b)
+    a_dim = np_utils.getitem(array_ops.shape(a), -1)
+    b_dim = np_utils.getitem(array_ops.shape(b), -1)
+
+    def maybe_pad_0(a, size_of_last_dim):
+
+      def pad_0(a):
+        return array_ops.pad(
+            a,
+            array_ops.concat([
+                array_ops.zeros([array_ops.rank(a) - 1, 2], dtypes.int32),
+                constant_op.constant([[0, 1]], dtypes.int32)
+            ],
+                             axis=0))
+
+      return np_utils.cond(
+          math_ops.equal(size_of_last_dim, 2), lambda: pad_0(a), lambda: a)
+
+    a = maybe_pad_0(a, a_dim)
+    b = maybe_pad_0(b, b_dim)
+    c = math_ops.cross(*np_utils.tf_broadcast(a, b))
+    if axis_c < 0:
+      axis_c = np_utils.add(axis_c, array_ops.rank(c))
+
+    def move_last_to_axis(a, axis):
+      r = array_ops.rank(a)
+      return array_ops.transpose(
+          a,
+          array_ops.concat(
+              [math_ops.range(axis), [r - 1],
+               math_ops.range(axis, r - 1)],
+              axis=0))
+
+    c = np_utils.cond(
+        (a_dim == 2) & (b_dim == 2),
+        lambda: c[..., 2],
+        lambda: np_utils.cond(  # pylint: disable=g-long-lambda
+            axis_c == np_utils.subtract(array_ops.rank(c), 1), lambda: c,
+            lambda: move_last_to_axis(c, axis_c)))
+    return c
+
+  return _bin_op(f, a, b)
+
+
+@np_utils.np_doc(np.power)
+def power(x1, x2):
+  return _bin_op(math_ops.pow, x1, x2)
+
+
+@np_utils.np_doc(np.float_power)
+def float_power(x1, x2):
+  return power(x1, x2)
+
+
+@np_utils.np_doc(np.arctan2)
+def arctan2(x1, x2):
+  return _bin_op(math_ops.atan2, x1, x2)
+
+
+@np_utils.np_doc(np.nextafter)
+def nextafter(x1, x2):
+  return _bin_op(math_ops.nextafter, x1, x2)
+
+
+@np_utils.np_doc(np.heaviside)
+def heaviside(x1, x2):  # pylint: disable=missing-function-docstring
+
+  def f(x1, x2):
+    return array_ops.where_v2(
+        x1 < 0, constant_op.constant(0, dtype=x2.dtype),
+        array_ops.where_v2(x1 > 0, constant_op.constant(1, dtype=x2.dtype), x2))
+
+  y = _bin_op(f, x1, x2)
+  if not np.issubdtype(y.dtype, np.inexact):
+    y = y.astype(np_dtypes.default_float_type())
+  return y
+
+
+@np_utils.np_doc(np.hypot)
+def hypot(x1, x2):
+  return sqrt(square(x1) + square(x2))
+
+
+@np_utils.np_doc(np.kron)
+def kron(a, b):  # pylint: disable=missing-function-docstring
+  # pylint: disable=protected-access,g-complex-comprehension
+  a, b = np_array_ops._promote_dtype(a, b)
+  ndim = max(a.ndim, b.ndim)
+  if a.ndim < ndim:
+    a = np_array_ops.reshape(a, np_array_ops._pad_left_to(ndim, a.shape))
+  if b.ndim < ndim:
+    b = np_array_ops.reshape(b, np_array_ops._pad_left_to(ndim, b.shape))
+  a_reshaped = np_array_ops.reshape(a, [i for d in a.shape for i in (d, 1)])
+  b_reshaped = np_array_ops.reshape(b, [i for d in b.shape for i in (1, d)])
+  out_shape = tuple(np.multiply(a.shape, b.shape))
+  return np_array_ops.reshape(a_reshaped * b_reshaped, out_shape)
+
+
+@np_utils.np_doc(np.outer)
+def outer(a, b):
+
+  def f(a, b):
+    return array_ops.reshape(a, [-1, 1]) * array_ops.reshape(b, [-1])
+
+  return _bin_op(f, a, b)
+
+
+# This can also be implemented via tf.reduce_logsumexp
+@np_utils.np_doc(np.logaddexp)
+def logaddexp(x1, x2):
+  amax = maximum(x1, x2)
+  delta = x1 - x2
+  return np_array_ops.where(
+      isnan(delta),
+      x1 + x2,  # NaNs or infinities of the same sign.
+      amax + log1p(exp(-abs(delta))))
+
+
+@np_utils.np_doc(np.logaddexp2)
+def logaddexp2(x1, x2):
+  amax = maximum(x1, x2)
+  delta = x1 - x2
+  return np_array_ops.where(
+      isnan(delta),
+      x1 + x2,  # NaNs or infinities of the same sign.
+      amax + log1p(exp2(-abs(delta))) / np.log(2))
+
+
+@np_utils.np_doc(np.polyval)
+def polyval(p, x):  # pylint: disable=missing-function-docstring
+
+  def f(p, x):
+    if p.shape.rank == 0:
+      p = array_ops.reshape(p, [1])
+    p = array_ops.unstack(p)
+    # TODO(wangpeng): Make tf version take a tensor for p instead of a list.
+    y = math_ops.polyval(p, x)
+    # If the polynomial is 0-order, numpy requires the result to be broadcast to
+    # `x`'s shape.
+    if len(p) == 1:
+      y = array_ops.broadcast_to(y, x.shape)
+    return y
+
+  return _bin_op(f, p, x)
+
+
+@np_utils.np_doc(np.isclose)
+def isclose(a, b, rtol=1e-05, atol=1e-08, equal_nan=False):  # pylint: disable=missing-docstring
+
+  def f(a, b):  # pylint: disable=missing-docstring
+    dtype = a.dtype
+    if np.issubdtype(dtype.as_numpy_dtype, np.inexact):
+      rtol_ = ops.convert_to_tensor(rtol, dtype.real_dtype)
+      atol_ = ops.convert_to_tensor(atol, dtype.real_dtype)
+      result = (math_ops.abs(a - b) <= atol_ + rtol_ * math_ops.abs(b))
+      if equal_nan:
+        result = result | (math_ops.is_nan(a) & math_ops.is_nan(b))
+      return result
+    else:
+      return a == b
+
+  return _bin_op(f, a, b)
+
+
+@np_utils.np_doc(np.allclose)
+def allclose(a, b, rtol=1e-05, atol=1e-08, equal_nan=False):
+  return np_array_ops.all(
+      isclose(a, b, rtol=rtol, atol=atol, equal_nan=equal_nan))
+
+
+def _tf_gcd(x1, x2):  # pylint: disable=missing-function-docstring
+
+  def _gcd_cond_fn(_, x2):
+    return math_ops.reduce_any(x2 != 0)
+
+  def _gcd_body_fn(x1, x2):
+    # math_ops.mod will raise an error when any element of x2 is 0. To avoid
+    # that, we change those zeros to ones. Their values don't matter because
+    # they won't be used.
+    x2_safe = array_ops.where_v2(x2 != 0, x2, constant_op.constant(1, x2.dtype))
+    x1, x2 = (array_ops.where_v2(x2 != 0, x2, x1),
+              array_ops.where_v2(x2 != 0, math_ops.mod(x1, x2_safe),
+                                 constant_op.constant(0, x2.dtype)))
+    return (array_ops.where_v2(x1 < x2, x2,
+                               x1), array_ops.where_v2(x1 < x2, x1, x2))
+
+  if (not np.issubdtype(x1.dtype.as_numpy_dtype, np.integer) or
+      not np.issubdtype(x2.dtype.as_numpy_dtype, np.integer)):
+    raise ValueError('Arguments to gcd must be integers.')
+  shape = array_ops.broadcast_static_shape(x1.shape, x2.shape)
+  x1 = array_ops.broadcast_to(x1, shape)
+  x2 = array_ops.broadcast_to(x2, shape)
+  value, _ = control_flow_ops.while_loop(_gcd_cond_fn, _gcd_body_fn,
+                                         (math_ops.abs(x1), math_ops.abs(x2)))
+  return value
+
+
+# Note that np.gcd may not be present in some supported versions of numpy.
+@np_utils.np_doc(None, np_fun_name='gcd')
+def gcd(x1, x2):
+  return _bin_op(_tf_gcd, x1, x2)
+
+
+# Note that np.lcm may not be present in some supported versions of numpy.
+@np_utils.np_doc(None, np_fun_name='lcm')
+def lcm(x1, x2):  # pylint: disable=missing-function-docstring
+
+  def f(x1, x2):
+    d = _tf_gcd(x1, x2)
+    # Same as the `x2_safe` trick above
+    d_safe = array_ops.where_v2(
+        math_ops.equal(d, 0), constant_op.constant(1, d.dtype), d)
+    return array_ops.where_v2(
+        math_ops.equal(d, 0), constant_op.constant(0, d.dtype),
+        math_ops.abs(x1 * x2) // d_safe)
+
+  return _bin_op(f, x1, x2)
+
+
+def _bitwise_binary_op(tf_fn, x1, x2):  # pylint: disable=missing-function-docstring
+
+  def f(x1, x2):
+    is_bool = (x1.dtype == dtypes.bool)
+    if is_bool:
+      assert x2.dtype == dtypes.bool
+      x1 = math_ops.cast(x1, dtypes.int8)
+      x2 = math_ops.cast(x2, dtypes.int8)
+    r = tf_fn(x1, x2)
+    if is_bool:
+      r = math_ops.cast(r, dtypes.bool)
+    return r
+
+  return _bin_op(f, x1, x2)
+
+
+@np_utils.np_doc(np.bitwise_and)
+def bitwise_and(x1, x2):
+  return _bitwise_binary_op(bitwise_ops.bitwise_and, x1, x2)
+
+
+@np_utils.np_doc(np.bitwise_or)
+def bitwise_or(x1, x2):
+  return _bitwise_binary_op(bitwise_ops.bitwise_or, x1, x2)
+
+
+@np_utils.np_doc(np.bitwise_xor)
+def bitwise_xor(x1, x2):
+  return _bitwise_binary_op(bitwise_ops.bitwise_xor, x1, x2)
+
+
+@np_utils.np_doc(np.bitwise_not)
+def bitwise_not(x):
+
+  def f(x):
+    if x.dtype == dtypes.bool:
+      return math_ops.logical_not(x)
+    return bitwise_ops.invert(x)
+
+  return _scalar(f, x)
+
+
+def _scalar(tf_fn, x, promote_to_float=False):
+  """Computes the tf_fn(x) for each element in `x`.
+
+  Args:
+    tf_fn: function that takes a single Tensor argument.
+    x: array_like. Could be an ndarray, a Tensor or any object that can be
+      converted to a Tensor using `ops.convert_to_tensor`.
+    promote_to_float: whether to cast the argument to a float dtype
+      (`np_dtypes.default_float_type`) if it is not already.
+
+  Returns:
+    An ndarray with the same shape as `x`. The default output dtype is
+    determined by `np_dtypes.default_float_type`, unless x is an ndarray with a
+    floating point type, in which case the output type is same as x.dtype.
+  """
+  x = np_array_ops.asarray(x)
+  if promote_to_float and not np.issubdtype(x.dtype, np.inexact):
+    x = x.astype(np_dtypes.default_float_type())
+  return np_utils.tensor_to_ndarray(tf_fn(x.data))
+
+
+@np_utils.np_doc(np.log)
+def log(x):
+  return _scalar(math_ops.log, x, True)
+
+
+@np_utils.np_doc(np.exp)
+def exp(x):
+  return _scalar(math_ops.exp, x, True)
+
+
+@np_utils.np_doc(np.sqrt)
+def sqrt(x):
+  return _scalar(math_ops.sqrt, x, True)
+
+
+@np_utils.np_doc(np.abs)
+def abs(x):  # pylint: disable=redefined-builtin
+  return _scalar(math_ops.abs, x)
+
+
+@np_utils.np_doc(np.absolute)
+def absolute(x):
+  return abs(x)
+
+
+@np_utils.np_doc(np.fabs)
+def fabs(x):
+  return abs(x)
+
+
+@np_utils.np_doc(np.ceil)
+def ceil(x):
+  return _scalar(math_ops.ceil, x, True)
+
+
+@np_utils.np_doc(np.floor)
+def floor(x):
+  return _scalar(math_ops.floor, x, True)
+
+
+@np_utils.np_doc(np.conj)
+def conj(x):
+  return _scalar(math_ops.conj, x)
+
+
+@np_utils.np_doc(np.negative)
+def negative(x):
+  return _scalar(math_ops.negative, x)
+
+
+@np_utils.np_doc(np.reciprocal)
+def reciprocal(x):
+  return _scalar(math_ops.reciprocal, x)
+
+
+@np_utils.np_doc(np.signbit)
+def signbit(x):
+
+  def f(x):
+    if x.dtype == dtypes.bool:
+      return array_ops.fill(x.shape, False)
+    return x < 0
+
+  return _scalar(f, x)
+
+
+@np_utils.np_doc(np.sin)
+def sin(x):
+  return _scalar(math_ops.sin, x, True)
+
+
+@np_utils.np_doc(np.cos)
+def cos(x):
+  return _scalar(math_ops.cos, x, True)
+
+
+@np_utils.np_doc(np.tan)
+def tan(x):
+  return _scalar(math_ops.tan, x, True)
+
+
+@np_utils.np_doc(np.sinh)
+def sinh(x):
+  return _scalar(math_ops.sinh, x, True)
+
+
+@np_utils.np_doc(np.cosh)
+def cosh(x):
+  return _scalar(math_ops.cosh, x, True)
+
+
+@np_utils.np_doc(np.tanh)
+def tanh(x):
+  return _scalar(math_ops.tanh, x, True)
+
+
+@np_utils.np_doc(np.arcsin)
+def arcsin(x):
+  return _scalar(math_ops.asin, x, True)
+
+
+@np_utils.np_doc(np.arccos)
+def arccos(x):
+  return _scalar(math_ops.acos, x, True)
+
+
+@np_utils.np_doc(np.arctan)
+def arctan(x):
+  return _scalar(math_ops.atan, x, True)
+
+
+@np_utils.np_doc(np.arcsinh)
+def arcsinh(x):
+  return _scalar(math_ops.asinh, x, True)
+
+
+@np_utils.np_doc(np.arccosh)
+def arccosh(x):
+  return _scalar(math_ops.acosh, x, True)
+
+
+@np_utils.np_doc(np.arctanh)
+def arctanh(x):
+  return _scalar(math_ops.atanh, x, True)
+
+
+@np_utils.np_doc(np.deg2rad)
+def deg2rad(x):
+
+  def f(x):
+    return x * (np.pi / 180.0)
+
+  return _scalar(f, x, True)
+
+
+@np_utils.np_doc(np.rad2deg)
+def rad2deg(x):
+  return x * (180.0 / np.pi)
+
+
+_tf_float_types = [
+    dtypes.bfloat16, dtypes.float16, dtypes.float32, dtypes.float64
+]
+
+
+@np_utils.np_doc(np.angle)
+def angle(z, deg=False):  # pylint: disable=missing-function-docstring
+
+  def f(x):
+    if x.dtype in _tf_float_types:
+      # Workaround for b/147515503
+      return array_ops.where_v2(x < 0, np.pi, 0)
+    else:
+      return math_ops.angle(x)
+
+  y = _scalar(f, z, True)
+  if deg:
+    y = rad2deg(y)
+  return y
+
+
+@np_utils.np_doc(np.cbrt)
+def cbrt(x):
+
+  def f(x):
+    # __pow__ can't handle negative base, so we use `abs` here.
+    rt = math_ops.abs(x)**(1.0 / 3)
+    return array_ops.where_v2(x < 0, -rt, rt)
+
+  return _scalar(f, x, True)
+
+
+@np_utils.np_doc(np.conjugate)
+def conjugate(x):
+  return _scalar(math_ops.conj, x)
+
+
+@np_utils.np_doc(np.exp2)
+def exp2(x):
+
+  def f(x):
+    return 2**x
+
+  return _scalar(f, x, True)
+
+
+@np_utils.np_doc(np.expm1)
+def expm1(x):
+  return _scalar(math_ops.expm1, x, True)
+
+
+@np_utils.np_doc(np.fix)
+def fix(x):
+
+  def f(x):
+    return array_ops.where_v2(x < 0, math_ops.ceil(x), math_ops.floor(x))
+
+  return _scalar(f, x, True)
+
+
+@np_utils.np_doc(np.iscomplex)
+def iscomplex(x):
+  return np_array_ops.imag(x) != 0
+
+
+@np_utils.np_doc(np.isreal)
+def isreal(x):
+  return np_array_ops.imag(x) == 0
+
+
+@np_utils.np_doc(np.iscomplexobj)
+def iscomplexobj(x):
+  x = np_array_ops.array(x)
+  return np.issubdtype(x.dtype, np.complexfloating)
+
+
+@np_utils.np_doc(np.isrealobj)
+def isrealobj(x):
+  return not iscomplexobj(x)
+
+
+@np_utils.np_doc(np.isnan)
+def isnan(x):
+  return _scalar(math_ops.is_nan, x, True)
+
+
+def _make_nan_reduction(onp_reduction, reduction, init_val):
+  """Helper to generate nan* functions."""
+
+  @np_utils.np_doc(onp_reduction)
+  def nan_reduction(a, axis=None, dtype=None, keepdims=False):
+    a = np_array_ops.array(a)
+    v = np_array_ops.array(init_val, dtype=a.dtype)
+    return reduction(
+        np_array_ops.where(isnan(a), v, a),
+        axis=axis,
+        dtype=dtype,
+        keepdims=keepdims)
+
+  return nan_reduction
+
+
+nansum = _make_nan_reduction(np.nansum, np_array_ops.sum, 0)
+nanprod = _make_nan_reduction(np.nanprod, np_array_ops.prod, 1)
+
+
+@np_utils.np_doc(np.nanmean)
+def nanmean(a, axis=None, dtype=None, keepdims=None):  # pylint: disable=missing-docstring
+  a = np_array_ops.array(a)
+  if np.issubdtype(a.dtype, np.bool_) or np.issubdtype(a.dtype, np.integer):
+    return np_array_ops.mean(a, axis=axis, dtype=dtype, keepdims=keepdims)
+  nan_mask = logical_not(isnan(a))
+  if dtype is None:
+    dtype = a.dtype
+  normalizer = np_array_ops.sum(
+      nan_mask, axis=axis, dtype=dtype, keepdims=keepdims)
+  return nansum(a, axis=axis, dtype=dtype, keepdims=keepdims) / normalizer
+
+
+@np_utils.np_doc(np.isfinite)
+def isfinite(x):
+  return _scalar(math_ops.is_finite, x, True)
+
+
+@np_utils.np_doc(np.isinf)
+def isinf(x):
+  return _scalar(math_ops.is_inf, x, True)
+
+
+@np_utils.np_doc(np.isneginf)
+def isneginf(x):
+  return x == np_array_ops.full_like(x, -np.inf)
+
+
+@np_utils.np_doc(np.isposinf)
+def isposinf(x):
+  return x == np_array_ops.full_like(x, np.inf)
+
+
+@np_utils.np_doc(np.log2)
+def log2(x):
+  return log(x) / np.log(2)
+
+
+@np_utils.np_doc(np.log10)
+def log10(x):
+  return log(x) / np.log(10)
+
+
+@np_utils.np_doc(np.log1p)
+def log1p(x):
+  return _scalar(math_ops.log1p, x, True)
+
+
+@np_utils.np_doc(np.positive)
+def positive(x):
+  return _scalar(lambda x: x, x)
+
+
+@np_utils.np_doc(np.sinc)
+def sinc(x):
+
+  def f(x):
+    pi_x = x * np.pi
+    return array_ops.where_v2(x == 0, array_ops.ones_like(x),
+                              math_ops.sin(pi_x) / pi_x)
+
+  return _scalar(f, x, True)
+
+
+@np_utils.np_doc(np.square)
+def square(x):
+  return _scalar(math_ops.square, x)
+
+
+@np_utils.np_doc(np.diff)
+def diff(a, n=1, axis=-1):  # pylint: disable=missing-function-docstring
+
+  def f(a):
+    nd = a.shape.rank
+    if (axis + nd if axis < 0 else axis) >= nd:
+      raise ValueError('axis %s is out of bounds for array of dimension %s' %
+                       (axis, nd))
+    if n < 0:
+      raise ValueError('order must be non-negative but got %s' % n)
+    slice1 = [slice(None)] * nd
+    slice2 = [slice(None)] * nd
+    slice1[axis] = slice(1, None)
+    slice2[axis] = slice(None, -1)
+    slice1 = tuple(slice1)
+    slice2 = tuple(slice2)
+    op = math_ops.not_equal if a.dtype == dtypes.bool else math_ops.subtract
+    for _ in range(n):
+      a = op(a[slice1], a[slice2])
+    return a
+
+  return _scalar(f, a)
+
+
+def _flip_args(f):
+  def _f(a, b):
+    return f(b, a)
+  return _f
+
+
+setattr(np_arrays.ndarray, '__abs__', absolute)
+setattr(np_arrays.ndarray, '__floordiv__', floor_divide)
+setattr(np_arrays.ndarray, '__rfloordiv__', _flip_args(floor_divide))
+setattr(np_arrays.ndarray, '__mod__', mod)
+setattr(np_arrays.ndarray, '__rmod__', _flip_args(mod))
+setattr(np_arrays.ndarray, '__add__', add)
+setattr(np_arrays.ndarray, '__radd__', _flip_args(add))
+setattr(np_arrays.ndarray, '__sub__', subtract)
+setattr(np_arrays.ndarray, '__rsub__', _flip_args(subtract))
+setattr(np_arrays.ndarray, '__mul__', multiply)
+setattr(np_arrays.ndarray, '__rmul__', _flip_args(multiply))
+setattr(np_arrays.ndarray, '__pow__', power)
+setattr(np_arrays.ndarray, '__rpow__', _flip_args(power))
+setattr(np_arrays.ndarray, '__truediv__', true_divide)
+setattr(np_arrays.ndarray, '__rtruediv__', _flip_args(true_divide))
+
+
+def _comparison(tf_fun, x1, x2, cast_bool_to_int=False):
+  dtype = np_utils.result_type(x1, x2)
+  # Cast x1 and x2 to the result_type if needed.
+  x1 = np_array_ops.array(x1, dtype=dtype)
+  x2 = np_array_ops.array(x2, dtype=dtype)
+  x1 = x1.data
+  x2 = x2.data
+  if cast_bool_to_int and x1.dtype == dtypes.bool:
+    x1 = math_ops.cast(x1, dtypes.int32)
+    x2 = math_ops.cast(x2, dtypes.int32)
+  return np_utils.tensor_to_ndarray(tf_fun(x1, x2))
+
+
+@np_utils.np_doc(np.equal)
+def equal(x1, x2):
+  return _comparison(math_ops.equal, x1, x2)
+
+
+@np_utils.np_doc(np.not_equal)
+def not_equal(x1, x2):
+  return _comparison(math_ops.not_equal, x1, x2)
+
+
+@np_utils.np_doc(np.greater)
+def greater(x1, x2):
+  return _comparison(math_ops.greater, x1, x2, True)
+
+
+@np_utils.np_doc(np.greater_equal)
+def greater_equal(x1, x2):
+  return _comparison(math_ops.greater_equal, x1, x2, True)
+
+
+@np_utils.np_doc(np.less)
+def less(x1, x2):
+  return _comparison(math_ops.less, x1, x2, True)
+
+
+@np_utils.np_doc(np.less_equal)
+def less_equal(x1, x2):
+  return _comparison(math_ops.less_equal, x1, x2, True)
+
+
+@np_utils.np_doc(np.array_equal)
+def array_equal(a1, a2):
+
+  def f(a1, a2):
+    if a1.shape != a2.shape:
+      return constant_op.constant(False)
+    return math_ops.reduce_all(math_ops.equal(a1, a2))
+
+  return _comparison(f, a1, a2)
+
+
+def _logical_binary_op(tf_fun, x1, x2):
+  x1 = np_array_ops.array(x1, dtype=np.bool_)
+  x2 = np_array_ops.array(x2, dtype=np.bool_)
+  return np_utils.tensor_to_ndarray(tf_fun(x1.data, x2.data))
+
+
+@np_utils.np_doc(np.logical_and)
+def logical_and(x1, x2):
+  return _logical_binary_op(math_ops.logical_and, x1, x2)
+
+
+@np_utils.np_doc(np.logical_or)
+def logical_or(x1, x2):
+  return _logical_binary_op(math_ops.logical_or, x1, x2)
+
+
+@np_utils.np_doc(np.logical_xor)
+def logical_xor(x1, x2):
+  return _logical_binary_op(math_ops.logical_xor, x1, x2)
+
+
+@np_utils.np_doc(np.logical_not)
+def logical_not(x):
+  x = np_array_ops.array(x, dtype=np.bool_)
+  return np_utils.tensor_to_ndarray(math_ops.logical_not(x.data))
+
+
+setattr(np_arrays.ndarray, '__invert__', logical_not)
+setattr(np_arrays.ndarray, '__lt__', less)
+setattr(np_arrays.ndarray, '__le__', less_equal)
+setattr(np_arrays.ndarray, '__gt__', greater)
+setattr(np_arrays.ndarray, '__ge__', greater_equal)
+setattr(np_arrays.ndarray, '__eq__', equal)
+setattr(np_arrays.ndarray, '__ne__', not_equal)
+
+
+@np_utils.np_doc(np.linspace)
+def linspace(  # pylint: disable=missing-docstring
+    start, stop, num=50, endpoint=True, retstep=False, dtype=float, axis=0):
+  if dtype:
+    dtype = np_utils.result_type(dtype)
+  start = np_array_ops.array(start, dtype=dtype).data
+  stop = np_array_ops.array(stop, dtype=dtype).data
+  if num < 0:
+    raise ValueError('Number of samples {} must be non-negative.'.format(num))
+  step = ops.convert_to_tensor(np.nan)
+  if endpoint:
+    result = math_ops.linspace(start, stop, num, axis=axis)
+    if num > 1:
+      step = (stop - start) / (num - 1)
+  else:
+    # math_ops.linspace does not support endpoint=False so we manually handle it
+    # here.
+    if num > 1:
+      step = ((stop - start) / num)
+      new_stop = math_ops.cast(stop, step.dtype) - step
+      start = math_ops.cast(start, new_stop.dtype)
+      result = math_ops.linspace(start, new_stop, num, axis=axis)
+    else:
+      result = math_ops.linspace(start, stop, num, axis=axis)
+  if dtype:
+    result = math_ops.cast(result, dtype)
+  if retstep:
+    return (np_arrays.tensor_to_ndarray(result),
+            np_arrays.tensor_to_ndarray(step))
+  else:
+    return np_arrays.tensor_to_ndarray(result)
+
+
+@np_utils.np_doc(np.logspace)
+def logspace(start, stop, num=50, endpoint=True, base=10.0, dtype=None, axis=0):
+  dtype = np_utils.result_type(start, stop, dtype)
+  result = linspace(
+      start, stop, num=num, endpoint=endpoint, dtype=dtype, axis=axis).data
+  result = math_ops.pow(math_ops.cast(base, result.dtype), result)
+  if dtype:
+    result = math_ops.cast(result, dtype)
+  return np_arrays.tensor_to_ndarray(result)
+
+
+@np_utils.np_doc(np.ptp)
+def ptp(a, axis=None, keepdims=None):
+  return (np_array_ops.amax(a, axis=axis, keepdims=keepdims) -
+          np_array_ops.amin(a, axis=axis, keepdims=keepdims))
+
+
+@np_utils.np_doc_only(np.concatenate)
+def concatenate(arys, axis=0):
+  if not isinstance(arys, (list, tuple)):
+    arys = [arys]
+  if not arys:
+    raise ValueError('Need at least one array to concatenate.')
+  dtype = np_utils.result_type(*arys)
+  arys = [np_array_ops.array(array, dtype=dtype).data for array in arys]
+  return np_arrays.tensor_to_ndarray(array_ops.concat(arys, axis))
+
+
+@np_utils.np_doc_only(np.tile)
+def tile(a, reps):  # pylint: disable=missing-function-docstring
+  a = np_array_ops.array(a).data
+  reps = np_array_ops.array(reps, dtype=dtypes.int32).reshape([-1]).data
+
+  a_rank = array_ops.rank(a)
+  reps_size = array_ops.size(reps)
+  reps = array_ops.pad(
+      reps, [[math_ops.maximum(a_rank - reps_size, 0), 0]], constant_values=1)
+  a_shape = array_ops.pad(
+      array_ops.shape(a), [[math_ops.maximum(reps_size - a_rank, 0), 0]],
+      constant_values=1)
+  a = array_ops.reshape(a, a_shape)
+
+  return np_arrays.tensor_to_ndarray(array_ops.tile(a, reps))
+
+
+@np_utils.np_doc(np.count_nonzero)
+def count_nonzero(a, axis=None):
+  return np_arrays.tensor_to_ndarray(
+      math_ops.count_nonzero(np_array_ops.array(a).data, axis))
+
+
+@np_utils.np_doc(np.argsort)
+def argsort(a, axis=-1, kind='quicksort', order=None):  # pylint: disable=missing-docstring
+  # TODO(nareshmodi): make string tensors also work.
+  if kind not in ('quicksort', 'stable'):
+    raise ValueError("Only 'quicksort' and 'stable' arguments are supported.")
+  if order is not None:
+    raise ValueError("'order' argument to sort is not supported.")
+  stable = (kind == 'stable')
+
+  a = np_array_ops.array(a).data
+
+  def _argsort(a, axis, stable):
+    if axis is None:
+      a = array_ops.reshape(a, [-1])
+      axis = 0
+
+    return sort_ops.argsort(a, axis, stable=stable)
+
+  tf_ans = control_flow_ops.cond(
+      math_ops.equal(array_ops.rank(a), 0), lambda: constant_op.constant([0]),
+      lambda: _argsort(a, axis, stable))
+
+  return np_array_ops.array(tf_ans, dtype=np.intp)
+
+
+@np_utils.np_doc(np.sort)
+def sort(a, axis=-1, kind='quicksort', order=None):  # pylint: disable=missing-docstring
+  if kind != 'quicksort':
+    raise ValueError("Only 'quicksort' is supported.")
+  if order is not None:
+    raise ValueError("'order' argument to sort is not supported.")
+
+  a = np_array_ops.array(a)
+
+  if axis is None:
+    result_t = sort_ops.sort(array_ops.reshape(a.data, [-1]), 0)
+    return np_utils.tensor_to_ndarray(result_t)
+  else:
+    return np_utils.tensor_to_ndarray(sort_ops.sort(a.data, axis))
+
+
+def _argminmax(fn, a, axis=None):
+  a = np_array_ops.array(a)
+  if axis is None:
+    # When axis is None numpy flattens the array.
+    a_t = array_ops.reshape(a.data, [-1])
+  else:
+    a_t = np_array_ops.atleast_1d(a).data
+  return np_utils.tensor_to_ndarray(fn(input=a_t, axis=axis))
+
+
+@np_utils.np_doc(np.argmax)
+def argmax(a, axis=None):
+  return _argminmax(math_ops.argmax, a, axis)
+
+
+@np_utils.np_doc(np.argmin)
+def argmin(a, axis=None):
+  return _argminmax(math_ops.argmin, a, axis)
+
+
+@np_utils.np_doc(np.append)
+def append(arr, values, axis=None):
+  if axis is None:
+    return concatenate([np_array_ops.ravel(arr), np_array_ops.ravel(values)], 0)
+  else:
+    return concatenate([arr, values], axis=axis)
+
+
+@np_utils.np_doc(np.average)
+def average(a, axis=None, weights=None, returned=False):  # pylint: disable=missing-docstring
+  if axis is not None and not isinstance(axis, six.integer_types):
+    # TODO(wangpeng): Support tuple of ints as `axis`
+    raise ValueError('`axis` must be an integer. Tuple of ints is not '
+                     'supported yet. Got type: %s' % type(axis))
+  a = np_array_ops.array(a)
+  if weights is None:  # Treat all weights as 1
+    if not np.issubdtype(a.dtype, np.inexact):
+      a = a.astype(
+          np_utils.result_type(a.dtype, np_dtypes.default_float_type()))
+    avg = math_ops.reduce_mean(a.data, axis=axis)
+    if returned:
+      if axis is None:
+        weights_sum = array_ops.size(a.data)
+      else:
+        weights_sum = array_ops.shape(a.data)[axis]
+      weights_sum = math_ops.cast(weights_sum, a.data.dtype)
+  else:
+    if np.issubdtype(a.dtype, np.inexact):
+      out_dtype = np_utils.result_type(a.dtype, weights)
+    else:
+      out_dtype = np_utils.result_type(a.dtype, weights,
+                                       np_dtypes.default_float_type())
+    a = np_array_ops.array(a, out_dtype).data
+    weights = np_array_ops.array(weights, out_dtype).data
+
+    def rank_equal_case():
+      control_flow_ops.Assert(
+          math_ops.reduce_all(array_ops.shape(a) == array_ops.shape(weights)),
+          [array_ops.shape(a), array_ops.shape(weights)])
+      weights_sum = math_ops.reduce_sum(weights, axis=axis)
+      avg = math_ops.reduce_sum(a * weights, axis=axis) / weights_sum
+      return avg, weights_sum
+
+    if axis is None:
+      avg, weights_sum = rank_equal_case()
+    else:
+
+      def rank_not_equal_case():
+        control_flow_ops.Assert(
+            array_ops.rank(weights) == 1, [array_ops.rank(weights)])
+        weights_sum = math_ops.reduce_sum(weights)
+        axes = ops.convert_to_tensor([[axis], [0]])
+        avg = math_ops.tensordot(a, weights, axes) / weights_sum
+        return avg, weights_sum
+
+      # We condition on rank rather than shape equality, because if we do the
+      # latter, when the shapes are partially unknown but the ranks are known
+      # and different, np_utils.cond will run shape checking on the true branch,
+      # which will raise a shape-checking error.
+      avg, weights_sum = np_utils.cond(
+          math_ops.equal(array_ops.rank(a), array_ops.rank(weights)),
+          rank_equal_case, rank_not_equal_case)
+
+  avg = np_array_ops.array(avg)
+  if returned:
+    weights_sum = np_array_ops.broadcast_to(weights_sum,
+                                            array_ops.shape(avg.data))
+    return avg, weights_sum
+  return avg
+
+
+@np_utils.np_doc(np.trace)
+def trace(a, offset=0, axis1=0, axis2=1, dtype=None):  # pylint: disable=missing-docstring
+  if dtype:
+    dtype = np_utils.result_type(dtype)
+  a = np_array_ops.asarray(a, dtype).data
+
+  if offset == 0:
+    a_shape = a.shape
+    if a_shape.rank is not None:
+      rank = len(a_shape)
+      if (axis1 == -2 or axis1 == rank - 2) and (axis2 == -1 or
+                                                 axis2 == rank - 1):
+        return np_utils.tensor_to_ndarray(math_ops.trace(a))
+
+  a = np_array_ops.diagonal(a, offset, axis1, axis2)
+  return np_array_ops.sum(a, -1, dtype)
+
+
+@np_utils.np_doc(np.meshgrid)
+def meshgrid(*xi, **kwargs):
+  """This currently requires copy=True and sparse=False."""
+  sparse = kwargs.get('sparse', False)
+  if sparse:
+    raise ValueError('meshgrid doesnt support returning sparse arrays yet')
+
+  copy = kwargs.get('copy', True)
+  if not copy:
+    raise ValueError('meshgrid only supports copy=True')
+
+  indexing = kwargs.get('indexing', 'xy')
+
+  xi = [np_array_ops.asarray(arg).data for arg in xi]
+  kwargs = {'indexing': indexing}
+
+  outputs = array_ops.meshgrid(*xi, **kwargs)
+  outputs = [np_utils.tensor_to_ndarray(output) for output in outputs]
+
+  return outputs
diff --git a/tensorflow/python/ops/numpy_ops/np_math_ops_test.py b/tensorflow/python/ops/numpy_ops/np_math_ops_test.py
new file mode 100644
index 00000000000..00d9a35a025
--- /dev/null
+++ b/tensorflow/python/ops/numpy_ops/np_math_ops_test.py
@@ -0,0 +1,332 @@
+# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Tests for tf numpy mathematical methods."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import itertools
+from absl.testing import parameterized
+import numpy as np
+from six.moves import range
+
+from tensorflow.python.framework import errors
+from tensorflow.python.framework import ops
+from tensorflow.python.ops.numpy_ops import np_array_ops
+from tensorflow.python.ops.numpy_ops import np_arrays
+from tensorflow.python.ops.numpy_ops import np_math_ops
+from tensorflow.python.platform import test
+
+
+class MathTest(test.TestCase, parameterized.TestCase):
+
+  def setUp(self):
+    super(MathTest, self).setUp()
+    self.array_transforms = [
+        lambda x: x,  # Identity,
+        ops.convert_to_tensor,
+        np.array,
+        lambda x: np.array(x, dtype=np.float32),
+        lambda x: np.array(x, dtype=np.float64),
+        np_array_ops.array,
+        lambda x: np_array_ops.array(x, dtype=np.float32),
+        lambda x: np_array_ops.array(x, dtype=np.float64),
+    ]
+    self.types = [np.int32, np.int64, np.float32, np.float64]
+
+  def _testBinaryOp(self,
+                    math_fun,
+                    np_fun,
+                    name,
+                    operands=None,
+                    extra_operands=None,
+                    check_promotion=True,
+                    check_promotion_result_type=True):
+
+    def run_test(a, b):
+      for fn in self.array_transforms:
+        arg1 = fn(a)
+        arg2 = fn(b)
+        self.match(
+            math_fun(arg1, arg2),
+            np_fun(arg1, arg2),
+            msg='{}({}, {})'.format(name, arg1, arg2))
+      # Tests type promotion
+      for type_a in self.types:
+        for type_b in self.types:
+          if not check_promotion and type_a != type_b:
+            continue
+          arg1 = np_array_ops.array(a, dtype=type_a)
+          arg2 = np_array_ops.array(b, dtype=type_b)
+          self.match(
+              math_fun(arg1, arg2),
+              np_fun(arg1, arg2),
+              msg='{}({}, {})'.format(name, arg1, arg2),
+              check_dtype=check_promotion_result_type)
+
+    if operands is None:
+      operands = [(5, 2), (5, [2, 3]), (5, [[2, 3], [6, 7]]), ([1, 2, 3], 7),
+                  ([1, 2, 3], [5, 6, 7])]
+    for operand1, operand2 in operands:
+      run_test(operand1, operand2)
+    if extra_operands is not None:
+      for operand1, operand2 in extra_operands:
+        run_test(operand1, operand2)
+
+  def testDot(self):
+    extra_operands = [([1, 2], [[5, 6, 7], [8, 9, 10]]),
+                      (np.arange(2 * 3 * 5).reshape([2, 3, 5]).tolist(),
+                       np.arange(5 * 7 * 11).reshape([7, 5, 11]).tolist())]
+    return self._testBinaryOp(
+        np_math_ops.dot, np.dot, 'dot', extra_operands=extra_operands)
+
+  def testMinimum(self):
+    # The numpy version has strange result type when promotion happens,
+    # so set check_promotion_result_type to False.
+    return self._testBinaryOp(
+        np_math_ops.minimum,
+        np.minimum,
+        'minimum',
+        check_promotion_result_type=False)
+
+  def testMaximum(self):
+    # The numpy version has strange result type when promotion happens,
+    # so set check_promotion_result_type to False.
+    return self._testBinaryOp(
+        np_math_ops.maximum,
+        np.maximum,
+        'maximum',
+        check_promotion_result_type=False)
+
+  def testMatmul(self):
+    operands = [([[1, 2]], [[3, 4, 5], [6, 7, 8]])]
+    return self._testBinaryOp(
+        np_math_ops.matmul, np.matmul, 'matmul', operands=operands)
+
+  def testMatmulError(self):
+    with self.assertRaisesRegex(ValueError, r''):
+      np_math_ops.matmul(
+          np_array_ops.ones([], np.int32), np_array_ops.ones([2, 3], np.int32))
+    with self.assertRaisesRegex(ValueError, r''):
+      np_math_ops.matmul(
+          np_array_ops.ones([2, 3], np.int32), np_array_ops.ones([], np.int32))
+
+  def _testUnaryOp(self, math_fun, np_fun, name):
+
+    def run_test(a):
+      for fn in self.array_transforms:
+        arg1 = fn(a)
+        self.match(
+            math_fun(arg1), np_fun(arg1), msg='{}({})'.format(name, arg1))
+
+    run_test(5)
+    run_test([2, 3])
+    run_test([[2, -3], [-6, 7]])
+
+  def testLog(self):
+    self._testUnaryOp(np_math_ops.log, np.log, 'log')
+
+  def testExp(self):
+    self._testUnaryOp(np_math_ops.exp, np.exp, 'exp')
+
+  def testTanh(self):
+    self._testUnaryOp(np_math_ops.tanh, np.tanh, 'tanh')
+
+  def testSqrt(self):
+    self._testUnaryOp(np_math_ops.sqrt, np.sqrt, 'sqrt')
+
+  def match(self, actual, expected, msg='', check_dtype=True):
+    self.assertIsInstance(actual, np_arrays.ndarray)
+    if check_dtype:
+      self.assertEqual(
+          actual.dtype, expected.dtype,
+          'Dtype mismatch.\nActual: {}\nExpected: {}\n{}'.format(
+              actual.dtype, expected.dtype, msg))
+    self.assertEqual(
+        actual.shape, expected.shape,
+        'Shape mismatch.\nActual: {}\nExpected: {}\n{}'.format(
+            actual.shape, expected.shape, msg))
+    np.testing.assert_almost_equal(actual.tolist(), expected.tolist())
+
+  def testArgsort(self):
+    self._testUnaryOp(np_math_ops.argsort, np.argsort, 'argsort')
+
+    # Test stability
+    r = np.arange(100)
+    a = np.zeros(100)
+    np.testing.assert_equal(np_math_ops.argsort(a, kind='stable'), r)
+
+  def testArgMaxArgMin(self):
+    data = [
+        0,
+        5,
+        [1],
+        [1, 2, 3],
+        [[1, 2, 3]],
+        [[4, 6], [7, 8]],
+        [[[4, 6], [9, 10]], [[7, 8], [12, 34]]],
+    ]
+    for fn, d in itertools.product(self.array_transforms, data):
+      arr = fn(d)
+      self.match(np_math_ops.argmax(arr), np.argmax(arr))
+      self.match(np_math_ops.argmin(arr), np.argmin(arr))
+      if hasattr(arr, 'shape'):
+        ndims = len(arr.shape)
+      else:
+        ndims = np_array_ops.array(arr, copy=False).ndim
+      if ndims == 0:
+        # Numpy flattens the scalar ndarray and treats it as a 1-d array of
+        # size 1.
+        ndims = 1
+      for axis in range(-ndims, ndims):
+        self.match(
+            np_math_ops.argmax(arr, axis=axis), np.argmax(arr, axis=axis))
+        self.match(
+            np_math_ops.argmin(arr, axis=axis), np.argmin(arr, axis=axis))
+
+  @parameterized.parameters([False, True])
+  def testIsCloseEqualNan(self, equal_nan):
+    a = np.asarray([1, 1, np.nan, 1, np.nan], np.float32)
+    b = np.asarray([1, 2, 1, np.nan, np.nan], np.float32)
+    self.match(
+        np_math_ops.isclose(a, b, equal_nan=equal_nan),
+        np.isclose(a, b, equal_nan=equal_nan))
+
+  def testAverageWrongShape(self):
+    with self.assertRaisesWithPredicateMatch(errors.InvalidArgumentError, r''):
+      np_math_ops.average(np.ones([2, 3]), weights=np.ones([2, 4]))
+    with self.assertRaisesWithPredicateMatch(errors.InvalidArgumentError, r''):
+      np_math_ops.average(np.ones([2, 3]), axis=0, weights=np.ones([2, 4]))
+    with self.assertRaisesWithPredicateMatch(errors.InvalidArgumentError, r''):
+      np_math_ops.average(np.ones([2, 3]), axis=0, weights=np.ones([]))
+    with self.assertRaisesWithPredicateMatch(errors.InvalidArgumentError, r''):
+      np_math_ops.average(np.ones([2, 3]), axis=0, weights=np.ones([5]))
+
+  def testClip(self):
+
+    def run_test(arr, *args, **kwargs):
+      check_dtype = kwargs.pop('check_dtype', True)
+      for fn in self.array_transforms:
+        arr = fn(arr)
+        self.match(
+            np_math_ops.clip(arr, *args, **kwargs),
+            np.clip(arr, *args, **kwargs),
+            check_dtype=check_dtype)
+
+    # NumPy exhibits weird typing behavior when a/a_min/a_max are scalars v/s
+    # lists, e.g.,
+    #
+    # np.clip(np.array(0, dtype=np.int32), -5, 5).dtype == np.int64
+    # np.clip(np.array([0], dtype=np.int32), -5, 5).dtype == np.int32
+    # np.clip(np.array([0], dtype=np.int32), [-5], [5]).dtype == np.int64
+    #
+    # So we skip matching type. In tf-numpy the type of the output array is
+    # always the same as the input array.
+    run_test(0, -1, 5, check_dtype=False)
+    run_test(-1, -1, 5, check_dtype=False)
+    run_test(5, -1, 5, check_dtype=False)
+    run_test(-10, -1, 5, check_dtype=False)
+    run_test(10, -1, 5, check_dtype=False)
+    run_test(10, None, 5, check_dtype=False)
+    run_test(10, -1, None, check_dtype=False)
+    run_test([0, 20, -5, 4], -1, 5, check_dtype=False)
+    run_test([0, 20, -5, 4], None, 5, check_dtype=False)
+    run_test([0, 20, -5, 4], -1, None, check_dtype=False)
+    run_test([0.5, 20.2, -5.7, 4.4], -1.5, 5.1, check_dtype=False)
+
+    run_test([0, 20, -5, 4], [-5, 0, -5, 0], [0, 5, 0, 5], check_dtype=False)
+    run_test([[1, 2, 3], [4, 5, 6]], [2, 0, 2], 5, check_dtype=False)
+    run_test([[1, 2, 3], [4, 5, 6]], 0, [5, 3, 1], check_dtype=False)
+
+  def testPtp(self):
+
+    def run_test(arr, *args, **kwargs):
+      for fn in self.array_transforms:
+        arg = fn(arr)
+        self.match(
+            np_math_ops.ptp(arg, *args, **kwargs), np.ptp(arg, *args, **kwargs))
+
+    run_test([1, 2, 3])
+    run_test([1., 2., 3.])
+    run_test([[1, 2], [3, 4]], axis=1)
+    run_test([[1, 2], [3, 4]], axis=0)
+    run_test([[1, 2], [3, 4]], axis=-1)
+    run_test([[1, 2], [3, 4]], axis=-2)
+
+  def testLinSpace(self):
+    array_transforms = [
+        lambda x: x,  # Identity,
+        ops.convert_to_tensor,
+        np.array,
+        lambda x: np.array(x, dtype=np.float32),
+        lambda x: np.array(x, dtype=np.float64),
+        np_array_ops.array,
+        lambda x: np_array_ops.array(x, dtype=np.float32),
+        lambda x: np_array_ops.array(x, dtype=np.float64)
+    ]
+
+    def run_test(start, stop, **kwargs):
+      for fn1 in array_transforms:
+        for fn2 in array_transforms:
+          arg1 = fn1(start)
+          arg2 = fn2(stop)
+          self.match(
+              np_math_ops.linspace(arg1, arg2, **kwargs),
+              np.linspace(arg1, arg2, **kwargs),
+              msg='linspace({}, {})'.format(arg1, arg2))
+
+    run_test(0, 1)
+    run_test(0, 1, num=10)
+    run_test(0, 1, endpoint=False)
+    run_test(0, -1)
+    run_test(0, -1, num=10)
+    run_test(0, -1, endpoint=False)
+
+  def testLogSpace(self):
+    array_transforms = [
+        lambda x: x,  # Identity,
+        ops.convert_to_tensor,
+        np.array,
+        lambda x: np.array(x, dtype=np.float32),
+        lambda x: np.array(x, dtype=np.float64),
+        np_array_ops.array,
+        lambda x: np_array_ops.array(x, dtype=np.float32),
+        lambda x: np_array_ops.array(x, dtype=np.float64)
+    ]
+
+    def run_test(start, stop, **kwargs):
+      for fn1 in array_transforms:
+        for fn2 in array_transforms:
+          arg1 = fn1(start)
+          arg2 = fn2(stop)
+          self.match(
+              np_math_ops.logspace(arg1, arg2, **kwargs),
+              np.logspace(arg1, arg2, **kwargs),
+              msg='logspace({}, {})'.format(arg1, arg2))
+
+    run_test(0, 5)
+    run_test(0, 5, num=10)
+    run_test(0, 5, endpoint=False)
+    run_test(0, 5, base=2.0)
+    run_test(0, -5)
+    run_test(0, -5, num=10)
+    run_test(0, -5, endpoint=False)
+    run_test(0, -5, base=2.0)
+
+
+if __name__ == '__main__':
+  ops.enable_eager_execution()
+  test.main()
diff --git a/tensorflow/python/ops/numpy_ops/np_random.py b/tensorflow/python/ops/numpy_ops/np_random.py
new file mode 100644
index 00000000000..801a7549b97
--- /dev/null
+++ b/tensorflow/python/ops/numpy_ops/np_random.py
@@ -0,0 +1,56 @@
+# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Random functions."""
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import numpy as np
+
+from tensorflow.python.framework import random_seed
+from tensorflow.python.ops import random_ops
+from tensorflow.python.ops.numpy_ops import np_utils
+
+DEFAULT_RANDN_DTYPE = np.float32
+
+
+def randn(*args):
+  """Returns samples from a normal distribution.
+
+  Uses `tf.random_normal`.
+
+  Args:
+    *args: The shape of the output array.
+
+  Returns:
+    An ndarray with shape `args` and dtype `float64`.
+  """
+  # TODO(wangpeng): Use new stateful RNG
+  if np_utils.isscalar(args):
+    args = (args,)
+  return np_utils.tensor_to_ndarray(
+      random_ops.random_normal(args, dtype=DEFAULT_RANDN_DTYPE))
+
+
+def seed(s):
+  """Sets the seed for the random number generator.
+
+  Uses `tf.set_random_seed`.
+
+  Args:
+    s: an integer.
+  """
+  # TODO(wangpeng): make the signature the same as numpy
+  random_seed.set_seed(s)
diff --git a/tensorflow/python/ops/numpy_ops/np_random_test.py b/tensorflow/python/ops/numpy_ops/np_random_test.py
new file mode 100644
index 00000000000..3423b2234e8
--- /dev/null
+++ b/tensorflow/python/ops/numpy_ops/np_random_test.py
@@ -0,0 +1,91 @@
+# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Tests for tf numpy random number methods."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import numpy as np
+from six.moves import range
+
+from tensorflow.python.framework import ops
+# Needed for ndarray.reshape.
+from tensorflow.python.ops.numpy_ops import np_array_ops  # pylint: disable=unused-import
+from tensorflow.python.ops.numpy_ops import np_random
+from tensorflow.python.platform import test
+
+
+class RandomTest(test.TestCase):
+
+  def assertNotAllClose(self, a, b, **kwargs):
+    try:
+      self.assertAllClose(a, b, **kwargs)
+    except AssertionError:
+      return
+    raise AssertionError('The two values are close at all %d elements' %
+                         np.size(a))
+
+  def testRandN(self):
+
+    def run_test(*args):
+      num_samples = 1000
+      tol = 0.1  # High tolerance to keep the # of samples low else the test
+      # takes a long time to run.
+      np_random.seed(10)
+      outputs = [np_random.randn(*args) for _ in range(num_samples)]
+
+      # Test output shape.
+      for output in outputs:
+        self.assertEqual(output.shape, tuple(args))
+        self.assertEqual(output.dtype.type, np_random.DEFAULT_RANDN_DTYPE)
+
+      if np.prod(args):  # Don't bother with empty arrays.
+        outputs = [output.tolist() for output in outputs]
+
+        # Test that the properties of normal distribution are satisfied.
+        mean = np.mean(outputs, axis=0)
+        stddev = np.std(outputs, axis=0)
+        self.assertAllClose(mean, np.zeros(args), atol=tol)
+        self.assertAllClose(stddev, np.ones(args), atol=tol)
+
+        # Test that outputs are different with different seeds.
+        np_random.seed(20)
+        diff_seed_outputs = [
+            np_random.randn(*args).tolist() for _ in range(num_samples)
+        ]
+        self.assertNotAllClose(outputs, diff_seed_outputs)
+
+        # Test that outputs are the same with the same seed.
+        np_random.seed(10)
+        same_seed_outputs = [
+            np_random.randn(*args).tolist() for _ in range(num_samples)
+        ]
+        self.assertAllClose(outputs, same_seed_outputs)
+
+    run_test()
+    run_test(0)
+    run_test(1)
+    run_test(5)
+    run_test(2, 3)
+    run_test(0, 2, 3)
+    run_test(2, 0, 3)
+    run_test(2, 3, 0)
+    run_test(2, 3, 5)
+
+
+if __name__ == '__main__':
+  ops.enable_eager_execution()
+  test.main()
diff --git a/tensorflow/python/ops/numpy_ops/np_utils.py b/tensorflow/python/ops/numpy_ops/np_utils.py
new file mode 100644
index 00000000000..0fa3ab8e225
--- /dev/null
+++ b/tensorflow/python/ops/numpy_ops/np_utils.py
@@ -0,0 +1,466 @@
+# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Utility functions for internal use."""
+# pylint: disable=g-direct-tensorflow-import
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+import inspect
+import numpy as np
+
+from tensorflow.python.framework import dtypes
+from tensorflow.python.framework import indexed_slices
+from tensorflow.python.framework import ops
+from tensorflow.python.framework import tensor_util
+from tensorflow.python.ops import array_ops
+from tensorflow.python.ops import control_flow_ops
+from tensorflow.python.ops import math_ops
+from tensorflow.python.ops.numpy_ops import np_arrays
+from tensorflow.python.ops.numpy_ops import np_dtypes
+from tensorflow.python.util import nest
+
+tensor_to_ndarray = np_arrays.tensor_to_ndarray
+
+
+def _canonicalize_axis(axis, rank):
+  return _canonicalize_axes([axis], rank)[0]
+
+
+def _canonicalize_axes(axes, rank):
+  rank = _maybe_static(rank)
+
+  if isinstance(rank, ops.Tensor):
+    canonicalizer = (
+        lambda axis: cond(axis < 0, lambda: axis + rank, lambda: axis))
+  else:
+    canonicalizer = lambda axis: axis+rank if axis < 0 else axis
+
+  return [canonicalizer(axis) for axis in axes]
+
+
+def _supports_signature():
+  return hasattr(inspect, 'signature')
+
+
+def _to_tf_type(dtype):
+  """Converts a native python or numpy type to TF DType.
+
+  Args:
+    dtype: Could be a python type, a numpy type or a TF DType.
+
+  Returns:
+    A tensorflow `DType`.
+  """
+  return dtypes.as_dtype(dtype)
+
+
+def _to_numpy_type(dtype):
+  """Converts a native python or TF DType to numpy type.
+
+  Args:
+    dtype: Could be a python type, a numpy type or a TF DType.
+
+  Returns:
+    A NumPy `dtype`.
+  """
+  if isinstance(dtype, dtypes.DType):
+    return dtype.as_numpy_dtype
+  return np.dtype(dtype)
+
+
+def finfo(dtype):
+  """Returns properties of floating point types.
+
+  Note that currently it just forwards to the numpy namesake, while tensorflow
+  and numpy dtypes may have different properties.
+
+  Args:
+    dtype: Could be a python type, a numpy type or a TF DType.
+
+  Returns:
+    A class describing properties of `dtype`, as described by
+    https://docs.scipy.org/doc/numpy/reference/generated/numpy.finfo.html
+  """
+  return np.finfo(_to_numpy_type(dtype))
+
+
+def isscalar(val):
+  """Returns whether `val` is a scalar value or scalar Tensor."""
+  if isinstance(val, (np.ndarray, np_arrays.ndarray, ops.Tensor)):
+    return len(val.shape) == 0  # pylint: disable=g-explicit-length-test
+  return np.isscalar(val)
+
+
+# Can't use np_doc because np.result_type is a builtin function.
+def result_type(*arrays_and_dtypes):
+  """Returns the type resulting from applying NumPy type promotion to arguments.
+
+  Args:
+    *arrays_and_dtypes: A list of array_like objects or dtypes.
+
+  Returns:
+    A numpy dtype.
+  """
+
+  def maybe_get_dtype(x):
+    # Don't put np.ndarray in this list, because np.result_type looks at the
+    # value (not just dtype) of np.ndarray to decide the result type.
+    if isinstance(x, (np_arrays.ndarray, np_arrays.ShardedNdArray, ops.Tensor,
+                      indexed_slices.IndexedSlices)):
+      return _to_numpy_type(x.dtype)
+    elif isinstance(x, dtypes.DType):
+      return _to_numpy_type(x)
+    return x
+
+  arrays_and_dtypes = [
+      maybe_get_dtype(x) for x in nest.flatten(arrays_and_dtypes)
+  ]
+  if not arrays_and_dtypes:
+    # If arrays_and_dtypes is an empty list, let numpy decide what the dtype is.
+    arrays_and_dtypes = [np.asarray([])]
+  return np_dtypes._result_type(*arrays_and_dtypes)  # pylint: disable=protected-access
+
+
+def promote_types(type1, type2):
+  """Returns the type resulting from applying NumPy type promotion.
+
+  Args:
+    type1: A numpy type.
+    type2: A numpy type.
+
+  Returns:
+    A numpy type.
+  """
+  type1 = _to_numpy_type(type1)
+  type2 = _to_numpy_type(type2)
+  return np_dtypes.canonicalize_dtype(np.promote_types(type1, type2))
+
+
+def _has_docstring(f):
+  return (f and hasattr(f, '__doc__') and isinstance(f.__doc__, str) and
+          f.__doc__)
+
+
+def _add_blank_line(s):
+  if s.endswith('\n'):
+    return s + '\n'
+  else:
+    return s + '\n\n'
+
+
+def _np_signature(f):
+  """An enhanced inspect.signature that can handle numpy.ufunc."""
+  # TODO(wangpeng): consider migrating away from inspect.signature.
+  # inspect.signature is supported in Python 3.3.
+  if not hasattr(inspect, 'signature'):
+    return None
+  if f is None:
+    return None
+  if not isinstance(f, np.ufunc):
+    try:
+      return inspect.signature(f)
+    except ValueError:
+      return None
+
+  def names_from_num(prefix, n):
+    if n <= 0:
+      return []
+    elif n == 1:
+      return [prefix]
+    else:
+      return [prefix + str(i + 1) for i in range(n)]
+
+  input_names = names_from_num('x', f.nin)
+  output_names = names_from_num('out', f.nout)
+  keyword_only_params = [('where', True), ('casting', 'same_kind'),
+                         ('order', 'K'), ('dtype', None), ('subok', True),
+                         ('signature', None), ('extobj', None)]
+  params = []
+  params += [
+      inspect.Parameter(name, inspect.Parameter.POSITIONAL_ONLY)
+      for name in input_names
+  ]
+  if f.nout > 1:
+    params += [
+        inspect.Parameter(
+            name, inspect.Parameter.POSITIONAL_ONLY, default=None)
+        for name in output_names
+    ]
+  params += [
+      inspect.Parameter(
+          'out',
+          inspect.Parameter.POSITIONAL_OR_KEYWORD,
+          default=None if f.nout == 1 else (None,) * f.nout)
+  ]
+  params += [
+      inspect.Parameter(name, inspect.Parameter.KEYWORD_ONLY, default=default)
+      for name, default in keyword_only_params
+  ]
+  return inspect.Signature(params)
+
+
+# Python 2 doesn't allow keyword-only argument. Python prior to 3.8 doesn't
+# allow positional-only argument. So we conflate positional-only, keyword-only
+# and positional-or-keyword arguments here.
+def _is_compatible_param_kind(a, b):
+
+  def relax(k):
+    if k in (inspect.Parameter.POSITIONAL_ONLY, inspect.Parameter.KEYWORD_ONLY):
+      return inspect.Parameter.POSITIONAL_OR_KEYWORD
+    return k
+
+  return relax(a) == relax(b)
+
+
+def np_doc(np_fun, np_fun_name=None):
+  """Attachs numpy docstring to a function.
+
+  Args:
+    np_fun: the numpy function whose docstring will be used.
+    np_fun_name: optional name for the np_fun symbol. At least one of np_fun or
+      np_fun_name shoud be set.
+
+  Returns:
+    A function decorator that attaches the docstring from `np_fun` to the
+    decorated function.
+  """
+  if np_fun is None:
+    assert np_fun_name is not None
+    try:
+      np_fun = getattr(np, str(np_fun_name))
+    except AttributeError:
+      np_fun = None
+  np_sig = _np_signature(np_fun)
+  if np_fun_name is None:
+    assert np_fun is not None
+    np_fun_name = np_fun.__name__
+
+  def decorator(f):
+    """The decorator."""
+    unsupported_params = []
+    if hasattr(inspect, 'signature') and np_sig is not None:
+      try:
+        sig = inspect.signature(f)
+      except ValueError:
+        sig = None
+      # TODO(wangpeng): Enable this.
+      # Looks like this may not work with different versions of numpy.
+      # if sig is not None:
+      #   for name, param in sig.parameters.items():
+      #     np_param = np_sig.parameters.get(name)
+      #     if np_param is None:
+      #       raise TypeError('Cannot find parameter "%s" in the numpy
+      #          function\'s ' 'signature' % name)
+      #     if not _is_compatible_param_kind(param.kind, np_param.kind):
+      #       raise TypeError(
+      #           'Parameter "%s" is of kind %s while in numpy it is of '
+      #           'kind %s' % (name, param.kind, np_param.kind))
+      #     has_default = (param.default != inspect.Parameter.empty)
+      #     np_has_default = (np_param.default != inspect.Parameter.empty)
+      #     if has_default != np_has_default:
+      #       raise TypeError('Parameter "%s" should%s have a default value' %
+      #                       (name, '' if np_has_default else ' not'))
+      #   for name in np_sig.parameters:
+      #     if name not in sig.parameters:
+      #       unsupported_params.append(name)
+    f.__doc__ = _np_doc_helper(f, np_fun, np_fun_name=np_fun_name,
+                               unsupported_params=unsupported_params)
+    return f
+
+  return decorator
+
+
+def _np_doc_helper(f, np_f, np_fun_name=None, unsupported_params=None):
+  """Helper to get docs."""
+  if not unsupported_params and not _has_docstring(f) and _has_docstring(np_f):
+      # TODO(wangpeng): It looks like code snippets in numpy doc don't work
+      # correctly with doctest. Fix that and remove the reformatting of the np_f
+      # comment, here and below.
+    return np_f.__doc__.replace('>>>', '>')
+  assert np_f or np_fun_name
+  if not np_fun_name:
+    np_fun_name = np_f.__name__
+  doc = 'TensorFlow variant of `numpy.%s`.\n\n' % np_fun_name
+  if unsupported_params:
+    doc += 'Unsupported arguments: ' + ', '.join(
+        '`' + name + '`' for name in unsupported_params) + '.\n\n'
+  if _has_docstring(f):
+    doc += f.__doc__
+    doc = _add_blank_line(doc)
+  if _has_docstring(np_f):
+    doc += 'Documentation for `numpy.%s`:\n\n' % np_f.__name__
+    doc += np_f.__doc__.replace('>>>', '>')
+  return doc
+
+
+def np_doc_only(np_f):
+  """Attachs numpy docstring to a function.
+
+  This differs from np_doc in that it doesn't check for a match in signature.
+
+  Args:
+    np_f: the numpy function whose docstring will be used.
+
+  Returns:
+    A function decorator that attaches the docstring from `np_f` to the
+    decorated function.
+  """
+
+  def decorator(f):
+    f.__doc__ = _np_doc_helper(f, np_f)
+    return f
+
+  return decorator
+
+
+def tf_broadcast(*args):
+  """Broadcast tensors.
+
+  Args:
+    *args: a list of tensors whose shapes are broadcastable against each other.
+
+  Returns:
+    Tensors broadcasted to the common shape.
+  """
+  if len(args) <= 1:
+    return args
+  sh = array_ops.shape(args[0])
+  for arg in args[1:]:
+    sh = array_ops.broadcast_dynamic_shape(sh, array_ops.shape(arg))
+  return [array_ops.broadcast_to(arg, sh) for arg in args]
+
+
+# TODO(wangpeng): Move the following functions to a separate file and check for
+#   float dtypes in each of them.
+
+
+def get_static_value(x):
+  """A version of tf.get_static_value that returns None on float dtypes.
+
+  It returns None on float dtypes in order to avoid breaking gradients.
+
+  Args:
+    x: a tensor.
+
+  Returns:
+    Same as `tf.get_static_value`, except that it returns None when `x` has a
+    float dtype.
+  """
+  if isinstance(x, ops.Tensor) and (x.dtype.is_floating or x.dtype.is_complex):
+    return None
+  return tensor_util.constant_value(x)
+
+
+def _maybe_static(x):
+  value = get_static_value(x)
+  if value is None:
+    return x
+  else:
+    return value
+
+
+# All the following functions exist becaues get_static_value can't handle
+# their TF counterparts.
+
+
+def cond(pred, true_fn, false_fn):
+  """A version of tf.cond that tries to evaluate the condition."""
+  v = get_static_value(pred)
+  if v is None:
+    return control_flow_ops.cond(pred, true_fn, false_fn)
+  if v:
+    return true_fn()
+  else:
+    return false_fn()
+
+
+def add(a, b):
+  """A version of tf.add that eagerly evaluates if possible."""
+  return _maybe_static(a) + _maybe_static(b)
+
+
+def subtract(a, b):
+  """A version of tf.subtract that eagerly evaluates if possible."""
+  return _maybe_static(a) - _maybe_static(b)
+
+
+def greater(a, b):
+  """A version of tf.greater that eagerly evaluates if possible."""
+  return _maybe_static(a) > _maybe_static(b)
+
+
+def greater_equal(a, b):
+  """A version of tf.greater_equal that eagerly evaluates if possible."""
+  return _maybe_static(a) >= _maybe_static(b)
+
+
+def less_equal(a, b):
+  """A version of tf.less_equal that eagerly evaluates if possible."""
+  return _maybe_static(a) <= _maybe_static(b)
+
+
+def logical_and(a, b):
+  """A version of tf.logical_and that eagerly evaluates if possible."""
+  a_value = get_static_value(a)
+  if a_value is not None:
+    if np.isscalar(a_value):
+      if a_value:
+        return _maybe_static(b)
+      else:
+        return a_value
+    else:
+      return a_value & _maybe_static(b)
+  else:
+    return a & _maybe_static(b)
+
+
+def logical_or(a, b):
+  """A version of tf.logical_or that eagerly evaluates if possible."""
+  a_value = get_static_value(a)
+  if a_value is not None:
+    if np.isscalar(a_value):
+      if a_value:
+        return a_value
+      else:
+        return _maybe_static(b)
+    else:
+      return a_value | _maybe_static(b)
+  else:
+    return a | _maybe_static(b)
+
+
+def getitem(a, slice_spec):
+  """A version of __getitem__ that eagerly evaluates if possible."""
+  return _maybe_static(a)[slice_spec]
+
+
+def reduce_all(input_tensor, axis=None, keepdims=False):
+  """A version of tf.reduce_all that eagerly evaluates if possible."""
+  v = get_static_value(input_tensor)
+  if v is None:
+    return math_ops.reduce_all(input_tensor, axis=axis, keepdims=keepdims)
+  else:
+    return v.all(axis=axis, keepdims=keepdims)
+
+
+def reduce_any(input_tensor, axis=None, keepdims=False):
+  """A version of tf.reduce_any that eagerly evaluates if possible."""
+  v = get_static_value(input_tensor)
+  if v is None:
+    return math_ops.reduce_any(input_tensor, axis=axis, keepdims=keepdims)
+  else:
+    return v.any(axis=axis, keepdims=keepdims)
diff --git a/tensorflow/python/ops/numpy_ops/np_utils_test.py b/tensorflow/python/ops/numpy_ops/np_utils_test.py
new file mode 100644
index 00000000000..6d0dfa51185
--- /dev/null
+++ b/tensorflow/python/ops/numpy_ops/np_utils_test.py
@@ -0,0 +1,92 @@
+# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Tests for utils.py."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from tensorflow.python.ops.numpy_ops import np_utils
+from tensorflow.python.platform import test
+
+
+class UtilsTest(test.TestCase):
+
+  # pylint: disable=unused-argument
+  def testNpDoc(self):
+
+    def np_fun(x):
+      """np_fun docstring."""
+      return
+
+    @np_utils.np_doc(np_fun)
+    def f():
+      """f docstring."""
+      return
+
+    expected = """TensorFlow variant of `numpy.np_fun`.
+
+f docstring.
+
+Documentation for `numpy.np_fun`:
+
+np_fun docstring."""
+    self.assertEqual(expected, f.__doc__)
+
+  def testNpDocName(self):
+
+    @np_utils.np_doc(None, np_fun_name='foo')
+    def f():
+      """f docstring."""
+      return
+    expected = """TensorFlow variant of `numpy.foo`.
+
+f docstring.
+
+"""
+    self.assertEqual(expected, f.__doc__)
+
+  def testNpDocErrors(self):
+
+    self.skipTest('Enable once np signature checking is done.')
+    # if not np_utils._supports_signature():
+    #   self.skipTest("inspect.signature not supported")
+
+    def np_fun(x, y=1, **kwargs):
+      return
+
+    # pylint: disable=unused-variable
+    with self.assertRaisesRegexp(TypeError, 'Cannot find parameter'):
+
+      @np_utils.np_doc(np_fun)
+      def f1(a):
+        return
+
+    with self.assertRaisesRegexp(TypeError, 'is of kind'):
+
+      @np_utils.np_doc(np_fun)
+      def f2(x, kwargs):
+        return
+
+    with self.assertRaisesRegexp(TypeError,
+                                 'Parameter "y" should have a default value'):
+
+      @np_utils.np_doc(np_fun)
+      def f3(x, y):
+        return
+
+
+if __name__ == '__main__':
+  test.main()
diff --git a/tensorflow/python/platform/build_info_test.py b/tensorflow/python/platform/build_info_test.py
index f0df0b756cc..be253885715 100644
--- a/tensorflow/python/platform/build_info_test.py
+++ b/tensorflow/python/platform/build_info_test.py
@@ -25,8 +25,10 @@ from tensorflow.python.platform import test
 class BuildInfoTest(test.TestCase):
 
   def testBuildInfo(self):
-    self.assertEqual(build_info.is_rocm_build, test.is_built_with_rocm())
-    self.assertEqual(build_info.is_cuda_build, test.is_built_with_cuda())
+    self.assertEqual(build_info.build_info['is_rocm_build'],
+                     test.is_built_with_rocm())
+    self.assertEqual(build_info.build_info['is_cuda_build'],
+                     test.is_built_with_cuda())
 
 
 if __name__ == '__main__':
diff --git a/tensorflow/python/platform/self_check.py b/tensorflow/python/platform/self_check.py
index f6cf7705e13..c10c4108c7d 100644
--- a/tensorflow/python/platform/self_check.py
+++ b/tensorflow/python/platform/self_check.py
@@ -20,6 +20,7 @@ from __future__ import print_function
 
 import os
 
+MSVCP_DLL_NAMES = "msvcp_dll_names"
 
 try:
   from tensorflow.python.platform import build_info
@@ -42,9 +43,9 @@ def preload_check():
     # we load the Python extension, so that we can raise an actionable error
     # message if they are not found.
     import ctypes  # pylint: disable=g-import-not-at-top
-    if hasattr(build_info, "msvcp_dll_names"):
+    if MSVCP_DLL_NAMES in build_info.build_info:
       missing = []
-      for dll_name in build_info.msvcp_dll_names.split(","):
+      for dll_name in build_info.build_info[MSVCP_DLL_NAMES].split(","):
         try:
           ctypes.WinDLL(dll_name)
         except OSError:
diff --git a/tensorflow/python/platform/sysconfig.py b/tensorflow/python/platform/sysconfig.py
index 721ad99c60a..3c1a96ac06c 100644
--- a/tensorflow/python/platform/sysconfig.py
+++ b/tensorflow/python/platform/sysconfig.py
@@ -24,6 +24,7 @@ import platform as _platform
 from tensorflow.python.framework.versions import CXX11_ABI_FLAG as _CXX11_ABI_FLAG
 from tensorflow.python.framework.versions import MONOLITHIC_BUILD as _MONOLITHIC_BUILD
 from tensorflow.python.framework.versions import VERSION as _VERSION
+from tensorflow.python.platform import build_info
 from tensorflow.python.util.tf_export import tf_export
 
 
@@ -84,3 +85,28 @@ def get_link_flags():
     else:
       flags.append('-l:libtensorflow_framework.so.%s' % ver)
   return flags
+
+
+@tf_export('sysconfig.get_build_info')
+def get_build_info():
+  """Get a dictionary describing TensorFlow's build environment.
+
+  Values are generated when TensorFlow is compiled, and are static for each
+  TensorFlow package. The return value is a dictionary with string keys such as:
+
+    - cuda_version
+    - cudnn_version
+    - is_cuda_build
+    - is_rocm_build
+    - msvcp_dll_names
+    - nvcuda_dll_name
+    - cudart_dll_name
+    - cudnn_dll_name
+
+  Note that the actual keys and values returned by this function is subject to
+  change across different versions of TensorFlow or across platforms.
+
+  Returns:
+    A Dictionary describing TensorFlow's build environment.
+  """
+  return build_info.build_info
diff --git a/tensorflow/python/platform/sysconfig_test.py b/tensorflow/python/platform/sysconfig_test.py
new file mode 100644
index 00000000000..3e5956bf4f7
--- /dev/null
+++ b/tensorflow/python/platform/sysconfig_test.py
@@ -0,0 +1,38 @@
+# Copyright 2019 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from tensorflow.python.platform import googletest
+from tensorflow.python.platform import sysconfig
+from tensorflow.python.platform import test
+
+
+class SysconfigTest(googletest.TestCase):
+
+  def test_get_build_info_works(self):
+    build_info = sysconfig.get_build_info()
+    self.assertIsInstance(build_info, dict)
+
+  def test_rocm_cuda_info_matches(self):
+    build_info = sysconfig.get_build_info()
+    self.assertEqual(build_info["is_rocm_build"], test.is_built_with_rocm())
+    self.assertEqual(build_info["is_cuda_build"], test.is_built_with_cuda())
+
+
+if __name__ == "__main__":
+  googletest.main()
diff --git a/tensorflow/python/tpu/BUILD b/tensorflow/python/tpu/BUILD
index 422c106432e..96b4fda7aa4 100644
--- a/tensorflow/python/tpu/BUILD
+++ b/tensorflow/python/tpu/BUILD
@@ -515,6 +515,23 @@ tf_py_test(
     ],
 )
 
+tpu_py_test(
+    name = "tpu_outside_compilation_test",
+    srcs = [
+        "tpu_outside_compilation_test.py",
+    ],
+    disable_experimental = True,
+    python_version = "PY3",
+    tags = ["no_oss"],
+    deps = [
+        ":tpu_lib",
+        "//tensorflow/python:variables",
+        "//tensorflow/python/distribute/cluster_resolver:cluster_resolver_lib",
+        "//tensorflow/python/eager:remote",
+        "//tensorflow/python/eager:test",
+    ],
+)
+
 tf_proto_library(
     name = "tensor_tracer_proto",
     srcs = ["tensor_tracer.proto"],
diff --git a/tensorflow/python/tpu/bfloat16.py b/tensorflow/python/tpu/bfloat16.py
index 70f71815e51..9761d7f7a0e 100644
--- a/tensorflow/python/tpu/bfloat16.py
+++ b/tensorflow/python/tpu/bfloat16.py
@@ -70,11 +70,13 @@ def _get_custom_getter():
 
 @tf_export(v1=['tpu.bfloat16_scope'])
 @tf_contextlib.contextmanager
-def bfloat16_scope():
+def bfloat16_scope(name=None):
   """Scope class for bfloat16 variables so that the model uses custom getter.
 
   This enables variables to be read as bfloat16 type when using get_variable.
   """
+  if name is None:
+    name = ''
   with variable_scope.variable_scope(
-      '', custom_getter=_get_custom_getter()) as varscope:
+      name, custom_getter=_get_custom_getter()) as varscope:
     yield varscope
diff --git a/tensorflow/python/tpu/bfloat16_test.py b/tensorflow/python/tpu/bfloat16_test.py
index 78157ea86c2..148523a4580 100644
--- a/tensorflow/python/tpu/bfloat16_test.py
+++ b/tensorflow/python/tpu/bfloat16_test.py
@@ -24,15 +24,50 @@ from tensorflow.python.framework import test_util
 from tensorflow.python.ops import variable_scope
 from tensorflow.python.platform import test
 from tensorflow.python.tpu import bfloat16
+from tensorflow.python.framework import ops
+from tensorflow.python.ops import math_ops
+from tensorflow.python.ops import variables
 
 
 class BFloat16ScopeTest(test.TestCase):
 
-  def testScopeName(self):
+  def testDefaultScopeName(self):
     """Test if name for the variable scope is propagated correctly."""
     with bfloat16.bfloat16_scope() as bf:
       self.assertEqual(bf.name, "")
 
+  def testCustomScopeName(self):
+    """Test if custom name for the variable scope is propagated correctly."""
+    name = 'bfloat16'
+    with bfloat16.bfloat16_scope('bfloat16') as bf:
+      self.assertEqual(bf.name, name)
+
+  def testVariableName(self):
+    """Test if custom name for the variable scope is propagated correctly."""
+    g = ops.Graph()
+    with g.as_default():
+      a = variables.Variable(2.2, name='var_a')
+      b = variables.Variable(3.3, name='var_b')
+      d = variables.Variable(4.4, name='var_b')
+      with g.name_scope('scope1'):
+        with bfloat16.bfloat16_scope('bf16'):
+          a = math_ops.cast(a, dtypes.bfloat16)
+          b = math_ops.cast(b, dtypes.bfloat16)
+          c = math_ops.add(a, b, name='addition')
+        with bfloat16.bfloat16_scope():
+          d = math_ops.cast(d, dtypes.bfloat16)
+          math_ops.add(c, d, name='addition')
+
+    g_ops = g.get_operations()
+    ops_name = []
+    for op in g_ops:
+      ops_name.append(str(op.name))
+
+    self.assertIn('scope1/bf16/addition', ops_name)
+    self.assertIn('scope1/bf16/Cast', ops_name)
+    self.assertIn('scope1/addition', ops_name)
+    self.assertIn('scope1/Cast', ops_name)
+
   @test_util.run_deprecated_v1
   def testRequestedDType(self):
     """Test if requested dtype is honored in the getter.
diff --git a/tensorflow/python/tpu/tensor_tracer.py b/tensorflow/python/tpu/tensor_tracer.py
index b4f99897094..c0536d84182 100644
--- a/tensorflow/python/tpu/tensor_tracer.py
+++ b/tensorflow/python/tpu/tensor_tracer.py
@@ -1844,7 +1844,7 @@ class TensorTracer(object):
           if len(processed_tensors) != 1:
             raise RuntimeError('Multiple stats are only allowed in compact '
                                'mode.')
-          processed_out_tensor = processed_tensors.values()[0]
+          processed_out_tensor = list(processed_tensors.values())[0]
           # Store the whole tensor in a buffer.
           trace_op = self._snapshot_tensor(processed_out_tensor)
         else:
diff --git a/tensorflow/python/tpu/tpu_embedding_v2.py b/tensorflow/python/tpu/tpu_embedding_v2.py
index 5a66f6ce8b9..d82dcc616c6 100644
--- a/tensorflow/python/tpu/tpu_embedding_v2.py
+++ b/tensorflow/python/tpu/tpu_embedding_v2.py
@@ -696,17 +696,33 @@ class TPUEmbedding(tracking.AutoTrackable):
       """Create all variables."""
       shape = (table.vocabulary_size, table.dim)
 
-      # We use functools.partial here for the initial_value so that we have a
-      # variable creation that is compatible with both the sharded variable
-      # creator and the normal variable creator. The sharded variable creator
-      # will extract the shape of the tensor from the functool.partial object to
-      # decide on the sharding.
-      parameters = tf_variables.Variable(
-          name=table.name,
-          initial_value=functools.partial(
-              table.initializer, shape=shape, dtype=dtypes.float32),
-          trainable=not self._using_tpu)
-      slot_vars = table.optimizer._create_slots(parameters)  # pylint: disable=protected-access
+      def getter(name, shape, dtype, initializer, trainable):
+        return tf_variables.Variable(
+            name=name,
+            initial_value=functools.partial(initializer, shape, dtype=dtype),
+            trainable=trainable)
+
+      def variable_creator(name, initializer, trainable=True):
+        # use add_variable_with_custom_getter here so that we take advantage of
+        # the checkpoint loading to allow restore before the variables get
+        # created which avoids double initialization.
+        return self._add_variable_with_custom_getter(
+            name=name,
+            initializer=initializer,
+            shape=shape,
+            dtype=dtypes.float32,
+            getter=getter,
+            trainable=trainable)
+
+      parameters = variable_creator(table.name, table.initializer,
+                                    trainable=not self._using_tpu)
+
+      def slot_creator(name, initializer):
+        return variable_creator(table.name + "/" + name,
+                                initializer,
+                                False)
+
+      slot_vars = table.optimizer._create_slots(parameters, slot_creator)  # pylint: disable=protected-access
       slot_vars["parameters"] = parameters
       return slot_vars
 
diff --git a/tensorflow/python/tpu/tpu_embedding_v2_test.py b/tensorflow/python/tpu/tpu_embedding_v2_test.py
index 78b5c9fa3bc..a8b21480919 100644
--- a/tensorflow/python/tpu/tpu_embedding_v2_test.py
+++ b/tensorflow/python/tpu/tpu_embedding_v2_test.py
@@ -39,6 +39,7 @@ from tensorflow.python.framework import dtypes
 from tensorflow.python.framework import ops
 from tensorflow.python.framework import sparse_tensor
 from tensorflow.python.framework import tensor_spec
+from tensorflow.python.module import module
 from tensorflow.python.ops import array_ops
 from tensorflow.python.ops import gen_math_ops
 from tensorflow.python.ops import init_ops_v2
@@ -56,7 +57,6 @@ from tensorflow.python.training import checkpoint_utils
 from tensorflow.python.training.tracking import util
 from tensorflow.python.util import nest
 
-
 FLAGS = flags.FLAGS
 flags.DEFINE_string('tpu', '', 'Name of TPU to connect to.')
 flags.DEFINE_string('project', None, 'Name of GCP project with TPU.')
@@ -161,6 +161,60 @@ class TPUEmbeddingCheckpointTest(parameterized.TestCase, test.TestCase):
         msg='Second mid level api should have retrieved the first model values.'
     )
 
+  def test_checkpoint_restore_before_variable_creation(self):
+
+    class TestModule(module.Module):
+
+      def __init__(self, initializer, rows):
+        self._initializer = initializer
+        self._rows = rows
+
+      def create_embedding(self):
+        table = tpu_embedding_v2_utils.TableConfig(
+            vocabulary_size=self._rows, dim=4, initializer=self._initializer,
+            combiner='sum', name='table')
+        feature_config = (tpu_embedding_v2_utils.FeatureConfig(
+            table=table, name='feature'),)
+        optimizer = tpu_embedding_v2_utils.SGD()
+
+        self.tpu_embedding = tpu_embedding_v2.TPUEmbedding(
+            feature_config, self._rows, optimizer)
+
+    # We need to clear the already loaded config provided by setUp method.
+    tpu_strategy_util.initialize_tpu_system(self.resolver)
+
+    with self.strategy.scope():
+      module1 = TestModule(init_ops_v2.Ones(),
+                           self.strategy.num_replicas_in_sync * 2)
+      module1.create_embedding()
+
+    checkpoint = util.Checkpoint(test_module=module1)
+    checkpoint.save(_get_tmpdir('restore_before_create', 'save'))
+
+    tpu_strategy_util.initialize_tpu_system(self.resolver)
+
+    with self.strategy.scope():
+      module2 = TestModule(init_ops_v2.Zeros(),
+                           self.strategy.num_replicas_in_sync * 2)
+
+    checkpoint = util.Checkpoint(test_module=module2)
+    checkpoint.restore(_get_tmpdir('restore_before_create', 'save-1'))
+
+    with self.strategy.scope():
+      module2.create_embedding()
+
+    def get_values(mid):
+      return mid._variables['table']['parameters'].variables[0].numpy()
+
+    self.assertAllClose(np.ones((self.strategy.num_replicas_in_sync * 2, 4)),
+                        get_values(module2.tpu_embedding))
+
+    # Fetch the values from the TPU to check that they are the same.
+    module2.tpu_embedding._retrieve_variables()
+
+    self.assertAllClose(np.ones((self.strategy.num_replicas_in_sync * 2, 4)),
+                        get_values(module2.tpu_embedding))
+
   def build_mid_level(self, embedding_values, optimizer,
                       initialize_tpu_embedding=True):
     """Creates an embedding api object initialized to embedding_values."""
@@ -172,7 +226,7 @@ class TPUEmbeddingCheckpointTest(parameterized.TestCase, test.TestCase):
     feature_config = (tpu_embedding_v2_utils.FeatureConfig(
         table=table, name='feature'),)
 
-    # batch_size here does not matter as we aren't traininig in any of these
+    # batch_size here does not matter as we aren't training in any of these
     # tests.
     return tpu_embedding_v2.TPUEmbedding(
         feature_config, 64, optimizer,
diff --git a/tensorflow/python/tpu/tpu_embedding_v2_utils.py b/tensorflow/python/tpu/tpu_embedding_v2_utils.py
index bba0d10a62f..9d7de203889 100644
--- a/tensorflow/python/tpu/tpu_embedding_v2_utils.py
+++ b/tensorflow/python/tpu/tpu_embedding_v2_utils.py
@@ -20,13 +20,11 @@ from __future__ import print_function
 from __future__ import unicode_literals
 
 import abc
-import functools
 import math
 import six
 
 from tensorflow.core.protobuf.tpu import optimization_parameters_pb2
 from tensorflow.python.ops import init_ops_v2
-from tensorflow.python.ops import variables as tf_variables
 from tensorflow.python.tpu.ops import tpu_ops
 from tensorflow.python.util.tf_export import tf_export
 
@@ -101,13 +99,13 @@ class _Optimizer(object):
     """Returns the retrieve function for the optimizer."""
     raise NotImplementedError
 
-  def _create_slots(self, table):
+  def _create_slots(self, table, variable_creator):
     """Creates slot variables for table.
 
-    Uses shape of table to create parallel slot variables.
-
     Args:
-      table: A Variable or equivalent.
+      table: The table variable to create slots for.
+      variable_creator: A function which creates variables. Takes parameters
+        'name', 'initializer'.
 
     Returns:
       A dict of variables, keyed by self._slot_names().
@@ -118,11 +116,7 @@ class _Optimizer(object):
       slots = {}
       for slot, initializer in zip(self._slot_names(),
                                    self._slot_initializers()):
-        slots[slot] = tf_variables.Variable(
-            name=table.name + "/" + slot,
-            initial_value=functools.partial(
-                initializer, shape=table.shape, dtype=table.dtype),
-            trainable=False)
+        slots[slot] = variable_creator(name=slot, initializer=initializer)
       return slots
 
 
diff --git a/tensorflow/python/tpu/tpu_outside_compilation_test.py b/tensorflow/python/tpu/tpu_outside_compilation_test.py
new file mode 100644
index 00000000000..f7ecb294c44
--- /dev/null
+++ b/tensorflow/python/tpu/tpu_outside_compilation_test.py
@@ -0,0 +1,170 @@
+# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+"""Tests for TPU outside compilation."""
+
+from __future__ import absolute_import
+from __future__ import division
+from __future__ import print_function
+
+from tensorflow.python.distribute import tpu_strategy as tpu_lib
+from tensorflow.python.distribute.cluster_resolver import tpu_cluster_resolver
+from tensorflow.python.eager import def_function
+from tensorflow.python.eager import remote
+from tensorflow.python.eager import test
+from tensorflow.python.framework import constant_op
+from tensorflow.python.ops import logging_ops
+from tensorflow.python.ops import variables
+from tensorflow.python.platform import flags
+from tensorflow.python.tpu import tpu
+from tensorflow.python.tpu import tpu_strategy_util
+
+FLAGS = flags.FLAGS
+flags.DEFINE_string("tpu", "", "Name of TPU to connect to.")
+flags.DEFINE_string("project", None, "Name of GCP project with TPU.")
+flags.DEFINE_string("zone", None, "Name of GCP zone with TPU.")
+
+
+def get_tpu_cluster_resolver():
+  resolver = tpu_cluster_resolver.TPUClusterResolver(
+      tpu=FLAGS.tpu,
+      zone=FLAGS.zone,
+      project=FLAGS.project,
+  )
+  return resolver
+
+
+def get_tpu_strategy():
+  resolver = get_tpu_cluster_resolver()
+  remote.connect_to_cluster(resolver)
+  tpu_strategy_util.initialize_tpu_system(resolver)
+  return tpu_lib.TPUStrategy(resolver)
+
+
+class TpuOutsideCompilationTest(test.TestCase):
+
+  def testResourceVariableAssignOnHost(self):
+    strategy = get_tpu_strategy()
+    with strategy.scope():
+      v = variables.Variable(
+          0.0, aggregation=variables.VariableAggregation.MEAN)
+    v2 = variables.Variable(0.0, aggregation=variables.VariableAggregation.MEAN)
+
+    def assign_fn():
+      v2.assign_add(4.0)
+
+    @def_function.function
+    def train_step():
+
+      def assign_add():
+        v.assign_add(2.0)
+        tpu.outside_compilation(assign_fn)
+        v.assign_add(3.0)
+
+      strategy.run(assign_add)
+      return
+
+    train_step()
+    self.assertAllEqual(4.0 * strategy.num_replicas_in_sync, v2.numpy())
+    self.assertAllEqual(5.0, v.numpy())
+
+  def testHostInputOnly(self):
+    strategy = get_tpu_strategy()
+
+    def outside_fn(x):
+      logging_ops.print_v2("Outside compiled", x)
+
+    @def_function.function
+    def train_step():
+
+      def tpu_fn(x):
+        x2 = x + 5.0
+        tpu.outside_compilation(outside_fn, x2)
+        return x2 + 5.0
+
+      return strategy.run(tpu_fn, args=(25.0,))
+
+    self.assertAllEqual(
+        strategy.experimental_local_results(train_step()),
+        constant_op.constant(35., shape=(strategy.num_replicas_in_sync)))
+
+  def testHostInputOutput(self):
+    strategy = get_tpu_strategy()
+
+    def outside_fn(x):
+      logging_ops.print_v2("Outside compiled", x)
+      return x + 6.0
+
+    @def_function.function
+    def train_step():
+
+      def tpu_fn(x):
+        x2 = x + 5.0
+        output = tpu.outside_compilation(outside_fn, x2)
+        return output
+
+      return strategy.run(tpu_fn, args=(25.0,))
+
+    self.assertAllEqual(
+        strategy.experimental_local_results(train_step()),
+        constant_op.constant(36., shape=(strategy.num_replicas_in_sync)))
+
+  def testOutsideCompilationControlFlowIf(self):
+    strategy = get_tpu_strategy()
+
+    def outside_fn(x):
+      logging_ops.print_v2("Outside compiled", x)
+      return x + 6.0
+
+    @def_function.function
+    def train_step():
+
+      def tpu_fn(x):
+        x2 = x + 5.0
+        if x < 50.0:
+          return tpu.outside_compilation(outside_fn, x2)
+        else:
+          return x2
+
+      return strategy.run(tpu_fn, args=(25.0,))
+
+    self.assertAllEqual(
+        strategy.experimental_local_results(train_step()),
+        constant_op.constant(36., shape=(strategy.num_replicas_in_sync)))
+
+  def testOutsideCompilationControlFlowWhile(self):
+    strategy = get_tpu_strategy()
+
+    def outside_fn(x):
+      logging_ops.print_v2("Outside compiled", x)
+      return x + 6.0
+
+    @def_function.function
+    def train_step():
+
+      def tpu_fn(x):
+        x2 = x + 5.0
+        while x2 < 50.0:
+          x2 = tpu.outside_compilation(outside_fn, x2)
+        return x2 + 4.0
+
+      return strategy.run(tpu_fn, args=(25.0,))
+
+    self.assertAllEqual(
+        strategy.experimental_local_results(train_step()),
+        constant_op.constant(58., shape=(strategy.num_replicas_in_sync)))
+
+
+if __name__ == "__main__":
+  test.main()
diff --git a/tensorflow/python/training/quantize_training_test.py b/tensorflow/python/training/quantize_training_test.py
index 813ea2416bf..6e5b2f9e641 100644
--- a/tensorflow/python/training/quantize_training_test.py
+++ b/tensorflow/python/training/quantize_training_test.py
@@ -42,8 +42,8 @@ class PywrapQuantizeTrainingTest(test.TestCase):
       b = constant_op.constant(7.0, shape=[1, 1])
       c = math_ops.matmul(a, b, name='matmul')
 
-      self.assertEquals(c.eval(), 42.0)
-      self.assertEquals(len(sess.graph_def.node), 3)
+      self.assertEqual(c.eval(), 42.0)
+      self.assertEqual(len(sess.graph_def.node), 3)
 
       result = quantize_training.do_quantize_training_on_graphdef(
           sess.graph_def, 8)
@@ -53,7 +53,7 @@ class PywrapQuantizeTrainingTest(test.TestCase):
 
   # Test that save/restoring works for EMA variables generated in the
   # quantized training rewrite.
-  @test_util.run_v1_only('b/120545219')
+  @test_util.run_v1_only('The API is only expect to work with v1 session mode.')
   def testQuantizedSaveRestore(self):
     save_path = os.path.join(self.get_temp_dir(), 'quantized_save_restore')
 
@@ -89,13 +89,13 @@ class PywrapQuantizeTrainingTest(test.TestCase):
       # When we restore the saved variabled, the quantization variables should
       # be restored as well.
       saver.restore(sess, save_path)
-      self.assertEquals(7.0, sess.run(g.get_tensor_by_name('b:0')))
-      self.assertEquals(6.0, sess.run(g.get_tensor_by_name('a/Min/Variable:0')))
-      self.assertEquals(6.0, sess.run(g.get_tensor_by_name('a/Max/Variable:0')))
-      self.assertEquals(7.0,
-                        sess.run(g.get_tensor_by_name('b/read/Min/Variable:0')))
-      self.assertEquals(7.0,
-                        sess.run(g.get_tensor_by_name('b/read/Max/Variable:0')))
+      self.assertEqual(7.0, sess.run(g.get_tensor_by_name('b:0')))
+      self.assertEqual(6.0, sess.run(g.get_tensor_by_name('a/Min/Variable:0')))
+      self.assertEqual(6.0, sess.run(g.get_tensor_by_name('a/Max/Variable:0')))
+      self.assertEqual(7.0,
+                       sess.run(g.get_tensor_by_name('b/read/Min/Variable:0')))
+      self.assertEqual(7.0,
+                       sess.run(g.get_tensor_by_name('b/read/Max/Variable:0')))
 
 
 if __name__ == '__main__':
diff --git a/tensorflow/python/util/tf_stack.cc b/tensorflow/python/util/tf_stack.cc
index 5c18c87ad32..aa9be6305ce 100644
--- a/tensorflow/python/util/tf_stack.cc
+++ b/tensorflow/python/util/tf_stack.cc
@@ -83,7 +83,8 @@ std::vector<FrameSummary> ExtractStack(ssize_t limit, const py::list& mappers,
   std::vector<FrameSummary> ret;
   // 16 is somewhat arbitrary, but TensorFlow stack traces tend to be deep.
   ret.reserve(limit < 0 ? 16 : static_cast<size_t>(limit));
-  for (; f != nullptr && (limit < 0 || ret.size() < limit); f = f->f_back) {
+  for (; f != nullptr && (limit < 0 || ret.size() < static_cast<size_t>(limit));
+       f = f->f_back) {
     const PyCodeObject* co = f->f_code;
     int lineno = PyFrame_GetLineNumber(const_cast<PyFrameObject*>(f));
     auto filename = py::reinterpret_borrow<py::str>(co->co_filename);
diff --git a/tensorflow/security/fuzzing/BUILD b/tensorflow/security/fuzzing/BUILD
new file mode 100644
index 00000000000..887e1a23cdf
--- /dev/null
+++ b/tensorflow/security/fuzzing/BUILD
@@ -0,0 +1,17 @@
+# Fuzzing TensorFlow.
+# Since we use OSSFuzz, gather all fuzzers into a single place.
+# This way, we can use tooling to determine the status of the fuzzing efforts.
+
+load(
+    "//tensorflow/security/fuzzing:tf_fuzzing.bzl",
+    "tf_fuzz_target",
+)
+
+package(
+    licenses = ["notice"],  # Apache 2.0
+)
+
+tf_fuzz_target(
+    name = "demo_fuzz",
+    srcs = ["demo_fuzz.cc"],
+)
diff --git a/tensorflow/security/fuzzing/demo_fuzz.cc b/tensorflow/security/fuzzing/demo_fuzz.cc
new file mode 100644
index 00000000000..71777591694
--- /dev/null
+++ b/tensorflow/security/fuzzing/demo_fuzz.cc
@@ -0,0 +1,32 @@
+/* Copyright 2020 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+#include <cstdint>
+#include <cstdlib>
+
+// This is a demo fuzzer to test that the entire framework functions correctly.
+// Once we start moving the existing fuzzers to this framework we will delete
+// this.
+// TODO(mihaimaruseac): Delete this when no longer needed
+void DemoFuzzer(const uint8_t* data, size_t size) {
+  // Trigger a small bug that should be found by the fuzzer quite quickly
+  if (size > 10 && size % 3 == 2)
+    if (data[0] > data[1])
+      if (data[5] % data[2] == data[3]) abort();
+}
+
+extern "C" int LLVMFuzzerTestOneInput(const uint8_t* data, size_t size) {
+  DemoFuzzer(data, size);
+  return 0;
+}
diff --git a/tensorflow/security/fuzzing/tf_fuzzing.bzl b/tensorflow/security/fuzzing/tf_fuzzing.bzl
new file mode 100644
index 00000000000..63a8cbac704
--- /dev/null
+++ b/tensorflow/security/fuzzing/tf_fuzzing.bzl
@@ -0,0 +1,80 @@
+"""Definitions for rules to fuzz TensorFlow."""
+
+# TensorFlow fuzzing can be done in open source too.
+#
+# For a fuzzer ${FUZZ} we have the following setup:
+#   - ${FUZZ}_fuzz.cc           : the implementation of the fuzzer
+#   - corpus/${FUZZ}/...        : public corpus for the fuzzer
+#   - dictionaries/${FUZZ}.dict : fuzzing dictionary for the fuzzer
+#   - ${FUZZ}_internal/...      : internal data for the fuzzer
+#
+# If a fuzzer needs some framework to build, we can use the ${FUZZ}_internal/
+# directory to hold the harness. Or, if the infrastructure needs to be shared
+# across multiple fuzzers (for example fuzzing ops), we can store it in other
+# places in TF or move it to a different folder here. We will decide on these
+# on a case by case basis, for now the ops fuzzing harness resides under
+# tensorflow/core/kernels/fuzzing.
+#
+# The internal folder can also contain proto definitions (if using proto-based
+# mutators to do structure aware fuzzing) or any other type of content that is
+# not classified elsewhere.
+
+# tf_cc_fuzz_target is a cc_test modified to include fuzzing support.
+def tf_fuzz_target(
+        name,
+        # Fuzzing specific arguments
+        fuzzing_dict = [],
+        corpus = [],
+        parsers = [],
+        # Reporting bugs arguments, not used in open source
+        componentid = None,
+        hotlists = [],
+        # Additional cc_test control
+        data = [],
+        deps = [],
+        tags = [],
+        # Remaining cc_test arguments
+        **kwargs):
+    """Specify how to build a TensorFlow fuzz target.
+
+    Args:
+      name: Mandatory name of the fuzzer target.
+
+      fuzzing_dict: An optional a set of dictionary files following
+        the AFL/libFuzzer dictionary syntax.
+
+      corpus: An optional set of files used as the initial test corpus
+        for the target. When doing "bazel test" in the default null-fuzzer
+        (unittest) mode, these files are automatically passed to the target
+        function.
+
+      parsers: An optional list of file extensions that the target supports.
+        Used by tools like autofuzz to reuse corpus sets across targets.
+
+      componentid: Used internally for reporting fuzz discovered bugs.
+
+      hotlists: Used internally for reporting fuzz discovered bugs.
+
+      data: Additional data dependencies passed to the underlying cc_test rule.
+
+      deps: An optional list of dependencies for the code you're fuzzing.
+
+      tags: Additional tags passed to the underlying cc_test rule.
+
+      **kwargs: Collects all remaining arguments and passes them to the
+        underlying cc_test rule generated by the macro.
+    """
+    componentid = None
+    hotlists = None
+
+    # Fuzzers in open source must be run manually
+    tags = tags + ["manual"]
+
+    # Now, redirect to cc_test
+    native.cc_test(
+        name = name,
+        deps = deps,  # TODO(mihaimaruseac): fuzzing lib?
+        data = data,  # TODO(mihaimaruseac): dict, corpus, parsers??
+        tags = tags,  # TODO(mihaimaruseac): fuzzing tags?
+        **kwargs
+    )
diff --git a/tensorflow/stream_executor/cuda/cuda_driver.cc b/tensorflow/stream_executor/cuda/cuda_driver.cc
index 3d011123d87..e30eb549a9c 100644
--- a/tensorflow/stream_executor/cuda/cuda_driver.cc
+++ b/tensorflow/stream_executor/cuda/cuda_driver.cc
@@ -308,9 +308,12 @@ static port::Status InternalInit() {
 
   if (res == CUDA_SUCCESS) {
     return port::Status::OK();
+  } else if (res == CUDA_ERROR_SHARED_OBJECT_INIT_FAILED) {
+    LOG(WARNING) << "failed call to cuInit: " << ToString(res);
+  } else {
+    LOG(ERROR) << "failed call to cuInit: " << ToString(res);
   }
 
-  LOG(ERROR) << "failed call to cuInit: " << ToString(res);
   Diagnostician::LogDiagnosticInformation();
   return port::Status(port::error::ABORTED,
                       absl::StrCat("failed call to cuInit: ", ToString(res)));
diff --git a/tensorflow/stream_executor/lib/statusor.h b/tensorflow/stream_executor/lib/statusor.h
index 7ba0954476b..738abf95893 100644
--- a/tensorflow/stream_executor/lib/statusor.h
+++ b/tensorflow/stream_executor/lib/statusor.h
@@ -92,7 +92,8 @@ class StatusOr : private internal_statusor::StatusOrData<T>,
   typedef internal_statusor::StatusOrData<T> Base;
 
  public:
-  typedef T element_type;
+  typedef T element_type;  // DEPRECATED: use `value_type`.
+  typedef T value_type;
 
   // Constructs a new StatusOr with Status::UNKNOWN status.  This is marked
   // 'explicit' to try to catch cases like 'return {};', where people think
diff --git a/tensorflow/tensorflow.bzl b/tensorflow/tensorflow.bzl
index 03c561f4fc1..2730685a26f 100644
--- a/tensorflow/tensorflow.bzl
+++ b/tensorflow/tensorflow.bzl
@@ -2615,6 +2615,10 @@ def tf_version_info_genrule(name, out):
         arguments = "--generate \"$@\" --git_tag_override=${GIT_TAG_OVERRIDE:-}",
     )
 
+def _dict_to_kv(d):
+    """Convert a dictionary to a space-joined list of key=value pairs."""
+    return " " + " ".join(["%s=%s" % (k, v) for k, v in d.items()])
+
 def tf_py_build_info_genrule(name, out):
     _local_genrule(
         name = name,
@@ -2622,16 +2626,16 @@ def tf_py_build_info_genrule(name, out):
         exec_tool = "//tensorflow/tools/build_info:gen_build_info",
         arguments =
             "--raw_generate \"$@\" " +
-            " --is_config_cuda " + if_cuda("True", "False") +
-            " --is_config_rocm " + if_rocm("True", "False") +
-            " --key_value " +
-            if_cuda(" cuda_version_number=${TF_CUDA_VERSION:-} cudnn_version_number=${TF_CUDNN_VERSION:-} ", "") +
-            if_windows(" msvcp_dll_names=msvcp140.dll,msvcp140_1.dll ", "") +
-            if_windows_cuda(" ".join([
-                "nvcuda_dll_name=nvcuda.dll",
-                "cudart_dll_name=cudart64_$(echo $${TF_CUDA_VERSION:-} | sed \"s/\\.//\").dll",
-                "cudnn_dll_name=cudnn64_${TF_CUDNN_VERSION:-}.dll",
-            ]), ""),
+            " --key_value" +
+            " is_rocm_build=" + if_rocm("True", "False") +
+            " is_cuda_build=" + if_cuda("True", "False") +
+            if_windows(_dict_to_kv({
+                "msvcp_dll_names": "msvcp140.dll,msvcp140_1.dll",
+            }), "") + if_windows_cuda(_dict_to_kv({
+                "nvcuda_dll_name": "nvcuda.dll",
+                "cudart_dll_name": "cudart{cuda_version}.dll",
+                "cudnn_dll_name": "cudnn{cudnn_version}.dll",
+            }), ""),
     )
 
 def cc_library_with_android_deps(
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.-model.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.-model.pbtxt
index d696021fcb4..ea2945b5bf6 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.-model.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.-model.pbtxt
@@ -108,6 +108,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.-sequential.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.-sequential.pbtxt
index b8486a27b9e..c1dea9335c0 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.-sequential.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.-sequential.pbtxt
@@ -110,6 +110,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.experimental.-linear-model.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.experimental.-linear-model.pbtxt
index 7bf71844fa6..ba87c0a2a7a 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.experimental.-linear-model.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.experimental.-linear-model.pbtxt
@@ -109,6 +109,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.experimental.-peephole-l-s-t-m-cell.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.experimental.-peephole-l-s-t-m-cell.pbtxt
index ee5e5b884a2..e366d0b1f52 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.experimental.-peephole-l-s-t-m-cell.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.experimental.-peephole-l-s-t-m-cell.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.experimental.-sequence-features.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.experimental.-sequence-features.pbtxt
index e2bef6beaaa..2b4ebd55410 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.experimental.-sequence-features.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.experimental.-sequence-features.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.experimental.-wide-deep-model.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.experimental.-wide-deep-model.pbtxt
index 87a7319639b..37fb2051f81 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.experimental.-wide-deep-model.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.experimental.-wide-deep-model.pbtxt
@@ -109,6 +109,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-abstract-r-n-n-cell.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-abstract-r-n-n-cell.pbtxt
index 8c80da861f6..363e4b7cf20 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-abstract-r-n-n-cell.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-abstract-r-n-n-cell.pbtxt
@@ -95,6 +95,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-activation.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-activation.pbtxt
index ef8efd606c4..409dfcd26e4 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-activation.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-activation.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-activity-regularization.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-activity-regularization.pbtxt
index 60578d2cc59..599d550fee2 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-activity-regularization.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-activity-regularization.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-add.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-add.pbtxt
index 3bd1f2c7623..b43eee1e6e5 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-add.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-add.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-additive-attention.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-additive-attention.pbtxt
index f6f8d3914b4..2edb8e028c1 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-additive-attention.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-additive-attention.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-alpha-dropout.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-alpha-dropout.pbtxt
index 3e408e96036..4f7c7d6ca27 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-alpha-dropout.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-alpha-dropout.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-attention.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-attention.pbtxt
index 4197c1a88f6..e335099c084 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-attention.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-attention.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-average-pooling1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-average-pooling1-d.pbtxt
index 153a801e1d8..80c8e0d63c1 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-average-pooling1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-average-pooling1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-average-pooling2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-average-pooling2-d.pbtxt
index 66e261111ae..4e95f083490 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-average-pooling2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-average-pooling2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-average-pooling3-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-average-pooling3-d.pbtxt
index b247490b067..370589fb876 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-average-pooling3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-average-pooling3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-average.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-average.pbtxt
index 0f1808332c5..8abc0add0dc 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-average.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-average.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-avg-pool1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-avg-pool1-d.pbtxt
index 567143eb41d..6938e84fe77 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-avg-pool1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-avg-pool1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-avg-pool2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-avg-pool2-d.pbtxt
index 56a2db85419..5ca4fd39173 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-avg-pool2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-avg-pool2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-avg-pool3-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-avg-pool3-d.pbtxt
index c0ab32fd7c2..378bf0e84a0 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-avg-pool3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-avg-pool3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-batch-normalization.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-batch-normalization.pbtxt
index 5ff17a5b422..f5ef78bc6d7 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-batch-normalization.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-batch-normalization.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-bidirectional.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-bidirectional.pbtxt
index ccb55ec0c52..60402f72e77 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-bidirectional.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-bidirectional.pbtxt
@@ -92,6 +92,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-concatenate.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-concatenate.pbtxt
index c44ff9e48e2..c6733e7e8ab 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-concatenate.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-concatenate.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv-l-s-t-m2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv-l-s-t-m2-d.pbtxt
index 43112cfe785..f77d613e354 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv-l-s-t-m2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv-l-s-t-m2-d.pbtxt
@@ -169,6 +169,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv1-d-transpose.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv1-d-transpose.pbtxt
index 3a592d713bc..03ccf45ea84 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv1-d-transpose.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv1-d-transpose.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv1-d.pbtxt
index 3319122f50b..f41ff3f2136 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv2-d-transpose.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv2-d-transpose.pbtxt
index 535243a2224..2370f378e57 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv2-d-transpose.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv2-d-transpose.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv2-d.pbtxt
index 30736595ce5..82caa2a7b1d 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv3-d-transpose.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv3-d-transpose.pbtxt
index afcc8822af6..e8adfd50fb2 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv3-d-transpose.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv3-d-transpose.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv3-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv3-d.pbtxt
index 84527b26a39..d49616f857c 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-conv3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution1-d-transpose.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution1-d-transpose.pbtxt
index d81e4546670..6332fcc5d4b 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution1-d-transpose.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution1-d-transpose.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution1-d.pbtxt
index ccb783c33bf..5c94f928d6b 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution2-d-transpose.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution2-d-transpose.pbtxt
index c1f49885d87..0de86bcf16a 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution2-d-transpose.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution2-d-transpose.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution2-d.pbtxt
index 179acdc7966..a0f0ff2e7ba 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution3-d-transpose.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution3-d-transpose.pbtxt
index 5d1d6d04505..c772d1243f4 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution3-d-transpose.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution3-d-transpose.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution3-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution3-d.pbtxt
index b7db8afd065..dc0637655e3 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-convolution3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-cropping1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-cropping1-d.pbtxt
index 29c0cd34098..090f4055906 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-cropping1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-cropping1-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-cropping2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-cropping2-d.pbtxt
index b1ecb7d1204..b010cfe1acd 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-cropping2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-cropping2-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-cropping3-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-cropping3-d.pbtxt
index 1e0fe6e6cf8..802627cc0ed 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-cropping3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-cropping3-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-cu-d-n-n-g-r-u.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-cu-d-n-n-g-r-u.pbtxt
index 4a2a4d19048..7c95b72997c 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-cu-d-n-n-g-r-u.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-cu-d-n-n-g-r-u.pbtxt
@@ -97,6 +97,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-cu-d-n-n-l-s-t-m.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-cu-d-n-n-l-s-t-m.pbtxt
index 282d24fe9e7..40f6c7a7338 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-cu-d-n-n-l-s-t-m.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-cu-d-n-n-l-s-t-m.pbtxt
@@ -97,6 +97,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-dense-features.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-dense-features.pbtxt
index 7ed6c7747a7..3d541caa034 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-dense-features.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-dense-features.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-dense.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-dense.pbtxt
index 025c35eca17..90d398e8975 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-dense.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-dense.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-depthwise-conv2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-depthwise-conv2-d.pbtxt
index d15459798cf..d754f8d1de3 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-depthwise-conv2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-depthwise-conv2-d.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-dot.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-dot.pbtxt
index 777248192c6..7189ec944a1 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-dot.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-dot.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-dropout.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-dropout.pbtxt
index fe114648bff..ecee07fae25 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-dropout.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-dropout.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-e-l-u.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-e-l-u.pbtxt
index 19429711e80..f24a778092b 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-e-l-u.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-e-l-u.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-embedding.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-embedding.pbtxt
index 5ac35db6734..f9e1721669a 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-embedding.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-embedding.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-flatten.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-flatten.pbtxt
index ff17ea72d45..b76d54415c7 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-flatten.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-flatten.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-g-r-u-cell.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-g-r-u-cell.pbtxt
index 146333b09f1..5a6976a8e54 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-g-r-u-cell.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-g-r-u-cell.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-g-r-u.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-g-r-u.pbtxt
index 86fb73d68f7..2eedd3a212e 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-g-r-u.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-g-r-u.pbtxt
@@ -152,6 +152,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-gaussian-dropout.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-gaussian-dropout.pbtxt
index 8cfe9f9c692..1d3b1e73c55 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-gaussian-dropout.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-gaussian-dropout.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-gaussian-noise.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-gaussian-noise.pbtxt
index a64897f8849..46caef8596c 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-gaussian-noise.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-gaussian-noise.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-average-pooling1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-average-pooling1-d.pbtxt
index 7363d9d6521..0afe2f7483f 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-average-pooling1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-average-pooling1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-average-pooling2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-average-pooling2-d.pbtxt
index 58a08cd2d94..8c001871ed6 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-average-pooling2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-average-pooling2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-average-pooling3-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-average-pooling3-d.pbtxt
index 1ec5624d8bf..c0e2d405e9a 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-average-pooling3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-average-pooling3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-avg-pool1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-avg-pool1-d.pbtxt
index 7931f0deb12..785db4a7762 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-avg-pool1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-avg-pool1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-avg-pool2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-avg-pool2-d.pbtxt
index 6db66c8ba9e..c288f835aac 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-avg-pool2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-avg-pool2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-avg-pool3-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-avg-pool3-d.pbtxt
index ffd750c0522..060e6fd7d12 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-avg-pool3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-avg-pool3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pool1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pool1-d.pbtxt
index 11762c021a9..fcf2cf4c0ac 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pool1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pool1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pool2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pool2-d.pbtxt
index 08043cb2926..5a9fcc6db6c 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pool2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pool2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pool3-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pool3-d.pbtxt
index f4155ac58aa..9d50d7e2d95 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pool3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pool3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pooling1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pooling1-d.pbtxt
index 6be5e6d4dea..b071143ab2b 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pooling1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pooling1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pooling2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pooling2-d.pbtxt
index 69719674f2f..49a548fc277 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pooling2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pooling2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pooling3-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pooling3-d.pbtxt
index 624163caa84..f090677378d 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pooling3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-global-max-pooling3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-input-layer.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-input-layer.pbtxt
index 39e79f7980b..e7952bb23c9 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-input-layer.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-input-layer.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-l-s-t-m-cell.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-l-s-t-m-cell.pbtxt
index 4f88e672708..16519f922d9 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-l-s-t-m-cell.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-l-s-t-m-cell.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-l-s-t-m.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-l-s-t-m.pbtxt
index 6a799057db0..1dc701cc7d6 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-l-s-t-m.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-l-s-t-m.pbtxt
@@ -148,6 +148,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-lambda.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-lambda.pbtxt
index 22fa730112f..5673f4d01a1 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-lambda.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-lambda.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-layer-normalization.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-layer-normalization.pbtxt
index d8e9445b8cb..99eafcb8fe3 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-layer-normalization.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-layer-normalization.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-layer.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-layer.pbtxt
index cdc76a45594..bc84f3b51c1 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-layer.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-layer.pbtxt
@@ -86,6 +86,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-leaky-re-l-u.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-leaky-re-l-u.pbtxt
index 06ffbc8fdf3..95c5d4bb39b 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-leaky-re-l-u.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-leaky-re-l-u.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-locally-connected1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-locally-connected1-d.pbtxt
index c2826298321..b72424921a4 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-locally-connected1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-locally-connected1-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-locally-connected2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-locally-connected2-d.pbtxt
index da6934bae44..985c2274379 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-locally-connected2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-locally-connected2-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-masking.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-masking.pbtxt
index 205bf1ed369..65ae9893b11 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-masking.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-masking.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pool1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pool1-d.pbtxt
index df8c2dcd736..57c01d47d04 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pool1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pool1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pool2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pool2-d.pbtxt
index 20a2d5162f4..88c1fb93eaf 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pool2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pool2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pool3-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pool3-d.pbtxt
index 0bddc075006..c5f18cb9a05 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pool3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pool3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pooling1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pooling1-d.pbtxt
index ac7827999ef..db5eb02deee 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pooling1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pooling1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pooling2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pooling2-d.pbtxt
index f3ae4bb7e5e..45e68e0d94c 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pooling2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pooling2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pooling3-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pooling3-d.pbtxt
index 419b64d142f..36ea8cbb6ae 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pooling3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-max-pooling3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-maximum.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-maximum.pbtxt
index 6535a951a1e..87afbb42ee7 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-maximum.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-maximum.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-minimum.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-minimum.pbtxt
index d54b4d1bb60..1de712f45ae 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-minimum.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-minimum.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-multiply.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-multiply.pbtxt
index 4dee52c2ac6..4502002d2c7 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-multiply.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-multiply.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-p-re-l-u.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-p-re-l-u.pbtxt
index 84025572e83..1543f27fab8 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-p-re-l-u.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-p-re-l-u.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-permute.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-permute.pbtxt
index 9483167cb23..47749db7b2a 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-permute.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-permute.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-r-n-n.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-r-n-n.pbtxt
index 7143160bed9..76bfe022b2b 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-r-n-n.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-r-n-n.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-re-l-u.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-re-l-u.pbtxt
index 31b6b03c1b8..48ac55d9b2e 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-re-l-u.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-re-l-u.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-repeat-vector.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-repeat-vector.pbtxt
index e5295928656..71753b21d64 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-repeat-vector.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-repeat-vector.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-reshape.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-reshape.pbtxt
index 8b4773bc4f2..8778dfa3f20 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-reshape.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-reshape.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-separable-conv1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-separable-conv1-d.pbtxt
index dae9b58bc55..1f7d7c0204d 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-separable-conv1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-separable-conv1-d.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-separable-conv2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-separable-conv2-d.pbtxt
index 53ee61ca723..e301442b952 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-separable-conv2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-separable-conv2-d.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-separable-convolution1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-separable-convolution1-d.pbtxt
index 28935f62922..6e52320dcb5 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-separable-convolution1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-separable-convolution1-d.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-separable-convolution2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-separable-convolution2-d.pbtxt
index 8c00f85609f..16e10561154 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-separable-convolution2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-separable-convolution2-d.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-simple-r-n-n-cell.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-simple-r-n-n-cell.pbtxt
index 068788d6b34..8491504b3ab 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-simple-r-n-n-cell.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-simple-r-n-n-cell.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-simple-r-n-n.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-simple-r-n-n.pbtxt
index ddb87d74337..f651e06aa50 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-simple-r-n-n.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-simple-r-n-n.pbtxt
@@ -140,6 +140,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-softmax.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-softmax.pbtxt
index cc5165ea47a..75737c0a415 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-softmax.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-softmax.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-spatial-dropout1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-spatial-dropout1-d.pbtxt
index 9fff96d8764..e70f19ac2e5 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-spatial-dropout1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-spatial-dropout1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-spatial-dropout2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-spatial-dropout2-d.pbtxt
index 24fbd03ee49..b5d4b00a220 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-spatial-dropout2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-spatial-dropout2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-spatial-dropout3-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-spatial-dropout3-d.pbtxt
index 50ec54308c9..52ace6ef50f 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-spatial-dropout3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-spatial-dropout3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-stacked-r-n-n-cells.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-stacked-r-n-n-cells.pbtxt
index 9de71f557f6..14f3ccf383d 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-stacked-r-n-n-cells.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-stacked-r-n-n-cells.pbtxt
@@ -95,6 +95,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-subtract.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-subtract.pbtxt
index 05bbc5ad1be..f85acb2945a 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-subtract.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-subtract.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-thresholded-re-l-u.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-thresholded-re-l-u.pbtxt
index 2d34bf8754c..dccb48bce49 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-thresholded-re-l-u.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-thresholded-re-l-u.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-time-distributed.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-time-distributed.pbtxt
index c153411811f..b02fba20dee 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-time-distributed.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-time-distributed.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-up-sampling1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-up-sampling1-d.pbtxt
index 07ca8e40761..84f58bf535e 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-up-sampling1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-up-sampling1-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-up-sampling2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-up-sampling2-d.pbtxt
index f5b5f8bbf85..b78fcc18e08 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-up-sampling2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-up-sampling2-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-up-sampling3-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-up-sampling3-d.pbtxt
index acfc1a33cfb..29851be0b77 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-up-sampling3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-up-sampling3-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-wrapper.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-wrapper.pbtxt
index 58082daa2fa..da355d1142f 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-wrapper.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-wrapper.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-zero-padding1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-zero-padding1-d.pbtxt
index 40e7a43ad53..3180441407e 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-zero-padding1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-zero-padding1-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-zero-padding2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-zero-padding2-d.pbtxt
index 473feb798f3..ecccc705e07 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-zero-padding2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-zero-padding2-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-zero-padding3-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-zero-padding3-d.pbtxt
index ab8fca29714..fc93d9957d7 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-zero-padding3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.-zero-padding3-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.-einsum-dense.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.-einsum-dense.pbtxt
index 8a782f6666f..8e8684bab54 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.-einsum-dense.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.-einsum-dense.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.-random-fourier-features.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.-random-fourier-features.pbtxt
index cadf62e0d37..9b4ddcd3f62 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.-random-fourier-features.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.-random-fourier-features.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-category-crossing.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-category-crossing.pbtxt
index 6cfcbf73e5d..73592f11b8d 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-category-crossing.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-category-crossing.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-category-encoding.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-category-encoding.pbtxt
index 5bd938d8fd6..26a016264d8 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-category-encoding.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-category-encoding.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
@@ -117,7 +121,7 @@ tf_class {
   }
   member_method {
     name: "__init__"
-    argspec: "args=[\'self\', \'max_tokens\', \'output_mode\', \'sparse\'], varargs=None, keywords=kwargs, defaults=[\'None\', \'count\', \'False\'], "
+    argspec: "args=[\'self\', \'max_tokens\', \'output_mode\', \'sparse\'], varargs=None, keywords=kwargs, defaults=[\'None\', \'binary\', \'False\'], "
   }
   member_method {
     name: "adapt"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-center-crop.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-center-crop.pbtxt
index 5640a4d1dcd..b03ca3dc11c 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-center-crop.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-center-crop.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-discretization.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-discretization.pbtxt
index 82ce2303a76..10f5b9ae1aa 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-discretization.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-discretization.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-hashing.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-hashing.pbtxt
index e4a5619058d..b29b32e1315 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-hashing.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-hashing.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-integer-lookup.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-integer-lookup.pbtxt
index c0052764039..44afdfc7d34 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-integer-lookup.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-integer-lookup.pbtxt
@@ -93,6 +93,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-normalization.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-normalization.pbtxt
index 88a89805ec9..e1a451f8f52 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-normalization.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-normalization.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-preprocessing-layer.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-preprocessing-layer.pbtxt
index 305b239c3e6..095471dc12a 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-preprocessing-layer.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-preprocessing-layer.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-contrast.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-contrast.pbtxt
index 3fc5402fb39..b9a330e8bcc 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-contrast.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-contrast.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-crop.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-crop.pbtxt
index 250880c9ae8..f0f1bc45b0e 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-crop.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-crop.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-flip.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-flip.pbtxt
index 39cd6af00a0..8b1d7734d44 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-flip.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-flip.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-height.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-height.pbtxt
index ce654bd1537..5a8f67f9487 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-height.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-height.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-rotation.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-rotation.pbtxt
index 95c9cb2dd73..0938ec3d684 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-rotation.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-rotation.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-translation.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-translation.pbtxt
index 92dfa72a7a5..cfee60206f1 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-translation.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-translation.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-width.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-width.pbtxt
index 88108bfe9aa..93b272d1105 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-width.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-width.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-zoom.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-zoom.pbtxt
index 85850223bcb..f0b50e92e66 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-zoom.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-random-zoom.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-rescaling.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-rescaling.pbtxt
index 60c0bc92f81..94b6c730699 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-rescaling.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-rescaling.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-resizing.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-resizing.pbtxt
index 5313dfe9907..30b20a1ac01 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-resizing.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-resizing.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-string-lookup.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-string-lookup.pbtxt
index 1deb932cb4e..0e671b5f78b 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-string-lookup.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-string-lookup.pbtxt
@@ -93,6 +93,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-text-vectorization.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-text-vectorization.pbtxt
index 4f5b0f480e4..1b9a745486c 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-text-vectorization.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.layers.experimental.preprocessing.-text-vectorization.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-a-u-c.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-a-u-c.pbtxt
index 56704ace966..96703a02f88 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-a-u-c.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-a-u-c.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "thresholds"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-accuracy.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-accuracy.pbtxt
index fb970c23732..e1b19dc5836 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-accuracy.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-accuracy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-binary-accuracy.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-binary-accuracy.pbtxt
index 09863e42eb1..64edac45aa4 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-binary-accuracy.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-binary-accuracy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-binary-crossentropy.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-binary-crossentropy.pbtxt
index eb033ce30a5..8130de478cd 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-binary-crossentropy.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-binary-crossentropy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-categorical-accuracy.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-categorical-accuracy.pbtxt
index 9de555e3427..7bfcf1f2bdd 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-categorical-accuracy.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-categorical-accuracy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-categorical-crossentropy.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-categorical-crossentropy.pbtxt
index fa41859b37e..4e37d89dba7 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-categorical-crossentropy.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-categorical-crossentropy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-categorical-hinge.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-categorical-hinge.pbtxt
index 3ebaddb0e58..9bb3b783982 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-categorical-hinge.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-categorical-hinge.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-cosine-similarity.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-cosine-similarity.pbtxt
index 7d8eafeb393..4defb3657ec 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-cosine-similarity.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-cosine-similarity.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-false-negatives.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-false-negatives.pbtxt
index 3fa0db2af91..628363c3347 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-false-negatives.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-false-negatives.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-false-positives.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-false-positives.pbtxt
index 4e2a380445e..73c30e9a080 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-false-positives.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-false-positives.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-hinge.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-hinge.pbtxt
index 66e416d57f6..fe3b145999f 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-hinge.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-hinge.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-k-l-divergence.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-k-l-divergence.pbtxt
index 6fbbe6b9336..e167d1a743c 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-k-l-divergence.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-k-l-divergence.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-log-cosh-error.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-log-cosh-error.pbtxt
index f7f8f79eb17..83d2c77a759 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-log-cosh-error.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-log-cosh-error.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-absolute-error.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-absolute-error.pbtxt
index adaf33d3608..90f18e32247 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-absolute-error.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-absolute-error.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-absolute-percentage-error.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-absolute-percentage-error.pbtxt
index 2f743849a8b..cab559c90cb 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-absolute-percentage-error.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-absolute-percentage-error.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-io-u.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-io-u.pbtxt
index 26fe404372d..091cf58e3e2 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-io-u.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-io-u.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-relative-error.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-relative-error.pbtxt
index 1d3eae22f8c..faf78e01f38 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-relative-error.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-relative-error.pbtxt
@@ -90,6 +90,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-squared-error.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-squared-error.pbtxt
index 3fe23a73576..d4cb6e5b7ad 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-squared-error.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-squared-error.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-squared-logarithmic-error.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-squared-logarithmic-error.pbtxt
index 2b98c31a6c7..32a1cffa204 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-squared-logarithmic-error.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-squared-logarithmic-error.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-tensor.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-tensor.pbtxt
index 772bf62a923..788f2e316b0 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-tensor.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean-tensor.pbtxt
@@ -92,6 +92,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "total"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean.pbtxt
index bc14d53dbee..76981e1950c 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-mean.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-metric.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-metric.pbtxt
index 8a6977835a0..65c65f71f6a 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-metric.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-metric.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-poisson.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-poisson.pbtxt
index ff7fdbb6382..22ba826b26e 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-poisson.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-poisson.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-precision-at-recall.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-precision-at-recall.pbtxt
index 81d8a8d94d6..c6bd13d4f1e 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-precision-at-recall.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-precision-at-recall.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-precision.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-precision.pbtxt
index 7a2dc1f7eaf..eaf6bbeb78d 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-precision.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-precision.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-recall-at-precision.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-recall-at-precision.pbtxt
index f57b210b9e8..7254e26db7f 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-recall-at-precision.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-recall-at-precision.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-recall.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-recall.pbtxt
index e62bad28d0c..1f0d977ea23 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-recall.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-recall.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-root-mean-squared-error.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-root-mean-squared-error.pbtxt
index 3ee6eb4995e..c6f2ea193d4 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-root-mean-squared-error.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-root-mean-squared-error.pbtxt
@@ -90,6 +90,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-sensitivity-at-specificity.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-sensitivity-at-specificity.pbtxt
index ddfadbdc66f..1c383c8dcf0 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-sensitivity-at-specificity.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-sensitivity-at-specificity.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-sparse-categorical-accuracy.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-sparse-categorical-accuracy.pbtxt
index f9c77f7a8a2..11c694eb234 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-sparse-categorical-accuracy.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-sparse-categorical-accuracy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-sparse-categorical-crossentropy.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-sparse-categorical-crossentropy.pbtxt
index 7e1abedfebb..c95806164ac 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-sparse-categorical-crossentropy.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-sparse-categorical-crossentropy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-sparse-top-k-categorical-accuracy.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-sparse-top-k-categorical-accuracy.pbtxt
index 4a222c840a1..01b883278e5 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-sparse-top-k-categorical-accuracy.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-sparse-top-k-categorical-accuracy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-specificity-at-sensitivity.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-specificity-at-sensitivity.pbtxt
index ce55374bd73..3d90bfc1ac9 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-specificity-at-sensitivity.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-specificity-at-sensitivity.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-squared-hinge.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-squared-hinge.pbtxt
index a9192f88606..25858b461b3 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-squared-hinge.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-squared-hinge.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-sum.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-sum.pbtxt
index a0def9553f5..065036aee12 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-sum.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-sum.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-top-k-categorical-accuracy.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-top-k-categorical-accuracy.pbtxt
index 24511bd678c..2fac29e2415 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-top-k-categorical-accuracy.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-top-k-categorical-accuracy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-true-negatives.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-true-negatives.pbtxt
index b840940d24a..8fbf24a66ad 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-true-negatives.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-true-negatives.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-true-positives.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-true-positives.pbtxt
index 87f6a87de98..4d1f2fd213f 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-true-positives.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.metrics.-true-positives.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.models.-model.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.models.-model.pbtxt
index 00c9fc22def..7439fc2dd6b 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.models.-model.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.models.-model.pbtxt
@@ -108,6 +108,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.keras.models.-sequential.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.keras.models.-sequential.pbtxt
index d3cca7311ee..24e8bf57611 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.keras.models.-sequential.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.keras.models.-sequential.pbtxt
@@ -110,6 +110,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.layers.-average-pooling1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.layers.-average-pooling1-d.pbtxt
index 5f89addb628..0fa2415c54f 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.layers.-average-pooling1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.layers.-average-pooling1-d.pbtxt
@@ -98,6 +98,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.layers.-average-pooling2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.layers.-average-pooling2-d.pbtxt
index 056e98d47d0..871adc80846 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.layers.-average-pooling2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.layers.-average-pooling2-d.pbtxt
@@ -98,6 +98,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.layers.-average-pooling3-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.layers.-average-pooling3-d.pbtxt
index 63eaa837e93..9b67ca9bee9 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.layers.-average-pooling3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.layers.-average-pooling3-d.pbtxt
@@ -98,6 +98,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.layers.-batch-normalization.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.layers.-batch-normalization.pbtxt
index f7199a19a28..905b8c028bb 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.layers.-batch-normalization.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.layers.-batch-normalization.pbtxt
@@ -98,6 +98,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.layers.-conv1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.layers.-conv1-d.pbtxt
index 9e67ceb7f45..ee778d9bead 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.layers.-conv1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.layers.-conv1-d.pbtxt
@@ -98,6 +98,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.layers.-conv2-d-transpose.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.layers.-conv2-d-transpose.pbtxt
index 52724953edc..7c6958e4d9f 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.layers.-conv2-d-transpose.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.layers.-conv2-d-transpose.pbtxt
@@ -99,6 +99,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.layers.-conv2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.layers.-conv2-d.pbtxt
index da144936c76..e88b843679c 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.layers.-conv2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.layers.-conv2-d.pbtxt
@@ -98,6 +98,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.layers.-conv3-d-transpose.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.layers.-conv3-d-transpose.pbtxt
index 8672e9bd666..f31110ba8fe 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.layers.-conv3-d-transpose.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.layers.-conv3-d-transpose.pbtxt
@@ -99,6 +99,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.layers.-conv3-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.layers.-conv3-d.pbtxt
index 13144714b0b..a3f7e557ddc 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.layers.-conv3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.layers.-conv3-d.pbtxt
@@ -98,6 +98,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.layers.-dense.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.layers.-dense.pbtxt
index 3fb0aa15efa..930706bec9d 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.layers.-dense.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.layers.-dense.pbtxt
@@ -97,6 +97,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.layers.-dropout.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.layers.-dropout.pbtxt
index 8ad21ade6a4..bb57573198a 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.layers.-dropout.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.layers.-dropout.pbtxt
@@ -97,6 +97,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.layers.-flatten.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.layers.-flatten.pbtxt
index e5b86e2be32..1dff295f333 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.layers.-flatten.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.layers.-flatten.pbtxt
@@ -97,6 +97,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.layers.-layer.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.layers.-layer.pbtxt
index cc325d01456..108d1b94bc6 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.layers.-layer.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.layers.-layer.pbtxt
@@ -95,6 +95,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.layers.-max-pooling1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.layers.-max-pooling1-d.pbtxt
index 8a8199b99d4..b65100b2ace 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.layers.-max-pooling1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.layers.-max-pooling1-d.pbtxt
@@ -98,6 +98,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.layers.-max-pooling2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.layers.-max-pooling2-d.pbtxt
index ebdeed19c77..8b44d2eca70 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.layers.-max-pooling2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.layers.-max-pooling2-d.pbtxt
@@ -98,6 +98,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.layers.-max-pooling3-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.layers.-max-pooling3-d.pbtxt
index a0a342c5b86..2f2b7e8830b 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.layers.-max-pooling3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.layers.-max-pooling3-d.pbtxt
@@ -98,6 +98,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.layers.-separable-conv1-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.layers.-separable-conv1-d.pbtxt
index dc81593ffde..ae06fe06994 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.layers.-separable-conv1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.layers.-separable-conv1-d.pbtxt
@@ -99,6 +99,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.layers.-separable-conv2-d.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.layers.-separable-conv2-d.pbtxt
index d4ab15d3217..330528e2df6 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.layers.-separable-conv2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.layers.-separable-conv2-d.pbtxt
@@ -99,6 +99,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.lite.experimental.nn.-t-f-lite-l-s-t-m-cell.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.lite.experimental.nn.-t-f-lite-l-s-t-m-cell.pbtxt
index 14be46801fa..fc8f8cb0478 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.lite.experimental.nn.-t-f-lite-l-s-t-m-cell.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.lite.experimental.nn.-t-f-lite-l-s-t-m-cell.pbtxt
@@ -106,6 +106,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.lite.experimental.nn.-tf-lite-r-n-n-cell.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.lite.experimental.nn.-tf-lite-r-n-n-cell.pbtxt
index 6b8e613e6d0..9cd3bbd8732 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.lite.experimental.nn.-tf-lite-r-n-n-cell.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.lite.experimental.nn.-tf-lite-r-n-n-cell.pbtxt
@@ -106,6 +106,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-basic-l-s-t-m-cell.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-basic-l-s-t-m-cell.pbtxt
index 2c95ad00100..f69f9d6fe7d 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-basic-l-s-t-m-cell.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-basic-l-s-t-m-cell.pbtxt
@@ -106,6 +106,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-basic-r-n-n-cell.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-basic-r-n-n-cell.pbtxt
index 2eae2ba5168..c9d09d08a67 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-basic-r-n-n-cell.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-basic-r-n-n-cell.pbtxt
@@ -106,6 +106,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-device-wrapper.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-device-wrapper.pbtxt
index 86adb8695ca..41bafa62ff9 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-device-wrapper.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-device-wrapper.pbtxt
@@ -107,6 +107,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-dropout-wrapper.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-dropout-wrapper.pbtxt
index 05491f86058..e2ddb7feafe 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-dropout-wrapper.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-dropout-wrapper.pbtxt
@@ -107,6 +107,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-g-r-u-cell.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-g-r-u-cell.pbtxt
index 17813a8de8b..11f02509c12 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-g-r-u-cell.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-g-r-u-cell.pbtxt
@@ -106,6 +106,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-l-s-t-m-cell.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-l-s-t-m-cell.pbtxt
index 21a0b8ab4eb..d8f03cda8b5 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-l-s-t-m-cell.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-l-s-t-m-cell.pbtxt
@@ -106,6 +106,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-multi-r-n-n-cell.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-multi-r-n-n-cell.pbtxt
index d859a4666a0..de531d7a11a 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-multi-r-n-n-cell.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-multi-r-n-n-cell.pbtxt
@@ -105,6 +105,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-r-n-n-cell.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-r-n-n-cell.pbtxt
index 29a592022b2..d678fae51f1 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-r-n-n-cell.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-r-n-n-cell.pbtxt
@@ -104,6 +104,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-residual-wrapper.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-residual-wrapper.pbtxt
index 96f625f7f55..b4449602033 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-residual-wrapper.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.nn.rnn_cell.-residual-wrapper.pbtxt
@@ -107,6 +107,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.raw_ops.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.raw_ops.pbtxt
index d4776eef338..e8cb384aa7b 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.raw_ops.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.raw_ops.pbtxt
@@ -982,7 +982,7 @@ tf_module {
   }
   member_method {
     name: "DebugIdentityV2"
-    argspec: "args=[\'input\', \'tfdbg_context_id\', \'op_name\', \'output_slot\', \'tensor_debug_mode\', \'debug_urls\', \'name\'], varargs=None, keywords=None, defaults=[\'\', \'\', \'-1\', \'-1\', \'[]\', \'None\'], "
+    argspec: "args=[\'input\', \'tfdbg_context_id\', \'op_name\', \'output_slot\', \'tensor_debug_mode\', \'debug_urls\', \'circular_buffer_size\', \'name\'], varargs=None, keywords=None, defaults=[\'\', \'\', \'-1\', \'-1\', \'[]\', \'1000\', \'None\'], "
   }
   member_method {
     name: "DebugNanCount"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.sysconfig.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.sysconfig.pbtxt
index 811ca18cdb4..7b05d382f6c 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.sysconfig.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.sysconfig.pbtxt
@@ -8,6 +8,10 @@ tf_module {
     name: "MONOLITHIC_BUILD"
     mtype: "<type \'int\'>"
   }
+  member_method {
+    name: "get_build_info"
+    argspec: "args=[], varargs=None, keywords=None, defaults=None"
+  }
   member_method {
     name: "get_compile_flags"
     argspec: "args=[], varargs=None, keywords=None, defaults=None"
diff --git a/tensorflow/tools/api/golden/v1/tensorflow.tpu.pbtxt b/tensorflow/tools/api/golden/v1/tensorflow.tpu.pbtxt
index 7f9f0e874f9..47213e34e3d 100644
--- a/tensorflow/tools/api/golden/v1/tensorflow.tpu.pbtxt
+++ b/tensorflow/tools/api/golden/v1/tensorflow.tpu.pbtxt
@@ -18,7 +18,7 @@ tf_module {
   }
   member_method {
     name: "bfloat16_scope"
-    argspec: "args=[], varargs=None, keywords=None, defaults=None"
+    argspec: "args=[\'name\'], varargs=None, keywords=None, defaults=[\'None\'], "
   }
   member_method {
     name: "core"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.-model.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.-model.pbtxt
index d696021fcb4..ea2945b5bf6 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.-model.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.-model.pbtxt
@@ -108,6 +108,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.-sequential.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.-sequential.pbtxt
index b8486a27b9e..c1dea9335c0 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.-sequential.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.-sequential.pbtxt
@@ -110,6 +110,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.experimental.-linear-model.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.experimental.-linear-model.pbtxt
index 7bf71844fa6..ba87c0a2a7a 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.experimental.-linear-model.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.experimental.-linear-model.pbtxt
@@ -109,6 +109,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.experimental.-peephole-l-s-t-m-cell.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.experimental.-peephole-l-s-t-m-cell.pbtxt
index ee5e5b884a2..e366d0b1f52 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.experimental.-peephole-l-s-t-m-cell.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.experimental.-peephole-l-s-t-m-cell.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.experimental.-sequence-features.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.experimental.-sequence-features.pbtxt
index e2bef6beaaa..2b4ebd55410 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.experimental.-sequence-features.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.experimental.-sequence-features.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.experimental.-wide-deep-model.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.experimental.-wide-deep-model.pbtxt
index 87a7319639b..37fb2051f81 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.experimental.-wide-deep-model.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.experimental.-wide-deep-model.pbtxt
@@ -109,6 +109,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-abstract-r-n-n-cell.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-abstract-r-n-n-cell.pbtxt
index 8c80da861f6..363e4b7cf20 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-abstract-r-n-n-cell.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-abstract-r-n-n-cell.pbtxt
@@ -95,6 +95,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-activation.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-activation.pbtxt
index ef8efd606c4..409dfcd26e4 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-activation.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-activation.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-activity-regularization.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-activity-regularization.pbtxt
index 60578d2cc59..599d550fee2 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-activity-regularization.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-activity-regularization.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-add.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-add.pbtxt
index 3bd1f2c7623..b43eee1e6e5 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-add.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-add.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-additive-attention.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-additive-attention.pbtxt
index f6f8d3914b4..2edb8e028c1 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-additive-attention.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-additive-attention.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-alpha-dropout.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-alpha-dropout.pbtxt
index 3e408e96036..4f7c7d6ca27 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-alpha-dropout.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-alpha-dropout.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-attention.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-attention.pbtxt
index 4197c1a88f6..e335099c084 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-attention.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-attention.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-average-pooling1-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-average-pooling1-d.pbtxt
index 153a801e1d8..80c8e0d63c1 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-average-pooling1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-average-pooling1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-average-pooling2-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-average-pooling2-d.pbtxt
index 66e261111ae..4e95f083490 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-average-pooling2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-average-pooling2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-average-pooling3-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-average-pooling3-d.pbtxt
index b247490b067..370589fb876 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-average-pooling3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-average-pooling3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-average.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-average.pbtxt
index 0f1808332c5..8abc0add0dc 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-average.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-average.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-avg-pool1-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-avg-pool1-d.pbtxt
index 567143eb41d..6938e84fe77 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-avg-pool1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-avg-pool1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-avg-pool2-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-avg-pool2-d.pbtxt
index 56a2db85419..5ca4fd39173 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-avg-pool2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-avg-pool2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-avg-pool3-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-avg-pool3-d.pbtxt
index c0ab32fd7c2..378bf0e84a0 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-avg-pool3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-avg-pool3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-batch-normalization.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-batch-normalization.pbtxt
index 6b1c609774e..049b9b2b8a6 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-batch-normalization.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-batch-normalization.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-bidirectional.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-bidirectional.pbtxt
index ccb55ec0c52..60402f72e77 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-bidirectional.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-bidirectional.pbtxt
@@ -92,6 +92,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-concatenate.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-concatenate.pbtxt
index c44ff9e48e2..c6733e7e8ab 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-concatenate.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-concatenate.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv-l-s-t-m2-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv-l-s-t-m2-d.pbtxt
index 43112cfe785..f77d613e354 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv-l-s-t-m2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv-l-s-t-m2-d.pbtxt
@@ -169,6 +169,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv1-d-transpose.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv1-d-transpose.pbtxt
index 3a592d713bc..03ccf45ea84 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv1-d-transpose.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv1-d-transpose.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv1-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv1-d.pbtxt
index 3319122f50b..f41ff3f2136 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv2-d-transpose.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv2-d-transpose.pbtxt
index 535243a2224..2370f378e57 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv2-d-transpose.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv2-d-transpose.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv2-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv2-d.pbtxt
index 30736595ce5..82caa2a7b1d 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv3-d-transpose.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv3-d-transpose.pbtxt
index afcc8822af6..e8adfd50fb2 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv3-d-transpose.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv3-d-transpose.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv3-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv3-d.pbtxt
index 84527b26a39..d49616f857c 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-conv3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution1-d-transpose.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution1-d-transpose.pbtxt
index d81e4546670..6332fcc5d4b 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution1-d-transpose.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution1-d-transpose.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution1-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution1-d.pbtxt
index ccb783c33bf..5c94f928d6b 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution2-d-transpose.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution2-d-transpose.pbtxt
index c1f49885d87..0de86bcf16a 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution2-d-transpose.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution2-d-transpose.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution2-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution2-d.pbtxt
index 179acdc7966..a0f0ff2e7ba 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution3-d-transpose.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution3-d-transpose.pbtxt
index 5d1d6d04505..c772d1243f4 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution3-d-transpose.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution3-d-transpose.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution3-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution3-d.pbtxt
index b7db8afd065..dc0637655e3 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-convolution3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-cropping1-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-cropping1-d.pbtxt
index 29c0cd34098..090f4055906 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-cropping1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-cropping1-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-cropping2-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-cropping2-d.pbtxt
index b1ecb7d1204..b010cfe1acd 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-cropping2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-cropping2-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-cropping3-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-cropping3-d.pbtxt
index 1e0fe6e6cf8..802627cc0ed 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-cropping3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-cropping3-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-dense-features.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-dense-features.pbtxt
index 3b4eb863387..29c3b428d23 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-dense-features.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-dense-features.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-dense.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-dense.pbtxt
index 025c35eca17..90d398e8975 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-dense.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-dense.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-depthwise-conv2-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-depthwise-conv2-d.pbtxt
index d15459798cf..d754f8d1de3 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-depthwise-conv2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-depthwise-conv2-d.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-dot.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-dot.pbtxt
index 777248192c6..7189ec944a1 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-dot.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-dot.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-dropout.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-dropout.pbtxt
index fe114648bff..ecee07fae25 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-dropout.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-dropout.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-e-l-u.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-e-l-u.pbtxt
index 19429711e80..f24a778092b 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-e-l-u.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-e-l-u.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-embedding.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-embedding.pbtxt
index 5ac35db6734..f9e1721669a 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-embedding.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-embedding.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-flatten.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-flatten.pbtxt
index ff17ea72d45..b76d54415c7 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-flatten.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-flatten.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-g-r-u-cell.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-g-r-u-cell.pbtxt
index df2fa8a2f5d..ac60b3f5b3f 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-g-r-u-cell.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-g-r-u-cell.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-g-r-u.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-g-r-u.pbtxt
index 24510d6a2bd..6d8f5008f65 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-g-r-u.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-g-r-u.pbtxt
@@ -154,6 +154,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-gaussian-dropout.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-gaussian-dropout.pbtxt
index 8cfe9f9c692..1d3b1e73c55 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-gaussian-dropout.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-gaussian-dropout.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-gaussian-noise.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-gaussian-noise.pbtxt
index a64897f8849..46caef8596c 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-gaussian-noise.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-gaussian-noise.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-average-pooling1-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-average-pooling1-d.pbtxt
index 7363d9d6521..0afe2f7483f 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-average-pooling1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-average-pooling1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-average-pooling2-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-average-pooling2-d.pbtxt
index 58a08cd2d94..8c001871ed6 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-average-pooling2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-average-pooling2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-average-pooling3-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-average-pooling3-d.pbtxt
index 1ec5624d8bf..c0e2d405e9a 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-average-pooling3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-average-pooling3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-avg-pool1-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-avg-pool1-d.pbtxt
index 7931f0deb12..785db4a7762 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-avg-pool1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-avg-pool1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-avg-pool2-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-avg-pool2-d.pbtxt
index 6db66c8ba9e..c288f835aac 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-avg-pool2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-avg-pool2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-avg-pool3-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-avg-pool3-d.pbtxt
index ffd750c0522..060e6fd7d12 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-avg-pool3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-avg-pool3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pool1-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pool1-d.pbtxt
index 11762c021a9..fcf2cf4c0ac 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pool1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pool1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pool2-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pool2-d.pbtxt
index 08043cb2926..5a9fcc6db6c 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pool2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pool2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pool3-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pool3-d.pbtxt
index f4155ac58aa..9d50d7e2d95 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pool3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pool3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pooling1-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pooling1-d.pbtxt
index 6be5e6d4dea..b071143ab2b 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pooling1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pooling1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pooling2-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pooling2-d.pbtxt
index 69719674f2f..49a548fc277 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pooling2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pooling2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pooling3-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pooling3-d.pbtxt
index 624163caa84..f090677378d 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pooling3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-global-max-pooling3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-input-layer.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-input-layer.pbtxt
index 39e79f7980b..e7952bb23c9 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-input-layer.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-input-layer.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-l-s-t-m-cell.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-l-s-t-m-cell.pbtxt
index 428b078e9d8..3682005bee1 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-l-s-t-m-cell.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-l-s-t-m-cell.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-l-s-t-m.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-l-s-t-m.pbtxt
index 0ddf19fcde9..56368eefc1b 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-l-s-t-m.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-l-s-t-m.pbtxt
@@ -150,6 +150,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-lambda.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-lambda.pbtxt
index 22fa730112f..5673f4d01a1 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-lambda.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-lambda.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-layer-normalization.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-layer-normalization.pbtxt
index d8e9445b8cb..99eafcb8fe3 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-layer-normalization.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-layer-normalization.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-layer.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-layer.pbtxt
index cdc76a45594..bc84f3b51c1 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-layer.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-layer.pbtxt
@@ -86,6 +86,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-leaky-re-l-u.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-leaky-re-l-u.pbtxt
index 06ffbc8fdf3..95c5d4bb39b 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-leaky-re-l-u.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-leaky-re-l-u.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-locally-connected1-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-locally-connected1-d.pbtxt
index c2826298321..b72424921a4 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-locally-connected1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-locally-connected1-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-locally-connected2-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-locally-connected2-d.pbtxt
index da6934bae44..985c2274379 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-locally-connected2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-locally-connected2-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-masking.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-masking.pbtxt
index 205bf1ed369..65ae9893b11 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-masking.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-masking.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pool1-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pool1-d.pbtxt
index df8c2dcd736..57c01d47d04 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pool1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pool1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pool2-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pool2-d.pbtxt
index 20a2d5162f4..88c1fb93eaf 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pool2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pool2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pool3-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pool3-d.pbtxt
index 0bddc075006..c5f18cb9a05 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pool3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pool3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pooling1-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pooling1-d.pbtxt
index ac7827999ef..db5eb02deee 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pooling1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pooling1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pooling2-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pooling2-d.pbtxt
index f3ae4bb7e5e..45e68e0d94c 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pooling2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pooling2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pooling3-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pooling3-d.pbtxt
index 419b64d142f..36ea8cbb6ae 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pooling3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-max-pooling3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-maximum.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-maximum.pbtxt
index 6535a951a1e..87afbb42ee7 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-maximum.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-maximum.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-minimum.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-minimum.pbtxt
index d54b4d1bb60..1de712f45ae 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-minimum.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-minimum.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-multiply.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-multiply.pbtxt
index 4dee52c2ac6..4502002d2c7 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-multiply.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-multiply.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-p-re-l-u.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-p-re-l-u.pbtxt
index 84025572e83..1543f27fab8 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-p-re-l-u.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-p-re-l-u.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-permute.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-permute.pbtxt
index 9483167cb23..47749db7b2a 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-permute.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-permute.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-r-n-n.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-r-n-n.pbtxt
index 7143160bed9..76bfe022b2b 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-r-n-n.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-r-n-n.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-re-l-u.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-re-l-u.pbtxt
index 31b6b03c1b8..48ac55d9b2e 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-re-l-u.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-re-l-u.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-repeat-vector.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-repeat-vector.pbtxt
index e5295928656..71753b21d64 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-repeat-vector.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-repeat-vector.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-reshape.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-reshape.pbtxt
index 8b4773bc4f2..8778dfa3f20 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-reshape.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-reshape.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-separable-conv1-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-separable-conv1-d.pbtxt
index dae9b58bc55..1f7d7c0204d 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-separable-conv1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-separable-conv1-d.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-separable-conv2-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-separable-conv2-d.pbtxt
index 53ee61ca723..e301442b952 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-separable-conv2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-separable-conv2-d.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-separable-convolution1-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-separable-convolution1-d.pbtxt
index 28935f62922..6e52320dcb5 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-separable-convolution1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-separable-convolution1-d.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-separable-convolution2-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-separable-convolution2-d.pbtxt
index 8c00f85609f..16e10561154 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-separable-convolution2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-separable-convolution2-d.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-simple-r-n-n-cell.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-simple-r-n-n-cell.pbtxt
index 068788d6b34..8491504b3ab 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-simple-r-n-n-cell.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-simple-r-n-n-cell.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-simple-r-n-n.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-simple-r-n-n.pbtxt
index ddb87d74337..f651e06aa50 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-simple-r-n-n.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-simple-r-n-n.pbtxt
@@ -140,6 +140,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-softmax.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-softmax.pbtxt
index cc5165ea47a..75737c0a415 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-softmax.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-softmax.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-spatial-dropout1-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-spatial-dropout1-d.pbtxt
index 9fff96d8764..e70f19ac2e5 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-spatial-dropout1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-spatial-dropout1-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-spatial-dropout2-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-spatial-dropout2-d.pbtxt
index 24fbd03ee49..b5d4b00a220 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-spatial-dropout2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-spatial-dropout2-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-spatial-dropout3-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-spatial-dropout3-d.pbtxt
index 50ec54308c9..52ace6ef50f 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-spatial-dropout3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-spatial-dropout3-d.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-stacked-r-n-n-cells.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-stacked-r-n-n-cells.pbtxt
index 9de71f557f6..14f3ccf383d 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-stacked-r-n-n-cells.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-stacked-r-n-n-cells.pbtxt
@@ -95,6 +95,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-subtract.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-subtract.pbtxt
index 05bbc5ad1be..f85acb2945a 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-subtract.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-subtract.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-thresholded-re-l-u.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-thresholded-re-l-u.pbtxt
index 2d34bf8754c..dccb48bce49 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-thresholded-re-l-u.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-thresholded-re-l-u.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-time-distributed.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-time-distributed.pbtxt
index c153411811f..b02fba20dee 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-time-distributed.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-time-distributed.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-up-sampling1-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-up-sampling1-d.pbtxt
index 07ca8e40761..84f58bf535e 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-up-sampling1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-up-sampling1-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-up-sampling2-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-up-sampling2-d.pbtxt
index f5b5f8bbf85..b78fcc18e08 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-up-sampling2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-up-sampling2-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-up-sampling3-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-up-sampling3-d.pbtxt
index acfc1a33cfb..29851be0b77 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-up-sampling3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-up-sampling3-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-wrapper.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-wrapper.pbtxt
index 58082daa2fa..da355d1142f 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-wrapper.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-wrapper.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-zero-padding1-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-zero-padding1-d.pbtxt
index 40e7a43ad53..3180441407e 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-zero-padding1-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-zero-padding1-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-zero-padding2-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-zero-padding2-d.pbtxt
index 473feb798f3..ecccc705e07 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-zero-padding2-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-zero-padding2-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-zero-padding3-d.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-zero-padding3-d.pbtxt
index ab8fca29714..fc93d9957d7 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-zero-padding3-d.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.-zero-padding3-d.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.-einsum-dense.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.-einsum-dense.pbtxt
index 8a782f6666f..8e8684bab54 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.-einsum-dense.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.-einsum-dense.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.-random-fourier-features.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.-random-fourier-features.pbtxt
index cadf62e0d37..9b4ddcd3f62 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.-random-fourier-features.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.-random-fourier-features.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.-sync-batch-normalization.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.-sync-batch-normalization.pbtxt
index a589ffff174..70d57f09355 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.-sync-batch-normalization.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.-sync-batch-normalization.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-category-crossing.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-category-crossing.pbtxt
index 6cfcbf73e5d..73592f11b8d 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-category-crossing.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-category-crossing.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-category-encoding.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-category-encoding.pbtxt
index fe0e45c3c67..6874daac890 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-category-encoding.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-category-encoding.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
@@ -115,7 +119,7 @@ tf_class {
   }
   member_method {
     name: "__init__"
-    argspec: "args=[\'self\', \'max_tokens\', \'output_mode\', \'sparse\'], varargs=None, keywords=kwargs, defaults=[\'None\', \'count\', \'False\'], "
+    argspec: "args=[\'self\', \'max_tokens\', \'output_mode\', \'sparse\'], varargs=None, keywords=kwargs, defaults=[\'None\', \'binary\', \'False\'], "
   }
   member_method {
     name: "adapt"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-center-crop.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-center-crop.pbtxt
index 5640a4d1dcd..b03ca3dc11c 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-center-crop.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-center-crop.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-discretization.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-discretization.pbtxt
index 82ce2303a76..10f5b9ae1aa 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-discretization.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-discretization.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-hashing.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-hashing.pbtxt
index e4a5619058d..b29b32e1315 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-hashing.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-hashing.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-integer-lookup.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-integer-lookup.pbtxt
index dedb0f3073f..ca87b769d6a 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-integer-lookup.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-integer-lookup.pbtxt
@@ -90,6 +90,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-normalization.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-normalization.pbtxt
index d52fffa12a3..7045e8512ad 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-normalization.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-normalization.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-preprocessing-layer.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-preprocessing-layer.pbtxt
index 305b239c3e6..095471dc12a 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-preprocessing-layer.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-preprocessing-layer.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-contrast.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-contrast.pbtxt
index 3fc5402fb39..b9a330e8bcc 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-contrast.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-contrast.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-crop.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-crop.pbtxt
index 250880c9ae8..f0f1bc45b0e 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-crop.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-crop.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-flip.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-flip.pbtxt
index 39cd6af00a0..8b1d7734d44 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-flip.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-flip.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-height.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-height.pbtxt
index ce654bd1537..5a8f67f9487 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-height.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-height.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-rotation.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-rotation.pbtxt
index 95c9cb2dd73..0938ec3d684 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-rotation.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-rotation.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-translation.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-translation.pbtxt
index 92dfa72a7a5..cfee60206f1 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-translation.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-translation.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-width.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-width.pbtxt
index 88108bfe9aa..93b272d1105 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-width.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-width.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-zoom.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-zoom.pbtxt
index 85850223bcb..f0b50e92e66 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-zoom.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-random-zoom.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-rescaling.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-rescaling.pbtxt
index 60c0bc92f81..94b6c730699 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-rescaling.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-rescaling.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-resizing.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-resizing.pbtxt
index 5313dfe9907..30b20a1ac01 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-resizing.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-resizing.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-string-lookup.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-string-lookup.pbtxt
index b419e779c48..516f2c08571 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-string-lookup.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-string-lookup.pbtxt
@@ -90,6 +90,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-text-vectorization.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-text-vectorization.pbtxt
index a33f65189fd..2106fa75996 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-text-vectorization.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.layers.experimental.preprocessing.-text-vectorization.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-a-u-c.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-a-u-c.pbtxt
index 56704ace966..96703a02f88 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-a-u-c.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-a-u-c.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "thresholds"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-accuracy.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-accuracy.pbtxt
index fb970c23732..e1b19dc5836 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-accuracy.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-accuracy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-binary-accuracy.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-binary-accuracy.pbtxt
index 09863e42eb1..64edac45aa4 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-binary-accuracy.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-binary-accuracy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-binary-crossentropy.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-binary-crossentropy.pbtxt
index eb033ce30a5..8130de478cd 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-binary-crossentropy.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-binary-crossentropy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-categorical-accuracy.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-categorical-accuracy.pbtxt
index 9de555e3427..7bfcf1f2bdd 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-categorical-accuracy.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-categorical-accuracy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-categorical-crossentropy.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-categorical-crossentropy.pbtxt
index fa41859b37e..4e37d89dba7 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-categorical-crossentropy.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-categorical-crossentropy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-categorical-hinge.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-categorical-hinge.pbtxt
index 3ebaddb0e58..9bb3b783982 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-categorical-hinge.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-categorical-hinge.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-cosine-similarity.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-cosine-similarity.pbtxt
index 7d8eafeb393..4defb3657ec 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-cosine-similarity.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-cosine-similarity.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-false-negatives.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-false-negatives.pbtxt
index 3fa0db2af91..628363c3347 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-false-negatives.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-false-negatives.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-false-positives.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-false-positives.pbtxt
index 4e2a380445e..73c30e9a080 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-false-positives.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-false-positives.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-hinge.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-hinge.pbtxt
index 66e416d57f6..fe3b145999f 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-hinge.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-hinge.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-k-l-divergence.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-k-l-divergence.pbtxt
index 6fbbe6b9336..e167d1a743c 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-k-l-divergence.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-k-l-divergence.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-log-cosh-error.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-log-cosh-error.pbtxt
index f7f8f79eb17..83d2c77a759 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-log-cosh-error.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-log-cosh-error.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-absolute-error.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-absolute-error.pbtxt
index adaf33d3608..90f18e32247 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-absolute-error.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-absolute-error.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-absolute-percentage-error.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-absolute-percentage-error.pbtxt
index 2f743849a8b..cab559c90cb 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-absolute-percentage-error.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-absolute-percentage-error.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-io-u.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-io-u.pbtxt
index 26fe404372d..091cf58e3e2 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-io-u.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-io-u.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-relative-error.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-relative-error.pbtxt
index 1d3eae22f8c..faf78e01f38 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-relative-error.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-relative-error.pbtxt
@@ -90,6 +90,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-squared-error.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-squared-error.pbtxt
index 3fe23a73576..d4cb6e5b7ad 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-squared-error.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-squared-error.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-squared-logarithmic-error.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-squared-logarithmic-error.pbtxt
index 2b98c31a6c7..32a1cffa204 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-squared-logarithmic-error.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-squared-logarithmic-error.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-tensor.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-tensor.pbtxt
index 772bf62a923..788f2e316b0 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-tensor.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean-tensor.pbtxt
@@ -92,6 +92,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "total"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean.pbtxt
index bc14d53dbee..76981e1950c 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-mean.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-metric.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-metric.pbtxt
index 8a6977835a0..65c65f71f6a 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-metric.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-metric.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-poisson.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-poisson.pbtxt
index ff7fdbb6382..22ba826b26e 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-poisson.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-poisson.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-precision-at-recall.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-precision-at-recall.pbtxt
index 81d8a8d94d6..c6bd13d4f1e 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-precision-at-recall.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-precision-at-recall.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-precision.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-precision.pbtxt
index 7a2dc1f7eaf..eaf6bbeb78d 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-precision.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-precision.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-recall-at-precision.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-recall-at-precision.pbtxt
index f57b210b9e8..7254e26db7f 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-recall-at-precision.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-recall-at-precision.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-recall.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-recall.pbtxt
index e62bad28d0c..1f0d977ea23 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-recall.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-recall.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-root-mean-squared-error.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-root-mean-squared-error.pbtxt
index 3ee6eb4995e..c6f2ea193d4 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-root-mean-squared-error.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-root-mean-squared-error.pbtxt
@@ -90,6 +90,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-sensitivity-at-specificity.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-sensitivity-at-specificity.pbtxt
index ddfadbdc66f..1c383c8dcf0 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-sensitivity-at-specificity.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-sensitivity-at-specificity.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-sparse-categorical-accuracy.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-sparse-categorical-accuracy.pbtxt
index f9c77f7a8a2..11c694eb234 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-sparse-categorical-accuracy.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-sparse-categorical-accuracy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-sparse-categorical-crossentropy.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-sparse-categorical-crossentropy.pbtxt
index 7e1abedfebb..c95806164ac 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-sparse-categorical-crossentropy.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-sparse-categorical-crossentropy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-sparse-top-k-categorical-accuracy.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-sparse-top-k-categorical-accuracy.pbtxt
index 4a222c840a1..01b883278e5 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-sparse-top-k-categorical-accuracy.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-sparse-top-k-categorical-accuracy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-specificity-at-sensitivity.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-specificity-at-sensitivity.pbtxt
index ce55374bd73..3d90bfc1ac9 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-specificity-at-sensitivity.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-specificity-at-sensitivity.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-squared-hinge.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-squared-hinge.pbtxt
index a9192f88606..25858b461b3 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-squared-hinge.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-squared-hinge.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-sum.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-sum.pbtxt
index a0def9553f5..065036aee12 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-sum.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-sum.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-top-k-categorical-accuracy.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-top-k-categorical-accuracy.pbtxt
index 24511bd678c..2fac29e2415 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-top-k-categorical-accuracy.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-top-k-categorical-accuracy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-true-negatives.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-true-negatives.pbtxt
index b840940d24a..8fbf24a66ad 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-true-negatives.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-true-negatives.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-true-positives.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-true-positives.pbtxt
index 87f6a87de98..4d1f2fd213f 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-true-positives.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.metrics.-true-positives.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.models.-model.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.models.-model.pbtxt
index 00c9fc22def..7439fc2dd6b 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.models.-model.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.models.-model.pbtxt
@@ -108,6 +108,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.keras.models.-sequential.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.keras.models.-sequential.pbtxt
index d3cca7311ee..24e8bf57611 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.keras.models.-sequential.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.keras.models.-sequential.pbtxt
@@ -110,6 +110,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-a-u-c.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-a-u-c.pbtxt
index 03451eb1ae7..e9177defa20 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-a-u-c.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-a-u-c.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "thresholds"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-accuracy.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-accuracy.pbtxt
index 7011c6d74fa..6b63d833e42 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-accuracy.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-accuracy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-binary-accuracy.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-binary-accuracy.pbtxt
index 1259bbab1eb..3d4d7f1ca97 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-binary-accuracy.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-binary-accuracy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-binary-crossentropy.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-binary-crossentropy.pbtxt
index edceee32ad0..bfa0226b05d 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-binary-crossentropy.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-binary-crossentropy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-categorical-accuracy.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-categorical-accuracy.pbtxt
index 442e48571cb..a1216947733 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-categorical-accuracy.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-categorical-accuracy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-categorical-crossentropy.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-categorical-crossentropy.pbtxt
index ac5beaf47db..2f67f041b9a 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-categorical-crossentropy.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-categorical-crossentropy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-categorical-hinge.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-categorical-hinge.pbtxt
index 0999ad8ba56..1c64601ad46 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-categorical-hinge.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-categorical-hinge.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-cosine-similarity.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-cosine-similarity.pbtxt
index a77ac482bdb..b2954042bbf 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-cosine-similarity.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-cosine-similarity.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-false-negatives.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-false-negatives.pbtxt
index 171820ce02d..7bfee8ccd5d 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-false-negatives.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-false-negatives.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-false-positives.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-false-positives.pbtxt
index 2ae474f4faa..44028896ecf 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-false-positives.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-false-positives.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-hinge.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-hinge.pbtxt
index daddc6e44da..fee3302e540 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-hinge.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-hinge.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-k-l-divergence.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-k-l-divergence.pbtxt
index 4b1e4bfb92c..e548b5628b5 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-k-l-divergence.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-k-l-divergence.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-log-cosh-error.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-log-cosh-error.pbtxt
index e37629b0dc6..177850aaabf 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-log-cosh-error.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-log-cosh-error.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-absolute-error.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-absolute-error.pbtxt
index c2267f83969..429d31b967d 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-absolute-error.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-absolute-error.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-absolute-percentage-error.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-absolute-percentage-error.pbtxt
index c7f57ed5244..be99116d61b 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-absolute-percentage-error.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-absolute-percentage-error.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-io-u.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-io-u.pbtxt
index 32b6b4e9fad..861283a1272 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-io-u.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-io-u.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-relative-error.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-relative-error.pbtxt
index 23e79579eb8..17604647fad 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-relative-error.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-relative-error.pbtxt
@@ -90,6 +90,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-squared-error.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-squared-error.pbtxt
index c924ffb55b9..e2ce76c2a1b 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-squared-error.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-squared-error.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-squared-logarithmic-error.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-squared-logarithmic-error.pbtxt
index f3d29557a17..4cc3f5c26e7 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-squared-logarithmic-error.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-squared-logarithmic-error.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-tensor.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-tensor.pbtxt
index 2a22c1a5b63..97ce4b030af 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-tensor.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean-tensor.pbtxt
@@ -92,6 +92,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "total"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean.pbtxt
index 94fb1937b76..de1ecd925e0 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-mean.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-metric.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-metric.pbtxt
index f3bb587cca5..28659b68e57 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-metric.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-metric.pbtxt
@@ -87,6 +87,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-poisson.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-poisson.pbtxt
index 0fa457f3553..37440a594da 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-poisson.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-poisson.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-precision-at-recall.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-precision-at-recall.pbtxt
index a26305e2d0f..8d1e8854aca 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-precision-at-recall.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-precision-at-recall.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-precision.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-precision.pbtxt
index 73743ba32a0..b53e97f9cbb 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-precision.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-precision.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-recall-at-precision.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-recall-at-precision.pbtxt
index 87f38a2d95d..95a9c0b64c1 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-recall-at-precision.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-recall-at-precision.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-recall.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-recall.pbtxt
index 4e339c3f772..9f4d76fecba 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-recall.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-recall.pbtxt
@@ -88,6 +88,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-root-mean-squared-error.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-root-mean-squared-error.pbtxt
index 0f57806d0f4..9d5461d1fb7 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-root-mean-squared-error.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-root-mean-squared-error.pbtxt
@@ -90,6 +90,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-sensitivity-at-specificity.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-sensitivity-at-specificity.pbtxt
index e82f86f28e7..6799b934475 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-sensitivity-at-specificity.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-sensitivity-at-specificity.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-sparse-categorical-accuracy.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-sparse-categorical-accuracy.pbtxt
index 60f2e2e89b5..8e930197163 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-sparse-categorical-accuracy.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-sparse-categorical-accuracy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-sparse-categorical-crossentropy.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-sparse-categorical-crossentropy.pbtxt
index a24216cd16e..f5a41280932 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-sparse-categorical-crossentropy.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-sparse-categorical-crossentropy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-sparse-top-k-categorical-accuracy.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-sparse-top-k-categorical-accuracy.pbtxt
index e4139b23999..463730dece5 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-sparse-top-k-categorical-accuracy.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-sparse-top-k-categorical-accuracy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-specificity-at-sensitivity.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-specificity-at-sensitivity.pbtxt
index cf55bba0ee9..4d1c97a60e0 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-specificity-at-sensitivity.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-specificity-at-sensitivity.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-squared-hinge.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-squared-hinge.pbtxt
index 997363b92bc..03092595d20 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-squared-hinge.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-squared-hinge.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-sum.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-sum.pbtxt
index 2a3c2a398eb..d738fa5a12a 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-sum.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-sum.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-top-k-categorical-accuracy.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-top-k-categorical-accuracy.pbtxt
index 006bbc68a40..1a91cba2f63 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-top-k-categorical-accuracy.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-top-k-categorical-accuracy.pbtxt
@@ -91,6 +91,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-true-negatives.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-true-negatives.pbtxt
index 5550520f3c7..2c0c7cbab48 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-true-negatives.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-true-negatives.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-true-positives.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-true-positives.pbtxt
index 06892683ac3..beb1e64b5ea 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.metrics.-true-positives.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.metrics.-true-positives.pbtxt
@@ -89,6 +89,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.nn.-r-n-n-cell-device-wrapper.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.nn.-r-n-n-cell-device-wrapper.pbtxt
index cdcf7f23edc..e88c74bf97b 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.nn.-r-n-n-cell-device-wrapper.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.nn.-r-n-n-cell-device-wrapper.pbtxt
@@ -98,6 +98,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.nn.-r-n-n-cell-dropout-wrapper.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.nn.-r-n-n-cell-dropout-wrapper.pbtxt
index 75e827a8ab9..d44b2862408 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.nn.-r-n-n-cell-dropout-wrapper.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.nn.-r-n-n-cell-dropout-wrapper.pbtxt
@@ -98,6 +98,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.nn.-r-n-n-cell-residual-wrapper.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.nn.-r-n-n-cell-residual-wrapper.pbtxt
index faf951f2153..a514f8ceea5 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.nn.-r-n-n-cell-residual-wrapper.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.nn.-r-n-n-cell-residual-wrapper.pbtxt
@@ -98,6 +98,10 @@ tf_class {
     name: "submodules"
     mtype: "<type \'property\'>"
   }
+  member {
+    name: "supports_masking"
+    mtype: "<type \'property\'>"
+  }
   member {
     name: "trainable"
     mtype: "<type \'property\'>"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.raw_ops.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.raw_ops.pbtxt
index d4776eef338..e8cb384aa7b 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.raw_ops.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.raw_ops.pbtxt
@@ -982,7 +982,7 @@ tf_module {
   }
   member_method {
     name: "DebugIdentityV2"
-    argspec: "args=[\'input\', \'tfdbg_context_id\', \'op_name\', \'output_slot\', \'tensor_debug_mode\', \'debug_urls\', \'name\'], varargs=None, keywords=None, defaults=[\'\', \'\', \'-1\', \'-1\', \'[]\', \'None\'], "
+    argspec: "args=[\'input\', \'tfdbg_context_id\', \'op_name\', \'output_slot\', \'tensor_debug_mode\', \'debug_urls\', \'circular_buffer_size\', \'name\'], varargs=None, keywords=None, defaults=[\'\', \'\', \'-1\', \'-1\', \'[]\', \'1000\', \'None\'], "
   }
   member_method {
     name: "DebugNanCount"
diff --git a/tensorflow/tools/api/golden/v2/tensorflow.sysconfig.pbtxt b/tensorflow/tools/api/golden/v2/tensorflow.sysconfig.pbtxt
index 811ca18cdb4..7b05d382f6c 100644
--- a/tensorflow/tools/api/golden/v2/tensorflow.sysconfig.pbtxt
+++ b/tensorflow/tools/api/golden/v2/tensorflow.sysconfig.pbtxt
@@ -8,6 +8,10 @@ tf_module {
     name: "MONOLITHIC_BUILD"
     mtype: "<type \'int\'>"
   }
+  member_method {
+    name: "get_build_info"
+    argspec: "args=[], varargs=None, keywords=None, defaults=None"
+  }
   member_method {
     name: "get_compile_flags"
     argspec: "args=[], varargs=None, keywords=None, defaults=None"
diff --git a/tensorflow/tools/build_info/BUILD b/tensorflow/tools/build_info/BUILD
index 556dd0c86f0..938ab967235 100644
--- a/tensorflow/tools/build_info/BUILD
+++ b/tensorflow/tools/build_info/BUILD
@@ -14,6 +14,7 @@ py_binary(
     srcs_version = "PY2AND3",
     tags = ["no-remote-exec"],
     deps = [
+        "@local_config_cuda//cuda:cuda_config_py",
         "@six_archive//:six",
     ],
 )
diff --git a/tensorflow/tools/build_info/gen_build_info.py b/tensorflow/tools/build_info/gen_build_info.py
index df9068fb3d1..19478ab4dc2 100755
--- a/tensorflow/tools/build_info/gen_build_info.py
+++ b/tensorflow/tools/build_info/gen_build_info.py
@@ -1,4 +1,4 @@
-# Lint as: python2, python3
+# Lint as: python3
 # Copyright 2017 The TensorFlow Authors. All Rights Reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -22,47 +22,38 @@ import argparse
 
 import six
 
+# cuda.cuda is only valid in OSS
+try:
+  from cuda.cuda import cuda_config  # pylint: disable=g-import-not-at-top
+except ImportError:
+  cuda_config = None
 
-def write_build_info(filename, is_config_cuda, is_config_rocm, key_value_list):
+
+def write_build_info(filename, key_value_list):
   """Writes a Python that describes the build.
 
   Args:
     filename: filename to write to.
-    is_config_cuda: Whether this build is using CUDA.
-    is_config_rocm: Whether this build is using ROCm.
     key_value_list: A list of "key=value" strings that will be added to the
-      module as additional fields.
-
-  Raises:
-    ValueError: If `key_value_list` includes the key "is_cuda_build", which
-      would clash with one of the default fields.
+      module's "build_info" dictionary as additional entries.
   """
-  module_docstring = "\"\"\"Generates a Python module containing information "
-  module_docstring += "about the build.\"\"\""
 
-  build_config_rocm_bool = "False"
-  build_config_cuda_bool = "False"
+  build_info = {}
 
-  if is_config_rocm == "True":
-    build_config_rocm_bool = "True"
-  elif is_config_cuda == "True":
-    build_config_cuda_bool = "True"
+  if cuda_config:
+    build_info.update(cuda_config.config)
 
-  key_value_pair_stmts = []
-  if key_value_list:
-    for arg in key_value_list:
-      key, value = six.ensure_str(arg).split("=")
-      if key == "is_cuda_build":
-        raise ValueError("The key \"is_cuda_build\" cannot be passed as one of "
-                         "the --key_value arguments.")
-      if key == "is_rocm_build":
-        raise ValueError("The key \"is_rocm_build\" cannot be passed as one of "
-                         "the --key_value arguments.")
-      key_value_pair_stmts.append("%s = %r" % (key, value))
-  key_value_pair_content = "\n".join(key_value_pair_stmts)
+  for arg in key_value_list:
+    key, value = six.ensure_str(arg).split("=")
+    if value.lower() == "true":
+      build_info[key] = True
+    elif value.lower() == "false":
+      build_info[key] = False
+    else:
+      build_info[key] = value.format(**build_info)
 
   contents = """
-# Copyright 2017 The TensorFlow Authors. All Rights Reserved.
+# Copyright 2020 The TensorFlow Authors. All Rights Reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
 # you may not use this file except in compliance with the License.
@@ -76,33 +67,19 @@ def write_build_info(filename, is_config_cuda, is_config_rocm, key_value_list):
 # See the License for the specific language governing permissions and
 # limitations under the License.
 # ==============================================================================
-%s
+\"\"\"Auto-generated module providing information about the build.\"\"\"
 from __future__ import absolute_import
 from __future__ import division
 from __future__ import print_function
 
-is_rocm_build = %s
-is_cuda_build = %s
-
-%s
-""" % (module_docstring, build_config_rocm_bool, build_config_cuda_bool,
-       key_value_pair_content)
+build_info = {build_info}
+""".format(build_info=build_info)
   open(filename, "w").write(contents)
 
 
 parser = argparse.ArgumentParser(
     description="""Build info injection into the PIP package.""")
 
-parser.add_argument(
-    "--is_config_cuda",
-    type=str,
-    help="'True' for CUDA GPU builds, 'False' otherwise.")
-
-parser.add_argument(
-    "--is_config_rocm",
-    type=str,
-    help="'True' for ROCm GPU builds, 'False' otherwise.")
-
 parser.add_argument("--raw_generate", type=str, help="Generate build_info.py")
 
 parser.add_argument(
@@ -110,10 +87,7 @@ parser.add_argument(
 
 args = parser.parse_args()
 
-if (args.raw_generate is not None) and (args.is_config_cuda is not None) and (
-    args.is_config_rocm is not None):
-  write_build_info(args.raw_generate, args.is_config_cuda, args.is_config_rocm,
-                   args.key_value)
+if args.raw_generate:
+  write_build_info(args.raw_generate, args.key_value)
 else:
-  raise RuntimeError(
-      "--raw_generate, --is_config_cuda and --is_config_rocm must be used")
+  raise RuntimeError("--raw_generate must be used.")
diff --git a/tensorflow/tools/ci_build/Dockerfile.custom_op_ubuntu_16 b/tensorflow/tools/ci_build/Dockerfile.custom_op_ubuntu_16
index c540d03df7f..305c5a7c381 100644
--- a/tensorflow/tools/ci_build/Dockerfile.custom_op_ubuntu_16
+++ b/tensorflow/tools/ci_build/Dockerfile.custom_op_ubuntu_16
@@ -75,7 +75,7 @@ RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.6 0
 # Install given tensorflow or tf-nightly version, if not specified, install the # latest official release
 ARG TF_PACKAGE=tensorflow
 ARG TF_PACKAGE_VERSION=
-RUN pip install ${TF_PACKAGE}${TF_PACKAGE_VERSION:+==${TF_PACKAGE_VERSION}}
+RUN pip3 install ${TF_PACKAGE}${TF_PACKAGE_VERSION:+==${TF_PACKAGE_VERSION}}
 
 # TODO(klimek): Figure out a better way to get the right include paths
 # forwarded when we install new packages.
diff --git a/tensorflow/tools/ci_build/Dockerfile.custom_op_ubuntu_16_cuda10.0 b/tensorflow/tools/ci_build/Dockerfile.custom_op_ubuntu_16_cuda10.0
index 1a1200095b1..0732cd834a3 100644
--- a/tensorflow/tools/ci_build/Dockerfile.custom_op_ubuntu_16_cuda10.0
+++ b/tensorflow/tools/ci_build/Dockerfile.custom_op_ubuntu_16_cuda10.0
@@ -75,7 +75,7 @@ RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.6 0
 # Install given tensorflow or tf-nightly version, if not specified, install the # latest official release
 ARG TF_PACKAGE=tensorflow
 ARG TF_PACKAGE_VERSION=
-RUN pip install ${TF_PACKAGE}${TF_PACKAGE_VERSION:+==${TF_PACKAGE_VERSION}}
+RUN pip3 install ${TF_PACKAGE}${TF_PACKAGE_VERSION:+==${TF_PACKAGE_VERSION}}
 
 # TODO(klimek): Figure out a better way to get the right include paths
 # forwarded when we install new packages.
diff --git a/tensorflow/tools/ci_build/Dockerfile.custom_op_ubuntu_16_cuda10.1 b/tensorflow/tools/ci_build/Dockerfile.custom_op_ubuntu_16_cuda10.1
index 01f34365ff2..4d58ad67df6 100644
--- a/tensorflow/tools/ci_build/Dockerfile.custom_op_ubuntu_16_cuda10.1
+++ b/tensorflow/tools/ci_build/Dockerfile.custom_op_ubuntu_16_cuda10.1
@@ -77,7 +77,7 @@ RUN update-alternatives --install /usr/bin/python python /usr/bin/python3.6 0
 # Install given tensorflow or tf-nightly version, if not specified, install the # latest official release
 ARG TF_PACKAGE=tensorflow
 ARG TF_PACKAGE_VERSION=
-RUN pip install ${TF_PACKAGE}${TF_PACKAGE_VERSION:+==${TF_PACKAGE_VERSION}}
+RUN pip3 install ${TF_PACKAGE}${TF_PACKAGE_VERSION:+==${TF_PACKAGE_VERSION}}
 
 # TODO(klimek): Figure out a better way to get the right include paths
 # forwarded when we install new packages.
diff --git a/tensorflow/tools/ci_build/builds/docker_cpu_pip.sh b/tensorflow/tools/ci_build/builds/docker_cpu_pip.sh
index cf0036fb98f..e5f02062942 100755
--- a/tensorflow/tools/ci_build/builds/docker_cpu_pip.sh
+++ b/tensorflow/tools/ci_build/builds/docker_cpu_pip.sh
@@ -44,13 +44,14 @@ bazel --output_base=/tmp test --define=no_tensorflow_py_deps=true \
       --test_lang_filters=py \
       --build_tests_only \
       -k \
-      --test_tag_filters=-no_oss,-oss_serial,-no_pip,-nopip \
+      --test_tag_filters=-no_oss,-oss_serial,-no_pip,-nopip,-gpu,-tpu \
       --test_size_filters=small,medium \
       --test_timeout 300,450,1200,3600 \
       --test_output=errors \
       -- //${PIP_TEST_ROOT}/tensorflow/python/... \
+      -//${PIP_TEST_ROOT}/tensorflow/python/compiler/xla:xla_test \
+      -//${PIP_TEST_ROOT}/tensorflow/python/distribute:parameter_server_strategy_test \
       -//${PIP_TEST_ROOT}/tensorflow/python:virtual_gpu_test \
-      -//${PIP_TEST_ROOT}/tensorflow/python:virtual_gpu_test_gpu \
-      -//${PIP_TEST_ROOT}/tensorflow/python:collective_ops_gpu_test \
-      -//${PIP_TEST_ROOT}/tensorflow/python:collective_ops_gpu_test_gpu
-
+      -//${PIP_TEST_ROOT}/tensorflow/python:collective_ops_gpu_test
+# The above tests are excluded because they seem to require a GPU.
+# TODO(yifeif): Investigate and potentially add an unconditional 'gpu' tag.
diff --git a/tensorflow/tools/ci_build/presubmit/ubuntu_16/sanity/build.sh b/tensorflow/tools/ci_build/presubmit/ubuntu_16/sanity/build.sh
index d111a3bb658..bd2f71abc68 100644
--- a/tensorflow/tools/ci_build/presubmit/ubuntu_16/sanity/build.sh
+++ b/tensorflow/tools/ci_build/presubmit/ubuntu_16/sanity/build.sh
@@ -29,7 +29,6 @@ function install_pylint () {
   # TODO(mihaimaruseac): this is used in the release build in the same way,
   # maybe extract out to a common?
   sudo python3 -m pip install setuptools --upgrade
-  sudo python2 -m pip install pylint==1.6.4
   sudo python3 -m pip install pylint==1.6.4
 }
 
diff --git a/tensorflow/tools/pip_package/setup.py b/tensorflow/tools/pip_package/setup.py
index 01a3696823d..92061b396ce 100644
--- a/tensorflow/tools/pip_package/setup.py
+++ b/tensorflow/tools/pip_package/setup.py
@@ -1,3 +1,4 @@
+# lint as: python3
 # Copyright 2015 The TensorFlow Authors. All Rights Reserved.
 #
 # Licensed under the Apache License, Version 2.0 (the "License");
@@ -113,7 +114,8 @@ CONSOLE_SCRIPTS = [
     # even though the command is not removed, just moved to a different wheel.
     'tensorboard = tensorboard.main:run_main',
     'tf_upgrade_v2 = tensorflow.tools.compatibility.tf_upgrade_v2_main:main',
-    'estimator_ckpt_converter = tensorflow_estimator.python.estimator.tools.checkpoint_converter:main',
+    'estimator_ckpt_converter = '
+    'tensorflow_estimator.python.estimator.tools.checkpoint_converter:main',
 ]
 # pylint: enable=line-too-long
 
@@ -152,11 +154,10 @@ class InstallHeaders(Command):
   """
   description = 'install C/C++ header files'
 
-  user_options = [('install-dir=', 'd',
-                   'directory to install header files to'),
-                  ('force', 'f',
-                   'force installation (overwrite existing files)'),
-                 ]
+  user_options = [
+      ('install-dir=', 'd', 'directory to install header files to'),
+      ('force', 'f', 'force installation (overwrite existing files)'),
+  ]
 
   boolean_options = ['force']
 
@@ -166,8 +167,7 @@ class InstallHeaders(Command):
     self.outfiles = []
 
   def finalize_options(self):
-    self.set_undefined_options('install',
-                               ('install_headers', 'install_dir'),
+    self.set_undefined_options('install', ('install_headers', 'install_dir'),
                                ('force', 'force'))
 
   def mkdir_and_copy_file(self, header):
@@ -227,9 +227,7 @@ so_lib_paths = [
 
 matches = []
 for path in so_lib_paths:
-  matches.extend(
-      ['../' + x for x in find_files('*', path) if '.py' not in x]
-  )
+  matches.extend(['../' + x for x in find_files('*', path) if '.py' not in x])
 
 if os.name == 'nt':
   EXTENSION_NAME = 'python/_pywrap_tensorflow_internal.pyd'
@@ -250,11 +248,10 @@ headers = (
     list(find_files('*.h', 'tensorflow/stream_executor')) +
     list(find_files('*.h', 'google/com_google_protobuf/src')) +
     list(find_files('*.inc', 'google/com_google_protobuf/src')) +
-    list(find_files('*', 'third_party/eigen3')) + list(
-        find_files('*.h', 'tensorflow/include/external/com_google_absl')) +
-    list(
-        find_files('*.inc', 'tensorflow/include/external/com_google_absl'))
-    + list(find_files('*', 'tensorflow/include/external/eigen_archive')))
+    list(find_files('*', 'third_party/eigen3')) +
+    list(find_files('*.h', 'tensorflow/include/external/com_google_absl')) +
+    list(find_files('*.inc', 'tensorflow/include/external/com_google_absl')) +
+    list(find_files('*', 'tensorflow/include/external/eigen_archive')))
 
 setup(
     name=project_name,
@@ -287,7 +284,7 @@ setup(
         'install': InstallCommand,
     },
     # PyPI package information.
-    classifiers=[
+    classifiers=sorted([
         'Development Status :: 5 - Production/Stable',
         'Intended Audience :: Developers',
         'Intended Audience :: Education',
@@ -305,7 +302,7 @@ setup(
         'Topic :: Software Development',
         'Topic :: Software Development :: Libraries',
         'Topic :: Software Development :: Libraries :: Python Modules',
-    ],
+    ]),
     license='Apache 2.0',
     keywords='tensorflow tensor machine learning',
 )
diff --git a/tensorflow/workspace.bzl b/tensorflow/workspace.bzl
index 1ac8a49028d..4d84614b1e8 100755
--- a/tensorflow/workspace.bzl
+++ b/tensorflow/workspace.bzl
@@ -164,11 +164,11 @@ def tf_repositories(path_prefix = "", tf_repo_name = ""):
 
     tf_http_archive(
         name = "XNNPACK",
-        sha256 = "dfa6181e238f0ca88a641952678cd7f3e38da541d8b731ce3fea1d0eeffb6101",
-        strip_prefix = "XNNPACK-b2217ddb5fa74db09d9da1326902269ae18e41ad",
+        sha256 = "30b468db7d85b5f4afb3fd60947d690bc1c29d4eccca8fffeabe5b5328621c0e",
+        strip_prefix = "XNNPACK-9d3a459441c272d82be14b579656b961066eba2c",
         urls = [
-            "https://storage.googleapis.com/mirror.tensorflow.org/github.com/google/XNNPACK/archive/b2217ddb5fa74db09d9da1326902269ae18e41ad.zip",
-            "https://github.com/google/XNNPACK/archive/b2217ddb5fa74db09d9da1326902269ae18e41ad.zip",
+            "https://storage.googleapis.com/mirror.tensorflow.org/github.com/google/XNNPACK/archive/9d3a459441c272d82be14b579656b961066eba2c.zip",
+            "https://github.com/google/XNNPACK/archive/9d3a459441c272d82be14b579656b961066eba2c.zip",
         ],
     )
 
@@ -237,11 +237,11 @@ def tf_repositories(path_prefix = "", tf_repo_name = ""):
         name = "eigen_archive",
         build_file = clean_dep("//third_party:eigen.BUILD"),
         patch_file = clean_dep("//third_party/eigen3:gpu_packet_math.patch"),
-        sha256 = "615be1295290c13039b0c980a4a55933be26b1e06194d86c6014876fa85c7c6b",  # SHARED_EIGEN_SHA
-        strip_prefix = "eigen-8719b9c5bc1a97e62d675c02495ed72dda6fae73",
+        sha256 = "caee94d191b26265ceab19193d35d5176bba206407343b967a08ddafc3ebd167",  # SHARED_EIGEN_SHA
+        strip_prefix = "eigen-c2ab36f47a34e572f37e3dd556ac8a04ab769277",
         urls = [
-            "https://storage.googleapis.com/mirror.tensorflow.org/gitlab.com/libeigen/eigen/-/archive/8719b9c5bc1a97e62d675c02495ed72dda6fae73/eigen-8719b9c5bc1a97e62d675c02495ed72dda6fae73.tar.gz",
-            "https://gitlab.com/libeigen/eigen/-/archive/8719b9c5bc1a97e62d675c02495ed72dda6fae73/eigen-8719b9c5bc1a97e62d675c02495ed72dda6fae73.tar.gz",
+            "https://storage.googleapis.com/mirror.tensorflow.org/gitlab.com/libeigen/eigen/-/archive/c2ab36f47a34e572f37e3dd556ac8a04ab769277/eigen-c2ab36f47a34e572f37e3dd556ac8a04ab769277.tar.gz",
+            "https://gitlab.com/libeigen/eigen/-/archive/c2ab36f47a34e572f37e3dd556ac8a04ab769277/eigen-c2ab36f47a34e572f37e3dd556ac8a04ab769277.tar.gz",
         ],
     )
 
@@ -655,8 +655,8 @@ def tf_repositories(path_prefix = "", tf_repo_name = ""):
     )
 
     # Check out LLVM and MLIR from llvm-project.
-    LLVM_COMMIT = "034a7b6604067b0ccb36c761a5782456b76c447e"
-    LLVM_SHA256 = "87bd4dd8c2620ae6371dcfeeb6f1583918945c829cb115020ad4bc0a74a079d7"
+    LLVM_COMMIT = "c4b5a66e44f031eb89c9d6ea32b144f1169bdbae"
+    LLVM_SHA256 = "8463cbed08a66c7171c831e9549076cf3fd4f7e6fe690b9b799d6afef2465007"
     LLVM_URLS = [
         "https://storage.googleapis.com/mirror.tensorflow.org/github.com/llvm/llvm-project/archive/{commit}.tar.gz".format(commit = LLVM_COMMIT),
         "https://github.com/llvm/llvm-project/archive/{commit}.tar.gz".format(commit = LLVM_COMMIT),
diff --git a/third_party/cpuinfo/BUILD.bazel b/third_party/cpuinfo/BUILD.bazel
index 5510905cbef..2a2be96d82f 100644
--- a/third_party/cpuinfo/BUILD.bazel
+++ b/third_party/cpuinfo/BUILD.bazel
@@ -160,6 +160,16 @@ cc_library(
     ],
 )
 
+cc_library(
+    name = "cpuinfo_with_unstripped_include_path",
+    hdrs = [
+        "include/cpuinfo.h",
+    ],
+    deps = [
+        ":cpuinfo_impl",
+    ],
+)
+
 ############################# Build configurations #############################
 
 config_setting(
diff --git a/third_party/gpus/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc.tpl b/third_party/gpus/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc.tpl
index 9cc06ef99f5..a48ef8bf35a 100755
--- a/third_party/gpus/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc.tpl
+++ b/third_party/gpus/crosstool/clang/bin/crosstool_wrapper_driver_is_not_gcc.tpl
@@ -269,7 +269,6 @@ def main():
   if args.x and args.x[0] == 'cuda':
     if args.cuda_log: Log('-x cuda')
     leftover = [pipes.quote(s) for s in leftover]
-    args.cuda_log = True
     if args.cuda_log: Log('using nvcc')
     return InvokeNvcc(leftover, log=args.cuda_log)
 
diff --git a/third_party/gpus/cuda/BUILD.tpl b/third_party/gpus/cuda/BUILD.tpl
index 6b5318f60c2..e5833e7cdbb 100644
--- a/third_party/gpus/cuda/BUILD.tpl
+++ b/third_party/gpus/cuda/BUILD.tpl
@@ -218,4 +218,9 @@ bzl_library(
     ],
 )
 
+py_library(
+    name = "cuda_config_py",
+    srcs = ["cuda/cuda_config.py"]
+)
+
 %{copy_rules}
diff --git a/third_party/gpus/cuda/BUILD.windows.tpl b/third_party/gpus/cuda/BUILD.windows.tpl
index 75d360fa17e..e83da1429dd 100644
--- a/third_party/gpus/cuda/BUILD.windows.tpl
+++ b/third_party/gpus/cuda/BUILD.windows.tpl
@@ -204,4 +204,9 @@ bzl_library(
     ],
 )
 
+py_library(
+    name = "cuda_config_py",
+    srcs = ["cuda/cuda_config.py"]
+)
+
 %{copy_rules}
diff --git a/third_party/gpus/cuda/cuda_config.py.tpl b/third_party/gpus/cuda/cuda_config.py.tpl
new file mode 100644
index 00000000000..eb041a17e8a
--- /dev/null
+++ b/third_party/gpus/cuda/cuda_config.py.tpl
@@ -0,0 +1,17 @@
+# Lint as: python3
+# Copyright 2019 The TensorFlow Authors. All Rights Reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# ==============================================================================
+
+config = %{cuda_config}
diff --git a/third_party/gpus/cuda_configure.bzl b/third_party/gpus/cuda_configure.bzl
index a0aefa77e9d..90ce206db01 100644
--- a/third_party/gpus/cuda_configure.bzl
+++ b/third_party/gpus/cuda_configure.bzl
@@ -824,6 +824,15 @@ filegroup(name="cudnn-include")
         "cuda/cuda/cuda_config.h",
     )
 
+    # Set up cuda_config.py, which is used by gen_build_info to provide
+    # static build environment info to the API
+    _tpl(
+        repository_ctx,
+        "cuda:cuda_config.py",
+        _py_tmpl_dict({}),
+        "cuda/cuda/cuda_config.py",
+    )
+
     # If cuda_configure is not configured to build with GPU support, and the user
     # attempts to build with --config=cuda, add a dummy build rule to intercept
     # this and fail with an actionable error message.
@@ -938,6 +947,7 @@ def _create_local_cuda_repository(repository_ctx):
         "crosstool:BUILD",
         "crosstool:cc_toolchain_config.bzl",
         "cuda:cuda_config.h",
+        "cuda:cuda_config.py",
     ]}
     tpl_paths["cuda:BUILD"] = _tpl_path(repository_ctx, "cuda:BUILD.windows" if is_windows(repository_ctx) else "cuda:BUILD")
     find_cuda_config_script = repository_ctx.path(Label("@org_tensorflow//third_party/gpus:find_cuda_config.py.gz.base64"))
@@ -1273,6 +1283,22 @@ def _create_local_cuda_repository(repository_ctx):
         },
     )
 
+    # Set up cuda_config.py, which is used by gen_build_info to provide
+    # static build environment info to the API
+    repository_ctx.template(
+        "cuda/cuda/cuda_config.py",
+        tpl_paths["cuda:cuda_config.py"],
+        _py_tmpl_dict({
+            "cuda_version": cuda_config.cuda_version,
+            "cudnn_version": cuda_config.cudnn_version,
+            "cuda_compute_capabilities": cuda_config.compute_capabilities,
+            "cpu_compiler": str(cc),
+        }),
+    )
+
+def _py_tmpl_dict(d):
+    return {"%{cuda_config}": str(d)}
+
 def _create_remote_cuda_repository(repository_ctx, remote_config_repo):
     """Creates pointers to a remotely configured repo set up to build with CUDA."""
     _tpl(
@@ -1301,6 +1327,11 @@ def _create_remote_cuda_repository(repository_ctx, remote_config_repo):
         config_repo_label(remote_config_repo, "cuda:cuda/cuda_config.h"),
         {},
     )
+    repository_ctx.template(
+        "cuda/cuda/cuda_config.py",
+        config_repo_label(remote_config_repo, "cuda:cuda/cuda_config.py"),
+        _py_tmpl_dict({}),
+    )
 
     repository_ctx.template(
         "crosstool/BUILD",
diff --git a/third_party/llvm/llvm.autogenerated.BUILD b/third_party/llvm/llvm.autogenerated.BUILD
index c16b62f635a..ecc9e48bfb2 100644
--- a/third_party/llvm/llvm.autogenerated.BUILD
+++ b/third_party/llvm/llvm.autogenerated.BUILD
@@ -577,6 +577,7 @@ gentbl(
         ("-gen-global-isel", "lib/Target/AMDGPU/AMDGPUGenGlobalISel.inc"),
         ("-gen-global-isel-combiner -combiners=AMDGPUPreLegalizerCombinerHelper", "lib/Target/AMDGPU/AMDGPUGenPreLegalizeGICombiner.inc"),
         ("-gen-global-isel-combiner -combiners=AMDGPUPostLegalizerCombinerHelper", "lib/Target/AMDGPU/AMDGPUGenPostLegalizeGICombiner.inc"),
+        ("-gen-global-isel-combiner -combiners=AMDGPURegBankCombinerHelper", "lib/Target/AMDGPU/AMDGPUGenRegBankGICombiner.inc"),
     ],
     tblgen = ":llvm-tblgen",
     td_file = "lib/Target/AMDGPU/AMDGPUGISel.td",
@@ -659,7 +660,7 @@ cc_library(
 )
 
 cc_library(
-    name = "aarch64_asm_parser",
+    name = "AArch64AsmParser",
     srcs = glob([
         "lib/Target/AArch64/AsmParser/*.c",
         "lib/Target/AArch64/AsmParser/*.cpp",
@@ -673,18 +674,23 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/AArch64"],
     deps = [
-        ":aarch64_desc",
-        ":aarch64_info",
-        ":aarch64_utils",
+        ":AArch64Desc",
+        ":AArch64Info",
+        ":AArch64Utils",
+        ":MC",
+        ":MCParser",
+        ":Support",
         ":config",
-        ":mc",
-        ":mc_parser",
-        ":support",
     ],
 )
 
+alias(
+    name = "aarch64_asm_parser",
+    actual = ":AArch64AsmParser",
+)
+
 cc_library(
-    name = "aarch64_code_gen",
+    name = "AArch64CodeGen",
     srcs = glob([
         "lib/Target/AArch64/*.c",
         "lib/Target/AArch64/*.cpp",
@@ -698,27 +704,32 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/AArch64"],
     deps = [
-        ":aarch64_desc",
-        ":aarch64_info",
-        ":aarch64_utils",
-        ":analysis",
-        ":asm_printer",
-        ":cf_guard",
-        ":code_gen",
+        ":AArch64Desc",
+        ":AArch64Info",
+        ":AArch64Utils",
+        ":Analysis",
+        ":AsmPrinter",
+        ":CFGuard",
+        ":CodeGen",
+        ":Core",
+        ":GlobalISel",
+        ":MC",
+        ":Scalar",
+        ":SelectionDAG",
+        ":Support",
+        ":Target",
+        ":TransformUtils",
         ":config",
-        ":core",
-        ":global_i_sel",
-        ":mc",
-        ":scalar",
-        ":selection_dag",
-        ":support",
-        ":target",
-        ":transform_utils",
     ],
 )
 
+alias(
+    name = "aarch64_code_gen",
+    actual = ":AArch64CodeGen",
+)
+
 cc_library(
-    name = "aarch64_desc",
+    name = "AArch64Desc",
     srcs = glob([
         "lib/Target/AArch64/MCTargetDesc/*.c",
         "lib/Target/AArch64/MCTargetDesc/*.cpp",
@@ -732,21 +743,26 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/AArch64"],
     deps = [
-        ":aarch64_info",
+        ":AArch64Info",
+        ":AArch64Utils",
+        ":BinaryFormat",
+        ":MC",
+        ":Support",
         ":aarch64_target_gen",
-        ":aarch64_utils",
         ":attributes_gen",
-        ":binary_format",
         ":config",
         ":intrinsic_enums_gen",
         ":intrinsics_impl_gen",
-        ":mc",
-        ":support",
     ],
 )
 
+alias(
+    name = "aarch64_desc",
+    actual = ":AArch64Desc",
+)
+
 cc_library(
-    name = "aarch64_disassembler",
+    name = "AArch64Disassembler",
     srcs = glob([
         "lib/Target/AArch64/Disassembler/*.c",
         "lib/Target/AArch64/Disassembler/*.cpp",
@@ -760,18 +776,23 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/AArch64"],
     deps = [
-        ":aarch64_desc",
-        ":aarch64_info",
-        ":aarch64_utils",
+        ":AArch64Desc",
+        ":AArch64Info",
+        ":AArch64Utils",
+        ":MC",
+        ":MCDisassembler",
+        ":Support",
         ":config",
-        ":mc",
-        ":mc_disassembler",
-        ":support",
     ],
 )
 
+alias(
+    name = "aarch64_disassembler",
+    actual = ":AArch64Disassembler",
+)
+
 cc_library(
-    name = "aarch64_info",
+    name = "AArch64Info",
     srcs = glob([
         "lib/Target/AArch64/TargetInfo/*.c",
         "lib/Target/AArch64/TargetInfo/*.cpp",
@@ -788,15 +809,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/AArch64"],
     deps = [
+        ":Support",
         ":code_gen",
         ":config",
-        ":support",
         ":target",
     ],
 )
 
+alias(
+    name = "aarch64_info",
+    actual = ":AArch64Info",
+)
+
 cc_library(
-    name = "aarch64_utils",
+    name = "AArch64Utils",
     srcs = glob([
         "lib/Target/AArch64/Utils/*.c",
         "lib/Target/AArch64/Utils/*.cpp",
@@ -811,15 +837,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/AArch64"],
     deps = [
+        ":Support",
         ":aarch64_target_gen",
         ":config",
         ":mc",
-        ":support",
     ],
 )
 
+alias(
+    name = "aarch64_utils",
+    actual = ":AArch64Utils",
+)
+
 cc_library(
-    name = "amdgpu_asm_parser",
+    name = "AMDGPUAsmParser",
     srcs = glob([
         "lib/Target/AMDGPU/AsmParser/*.c",
         "lib/Target/AMDGPU/AsmParser/*.cpp",
@@ -833,18 +864,23 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/AMDGPU"],
     deps = [
-        ":amdgpu_desc",
-        ":amdgpu_info",
-        ":amdgpu_utils",
+        ":AMDGPUDesc",
+        ":AMDGPUInfo",
+        ":AMDGPUUtils",
+        ":MC",
+        ":MCParser",
+        ":Support",
         ":config",
-        ":mc",
-        ":mc_parser",
-        ":support",
     ],
 )
 
+alias(
+    name = "amdgpu_asm_parser",
+    actual = ":AMDGPUAsmParser",
+)
+
 cc_library(
-    name = "amdgpu_code_gen",
+    name = "AMDGPUCodeGen",
     srcs = glob([
         "lib/Target/AMDGPU/*.c",
         "lib/Target/AMDGPU/*.cpp",
@@ -858,30 +894,35 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/AMDGPU"],
     deps = [
-        ":amdgpu_desc",
-        ":amdgpu_info",
-        ":amdgpu_utils",
-        ":analysis",
-        ":asm_printer",
-        ":binary_format",
-        ":code_gen",
+        ":AMDGPUDesc",
+        ":AMDGPUInfo",
+        ":AMDGPUUtils",
+        ":Analysis",
+        ":AsmPrinter",
+        ":BinaryFormat",
+        ":CodeGen",
+        ":Core",
+        ":GlobalISel",
+        ":IPO",
+        ":MC",
+        ":MIRParser",
+        ":Scalar",
+        ":SelectionDAG",
+        ":Support",
+        ":Target",
+        ":TransformUtils",
+        ":Vectorize",
         ":config",
-        ":core",
-        ":global_i_sel",
-        ":ipo",
-        ":mc",
-        ":mir_parser",
-        ":scalar",
-        ":selection_dag",
-        ":support",
-        ":target",
-        ":transform_utils",
-        ":vectorize",
     ],
 )
 
+alias(
+    name = "amdgpu_code_gen",
+    actual = ":AMDGPUCodeGen",
+)
+
 cc_library(
-    name = "amdgpu_desc",
+    name = "AMDGPUDesc",
     srcs = glob([
         "lib/Target/AMDGPU/MCTargetDesc/*.c",
         "lib/Target/AMDGPU/MCTargetDesc/*.cpp",
@@ -895,18 +936,23 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/AMDGPU"],
     deps = [
-        ":amdgpu_info",
-        ":amdgpu_utils",
-        ":binary_format",
+        ":AMDGPUInfo",
+        ":AMDGPUUtils",
+        ":BinaryFormat",
+        ":Core",
+        ":MC",
+        ":Support",
         ":config",
-        ":core",
-        ":mc",
-        ":support",
     ],
 )
 
+alias(
+    name = "amdgpu_desc",
+    actual = ":AMDGPUDesc",
+)
+
 cc_library(
-    name = "amdgpu_disassembler",
+    name = "AMDGPUDisassembler",
     srcs = glob([
         "lib/Target/AMDGPU/Disassembler/*.c",
         "lib/Target/AMDGPU/Disassembler/*.cpp",
@@ -920,18 +966,23 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/AMDGPU"],
     deps = [
-        ":amdgpu_desc",
-        ":amdgpu_info",
-        ":amdgpu_utils",
+        ":AMDGPUDesc",
+        ":AMDGPUInfo",
+        ":AMDGPUUtils",
+        ":MC",
+        ":MCDisassembler",
+        ":Support",
         ":config",
-        ":mc",
-        ":mc_disassembler",
-        ":support",
     ],
 )
 
+alias(
+    name = "amdgpu_disassembler",
+    actual = ":AMDGPUDisassembler",
+)
+
 cc_library(
-    name = "amdgpu_info",
+    name = "AMDGPUInfo",
     srcs = glob([
         "lib/Target/AMDGPU/TargetInfo/*.c",
         "lib/Target/AMDGPU/TargetInfo/*.cpp",
@@ -945,16 +996,21 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/AMDGPU"],
     deps = [
+        ":Support",
         ":amdgpu_r600_target_gen",
         ":amdgpu_target_gen",
         ":config",
         ":core",
-        ":support",
     ],
 )
 
+alias(
+    name = "amdgpu_info",
+    actual = ":AMDGPUInfo",
+)
+
 cc_library(
-    name = "amdgpu_utils",
+    name = "AMDGPUUtils",
     srcs = glob([
         "lib/Target/AMDGPU/Utils/*.c",
         "lib/Target/AMDGPU/Utils/*.cpp",
@@ -968,18 +1024,23 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/AMDGPU"],
     deps = [
+        ":BinaryFormat",
+        ":Core",
+        ":MC",
+        ":Support",
         ":amdgpu_r600_target_gen",
         ":amdgpu_target_gen",
-        ":binary_format",
         ":config",
-        ":core",
-        ":mc",
-        ":support",
     ],
 )
 
+alias(
+    name = "amdgpu_utils",
+    actual = ":AMDGPUUtils",
+)
+
 cc_library(
-    name = "arc_code_gen",
+    name = "ARCCodeGen",
     srcs = glob([
         "lib/Target/ARC/*.c",
         "lib/Target/ARC/*.cpp",
@@ -993,23 +1054,28 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/ARC"],
     deps = [
-        ":analysis",
-        ":arc_desc",
-        ":arc_info",
-        ":asm_printer",
-        ":code_gen",
+        ":ARCDesc",
+        ":ARCInfo",
+        ":Analysis",
+        ":AsmPrinter",
+        ":CodeGen",
+        ":Core",
+        ":MC",
+        ":SelectionDAG",
+        ":Support",
+        ":Target",
+        ":TransformUtils",
         ":config",
-        ":core",
-        ":mc",
-        ":selection_dag",
-        ":support",
-        ":target",
-        ":transform_utils",
     ],
 )
 
+alias(
+    name = "arc_code_gen",
+    actual = ":ARCCodeGen",
+)
+
 cc_library(
-    name = "arc_desc",
+    name = "ARCDesc",
     srcs = glob([
         "lib/Target/ARC/MCTargetDesc/*.c",
         "lib/Target/ARC/MCTargetDesc/*.cpp",
@@ -1023,15 +1089,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/ARC"],
     deps = [
-        ":arc_info",
+        ":ARCInfo",
+        ":MC",
+        ":Support",
         ":config",
-        ":mc",
-        ":support",
     ],
 )
 
+alias(
+    name = "arc_desc",
+    actual = ":ARCDesc",
+)
+
 cc_library(
-    name = "arc_disassembler",
+    name = "ARCDisassembler",
     srcs = glob([
         "lib/Target/ARC/Disassembler/*.c",
         "lib/Target/ARC/Disassembler/*.cpp",
@@ -1045,15 +1116,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/ARC"],
     deps = [
-        ":arc_info",
+        ":ARCInfo",
+        ":MCDisassembler",
+        ":Support",
         ":config",
-        ":mc_disassembler",
-        ":support",
     ],
 )
 
+alias(
+    name = "arc_disassembler",
+    actual = ":ARCDisassembler",
+)
+
 cc_library(
-    name = "arc_info",
+    name = "ARCInfo",
     srcs = glob([
         "lib/Target/ARC/TargetInfo/*.c",
         "lib/Target/ARC/TargetInfo/*.cpp",
@@ -1067,13 +1143,18 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/ARC"],
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "arc_info",
+    actual = ":ARCInfo",
+)
+
 cc_library(
-    name = "arm_asm_parser",
+    name = "ARMAsmParser",
     srcs = glob([
         "lib/Target/ARM/AsmParser/*.c",
         "lib/Target/ARM/AsmParser/*.cpp",
@@ -1087,18 +1168,23 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/ARM"],
     deps = [
-        ":arm_desc",
-        ":arm_info",
-        ":arm_utils",
+        ":ARMDesc",
+        ":ARMInfo",
+        ":ARMUtils",
+        ":MC",
+        ":MCParser",
+        ":Support",
         ":config",
-        ":mc",
-        ":mc_parser",
-        ":support",
     ],
 )
 
+alias(
+    name = "arm_asm_parser",
+    actual = ":ARMAsmParser",
+)
+
 cc_library(
-    name = "arm_code_gen",
+    name = "ARMCodeGen",
     srcs = glob([
         "lib/Target/ARM/*.c",
         "lib/Target/ARM/*.cpp",
@@ -1112,27 +1198,32 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/ARM"],
     deps = [
-        ":analysis",
-        ":arm_desc",
-        ":arm_info",
-        ":arm_utils",
-        ":asm_printer",
-        ":cf_guard",
-        ":code_gen",
+        ":ARMDesc",
+        ":ARMInfo",
+        ":ARMUtils",
+        ":Analysis",
+        ":AsmPrinter",
+        ":CFGuard",
+        ":CodeGen",
+        ":Core",
+        ":GlobalISel",
+        ":MC",
+        ":Scalar",
+        ":SelectionDAG",
+        ":Support",
+        ":Target",
+        ":TransformUtils",
         ":config",
-        ":core",
-        ":global_i_sel",
-        ":mc",
-        ":scalar",
-        ":selection_dag",
-        ":support",
-        ":target",
-        ":transform_utils",
     ],
 )
 
+alias(
+    name = "arm_code_gen",
+    actual = ":ARMCodeGen",
+)
+
 cc_library(
-    name = "arm_desc",
+    name = "ARMDesc",
     srcs = glob([
         "lib/Target/ARM/MCTargetDesc/*.c",
         "lib/Target/ARM/MCTargetDesc/*.cpp",
@@ -1148,22 +1239,27 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/ARM"],
     deps = [
-        ":arm_info",
+        ":ARMInfo",
+        ":ARMUtils",
+        ":BinaryFormat",
+        ":MC",
+        ":MCDisassembler",
+        ":Support",
         ":arm_target_gen",
-        ":arm_utils",
         ":attributes_gen",
-        ":binary_format",
         ":config",
         ":intrinsic_enums_gen",
         ":intrinsics_impl_gen",
-        ":mc",
-        ":mc_disassembler",
-        ":support",
     ],
 )
 
+alias(
+    name = "arm_desc",
+    actual = ":ARMDesc",
+)
+
 cc_library(
-    name = "arm_disassembler",
+    name = "ARMDisassembler",
     srcs = glob([
         "lib/Target/ARM/Disassembler/*.c",
         "lib/Target/ARM/Disassembler/*.cpp",
@@ -1177,17 +1273,22 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/ARM"],
     deps = [
-        ":arm_desc",
-        ":arm_info",
-        ":arm_utils",
+        ":ARMDesc",
+        ":ARMInfo",
+        ":ARMUtils",
+        ":MCDisassembler",
+        ":Support",
         ":config",
-        ":mc_disassembler",
-        ":support",
     ],
 )
 
+alias(
+    name = "arm_disassembler",
+    actual = ":ARMDisassembler",
+)
+
 cc_library(
-    name = "arm_info",
+    name = "ARMInfo",
     srcs = glob([
         "lib/Target/ARM/TargetInfo/*.c",
         "lib/Target/ARM/TargetInfo/*.cpp",
@@ -1202,15 +1303,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/ARM"],
     deps = [
+        ":Support",
         ":arm_target_gen",
         ":config",
-        ":support",
         ":target",
     ],
 )
 
+alias(
+    name = "arm_info",
+    actual = ":ARMInfo",
+)
+
 cc_library(
-    name = "arm_utils",
+    name = "ARMUtils",
     srcs = glob([
         "lib/Target/ARM/Utils/*.c",
         "lib/Target/ARM/Utils/*.cpp",
@@ -1225,15 +1331,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/ARM"],
     deps = [
+        ":Support",
         ":arm_target_gen",
         ":config",
         ":mc",
-        ":support",
     ],
 )
 
+alias(
+    name = "arm_utils",
+    actual = ":ARMUtils",
+)
+
 cc_library(
-    name = "avr_asm_parser",
+    name = "AVRAsmParser",
     srcs = glob([
         "lib/Target/AVR/AsmParser/*.c",
         "lib/Target/AVR/AsmParser/*.cpp",
@@ -1247,17 +1358,22 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/AVR"],
     deps = [
-        ":avr_desc",
-        ":avr_info",
+        ":AVRDesc",
+        ":AVRInfo",
+        ":MC",
+        ":MCParser",
+        ":Support",
         ":config",
-        ":mc",
-        ":mc_parser",
-        ":support",
     ],
 )
 
+alias(
+    name = "avr_asm_parser",
+    actual = ":AVRAsmParser",
+)
+
 cc_library(
-    name = "avr_code_gen",
+    name = "AVRCodeGen",
     srcs = glob([
         "lib/Target/AVR/*.c",
         "lib/Target/AVR/*.cpp",
@@ -1271,21 +1387,26 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/AVR"],
     deps = [
-        ":asm_printer",
-        ":avr_desc",
-        ":avr_info",
-        ":code_gen",
+        ":AVRDesc",
+        ":AVRInfo",
+        ":AsmPrinter",
+        ":CodeGen",
+        ":Core",
+        ":MC",
+        ":SelectionDAG",
+        ":Support",
+        ":Target",
         ":config",
-        ":core",
-        ":mc",
-        ":selection_dag",
-        ":support",
-        ":target",
     ],
 )
 
+alias(
+    name = "avr_code_gen",
+    actual = ":AVRCodeGen",
+)
+
 cc_library(
-    name = "avr_desc",
+    name = "AVRDesc",
     srcs = glob([
         "lib/Target/AVR/MCTargetDesc/*.c",
         "lib/Target/AVR/MCTargetDesc/*.cpp",
@@ -1299,15 +1420,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/AVR"],
     deps = [
-        ":avr_info",
+        ":AVRInfo",
+        ":MC",
+        ":Support",
         ":config",
-        ":mc",
-        ":support",
     ],
 )
 
+alias(
+    name = "avr_desc",
+    actual = ":AVRDesc",
+)
+
 cc_library(
-    name = "avr_disassembler",
+    name = "AVRDisassembler",
     srcs = glob([
         "lib/Target/AVR/Disassembler/*.c",
         "lib/Target/AVR/Disassembler/*.cpp",
@@ -1321,15 +1447,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/AVR"],
     deps = [
-        ":avr_info",
+        ":AVRInfo",
+        ":MCDisassembler",
+        ":Support",
         ":config",
-        ":mc_disassembler",
-        ":support",
     ],
 )
 
+alias(
+    name = "avr_disassembler",
+    actual = ":AVRDisassembler",
+)
+
 cc_library(
-    name = "avr_info",
+    name = "AVRInfo",
     srcs = glob([
         "lib/Target/AVR/TargetInfo/*.c",
         "lib/Target/AVR/TargetInfo/*.cpp",
@@ -1343,13 +1474,18 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/AVR"],
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "avr_info",
+    actual = ":AVRInfo",
+)
+
 cc_library(
-    name = "aggressive_inst_combine",
+    name = "AggressiveInstCombine",
     srcs = glob([
         "lib/Transforms/AggressiveInstCombine/*.c",
         "lib/Transforms/AggressiveInstCombine/*.cpp",
@@ -1363,16 +1499,21 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":analysis",
+        ":Analysis",
+        ":Core",
+        ":Support",
+        ":TransformUtils",
         ":config",
-        ":core",
-        ":support",
-        ":transform_utils",
     ],
 )
 
+alias(
+    name = "aggressive_inst_combine",
+    actual = ":AggressiveInstCombine",
+)
+
 cc_library(
-    name = "analysis",
+    name = "Analysis",
     srcs = glob([
         "lib/Analysis/*.c",
         "lib/Analysis/*.cpp",
@@ -1388,17 +1529,22 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":binary_format",
+        ":BinaryFormat",
+        ":Core",
+        ":Object",
+        ":ProfileData",
+        ":Support",
         ":config",
-        ":core",
-        ":object",
-        ":profile_data",
-        ":support",
     ],
 )
 
+alias(
+    name = "analysis",
+    actual = ":Analysis",
+)
+
 cc_library(
-    name = "asm_parser",
+    name = "AsmParser",
     srcs = glob([
         "lib/AsmParser/*.c",
         "lib/AsmParser/*.cpp",
@@ -1412,15 +1558,20 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":binary_format",
+        ":BinaryFormat",
+        ":Core",
+        ":Support",
         ":config",
-        ":core",
-        ":support",
     ],
 )
 
+alias(
+    name = "asm_parser",
+    actual = ":AsmParser",
+)
+
 cc_library(
-    name = "asm_printer",
+    name = "AsmPrinter",
     srcs = glob([
         "lib/CodeGen/AsmPrinter/*.c",
         "lib/CodeGen/AsmPrinter/*.cpp",
@@ -1435,24 +1586,29 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":analysis",
-        ":binary_format",
-        ":code_gen",
+        ":Analysis",
+        ":BinaryFormat",
+        ":CodeGen",
+        ":Core",
+        ":DebugInfoCodeView",
+        ":DebugInfoDWARF",
+        ":DebugInfoMSF",
+        ":MC",
+        ":MCParser",
+        ":Remarks",
+        ":Support",
+        ":Target",
         ":config",
-        ":core",
-        ":debug_info_code_view",
-        ":debug_info_dwarf",
-        ":debug_info_msf",
-        ":mc",
-        ":mc_parser",
-        ":remarks",
-        ":support",
-        ":target",
     ],
 )
 
+alias(
+    name = "asm_printer",
+    actual = ":AsmPrinter",
+)
+
 cc_library(
-    name = "bpf_asm_parser",
+    name = "BPFAsmParser",
     srcs = glob([
         "lib/Target/BPF/AsmParser/*.c",
         "lib/Target/BPF/AsmParser/*.cpp",
@@ -1466,17 +1622,22 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/BPF"],
     deps = [
-        ":bpf_desc",
-        ":bpf_info",
+        ":BPFDesc",
+        ":BPFInfo",
+        ":MC",
+        ":MCParser",
+        ":Support",
         ":config",
-        ":mc",
-        ":mc_parser",
-        ":support",
     ],
 )
 
+alias(
+    name = "bpf_asm_parser",
+    actual = ":BPFAsmParser",
+)
+
 cc_library(
-    name = "bpf_code_gen",
+    name = "BPFCodeGen",
     srcs = glob([
         "lib/Target/BPF/*.c",
         "lib/Target/BPF/*.cpp",
@@ -1490,21 +1651,26 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/BPF"],
     deps = [
-        ":asm_printer",
-        ":bpf_desc",
-        ":bpf_info",
-        ":code_gen",
+        ":AsmPrinter",
+        ":BPFDesc",
+        ":BPFInfo",
+        ":CodeGen",
+        ":Core",
+        ":MC",
+        ":SelectionDAG",
+        ":Support",
+        ":Target",
         ":config",
-        ":core",
-        ":mc",
-        ":selection_dag",
-        ":support",
-        ":target",
     ],
 )
 
+alias(
+    name = "bpf_code_gen",
+    actual = ":BPFCodeGen",
+)
+
 cc_library(
-    name = "bpf_desc",
+    name = "BPFDesc",
     srcs = glob([
         "lib/Target/BPF/MCTargetDesc/*.c",
         "lib/Target/BPF/MCTargetDesc/*.cpp",
@@ -1518,15 +1684,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/BPF"],
     deps = [
-        ":bpf_info",
+        ":BPFInfo",
+        ":MC",
+        ":Support",
         ":config",
-        ":mc",
-        ":support",
     ],
 )
 
+alias(
+    name = "bpf_desc",
+    actual = ":BPFDesc",
+)
+
 cc_library(
-    name = "bpf_disassembler",
+    name = "BPFDisassembler",
     srcs = glob([
         "lib/Target/BPF/Disassembler/*.c",
         "lib/Target/BPF/Disassembler/*.cpp",
@@ -1540,15 +1711,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/BPF"],
     deps = [
-        ":bpf_info",
+        ":BPFInfo",
+        ":MCDisassembler",
+        ":Support",
         ":config",
-        ":mc_disassembler",
-        ":support",
     ],
 )
 
+alias(
+    name = "bpf_disassembler",
+    actual = ":BPFDisassembler",
+)
+
 cc_library(
-    name = "bpf_info",
+    name = "BPFInfo",
     srcs = glob([
         "lib/Target/BPF/TargetInfo/*.c",
         "lib/Target/BPF/TargetInfo/*.cpp",
@@ -1562,13 +1738,18 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/BPF"],
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "bpf_info",
+    actual = ":BPFInfo",
+)
+
 cc_library(
-    name = "binary_format",
+    name = "BinaryFormat",
     srcs = glob([
         "lib/BinaryFormat/*.c",
         "lib/BinaryFormat/*.cpp",
@@ -1584,13 +1765,18 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "binary_format",
+    actual = ":BinaryFormat",
+)
+
 cc_library(
-    name = "bit_reader",
+    name = "BitReader",
     srcs = glob([
         "lib/Bitcode/Reader/*.c",
         "lib/Bitcode/Reader/*.cpp",
@@ -1605,15 +1791,20 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":bitstream_reader",
+        ":BitstreamReader",
+        ":Core",
+        ":Support",
         ":config",
-        ":core",
-        ":support",
     ],
 )
 
+alias(
+    name = "bit_reader",
+    actual = ":BitReader",
+)
+
 cc_library(
-    name = "bit_writer",
+    name = "BitWriter",
     srcs = glob([
         "lib/Bitcode/Writer/*.c",
         "lib/Bitcode/Writer/*.cpp",
@@ -1630,17 +1821,22 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":analysis",
+        ":Analysis",
+        ":Core",
+        ":MC",
+        ":Object",
+        ":Support",
         ":config",
-        ":core",
-        ":mc",
-        ":object",
-        ":support",
     ],
 )
 
+alias(
+    name = "bit_writer",
+    actual = ":BitWriter",
+)
+
 cc_library(
-    name = "bitstream_reader",
+    name = "BitstreamReader",
     srcs = glob([
         "lib/Bitstream/Reader/*.c",
         "lib/Bitstream/Reader/*.cpp",
@@ -1654,13 +1850,18 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "bitstream_reader",
+    actual = ":BitstreamReader",
+)
+
 cc_library(
-    name = "cf_guard",
+    name = "CFGuard",
     srcs = glob([
         "lib/Transforms/CFGuard/*.c",
         "lib/Transforms/CFGuard/*.cpp",
@@ -1674,14 +1875,19 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Core",
+        ":Support",
         ":config",
-        ":core",
-        ":support",
     ],
 )
 
+alias(
+    name = "cf_guard",
+    actual = ":CFGuard",
+)
+
 cc_library(
-    name = "code_gen",
+    name = "CodeGen",
     srcs = glob([
         "lib/CodeGen/*.c",
         "lib/CodeGen/*.cpp",
@@ -1696,23 +1902,28 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":analysis",
-        ":bit_reader",
-        ":bit_writer",
+        ":Analysis",
+        ":BitReader",
+        ":BitWriter",
+        ":Core",
+        ":MC",
+        ":ProfileData",
+        ":Scalar",
+        ":Support",
+        ":Target",
+        ":TransformUtils",
         ":config",
-        ":core",
         ":instrumentation",
-        ":mc",
-        ":profile_data",
-        ":scalar",
-        ":support",
-        ":target",
-        ":transform_utils",
     ],
 )
 
+alias(
+    name = "code_gen",
+    actual = ":CodeGen",
+)
+
 cc_library(
-    name = "core",
+    name = "Core",
     srcs = glob([
         "lib/IR/*.c",
         "lib/IR/*.cpp",
@@ -1734,11 +1945,13 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":BinaryFormat",
+        ":Remarks",
+        ":Support",
         ":aarch64_enums_gen",
         ":amdgcn_enums_gen",
         ":arm_enums_gen",
         ":attributes_gen",
-        ":binary_format",
         ":bpf_enums_gen",
         ":config",
         ":hexagon_enums_gen",
@@ -1748,18 +1961,21 @@ cc_library(
         ":nvvm_enums_gen",
         ":ppc_enums_gen",
         ":r600_enums_gen",
-        ":remarks",
         ":riscv_enums_gen",
         ":s390_enums_gen",
-        ":support",
         ":wasm_enums_gen",
         ":x86_enums_gen",
         ":xcore_enums_gen",
     ],
 )
 
+alias(
+    name = "core",
+    actual = ":Core",
+)
+
 cc_library(
-    name = "coroutines",
+    name = "Coroutines",
     srcs = glob([
         "lib/Transforms/Coroutines/*.c",
         "lib/Transforms/Coroutines/*.cpp",
@@ -1773,18 +1989,23 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":analysis",
+        ":Analysis",
+        ":Core",
+        ":IPO",
+        ":Scalar",
+        ":Support",
+        ":TransformUtils",
         ":config",
-        ":core",
-        ":ipo",
-        ":scalar",
-        ":support",
-        ":transform_utils",
     ],
 )
 
+alias(
+    name = "coroutines",
+    actual = ":Coroutines",
+)
+
 cc_library(
-    name = "coverage",
+    name = "Coverage",
     srcs = glob([
         "lib/ProfileData/Coverage/*.c",
         "lib/ProfileData/Coverage/*.cpp",
@@ -1798,16 +2019,21 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Core",
+        ":Object",
+        ":ProfileData",
+        ":Support",
         ":config",
-        ":core",
-        ":object",
-        ":profile_data",
-        ":support",
     ],
 )
 
+alias(
+    name = "coverage",
+    actual = ":Coverage",
+)
+
 cc_library(
-    name = "dwarf_linker",
+    name = "DWARFLinker",
     srcs = glob([
         "lib/DWARFLinker/*.c",
         "lib/DWARFLinker/*.cpp",
@@ -1821,18 +2047,23 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":asm_printer",
-        ":code_gen",
+        ":AsmPrinter",
+        ":CodeGen",
+        ":DebugInfoDWARF",
+        ":MC",
+        ":Object",
+        ":Support",
         ":config",
-        ":debug_info_dwarf",
-        ":mc",
-        ":object",
-        ":support",
     ],
 )
 
+alias(
+    name = "dwarf_linker",
+    actual = ":DWARFLinker",
+)
+
 cc_library(
-    name = "debug_info_code_view",
+    name = "DebugInfoCodeView",
     srcs = glob([
         "lib/DebugInfo/CodeView/*.c",
         "lib/DebugInfo/CodeView/*.cpp",
@@ -1846,15 +2077,20 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":DebugInfoMSF",
+        ":Support",
         ":binary_format",
         ":config",
-        ":debug_info_msf",
-        ":support",
     ],
 )
 
+alias(
+    name = "debug_info_code_view",
+    actual = ":DebugInfoCodeView",
+)
+
 cc_library(
-    name = "debug_info_dwarf",
+    name = "DebugInfoDWARF",
     srcs = glob([
         "lib/DebugInfo/DWARF/*.c",
         "lib/DebugInfo/DWARF/*.cpp",
@@ -1868,16 +2104,21 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":binary_format",
+        ":BinaryFormat",
+        ":MC",
+        ":Object",
+        ":Support",
         ":config",
-        ":mc",
-        ":object",
-        ":support",
     ],
 )
 
+alias(
+    name = "debug_info_dwarf",
+    actual = ":DebugInfoDWARF",
+)
+
 cc_library(
-    name = "debug_info_gsym",
+    name = "DebugInfoGSYM",
     srcs = glob([
         "lib/DebugInfo/GSYM/*.c",
         "lib/DebugInfo/GSYM/*.cpp",
@@ -1891,16 +2132,21 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":DebugInfoDWARF",
+        ":MC",
+        ":Object",
+        ":Support",
         ":config",
-        ":debug_info_dwarf",
-        ":mc",
-        ":object",
-        ":support",
     ],
 )
 
+alias(
+    name = "debug_info_gsym",
+    actual = ":DebugInfoGSYM",
+)
+
 cc_library(
-    name = "debug_info_msf",
+    name = "DebugInfoMSF",
     srcs = glob([
         "lib/DebugInfo/MSF/*.c",
         "lib/DebugInfo/MSF/*.cpp",
@@ -1914,13 +2160,18 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "debug_info_msf",
+    actual = ":DebugInfoMSF",
+)
+
 cc_library(
-    name = "debug_info_pdb",
+    name = "DebugInfoPDB",
     srcs = glob([
         "lib/DebugInfo/PDB/*.c",
         "lib/DebugInfo/PDB/*.cpp",
@@ -1934,17 +2185,22 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":binary_format",
+        ":BinaryFormat",
+        ":DebugInfoCodeView",
+        ":DebugInfoMSF",
+        ":Object",
+        ":Support",
         ":config",
-        ":debug_info_code_view",
-        ":debug_info_msf",
-        ":object",
-        ":support",
     ],
 )
 
+alias(
+    name = "debug_info_pdb",
+    actual = ":DebugInfoPDB",
+)
+
 cc_library(
-    name = "demangle",
+    name = "Demangle",
     srcs = glob([
         "lib/Demangle/*.c",
         "lib/Demangle/*.cpp",
@@ -1960,8 +2216,13 @@ cc_library(
     deps = [":config"],
 )
 
+alias(
+    name = "demangle",
+    actual = ":Demangle",
+)
+
 cc_library(
-    name = "dlltool_driver",
+    name = "DlltoolDriver",
     srcs = glob([
         "lib/ToolDrivers/llvm-dlltool/*.c",
         "lib/ToolDrivers/llvm-dlltool/*.cpp",
@@ -1975,15 +2236,20 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Object",
+        ":Option",
+        ":Support",
         ":config",
-        ":object",
-        ":option",
-        ":support",
     ],
 )
 
+alias(
+    name = "dlltool_driver",
+    actual = ":DlltoolDriver",
+)
+
 cc_library(
-    name = "execution_engine",
+    name = "ExecutionEngine",
     srcs = glob([
         "lib/ExecutionEngine/*.c",
         "lib/ExecutionEngine/*.cpp",
@@ -1997,18 +2263,23 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Core",
+        ":MC",
+        ":Object",
+        ":RuntimeDyld",
+        ":Support",
+        ":Target",
         ":config",
-        ":core",
-        ":mc",
-        ":object",
-        ":runtime_dyld",
-        ":support",
-        ":target",
     ],
 )
 
+alias(
+    name = "execution_engine",
+    actual = ":ExecutionEngine",
+)
+
 cc_library(
-    name = "extensions",
+    name = "Extensions",
     srcs = glob([
         "lib/Extensions/*.c",
         "lib/Extensions/*.cpp",
@@ -2024,8 +2295,13 @@ cc_library(
     deps = [":config"],
 )
 
+alias(
+    name = "extensions",
+    actual = ":Extensions",
+)
+
 cc_library(
-    name = "frontend_open_mp",
+    name = "FrontendOpenMP",
     srcs = glob([
         "lib/Frontend/OpenMP/*.c",
         "lib/Frontend/OpenMP/*.cpp",
@@ -2039,15 +2315,20 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Core",
+        ":Support",
+        ":TransformUtils",
         ":config",
-        ":core",
-        ":support",
-        ":transform_utils",
     ],
 )
 
+alias(
+    name = "frontend_open_mp",
+    actual = ":FrontendOpenMP",
+)
+
 cc_library(
-    name = "fuzz_mutate",
+    name = "FuzzMutate",
     srcs = glob([
         "lib/FuzzMutate/*.c",
         "lib/FuzzMutate/*.cpp",
@@ -2061,19 +2342,24 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":analysis",
-        ":bit_reader",
-        ":bit_writer",
+        ":Analysis",
+        ":BitReader",
+        ":BitWriter",
+        ":Core",
+        ":Scalar",
+        ":Support",
+        ":Target",
         ":config",
-        ":core",
-        ":scalar",
-        ":support",
-        ":target",
     ],
 )
 
+alias(
+    name = "fuzz_mutate",
+    actual = ":FuzzMutate",
+)
+
 cc_library(
-    name = "global_i_sel",
+    name = "GlobalISel",
     srcs = glob([
         "lib/CodeGen/GlobalISel/*.c",
         "lib/CodeGen/GlobalISel/*.cpp",
@@ -2087,20 +2373,25 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":analysis",
-        ":code_gen",
+        ":Analysis",
+        ":CodeGen",
+        ":Core",
+        ":MC",
+        ":SelectionDAG",
+        ":Support",
+        ":Target",
+        ":TransformUtils",
         ":config",
-        ":core",
-        ":mc",
-        ":selection_dag",
-        ":support",
-        ":target",
-        ":transform_utils",
     ],
 )
 
+alias(
+    name = "global_i_sel",
+    actual = ":GlobalISel",
+)
+
 cc_library(
-    name = "hexagon_asm_parser",
+    name = "HexagonAsmParser",
     srcs = glob([
         "lib/Target/Hexagon/AsmParser/*.c",
         "lib/Target/Hexagon/AsmParser/*.cpp",
@@ -2114,17 +2405,22 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/Hexagon"],
     deps = [
+        ":HexagonDesc",
+        ":HexagonInfo",
+        ":MC",
+        ":MCParser",
+        ":Support",
         ":config",
-        ":hexagon_desc",
-        ":hexagon_info",
-        ":mc",
-        ":mc_parser",
-        ":support",
     ],
 )
 
+alias(
+    name = "hexagon_asm_parser",
+    actual = ":HexagonAsmParser",
+)
+
 cc_library(
-    name = "hexagon_code_gen",
+    name = "HexagonCodeGen",
     srcs = glob([
         "lib/Target/Hexagon/*.c",
         "lib/Target/Hexagon/*.cpp",
@@ -2138,26 +2434,31 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/Hexagon"],
     deps = [
-        ":analysis",
-        ":asm_printer",
-        ":code_gen",
+        ":Analysis",
+        ":AsmPrinter",
+        ":CodeGen",
+        ":Core",
+        ":HexagonAsmParser",
+        ":HexagonDesc",
+        ":HexagonInfo",
+        ":IPO",
+        ":MC",
+        ":Scalar",
+        ":SelectionDAG",
+        ":Support",
+        ":Target",
+        ":TransformUtils",
         ":config",
-        ":core",
-        ":hexagon_asm_parser",
-        ":hexagon_desc",
-        ":hexagon_info",
-        ":ipo",
-        ":mc",
-        ":scalar",
-        ":selection_dag",
-        ":support",
-        ":target",
-        ":transform_utils",
     ],
 )
 
+alias(
+    name = "hexagon_code_gen",
+    actual = ":HexagonCodeGen",
+)
+
 cc_library(
-    name = "hexagon_desc",
+    name = "HexagonDesc",
     srcs = glob([
         "lib/Target/Hexagon/MCTargetDesc/*.c",
         "lib/Target/Hexagon/MCTargetDesc/*.cpp",
@@ -2171,15 +2472,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/Hexagon"],
     deps = [
+        ":HexagonInfo",
+        ":MC",
+        ":Support",
         ":config",
-        ":hexagon_info",
-        ":mc",
-        ":support",
     ],
 )
 
+alias(
+    name = "hexagon_desc",
+    actual = ":HexagonDesc",
+)
+
 cc_library(
-    name = "hexagon_disassembler",
+    name = "HexagonDisassembler",
     srcs = glob([
         "lib/Target/Hexagon/Disassembler/*.c",
         "lib/Target/Hexagon/Disassembler/*.cpp",
@@ -2193,17 +2499,22 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/Hexagon"],
     deps = [
+        ":HexagonDesc",
+        ":HexagonInfo",
+        ":MC",
+        ":MCDisassembler",
+        ":Support",
         ":config",
-        ":hexagon_desc",
-        ":hexagon_info",
-        ":mc",
-        ":mc_disassembler",
-        ":support",
     ],
 )
 
+alias(
+    name = "hexagon_disassembler",
+    actual = ":HexagonDisassembler",
+)
+
 cc_library(
-    name = "hexagon_info",
+    name = "HexagonInfo",
     srcs = glob([
         "lib/Target/Hexagon/TargetInfo/*.c",
         "lib/Target/Hexagon/TargetInfo/*.cpp",
@@ -2217,13 +2528,18 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/Hexagon"],
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "hexagon_info",
+    actual = ":HexagonInfo",
+)
+
 cc_library(
-    name = "ipo",
+    name = "IPO",
     srcs = glob([
         "lib/Transforms/IPO/*.c",
         "lib/Transforms/IPO/*.cpp",
@@ -2240,28 +2556,33 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":aggressive_inst_combine",
-        ":analysis",
-        ":bit_reader",
-        ":bit_writer",
+        ":AggressiveInstCombine",
+        ":Analysis",
+        ":BitReader",
+        ":BitWriter",
+        ":Core",
+        ":FrontendOpenMP",
+        ":IRReader",
+        ":InstCombine",
+        ":Instrumentation",
+        ":Linker",
+        ":Object",
+        ":ProfileData",
+        ":Scalar",
+        ":Support",
+        ":TransformUtils",
+        ":Vectorize",
         ":config",
-        ":core",
-        ":frontend_open_mp",
-        ":inst_combine",
-        ":instrumentation",
-        ":ir_reader",
-        ":linker",
-        ":object",
-        ":profile_data",
-        ":scalar",
-        ":support",
-        ":transform_utils",
-        ":vectorize",
     ],
 )
 
+alias(
+    name = "ipo",
+    actual = ":IPO",
+)
+
 cc_library(
-    name = "ir_reader",
+    name = "IRReader",
     srcs = glob([
         "lib/IRReader/*.c",
         "lib/IRReader/*.cpp",
@@ -2275,16 +2596,21 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":asm_parser",
-        ":bit_reader",
+        ":AsmParser",
+        ":BitReader",
+        ":Core",
+        ":Support",
         ":config",
-        ":core",
-        ":support",
     ],
 )
 
+alias(
+    name = "ir_reader",
+    actual = ":IRReader",
+)
+
 cc_library(
-    name = "inst_combine",
+    name = "InstCombine",
     srcs = glob([
         "lib/Transforms/InstCombine/*.c",
         "lib/Transforms/InstCombine/*.cpp",
@@ -2298,17 +2624,22 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":analysis",
+        ":Analysis",
+        ":Core",
+        ":Support",
+        ":TransformUtils",
         ":config",
-        ":core",
         ":instcombine_transforms_gen",
-        ":support",
-        ":transform_utils",
     ],
 )
 
+alias(
+    name = "inst_combine",
+    actual = ":InstCombine",
+)
+
 cc_library(
-    name = "instrumentation",
+    name = "Instrumentation",
     srcs = glob([
         "lib/Transforms/Instrumentation/*.c",
         "lib/Transforms/Instrumentation/*.cpp",
@@ -2326,18 +2657,23 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":analysis",
+        ":Analysis",
+        ":Core",
+        ":MC",
+        ":ProfileData",
+        ":Support",
+        ":TransformUtils",
         ":config",
-        ":core",
-        ":mc",
-        ":profile_data",
-        ":support",
-        ":transform_utils",
     ],
 )
 
+alias(
+    name = "instrumentation",
+    actual = ":Instrumentation",
+)
+
 cc_library(
-    name = "interpreter",
+    name = "Interpreter",
     srcs = glob([
         "lib/ExecutionEngine/Interpreter/*.c",
         "lib/ExecutionEngine/Interpreter/*.cpp",
@@ -2351,16 +2687,21 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":code_gen",
+        ":CodeGen",
+        ":Core",
+        ":ExecutionEngine",
+        ":Support",
         ":config",
-        ":core",
-        ":execution_engine",
-        ":support",
     ],
 )
 
+alias(
+    name = "interpreter",
+    actual = ":Interpreter",
+)
+
 cc_library(
-    name = "jit_link",
+    name = "JITLink",
     srcs = glob([
         "lib/ExecutionEngine/JITLink/*.c",
         "lib/ExecutionEngine/JITLink/*.cpp",
@@ -2374,15 +2715,20 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":binary_format",
+        ":BinaryFormat",
+        ":Object",
+        ":Support",
         ":config",
-        ":object",
-        ":support",
     ],
 )
 
+alias(
+    name = "jit_link",
+    actual = ":JITLink",
+)
+
 cc_library(
-    name = "lto",
+    name = "LTO",
     srcs = glob([
         "lib/LTO/*.c",
         "lib/LTO/*.cpp",
@@ -2396,32 +2742,37 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":aggressive_inst_combine",
-        ":analysis",
-        ":binary_format",
-        ":bit_reader",
-        ":bit_writer",
-        ":code_gen",
+        ":AggressiveInstCombine",
+        ":Analysis",
+        ":BinaryFormat",
+        ":BitReader",
+        ":BitWriter",
+        ":CodeGen",
+        ":Core",
+        ":Extensions",
+        ":IPO",
+        ":InstCombine",
+        ":Linker",
+        ":MC",
+        ":ObjCARC",
+        ":Object",
+        ":Passes",
+        ":Remarks",
+        ":Scalar",
+        ":Support",
+        ":Target",
+        ":TransformUtils",
         ":config",
-        ":core",
-        ":extensions",
-        ":inst_combine",
-        ":ipo",
-        ":linker",
-        ":mc",
-        ":objc_arc",
-        ":object",
-        ":passes",
-        ":remarks",
-        ":scalar",
-        ":support",
-        ":target",
-        ":transform_utils",
     ],
 )
 
+alias(
+    name = "lto",
+    actual = ":LTO",
+)
+
 cc_library(
-    name = "lanai_asm_parser",
+    name = "LanaiAsmParser",
     srcs = glob([
         "lib/Target/Lanai/AsmParser/*.c",
         "lib/Target/Lanai/AsmParser/*.cpp",
@@ -2435,17 +2786,22 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/Lanai"],
     deps = [
+        ":LanaiDesc",
+        ":LanaiInfo",
+        ":MC",
+        ":MCParser",
+        ":Support",
         ":config",
-        ":lanai_desc",
-        ":lanai_info",
-        ":mc",
-        ":mc_parser",
-        ":support",
     ],
 )
 
+alias(
+    name = "lanai_asm_parser",
+    actual = ":LanaiAsmParser",
+)
+
 cc_library(
-    name = "lanai_code_gen",
+    name = "LanaiCodeGen",
     srcs = glob([
         "lib/Target/Lanai/*.c",
         "lib/Target/Lanai/*.cpp",
@@ -2459,24 +2815,29 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/Lanai"],
     deps = [
-        ":analysis",
-        ":asm_printer",
-        ":code_gen",
+        ":Analysis",
+        ":AsmPrinter",
+        ":CodeGen",
+        ":Core",
+        ":LanaiAsmParser",
+        ":LanaiDesc",
+        ":LanaiInfo",
+        ":MC",
+        ":SelectionDAG",
+        ":Support",
+        ":Target",
+        ":TransformUtils",
         ":config",
-        ":core",
-        ":lanai_asm_parser",
-        ":lanai_desc",
-        ":lanai_info",
-        ":mc",
-        ":selection_dag",
-        ":support",
-        ":target",
-        ":transform_utils",
     ],
 )
 
+alias(
+    name = "lanai_code_gen",
+    actual = ":LanaiCodeGen",
+)
+
 cc_library(
-    name = "lanai_desc",
+    name = "LanaiDesc",
     srcs = glob([
         "lib/Target/Lanai/MCTargetDesc/*.c",
         "lib/Target/Lanai/MCTargetDesc/*.cpp",
@@ -2490,16 +2851,21 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/Lanai"],
     deps = [
+        ":LanaiInfo",
+        ":MC",
+        ":MCDisassembler",
+        ":Support",
         ":config",
-        ":lanai_info",
-        ":mc",
-        ":mc_disassembler",
-        ":support",
     ],
 )
 
+alias(
+    name = "lanai_desc",
+    actual = ":LanaiDesc",
+)
+
 cc_library(
-    name = "lanai_disassembler",
+    name = "LanaiDisassembler",
     srcs = glob([
         "lib/Target/Lanai/Disassembler/*.c",
         "lib/Target/Lanai/Disassembler/*.cpp",
@@ -2513,17 +2879,22 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/Lanai"],
     deps = [
+        ":LanaiDesc",
+        ":LanaiInfo",
+        ":MC",
+        ":MCDisassembler",
+        ":Support",
         ":config",
-        ":lanai_desc",
-        ":lanai_info",
-        ":mc",
-        ":mc_disassembler",
-        ":support",
     ],
 )
 
+alias(
+    name = "lanai_disassembler",
+    actual = ":LanaiDisassembler",
+)
+
 cc_library(
-    name = "lanai_info",
+    name = "LanaiInfo",
     srcs = glob([
         "lib/Target/Lanai/TargetInfo/*.c",
         "lib/Target/Lanai/TargetInfo/*.cpp",
@@ -2537,13 +2908,18 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/Lanai"],
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "lanai_info",
+    actual = ":LanaiInfo",
+)
+
 cc_library(
-    name = "lib_driver",
+    name = "LibDriver",
     srcs = glob([
         "lib/ToolDrivers/llvm-lib/*.c",
         "lib/ToolDrivers/llvm-lib/*.cpp",
@@ -2557,17 +2933,22 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":binary_format",
-        ":bit_reader",
+        ":BinaryFormat",
+        ":BitReader",
+        ":Object",
+        ":Option",
+        ":Support",
         ":config",
-        ":object",
-        ":option",
-        ":support",
     ],
 )
 
+alias(
+    name = "lib_driver",
+    actual = ":LibDriver",
+)
+
 cc_library(
-    name = "line_editor",
+    name = "LineEditor",
     srcs = glob([
         "lib/LineEditor/*.c",
         "lib/LineEditor/*.cpp",
@@ -2581,13 +2962,18 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "line_editor",
+    actual = ":LineEditor",
+)
+
 cc_library(
-    name = "linker",
+    name = "Linker",
     srcs = glob([
         "lib/Linker/*.c",
         "lib/Linker/*.cpp",
@@ -2601,15 +2987,20 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Core",
+        ":Support",
+        ":TransformUtils",
         ":config",
-        ":core",
-        ":support",
-        ":transform_utils",
     ],
 )
 
+alias(
+    name = "linker",
+    actual = ":Linker",
+)
+
 cc_library(
-    name = "mc",
+    name = "MC",
     srcs = glob([
         "lib/MC/*.c",
         "lib/MC/*.cpp",
@@ -2623,15 +3014,22 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":BinaryFormat",
+        ":DebugInfoCodeView",
+        ":Support",
         ":binary_format",
         ":config",
         ":debug_info_code_view",
-        ":support",
     ],
 )
 
+alias(
+    name = "mc",
+    actual = ":MC",
+)
+
 cc_library(
-    name = "mca",
+    name = "MCA",
     srcs = glob([
         "lib/MCA/*.c",
         "lib/MCA/*.cpp",
@@ -2645,14 +3043,19 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":MC",
+        ":Support",
         ":config",
-        ":mc",
-        ":support",
     ],
 )
 
+alias(
+    name = "mca",
+    actual = ":MCA",
+)
+
 cc_library(
-    name = "mc_disassembler",
+    name = "MCDisassembler",
     srcs = glob([
         "lib/MC/MCDisassembler/*.c",
         "lib/MC/MCDisassembler/*.cpp",
@@ -2666,14 +3069,19 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":MC",
+        ":Support",
         ":config",
-        ":mc",
-        ":support",
     ],
 )
 
+alias(
+    name = "mc_disassembler",
+    actual = ":MCDisassembler",
+)
+
 cc_library(
-    name = "mcjit",
+    name = "MCJIT",
     srcs = glob([
         "lib/ExecutionEngine/MCJIT/*.c",
         "lib/ExecutionEngine/MCJIT/*.cpp",
@@ -2687,18 +3095,23 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Core",
+        ":ExecutionEngine",
+        ":Object",
+        ":RuntimeDyld",
+        ":Support",
+        ":Target",
         ":config",
-        ":core",
-        ":execution_engine",
-        ":object",
-        ":runtime_dyld",
-        ":support",
-        ":target",
     ],
 )
 
+alias(
+    name = "mcjit",
+    actual = ":MCJIT",
+)
+
 cc_library(
-    name = "mc_parser",
+    name = "MCParser",
     srcs = glob([
         "lib/MC/MCParser/*.c",
         "lib/MC/MCParser/*.cpp",
@@ -2712,14 +3125,19 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":MC",
+        ":Support",
         ":config",
-        ":mc",
-        ":support",
     ],
 )
 
+alias(
+    name = "mc_parser",
+    actual = ":MCParser",
+)
+
 cc_library(
-    name = "mir_parser",
+    name = "MIRParser",
     srcs = glob([
         "lib/CodeGen/MIRParser/*.c",
         "lib/CodeGen/MIRParser/*.cpp",
@@ -2733,19 +3151,24 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":asm_parser",
-        ":binary_format",
-        ":code_gen",
+        ":AsmParser",
+        ":BinaryFormat",
+        ":CodeGen",
+        ":Core",
+        ":MC",
+        ":Support",
+        ":Target",
         ":config",
-        ":core",
-        ":mc",
-        ":support",
-        ":target",
     ],
 )
 
+alias(
+    name = "mir_parser",
+    actual = ":MIRParser",
+)
+
 cc_library(
-    name = "ml_policies",
+    name = "MLPolicies",
     srcs = glob([
         "lib/Analysis/ML/*.c",
         "lib/Analysis/ML/*.cpp",
@@ -2759,14 +3182,19 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Core",
+        ":Support",
         ":config",
-        ":core",
-        ":support",
     ],
 )
 
+alias(
+    name = "ml_policies",
+    actual = ":MLPolicies",
+)
+
 cc_library(
-    name = "msp430_asm_parser",
+    name = "MSP430AsmParser",
     srcs = glob([
         "lib/Target/MSP430/AsmParser/*.c",
         "lib/Target/MSP430/AsmParser/*.cpp",
@@ -2780,17 +3208,22 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/MSP430"],
     deps = [
+        ":MC",
+        ":MCParser",
+        ":MSP430Desc",
+        ":MSP430Info",
+        ":Support",
         ":config",
-        ":mc",
-        ":mc_parser",
-        ":msp430_desc",
-        ":msp430_info",
-        ":support",
     ],
 )
 
+alias(
+    name = "msp430_asm_parser",
+    actual = ":MSP430AsmParser",
+)
+
 cc_library(
-    name = "msp430_code_gen",
+    name = "MSP430CodeGen",
     srcs = glob([
         "lib/Target/MSP430/*.c",
         "lib/Target/MSP430/*.cpp",
@@ -2804,21 +3237,26 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/MSP430"],
     deps = [
-        ":asm_printer",
-        ":code_gen",
+        ":AsmPrinter",
+        ":CodeGen",
+        ":Core",
+        ":MC",
+        ":MSP430Desc",
+        ":MSP430Info",
+        ":SelectionDAG",
+        ":Support",
+        ":Target",
         ":config",
-        ":core",
-        ":mc",
-        ":msp430_desc",
-        ":msp430_info",
-        ":selection_dag",
-        ":support",
-        ":target",
     ],
 )
 
+alias(
+    name = "msp430_code_gen",
+    actual = ":MSP430CodeGen",
+)
+
 cc_library(
-    name = "msp430_desc",
+    name = "MSP430Desc",
     srcs = glob([
         "lib/Target/MSP430/MCTargetDesc/*.c",
         "lib/Target/MSP430/MCTargetDesc/*.cpp",
@@ -2832,15 +3270,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/MSP430"],
     deps = [
+        ":MC",
+        ":MSP430Info",
+        ":Support",
         ":config",
-        ":mc",
-        ":msp430_info",
-        ":support",
     ],
 )
 
+alias(
+    name = "msp430_desc",
+    actual = ":MSP430Desc",
+)
+
 cc_library(
-    name = "msp430_disassembler",
+    name = "MSP430Disassembler",
     srcs = glob([
         "lib/Target/MSP430/Disassembler/*.c",
         "lib/Target/MSP430/Disassembler/*.cpp",
@@ -2854,15 +3297,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/MSP430"],
     deps = [
+        ":MCDisassembler",
+        ":MSP430Info",
+        ":Support",
         ":config",
-        ":mc_disassembler",
-        ":msp430_info",
-        ":support",
     ],
 )
 
+alias(
+    name = "msp430_disassembler",
+    actual = ":MSP430Disassembler",
+)
+
 cc_library(
-    name = "msp430_info",
+    name = "MSP430Info",
     srcs = glob([
         "lib/Target/MSP430/TargetInfo/*.c",
         "lib/Target/MSP430/TargetInfo/*.cpp",
@@ -2876,13 +3324,18 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/MSP430"],
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "msp430_info",
+    actual = ":MSP430Info",
+)
+
 cc_library(
-    name = "mips_asm_parser",
+    name = "MipsAsmParser",
     srcs = glob([
         "lib/Target/Mips/AsmParser/*.c",
         "lib/Target/Mips/AsmParser/*.cpp",
@@ -2896,17 +3349,22 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/Mips"],
     deps = [
+        ":MC",
+        ":MCParser",
+        ":MipsDesc",
+        ":MipsInfo",
+        ":Support",
         ":config",
-        ":mc",
-        ":mc_parser",
-        ":mips_desc",
-        ":mips_info",
-        ":support",
     ],
 )
 
+alias(
+    name = "mips_asm_parser",
+    actual = ":MipsAsmParser",
+)
+
 cc_library(
-    name = "mips_code_gen",
+    name = "MipsCodeGen",
     srcs = glob([
         "lib/Target/Mips/*.c",
         "lib/Target/Mips/*.cpp",
@@ -2920,23 +3378,28 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/Mips"],
     deps = [
-        ":analysis",
-        ":asm_printer",
-        ":code_gen",
+        ":Analysis",
+        ":AsmPrinter",
+        ":CodeGen",
+        ":Core",
+        ":GlobalISel",
+        ":MC",
+        ":MipsDesc",
+        ":MipsInfo",
+        ":SelectionDAG",
+        ":Support",
+        ":Target",
         ":config",
-        ":core",
-        ":global_i_sel",
-        ":mc",
-        ":mips_desc",
-        ":mips_info",
-        ":selection_dag",
-        ":support",
-        ":target",
     ],
 )
 
+alias(
+    name = "mips_code_gen",
+    actual = ":MipsCodeGen",
+)
+
 cc_library(
-    name = "mips_desc",
+    name = "MipsDesc",
     srcs = glob([
         "lib/Target/Mips/MCTargetDesc/*.c",
         "lib/Target/Mips/MCTargetDesc/*.cpp",
@@ -2950,15 +3413,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/Mips"],
     deps = [
+        ":MC",
+        ":MipsInfo",
+        ":Support",
         ":config",
-        ":mc",
-        ":mips_info",
-        ":support",
     ],
 )
 
+alias(
+    name = "mips_desc",
+    actual = ":MipsDesc",
+)
+
 cc_library(
-    name = "mips_disassembler",
+    name = "MipsDisassembler",
     srcs = glob([
         "lib/Target/Mips/Disassembler/*.c",
         "lib/Target/Mips/Disassembler/*.cpp",
@@ -2972,15 +3440,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/Mips"],
     deps = [
+        ":MCDisassembler",
+        ":MipsInfo",
+        ":Support",
         ":config",
-        ":mc_disassembler",
-        ":mips_info",
-        ":support",
     ],
 )
 
+alias(
+    name = "mips_disassembler",
+    actual = ":MipsDisassembler",
+)
+
 cc_library(
-    name = "mips_info",
+    name = "MipsInfo",
     srcs = glob([
         "lib/Target/Mips/TargetInfo/*.c",
         "lib/Target/Mips/TargetInfo/*.cpp",
@@ -2994,13 +3467,18 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/Mips"],
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "mips_info",
+    actual = ":MipsInfo",
+)
+
 cc_library(
-    name = "nvptx_code_gen",
+    name = "NVPTXCodeGen",
     srcs = glob([
         "lib/Target/NVPTX/*.c",
         "lib/Target/NVPTX/*.cpp",
@@ -3014,26 +3492,31 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/NVPTX"],
     deps = [
-        ":analysis",
-        ":asm_printer",
-        ":code_gen",
+        ":Analysis",
+        ":AsmPrinter",
+        ":CodeGen",
+        ":Core",
+        ":IPO",
+        ":MC",
+        ":NVPTXDesc",
+        ":NVPTXInfo",
+        ":Scalar",
+        ":SelectionDAG",
+        ":Support",
+        ":Target",
+        ":TransformUtils",
+        ":Vectorize",
         ":config",
-        ":core",
-        ":ipo",
-        ":mc",
-        ":nvptx_desc",
-        ":nvptx_info",
-        ":scalar",
-        ":selection_dag",
-        ":support",
-        ":target",
-        ":transform_utils",
-        ":vectorize",
     ],
 )
 
+alias(
+    name = "nvptx_code_gen",
+    actual = ":NVPTXCodeGen",
+)
+
 cc_library(
-    name = "nvptx_desc",
+    name = "NVPTXDesc",
     srcs = glob([
         "lib/Target/NVPTX/MCTargetDesc/*.c",
         "lib/Target/NVPTX/MCTargetDesc/*.cpp",
@@ -3048,15 +3531,20 @@ cc_library(
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/NVPTX"],
     deps = [
         "nvptx_target_gen",
+        ":MC",
+        ":NVPTXInfo",
+        ":Support",
         ":config",
-        ":mc",
-        ":nvptx_info",
-        ":support",
     ],
 )
 
+alias(
+    name = "nvptx_desc",
+    actual = ":NVPTXDesc",
+)
+
 cc_library(
-    name = "nvptx_info",
+    name = "NVPTXInfo",
     srcs = glob([
         "lib/Target/NVPTX/TargetInfo/*.c",
         "lib/Target/NVPTX/TargetInfo/*.cpp",
@@ -3073,16 +3561,21 @@ cc_library(
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/NVPTX"],
     deps = [
         "nvptx_target_gen",
+        ":Support",
         ":attributes_gen",
         ":config",
         ":core",
-        ":support",
         ":target",
     ],
 )
 
+alias(
+    name = "nvptx_info",
+    actual = ":NVPTXInfo",
+)
+
 cc_library(
-    name = "objc_arc",
+    name = "ObjCARC",
     srcs = glob([
         "lib/Transforms/ObjCARC/*.c",
         "lib/Transforms/ObjCARC/*.cpp",
@@ -3097,16 +3590,21 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":analysis",
+        ":Analysis",
+        ":Core",
+        ":Support",
+        ":TransformUtils",
         ":config",
-        ":core",
-        ":support",
-        ":transform_utils",
     ],
 )
 
+alias(
+    name = "objc_arc",
+    actual = ":ObjCARC",
+)
+
 cc_library(
-    name = "object",
+    name = "Object",
     srcs = glob([
         "lib/Object/*.c",
         "lib/Object/*.cpp",
@@ -3120,19 +3618,24 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":binary_format",
-        ":bit_reader",
+        ":BinaryFormat",
+        ":BitReader",
+        ":Core",
+        ":MC",
+        ":MCParser",
+        ":Support",
+        ":TextAPI",
         ":config",
-        ":core",
-        ":mc",
-        ":mc_parser",
-        ":support",
-        ":text_api",
     ],
 )
 
+alias(
+    name = "object",
+    actual = ":Object",
+)
+
 cc_library(
-    name = "object_yaml",
+    name = "ObjectYAML",
     srcs = glob([
         "lib/ObjectYAML/*.c",
         "lib/ObjectYAML/*.cpp",
@@ -3146,16 +3649,21 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":DebugInfoCodeView",
+        ":MC",
+        ":Object",
+        ":Support",
         ":config",
-        ":debug_info_code_view",
-        ":mc",
-        ":object",
-        ":support",
     ],
 )
 
+alias(
+    name = "object_yaml",
+    actual = ":ObjectYAML",
+)
+
 cc_library(
-    name = "option",
+    name = "Option",
     srcs = glob([
         "lib/Option/*.c",
         "lib/Option/*.cpp",
@@ -3169,13 +3677,18 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "option",
+    actual = ":Option",
+)
+
 cc_library(
-    name = "orc_error",
+    name = "OrcError",
     srcs = glob([
         "lib/ExecutionEngine/OrcError/*.c",
         "lib/ExecutionEngine/OrcError/*.cpp",
@@ -3189,13 +3702,18 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "orc_error",
+    actual = ":OrcError",
+)
+
 cc_library(
-    name = "orc_jit",
+    name = "OrcJIT",
     srcs = glob([
         "lib/ExecutionEngine/Orc/*.c",
         "lib/ExecutionEngine/Orc/*.cpp",
@@ -3209,23 +3727,28 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Core",
+        ":ExecutionEngine",
+        ":JITLink",
+        ":MC",
+        ":Object",
+        ":OrcError",
+        ":Passes",
+        ":RuntimeDyld",
+        ":Support",
+        ":Target",
+        ":TransformUtils",
         ":config",
-        ":core",
-        ":execution_engine",
-        ":jit_link",
-        ":mc",
-        ":object",
-        ":orc_error",
-        ":passes",
-        ":runtime_dyld",
-        ":support",
-        ":target",
-        ":transform_utils",
     ],
 )
 
+alias(
+    name = "orc_jit",
+    actual = ":OrcJIT",
+)
+
 cc_library(
-    name = "passes",
+    name = "Passes",
     srcs = glob([
         "lib/Passes/*.c",
         "lib/Passes/*.cpp",
@@ -3239,26 +3762,31 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":aggressive_inst_combine",
-        ":analysis",
-        ":code_gen",
+        ":AggressiveInstCombine",
+        ":Analysis",
+        ":CodeGen",
+        ":Core",
+        ":Coroutines",
+        ":IPO",
+        ":InstCombine",
+        ":Instrumentation",
+        ":MLPolicies",
+        ":Scalar",
+        ":Support",
+        ":Target",
+        ":TransformUtils",
+        ":Vectorize",
         ":config",
-        ":core",
-        ":coroutines",
-        ":inst_combine",
-        ":instrumentation",
-        ":ipo",
-        ":ml_policies",
-        ":scalar",
-        ":support",
-        ":target",
-        ":transform_utils",
-        ":vectorize",
     ],
 )
 
+alias(
+    name = "passes",
+    actual = ":Passes",
+)
+
 cc_library(
-    name = "powerpc_asm_parser",
+    name = "PowerPCAsmParser",
     srcs = glob([
         "lib/Target/PowerPC/AsmParser/*.c",
         "lib/Target/PowerPC/AsmParser/*.cpp",
@@ -3272,17 +3800,22 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/PowerPC"],
     deps = [
+        ":MC",
+        ":MCParser",
+        ":PowerPCDesc",
+        ":PowerPCInfo",
+        ":Support",
         ":config",
-        ":mc",
-        ":mc_parser",
-        ":powerpc_desc",
-        ":powerpc_info",
-        ":support",
     ],
 )
 
+alias(
+    name = "powerpc_asm_parser",
+    actual = ":PowerPCAsmParser",
+)
+
 cc_library(
-    name = "powerpc_code_gen",
+    name = "PowerPCCodeGen",
     srcs = glob([
         "lib/Target/PowerPC/*.c",
         "lib/Target/PowerPC/*.cpp",
@@ -3296,24 +3829,29 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/PowerPC"],
     deps = [
-        ":analysis",
-        ":asm_printer",
-        ":code_gen",
+        ":Analysis",
+        ":AsmPrinter",
+        ":CodeGen",
+        ":Core",
+        ":MC",
+        ":PowerPCDesc",
+        ":PowerPCInfo",
+        ":Scalar",
+        ":SelectionDAG",
+        ":Support",
+        ":Target",
+        ":TransformUtils",
         ":config",
-        ":core",
-        ":mc",
-        ":powerpc_desc",
-        ":powerpc_info",
-        ":scalar",
-        ":selection_dag",
-        ":support",
-        ":target",
-        ":transform_utils",
     ],
 )
 
+alias(
+    name = "powerpc_code_gen",
+    actual = ":PowerPCCodeGen",
+)
+
 cc_library(
-    name = "powerpc_desc",
+    name = "PowerPCDesc",
     srcs = glob([
         "lib/Target/PowerPC/MCTargetDesc/*.c",
         "lib/Target/PowerPC/MCTargetDesc/*.cpp",
@@ -3327,20 +3865,25 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/PowerPC"],
     deps = [
+        ":BinaryFormat",
+        ":MC",
+        ":PowerPCInfo",
+        ":Support",
         ":attributes_gen",
-        ":binary_format",
         ":config",
         ":intrinsic_enums_gen",
         ":intrinsics_impl_gen",
-        ":mc",
-        ":powerpc_info",
         ":powerpc_target_gen",
-        ":support",
     ],
 )
 
+alias(
+    name = "powerpc_desc",
+    actual = ":PowerPCDesc",
+)
+
 cc_library(
-    name = "powerpc_disassembler",
+    name = "PowerPCDisassembler",
     srcs = glob([
         "lib/Target/PowerPC/Disassembler/*.c",
         "lib/Target/PowerPC/Disassembler/*.cpp",
@@ -3354,15 +3897,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/PowerPC"],
     deps = [
+        ":MCDisassembler",
+        ":PowerPCInfo",
+        ":Support",
         ":config",
-        ":mc_disassembler",
-        ":powerpc_info",
-        ":support",
     ],
 )
 
+alias(
+    name = "powerpc_disassembler",
+    actual = ":PowerPCDisassembler",
+)
+
 cc_library(
-    name = "powerpc_info",
+    name = "PowerPCInfo",
     srcs = glob([
         "lib/Target/PowerPC/TargetInfo/*.c",
         "lib/Target/PowerPC/TargetInfo/*.cpp",
@@ -3378,17 +3926,22 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/PowerPC"],
     deps = [
+        ":Support",
         ":attributes_gen",
         ":config",
         ":core",
         ":powerpc_target_gen",
-        ":support",
         ":target",
     ],
 )
 
+alias(
+    name = "powerpc_info",
+    actual = ":PowerPCInfo",
+)
+
 cc_library(
-    name = "profile_data",
+    name = "ProfileData",
     srcs = glob([
         "lib/ProfileData/*.c",
         "lib/ProfileData/*.cpp",
@@ -3402,14 +3955,19 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Core",
+        ":Support",
         ":config",
-        ":core",
-        ":support",
     ],
 )
 
+alias(
+    name = "profile_data",
+    actual = ":ProfileData",
+)
+
 cc_library(
-    name = "riscv_asm_parser",
+    name = "RISCVAsmParser",
     srcs = glob([
         "lib/Target/RISCV/AsmParser/*.c",
         "lib/Target/RISCV/AsmParser/*.cpp",
@@ -3423,18 +3981,23 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/RISCV"],
     deps = [
+        ":MC",
+        ":MCParser",
+        ":RISCVDesc",
+        ":RISCVInfo",
+        ":RISCVUtils",
+        ":Support",
         ":config",
-        ":mc",
-        ":mc_parser",
-        ":riscv_desc",
-        ":riscv_info",
-        ":riscv_utils",
-        ":support",
     ],
 )
 
+alias(
+    name = "riscv_asm_parser",
+    actual = ":RISCVAsmParser",
+)
+
 cc_library(
-    name = "riscv_code_gen",
+    name = "RISCVCodeGen",
     srcs = glob([
         "lib/Target/RISCV/*.c",
         "lib/Target/RISCV/*.cpp",
@@ -3448,24 +4011,29 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/RISCV"],
     deps = [
-        ":analysis",
-        ":asm_printer",
-        ":code_gen",
+        ":Analysis",
+        ":AsmPrinter",
+        ":CodeGen",
+        ":Core",
+        ":GlobalISel",
+        ":MC",
+        ":RISCVDesc",
+        ":RISCVInfo",
+        ":RISCVUtils",
+        ":SelectionDAG",
+        ":Support",
+        ":Target",
         ":config",
-        ":core",
-        ":global_i_sel",
-        ":mc",
-        ":riscv_desc",
-        ":riscv_info",
-        ":riscv_utils",
-        ":selection_dag",
-        ":support",
-        ":target",
     ],
 )
 
+alias(
+    name = "riscv_code_gen",
+    actual = ":RISCVCodeGen",
+)
+
 cc_library(
-    name = "riscv_desc",
+    name = "RISCVDesc",
     srcs = glob([
         "lib/Target/RISCV/MCTargetDesc/*.c",
         "lib/Target/RISCV/MCTargetDesc/*.cpp",
@@ -3479,16 +4047,21 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/RISCV"],
     deps = [
+        ":MC",
+        ":RISCVInfo",
+        ":RISCVUtils",
+        ":Support",
         ":config",
-        ":mc",
-        ":riscv_info",
-        ":riscv_utils",
-        ":support",
     ],
 )
 
+alias(
+    name = "riscv_desc",
+    actual = ":RISCVDesc",
+)
+
 cc_library(
-    name = "riscv_disassembler",
+    name = "RISCVDisassembler",
     srcs = glob([
         "lib/Target/RISCV/Disassembler/*.c",
         "lib/Target/RISCV/Disassembler/*.cpp",
@@ -3502,15 +4075,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/RISCV"],
     deps = [
+        ":MCDisassembler",
+        ":RISCVInfo",
+        ":Support",
         ":config",
-        ":mc_disassembler",
-        ":riscv_info",
-        ":support",
     ],
 )
 
+alias(
+    name = "riscv_disassembler",
+    actual = ":RISCVDisassembler",
+)
+
 cc_library(
-    name = "riscv_info",
+    name = "RISCVInfo",
     srcs = glob([
         "lib/Target/RISCV/TargetInfo/*.c",
         "lib/Target/RISCV/TargetInfo/*.cpp",
@@ -3524,13 +4102,18 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/RISCV"],
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "riscv_info",
+    actual = ":RISCVInfo",
+)
+
 cc_library(
-    name = "riscv_utils",
+    name = "RISCVUtils",
     srcs = glob([
         "lib/Target/RISCV/Utils/*.c",
         "lib/Target/RISCV/Utils/*.cpp",
@@ -3544,13 +4127,18 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/RISCV"],
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "riscv_utils",
+    actual = ":RISCVUtils",
+)
+
 cc_library(
-    name = "remarks",
+    name = "Remarks",
     srcs = glob([
         "lib/Remarks/*.c",
         "lib/Remarks/*.cpp",
@@ -3564,14 +4152,19 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":bitstream_reader",
+        ":BitstreamReader",
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "remarks",
+    actual = ":Remarks",
+)
+
 cc_library(
-    name = "runtime_dyld",
+    name = "RuntimeDyld",
     srcs = glob([
         "lib/ExecutionEngine/RuntimeDyld/*.c",
         "lib/ExecutionEngine/RuntimeDyld/*.cpp",
@@ -3593,16 +4186,21 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":MC",
+        ":Object",
+        ":Support",
         ":config",
-        ":mc",
         ":mc_disassembler",
-        ":object",
-        ":support",
     ],
 )
 
+alias(
+    name = "runtime_dyld",
+    actual = ":RuntimeDyld",
+)
+
 cc_library(
-    name = "scalar",
+    name = "Scalar",
     srcs = glob([
         "lib/Transforms/Scalar/*.c",
         "lib/Transforms/Scalar/*.cpp",
@@ -3621,19 +4219,24 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":aggressive_inst_combine",
-        ":analysis",
+        ":AggressiveInstCombine",
+        ":Analysis",
+        ":Core",
+        ":InstCombine",
+        ":Support",
+        ":TransformUtils",
         ":config",
-        ":core",
-        ":inst_combine",
-        ":support",
         ":target",
-        ":transform_utils",
     ],
 )
 
+alias(
+    name = "scalar",
+    actual = ":Scalar",
+)
+
 cc_library(
-    name = "selection_dag",
+    name = "SelectionDAG",
     srcs = glob([
         "lib/CodeGen/SelectionDAG/*.c",
         "lib/CodeGen/SelectionDAG/*.cpp",
@@ -3647,19 +4250,24 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":analysis",
-        ":code_gen",
+        ":Analysis",
+        ":CodeGen",
+        ":Core",
+        ":MC",
+        ":Support",
+        ":Target",
+        ":TransformUtils",
         ":config",
-        ":core",
-        ":mc",
-        ":support",
-        ":target",
-        ":transform_utils",
     ],
 )
 
+alias(
+    name = "selection_dag",
+    actual = ":SelectionDAG",
+)
+
 cc_library(
-    name = "sparc_asm_parser",
+    name = "SparcAsmParser",
     srcs = glob([
         "lib/Target/Sparc/AsmParser/*.c",
         "lib/Target/Sparc/AsmParser/*.cpp",
@@ -3673,17 +4281,22 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/Sparc"],
     deps = [
+        ":MC",
+        ":MCParser",
+        ":SparcDesc",
+        ":SparcInfo",
+        ":Support",
         ":config",
-        ":mc",
-        ":mc_parser",
-        ":sparc_desc",
-        ":sparc_info",
-        ":support",
     ],
 )
 
+alias(
+    name = "sparc_asm_parser",
+    actual = ":SparcAsmParser",
+)
+
 cc_library(
-    name = "sparc_code_gen",
+    name = "SparcCodeGen",
     srcs = glob([
         "lib/Target/Sparc/*.c",
         "lib/Target/Sparc/*.cpp",
@@ -3697,21 +4310,26 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/Sparc"],
     deps = [
-        ":asm_printer",
-        ":code_gen",
+        ":AsmPrinter",
+        ":CodeGen",
+        ":Core",
+        ":MC",
+        ":SelectionDAG",
+        ":SparcDesc",
+        ":SparcInfo",
+        ":Support",
+        ":Target",
         ":config",
-        ":core",
-        ":mc",
-        ":selection_dag",
-        ":sparc_desc",
-        ":sparc_info",
-        ":support",
-        ":target",
     ],
 )
 
+alias(
+    name = "sparc_code_gen",
+    actual = ":SparcCodeGen",
+)
+
 cc_library(
-    name = "sparc_desc",
+    name = "SparcDesc",
     srcs = glob([
         "lib/Target/Sparc/MCTargetDesc/*.c",
         "lib/Target/Sparc/MCTargetDesc/*.cpp",
@@ -3725,15 +4343,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/Sparc"],
     deps = [
+        ":MC",
+        ":SparcInfo",
+        ":Support",
         ":config",
-        ":mc",
-        ":sparc_info",
-        ":support",
     ],
 )
 
+alias(
+    name = "sparc_desc",
+    actual = ":SparcDesc",
+)
+
 cc_library(
-    name = "sparc_disassembler",
+    name = "SparcDisassembler",
     srcs = glob([
         "lib/Target/Sparc/Disassembler/*.c",
         "lib/Target/Sparc/Disassembler/*.cpp",
@@ -3747,15 +4370,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/Sparc"],
     deps = [
+        ":MCDisassembler",
+        ":SparcInfo",
+        ":Support",
         ":config",
-        ":mc_disassembler",
-        ":sparc_info",
-        ":support",
     ],
 )
 
+alias(
+    name = "sparc_disassembler",
+    actual = ":SparcDisassembler",
+)
+
 cc_library(
-    name = "sparc_info",
+    name = "SparcInfo",
     srcs = glob([
         "lib/Target/Sparc/TargetInfo/*.c",
         "lib/Target/Sparc/TargetInfo/*.cpp",
@@ -3769,13 +4397,18 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/Sparc"],
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "sparc_info",
+    actual = ":SparcInfo",
+)
+
 cc_library(
-    name = "support",
+    name = "Support",
     srcs = glob([
         "lib/Support/*.c",
         "lib/Support/*.cpp",
@@ -3799,14 +4432,19 @@ cc_library(
     ],
     copts = llvm_copts,
     deps = [
+        ":Demangle",
         ":config",
-        ":demangle",
         "@zlib",
     ],
 )
 
+alias(
+    name = "support",
+    actual = ":Support",
+)
+
 cc_library(
-    name = "symbolize",
+    name = "Symbolize",
     srcs = glob([
         "lib/DebugInfo/Symbolize/*.c",
         "lib/DebugInfo/Symbolize/*.cpp",
@@ -3820,17 +4458,22 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":DebugInfoDWARF",
+        ":DebugInfoPDB",
+        ":Demangle",
+        ":Object",
+        ":Support",
         ":config",
-        ":debug_info_dwarf",
-        ":debug_info_pdb",
-        ":demangle",
-        ":object",
-        ":support",
     ],
 )
 
+alias(
+    name = "symbolize",
+    actual = ":Symbolize",
+)
+
 cc_library(
-    name = "system_z_asm_parser",
+    name = "SystemZAsmParser",
     srcs = glob([
         "lib/Target/SystemZ/AsmParser/*.c",
         "lib/Target/SystemZ/AsmParser/*.cpp",
@@ -3844,17 +4487,22 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/SystemZ"],
     deps = [
+        ":MC",
+        ":MCParser",
+        ":Support",
+        ":SystemZDesc",
+        ":SystemZInfo",
         ":config",
-        ":mc",
-        ":mc_parser",
-        ":support",
-        ":system_z_desc",
-        ":system_z_info",
     ],
 )
 
+alias(
+    name = "system_z_asm_parser",
+    actual = ":SystemZAsmParser",
+)
+
 cc_library(
-    name = "system_z_code_gen",
+    name = "SystemZCodeGen",
     srcs = glob([
         "lib/Target/SystemZ/*.c",
         "lib/Target/SystemZ/*.cpp",
@@ -3868,23 +4516,28 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/SystemZ"],
     deps = [
-        ":analysis",
-        ":asm_printer",
-        ":code_gen",
+        ":Analysis",
+        ":AsmPrinter",
+        ":CodeGen",
+        ":Core",
+        ":MC",
+        ":Scalar",
+        ":SelectionDAG",
+        ":Support",
+        ":SystemZDesc",
+        ":SystemZInfo",
+        ":Target",
         ":config",
-        ":core",
-        ":mc",
-        ":scalar",
-        ":selection_dag",
-        ":support",
-        ":system_z_desc",
-        ":system_z_info",
-        ":target",
     ],
 )
 
+alias(
+    name = "system_z_code_gen",
+    actual = ":SystemZCodeGen",
+)
+
 cc_library(
-    name = "system_z_desc",
+    name = "SystemZDesc",
     srcs = glob([
         "lib/Target/SystemZ/MCTargetDesc/*.c",
         "lib/Target/SystemZ/MCTargetDesc/*.cpp",
@@ -3898,15 +4551,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/SystemZ"],
     deps = [
+        ":MC",
+        ":Support",
+        ":SystemZInfo",
         ":config",
-        ":mc",
-        ":support",
-        ":system_z_info",
     ],
 )
 
+alias(
+    name = "system_z_desc",
+    actual = ":SystemZDesc",
+)
+
 cc_library(
-    name = "system_z_disassembler",
+    name = "SystemZDisassembler",
     srcs = glob([
         "lib/Target/SystemZ/Disassembler/*.c",
         "lib/Target/SystemZ/Disassembler/*.cpp",
@@ -3920,17 +4578,22 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/SystemZ"],
     deps = [
+        ":MC",
+        ":MCDisassembler",
+        ":Support",
+        ":SystemZDesc",
+        ":SystemZInfo",
         ":config",
-        ":mc",
-        ":mc_disassembler",
-        ":support",
-        ":system_z_desc",
-        ":system_z_info",
     ],
 )
 
+alias(
+    name = "system_z_disassembler",
+    actual = ":SystemZDisassembler",
+)
+
 cc_library(
-    name = "system_z_info",
+    name = "SystemZInfo",
     srcs = glob([
         "lib/Target/SystemZ/TargetInfo/*.c",
         "lib/Target/SystemZ/TargetInfo/*.cpp",
@@ -3944,13 +4607,18 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/SystemZ"],
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "system_z_info",
+    actual = ":SystemZInfo",
+)
+
 cc_library(
-    name = "tablegen",
+    name = "TableGen",
     srcs = glob([
         "lib/TableGen/*.c",
         "lib/TableGen/*.cpp",
@@ -3966,14 +4634,19 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Support",
         ":config",
         ":mc",
-        ":support",
     ],
 )
 
+alias(
+    name = "tablegen",
+    actual = ":TableGen",
+)
+
 cc_library(
-    name = "target",
+    name = "Target",
     srcs = glob([
         "lib/Target/*.c",
         "lib/Target/*.cpp",
@@ -3992,16 +4665,21 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":analysis",
+        ":Analysis",
+        ":Core",
+        ":MC",
+        ":Support",
         ":config",
-        ":core",
-        ":mc",
-        ":support",
     ],
 )
 
+alias(
+    name = "target",
+    actual = ":Target",
+)
+
 cc_library(
-    name = "testing_support",
+    name = "TestingSupport",
     srcs = glob([
         "lib/Testing/Support/*.c",
         "lib/Testing/Support/*.cpp",
@@ -4015,13 +4693,18 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "testing_support",
+    actual = ":TestingSupport",
+)
+
 cc_library(
-    name = "text_api",
+    name = "TextAPI",
     srcs = glob([
         "lib/TextAPI/*.c",
         "lib/TextAPI/*.cpp",
@@ -4049,14 +4732,19 @@ cc_library(
     ],
     copts = llvm_copts,
     deps = [
-        ":binary_format",
+        ":BinaryFormat",
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "text_api",
+    actual = ":TextAPI",
+)
+
 cc_library(
-    name = "transform_utils",
+    name = "TransformUtils",
     srcs = glob([
         "lib/Transforms/Utils/*.c",
         "lib/Transforms/Utils/*.cpp",
@@ -4072,15 +4760,49 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":analysis",
+        ":Analysis",
+        ":Core",
+        ":Support",
         ":config",
-        ":core",
-        ":support",
     ],
 )
 
+alias(
+    name = "transform_utils",
+    actual = ":TransformUtils",
+)
+
 cc_library(
-    name = "ve_code_gen",
+    name = "VEAsmParser",
+    srcs = glob([
+        "lib/Target/VE/AsmParser/*.c",
+        "lib/Target/VE/AsmParser/*.cpp",
+        "lib/Target/VE/AsmParser/*.inc",
+    ]),
+    hdrs = glob([
+        "include/llvm/Target/VE/AsmParser/*.h",
+        "include/llvm/Target/VE/AsmParser/*.def",
+        "include/llvm/Target/VE/AsmParser/*.inc",
+        "lib/Target/VE/AsmParser/*.h",
+    ]),
+    copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/VE"],
+    deps = [
+        ":MC",
+        ":MCParser",
+        ":Support",
+        ":VEDesc",
+        ":VEInfo",
+        ":config",
+    ],
+)
+
+alias(
+    name = "ve_asm_parser",
+    actual = ":VEAsmParser",
+)
+
+cc_library(
+    name = "VECodeGen",
     srcs = glob([
         "lib/Target/VE/*.c",
         "lib/Target/VE/*.cpp",
@@ -4094,22 +4816,27 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/VE"],
     deps = [
-        ":analysis",
-        ":asm_printer",
-        ":code_gen",
+        ":Analysis",
+        ":AsmPrinter",
+        ":CodeGen",
+        ":Core",
+        ":MC",
+        ":SelectionDAG",
+        ":Support",
+        ":Target",
+        ":VEDesc",
+        ":VEInfo",
         ":config",
-        ":core",
-        ":mc",
-        ":selection_dag",
-        ":support",
-        ":target",
-        ":ve_desc",
-        ":ve_info",
     ],
 )
 
+alias(
+    name = "ve_code_gen",
+    actual = ":VECodeGen",
+)
+
 cc_library(
-    name = "ve_desc",
+    name = "VEDesc",
     srcs = glob([
         "lib/Target/VE/MCTargetDesc/*.c",
         "lib/Target/VE/MCTargetDesc/*.cpp",
@@ -4123,15 +4850,47 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/VE"],
     deps = [
+        ":MC",
+        ":Support",
+        ":VEInfo",
         ":config",
-        ":mc",
-        ":support",
-        ":ve_info",
     ],
 )
 
+alias(
+    name = "ve_desc",
+    actual = ":VEDesc",
+)
+
 cc_library(
-    name = "ve_info",
+    name = "VEDisassembler",
+    srcs = glob([
+        "lib/Target/VE/Disassembler/*.c",
+        "lib/Target/VE/Disassembler/*.cpp",
+        "lib/Target/VE/Disassembler/*.inc",
+    ]),
+    hdrs = glob([
+        "include/llvm/Target/VE/Disassembler/*.h",
+        "include/llvm/Target/VE/Disassembler/*.def",
+        "include/llvm/Target/VE/Disassembler/*.inc",
+        "lib/Target/VE/Disassembler/*.h",
+    ]),
+    copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/VE"],
+    deps = [
+        ":MCDisassembler",
+        ":Support",
+        ":VEInfo",
+        ":config",
+    ],
+)
+
+alias(
+    name = "ve_disassembler",
+    actual = ":VEDisassembler",
+)
+
+cc_library(
+    name = "VEInfo",
     srcs = glob([
         "lib/Target/VE/TargetInfo/*.c",
         "lib/Target/VE/TargetInfo/*.cpp",
@@ -4145,13 +4904,18 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/VE"],
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "ve_info",
+    actual = ":VEInfo",
+)
+
 cc_library(
-    name = "vectorize",
+    name = "Vectorize",
     srcs = glob([
         "lib/Transforms/Vectorize/*.c",
         "lib/Transforms/Vectorize/*.cpp",
@@ -4167,17 +4931,22 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
-        ":analysis",
+        ":Analysis",
+        ":Core",
+        ":Support",
+        ":TransformUtils",
         ":config",
-        ":core",
         ":scalar",
-        ":support",
-        ":transform_utils",
     ],
 )
 
+alias(
+    name = "vectorize",
+    actual = ":Vectorize",
+)
+
 cc_library(
-    name = "web_assembly_asm_parser",
+    name = "WebAssemblyAsmParser",
     srcs = glob([
         "lib/Target/WebAssembly/AsmParser/*.c",
         "lib/Target/WebAssembly/AsmParser/*.cpp",
@@ -4191,16 +4960,21 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/WebAssembly"],
     deps = [
+        ":MC",
+        ":MCParser",
+        ":Support",
+        ":WebAssemblyInfo",
         ":config",
-        ":mc",
-        ":mc_parser",
-        ":support",
-        ":web_assembly_info",
     ],
 )
 
+alias(
+    name = "web_assembly_asm_parser",
+    actual = ":WebAssemblyAsmParser",
+)
+
 cc_library(
-    name = "web_assembly_code_gen",
+    name = "WebAssemblyCodeGen",
     srcs = glob([
         "lib/Target/WebAssembly/*.c",
         "lib/Target/WebAssembly/*.cpp",
@@ -4214,25 +4988,30 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/WebAssembly"],
     deps = [
-        ":analysis",
-        ":asm_printer",
-        ":binary_format",
-        ":code_gen",
+        ":Analysis",
+        ":AsmPrinter",
+        ":BinaryFormat",
+        ":CodeGen",
+        ":Core",
+        ":MC",
+        ":Scalar",
+        ":SelectionDAG",
+        ":Support",
+        ":Target",
+        ":TransformUtils",
+        ":WebAssemblyDesc",
+        ":WebAssemblyInfo",
         ":config",
-        ":core",
-        ":mc",
-        ":scalar",
-        ":selection_dag",
-        ":support",
-        ":target",
-        ":transform_utils",
-        ":web_assembly_desc",
-        ":web_assembly_info",
     ],
 )
 
+alias(
+    name = "web_assembly_code_gen",
+    actual = ":WebAssemblyCodeGen",
+)
+
 cc_library(
-    name = "web_assembly_desc",
+    name = "WebAssemblyDesc",
     srcs = glob([
         "lib/Target/WebAssembly/MCTargetDesc/*.c",
         "lib/Target/WebAssembly/MCTargetDesc/*.cpp",
@@ -4246,15 +5025,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/WebAssembly"],
     deps = [
+        ":MC",
+        ":Support",
+        ":WebAssemblyInfo",
         ":config",
-        ":mc",
-        ":support",
-        ":web_assembly_info",
     ],
 )
 
+alias(
+    name = "web_assembly_desc",
+    actual = ":WebAssemblyDesc",
+)
+
 cc_library(
-    name = "web_assembly_disassembler",
+    name = "WebAssemblyDisassembler",
     srcs = glob([
         "lib/Target/WebAssembly/Disassembler/*.c",
         "lib/Target/WebAssembly/Disassembler/*.cpp",
@@ -4268,17 +5052,22 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/WebAssembly"],
     deps = [
+        ":MC",
+        ":MCDisassembler",
+        ":Support",
+        ":WebAssemblyDesc",
+        ":WebAssemblyInfo",
         ":config",
-        ":mc",
-        ":mc_disassembler",
-        ":support",
-        ":web_assembly_desc",
-        ":web_assembly_info",
     ],
 )
 
+alias(
+    name = "web_assembly_disassembler",
+    actual = ":WebAssemblyDisassembler",
+)
+
 cc_library(
-    name = "web_assembly_info",
+    name = "WebAssemblyInfo",
     srcs = glob([
         "lib/Target/WebAssembly/TargetInfo/*.c",
         "lib/Target/WebAssembly/TargetInfo/*.cpp",
@@ -4292,13 +5081,18 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/WebAssembly"],
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "web_assembly_info",
+    actual = ":WebAssemblyInfo",
+)
+
 cc_library(
-    name = "windows_manifest",
+    name = "WindowsManifest",
     srcs = glob([
         "lib/WindowsManifest/*.c",
         "lib/WindowsManifest/*.cpp",
@@ -4312,13 +5106,18 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "windows_manifest",
+    actual = ":WindowsManifest",
+)
+
 cc_library(
-    name = "x86_asm_parser",
+    name = "X86AsmParser",
     srcs = glob([
         "lib/Target/X86/AsmParser/*.c",
         "lib/Target/X86/AsmParser/*.cpp",
@@ -4332,17 +5131,22 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/X86"],
     deps = [
+        ":MC",
+        ":MCParser",
+        ":Support",
+        ":X86Desc",
+        ":X86Info",
         ":config",
-        ":mc",
-        ":mc_parser",
-        ":support",
-        ":x86_desc",
-        ":x86_info",
     ],
 )
 
+alias(
+    name = "x86_asm_parser",
+    actual = ":X86AsmParser",
+)
+
 cc_library(
-    name = "x86_code_gen",
+    name = "X86CodeGen",
     srcs = glob([
         "lib/Target/X86/*.c",
         "lib/Target/X86/*.cpp",
@@ -4356,26 +5160,31 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/X86"],
     deps = [
-        ":analysis",
-        ":asm_printer",
-        ":cf_guard",
-        ":code_gen",
+        ":Analysis",
+        ":AsmPrinter",
+        ":CFGuard",
+        ":CodeGen",
+        ":Core",
+        ":GlobalISel",
+        ":MC",
+        ":ProfileData",
+        ":SelectionDAG",
+        ":Support",
+        ":Target",
+        ":X86Desc",
+        ":X86Info",
         ":config",
-        ":core",
-        ":global_i_sel",
-        ":mc",
-        ":profile_data",
-        ":selection_dag",
-        ":support",
-        ":target",
         ":x86_defs",
-        ":x86_desc",
-        ":x86_info",
     ],
 )
 
+alias(
+    name = "x86_code_gen",
+    actual = ":X86CodeGen",
+)
+
 cc_library(
-    name = "x86_desc",
+    name = "X86Desc",
     srcs = glob([
         "lib/Target/X86/MCTargetDesc/*.c",
         "lib/Target/X86/MCTargetDesc/*.cpp",
@@ -4389,17 +5198,22 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/X86"],
     deps = [
-        ":binary_format",
+        ":BinaryFormat",
+        ":MC",
+        ":MCDisassembler",
+        ":Support",
+        ":X86Info",
         ":config",
-        ":mc",
-        ":mc_disassembler",
-        ":support",
-        ":x86_info",
     ],
 )
 
+alias(
+    name = "x86_desc",
+    actual = ":X86Desc",
+)
+
 cc_library(
-    name = "x86_disassembler",
+    name = "X86Disassembler",
     srcs = glob([
         "lib/Target/X86/Disassembler/*.c",
         "lib/Target/X86/Disassembler/*.cpp",
@@ -4413,15 +5227,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/X86"],
     deps = [
+        ":MCDisassembler",
+        ":Support",
+        ":X86Info",
         ":config",
-        ":mc_disassembler",
-        ":support",
-        ":x86_info",
     ],
 )
 
+alias(
+    name = "x86_disassembler",
+    actual = ":X86Disassembler",
+)
+
 cc_library(
-    name = "x86_info",
+    name = "X86Info",
     srcs = glob([
         "lib/Target/X86/TargetInfo/*.c",
         "lib/Target/X86/TargetInfo/*.cpp",
@@ -4436,15 +5255,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/X86"],
     deps = [
+        ":Support",
         ":config",
         ":mc",
-        ":support",
         ":x86_target_gen",
     ],
 )
 
+alias(
+    name = "x86_info",
+    actual = ":X86Info",
+)
+
 cc_library(
-    name = "x_core_code_gen",
+    name = "XCoreCodeGen",
     srcs = glob([
         "lib/Target/XCore/*.c",
         "lib/Target/XCore/*.cpp",
@@ -4458,23 +5282,28 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/XCore"],
     deps = [
-        ":analysis",
-        ":asm_printer",
-        ":code_gen",
+        ":Analysis",
+        ":AsmPrinter",
+        ":CodeGen",
+        ":Core",
+        ":MC",
+        ":SelectionDAG",
+        ":Support",
+        ":Target",
+        ":TransformUtils",
+        ":XCoreDesc",
+        ":XCoreInfo",
         ":config",
-        ":core",
-        ":mc",
-        ":selection_dag",
-        ":support",
-        ":target",
-        ":transform_utils",
-        ":x_core_desc",
-        ":x_core_info",
     ],
 )
 
+alias(
+    name = "x_core_code_gen",
+    actual = ":XCoreCodeGen",
+)
+
 cc_library(
-    name = "x_core_desc",
+    name = "XCoreDesc",
     srcs = glob([
         "lib/Target/XCore/MCTargetDesc/*.c",
         "lib/Target/XCore/MCTargetDesc/*.cpp",
@@ -4488,15 +5317,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/XCore"],
     deps = [
+        ":MC",
+        ":Support",
+        ":XCoreInfo",
         ":config",
-        ":mc",
-        ":support",
-        ":x_core_info",
     ],
 )
 
+alias(
+    name = "x_core_desc",
+    actual = ":XCoreDesc",
+)
+
 cc_library(
-    name = "x_core_disassembler",
+    name = "XCoreDisassembler",
     srcs = glob([
         "lib/Target/XCore/Disassembler/*.c",
         "lib/Target/XCore/Disassembler/*.cpp",
@@ -4510,15 +5344,20 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/XCore"],
     deps = [
+        ":MCDisassembler",
+        ":Support",
+        ":XCoreInfo",
         ":config",
-        ":mc_disassembler",
-        ":support",
-        ":x_core_info",
     ],
 )
 
+alias(
+    name = "x_core_disassembler",
+    actual = ":XCoreDisassembler",
+)
+
 cc_library(
-    name = "x_core_info",
+    name = "XCoreInfo",
     srcs = glob([
         "lib/Target/XCore/TargetInfo/*.c",
         "lib/Target/XCore/TargetInfo/*.cpp",
@@ -4532,13 +5371,18 @@ cc_library(
     ]),
     copts = llvm_copts + ["-Iexternal/llvm-project/llvm/lib/Target/XCore"],
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
+alias(
+    name = "x_core_info",
+    actual = ":XCoreInfo",
+)
+
 cc_library(
-    name = "x_ray",
+    name = "XRay",
     srcs = glob([
         "lib/XRay/*.c",
         "lib/XRay/*.cpp",
@@ -4552,12 +5396,17 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Object",
+        ":Support",
         ":config",
-        ":object",
-        ":support",
     ],
 )
 
+alias(
+    name = "x_ray",
+    actual = ":XRay",
+)
+
 cc_library(
     name = "gtest",
     srcs = glob([
@@ -4573,8 +5422,8 @@ cc_library(
     ]),
     copts = llvm_copts,
     deps = [
+        ":Support",
         ":config",
-        ":support",
     ],
 )
 
@@ -4597,3 +5446,98 @@ cc_library(
         ":gtest",
     ],
 )
+
+alias(
+    name = "aarch64_target",
+    actual = ":aarch64_code_gen",
+)
+
+alias(
+    name = "aarch64_target_disassembler",
+    actual = ":aarch64_disassembler",
+)
+
+alias(
+    name = "arm_target",
+    actual = ":arm_code_gen",
+)
+
+alias(
+    name = "arm_target_disassembler",
+    actual = ":arm_disassembler",
+)
+
+alias(
+    name = "codegen",
+    actual = ":code_gen",
+)
+
+alias(
+    name = "frontend_openmp",
+    actual = ":frontend_open_mp",
+)
+
+alias(
+    name = "ipo_transforms",
+    actual = ":ipo",
+)
+
+alias(
+    name = "ir",
+    actual = ":core",
+)
+
+alias(
+    name = "machine_code",
+    actual = ":mc",
+)
+
+alias(
+    name = "machine_code_disassembler",
+    actual = ":mc_disassembler",
+)
+
+alias(
+    name = "nvptx_target",
+    actual = ":nvptx_code_gen",
+)
+
+alias(
+    name = "objcarc_transforms",
+    actual = ":objc_arc",
+)
+
+alias(
+    name = "orcjit",
+    actual = ":orc_jit",
+)
+
+alias(
+    name = "powerpc_target",
+    actual = ":powerpc_code_gen",
+)
+
+alias(
+    name = "powerpc_target_disassembler",
+    actual = ":powerpc_disassembler",
+)
+
+alias(
+    name = "scalar_transforms",
+    actual = ":scalar",
+)
+
+alias(
+    name = "target_base",
+    actual = ":target",
+)
+
+alias(
+    name = "x86_target",
+    actual = ":x86_code_gen",
+)
+
+alias(
+    name = "x86_target_disassembler",
+    actual = ":x86_disassembler",
+)
diff --git a/third_party/mlir/BUILD b/third_party/mlir/BUILD
index e3cec7d7104..27159203cf9 100644
--- a/third_party/mlir/BUILD
+++ b/third_party/mlir/BUILD
@@ -284,7 +284,7 @@ cc_library(
         ":IR",
         ":SideEffects",
         ":VectorOps",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
@@ -313,7 +313,7 @@ cc_library(
         ":VectorOps",
         ":VectorToLLVM",
         ":VectorToSCF",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
@@ -686,6 +686,25 @@ gentbl(
     ],
 )
 
+gentbl(
+    name = "MLIRShapeCanonicalizationIncGen",
+    strip_include_prefix = "include/mlir/Dialect/Shape",
+    tbl_outs = [
+        (
+            "-gen-rewriters",
+            "include/mlir/Dialect/Shape/IR/ShapeCanonicalization.inc",
+        ),
+    ],
+    tblgen = ":mlir-tblgen",
+    td_file = "lib/Dialect/Shape/IR/ShapeCanonicalization.td",
+    td_srcs = [
+        ":StdOpsTdFiles",
+        "include/mlir/Dialect/Shape/IR/ShapeBase.td",
+        "include/mlir/Dialect/Shape/IR/ShapeOps.td",
+        "include/mlir/Interfaces/InferTypeOpInterface.td",
+    ],
+)
+
 cc_library(
     name = "Shape",
     srcs = glob(
@@ -704,6 +723,7 @@ cc_library(
         ":Dialect",
         ":IR",
         ":InferTypeOpInterface",
+        ":MLIRShapeCanonicalizationIncGen",
         ":ShapeOpsIncGen",
         ":SideEffects",
         ":Support",
@@ -711,6 +731,31 @@ cc_library(
     ],
 )
 
+cc_library(
+    name = "ShapeToStandard",
+    srcs = glob([
+        "lib/Conversion/ShapeToStandard/*.cpp",
+        "lib/Conversion/ShapeToStandard/*.h",
+    ]) + ["lib/Conversion/PassDetail.h"],
+    hdrs = glob([
+        "include/mlir/Conversion/ShapeToStandard/*.h",
+    ]),
+    includes = ["include"],
+    deps = [
+        ":Affine",
+        ":ConversionPassIncGen",
+        ":IR",
+        ":Pass",
+        ":SCFDialect",
+        ":Shape",
+        ":StandardOps",
+        ":Support",
+        ":Transforms",
+        "@llvm-project//llvm:ir",
+        "@llvm-project//llvm:support",
+    ],
+)
+
 cc_library(
     name = "StandardOps",
     srcs = glob(
@@ -901,7 +946,7 @@ cc_library(
         ":IR",
         ":LLVMAVX512IncGen",
         ":LLVMDialect",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
@@ -934,7 +979,7 @@ cc_library(
         ":LLVMAVX512ConversionIncGen",
         ":LLVMIRModuleTranslation",
         ":Translation",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
@@ -975,7 +1020,7 @@ cc_library(
         "@llvm-project//llvm:asm_parser",
         "@llvm-project//llvm:bit_reader",
         "@llvm-project//llvm:bit_writer",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
@@ -1303,10 +1348,10 @@ cc_library(
         ":Pass",
         ":Support",
         ":TargetNVVMIR",
-        "@llvm-project//llvm:core",
-        "@llvm-project//llvm:nvptx_code_gen",
+        "@llvm-project//llvm:ir",
+        "@llvm-project//llvm:nvptx_target",
         "@llvm-project//llvm:support",
-        "@llvm-project//llvm:target",
+        "@llvm-project//llvm:target_base",
     ],
 )
 
@@ -1430,7 +1475,7 @@ cc_library(
         ":StandardOps",
         ":Support",
         "@llvm-project//llvm:asm_parser",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
@@ -1502,7 +1547,7 @@ cc_library(
         ":StandardOps",
         ":Support",
         "@llvm-project//llvm:asm_parser",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
@@ -2115,7 +2160,7 @@ cc_library(
         ":Support",
         ":TransformUtils",
         ":Transforms",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
@@ -2337,8 +2382,8 @@ cc_library(
         ":LLVMIRTransforms",
         ":OpenMPDialect",
         ":Support",
-        "@llvm-project//llvm:core",
-        "@llvm-project//llvm:frontend_open_mp",
+        "@llvm-project//llvm:frontend_openmp",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
         "@llvm-project//llvm:transform_utils",
     ],
@@ -2362,7 +2407,7 @@ cc_library(
         ":Support",
         ":TargetLLVMAVX512Intr",
         ":Translation",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:ir_reader",
         "@llvm-project//llvm:support",
     ],
@@ -2386,7 +2431,7 @@ cc_library(
         ":NVVMDialect",
         ":Support",
         ":Translation",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
@@ -2409,7 +2454,7 @@ cc_library(
         ":ROCDLDialect",
         ":Support",
         ":Translation",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
@@ -2432,15 +2477,15 @@ cc_library(
         ":Translation",
         "@llvm-project//llvm:bit_reader",
         "@llvm-project//llvm:bit_writer",
-        "@llvm-project//llvm:core",
         "@llvm-project//llvm:execution_engine",
-        "@llvm-project//llvm:mc",
-        "@llvm-project//llvm:orc_jit",
+        "@llvm-project//llvm:ir",
+        "@llvm-project//llvm:machine_code",
+        "@llvm-project//llvm:orcjit",
         "@llvm-project//llvm:support",
-        "@llvm-project//llvm:target",  # fixdeps: keep
+        "@llvm-project//llvm:target_base",  # fixdeps: keep
         "@llvm-project//llvm:transform_utils",
-        "@llvm-project//llvm:x86_code_gen",  # fixdeps: keep
-        "@llvm-project//llvm:x86_disassembler",  # fixdeps: keep
+        "@llvm-project//llvm:x86_target",  # fixdeps: keep
+        "@llvm-project//llvm:x86_target_disassembler",  # fixdeps: keep
     ],
 )
 
@@ -2455,10 +2500,10 @@ cc_library(
     includes = ["include"],
     deps = [
         "@llvm-project//llvm:analysis",
-        "@llvm-project//llvm:core",
-        "@llvm-project//llvm:ipo",
+        "@llvm-project//llvm:ipo_transforms",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
-        "@llvm-project//llvm:target",
+        "@llvm-project//llvm:target_base",
     ],
 )
 
@@ -2486,6 +2531,7 @@ cc_library(
         ":Parser",
         ":Pass",
         ":SCFTransforms",
+        ":ShapeToStandard",
         ":StandardOpsTransforms",
         ":StandardToSPIRVConversions",
         ":Support",
@@ -2583,6 +2629,7 @@ cc_library(
         ":SPIRVLowering",
         ":SPIRVPassIncGen",
         ":Shape",
+        ":ShapeToStandard",
         ":StandardOps",
         ":StandardOpsTransforms",
         ":StandardOpsTransformsPassIncGen",
@@ -2659,8 +2706,8 @@ cc_library(
         ":Parser",
         ":Pass",
         ":Support",
-        "@llvm-project//llvm:core",
-        "@llvm-project//llvm:orc_jit",
+        "@llvm-project//llvm:ir",
+        "@llvm-project//llvm:orcjit",
         "@llvm-project//llvm:support",
     ],
 )
@@ -3155,7 +3202,7 @@ cc_library(
         ":Transforms",
         ":VectorToLLVM",
         ":VectorToSCF",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
@@ -3180,7 +3227,7 @@ cc_library(
         ":StandardOps",
         ":Support",
         ":Transforms",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
@@ -3269,6 +3316,7 @@ cc_library(
         "include/mlir/Dialect/Linalg/EDSC/Builders.h",
         "include/mlir/Dialect/Linalg/EDSC/FoldedIntrinsics.h",
         "include/mlir/Dialect/Linalg/Passes.h",
+        "include/mlir/Dialect/Linalg/Transforms/Hoisting.h",
         "include/mlir/Dialect/Linalg/Transforms/Transforms.h",
         "include/mlir/Dialect/Linalg/Utils/Utils.h",
     ],
@@ -3288,13 +3336,14 @@ cc_library(
         ":LinalgStructuredOpsIncGen",
         ":Pass",
         ":SCFDialect",
+        ":SCFTransforms",
         ":StandardOps",
         ":Support",
         ":TransformUtils",
         ":Transforms",
         ":TransformsPassIncGen",
         ":VectorOps",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
@@ -3386,7 +3435,7 @@ cc_library(
         ":Support",
         ":Transforms",
         ":VectorOps",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
@@ -3414,7 +3463,7 @@ cc_library(
         ":Support",
         ":Transforms",
         ":VectorOps",
-        "@llvm-project//llvm:core",
+        "@llvm-project//llvm:ir",
         "@llvm-project//llvm:support",
     ],
 )
diff --git a/third_party/toolchains/preconfig/generate/generate.bzl b/third_party/toolchains/preconfig/generate/generate.bzl
index 1c8a4dfb052..95e46c73c51 100644
--- a/third_party/toolchains/preconfig/generate/generate.bzl
+++ b/third_party/toolchains/preconfig/generate/generate.bzl
@@ -41,7 +41,7 @@ def _tensorflow_rbe_config(name, compiler, python_version, os, rocm_version = No
         env.update({
             "TF_NEED_CUDA": "1",
             "TF_CUDA_CLANG": "1" if compiler.endswith("clang") else "0",
-            "TF_CUDA_COMPUTE_CAPABILITIES": "3.0,6.0",
+            "TF_CUDA_COMPUTE_CAPABILITIES": "3.5,6.0",
             "TF_ENABLE_XLA": "1",
             "TF_CUDNN_VERSION": cudnn_version,
             "TF_CUDA_VERSION": cuda_version,
diff --git a/third_party/toolchains/remote_config/rbe_config.bzl b/third_party/toolchains/remote_config/rbe_config.bzl
index 744496e8335..08c115ab3af 100644
--- a/third_party/toolchains/remote_config/rbe_config.bzl
+++ b/third_party/toolchains/remote_config/rbe_config.bzl
@@ -37,7 +37,7 @@ def _tensorflow_rbe_config(name, compiler, python_versions, os, rocm_version = N
         env.update({
             "TF_NEED_CUDA": "1",
             "TF_CUDA_CLANG": "1" if compiler.endswith("clang") else "0",
-            "TF_CUDA_COMPUTE_CAPABILITIES": "3.0,6.0",
+            "TF_CUDA_COMPUTE_CAPABILITIES": "3.5,6.0",
             "TF_ENABLE_XLA": "1",
             "TF_CUDNN_VERSION": cudnn_version,
             "TF_CUDA_VERSION": cuda_version,