c29e9f25e7
An exception is being thrown when objects that use `CapturableResourceDeleter` are garbage collected at the end of a program's life. This can happen in very normal circumstances, such as when using `saved_model_cli` to inspect a model. The cause of the exception appears to be a race condition with garbage collection between `CapturableResourceDeleter` and `ScopedTFFunction`. Both define a custom finalizer (`__del__`); `CaptureableResourceDeleter`'s finalizer ultimately calls a concrete function which calls an `_EagerDefinedFunction` which attempts to load and execute a `ScopedTFFunction`. In the case of multiple objects in a reference cycle all going unreachable during the same garbage collection cycle, we get no guaranteed ordering for which of the objects will be collected first. In the case of the exception, `ScopedTFFunction` is collected first and its underlying function is deleted. Later, `CapturableResourceDeleter` is called, which fails, since the function it's trying to call is gone. PiperOrigin-RevId: 358292164 Change-Id: I9162d5de622f5c1ec9b2954647b9958a7d3d87b6
TensorFlow SavedModel
[TOC]
Overview
SavedModel is the universal serialization format for TensorFlow models.
SavedModel provides a language-neutral format to save machine-learning models that is recoverable and hermetic. It enables higher-level systems and tools to produce, consume and transform TensorFlow models.
Guides
- Using the SavedModel Format
- Save and load Keras models
- Save and load with checkpointing in Keras
- Training checkpoints
- Save and load a model using a distribution strategy
Public API
tf.saved_model.savetf.saved_model.loadtf.saved_model.SaveOptionstf.saved_model.LoadOptionstf.saved_model.Assettf.saved_model.contains_saved_model
Related Modules and Functions
The SavedModel Format
A SavedModel directory has the following structure:
assets/
assets.extra/
variables/
variables.data-?????-of-?????
variables.index
saved_model.pb
- SavedModel protocol buffer
saved_model.pborsaved_model.pbtxt- Includes the graph definitions as
MetaGraphDefprotocol buffers.
- Assets
- Subfolder called
assets. - Contains auxiliary files such as vocabularies, etc.
- Subfolder called
- Extra assets
- Subfolder where higher-level libraries and users can add their own assets that co-exist with the model, but are not loaded by the graph.
- This subfolder is not managed by the SavedModel libraries.
- Variables
- Subfolder called
variables.variables.data-?????-of-?????variables.index
- Subfolder called
Stripping Default valued attributes
The SavedModelBuilder class allows users to control whether default-valued
attributes must be stripped from the NodeDefs while adding a meta graph to the
SavedModel bundle. Both SavedModelBuilder.add_meta_graph_and_variables and
SavedModelBuilder.add_meta_graph methods accept a Boolean flag
strip_default_attrs that controls this behavior.
If strip_default_attrs is False, the exported MetaGraphDef will have the
default valued attributes in all it's NodeDef instances. This can break forward
compatibility with a sequence of events such as the following:
- An existing Op (
Foo) is updated to include a new attribute (T) with a default (bool) at version 101. - A model producer (such as a Trainer) binary picks up this change
(version 101) to the OpDef and re-exports an existing model that uses Op
Foo. - A model consumer (such as Tensorflow Serving) running an older binary
(version 100) doesn't have attribute
Tfor OpFoo, but tries to import this model. The model consumer doesn't recognize attributeTin a NodeDef that uses OpFooand therefore fails to load the model.
By setting strip_default_attrs to True, the model producers can strip away
any default valued attributes in the NodeDefs. This helps ensure that newly
added attributes with defaults don't cause older model consumers to fail loading
models regenerated with newer training binaries.
TIP: If you care about forward compatibility, then set strip_default_attrs
to True while using SavedModelBuilder.add_meta_graph_and_variables and
SavedModelBuilder.add_meta_graph.