experiments
/
STT-tensorflow
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Update the documentation for Model metadata 

PiperOrigin-RevId: 317131186
Change-Id: Ida1cd7fc729d7d601c32d646bd650fe1aa539591
This commit is contained in:
Lu Wang 2020-06-18 10:29:58 -07:00 committed by TensorFlower Gardener
parent 455750f362
commit 81041bcd82
1 changed files with 60 additions and 12 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

72
tensorflow/lite/g3doc/convert/metadata.md
View File

@ -9,7 +9,8 @@ input / output information. The metadata consists of both
* human readable parts which convey the best practice when using the model, * human readable parts which convey the best practice when using the model,
and and
* machine readable parts that can be leveraged by code generators, such as * machine readable parts that can be leveraged by code generators, such as
[the TensorFlow Lite Android code generator](../guide/codegen.md). [the TensorFlow Lite Android code generator](../guide/codegen.md) and
[the Android Studio ML Binding feature](https://developer.android.com/studio/preview/features#tensor-flow-lite-models).
## Setup the metadata tools ## Setup the metadata tools
@ -32,11 +33,21 @@ There are three parts to the
[model metadata](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/experimental/support/metadata/metadata_schema.fbs): [model metadata](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/experimental/support/metadata/metadata_schema.fbs):
1. **Model information** - Overall description of the model as well as items 1. **Model information** - Overall description of the model as well as items
such as licence terms. such as licence terms. See
[ModelMetadata](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/experimental/support/metadata/metadata_schema.fbs#L515).
2. **Input information** - Description of the inputs and pre-processing 2. **Input information** - Description of the inputs and pre-processing
required such as normalization. required such as normalization. See
[SubGraphMetadata.input_tensor_metadata](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/experimental/support/metadata/metadata_schema.fbs#L500).
3. **Output information** - Description of the output and post-processing 3. **Output information** - Description of the output and post-processing
required such as mapping to labels. required such as mapping to labels. See
[SubGraphMetadata.output_tensor_metadata](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/experimental/support/metadata/metadata_schema.fbs#L509).
Since TensorFlow Lite only supports single subgraph at this point, the
[TensorFlow Lite code generator](../guide/codegen.md) and
[the Android Studio ML Binding feature](https://developer.android.com/studio/preview/features#tensor-flow-lite-models)
will use `ModelMetadata.name` and `ModelMetadata.description`, instead of
`SubGraphMetadata.name` and `SubGraphMetadata.description`, when displaying
metadata and generating code.
### Supported Input / Output types ### Supported Input / Output types
@ -51,6 +62,29 @@ Lite metadata:
* Bounding box - Rectangular shape bounding boxes. The schema supports * Bounding box - Rectangular shape bounding boxes. The schema supports
[a variety of numbering schemes](https://github.com/tensorflow/tensorflow/blob/268853ee81edab09e07f455cc918f7ef9a421485/tensorflow/lite/experimental/support/metadata/metadata_schema.fbs#L165). [a variety of numbering schemes](https://github.com/tensorflow/tensorflow/blob/268853ee81edab09e07f455cc918f7ef9a421485/tensorflow/lite/experimental/support/metadata/metadata_schema.fbs#L165).
### Pack the associated files
TensorFlow Lite models may come with different associated files. For example,
natural language models usually have vocab files that map word pieces to word
IDs; classification models may have label files that indicate object categories.
Without the associated files (if there are), a model will not function well.
The associated files can now be bundled with the model through the metadata
Python library. The new TensorFlow Lite model becomes a zip file that contains
both the model and the associated files. It can be unpacked with common zip
tools. This new model format keeps using the same file extension, `.tflite`. It
is compatible with existing TFLite framework and Interpreter. See
[Pack mtadata and associated files into the model](#pack-metadata-and-associated-files-into-the-model)
for more details.
The associate file information can be recored in the metadata. Depending on the
file type and where the file is attached to (i.e. `ModelMetadata`,
`SubGraphMetadata`, and `TensorMetadata`),
[the TensorFlow Lite Android code generator](../guide/codegen.md) may apply
corresponding pre/post processing automatically to the object. See
[the \<Codegen usage\> section of each associate file type](https://github.com/tensorflow/tensorflow/blob/268853ee81edab09e07f455cc918f7ef9a421485/tensorflow/lite/experimental/support/metadata/metadata_schema.fbs#L37-L77)
in the schema for more details.
### Examples ### Examples
Note: The export directory specified has to exist before you run the script; it Note: The export directory specified has to exist before you run the script; it
@ -63,7 +97,9 @@ types of models here:
Download the script Download the script
[here](https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification/metadata/metadata_writer_for_image_classifier.py) [here](https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification/metadata/metadata_writer_for_image_classifier.py)
and run the script like this: , which populates metadata to
[mobilenet_v1_0.75_160_quantized.tflite](https://tfhub.dev/tensorflow/lite-model/mobilenet_v1_0.75_160_quantized/1/default/1).
Run the script like this:
```sh ```sh
python ./metadata_writer_for_image_classifier.py \ python ./metadata_writer_for_image_classifier.py \
@ -72,8 +108,11 @@ python ./metadata_writer_for_image_classifier.py \
--export_directory=model_with_metadata --export_directory=model_with_metadata
``` ```
The rest of this guide will highlight some of the key sections in the image To populate metadata for other image classification models, add the model specs
classification example to illustrate the key elements. like
[this](https://github.com/tensorflow/examples/blob/master/lite/examples/image_classification/metadata/metadata_writer_for_image_classifier.py#L63-L74)
into the script. The rest of this guide will highlight some of the key sections
in the image classification example to illustrate the key elements.
### Deep dive into the image classification example ### Deep dive into the image classification example
@ -173,7 +212,7 @@ label_file.type = _metadata_fb.AssociatedFileType.TENSOR_AXIS_LABELS
output_meta.associatedFiles = [label_file] output_meta.associatedFiles = [label_file]
``` ```
#### Put it all together #### Create the metadata Flatbuffers
The following code combines the model information with the input and output The following code combines the model information with the input and output
information: information:
@ -192,8 +231,10 @@ b.Finish(
metadata_buf = b.Output() metadata_buf = b.Output()
``` ```
Once the data structure is ready, the metadata is written into the TFLite file #### Pack metadata and associated files into the model
via the `populate` method:
Once the metadata Flatbuffers is created, the metadata and the label file are
written into the TFLite file via the `populate` method:
```python ```python
populator = _metadata.MetadataPopulator.with_model_file(model_file) populator = _metadata.MetadataPopulator.with_model_file(model_file)
@ -202,9 +243,16 @@ populator.load_associated_files(["your_path_to_label_file"])
populator.populate() populator.populate()
``` ```
#### Verify the metadata You can pack as many associated files as you want into the model through
`load_associated_files`. However, it is required to pack at least those files
documented in the metadata. In this example, packing the lable file is
mandatory.
You can read the metadata in a TFLite file using the `MetadataDisplayer`: ### Visualize the metadata
You can use [Netron](https://github.com/lutzroeder/netron) to visualize your
metadata, or you can read the metadata from a TensorFlow Lite model into a json
format using the `MetadataDisplayer`:
```python ```python
displayer = _metadata.MetadataDisplayer.with_model_file(export_model_path) displayer = _metadata.MetadataDisplayer.with_model_file(export_model_path)