From 81041bcd8267056741c8b4766bd8018a7270f454 Mon Sep 17 00:00:00 2001 From: Lu Wang Date: Thu, 18 Jun 2020 10:29:58 -0700 Subject: [PATCH] Update the documentation for Model metadata PiperOrigin-RevId: 317131186 Change-Id: Ida1cd7fc729d7d601c32d646bd650fe1aa539591 --- tensorflow/lite/g3doc/convert/metadata.md | 72 +++++++++++++++++++---- 1 file changed, 60 insertions(+), 12 deletions(-) diff --git a/tensorflow/lite/g3doc/convert/metadata.md b/tensorflow/lite/g3doc/convert/metadata.md index 29b2c5ce2b3..cd86333b305 100644 --- a/tensorflow/lite/g3doc/convert/metadata.md +++ b/tensorflow/lite/g3doc/convert/metadata.md @@ -9,7 +9,8 @@ input / output information. The metadata consists of both * human readable parts which convey the best practice when using the model, and * machine readable parts that can be leveraged by code generators, such as - [the TensorFlow Lite Android code generator](../guide/codegen.md). + [the TensorFlow Lite Android code generator](../guide/codegen.md) and + [the Android Studio ML Binding feature](https://developer.android.com/studio/preview/features#tensor-flow-lite-models). ## Setup the metadata tools @@ -32,11 +33,21 @@ There are three parts to the [model metadata](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/experimental/support/metadata/metadata_schema.fbs): 1. **Model information** - Overall description of the model as well as items - such as licence terms. + such as licence terms. See + [ModelMetadata](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/experimental/support/metadata/metadata_schema.fbs#L515). 2. **Input information** - Description of the inputs and pre-processing - required such as normalization. + required such as normalization. See + [SubGraphMetadata.input_tensor_metadata](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/experimental/support/metadata/metadata_schema.fbs#L500). 3. **Output information** - Description of the output and post-processing - required such as mapping to labels. + required such as mapping to labels. See + [SubGraphMetadata.output_tensor_metadata](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/experimental/support/metadata/metadata_schema.fbs#L509). + +Since TensorFlow Lite only supports single subgraph at this point, the +[TensorFlow Lite code generator](../guide/codegen.md) and +[the Android Studio ML Binding feature](https://developer.android.com/studio/preview/features#tensor-flow-lite-models) +will use `ModelMetadata.name` and `ModelMetadata.description`, instead of +`SubGraphMetadata.name` and `SubGraphMetadata.description`, when displaying +metadata and generating code. ### Supported Input / Output types @@ -51,6 +62,29 @@ Lite metadata: * Bounding box - Rectangular shape bounding boxes. The schema supports [a variety of numbering schemes](https://github.com/tensorflow/tensorflow/blob/268853ee81edab09e07f455cc918f7ef9a421485/tensorflow/lite/experimental/support/metadata/metadata_schema.fbs#L165). +### Pack the associated files + +TensorFlow Lite models may come with different associated files. For example, +natural language models usually have vocab files that map word pieces to word +IDs; classification models may have label files that indicate object categories. +Without the associated files (if there are), a model will not function well. + +The associated files can now be bundled with the model through the metadata +Python library. The new TensorFlow Lite model becomes a zip file that contains +both the model and the associated files. It can be unpacked with common zip +tools. This new model format keeps using the same file extension, `.tflite`. It +is compatible with existing TFLite framework and Interpreter. See +[Pack mtadata and associated files into the model](#pack-metadata-and-associated-files-into-the-model) +for more details. + +The associate file information can be recored in the metadata. Depending on the +file type and where the file is attached to (i.e. `ModelMetadata`, +`SubGraphMetadata`, and `TensorMetadata`), +[the TensorFlow Lite Android code generator](../guide/codegen.md) may apply +corresponding pre/post processing automatically to the object. See +[the \ section of each associate file type](https://github.com/tensorflow/tensorflow/blob/268853ee81edab09e07f455cc918f7ef9a421485/tensorflow/lite/experimental/support/metadata/metadata_schema.fbs#L37-L77) +in the schema for more details. + ### Examples Note: The export directory specified has to exist before you run the script; it @@ -63,7 +97,9 @@ types of models here: Download the script [here](https://github.com/tensorflow/examples/tree/master/lite/examples/image_classification/metadata/metadata_writer_for_image_classifier.py) -and run the script like this: +, which populates metadata to +[mobilenet_v1_0.75_160_quantized.tflite](https://tfhub.dev/tensorflow/lite-model/mobilenet_v1_0.75_160_quantized/1/default/1). +Run the script like this: ```sh python ./metadata_writer_for_image_classifier.py \ @@ -72,8 +108,11 @@ python ./metadata_writer_for_image_classifier.py \ --export_directory=model_with_metadata ``` -The rest of this guide will highlight some of the key sections in the image -classification example to illustrate the key elements. +To populate metadata for other image classification models, add the model specs +like +[this](https://github.com/tensorflow/examples/blob/master/lite/examples/image_classification/metadata/metadata_writer_for_image_classifier.py#L63-L74) +into the script. The rest of this guide will highlight some of the key sections +in the image classification example to illustrate the key elements. ### Deep dive into the image classification example @@ -173,7 +212,7 @@ label_file.type = _metadata_fb.AssociatedFileType.TENSOR_AXIS_LABELS output_meta.associatedFiles = [label_file] ``` -#### Put it all together +#### Create the metadata Flatbuffers The following code combines the model information with the input and output information: @@ -192,8 +231,10 @@ b.Finish( metadata_buf = b.Output() ``` -Once the data structure is ready, the metadata is written into the TFLite file -via the `populate` method: +#### Pack metadata and associated files into the model + +Once the metadata Flatbuffers is created, the metadata and the label file are +written into the TFLite file via the `populate` method: ```python populator = _metadata.MetadataPopulator.with_model_file(model_file) @@ -202,9 +243,16 @@ populator.load_associated_files(["your_path_to_label_file"]) populator.populate() ``` -#### Verify the metadata +You can pack as many associated files as you want into the model through +`load_associated_files`. However, it is required to pack at least those files +documented in the metadata. In this example, packing the lable file is +mandatory. -You can read the metadata in a TFLite file using the `MetadataDisplayer`: +### Visualize the metadata + +You can use [Netron](https://github.com/lutzroeder/netron) to visualize your +metadata, or you can read the metadata from a TensorFlow Lite model into a json +format using the `MetadataDisplayer`: ```python displayer = _metadata.MetadataDisplayer.with_model_file(export_model_path)