experiments/STT-tensorflow
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Documentation for TFLite Core ML delegate

Browse Source 
PiperOrigin-RevId: 304319290
Change-Id: I99515c35f8cb4aae8ab4ba62a85d77610739909e
This commit is contained in:
Taehee Jeong 2020-04-01 20:49:33 -07:00 committed by TensorFlower Gardener
parent 54822b543e
commit b3b6d2ab7b
4 changed files with 239 additions and 29 deletions
tensorflow/lite
experimental/delegates/coreml
README.md
g3doc
_book.yaml
performance
coreml_delegate.mddelegates.md

24
tensorflow/lite/experimental/delegates/coreml/README.md
View File

@ -8,19 +8,19 @@ which results in faster model inference on iOS devices.
## Supported iOS versions and processors
* iOS 11 and later. In the older iOS version, Core ML delegate will
* iOS 12 and later. In the older iOS versions, Core ML delegate will
automatically fallback to CPU.
* When running on iPhone Xs and later, it will use Neural Engine for faster
inference.
## Update code to use core ML delegate
## Update code to use Core ML delegate
### Swift
Initialize TensorFlow Lite interpreter with Core ML delegate.
```swift
let coreMLDelegate = CoreMLDelegate()
let coreMlDelegate = CoreMLDelegate()
let interpreter = try Interpreter(modelPath: modelPath,
delegates: [coreMLDelegate])
```
@ -91,13 +91,13 @@ Following ops are supported by the Core ML delegate.
* Does Core ML delegate support fallback to CPU if a graph contains unsupported
ops?
* Yes
* Yes.
* Does Core ML delegate work on iOS Simulator?
* Yes. The library includes x86 and x86_64 targets so it can run on
a simulator, but you will not see performance boost over CPU.
* Does TensorFlow Lite and CoreML delegate support MacOS?
* TensorFlow Lite is only tested on iOS but not MacOS.
* Is custom TF Lite ops supported?
* Does TensorFlow Lite and Core ML delegate support macOS?
* TensorFlow Lite is only tested on iOS but not macOS.
* Are custom TF Lite ops supported?
* No, CoreML delegate does not support custom ops and they will fallback to
CPU.
@ -110,8 +110,8 @@ Following ops are supported by the Core ML delegate.
/// TensorFlow Lite graph operations.
///
/// - Important: This is an experimental interface that is subject to change.
public final class CoreMlDelegate: Delegate {
/// The configuration options for the `CoreMlDelegate`.
public final class CoreMLDelegate: Delegate {
/// The configuration options for the `CoreMLDelegate`.
public let options: Options
// Conformance to the `Delegate` protocol.
@ -121,7 +121,7 @@ public final class CoreMlDelegate: Delegate {
///
/// - Parameters:
/// - options: Configurations for the delegate. The default is a new instance of
/// `CoreMlDelegate.Options` with the default configuration values.
/// `CoreMLDelegate.Options` with the default configuration values.
public init(options: Options = Options()) {
self.options = options
var delegateOptions = TfLiteCoreMlDelegateOptions()
@ -133,8 +133,8 @@ public final class CoreMlDelegate: Delegate {
}
}
extension CoreMlDelegate {
/// Options for configuring the `CoreMlDelegate`.
extension CoreMLDelegate {
/// Options for configuring the `CoreMLDelegate`.
public struct Options: Equatable, Hashable {
/// Creates a new instance with the default values.
public init() {}

3
tensorflow/lite/g3doc/_book.yaml
View File

@ -127,6 +127,9 @@ upper_tabs:
- title: "Hexagon delegate"
path: /lite/performance/hexagon_delegate
status: experimental
- title: "Core ML delegate"
path: /lite/performance/coreml_delegate
status: experimental
- heading: "Optimize a model"
- title: "Overview"

188
tensorflow/lite/g3doc/performance/coreml_delegate.md Normal file
View File

@ -0,0 +1,188 @@
# Tensorflow Lite Core ML Delegate
TensorFlow Lite Core ML Delegate enables running TensorFlow Lite models on
[Core ML framework](https://developer.apple.com/documentation/coreml),
which results in faster model inference on iOS devices.
Note: This delegate is in experimental (beta) phase.
Note: Core ML delegate is using Core ML version 2.1.
**Supported iOS versions and devices:**
* iOS 12 and later. In the older iOS versions, Core ML delegate will
automatically fallback to CPU.
* When running on iPhone Xs and later, it will use Neural Engine for faster
inference.
**Supported models**
The Core ML delegate currently supports float32 models.
## Trying the Core ML delegate on your own model
The Core ML delegate is already included in nightly release of TensorFlow lite
CocoaPods. To use Core ML delegate, change your TensorFlow lite pod
(`TensorflowLiteC` for C++ API, and `TensorFlowLiteSwift` for Swift) version to
`0.0.1-nightly` in your `Podfile`.
```
target 'YourProjectName'
# pod 'TensorFlowLiteSwift'
pod 'TensorFlowLiteSwift', '~> 0.0.1-nightly'
```
Note: After updating `Podfile`, you should run `pod update` to reflect changes.
### Swift
Initialize TensorFlow Lite interpreter with the Core ML delegate.
```swift
let coreMlDelegate = CoreMLDelegate()
let interpreter = try Interpreter(modelPath: modelPath,
delegates: [coreMlDelegate])
```
### Objective-C++
The Core ML delegate uses C++ API for Objective-C++ codes.
#### Step 1. Include `coreml_delegate.h`.
```objectivec++
#include "tensorflow/lite/experimental/delegates/coreml/coreml_delegate.h"
```
#### Step 2. Create a delegate and initialize a TensorFlow Lite Interpreter
After initializing the interpreter, call `interpreter->ModifyGraphWithDelegate`
with initialized Core ML delegate to apply the delegate.
```objectivec++
// initializer interpreter with model.
tflite::InterpreterBuilder(*model, resolver)(&interpreter);
// Add following section to use the Core ML delegate.
TfLiteCoreMlDelegateOptions options = {};
delegate = TfLiteCoreMlDelegateCreate(&options);
interpreter->ModifyGraphWithDelegate(delegate);
// ...
```
#### Step 3. Dispose the delegate when it is no longer used.
Add this code to the section where you dispose of the delegate (e.g. `dealloc`
of class).
```objectivec++
TfLiteCoreMlDelegateDelete(delegate);
```
## Supported ops
Following ops are supported by the Core ML delegate.
* Add
* Only certain shapes are broadcastable. In Core ML tensor layout,
following tensor shapes are broadcastable. `[B, C, H, W]`, `[B, C, 1,
1]`, `[B, 1, H, W]`, `[B, 1, 1, 1]`.
* AveragePool2D
* Concat
* Conv2D
* Weights and bias should be constant.
* DepthwiseConv2D
* Weights and bias should be constant.
* Hardswish
* Logistic (aka Sigmoid)
* MaxPool2D
* Mul
* Only certain shapes are broadcastable. In Core ML tensor layout,
following tensor shapes are broadcastable. `[B, C, H, W]`, `[B, C, 1,
1]`, `[B, 1, H, W]`, `[B, 1, 1, 1]`.
* Relu
* ReluN1To1
* Relu6
* Reshape
* ResizeBilinear
* SoftMax
* Tanh
## Feedback
For issues, please create a
[GitHub](https://github.com/tensorflow/tensorflow/issues/new?template=50-other-issues.md)
issue with all the necessary details to reproduce.
## FAQ
* Does CoreML delegate support fallback to CPU if a graph contains unsupported
ops?
* Yes
* Does CoreML delegate work on iOS Simulator?
* Yes. The library includes x86 and x86_64 targets so it can run on
a simulator, but you will not see performance boost over CPU.
* Does TensorFlow Lite and CoreML delegate support MacOS?
* TensorFlow Lite is only tested on iOS but not MacOS.
* Is custom TF Lite ops supported?
* No, CoreML delegate does not support custom ops and they will fallback to
CPU.
## APIs
### Core ML delegate Swift API
```swift
/// A delegate that uses the `Core ML` framework for performing
/// TensorFlow Lite graph operations.
///
/// - Important: This is an experimental interface that is subject to change.
public final class CoreMLDelegate: Delegate {
/// The configuration options for the `CoreMLDelegate`.
public let options: Options
// Conformance to the `Delegate` protocol.
public private(set) var cDelegate: CDelegate
* /// Creates a new instance configured with the given `options`.
///
/// - Parameters:
/// - options: Configurations for the delegate. The default is a new instance of
/// `CoreMLDelegate.Options` with the default configuration values.
public init(options: Options = Options()) {
self.options = options
var delegateOptions = TfLiteCoreMlDelegateOptions()
cDelegate = TfLiteCoreMlDelegateCreate(&delegateOptions)
}
deinit {
TfLiteCoreMlDelegateDelete(cDelegate)
}
}
extension CoreMLDelegate {
/// Options for configuring the `CoreMLDelegate`.
public struct Options: Equatable, Hashable {
/// Creates a new instance with the default values.
public init() {}
}
}
```
### Core ML delegate C++ API
```c++
typedef struct {
// We have dummy for now as we can't have empty struct in C.
char dummy;
} TfLiteCoreMlDelegateOptions;
// Return a delegate that uses CoreML for ops execution.
// Must outlive the interpreter.
TfLiteDelegate* TfLiteCoreMlDelegateCreate(
const TfLiteCoreMlDelegateOptions* options);
// Do any needed cleanup and delete 'delegate'.
void TfLiteCoreMlDelegateDelete(TfLiteDelegate* delegate);
```

49
tensorflow/lite/g3doc/performance/delegates.md
View File

@ -1,17 +1,18 @@
# TensorFlow Lite delegates
Note: Delegate API is still experimental and is subject to change.
## What is a TensorFlow Lite delegate?
A TensorFlow Lite delegate is a way to delegate part or all of graph execution to another executor.
A TensorFlow Lite delegate is a way to delegate part or all of graph execution
to another executor.
## Why should I use delegates?
Running inference on compute-heavy machine learning models on mobile devices is resource demanding due to the devices' limited processing and power.
Running inference on compute-heavy machine learning models on mobile devices is
resource demanding due to the devices' limited processing and power.
Instead of relying on the CPU, some devices have hardware accelerators, such as GPU or DSP, that allows for better performance and higher energy efficiency.
Instead of relying on the CPU, some devices have hardware accelerators, such as
GPU or DSP, that allows for better performance and higher energy efficiency.
## Using the built-in delegates
@ -33,6 +34,12 @@ TensorFlow Lite provides the following delegates for hardware acceleration:
can be used on devices older version of Android OS that does not fully
support NNAPI. See [TensorFlow Lite Hexagon delegate](hexagon_delegate.md)
for more detail.
* **Core ML delegate for newer iPhones and iPads** - For newer iPhones and
iPads where Neural Engine is available, you can use Core ML delegate to
accelerate inference for 32-bit float based models. Neural Engine is
available Apple mobile devices with A12 SoC or higher. For an overview of
the Core ML delegate and step-by-step instructions, see
[TensorFlow Lite Core ML delegate](coreml_delegate.md).
## How do delegates work?
@ -40,16 +47,25 @@ Let's say we have a simple model graph such as the following:
![Original graph](../images/performance/tflite_delegate_graph_1.png "Original Graph")
If a delegate was provided for specific operations, then TensorFlow Lite will split the graph into multiple subgraphs where each subgraph will be handled by a delegate.
If a delegate was provided for specific operations, then TensorFlow Lite will
split the graph into multiple subgraphs where each subgraph will be handled by a
delegate.
Let's assume that there is a delegate "MyDelegate," which has a faster implementation for Conv2D and Mean operations. The resulting main graph will be updated to look like below.
Let's assume that there is a delegate "MyDelegate," which has a faster
implementation for Conv2D and Mean operations. The resulting main graph will be
updated to look like below.
![Graph with delegate](../images/performance/tflite_delegate_graph_2.png "Graph with delegate")
Each subgraph that is handled by a delegate will be replaced with a node that evaluates the subgraph on its invoked call.
Depending on the model, the final graph can end up with one node, which means that all of the graphs were delegated or multiple nodes handled the subgraphs. In general, you don't want to have multiple subgraphs handled by the delegate, since each time you switch from delegate to the main graph, there is an overhead for passing the results from the subgraph to the main graph. It's not always safe to share memory.
Each subgraph that is handled by a delegate will be replaced with a node that
evaluates the subgraph on its invoked call.
Depending on the model, the final graph can end up with one node, which means
that all of the graphs were delegated or multiple nodes handled the subgraphs.
In general, you don't want to have multiple subgraphs handled by the delegate,
since each time you switch from delegate to the main graph, there is an overhead
for passing the results from the subgraph to the main graph. It's not always
safe to share memory.
## How to add a delegate
@ -57,12 +73,15 @@ _Note that the API used below is experimental and is subject to change._
Based on the previous section, to add a delegate, we need to do the following:
1. Define a kernel node that is responsible for evaluating the delegate
subgraph
1. Create an instance of
[TfLiteDelegate](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/c/common.h#L611),
which is responsible for registering the kernel node and claiming the nodes
that the delegate can execute
1. Define a kernel node that is responsible for evaluating the delegate subgraph
1. Create an instance of [TfLiteDelegate](https://github.com/tensorflow/tensorflow/blob/master/tensorflow/lite/c/common.h#L611), which is responsible for registering the kernel node and claiming the nodes that the delegate can execute
To see it in code, let's define a delegate and call it "MyDelegate," which can execute Conv2D and Mean operations faster.
To see it in code, let's define a delegate and call it "MyDelegate," which can
execute Conv2D and Mean operations faster.
```
// This is where the execution of the operations or whole graph happens.