TFLM: add HIMAX WE1 EVB to support TFLM example(magic wand and micro speech)

2020-07-14 17:42:11 +08:00 · 2020-07-14 17:42:11 +08:00 · 820519bcf3
commit 820519bcf3
parent 4e0c4f226e
7 changed files with 441 additions and 2 deletions
--- a/tensorflow/lite/micro/examples/magic_wand/README.md
+++ b/tensorflow/lite/micro/examples/magic_wand/README.md
@ -12,6 +12,7 @@ then outputs the gesture to the serial port.

 -   [Getting started](#getting-started)
 -   [Deploy to Arduino](#deploy-to-arduino)
+-   [Deploy to Himax WE1 EVB](#deploy-to-himax-we1-evb)
 -   [Deploy to SparkFun Edge](#deploy-to-sparkfun-edge)
 -   [Run the tests on a development machine](#run-the-tests-on-a-development-machine)
 -   [Train your own model](#train-your-own-model)
@ -140,6 +141,138 @@ SLOPE:
 * * * * * * * *
 ```

+## Deploy to Himax WE1 EVB
+
+The following instructions will help you build and deploy this example to
+[HIMAX WE1 EVB](https://github.com/HimaxWiseEyePlus/bsp_tflu/tree/master/HIMAX_WE1_EVB_board_brief)
+board. To undstand more about using this board, please check
+[HIMAX WE1 EVB user guide](https://github.com/HimaxWiseEyePlus/bsp_tflu/tree/master/HIMAX_WE1_EVB_user_guide).
+
+### Initial Setup
+
+To use the HIMAX WE1 EVB, please make sure following software are installed:
+
+#### MetaWare Development Toolkit
+
+See
+[Install the Synopsys DesignWare ARC MetaWare Development Toolkit](/tensorflow/lite/micro/tools/make/targets/arc/README.md#install-the-synopsys-designware-arc-metaware-development-toolkit)
+section for instructions on toolchain installation.
+
+#### Make Tool version
+
+A `'make'` tool is required for deploying Tensorflow Lite Micro applications on
+HIMAX WE1 EVB, See
+[Check make tool version](/tensorflow/lite/micro/tools/make/targets/arc/README.md#make-tool)
+section for proper environment.
+
+#### Serial Terminal Emulation Application
+
+There are 2 main purposes for HIMAX WE1 EVB Debug UART port
+
+-   print application output
+-   burn application to flash by using xmodem send application binary
+
+You can use any terminal emulation program (like [PuTTY](https://www.putty.org/)
+or [minicom](https://linux.die.net/man/1/minicom)).
+
+### Generate Example Project
+
+The example project for HIMAX WE1 EVB platform can be generated with the
+following command:
+
+Download related third party data
+
+```
+make -f tensorflow/lite/micro/tools/make/Makefile TARGET=himax_we1_evb third_party_downloads
+```
+
+Generate magic wand project
+
+```
+make -f tensorflow/lite/micro/tools/make/Makefile generate_magic_wand_make_project TARGET=himax_we1_evb
+```
+
+### Build and Burn Example
+
+Following the Steps to run magic wand example at HIMAX WE1 EVB platform.
+
+1.  Go to the generated example project directory.
+
+    ```
+    cd tensorflow/lite/micro/tools/make/gen/himax_we1_evb_arc/prj/magic_wand/make
+    ```
+
+2.  Build the example using
+
+    ```
+    make app
+    ```
+
+3.  After example build finish, copy ELF file and map file to image generate
+    tool directory. \
+    image generate tool directory located at
+    `'tensorflow/lite/micro/tools/make/downloads/himax_we1_sdk/image_gen_linux_v3/'`
+
+    ```
+    cp magic_wand.elf himax_we1_evb.map ../../../../../downloads/himax_we1_sdk/image_gen_linux_v3/
+    ```
+
+4.  Go to flash image generate tool directory.
+
+    ```
+    cd ../../../../../downloads/himax_we1_sdk/image_gen_linux_v3/
+    ```
+
+5.  run image generate tool, generate flash image file.
+
+    *   Before running image generate tool, by typing `sudo chmod +x image_gen`
+        and `sudo chmod +x sign_tool` to make sure it is executable.
+
+    ```
+    image_gen -e magic_wand.elf -m himax_we1_evb.map -o out.img
+    ```
+
+6.  Download flash image file to HIMAX WE1 EVB by UART:
+
+    *   more detail about download image through UART can be found at
+        [HIMAX WE1 EVB update Flash image](https://github.com/HimaxWiseEyePlus/bsp_tflu/tree/master/HIMAX_WE1_EVB_user_guide#flash-image-update)
+
+After these steps, press reset button on the HIMAX WE1 EVB, you will see
+application output in the serial terminal. Perform following gestures `'Wing'`,`'Ring'`,`'Slope'` and you can see the otuput in serial terminal. 
+
+```
+WING:
+*         *         *
+ *       * *       *
+  *     *   *     *
+   *   *     *   *
+    * *       * *
+     *         *
+```
+
+```
+RING:
+          *
+       *     *
+     *         *
+    *           *
+     *         *
+       *     *
+          *
+```
+
+```
+SLOPE:
+        *
+       *
+      *
+     *
+    *
+   *
+  *
+ * * * * * * * *
+```
+
 ## Deploy to SparkFun Edge

 The following instructions will help you build and deploy this sample on the
--- a/tensorflow/lite/micro/examples/magic_wand/himax_we1_evb/accelerometer_handler.cc
+++ b/tensorflow/lite/micro/examples/magic_wand/himax_we1_evb/accelerometer_handler.cc
@ -0,0 +1,89 @@
+/* Copyright 2019 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include "tensorflow/lite/micro/examples/magic_wand/accelerometer_handler.h"
+
+#include "hx_drv_tflm.h"
+
+int begin_index = 0;
+
+namespace {
+// Ring buffer size
+constexpr int ring_buffer_size = 600;
+// Ring buffer
+float save_data[ring_buffer_size] = {0.0};
+// Flag to start detect gesture
+bool pending_initial_data = true;
+// Available data count in accelerometer FIFO
+int available_count = 0;
+
+}  // namespace
+
+TfLiteStatus SetupAccelerometer(tflite::ErrorReporter* error_reporter) {
+  if (hx_drv_accelerometer_initial() != HX_DRV_LIB_PASS) {
+    TF_LITE_REPORT_ERROR(error_reporter, "setup fail");
+    return kTfLiteError;
+  }
+
+  TF_LITE_REPORT_ERROR(error_reporter, "setup done");
+
+  return kTfLiteOk;
+}
+
+bool ReadAccelerometer(tflite::ErrorReporter* error_reporter, float* input,
+                       int length) {
+  // Check how many accelerometer data
+  available_count = hx_drv_accelerometer_available_count();
+
+  if (available_count == 0) return false;
+
+  for (int i = 0; i < available_count; i++) {
+    float x, y, z;
+    hx_drv_accelerometer_receive(&x, &y, &z);
+
+    const float norm_x = -x;
+    const float norm_y = y;
+    const float norm_z = z;
+
+    // Save data in milli-g unit
+    save_data[begin_index++] = norm_x * 1000;
+    save_data[begin_index++] = norm_y * 1000;
+    save_data[begin_index++] = norm_z * 1000;
+
+    // If reach end of buffer, return to 0 position
+    if (begin_index >= ring_buffer_size) begin_index = 0;
+  }
+
+  // Check if data enough for prediction
+  if (pending_initial_data && begin_index >= 200) {
+    pending_initial_data = false;
+  }
+
+  // Return if we don't have enough data
+  if (pending_initial_data) {
+    return false;
+  }
+
+  // Copy the requested number of bytes to the provided input tensor
+  for (int i = 0; i < length; ++i) {
+    int ring_array_index = begin_index + i - length;
+    if (ring_array_index < 0) {
+      ring_array_index += ring_buffer_size;
+    }
+    input[i] = save_data[ring_array_index];
+  }
+
+  return true;
+}
--- a/tensorflow/lite/micro/examples/micro_speech/README.md
+++ b/tensorflow/lite/micro/examples/micro_speech/README.md
@ -22,6 +22,7 @@ kilobytes of Flash.
 -   [Deploy to SparkFun Edge](#deploy-to-sparkfun-edge)
 -   [Deploy to STM32F746](#deploy-to-STM32F746)
 -   [Deploy to NXP FRDM K66F](#deploy-to-nxp-frdm-k66f)
+-   [Deploy to HIMAX WE1 EVB](#deploy-to-himax-we1-evb)
 -   [Run on macOS](#run-on-macos)
 -   [Run the tests on a development machine](#run-the-tests-on-a-development-machine)
 -   [Train your own model](#train-your-own-model)
@ -562,6 +563,105 @@ using [ARM Mbed](https://github.com/ARMmbed/mbed-cli).
    in black color. If there is no output on the serial port, you can connect
    headphone to headphone port to check if audio loopback path is working.

+## Deploy to HIMAX WE1 EVB
+
+The following instructions will help you build and deploy this example to
+[HIMAX WE1 EVB](https://github.com/HimaxWiseEyePlus/bsp_tflu/tree/master/HIMAX_WE1_EVB_board_brief)
+board. To undstand more about using this board, please check 
+[HIMAX WE1 EVB user guide](https://github.com/HimaxWiseEyePlus/bsp_tflu/tree/master/HIMAX_WE1_EVB_user_guide).
+
+### Initial Setup
+
+To use the HIMAX WE1 EVB, please make sure following software are installed:
+
+#### MetaWare Development Toolkit
+
+See
+[Install the Synopsys DesignWare ARC MetaWare Development Toolkit](/tensorflow/lite/micro/tools/make/targets/arc/README.md#install-the-synopsys-designware-arc-metaware-development-toolkit)
+section for instructions on toolchain installation.
+
+#### Make Tool version
+
+A `'make'` tool is required for deploying Tensorflow Lite Micro
+applications on HIMAX WE1 EVB, See 
+[Check make tool version](/tensorflow/lite/micro/tools/make/targets/arc/README.md#make-tool)
+section for proper environment.
+
+#### Serial Terminal Emulation Application
+
+There are 2 main purposes for HIMAX WE1 EVB Debug UART port 
+
+- print application output
+- burn application to flash by using xmodem send application binary
+
+You can use any terminal emulation program (like [PuTTY](https://www.putty.org/) or [minicom](https://linux.die.net/man/1/minicom)).
+
+
+### Generate Example Project
+
+The example project for HIMAX WE1 EVB platform can be generated with the following
+command:
+
+Download related third party data
+
+```
+make -f tensorflow/lite/micro/tools/make/Makefile TARGET=himax_we1_evb third_party_downloads
+```
+
+Generate micro speech project
+
+```
+make -f tensorflow/lite/micro/tools/make/Makefile generate_micro_speech_make_project TARGET=himax_we1_evb 
+```
+
+### Build and Burn Example
+
+Following the Steps to run micro speech example at HIMAX WE1 EVB platform. 
+
+1.  Go to the generated example project directory.
+
+    ```
+    cd tensorflow/lite/micro/tools/make/gen/himax_we1_evb_arc/prj/micro_speech/make
+    ```
+    
+2.  Build the example using
+
+    ```
+    make app
+    ```
+
+3.  After example build finish, copy ELF file and map file to image generate tool directory.  
+    image generate tool directory  located at `'tensorflow/lite/micro/tools/make/downloads/himax_we1_sdk/image_gen_linux_v3/'` 
+
+    ```
+    cp micro_speech.elf himax_we1_evb.map ../../../../../downloads/himax_we1_sdk/image_gen_linux_v3/
+    ```
+
+4.  Go to flash image generate tool directory. 
+
+    ```
+    cd ../../../../../downloads/himax_we1_sdk/image_gen_linux_v3/
+    ```
+
+5.  run image generate tool, generate flash image file. 
+
+    *   Before running image generate tool, by typing `sudo chmod +x image_gen`
+        and `sudo chmod +x sign_tool` to make sure it is executable. 
+
+    ```
+    image_gen -e micro_speech.elf -m himax_we1_evb.map -o out.img
+    ```    
+       
+
+6.  Download flash image file to HIMAX WE1 EVB by UART:
+
+    *   more detail about download image through UART can be found at [HIMAX WE1 EVB update Flash image](https://github.com/HimaxWiseEyePlus/bsp_tflu/tree/master/HIMAX_WE1_EVB_user_guide#flash-image-update)
+
+After these steps, press reset button on the HIMAX WE1 EVB, you will see application output in the serial
+terminal and lighting LED.
+
+![Animation on Himax WE1 EVB](images/animation_on_himax_we1_evb.gif)
+
 ## Run on macOS

 The example contains an audio provider compatible with macOS. If you have access
--- a/tensorflow/lite/micro/examples/micro_speech/himax_we1_evb/audio_provider.cc
+++ b/tensorflow/lite/micro/examples/micro_speech/himax_we1_evb/audio_provider.cc
@ -0,0 +1,58 @@
+/* Copyright 2018 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include "tensorflow/lite/micro/examples/micro_speech/audio_provider.h"
+
+#include "tensorflow/lite/micro/examples/micro_speech/micro_features/micro_model_settings.h"
+
+#include "hx_drv_tflm.h"
+
+namespace {
+// Feedback silence buffer when beginning start_ms <= 0
+int16_t g_silence[kMaxAudioSampleSize] = {0};
+// Latest time-stamp
+int32_t g_latest_audio_timestamp = 0;
+// config about audio data size and address
+hx_drv_mic_data_config_t mic_config;
+// Flag for check if audio is initialize or not
+bool g_is_audio_initialized = false;
+}  // namespace
+
+TfLiteStatus GetAudioSamples(tflite::ErrorReporter* error_reporter,
+                             int start_ms, int duration_ms,
+                             int* audio_samples_size, int16_t** audio_samples) {
+  if (!g_is_audio_initialized) {
+    if (hx_drv_mic_initial() != HX_DRV_LIB_PASS) return kTfLiteError;
+
+    hx_drv_mic_on();
+    g_is_audio_initialized = true;
+  }
+
+  if (start_ms > 0) {
+    hx_drv_mic_capture(&mic_config);
+  } else {
+    mic_config.data_size = kMaxAudioSampleSize;
+    mic_config.data_address = (uint32_t)g_silence;
+  }
+
+  *audio_samples_size = mic_config.data_size;
+  *audio_samples = (int16_t*)mic_config.data_address;
+  return kTfLiteOk;
+}
+
+int32_t LatestAudioTimestamp() {
+  hx_drv_mic_timestamp_get(&g_latest_audio_timestamp);
+  return g_latest_audio_timestamp;
+}
--- a/tensorflow/lite/micro/examples/micro_speech/himax_we1_evb/command_responder.cc
+++ b/tensorflow/lite/micro/examples/micro_speech/himax_we1_evb/command_responder.cc
@ -0,0 +1,59 @@
+/* Copyright 2019 The TensorFlow Authors. All Rights Reserved.
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.
+==============================================================================*/
+
+#include "tensorflow/lite/micro/examples/micro_speech/command_responder.h"
+#include "hx_drv_tflm.h"
+
+static int32_t last_command_time = 0;
+static uint32_t loop = 0;
+static bool all_on = 0;
+
+void RespondToCommand(tflite::ErrorReporter* error_reporter,
+                      int32_t current_time, const char* found_command,
+                      uint8_t score, bool is_new_command) {
+  loop++;
+  if (is_new_command) {
+    TF_LITE_REPORT_ERROR(error_reporter, "Heard %s (%d) @%dms", found_command,
+                         score, current_time);
+    if (found_command[0] == 'y') {
+      last_command_time = current_time;
+      hx_drv_led_off(HX_DRV_LED_RED);
+      hx_drv_led_on(HX_DRV_LED_GREEN);
+    } else if (found_command[0] == 'n') {
+      last_command_time = current_time;
+      hx_drv_led_off(HX_DRV_LED_GREEN);
+      hx_drv_led_on(HX_DRV_LED_RED);
+    }
+  }
+
+  if (last_command_time != 0) {
+    if (last_command_time < (current_time - 3000)) {
+      last_command_time = 0;
+      hx_drv_led_off(HX_DRV_LED_GREEN);
+      hx_drv_led_off(HX_DRV_LED_RED);
+    }
+  } else {
+    if ((loop % 10) == 0) {
+      if (all_on) {
+        hx_drv_led_on(HX_DRV_LED_RED);
+        hx_drv_led_on(HX_DRV_LED_GREEN);
+      } else {
+        hx_drv_led_off(HX_DRV_LED_RED);
+        hx_drv_led_off(HX_DRV_LED_GREEN);
+      }
+      all_on = !all_on;
+    }
+  }
+}
--- a/tensorflow/lite/micro/examples/micro_speech/images/animation_on_himax_we1_evb.gif
+++ b/tensorflow/lite/micro/examples/micro_speech/images/animation_on_himax_we1_evb.gif
--- a/tensorflow/lite/micro/tools/make/third_party_downloads.inc
+++ b/tensorflow/lite/micro/tools/make/third_party_downloads.inc
@ -86,7 +86,7 @@ XTENSA_HIFI4_MD5 :="f234764928f9a42901df33a27e118c8b"
 ETHOSU_URL := "https://git.mlplatform.org/ml/ethos-u/ethos-u-core-driver.git/snapshot/ethos-u-core-driver-bcb5aaa99756f1b5c1295b079ebdd60996bc75a5.tar.gz"
 ETHOSU_MD5 := "d2073c8d88fc167fd5c46b5dcda58ea1"

-HIMAX_WE1_SDK_URL ="https://www.himax.com.tw/we-i/himax_we1_sdk_v02.zip"
-HIMAX_WE1_SDK_MD5 ="9a4b2f29b16052764e437b64bdcba816"
+HIMAX_WE1_SDK_URL ="https://www.himax.com.tw/we-i/himax_we1_sdk_v03.zip"
+HIMAX_WE1_SDK_MD5 ="1cd9b17f3fdb3e9a1dfd1cc356694325"