experiments/STT-tensorflow
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Refactor Python PIP build scripts for ARM/ARM64

Browse Source 
- Simplify build commands with only container name without providing environment variables.
- Add Python 3.8 support.
- Update build_raspberry_pi.sh to build generate aarch64 binary.

[Python 3.8 PIP package for ARM64]
The following command is used to build Python 3.8 PIP package for aarch64.
$ tensorflow/tools/ci_build/ci_build.sh PI-PYTHON38 \
    tensorflow/tools/ci_build/pi/build_raspberry_pi.sh AARCH64

[Test on Ubuntu 20.04 ARM64]
I've tested the PIP package on Ubuntu 20.04 64bit with RPI4.
There was an issue on installing h5py on Ubuntu 20.04. The following command
was needed to install h5py package.
$ HDF5_DIR=/usr/lib/aarch64-linux-gnu/hdf5/serial CC=h5cc pip3 install h5py

This PR handles #30716 issue.

PiperOrigin-RevId: 323111228
Change-Id: I408149305a082afe105215fc380f602961b2fdf0
This commit is contained in:
Terry Heo 2020-07-24 18:36:02 -07:00 committed by TensorFlower Gardener
parent a1be5430d4
commit 98b4246afa
6 changed files with 72 additions and 29 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

21
tensorflow/lite/tools/pip_package/README.md
View File

@ -65,32 +65,28 @@ tensorflow/lite/tools/pip_package/build_pip_package_with_bazel.sh
### Cross build for armhf Python 3.5
```sh
CI_DOCKER_EXTRA_PARAMS="-e CI_BUILD_PYTHON=python3 -e CROSSTOOL_PYTHON_INCLUDE_PATH=/usr/include/python3.5" \
tensorflow/tools/ci_build/ci_build.sh PI-PYTHON3 \
tensorflow/tools/ci_build/ci_build.sh PI-PYTHON3 \
tensorflow/lite/tools/pip_package/build_pip_package_with_bazel.sh armhf
```
### Cross build for armhf Python 3.7
```sh
CI_DOCKER_EXTRA_PARAMS="-e CI_BUILD_PYTHON=python3.7 -e CROSSTOOL_PYTHON_INCLUDE_PATH=/usr/include/python3.7" \
tensorflow/tools/ci_build/ci_build.sh PI-PYTHON37 \
tensorflow/tools/ci_build/ci_build.sh PI-PYTHON37 \
tensorflow/lite/tools/pip_package/build_pip_package_with_bazel.sh armhf
```
### Cross build for aarch64 Python 3.5
```sh
CI_DOCKER_EXTRA_PARAMS="-e CI_BUILD_PYTHON=python3 -e CROSSTOOL_PYTHON_INCLUDE_PATH=/usr/include/python3.5" \
tensorflow/tools/ci_build/ci_build.sh PI-PYTHON3 \
tensorflow/tools/ci_build/ci_build.sh PI-PYTHON3 \
tensorflow/lite/tools/pip_package/build_pip_package_with_bazel.sh aarch64
```
### Cross build for aarch64 Python 3.7
### Cross build for aarch64 Python 3.8
```sh
CI_DOCKER_EXTRA_PARAMS="-e CI_BUILD_PYTHON=python3.7 -e CROSSTOOL_PYTHON_INCLUDE_PATH=/usr/include/python3.7" \
tensorflow/tools/ci_build/ci_build.sh PI-PYTHON37 \
tensorflow/tools/ci_build/ci_build.sh PI-PYTHON38 \
tensorflow/lite/tools/pip_package/build_pip_package_with_bazel.sh aarch64
```
@ -109,12 +105,11 @@ CUSTOM_BAZEL_FLAGS=--define=tflite_pip_with_flex=true \
tensorflow/lite/tools/pip_package/build_pip_package_with_bazel.sh
```
### Cross build with Flex for armhf Python 3.5
### Cross build with Flex for armhf Python 3.7
```sh
CI_DOCKER_EXTRA_PARAMS="-e CUSTOM_BAZEL_FLAGS=--define=tflite_pip_with_flex=true \
-e CI_BUILD_PYTHON=python3 -e CROSSTOOL_PYTHON_INCLUDE_PATH=/usr/include/python3.5" \
tensorflow/tools/ci_build/ci_build.sh PI-PYTHON3 \
CI_DOCKER_EXTRA_PARAMS="-e CUSTOM_BAZEL_FLAGS=--define=tflite_pip_with_flex=true" \
tensorflow/tools/ci_build/ci_build.sh PI-PYTHON37 \
tensorflow/lite/tools/pip_package/build_pip_package_with_bazel.sh armhf
```

5
tensorflow/tools/ci_build/Dockerfile.pi-python3
View File

@ -1,6 +1,9 @@
FROM ubuntu:16.04
LABEL maintainer="Jan Prach <jendap@google.com>"
LABEL maintainer="Terry Heo <terryheo@google.com>"
ENV CI_BUILD_PYTHON=python3
ENV CROSSTOOL_PYTHON_INCLUDE_PATH=/usr/include/python3.5
# Copy and run the install scripts.
COPY install/*.sh /install/

10
tensorflow/tools/ci_build/Dockerfile.pi-python37
View File

@ -1,6 +1,9 @@
FROM ubuntu:16.04
LABEL maintainer="Jan Prach <jendap@google.com>"
LABEL maintainer="Terry Heo <terryheo@google.com>"
ENV CI_BUILD_PYTHON=python3.7
ENV CROSSTOOL_PYTHON_INCLUDE_PATH=/usr/include/python3.7
# Copy and run the install scripts.
COPY install/*.sh /install/
@ -10,10 +13,7 @@ RUN add-apt-repository -y ppa:openjdk-r/ppa && \
RUN /install/install_deb_packages.sh
# The following line installs the Python 3.7 cross-compilation toolchain.
RUN /install/install_pi_python37_toolchain.sh
RUN /install/install_pip_packages_by_version.sh "/usr/local/bin/pip3.7"
RUN ln -sf /usr/local/lib/python3.7/dist-packages/numpy/core/include/numpy /usr/include/python3.7/numpy
RUN /install/install_pi_python3x_toolchain.sh "3.7"
RUN /install/install_bazel.sh
RUN /install/install_proto3.sh

28
tensorflow/tools/ci_build/Dockerfile.pi-python38 Normal file
View File

@ -0,0 +1,28 @@
FROM ubuntu:16.04
LABEL maintainer="Terry Heo <terryheo@google.com>"
ENV CI_BUILD_PYTHON=python3.8
ENV CROSSTOOL_PYTHON_INCLUDE_PATH=/usr/include/python3.8
# Copy and run the install scripts.
COPY install/*.sh /install/
RUN /install/install_bootstrap_deb_packages.sh
RUN add-apt-repository -y ppa:openjdk-r/ppa && \
add-apt-repository -y ppa:george-edison55/cmake-3.x
RUN /install/install_deb_packages.sh
# The following line installs the Python 3.8 cross-compilation toolchain.
RUN /install/install_pi_python3x_toolchain.sh "3.8"
RUN /install/install_bazel.sh
RUN /install/install_proto3.sh
RUN /install/install_buildifier.sh
RUN /install/install_auditwheel.sh
RUN /install/install_golang.sh
# Set up the master bazelrc configuration file.
COPY install/.bazelrc /etc/bazel.bazelrc
# XLA is not needed for PI
ENV TF_ENABLE_XLA=0

18
tensorflow/tools/ci_build/install/install_pi_python37_toolchain.sh → tensorflow/tools/ci_build/install/install_pi_python3x_toolchain.sh
View File

@ -14,6 +14,7 @@
# limitations under the License.
# ==============================================================================
PYTHON_VERSION=$1
dpkg --add-architecture armhf
dpkg --add-architecture arm64
echo 'deb [arch=arm64,armhf] http://ports.ubuntu.com/ xenial main restricted universe multiverse' >> /etc/apt/sources.list.d/armhf.list
@ -23,8 +24,15 @@ echo 'deb [arch=arm64,armhf] http://ports.ubuntu.com/ xenial-backports main rest
sed -i 's#deb http://archive.ubuntu.com/ubuntu/#deb [arch=amd64] http://archive.ubuntu.com/ubuntu/#g' /etc/apt/sources.list
yes | add-apt-repository ppa:deadsnakes/ppa
apt-get update
apt-get install -y python3.7 python3.7-dev
#/usr/local/bin/python3.7 is needed to use /install/install_pip_packages_by_version.sh
ln -sf /usr/bin/python3.7 /usr/local/bin/python3.7
apt-get install -y libpython3.7-dev:armhf
apt-get install -y libpython3.7-dev:arm64
apt-get install -y python${PYTHON_VERSION} python${PYTHON_VERSION}-dev
#/usr/local/bin/python3.x is needed to use /install/install_pip_packages_by_version.sh
ln -sf /usr/bin/python${PYTHON_VERSION} /usr/local/bin/python${PYTHON_VERSION}
apt-get install -y libpython${PYTHON_VERSION}-dev:armhf
apt-get install -y libpython${PYTHON_VERSION}-dev:arm64
if [[ "${PYTHON_VERSION}" == "3.8" ]]; then
apt-get install -y python${PYTHON_VERSION}-distutils
fi
/install/install_pip_packages_by_version.sh "/usr/local/bin/pip${PYTHON_VERSION}"
ln -sf /usr/local/lib/python${PYTHON_VERSION}/dist-packages/numpy/core/include/numpy /usr/include/python${PYTHON_VERSION}/numpy

19
tensorflow/tools/ci_build/pi/build_raspberry_pi.sh
View File

@ -38,9 +38,11 @@ export TF_ENABLE_XLA=0
yes '' | ./configure
# Fix for curl build problem in 32-bit, see https://stackoverflow.com/questions/35181744/size-of-array-curl-rule-01-is-negative
sudo sed -i 's/define CURL_SIZEOF_LONG 8/define CURL_SIZEOF_LONG 4/g' /usr/include/curl/curlbuild.h
sudo sed -i 's/define CURL_SIZEOF_CURL_OFF_T 8/define CURL_SIZEOF_CURL_OFF_T 4/g' /usr/include/curl/curlbuild.h
if [[ $1 != "AARCH64" ]]; then
# Fix for curl build problem in 32-bit, see https://stackoverflow.com/questions/35181744/size-of-array-curl-rule-01-is-negative
sudo sed -i 's/define CURL_SIZEOF_LONG 8/define CURL_SIZEOF_LONG 4/g' /usr/include/curl/curlbuild.h
sudo sed -i 's/define CURL_SIZEOF_CURL_OFF_T 8/define CURL_SIZEOF_CURL_OFF_T 4/g' /usr/include/curl/curlbuild.h
fi
# The system-installed OpenSSL headers get pulled in by the latest BoringSSL
# release on this configuration, so move them before we build:
@ -79,6 +81,7 @@ if [[ $1 == "PI_ONE" ]]; then
make PREFIX=${OPENBLAS_INSTALL_PATH} install
PI_COPTS="--copt=-march=armv6 --copt=-mfpu=vfp
--cpu=armeabi --crosstool_top=@local_config_arm_compiler//:toolchain
--copt=-DUSE_GEMM_FOR_CONV --copt=-DUSE_OPENBLAS
--copt=-isystem --copt=${OPENBLAS_INSTALL_PATH}/include/
--copt=-std=gnu11 --copt=-DS_IREAD=S_IRUSR --copt=-DS_IWRITE=S_IWUSR
@ -87,8 +90,15 @@ if [[ $1 == "PI_ONE" ]]; then
--linkopt=-l:libopenblas.a"
echo "Building for the Pi One/Zero, with no NEON support"
WHEEL_ARCH=linux_armv6l
elif [[ $1 == "AARCH64" ]]; then
PI_COPTS="--config=elinux_aarch64
--copt=-std=gnu11
--copt=-O3"
WHEEL_ARCH=linux_aarch64
echo "Building for the aarch64"
else
PI_COPTS="--copt=-march=armv7-a --copt=-mfpu=neon-vfpv4
--cpu=armeabi --crosstool_top=@local_config_arm_compiler//:toolchain
--copt=-std=gnu11 --copt=-DS_IREAD=S_IRUSR --copt=-DS_IWRITE=S_IWUSR
--copt=-O3 --copt=-fno-tree-pre --copt=-fpermissive
--copt=-U__GCC_HAVE_SYNC_COMPARE_AND_SWAP_1
@ -107,8 +117,7 @@ cd ${WORKSPACE_PATH}
bazel build -c opt ${PI_COPTS} \
--config=monolithic \
--copt=-funsafe-math-optimizations --copt=-ftree-vectorize \
--copt=-fomit-frame-pointer --cpu=armeabi \
--crosstool_top=@local_config_arm_compiler//:toolchain \
--copt=-fomit-frame-pointer \
--define tensorflow_mkldnn_contraction_kernel=0 \
--verbose_failures \
//tensorflow:libtensorflow.so \