experiments/STT-tensorflow
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Speed up install of pip packages; solve test failures

Browse Source 
Previously, pip install of scipy and sklearn builds from source and takes a long time. This CL makes the packages install from binary whl files downloaded from PyPI.

Docker build (Dockerfile.cpu) performance improvement (no caching):
Without this CL: 24'23''
With this CL:    9'48''

The CL also solves two build failures that surfaced lately, including:
http://ci.tensorflow.org/view/Nightly/job/nigntly-matrix-linux-gpu/TF_BUILD_CONTAINER_TYPE=GPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON3,label=gpu-linux/105/console
http://ci.tensorflow.org/view/Nightly/job/nightly-matrix-cpu/TF_BUILD_CONTAINER_TYPE=CPU,TF_BUILD_IS_OPT=OPT,TF_BUILD_IS_PIP=PIP,TF_BUILD_PYTHON_VERSION=PYTHON3,label=cpu-slave/104/console
Change: 123201123
This commit is contained in:
Shanqing Cai 2016-05-25 03:33:15 -08:00 committed by TensorFlower Gardener
parent fd32666744
commit 0db67854fd
3 changed files with 55 additions and 9 deletions
tensorflow/tools/ci_build
builds
pip.sh
install
install_deb_packages.shinstall_pip_packages.sh

2
tensorflow/tools/ci_build/builds/pip.sh
View File

@ -187,7 +187,7 @@ source "${VENV_DIR}/bin/activate" || \
# Force tensorflow reinstallation. Otherwise it may not get installed from
# last build if it had the same version number as previous build.
PIP_FLAGS="--upgrade --no-deps --force-reinstall"
PIP_FLAGS="--upgrade --force-reinstall"
pip install -v ${PIP_FLAGS} ${WHL_PATH} || \
die "pip install (forcing to reinstall tensorflow) FAILED"
echo "Successfully installed pip package ${WHL_PATH}"

2
tensorflow/tools/ci_build/install/install_deb_packages.sh
View File

@ -39,12 +39,10 @@ apt-get install -y \
openjdk-8-jre-headless \
pkg-config \
python-dev \
python-numpy \
python-pandas \
python-pip \
python-virtualenv \
python3-dev \
python3-numpy \
python3-pandas \
python3-pip \
sudo \

60
tensorflow/tools/ci_build/install/install_pip_packages.sh
View File

@ -16,13 +16,52 @@
set -e
# Use pip to install scipy to get the latest version, instead of 0.13 through
# apt-get
pip install scipy==0.15.1
pip3 install scipy==0.15.1
# Install pip packages from whl files to avoid the time-consuming process of
# building from source.
pip install sklearn
pip3 install scikit-learn
# Use pip to install numpy to the latest version, instead of 1.8.2 through
# apt-get
wget -q https://pypi.python.org/packages/06/92/3c786303889e6246971ad4c48ac2b4e37a1b1c67c0dc2106dc85cb15c18e/numpy-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl#md5=6ffb66ff78c28c55bfa09a2ceee487df
mv numpy-1.11.0-cp27-cp27mu-manylinux1_x86_64.whl \
numpy-1.11.0-cp27-none-linux_x86_64.whl
pip install numpy-1.11.0-cp27-none-linux_x86_64.whl
rm numpy-1.11.0-cp27-none-linux_x86_64.whl
wget -q https://pypi.python.org/packages/ea/ca/5e48a68be496e6f79c3c8d90f7c03ea09bbb154ea4511f5b3d6c825cefe5/numpy-1.11.0-cp34-cp34m-manylinux1_x86_64.whl#md5=08a002aeffa20354aa5045eadb549361
mv numpy-1.11.0-cp34-cp34m-manylinux1_x86_64.whl \
numpy-1.11.0-cp34-cp34m-linux_x86_64.whl
pip3 install numpy-1.11.0-cp34-cp34m-linux_x86_64.whl
rm numpy-1.11.0-cp34-cp34m-linux_x86_64.whl
# Use pip to install scipy to get the latest version, instead of 0.13 through
# apt-get.
# pip install scipy==0.17.1
wget -q https://pypi.python.org/packages/8a/de/326cf31a5a3ba0c01c40cdd78f7140b0510ed80e6d5ec5b2ec173c72df03/scipy-0.17.1-cp27-cp27mu-manylinux1_x86_64.whl#md5=8d0df61ceba78a2796f8d90fc979576f
mv scipy-0.17.1-cp27-cp27mu-manylinux1_x86_64.whl \
scipy-0.17.1-cp27-none-linux_x86_64.whl
pip install scipy-0.17.1-cp27-none-linux_x86_64.whl
rm scipy-0.17.1-cp27-none-linux_x86_64.whl
# pip3 install scipy==0.17.1
wget -q https://pypi.python.org/packages/eb/2e/76aff3b25dd06cab06622f82a4790ff5002ab686e940847bb2503b4b2122/scipy-0.17.1-cp34-cp34m-manylinux1_x86_64.whl#md5=bb39b9e1d16fa220967ad7edd39a8b28
mv scipy-0.17.1-cp34-cp34m-manylinux1_x86_64.whl \
scipy-0.17.1-cp34-cp34m-linux_x86_64.whl
pip3 install scipy-0.17.1-cp34-cp34m-linux_x86_64.whl
rm scipy-0.17.1-cp34-cp34m-linux_x86_64.whl
# pip install sklearn
wget -q https://pypi.python.org/packages/bf/80/06e77e5a682c46a3880ec487a5f9d910f5c8d919df9aca58052089687c7e/scikit_learn-0.17.1-cp27-cp27mu-manylinux1_x86_64.whl#md5=337b91f502138ba7fd722803138f6dfd
mv scikit_learn-0.17.1-cp27-cp27mu-manylinux1_x86_64.whl \
scikit_learn-0.17.1-cp27-none-linux_x86_64.whl
pip install scikit_learn-0.17.1-cp27-none-linux_x86_64.whl
rm scikit_learn-0.17.1-cp27-none-linux_x86_64.whl
# pip3 install scikit-learn
wget -q https://pypi.python.org/packages/7e/f1/1cc8a1ae2b4de89bff0981aee904ff05779c49a4c660fa38178f9772d3a7/scikit_learn-0.17.1-cp34-cp34m-manylinux1_x86_64.whl#md5=a722a7372b64ec9f7b49a2532d21372b
mv scikit_learn-0.17.1-cp34-cp34m-manylinux1_x86_64.whl \
scikit_learn-0.17.1-cp34-cp34m-linux_x86_64.whl
pip3 install scikit_learn-0.17.1-cp34-cp34m-linux_x86_64.whl
rm scikit_learn-0.17.1-cp34-cp34m-linux_x86_64.whl
# Benchmark tests require the following:
pip install psutil
@ -33,3 +72,12 @@ pip3 install py-cpuinfo
# pylint tests require the following:
pip install pylint
pip3 install pylint
# Remove packages in /usr/lib/python* that may interfere with packages in
# /usr/local/lib. These packages may get installed inadvertantly with packages
# such as apt-get python-pandas. Their older versions can mask the more recent
# versions installed above with pip and cause test failures.
rm -rf /usr/lib/python2.7/dist-packages/numpy \
/usr/lib/python2.7/dist-packages/scipy \
/usr/lib/python3/dist-packages/numpy \
/usr/lib/python3/dist-packages/scipy