experiments/STT-tensorflow
1
0
Fork 0
Code Issues Pull Requests Projects Releases Wiki Activity

Upgrade Dockerfile assembler system

Browse Source 
This is a big upgrade to the Dockerfile assembler I wrote a couple of
months ago. The spec has changed, the script has been rewritten, and
there are new features throughout:

- The assembler can build and upload images to Docker Hub.
- The assembler can also run tests (!), although the testing system is
  extremely rudimentary. It could be expanded with parallelism later, if
  execution time becomes a problem.
- spec.yml is totally different, and now defines both dockerfiles and
  images. It handles the combinatorial explosion of multiple optional features
  without excessive duplication, unlike the previous spec format.
- Partials are the same, but I dumped the extensive dockerfile
  documentation support because I don't think anyone would have used it.
- Dockerfiles are handled under the same kind of system as images, which
  is neat. The new Dockerfiles aren't so duplicated.
- I've upgraded the images with new tensorflow tutorial files (jupyter
  only) and fixed some others that didn't actually work.
- I've improved the development documentation by suggesting aliases.
- Added "static-dockerfiles" directory to track independent Dockerfiles.

These changes should better support changes like .
This commit is contained in:
Austin Anderson 2018-08-24 17:01:55 -07:00
parent 94008f8c57
commit 675f415603
29 changed files with 1031 additions and 833 deletions

1
tensorflow/tools/dockerfiles/.gitignore vendored Normal file
View File

@ -0,0 +1 @@
dockerfiles/*.temp.Dockerfile

49
tensorflow/tools/dockerfiles/README.md
View File

@ -1,8 +1,12 @@
# TensorFlow Dockerfiles
This directory houses TensorFlow's Dockerfiles. **DO NOT EDIT THE DOCKERFILES
MANUALLY!** They are maintained by `assembler.py`, which builds Dockerfiles from
the files in `partials/` and the rules in `spec.yml`. See [the Contributing
This directory houses TensorFlow's Dockerfiles and the infrastructure used to
create and deploy them to [Docker
Hub](https://hub.docker.com/r/tensorflow/tensorflow).
**DO NOT EDIT THE DOCKERFILES/ DIRECTORY MANUALLY!** The files within are
maintained by `assembler.py`, which builds Dockerfiles from the files in
`partials/` and the rules in `spec.yml`. See [the Contributing
section](#contributing) for more information.
These Dockerfiles are planned to replace the Dockerfiles used to generate
@ -20,10 +24,10 @@ $ docker build -f ./dockerfiles/cpu.Dockerfile -t tf .
Each Dockerfile has its own set of available `--build-arg`s which are documented
in the Dockerfile itself.
## Running
## Running Locally Built Images
After building the image with the tag `tf` (for example), use `docker run` to
run the images. Examples are below.
run the images.
Note for new Docker users: the `-v` and `-u` flags share directories between
the Docker container and your machine, and very important. Without
@ -42,8 +46,10 @@ $ docker run -u $(id -u):$(id -g) -v $(pwd):/my-devel -it tf
# GPU-based images (set up nvidia-docker2 first)
$ docker run --runtime=nvidia -u $(id -u):$(id -g) -v $(pwd):/my-devel -it tf
# Images with Jupyter run on port 8888, and needs a volume for notebooks
$ docker run --user $(id -u):$(id -g) -p 8888:8888 -v $(pwd):/notebooks -it tf
# Images with Jupyter run on port 8888 and need a volume for your notebooks
# You can change $(PWD) to the full path to a directory if your notebooks
# live outside the current directory.
$ docker run --user $(id -u):$(id -g) -p 8888:8888 -v $(PWD):/tf/notebooks -it tf
```
These images do not come with the TensorFlow source code -- but the development
@ -60,11 +66,32 @@ You can use the `Dockerfile` in this directory to build an editing environment
that has all of the Python dependencies you'll need:
```bash
$ docker build -t tf-assembler -f assembler.Dockerfile .
# Build the tools-helper image so you can run the assembler
$ docker build -t tf-tools -f tools.Dockerfile .
# Set --user to set correct permissions on generated files
$ docker run --user $(id -u):$(id -g) -it -v $(pwd):/tf tf-assembler bash
$ docker run --user $(id -u):$(id -g) -it -v $(pwd):/tf tf-tools bash
# In the container...
/tf $ python3 ./assembler.py -o dockerfiles -s spec.yml
# Next you can make a handy alias depending on what you're doing. When building
# Docker images, you need to run as root with docker.sock mounted so that the
# container can run Docker commands. When assembling Dockerfiles, though, you'll
# want to run as your user so that new files have the right permissions.
# If you're BUILDING OR DEPLOYING DOCKER IMAGES, run as root with docker.sock:
$ alias asm_images="docker run --rm -v $(pwd):/tf -v /var/run/docker.sock:/var/run/docker.sock tf-tools python3 assembler.py "
# If you're REBUILDING OR ADDING DOCKERFILES, remove docker.sock and add -u:
$ alias asm_dockerfiles="docker run --rm -u $(id -u):$(id -g) -v $(pwd):/tf tf-tools python3 assembler.py "
# Check flags
$ asm_dockerfiles --help
# Assemble all of the Dockerfiles
$ asm_dockerfiles --release ubuntu-dockerfiles --construct_dockerfiles
# Build all of the "nightly" images on your local machine:
$ asm_images --release nightly --build_images
# Build version release for version 99.0, except "gpu" tags:
$ asm_images --release versioned --arg _TAG_PREFIX=99.0 --build_images --exclude_tags_matching '*.gpu.*'
```

944
tensorflow/tools/dockerfiles/assembler.py
View File

File diff suppressed because it is too large Load Diff

63
tensorflow/tools/dockerfiles/dockerfiles/cpu-devel-jupyter.Dockerfile
View File

@ -16,27 +16,12 @@
# THIS IS A GENERATED DOCKERFILE.
#
# This file was assembled from multiple pieces, whose use is documented
# below. Please refer to the the TensorFlow dockerfiles documentation for
# more information. Build args are documented as their default value.
#
# Ubuntu-based, CPU-only environment for developing changes for TensorFlow, with Jupyter included.
#
# Start from Ubuntu, with TF development packages (no GPU support)
# --build-arg UBUNTU_VERSION=16.04
# ( no description )
#
# Python is required for TensorFlow and other libraries.
# --build-arg USE_PYTHON_3_NOT_2=True
# Install python 3 over Python 2
#
# Install the latest version of Bazel and Python development tools.
#
# Configure TensorFlow's shell prompt and login tools.
#
# Launch Jupyter on execution instead of a bash prompt.
# throughout. Please refer to the the TensorFlow dockerfiles documentation
# for more information.
ARG UBUNTU_VERSION=16.04
FROM ubuntu:${UBUNTU_VERSION}
FROM ubuntu:${UBUNTU_VERSION} AS base
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
@ -48,7 +33,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
libpng12-dev \
libzmq3-dev \
pkg-config \
python-dev \
rsync \
software-properties-common \
unzip \
@ -59,8 +43,11 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
&& \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
ENV CI_BUILD_PYTHON python
ARG USE_PYTHON_3_NOT_2=True
ARG USE_PYTHON_3_NOT_2
ARG _PY_SUFFIX=${USE_PYTHON_3_NOT_2:+3}
ARG PYTHON=python${_PY_SUFFIX}
ARG PIP=pip${_PY_SUFFIX}
@ -72,10 +59,13 @@ RUN apt-get update && apt-get install -y \
${PYTHON} \
${PYTHON}-pip
RUN ${PIP} install --upgrade \
RUN ${PIP} --no-cache-dir install --upgrade \
pip \
setuptools
# Some TF tools expect a "python" binary
RUN ln -s $(which ${PYTHON}) /usr/local/bin/python
RUN apt-get update && apt-get install -y \
build-essential \
curl \
@ -84,6 +74,20 @@ RUN apt-get update && apt-get install -y \
${PYTHON}-dev \
swig
RUN ${PIP} --no-cache-dir install \
Pillow \
h5py \
keras_applications \
keras_preprocessing \
matplotlib \
mock \
numpy \
scipy \
sklearn \
pandas \
&& test "${USE_PYTHON_3_NOT_2}" -eq 1 && true || ${PIP} --no-cache-dir install \
enum34
# Install bazel
RUN echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | tee /etc/apt/sources.list.d/bazel.list && \
curl https://bazel.build/bazel-release.pub.gpg | apt-key add - && \
@ -93,11 +97,18 @@ RUN echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8
COPY bashrc /etc/bash.bashrc
RUN chmod a+rwx /etc/bash.bashrc
RUN ${PIP} install jupyter
RUN ${PIP} install jupyter matplotlib
RUN mkdir /notebooks && chmod a+rwx /notebooks
RUN mkdir -p /tf/tensorflow-tutorials && chmod -R a+rwx /tf/
RUN mkdir /.local && chmod a+rwx /.local
WORKDIR /notebooks
RUN apt-get install -y --no-install-recommends wget
WORKDIR /tf/tensorflow-tutorials
RUN wget https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/keras/basic_classification.ipynb
RUN wget https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/keras/basic_text_classification.ipynb
RUN apt-get autoremove -y && apt-get remove -y wget
WORKDIR /tf
EXPOSE 8888
CMD ["bash", "-c", "source /etc/bash.bashrc && jupyter notebook --notebook-dir=/notebooks --ip 0.0.0.0 --no-browser --allow-root"]
RUN ${PYTHON} -m ipykernel.kernelspec
CMD ["bash", "-c", "source /etc/bash.bashrc && jupyter notebook --notebook-dir=/tf --ip 0.0.0.0 --no-browser --allow-root"]

46
tensorflow/tools/dockerfiles/dockerfiles/cpu-devel.Dockerfile
View File

@ -16,25 +16,12 @@
# THIS IS A GENERATED DOCKERFILE.
#
# This file was assembled from multiple pieces, whose use is documented
# below. Please refer to the the TensorFlow dockerfiles documentation for
# more information. Build args are documented as their default value.
#
# Ubuntu-based, CPU-only environment for developing changes for TensorFlow.
#
# Start from Ubuntu, with TF development packages (no GPU support)
# --build-arg UBUNTU_VERSION=16.04
# ( no description )
#
# Python is required for TensorFlow and other libraries.
# --build-arg USE_PYTHON_3_NOT_2=True
# Install python 3 over Python 2
#
# Install the latest version of Bazel and Python development tools.
#
# Configure TensorFlow's shell prompt and login tools.
# throughout. Please refer to the the TensorFlow dockerfiles documentation
# for more information.
ARG UBUNTU_VERSION=16.04
FROM ubuntu:${UBUNTU_VERSION}
FROM ubuntu:${UBUNTU_VERSION} AS base
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
@ -46,7 +33,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
libpng12-dev \
libzmq3-dev \
pkg-config \
python-dev \
rsync \
software-properties-common \
unzip \
@ -57,8 +43,11 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
&& \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
ENV CI_BUILD_PYTHON python
ARG USE_PYTHON_3_NOT_2=True
ARG USE_PYTHON_3_NOT_2
ARG _PY_SUFFIX=${USE_PYTHON_3_NOT_2:+3}
ARG PYTHON=python${_PY_SUFFIX}
ARG PIP=pip${_PY_SUFFIX}
@ -70,10 +59,13 @@ RUN apt-get update && apt-get install -y \
${PYTHON} \
${PYTHON}-pip
RUN ${PIP} install --upgrade \
RUN ${PIP} --no-cache-dir install --upgrade \
pip \
setuptools
# Some TF tools expect a "python" binary
RUN ln -s $(which ${PYTHON}) /usr/local/bin/python
RUN apt-get update && apt-get install -y \
build-essential \
curl \
@ -82,6 +74,20 @@ RUN apt-get update && apt-get install -y \
${PYTHON}-dev \
swig
RUN ${PIP} --no-cache-dir install \
Pillow \
h5py \
keras_applications \
keras_preprocessing \
matplotlib \
mock \
numpy \
scipy \
sklearn \
pandas \
&& test "${USE_PYTHON_3_NOT_2}" -eq 1 && true || ${PIP} --no-cache-dir install \
enum34
# Install bazel
RUN echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | tee /etc/apt/sources.list.d/bazel.list && \
curl https://bazel.build/bazel-release.pub.gpg | apt-key add - && \

52
tensorflow/tools/dockerfiles/dockerfiles/cpu-jupyter.Dockerfile
View File

@ -16,31 +16,14 @@
# THIS IS A GENERATED DOCKERFILE.
#
# This file was assembled from multiple pieces, whose use is documented
# below. Please refer to the the TensorFlow dockerfiles documentation for
# more information. Build args are documented as their default value.
#
# Ubuntu-based, CPU-only environment for using TensorFlow, with Jupyter included.
#
# Start from Ubuntu (no GPU support)
# --build-arg UBUNTU_VERSION=16.04
# ( no description )
#
# Python is required for TensorFlow and other libraries.
# --build-arg USE_PYTHON_3_NOT_2=True
# Install python 3 over Python 2
#
# Install the TensorFlow Python package.
# --build-arg TF_PACKAGE=tensorflow (tensorflow|tensorflow-gpu|tf-nightly|tf-nightly-gpu)
# The specific TensorFlow Python package to install
#
# Configure TensorFlow's shell prompt and login tools.
#
# Launch Jupyter on execution instead of a bash prompt.
# throughout. Please refer to the the TensorFlow dockerfiles documentation
# for more information.
ARG UBUNTU_VERSION=16.04
FROM ubuntu:${UBUNTU_VERSION}
ARG USE_PYTHON_3_NOT_2=True
FROM ubuntu:${UBUNTU_VERSION} as base
ARG USE_PYTHON_3_NOT_2
ARG _PY_SUFFIX=${USE_PYTHON_3_NOT_2:+3}
ARG PYTHON=python${_PY_SUFFIX}
ARG PIP=pip${_PY_SUFFIX}
@ -52,21 +35,36 @@ RUN apt-get update && apt-get install -y \
${PYTHON} \
${PYTHON}-pip
RUN ${PIP} install --upgrade \
RUN ${PIP} --no-cache-dir install --upgrade \
pip \
setuptools
# Some TF tools expect a "python" binary
RUN ln -s $(which ${PYTHON}) /usr/local/bin/python
# Options:
# tensorflow
# tensorflow-gpu
# tf-nightly
# tf-nightly-gpu
ARG TF_PACKAGE=tensorflow
RUN ${PIP} install ${TF_PACKAGE}
COPY bashrc /etc/bash.bashrc
RUN chmod a+rwx /etc/bash.bashrc
RUN ${PIP} install jupyter
RUN ${PIP} install jupyter matplotlib
RUN mkdir /notebooks && chmod a+rwx /notebooks
RUN mkdir -p /tf/tensorflow-tutorials && chmod -R a+rwx /tf/
RUN mkdir /.local && chmod a+rwx /.local
WORKDIR /notebooks
RUN apt-get install -y --no-install-recommends wget
WORKDIR /tf/tensorflow-tutorials
RUN wget https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/keras/basic_classification.ipynb
RUN wget https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/keras/basic_text_classification.ipynb
RUN apt-get autoremove -y && apt-get remove -y wget
WORKDIR /tf
EXPOSE 8888
CMD ["bash", "-c", "source /etc/bash.bashrc && jupyter notebook --notebook-dir=/notebooks --ip 0.0.0.0 --no-browser --allow-root"]
RUN ${PYTHON} -m ipykernel.kernelspec
CMD ["bash", "-c", "source /etc/bash.bashrc && jupyter notebook --notebook-dir=/tf --ip 0.0.0.0 --no-browser --allow-root"]

35
tensorflow/tools/dockerfiles/dockerfiles/cpu.Dockerfile
View File

@ -16,29 +16,14 @@
# THIS IS A GENERATED DOCKERFILE.
#
# This file was assembled from multiple pieces, whose use is documented
# below. Please refer to the the TensorFlow dockerfiles documentation for
# more information. Build args are documented as their default value.
#
# Ubuntu-based, CPU-only environment for using TensorFlow
#
# Start from Ubuntu (no GPU support)
# --build-arg UBUNTU_VERSION=16.04
# ( no description )
#
# Python is required for TensorFlow and other libraries.
# --build-arg USE_PYTHON_3_NOT_2=True
# Install python 3 over Python 2
#
# Install the TensorFlow Python package.
# --build-arg TF_PACKAGE=tensorflow (tensorflow|tensorflow-gpu|tf-nightly|tf-nightly-gpu)
# The specific TensorFlow Python package to install
#
# Configure TensorFlow's shell prompt and login tools.
# throughout. Please refer to the the TensorFlow dockerfiles documentation
# for more information.
ARG UBUNTU_VERSION=16.04
FROM ubuntu:${UBUNTU_VERSION}
ARG USE_PYTHON_3_NOT_2=True
FROM ubuntu:${UBUNTU_VERSION} as base
ARG USE_PYTHON_3_NOT_2
ARG _PY_SUFFIX=${USE_PYTHON_3_NOT_2:+3}
ARG PYTHON=python${_PY_SUFFIX}
ARG PIP=pip${_PY_SUFFIX}
@ -50,10 +35,18 @@ RUN apt-get update && apt-get install -y \
${PYTHON} \
${PYTHON}-pip
RUN ${PIP} install --upgrade \
RUN ${PIP} --no-cache-dir install --upgrade \
pip \
setuptools
# Some TF tools expect a "python" binary
RUN ln -s $(which ${PYTHON}) /usr/local/bin/python
# Options:
# tensorflow
# tensorflow-gpu
# tf-nightly
# tf-nightly-gpu
ARG TF_PACKAGE=tensorflow
RUN ${PIP} install ${TF_PACKAGE}

75
tensorflow/tools/dockerfiles/dockerfiles/nvidia-devel-jupyter.Dockerfile → tensorflow/tools/dockerfiles/dockerfiles/gpu-devel-jupyter.Dockerfile
View File

@ -16,28 +16,12 @@
# THIS IS A GENERATED DOCKERFILE.
#
# This file was assembled from multiple pieces, whose use is documented
# below. Please refer to the the TensorFlow dockerfiles documentation for
# more information. Build args are documented as their default value.
#
# Ubuntu-based, Nvidia-GPU-enabled environment for developing changes for TensorFlow, with Jupyter included.
#
# Start from Nvidia's Ubuntu base image with CUDA and CuDNN, with TF development
# packages.
# --build-arg UBUNTU_VERSION=16.04
# ( no description )
#
# Python is required for TensorFlow and other libraries.
# --build-arg USE_PYTHON_3_NOT_2=True
# Install python 3 over Python 2
#
# Install the latest version of Bazel and Python development tools.
#
# Configure TensorFlow's shell prompt and login tools.
#
# Launch Jupyter on execution instead of a bash prompt.
# throughout. Please refer to the the TensorFlow dockerfiles documentation
# for more information.
ARG UBUNTU_VERSION=16.04
FROM nvidia/cuda:9.0-base-ubuntu${UBUNTU_VERSION}
FROM nvidia/cuda:9.0-base-ubuntu${UBUNTU_VERSION} as base
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
@ -60,6 +44,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
libpng12-dev \
libzmq3-dev \
pkg-config \
python-dev \
rsync \
software-properties-common \
unzip \
@ -82,11 +67,19 @@ RUN mkdir /usr/local/cuda-9.0/lib && \
ln -s /usr/lib/x86_64-linux-gnu/libnccl.so.2 /usr/local/cuda/lib/libnccl.so.2 && \
ln -s /usr/include/nccl.h /usr/local/cuda/include/nccl.h
# TODO(tobyboyd): Remove after license is excluded from BUILD file.
RUN gunzip /usr/share/doc/libnccl2/NCCL-SLA.txt.gz && \
cp /usr/share/doc/libnccl2/NCCL-SLA.txt /usr/local/cuda/
# Configure the build for our CUDA configuration.
ENV CI_BUILD_PYTHON python
ENV LD_LIBRARY_PATH /usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH
ENV TF_NEED_CUDA 1
ENV TF_NEED_TENSORRT 1
ENV TF_CUDA_COMPUTE_CAPABILITIES=3.5,5.2,6.0,6.1,7.0
ENV TF_CUDA_VERSION=9.0
ENV TF_CUDNN_VERSION=7
ARG USE_PYTHON_3_NOT_2=True
# NCCL 2.x
ENV TF_NCCL_VERSION=2
ARG USE_PYTHON_3_NOT_2
ARG _PY_SUFFIX=${USE_PYTHON_3_NOT_2:+3}
ARG PYTHON=python${_PY_SUFFIX}
ARG PIP=pip${_PY_SUFFIX}
@ -98,10 +91,13 @@ RUN apt-get update && apt-get install -y \
${PYTHON} \
${PYTHON}-pip
RUN ${PIP} install --upgrade \
RUN ${PIP} --no-cache-dir install --upgrade \
pip \
setuptools
# Some TF tools expect a "python" binary
RUN ln -s $(which ${PYTHON}) /usr/local/bin/python
RUN apt-get update && apt-get install -y \
build-essential \
curl \
@ -110,6 +106,20 @@ RUN apt-get update && apt-get install -y \
${PYTHON}-dev \
swig
RUN ${PIP} --no-cache-dir install \
Pillow \
h5py \
keras_applications \
keras_preprocessing \
matplotlib \
mock \
numpy \
scipy \
sklearn \
pandas \
&& test "${USE_PYTHON_3_NOT_2}" -eq 1 && true || ${PIP} --no-cache-dir install \
enum34
# Install bazel
RUN echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | tee /etc/apt/sources.list.d/bazel.list && \
curl https://bazel.build/bazel-release.pub.gpg | apt-key add - && \
@ -119,11 +129,18 @@ RUN echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8
COPY bashrc /etc/bash.bashrc
RUN chmod a+rwx /etc/bash.bashrc
RUN ${PIP} install jupyter
RUN ${PIP} install jupyter matplotlib
RUN mkdir /notebooks && chmod a+rwx /notebooks
RUN mkdir -p /tf/tensorflow-tutorials && chmod -R a+rwx /tf/
RUN mkdir /.local && chmod a+rwx /.local
WORKDIR /notebooks
RUN apt-get install -y --no-install-recommends wget
WORKDIR /tf/tensorflow-tutorials
RUN wget https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/keras/basic_classification.ipynb
RUN wget https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/keras/basic_text_classification.ipynb
RUN apt-get autoremove -y && apt-get remove -y wget
WORKDIR /tf
EXPOSE 8888
CMD ["bash", "-c", "source /etc/bash.bashrc && jupyter notebook --notebook-dir=/notebooks --ip 0.0.0.0 --no-browser --allow-root"]
RUN ${PYTHON} -m ipykernel.kernelspec
CMD ["bash", "-c", "source /etc/bash.bashrc && jupyter notebook --notebook-dir=/tf --ip 0.0.0.0 --no-browser --allow-root"]

58
tensorflow/tools/dockerfiles/dockerfiles/nvidia-devel.Dockerfile → tensorflow/tools/dockerfiles/dockerfiles/gpu-devel.Dockerfile
View File

@ -16,26 +16,12 @@
# THIS IS A GENERATED DOCKERFILE.
#
# This file was assembled from multiple pieces, whose use is documented
# below. Please refer to the the TensorFlow dockerfiles documentation for
# more information. Build args are documented as their default value.
#
# Ubuntu-based, Nvidia-GPU-enabled environment for developing changes for TensorFlow.
#
# Start from Nvidia's Ubuntu base image with CUDA and CuDNN, with TF development
# packages.
# --build-arg UBUNTU_VERSION=16.04
# ( no description )
#
# Python is required for TensorFlow and other libraries.
# --build-arg USE_PYTHON_3_NOT_2=True
# Install python 3 over Python 2
#
# Install the latest version of Bazel and Python development tools.
#
# Configure TensorFlow's shell prompt and login tools.
# throughout. Please refer to the the TensorFlow dockerfiles documentation
# for more information.
ARG UBUNTU_VERSION=16.04
FROM nvidia/cuda:9.0-base-ubuntu${UBUNTU_VERSION}
FROM nvidia/cuda:9.0-base-ubuntu${UBUNTU_VERSION} as base
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
@ -58,6 +44,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
libpng12-dev \
libzmq3-dev \
pkg-config \
python-dev \
rsync \
software-properties-common \
unzip \
@ -80,11 +67,19 @@ RUN mkdir /usr/local/cuda-9.0/lib && \
ln -s /usr/lib/x86_64-linux-gnu/libnccl.so.2 /usr/local/cuda/lib/libnccl.so.2 && \
ln -s /usr/include/nccl.h /usr/local/cuda/include/nccl.h
# TODO(tobyboyd): Remove after license is excluded from BUILD file.
RUN gunzip /usr/share/doc/libnccl2/NCCL-SLA.txt.gz && \
cp /usr/share/doc/libnccl2/NCCL-SLA.txt /usr/local/cuda/
# Configure the build for our CUDA configuration.
ENV CI_BUILD_PYTHON python
ENV LD_LIBRARY_PATH /usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH
ENV TF_NEED_CUDA 1
ENV TF_NEED_TENSORRT 1
ENV TF_CUDA_COMPUTE_CAPABILITIES=3.5,5.2,6.0,6.1,7.0
ENV TF_CUDA_VERSION=9.0
ENV TF_CUDNN_VERSION=7
ARG USE_PYTHON_3_NOT_2=True
# NCCL 2.x
ENV TF_NCCL_VERSION=2
ARG USE_PYTHON_3_NOT_2
ARG _PY_SUFFIX=${USE_PYTHON_3_NOT_2:+3}
ARG PYTHON=python${_PY_SUFFIX}
ARG PIP=pip${_PY_SUFFIX}
@ -96,10 +91,13 @@ RUN apt-get update && apt-get install -y \
${PYTHON} \
${PYTHON}-pip
RUN ${PIP} install --upgrade \
RUN ${PIP} --no-cache-dir install --upgrade \
pip \
setuptools
# Some TF tools expect a "python" binary
RUN ln -s $(which ${PYTHON}) /usr/local/bin/python
RUN apt-get update && apt-get install -y \
build-essential \
curl \
@ -108,6 +106,20 @@ RUN apt-get update && apt-get install -y \
${PYTHON}-dev \
swig
RUN ${PIP} --no-cache-dir install \
Pillow \
h5py \
keras_applications \
keras_preprocessing \
matplotlib \
mock \
numpy \
scipy \
sklearn \
pandas \
&& test "${USE_PYTHON_3_NOT_2}" -eq 1 && true || ${PIP} --no-cache-dir install \
enum34
# Install bazel
RUN echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | tee /etc/apt/sources.list.d/bazel.list && \
curl https://bazel.build/bazel-release.pub.gpg | apt-key add - && \

61
tensorflow/tools/dockerfiles/dockerfiles/nvidia-jupyter.Dockerfile → tensorflow/tools/dockerfiles/dockerfiles/gpu-jupyter.Dockerfile
View File

@ -16,30 +16,13 @@
# THIS IS A GENERATED DOCKERFILE.
#
# This file was assembled from multiple pieces, whose use is documented
# below. Please refer to the the TensorFlow dockerfiles documentation for
# more information. Build args are documented as their default value.
#
# Ubuntu-based, Nvidia-GPU-enabled environment for using TensorFlow, with Jupyter included.
#
# NVIDIA with CUDA and CuDNN, no dev stuff
# --build-arg UBUNTU_VERSION=16.04
# ( no description )
#
# Python is required for TensorFlow and other libraries.
# --build-arg USE_PYTHON_3_NOT_2=True
# Install python 3 over Python 2
#
# Install the TensorFlow Python package.
# --build-arg TF_PACKAGE=tensorflow-gpu (tensorflow|tensorflow-gpu|tf-nightly|tf-nightly-gpu)
# The specific TensorFlow Python package to install
#
# Configure TensorFlow's shell prompt and login tools.
#
# Launch Jupyter on execution instead of a bash prompt.
# throughout. Please refer to the the TensorFlow dockerfiles documentation
# for more information.
FROM nvidia/cuda:9.0-base-ubuntu16.04
ARG UBUNTU_VERSION=16.04
FROM nvidia/cuda:9.0-base-ubuntu${UBUNTU_VERSION} as base
# Pick up some TF dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
cuda-command-line-tools-9-0 \
@ -48,6 +31,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
cuda-curand-9-0 \
cuda-cusolver-9-0 \
cuda-cusparse-9-0 \
curl \
libcudnn7=7.2.1.38-1+cuda9.0 \
libnccl2=2.2.13-1+cuda9.0 \
libfreetype6-dev \
@ -55,6 +39,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
libpng12-dev \
libzmq3-dev \
pkg-config \
rsync \
software-properties-common \
unzip \
&& \
@ -66,7 +51,10 @@ RUN apt-get update && \
apt-get update && \
apt-get install libnvinfer4=4.1.2-1+cuda9.0
ARG USE_PYTHON_3_NOT_2=True
# For CUDA profiling, TensorFlow requires CUPTI.
ENV LD_LIBRARY_PATH /usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH
ARG USE_PYTHON_3_NOT_2
ARG _PY_SUFFIX=${USE_PYTHON_3_NOT_2:+3}
ARG PYTHON=python${_PY_SUFFIX}
ARG PIP=pip${_PY_SUFFIX}
@ -78,21 +66,36 @@ RUN apt-get update && apt-get install -y \
${PYTHON} \
${PYTHON}-pip
RUN ${PIP} install --upgrade \
RUN ${PIP} --no-cache-dir install --upgrade \
pip \
setuptools
ARG TF_PACKAGE=tensorflow-gpu
# Some TF tools expect a "python" binary
RUN ln -s $(which ${PYTHON}) /usr/local/bin/python
# Options:
# tensorflow
# tensorflow-gpu
# tf-nightly
# tf-nightly-gpu
ARG TF_PACKAGE=tensorflow
RUN ${PIP} install ${TF_PACKAGE}
COPY bashrc /etc/bash.bashrc
RUN chmod a+rwx /etc/bash.bashrc
RUN ${PIP} install jupyter
RUN ${PIP} install jupyter matplotlib
RUN mkdir /notebooks && chmod a+rwx /notebooks
RUN mkdir -p /tf/tensorflow-tutorials && chmod -R a+rwx /tf/
RUN mkdir /.local && chmod a+rwx /.local
WORKDIR /notebooks
RUN apt-get install -y --no-install-recommends wget
WORKDIR /tf/tensorflow-tutorials
RUN wget https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/keras/basic_classification.ipynb
RUN wget https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/keras/basic_text_classification.ipynb
RUN apt-get autoremove -y && apt-get remove -y wget
WORKDIR /tf
EXPOSE 8888
CMD ["bash", "-c", "source /etc/bash.bashrc && jupyter notebook --notebook-dir=/notebooks --ip 0.0.0.0 --no-browser --allow-root"]
RUN ${PYTHON} -m ipykernel.kernelspec
CMD ["bash", "-c", "source /etc/bash.bashrc && jupyter notebook --notebook-dir=/tf --ip 0.0.0.0 --no-browser --allow-root"]

44
tensorflow/tools/dockerfiles/dockerfiles/nvidia.Dockerfile → tensorflow/tools/dockerfiles/dockerfiles/gpu.Dockerfile
View File

@ -16,28 +16,13 @@
# THIS IS A GENERATED DOCKERFILE.
#
# This file was assembled from multiple pieces, whose use is documented
# below. Please refer to the the TensorFlow dockerfiles documentation for
# more information. Build args are documented as their default value.
#
# Ubuntu-based, Nvidia-GPU-enabled environment for using TensorFlow.
#
# NVIDIA with CUDA and CuDNN, no dev stuff
# --build-arg UBUNTU_VERSION=16.04
# ( no description )
#
# Python is required for TensorFlow and other libraries.
# --build-arg USE_PYTHON_3_NOT_2=True
# Install python 3 over Python 2
#
# Install the TensorFlow Python package.
# --build-arg TF_PACKAGE=tensorflow-gpu (tensorflow|tensorflow-gpu|tf-nightly|tf-nightly-gpu)
# The specific TensorFlow Python package to install
#
# Configure TensorFlow's shell prompt and login tools.
# throughout. Please refer to the the TensorFlow dockerfiles documentation
# for more information.
FROM nvidia/cuda:9.0-base-ubuntu16.04
ARG UBUNTU_VERSION=16.04
FROM nvidia/cuda:9.0-base-ubuntu${UBUNTU_VERSION} as base
# Pick up some TF dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
cuda-command-line-tools-9-0 \
@ -46,6 +31,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
cuda-curand-9-0 \
cuda-cusolver-9-0 \
cuda-cusparse-9-0 \
curl \
libcudnn7=7.2.1.38-1+cuda9.0 \
libnccl2=2.2.13-1+cuda9.0 \
libfreetype6-dev \
@ -53,6 +39,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
libpng12-dev \
libzmq3-dev \
pkg-config \
rsync \
software-properties-common \
unzip \
&& \
@ -64,7 +51,10 @@ RUN apt-get update && \
apt-get update && \
apt-get install libnvinfer4=4.1.2-1+cuda9.0
ARG USE_PYTHON_3_NOT_2=True
# For CUDA profiling, TensorFlow requires CUPTI.
ENV LD_LIBRARY_PATH /usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH
ARG USE_PYTHON_3_NOT_2
ARG _PY_SUFFIX=${USE_PYTHON_3_NOT_2:+3}
ARG PYTHON=python${_PY_SUFFIX}
ARG PIP=pip${_PY_SUFFIX}
@ -76,11 +66,19 @@ RUN apt-get update && apt-get install -y \
${PYTHON} \
${PYTHON}-pip
RUN ${PIP} install --upgrade \
RUN ${PIP} --no-cache-dir install --upgrade \
pip \
setuptools
ARG TF_PACKAGE=tensorflow-gpu
# Some TF tools expect a "python" binary
RUN ln -s $(which ${PYTHON}) /usr/local/bin/python
# Options:
# tensorflow
# tensorflow-gpu
# tf-nightly
# tf-nightly-gpu
ARG TF_PACKAGE=tensorflow
RUN ${PIP} install ${TF_PACKAGE}
COPY bashrc /etc/bash.bashrc

15
tensorflow/tools/dockerfiles/partials/jupyter.partial.Dockerfile
View File

@ -1,8 +1,15 @@
RUN ${PIP} install jupyter
RUN ${PIP} install jupyter matplotlib
RUN mkdir /notebooks && chmod a+rwx /notebooks
RUN mkdir -p /tf/tensorflow-tutorials && chmod -R a+rwx /tf/
RUN mkdir /.local && chmod a+rwx /.local
WORKDIR /notebooks
RUN apt-get install -y --no-install-recommends wget
WORKDIR /tf/tensorflow-tutorials
RUN wget https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/keras/basic_classification.ipynb
RUN wget https://raw.githubusercontent.com/tensorflow/docs/master/site/en/tutorials/keras/basic_text_classification.ipynb
RUN apt-get autoremove -y && apt-get remove -y wget
WORKDIR /tf
EXPOSE 8888
CMD ["bash", "-c", "source /etc/bash.bashrc && jupyter notebook --notebook-dir=/notebooks --ip 0.0.0.0 --no-browser --allow-root"]
RUN ${PYTHON} -m ipykernel.kernelspec
CMD ["bash", "-c", "source /etc/bash.bashrc && jupyter notebook --notebook-dir=/tf --ip 0.0.0.0 --no-browser --allow-root"]

7
tensorflow/tools/dockerfiles/partials/tensorflow.partial.Dockerfile
View File

@ -1,2 +1,7 @@
ARG TF_PACKAGE
# Options:
# tensorflow
# tensorflow-gpu
# tf-nightly
# tf-nightly-gpu
ARG TF_PACKAGE=tensorflow
RUN ${PIP} install ${TF_PACKAGE}

0
tensorflow/tools/dockerfiles/partials/test-import.partial.Dockerfile Normal file
View File

2
tensorflow/tools/dockerfiles/partials/ubuntu.partial.Dockerfile
View File

@ -1,2 +0,0 @@
ARG UBUNTU_VERSION=16.04
FROM ubuntu:${UBUNTU_VERSION}

14
tensorflow/tools/dockerfiles/partials/bazel.partial.Dockerfile → tensorflow/tools/dockerfiles/partials/ubuntu/bazel.partial.Dockerfile
View File

@ -6,6 +6,20 @@ RUN apt-get update && apt-get install -y \
${PYTHON}-dev \
swig
RUN ${PIP} --no-cache-dir install \
Pillow \
h5py \
keras_applications \
keras_preprocessing \
matplotlib \
mock \
numpy \
scipy \
sklearn \
pandas \
&& test "${USE_PYTHON_3_NOT_2}" -eq 1 && true || ${PIP} --no-cache-dir install \
enum34
# Install bazel
RUN echo "deb [arch=amd64] http://storage.googleapis.com/bazel-apt stable jdk1.8" | tee /etc/apt/sources.list.d/bazel.list && \
curl https://bazel.build/bazel-release.pub.gpg | apt-key add - && \

7
tensorflow/tools/dockerfiles/partials/ubuntu-devel.partial.Dockerfile → tensorflow/tools/dockerfiles/partials/ubuntu/cpu-devel.partial.Dockerfile
View File

@ -1,5 +1,4 @@
ARG UBUNTU_VERSION=16.04
FROM ubuntu:${UBUNTU_VERSION}
FROM ubuntu:${UBUNTU_VERSION} AS base
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
@ -11,7 +10,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
libpng12-dev \
libzmq3-dev \
pkg-config \
python-dev \
rsync \
software-properties-common \
unzip \
@ -22,3 +20,6 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
&& \
apt-get clean && \
rm -rf /var/lib/apt/lists/*
ENV CI_BUILD_PYTHON python

1
tensorflow/tools/dockerfiles/partials/ubuntu/cpu.partial.Dockerfile Normal file
View File

@ -0,0 +1 @@
FROM ubuntu:${UBUNTU_VERSION} as base

18
tensorflow/tools/dockerfiles/partials/nvidia-devel.partial.Dockerfile → tensorflow/tools/dockerfiles/partials/ubuntu/nvidia-devel.partial.Dockerfile
View File

@ -1,5 +1,4 @@
ARG UBUNTU_VERSION=16.04
FROM nvidia/cuda:9.0-base-ubuntu${UBUNTU_VERSION}
FROM nvidia/cuda:9.0-base-ubuntu${UBUNTU_VERSION} as base
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
@ -22,6 +21,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
libpng12-dev \
libzmq3-dev \
pkg-config \
python-dev \
rsync \
software-properties-common \
unzip \
@ -44,6 +44,14 @@ RUN mkdir /usr/local/cuda-9.0/lib && \
ln -s /usr/lib/x86_64-linux-gnu/libnccl.so.2 /usr/local/cuda/lib/libnccl.so.2 && \
ln -s /usr/include/nccl.h /usr/local/cuda/include/nccl.h
# TODO(tobyboyd): Remove after license is excluded from BUILD file.
RUN gunzip /usr/share/doc/libnccl2/NCCL-SLA.txt.gz && \
cp /usr/share/doc/libnccl2/NCCL-SLA.txt /usr/local/cuda/
# Configure the build for our CUDA configuration.
ENV CI_BUILD_PYTHON python
ENV LD_LIBRARY_PATH /usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH
ENV TF_NEED_CUDA 1
ENV TF_NEED_TENSORRT 1
ENV TF_CUDA_COMPUTE_CAPABILITIES=3.5,5.2,6.0,6.1,7.0
ENV TF_CUDA_VERSION=9.0
ENV TF_CUDNN_VERSION=7
# NCCL 2.x
ENV TF_NCCL_VERSION=2

8
tensorflow/tools/dockerfiles/partials/nvidia.partial.Dockerfile → tensorflow/tools/dockerfiles/partials/ubuntu/nvidia.partial.Dockerfile
View File

@ -1,6 +1,5 @@
FROM nvidia/cuda:9.0-base-ubuntu16.04
FROM nvidia/cuda:9.0-base-ubuntu${UBUNTU_VERSION} as base
# Pick up some TF dependencies
RUN apt-get update && apt-get install -y --no-install-recommends \
build-essential \
cuda-command-line-tools-9-0 \
@ -9,6 +8,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
cuda-curand-9-0 \
cuda-cusolver-9-0 \
cuda-cusparse-9-0 \
curl \
libcudnn7=7.2.1.38-1+cuda9.0 \
libnccl2=2.2.13-1+cuda9.0 \
libfreetype6-dev \
@ -16,6 +16,7 @@ RUN apt-get update && apt-get install -y --no-install-recommends \
libpng12-dev \
libzmq3-dev \
pkg-config \
rsync \
software-properties-common \
unzip \
&& \
@ -26,3 +27,6 @@ RUN apt-get update && \
apt-get install nvinfer-runtime-trt-repo-ubuntu1604-4.0.1-ga-cuda9.0 && \
apt-get update && \
apt-get install libnvinfer4=4.1.2-1+cuda9.0
# For CUDA profiling, TensorFlow requires CUPTI.
ENV LD_LIBRARY_PATH /usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH

5
tensorflow/tools/dockerfiles/partials/python.partial.Dockerfile → tensorflow/tools/dockerfiles/partials/ubuntu/python.partial.Dockerfile
View File

@ -10,6 +10,9 @@ RUN apt-get update && apt-get install -y \
${PYTHON} \
${PYTHON}-pip
RUN ${PIP} install --upgrade \
RUN ${PIP} --no-cache-dir install --upgrade \
pip \
setuptools
# Some TF tools expect a "python" binary
RUN ln -s $(which ${PYTHON}) /usr/local/bin/python

0
tensorflow/tools/dockerfiles/partials/ubuntu/test-devel.partial.Dockerfile Normal file
View File

1
tensorflow/tools/dockerfiles/partials/ubuntu/version.partial.Dockerfile Normal file
View File

@ -0,0 +1 @@
ARG UBUNTU_VERSION=16.04

306
tensorflow/tools/dockerfiles/spec.yml
View File

@ -1,195 +1,135 @@
# ======
# HEADER
# ======
#
# This is commented-out and prepended to each generated Dockerfile.
header: |
Copyright 2018 The TensorFlow Authors. All Rights Reserved.
# Copyright 2018 The TensorFlow Authors. All Rights Reserved.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# ============================================================================
#
# THIS IS A GENERATED DOCKERFILE.
#
# This file was assembled from multiple pieces, whose use is documented
# throughout. Please refer to the the TensorFlow dockerfiles documentation
# for more information.
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License.
============================================================================
THIS IS A GENERATED DOCKERFILE.
This file was assembled from multiple pieces, whose use is documented
below. Please refer to the the TensorFlow dockerfiles documentation for
more information. Build args are documented as their default value.
# ========
# PARTIALS
# ========
# A combinatorial explosion of Docker images and Dockerfiles.
# Each "release" defines all of the ways to combine related but separate chunks
# of functionality ("slices") by listing all of the "slice sets" to use when
# building.
#
# Represent and document pieces of a Dockerfile. Spec:
#
# name: the name of the partial, is referenced from the images section
# desc: A description, inserted later into the Dockerfile
# file: Alternative file prefix, e.g. file.partial.Dockerfile. The default is
# the name of the partial.
# args: A dict of ARGs in the Dockerfile; each entry has the format
# ARG_NAME: VALUE where VALUE is one of:
# - a dict:
# desc: Documentation for the arg
# default: Default value for the arg; is written to the Dockerfile
# options: List of strings, part of documentation
# - a concrete value: the same as a dictionary with default: [value].
# For example, a release that uses {nightly}{py} would create 4 Dockerfiles
# (which could become images or concrete Dockerfiles), because the "nightly"
# and "py" slice sets both have two entries:
#
# - nightly (no -py2 because the Python 2 slice set has add_to_name: ""
# - nightly-py3
# - nightly-gpu (similar)
# - nightly-gpu-py3
partials:
ubuntu:
desc: Start from Ubuntu (no GPU support)
args:
UBUNTU_VERSION: 16.04
releases:
nightly:
tag_specs:
- "{nightly}{py}{jupyter}"
ubuntu-devel:
desc: Start from Ubuntu, with TF development packages (no GPU support)
args:
UBUNTU_VERSION: 16.04
versioned:
tag_specs:
- "{_TAG_PREFIX}{ubuntu}{py}{jupyter}"
bazel:
desc: Install the latest version of Bazel and Python development tools.
ubuntu-dockerfiles:
is_dockerfiles: true
upload_images: false
tag_specs:
- "{ubuntu}{jupyter}"
nvidia:
desc: NVIDIA with CUDA and CuDNN, no dev stuff
args:
UBUNTU_VERSION: 16.04
slice_sets:
nvidia-devel:
desc: >
Start from Nvidia's Ubuntu base image with CUDA and CuDNN, with TF
development packages.
args:
UBUNTU_VERSION: 16.04
py:
- add_to_name: ""
args:
- USE_PYTHON_3_NOT_2=
- add_to_name: "-py3"
args:
- USE_PYTHON_3_NOT_2=1
python:
desc: Python is required for TensorFlow and other libraries.
args:
USE_PYTHON_3_NOT_2:
default: true
desc: Install python 3 over Python 2
tensorflow:
desc: Install the TensorFlow Python package.
args:
TF_PACKAGE:
default: tensorflow
options:
- tensorflow
- tensorflow-gpu
- tf-nightly
- tf-nightly-gpu
desc: The specific TensorFlow Python package to install
shell:
desc: Configure TensorFlow's shell prompt and login tools.
jupyter:
desc: Launch Jupyter on execution instead of a bash prompt.
- add_to_name: ""
- add_to_name: "-jupyter"
partials:
- jupyter
# ======
# IMAGES
# ======
#
# Represent Dockerfiles. Spec:
#
# name: the name of the image, possibly referenced by other images
# desc: A description, inserted later into the Dockerfile
# create-dockerfile: Create a dockerfile based on this. Useful for creating
# extensible base images that don't need a file. Default is true.
# partials: List of VALUEs, where a VALUE is either:
# - the name of a partial, which inserts that partial into this image
# - image: [name of another image], which inserts the partials from that
# image into this image
# arg-defaults: List of VALUEs, where a VALUE is either:
# - ARG_NAME: VALUE, which sets the ARG_NAME to VALUE wherever it appears
# in this image's partials
# - [name of another image], which loads the default args from that image
images:
ubuntu:
- add_to_name: ""
dockerfile_exclusive_name: "cpu"
partials:
- ubuntu/version
- ubuntu/cpu
- ubuntu/python
- tensorflow
- shell
- add_to_name: "-gpu"
dockerfile_exclusive_name: "gpu"
args:
- TF_PACKAGE=tensorflow-gpu
partials:
- ubuntu/version
- ubuntu/nvidia
- ubuntu/python
- tensorflow
- shell
tests:
- import-gpu.sh
test_runtime: nvidia
- add_to_name: "-devel"
dockerfile_exclusive_name: "cpu-devel"
partials:
- ubuntu/version
- ubuntu/cpu-devel
- ubuntu/python
- ubuntu/bazel
- shell
tests:
- build-cpu.sh
- add_to_name: "-gpu-devel"
dockerfile_exclusive_name: "gpu-devel"
partials:
- ubuntu/version
- ubuntu/nvidia-devel
- ubuntu/python
- ubuntu/bazel
- shell
tests:
- build-gpu.sh
test_runtime: nvidia
nodev:
create-dockerfile: false
partials:
- python
- tensorflow
- shell
dev:
create-dockerfile: false
partials:
- python
- bazel
- shell
cpu:
desc: Ubuntu-based, CPU-only environment for using TensorFlow
partials:
- ubuntu
- image: nodev
cpu-devel:
desc: >
Ubuntu-based, CPU-only environment for developing changes for
TensorFlow.
partials:
- ubuntu-devel
- image: dev
nvidia:
desc: Ubuntu-based, Nvidia-GPU-enabled environment for using TensorFlow.
arg-defaults:
- TF_PACKAGE: tensorflow-gpu
partials:
- nvidia
- image: nodev
nvidia-devel:
desc: >
Ubuntu-based, Nvidia-GPU-enabled environment for developing changes
for TensorFlow.
arg-defaults:
- TF_PACKAGE: tensorflow-gpu
partials:
- nvidia-devel
- image: dev
cpu-jupyter:
desc: >
Ubuntu-based, CPU-only environment for using TensorFlow, with Jupyter
included.
partials:
- image: cpu
- jupyter
cpu-devel-jupyter:
desc: >
Ubuntu-based, CPU-only environment for developing changes for
TensorFlow, with Jupyter included.
partials:
- image: cpu-devel
- jupyter
nvidia-jupyter:
desc: >
Ubuntu-based, Nvidia-GPU-enabled environment for using TensorFlow, with
Jupyter included.
arg-defaults:
- nvidia
partials:
- image: nvidia
- jupyter
nvidia-devel-jupyter:
desc: >
Ubuntu-based, Nvidia-GPU-enabled environment for developing changes for
TensorFlow, with Jupyter included.
arg-defaults:
- nvidia-devel
partials:
- image: nvidia-devel
- jupyter
nightly:
- add_to_name: "nightly"
partials:
- ubuntu/version
- ubuntu/cpu
- ubuntu/python
- tensorflow
- shell
args:
- TF_PACKAGE=tf-nightly
tests:
- import.sh
- add_to_name: "nightly-gpu"
partials:
- ubuntu/version
- ubuntu/nvidia
- ubuntu/python
- tensorflow
- shell
test_runtime: nvidia
tests:
- import-gpu.sh
args:
- TF_PACKAGE=tf-nightly-gpu

22
tensorflow/tools/dockerfiles/tests/build-cpu.sh Executable file
View File

@ -0,0 +1,22 @@
#!/usr/bin/env bash
# Download and build TensorFlow.
set -euxo pipefail
git clone --branch=master --depth=1 https://github.com/tensorflow/tensorflow.git /tensorflow
cd /tensorflow
ln -s $(which ${PYTHON}) /usr/local/bin/python
# For optimized builds appropriate for the hardware platform of your choosing, uncomment below...
# For ivy-bridge or sandy-bridge
# --copt=-march="ivybridge" \
# for haswell, broadwell, or skylake
# --copt=-march="haswell" \
tensorflow/tools/ci_build/builds/configured CPU \
bazel build -c opt --copt=-mavx --cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" \
tensorflow/tools/pip_package:build_pip_package && \
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/pip && \
pip --no-cache-dir install --upgrade /tmp/pip/tensorflow-*.whl && \
rm -rf /tmp/pip && \
rm -rf /root/.cache

20
tensorflow/tools/dockerfiles/tests/build-gpu.sh Executable file
View File

@ -0,0 +1,20 @@
#!/usr/bin/env bash
# Download and build TensorFlow.
set -euxo pipefail
git clone --branch=master --depth=1 https://github.com/tensorflow/tensorflow.git /tensorflow
cd /tensorflow
ln -s $(which ${PYTHON}) /usr/local/bin/python
ln -s /usr/local/cuda/lib64/stubs/libcuda.so /usr/local/cuda/lib64/stubs/libcuda.so.1
LD_LIBRARY_PATH=/usr/local/cuda/lib64/stubs:${LD_LIBRARY_PATH} \
tensorflow/tools/ci_build/builds/configured GPU \
bazel build -c opt --copt=-mavx --config=cuda \
--cxxopt="-D_GLIBCXX_USE_CXX11_ABI=0" \
tensorflow/tools/pip_package:build_pip_package && \
rm /usr/local/cuda/lib64/stubs/libcuda.so.1 && \
bazel-bin/tensorflow/tools/pip_package/build_pip_package /tmp/pip && \
pip --no-cache-dir install --upgrade /tmp/pip/tensorflow-*.whl && \
rm -rf /tmp/pip && \
rm -rf /root/.cache

2
tensorflow/tools/dockerfiles/tests/import-gpu.sh Executable file
View File

@ -0,0 +1,2 @@
#!/usr/bin/env bash
python -c 'import tensorflow as tf; tf.test.is_gpu_available() or exit(1)'

3
tensorflow/tools/dockerfiles/tests/import.sh Executable file
View File

@ -0,0 +1,3 @@
#!/usr/bin/env bash
set -euxo pipefail
python -c 'import tensorflow as tf'

5
tensorflow/tools/dockerfiles/assembler.Dockerfile → tensorflow/tools/dockerfiles/tools.Dockerfile
View File

@ -20,8 +20,9 @@
FROM debian:stretch
LABEL maintainer="Austin Anderson <angerson@google.com>"
RUN apt-get update && apt-get install -y python3 python3-pip bash
RUN pip3 install --upgrade pip setuptools pyyaml absl-py cerberus
RUN apt-get update && apt-get install -y python3 python3-pip bash curl
RUN curl -sSL https://get.docker.com/ | sh
RUN pip3 install --upgrade pip setuptools pyyaml absl-py cerberus docker
WORKDIR /tf
VOLUME ["/tf"]