Using Nvidia CUDA with repo2docker

[jzf2101]

Related to jupyterhub/team-compass#52 and jupyterhub/team-compass#85 and the fork from @spMohanty that includes CUDA in the base image

@choldgraf was curious about how we could vary the base image in r2d. @minrk thought we could create an additional configuration file to specify the base image eg runtime.txt we should probably require ubuntu as the OS?

This was mentioned in this month's meeting

[betatim]

Should we re-title this issue to "Uses-cases for changing the base image"? Otherwise this will become a mixture of "how to do GPU support" and "base images for X and Y and Z".

[spMohanty]

@jzf2101 : Good to seem some momentum around this.
repo2docker with GPU support is something we have been actively using on crowdAI. We managed to collect about 800 repositories (and numerous tags in each of the repositories), amounting to ~1TB of code, that are repo2docker (the crowdai-fork) compatible , for two of the NIPS competitions this year that I was co-organizing. And we have a few more challenges on crowdAI coming up which will aggresively use the same setup.

The way we do it, is very hacky, and non-standard. We simply change the base image to nvidia/cuda:9.0-cudnn7-runtime-ubuntu16.04. Which ties all submissions to cuda-9.0, and cudnn7, while in many cases, users would want to choose which cuda version and cudnn version they want to build against.

Unfortunately, given the licensing situations with cuda and cudnn, the only way we can have cuda/cudnn in built images, if we build on top of the official base images released by nvidia : https://hub.docker.com/r/nvidia/cuda/tags/
All of them are built on top of Ubuntu 14.04/16.04/18.04.

We have seen some weird behaviour in terms of reproducibility with building dockerimages with these images as the base image, as for some reason, they abruptly update the old tags, sometimes breaking production code, or some weird dependencies. Wish they versioned the images better. But this would be an important consideration from repo2dockers point of view.

So ideally, in context of GPU support, the configurations that we would need inruntime.txt would be the cuda version, and the cudnn version, when present, the buildpack should try to build a GPU compatible image (with an appropriate message ofcourse).

And we would really love to see this released as a part of repo2docker. our fork is already 324 commits behind, and as a small team focused on quite a few things, it might not be possible to keep catching up with repo2docker all the time :D

But as I mentioned, its great to see some momentum around GPU support here, and would love to see this shipped with repo2docker soon :D

[betatim]

The Dockerfile shown in jupyterhub/zero-to-jupyterhub-k8s#994 (comment) suggests that we do not need to switch base images to be able to support running on a Nvidia GPU. Is there something missing from the linked Dockerfile (it has only been tested on GKE)?

If that Dockerfile also works on a local machine (maybe with nvidia-docker) then this isn't a use case that motivates adding the ability to switch base images.

[spMohanty]

@betatim : I quickly skimmed through it, but I think I understand whats hapenning with it.

In case of GPU support for the k8 service on GKE, which I think is still in beta mode, mounts the cuda drivers from the host nodes. GKE ensures that the cuda drivers are all accessible at a particular location (usually /usr/local/nvidia) on the host machine, so that the k8 pods can mount them on the go.

But thats something, which would be hard for us (and many other repo2docker users) to use in the long run, because there you basically pin a whole cluster to a particular cuda version. And this might have many unintended side effects, for instance the tensorflow-gpu pip packages was not supported with cuda-9.1, and users might want to explicitly choose the cuda and cudnn versions they want to run their code against, which might have huge impacts on the performance of their code.

Hence, in an ideal case, we would want each repository to specify its own cuda/cudnn versions, and build off from a base image which have the necessary drivers in the image itself.

And that Dockerfile will not work on local machine, unless you ensure that cuda is present at the correct location, and is mounted onto the container during runtime at the appropriate location(s).

[jzf2101]

@betatim suggests we may need to call:

repo2docker --no-run https://myrepo.com
will build an image from the repo
that you can then run with nvidia-docker run <name-of-image-here>

[spMohanty]

@jzf2101 : Ahh ! well nvidia-docker will install the basic nvidia drivers depending on the gpu on the host machine, but not the full cuda toolkit. The nvidia drivers enable the container to "see" the GPUs on the host machine, but they still expect a CUDA capable container to be able to use the GPUs in any sensible way.

https://devtalk.nvidia.com/default/topic/1033038/cuda-setup-and-installation/does-nvidia-docker-install-cuda-and-nvidia-driver-in-docker-/

And the licensing of CUDA (and cudnn, etc), makes it very non trivial to efficiently package them into prebuilt images, hence most people just start off the official nvidia/cuda base images. Installing cuda through scripts etc, would require you to expressly agree to the terms of usage by nvidia during the installation, which cannot be abstracted away in automated build processes (at least in a legal way).

[jzf2101]

So you're not calling the nvidia runtime when you run crowdai-repo2docker?

[spMohanty]

@jzf2101 : No, when we use crowdai-repo2docker, then we do not even need nvidia-docker, the goal there is to build a CUDA capable docker image.

When we run the docker image, then we need nvidia-docker if running locally, or the nvidia-gpu-device-plugin daemonset on a k8 cluster (while requesting for nvidia/gpu:1).

[choldgraf]

what if we did the following:

• Added a new parameter to repo2docker, something like --base-image that for now would accept something like org/image:tag.
• This image, if provided, would replace what's in the buildpack-deps image: https://github.com/jupyter/repo2docker/blob/master/repo2docker/buildpacks/base.py#L13
• We provide a warning if this is provided that says something like

  Warning: Alternate base image provided. Reproducibility or compatibility with repo2docker build-packs is not guaranteed. repo2docker will only work with base images running <LIST-OF-REQUIRED-THINGS>.
  

• Add a documentation page that more explicitly lays out the requirements in the base image.

What do folks think?

[jzf2101]

FWIW I think we should provide documentation on how to also avoid swapping the image in the case that you DO want CDA drivers.

[betatim]

To use your GPU there are two things that need doing: install various things inside your container and install various things on the host.

repo2docker can't help you with installing things on the host. In the comment I linked to above they are running on GKE and use some GKE specific way to install things on all the nodes in the cluster. I'd expect users to take care of all that "somehow".

The next question is how do we get stuff into the docker image. This is where a lot of people use the Nvidia image as a base image.

Taking a look at the Dockerfile for nvidia/cuda it appears to install some apt packages and set some environment variables. What I'd like to confirm is if the Dockerfile in this comment does the same as the Nvidia Dockerfile does. My guess is the answer is yes, but I don't have access to a GPU to check.

# For the latest tag, see: https://hub.docker.com/r/jupyter/datascience-notebook/tags/
FROM jupyter/datascience-notebook:f2889d7ae7d6

# GPU powered ML
# ----------------------------------------
RUN conda install -c conda-forge --yes --quiet \
    tensorflow-gpu \
    cudatoolkit=9.0 && \
    conda clean -tipsy && \
    fix-permissions $CONDA_DIR && \
    fix-permissions /home/$NB_USER

# Allow drivers installed by the nvidia-driver-installer to be located
ENV LD_LIBRARY_PATH=${LD_LIBRARY_PATH}:/usr/local/nvidia/lib64
# Also, utilities like `nvidia-smi` are installed here
ENV PATH=${PATH}:/usr/local/nvidia/bin

Instead of installing apt packages it uses conda-forge packages. This is easier for us to do in repo2docker as we currently do not support adding extra PPAs before installing user specified apt packages.

This would give the user control over which version of CUDA to install inside the image no? It would have to match the drivers installed on the host but that is also the case when you use the Nvidia provided image.