ARC and dind (and Nvidia)

[robindarby]

So, I'm trying to run a docker image that requires GPU.
I've managed to get dind working, using the helm template, but setting containerMode.type="dind" - great.

However, because I need to run the build image with GPU, I need to use a custom template definition; that's okay, I'm brave.
Using the standard template, I understand that I need to customize the runner container:

     containers:
     - name: runner
       image: xyz-nvidia

I'm planning on using https://github.com/actions/runner as my base/template, adding nvidia-container-toolkit.

I'm assuming (?) I also need to customize the dind image also docker:dind:

     - name: dind
       image: docker:dind

however, this is based on Alpine Linux (which makes it hard to install the Nvidia stuff).
Are there any examples of a Ubuntu-based dind image, i.e. what the helm values.yaml would look like?

[robindarby]

Okay; so if anyone comes accross this question in the future and is facing the same issue, here is what worked for me.

Basically, you need a custom ARC values template, something like:

template:
   spec:
     initContainers:
     - name: init-dind-externals
       image: ghcr.io/my-org/cuda-actions-runner:0.0.1 
       command: ["cp", "-r", "-v", "/home/runner/externals/.", "/home/runner/tmpDir/"]
       volumeMounts:
         - name: dind-externals
           mountPath: /home/runner/tmpDir
     containers:
     - name: runner
       image: ghcr.io/my-org/cuda-actions-runner:0.0.1 
       command: ["/home/runner/run.sh"]
       env:
         - name: DOCKER_HOST
           value: unix:///var/run/docker.sock
       volumeMounts:
         - name: work
           mountPath: /home/runner/_work
         - name: dind-sock
           mountPath: /var/run
     - name: dind
       image: ghcr.io/my-org/dind-nvidia:0.0.3
       args:
         - dockerd
         - --host=unix:///var/run/docker.sock
         - --group=$(DOCKER_GROUP_GID)
       env:
         - name: DOCKER_GROUP_GID
           value: "123"
       securityContext:
         privileged: true
       volumeMounts:
         - name: work
           mountPath: /home/runner/_work
         - name: dind-sock
           mountPath: /var/run
         - name: dind-externals
           mountPath: /home/runner/externals
     volumes:
     - name: work
       emptyDir: {}
     - name: dind-sock
       emptyDir: {}
     - name: dind-externals
       emptyDir: {}
     imagePullSecrets:
      - name: ghcr-login-secret

What we are doing here is providing our own POD spec for our “runner” and “docker in docker” sidecar.

The following is where we define the runner; this is the container that runs the project docker file:

initContainers:
     - name: init-dind-externals
       image: ghcr.io/my-org/cuda-actions-runner:0.0.1 
       command: ["cp", "-r", "-v", "/home/runner/externals/.", "/home/runner/tmpDir/"]
       volumeMounts:
         - name: dind-externals
           mountPath: /home/runner/tmpDir
     containers:
     - name: runner
       image: ghcr.io/my-org/cuda-actions-runner:0.0.1 
       command: ["/home/runner/run.sh"]
       env:
         - name: DOCKER_HOST
           value: unix:///var/run/docker.sock

We use a custom action runner that installs and sets up CUDA for us based on: https://github.com/actions/runner

We also need to install CUDA and docker in the sidecar:

- name: dind
       image: ghcr.io/my-org/dind-nvidia:0.0.3

again, this is custom, based on: https://github.com/Henderake/dind-nvidia-docker

[CuriousDolphin]

can you share dockerfile for dind-nvidia and cuda-actions-runner ?

[CuriousDolphin]

can you share dockerfile for dind-nvidia and cuda-actions-runner ?

Actually it was not complex,I thank you for the insights.I created a repository with both images, dind and actions-runner, both based on CUDA 12 and Ubuntu 22 and pushed to dockerhub, if it can be useful to anyone:
https://github.com/CuriousDolphin/nvidia-dind-actions-runners

[robindarby]

In the end, it wasn't all that complicated, but better documentation would have helped :(

[apanzerj]

Hi!

My one concern is that you basically yoinked the template they use and put it as the template in the values. The fear I have is that if there is underlying change the default template things will break when your variation on the template is not updated too.

Not sure if that makes sense...