containerMode: kubernetes not able to use ephemeral volumes for "job" pods?

[bquenin]

Hi everyone!

When using containerMode: kubernetes runner, it seems that we must provide a workVolumeClaimTemplate as stated in the documentation here:

The runner will dynamically spin up pods and k8s jobs in the runner's namespace to run the workflow, so a workVolumeClaimTemplate is required for the runner's working directory, and a service account with the appropriate permissions.

If I understand correctly, the constraint of specifying a volume claim comes from the fact that the k8s runner will spawn a new "job" pod, which can be scheduled on any node, i.e. not necessarily on the same node as of the runner, and as both need to share some information, they need to use a persistent volume.

So in the case of an ephemeral k8s runner, the following steps will happen:

1. a runner is scheduled by the ARC and request for a persistent volume. This persistent volume is provisioned and provided by the underlying cloud provider, typically, an EBS volume.
2. a job is assigned to the runner. The runner will spawn a single job pod regardless the number of jobs in the workflow and this pod will inherit the same volume as the runner.
3. once the job is done. Both the runner pod and the job pod are scheduled for deletion.
4. Rinse and repeat

Please correct me if I'm wrong but this means that for each job in a workflow, a new EBS volume is requested. If that's the case, it seems that this solution doesn't scale very well.

Another approach would be to use a non ephemeral k8s runner. In that case, there is a one EBS volume per runner. It is better in terms of scalability. However, I'm not sure of what is shared exactly between the runner pod and the "job" pod.

As we want to implement reproducible builds:

• What are the implications of using non ephemeral k8s runners on the "job" pod? Would there be anything leaking between jobs?
• Is there a way to use ephemeral volumes on the runner pods or "job"pods in kubernetes mode as we neither want to share nor persist anything?
• What does the runner pod share with the job pod?

Any help would be greatly appreciated!

[bquenin]

Hi everyone, while experimenting with the k8s runner, I've discovered that it's actually using generic ephemeral volumes, which is not mentioned in the documentation.

It is possible to use any local storage provider such as:

• https://github.com/rancher/local-path-provisioner
• https://github.com/kubernetes-sigs/sig-storage-local-static-provisioner

[nebuk89]

@Vijay-train - a note for something we should work out how to add into our docs!