ARC with containerMode.type: "kubernetes": -workflow container does not inherit -runner PodSpec

[bjoto]

Hey, thanks for a great project! I'm setting up ARC to work with the RISC-V Linux kernel CI.

I have ARC setup with containerMode.type: "kubernetes", and the following set up:

githubConfigUrl: "https://github.com/bjoto/linux" # move to proper github.com/linux-riscv/linux-riscv
githubConfigSecret:
  github_token: "<redacted>"

maxRunners: 10
minRunners: 0

containerMode:
  type: "kubernetes"

template:
  spec:
	securityContext:
	  fsGroup: 123
	nodeSelector:
	  cloud.google.com/compute-class: Scale-Out
	  kubernetes.io/arch: arm64
	containers:
	  - name: runner
		image: ghcr.io/actions/actions-runner:latest
		command: ["/home/runner/run.sh"]
		resources:
		  requests:
			cpu: 32
			memory: 96Gi
	volumes:
	  - name: work
		ephemeral:
		  volumeClaimTemplate:
			spec:
			  accessModes: [ "ReadWriteOnce" ]
			  resources:
				requests:
				  storage: 100Gi

Now, when a workflow is running, the resources for the runner is not inherited to the workflow:

Name:             arc-runner-set-2-676kh-runner-bnttp
Containers:
  runner:
	Container ID:  containerd://e8d7e1ddd2fb180b4c2b6c63304543c253fcb0d2a6b2df1bc222506b32eaeb65
	Image:         ghcr.io/actions/actions-runner:latest
	Image ID:      ghcr.io/actions/actions-runner@sha256:e505724e0dbb070454912b159645932f74dda9aed03c48c463f33e270a6cbfa1
	Port:          <none>
	Host Port:     <none>
	Command:
	  /home/runner/run.sh
	State:          Running
	  Started:      Thu, 14 Dec 2023 09:32:31 +0100
	Ready:          True
	Restart Count:  0
	Limits:
	  cpu:                32
	  ephemeral-storage:  1Gi
	  memory:             128Gi
	Requests:
	  cpu:                32
	  ephemeral-storage:  1Gi
	  memory:             128Gi
	Environment:
	  ACTIONS_RUNNER_CONTAINER_HOOKS:          /home/runner/k8s/index.js

but the workflow:

Name:             arc-runner-set-2-676kh-runner-bnttp-workflow
Containers:
  job:
	Container ID:  containerd://5dbf9462d18fe2192cbddc2b654a1d9e81ff8d438025f1c393b251a7cf4c068e
	Image:         ghcr.io/linux-riscv/pw-builder:latest
	Image ID:      ghcr.io/linux-riscv/pw-builder@sha256:b5648c40b99438ca6d5a2c6fe037b6da2320b878579fafbf817fd52e41dd4cc8
	Port:          <none>
	Host Port:     <none>
	Command:
	  tail
	Args:
	  -f
	  /dev/null
	State:          Running
	  Started:      Thu, 14 Dec 2023 09:33:00 +0100
	Ready:          True
	Restart Count:  0
	Limits:
	  cpu:                500m
	  ephemeral-storage:  1Gi
	  memory:             2Gi
	Requests:
	  cpu:                500m
	  ephemeral-storage:  1Gi
	  memory:             2Gi

Is it possible to configure the resources of workflow pod, or let it inherit the from the runner? It looks like it should be possible with actions/runner-container-hooks#75 ...

Is that part of the the released ARC?

[nikola-jokic]

Hey @bjoto,

Thank you for your kind words! This is intended behavior, since resources required by the runner may be different from resources required by workflow pods. In order to configure it, one way is to provide a config map in the format described in the ADR, and then mount it to the runner pod. You should also expose an environment variable specifying a file path from which the hook will read the content, modify the pod and create one.

We are planning to include hook extension example in the docs ☺️

[bjoto]

I tried the following, for the runner container:

...
        env:
          - name: ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE
            value: "/home/runner/config/worker-podspec.yaml"
        volumeMounts:
          - name: worker-podspec-volume
            mountPath: /home/runner/config
            readOnly: true
    volumes:
      - name: worker-podspec-volume
        configMap:
          name: worker-cm
          items:
            - key: worker-podspec.yaml
              path: worker-podspec.yaml
...

where the podspec is:

spec:
  containers:
  - name: $job
    resources:
      requests:
        cpu: 32
        memory: 96Gi        
    env:
    - name: ENV1
      value: "testest"

but for some reason the resources addition didn't work... The node preparejob failed.

Removing the "resources" worked (as in the ENV1 was in the runner).

Is it possible to turn on debug print somehow?

[bjoto]

Ugh, right. I just dug into actions/runner-container-hooks#111 and actions/runner-container-hooks#112. The ReadWriteMany is not a viable route for this.

actions/runner-container-hooks#112 (comment) -- I guess I need to look for workarounds... ;-)

[nikola-jokic]

I would suggest not using too many resources for the runner, since most of the work will be performed within the workflow container. It needs just enough so it can handle communication with the actions service and invoke node to call the hook. But the step execution will happen within the workflow pod. I hope you will find the right balance ☺️

[bjoto]

Thanks for getting back!

More information/background. I'm moving my existing GCE-based runner to GKE.

The RISC-V Linux kernel build jobs/test are massive, and need beefy runners. Today, we have a couple of dedicated GCE VMs, but we'd like to move to GKE/autopilot for cost-reasons.

The RISC-V kernel CI uses container builds, and hence container-mode "kubernetes" seems to be good fit. Ideally I'd like to allocate the build same node for the runner and workflow, but as actions/runner-container-hooks#111 actions/runner-container-hooks#112 states, today it's only doable with RWX, and typically different nodes, since I'm requesting large nodes.

I took the RWX-variant for a run, and for information I'm sharing the setup. Note that I'm using my own build of the the runner, but it's only a rebuild of https://github.com/actions/runner/blob/main/images/Dockerfile to get the 0.5.0 version of runner-container-hooks, which is AFAIU not part of the release.

githubConfigUrl: "https://github.com/bjoto/linux" # TODO: change to official linux-riscv repo
githubConfigSecret:
  github_token: "REDACTED"

maxRunners: 10
minRunners: 0

containerMode:
  type: "kubernetes"

template:
  spec:
    securityContext:
      fsGroup: 123
    containers:
      - name: runner
        image: ghcr.io/linux-riscv/runner-multi:latest
        command: ["/home/runner/run.sh"]
        env:
          - name: ACTIONS_RUNNER_CONTAINER_HOOK_TEMPLATE
            value: "/home/runner/config/worker-podspec.yaml"
          - name: ACTIONS_RUNNER_USE_KUBE_SCHEDULER
            value: "true"
        volumeMounts:
          - name: worker-podspec-volume
            mountPath: /home/runner/config
            readOnly: true
    volumes:
      - name: worker-podspec-volume
        configMap:
          name: worker-cm
          items:
            - key: worker-podspec.yaml
              path: worker-podspec.yaml
      - name: work
        ephemeral:
          volumeClaimTemplate:
            spec:
              storageClassName: "premium-rwx"
              accessModes: [ "ReadWriteMany" ]
              resources:
                requests:
                  storage: 100Gi

where the workflow customization in configmap is:

spec:
  nodeSelector:
    cloud.google.com/compute-class: Scale-Out
    kubernetes.io/arch: amd64
  securityContext:
    fsGroup: 123
  containers:
  - name: $job
    resources:
      requests:
        cpu: 32
        memory: 96Gi        
    volumeMounts:
        - mountPath: "/build"
          name: build-volume
  volumes:
  - name: build-volume
    ephemeral:
      volumeClaimTemplate:
        metadata:
          labels:
            type: ephemeral
        spec:
          accessModes: ["ReadWriteOnce"]
          resources:
            requests:
              storage: 1Ti

I'll let this run for a while, and hopefully there will support for RWO, running the runner and the workflow on the same node with some fancy "allocate resources upfront" mechanism.

[bjoto]

FWIW, I also seem to hit some issue that some steps are not reported back correctly to the runner, but hangs.

This step https://github.com/bjoto/linux/blob/arc-test/.github/workflows/patches.yml#L32 succeeds after ~1.5h (verified via kubectl exec) but the runner doesn't pick the results up, but hangs/spins.

What is the best way to turn on debug logging for the runner + runner-container-hooks?

[bjoto]

Got it running w/ debug. For some reason "Command.RunScriptStep" never completes, even though the script is successfully done running in the workflow container.

https://github.com/bjoto/linux/actions/runs/7244433043/job/19757241356

(I might close the discussion for now -- it's diverging ;-))

[ArieLevs]

@bjoto were you able to resolve this/spot the issue?
I have the same scenario, as soon as I set resources part of my hook extension configmap, the workflow pod fails to start, once its removed all is working again