ARC Runner Job step getting stuck until timeout or prematurely starting next Job step while the previous step process is still in progress #4019
Unanswered
ahamednijamudeen
asked this question in
Questions
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
-
We are running Github ARC controller and Runner Helm chart version
0.10.1.in Azure Kubernetesv1.31.2. Running the ARC inKubernetesKind.In our workflow, we have Job step with long running Postgres
pg_dumpprocess which can take couple of hours to complete due to size of the database. What I have been seeing when the Job is triggered is that workflow step stop printing any logs after few minutes even though thepg_dumpprocess seems to still running in the workflow pod, sometime the step just hangs without any progress and times out after 6 hours even though the process within the workflow pod seems to have finished after some time.in some cases, the JOB proceeds to the next step of
clean upandpg_restorewhile the previous step processpg_dumpis still in progress within the workflow pod.Below process output from workflow POD when the
pg_dumpwas running which show the process executed via shell scriptshAfter couple of minute, the Job proceed to the next step of
clean upand eventually topg_restorewhile the previous processpg_dumphasn't completed.. below is the process listing which shows both are active and somehow the Job Step determinedpg_dumpstep is completedHere's the TOP command from workflow pod that shows both process from different step are in progress . I see the CPU utilization of
pg_dumpis close to ~100% alwayspg_dumpprocess running close to19minute andpg_restorefrom different step running in parallel for~2minute.Below is the Runner POD Log during the step movement between
pg_dump,clean_upandpg_restoreBelow is our Runner POD spec. we are using Flux CD to sync in our environment
Beta Was this translation helpful? Give feedback.
All reactions