`Pod failed: Error (exit code 255)` but no retry

[tooptoop4]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

i have workflow with 2 steps, first step succeeded, but 2nd step shows no logs at all and has Error (exit code 255) in the UI.

if pod couldn't start i would expect it to retry (since i have retryPolicy: "OnError")

note: i have previously run same workflow type 1000s of times successfully, so seems like rare scenario

here are kubectl events output:

{"time":"2023-07-13T22:00:07.575403594Z","stream":"stdout","_p":"F","log":"auth                18m         Normal    TaintManagerEviction           pod/failedpod                                            taint-controller                                                   Cancelling deletion of Pod auth/failedpod                                                                                                                                                                      18m          2       failedpod.17718b4a8ba756ef
{"time":"2023-07-13T22:00:07.57589893Z","stream":"stdout","_p":"F","log":"auth                18m         Warning   FailedCreatePodSandBox         pod/failedpod                                            kubelet, redactnode          Failed to create pod sandbox: rpc error: code = Unknown desc = failed to setup network for sandbox \"redactid\": plugin type=\"aws-cni\" name=\"aws-cni\" failed (add): add cmd: failed to assign an IP address to container   18m          1       failedpod.17718b4a8d4a3809
{"time":"2023-07-13T22:00:07.575931441Z","stream":"stdout","_p":"F","log":"auth                18m         Warning   FailedScheduling               pod/earlierstepsuccesspod                                                  default-scheduler                                                  0/3 nodes are available: 1 Insufficient memory, 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.                                                                                                                         20m          8       earlierstepsuccesspod.17718b374ee51bf4
{"time":"2023-07-13T22:00:07.577291549Z","stream":"stdout","_p":"F","log":"auth                19m         Normal    TriggeredScaleUp               pod/earlierstepsuccesspod                                                  cluster-autoscaler                                                 pod triggered scale-up: [{redactnodegroup 1->2 (max: 5)}]                                                                                                                                                        19m          1       earlierstepsuccesspod.17718b4330a0da1e
{"time":"2023-07-13T22:00:07.577306969Z","stream":"stdout","_p":"F","log":"auth                18m         Warning   WorkflowNodeFailed             workflow/redactwf                                                                  workflow-controller                                                Failed node redactwf(0)[3].redactfailedstep: Error (exit code 255)                                                                                                                                                                        18m          1       redactwf.17718b53784fccf4
{"time":"2023-07-13T22:00:07.577789875Z","stream":"stdout","_p":"F","log":"auth                18m         Warning   WorkflowNodeFailed             workflow/redactwf                                                                  workflow-controller                                                Failed node redactwf: Max duration limit exceeded                                                                                                                                                                                        18m          1       redactwf.17718b53785008cc
{"time":"2023-07-13T22:00:07.577813436Z","stream":"stdout","_p":"F","log":"auth                18m         Warning   WorkflowNodeFailed             workflow/redactwf                                                                  workflow-controller                                                Failed node redactwf(0): child 'redactwf-3286042985' failed                                                                                                                                      18m          1       redactwf.17718b5378502230
{"time":"2023-07-13T22:00:07.578257592Z","stream":"stdout","_p":"F","log":"auth                18m         Warning   WorkflowNodeFailed             workflow/redactwf                                                                  workflow-controller                                                Failed node redactwf(0)[3].redactfailedstep(0): Error (exit code 255)                                                                                                                                                                     18m          1       redactwf.17718b53785058b8
{"time":"2023-07-13T22:00:07.578269782Z","stream":"stdout","_p":"F","log":"auth                18m         Warning   WorkflowNodeFailed             workflow/redactwf                                                                  workflow-controller                                                Failed node redactwf(0)[3]: child 'redactwf-3286042985' failed                                                                                                                                   18m          1       redactwf.17718b5378506b50
{"time":"2023-07-13T22:00:07.578732638Z","stream":"stdout","_p":"F","log":"auth                17m         Warning   WorkflowFailed                 workflow/redactwf                                                                  workflow-controller                                                Max duration limit exceeded                                                                                                                                                                                                                                                      17m          1       redactwf.17718b5b3bd9d328

here are cluster-autoscaler logs:

I0713 21:40:05.075965       1 static_autoscaler.go:731] Pod successfulearlierstep is 0.997 seconds old, too new to consider unschedulable
I0713 21:40:15.185394       1 static_autoscaler.go:731] Pod successfulearlierstep is 11.101 seconds old, too new to consider unschedulable
I0713 21:40:25.349191       1 static_autoscaler.go:731] Pod successfulearlierstep is 21.215 seconds old, too new to consider unschedulable
I0713 21:40:35.599944       1 static_autoscaler.go:731] Pod successfulearlierstep is 31.381 seconds old, too new to consider unschedulable
I0713 21:40:45.691346       1 static_autoscaler.go:731] Pod successfulearlierstep is 41.619 seconds old, too new to consider unschedulable
I0713 21:40:55.784535       1 klogx.go:86] Pod auth/successfulearlierstep is unschedulable
I0713 21:40:55.946754       1 event_sink_logging_wrapper.go:48] Event(v1.ObjectReference{Kind:\"Pod\", Namespace:\"auth\", Name:\"successfulearlierstep\", UID:\"redact\", APIVersion:\"v1\", ResourceVersion:\"redact\", FieldPath:\"\"}): type: 'Normal' reason: 'TriggeredScaleUp' pod triggered scale-up: [{redactnodegroup 1->2 (max: 5)}]
I0713 21:41:06.037378       1 filter_out_schedulable.go:162] Pod auth.successfulearlierstep marked as unschedulable can be scheduled on node template-node-for-eks-redact-upcoming-0. Ignoring in scale up.
I0713 21:41:16.137090       1 filter_out_schedulable.go:162] Pod auth.successfulearlierstep marked as unschedulable can be scheduled on node redactnode2. Ignoring in scale up.
I0713 21:41:36.500233       1 cluster.go:170] node redactnode1 cannot be removed: auth/failedpod is not replicated

Version

3.4.8

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

n/a

Logs from the workflow controller

kubectl logs -n argo deploy/workflow-controller | grep ${workflow}

"log":"time=\"2023-07-13T21:40:04.837Z\" level=info msg=\"Processing workflow\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:40:04.861Z\" level=info msg=\"Updated phase  -> Running\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:40:04.863Z\" level=info msg=\"Retry node redactwf initialized Running\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:40:04.865Z\" level=info msg=\"Steps node redactwf238367780 initialized Running\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:40:04.865Z\" level=info msg=\"StepGroup node redactwf3158130894 initialized Running\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:40:04.867Z\" level=info msg=\"Retry node redactwf561794156 initialized Running\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:40:04.868Z\" level=info msg=\"Pod node redactwf3209474919 initialized Pending\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:40:04.901Z\" level=info msg=\"Created pod: redactwf(0)[0].msg-split(0) (redactearliersuccessfulstep)\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:40:04.901Z\" level=info msg=\"Workflow step group node redactwf3158130894 not yet completed\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:40:04.901Z\" level=info msg=\"TaskSet Reconciliation\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:40:04.901Z\" level=info msg=reconcileAgentPod namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:40:04.920Z\" level=info msg=\"Workflow update successful\" namespace=auth phase=Running resourceVersion=11388036 workflow=redactwf
"log":"time=\"2023-07-13T21:40:14.838Z\" level=info msg=\"Processing workflow\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:40:14.842Z\" level=info msg=\"Task-result reconciliation\" namespace=auth numObjs=0 workflow=redactwf
"log":"time=\"2023-07-13T21:40:14.843Z\" level=info msg=\"node changed\" namespace=auth new.message=\"Unschedulable: 0/3 nodes are available: 1 Insufficient memory, 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.\" new.phase=Pending new.progress=0/1 nodeID=redactwf3209474919 old.message= old.phase=Pending old.progress=0/1 workflow=redactwf
"log":"time=\"2023-07-13T21:40:14.847Z\" level=info msg=\"Workflow step group node redactwf3158130894 not yet completed\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:40:14.847Z\" level=info msg=\"TaskSet Reconciliation\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:40:14.847Z\" level=info msg=reconcileAgentPod namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:40:14.865Z\" level=info msg=\"Workflow update successful\" namespace=auth phase=Running resourceVersion=11388333 workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.042Z\" level=info msg=\"Processing workflow\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.047Z\" level=info msg=\"Task-result reconciliation\" namespace=auth numObjs=1 workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.047Z\" level=info msg=\"task-result changed\" namespace=auth nodeID=redactwf3209474919 workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.047Z\" level=info msg=\"node changed\" namespace=auth new.message= new.phase=Succeeded new.progress=0/1 nodeID=redactwf3209474919 old.message=\"Unschedulable: 0/3 nodes are available: 1 Insufficient memory, 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.\" old.phase=Pending old.progress=0/1 workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"node redactwf561794156 phase Running -> Succeeded\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"node redactwf561794156 finished: 2023-07-13 21:41:27.052110693 +0000 UTC\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"Step group node redactwf3158130894 successful\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"node redactwf3158130894 phase Running -> Succeeded\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"node redactwf3158130894 finished: 2023-07-13 21:41:27.052172772 +0000 UTC\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"StepGroup node redactwf2017105707 initialized Running\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"SG Outbound nodes of redactwf561794156 are [redactwf3209474919]\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"Skipping redactwf(0)[1].dts: when ''redactimg' == 'redactfalse'' evaluated false\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"Skipped node redactwf935157812 initialized Skipped (message: when ''redactimg' == 'redactfalse'' evaluated false)\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"Step group node redactwf2017105707 successful\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"node redactwf2017105707 phase Running -> Succeeded\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"node redactwf2017105707 finished: 2023-07-13 21:41:27.052633731 +0000 UTC\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"StepGroup node redactwf2017650160 initialized Running\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"SG Outbound nodes of redactwf935157812 are [redactwf935157812]\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"Skipping redactwf(0)[2].redactfalse-call: when ''redactimg' == 'redactfalse'' evaluated false\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"Skipped node redactwf1679237449 initialized Skipped (message: when ''redactimg' == 'redactfalse'' evaluated false)\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"Step group node redactwf2017650160 successful\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"node redactwf2017650160 phase Running -> Succeeded\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"node redactwf2017650160 finished: 2023-07-13 21:41:27.05291968 +0000 UTC\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"StepGroup node redactwf3158277989 initialized Running\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.052Z\" level=info msg=\"SG Outbound nodes of redactwf1679237449 are [redactwf1679237449]\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.055Z\" level=info msg=\"Retry node redactwf3286042985 initialized Running\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.057Z\" level=info msg=\"Pod node redactwf1959261136 initialized Pending\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.101Z\" level=info msg=\"Created pod: redactwf(0)[3].redactfailedstep(0) (redactfailedpod)\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.101Z\" level=info msg=\"Workflow step group node redactwf3158277989 not yet completed\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.101Z\" level=info msg=\"TaskSet Reconciliation\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.102Z\" level=info msg=reconcileAgentPod namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:27.126Z\" level=info msg=\"Workflow update successful\" namespace=auth phase=Running resourceVersion=11389509 workflow=redactwf
"log":"time=\"2023-07-13T21:41:37.107Z\" level=info msg=\"Processing workflow\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:37.112Z\" level=info msg=\"Task-result reconciliation\" namespace=auth numObjs=1 workflow=redactwf
"log":"time=\"2023-07-13T21:41:37.112Z\" level=info msg=\"node changed\" namespace=auth new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=redactwf1959261136 old.message= old.phase=Pending old.progress=0/1 workflow=redactwf
"log":"time=\"2023-07-13T21:41:37.112Z\" level=info msg=\"node unchanged\" namespace=auth nodeID=redactwf3209474919 workflow=redactwf
"log":"time=\"2023-07-13T21:41:37.114Z\" level=info msg=\"SG Outbound nodes of redactwf561794156 are [redactwf3209474919]\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:37.115Z\" level=info msg=\"SG Outbound nodes of redactwf935157812 are [redactwf935157812]\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:37.115Z\" level=info msg=\"SG Outbound nodes of redactwf1679237449 are [redactwf1679237449]\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:37.117Z\" level=info msg=\"Workflow step group node redactwf3158277989 not yet completed\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:37.117Z\" level=info msg=\"TaskSet Reconciliation\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:37.117Z\" level=info msg=reconcileAgentPod namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:37.138Z\" level=info msg=\"Workflow update successful\" namespace=auth phase=Running resourceVersion=11389662 workflow=redactwf
"log":"time=\"2023-07-13T21:41:53.555Z\" level=info msg=\"Processing workflow\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:53.560Z\" level=info msg=\"Task-result reconciliation\" namespace=auth numObjs=1 workflow=redactwf
"log":"time=\"2023-07-13T21:41:53.560Z\" level=info msg=\"node unchanged\" namespace=auth nodeID=redactwf1959261136 workflow=redactwf
"log":"time=\"2023-07-13T21:41:53.560Z\" level=info msg=\"node unchanged\" namespace=auth nodeID=redactwf3209474919 workflow=redactwf
"log":"time=\"2023-07-13T21:41:53.562Z\" level=info msg=\"SG Outbound nodes of redactwf561794156 are [redactwf3209474919]\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:53.562Z\" level=info msg=\"SG Outbound nodes of redactwf935157812 are [redactwf935157812]\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:53.562Z\" level=info msg=\"SG Outbound nodes of redactwf1679237449 are [redactwf1679237449]\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:53.565Z\" level=info msg=\"Workflow step group node redactwf3158277989 not yet completed\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:53.565Z\" level=info msg=\"TaskSet Reconciliation\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:53.565Z\" level=info msg=reconcileAgentPod namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:41:56.940Z\" level=info msg=\"insignificant pod change\" key=auth/redactfailedpod
"log":"time=\"2023-07-13T21:42:05.748Z\" level=info msg=\"Processing workflow\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.754Z\" level=info msg=\"Task-result reconciliation\" namespace=auth numObjs=1 workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.754Z\" level=info msg=\"Pod failed: Error (exit code 255)\" displayName=\"redactfailedstep(0)\" namespace=auth pod=redactfailedpod templateName=redactfailedstep workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.754Z\" level=info msg=\"marking node as failed since init container has non-zero exit code\" namespace=auth new.phase=Failed workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.754Z\" level=info msg=\"node changed\" namespace=auth new.message=\"Error (exit code 255)\" new.phase=Failed new.progress=0/1 nodeID=redactwf1959261136 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.755Z\" level=info msg=\"node unchanged\" namespace=auth nodeID=redactwf3209474919 workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.757Z\" level=info msg=\"SG Outbound nodes of redactwf561794156 are [redactwf3209474919]\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.757Z\" level=info msg=\"SG Outbound nodes of redactwf935157812 are [redactwf935157812]\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.757Z\" level=info msg=\"SG Outbound nodes of redactwf1679237449 are [redactwf1679237449]\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.760Z\" level=info msg=\"node has maxDuration set, setting executionDeadline to: Thu Jul 13 21:42:27 +0000 (21 seconds from now)\" namespace=auth node=\"redactwf(0)[3].redactfailedstep\" workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.760Z\" level=info msg=\"Node not set to be retried after status: Failed\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.761Z\" level=info msg=\"node redactwf3286042985 phase Running -> Failed\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.761Z\" level=info msg=\"node redactwf3286042985 message: Error (exit code 255)\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.761Z\" level=info msg=\"node redactwf3286042985 finished: 2023-07-13 21:42:05.761033041 +0000 UTC\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.761Z\" level=info msg=\"Step group node redactwf3158277989 deemed failed: child 'redactwf3286042985' failed\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.761Z\" level=info msg=\"node redactwf3158277989 phase Running -> Failed\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.761Z\" level=info msg=\"node redactwf3158277989 message: child 'redactwf3286042985' failed\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.761Z\" level=info msg=\"node redactwf3158277989 finished: 2023-07-13 21:42:05.761160952 +0000 UTC\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.761Z\" level=info msg=\"step group redactwf3158277989 was unsuccessful: child 'redactwf3286042985' failed\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.761Z\" level=info msg=\"Outbound nodes of redactwf3286042985 is [redactwf1959261136]\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.761Z\" level=info msg=\"Outbound nodes of redactwf238367780 is [redactwf1959261136]\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.761Z\" level=info msg=\"node redactwf238367780 phase Running -> Failed\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.761Z\" level=info msg=\"node redactwf238367780 message: child 'redactwf3286042985' failed\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.761Z\" level=info msg=\"node redactwf238367780 finished: 2023-07-13 21:42:05.761309303 +0000 UTC\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.761Z\" level=info msg=\"Checking daemoned children of redactwf238367780\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.777Z\" level=info msg=\"Max duration limit exceeded. Failing...\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.777Z\" level=info msg=\"node redactwf phase Running -> Failed\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.777Z\" level=info msg=\"node redactwf message: Max duration limit exceeded\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.778Z\" level=info msg=\"node redactwf finished: 2023-07-13 21:42:05.778684492 +0000 UTC\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.778Z\" level=info msg=\"TaskSet Reconciliation\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.779Z\" level=info msg=reconcileAgentPod namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.779Z\" level=info msg=\"Running OnExit handler: exit-handler\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.782Z\" level=info msg=\"Retry node redactwf4053160012 initialized Running\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.783Z\" level=info msg=\"Steps node redactwf468718407 initialized Running\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.784Z\" level=info msg=\"StepGroup node redactwf2768388195 initialized Running\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.786Z\" level=info msg=\"Retry node redactwf995095700 initialized Running\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.787Z\" level=info msg=\"Pod node redactwf2851953343 initialized Pending\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.827Z\" level=info msg=\"Created pod: redactwf.onExit(0)[0].notifyError(0) (redactwfsendmail-2851953343)\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.827Z\" level=info msg=\"Workflow step group node redactwf2768388195 not yet completed\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:05.857Z\" level=info msg=\"Workflow update successful\" namespace=auth phase=Running resourceVersion=11390034 workflow=redactwf
"log":"time=\"2023-07-13T21:42:15.833Z\" level=info msg=\"Processing workflow\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:15.841Z\" level=info msg=\"Task-result reconciliation\" namespace=auth numObjs=1 workflow=redactwf
"log":"time=\"2023-07-13T21:42:15.841Z\" level=info msg=\"node changed\" namespace=auth new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=redactwf2851953343 old.message= old.phase=Pending old.progress=0/1 workflow=redactwf
"log":"time=\"2023-07-13T21:42:15.841Z\" level=info msg=\"node unchanged\" namespace=auth nodeID=redactwf3209474919 workflow=redactwf
"log":"time=\"2023-07-13T21:42:15.841Z\" level=info msg=\"Pod failed: Error (exit code 255)\" displayName=\"redactfailedstep(0)\" namespace=auth pod=redactfailedpod templateName=redactfailedstep workflow=redactwf
"log":"time=\"2023-07-13T21:42:15.841Z\" level=info msg=\"marking node as failed since init container has non-zero exit code\" namespace=auth new.phase=Failed workflow=redactwf
"log":"time=\"2023-07-13T21:42:15.841Z\" level=info msg=\"node unchanged\" namespace=auth nodeID=redactwf1959261136 workflow=redactwf
"log":"time=\"2023-07-13T21:42:15.844Z\" level=info msg=\"TaskSet Reconciliation\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:15.844Z\" level=info msg=reconcileAgentPod namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:15.844Z\" level=info msg=\"Running OnExit handler: exit-handler\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:15.848Z\" level=info msg=\"Workflow step group node redactwf2768388195 not yet completed\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:15.874Z\" level=info msg=\"Workflow update successful\" namespace=auth phase=Running resourceVersion=11390148 workflow=redactwf
"log":"time=\"2023-07-13T21:42:29.151Z\" level=info msg=\"Processing workflow\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:29.157Z\" level=info msg=\"Task-result reconciliation\" namespace=auth numObjs=2 workflow=redactwf
"log":"time=\"2023-07-13T21:42:29.157Z\" level=info msg=\"task-result changed\" namespace=auth nodeID=redactwf2851953343 workflow=redactwf
"log":"time=\"2023-07-13T21:42:29.157Z\" level=info msg=\"node changed\" namespace=auth new.message= new.phase=Running new.progress=0/1 nodeID=redactwf2851953343 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=redactwf
"log":"time=\"2023-07-13T21:42:29.157Z\" level=info msg=\"Pod failed: Error (exit code 255)\" displayName=\"redactfailedstep(0)\" namespace=auth pod=redactfailedpod templateName=redactfailedstep workflow=redactwf
"log":"time=\"2023-07-13T21:42:29.157Z\" level=info msg=\"marking node as failed since init container has non-zero exit code\" namespace=auth new.phase=Failed workflow=redactwf
"log":"time=\"2023-07-13T21:42:29.157Z\" level=info msg=\"node unchanged\" namespace=auth nodeID=redactwf1959261136 workflow=redactwf
"log":"time=\"2023-07-13T21:42:29.157Z\" level=info msg=\"node unchanged\" namespace=auth nodeID=redactwf3209474919 workflow=redactwf
"log":"time=\"2023-07-13T21:42:29.160Z\" level=info msg=\"TaskSet Reconciliation\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:29.160Z\" level=info msg=reconcileAgentPod namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:29.160Z\" level=info msg=\"Running OnExit handler: exit-handler\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:29.162Z\" level=info msg=\"cleaning up pod\" action=terminateContainers key=auth/redactwfsendmail-2851953343/terminateContainers
"log":"time=\"2023-07-13T21:42:29.163Z\" level=info msg=\"Workflow step group node redactwf2768388195 not yet completed\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:29.191Z\" level=info msg=\"Workflow update successful\" namespace=auth phase=Running resourceVersion=11390263 workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.186Z\" level=info msg=\"Processing workflow\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.193Z\" level=info msg=\"Task-result reconciliation\" namespace=auth numObjs=2 workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.194Z\" level=info msg=\"node changed\" namespace=auth new.message= new.phase=Succeeded new.progress=0/1 nodeID=redactwf2851953343 old.message= old.phase=Running old.progress=0/1 workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.194Z\" level=info msg=\"Pod failed: Error (exit code 255)\" displayName=\"redactfailedstep(0)\" namespace=auth pod=redactfailedpod templateName=redactfailedstep workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.194Z\" level=info msg=\"marking node as failed since init container has non-zero exit code\" namespace=auth new.phase=Failed workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.194Z\" level=info msg=\"node unchanged\" namespace=auth nodeID=redactwf1959261136 workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.194Z\" level=info msg=\"node unchanged\" namespace=auth nodeID=redactwf3209474919 workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.197Z\" level=info msg=\"TaskSet Reconciliation\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.197Z\" level=info msg=reconcileAgentPod namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.197Z\" level=info msg=\"Running OnExit handler: exit-handler\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.201Z\" level=info msg=\"node redactwf995095700 phase Running -> Succeeded\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.201Z\" level=info msg=\"node redactwf995095700 finished: 2023-07-13 21:42:39.201333097 +0000 UTC\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.201Z\" level=info msg=\"Step group node redactwf2768388195 successful\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.201Z\" level=info msg=\"node redactwf2768388195 phase Running -> Succeeded\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.201Z\" level=info msg=\"node redactwf2768388195 finished: 2023-07-13 21:42:39.201404728 +0000 UTC\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.201Z\" level=info msg=\"Outbound nodes of redactwf995095700 is [redactwf2851953343]\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.201Z\" level=info msg=\"Outbound nodes of redactwf468718407 is [redactwf2851953343]\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.201Z\" level=info msg=\"node redactwf468718407 phase Running -> Succeeded\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.201Z\" level=info msg=\"node redactwf468718407 finished: 2023-07-13 21:42:39.201507048 +0000 UTC\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.201Z\" level=info msg=\"Checking daemoned children of redactwf468718407\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.202Z\" level=info msg=\"node redactwf4053160012 phase Running -> Succeeded\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.203Z\" level=info msg=\"node redactwf4053160012 finished: 2023-07-13 21:42:39.203016228 +0000 UTC\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.203Z\" level=info msg=\"Updated phase Running -> Failed\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.203Z\" level=info msg=\"Updated message  -> Max duration limit exceeded\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.203Z\" level=info msg=\"Marking workflow completed\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.203Z\" level=info msg=\"Marking workflow as pending archiving\" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.203Z\" level=info msg=\"Checking daemoned children of \" namespace=auth workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.208Z\" level=info msg=\"cleaning up pod\" action=deletePod key=auth/redactwf1340600742-agent/deletePod
"log":"time=\"2023-07-13T21:42:39.226Z\" level=info msg=\"Workflow update successful\" namespace=auth phase=Failed resourceVersion=11390354 workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.256Z\" level=info msg=\"archiving workflow\" namespace=auth uid=0c69ea4f-3318-4a21-9134-ab5319a0d71d workflow=redactwf
"log":"time=\"2023-07-13T21:42:39.317Z\" level=info msg=\"Queueing Failed workflow auth/redactwf for delete in 1m5s due to TTL\"
"log":"time=\"2023-07-13T21:42:44.255Z\" level=info msg=\"cleaning up pod\" action=deletePod key=auth/redactwfsendmail-2851953343/deletePod
"log":"time=\"2023-07-13T21:42:44.255Z\" level=info msg=\"cleaning up pod\" action=deletePod key=auth/redactearliersuccessfulstep/deletePod
"log":"time=\"2023-07-13T21:42:44.255Z\" level=info msg=\"cleaning up pod\" action=deletePod key=auth/redactfailedpod/deletePod
"log":"time=\"2023-07-13T21:42:59.163Z\" level=info msg=\"cleaning up pod\" action=killContainers key=auth/redactwfsendmail-2851953343/killContainers
"log":"time=\"2023-07-13T21:43:45.001Z\" level=info msg=\"Deleting garbage collected workflow 'auth/redactwf'\"
"log":"time=\"2023-07-13T21:43:45.016Z\" level=info msg=\"Successfully deleted 'auth/redactwf'\"

Logs from in your workflow's wait container

kubectl logs -n argo -c wait -l workflows.argoproj.io/workflow=${workflow},workflow.argoproj.io/phase!=Succeeded

there were none at all

[tooptoop4]

even with this retrystrategy, argo doesn't retry :( controller shows marking node as failed since init container has non-zero exit code

          retryPolicy: "Always"
          expression: 'lastRetry.status == "Error" or (lastRetry.status == "Failed" and asInt(lastRetry.exitCode) in [255])'

time=\"2023-07-18T04:10:16.553Z\" level=info msg=\"node changed\" namespace=auth new.message=\"Unschedulable: 0/3 nodes are available: 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.\" new.phase=Pending new.progress=0/1 nodeID=redactwf-1842505573 old.message= old.phase=Pending old.progress=0/1 workflow=redactwf"

time=\"2023-07-18T04:11:44.545Z\" level=info msg=\"node changed\" namespace=auth new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=redactwf-1842505573 old.message=\"Unschedulable: 0/3 nodes are available: 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found for incoming pod.\" old.phase=Pending old.progress=0/1 workflow=redactwf"

time=\"2023-07-18T04:12:11.972Z\" level=info msg=\"node changed\" namespace=auth new.message=\"Error (exit code 255)\" new.phase=Failed new.progress=0/1 nodeID=redactwf-1842505573 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=redactwf"

[stale]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[tooptoop4]

i am still getting this error in 3.4.11 with k8s 1.27
i think the way to reproduce this is to have small (ie < 4) number of nodes in k8s and try to run 70 different workflows at the same time, where each workflow is a single pod that mounts 12 configmaps and that pod has resources.requests defined consuming 15gb memory and 10gb ephemeral storage

kubectl events mentions:
FailedScheduling pod/redact default-scheduler 0/7 nodes are available: 1 Too many pods, 1 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 5 Insufficient memory. preemption: 0/7 nodes are available: 1 Preemption is not helpful for scheduling, 6 No preemption victims found for incoming MountVolume.SetUp failed for volume \"redact\" : object \"redact\"/\"redact\" not registered

workflow controller mentions:

2024-02-19T16:03:28.451Z\" level=info msg=\"node changed\" namespace=myns new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=mywf-2396499528 old.message=\"Unschedulable: 0/10 nodes are available: 1 Too many pods, 2 Insufficient cpu, 2 node(s) didn't match Pod's node affinity/selector, 2 node(s) had untolerated taint {node.kubernetes.io/not-ready: }, 5 Insufficient memory. preemption: 0/10 nodes are available: 4 Preemption is not helpful for scheduling, 6 No preemption victims found for incoming pod..\" old.phase=Pending old.progress=0/1 workflow=mywf
2024-02-19T16:03:28.456Z\" level=info msg=\"Workflow step group node mywf-4256579085 not yet completed\" namespace=myns workflow=mywf
2024-02-19T16:03:28.456Z\" level=info msg=\"TaskSet Reconciliation\" namespace=myns workflow=mywf
2024-02-19T16:03:28.456Z\" level=info msg=reconcileAgentPod namespace=myns workflow=mywf
2024-02-19T16:03:28.508Z\" level=info msg=\"Workflow update successful\" namespace=myns phase=Running resourceVersion=183000622 workflow=mywf
2024-02-19T16:04:06.123Z\" level=info msg=\"Processing workflow\" namespace=myns workflow=mywf
2024-02-19T16:04:06.127Z\" level=info msg=\"Task-result reconciliation\" namespace=myns numObjs=0 workflow=mywf
2024-02-19T16:04:06.127Z\" level=info msg=\"Pod failed: Unknown (exit code 255)\" displayName=\"red(0)\" namespace=myns pod=mywf-red-2396499528 templateName=red workflow=mywf
2024-02-19T16:04:06.128Z\" level=info msg=\"marking node as failed since init container has non-zero exit code\" namespace=myns new.phase=Failed workflow=mywf
2024-02-19T16:04:06.128Z\" level=info msg=\"node changed\" namespace=myns new.message=\"Unknown (exit code 255)\" new.phase=Failed new.progress=0/1 nodeID=mywf-2396499528 old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=mywf
2024-02-19T16:04:06.132Z\" level=info msg=\"node mywf-2024885873 phase Pending -> Failed\" namespace=myns workflow=mywf