workflow is in running status after initcontainer failed

[capacman]

Pre-requisites

• I have double-checked my configuration
• I can confirm the issues exists when I tested with :latest
• I'd like to contribute the fix myself (see contributing guide)

What happened/what you expected to happen?

When i submit a workflow with an initcontainer and if that initcontainer fails workflows stuck at running state. I expect workflow to fail.

Version

v.3.4.5

Paste a small workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflows that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  name: init-fail
spec:
  entrypoint: init-container-example
  templates:
  - name: init-container-example
    container:
      image: alpine:latest
      command: ["echo", "bye"]
      volumeMounts:
      - name: foo
        mountPath: /foo
    initContainers:
    - name: hello
      image: alpine:latest
      command: ["abcd"]
      mirrorVolumeMounts: true
  volumes:
    - name: foo
      emptyDir: {}

Logs from the workflow controller

time="2023-03-20T19:35:43.494Z" level=info msg="Processing workflow" namespace=henkan-argo-test-flows workflow=init-fail
time="2023-03-20T19:35:43.497Z" level=info msg="Updated phase  -> Running" namespace=henkan-argo-test-flows workflow=init-fail
time="2023-03-20T19:35:43.498Z" level=info msg="Pod node init-fail initialized Pending" namespace=henkan-argo-test-flows workflow=init-fail
time="2023-03-20T19:35:43.524Z" level=info msg="Created pod: init-fail (init-fail)" namespace=henkan-argo-test-flows workflow=init-fail
time="2023-03-20T19:35:43.524Z" level=info msg="TaskSet Reconciliation" namespace=henkan-argo-test-flows workflow=init-fail
time="2023-03-20T19:35:43.524Z" level=info msg=reconcileAgentPod namespace=henkan-argo-test-flows workflow=init-fail
time="2023-03-20T19:35:43.530Z" level=info msg="Workflow update successful" namespace=henkan-argo-test-flows phase=Running resourceVersion=250900670 workflow=init-fail
time="2023-03-20T19:35:46.527Z" level=info msg="Processing workflow" namespace=henkan-argo-test-flows workflow=init-fail
time="2023-03-20T19:35:46.528Z" level=info msg="Task-result reconciliation" namespace=henkan-argo-test-flows numObjs=0 workflow=init-fail
time="2023-03-20T19:35:46.528Z" level=info msg="node changed" namespace=henkan-argo-test-flows new.message=PodInitializing new.phase=Pending new.progress=0/1 nodeID=init-fail old.message= old.phase=Pending old.progress=0/1 workflow=init-fail
time="2023-03-20T19:35:46.528Z" level=info msg="TaskSet Reconciliation" namespace=henkan-argo-test-flows workflow=init-fail
time="2023-03-20T19:35:46.528Z" level=info msg=reconcileAgentPod namespace=henkan-argo-test-flows workflow=init-fail
time="2023-03-20T19:35:46.535Z" level=info msg="Workflow update successful" namespace=henkan-argo-test-flows phase=Running resourceVersion=250900702 workflow=init-fail
time="2023-03-20T19:35:49.536Z" level=info msg="Processing workflow" namespace=henkan-argo-test-flows workflow=init-fail
time="2023-03-20T19:35:49.537Z" level=info msg="Task-result reconciliation" namespace=henkan-argo-test-flows numObjs=0 workflow=init-fail
time="2023-03-20T19:35:49.537Z" level=info msg="Pod failed: Pod failed before main container starts" displayName=init-fail namespace=henkan-argo-test-flows pod=init-fail templateName=init-container-example workflow=init-fail
time="2023-03-20T19:35:49.537Z" level=info msg="leaving phase un-changed: wait container is not yet terminated " namespace=henkan-argo-test-flows new.phase=Error workflow=init-fail
time="2023-03-20T19:35:49.537Z" level=info msg="node changed" namespace=henkan-argo-test-flows new.message="Pod failed before main container starts" new.phase=Pending new.progress=0/1 nodeID=init-fail old.message=PodInitializing old.phase=Pending old.progress=0/1 workflow=init-fail
time="2023-03-20T19:35:49.537Z" level=info msg="TaskSet Reconciliation" namespace=henkan-argo-test-flows workflow=init-fail
time="2023-03-20T19:35:49.537Z" level=info msg=reconcileAgentPod namespace=henkan-argo-test-flows workflow=init-fail
time="2023-03-20T19:35:49.546Z" level=info msg="Workflow update successful" namespace=henkan-argo-test-flows phase=Running resourceVersion=250900729 workflow=init-fail
time="2023-03-20T19:35:52.547Z" level=info msg="Processing workflow" namespace=henkan-argo-test-flows workflow=init-fail
time="2023-03-20T19:35:52.548Z" level=info msg="Task-result reconciliation" namespace=henkan-argo-test-flows numObjs=0 workflow=init-fail
time="2023-03-20T19:35:52.548Z" level=info msg="Pod failed: Pod failed before main container starts" displayName=init-fail namespace=henkan-argo-test-flows pod=init-fail templateName=init-container-example workflow=init-fail
time="2023-03-20T19:35:52.548Z" level=info msg="leaving phase un-changed: wait container is not yet terminated " namespace=henkan-argo-test-flows new.phase=Error workflow=init-fail
time="2023-03-20T19:35:52.548Z" level=info msg="node unchanged" namespace=henkan-argo-test-flows nodeID=init-fail workflow=init-fail
time="2023-03-20T19:35:52.548Z" level=info msg="TaskSet Reconciliation" namespace=henkan-argo-test-flows workflow=init-fail
time="2023-03-20T19:35:52.548Z" level=info msg=reconcileAgentPod namespace=henkan-argo-test-flows workflow=init-fail

Logs from in your workflow's wait container

Error from server (BadRequest): container "wait" in pod "init-fail" is waiting to start: PodInitializing

[terrytangyuan]

I cannot reproduce this. My workflow failed locally. Please check if you are using v3.4.5. Also it would be helpful if you can paste your entire pod via kubectl get pod init-fail -o yaml.

[capacman]

I noticed that it is running correctly when i submit workflow to the same namespace that argo-workflow installed. But if i submit init-fail to different namespace then it stuck. Is there anything i should check?

When i check controller and server pods here are the container statuses so i think it is v.3.4.5. It is installed with helm:

  containerStatuses:
  - containerID: containerd://9a83ecbaf680c6793a0cc10a7f3071ba92b79bfa9bb136bbbfdbb312c6d391c0
    image: quay.io/argoproj/argocli:v3.4.5
    imageID: quay.io/argoproj/argocli@sha256:3370abbce70c6b5c29bbc8b0fcb315f31497d1a5a8fec899184e37c32760c86e
    lastState: {}
    name: argo-server
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-03-22T11:40:58Z"

  containerStatuses:
  - containerID: containerd://473ca739552249ba8d2bb60338c74db2f7f9a3b1a9ef6d50f5aa5efb54aeb769
    image: quay.io/argoproj/workflow-controller:v3.4.5
    imageID: quay.io/argoproj/workflow-controller@sha256:1d7d9691080d18f066a4ccdbd3138eb735d3da1172561afa5762e58e30f9ba85
    lastState: {}
    name: controller
    ready: true
    restartCount: 0
    started: true
    state:
      running:
        startedAt: "2023-03-22T11:40:39Z"

I below output of kubectl get pod init-fail -o yaml -n henkan-argo-test-flows (argo-workflow installed in ns henkan-server):

apiVersion: v1
kind: Pod
metadata:
  annotations:
    kubectl.kubernetes.io/default-container: main
    workflows.argoproj.io/node-id: init-fail
    workflows.argoproj.io/node-name: init-fail
  creationTimestamp: "2023-03-23T07:04:09Z"
  labels:
    workflows.argoproj.io/completed: "false"
    workflows.argoproj.io/workflow: init-fail
  name: init-fail
  namespace: henkan-argo-test-flows
  ownerReferences:
  - apiVersion: argoproj.io/v1alpha1
    blockOwnerDeletion: true
    controller: true
    kind: Workflow
    name: init-fail
    uid: ff17d83d-4e6e-4ab6-aca5-16e4ae49269d
  resourceVersion: "252611173"
  uid: b4191713-7789-4bdc-9182-0feb6268db41
spec:
  containers:
  - command:
    - argoexec
    - wait
    - --loglevel
    - info
    - --log-format
    - text
    env:
    - name: ARGO_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: ARGO_POD_UID
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.uid
    - name: GODEBUG
      value: x509ignoreCN=0
    - name: ARGO_WORKFLOW_NAME
      value: init-fail
    - name: WAIT_CONTAINER_STATUS_CHECK_INTERVAL
      value: 2s
    - name: RESOURCE_STATE_CHECK_INTERVAL
      value: 2s
    - name: ARGO_CONTAINER_NAME
      value: wait
    - name: ARGO_TEMPLATE
      value: '{"name":"init-container-example","inputs":{},"outputs":{},"metadata":{},"container":{"name":"","image":"alpine:latest","command":["echo","bye"],"resources":{},"volumeMounts":[{"name":"foo","mountPath":"/foo"}]},"initContainers":[{"name":"hello","image":"alpine:latest","command":["abcd"],"resources":{},"mirrorVolumeMounts":true}]}'
    - name: ARGO_NODE_ID
      value: init-fail
    - name: ARGO_INCLUDE_SCRIPT_OUTPUT
      value: "false"
    - name: ARGO_DEADLINE
      value: "0001-01-01T00:00:00Z"
    - name: ARGO_PROGRESS_FILE
      value: /var/run/argo/progress
    - name: ARGO_PROGRESS_PATCH_TICK_DURATION
      value: 1m0s
    - name: ARGO_PROGRESS_FILE_TICK_DURATION
      value: 3s
    image: quay.io/argoproj/argoexec:v3.4.5
    imagePullPolicy: IfNotPresent
    name: wait
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /mainctrfs/foo
      name: foo
    - mountPath: /tmp
      name: tmp-dir-argo
      subPath: "0"
    - mountPath: /var/run/argo
      name: var-run-argo
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-q2z6n
      readOnly: true
  - command:
    - /var/run/argo/argoexec
    - emissary
    - --loglevel
    - info
    - --log-format
    - text
    - --
    - echo
    - bye
    env:
    - name: ARGO_CONTAINER_NAME
      value: main
    - name: ARGO_TEMPLATE
      value: '{"name":"init-container-example","inputs":{},"outputs":{},"metadata":{},"container":{"name":"","image":"alpine:latest","command":["echo","bye"],"resources":{},"volumeMounts":[{"name":"foo","mountPath":"/foo"}]},"initContainers":[{"name":"hello","image":"alpine:latest","command":["abcd"],"resources":{},"mirrorVolumeMounts":true}]}'
    - name: ARGO_NODE_ID
      value: init-fail
    - name: ARGO_INCLUDE_SCRIPT_OUTPUT
      value: "false"
    - name: ARGO_DEADLINE
      value: "0001-01-01T00:00:00Z"
    - name: ARGO_PROGRESS_FILE
      value: /var/run/argo/progress
    - name: ARGO_PROGRESS_PATCH_TICK_DURATION
      value: 1m0s
    - name: ARGO_PROGRESS_FILE_TICK_DURATION
      value: 3s
    image: alpine:latest
    imagePullPolicy: Always
    name: main
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /foo
      name: foo
    - mountPath: /var/run/argo
      name: var-run-argo
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-q2z6n
      readOnly: true
  dnsPolicy: ClusterFirst
  enableServiceLinks: true
  initContainers:
  - command:
    - argoexec
    - init
    - --loglevel
    - info
    - --log-format
    - text
    env:
    - name: ARGO_POD_NAME
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.name
    - name: ARGO_POD_UID
      valueFrom:
        fieldRef:
          apiVersion: v1
          fieldPath: metadata.uid
    - name: GODEBUG
      value: x509ignoreCN=0
    - name: ARGO_WORKFLOW_NAME
      value: init-fail
    - name: WAIT_CONTAINER_STATUS_CHECK_INTERVAL
      value: 2s
    - name: RESOURCE_STATE_CHECK_INTERVAL
      value: 2s
    - name: ARGO_CONTAINER_NAME
      value: init
    - name: ARGO_TEMPLATE
      value: '{"name":"init-container-example","inputs":{},"outputs":{},"metadata":{},"container":{"name":"","image":"alpine:latest","command":["echo","bye"],"resources":{},"volumeMounts":[{"name":"foo","mountPath":"/foo"}]},"initContainers":[{"name":"hello","image":"alpine:latest","command":["abcd"],"resources":{},"mirrorVolumeMounts":true}]}'
    - name: ARGO_NODE_ID
      value: init-fail
    - name: ARGO_INCLUDE_SCRIPT_OUTPUT
      value: "false"
    - name: ARGO_DEADLINE
      value: "0001-01-01T00:00:00Z"
    - name: ARGO_PROGRESS_FILE
      value: /var/run/argo/progress
    - name: ARGO_PROGRESS_PATCH_TICK_DURATION
      value: 1m0s
    - name: ARGO_PROGRESS_FILE_TICK_DURATION
      value: 3s
    image: quay.io/argoproj/argoexec:v3.4.5
    imagePullPolicy: IfNotPresent
    name: init
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /var/run/argo
      name: var-run-argo
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-q2z6n
      readOnly: true
  - command:
    - abcd
    env:
    - name: ARGO_CONTAINER_NAME
      value: hello
    - name: ARGO_TEMPLATE
      value: '{"name":"init-container-example","inputs":{},"outputs":{},"metadata":{},"container":{"name":"","image":"alpine:latest","command":["echo","bye"],"resources":{},"volumeMounts":[{"name":"foo","mountPath":"/foo"}]},"initContainers":[{"name":"hello","image":"alpine:latest","command":["abcd"],"resources":{},"mirrorVolumeMounts":true}]}'
    - name: ARGO_NODE_ID
      value: init-fail
    - name: ARGO_INCLUDE_SCRIPT_OUTPUT
      value: "false"
    - name: ARGO_DEADLINE
      value: "0001-01-01T00:00:00Z"
    - name: ARGO_PROGRESS_FILE
      value: /var/run/argo/progress
    - name: ARGO_PROGRESS_PATCH_TICK_DURATION
      value: 1m0s
    - name: ARGO_PROGRESS_FILE_TICK_DURATION
      value: 3s
    image: alpine:latest
    imagePullPolicy: Always
    name: hello
    resources: {}
    terminationMessagePath: /dev/termination-log
    terminationMessagePolicy: File
    volumeMounts:
    - mountPath: /foo
      name: foo
    - mountPath: /var/run/argo
      name: var-run-argo
    - mountPath: /var/run/secrets/kubernetes.io/serviceaccount
      name: kube-api-access-q2z6n
      readOnly: true
  nodeName: henkan-demo-node001
  preemptionPolicy: PreemptLowerPriority
  priority: 0
  restartPolicy: Never
  schedulerName: default-scheduler
  securityContext: {}
  serviceAccount: default
  serviceAccountName: default
  terminationGracePeriodSeconds: 30
  tolerations:
  - effect: NoExecute
    key: node.kubernetes.io/not-ready
    operator: Exists
    tolerationSeconds: 300
  - effect: NoExecute
    key: node.kubernetes.io/unreachable
    operator: Exists
    tolerationSeconds: 300
  volumes:
  - emptyDir: {}
    name: var-run-argo
  - emptyDir: {}
    name: tmp-dir-argo
  - emptyDir: {}
    name: foo
  - name: kube-api-access-q2z6n
    projected:
      defaultMode: 420
      sources:
      - serviceAccountToken:
          expirationSeconds: 3607
          path: token
      - configMap:
          items:
          - key: ca.crt
            path: ca.crt
          name: kube-root-ca.crt
      - downwardAPI:
          items:
          - fieldRef:
              apiVersion: v1
              fieldPath: metadata.namespace
            path: namespace
status:
  conditions:
  - lastProbeTime: null
    lastTransitionTime: "2023-03-23T07:04:09Z"
    message: 'containers with incomplete status: [hello]'
    reason: ContainersNotInitialized
    status: "False"
    type: Initialized
  - lastProbeTime: null
    lastTransitionTime: "2023-03-23T07:04:09Z"
    reason: PodFailed
    status: "False"
    type: Ready
  - lastProbeTime: null
    lastTransitionTime: "2023-03-23T07:04:09Z"
    reason: PodFailed
    status: "False"
    type: ContainersReady
  - lastProbeTime: null
    lastTransitionTime: "2023-03-23T07:04:09Z"
    status: "True"
    type: PodScheduled
  containerStatuses:
  - image: alpine:latest
    imageID: ""
    lastState: {}
    name: main
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        reason: PodInitializing
  - image: quay.io/argoproj/argoexec:v3.4.5
    imageID: ""
    lastState: {}
    name: wait
    ready: false
    restartCount: 0
    started: false
    state:
      waiting:
        reason: PodInitializing
  hostIP: 172.12.2.64
  initContainerStatuses:
  - containerID: containerd://3ab3bee87c7e541196f7e465ebdb16d2e4db9c922310fe9fd61401475da601b2
    image: quay.io/argoproj/argoexec:v3.4.5
    imageID: quay.io/argoproj/argoexec@sha256:3f2ba3b355eccc61ec0e3b5c9fc547f4e9ff5821bf6f08485a6a5fc99bf7731b
    lastState: {}
    name: init
    ready: true
    restartCount: 0
    state:
      terminated:
        containerID: containerd://3ab3bee87c7e541196f7e465ebdb16d2e4db9c922310fe9fd61401475da601b2
        exitCode: 0
        finishedAt: "2023-03-23T07:04:10Z"
        reason: Completed
        startedAt: "2023-03-23T07:04:09Z"
  - containerID: containerd://aac7b8be10511bd71f9e50e01ac8f0ff12d394d6ea52492db8ed7a53c9da7718
    image: docker.io/library/alpine:latest
    imageID: docker.io/library/alpine@sha256:69665d02cb32192e52e07644d76bc6f25abeb5410edc1c7a81a10ba3f0efb90a
    lastState: {}
    name: hello
    ready: false
    restartCount: 0
    state:
      terminated:
        containerID: containerd://aac7b8be10511bd71f9e50e01ac8f0ff12d394d6ea52492db8ed7a53c9da7718
        exitCode: 128
        finishedAt: "2023-03-23T07:04:11Z"
        message: 'failed to create containerd task: failed to create shim task: OCI
          runtime create failed: runc create failed: unable to start container process:
          exec: "abcd": executable file not found in $PATH: unknown'
        reason: StartError
        startedAt: "1970-01-01T00:00:00Z"
  phase: Failed
  podIP: 10.42.0.164
  podIPs:
  - ip: 10.42.0.164
  qosClass: BestEffort
  startTime: "2023-03-23T07:04:09Z"

[capacman]

In the mean time it stucks at running state randomly even if i submit it to the same namespace argo-workflow installed.

Workflows marked as failed has below message in controller logs:
"marking node as failed since init container has non-zero exit code"
but workflows remains at running does not have above log but have :
"Pod failed before main container starts"

[terrytangyuan]

Related: #10045 (comment)

[terrytangyuan]

Can you paste your workflow YAML including the statuses?

[capacman]

below workflow stuck at running. Same workflow transition to failed time to time. So This issue occurs randomly

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"argoproj.io/v1alpha1","kind":"Workflow","metadata":{"annotations":{},"name":"init-fail","namespace":"default"},"spec":{"entrypoint":"init-container-example","templates":[{"container":{"command":["echo","bye"],"image":"alpine:latest","volumeMounts":[{"mountPath":"/foo","name":"foo"}]},"initContainers":[{"command":["abcd"],"image":"alpine:latest","mirrorVolumeMounts":true,"name":"hello"}],"name":"init-container-example"}],"volumes":[{"emptyDir":{},"name":"foo"}]}}
    workflows.argoproj.io/pod-name-format: v2
  creationTimestamp: "2023-03-23T14:34:29Z"
  generation: 4
  labels:
    workflows.argoproj.io/phase: Running
    workflows.henkan.io/submit: "true"
  name: init-fail
  namespace: default
  resourceVersion: "252846199"
  uid: 73c27f8e-ccd9-4053-b7ae-0637e632da2a
spec:
  arguments: {}
  entrypoint: init-container-example
  podGC:
    strategy: OnWorkflowCompletion
  templates:
  - container:
      command:
      - echo
      - bye
      image: alpine:latest
      name: ""
      resources: {}
      volumeMounts:
      - mountPath: /foo
        name: foo
    initContainers:
    - command:
      - abcd
      image: alpine:latest
      mirrorVolumeMounts: true
      name: hello
      resources: {}
    inputs: {}
    metadata: {}
    name: init-container-example
    outputs: {}
  ttlStrategy:
    secondsAfterCompletion: 15
  volumes:
  - emptyDir: {}
    name: foo
status:
  artifactGCStatus:
    notSpecified: true
  artifactRepositoryRef:
    artifactRepository: {}
    default: true
  conditions:
  - status: "False"
    type: PodRunning
  finishedAt: null
  nodes:
    init-fail:
      displayName: init-fail
      finishedAt: null
      hostNodeName: henkan-demo-node002
      id: init-fail
      message: Pod failed before main container starts
      name: init-fail
      phase: Pending
      progress: 0/1
      startedAt: "2023-03-23T14:34:29Z"
      templateName: init-container-example
      templateScope: local/init-fail
      type: Pod
  phase: Running
  progress: 0/1
  startedAt: "2023-03-23T14:34:29Z"

[terrytangyuan]

Thanks should be fixed by #10740