Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

DAG outputs from task parameter don't work with a retry strategy #14228

Open
3 of 4 tasks
elliotgunton opened this issue Feb 26, 2025 · 1 comment · May be fixed by #14299
Open
3 of 4 tasks

DAG outputs from task parameter don't work with a retry strategy #14228

elliotgunton opened this issue Feb 26, 2025 · 1 comment · May be fixed by #14299
Labels
area/retryStrategy Template-level retryStrategy type/bug

Comments

@elliotgunton
Copy link
Contributor

elliotgunton commented Feb 26, 2025 •
edited
Loading

Pre-requisites

  • I have double-checked my configuration
  • I have tested with the :latest image tag (i.e. quay.io/argoproj/workflow-controller:latest) and can confirm the issue still exists on :latest. If not, I have explained why, in detail, in my description below.
  • I have searched existing issues and could not find a match for this bug
  • I'd like to contribute the fix myself (see contributing guide)

What happened? What did you expect to happen?

  • With a retryStrategy on the Workflow spec, define a DAG with:
    • a "fanout" task that uses withItems, the template should have an output parameter
    • an output parameter from the DAG, that takes the output parameter from the fanout task using valueFrom
  • The task will succeed (so no retry occurs for the tasks themselves)
  • The DAG node will fail with Unable to resolve: "tasks.echo-list.outputs.parameters.dummy-output"
  • Then the DAG node will retry until retries are exhausted and the workflow will fail

If you remove the retryStrategy from the spec, the DAG node succeeds first time and the workflow completes successfully. Note the problem also doesn't occur when using steps.

Looks like a similar root cause (retry strategy) as #13691

Further notes:

  • The bug occurs with retryStrategy set either in the workflow or the workflow controller configmap workflowDefaults
  • If the workflow controller config map sets a retryStrategy, there doesn't seem to be a way to remove retryStrategy on the workflow, as argo will add the value from the configmap even if the workflow has retryStrategy: {}
    • Also note that just the presence of an empty retryStrategy: {} will break the workflow (even if it's not set in the config map)

Version(s)

v3.6.4

Paste a minimal workflow that reproduces the issue. We must be able to run the workflow; don't enter a workflow that uses private images.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
  generateName: fanout-dag-workflow-
spec:
  retryStrategy:
    retryPolicy: OnError
    limit: 1
  entrypoint: fanout-dag-with-output
  templates:
  - name: echo-value
    container:
      args:
      - '{{inputs.parameters.message}}'
      command:
      - echo
      image: busybox
    inputs:
      parameters:
      - name: message
    outputs:
      parameters:
      - name: dummy-output
        value: test
  - name: fanout-dag-with-output
    dag:
      tasks:
      - name: echo-list
        template: echo-value
        arguments:
          parameters:
          - name: message
            value: '{{item}}'
        withItems:
        - 1
        - 2
        - 3
    outputs:
      parameters:
      - name: dummy-dag-output
        valueFrom:
          parameter: '{{tasks.echo-list.outputs.parameters.dummy-output}}'

Logs from the workflow controller

time="2025-02-26T11:19:12.697Z" level=info msg="Processing workflow" Phase= ResourceVersion=3600 namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.700Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.700Z" level=info msg="Updated phase  -> Running" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.700Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.700Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.700Z" level=info msg="Retry node fanout-dag-workflow-qpdvj initialized Running" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.701Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.701Z" level=info msg="DAG node fanout-dag-workflow-qpdvj-117315324 initialized Running" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.701Z" level=warning msg="was unable to obtain the node for fanout-dag-workflow-qpdvj-2007032986, taskName echo-list"
time="2025-02-26T11:19:12.701Z" level=info msg="TaskGroup node fanout-dag-workflow-qpdvj-2007032986 initialized Running (message: )" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.701Z" level=warning msg="was unable to obtain the node for fanout-dag-workflow-qpdvj-3726547270, taskName echo-list(0:1)"
time="2025-02-26T11:19:12.701Z" level=info msg="All of node fanout-dag-workflow-qpdvj(0).echo-list(0:1) dependencies [] completed" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.701Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.701Z" level=info msg="Retry node fanout-dag-workflow-qpdvj-3726547270 initialized Running" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.701Z" level=info msg="Pod node fanout-dag-workflow-qpdvj-1844115781 initialized Pending" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.707Z" level=info msg="Created pod: fanout-dag-workflow-qpdvj(0).echo-list(0:1)(0) (fanout-dag-workflow-qpdvj-echo-value-1844115781)" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.707Z" level=warning msg="was unable to obtain the node for fanout-dag-workflow-qpdvj-639373036, taskName echo-list(1:2)"
time="2025-02-26T11:19:12.707Z" level=info msg="All of node fanout-dag-workflow-qpdvj(0).echo-list(1:2) dependencies [] completed" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.707Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.707Z" level=info msg="Retry node fanout-dag-workflow-qpdvj-639373036 initialized Running" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.707Z" level=info msg="Pod node fanout-dag-workflow-qpdvj-2384080615 initialized Pending" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.711Z" level=info msg="Created pod: fanout-dag-workflow-qpdvj(0).echo-list(1:2)(0) (fanout-dag-workflow-qpdvj-echo-value-2384080615)" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.711Z" level=warning msg="was unable to obtain the node for fanout-dag-workflow-qpdvj-633535614, taskName echo-list(2:3)"
time="2025-02-26T11:19:12.711Z" level=info msg="All of node fanout-dag-workflow-qpdvj(0).echo-list(2:3) dependencies [] completed" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.711Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.711Z" level=info msg="Retry node fanout-dag-workflow-qpdvj-633535614 initialized Running" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.711Z" level=info msg="Pod node fanout-dag-workflow-qpdvj-3098323245 initialized Pending" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.717Z" level=info msg="Created pod: fanout-dag-workflow-qpdvj(0).echo-list(2:3)(0) (fanout-dag-workflow-qpdvj-echo-value-3098323245)" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.717Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.717Z" level=info msg=reconcileAgentPod namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.724Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=3610 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.724Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=3610 namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.724Z" level=info msg="Task-result reconciliation" namespace=default numObjs=0 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.725Z" level=info msg="node unchanged" namespace=default nodeID=fanout-dag-workflow-qpdvj-3098323245 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.725Z" level=info msg="node changed" namespace=default new.message= new.phase=Pending new.progress=0/1 nodeID=fanout-dag-workflow-qpdvj-1844115781 old.message= old.phase=Pending old.progress=0/1 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.725Z" level=info msg="node changed" namespace=default new.message= new.phase=Pending new.progress=0/1 nodeID=fanout-dag-workflow-qpdvj-2384080615 old.message= old.phase=Pending old.progress=0/1 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.726Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.727Z" level=info msg=reconcileAgentPod namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:12.732Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=3615 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.712Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=3615 namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.712Z" level=info msg="Task-result reconciliation" namespace=default numObjs=3 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.712Z" level=info msg="task-result changed" namespace=default nodeID=fanout-dag-workflow-qpdvj-2384080615 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.713Z" level=info msg="task-result changed" namespace=default nodeID=fanout-dag-workflow-qpdvj-1844115781 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.713Z" level=info msg="task-result changed" namespace=default nodeID=fanout-dag-workflow-qpdvj-3098323245 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.714Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=fanout-dag-workflow-qpdvj-3098323245 old.message= old.phase=Pending old.progress=0/1 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.714Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=fanout-dag-workflow-qpdvj-1844115781 old.message= old.phase=Pending old.progress=0/1 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.714Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=fanout-dag-workflow-qpdvj-2384080615 old.message= old.phase=Pending old.progress=0/1 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.717Z" level=info msg="node fanout-dag-workflow-qpdvj-3726547270 phase Running -> Succeeded" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.717Z" level=info msg="node fanout-dag-workflow-qpdvj-3726547270 finished: 2025-02-26 11:19:22.717867763 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.718Z" level=info msg="node fanout-dag-workflow-qpdvj-639373036 phase Running -> Succeeded" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.718Z" level=info msg="node fanout-dag-workflow-qpdvj-639373036 finished: 2025-02-26 11:19:22.718070346 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.718Z" level=info msg="node fanout-dag-workflow-qpdvj-633535614 phase Running -> Succeeded" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.718Z" level=info msg="node fanout-dag-workflow-qpdvj-633535614 finished: 2025-02-26 11:19:22.718231638 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.718Z" level=info msg="node fanout-dag-workflow-qpdvj-2007032986 phase Running -> Succeeded" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.718Z" level=info msg="node fanout-dag-workflow-qpdvj-2007032986 finished: 2025-02-26 11:19:22.718291929 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.718Z" level=error msg="unable to get outputs" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.718Z" level=error msg="Mark error node" error="Unable to resolve: \"tasks.echo-list.outputs.parameters.dummy-output\"" namespace=default nodeName="fanout-dag-workflow-qpdvj(0)" workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.718Z" level=info msg="node fanout-dag-workflow-qpdvj-117315324 phase Running -> Error" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.718Z" level=info msg="node fanout-dag-workflow-qpdvj-117315324 message: Unable to resolve: \"tasks.echo-list.outputs.parameters.dummy-output\"" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.718Z" level=info msg="node fanout-dag-workflow-qpdvj-117315324 finished: 2025-02-26 11:19:22.718627346 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.718Z" level=info msg="Retry Policy: OnError (onFailed: false, onError true)" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.718Z" level=info msg="1 child nodes of fanout-dag-workflow-qpdvj failed. Trying again..." namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.719Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.719Z" level=info msg="DAG node fanout-dag-workflow-qpdvj-3808141241 initialized Running" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.719Z" level=warning msg="was unable to obtain the node for fanout-dag-workflow-qpdvj-1874910499, taskName echo-list"
time="2025-02-26T11:19:22.719Z" level=info msg="TaskGroup node fanout-dag-workflow-qpdvj-1874910499 initialized Running (message: )" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.719Z" level=warning msg="was unable to obtain the node for fanout-dag-workflow-qpdvj-3575878853, taskName echo-list(0:1)"
time="2025-02-26T11:19:22.719Z" level=info msg="All of node fanout-dag-workflow-qpdvj(1).echo-list(0:1) dependencies [] completed" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.719Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.719Z" level=info msg="Retry node fanout-dag-workflow-qpdvj-3575878853 initialized Running" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.719Z" level=info msg="Pod node fanout-dag-workflow-qpdvj-3433289636 initialized Pending" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.732Z" level=info msg="Created pod: fanout-dag-workflow-qpdvj(1).echo-list(0:1)(0) (fanout-dag-workflow-qpdvj-echo-value-3433289636)" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.732Z" level=warning msg="was unable to obtain the node for fanout-dag-workflow-qpdvj-2271645639, taskName echo-list(1:2)"
time="2025-02-26T11:19:22.732Z" level=info msg="All of node fanout-dag-workflow-qpdvj(1).echo-list(1:2) dependencies [] completed" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.732Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.732Z" level=info msg="Retry node fanout-dag-workflow-qpdvj-2271645639 initialized Running" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.733Z" level=info msg="Pod node fanout-dag-workflow-qpdvj-1525886202 initialized Pending" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.744Z" level=info msg="Created pod: fanout-dag-workflow-qpdvj(1).echo-list(1:2)(0) (fanout-dag-workflow-qpdvj-echo-value-1525886202)" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.744Z" level=warning msg="was unable to obtain the node for fanout-dag-workflow-qpdvj-2293048533, taskName echo-list(2:3)"
time="2025-02-26T11:19:22.744Z" level=info msg="All of node fanout-dag-workflow-qpdvj(1).echo-list(2:3) dependencies [] completed" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.744Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.744Z" level=info msg="Retry node fanout-dag-workflow-qpdvj-2293048533 initialized Running" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.744Z" level=info msg="Pod node fanout-dag-workflow-qpdvj-3831230004 initialized Pending" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.755Z" level=info msg="Created pod: fanout-dag-workflow-qpdvj(1).echo-list(2:3)(0) (fanout-dag-workflow-qpdvj-echo-value-3831230004)" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.755Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.755Z" level=info msg=reconcileAgentPod namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.771Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=3698 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:22.778Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/fanout-dag-workflow-qpdvj-echo-value-2384080615/labelPodCompleted
time="2025-02-26T11:19:22.778Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/fanout-dag-workflow-qpdvj-echo-value-1844115781/labelPodCompleted
time="2025-02-26T11:19:22.778Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/fanout-dag-workflow-qpdvj-echo-value-3098323245/labelPodCompleted
time="2025-02-26T11:19:32.736Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=3698 namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.736Z" level=info msg="Task-result reconciliation" namespace=default numObjs=6 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.737Z" level=info msg="task-result changed" namespace=default nodeID=fanout-dag-workflow-qpdvj-3433289636 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.737Z" level=info msg="task-result changed" namespace=default nodeID=fanout-dag-workflow-qpdvj-3831230004 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.737Z" level=info msg="task-result changed" namespace=default nodeID=fanout-dag-workflow-qpdvj-1525886202 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.737Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=fanout-dag-workflow-qpdvj-3831230004 old.message= old.phase=Pending old.progress=0/1 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.737Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=fanout-dag-workflow-qpdvj-3433289636 old.message= old.phase=Pending old.progress=0/1 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.737Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=fanout-dag-workflow-qpdvj-1525886202 old.message= old.phase=Pending old.progress=0/1 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.745Z" level=info msg="node fanout-dag-workflow-qpdvj-3575878853 phase Running -> Succeeded" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.745Z" level=info msg="node fanout-dag-workflow-qpdvj-3575878853 finished: 2025-02-26 11:19:32.745818711 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.745Z" level=info msg="node fanout-dag-workflow-qpdvj-2271645639 phase Running -> Succeeded" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.745Z" level=info msg="node fanout-dag-workflow-qpdvj-2271645639 finished: 2025-02-26 11:19:32.745986003 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.746Z" level=info msg="node fanout-dag-workflow-qpdvj-2293048533 phase Running -> Succeeded" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.746Z" level=info msg="node fanout-dag-workflow-qpdvj-2293048533 finished: 2025-02-26 11:19:32.746124003 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.746Z" level=info msg="node fanout-dag-workflow-qpdvj-1874910499 phase Running -> Succeeded" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.746Z" level=info msg="node fanout-dag-workflow-qpdvj-1874910499 finished: 2025-02-26 11:19:32.746153211 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.746Z" level=error msg="unable to get outputs" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.746Z" level=error msg="Mark error node" error="Unable to resolve: \"tasks.echo-list.outputs.parameters.dummy-output\"" namespace=default nodeName="fanout-dag-workflow-qpdvj(1)" workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.746Z" level=info msg="node fanout-dag-workflow-qpdvj-3808141241 phase Running -> Error" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.746Z" level=info msg="node fanout-dag-workflow-qpdvj-3808141241 message: Unable to resolve: \"tasks.echo-list.outputs.parameters.dummy-output\"" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.746Z" level=info msg="node fanout-dag-workflow-qpdvj-3808141241 finished: 2025-02-26 11:19:32.746266711 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.746Z" level=info msg="Retry Policy: OnError (onFailed: false, onError true)" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.746Z" level=info msg="2 child nodes of fanout-dag-workflow-qpdvj failed. Trying again..." namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.746Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.746Z" level=info msg="DAG node fanout-dag-workflow-qpdvj-184131610 initialized Running" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.746Z" level=warning msg="was unable to obtain the node for fanout-dag-workflow-qpdvj-756893016, taskName echo-list"
time="2025-02-26T11:19:32.746Z" level=info msg="TaskGroup node fanout-dag-workflow-qpdvj-756893016 initialized Running (message: )" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.746Z" level=warning msg="was unable to obtain the node for fanout-dag-workflow-qpdvj-3453266348, taskName echo-list(0:1)"
time="2025-02-26T11:19:32.746Z" level=info msg="All of node fanout-dag-workflow-qpdvj(2).echo-list(0:1) dependencies [] completed" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.746Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.748Z" level=info msg="Retry node fanout-dag-workflow-qpdvj-3453266348 initialized Running" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.748Z" level=info msg="Pod node fanout-dag-workflow-qpdvj-1306772775 initialized Pending" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.760Z" level=info msg="Created pod: fanout-dag-workflow-qpdvj(2).echo-list(0:1)(0) (fanout-dag-workflow-qpdvj-echo-value-1306772775)" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.761Z" level=warning msg="was unable to obtain the node for fanout-dag-workflow-qpdvj-3548922614, taskName echo-list(1:2)"
time="2025-02-26T11:19:32.761Z" level=info msg="All of node fanout-dag-workflow-qpdvj(2).echo-list(1:2) dependencies [] completed" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.761Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.761Z" level=info msg="Retry node fanout-dag-workflow-qpdvj-3548922614 initialized Running" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.761Z" level=info msg="Pod node fanout-dag-workflow-qpdvj-3484000245 initialized Pending" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.769Z" level=info msg="Created pod: fanout-dag-workflow-qpdvj(2).echo-list(1:2)(0) (fanout-dag-workflow-qpdvj-echo-value-3484000245)" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.769Z" level=warning msg="was unable to obtain the node for fanout-dag-workflow-qpdvj-3354717764, taskName echo-list(2:3)"
time="2025-02-26T11:19:32.769Z" level=info msg="All of node fanout-dag-workflow-qpdvj(2).echo-list(2:3) dependencies [] completed" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.769Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.769Z" level=info msg="Retry node fanout-dag-workflow-qpdvj-3354717764 initialized Running" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.769Z" level=info msg="Pod node fanout-dag-workflow-qpdvj-1844869295 initialized Pending" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.782Z" level=info msg="Created pod: fanout-dag-workflow-qpdvj(2).echo-list(2:3)(0) (fanout-dag-workflow-qpdvj-echo-value-1844869295)" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.782Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.782Z" level=info msg=reconcileAgentPod namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.799Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=3798 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:32.807Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/fanout-dag-workflow-qpdvj-echo-value-1525886202/labelPodCompleted
time="2025-02-26T11:19:32.807Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/fanout-dag-workflow-qpdvj-echo-value-3831230004/labelPodCompleted
time="2025-02-26T11:19:32.808Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/fanout-dag-workflow-qpdvj-echo-value-3433289636/labelPodCompleted
time="2025-02-26T11:19:42.763Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=3798 namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.764Z" level=info msg="Task-result reconciliation" namespace=default numObjs=9 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.764Z" level=info msg="task-result changed" namespace=default nodeID=fanout-dag-workflow-qpdvj-1306772775 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.764Z" level=info msg="task-result changed" namespace=default nodeID=fanout-dag-workflow-qpdvj-1844869295 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.764Z" level=info msg="task-result changed" namespace=default nodeID=fanout-dag-workflow-qpdvj-3484000245 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.764Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=fanout-dag-workflow-qpdvj-1844869295 old.message= old.phase=Pending old.progress=0/1 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.764Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=fanout-dag-workflow-qpdvj-1306772775 old.message= old.phase=Pending old.progress=0/1 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.764Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=fanout-dag-workflow-qpdvj-3484000245 old.message= old.phase=Pending old.progress=0/1 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.765Z" level=info msg="node fanout-dag-workflow-qpdvj-3453266348 phase Running -> Succeeded" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.765Z" level=info msg="node fanout-dag-workflow-qpdvj-3453266348 finished: 2025-02-26 11:19:42.765759383 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.765Z" level=info msg="node fanout-dag-workflow-qpdvj-3548922614 phase Running -> Succeeded" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.765Z" level=info msg="node fanout-dag-workflow-qpdvj-3548922614 finished: 2025-02-26 11:19:42.765955091 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.766Z" level=info msg="node fanout-dag-workflow-qpdvj-3354717764 phase Running -> Succeeded" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.766Z" level=info msg="node fanout-dag-workflow-qpdvj-3354717764 finished: 2025-02-26 11:19:42.766074341 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.766Z" level=info msg="node fanout-dag-workflow-qpdvj-756893016 phase Running -> Succeeded" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.766Z" level=info msg="node fanout-dag-workflow-qpdvj-756893016 finished: 2025-02-26 11:19:42.766107175 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.766Z" level=error msg="unable to get outputs" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.766Z" level=error msg="Mark error node" error="Unable to resolve: \"tasks.echo-list.outputs.parameters.dummy-output\"" namespace=default nodeName="fanout-dag-workflow-qpdvj(2)" workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.766Z" level=info msg="node fanout-dag-workflow-qpdvj-184131610 phase Running -> Error" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.766Z" level=info msg="node fanout-dag-workflow-qpdvj-184131610 message: Unable to resolve: \"tasks.echo-list.outputs.parameters.dummy-output\"" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.766Z" level=info msg="node fanout-dag-workflow-qpdvj-184131610 finished: 2025-02-26 11:19:42.76625555 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.766Z" level=info msg="Retry Policy: OnError (onFailed: false, onError true)" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.766Z" level=info msg="3 child nodes of fanout-dag-workflow-qpdvj failed. Trying again..." namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.766Z" level=info msg="was unable to obtain node for , letting display name to be nodeName" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.766Z" level=info msg="DAG node fanout-dag-workflow-qpdvj-3874957527 initialized Running" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.766Z" level=warning msg="was unable to obtain the node for fanout-dag-workflow-qpdvj-2383922441, taskName echo-list"
time="2025-02-26T11:19:42.766Z" level=info msg="TaskGroup node fanout-dag-workflow-qpdvj-2383922441 initialized Running (message: )" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.766Z" level=warning msg="was unable to obtain the node for fanout-dag-workflow-qpdvj-4158612323, taskName echo-list(0:1)"
time="2025-02-26T11:19:42.766Z" level=info msg="All of node fanout-dag-workflow-qpdvj(3).echo-list(0:1) dependencies [] completed" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.766Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.767Z" level=info msg="Retry node fanout-dag-workflow-qpdvj-4158612323 initialized Running" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.767Z" level=info msg="Pod node fanout-dag-workflow-qpdvj-2003371822 initialized Pending" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.778Z" level=info msg="Created pod: fanout-dag-workflow-qpdvj(3).echo-list(0:1)(0) (fanout-dag-workflow-qpdvj-echo-value-2003371822)" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.778Z" level=warning msg="was unable to obtain the node for fanout-dag-workflow-qpdvj-2046826465, taskName echo-list(1:2)"
time="2025-02-26T11:19:42.778Z" level=info msg="All of node fanout-dag-workflow-qpdvj(3).echo-list(1:2) dependencies [] completed" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.779Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.780Z" level=info msg="Retry node fanout-dag-workflow-qpdvj-2046826465 initialized Running" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.781Z" level=info msg="Pod node fanout-dag-workflow-qpdvj-2719422904 initialized Pending" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.790Z" level=info msg="Created pod: fanout-dag-workflow-qpdvj(3).echo-list(1:2)(0) (fanout-dag-workflow-qpdvj-echo-value-2719422904)" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.790Z" level=warning msg="was unable to obtain the node for fanout-dag-workflow-qpdvj-3102518611, taskName echo-list(2:3)"
time="2025-02-26T11:19:42.790Z" level=info msg="All of node fanout-dag-workflow-qpdvj(3).echo-list(2:3) dependencies [] completed" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.790Z" level=warning msg="Node was nil, will be initialized as type Skipped" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.791Z" level=info msg="Retry node fanout-dag-workflow-qpdvj-3102518611 initialized Running" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.791Z" level=info msg="Pod node fanout-dag-workflow-qpdvj-402549470 initialized Pending" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.796Z" level=info msg="Created pod: fanout-dag-workflow-qpdvj(3).echo-list(2:3)(0) (fanout-dag-workflow-qpdvj-echo-value-402549470)" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.796Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.796Z" level=info msg=reconcileAgentPod namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.808Z" level=info msg="Workflow update successful" namespace=default phase=Running resourceVersion=3899 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:42.814Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/fanout-dag-workflow-qpdvj-echo-value-3484000245/labelPodCompleted
time="2025-02-26T11:19:42.814Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/fanout-dag-workflow-qpdvj-echo-value-1844869295/labelPodCompleted
time="2025-02-26T11:19:42.814Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/fanout-dag-workflow-qpdvj-echo-value-1306772775/labelPodCompleted
time="2025-02-26T11:19:52.785Z" level=info msg="Processing workflow" Phase=Running ResourceVersion=3899 namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.785Z" level=info msg="Task-result reconciliation" namespace=default numObjs=12 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.785Z" level=info msg="task-result changed" namespace=default nodeID=fanout-dag-workflow-qpdvj-2003371822 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.785Z" level=info msg="task-result changed" namespace=default nodeID=fanout-dag-workflow-qpdvj-2719422904 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.785Z" level=info msg="task-result changed" namespace=default nodeID=fanout-dag-workflow-qpdvj-402549470 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.786Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=fanout-dag-workflow-qpdvj-402549470 old.message= old.phase=Pending old.progress=0/1 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.786Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=fanout-dag-workflow-qpdvj-2003371822 old.message= old.phase=Pending old.progress=0/1 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.786Z" level=info msg="node changed" namespace=default new.message= new.phase=Succeeded new.progress=0/1 nodeID=fanout-dag-workflow-qpdvj-2719422904 old.message= old.phase=Pending old.progress=0/1 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.786Z" level=info msg="node fanout-dag-workflow-qpdvj-4158612323 phase Running -> Succeeded" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.787Z" level=info msg="node fanout-dag-workflow-qpdvj-4158612323 finished: 2025-02-26 11:19:52.787090429 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.787Z" level=info msg="node fanout-dag-workflow-qpdvj-2046826465 phase Running -> Succeeded" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.787Z" level=info msg="node fanout-dag-workflow-qpdvj-2046826465 finished: 2025-02-26 11:19:52.787255763 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.787Z" level=info msg="node fanout-dag-workflow-qpdvj-3102518611 phase Running -> Succeeded" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.787Z" level=info msg="node fanout-dag-workflow-qpdvj-3102518611 finished: 2025-02-26 11:19:52.787592929 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.787Z" level=info msg="node fanout-dag-workflow-qpdvj-2383922441 phase Running -> Succeeded" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.787Z" level=info msg="node fanout-dag-workflow-qpdvj-2383922441 finished: 2025-02-26 11:19:52.787642471 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.787Z" level=error msg="unable to get outputs" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.787Z" level=error msg="Mark error node" error="Unable to resolve: \"tasks.echo-list.outputs.parameters.dummy-output\"" namespace=default nodeName="fanout-dag-workflow-qpdvj(3)" workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.787Z" level=info msg="node fanout-dag-workflow-qpdvj-3874957527 phase Running -> Error" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.787Z" level=info msg="node fanout-dag-workflow-qpdvj-3874957527 message: Unable to resolve: \"tasks.echo-list.outputs.parameters.dummy-output\"" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.787Z" level=info msg="node fanout-dag-workflow-qpdvj-3874957527 finished: 2025-02-26 11:19:52.787867013 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.788Z" level=info msg="Retry Policy: OnError (onFailed: false, onError true)" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.788Z" level=info msg="No more retries left. Failing..." namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.788Z" level=info msg="node fanout-dag-workflow-qpdvj phase Running -> Error" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.788Z" level=info msg="node fanout-dag-workflow-qpdvj message: No more retries left" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.788Z" level=info msg="node fanout-dag-workflow-qpdvj finished: 2025-02-26 11:19:52.788114596 +0000 UTC" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.788Z" level=info msg="TaskSet Reconciliation" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.788Z" level=info msg=reconcileAgentPod namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.788Z" level=info msg="Updated phase Running -> Error" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.788Z" level=info msg="Updated message  -> No more retries left" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.788Z" level=info msg="Marking workflow completed" namespace=default workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.805Z" level=info msg="Workflow update successful" namespace=default phase=Error resourceVersion=3990 workflow=fanout-dag-workflow-qpdvj
time="2025-02-26T11:19:52.850Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/fanout-dag-workflow-qpdvj-echo-value-2003371822/labelPodCompleted
time="2025-02-26T11:19:52.850Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/fanout-dag-workflow-qpdvj-echo-value-2719422904/labelPodCompleted
time="2025-02-26T11:19:52.850Z" level=info msg="cleaning up pod" action=labelPodCompleted key=default/fanout-dag-workflow-qpdvj-echo-value-402549470/labelPodCompleted

Logs from in your workflow's wait container

time="2025-02-26T11:19:24.109Z" level=info msg="Starting Workflow Executor" version=v3.6.4
time="2025-02-26T11:19:24.110Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2025-02-26T11:19:24.110Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=default podName=fanout-dag-workflow-qpdvj-echo-value-1525886202 templateName=echo-value version="&Version{Version:v3.6.4,BuildDate:2025-02-10T11:50:49Z,GitCommit:77552d852da5c53a1d8c9918b3474d8bff06260a,GitTag:v3.6.4,GitTreeState:clean,GoVersion:go1.23.4,Compiler:gc,Platform:linux/arm64,}"
time="2025-02-26T11:19:24.115Z" level=info msg="Starting deadline monitor"
time="2025-02-26T11:19:28.120Z" level=info msg="Main container completed" error="<nil>"
time="2025-02-26T11:19:28.120Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2025-02-26T11:19:28.120Z" level=info msg="Saving output parameters"
time="2025-02-26T11:19:28.120Z" level=info msg="Saving path output parameter: dummy-output"
time="2025-02-26T11:19:28.120Z" level=info msg="No output artifacts"
time="2025-02-26T11:19:28.135Z" level=info msg="Alloc=17242 TotalAlloc=21036 Sys=31317 NumGC=3 Goroutines=8"
time="2025-02-26T11:19:14.440Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=default podName=fanout-dag-workflow-qpdvj-echo-value-1844115781 templateName=echo-value version="&Version{Version:v3.6.4,BuildDate:2025-02-10T11:50:49Z,GitCommit:77552d852da5c53a1d8c9918b3474d8bff06260a,GitTag:v3.6.4,GitTreeState:clean,GoVersion:go1.23.4,Compiler:gc,Platform:linux/arm64,}"
time="2025-02-26T11:19:14.447Z" level=info msg="Starting deadline monitor"
time="2025-02-26T11:19:19.456Z" level=info msg="Main container completed" error="<nil>"
time="2025-02-26T11:19:19.457Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2025-02-26T11:19:19.457Z" level=info msg="Saving output parameters"
time="2025-02-26T11:19:19.457Z" level=info msg="Saving path output parameter: dummy-output"
time="2025-02-26T11:19:19.457Z" level=info msg="No output artifacts"
time="2025-02-26T11:19:19.469Z" level=info msg="Alloc=16794 TotalAlloc=21078 Sys=31061 NumGC=3 Goroutines=8"
time="2025-02-26T11:19:19.473Z" level=info msg="Deadline monitor stopped"
time="2025-02-26T11:19:19.473Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2025-02-26T11:19:44.347Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2025-02-26T11:19:44.347Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=default podName=fanout-dag-workflow-qpdvj-echo-value-2003371822 templateName=echo-value version="&Version{Version:v3.6.4,BuildDate:2025-02-10T11:50:49Z,GitCommit:77552d852da5c53a1d8c9918b3474d8bff06260a,GitTag:v3.6.4,GitTreeState:clean,GoVersion:go1.23.4,Compiler:gc,Platform:linux/arm64,}"
time="2025-02-26T11:19:44.351Z" level=info msg="Starting deadline monitor"
time="2025-02-26T11:19:48.357Z" level=info msg="Main container completed" error="<nil>"
time="2025-02-26T11:19:48.357Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2025-02-26T11:19:48.357Z" level=info msg="Saving output parameters"
time="2025-02-26T11:19:48.357Z" level=info msg="Saving path output parameter: dummy-output"
time="2025-02-26T11:19:48.357Z" level=info msg="No output artifacts"
time="2025-02-26T11:19:48.370Z" level=info msg="Alloc=16711 TotalAlloc=21020 Sys=26965 NumGC=3 Goroutines=8"
time="2025-02-26T11:19:48.374Z" level=info msg="Deadline monitor stopped"
time="2025-02-26T11:19:44.330Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2025-02-26T11:19:44.332Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=default podName=fanout-dag-workflow-qpdvj-echo-value-2719422904 templateName=echo-value version="&Version{Version:v3.6.4,BuildDate:2025-02-10T11:50:49Z,GitCommit:77552d852da5c53a1d8c9918b3474d8bff06260a,GitTag:v3.6.4,GitTreeState:clean,GoVersion:go1.23.4,Compiler:gc,Platform:linux/arm64,}"
time="2025-02-26T11:19:44.341Z" level=info msg="Starting deadline monitor"
time="2025-02-26T11:19:49.349Z" level=info msg="Main container completed" error="<nil>"
time="2025-02-26T11:19:49.349Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2025-02-26T11:19:49.349Z" level=info msg="Saving output parameters"
time="2025-02-26T11:19:49.349Z" level=info msg="Saving path output parameter: dummy-output"
time="2025-02-26T11:19:49.349Z" level=info msg="No output artifacts"
time="2025-02-26T11:19:49.356Z" level=info msg="Alloc=14633 TotalAlloc=21064 Sys=26965 NumGC=3 Goroutines=8"
time="2025-02-26T11:19:49.359Z" level=info msg="Deadline monitor stopped"
time="2025-02-26T11:19:14.442Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=default podName=fanout-dag-workflow-qpdvj-echo-value-3098323245 templateName=echo-value version="&Version{Version:v3.6.4,BuildDate:2025-02-10T11:50:49Z,GitCommit:77552d852da5c53a1d8c9918b3474d8bff06260a,GitTag:v3.6.4,GitTreeState:clean,GoVersion:go1.23.4,Compiler:gc,Platform:linux/arm64,}"
time="2025-02-26T11:19:14.447Z" level=info msg="Starting deadline monitor"
time="2025-02-26T11:19:18.454Z" level=info msg="Main container completed" error="<nil>"
time="2025-02-26T11:19:18.454Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2025-02-26T11:19:18.454Z" level=info msg="Saving output parameters"
time="2025-02-26T11:19:18.454Z" level=info msg="Saving path output parameter: dummy-output"
time="2025-02-26T11:19:18.454Z" level=info msg="No output artifacts"
time="2025-02-26T11:19:18.468Z" level=info msg="Alloc=16697 TotalAlloc=21035 Sys=30805 NumGC=3 Goroutines=8"
time="2025-02-26T11:19:18.472Z" level=info msg="stopping progress monitor (context done)" error="context canceled"
time="2025-02-26T11:19:18.472Z" level=info msg="Deadline monitor stopped"
time="2025-02-26T11:19:34.725Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2025-02-26T11:19:34.725Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=default podName=fanout-dag-workflow-qpdvj-echo-value-3484000245 templateName=echo-value version="&Version{Version:v3.6.4,BuildDate:2025-02-10T11:50:49Z,GitCommit:77552d852da5c53a1d8c9918b3474d8bff06260a,GitTag:v3.6.4,GitTreeState:clean,GoVersion:go1.23.4,Compiler:gc,Platform:linux/arm64,}"
time="2025-02-26T11:19:34.735Z" level=info msg="Starting deadline monitor"
time="2025-02-26T11:19:39.742Z" level=info msg="Main container completed" error="<nil>"
time="2025-02-26T11:19:39.742Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2025-02-26T11:19:39.742Z" level=info msg="Saving output parameters"
time="2025-02-26T11:19:39.742Z" level=info msg="Saving path output parameter: dummy-output"
time="2025-02-26T11:19:39.742Z" level=info msg="No output artifacts"
time="2025-02-26T11:19:39.755Z" level=info msg="Alloc=16439 TotalAlloc=21087 Sys=26965 NumGC=3 Goroutines=8"
time="2025-02-26T11:19:39.760Z" level=info msg="Deadline monitor stopped"
time="2025-02-26T11:19:24.106Z" level=info msg="Starting Workflow Executor" version=v3.6.4
time="2025-02-26T11:19:24.108Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2025-02-26T11:19:24.108Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=default podName=fanout-dag-workflow-qpdvj-echo-value-3831230004 templateName=echo-value version="&Version{Version:v3.6.4,BuildDate:2025-02-10T11:50:49Z,GitCommit:77552d852da5c53a1d8c9918b3474d8bff06260a,GitTag:v3.6.4,GitTreeState:clean,GoVersion:go1.23.4,Compiler:gc,Platform:linux/arm64,}"
time="2025-02-26T11:19:24.115Z" level=info msg="Starting deadline monitor"
time="2025-02-26T11:19:29.120Z" level=info msg="Main container completed" error="<nil>"
time="2025-02-26T11:19:29.120Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2025-02-26T11:19:29.120Z" level=info msg="Saving output parameters"
time="2025-02-26T11:19:29.120Z" level=info msg="Saving path output parameter: dummy-output"
time="2025-02-26T11:19:29.120Z" level=info msg="No output artifacts"
time="2025-02-26T11:19:29.133Z" level=info msg="Alloc=15556 TotalAlloc=21105 Sys=27221 NumGC=3 Goroutines=8"
time="2025-02-26T11:19:34.731Z" level=info msg="Starting Workflow Executor" version=v3.6.4
time="2025-02-26T11:19:34.734Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2025-02-26T11:19:34.735Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=default podName=fanout-dag-workflow-qpdvj-echo-value-1306772775 templateName=echo-value version="&Version{Version:v3.6.4,BuildDate:2025-02-10T11:50:49Z,GitCommit:77552d852da5c53a1d8c9918b3474d8bff06260a,GitTag:v3.6.4,GitTreeState:clean,GoVersion:go1.23.4,Compiler:gc,Platform:linux/arm64,}"
time="2025-02-26T11:19:34.739Z" level=info msg="Starting deadline monitor"
time="2025-02-26T11:19:37.743Z" level=info msg="Main container completed" error="<nil>"
time="2025-02-26T11:19:37.743Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2025-02-26T11:19:37.743Z" level=info msg="Saving output parameters"
time="2025-02-26T11:19:37.743Z" level=info msg="Saving path output parameter: dummy-output"
time="2025-02-26T11:19:37.743Z" level=info msg="No output artifacts"
time="2025-02-26T11:19:37.752Z" level=info msg="Alloc=15927 TotalAlloc=21107 Sys=26965 NumGC=3 Goroutines=8"
time="2025-02-26T11:19:34.723Z" level=info msg="Starting Workflow Executor" version=v3.6.4
time="2025-02-26T11:19:34.725Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2025-02-26T11:19:34.725Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=default podName=fanout-dag-workflow-qpdvj-echo-value-1844869295 templateName=echo-value version="&Version{Version:v3.6.4,BuildDate:2025-02-10T11:50:49Z,GitCommit:77552d852da5c53a1d8c9918b3474d8bff06260a,GitTag:v3.6.4,GitTreeState:clean,GoVersion:go1.23.4,Compiler:gc,Platform:linux/arm64,}"
time="2025-02-26T11:19:34.735Z" level=info msg="Starting deadline monitor"
time="2025-02-26T11:19:38.739Z" level=info msg="Main container completed" error="<nil>"
time="2025-02-26T11:19:38.739Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2025-02-26T11:19:38.739Z" level=info msg="Saving output parameters"
time="2025-02-26T11:19:38.739Z" level=info msg="Saving path output parameter: dummy-output"
time="2025-02-26T11:19:38.739Z" level=info msg="No output artifacts"
time="2025-02-26T11:19:38.749Z" level=info msg="Alloc=15896 TotalAlloc=21068 Sys=27221 NumGC=3 Goroutines=8"
time="2025-02-26T11:19:14.432Z" level=info msg="Starting Workflow Executor" version=v3.6.4
time="2025-02-26T11:19:14.434Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2025-02-26T11:19:14.434Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=default podName=fanout-dag-workflow-qpdvj-echo-value-2384080615 templateName=echo-value version="&Version{Version:v3.6.4,BuildDate:2025-02-10T11:50:49Z,GitCommit:77552d852da5c53a1d8c9918b3474d8bff06260a,GitTag:v3.6.4,GitTreeState:clean,GoVersion:go1.23.4,Compiler:gc,Platform:linux/arm64,}"
time="2025-02-26T11:19:14.445Z" level=info msg="Starting deadline monitor"
time="2025-02-26T11:19:17.450Z" level=info msg="Main container completed" error="<nil>"
time="2025-02-26T11:19:17.450Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2025-02-26T11:19:17.450Z" level=info msg="Saving output parameters"
time="2025-02-26T11:19:17.450Z" level=info msg="Saving path output parameter: dummy-output"
time="2025-02-26T11:19:17.450Z" level=info msg="No output artifacts"
time="2025-02-26T11:19:17.472Z" level=info msg="Alloc=17285 TotalAlloc=21089 Sys=30805 NumGC=3 Goroutines=8"
time="2025-02-26T11:19:24.106Z" level=info msg="Starting Workflow Executor" version=v3.6.4
time="2025-02-26T11:19:24.108Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2025-02-26T11:19:24.108Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=default podName=fanout-dag-workflow-qpdvj-echo-value-3433289636 templateName=echo-value version="&Version{Version:v3.6.4,BuildDate:2025-02-10T11:50:49Z,GitCommit:77552d852da5c53a1d8c9918b3474d8bff06260a,GitTag:v3.6.4,GitTreeState:clean,GoVersion:go1.23.4,Compiler:gc,Platform:linux/arm64,}"
time="2025-02-26T11:19:24.115Z" level=info msg="Starting deadline monitor"
time="2025-02-26T11:19:27.120Z" level=info msg="Main container completed" error="<nil>"
time="2025-02-26T11:19:27.120Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2025-02-26T11:19:27.120Z" level=info msg="Saving output parameters"
time="2025-02-26T11:19:27.120Z" level=info msg="Saving path output parameter: dummy-output"
time="2025-02-26T11:19:27.120Z" level=info msg="No output artifacts"
time="2025-02-26T11:19:27.133Z" level=info msg="Alloc=16541 TotalAlloc=21025 Sys=26453 NumGC=3 Goroutines=8"
time="2025-02-26T11:19:44.334Z" level=info msg="Using executor retry strategy" Duration=1s Factor=1.6 Jitter=0.5 Steps=5
time="2025-02-26T11:19:44.334Z" level=info msg="Executor initialized" deadline="0001-01-01 00:00:00 +0000 UTC" includeScriptOutput=false namespace=default podName=fanout-dag-workflow-qpdvj-echo-value-402549470 templateName=echo-value version="&Version{Version:v3.6.4,BuildDate:2025-02-10T11:50:49Z,GitCommit:77552d852da5c53a1d8c9918b3474d8bff06260a,GitTag:v3.6.4,GitTreeState:clean,GoVersion:go1.23.4,Compiler:gc,Platform:linux/arm64,}"
time="2025-02-26T11:19:44.341Z" level=info msg="Starting deadline monitor"
time="2025-02-26T11:19:47.345Z" level=info msg="Main container completed" error="<nil>"
time="2025-02-26T11:19:47.345Z" level=info msg="No Script output reference in workflow. Capturing script output ignored"
time="2025-02-26T11:19:47.345Z" level=info msg="Saving output parameters"
time="2025-02-26T11:19:47.345Z" level=info msg="Saving path output parameter: dummy-output"
time="2025-02-26T11:19:47.345Z" level=info msg="No output artifacts"
time="2025-02-26T11:19:47.357Z" level=info msg="Alloc=15381 TotalAlloc=21169 Sys=27733 NumGC=3 Goroutines=8"
time="2025-02-26T11:19:47.362Z" level=info msg="Deadline monitor stopped"
@jswxstw jswxstw added the area/retryStrategy Template-level retryStrategy label Mar 3, 2025
@jswxstw
Copy link
Member

jswxstw commented Mar 3, 2025

Looks like a similar root cause (retry strategy) as #13691

Not the same.

I ran the workflow you provided and it turned to Error with message Workflow operation error, which is different from yours. ps. I tested it with the latest.

I found that processAggregateNodeOutputs will fail if Steps or DAG has a retryStrategy and it panics here:

argo-workflows/workflow/controller/operator.go

Lines 3273 to 3274 in b005487

// need to sort the child node list so that the order of outputs are preserved
sort.Sort(loopNodes(childNodes))

The name of the child node has extra brackets and it panics here: s = strings.SplitN(s, "(", 2)[1]

argo-workflows/workflow/controller/operator.go

Lines 3238 to 3252 in b005487

func parseLoopIndex(s string) int {
s = strings.SplitN(s, "(", 2)[1]
s = strings.SplitN(s, ":", 2)[0]
val, err := strconv.Atoi(s)
if err != nil {
panic(fmt.Sprintf("failed to parse '%s' as int: %v", s, err))
}
return val
}
func (n loopNodes) Less(i, j int) bool {
left := parseLoopIndex(n[i].Name)
right := parseLoopIndex(n[j].Name)
return left < right
}

jswxstw added a commit to jswxstw/argo-workflows that referenced this issue Mar 14, 2025
@jswxstw
fix: process aggregate outputs for nodes with retries. Fixes argoproj…
354b62a 
…#14228

Signed-off-by: oninowang <[email protected]>
jswxstw added a commit to jswxstw/argo-workflows that referenced this issue Mar 14, 2025
@jswxstw
fix: process aggregate outputs for nodes with retries. Fixes argoproj…
7c73ab0 
…#14228

Signed-off-by: oninowang <[email protected]>
@jswxstw jswxstw linked a pull request Mar 14, 2025 that will close this issue
Open
jswxstw added a commit to jswxstw/argo-workflows that referenced this issue Mar 17, 2025
@jswxstw
fix: process aggregate outputs for nodes with retries. Fixes argoproj…
3c50fe2 
…#14228

Signed-off-by: oninowang <[email protected]>
jswxstw added a commit to jswxstw/argo-workflows that referenced this issue Mar 17, 2025
@jswxstw
fix: process aggregate outputs for nodes with retries. Fixes argoproj…
18f55ac 
…#14228

Signed-off-by: oninowang <[email protected]>
jswxstw added a commit to jswxstw/argo-workflows that referenced this issue Mar 18, 2025
@jswxstw
fix: process aggregate outputs for nodes with retries. Fixes argoproj…
feb6224 
…#14228

Signed-off-by: oninowang <[email protected]>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
area/retryStrategy Template-level retryStrategy type/bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

fix: process aggregate outputs for nodes with retries. Fixes #14228 jswxstw/argo-workflows
2 participants
@elliotgunton @jswxstw