BlueGreen Rollout Stuck in Progressing Blocking new Rollout

[Suraiya-Hameed]

Checklist:

• I've included steps to reproduce the bug.
• I've included the version of argo rollouts.

Describe the bug

While using BlueGreen strategy in Argo Rollouts (v1.8.2), I encountered a scenario where a rollout became stuck in a "processing" state and could not be recovered or promoted to stable. After triggering a new deployment, the new ReplicaSet is scaled fully and marked as preview, despite previewReplicaCount being set to 1. This new RS never gets promoted to stable/active, and the old RS remains marked as stable. I have attempted multiple recovery steps including:

• Deploying new versions (new Helm releases)
• Helm rollback
• Manual promotion (kubectl argo rollouts promote)
• Restarting the rollout

None of these actions have resolved the issue. I’m unable to deploy any new version, and the rollout remains stuck in this broken state.

This issue occurred once and I have not been able to reproduce it again. One notable event in metrics was a scaling event (likely via HPA or KEDA) that occurred during the rollout, which may be relevant. I have both prePromotionAnalysis and postPromotionAnalysis configured with auto-promotion enabled.

Both the preview and active/stable RS gets scaled on scaling event.

To Reproduce

Unfortunately, this issue is not consistently reproducible. Here’s what led up to the incident:

1. Apply a rollout with BlueGreen strategy
2. Ensure previewReplicaCount is set (in our case, to 1)
3. Trigger a new rollout (e.g. Helm upgrade)
4. Observe rollout gets stuck — new RS is fully scaled and stuck in preview
5. Attempting manual promotion or restarting rollout fails to resolve

Expected behavior

New ReplicaSet should be marked as preview with only previewReplicaCount replicas, proceed through pre- and post-promotion analysis, and then be promoted to stable. Old ReplicaSet should be demoted and eventually scaled down.

Screenshots

Version

1.8.2
Logs

# Paste the logs from the rollout controller

# Logs for the entire controller:
kubectl logs -n argo-rollouts deployment/argo-rollouts

# Logs for a specific rollout:
kubectl logs -n argo-rollouts deployment/argo-rollouts | grep rollout=<ROLLOUTNAME

time="2025-05-23T22:24:25Z" level=info msg="Reconciling Pre Promotion Analysis" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:24:25Z" level=info msg="Reconciling Post Promotion Analysis" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:24:25Z" level=info msg="Timed out (false) [last progress check: 2025-05-23 22:20:00 +0000 UTC - now: 2025-05-23 22:24:25.993005501 +0000 UTC m=+15950.933307485]" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:24:26Z" level=info msg="Patched: {\"status\":{\"availableReplicas\":10,\"conditions\":[{\"lastTransitionTime\":\"2025-05-22T21:33:55Z\",\"lastUpdateTime\":\"2025-05-22T21:33:55Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2025-05-22T21:34:43Z\",\"lastUpdateTime\":\"2025-05-22T21:34:43Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2025-05-23T22:09:38Z\",\"lastUpdateTime\":\"2025-05-23T22:20:00Z\",\"message\":\"ReplicaSet \\\"simpleapp-instance-3-6d9b8c7dbb\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"},{\"lastTransitionTime\":\"2025-05-23T22:24:25Z\",\"lastUpdateTime\":\"2025-05-23T22:24:25Z\",\"message\":\"Rollout does not have minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"False\",\"type\":\"Available\"}],\"message\":\"updated replicas are still becoming available\",\"readyReplicas\":10}}" generation=190 namespace=simpleapp-ns-3 resourceVersion=9475232 rollout=simpleapp-instance-3
time="2025-05-23T22:24:26Z" level=info msg="persisted to informer" generation=190 namespace=simpleapp-ns-3 resourceVersion=9479503 rollout=simpleapp-instance-3
time="2025-05-23T22:24:26Z" level=info msg="Reconciliation completed" generation=190 namespace=simpleapp-ns-3 resourceVersion=9475232 rollout=simpleapp-instance-3 time_ms=30.673155
time="2025-05-23T22:24:26Z" level=info msg="Started syncing rollout" generation=190 namespace=simpleapp-ns-3 resourceVersion=9479503 rollout=simpleapp-instance-3
time="2025-05-23T22:24:26Z" level=info msg="Reconciling stable ReplicaSet 'simpleapp-instance-3-6c55dfcff'" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:24:26Z" level=info msg="Reconciling Pre Promotion Analysis" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:24:26Z" level=info msg="Reconciling Post Promotion Analysis" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:24:26Z" level=info msg="Timed out (false) [last progress check: 2025-05-23 22:20:00 +0000 UTC - now: 2025-05-23 22:24:26.093617553 +0000 UTC m=+15951.033919537]" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:24:26Z" level=info msg="No status changes. Skipping patch" generation=190 namespace=simpleapp-ns-3 resourceVersion=9479503 rollout=simpleapp-instance-3
time="2025-05-23T22:24:26Z" level=info msg="Queueing up rollout for a progress after 333s" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:24:26Z" level=info msg="Reconciliation completed" generation=190 namespace=simpleapp-ns-3 resourceVersion=9479503 rollout=simpleapp-instance-3 time_ms=73.379848
time="2025-05-23T22:24:42Z" level=info msg="Started syncing rollout" generation=190 namespace=simpleapp-ns-3 resourceVersion=9479503 rollout=simpleapp-instance-3
time="2025-05-23T22:24:42Z" level=info msg="Reconciling stable ReplicaSet 'simpleapp-instance-3-6c55dfcff'" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:24:42Z" level=info msg="Reconciling Pre Promotion Analysis" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:24:42Z" level=info msg="Reconciling Post Promotion Analysis" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:24:42Z" level=info msg="Patched: {\"status\":{\"availableReplicas\":11,\"conditions\":[{\"lastTransitionTime\":\"2025-05-22T21:33:55Z\",\"lastUpdateTime\":\"2025-05-22T21:33:55Z\",\"message\":\"Rollout is not healthy\",\"reason\":\"RolloutHealthy\",\"status\":\"False\",\"type\":\"Healthy\"},{\"lastTransitionTime\":\"2025-05-22T21:34:43Z\",\"lastUpdateTime\":\"2025-05-22T21:34:43Z\",\"message\":\"RolloutCompleted\",\"reason\":\"RolloutCompleted\",\"status\":\"False\",\"type\":\"Completed\"},{\"lastTransitionTime\":\"2025-05-23T22:09:38Z\",\"lastUpdateTime\":\"2025-05-23T22:24:42Z\",\"message\":\"ReplicaSet \\\"simpleapp-instance-3-6d9b8c7dbb\\\" is progressing.\",\"reason\":\"ReplicaSetUpdated\",\"status\":\"True\",\"type\":\"Progressing\"},{\"lastTransitionTime\":\"2025-05-23T22:24:42Z\",\"lastUpdateTime\":\"2025-05-23T22:24:42Z\",\"message\":\"Rollout has minimum availability\",\"reason\":\"AvailableReason\",\"status\":\"True\",\"type\":\"Available\"}],\"message\":\"active service cutover pending\",\"readyReplicas\":11}}" generation=190 namespace=simpleapp-ns-3 resourceVersion=9479503 rollout=simpleapp-instance-3
time="2025-05-23T22:24:42Z" level=info msg="persisted to informer" generation=190 namespace=simpleapp-ns-3 resourceVersion=9479675 rollout=simpleapp-instance-3
time="2025-05-23T22:24:42Z" level=info msg="Reconciliation completed" generation=190 namespace=simpleapp-ns-3 resourceVersion=9479503 rollout=simpleapp-instance-3 time_ms=29.567562
time="2025-05-23T22:24:42Z" level=info msg="Started syncing rollout" generation=190 namespace=simpleapp-ns-3 resourceVersion=9479675 rollout=simpleapp-instance-3
time="2025-05-23T22:24:42Z" level=info msg="Reconciling stable ReplicaSet 'simpleapp-instance-3-6c55dfcff'" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:24:42Z" level=info msg="Reconciling Pre Promotion Analysis" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:24:42Z" level=info msg="Reconciling Post Promotion Analysis" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:24:42Z" level=info msg="Timed out (false) [last progress check: 2025-05-23 22:24:42 +0000 UTC - now: 2025-05-23 22:24:42.326139903 +0000 UTC m=+15967.266441887]" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:24:42Z" level=info msg="No status changes. Skipping patch" generation=190 namespace=simpleapp-ns-3 resourceVersion=9479675 rollout=simpleapp-instance-3
time="2025-05-23T22:24:42Z" level=info msg="Queueing up rollout for a progress after 599s" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:24:42Z" level=info msg="Reconciliation completed" generation=190 namespace=simpleapp-ns-3 resourceVersion=9479675 rollout=simpleapp-instance-3 time_ms=3.029035
time="2025-05-23T22:29:11Z" level=info msg="syncing service" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3 service=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="syncing service" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3 service=simpleapp-instance-3-preview
time="2025-05-23T22:29:11Z" level=info msg="Started syncing rollout" generation=190 namespace=simpleapp-ns-3 resourceVersion=9479675 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="Reconciling stable ReplicaSet 'simpleapp-instance-3-6c55dfcff'" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="Reconciling Pre Promotion Analysis" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="Reconciling Post Promotion Analysis" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="Timed out (false) [last progress check: 2025-05-23 22:24:42 +0000 UTC - now: 2025-05-23 22:29:11.797111677 +0000 UTC m=+16236.737413651]" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="No status changes. Skipping patch" generation=190 namespace=simpleapp-ns-3 resourceVersion=9479675 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="Queueing up rollout for a progress after 330s" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="Reconciliation completed" generation=190 namespace=simpleapp-ns-3 resourceVersion=9479675 rollout=simpleapp-instance-3 time_ms=5.669488
time="2025-05-23T22:29:11Z" level=info msg="Started syncing rollout" generation=190 namespace=simpleapp-ns-3 resourceVersion=9479675 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="Reconciling stable ReplicaSet 'simpleapp-instance-3-6c55dfcff'" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="Reconciling Pre Promotion Analysis" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="Reconciling Post Promotion Analysis" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="Timed out (false) [last progress check: 2025-05-23 22:24:42 +0000 UTC - now: 2025-05-23 22:29:11.891613395 +0000 UTC m=+16236.831915379]" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="No status changes. Skipping patch" generation=190 namespace=simpleapp-ns-3 resourceVersion=9479675 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="Queueing up rollout for a progress after 330s" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="Reconciliation completed" generation=190 namespace=simpleapp-ns-3 resourceVersion=9479675 rollout=simpleapp-instance-3 time_ms=94.977824
time="2025-05-23T22:29:11Z" level=info msg="Started syncing rollout" generation=190 namespace=simpleapp-ns-3 resourceVersion=9479675 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="Reconciling stable ReplicaSet 'simpleapp-instance-3-6c55dfcff'" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="Reconciling Pre Promotion Analysis" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="Reconciling Post Promotion Analysis" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="Timed out (false) [last progress check: 2025-05-23 22:24:42 +0000 UTC - now: 2025-05-23 22:29:11.997078567 +0000 UTC m=+16236.937380541]" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="No status changes. Skipping patch" generation=190 namespace=simpleapp-ns-3 resourceVersion=9479675 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="Queueing up rollout for a progress after 330s" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T22:29:11Z" level=info msg="Reconciliation completed" generation=190 namespace=simpleapp-ns-3 resourceVersion=9479675 rollout=simpleapp-instance-3 time_ms=104.95787800000001

Error logs

time="2025-05-23T16:57:03Z" level=info msg="failed to sync ephemeral metadata nil to ReplicaSet simpleapp-instance-3-7ff87c4998: error updating replicaset in updateReplicaSet simpleapp-instance-3-7ff87c4998: Operation cannot be fulfilled on replicasets.apps \"simpleapp-instance-3-7ff87c4998\": the object has been modified; please apply your changes to the latest version and try again" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T16:57:03Z" level=error msg="roCtx.reconcile err failed to sync ephemeral metadata: error updating replicaset in updateReplicaSet simpleapp-instance-3-7ff87c4998: Operation cannot be fulfilled on replicasets.apps \"simpleapp-instance-3-7ff87c4998\": the object has been modified; please apply your changes to the latest version and try again" generation=169 namespace=simpleapp-ns-3 resourceVersion=9217413 rollout=simpleapp-instance-3
time="2025-05-23T16:57:03Z" level=error msg="rollout syncHandler error: failed to sync ephemeral metadata: error updating replicaset in updateReplicaSet simpleapp-instance-3-7ff87c4998: Operation cannot be fulfilled on replicasets.apps \"simpleapp-instance-3-7ff87c4998\": the object has been modified; please apply your changes to the latest version and try again" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T17:11:50Z" level=info msg="failed to sync ephemeral metadata &PodTemplateMetadata{Labels:map[string]string{role: preview,},Annotations:map[string]string{},} to ReplicaSet simpleapp-instance-3-7ff87c4998: error updating replicaset in updateReplicaSet simpleapp-instance-3-7ff87c4998: Operation cannot be fulfilled on replicasets.apps \"simpleapp-instance-3-7ff87c4998\": the object has been modified; please apply your changes to the latest version and try again" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T17:11:50Z" level=error msg="roCtx.reconcile err failed to sync ephemeral metadata: error updating replicaset in updateReplicaSet simpleapp-instance-3-7ff87c4998: Operation cannot be fulfilled on replicasets.apps \"simpleapp-instance-3-7ff87c4998\": the object has been modified; please apply your changes to the latest version and try again" generation=170 namespace=simpleapp-ns-3 resourceVersion=9250514 rollout=simpleapp-instance-3
time="2025-05-23T17:11:50Z" level=error msg="rollout syncHandler error: failed to sync ephemeral metadata: error updating replicaset in updateReplicaSet simpleapp-instance-3-7ff87c4998: Operation cannot be fulfilled on replicasets.apps \"simpleapp-instance-3-7ff87c4998\": the object has been modified; please apply your changes to the latest version and try again" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T17:11:51Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on pods \"simpleapp-instance-3-58f9d87cb9-4m4s6\": the object has been modified; please apply your changes to the latest version and try again" generation=170 namespace=simpleapp-ns-3 resourceVersion=9250514 rollout=simpleapp-instance-3
time="2025-05-23T17:11:51Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on pods \"simpleapp-instance-3-58f9d87cb9-4m4s6\": the object has been modified; please apply your changes to the latest version and try again" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T17:13:51Z" level=info msg="failed to sync ephemeral metadata &PodTemplateMetadata{Labels:map[string]string{role: preview,},Annotations:map[string]string{},} to ReplicaSet simpleapp-instance-3-58f9d87cb9: error updating replicaset in updateReplicaSet simpleapp-instance-3-58f9d87cb9: Operation cannot be fulfilled on replicasets.apps \"simpleapp-instance-3-58f9d87cb9\": the object has been modified; please apply your changes to the latest version and try again" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T17:13:51Z" level=error msg="roCtx.reconcile err failed to sync ephemeral metadata: error updating replicaset in updateReplicaSet simpleapp-instance-3-58f9d87cb9: Operation cannot be fulfilled on replicasets.apps \"simpleapp-instance-3-58f9d87cb9\": the object has been modified; please apply your changes to the latest version and try again" generation=171 namespace=simpleapp-ns-3 resourceVersion=9255787 rollout=simpleapp-instance-3
time="2025-05-23T17:13:51Z" level=error msg="rollout syncHandler error: failed to sync ephemeral metadata: error updating replicaset in updateReplicaSet simpleapp-instance-3-58f9d87cb9: Operation cannot be fulfilled on replicasets.apps \"simpleapp-instance-3-58f9d87cb9\": the object has been modified; please apply your changes to the latest version and try again" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T17:13:51Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on pods \"simpleapp-instance-3-58f9d87cb9-b4xlj\": the object has been modified; please apply your changes to the latest version and try again" generation=171 namespace=simpleapp-ns-3 resourceVersion=9255787 rollout=simpleapp-instance-3
time="2025-05-23T17:13:51Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on pods \"simpleapp-instance-3-58f9d87cb9-b4xlj\": the object has been modified; please apply your changes to the latest version and try again" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T17:17:37Z" level=info msg="failed to sync ephemeral metadata &PodTemplateMetadata{Labels:map[string]string{role: preview,},Annotations:map[string]string{},} to ReplicaSet simpleapp-instance-3-5ddf795d47: error updating replicaset in updateReplicaSet simpleapp-instance-3-5ddf795d47: Operation cannot be fulfilled on replicasets.apps \"simpleapp-instance-3-5ddf795d47\": the object has been modified; please apply your changes to the latest version and try again" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T17:17:37Z" level=error msg="roCtx.reconcile err failed to sync ephemeral metadata: error updating replicaset in updateReplicaSet simpleapp-instance-3-5ddf795d47: Operation cannot be fulfilled on replicasets.apps \"simpleapp-instance-3-5ddf795d47\": the object has been modified; please apply your changes to the latest version and try again" generation=172 namespace=simpleapp-ns-3 resourceVersion=9260020 rollout=simpleapp-instance-3
time="2025-05-23T17:17:37Z" level=error msg="rollout syncHandler error: failed to sync ephemeral metadata: error updating replicaset in updateReplicaSet simpleapp-instance-3-5ddf795d47: Operation cannot be fulfilled on replicasets.apps \"simpleapp-instance-3-5ddf795d47\": the object has been modified; please apply your changes to the latest version and try again" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3
time="2025-05-23T17:17:37Z" level=error msg="roCtx.reconcile err Operation cannot be fulfilled on pods \"simpleapp-instance-3-5ddf795d47-72khp\": the object has been modified; please apply your changes to the latest version and try again" generation=172 namespace=simpleapp-ns-3 resourceVersion=9260020 rollout=simpleapp-instance-3
time="2025-05-23T17:17:37Z" level=error msg="rollout syncHandler error: Operation cannot be fulfilled on pods \"simpleapp-instance-3-5ddf795d47-72khp\": the object has been modified; please apply your changes to the latest version and try again" namespace=simpleapp-ns-3 rollout=simpleapp-instance-3

---

Message from the maintainers:

Impacted by this bug? Give it a 👍. We prioritize the issues with the most 👍.

[Suraiya-Hameed]

After trying various options to get out of this state without deleting and re-creating a rollout, what eventually worked was pausing the rollout and then promoting it.

[kostis-codefresh]

Could you post your Rollout yaml please (anonymized if needed)?

[Suraiya-Hameed]

apiVersion: argoproj.io/v1alpha1
kind: Rollout
metadata:
  annotations:
    meta.helm.sh/release-name: simpleapp-instance-3
    meta.helm.sh/release-namespace: simpleapp-ns-3
    rollout.argoproj.io/revision: "4"
  creationTimestamp: "2025-05-27T00:34:29Z"
  generation: 9
  labels:
    app.kubernetes.io/instance: simpleapp-instance-3
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: simpleapp-instance-3
    helm.sh/chart: <>
  name: simpleapp-instance-3
  namespace: simpleapp-ns-3
  resourceVersion: "12964094"
  uid: c54c2cd7-a7aa-4923-b9a5-5c27433bbd64
spec:
  replicas: 11
  revisionHistoryLimit: 4
  selector:
    matchLabels:
      app.kubernetes.io/instance: simpleapp-instance-3
      app.kubernetes.io/name: simpleapp-instance-3
  strategy:
    blueGreen:
      activeMetadata:
        labels:
          role: active
      activeService: simpleapp-instance-3
      autoPromotionEnabled: true
      postPromotionAnalysis:
        analysisRunMetadata: {}
        templates:
        - templateName: example-post-promotion-analysis
      prePromotionAnalysis:
        analysisRunMetadata: {}
        templates:
        - templateName: example-analysis
      previewMetadata:
        labels:
          role: preview
      previewReplicaCount: 1
      previewService: simpleapp-instance-3-preview
  template:
    metadata:
      labels:
        app.kubernetes.io/instance: simpleapp-instance-3
        app.kubernetes.io/name: simpleapp-instance-3
        azure.workload.identity/use: "true"
    spec:
      affinity:
        podAntiAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
          - podAffinityTerm:
              labelSelector:
                matchLabels:
                  app.kubernetes.io/name: simpleapp-instance-3
              topologyKey: kubernetes.io/hostname
            weight: 1
      containers:
      - env:
        - name: processToMonitor
        image: <>
        imagePullPolicy: IfNotPresent
        lifecycle:
          postStart:
            exec:
              command:
              - bash
              - -c
              - until <>; do sleep 1; done
        livenessProbe:
          failureThreshold: 3
          initialDelaySeconds: 15
          periodSeconds: 20
          successThreshold: 1
          tcpSocket:
            port: 11013
          timeoutSeconds: 1
        name: mdm
        readinessProbe:
          exec:
            command:
            - bash
            - -c
            - if ! <>; then exit
              1; fi
          failureThreshold: 3
          initialDelaySeconds: 5
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 3
        resources:
          limits:
            cpu: 200m
            memory: 256Mi
          requests:
            cpu: 200m
            memory: 256Mi
        volumeMounts:
        - mountPath: /tmp/mdm/
          name: vault-folder
          readOnly: true
          subPath: <>
      - env:
        - name: DATA_CENTER
          value: <>
        - name: APP_NAME
          value: simpleapp-instance-3
        - name: NODE_NAME
          valueFrom:
            fieldRef:
              fieldPath: spec.nodeName
        - name: NAMESPACE_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.namespace
        - name: POD_NAME
          valueFrom:
            fieldRef:
              fieldPath: metadata.name
        image: <>
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 3
          httpGet:
            path: /healthz/liveness
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 15
          periodSeconds: 15
          timeoutSeconds: 15
        name: <>
        ports:
        - containerPort: 8080
          name: http
          protocol: TCP
        readinessProbe:
          failureThreshold: 3
          httpGet:
            path: healthz/readiness
            port: 8080
            scheme: HTTP
          initialDelaySeconds: 15
          periodSeconds: 20
          successThreshold: 1
          timeoutSeconds: 15
        resources:
          limits:
            cpu: 300m
            memory: 300Mi
          requests:
            cpu: 200m
            memory: 256Mi
        securityContext:
          allowPrivilegeEscalation: false
          privileged: false
          readOnlyRootFilesystem: false
          runAsGroup: 101
          runAsNonRoot: true
          runAsUser: 101
        volumeMounts:
        - mountPath: <>
          name: <>
        - mountPath: /vault-folder
          name: vault-folder
          readOnly: true
      dnsConfig:
        options:
        - name: single-request-reopen
        - name: timeout
          value: "1"
        - name: attempts
          value: "5"
      initContainers:
      - env:
        - name: TIMEOUT
          value: "60"
        image: <>
        name: <>
        resources: {}
        volumeMounts:
        - mountPath: /vault-folder
          name: vault-folder
      serviceAccount: simpleapp-instance-3
      serviceAccountName: simpleapp-instance-3
      terminationGracePeriodSeconds: 60
      tolerations:
      - effect: NoSchedule
        key: beta.kubernetes.io/instance-type
        operator: Equal
        value: Standard_D32s_v3
      volumes:
      - hostPath:
          path: <>
        name: <>
      - emptyDir:
          medium: Memory
        name: vault-folder
status:
  HPAReplicas: 11
  availableReplicas: 11
  blueGreen:
    activeSelector: 5dc468b5c6
    postPromotionAnalysisRunStatus:
      name: simpleapp-instance-3-5dc468b5c6-4-post
      status: Successful
    prePromotionAnalysisRunStatus:
      name: simpleapp-instance-3-5dc468b5c6-4-pre
      status: Successful
    previewSelector: 5dc468b5c6
  canary: {}
  conditions:
  - lastTransitionTime: "2025-05-27T00:53:33Z"
    lastUpdateTime: "2025-05-27T00:53:33Z"
    message: Rollout has minimum availability
    reason: AvailableReason
    status: "True"
    type: Available
  - lastTransitionTime: "2025-05-27T16:56:15Z"
    lastUpdateTime: "2025-05-27T16:56:15Z"
    message: Rollout is healthy
    reason: RolloutHealthy
    status: "True"
    type: Healthy
  - lastTransitionTime: "2025-05-27T00:34:30Z"
    lastUpdateTime: "2025-05-27T16:56:15Z"
    message: ReplicaSet "simpleapp-instance-3-5dc468b5c6" has successfully progressed.
    reason: NewReplicaSetAvailable
    status: "True"
    type: Progressing
  - lastTransitionTime: "2025-05-27T16:56:24Z"
    lastUpdateTime: "2025-05-27T16:56:24Z"
    message: RolloutCompleted
    reason: RolloutCompleted
    status: "True"
    type: Completed
  currentPodHash: 5dc468b5c6
  observedGeneration: "9"
  phase: Healthy
  readyReplicas: 11
  replicas: 11
  selector: app.kubernetes.io/instance=simpleapp-instance-3,app.kubernetes.io/name=simpleapp-instance-3,rollouts-pod-template-hash=5dc468b5c6
  stableRS: 5dc468b5c6
  updatedReplicas: 11
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  annotations:
    meta.helm.sh/release-name: simpleapp-instance-3
    meta.helm.sh/release-namespace: simpleapp-ns-3
  creationTimestamp: "2025-05-20T21:54:35Z"
  generation: 12
  labels:
    app.kubernetes.io/managed-by: Helm
  name: example-analysis
  namespace: simpleapp-ns-3
  resourceVersion: "12388479"
  uid: 3b25fbdf-2d78-4791-b288-6138290fa332
spec:
  metrics:
  - count: 1
    failureLimit: 0
    initialDelay: 5s
    interval: 1m
    name: pass
    provider:
      job:
        spec:
          backoffLimit: 0
          template:
            spec:
              containers:
              - args:
                - exit 0
                command:
                - sh
                - -c
                env:
                - name: DATA_CENTER
                  value: <>
                image: <>
                name: pass
              restartPolicy: OnFailure
  - count: 1
    failureLimit: 0
    initialDelay: 5s
    interval: 1m
    name: verify-workload-identity
    provider:
      job:
        spec:
          backoffLimit: 0
          template:
            metadata:
              labels:
                azure.workload.identity/use: "true"
            spec:
              containers:
              - args:
                - exit 0
                command:
                - sh
                - -c
                env:
                - name: DATA_CENTER
                  value: <>
                image: <>
                name: <>
              restartPolicy: OnFailure
              serviceAccountName: simpleapp-instance-3
  - count: 5
    failureLimit: 0
    interval: 30s
    name: prom-ql
    provider:
      prometheus:
        address: http://<>
        query: scalar(count(kube_pod_status_phase{phase="Running", namespace="simpleapp-ns-3",
          pod=~"simpleapp-.*"}))
    successCondition: result >= 4
  - count: 5
    failureLimit: 0
    interval: 30s
    name: prom-pod-cpu
    provider:
      prometheus:
        address: http://<>
        query: scalar(avg(rate(container_cpu_usage_seconds_total{namespace="simpleapp-ns-3",
          pod=~"simpleapp-.*"}[5m])) * 100)
    successCondition: result > 0
  - count: 5
    failureLimit: 0
    interval: 30s
    name: prom-pod-memory
    provider:
      prometheus:
        address: http://<>
        query: scalar(avg(container_memory_usage_bytes{namespace="simpleapp-ns-3",
          pod=~"simpleapp-.*"}) / 1024 / 1024)
    successCondition: result > 0
  - count: 5
    failureLimit: 0
    interval: 30s
    name: prom-rollout-status
    provider:
      prometheus:
        address: http://<>
        query: scalar(count(argo_rollouts:rollout_information{namespace="simpleapp-ns-3",phase="Progressing"}))
    successCondition: result == 1
---
apiVersion: argoproj.io/v1alpha1
kind: AnalysisTemplate
metadata:
  annotations:
    meta.helm.sh/release-name: simpleapp-instance-3
    meta.helm.sh/release-namespace: simpleapp-ns-3
  creationTimestamp: "2025-05-20T21:54:35Z"
  generation: 8
  labels:
    app.kubernetes.io/managed-by: Helm
  name: example-post-promotion-analysis
  namespace: simpleapp-ns-3
  resourceVersion: "12388483"
  uid: d9b38322-47d4-4c4b-bcf7-319d51025708
spec:
  metrics:
  - count: 1
    failureLimit: 0
    initialDelay: 5s
    interval: 1m
    name: post-promotion-check
    provider:
      job:
        spec:
          backoffLimit: 0
          template:
            spec:
              containers:
              - args:
                - exit 0
                command:
                - sh
                - -c
                env:
                - name: DATA_CENTER
                  value: wus2
                image: <>
                name: post-promotion-check
              restartPolicy: OnFailure

[Suraiya-Hameed]

I made some more progress on this issue, it happens only when previewReplicaCount and prePromotionAnalysis are used together.

There must be some race condition as I can't reproduce it consistently, but here is the sequence of events

• Rollout Version X, it marked stable
• Deploy Version Y → PrePromotionAnalysis starts ✅
• Deploy Version Z while Y's AnalysisRun is running → Version Z is marked as preview - No AnalysisRun is created for Z and number of Version Z pod equals to Desired Replicas (previewReplicaCount isn't honored)
• Any new Version deployed after this is marked as preview (No AnalysisRun is created and previewReplicaCount isn't honored) - new Version never gets promoted to Stable

[ajax-bychenok-y]

We have similar situation and we have only Rollout object and no additional Analysis or so.
We have just simple dummy service with health checks etc. And 2 images with different tags:

1111111111111111.dkr.ecr.eu-west-1.amazonaws.com/my-company/k8s-dummy-svc:0.0.1-5.c3f852b-SNAPSHOT
1111111111111111.dkr.ecr.eu-west-1.amazonaws.com/my-company/k8s-dummy-svc:0.0.1-7.a789b21-SNAPSHOT

And I am constatly releasing these version using BlueGreen strategy and AutoPromotion disabled (it always needs manual promote after Paused state). And from time to time I am getting constant "Progressing" state w/o "Paused" (as should be):

$ kubectl argo rollouts get rollout k8s-dummy-svc -n sandbox-yuriib

Name:            k8s-dummy-svc
Namespace:       sandbox-yuriib
Status:          ◌ Progressing
Message:         active service cutover pending
Strategy:        BlueGreen
Images:          111111111111111.dkr.ecr.eu-west-1.amazonaws.com/my-company/k8s-dummy-svc:0.0.1-5.c3f852b-SNAPSHOT (preview)
                 111111111111111.dkr.ecr.eu-west-1.amazonaws.com/my-company/k8s-dummy-svc:0.0.1-7.a789b21-SNAPSHOT (stable, active)
Replicas:
  Desired:       2
  Current:       4
  Updated:       2
  Ready:         2
  Available:     2

NAME                                       KIND        STATUS         AGE   INFO
⟳ k8s-dummy-svc                            Rollout     ◌ Progressing  3h9m
├──# revision:9
│  └──⧉ k8s-dummy-svc-76474bc745           ReplicaSet  ✔ Healthy      97m   preview
│     ├──□ k8s-dummy-svc-76474bc745-4tjc4  Pod         ✔ Running      11m   ready:1/1
│     └──□ k8s-dummy-svc-76474bc745-9jrsw  Pod         ✔ Running      11m   ready:1/1
├──# revision:8
│  └──⧉ k8s-dummy-svc-754b657cf4           ReplicaSet  ✔ Healthy      61m   stable,active
│     ├──□ k8s-dummy-svc-754b657cf4-2pq4s  Pod         ✔ Running      61m   ready:1/1
│     └──□ k8s-dummy-svc-754b657cf4-z9ft7  Pod         ✔ Running      61m   ready:1/1
└──# revision:4
   └──⧉ k8s-dummy-svc-c6bbcfdc9            ReplicaSet  • ScaledDown   165m

As you can see, everything is healthy and ready here. But why status is always Progressing? Does Argo Rollouts make any additional healthy checks?

Only promote --full helps me in this case which skips all steps or so. But I feel this is not how it should work.

[ajax-bychenok-y]

Seems that full disable of history rollout.revisionHistoryLimit: 0 fixes rollout stuck. But it's hard to call it a solution. How to rollback w/o it?