Yeah this is a serious issue that affects me as well. I use an HPA and if I don't set the replicas field to the current replica count, it wipes out all but one of my pods before recreating them. I have persistent client connections I can't afford to lose in this way.

I came across this issue while debugging the same behavior using autoscaler/v2 HPA in the CNCF slack channel for argo rollouts as well: https://cloud-native.slack.com/archives/C01U781DW2E/p1736195993217089

Our expectations is that during new version rollout, the number of replicas is carried-over from previous version.

What do you mean by "new version rollout"? Do you have a process that will delete the rollout object and recreate it? Or something that removes the spec.replicas field? I would not expect the .spec.replicas to ever be nil after the initial creation, unless it is explicitly removed.

Kubernetes audit logs can be useful to find who updated an object. Sometimes, the ManagedFields can also provide some information.

I believe I'm seeing that same bug, here's a timeline of a change that does two things:

• updates a rollout manifest from having a fixed number of replicas to no predefined number of replicas (55->nil)
• creates an HPA (scaleTargetRef: kind: Rollout name: my-service) covering the application pods with a range of [50,60]
• does so as part of a deploy using setWeight: 20% as its first step
• revision A is the currently active, healthy deploy/replicaset

I've pulled the timestamps and strings from audit logs. Timestamps (requestReceivedTimestamp) are typically when rollouts-controller creates the corresponding Event. Actual resource creation might differ mildly.

1. At 2025-01-09T15:38:45.012840Z, argo-rollouts creates new my-service HPA with [min:50, max:60] replicas.

2. 2025-01-09T15:38:48.869Z, argo-rollouts shrinks the replicaset currently serving traffic:
   Scaled down ReplicaSet my-service-6d78cbf799 (revision A) from 55 to 1

3. 2025-01-09T15:38:50.530422Z, argo-rollouts creates the replicaset for the next version
   Created ReplicaSet my-service-7445f6cb6d (revision A+1)

4. 2025-01-09T15:38:51.496034Z
   Scaled up ReplicaSet my-service-7445f6cb6d (revision A+1) from 0 to 1

5. 2025-01-09T15:39:00.053656Z, the hpa autoscaler reports

type: AbleToScale
status: "True"
lastTransitionTime: "2025-01-09T15:39:00Z"
reason: SucceededRescale
message: the HPA controller was able to update the target scale to 50

6. 2025-01-09T15:39:01.085Z, argo-rollout sets the stable replicaset to 40:
   Scaled up ReplicaSet my-service-6d78cbf799 (revision A) from 1 to 40

7. 2025-01-09T15:39:02.202262Z argo-rollout sets new version to 10:
   Scaled up ReplicaSet my-service-7445f6cb6d (revision A+1) from 1 to 10

Note that ~2s before argo even created the replicaset for the next version, it scaled down the existing, healthy replicaset to 1. 10 seconds later, the HPA sets the scale to the minimum. Argo then proceeds with starting the canary run.

Between 15:38:48 and 15:39:01 (~13s) we're actively shutting down most healthy pods (rev A) of this service, before we start replacing them with another set of 40 pods (rev A again) and then proceed with a regular deploy (not without having significant service impact from the unexpected shrinking).

@chrono So what is impactful is

updates a rollout manifest from having a fixed number of replicas to no predefined number of replicas (55->nil)

By doing this, rollout is told that no replicas are specified, so it defauts to 1 as expected.

Instead, you could deploy the HPA first and let it manage the replicas field. If you use Argo CD to manage the replicas, then you can remove the config for Git and use IgnoreDifferences so you never apply a nil replicas config.

By doing this, rollout is told that no replicas are specified, so it defauts to 1 as expected.

That's the point of the bug. There's already a live Rollout with spec.replicas, which is much better "default" than 1.

you could deploy the HPA first and let it manage the replicas field. If you use Argo CD to manage the replicas, then you can remove the config for Git and use IgnoreDifferences so you never apply a nil replicas config.

I could also try to teach every developer to kubectl apply replicasets by hand, instead of using ArgoCD.

Why Argo CD?¶
Application definitions, configurations, and environments should be declarative and version controlled. Application deployment and lifecycle management should be automated, auditable, and easy to understand.

That's why folks are using ArgoCD. As an abstraction so that practitioners don't have to understand and execute nitty gritty details. And if things can't be done in a safe, predictable, manner -- perhaps the tooling should fail safe and throw the hands up "I need manual intervention here".

At this point, Argo CD does not know the behavior of your Kubernetes resources. It is told to match your desired state, and it does that by removing the replicas field as instructed. Argo Rollouts also does as instructed: It has no replicas defined in the manifest, so it default to 1 (Same as the Deployment behavior). It does not have any guarantee that what is in .status.replicas matches what was previously in .spec.replicas.

I believe the underlying issue is an Argo CD issue to stop managing the "replicas" field in the desired git state, without removing it from the live state. This use case seems documented in Argo CD Documentation for Deployment, but I believe would also apply on Rollout objects since they have the same behavior.

Almost any behavior would be better than what currently happens. The 'replicas' in the rollout should be what is the final state, but it immediately updates replica counts on the existing kubernetes objects. Let's say I have a replicaset controlled by HPA and although my original rollout said 5 pods, it is currently running 10. If I run a new rollout, only updating the image tag, it will immediately destroy 5 of the active pods to bring that original replicaset in line with the spec. If I don't specify replicas, we get slashed to 1 replica.
Should users have to dynamically specify this replica count on every rollout? I think there must be a better way. Perhaps a new field 'minReplicas' or 'useExistingReplicaCount' or SOMETHING that might allow Argo rollouts to retain existing replica counts decided by HPA rather than wiping them out one way or another.

Perhaps it's useful to temporarily ignore the HPA in this discussion and focus on the Rollout, and its management of Replicasets.

A developer for some [leave this undefined] reason unsets the replica count, or sets it to some value that will not result in a healthy system. When they're doing this change as part of change using a deployment strategy like canary or bluegreen then I'd expect argo doing its best to treat this new configuration as part of the change it's trying to validate and roll out.

So I wouldn't expect argo to scale down the old, healthy, replicaset to the new (potentially unsafe / broken / untested) value immediately before starting the deployment strategy. I would expect that argo tries to introduce the new configuration as part of the strategy, and allow automatic health assessments to judge its performance.

If I run a new rollout, only updating the image tag, it will immediately destroy 5 of the active pods to bring that original replicaset in line with the spec. If I don't specify replicas, we get slashed to 1 replica.
Should users have to dynamically specify this replica count on every rollout?

@insylogo The HPA will set the spec.replicas field based on its recommendation. Rollout will always respect the value that is provided as part of spec.replicas manifest configuration live in Kubernetes. So if you create a rollout with 5 replicas by setting spec.replicas: 5 in the manifest, it will scale it to 5. If the HPA update the spec.replicas to 10, then rollout will scale it to 10 replicas. If you preform a manual action that update the rollout manifest and set the spec.replicas to 5 again, then rollout will scale down to 5 replicas. If you perform another update where you explicitly remove the spec.replicas field (kubectl apply/patch/edit may do strategic merge and might not remove the replicas field, but Argo CD or another GitOps system might depending on their configuration), then rollout will respect what has been set in the spec, and scale it down to 1 since that is what is present in the manifest spec.

You mention that your HPA controls the ReplicaSet. The answer above is assuming that the Rollout is configured in your scaleTargetRef, as shown in this documentation. Does your scaleTargetRef is your Rollout like in the example?

I would expect that argo tries to introduce the new configuration as part of the strategy, and allow automatic health assessments to judge its performance.

@chrono That would be an interesting Idea to solve the problem, but spec.replicas is not part of the canary/blue-green strategy. Otherwise, scaling up/down the rollout object (either manually, with an HPA or any other means) would cause constant new rollouts. Because the health of the effective Pod spec has already been assessed, a resize operation does not need to go through the same lengthly hops as a new Pod Spec would.

In my opinion, I think this is not a problem that should be solved within the Rollout or Deployment controller. I think a feature could be added in Argo CD to help prevent these kind of issues since ignoreDifferences and RespectIgnoreDifferences may be too complex to configure in each Application. If you are using another tool that updates the spec.replicas field, then this tool will also need modification to not set the spec.replicas to an undesired value.

Something that could be added in my opinion is a validation that would fail the resource update if the .spec.replicas is removed on an existing Rollout resource. This way the alternative would either be to do a Kubernetes Replace operation, or explicitly set the field to spec.replicas: 1 if that is the intended behavior.

if things can't be done in a safe, predictable, manner -- perhaps the tooling should fail safe and throw the hands up "I need manual intervention here".

I think that would meet this behavior. If you are using Argo CD, the sync would fail, blocked by this validation if the Application is not configured to not dangerously override the spec.replicas.

But the field status.replicas (total pods in all ReplicaSet)does not have the same intent as .spec.replicas (Number of desired stable replicas) and should not be used as a default in my opinion.