Skip to content

OCPBUGS-57423: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition #5119

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
wants to merge 1 commit into
base: release-4.16
Choose a base branch
from

Conversation

wking
Copy link
Member

@wking wking commented Jun 12, 2025

Manually picking #4760 / #5111 back to 4.16, which is using coStatus instead of coStatusCondition, because it lacks #4321. Otherwise identical:

$ diff -U1 <(git show --oneline 377a78bb9fd2) <(git show --oneline 6bf0d7135d09)
--- /dev/fd/63  2025-06-12 10:41:31.352490108 -0700
+++ /dev/fd/62  2025-06-12 10:41:31.352490108 -0700
@@ -1,7 +1,7 @@
-377a78bb9 pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition
+6bf0d7135 pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition
 diff --git a/pkg/operator/status.go b/pkg/operator/status.go
-index 63699ec5c..eb015e005 100644
+index 69d37ad29..21a5ccc16 100644
 --- a/pkg/operator/status.go
 +++ b/pkg/operator/status.go
-@@ -285,13 +285,6 @@ func (optr *Operator) syncUpgradeableStatus(co *configv1.ClusterOperator) error
+@@ -316,13 +316,6 @@ func (optr *Operator) syncUpgradeableStatus() error {
                        break
@@ -12,5 +12,5 @@
 -      if updating && !degraded && !interrupted {
--              coStatusCondition.Status = configv1.ConditionFalse
--              coStatusCondition.Reason = "PoolUpdating"
--              coStatusCondition.Message = "One or more machine config pools are updating, please see `oc get mcp` for further details"
+-              coStatus.Status = configv1.ConditionFalse
+-              coStatus.Reason = "PoolUpdating"
+-              coStatus.Message = "One or more machine config pools are updating, please see `oc get mcp` for further details"
 -      }

956e787 (Implement Upgrade-Monitor, FeatureGate, and
MachineConfigNode types, 2023-11-28, openshift#4012) had added the "this should
no longer trigger when adding a node to a pool" comment, but
unfortunately, it's still triggering.  For example, in [1]:

  $ curl -s https://storage.googleapis.com/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.19-e2e-gcp-ovn-serial-crun/1868424902256627712/build-log.txt | grep 'PoolUpdating' | sort | uniq
  time="2024-12-16T01:43:52Z" level=info msg="operator status: processing event" event="Dec 16 00:55:35.662 W clusteroperator/machine-config condition/Upgradeable reason/PoolUpdating status/False One or more machine config pools are updating, please see `oc get mcp` for further details" operator=machine-config

Checking PromeCIeus, the Upgradeable=False window seems to have been
00:56 through 00:59, which correlates with the scale-up/scale-down of
the serial suite:

  $ curl -s https://storage.googleapis.com/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.19-e2e-gcp-ovn-serial-crun/1868424902256627712/build-log.txt | grep 'Managed cluster should grow and decrease when scaling different machineSets simultaneously'
  started: 0/20/74 "[sig-cluster-lifecycle][Feature:Machines][Serial] Managed cluster should grow and decrease when scaling different machineSets simultaneously [Timeout:30m][apigroup:machine.openshift.io] [Suite:openshift/conformance/serial]"
  passed: (5m42s) 2024-12-16T00:57:49 "[sig-cluster-lifecycle][Feature:Machines][Serial] Managed cluster should grow and decrease when scaling different machineSets simultaneously [Timeout:30m][apigroup:machine.openshift.io] [Suite:openshift/conformance/serial]"

confirmed via MCC logs:

  $ curl -s https://gcsweb-ci.apps.ci.l2s4.p1.openshiftapps.com/gcs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.19-e2e-gcp-ovn-serial-crun/1868424902256627712/artifacts/e2e-gcp-ovn-serial-crun/gather-extra/artifacts/pods/openshift-machine-config-operator_machine-config-controller-6f4f46457c-v8b2l_machine-config-controller.log | grep rendered-
  I1216 00:55:35.430231       1 node_controller.go:584] Pool worker[zone=us-central1-f]: node ci-op-k8c03v6z-9149a-r27w7-worker-f-t7rmb: changed annotation machineconfiguration.openshift.io/currentConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2
  I1216 00:55:35.430252       1 node_controller.go:584] Pool worker[zone=us-central1-f]: node ci-op-k8c03v6z-9149a-r27w7-worker-f-t7rmb: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2
  I1216 00:55:36.174629       1 node_controller.go:584] Pool worker[zone=us-central1-a]: node ci-op-k8c03v6z-9149a-r27w7-worker-a-f7hkj: changed annotation machineconfiguration.openshift.io/currentConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2
  I1216 00:55:36.174738       1 node_controller.go:584] Pool worker[zone=us-central1-a]: node ci-op-k8c03v6z-9149a-r27w7-worker-a-f7hkj: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2
  I1216 00:55:41.296273       1 node_controller.go:584] Pool worker[zone=us-central1-b]: node ci-op-k8c03v6z-9149a-r27w7-worker-b-554bt: changed annotation machineconfiguration.openshift.io/currentConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2
  I1216 00:55:41.296306       1 node_controller.go:584] Pool worker[zone=us-central1-b]: node ci-op-k8c03v6z-9149a-r27w7-worker-b-554bt: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2
  I1216 00:55:47.106173       1 node_controller.go:584] Pool worker[zone=us-central1-c]: node ci-op-k8c03v6z-9149a-r27w7-worker-c-hshj2: changed annotation machineconfiguration.openshift.io/currentConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2
  I1216 00:55:47.106201       1 node_controller.go:584] Pool worker[zone=us-central1-c]: node ci-op-k8c03v6z-9149a-r27w7-worker-c-hshj2: changed annotation machineconfiguration.openshift.io/desiredConfig = rendered-worker-6d0e61dc44f24db3272625b901024ed2

In this commit, I'm dropping the code that had been moving the
ClusterOperator to Upgradeable=False on PoolUpdating entirely, instead
of hoping that it doesn't trip.  I haven't dug into why the code had
still been tripping.  But we want to stay Upgradeable=True while new
nodes scale in, because clusters where nodes are joining should still
be able to update to 4.(y+1).  There are node-vs.-control-plane skew
issues that should block updates to 4.(y+1), but they're enforced by
the Kube API server operator [2], and don't need the MCO chipping in.

[1]: https://prow.ci.openshift.org/view/gs/test-platform-results/logs/periodic-ci-openshift-release-master-nightly-4.19-e2e-gcp-ovn-serial-crun/1868424902256627712
[2]: openshift/cluster-kube-apiserver-operator@9ce4f74
@wking
Copy link
Member Author

wking commented Jun 12, 2025

/jira cherry-pick OCPBUGS-57135

@openshift-ci openshift-ci bot requested review from jkyros and sinnykumari June 12, 2025 17:42
Copy link
Contributor

openshift-ci bot commented Jun 12, 2025

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: wking
Once this PR has been reviewed and has the lgtm label, please assign djoshy for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot
Copy link
Contributor

@wking: Jira Issue OCPBUGS-57135 has been cloned as Jira Issue OCPBUGS-57423. Will retitle bug to link to clone.
/retitle OCPBUGS-57423: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition

In response to this:

/jira cherry-pick OCPBUGS-57135

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot changed the title pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition OCPBUGS-57423: pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition Jun 12, 2025
@openshift-ci-robot openshift-ci-robot added jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. labels Jun 12, 2025
@openshift-ci-robot
Copy link
Contributor

@wking: This pull request references Jira Issue OCPBUGS-57423, which is valid. The bug has been moved to the POST state.

7 validation(s) were run on this bug
  • bug is open, matching expected state (open)
  • bug target version (4.16.z) matches configured target version for branch (4.16.z)
  • bug is in the state New, which is one of the valid states (NEW, ASSIGNED, POST)
  • release note text is set and does not match the template
  • dependent bug Jira Issue OCPBUGS-57135 is in the state Verified, which is one of the valid states (VERIFIED, RELEASE PENDING, CLOSED (ERRATA), CLOSED (CURRENT RELEASE), CLOSED (DONE), CLOSED (DONE-ERRATA))
  • dependent Jira Issue OCPBUGS-57135 targets the "4.17.z" version, which is one of the valid target versions: 4.17.0, 4.17.z
  • bug has dependents

Requesting review from QA contact:
/cc @sergiordlr

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

Manually picking #4760 / #5111 back to 4.16, which is using coStatus instead of coStatusCondition, because it lacks #4321. Otherwise identical:

$ diff -U1 <(git show --oneline 377a78bb9fd2) <(git show --oneline 6bf0d7135d09)
--- /dev/fd/63  2025-06-12 10:41:31.352490108 -0700
+++ /dev/fd/62  2025-06-12 10:41:31.352490108 -0700
@@ -1,7 +1,7 @@
-377a78bb9 pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition
+6bf0d7135 pkg/operator/status: Drop PoolUpdating as an Upgradeable=False condition
diff --git a/pkg/operator/status.go b/pkg/operator/status.go
-index 63699ec5c..eb015e005 100644
+index 69d37ad29..21a5ccc16 100644
--- a/pkg/operator/status.go
+++ b/pkg/operator/status.go
-@@ -285,13 +285,6 @@ func (optr *Operator) syncUpgradeableStatus(co *configv1.ClusterOperator) error
+@@ -316,13 +316,6 @@ func (optr *Operator) syncUpgradeableStatus() error {
                       break
@@ -12,5 +12,5 @@
-      if updating && !degraded && !interrupted {
--              coStatusCondition.Status = configv1.ConditionFalse
--              coStatusCondition.Reason = "PoolUpdating"
--              coStatusCondition.Message = "One or more machine config pools are updating, please see `oc get mcp` for further details"
+-              coStatus.Status = configv1.ConditionFalse
+-              coStatus.Reason = "PoolUpdating"
+-              coStatus.Message = "One or more machine config pools are updating, please see `oc get mcp` for further details"
-      }

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

@openshift-ci openshift-ci bot requested a review from sergiordlr June 12, 2025 17:42
@wking
Copy link
Member Author

wking commented Jun 13, 2025

unit TestBuildControllerMultipleOptedInPools/Happy_Path_Multiple_Configs/master/CustomBuildPod panic doesn't look like it's touching any of the code I'm changing.

/retest-required

Copy link
Contributor

openshift-ci bot commented Jun 13, 2025

@wking: The following tests failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name Commit Details Required Rerun command
ci/prow/e2e-azure-ovn-upgrade-out-of-change 6bf0d71 link false /test e2e-azure-ovn-upgrade-out-of-change
ci/prow/e2e-gcp-op-single-node 6bf0d71 link true /test e2e-gcp-op-single-node
ci/prow/e2e-gcp-op-techpreview 6bf0d71 link false /test e2e-gcp-op-techpreview

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

@sergiordlr
Copy link

/label cherry-pick-approved

@openshift-ci openshift-ci bot added the cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. label Jun 19, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
cherry-pick-approved Indicates a cherry-pick PR into a release branch has been approved by the release branch manager. jira/valid-bug Indicates that a referenced Jira bug is valid for the branch this PR is targeting. jira/valid-reference Indicates that this PR references a valid Jira ticket of any type.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants