OCPBUGS-26008: installer pods: consider Pods with phase Failed and no container statuses as terminated

[everettraven]

OCPBUGS-26008 identified an edge case where installer pods could get stuck and continuously time out while waiting for an installer pod that entered a state where it had a .status.phase = Failed with .status.reason = NodeShutdown.

Because the installer pods have a restartPolicy = Never, the installer pod in this case never ran and future installer pods were stuck waiting for the previous installer pod to get populated with container statuses. Because it failed due to a NodeShutdown, the containers never ran and thus no container statuses were populated.

This PR updates the logic installer pods use to check that all other installer pods are terminated to include installer pods with phase Failed and no container statuses populated as "terminated".

[openshift-ci-robot]

@everettraven: This pull request references Jira Issue OCPBUGS-26008, which is invalid:

• expected the bug to target the "4.20.0" version, but no target version was set

Comment /jira refresh to re-evaluate validity if changes to the Jira bug are made, or edit the title of this pull request to link to a different bug.

The bug has been updated to refer to the pull request using the external bug tracker.

In response to this:

OCPBUGS-26008 identified an edge case where installer pods could get stuck and continuously time out while waiting for an installer pod that entered a state where it had a .status.phase = Failed with .status.reason = NodeShutdown.

Because the installer pods have a restartPolicy = Never, the installer pod in this case never ran and future installer pods were stuck waiting for the previous installer pod to get populated with container statuses. Because it failed due to a NodeShutdown, the containers never ran and thus no container statuses were populated.

This PR updates the logic installer pods use to check that all other installer pods are terminated to include installer pods with phase Failed and no container statuses populated as "terminated".

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: everettraven
Once this PR has been reviewed and has the lgtm label, please assign dgrisonnet for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Needs approval from an approver in each of these files:

• pkg/operator/staticpod/OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[everettraven]

This is an open question for reviewers - I couldn't think of any specific scenarios where this logic shouldn't apply, but I could definitely be missing something.

[everettraven]

Not sure why unit tests failed here. Running make test-unit works fine locally. Retesting

/retest

[everettraven]

/retest

[openshift-ci]

@everettraven: all tests passed!

Full PR test history. Your PR dashboard.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.