Skip to content

add -eou pipefail to remaining job templates #6738

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jan 16, 2020

Conversation

bparees
Copy link
Contributor

@bparees bparees commented Jan 16, 2020

it's understood this PR may break jobs that can't tolerate -euo pipefail. Expectation is for teams to fix those jobs to tolerate it, not to change the bash header. We want consistent bash headers.

@openshift-ci-robot openshift-ci-robot added the size/S Denotes a PR that changes 10-29 lines, ignoring generated files. label Jan 16, 2020
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: bparees

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Jan 16, 2020
@bparees
Copy link
Contributor Author

bparees commented Jan 16, 2020

/assign @smarterclayton @stevekuznetsov

note that the last set of updates broke at least on job that had to be fixed with:
#6723

i have not audited this to see if there are any undefined env vars in use in these templates... i can take the -u off if you think that's better.

#!/bin/sh -x

#!/bin/sh
set -euo pipefail
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

do we need to move to Bash shebangs for pipefail? I don't know if it's POSIX

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we should make the header of the files consistent

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

(didn't move to /bin/bash i mean)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

No pipefail in POSIX. I think we should move to #!/bin/bash.

@bparees bparees force-pushed the pipefail branch 2 times, most recently from ad45130 to e1ad773 Compare January 16, 2020 18:21
@openshift-ci-robot openshift-ci-robot added size/M Denotes a PR that changes 30-99 lines, ignoring generated files. and removed size/S Denotes a PR that changes 10-29 lines, ignoring generated files. labels Jan 16, 2020
@bparees
Copy link
Contributor Author

bparees commented Jan 16, 2020

@smarterclayton updated all invocations to be

        #!/bin/bash
        set -euo pipefail

per your request. ptal.

@bparees bparees force-pushed the pipefail branch 4 times, most recently from 6c9997b to 8238f37 Compare January 16, 2020 18:25
@smarterclayton
Copy link
Contributor

Looks fine, might want to double check the rehearsals as necessary.

@bparees bparees added the lgtm Indicates that a PR is ready to be merged. label Jan 16, 2020
@bparees
Copy link
Contributor Author

bparees commented Jan 16, 2020

I audited the rehearsal failures, none of them appear to be caused by these changes.

@@ -1855,6 +1856,7 @@ objects:
- -c
- |
#!/bin/bash
set -euo pipefail
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Maybe not fail here... as teardown can have failures but should attempt to try to completion.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

the teardown script has the following protections that make this ok i think:

  1. trap 'teardown' EXIT
       function teardown() {
          set +e

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@jstuever agree?

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hold on. teardown shouldn't need to set +e. We have a "correct" teardown on most things - if we have a non standard teardown it would only apply in that context of one job (i.e. don't go adding set +e to teardown that don't have it already)

@bparees
Copy link
Contributor Author

bparees commented Jan 16, 2020

/hold

@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 16, 2020
@openshift-ci-robot
Copy link
Contributor

@bparees: The following tests failed, say /retest to rerun all failed tests:

Test name Commit Details Rerun command
ci/rehearse/cri-o/cri-o/release-1.13/e2e-aws bee15b9 link /test pj-rehearse
ci/rehearse/openshift/openshift-ansible/release-3.11/e2e-atomic bee15b9 link /test pj-rehearse
ci/rehearse/openshift/installer/master/e2e-gcp-upi bee15b9 link /test pj-rehearse
ci/rehearse/openshift/cloud-credential-operator/master/e2e-gcp bee15b9 link /test pj-rehearse
ci/rehearse/openshift/cloud-credential-operator/master/e2e-openstack bee15b9 link /test pj-rehearse
ci/rehearse/openshift/installer/fcos/e2e-vsphere bee15b9 link /test pj-rehearse
ci/rehearse/openshift/installer/master/e2e-ovirt bee15b9 link /test pj-rehearse
ci/rehearse/openshift/cluster-api-provider-aws/master/e2e-aws-operator bee15b9 link /test pj-rehearse
ci/rehearse/openshift/installer/master/e2e-aws-scaleup-rhel7 bee15b9 link /test pj-rehearse
ci/rehearse/openshift/cluster-api-provider-gcp/master/e2e-gcp-operator bee15b9 link /test pj-rehearse
ci/rehearse/openshift/installer/master/e2e-metal bee15b9 link /test pj-rehearse
ci/rehearse/openshift/installer/master/e2e-aws-proxy bee15b9 link /test pj-rehearse
ci/rehearse/openshift/cloud-credential-operator/master/e2e-azure bee15b9 link /test pj-rehearse
ci/prow/pj-rehearse bee15b9 link /test pj-rehearse
ci/rehearse/openshift/image-registry/release-3.11/e2e-gcp bee15b9 link /test pj-rehearse

Full PR test history. Your PR dashboard. Please help us cut down on flakes by linking to an open issue when you hit one in your PR.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. I understand the commands that are listed here.

@bparees
Copy link
Contributor Author

bparees commented Jan 16, 2020

/hold cancel

@openshift-ci-robot openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Jan 16, 2020
@openshift-merge-robot openshift-merge-robot merged commit 20e1ed1 into openshift:master Jan 16, 2020
@openshift-ci-robot
Copy link
Contributor

@bparees: Updated the following 35 configmaps:

  • prow-job-cluster-launch-installer-openstack-e2e configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-installer-openstack-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-openstack-e2e.yaml
  • prow-job-cluster-launch-installer-upi-e2e configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-installer-upi-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-upi-e2e.yaml
  • prow-job-cluster-launch-e2e-openshift-ansible configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-e2e-openshift-ansible.yaml using file ci-operator/templates/openshift/openshift-ansible/cluster-launch-e2e-openshift-ansible.yaml
  • prow-job-cluster-launch-e2e-openshift-jenkins configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-e2e-openshift-jenkins.yaml using file ci-operator/templates/openshift/openshift-ansible/cluster-launch-e2e-openshift-jenkins.yaml
  • prow-job-cluster-launch-installer-custom-test-image configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-installer-custom-test-image.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-custom-test-image.yaml
  • prow-job-cluster-launch-installer-ovirt-e2e configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-installer-ovirt-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-ovirt-e2e.yaml
  • prow-job-cluster-launch-installer-upi-e2e configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-installer-upi-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-upi-e2e.yaml
  • prow-job-cluster-launch-e2e-openshift-ansible configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-e2e-openshift-ansible.yaml using file ci-operator/templates/openshift/openshift-ansible/cluster-launch-e2e-openshift-ansible.yaml
  • prow-job-cluster-launch-e2e configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-e2e.yaml using file ci-operator/templates/openshift/openshift-ansible/cluster-launch-e2e.yaml
  • prow-job-cluster-launch-installer-src configmap in namespace ci at cluster ci/api-build01-ci-devcluster-openshift-com:6443 using the following files:
    • key cluster-launch-installer-src.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-src.yaml
  • prow-job-cluster-launch-installer-upi-e2e configmap in namespace ci at cluster ci/api-build01-ci-devcluster-openshift-com:6443 using the following files:
    • key cluster-launch-installer-upi-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-upi-e2e.yaml
  • prow-job-cluster-launch-e2e-openshift-jenkins configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-e2e-openshift-jenkins.yaml using file ci-operator/templates/openshift/openshift-ansible/cluster-launch-e2e-openshift-jenkins.yaml
  • prow-job-cluster-launch-e2e-upgrade configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-e2e-upgrade.yaml using file ci-operator/templates/openshift/openshift-ansible/cluster-launch-e2e-upgrade.yaml
  • prow-job-cluster-launch-installer-libvirt-e2e configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-installer-libvirt-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-libvirt-e2e.yaml
  • prow-job-cluster-launch-installer-metal-e2e configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-installer-metal-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-metal-e2e.yaml
  • prow-job-cluster-launch-installer-src configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-installer-src.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-src.yaml
  • prow-job-cluster-launch-installer-upi-src configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-installer-upi-src.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-upi-src.yaml
  • prow-job-cluster-launch-installer-custom-test-image configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-installer-custom-test-image.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-custom-test-image.yaml
  • prow-job-cluster-launch-installer-upi-src configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-installer-upi-src.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-upi-src.yaml
  • prow-job-cluster-launch-installer-e2e configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-installer-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-e2e.yaml
  • prow-job-cluster-launch-installer-ovirt-e2e configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-installer-ovirt-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-ovirt-e2e.yaml
  • prow-job-endurance-install configmap in namespace ci at cluster default using the following files:
    • key endurance-install.yaml using file ci-operator/templates/openshift/endurance/endurance-install.yaml
  • prow-job-endurance-install configmap in namespace ci-stg at cluster default using the following files:
    • key endurance-install.yaml using file ci-operator/templates/openshift/endurance/endurance-install.yaml
  • prow-job-cluster-launch-installer-e2e configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-installer-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-e2e.yaml
  • prow-job-cluster-launch-installer-e2e configmap in namespace ci at cluster ci/api-build01-ci-devcluster-openshift-com:6443 using the following files:
    • key cluster-launch-installer-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-e2e.yaml
  • prow-job-cluster-launch-installer-openstack-e2e configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-installer-openstack-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-openstack-e2e.yaml
  • prow-job-cluster-launch-e2e configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-e2e.yaml using file ci-operator/templates/openshift/openshift-ansible/cluster-launch-e2e.yaml
  • prow-job-cluster-launch-e2e-azure configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-e2e-azure.yaml using file ci-operator/templates/openshift/openshift-azure/cluster-launch-e2e-azure.yaml
  • prow-job-cluster-launch-installer-libvirt-e2e configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-installer-libvirt-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-libvirt-e2e.yaml
  • prow-job-cluster-launch-installer-metal-e2e configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-installer-metal-e2e.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-metal-e2e.yaml
  • prow-job-cluster-launch-installer-src configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-installer-src.yaml using file ci-operator/templates/openshift/installer/cluster-launch-installer-src.yaml
  • prow-job-cluster-launch-e2e-upgrade configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-launch-e2e-upgrade.yaml using file ci-operator/templates/openshift/openshift-ansible/cluster-launch-e2e-upgrade.yaml
  • prow-job-cluster-scaleup-e2e-40 configmap in namespace ci at cluster default using the following files:
    • key cluster-scaleup-e2e-40.yaml using file ci-operator/templates/openshift/openshift-ansible/cluster-scaleup-e2e-40.yaml
  • prow-job-cluster-scaleup-e2e-40 configmap in namespace ci-stg at cluster default using the following files:
    • key cluster-scaleup-e2e-40.yaml using file ci-operator/templates/openshift/openshift-ansible/cluster-scaleup-e2e-40.yaml
  • prow-job-cluster-launch-e2e-azure configmap in namespace ci at cluster default using the following files:
    • key cluster-launch-e2e-azure.yaml using file ci-operator/templates/openshift/openshift-azure/cluster-launch-e2e-azure.yaml

In response to this:

it's understood this PR may break jobs that can't tolerate -euo pipefail. Expectation is for teams to fix those jobs to tolerate it, not to change the bash header. We want consistent bash headers.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@bparees bparees deleted the pipefail branch January 17, 2020 14:47
wking added a commit to wking/ci-tools that referenced this pull request Jan 24, 2020
Bringing over a number of changes which have landed in
ci-operator/templates/openshift/installer/cluster-launch-installer-e2e.yaml
as of openshift/release@c64d5d7d8f (Merge pull request #6845 from
wking/shared-subnets-for-other-regions, 2020-01-23).

One series was AWS region sharding:

* openshift/release@b7179335a3
  (ci-operator/templates/openshift/installer/cluster-launch-installer-*:
  Random AWS regions for IPI, 2020-01-23, openshift/release#6833).
* openshift/release@7e38260d25
  (ci-operator/templates/openshift/installer: Shared subnets for new
  regions, 2020-01-23, openshift/release#6845).

Another series was the password removal followed by a bunch of
pipefail fumbling ;)

* openshift/release@4847cb5477 (clean up install log output,
  2020-01-13, openshift/release#6692).
* openshift/release@5c6ca8a506 (templates: Use 'pipefail' so that grep
  doesn't mask install failures, 2020-01-15, openshift/release#6718).
* openshift/release@bee15b9fa8 (add -eou pipefail to remaining job
  templates, 2020-01-16, openshift/release#6738)
* openshift/release@07bd61d677 (Revert "add -eou pipefail to remaining
  job templates", 2020-01-17, openshift/release#6748).
* openshift/release@ca655477ca (Revert "Revert "add -eou pipefail to
  remaining job templates"", 2020-01-17, openshift/release#6750).
* openshift/release@9d7453156b (tolerate undefined env vars in
  teardown, 2020-01-17, openshift/release#6750).

And there was also:

* openshift/release@c6c2efb3fc (templates: Add ipv6 variant that
  triggers azure singlestack, 2020-01-22, openshift/release#6809).
* openshift/release@752455a47f (templates: fix check for unset
  variable, 2020-01-16, openshift/release#6723).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. size/M Denotes a PR that changes 30-99 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

7 participants