Skip to content

Revert "core-services/prow/02_config/_boskos: Shard AWS, Azure, and GCP by region" #12842

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

wking
Copy link
Member

@wking wking commented Oct 15, 2020

This reverts commit 0cb81d0, #12589.

Telemetry isn't showing any free leases on these platforms, which is freaking me out a bit. It's possible that the dashboards just don't handle static names gracefully, but I'm going to float a revert in case something breaks and we want to buy some time for a post-mortem.

/hold

Probably no sense in landing this before we actually see jobs failing on lease aquisition, but please pull the hold if we do see that.

…CP by region"

This reverts commit 0cb81d0, openshift#12589.

Telemetry isn't showing any free leases on these platforms, which is
freaking me out a bit.  It's possible that the dashboards just don't
handle static names gracefully, but I'm going to float a revert in
case something breaks and we want to buy some time for a post-mortem.
@openshift-ci-robot openshift-ci-robot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 15, 2020
@openshift-ci-robot openshift-ci-robot added the approved Indicates a PR has been approved by an approver from all required OWNERS files. label Oct 15, 2020
@petr-muller
Copy link
Member

/hold cancel

Boskos dashboard shows the lease capacity dropped to 40 in AWS when the PR merged, we're seeing alerts about that, and jobs significantly started to fail to acquire leases: https://search.ci.openshift.org/chart?search=failed+to+acquire+lease&maxAge=48h&context=1&type=bug%2Bjunit&name=&maxMatches=5&maxBytes=20971520&groupBy=job

boskos

@openshift-ci-robot openshift-ci-robot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Oct 15, 2020
@openshift-ci-robot openshift-ci-robot added the lgtm Indicates that a PR is ready to be merged. label Oct 15, 2020
@openshift-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: petr-muller, wking

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@openshift-merge-robot openshift-merge-robot merged commit 8450c34 into openshift:master Oct 15, 2020
@openshift-ci-robot
Copy link
Contributor

@wking: Updated the following 2 configmaps:

  • resources configmap in namespace ci at cluster app.ci using the following files:
    • key boskos.yaml using file core-services/prow/02_config/_boskos.yaml
  • resources configmap in namespace ci at cluster api.ci using the following files:
    • key boskos.yaml using file core-services/prow/02_config/_boskos.yaml

In response to this:

This reverts commit 0cb81d0, #12589.

Telemetry isn't showing any free leases on these platforms, which is freaking me out a bit. It's possible that the dashboards just don't handle static names gracefully, but I'm going to float a revert in case something breaks and we want to buy some time for a post-mortem.

/hold

Probably no sense in landing this before we actually see jobs failing on lease aquisition, but please pull the hold if we do see that.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

@wking wking deleted the revert-per-region-leases branch October 15, 2020 14:23
wking added a commit to wking/openshift-release that referenced this pull request Dec 10, 2020
…e, and GCP by region""

This reverts commit 8a22fc4, openshift#12842.

Boskos' config reloading on dynamic -> static pivots has been fixed by
kubernetes-sigs/boskos@3834f37d8a (Config sync: Avoid deadlock when
static -> dynamic -> static, 2020-12-03, kubernetes-sigs/boskos#54),
so we can take another run at static leases for these platforms.  Not
a clean re-revert, because 4705f26 (core-services/prow/02_config:
Drop GCP Boskos leases to 80, 2020-12-02, openshift#14032) landed in the
meantime, but it was easy to update from 120 to 80 here.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants