Skip to content

🐛 Bump timeout for CI peribolos to 4h #666

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 1 commit into from
Jun 3, 2025

Conversation

cardil
Copy link
Contributor

@cardil cardil commented Jun 3, 2025

Changes

  • 🐛 Bump timeout for CI peribolos to 4h

Context

Should fix issues like https://prow.knative.dev/view/gs/knative-prow/logs/ci-knative-extensions-peribolos/1929946357959757824

Which issue(s) this PR fixes:

Related to #662
Follow up on #665

@knative-prow knative-prow bot requested review from evankanderson and upodroid June 3, 2025 20:21
Copy link

knative-prow bot commented Jun 3, 2025

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: cardil

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@knative-prow knative-prow bot added approved Indicates a PR has been approved by an approver from all required OWNERS files. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files. labels Jun 3, 2025
@cardil
Copy link
Contributor Author

cardil commented Jun 3, 2025

/assign @dsimansk

@evankanderson
Copy link
Member

It looks like our GitHub API calls are taking 10+ seconds over the 2 hour window. Do we know why the latency is so high.

Approving for now, but I'm slightly concerned about the duration.

/lgtm

@knative-prow knative-prow bot added the lgtm Indicates that a PR is ready to be merged. label Jun 3, 2025
@knative-prow knative-prow bot merged commit 0d2df49 into knative:main Jun 3, 2025
17 checks passed
Copy link

knative-prow bot commented Jun 3, 2025

@cardil: Updated the job-config configmap in namespace default at cluster default using the following files:

  • key peribolos.yaml using file prow/jobs/custom/peribolos.yaml

In response to this:

Changes

  • 🐛 Bump timeout for CI peribolos to 4h

Context

Should fix issues like https://prow.knative.dev/view/gs/knative-prow/logs/ci-knative-extensions-peribolos/1929946357959757824

Which issue(s) this PR fixes:

Related to #662
Follow up on #665

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

@cardil cardil deleted the bugfix/bump-peribolos-timeout branch June 3, 2025 21:26
@dsimansk
Copy link
Contributor

dsimansk commented Jun 4, 2025

It looks like our GitHub API calls are taking 10+ seconds over the 2 hour window. Do we know why the latency is so high.

Approving for now, but I'm slightly concerned about the duration.

/lgtm
A bit of context is here: https://cloud-native.slack.com/archives/C04LY4M2G49/p1748886167174489

We've been running into "secondary limits" per #662.

So far we dropped those a few times:

  1. From former hourly 2000, burst 1000 - job failed in a couple of minutes
  2. To 1000, 500 - job failed within 30 min
  3. To prow's default 300, 100 - it can succeed with enough time

I agree, it takes too long to complete. In case of knative org the duration is around 1 hour. But knative-extensions was closing to 2h mark, intermittently surpassing and getting terminated. I think --fix-team-repos is causing the issue, posting too many mutating requests that are "priced" higher that get queries etc. The job usually failed during execution of RemoveTeamRepoBySlug as it goes through a list of teams and applying the change on every repo, resulting in too many permutations.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm Indicates that a PR is ready to be merged. size/XS Denotes a PR that changes 0-9 lines, ignoring generated files.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants