Skip to content

GitHub CI queuing #153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
andres-erbsen opened this issue May 31, 2025 · 7 comments
Open

GitHub CI queuing #153

andres-erbsen opened this issue May 31, 2025 · 7 comments

Comments

@andres-erbsen
Copy link
Collaborator

I've started noticing that it takes 15min+ for non-basic checks to start, even though they are queued:

Image

I don't understand why this would happen yet, but I guess I can still write down the annoyance.

@JasonGross
Copy link
Member

JasonGross commented May 31, 2025

GitHub runners are shared between all repositories of an organization. I believe GitHub allocates 20 runners to each organization (not 100% sure about this, I've only verified this on my personal account). Checks are only queued once all dependencies complete, and are processed in order. I've seen wait times of 16+ hours on Fiat Crypto if I push a bunch of PRs simultaneously.

@Zimmi48
Copy link
Member

Zimmi48 commented Jun 2, 2025

We could perhaps look into whether GitHub does open source sponsoring by allowing more concurrent runners.

@andres-erbsen
Copy link
Collaborator Author

On our side, one thing I can think of is that we could stop testing reverse dependencies of stdlib on the previous rocq release.

@andres-erbsen
Copy link
Collaborator Author

There might be something more subtle going on with dynamically generated jobs. The Alpine-based quick build job I recently added does not seem to get delayed.

Image

@JasonGross
Copy link
Member

Is there something going on that's not explained by "Checks are only queued once all dependencies complete" + max 20 simultaneous jobs? The behavior I'd expect to see is that the static jobs with no dependencies run, and then initial dynamic jobs (up to 20) run, and the rest get queued. Are you seeing some other behavior?

@andres-erbsen
Copy link
Collaborator Author

I'm not sure how to check simultaneous jobs, but the above screenshot shows that there are no running jobs and yet some are queued (but they can't be waiting on dependencies because no jobs are running).

@JasonGross
Copy link
Member

there are no running jobs

You have to check for running jobs across all repositories in the rocq-prover organization: https://github.com/organizations/rocq-prover/settings/actions/hosted-runners

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants