Description
We're having a problem where our longtests are not completing, thus making it impossible to merge anything that needs to run them.
The timeout of the job is set to 120 minutes and if I recall correctly they usually run around 80 minutes in practice, but they're regularly terminating without finishing at around 45 minutes. I suspect this is a system timeout that I don't have direct control over, and I'll be filing a ticket to see if I can find that out. But in the meantime, I think we need to split up the longtest job -- we're going to need to do this eventually because the more checkers we add the longer it's going to get.
Some thoughts:
- our very longest tests are the language parser ones. They could easily be run in their own separate job.
- We currently run longtests+regular tests all in the same job, we could split those into two
- We have something like 500 binary checker tests, maybe more. It's likely that we'll want to group them somehow and run them as probably 2-4 separate test jobs. I'd prefer this be done automatically, so we don't have to assign tests to a job every time a new test is added, so what we want is for pytest to have a way to divide them up into separate groups automatically and only run one set. I have no idea if pytest has a way for this to happen already or if we'll have to build something.
If anyone wants to work on this, I'd suggest starting with moving the language parser tests into a separate job (have that be one PR) and them move on to the other things as needed.
This shouldn't actually be that hard but it may take a few iterations before we get something that works consistently.