[Request] Limiting the maximum number of concurrent processors

[diyaa-zen]

Is it possible to support such usage ?
diyaa-zen@25b771f

The runtime logger tends to be unreliable, particularly when multiple builds are running simultaneously and competing for system resources. For example, I aim to have 16 processors running concurrently for 32000 specs, but more often than not, I get one processor is active extending the test suite over a 20 minutes to complete. This significantly slows down the entire test suite.

I'm using this alongside the runtime logger, and it works well with a variability of around ±3 minutes.
The only issue I encountered and fixed in our repo was with the runtime logger, rspec does clear the log file each times it requires a formatter, so I ended up with custom implementation.

[grosser]

setting num_processes to the max you want will not create more workers if there are not enough items,
so I'm not sure why setting max instead of setting count would be any different

that rspec bug sounds interesting, fix for that would be nice, workaround could be opening 1 file per process and then joining but that could be hacky

[diyaa-zen]

Thanks for the quick reply,

I think I didn't do well explaining what I'm trying to achieve.

I've have a machine with ~90 cores this is used for CI/CD and I have parallel_specs as part of the pipeline I can have different builds running simultaneously (max of 5), I'm trying to achieve below

• Don't let a single builds hogs all the resources
• Try to Maintain 16 parallel processes of rspec in all the times to minimize the idle time

above is difficult to achieve with a static n processes that run at the same time when you have a large number of specs and system with heavy resources contention, the runtime log alone used to work but with growing load and specs count it's not accurate.

so in my case I'm dividing the specs runs into 48 division that runs in 16 threads at max.

think of it like batching 48 jobs into 16 workers.

PARALLEL_TEST_PROCESSORS=48
sh 'bundle exec parallel_rspec --maximum_parallel 16 --runtime-log ./tmp/rspec_runtime.rspec.v2.txt ./spec/'

I'm just checking the possibility of supporting such use case(I can work on it), otherwise I'll keep using the fork.

appreciate your opinion on this !

[grosser]

if you have 90 cpus and 5 runners then that is 18 core per runner if you just hardcode that.
setting --maximum-parallel 16 would cap it at 16 so you'd already have less than that.

Setting PARALLEL_TEST_PROCESSORS=48 and --maximum-parallel 16 is the same as PARALLEL_TEST_PROCESSORS=16.

I sounds like you want the individual runners to communicate with each other on how much to run in parallel, but that feature does not exist.
If you can inspect the current ci state you could set parallel processes dynamically (machine is empty -> use 70, machine is 4/5 full -> use 18)

The runtime log should be produces in a serial run to be reliable for balancing, so I'd suggest to either run that locally and check it in, or run it in a weekly cron with a single processor and then store the result in a cache.