You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
On macOS, running tests in the min-deps pixi env consistently hang (at about the 95% mark on my machine). To reproduce, run the following on a macOS machine (YMMV, depending on hardware and OS version):
pixi run -e min-deps run-tests
This appears to be an OS-specific issue that surfaces with having pytest run tests in parallel across multiple CPUs because while this hangs on my macOS machine (current code on develop branch), the GitHub workflow does not hang (Linux OS).
Further, this also appears to occur only for the min-deps pixi env, but without understanding why this does not (currently) happen in other pixi envs, I imagine that it could occur in other envs, depending on the specific system, and has the potential to occur at some later point with the addition of more tests.
I found that setting the --dist option to a value other than the default of load (when -n auto is also used) avoids having tests hang:
pixi run -e min-deps run-tests --dist loadscope
Using --dist loadfile also works, but causes tests to run more slowly than using loadscope.
For reference, here's the relevant section of the pytest help message:
$ pixi run -e min-deps pytest -h
...
distributed and subprocess testing:
-n, --numprocesses numprocesses
Shortcut for '--dist=load --tx=NUM*popen'.
With 'logical', attempt to detect logical CPU count (requires psutil, falls back to 'auto').
With 'auto', attempt to detect physical CPU count. If physical CPU count cannot be determined, falls back to 1.
Forced to 0 (disabled) when used with --pdb.
--maxprocesses=maxprocesses
Limit the maximum number of workers to process the tests when using --numprocesses with 'auto' or 'logical'
--max-worker-restart=MAXWORKERRESTART
Maximum number of workers that can be restarted when crashed (set to zero to disable this feature)
--dist=distmode Set mode for distributing tests to exec environments.
each: Send each test to all available environments.
load: Load balance by sending any pending test to any available environment.
loadscope: Load balance by sending pending groups of tests in the same scope to any available environment.
loadfile: Load balance by sending test grouped by file to any available environment.
loadgroup: Like 'load', but sends tests marked with 'xdist_group' to the same worker.
worksteal: Split the test suite between available environments, then re-balance when any worker runs out of tests.
(default) no: Run tests inprocess, don't distribute.
--tx=xspec Add a test execution environment. Some examples:
--tx popen//python=python2.5 --tx socket=192.168.1.102:8888
--tx [email protected]//chdir=testcache
...
I'm almost certain that adding this option when running on Linux, will not cause a problem, so I'm going to open a PR that simply adds the option to the task in pyproject.yml so that we can better avoid test hangs across both OSs (macOS and Linux).
More broadly, I'm also fairly confident that this issue is related to #390, as I still see a large number of warnings related to failing to close open files, thus I will also submit yet another PR to further eliminate these occurrences.
The text was updated successfully, but these errors were encountered:
chuckwondo
added a commit
to chuckwondo/VirtualiZarr
that referenced
this issue
Apr 7, 2025
On macOS, running tests in the
min-deps
pixi env consistently hang (at about the 95% mark on my machine). To reproduce, run the following on a macOS machine (YMMV, depending on hardware and OS version):This appears to be an OS-specific issue that surfaces with having pytest run tests in parallel across multiple CPUs because while this hangs on my macOS machine (current code on
develop
branch), the GitHub workflow does not hang (Linux OS).Further, this also appears to occur only for the
min-deps
pixi env, but without understanding why this does not (currently) happen in other pixi envs, I imagine that it could occur in other envs, depending on the specific system, and has the potential to occur at some later point with the addition of more tests.I found that setting the
--dist
option to a value other than the default ofload
(when-n auto
is also used) avoids having tests hang:Using
--dist loadfile
also works, but causes tests to run more slowly than usingloadscope
.For reference, here's the relevant section of the
pytest
help message:I'm almost certain that adding this option when running on Linux, will not cause a problem, so I'm going to open a PR that simply adds the option to the task in pyproject.yml so that we can better avoid test hangs across both OSs (macOS and Linux).
More broadly, I'm also fairly confident that this issue is related to #390, as I still see a large number of warnings related to failing to close open files, thus I will also submit yet another PR to further eliminate these occurrences.
The text was updated successfully, but these errors were encountered: