Skip to content

tests may hang on macOS #535

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
chuckwondo opened this issue Apr 7, 2025 · 0 comments · Fixed by #536
Closed

tests may hang on macOS #535

chuckwondo opened this issue Apr 7, 2025 · 0 comments · Fixed by #536

Comments

@chuckwondo
Copy link
Contributor

On macOS, running tests in the min-deps pixi env consistently hang (at about the 95% mark on my machine). To reproduce, run the following on a macOS machine (YMMV, depending on hardware and OS version):

pixi run -e min-deps run-tests

This appears to be an OS-specific issue that surfaces with having pytest run tests in parallel across multiple CPUs because while this hangs on my macOS machine (current code on develop branch), the GitHub workflow does not hang (Linux OS).

Further, this also appears to occur only for the min-deps pixi env, but without understanding why this does not (currently) happen in other pixi envs, I imagine that it could occur in other envs, depending on the specific system, and has the potential to occur at some later point with the addition of more tests.

I found that setting the --dist option to a value other than the default of load (when -n auto is also used) avoids having tests hang:

pixi run -e min-deps run-tests --dist loadscope

Using --dist loadfile also works, but causes tests to run more slowly than using loadscope.

For reference, here's the relevant section of the pytest help message:

$ pixi run -e min-deps pytest -h
...
distributed and subprocess testing:
  -n, --numprocesses numprocesses
                        Shortcut for '--dist=load --tx=NUM*popen'.
                        With 'logical', attempt to detect logical CPU count (requires psutil, falls back to 'auto').
                        With 'auto', attempt to detect physical CPU count. If physical CPU count cannot be determined, falls back to 1.
                        Forced to 0 (disabled) when used with --pdb.
  --maxprocesses=maxprocesses
                        Limit the maximum number of workers to process the tests when using --numprocesses with 'auto' or 'logical'
  --max-worker-restart=MAXWORKERRESTART
                        Maximum number of workers that can be restarted when crashed (set to zero to disable this feature)
  --dist=distmode       Set mode for distributing tests to exec environments.
                        each: Send each test to all available environments.
                        load: Load balance by sending any pending test to any available environment.
                        loadscope: Load balance by sending pending groups of tests in the same scope to any available environment.
                        loadfile: Load balance by sending test grouped by file to any available environment.
                        loadgroup: Like 'load', but sends tests marked with 'xdist_group' to the same worker.
                        worksteal: Split the test suite between available environments, then re-balance when any worker runs out of tests.
                        (default) no: Run tests inprocess, don't distribute.
  --tx=xspec            Add a test execution environment. Some examples:
                        --tx popen//python=python2.5 --tx socket=192.168.1.102:8888
                        --tx [email protected]//chdir=testcache
...

I'm almost certain that adding this option when running on Linux, will not cause a problem, so I'm going to open a PR that simply adds the option to the task in pyproject.yml so that we can better avoid test hangs across both OSs (macOS and Linux).

More broadly, I'm also fairly confident that this issue is related to #390, as I still see a large number of warnings related to failing to close open files, thus I will also submit yet another PR to further eliminate these occurrences.

chuckwondo added a commit to chuckwondo/VirtualiZarr that referenced this issue Apr 7, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant