multiprocessing producing EOFError #29

DavidEHayden · 2022-10-18T19:07:18Z

Using a fresh install of python 3.10, I install numpy and then nctoolkit using pip3, as per the installation instructions. Then running "import nctoolkit as nc" I get the following error:

Traceback (most recent call last):
File "", line 1, in
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 116, in spawn_main
exitcode = _main(fd, parent_sentinel)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 125, in _main
prepare(preparation_data)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 236, in prepare
_fixup_main_from_path(data['init_main_from_path'])
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 287, in _fixup_main_from_path
main_content = runpy.run_path(main_path,
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 289, in run_path
return _run_module_code(code, init_globals, run_name,
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 96, in _run_module_code
_run_code(code, mod_globals, init_globals,
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/runpy.py", line 86, in _run_code
exec(code, run_globals)
File "/Users/davidhayden/Desktop/Dataset/PythonProject/main.py", line 2, in
import nctoolkit
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/nctoolkit/init.py", line 1, in
from nctoolkit.api import (
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/nctoolkit/api.py", line 18, in
from nctoolkit.cleanup import cleanup, clean_all, temp_check
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/nctoolkit/cleanup.py", line 9, in
from nctoolkit.remove import nc_remove
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/nctoolkit/remove.py", line 2, in
from nctoolkit.session import session_info, get_tempdirs, get_safe
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/nctoolkit/session.py", line 7, in
mgr = Manager()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", line 57, in Manager
m.start()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/managers.py", line 562, in start
self._process.start()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", line 288, in _Popen
return Popen(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 32, in init
super().init(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/popen_spawn_posix.py", line 42, in _launch
prep_data = spawn.get_preparation_data(process_obj._name)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 154, in get_preparation_data
_check_not_importing_main()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/spawn.py", line 134, in _check_not_importing_main
raise RuntimeError('''
RuntimeError:
An attempt has been made to start a new process before the
current process has finished its bootstrapping phase.

    This probably means that you are not using fork to start your
    child processes and you have forgotten to use the proper idiom
    in the main module:

        if __name__ == '__main__':
            freeze_support()
            ...

    The "freeze_support()" line can be omitted if the program
    is not going to be frozen to produce an executable.

Traceback (most recent call last):
File "/Users/davidhayden/Desktop/Dataset/PythonProject/main.py", line 2, in
import nctoolkit
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/nctoolkit/init.py", line 1, in
from nctoolkit.api import (
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/nctoolkit/api.py", line 18, in
from nctoolkit.cleanup import cleanup, clean_all, temp_check
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/nctoolkit/cleanup.py", line 9, in
from nctoolkit.remove import nc_remove
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/nctoolkit/remove.py", line 2, in
from nctoolkit.session import session_info, get_tempdirs, get_safe
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/site-packages/nctoolkit/session.py", line 7, in
mgr = Manager()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/context.py", line 57, in Manager
m.start()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/managers.py", line 566, in start
self._address = reader.recv()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/connection.py", line 255, in recv
buf = self._recv_bytes()
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/connection.py", line 419, in _recv_bytes
buf = self._recv(4)
File "/Library/Frameworks/Python.framework/Versions/3.10/lib/python3.10/multiprocessing/connection.py", line 388, in _recv
raise EOFError
EOFError

I have tried to reinstall python and tried different virtual environments however I always get the same error.

The text was updated successfully, but these errors were encountered:

robertjwilson · 2022-10-18T21:37:05Z

Thanks @DavidEHayden. I am travelling for work right now, but will look into this in a couple of days. It looks like an upgrade to a dependency has created a conflict. Can you confirm your operating system? I can likely fix this problem by figuring out the offending dependency and pinning the version.

DavidEHayden · 2022-10-19T08:34:48Z

Hello, thanks for getting back to me. I am running macOS Monterey 12.6. I have also included my current packages and their versions if this helps in hunting down the dependancy. Thanks for the help.

robertjwilson · 2022-10-19T09:56:17Z

Thanks @DavidEHayden. That should help. I have just updated the macOS testing on github actions to python 3.10 to test it there. An update to a dependency has definitely broken something. Not only does numpy not seem to install in the macOS testing, the Linux tests are now failing. The only thing that will have changed since tests were last run are to the dependencies. I suspect it's the update to numpy a few days ago that has broken things. This should be easy to fix

robertjwilson · 2022-10-21T08:04:54Z

Hi @DavidEHayden. I've just got testing working for Python 3.10 on Monteray: here. It seems to work fine. Tests were actually failing because the package uses CDO as a system dependency and an update to CDO caused one of the tests to fail. So nctoolkit definitely seems fully compatible with the latest versions of packages on pypi.

Based on your error, the difficulty is caused by importing nctoolkit dependencies. nctoolkit itself is not the issue. Though ideally I want to be able to ensure problem-versions of dependencies are excluded when installing nctoolkit.

Are you able to import multiprocessing?

You are using an old(ish) version of pip, so it's possible something is going wrong at that end.

platipodium · 2023-05-27T13:05:30Z

I am experiencing the same error on Mac M1 running Ventura 13.4 and python11 (through mamba + pip git). Showstopper for verifying your submission to JOSS openjournals/joss-reviews#5494

robertjwilson · 2023-06-02T18:27:40Z

I am unable to reproduce this problem @platipodium. I've set up a Mac M1 testing environment on CircleCI and the tests passed OK. This was done with Python 3.11 and Ventura, and the pypi version of nctoolkit.

Test output available here

CircleCI config and install methods here.

Can you provide further info on your environment @platipodium? I can figure out if there is a problem with specific dependency versions. This is an error from subprocess, not nctoolkit. So this is almost certainly something external to nctoolkit, but one that maybe needs solved by dependency pinning.

platipodium · 2023-06-02T19:53:35Z

My reproducible installation:

Steps to reproduce

conda create -n nctoolkit python=3.11
conda activate nctoolkit
conda install mamba
mamba install nctoolkit
pip install git+https://github.com/pmlmodelling/ncplot.git
pip install git+https://github.com/pmlmodelling/nctoolkit.git
echo "import nctoolkit" >> test.py

platipodium · 2023-06-02T19:54:17Z

But my bad, I tried to run nctoolkit as part of a script ...

python test.py
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/spawn.py", line 120, in spawn_main
    exitcode = _main(fd, parent_sentinel)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/spawn.py", line 129, in _main
    prepare(preparation_data)
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/spawn.py", line 240, in prepare
    _fixup_main_from_path(data['init_main_from_path'])
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/spawn.py", line 291, in _fixup_main_from_path
    main_content = runpy.run_path(main_path,
                   ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen runpy>", line 291, in run_path
  File "<frozen runpy>", line 98, in _run_module_code
  File "<frozen runpy>", line 88, in _run_code
  File "/Users/Lemmen/devel/python/nctoolkit-test/nctoolkit-test.py", line 10, in <module>
    import nctoolkit as nc
  File "/opt/conda/envs/nctoolkit/lib/python3.11/site-packages/nctoolkit/__init__.py", line 1, in <module>
    from nctoolkit.api import (
  File "/opt/conda/envs/nctoolkit/lib/python3.11/site-packages/nctoolkit/api.py", line 17, in <module>
    from nctoolkit.cleanup import cleanup, clean_all, temp_check
  File "/opt/conda/envs/nctoolkit/lib/python3.11/site-packages/nctoolkit/cleanup.py", line 9, in <module>
    from nctoolkit.remove import nc_remove
  File "/opt/conda/envs/nctoolkit/lib/python3.11/site-packages/nctoolkit/remove.py", line 2, in <module>
    from nctoolkit.session import session_info, get_tempdirs, get_safe
  File "/opt/conda/envs/nctoolkit/lib/python3.11/site-packages/nctoolkit/session.py", line 7, in <module>
    mgr = Manager()
          ^^^^^^^^^
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/managers.py", line 563, in start
    self._process.start()
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/process.py", line 121, in start
    self._popen = self._Popen(self)
                  ^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/context.py", line 288, in _Popen
    return Popen(process_obj)
           ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 32, in __init__
    super().__init__(process_obj)
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/popen_fork.py", line 19, in __init__
    self._launch(process_obj)
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/popen_spawn_posix.py", line 42, in _launch
    prep_data = spawn.get_preparation_data(process_obj._name)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/spawn.py", line 158, in get_preparation_data
    _check_not_importing_main()
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/spawn.py", line 138, in _check_not_importing_main
    raise RuntimeError('''
RuntimeError:
        An attempt has been made to start a new process before the
        current process has finished its bootstrapping phase.

        This probably means that you are not using fork to start your
        child processes and you have forgotten to use the proper idiom
        in the main module:

            if __name__ == '__main__':
                freeze_support()
                ...

        The "freeze_support()" line can be omitted if the program
        is not going to be frozen to produce an executable.
Traceback (most recent call last):
  File "/Users/Lemmen/devel/python/nctoolkit-test/nctoolkit-test.py", line 10, in <module>
    import nctoolkit as nc
  File "/opt/conda/envs/nctoolkit/lib/python3.11/site-packages/nctoolkit/__init__.py", line 1, in <module>
    from nctoolkit.api import (
  File "/opt/conda/envs/nctoolkit/lib/python3.11/site-packages/nctoolkit/api.py", line 17, in <module>
    from nctoolkit.cleanup import cleanup, clean_all, temp_check
  File "/opt/conda/envs/nctoolkit/lib/python3.11/site-packages/nctoolkit/cleanup.py", line 9, in <module>
    from nctoolkit.remove import nc_remove
  File "/opt/conda/envs/nctoolkit/lib/python3.11/site-packages/nctoolkit/remove.py", line 2, in <module>
    from nctoolkit.session import session_info, get_tempdirs, get_safe
  File "/opt/conda/envs/nctoolkit/lib/python3.11/site-packages/nctoolkit/session.py", line 7, in <module>
    mgr = Manager()
          ^^^^^^^^^
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/managers.py", line 567, in start
    self._address = reader.recv()
                    ^^^^^^^^^^^^^
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/connection.py", line 249, in recv
    buf = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/connection.py", line 413, in _recv_bytes
    buf = self._recv(4)
          ^^^^^^^^^^^^^
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/connection.py", line 382, in _recv
    raise EOFError
EOFError

So I guess I am not supposed to do that?

platipodium · 2023-06-02T19:54:48Z

Working interactively is fine :=)

python
>> import nctoolkit
>> quit()

robertjwilson · 2023-06-05T09:07:18Z

OK, I'm able to reproduce the scripting issue @platipodium.

I've pinned it down to behaviour in the subprocess package. This script (pure subprocess) fails in macOS when it is run in a script: https://github.com/robertjwilson/nctoolkit/blob/master/testing/test_scripting.py.

However, running it interactively or from pytest is totally fine. On linux, no problems at all. I'll investigate this today and see if there is a workaround.

This only applies to doing things in parallel in nctoolkit, so if it's a bug in subprocess without an easy workaround, I can just turn off the ability to do things in parallel in macOS for now.

platipodium · 2023-06-05T12:05:45Z

@robertjwilson I think it would be totally fine to document this properly or catch it when the script is trying to execute on macOS in parallel. While it is not fixed, of course.

robertjwilson · 2023-06-06T07:43:32Z

I have released version 0.9.3 which should fully resolve this issue @platipodium. The problem appears to be a bug within multiprocessing on macOS. After investigating it for a couple of hours, I found a simple solution is to just replace multiprocessing with multiprocess. All that was needed to fix the issue was to change the import from import multiprocessing as mp to import multiprocess as mp on macOS. Linux still uses multiprocessing.

https://github.com/pmlmodelling/nctoolkit/blob/master/testing/test_parallel.py

The new version is on pypi and conda.

This should sort the issue out, but ideally I don't want another dependency, so I'll keep this open or open another issue to figure out how to drop multiprocess.

platipodium · 2023-06-06T08:16:16Z

The error persists for me, after updating to 0.9.3

pip install git+https://github.com/pmlmodelling/nctoolkit.git

python
>>> import nctoolkit
nctoolkit is using Climate Data Operators version 2.2.0
>>> nctoolkit.__version__
'0.9.3'
>>> quit()

python test_parallel.py
...
  File "/opt/conda/envs/nctoolkit/lib/python3.11/site-packages/nctoolkit/session.py", line 7, in <module>
    mgr = Manager()
          ^^^^^^^^^
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/context.py", line 57, in Manager
    m.start()
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/managers.py", line 567, in start
    self._address = reader.recv()
                    ^^^^^^^^^^^^^
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/connection.py", line 249, in recv
    buf = self._recv_bytes()
          ^^^^^^^^^^^^^^^^^^
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/connection.py", line 413, in _recv_bytes
    buf = self._recv(4)
          ^^^^^^^^^^^^^
  File "/opt/conda/envs/nctoolkit/lib/python3.11/multiprocessing/connection.py", line 382, in _recv
    raise EOFError
EOFError

So it is still loading multiprocessing not multiprocess (which was properly installed by pip as a dependency)

platipodium · 2023-06-06T08:18:30Z

I have released version 0.9.3 which should fully resolve this issue @platipodium. The problem appears to be a bug within multiprocessing on macOS. After investigating it for a couple of hours, I found a simple solution is to just replace multiprocessing with multiprocess. All that was needed to fix the issue was to change the import from import multiprocessing as mp to import multiprocess as mp on macOS. Linux still uses multiprocessing.

https://github.com/pmlmodelling/nctoolkit/blob/master/testing/test_parallel.py

The new version is on pypi and conda.

This should sort the issue out, but ideally I don't want another dependency, so I'll keep this open or open another issue to figure out how to drop multiprocess.

Please create a new issue for this, such that we can close this one and it appears on the JOSS review issues as closed, as well. Once this is fixed :=)

robertjwilson · 2023-06-06T08:22:19Z

Can you double check that @platipodium? It looks like your pip install has not actually installed the latest version. Line 7 in session.py is actually blank. https://github.com/pmlmodelling/nctoolkit/blob/master/nctoolkit/session.py

I think you need to add --force-reinstall or something to that effect to the pip call.

platipodium · 2023-06-06T08:55:44Z

Sure, but just FYI:

> mamba list
nctoolkit                 0.9.3                    pypi_0    pypi

So the version seems to be the updated one ...

> pip install --force-reinstall git+https://github.com/pmlmodelling/nctoolkit.git
...
  Attempting uninstall: nctoolkit
    Found existing installation: nctoolkit 0.9.3
    Uninstalling nctoolkit-0.9.3:
      Successfully uninstalled nctoolkit-0.9.3
...

> mamba list |grep nctool
nctoolkit                 0.9.3                    pypi_0    pypi

So without an apparent change in the version, the issue is now fixed. So something is off, but this issue can be closed.

robertjwilson · 2023-07-12T14:30:47Z

This appears to have been fixed. I will close it for. If @DavidEHayden is still having it I will reopen.

platipodium mentioned this issue May 27, 2023

[REVIEW]: nctoolkit: A Python package for netCDF analysis and post-processing openjournals/joss-reviews#5494

Closed

robertjwilson mentioned this issue Jun 21, 2023

[JOSS] Can not use multiprocessing #82

Closed

robertjwilson closed this as completed Jul 12, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

multiprocessing producing EOFError #29

multiprocessing producing EOFError #29

DavidEHayden commented Oct 18, 2022

robertjwilson commented Oct 18, 2022

DavidEHayden commented Oct 19, 2022

robertjwilson commented Oct 19, 2022

robertjwilson commented Oct 21, 2022

platipodium commented May 27, 2023

robertjwilson commented Jun 2, 2023

platipodium commented Jun 2, 2023

platipodium commented Jun 2, 2023

platipodium commented Jun 2, 2023

robertjwilson commented Jun 5, 2023

platipodium commented Jun 5, 2023

robertjwilson commented Jun 6, 2023

platipodium commented Jun 6, 2023

platipodium commented Jun 6, 2023

robertjwilson commented Jun 6, 2023

platipodium commented Jun 6, 2023

robertjwilson commented Jul 12, 2023

multiprocessing producing EOFError #29

multiprocessing producing EOFError #29

Comments

DavidEHayden commented Oct 18, 2022

robertjwilson commented Oct 18, 2022

DavidEHayden commented Oct 19, 2022

robertjwilson commented Oct 19, 2022

robertjwilson commented Oct 21, 2022

platipodium commented May 27, 2023

robertjwilson commented Jun 2, 2023

platipodium commented Jun 2, 2023

Steps to reproduce

platipodium commented Jun 2, 2023

platipodium commented Jun 2, 2023

robertjwilson commented Jun 5, 2023

platipodium commented Jun 5, 2023

robertjwilson commented Jun 6, 2023

platipodium commented Jun 6, 2023

platipodium commented Jun 6, 2023

robertjwilson commented Jun 6, 2023

platipodium commented Jun 6, 2023

robertjwilson commented Jul 12, 2023