[JOSS] Can not use multiprocessing #82
Comments
|
Thanks for spotting this. I changed the nctoolkit internals for macOS 2 weeks ago to fix this bug on macOS: #29 But I forgot to update this doc. Essentially, nctoolkit now uses multiprocess when running in parallel on macOS, but multiprocessing on Linux, due to weird differences in behaviour introduced in Python 3.7. So the example needs to use multiprocess on macOS. I've just added this to the docs: https://nctoolkit.readthedocs.io/en/latest/parallel.html |
|
OK, multiprocess does not crash like before. But I think I'm misunderstanding this example. This is copied and pasted from the documentation. import nctoolkit as nc
nc.options(parallel = True)
def process_chain(infile, outfile):
ds = nc.open_data(ff)
ds.assign(tos = lambda x: x.sst + 273.15)
ds.tmean()
ds.to_nc(outfile)
ensemble = nc.create_ensemble("data/ensemble")
import multiprocess as mp
pool = mp.Pool(2)
for ff in ensemble:
pool.apply_async(process_chain, [ff, ff.replace("ensemble", "new")])
pool.close()
pool.join()Is this assertion correct? Because it fails on my machine, so import os
assert os.path.exists("data/new")Also, is this correct? |
|
Ahhh I see, I need to create the directory But I think there's still a problem because there are silent failures. |
|
I've modified the example so it generates directories and the function is now fixed https://nctoolkit.readthedocs.io/en/latest/parallel.html |
|
I don't think it's fixed. The new directory has the wrong path. As I mentioned in my previous comment, isn't it an issue that the code silently fails? |
|
OK. Git screw up. I thought a fix was commited and pushed but it wasn't. Now definitely fixed https://nctoolkit.readthedocs.io/en/latest/parallel.html If this is still failing silently can you confirm the OS/Python? |
|
The code works now. Still, if I add a bug on purpose in the function, no Exception is raised (python 3.11, macOS). |
Describe the bug
I'm not able to run this: https://nctoolkit.readthedocs.io/en/latest/parallel.html#parallel-processing-using-multiprocessing
I tried both from jupyterlab and from ipython.
To Reproduce
where
data/ensembleis this.I get the following (truncated here because it's too long for GH issues), and then it just hangs:
Desktop (please complete the following information):
Additional context
openjournals/joss-reviews#5494
The text was updated successfully, but these errors were encountered: