Cannot use GaussianMixtures.jl inside Pluto because parallelization uses ALL workers

[jamblejoe]

Plut.jl is a reactive notebook server written in Julia. It spawns different notebooks in separate Julia processes. The by default enabled parallelization of em! tries to access other running processes. Because most likely GaussianMixtures.jl is not defined in these processes, em! errors. But worse is trying to run code in other notebook processes at all. I fixed that for me now by disabling process parallelization altogether. In the future, more fine-grained control of which Julia processes should be used to parallelize em! would be great!

[davidavdav]

You are probably right. This package was written in the Julia-0.3 era. The code is probably trying to parallelize at severar places beyond the control of the user.

Is it that your notebook server runs multiple threads?

Perhaps something like @everywhere using GaussianMixtures would help for now?

I suppose I didn't have a clue on how to manage package-global settings like "Use so many threads for this package".

[jamblejoe]

@davidavdav First of all thank you for the effort you put into this package. I am actually surprised that it still works so well considering that I run Julia-1.9. So, the Pluto notebook server runs in Julia. Every notebook is started in a separate Julia process so that they are encapsulated and do not interact with each other. This allows fine-grained control of code loading, e.g. one Project.toml for every notebook.

It seems that Pluto is using Distributed.jl under the hood, but apparently, there is another way of spawning Julia processes for the individual notebooks fonsp/Pluto.jl#300, which uses Malt.jl. This allows spawning workers in each notebook with Distributed.jl. I will give it a try.

[BioTurboNick]

I suppose this is the issue I just started running into. It will work when I first load the notebook, but fails thereafter - possibly because other notebooks have been opened.