InterpreterPoolExecutor does not work as replacement of ThreadPoolExecutor.

[corona10]

According to PEP-734, it should work as same as ThreadPoolExecutor, but it looks not.

Code

import concurrent.futures
from concurrent.futures import InterpreterPoolExecutor, ThreadPoolExecutor


def fib(n):
    if n < 2:
        return n
    return fib(n-1) + fib(n-2)

with InterpreterPoolExecutor(max_workers=5) as executor:
    futures = {executor.submit(fib, n): n for n in range(10)}
    for future in concurrent.futures.as_completed(futures):
        n = futures[future]
        data = future.result()
        print(f"fib({n}): {data}")

Execution (InterpreterPoolExecutor)

Exception: AttributeError: module '__main__' has no attribute 'fib'

The above exception was the direct cause of the following exception:

concurrent.interpreters.NotShareableError: object could not be unpickled

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/Users/xxxx/oss/cpython/op.py", line 14, in <module>
    data = future.result()
  File "/Users/xxxx/oss/cpython/Lib/concurrent/futures/_base.py", line 443, in result
    return self.__get_result()
           ~~~~~~~~~~~~~~~~~^^
  File "/Users/xxxx/oss/cpython/Lib/concurrent/futures/_base.py", line 395, in __get_result
    raise self._exception
  File "/Users/xxxx/oss/cpython/Lib/concurrent/futures/thread.py", line 86, in run
    result = ctx.run(self.task)
  File "/Users/denny.na/oss/cpython/Lib/concurrent/futures/interpreter.py", line 85, in run
    return self.interp.call(do_call, self.results, *task)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xxxx/oss/cpython/Lib/concurrent/interpreters/__init__.py", line 233, in call
    return self._call(callable, args, kwargs)
           ~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/xxxx/oss/cpython/Lib/concurrent/interpreters/__init__.py", line 215, in _call
    res, excinfo = _interpreters.call(self._id, callable, args, kwargs, restrict=True)
                   ~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
concurrent.interpreters.NotShareableError: args not shareable

Execution (ThreadPoolExecutor)

fib(0): 0
fib(7): 13
fib(2): 1
fib(6): 8
fib(1): 1
fib(4): 3
fib(3): 2
fib(5): 5
fib(9): 34
fib(8): 21

[corona10]

cc @hugovk We may need to add label release blocker for 3.14

[xuantengh]

I think this is because do_call is treated as a stateful function that cannot be shared between multiple interpreters:

$3 = {total = 14, locals = {total = 5, args = {total = 4, numposonly = 0, numposorkw = 4, numkwonly = 0, varargs = 0, varkwargs = 0}, numpure = 1, cells = {total = 0, numargs = 0, numothers = 0},
    hidden = {total = 0, numpure = 0, numcells = 0}}, numfree = 0, unbound = {total = 9, globals = {total = 5, numglobal = 3, numbuiltin = 3, numunknown = 0}, numattrs = 4, numunknown = 0}}

The _interp_call_pack function will check whether the function is stateless:

cpython/Modules/_interpretersmodule.c

Lines 467 to 488 in 42b251b

	static int
	_interp_call_pack(PyThreadState *tstate, struct interp_call *call,
	PyObject *func, PyObject *args, PyObject *kwargs)
	{
	xidata_fallback_t fallback = _PyXIDATA_FULL_FALLBACK;
	assert(call->func == NULL);
	assert(call->args == NULL);
	assert(call->kwargs == NULL);
	// Handle the func.
	if (!PyCallable_Check(func)) {
	_PyErr_Format(tstate, PyExc_TypeError,
	"expected a callable, got %R", func);
	return -1;
	}
	if (_PyFunction_GetXIData(tstate, func, &call->_preallocated.func) < 0) {
	PyObject *exc = _PyErr_GetRaisedException(tstate);
	if (_PyPickle_GetXIData(tstate, func, &call->_preallocated.func) < 0) {
	_PyErr_SetRaisedException(tstate, exc);
	return -1;
	}
	Py_DECREF(exc);
	}

_interp_call_pack -> _PyFunction_GetXIData -> _PyFunction_VerifyStateless -> _PyCode_VerifyStateless -> _PyCode_CheckNoExternalState

where do_call fails due to:

cpython/Objects/codeobject.c

Lines 2006 to 2008 in 42b251b

	else if (counts->unbound.globals.numglobal > 0) {
	errmsg = "globals not supported";
	}

And the unbound vars seem to be: ['BaseException', 'interpreters', 'print', 'sys', 'traceback']

[corona10]

@xuantengh
I am waiting for @ericsnowcurrently 's reply. If this is the intention, most people will treat InterpreterPoolExecutor as the same as ThreadPoolExecutor, but actual behavior does not seem to be so; it should be documented.
But if it is not intended, it should be resolved before officially releasing 3.14.

[efimov-mikhail]

JFTR: it works if we slightly rewrite this code:

import concurrent.futures
from concurrent.futures import InterpreterPoolExecutor, ThreadPoolExecutor

def calc_fib(n):
    def fib(n):
        if n < 2:
            return n
        return fib(n-1) + fib(n-2)

    return fib(n)

with InterpreterPoolExecutor(max_workers=5) as executor:
    futures = {executor.submit(calc_fib, n): n for n in range(10)}
    for future in concurrent.futures.as_completed(futures):
        n = futures[future]
        data = future.result()
        print(f"fib({n}): {data}")

[Tapeline]

As fib accesses the global scope to get itself it won't work. Thx to @efimov-mikhail for his example. I think we should consider adding some note to docs/howto about subinterpreters, since recursive functions are pretty commonly used for calculations that we want to parallelise.

[xuantengh]

I think we should consider adding some note to docs/howto about subinterpreters, since recursive functions are pretty commonly used for calculations that we want to parallelise.

Maybe I can file a PR for this?

[paultiq]

There used to be a comment in concurrent.futures.rst for InterpreterPoolExecutor saying: Functions defined in the main module cannot be pickled and thus cannot be used.

This comment was removed in #133957

[Tapeline]

I think we should consider adding some note to docs/howto about subinterpreters, since recursive functions are pretty commonly used for calculations that we want to parallelise.

Maybe I can file a PR for this?

It would be great, but it totally needs prior discussion. I think we should add a big section about recipes with subinterpreters, because there are more problems, than just defining recursive functions. E.g. top-level imports don't work either for the same reason. We should define preferred way to handle all these cases (ofc the discussion should involve Eric Snow and others involved in subinterpreters). We need to open another issue (or a discuss.python.org topic, potentially) :)