Parallel import of modules causes a ModuleNotFoundError

[eendebakpt]

Bug report

Bug description:

Using the free-threading build parallel import of modules causes a ModuleNotFoundError. The module that cannot be found can differ. A minimal reproducer:

import sys
from threading import Thread, Barrier
from importlib import import_module, reload
import time


# Setup threads
t0 = time.perf_counter()
number_of_threads = 8
barrier = Barrier(number_of_threads)

pmods = []


def work(ii):
    # Method doing the actual import
    barrier.wait()
    while True:
        try:
            m = pmods.pop()
            mod = import_module(m)
            # print(f'  {ii}: {m} done')
        except IndexError:
            return


worker_threads = []
for ii in range(number_of_threads):
    worker_threads.append(Thread(target=work, args=[ii]))

dt = time.perf_counter() - t0
print(f"setup threads {dt*1e3:.2f}")


def parallel_import(modules: list[str]):
    global pmods
    pmods += modules
    for t in worker_threads:
        t.start()
    for t in worker_threads:
        t.join()


def seq_import(modules: list[str]):
    for m in modules:
        mod = import_module(m)


import_method = seq_import
import_method = parallel_import

mods = [
    "abc",
    "functools",
    "weakref",
    "linecache",
    "glob",
    "annotationlib",
    "argparse",
    "fnmatch",
    "itertools",
    "operator",
    "string",
    "re",
    "collections",
    "sqlite3",
    "pathlib",
    "urllib",
    "typing",
    "csv",
    "uuid",
]

mods += [
    "setuptools",
    "sympy",
    "django",
    "boto3",
]  # these are some packages that need to be installed. without these the reproduction is the issue is much harder

t0 = time.perf_counter()
import_method(mods)
dt = time.perf_counter() - t0
print(f"{import_method.__name__} {dt*1e3:.2f}")

Note: this script fails on my system about 1 in 4 times. Changing the modules imported (or the order, or the number of threads) can effect this.

Modules that have given issues: collections.abs, glob, sympy, django.utils.regex_helper. This suggests it is a general issue, and not a particular package.

CPython versions tested on:

CPython main branch

Operating systems tested on:

Windows

Linked PRs

• gh-130094: Fix race conditions in importlib #130101

[colesbury]

I'm seeing failures in the GIL-enabled build too (3.14.0a5 from uv). I made some slight modifications to the script to exit on error and set sys.setswitchinterval to a small value:

https://gist.github.com/colesbury/90493ef5b6f818bbbfde6922ffc8ea8a

setup threads 0.05
  4: sympy failed: SymPy now depends on mpmath as an external library. See https://docs.sympy.org/latest/install.html#mpmath for more information.
Traceback (most recent call last):
  File "/raid/sgross/tmp/.venv/lib/python3.14/site-packages/sympy/__init__.py", line 22, in <module>
    import mpmath
  File "/raid/sgross/tmp/.venv/lib/python3.14/site-packages/mpmath/__init__.py", line 5, in <module>
    from .ctx_fp import FPContext
  File "/raid/sgross/tmp/.venv/lib/python3.14/site-packages/mpmath/ctx_fp.py", line 1, in <module>
    from .ctx_base import StandardBaseContext
  File "/raid/sgross/tmp/.venv/lib/python3.14/site-packages/mpmath/ctx_base.py", line 3, in <module>
    from .libmp.backend import xrange
  File "/raid/sgross/tmp/.venv/lib/python3.14/site-packages/mpmath/libmp/__init__.py", line 40, in <module>
    from .libhyper import (NoConvergence, make_hyp_summator,
    ...<3 lines>...
      mpf_ellipk, mpc_ellipk, mpf_ellipe, mpc_ellipe)
ModuleNotFoundError: No module named 'mpmath.libmp.libhyper'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/raid/sgross/tmp/parallel_imports.py", line 25, in work
    mod = import_module(m)
  File "/private/home/sgross/.local/share/uv/python/cpython-3.14.0a5-linux-x86_64-gnu/lib/python3.14/importlib/__init__.py", line 88, in import_module
    return _bootstrap._gcd_import(name[level:], package, level)
           ~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<frozen importlib._bootstrap>", line 1386, in _gcd_import
  File "<frozen importlib._bootstrap>", line 1359, in _find_and_load
  File "<frozen importlib._bootstrap>", line 1330, in _find_and_load_unlocked
  File "<frozen importlib._bootstrap>", line 935, in _load_unlocked
  File "<frozen importlib._bootstrap_external>", line 762, in exec_module
  File "<frozen importlib._bootstrap>", line 488, in _call_with_frames_removed
  File "/raid/sgross/tmp/.venv/lib/python3.14/site-packages/sympy/__init__.py", line 24, in <module>
    raise ImportError("SymPy now depends on mpmath as an external library. "
    "See https://docs.sympy.org/latest/install.html#mpmath for more information.")
ImportError: SymPy now depends on mpmath as an external library. See https://docs.sympy.org/latest/install.html#mpmath for more information.

[colesbury]

One race condition is in setuptools, which can add and removes entries in sys.meta_path. If that happens concurrently with the iteration of sys.meta_path, we can end up skipping entries and failing to import a module:

https://github.com/pypa/setuptools/blob/a3cc40f4b8fc90e8f57e3a59a54980fb1f408207/_distutils_hack/__init__.py#L226-L234

cpython/Lib/importlib/_bootstrap.py

Lines 1254 to 1262 in 05e89c3

	for finder in meta_path:
	with _ImportLockContext():
	try:
	find_spec = finder.find_spec
	except AttributeError:
	continue
	else:
	spec = find_spec(name, path, target)
	if spec is not None:

We should probably make a copy of sys.meta_path before iterating over it

[colesbury]

@eendebakpt - can you try the linked PR (#130101) and verify that it fixes the issue? I haven't seen any import errors locally with it anymore.

[colesbury]

This should be fixed in main now.