Creating venvs via standard library's `venv.create(..., with_pip=True)` fails for binaries linked against libpython dynamically

[smheidrich]

Minimal example:

import subprocess
import venv
from tempfile import mkdtemp
from traceback import print_exc

tmp_dir_path = mkdtemp(prefix="venv-create-bug-example.")
print(f"creating venv in: {tmp_dir_path}")
try:
  venv.create(tmp_dir_path, with_pip=True)
except subprocess.CalledProcessError as e:
  print_exc()
  print(f"subprocess stdout:\n{e.stdout}")
  print(f"subprocess stderr:\n{e.stderr}")
  exit(1)

Running this with a python-build-standalone binary that has a shared library dependency on libpython*.so, e.g. cpython-3.12.7+20241016-x86_64-unknown-linux-gnu-install_only will make this fail with (extra newlines inserted for readability):

creating venv in: /tmp/venv-create-bug-example.pj058eb3
Traceback (most recent call last):
  ...
subprocess.CalledProcessError: Command
'['/tmp/venv-create-bug-example.pj058eb3/bin/python', '-m', 'ensurepip', '--upgrade', '--default-pip']'
returned non-zero exit status 127.
subprocess stdout:
b'/tmp/venv-create-bug-example.pj058eb3/bin/python:
error while loading shared libraries:
/tmp/venv-create-bug-example.pj058eb3/bin/../lib/libpython3.12.so.1.0:
cannot open shared object file: No such file or directory\n'
...

Obviously, this is because the python binary links against libpython*.so dynamically with a relative path, and there no longer is a libpython*.so at this path when the Python executable (or a symlink to it) finds itself in the venv.

See also the corresponding issue in uv:

• libpython missing uv#6812

[willnode]

Maybe helps

$ cd ~/.pyenv/versions/3.11.10
$ ld bin/python
ld: warning: $ORIGIN/../lib/libpython3.11.so.1.0, needed by bin/python, not found (try using -rpath or -rpath-link)
ld: warning: cannot find entry symbol _start; not setting start address
ld: bin/python: undefined reference to `Py_BytesMain'
$ ln -s lib ../lib
$ export LD_LIBRARY_PATH=~/.pyenv/versions/3.11.10:$LD_LIBRARY_PAT
$ ld bin/python
ld: warning: cannot find entry symbol _start; not setting start address

Adding LD_LIBRARY_PATH works so...

// your test file
$ nano x.py
$ python x.py 
creating venv in: /tmp/venv-create-bug-example.fg3ebyf5
$ 

[geofft]

This is a little bit different from astral-sh/uv#6812. While they're both sort of about libpython.so not being in the venv, the impact is different. In the uv issue, running bin/python works fine, but ldd bin/python reports a spurious issue (which is a bug in ldd), and some build systems are unhappy about not being able to find libpython (which might be because they are trusting ldd, or might be some other, build-system-specific cause). The issue here is that bin/python just doesn't work at all and the venv is completely broken.

You can repro this with the CLI tool:

$ python/bin/python -m venv --copies /tmp/u
Error: Command '['/tmp/u/bin/python', '-m', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 127.
$ /tmp/u/bin/python
/tmp/u/bin/python: error while loading shared libraries: /tmp/u/bin/../lib/libpython3.13.so.1.0: cannot open shared object file: No such file or directory

The reason is that venv.main() decides between copies and symlinks based on os.name == 'nt', whereas venv.EnvBuilder/venv.create just defaults to symlinks=False. This means, incidentally, that you can work around this issue with venv.create(...symlinks=True), which is probably a good idea for its own sake in that it matches the default behavior of python -m venv on UNIX.

Some fun facts about how other distributions of Python approach this:

• Nix/NixOS builds their bin/python linking libpython.so, but because they hard-code paths in /nix and set rpaths, a venv created with --copies works fine because the copied Python binary still has the rpath set pointing to the original location of libpython.so. This won't quite work for us because we don't know in advance where someone is unpacking the interpreter (and for e.g. uv it's in the user's homedir which varies with their username). We could modify the copied bin/python to set an rpath, but making the venv bin/python no longer bit-for-bit identical to the original one seems like it might break something else unexpected.
• The system Python on macOS is patched to prevent using --copies and to change the default behavior of EnvBuilder, and a comment specifically calls out the equivalent problem as the reason they don't allow it:

$ diff -ur <(git -C src/cpython show v3.9.0:Lib/venv/__init__.py) /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/venv/__init__.py
--- /dev/fd/63	2025-01-27 15:22:52
+++ /Applications/Xcode.app/Contents/Developer/Library/Frameworks/Python3.framework/Versions/3.9/lib/python3.9/venv/__init__.py	2024-11-11 06:18:32
@@ -16,6 +16,21 @@
 CORE_VENV_DEPS = ('pip', 'setuptools')
 logger = logging.getLogger(__name__)
 
+def should_use_symlinks(symlinks=None):
+    if symlinks:
+        return True
+    else:
+        # The python we build for Xcode uses @executable_path/../Python3 to find Python3.framework
+        # A venv created without symlinks will not be able to load the framework, because
+        # @executable_path would be in the venv, not in the original location within Xcode.
+        symlinks_are_required = sysconfig.get_config_var('TRAIN_STYLE') == 'DT'
+        if symlinks is None:
+            return symlinks_are_required
+        else:
+            if symlinks_are_required:
+                raise Exception("This build of python cannot create venvs without using symlinks")
+            else:
+                return symlinks
 
 class EnvBuilder:
     """
@@ -44,11 +59,11 @@
     """
 
     def __init__(self, system_site_packages=False, clear=False,
-                 symlinks=False, upgrade=False, with_pip=False, prompt=None,
+                 symlinks=None, upgrade=False, with_pip=False, prompt=None,
                  upgrade_deps=False):
         self.system_site_packages = system_site_packages
         self.clear = clear
-        self.symlinks = symlinks
+        self.symlinks = should_use_symlinks(symlinks)
         self.upgrade = upgrade
         self.with_pip = with_pip
         if prompt == '.':  # see bpo-38901
@@ -404,7 +419,7 @@
 
 
 def create(env_dir, system_site_packages=False, clear=False,
-           symlinks=False, with_pip=False, prompt=None, upgrade_deps=False):
+           symlinks=None, with_pip=False, prompt=None, upgrade_deps=False):
     """Create a virtual environment in a directory."""
     builder = EnvBuilder(system_site_packages=system_site_packages,
                          clear=clear, symlinks=symlinks, with_pip=with_pip,

(I have no idea what TRAIN_STYLE is. Some internal Apple thing about release vs. internal builds? "Developer Tools"?)

[FFY00]

#505 only mitigates the issue, instead of fixing the source.

I feel like the proper fix should be around setting RPATH of the interpreter so that the link can be resolved correctly. This is a bit complex given the relocatable nature of PBS, but given that we are talking about virtual environments, which need to have a base installation, I think it may be reasonable to set the RPATH of the interpreter to point to the base installation.

If the base installation itself is actually meant to be relocatable (as-in it may be moved around), then we might want to warn users about its fragility when creating virtual environments.

[geofft]

To be clear #505 is just working around this issue for this repository's own Python code in the build scripts (when the Python interpreter itself is from python-build-standalone)—it's not intended as a fix for the outputs of this repository / does not change the outputs.

I'm inclined to say that copy-based venvs are just unsupported for python-build-standalone (as they are for Xcode's Pythons), but yeah, having venv do some rpath patching after it creates the environment would also work, at the risk of maybe breaking code signatures or other things that depend on the copy being bit-for-bit identical to the binary we distribute.

[FFY00]

Does that realistically matter in systems where symlinks are available?

Copy-based venvs exist primarily for Windows, where symlink support is problematic. Are there any use-cases for copy-based venvs in systems like macOS, which implement the kind of code signing you mentioned?

If not, I think the best solution is to simply adjust the UX so that users don't run into these scenarios by mistake.

We could fix the issue with a more complex solution like requiring certain tooling to set the RPATH properly, but I think that if we can avoid that kind of complexity, that would be best. Practically, I think this mostly becomes a non-issue if there aren't any real need world use-cases.

[zanieb]

Are there any use-cases for copy-based venvs in systems like macOS, which implement the kind of code signing you mentioned?

Just to share more context, people also want fully portable virtual environments, e.g.,

• Bundle Python Interpreter in virtual environments uv#7865
• Support copy or hardlink python interpreter when creating venv uv#6782

but I think that's distinct from a typical copy-based venv.

[geofft]

Even with python3 -m venv --copies, you're still left with a reference in pyvenv.cfg to the original installation's standard library, so I think the premises of 7865 that the given commands are sufficient to get you a venv that's "self contained and does not rely on the user's local files" is not true and the premise of 6782 that venv --copies does what they want is not true. I definitely see the value of giving users a way to get something that is truly self-contained, but I recall seeing somewhere an argument that this is no longer a virtual environment, and what you want at that point is just another unpack of a python-build-standalone distribution and installing packages straight into that distribution.

I don't know of any use case on UNIX actually wants copy-based venvs (as I argued in python/cpython#129382). My only hesitation was that it's a thing the code currently supports and currently defaults to, but I would be more than happy to break it and say symlinks only. Apple seems to have gotten away with doing that.