Virtualenvs created from links are broken

[konstin]

Unlike a system interpreter or a pyenv installation, pbs can't create venv from a symlink to the pbs installation location. PYTHONHOME is in the wrong location (that of the link, i think), which causes the encoding error.

I've only tested linux, i expect windows isn't affected (different venv mechanism, no symlinks by default), but i expect macos is affected, too.

FROM ubuntu

RUN apt update && apt install -yy wget curl tar python3

# Install pbs in one dir
RUN mkdir /root/a
WORKDIR /root/a
RUN wget https://github.com/indygreg/python-build-standalone/releases/download/20241016/cpython-3.12.7+20241016-x86_64_v3-unknown-linux-gnu-install_only_stripped.tar.gz
RUN tar xf cpython-3.12.7+20241016-x86_64_v3-unknown-linux-gnu-install_only_stripped.tar.gz

# Go to another dir
RUN mkdir /root/b
WORKDIR /root/b

# Symlink the system interpreter for a level of indirection
RUN ln -s /usr/bin/python3.12 python3.12
RUN /usr/bin/python3.12 -m venv --without-pip python3.12-venv-direct
RUN ./python3.12 -m venv --without-pip python3.12-venv-link

# Symlink the pds interpreter for a level of indirection
RUN ln -s /root/a/python/bin/python3 pbs3.12
RUN /root/a/python/bin/python -m venv --without-pip pbs3.12-venv-direct
RUN ./pbs3.12 -m venv --without-pip pbs3.12-venv-link

CMD ["bash", "-c", "./python3.12-venv-direct/bin/python -c 'import sys; print(1, sys._base_executable)' \
    && ./python3.12-venv-link/bin/python -c 'import sys; print(2, sys._base_executable)' \
    && ./pbs3.12-venv-direct/bin/python -c 'import sys; print(3, sys._base_executable)' \
    && ./pbs3.12-venv-link/bin/python -c 'import sys; print(4, sys._base_executable)'"]

1 /usr/bin/python3.12
2 /usr/bin/python3.12
3 /root/a/python/bin/python3.12
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = './pbs3.12-venv-link/bin/python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/install/lib/python3.12'
  sys._base_executable = '/root/a/python/bin/python3.12'
  sys.base_prefix = '/install'
  sys.base_exec_prefix = '/install'
  sys.platlibdir = 'lib'
  sys.executable = '/root/b/pbs3.12-venv-link/bin/python'
  sys.prefix = '/install'
  sys.exec_prefix = '/install'
  sys.path = [
    '/install/lib/python312.zip',
    '/install/lib/python3.12',
    '/install/lib/python3.12/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007dce45bbb740 (most recent call first):
  <no Python frame>

[zanieb]

Do you think this is a case of astral-sh/uv#8429 or are you suggesting we can fix this at build time?

[konstin]

I expect a sys variable is resolved incorrectly during interpreter startup, causing the wrong home in pyvenv.cfg. Ideally, we fix this in the pbs interpreter startup sequence itself, so that the venv module gets the right paths:

$ for file in *venv*/pyvenv.cfg; do echo && echo $file && cat $file; done

pbs3.12-venv-direct/pyvenv.cfg
home = /root/a/python/bin
include-system-site-packages = false
version = 3.12.7
executable = /root/a/python/bin/python3.12
command = /root/a/python/bin/python -m venv --without-pip /root/b/pbs3.12-venv-direct

pbs3.12-venv-link/pyvenv.cfg
home = /root/b
include-system-site-packages = false
version = 3.12.7
executable = /root/a/python/bin/python3.12
command = /root/b/pbs3.12 -m venv --without-pip /root/b/pbs3.12-venv-link

python3.12-venv-direct/pyvenv.cfg
home = /usr/bin
include-system-site-packages = false
version = 3.12.3
executable = /usr/bin/python3.12
command = /usr/bin/python3.12 -m venv --without-pip /root/b/python3.12-venv-direct

python3.12-venv-link/pyvenv.cfg
home = /root/b
include-system-site-packages = false
version = 3.12.3
executable = /usr/bin/python3.12
command = /root/b/python3.12 -m venv --without-pip /root/b/python3.12-venv-link```

[zanieb]

I need a little more exposition alongside these examples, I don't follow what you're demonstrating there.

I would be surprised if this wasn't a sysconfig issue, but of course it'd be great to fix here if we can.

[konstin]

As shown, pbs creates broken venv when the base interpreter is a symlink. This is a bug, since it should be able to do this, as system interpreter and pyenv succeed at this.

The crash happens because home in pyvenv.cfg is wrong, which is derived from sys._base_executable.

https://github.com/python/cpython/blob/d48cc82ed25e26b02eb97c6263d95dcaa1e9111b/Lib/venv/__init__.py#L162-L175

https://github.com/python/cpython/blob/d48cc82ed25e26b02eb97c6263d95dcaa1e9111b/Lib/venv/__init__.py#L222

This bug is the most likely candidate to crash the test suite in astral-sh/uv#8481 (comment).

We need to figure out how sys._base_executable is set and at some level, patch it to the right value. What exactly is going on inside the sys module and why it only affects pbs is unclear at this point.

[charliermarsh]

Interestingly, after symlinking, some of the sysconfig.get_config_vars() still point to the symlink location:

>>> sysconfig.get_config_vars()["base"]
'/Users/crmarsh/.local/share/uv/python/cpython-3.12.6-macos-aarch64-none'
>>> sysconfig.get_config_vars()["projectbase"]
'/Users/crmarsh/.local/share/uv/python/cpython-3.12.6-macos-aarch64-none/bin'

[ofek]

That would definitely affect compilations of extension modules.

[charliermarsh]

Ohh interesting, it's because Python does:

_PROJECT_BASE = os.path.dirname(_safe_realpath(sys.executable))

[indygreg]

I'm pretty sure I had to do something in PyOxidizer to work around this. Essentially mucking with the Python interpreter config to get it to resolve paths correctly because the default logic was insufficient.

We should consider involving a CPython maintainer on this one as the behavior is a smell. ISTR Windows having special logic for path resolution because portable installs are a thing on Windows. But since most UNIX-like installs are to a fixed path, CPython doesn't support dynamic paths as well as we'd like.

If CPython doesn't recognize the issue, we may have to patch our CPython build to handle dynamic install paths/symlibks correctly. This could be done with a custom site customize file. But since this included file can be disabled, we may need to patch C code.

[ncoghlan]

I haven't looked closely at this specific case, but we've definitely removed realpath calls from CPython before because resolving the symlink turned out to be the wrong thing to do (or added them, when things needed to be located relative to the underlying link target).

(Tangentially related, python/cpython#124825 is a proposal I filed to revisit adding lexical relative path resolution to pathlib, since I needed it to get venvstacks to work reliably on macOS. That's another project that uses python-build-standalone for its base runtime layers, so this could have been the underlying issue)

[charliermarsh]

I'm looking into this a bit.

[charliermarsh]

Ok, so for reference, the way prefix gets set in these builds is here.

If the executable is at /Users/crmarsh/python/install/bin/python, then executable_dir is /Users/crmarsh/python/install/bin and STDLIB_LANDMARKS is ['lib/python3.13/os.py', 'lib/python3.13/os.pyc']. So we look up the path and find /Users/crmarsh/python/install/lib/python3.13/os.py, and prefix gets set to /Users/crmarsh/python/install.

[charliermarsh]

Ok... If you symlink an interpreter (but don't create a virtualenv), then executable_dir is still the realpath of the containing directory.

So, above, it would still be /Users/crmarsh/python/install/bin even if you symlinked /Users/crmarsh/python/install/bin/python to /Users/crmarsh/foo. That's why symlinked interpreters still work.

Specifically, when you're not in a virtualenv, the executable_dir gets set here.

However... if you create a virtualenv from a symlinked interpreter, then home gets set to the directory containing the symlink. Above, that would be home = /Users/crmarsh. In getpath.py, we'd then set executable_dir based on home here instead of reading from the realpath or similar.

So we'd then fail to find lib/python3.13/os.py when searching up the path, hence the failure.

[charliermarsh]

I don't know if we should be setting home differently (in uv), or if we should be changing getpath.py to handle home differently.

[charliermarsh]

One thing to note in the "symlink to ./bin/python" case: base_executable does not resolve the symlink, but executable_dir (and the resulting prefix) does.

---

Sorry, my own scratch notes...

When I symlink ./install/bin/python to ./foo:

• base_executable: /Users/crmarsh/workspace/python-build-standalone/dist/foo
• real_executable: /Users/crmarsh/workspace/python-build-standalone/dist/python/install/bin/python3.13
• executable_dir: /Users/crmarsh/workspace/python-build-standalone/dist/python/install/bin
• prefix: /Users/crmarsh/workspace/python-build-standalone/dist/python/install

When I symlink ./install to ./bar:

• base_executable: /Users/crmarsh/workspace/python-build-standalone/dist/bar/bin/python
• real_executable: /Users/crmarsh/workspace/python-build-standalone/dist/bar/bin/python3.13
• executable_dir: /Users/crmarsh/workspace/python-build-standalone/dist/bar/bin
• prefix: /Users/crmarsh/workspace/python-build-standalone/dist/bar

Ahhh, ok. The reason for this, I think, is that the realpath used in getpath.py only resolves a symlinked file, and not any path segments. That explains why the real_executable looks like this.

---

When we launch from a virtualenv, we take whatever is in home, and:

• home becomes executable_dir (so we search for prefix from there).
• base_executable is typically computed via realpath(executable). But if executable is not a symlink, then it's joinpath(executable_dir, basename(executable)), so home does have an impact in that latter case.

---

The combination of these behaviors means that whatever we put in home needs to be a valid search path for STDLIB_LANDMARKS.

We can solve this in two ways:

1. virtualenv creators (like uv) could recognize that the base_executable is going to lead to an invalid home, then try real_executable instead (or, like, iteratively resolve the symlink and try to find the first valid home).
2. getpath.py could be changed to ensure that we do resolve base_executable in the first case described above (symlink to a Python executable directly).

(1) seems easier to me, though (2) might be the "better" fix? I'm just not confident on how we would implement it or how it would affect other, non-PBS Pythons.

[gvwilson]

@charliermarsh this one's biting us at work as well

[zanieb]

@gvwilson Can you explain a bit more about your setup? What are you using symlinked interpreters for? How are you creating your virtual environments?

[hatched]

Hi @zanieb, I'll be replying for @gvwilson 😁

This issue appears to be presenting itself on Linux (Fedora) in an AppImage.
It works as expected on macos and if distributed via an rpm where uv is installed globally.

The venv is being created with: uv venv path/to/env
Then we run a script with: uv run --directory path/to/env --with /path/to/wheel --isolated start
We attempted to provide the venv command with the --python or --python-preference flags but they didn't appear to be respected as the 3.12 version was always used.

Here is the configuration dump.... the /tmp/.mount... path is correct, but the ones which have /home/username/.cache are not.

Python path configuration:
  PYTHONHOME = '/tmp/.mount_P2c9luo/usr/'
  PYTHONPATH = '/tmp/.mount_P2c9luo/usr/share/pyshared/:'
  program name = '/home/username/.cache/uv/archive-v0/fNTQaShMKfyhiIbhkOWUq/bin/python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/tmp/.mount_P2c9luo/usr/lib64/python3.12'
  sys._base_executable = '/home/username/.cache/uv/archive-v0/fNTQaShMKfyhiIbhkOWUq/bin/python'
  sys.base_prefix = '/tmp/.mount_P2c9luo/usr/'
  sys.base_exec_prefix = '/tmp/.mount_P2c9luo/usr/'
  sys.platlibdir = 'lib64'
  sys.executable = '/home/username/.cache/uv/archive-v0/fNTQaShMKfyhiIbhkOWUq/bin/python'
  sys.prefix = '/tmp/.mount_P2c9luo/usr/'
  sys.exec_prefix = '/tmp/.mount_P2c9luo/usr/'
  sys.path = [
    '/tmp/.mount_P2c9luo/usr/share/pyshared',
    '/home/username/.local/share/com.P-s.app/environments/application_backend_internal',
    '/tmp/.mount_P2c9luo/usr/lib64/python312.zip',
    '/tmp/.mount_P2c9luo/usr/lib64/python3.12',
    '/tmp/.mount_P2c9luo/usr/lib64/python3.12/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

[zanieb]

Are you managing your python-build-standalone distributions with uv or manually? If the former, I'd appreciate a complete bug report over in uv.

I'm having a bit of a hard time following what's going on. A complete reproduction with the commands you're using (and their output) would be really helpful.

[hatched]

Those are being managed with uv.

I'll try and create a minimal repro and create a new issue in uv, thanks!

[henryiii]

Here's a reproducer for uvx cibuildwheel:

$ docker run --rm -it ubuntu:24.10
# apt update && apt install -y curl git
# curl -LsSf https://astral.sh/uv/install.sh | sh
# source $HOME/.local/bin/env
# uv python install 3.12
# git clone https://github.com/scikit-hep/boost-histogram
# cd boost-histogram
# uvx cibuildwheel --only cp312-pyodide_wasm32

     _ _       _ _   _       _           _
 ___|_| |_ _ _|_| |_| |_ _ _| |_ ___ ___| |
|  _| | . | | | | | . | | | |   | -_| -_| |
|___|_|___|___|_|_|___|_____|_|_|___|___|_|

cibuildwheel version 2.23.1

Build options:
  platform: pyodide
  allow_empty: False
  architectures: wasm32
  build_selector:
    build_config: cp312-pyodide_wasm32
    skip_config:
    requires_python: >=3.8
    enable: ['cpython-freethreading', 'cpython-prerelease', 'pypy']
  output_dir: /boost-histogram/wheelhouse
  package_dir: /boost-histogram
  test_selector:
    skip_config: cp*-musllinux_* cp313t-*win* pp311-*
  before_all:
  before_build:
  before_test:
  build_frontend:
    *: build[uv]
    cp312-pyodide_wasm32:
      name: build
      args: ['--exports', 'whole_archive']
  build_verbosity: 0
  config_settings:
  container_engine: docker
  dependency_constraints: pinned
  environment:
    PIP_ONLY_BINARY="numpy"
    PIP_PREFER_BINARY="1"
  manylinux_images: None
  musllinux_images: None
  repair_command:
  test_command:
    *: pytest -n auto --benchmark-disable {project}/tests
    cp312-pyodide_wasm32: pytest --benchmark-disable {project}/tests
  test_extras:
  test_groups:
    test
  test_requires:
    cloudpickle
    hypothesis>=6.0
    pytest-benchmark
    pytest>=6.0
    pytest-xdist

Cache folder: /root/.cache/cibuildwheel

Here we go!


Building cp312-pyodide_wasm32 wheel
CPython 3.12 Pyodide

Setting up build environment...

+ Download https://github.com/pypa/get-virtualenv/blob/20.29.3/public/virtualenv.pyz?raw=true to /root/.cache/cibuildwheel/virtualenv-20.29.3.pyz
+ /root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/bin/python -sS /root/.cache/cibuildwheel/virtualenv-20.29.3.pyz --activators= --no-periodic-update --pip=embed --no-setuptools --no-wheel --python /root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/bin/python /tmp/cibw-run-zmv0cabc/cp312-pyodide_wasm32/build/venv
created virtual environment CPython3.12.9.final.0-64 in 259ms
  creator CPython3Posix(dest=/tmp/cibw-run-zmv0cabc/cp312-pyodide_wasm32/build/venv, clear=False, no_vcs_ignore=False, global=False)
  seeder FromAppData(download=False, pip=embed, via=copy, app_data_dir=/root/.local/share/virtualenv)
    added seed packages: pip==25.0.1
+ python -m pip install --upgrade pip -c /root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/lib/python3.12/site-packages/cibuildwheel/resources/constraints-pyodide312.txt
Could not find platform independent libraries <prefix>
Could not find platform dependent libraries <exec_prefix>
Python path configuration:
  PYTHONHOME = (not set)
  PYTHONPATH = (not set)
  program name = '/tmp/cibw-run-zmv0cabc/cp312-pyodide_wasm32/build/venv/bin/python'
  isolated = 0
  environment = 1
  user site = 1
  safe_path = 0
  import site = 1
  is in build tree = 0
  stdlib dir = '/install/lib/python3.12'
  sys._base_executable = '/root/.local/share/uv/python/cpython-3.12.9-linux-x86_64-gnu/bin/python3.12'
  sys.base_prefix = '/install'
  sys.base_exec_prefix = '/install'
  sys.platlibdir = 'lib'
  sys.executable = '/tmp/cibw-run-zmv0cabc/cp312-pyodide_wasm32/build/venv/bin/python'
  sys.prefix = '/install'
  sys.exec_prefix = '/install'
  sys.path = [
    '/install/lib/python312.zip',
    '/install/lib/python3.12',
    '/install/lib/python3.12/lib-dynload',
  ]
Fatal Python error: init_fs_encoding: failed to get the Python codec of the filesystem encoding
Python runtime state: core initialized
ModuleNotFoundError: No module named 'encodings'

Current thread 0x00007f8c21d3c740 (most recent call first):
  <no Python frame>
Traceback (most recent call last):
  File "/root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/bin/cibuildwheel", line 12, in <module>
    sys.exit(main())
             ^^^^^^
  File "/root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/lib/python3.12/site-packages/cibuildwheel/__main__.py", line 49, in main
    main_inner(global_options)
  File "/root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/lib/python3.12/site-packages/cibuildwheel/__main__.py", line 184, in main_inner
    build_in_directory(args)
  File "/root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/lib/python3.12/site-packages/cibuildwheel/__main__.py", line 351, in build_in_directory
    platform_module.build(options, tmp_path)
  File "/root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/lib/python3.12/site-packages/cibuildwheel/pyodide.py", line 244, in build
    env = setup_python(
          ^^^^^^^^^^^^^
  File "/root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/lib/python3.12/site-packages/cibuildwheel/pyodide.py", line 131, in setup_python
    call(
  File "/root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/lib/python3.12/site-packages/cibuildwheel/util.py", line 154, in call
    result = subprocess.run(
             ^^^^^^^^^^^^^^^
  File "/root/.local/share/uv/python/cpython-3.12.9-linux-x86_64-gnu/lib/python3.12/subprocess.py", line 573, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '['/tmp/cibw-run-zmv0cabc/cp312-pyodide_wasm32/build/venv/bin/python', '-m', 'pip', 'install', '--upgrade', 'pip', '-c', '/root/.cache/uv/archive-v0/m8zF1t0bWCOV1lDOploP5/lib/python3.12/site-packages/cibuildwheel/resources/constraints-pyodide312.txt']' returned non-zero exit status 1.

pypa/cibuildwheel#2327

(also posted this on the linked uv issue)