Python dev headers missing in venv

[arogozhnikov]

I think python dev headers are now missing when uv installs venv; repro (start from ubuntu 22.04):

confirmed on uv==0.5.0 and 0.5.4, not tested on other versions

export $VIRTUAL_ENV=/tmp/.venv
curl -LsSf https://astral.sh/uv/0.5.0/install.sh | sh
. $HOME/.local/bin/env
uv venv --python 3.10 $VIRTUAL_ENV 
. $VIRTUAL_ENV/bin/activate
uv pip install ihm

output:

[stderr]
src/ihm_format_wrap_1.7.c:203:11: fatal error: Python.h: No such file or directory
  203 | # include <Python.h>
      |           ^~~~~~~~~~
compilation terminated.
error: command '/usr/bin/x86_64-linux-gnu-gcc' failed with exit code 1
  Caused by: This error likely indicates that you need to install a library that provides "Python.h" for ihm@1.7

Annoyingly, it used to work - I was able to install this package, but a couple of weeks ago binaries shipped with uv were swapped for some other builds, this affected even previous versions of uv.

[samypr100]

Similar to #8966 (comment)

TL;DR Try sysconfigpatcher to adjust the uv python downloads to have the proper pkgconfig and sysconfig metadata for your system.

[zanieb]

but a couple of weeks ago binaries shipped with uv were swapped for some other builds, this affected even previous versions of uv.

This doesn't quite make sense, we didn't change previous versions of uv and the Python distributions we use are hard-coded for each release and cannot change.

[zanieb]

If you're using a system interpreter, you should install the headers, e.g., the following works:

docker run --platform linux/x86_64 -it ghcr.io/astral-sh/uv:bookworm /bin/bash -c "apt update && apt install python3-dev && uv venv && uv pip install ihm"

I can't reproduce this failure using a managed interpreter:

docker run --platform linux/x86_64 -it ghcr.io/astral-sh/uv:bookworm /bin/bash -c "apt update && apt install clang && uv venv -p 3.10 && uv pip install ihm"

[arogozhnikov]

this works:

docker run --platform linux/x86_64 -it --rm ghcr.io/astral-sh/uv:bookworm /bin/bash -c "apt update && apt install clang --yes && uv venv -p 3.10 && uv pip install ihm"

this doesn't

docker run --platform linux/x86_64 -it --rm ghcr.io/astral-sh/uv:bookworm /bin/bash -c "apt update && apt install clang --yes && uv venv -p 3.11 && uv pip install ihm"

[wjakob]

I am also having this issue, and using Docker is not a feasible workaround. Could UV not be modified so that installs the Python headers when creating a venv?

[samypr100]

👋 @wjakob, the docker examples here are to provide an MRE, you can apply the same steps outside of docker.

[wjakob]

This is on a machine where I don't have root access, hence installing the headers via apt is not an option (also, that would be for the OS Python version, which is often quite outdated). With uv not installing them as part of the virtual environment, it is impossible to build C extensions within the environment.

[zanieb]

@arogozhnikov your failing example uses the system interpreter (3.11 is installed in the image already)

root@c0d1cea75e52:/# uv python list
cpython-3.13.0+freethreaded-linux-x86_64-gnu    <download available>
cpython-3.12.7-linux-x86_64-gnu                 <download available>
cpython-3.11.10-linux-x86_64-gnu                <download available>
cpython-3.11.2-linux-x86_64-gnu                 /usr/bin/python3.11
cpython-3.11.2-linux-x86_64-gnu                 /usr/bin/python3 -> python3.11
cpython-3.11.2-linux-x86_64-gnu                 /bin/python3.11
cpython-3.11.2-linux-x86_64-gnu                 /bin/python3 -> python3.11
cpython-3.10.15-linux-x86_64-gnu                <download available>
cpython-3.9.20-linux-x86_64-gnu                 <download available>
cpython-3.8.20-linux-x86_64-gnu                 <download available>
cpython-3.7.9-linux-x86_64-gnu                  <download available>
pypy-3.10.14-linux-x86_64-gnu                   <download available>
pypy-3.9.19-linux-x86_64-gnu                    <download available>
pypy-3.8.16-linux-x86_64-gnu                    <download available>
pypy-3.7.13-linux-x86_64-gnu                    <download available>

That's why you need to install development headers via the system package manager.

If you force uv to use a managed interpreter with --python-preference only-manage, the example works

docker run --platform linux/x86_64 -it --rm ghcr.io/astral-sh/uv:bookworm /bin/bash -c "apt update && apt install clang --yes && uv venv -p 3.11 --python-preference only-managed && uv pip install ihm"

[zanieb]

@wjakob please share a reproducible example

[samypr100]

docker run --platform linux/x86_64 -it --rm ghcr.io/astral-sh/uv:bookworm /bin/bash -c "apt update && apt install clang --yes && uv venv -p 3.11 --python-preference only-managed && uv pip install ihm"

Thanks @zanieb, I've been incorrectly under assumption OP wanted to use gcc and not clang, hence my suggestions to use sysconfigpatcher.

docker run --platform linux/x86_64 -it --rm ghcr.io/astral-sh/uv:bookworm /bin/bash -c "uv venv -p 3.11.10 && uv tool install git+https://github.com/bluss/sysconfigpatcher && uv tool run sysconfigpatcher /root/.local/share/uv/python/cpython-3.11.10-linux-x86_64-gnu && uv pip install ihm"

Do you think it's worth expanding in docs when is it reasonable to attempt to use sysconfigpatcher by adding a note to python-versions.md or maybe a troubleshooting faq.

[zanieb]

Do you think it's worth expanding in docs when is it reasonable to attempt to use sysconfigpatcher by adding a note to python-versions.md or maybe a troubleshooting faq.

It's sort of a priority for me to understand the problems people are encountering and fix it here rather than recommend sysconfigpatcher — it's nice that it exists and I'm sure we'll learn from it but I'd prefer a first-party solution.

[Tristan971]

please share a reproducible example

Here's one:

FROM fedora:41

RUN dnf update -y && \
    dnf group install -y 'development-tools' && \
    dnf install -y \
      gcc \
      gcc-c++ \
      gtk3 \
      gtk3-devel \
      libxcrypt-compat \
      python3 \
      python3-devel \
      which

RUN curl -LsSf https://astral.sh/uv/install.sh | sh

Run the resulting image then:

# just ensuring we don't use the system-managed one which is 3.13
[root@0a8793bee0ed ~]# uv init --python ">=3.10,<3.11" repro
[root@0a8793bee0ed ~]# cd repro
[root@0a8793bee0ed repro]# uv add 'wxpython>=4.0.0'
# ... takes a minute then fails with...
...
      The configuration failed
      (complete log in /root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src/build/waf/3.10/gtk3/config.log)

Then we go and look at the detailed logs

[root@0a8793bee0ed repro]# cat /root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src/build/waf/3.10/gtk3/config.log
# ... bunch of noise until ...
['/usr/bin/g++', '-g', '-fwrapv', '-O3', '-fPIC', '-fno-strict-aliasing', '-I/install/include/python3.10', '-DPYTHONDIR="/root/.cache/uv/builds-v0/.tmpn5kGVI/lib/python3.10/site-packages"', '-DPYTHONARCHDIR="/root/.cache/uv/builds-v0/.tmpn5kGVI/lib/python3.10/site-packages"', '-DNDEBUG', '../test.cpp', '-c', '-o/root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src/build/waf/3.10/gtk3/.conf_check_ef50035bd89e2895b752adc35a4fe169/testbuild/test.cpp.1.o']
err: ../test.cpp:2:10: fatal error: Python.h: No such file or directory
    2 | #include <Python.h>
      |          ^~~~~~~~~~
compilation terminated.

from /root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src: Test does not build: Traceback (most recent call last):
  File "/root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src/bin/.waf3-2.1.2-5b48554f897113218781486acfe4d78e/waflib/Configure.py", line 337, in run_build
    bld.compile()
  File "/root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src/bin/.waf3-2.1.2-5b48554f897113218781486acfe4d78e/waflib/Build.py", line 176, in compile
    raise Errors.BuildError(self.producer.error)
waflib.Errors.BuildError: Build failed
 -> task in 'testprog' failed with exit status 1 (run with -v to display more information)

Broken python installation? Get python-config now!
from /root/.cache/uv/sdists-v6/pypi/wxpython/4.2.2/0ju17OLUIdGAvvjIAN9Zx/src: The configuration failed

I can confirm that using sysconfigpatcher fixes this issue (the wxpython build takes a bit of an eternity, but it eventually does succeed)

[root@0a8793bee0ed repro]# uv tool install 'git+https://github.com/bluss/sysconfigpatcher'
 Updated https://github.com/bluss/sysconfigpatcher (c1ebf8a)
Resolved 1 package in 0.49ms
   Built sysconfigpatcher @ git+https://github.com/bluss/sysconfigpatcher@c1ebf8ab9274dcde255484d93ce0f1fd1f76a248
Prepared 1 package in 322ms
Installed 1 package in 0.99ms
 + sysconfigpatcher==0.3.1 (from git+https://github.com/bluss/sysconfigpatcher@c1ebf8ab9274dcde255484d93ce0f1fd1f76a248)
Installed 1 executable: sysconfigpatcher
[root@0a8793bee0ed repro]# realpath .venv/bin/python
/root/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/bin/python3.10
[root@0a8793bee0ed repro]# sysconfigpatcher /root/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/bin/python3.10
INFO: Patched /root/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/python3.10/_sysconfigdata__linux_x86_64-linux-gnu.py
INFO: Patched /root/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/pkgconfig/python-3.10-embed.pc
INFO: Patched /root/.local/share/uv/python/cpython-3.10.16-linux-x86_64-gnu/lib/pkgconfig/python-3.10.pc
[root@0a8793bee0ed repro]# uv add 'wxpython>=4.0.0'
Resolved 4 packages in 36ms
   Built wxpython==4.2.2
Prepared 1 package in 2m 58s
Installed 3 packages in 29ms
 + numpy==2.2.0
 + six==1.17.0
 + wxpython==4.2.2

Also worth noting that UV's manager isn't the only one that leads to these kinds of issues; back then I experienced a similar (but not the same exact) issue with pyenv+poetry, linked here https://gitlab.com/ita1024/waf/-/issues/2420