Installing ruff via pip on NixOs does not work

[heitorPB]

Installing ruff via pip on NixOs systems does not work:

$ cd /tmp
$ python3 -m venv venv
$ source venv/bin/activate
(venv)
$ pip install ruff
Collecting ruff
  Using cached ruff-0.0.212-py3-none-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.3 MB)
Installing collected packages: ruff
Successfully installed ruff-0.0.212

[notice] A new release of pip available: 22.2.2 -> 22.3.1
[notice] To update, run: pip install --upgrade pip
(venv)
$ ruff
-bash: /tmp/venv/bin/ruff: No such file or directory
(venv)

$ file /tmp/venv/bin/ruff
/tmp/venv/bin/ruff: ELF 64-bit LSB pie executable, x86-64, version 1 (SYSV), dynamically linked, interpreter /lib64/ld-linux-x86-64.so.2, BuildID[sha1]=1ec33df614fc5020de42d625366e4684ae45a0b6, for GNU/Linux 2.6.32, stripped
(venv)

The dynamic loader ("ELF interpreter") is not in the path hardcoded in the binary, although ldd shows the right path:

$ ldd /tmp/venv/bin/ruff
        linux-vdso.so.1 (0x00007f8623e51000)
        libgcc_s.so.1 => /nix/store/hsk71z8admvgykn7vzjy11dfnar9f4r1-glibc-2.35-163/lib/libgcc_s.so.1 (0x00007f86233eb000)
        librt.so.1 => /nix/store/hsk71z8admvgykn7vzjy11dfnar9f4r1-glibc-2.35-163/lib/librt.so.1 (0x00007f86233e6000)
        libpthread.so.0 => /nix/store/hsk71z8admvgykn7vzjy11dfnar9f4r1-glibc-2.35-163/lib/libpthread.so.0 (0x00007f86233e1000)
        libm.so.6 => /nix/store/hsk71z8admvgykn7vzjy11dfnar9f4r1-glibc-2.35-163/lib/libm.so.6 (0x00007f8623301000)
        libdl.so.2 => /nix/store/hsk71z8admvgykn7vzjy11dfnar9f4r1-glibc-2.35-163/lib/libdl.so.2 (0x00007f86232fc000)
        libc.so.6 => /nix/store/hsk71z8admvgykn7vzjy11dfnar9f4r1-glibc-2.35-163/lib/libc.so.6 (0x00007f86230f1000)
        /lib64/ld-linux-x86-64.so.2 => /nix/store/hsk71z8admvgykn7vzjy11dfnar9f4r1-glibc-2.35-163/lib64/ld-linux-x86-64.so.2 (0x00007f8623e53000)
(venv)

I was able to get it working using patchelf, but it is not a good workflow when using a Python package manager (e.g. hatch, pdm, poetry, etc) to handle all the dev dependencies of the project.

[messense]

Can you run it with LD_DEBUG=libs,files env var to see what's going on?

LD_DEBUG=libs,files /tmp/venv/bin/ruff

[messense]

https://nixos.wiki/wiki/Packaging/Binaries

Downloading and attempting to run a binary on NixOS will almost never work. This is due to hard-coded paths in the executable.

https://nixos.wiki/wiki/Packaging/Binaries#The_Dynamic_Loader

I think it's by design in NixOS? You should probably look into packaging ruff in nixpkgs.

Edit: Looks like it's already packaged: https://search.nixos.org/packages?channel=unstable&show=ruff&from=0&size=50&sort=relevance&type=packages&query=ruff

[charliermarsh]

Given the above, going to close for now.

[heitorPB]

@messense,

Can you run it with LD_DEBUG=libs,files env var to see what's going on?

The binary does not run, due to the different path of the Dynamic Loader.

@charliermarsh,

ruff does work if I install it on my machine via the OS package manager, but that is not a solution. The issue is that many Python projects use ruff as a linter defined in pyproject.toml/requirements.txt, and that gets installed via pip/pdm/hatch/poetry/etc in a local environment (a.k.a. virtual env). And then running the linter/test/format scripts simply does not work.

As an example:

$ git clone git@github.com:tiangolo/fastapi.git
$ cd fastapi/
$ python -m venv env
$ source ./env/bin/activate
$ pip install -e ."[dev,doc,test]"
$ bash ./scripts/format.sh
+ ruff fastapi tests docs_src scripts --fix
scripts/format.sh: line 4: /home/h/projects/fastapi/venv/bin/ruff: No such file or directory
+ black fastapi tests docs_src scripts
All done! ✨ 🍰 ✨
804 files left unchanged.
+ isort fastapi tests docs_src scripts

As you can see, ruff is just erroring out :(

In some cases, we can install ruff via the OS package manager, but some Python projects "embed" it, or use/require/pin a specific version.

[messense]

IMHO since you choose NixOS you should embrace their rules, it is known that pip install native binaries and extension modules may not work on NixOS, there are some existing issues in nixpkgs:

• python's pip downloads prebuilt binaries that don't work NixOS/nixpkgs#142383
• python: can not use imperatively installed packages as libraries NixOS/nixpkgs#10597

Ideally it should be fixed in NixOS, for now as a workaround I think you can install ruff from sdist instead

pip install --no-binary :all: ruff

That said, ruff may be able to provide fully static musl libc binaries but it might run much slower than the current glibc ones.

[heitorPB]

for now as a workaround I think you can install ruff from sdist instead

pip install --no-binary :all: ruff

Unfortunately, that did not work: I don't have rust compilers. I could install cargo/rustc/etc, but that does not provide a "work out of the box" experience. pip install flake8 just works and I was expecting that for ruff.

That said, ruff may be able to provide fully static musl libc binaries but it might run much slower than the current glibc ones.

Is this on the roadmap? Would it be possible to have an "alternate" installation for ruff that uses musl statically linked instead? For example: pip install ruff[musl]?

[messense]

pip install flake8 just works and I was expecting that for ruff.

flake8 is a pure Python wheel so it's very different than ruff which is a compiled Rust binary.

Is this on the roadmap?

That's a question for @charliermarsh, IMO the default should be glibc version unless we can prove that musl libc version is on par with its performance.

Would it be possible to have an "alternate" installation for ruff that uses musl statically linked instead? For example: pip install ruff[musl]?

pip install ruff[musl] won't work because [musl] is for specifying extra dependencies. A new ruff-musl/ruff-static pip package could work.

Maybe ruff eventual provides musl libc binaries that works for you but there are more similar pip binary projects that won't, so IMHO the best solution is to ask/sponsor nixpkgs to fix NixOS/nixpkgs#142383.

[akx]

I guess a musl-linked ruff could be shipped in a musllinux tagged wheel... musllinux seems to be supported by cibuildwheel, for what it's worth.