Python package build siliently fails if Swig is not installed

[ignaloidas]

Tried to pip install libnvme, as written in the docs, but after a seemingly successful install the package was not importable. After a fair bit of debugging, it seems that because the default in meson for python is auto (that is, build if all required tools are available), and the build process for the python package from pypi doesn't change that, meson just doesn't build the python package if python-specific dependencies aren't available.

[martin-belanger]

Were you trying to pip install a package that you built yourself or a package that was built by an upstream distro?

The build environment is configured for developers by default. It's not just the "python" configuration parameter, but it applies to other config parameters such as "prefix", "sysconfdir", "json-c", "keyutils", etc. For developers, we want the build environment to be easily configurable (i.e. not have to manually specify a lot of config parameters every time we build). That's why the default for many parameters is "auto".

If you want to build libnvme with specific options enabled (python, json-c, etc), then you must specify each option as "enabled" or "disabled" (e.g. -Dpython=enabled). Doing so will allow meson to stop if a dependency is missing. Also, if you want your modules, libraries, etc, to be installed in a specific location (e.g. /usr/ vs /usr/local/), then you must specify the "prefix" and "sysconfdir" (e.g. --prefix /usr --sysconfdir /etc).

[ignaloidas]

I was trying to pip install the package as published in PyPI: https://pypi.org/project/libnvme/ - essentially using the standard way to install python packages. As far as I can see what's in PyPI is essentially a source tarball that gets built as defined in the pyproject.toml file. And seems like it essentially does a default meson build without any options set, and then only keeps the python parts (if any).

There is no way (nor does it make sense) to manually set build options when installing via pip form PyPI. They should probably be set in pyproject.toml, and I looked a bit into that, but I'm not experienced with Meson and how it's Python build pipeline works and couldn't find a way to fix it.

[martin-belanger]

I was trying to pip install the package as published in PyPI: https://pypi.org/project/libnvme/ - essentially using the standard way to install python packages.

Thanks for the clarification. In that case, you are probably correct in assuming that the PyPI package is built with the wrong config options. It should be built with explicit options and not rely on the implicit ones. In other words, that means using -Dpython=enabled as in this example:

meson setup .build -Dpython=enabled 
meson compile -C .build

The options --prefix, --sysconfdir, and --buildtype should probably be configured as well, but I do not know what the intention of the maintainer (wagi) is regarding this package.

meson setup .build -Dpython=enabled --prefix /usr --sysconfdir /etc --buildtype release
meson compile -C .build

Note that using the above options for --prefix, --sysconfdir, and --buildtype would basically cause the standard libnvme libraries that may already be present in /usr/ to be overwritten by the PyPI package. Again, not sure what the maintainer wants to do.

Also note that some distros offer the Python wrapper for libnvme as a distro-specific package (e.g. apt-get install python3-libnvme). This may not be available on all distros though. So that may be another option for you instead of using PyPI and this would ensure that you run compatible versions of the libnvme modules and libraries.

[glimchb]

the python wheel build before uploading to pypi was happening in the python container here:

libnvme/.github/workflows/release-python.yml

Line 19 in 79ec876

image: ghcr.io/linux-nvme/debian.python:latest

so all the dependencies should be already met, that's why we don;t need to change auto option...

BUT

because we stopped publishing wheels here c95ff54

it means the code is compiled upon installation... which means your environment must meet the requirements...
we can't install python for you, that's a separate package, we can just specify dependencies...

maybe documentation must be clearer... you ask ask @igaw for more info, since he made recent changes to how pypi is published...

[ignaloidas]

If you do have swig available, it does compile a wrapper .so that links to the system libnvme.so, and this would be fine for my use case. The problem is when swig isn't available - the installation command still succeeds, but the wrapper isn't compiled, so de facto the command fails. Other projects that offer source distributions usually fail installing if there's dependencies missing, and that would be the preferred way to handle things (on some systems, wheels might not be available at all - source builds would need to happen anyways).

[igaw]

I can't remember the details for c95ff54 :

"Pre building the binaries with wheels is difficult, as we would also need to ship the libnvme.so with the Python binding." IIRC this caused troubles as well. And overwriting distro packages is not really an option IMO.

Documentation is highly appreciated. I've started a wiki for nvme-cli, we could add it there for the time being. Feel free to add something there: https://github.com/linux-nvme/nvme-cli/wiki

    build_python_bindings = py3_dep.found() and swig.found() and header_found

Alright, this is in since day one and does exactly what you see. No swig no bindings.
I don't know where to add -Dswig=enabled.

(The longer I work on this project the more I get to the conclusion, upstream source project should not provide any binary packages, when most distributions package the project already. It adds a lot of maintenance burdens for a couple of users)

[glimchb]

if you installed python3 and swig before runnign pip install... then everything will work

so I guess we can do one of those

• fail pip install if python3 or swig are missing (pre-requisites)
• enable both -Dswig=enabled and -Dpython=enabled in https://github.com/linux-nvme/libnvme/blob/master/pyproject.toml - need to research how to do it...

maybe we can remove this if condition and always build_python_bindings true ?

libnvme/libnvme/meson.build

Line 20 in 7b9f86f

if build_python_bindings

i don't remember why we wanted to have 2 options...

[igaw]

i don't remember why we wanted to have 2 options...

Me neither. I suppose if the user asks for Python the swig dependency could be added as hard dependency. At least from what I understand it would fix the problem.

[martin-belanger]

i don't remember why we wanted to have 2 options...

Me neither. I suppose if the user asks for Python the swig dependency could be added as hard dependency. At least from what I understand it would fix the problem.

If you look in meson_options.txt, there is no swig option. As @igaw said, when the user asks for Python (-Dpython=enabled), then Swig becomes a mandatory dependency. When -Dpython=auto, Python is optionally built only if Swig is already installed.

[eli-schwartz]

There is no way (nor does it make sense) to manually set build options when installing via pip form PyPI. They should probably be set in pyproject.toml, and I looked a bit into that, but I'm not experienced with Meson and how it's Python build pipeline works and couldn't find a way to fix it.

That is certainly a falsehood: https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-C

It is possible to manually set build options when installing via pip from PyPI. It makes sense to be able to manually set build options when engaging in the act of installing software, so if pip was not able to do so then that would be a bug in pip, but as it happens there is not a bug in pip.