API mode / C sources does not build with default setuptools / build

[dgdguk]

When compiling a CFFI module that compiles source code (these instructions), if that source code contains headers, and those headers are used in the C code, it fails with the default invocation of python -m build. This is because CFFI is not able to tell setuptools about the header files, and by default python -m build generates an isolated enviroment containing only the files that setuptools knows about, which will not include the required headers.

A workaround is to pass something like package_data={"": ["*.h"]}, to setuptools.setup, which will grab all .h files, but it may be worth either a) documenting this behaviour or b) augmenting set_source with a headers parameter so it can mark those headers as being required.

[arigo]

I'm the author of CFFI but I'm not following the current evolution of setuptools. I don't know in which situations this is a problem and the proper way to fix it. I would be happy to accept a pull request if you or someone else feels like implementing it.

[dgdguk]

Well, I think first thing to do would be to add a comment to the documentation so that people can work around it. If you want to see the issue in action, then here's some steps for that:

1. Pull https://github.com/dgdguk/convolutionalfixedsum
2. In setup.py, comment out line 20: package_data={"": ["*.h"]},
3. Run pip install build, followed by python -m build; this will crash, with the C compiler complaining it cannot find the headers.

This is by no means a critical issue at the moment; it's specific to the build tooling, so building a package with pip should still work. However, the PyPA is already recommending using build over pip, and I suspect that it's going to start becoming the default in a lot of places.

As for patches - I've not currently got the time for it, but I might be able to contribute something over the summer, if no-one gets to it before me.

[arigo]

That code calls set_source(...include_dirs=[str(analytical_base)]), though. This argument is passed straight from CFFI to setuptools, like most keyword arguments to set_source(). Does it mean that setuptools still accepts, but now ignores, the include_dirs argument in this case? If setuptools still accepts the include_dirs argument but no longer passes it to the C compiler as an -I option, that's rather strange (but not unexpected, because changing things in incompatible ways every two years seems to be common for setuptools).

[dgdguk]

I believe the problem is the isolated build environment that build creates. Basically, build copies all of the files that setuptools has been told about into a separate directory, with a seperate venv and everything, to make sure that the build doesn't rely on random files that setuptools doesn't know about and that someone with the source code can reproduce the build.

Unfortunately, while setuptools knows about all of the .c files, it doesn't know about the .h files. So it dutifully copies over all the .c files without the .h files, and then include_dirs ends up pointing to a folder which does not contain the .h files.