sysconfig data is generated with the host Python's Makefile when cross-compiling

[FFY00]

Bug report

Bug description:

Currently, the POSIX sysconfig data is generated with the Makefile from the host interpreter, instead of the target. This results in incorrect data, as demonstrated by https://github.com/python/cpython/actions/runs/12090193539/job/33716932397?pr=127426.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

• GH-127429: fix sysconfig data generation on cross-builds #127430

[zooba]

I still see this in my tests where I deliberately relocate Python after building.

Can prefix be made variable in the JSON file? I guess that means doing some reverse substitutions when emitting it.

Alternatively, can we skip the test when the current prefix doesn't match the one in the JSON file?

[FFY00]

We could ignore the prefix if it doesn't match the running interpreter.

Though, in your use-case, is the install expected to be relocated during its lifetime, or do you just do it as an intermediate step?

[zooba]

I'm building it and packaging it up so that other people can download a pre-built interpreter into their builds. I can't control where they extract it to, so finding the prefix at runtime is basically necessary.

After it's been run for the first time it's unlikely to move again, but I don't get to have an executable install step. So ideally we'd just calculate the prefix each time we launch and never hard code it.

[FFY00]

Alright, I am not too worried about those use-cases. My recommendation would be to probably manually set the prefixes to null after building.

For all practical purposes, this file is not critical, we don't load data from it currently, and if we move towards that, we will be definitely keep calculating the prefix at runtime. Right now it is provided as a nice-to-have for introspection, but not for folks to rely on always existing.

Regarding the test, yep, we should definitely either ignore the prefixes when comparing the data, or skip it, so that it passes on the situation you described.

[zooba]

manually set the prefixes to null after building.

Can you be more specific about what you mean by this? Do I try and patch the generated JSON file?

I wonder if there's a way we can override configure --prefix=... to not generate a prefix at all? It won't install, obviously, but we should be able to build, right? (Or it will require a parameter to make install, but still won't embed the prefix in the build.)

[FFY00]

Yep. Either that, or just don't ship the JSON file. These are just my suggestions anyway, feel free to go for what you feel most comfortable with.

I think you should actually be able to build with an empty prefix (empty string), and install to a relocatable location using DESTDIR.

[zooba]

Well this isn't an error I expected to see:

Traceback (most recent call last):
  File "/mnt/vss/_work/1/s/cpython/Lib/site.py", line 80, in <module>
    PREFIXES = [sys.prefix, sys.exec_prefix]
                ^^^^^^^^^^
AttributeError: module 'sys' has no attribute 'prefix'

But yeah, passing an empty --prefix got way further than I expected. Might only be a few tweaks needed for that to be a viable approach.

[FFY00]

Huh, interesting!

[FFY00]

Fixed via GH-127430. The relocating Python after build use-case should probably be covered in GH-119696.