GH-107956: install a static build description file #108483

FFY00 · 2023-08-25T14:23:41Z

This is a prototype, ready for an initial review. My main goal right now is to solidify/settle the key proposal implementation details (format, name, install location, etc).

Issue: Install a static installation description file as part of the Python installation #107956

FFY00 · 2023-08-25T14:29:53Z

After reaching a consensus on the key implementation details, I will prepare this PR for a final review (add documentation, tests, validate/fix cross-builds, etc.).

We can then move to adding more information to file, which involves further discussions of what should be included and how.

FFY00 · 2023-09-01T03:28:53Z

Friendly ping.

I know the details in question here are fairly unconstrained, so it can be very subjective, making it a maybe difficult thing to review, but the sooner we have a consensus and set the initial details, the sooner we can progress to the more technical and actually useful followup discussions. I have some employer time now to work on this, so I wanted to make sure I take advantage of it and help move this forward.

AA-Turner

The implementation looks good (one comment about Windows aside). I think this should also be added to PCBuild?

I've no strong thoughts on the filename; I agree with your thoughts on TOML vs JSON on the issue, though JSON would be slightly easier to write (no custom writer in the script etc).

A

AA-Turner · 2023-09-01T04:42:37Z

Tools/build/generate_install_details_file.py

+def toml_dump(fd: io.TextIOBase, data: ConfigData) -> None:
+    """**Very** basic TOML writer. It only implements what is necessary for this use-case."""
+    def toml_repr(obj: object) -> str:
+        if isinstance(obj, dict):
+            return '{ ' + ', '.join(f'{k} = {toml_repr(v)}' for k, v in obj.items()) + ' }'
+        else:
+            return repr(obj)
+
+    for section, entries in data.items():
+        print(f'[{section}]', file=fd)
+        for name, value in entries.items():
+            print(f'{name} = {toml_repr(value)}', file=fd)
+        fd.write('\n')


This doesn't work on Windows, as repr(value) escapes backslashes and uses single quotes, which TOML interprets as literal strings.

I think it would also be nice if the parser emitted dicts as tables:

[python.version_parts] major = 3 minor = 11 micro = 5 releaselevel = 'final' serial = 0

The following suggestion does both, but the fix for Windows is just changing repr(obj) to '{obj'}.

Suggested change

def toml_dump(fd: io.TextIOBase, data: ConfigData) -> None:

"""**Very** basic TOML writer. It only implements what is necessary for this use-case."""

def toml_repr(obj: object) -> str:

if isinstance(obj, dict):

return '{ ' + ', '.join(f'{k} = {toml_repr(v)}' for k, v in obj.items()) + ' }'

else:

return repr(obj)

for section, entries in data.items():

print(f'[{section}]', file=fd)

for name, value in entries.items():

print(f'{name} = {toml_repr(value)}', file=fd)

fd.write('\n')

def toml_dump(fd: io.TextIOBase, data: list[tuple[str, dict[str, ConfigItem]]]) -> None:

"""**Very** basic TOML writer. It only implements what is necessary for this use-case."""

for section, entries in data:

if fd.tell() != 0: # don't emit a newline before the first section

fd.write('\n')

fd.write(f'[{section}]\n')

subsections = []

for name, value in entries.items():

if isinstance(value, dict):

subsections.append((f'{section}.{name}', value))

elif isinstance(value, str):

fd.write(f"{name} = '{value}'\n")

else:

fd.write(f"{name} = {value!r}\n")

toml_dump(fd, subsections)

(This also involves changing Line 57 to toml_dump(f, config.items()).

This doesn't work on Windows, as repr(value) escapes backslashes and uses single quotes, which TOML interprets as literal strings.

Thank you for catching this! I actually wasn't aware of the difference between basic and literal strings in TOML. I don't think we should run into any such instances here, at least for now, given the small scope, but we should try to avoid that if possible.

I think it would also be nice if the parser emitted dicts as tables:

[python.version_parts] major = 3 minor = 11 micro = 5 releaselevel = 'final' serial = 0

Yeah, I don't have any very strong thoughts about it, so we can go with whatever makes the implementation simpler.

The following suggestion does both, but the fix for Windows is just changing repr(obj) to '{obj'}.

That causes other issues, like escaping quotes. I think I am just gonna make an escape table and call str.translate.

The problem is more that in the previous version, e.g. the executable string would be 'S:\\Development\\tmp\\venv\\Scripts\\python.exe' on Windows, which then was printed to the TOML file as executable = 'S:\\Development\\tmp\\venv\\Scripts\\python.exe', but then we hit the literal-vs-escaped string behaviour in TOML.

The PR as it stands now (with translation table) prints the following to TOML: executable = "S:\\\\Development\\\\tmp\\\\venv\\\\Scripts\\\\python.exe" -- the quotation marks are right but we've now got doubled-up backslashes!

I think that the fix is:

- 0x005C: r'\\\\', + 0x005C: '\\\\',

A

This thread makes me want to vote more strongly for either JSON or simple key=value with no quoting or escaping.

TOML parsers are not universal, which means a lot of tools are going to end up hacking together just enough of a parser to make things work enough of the time. Better to go with either something that's universal-enough, or something that's trivial to parse.

I understand.

This thread makes me want to vote more strongly for either JSON or simple key=value with no quoting or escaping.

IMO the simple key=value option wouldn't be a great option for the following reasons:

Being a custom format, it requires a custom parser anyway, which no matter how simple we intend it to be, it'd probably end up resulting in more user code

If we support different data types, the user implementation wouldn't be trivial, which I think goes against the purpose of having a "simple" mapping

If everything is a string, since we will still need to represent non-string data (eg. ints, bools, etc.), I think users would either end up implementing the type parsing code anyway, or it would be much more cumbersome for them to use the data, as they would likely still need to convert it to a useful type so that they can actually use the information

The lack of sections makes organization worse, makes it less ergonomic to have direct maps to the sysconfig API ¹, and makes it more cumbersome for users to use the data (eg. the version_parts field would need to be split in 5 different fields, on top of that, each likely with a large prefix)

That said, JSON is a reasonable choice, I just dislike how annoying it is to edit it, maybe even painful if you aren't very well familiarized with it. Not that is a big deal, it's just my personal opinion.

TOML parsers are not universal, which means a lot of tools are going to end up hacking together just enough of a parser to make things work enough of the time. Better to go with either something that's universal-enough, or something that's trivial to parse.

That is fair, I guess, though all major languages I am aware of have a TOML parser library, and looking at TOML's adoption rate I think that will only improve, so, realistically, I don't think this is really an issue. Though, to be clear, I do completely understand the concern.

Anyway, maybe I am just being a bit too opinionated here 🤣

Footnotes

I would like to have direct mappings, where possible, from this file to the data structures in the new sysconfig API. ↩

Taking this back to the issue #107956

Tools/build/generate_install_details_file.py

brettcannon · 2023-09-01T23:16:46Z

Friendly ping.

If that was meant for me, I have been waiting for the issue discussion to finish and this PR to come out of draft.

FFY00 · 2023-09-04T21:47:19Z

If that was meant for me, I have been waiting for the issue discussion to finish and this PR to come out of draft.

Oh okay. I don't think there's anything necessarily unresolved in the issue regarding the implementation choices made in this PR -- the feedback I got there seems compatible with this PR. There wasn't really a clear plan set in the issue, so my thought was to get and ACK/LGTM regarding the implementation choices made here while in draft state, to give more opportunity for tweaks/changes before I put in the effort to finalize this PR for review.

I'll just move the discussion back to the issue then. I will ask there for feedback there on the specific choices this PR makes.

bedevere-bot · 2023-10-05T17:57:27Z

🤖 New build scheduled with the buildbot fleet by @FFY00 for commit fdb10c0 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

bedevere-bot · 2023-10-05T23:05:58Z

🤖 New build scheduled with the buildbot fleet by @FFY00 for commit 12a7b04 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

bedevere-bot · 2023-10-06T01:18:52Z

🤖 New build scheduled with the buildbot fleet by @FFY00 for commit afa71ab 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

bedevere-bot · 2023-10-06T14:01:28Z

🤖 New build scheduled with the buildbot fleet by @FFY00 for commit 01a1665 🤖

If you want to schedule another build, you need to add the 🔨 test-with-buildbots label again.

FFY00 · 2023-10-10T23:02:37Z

I believe the left over buildbot failures are unrelated. I think this PR is ready for review.

zooba · 2023-10-11T22:13:39Z

I do plan to get to this, just haven't had time today. Hopefully tomorrow!

Doc/using/introspection.rst

zooba · 2023-10-12T15:23:52Z

Doc/using/introspection.rst

+++++++++++++++++++++
+
+:Type: ``str``
+:Description: An absolute path pointing to an interpreter from the installation.


Can we make these paths (optionally) relative from the directory containing the file? That way it can be static for most scenarios, instead of needing to be dynamically generated (and e.g. in the Windows Store package we can't dynamically generate it and also put it into the expected location, because it's read-only).

Alternatively, include a few variables like {prefix} that the reader can substitute themselves.

Yeah, I think that makes sense for use-cases like yours.

zooba · 2023-10-12T15:25:45Z

Doc/using/introspection.rst

+~~~~~~~~
+
+On standard CPython installations, when available, the details file will be
+installed in the same directory as the standard library, with the name


In some cases the standard library is split. Can we name a specific file it will be alongside? (e.g. the tests below expect it to be next to sysconfig.py)

Also, on Windows, this implies that everyone should hard-code Lib into their paths, because the only paths you can resolve without running Python are sys.prefix and sys.executable. (I'm not sure how best to go about locating the stdlib on other platforms either, to be honest, but I trust you for those.)

In some cases the standard library is split. Can we name a specific file it will be alongside? (e.g. the tests below expect it to be next to sysconfig.py)

Right, I will clarify it is the pure part of the standard library, which corresponds to the sysconfig stdlib path.

Also, on Windows, this implies that everyone should hard-code Lib into their paths, because the only paths you can resolve without running Python are sys.prefix and sys.executable. (I'm not sure how best to go about locating the stdlib on other platforms either, to be honest, but I trust you for those.)

I don't see a great solution for this, given that sys.prefix may contain multiple installations. My main two target use-cases are cross-compilation, where the user should be able to locate install-details.json themselves, and Python launchers, where, knowing that stdlib is within sys.prefix, should be able to search for the **/install-details.txt pattern in order to discover the available installation within a prefix.

The approach for Python launchers is not optimal, but it can be fixed later. Currently, I am just focusing on getting the file format down, and making the file available somewhere. Afterwards, we can look at the discoverability issue and consider for eg. having a global registry where these files could be additionally installed (eg. on Linux this could be the ~/.config/python/installations and /etc/python/installations paths), but that's a whole other issue, and I am trying to break things down into smaller chunks so that I can actually hope to make meaningful progress.

What do you think?

zooba · 2023-10-12T15:27:24Z

Doc/using/introspection.rst

+
+.. availability:: POSIX
+
+A ``python-config`` script may be available alongside the interpreter


Will this script grow an option to reveal the location of the install-details.json file?

I think that's reasonable, but I'd rather put it in the sysconfig CLI, which is less problematic.

Except we need a way to locate the file without launching the runtime? Which is why python-config is a shell script and not a Python script.

You're right 🤦

I just remembered that now when looking at GH-77620.

zooba · 2023-10-12T15:29:12Z

Doc/using/introspection.rst

+.. TODO: Currently, we don't have any documentation covering python-config, but
+         if/when we add some, we refer to it here, instead.
+
+.. availability:: POSIX


For Windows, probably the best equivalent is to recommend a PEP 514 addition to specify the full path to install-details.json. Probably an InstallDetailsPath value alongside the existing ExecutablePath one (mentioned in this section).

zooba · 2023-10-12T15:33:15Z

Tools/build/generate_install_details_file.py

+                for field in ('major', 'minor', 'micro', 'releaselevel', 'serial')
+            },
+            'executable': executable,
+            'stdlib': sysconfig.get_path('stdlib'),


This assumes that we're running in the target interpreter, yeah? So we need another way to generate this file for when we're cross-compiling (which is a normal part of the Windows build process).

Is there an easy way to have the script pick up all the makefile variables during a build without having to actually parse the makefile?

No, but we can add a CLI option to specify the value, like we do for executable. This way you just give the script the value when you call it in the Makefile, or other build system.

I'd like to make this script as standalone as possible.

zooba · 2023-10-12T15:34:28Z

I forget where we landed on needing a PEP for this or not, but right now I feel like we probably do.

FFY00 · 2023-10-12T18:35:01Z

Is that really required, considering this is an optional feature?

zooba · 2023-10-12T19:31:07Z

We tend to formalise specs for things that we expect others to follow to properly record the discussion. It doesn't make it a requirement to do the thing, just lays out the expectations that if you do want to include this file, here's how it should look.

The doc in the PR is pretty close to how I'd expect it to look. PEP 514 is a similar sort of thing, so I wouldn't expect it to be drastically longer or different to that one.

FFY00 · 2023-10-12T22:40:33Z

That makes sense, but I didn't want the spec to be formalized by a PEP, and instead be a living specification which we can extend and iterate over time. In fact, this PR just includes a couple of initial data keys, I was planning to add more afterwards.

I can write a PEP with an initial specification, though, but I'll make it clear there that the up-to-date version of the spec is on CPython docs.

Hoping that the PEP process won't take that much time 😅

zooba · 2023-10-12T23:14:06Z

I can write a PEP with an initial specification, though, but I'll make it clear there that the up-to-date version of the spec is on CPython docs.

This is how it's supposed to work anyway. It's only when we never properly document things that the PEP remains the canonical source (or when it's informational, which PEP 514 is, so don't copy that bit).

As long as you specify how tools should handle unexpected keys or missing keys, and include a naming convention for custom/non-standard keys, this ought to be fine. (In PEP 578 I only listed examples of audit hooks, to avoid having to argue about each one before even getting started.)

Hoping that the PEP process won't take that much time 😅

At least you're the most motivated person to work on it 😄 I took on the distutils one because nobody wanted to, and that turned into way more work than I hoped.

FFY00 · 2023-10-12T23:18:42Z

As long as you specify how tools should handle unexpected keys or missing keys, and include a naming convention for custom/non-standard keys, this ought to be fine. (In PEP 578 I only listed examples of audit hooks, to avoid having to argue about each one before even getting started.)

Alright. I will add a schema key then, for future-proofing, if we ever need to iterate over the format.

Signed-off-by: Filipe Laíns <[email protected]>

Co-authored-by: Steve Dower <[email protected]> Signed-off-by: Filipe Laíns <[email protected]>

FFY00 · 2024-02-24T16:07:36Z

Rebased on top of main without making any changes.

Signed-off-by: Filipe Laíns <[email protected]>

FFY00 · 2025-03-03T10:12:25Z

Closing as superseded by GH-130069.

FFY00 requested review from brettcannon and zooba August 25, 2023 14:23

bedevere-bot mentioned this pull request Aug 25, 2023

Install a static installation description file as part of the Python installation #107956

Closed

FFY00 force-pushed the gh-107956 branch from 72ae9ca to bed56f7 Compare August 25, 2023 14:26

FFY00 mentioned this pull request Sep 1, 2023

Prototype support for --python.target-config mesonbuild/meson#12193

Closed

AA-Turner reviewed Sep 1, 2023

View reviewed changes

FFY00 marked this pull request as ready for review October 5, 2023 17:57

bedevere-app bot added the awaiting review label Oct 5, 2023

FFY00 added 🔨 test-with-buildbots Test PR w/ buildbots; report in status section and removed awaiting review labels Oct 5, 2023

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 5, 2023

FFY00 added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 5, 2023

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 5, 2023

FFY00 added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 6, 2023

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 6, 2023

FFY00 added the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 6, 2023

bedevere-bot removed the 🔨 test-with-buildbots Test PR w/ buildbots; report in status section label Oct 6, 2023

zooba reviewed Oct 12, 2023

View reviewed changes

FFY00 and others added 14 commits February 24, 2024 10:28

pythonGH-107956: install a static description file

5031753

Signed-off-by: Filipe Laíns <[email protected]>

escape strings correctly

4da5996

Signed-off-by: Filipe Laíns <[email protected]>

compare the data loaded by tomllib to what we wrote in the sanity check

4047d86

Signed-off-by: Filipe Laíns <[email protected]>

change the format to JSON

b71fab2

Signed-off-by: Filipe Laíns <[email protected]>

add tests

de5729e

Signed-off-by: Filipe Laíns <[email protected]>

add news

4155f2c

Signed-off-by: Filipe Laíns <[email protected]>

fix test_python_stdlib

4adb942

Signed-off-by: Filipe Laíns <[email protected]>

fix the install-details.json installation

d1eaf10

Signed-off-by: Filipe Laíns <[email protected]>

get the interpreter path from the build system

5b3fe21

Signed-off-by: Filipe Laíns <[email protected]>

fix WASM tests

7e7c59a

Signed-off-by: Filipe Laíns <[email protected]>

add missing dependency to the desciprion file install target

f792b64

Signed-off-by: Filipe Laíns <[email protected]>

fix descfileinstall target dependency

b8d6aed

Signed-off-by: Filipe Laíns <[email protected]>

skip install-details.json test on wasi and emscripten when not available

15f8a1a

Signed-off-by: Filipe Laíns <[email protected]>

Apply suggestions from code review

c96e447

Co-authored-by: Steve Dower <[email protected]> Signed-off-by: Filipe Laíns <[email protected]>

FFY00 force-pushed the gh-107956 branch from 4434187 to c96e447 Compare February 24, 2024 16:06

rename file to build-details.json

217682b

Signed-off-by: Filipe Laíns <[email protected]>

FFY00 changed the title ~~GH-107956: install a static description file~~ GH-107956: install a static build description file Feb 24, 2024

brettcannon removed their request for review September 13, 2024 22:23

encukou added type-feature A feature request or enhancement topic-installation labels Jan 24, 2025

FFY00 closed this Mar 3, 2025


		.. availability:: POSIX

		A ``python-config`` script may be available alongside the interpreter

GH-107956: install a static build description file #108483

GH-107956: install a static build description file #108483

Conversation

FFY00 commented Aug 25, 2023 • edited by bedevere-bot Loading

FFY00 commented Aug 25, 2023

FFY00 commented Sep 1, 2023

AA-Turner left a comment

Choose a reason for hiding this comment

AA-Turner Sep 1, 2023 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

FFY00 Sep 5, 2023 • edited Loading

Choose a reason for hiding this comment

Footnotes

Choose a reason for hiding this comment

brettcannon commented Sep 1, 2023

FFY00 commented Sep 4, 2023

bedevere-bot commented Oct 5, 2023

bedevere-bot commented Oct 5, 2023

bedevere-bot commented Oct 6, 2023

bedevere-bot commented Oct 6, 2023

FFY00 commented Oct 10, 2023

zooba commented Oct 11, 2023

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

zooba commented Oct 12, 2023

FFY00 commented Oct 12, 2023

zooba commented Oct 12, 2023

FFY00 commented Oct 12, 2023

zooba commented Oct 12, 2023 • edited Loading

FFY00 commented Oct 12, 2023

FFY00 commented Feb 24, 2024

FFY00 commented Mar 3, 2025

FFY00 commented Aug 25, 2023 •

edited by bedevere-bot

Loading

AA-Turner Sep 1, 2023 •

edited

Loading

FFY00 Sep 5, 2023 •

edited

Loading

zooba commented Oct 12, 2023 •

edited

Loading