Support for PEP-751 lockfiles

[ndellosa95]

Is your feature request related to a problem? Please describe.
PEP-751, a standardized lockfile format for Python, is accepted. While we need to wait for the tools Pants leverages to generate lockfiles to support it, we can use this ticket to track any necessary changes within Pants itself.

Describe the solution you'd like
pants generate-lockfiles generates a PEP-751 lockfile. I've opened a related ticket on the pex side for this work. Presumably Pants will just need to delegate to pex, which will delegate to pip, once this is implemented there.

Describe alternatives you've considered
Uv will support this as well at some point too. It would still be ideal to support uv as an alternative to pex in the future.

[jsirois]

@ndellosa95 I don't think you read that uv ticket closely enough. I was very involved in those PEP discussions and bowed out when they decided to restrict the PEP to "a modern Pip requirements" format. Pex will be able to use it as a lock export format but not as a lock file format for the same reasons as uv. It does not support subsetting with its required features. To support subsetting optional features need to be used and tool-specific metadata needs to be added and the lock is then not portable (uv tool specific metadata != Pex tool specific metadata). This was an explicit punt on Brett's behalf to try to get something out the door.

[jsirois]

@ndellosa95 I asked this on the Pex ticket, but can you be more specific in describing how you'd use this new lock format / what the perceived benefit is? At some point, it will be supported by installers (think AWS Lambda), at which point pex3 lock export support will be useful. Is that the sort of thing you care about?

[cognifloyd]

Getting away from the pex-specific JSON format lockfiles would be my ideal resolution.

Pants has metadata about the lockfile that is designed to stay in the same file. Pants is not allowed to rely on the data contained within the non-standardized lockfile format that is subject to change in any version of PEX (it is not a documented API), so pants actually breaks the JSON by adding its metadata in a comment that gets stripped before passing the lockfile to PEX.

If pex could produce and consume this PEP-751 lockfile format, even if that means storing valuable metadata in PEX-specific extensions, then pants could make use of any of the fields that are standardized, possibly eliminating the need for that awful comment. Pants would still treat any pex extensions as opaque, not relying on the ability to read that pex-specific metadata. And if the standardized fields do not provide enough info to fully replace that comment, then pants can add its own pants-specific extensions to store the data it needs getting away from storing it in a comment.

Is this a portable use of PEP-751 that other tools could consume? No. But that's ok. Porting the lockfile to some other tool, like uv, is orthogonal to the ability for pants to work better with the lockfiles that pants asks PEX to generate and that pants passes to PEX when building pexes.

[jsirois]

@cognifloyd I contend at least part of your argument is bogus:

so pants actually breaks the JSON by adding its metadata in a comment that gets stripped before passing the lockfile to PEX.

That is a Pants problem and Pants needs to solve it independent of Pex lockfiles. Think https://docs.npmjs.com/cli/v9/configuring-npm/package-lock-json - is Pants going to try to tell the js ecosystem they need to switch to a lock format that accepts comments?

Pants could be sticking the comment in a top-level JSON field - I have suggested this before. I.E.:

:; pex3 lock create cowsay --indent 2 -o lock.json

# Add a comment in the sane way.
:; vi lock.json

{
  "__pants_lock_metadata__": {
    "datum1": "Pex is gracious and is not bothered by this."
  },
  "allow_builds": true,
  "allow_prereleases": false,
  "allow_wheels": true,
  "build_isolation": true,
  "constraints": [],
  "elide_unused_requires_dist": false,
  "excluded": [],
  "locked_resolves": [
    {
      "locked_requirements": [
        {
          "artifacts": [
            {
              "algorithm": "sha256",
              "hash": "274b1e6fc1b966d53976333eb90ac94cb07a450a700b455af9fbdf882244b30a",
              "url": "https://files.pythonhosted.org/packages/f1/13/63c0a02c44024ee16f664e0b36eefeb22d54e93531630bd99e237986f534/cowsay-6.1-py3-none-any.whl"
            }
          ],
          "project_name": "cowsay",
          "requires_dists": [],
          "requires_python": ">=3.8",
          "version": "6.1"
        }
      ],
      "platform_tag": [
        "cp311",
        "cp311",
        "manylinux_2_41_x86_64"
      ]
    }
  ],
  "only_builds": [],
  "only_wheels": [],
  "overridden": [],
  "path_mappings": {},
  "pex_version": "2.33.8",
  "pip_version": "20.3.4-patched",
  "prefer_older_binary": false,
  "requirements": [
    "cowsay"
  ],
  "requires_python": [],
  "resolver_version": "pip-legacy-resolver",
  "style": "strict",
  "target_systems": [],
  "transitive": true,
  "use_pep517": null,
  "use_system_time": false
}

And does Pex care? Its format parser makes a point of not caring:

:; jq .__pants_lock_metadata__ lock.json 
{
  "datum1": "Pex is gracious and is not bothered by this."
}

:; pex --lock lock.json -c cowsay -- -t Moo!
  ____
| Moo! |
  ====
    \
     \
       ^__^
       (oo)\_______
       (__)\       )\/\
           ||----w |
           ||     ||

Another way around this is for Pants to wholesale stop using side-effects on user files to store its own state. I've also suggested this here: #14281

@cognifloyd there are likely good arguments to support PEP-751, but I will ruthlessly point out bad ones to clear away woolly thinking and get to the heart of any real motivations that stand up to scrutiny.

[jsirois]

And, late to notice, but:

then pants could make use of any of the fields that are standardized, possibly eliminating the need for that awful comment.

The current Pants metadata:

pants/3rdparty/python/mypy.lock

Lines 1 to 21 in aec9dc3

	// This lockfile was autogenerated by Pants. To regenerate, run:
	//
	// pants generate-lockfiles --resolve=mypy
	//
	// --- BEGIN PANTS LOCKFILE METADATA: DO NOT EDIT OR REMOVE ---
	// {
	// "version": 3,
	// "valid_for_interpreter_constraints": [
	// "CPython==3.11.*"
	// ],
	// "generated_with_requirements": [
	// "mypy-typing-asserts",
	// "mypy==1.13.0",
	// "strawberry-graphql==0.240.4"
	// ],
	// "manylinux": "manylinux2014",
	// "requirement_constraints": [],
	// "only_binary": [],
	// "no_binary": []
	// }
	// --- END PANTS LOCKFILE METADATA ---

The fields therein covered by PEP-751 are:

• valid_for_interpreter_constraints -> https://peps.python.org/pep-0751/#requires-python (although just optional)
• generated_with_requirements: None
• manylinux: None (Altough I think Pants could drop this field - its only used for --platform locks which it does not generate or consume).
• requirement_constraints: None
• only_binary: None
• no_binary: None

Someone may want to check me on that homework, but I think I have it right.

[jsirois]

So, I want to make a meta-comment here that's not very friendly. Pants issues of a certain sort - there are several - have this pile-on but half or not at all thought through comment nature. I stepped away from Pants for reasons in this ballpark. I expect more - do some homework, have good arguments and stay nicer than me at the same time. The community will improve for it.

[jsirois]

As of the 2.39.0 release Pex now supports both exporting pylock.toml via pex3 lock export --format pep-751 and consuming pylock.toml to build PEXes via pex --pylock. In both cases, Pex sticks to standard fields and uses no [tool... entries. Since so much is optional in the standard fields, it's not guaranteed Pex can subset a pylock.toml, which is important for Pants current use case. Notably, Pex can subset its own lock exports, but not, as of today, uv's. Those, as of today, are pretty bare bones and leave out dependencies metadata required to subset. See: astral-sh/uv#13032 (comment)

All of the above is covered in ITs.