add pip-compatible --group flag to uv pip install and uv pip compile

[Gankra]

This is a minimal redux of #10861 to be compatible with uv pip.

This implements the interface described in: pypa/pip#13065 (comment) for uv pip install and uv pip compile. Namely --group <[path:]name>, where path when not defined defaults to pyproject.toml.

In that interface they add --group to pip install, pip download, and pip wheel. Notably we do not define uv pip download and uv pip wheel, so for parity we only need to implement uv pip install. However, we also support uv pip compile which is not part of pip itself, and --group makes sense there too.

---

The behaviour of --group for uv pip commands makes sense for the cases upstream pip supports, but has confusing meanings in cases that only we support (because reading pyproject.tomls is New Tech to them but heavily supported by us). Specifically case (h) below is a concerning footgun, and case (e) below may get complaints from people who aren't well-versed in dependency-groups-as-they-pertain-to-wheels.

Only Group Flags

Group flags on their own work reasonably and uncontroversially, except perhaps that they don't do very clever automatic project discovery.

a) uv pip install --group path/to/pyproject.toml:mygroup pulls up path/to/project.toml and installs all the packages listed by its mygroup dependency-group (essentially treating it like another kind of requirements.txt). In this regard it functions similarly to --only-group in the rest of uv's interface.

b) uv pip install --group mygroup is just sugar for uv pip install --group pyproject.toml:mygroup (note that no project discovery occurs, upstream pip simply hardcodes the path "pyproject.toml" here and we reproduce that.)

c) uv pip install --group a/pyproject.toml:groupx --group b/pyproject.toml:groupy, and any other instance of multiple --group flags, can be understood as completely independent requests for the given groups at the given files.

Groups With Named Packages

Groups being mixed with named packages also work in a fairly unsurprising way, especially if you understand that things like dependency-groups are not really supposed to exist on pypi, they're just for local development.

d) uv pip install mypackage --group path/to/pyproject.toml:mygroup much like multiple instances of --group the two requests here are essentially completely independent: pleases install mypackage, and please also install path/to/pyproject.toml:mygroup.

e) uv pip install mypackage --group mygroup is exactly the same, but this is where it becomes possible for someone to be a little confused, as you might think mygroup is supposed to refer to mypackage in some way (it can't). But no, it's sourcing pyproject.toml:mygroup from the current working directory.

Groups With Requirements/Sourcetrees/Editables

Requirements and sourcetrees are where I expect users to get confused. It behaves exactly the same as it does in the previous sections but you would absolutely be forgiven for expecting a different behaviour. Especially because --group with the rest of uv does do something different.

f) uv pip install -r a/pyproject.toml --group b/pyproject.toml:mygroup is again just two independent requests (install a/pyproject.toml's dependencies, and b/pyproject.toml's mygroup).

g) uv pip install -r pyproject.toml --group mygroup is exactly like the previous case but incidentally the two requests refer to the same file. What the user wanted to happen is almost certainly happening, but they are likely getting "lucky" here that they're requesting something simple.

h) uv pip install -r a/pyproject.toml --group mygroup is again exactly the same but the user is likely to get surprised and upset as this invocation actually sources two different files (install a/pyproject.toml's dependencies, and pyproject.toml's mygroup)! I would expect most people to assume the --group flag here is covering all applicable requirements/sourcetrees/editables, but no, it continues to be a totally independent reference to a file with a hardcoded relative path.

---

Fixes #8590
Fixes #8969

[Gankra]

I started doing a cleanup pass to compute a BTreeMap<PathBuf, Vec<GroupName>> that would be used by subsequent steps but it was just more code and value threading for no real benefit over the current impl that just does a filter(group.path == path) later.

[henryiii]

FYI, pip support was merged.

[zanieb]

Are these conflicts present in the pip implementation?

[zanieb]

(it doesn't really look like it?)

[Gankra]

I thought -r was exclusive to our CLI (I didn't check the others though).

[zanieb]

Yeah -r is exclusive to us, but I'd expect pip install -e . --group foo and pip install anyio --group foo to work unless they explicitly choose for it not to upstream.

[zanieb]

(Whether -r should have the same semantics as these other options, I'm not sure)

[zanieb]

One problem is that uv pip install -r pyproject.toml --extra foo is the "correct" use of --extra but uv pip install -r pyproject.toml --group foo would be invalid? Even more confusing, if we allow it, uv pip install -r foo/pyproject.toml --group bar would install bar from ./pyproject.toml instead of foo/pyproject.toml.

[Gankra]

Yeah that's part of why I opted to disallow it. The upstream pip semantics are really confusing to pair with anything else.

[zanieb]

I think it's important that we're compatible with pip on whether --group can be used with <package> and --editable though, which introduces all the same complexities.

[Gankra]

I implemented/pushed the following changes, because I needed to wrap my head around them in code/tests to think them through semantically:

• totally unrestrict --group from all other sources in uv-pip-install
  • semantically i still have it that --group just installs that group in a pyproject.toml (still hardcoded ./pyproject.toml default for now, see below). All the following cases are accepted:
    • "unrelated": uv pip install somepackage --group bar (essentially sugar for two separate uv pip install invocations as it is in upstream pip install)
    • "nice": uv pip install -r pyproject.toml --group bar (installs both the package's deps and its group bar) (but this is the user of the cli potentially getting "lucky")
    • "confusing": uv pip install -r foo/project.toml --group bar (installs foo/pyproject.toml's deps and ./pyproject.toml:bar)
  • implementation-wise this was a bit hairy as we have to kind of detect if we're doing --group or --only-group by looking at all the other inputs, but it ends up making sense (really it's always --only-group-but-additive-only... it's just an artifact of the internal source_tree stuff and wanting to dedupe files that we need to do anything complex).
• enable the same semantics for uv-pip-compile
  • ...this one I "implemented" and then I went to test it and realized that the semantics for pip-install are absolute nonsense here, i think? i think the old-old-old impl from last month where the --group just applies to every selected source is what you want here..? I'm not totally sure, I don't have a good intuition of what uses of this would look like... but I get the impression that uv pip compile --group pyproject.toml:foo is essentially gibberish? Or am I wrong?
• did not add project discovery in lieu of hardcoding pyproject.toml -- I was going to do this but then I realized that nowhere in the pip interface do we ever seem to invoke project discovery? Which was surprising. This seems like a big semantic departure if we start doing that? At very least needs to be discussed more.

[Gankra]

Zanie convinced me that pip-compile is fine actually, so I just loosened the CLI restriction to make it accepted.

I'm just adding more tests now.

[zanieb]

Can you share the current state of this? It looks like you updated the PR summary to cover the behavior we discussed?

[Gankra]

I believe the PR is in the final state I'd like. Possibly it could have Even More tests but I'm not sure it would add much. (Just rebased)

[zanieb]

This looks pretty good. I have mostly minor comments, though the --project bit might be blocking?

We should add this to the documentation at

• https://docs.astral.sh/uv/pip/compile/
• https://docs.astral.sh/uv/pip/packages/

We may want to mark --group on uv pip compile as "in preview" until support lands in pip-tools. See jazzband/pip-tools#2062. It looks like they're just considering using :: instead of : as a separator? Moving this to preview would involve a runtime warning and a note in the doc.

[zanieb]

I'm pretty sure this makes sense, but it might be relevant for Charlie to take a look at before merging.

[Gankra]

Ooookay I think that's all review feedback addressed!

[zanieb]

Probably want "in the pyproject.toml"

[Gankra]

This is actually a path we're displaying, maybe "was not found: {path}"?

[zanieb]

You could say "The dependency group 'bar' was not found in the project at {}"

It's a little weird for the working directory, maybe we'd want a ./ there?

[Gankra]

If we wanna do that I'd make it a change to path.user_display() across the board (I think it's fine as is...)

[zanieb]

I don't think it makes sense to do it across the board. "in the project at pyproject.toml" reads pretty weird. A little better if it's in backticks though. We can always special case that though, "in the project in the working directory" or something.

[zanieb]

I think this is ready. It's hard to keep track of everything with the scope of the changes.

I have a few stylistic nits remaining and want to see more user-facing documentation explaining how --group interacts with other options and how it differs from the rest of the uv interface. We can push that into a separate pull request, but it's very important. I'm happy to help with that if you need it.

[Gankra]

Rebased and addressed review in latest commit.

[zanieb]

Backticks on the path?

[Gankra]

Oh I assumed we purposefully avoided that in our style so the path was easy to copy/alt-click in terminals

[Gankra]

(This is why I suggested the : {path} approach)

[zanieb]

We either do : {path} or use backticks.

[zanieb]

I would not say "This is an attempt" in public facing copy. I'd just omit that whole sentence.

I think the second bit is a separate note? I'd use !!! note for the first part. Then add a separate !!! important that says

A --group flag has not yet been implemented in pip-tools and we may change our semantics to match the upstream behavior when it is finalized. The upstream behavior can be tracked in ...

I'd move this to the first use of pip compile --group above a bit so there's not two admonitions in a row.

[Gankra]

heh, yeah I wanted two separate notes but got stuck on the back-to-backness. adding distance between them is a fine compromise.

[zanieb]

I wouldn't say "This matches the behaviour that pip install is shipping." — it'll be stale once they ship it and is a bit superfluous here as the assumption is we match pip's behavior.

[zanieb]

You could say, "As in pip, " if you want to mention that it is done to match pip's behavior.

[Gankra]

Ooh I like that. I felt the need to specify that because I don't think we would have picked that behaviour in a vacuum and I expect some people will want to suggest changing it. Maybe too defensive 😅

[Gankra]

Done for reals?

[zanieb]

Yeah let's go :D