Pip should reorganise its internal API along the lines of having more modular commands

[pfmoore]

Personally, I wish that functionality like this could be separated out into more modular tools, because I don't see this as being a core piece of pip functionality, rather it's a piece of functionality that needs to be in pip just to re-use chunks of pip's machinery. But that's a whole other discussion, and not for now.

Originally posted by @pfmoore in #10748 (comment)

[pfmoore]

Making this an issue just to track the basic ideas. We already have a fairly modular internal structure, and there is ongoing work to refactor and tidy up pip's code base. So this isn't, in many senses, anything new.

What I'm really thinking here is that while we may not want to have a "pip API" to allow pip to be used as a library, we should design (and document the design) in a way that our implementations of pip subcommands are as near as possible standalone scripts, and the rest of pip is structured as a library that can be called by those commands.

Step 1 here would be to review the code base to check how much we're already doing this. This isn't a new idea, and it's been a while since I looked at the overall "shape" of the codebase, so it's possible that we are close to this already. (On the other hand, it's equally possible that we have a bunch of closely coupled modules, because expediency wins over perfection every time 🙂)

There's also a downside, in that no matter how much we say "pip is an application, and doesn't have a programmatic API", if we write the code as if it had an API, people will use it that way...

At some point, I'll do a review and add more concrete points here. If others want to do the same, that's fine as well.

[cosmicexplorer]

Personally, I wish that functionality like this could be separated out into more modular tools, because I don't see this as being a core piece of pip functionality, rather it's a piece of functionality that needs to be in pip just to re-use chunks of pip's machinery. But that's a whole other discussion, and not for now.

This is a really interesting point that I hadn't previously considered. In the above comment I describe how pex invokes pip by command-line, but with chagrin I realize that I was speaking past you a little, since you're explicitly saying that a library API would help to make this resolve goal less central to pip's application API, which would justify pip itself having a library API, not an external tool. Sorry for the noise--I think that makes sense!

For what it's worth, my perception is that pex (for example) would be able to invoke a pip library API at least as easily as a cli, so for pex's use case (in pex-tool/pex#1137 we discuss pex's ability to produce a .dot visualization of the pip resolve, for example), it would probably not be too difficult to invoke pip as a library API for the case of just doing a resolve (especially if more work on fast-deps can speed up the resolve). So I would like to amend my above comment to say: I totally get this now, and it would likely work for use cases that need internal pip information without needing to make a whole new command to extract it.

[pradyunsg]

What do you think this additional modularity looks like?

[pradyunsg]

I ask, because it's unclear to me what exactly you want to make more modular? Registration of commands, or options, or decoupling the core logic from the rest of pip? Or something else entirely?

[pradyunsg]

Right now, the only thing that the commands don't do is register themselves -- that is a hard-coded list because imports are expensive.

Everything else about a command is basically defined in the commands/*.py file; using some options defined in cmdoptions.py; and I'm not sure what this issue wants to change about this setup?

The only things that's a bit annoying is the cmdoptions vs inline options thing, but that doesn't directly affect modularity.

[pfmoore]

What do you think this additional modularity looks like?

As I say, I've not looked into this yet, it's "just a feeling", and this issue is intended mostly for my own benefit to remind me to take a proper look and see what, if anything, is actionable.

But what I'm mainly thinking about is having it be possible¹ for things like the pip resolve command that triggered this thought to be written as standalone commands. So rather than changing a bunch of pip's internals, the command would have all of its dedicated logic but still be able to use pip's machinery like the finder, the resolver, the network stack and logging, the build isolation and pyproject.toml support, etc.

The benefits I'm imagining are cleaner separation of concerns, easier prototyping of new commands and/or reorganising the UI, and better maintainability (mostly from the increased modularity). It also might allow us to spin parts of pip off as external libraries, but that would involve decoupling the implementations and not just the APIs, so may not be realistic.

Footnotes

1. But not recommended, I'm not suggesting pip have an API that we support and which has any sort of backward compatibility! ↩

[pradyunsg]

OK, cool. I don't think there's anything particularly actionable here; but if someone else wants to take a look or systematically evaluate things, I'm not gonna block anyone. :)