Add support for publishing to PyPI with a trusted publisher

[sisp]

• I have searched the issues of this repo and believe that this is not a duplicate.
• I have searched the FAQ and general documentation and believe that my question is not already covered.

Feature Request

pypi.org supports publishing packages via OIDC (with a "trusted publisher"). Since one of Poetry's benefits is building and publishing packages with a single CLI, I think it would be great to support this authentication scheme as well. Also pdm supports it already.

Currently, the best approach seems to be using the GitHub Action pypa/gh-action-pypi-publish after building the distribution files using Poetry:

uses: pypa/gh-action-pypi-publish@release/v1
with:
  packages-dir: dist/

[dimbleby]

As I understand it, this needs no change at all in poetry itself: an API token derived from OIDC is a token like any other. It just happens to be short-lived.

So I guess this feature request is asking for a github action?

This repository does not currently publish any github actions. In my opinion that's fine, it's something that others can build on poetry if they want to. No need to for it to be another thing for maintainers here to worry about.

Also for what it's worth: if there's already a satisfactory github action out there in the world that publishes using the authentication method that you like - I'd say just use that! twine is a fine tool, I don't know any reason why a github action that replaced twine upload with poetry publish would be better.

[dimbleby]

I looked at pdm, fwiw that will do the pypi oidc exchange itself, if the github-specific environment variables are present (as set up by a workflow with id-token: write). See here.

In my opinion that is such a github-workflows-only piece of code that it belongs better in a github action than it does in poetry. (Or in pdm, for that matter!)

But clearly the pdm project takes a different view, perhaps someone here will think that poetry should too.

[sisp]

I see your point, @dimbleby, about this being very specific to GitHub workflows. In addition, PyPI's docs on security state:

• Limit the scope of your publishing job: your publishing job should (ideally) have only two steps:
• Retrieve the publishable distribution files from a separate build job;
• Publish the distributions using pypa/gh-action-pypi-publish@release/v1.

This means, there should be 2 jobs:

1. Build the distribution files using Poetry.
2. Publish the distribution files.

And for job #​2, there is indeed a perfectly fine and official solution from the PyPA. Perhaps it's a better separation of concerns and less code to maintain for the Poetry project to not duplicate the OIDC logic in Poetry. But then, I think it might be useful to provide a suggested workflow snippet in the Poetry docs.

[dimbleby]

just ran into https://github.com/tschm/token-mint-action, which provides a github action for obtaining the necessary token - and reports that it did its testing with a poetry published project.

so I double down on: there's no need for poetry to do anything here - and recommend closing this.

[kkirsche]

If it's already supported, would it make sense to document this more fully to show how users can leverage this functionality to avoid future issues created for this?

[dimbleby]

sure, go for it

[matfax]

If it's already supported, would it make sense to document this more fully to show how users can leverage this functionality to avoid future issues created for this?

Poetry itself doesn't support trusted publishing, but the mentioned GitHub action can generate the token that poetry can use. I think it would still be best practice to have poetry support this feature natively since it concerns a security-sensitive matter.

[dimbleby]

it would still be best practice to have poetry support this feature natively since it concerns a security-sensitive matter.

What "best practice"? You might just as well say that "it would be best practice to have something other than poetry handle this, since it concerns a security-sensitive matter" - both sentences are equally (un)convincing.

If you are worried about security then in fact I expect you will have a better time auditing the tiny amount of code in tschm/token-mint-action than in poetry and all of its dependencies. Or, you will be better off with neither and instead using pypa/gh-action-pypi-publish: that's much more widely used and therefore much more likely to have eyes on it.

[matfax]

What "best practice"?

Generally, it's not best practice to include more and more dependencies if it can be integrated with comparably little effort. Poetry is often appreciated for its user-friendly publishing process, this would furthermore add some degree of complexity. It makes the debugging and monitoring of a security-sensitive matter more complex. Is it somehow possible that the token from the GitHub action leaks, etc.?

If you are worried about security then in fact I expect you will have a better time auditing the tiny amount of code in tschm/token-mint-action than in poetry and all of its dependencies.

There's one issue with your comparison. Poetry uses locked dependencies. This means that every dependency audit remains valid into the future, since the wheels are stored by a trusted party. GitHub actions can be modified when tags are used for reference. So you would have to pin the recommendation by commit hash, and audit every update as such. To be fair, it's possible that GitHub protection rules can guarantee a certain level of tag security, but I'm not sure how transparent this is. Pinning would remain the secure equivalent to poetry's locking.

[dimbleby]

Poetry uses locked dependencies. This means that every dependency audit remains valid into the future, since the wheels are stored by a trusted party. GitHub actions can be modified when tags are used for reference

pretty much the exact opposite, actually

• installing poetry itself does not use the poetry lockfile and so today's install and tomorrow's install will likely pick up different dependencies
• depending on a github action at a specific sha does ensure that you will get the same action each time.

[matfax]

• installing poetry itself does not use the poetry lockfile and so today's install and tomorrow's install will likely pick up different dependencies

I just noticed that the project contains the lock file. What would be the reason for ignoring it? That all dependencies have their own trusted independent security supervision? Not that it affects the reasoning for my argument.

• depending on a github action at a specific sha does ensure that you will get the same action each time.

That's what I meant by pinning. Tagging a version, however, does not ensure that.

[dimbleby]

the tool that understands poetry.lock is poetry. Using poetry to install poetry would be... unusual.

you think that avoiding extra dependencies is valuable. I think that poetry already has more features than maintainers can cope with; and that this is a great example of something that is already solved by several other tools: poetry doesn't have to do everything.

We can agree to disagree. Unless and until anyone submits a merge request, no-one has to make a decision either way.

[matfax]

the tool that understands poetry.lock is poetry. Using poetry to install poetry would be... unusual.

We have the poetry export for that, don't we?

you think that avoiding extra dependencies is valuable.

I don't think that. And I mostly support the integration of smaller dependencies that don't have many maintainers, even if there is a greater risk that they have to be migrated to another dependency at some time in the future. This instance isn't a simple dependency though, it's more about complexity.

We can agree to disagree. Unless and until anyone submits a merge request, no-one has to make a decision either way.

Of course, I only suggested leaving this issue open because it's not really "supported" by poetry. Unless there's an agreement that this falls out of scope. I'm not even a contributor, only a user who cares. Cheers!

[tschm]

tschm here. I am one of the authors of https://github.com/tschm/token-mint-action. This action is using poetry as an example but it also has a place in a world without poetry. So, I don't think it should be integrated into poetry. I would like to point you to pypi/warehouse#13555... I think folks there are not happy with poetry publish in the first place.

[edgarrmondragon]

fwiw it's no longer GitHub-specific since trusted publishers support other providers now https://blog.pypi.org/posts/2024-04-17-expanding-trusted-publisher-support/

[dimbleby]

that is true: but it does not mean that writing code in poetry to mint tokens in a github pipeline would help the other cases. Eg the exact means for getting a token from a gitlab pipeline is different than the mechanism for getting a token from a github pipeline, and so on.

to my mind this speaks against doing anything in poetry: all the more reason to leave token-minting to CI pipeline actions written for the environment in which they run.

[edgarrmondragon]

Yeah, just wanted to comment on the update. For GitHub in particular I'd still recommend using hynek/build-and-inspect-python-package and pypa/gh-action-pypi-publish for publishing to PyPI.

[johannesjh]

twine >= 6.1.0 supports publishing to PyPI using the "trusted publisher" openid-connect authentication mechanism, see the corresponding issue pypa/twine#999 and merge request pypa/twine#1194