Release process is cumbersome

[jaraco]

In #493, I set out to make the release process as lightweight as possible while keeping a reasonable level of reliability and auditability.

Today, I went to cut a quick release based on a PR, but ran into several steps that made it difficult.

First, the release process demands a PR, which implies:

• check out the code
• create a branch
• prepare the release
• push new branch
• create pr
• switch to browser
• wait minutes for CI to finish
  • fix noisy test failures (in code that has previously passed review)
• squash merge the changes
• delete the upstream branch
• switch back to cli
• switch back to master
• pull the changes
• locate and delete the now orphaned commit (due to squashed merge)
• tag the release and push the tag

Previously, this approach could be shortened to:

• check out the code
• prepare the release
• tag the release and push the tag

I tried expediting the process by merging locally and pushing that commit to master, but master is locked, so a PR is required.

This additional process multiplies the cost of cutting a release by several orders of magnitude and makes releasing something that requires a larger investment than necessary and will limit the frequency of releases.

I'd much rather see a streamlined process again.

[bhrutledge]

That does seem like a long list. Related:

• Prepare for 3.3 release by jaraco · Pull Request #682 · pypa/twine
• Fix/mitigate flaky integration tests · Issue #684 · pypa/twine
• Replace Travis with GitHub Actions · Issue #650 · pypa/twine
  • Suggests an action for releasing
• Adopt a new format for the changelog · Issue #546 · pypa/twine
  • Mentions the impact on the release process

[bhrutledge]

Having just released 3.3.0 using my local command line (as instructed by https://twine.readthedocs.io/en/latest/contributing.html#making-a-new-release), I'm curious about moving more/all of that to GitHub, via Releases and Actions.

For example, I was tempted to use https://github.com/pypa/twine/releases/new to give a nice, succinct changelog, ala https://github.com/pypa/bandersnatch/releases, https://github.com/pypa/packaging/releases, and https://github.com/pytest-dev/pytest/releases. However, I think that would require transforming the reStructuredText changelog into Markdown.

Any suggestions, or good examples of this (esp. given our adoption of towncrier)?

[sigmavirus24]

@bhrutledge GitHub releases really are just a lot of extra work for not a lot of benefit from everything I've been able to tell. I suspect we could do something with annotated tag messages git tag -am to get release information in there. (Releases can be autogenerated from an annotated (a.k.a., not light-weight) tag.) I don't think it needs to be markdown because we genuinely don't need to be making the process more cumbersome.

Genuinely though, I think shooting for GitHub releases is trying to solve the wrong problem. Also, only one of the examples actually uploads the build artifacts from PyPI (packaging) and that should probably be automated rather than done manually. Also, what's the value in putting release assets on GitHub and dissuading people from using PyPI? Why? It's not as if folks are also uploading the signature files there (.asc) to make them more discoverable, so there's little value in having wheels there too.

[bhrutledge]

@sigmavirus24 I think the overloading of the term "releases" might be causing some confusion. To clarify, I'm not advocating for replacing PyPI as the distribution mechanism. In fact, I'm not advocating strongly for anything, just bike-shedding alternatives, motivated by two things:

1. Removing reliance on local command line execution to upload a release to PyPI
2. Improving the experience for folks browsing Twine's repo

Re: motivation 1, creating a GitHub Release (starting at https://github.com/pypa/twine/releases/new) creates a tag, which I think would kick off the releases action, so that a maintainer could publish to PyPI entirely from the GitHub UI. FWIW, GitHub Releases can also be created using the GitHub CLI.

Re: motivation 2, I have always appreciated projects that have robust GitHub Release notes, similar to the repositories that I listed above. I often look there (in addition to hunting for a changelog), because it's quick, and if they exist, they're often nicely-formatted. I don't care about the assets.

Again, for clarity, I'm not advocating for replacing Twine's current changelog. Rather, I'm pondering duplicating its contents in a common location to improve discoverability. But, that is a non-trivial extra step, and care has to be taken to keep GitHub Releases in sync with the official PyPI release.

Also, we're already using annotated tags, because -m implies -a, so git tag -m "Release v3.3.0" 3.3.0 results in https://github.com/pypa/twine/releases/tag/3.3.0.

[sigmavirus24]

So using GitHub Actions to perform a release to PyPI is absolutely one thing. I'm all for automation of that. That's orthogonal to either building extra automation to put release notes in a 3rd (4th? 5th?) place.

I'm strongly against release notes in GitHub Releases because the Release notes can be amended and updated. Who makes certain they're up-to-date there? GitHub Releases don't show you the entirety of release notes for a release so you can't view all the notes for all versions at once, you have to open several pages to get to that. Any decent tooling (including GitHub's own dependabot) knows how to find change-logs and the better automation tools know how to embed that information in the pull request (without it needing to be attached to a GitHub release).

On the whole, putting the notes there doesn't seem to be a standard in the PyPA (as nothing seems to really be standardized) and doesn't buy us anything other than extra maintenance, plus inconsistency. The old versions don't have those notes. And if someone other than you performs the release and forgets to copy them there, people will open issues to "remind" us to put them there.

I often look there (in addition to hunting for a changelog), because it's quick, and if they exist, they're often nicely-formatted. I don't care about the assets.

To be clear, if I'm landing on a project's repository, it's 1 click to find the CHANGES/HISTORY/CHANGELOG file in the root rather than several to look at a single release via GitHub releases. Most projects, including this one in particular, use a mark up language which GitHub renders 99% correctly (and is often rendered 100% correctly on whatever documentation site the project uses).

Finally, creating a release creates implied assets that GitHub generates which you cannot disable last time I checked. So then you have 4 different 'canonical' assets. The 2 uploaded to PyPI (what you and I would rightly consider the real assets). The 2 GitHub creates which are now implicitly "official"/"canonical" as they're on the official project page (the hosted GitHub repository). This has generated confusion on other projects I'm part of and people then wonder why we're distributing assets that have the whole repository in them and why we don't distribute binaries.

But, that is a non-trivial extra step, and care has to be taken to keep GitHub Releases in sync with the official PyPI release.

And we'd need to start parsing the changes file to detect updates/amendments to the release section that corresponds with a given GitHub release and update that too. Otherwise we will get issues about "These notes say X, Y, and Z but these other notes also mention T, U, and V. What's the right thing and why haven't you updated the GitHub release?"

Also, we're already using annotated tags

Right, I wrote that because annotated tags create "releases" on GitHub just without the changelog embedded. I'm wondering if we can simplify things and shove the changelog in that without messing around with the GitHub CLI or Releases (it still suffers the problem of not having a completely up-to-date set of notes if something is amended after the fact).

[bhrutledge]

Those are excellent points; happy to set GitHub Releases aside. Thanks for indulging me.

[bhrutledge]

On a whim, I created a lightweight release for 3.4.2 at https://github.com/pypa/twine/releases, which simply links to the PyPI page and the changelog. This feels like minimal overhead to make releases more discoverable, and could be automated via a GitHub Action.

If there's appetite for this, I'd probably want to:

• Show "Releases" in the sidebar ala https://github.com/pypa/virtualenv
• Update the releasing guide to trigger publishing to PyPI via a GitHub Release instead of a local git tag and git push
  • This could probably also be a GitHub Action, that automates the Release description
• Create a manually-executed GitHub Action for running tox -e changelog and opening a PR
• Update the releasing guide to use that Action instead of a local command

Again, mostly just bike-shedding here, based on my experience searching and browsing projects on GitHub, and because I seem to enjoy this kind of thing.

[bhrutledge]

Note for future releasers: currently, the release job depends on the tests in the same workflow:

twine/.github/workflows/main.yml

Lines 72 to 76 in d82cd00

	release:
	needs:
	- lint
	- test
	- docs

but not the integration (aka end-to-end) tests:

twine/.github/workflows/integration.yml

Lines 17 to 18 in d82cd00

	jobs:
	test:

This is because the integration tests have been flaky; see #684. It would be nice if we could confidently say "failing integration tests should block a release" and move those tests back into the main workflow, to be part of the release pipeline. Until then, it's up to releasers to wait for the result before tagging the release, as noted in the release instructions:

Merge the pull request, and ensure the GitHub Actions build passes