PLIP: Switch to PEP 420 (Implicit namespace packages)

[mauritsvanrees]

Responsible Persons

Proposer: Maurits van Rees

Seconder:

Abstract

Move from setuptools/pkg_resources namespaces to native namespaces.
This is a breaking change, so is for Plone 7.

Motivation

Use of pkg_resources from setuptools is deprecated. Native namespaces have been available since Python 3.3 and are now the recommended and most easy way to define namespaces.

Assumptions

• Zope wants to make the same move.
  Since it is best to move all packages within one namespace at the same time, we need to coordinate with Zope for the Products namespace.
• Support for non-native namespaces is slowly being removed from pip and setuptools and other packaging related software. Bugs may be less likely to get a fix.
• We want to be less dependent on setuptools. There are other ways to handle packaging and distribution nowadays.
• Theoretically you can combine native namespaces and pkg_resources namespaces within the same namespace, but this depends on the tools that you use, and whether you make a development checkout of a package. A combination that works today, might fail tomorrow with a newer setuptools or pip version. It is best to move over all packages from a single namespace at the same time.
• Note that there should be no problem combining different namespace variants in different namespaces: if all zope.* packages have moved to non-native namespaces, all plone.* packages could still use the old pkg_resources-style namespaces.

Proposal & Implementation

• In Plone 7.0, all repositories in the GitHub plone organisation that are used by the Plone package should move to native namespaces. Others too where possible, and where we have control.
• In the upgrade guide, give add-on authors the advice to do the same. This goes especially for the collective namespace.
• The main changes per package:
  • Make a maintenance branch for this package on Plone 6. We want to leave room for new features in Plone 6, so the branch should be something like 3.x, not 3.0.x.
  • Remove __init__.py from the namespace packages. In Products.CMFPlone we would remove Products/__init__.py.
  • Make sure all packages (directories) in the package are found, and are included when creating a distribution. This includes for example the browser, profiles, and tests directories. There are multiple ways to do this, but I propose we use a src-layout as this is supposed to make everything automatic. For example move Products to src/Products. See discussion here. This is optional, but seems best. It avoids problems like this when moving zest.releaser over to using native namespaces. It will make some QA code easier, for example in plone/meta, because all code will be in src, instead of in any of {src, plone, Products, etc}.
  • It is fine to still have a setup.py: this PLIP is not about getting rid of setuptools. We must then declare setuptools as build backend in pyproject.toml, which we are already starting to do in Plone 6.
  • Remove any Plone 6 PyPI classifiers and add the Plone 7.0 classifier (once it exists).
  • Increase the major version of the package, because this is a breaking change.
  • Make a quick alpha release on PyPI with only these changes (and any other breaking changes that may have already been added).
  • Compare the number of tests before and after: there are various ways for tests to get lost (tests directory not included in the distribution), or not be discovered (the tests directory is there, but the test runner can't find it).
• We would have an early Plone 7.0.0a1 that is not much different from the latest Plone 6, except for the namespace implementation.
• The order does not matter too much, but this would be one way:
  • First get a Zope 6 alpha release with all those packages moved to native namespaces. This includes the Products namespace. The pure Zope community can use help from the wider Plone community in this.
  • Do the Plone part of the Products namespace.
  • plone.*, including plone.app.*.
  • Others, like collective, plonetheme, borg, zc, z3c, repoze, five.
  • We could also start with small namespaces. We have two namespaces that only have one package in core Plone: plonetheme.barceloneta and borg.localrole.
• If we do this during a sprint, we may need to temporarily disable Jenkins for Plone 7, to avoid complete overload.

Deliverables

• For each package, the main/master branch needs to be for Plone 7 only. Each package needs an alpha release on PyPI.
• A full Plone 7.0.0a1 release.
• Documentation:
  • Document what/why/how we did this.
  • Include links to several pull requests as example.
  • Advise add-on authors to do the same for their packages, especially in the collective namespace.

Risks

• Core Plone development will likely be broken for a while, until an entire namespace has been taken care of.
• We may temporarily lose some tests in a package if we are not careful. We may only notice this later, and then find they have been broken in the meantime.
• Some tooling may need updates. zope.testrunner was already updated, and z3c.dependencychecker as well.

Participants

• Maurits van Rees, Release Manager. I will need to do a lot of releases. :-) I can do some packages too, but it would be good if others did most of them. A group of people can do a lot in a day of sprinting or on a Plone TuneUp day.

Links:

• Similar Zope issue: Switch to PEP 420 (Implicit namespace packages) zopefoundation/meta#194
• The Python Packaging Guide:
  • packaging namespace packages
  • src-layout versus flat-layout
  • Is setup.py deprecated? Short answer: no.
• Setuptools:
  • pkg_resources
  • Package discovery and namespace packages
• The PyPA sample namespace packages repo has sample namespace packages for each of the three possible namespace variants. The readme has remarks on staggered migrations, including a table. Relevant to us in the table are the lines with cross_pep420_pkg_resources (or cross_pkg_resources_pep420 but that is basically duplicate data). The takeaway from that table, is that currently you can install a pep420 (native) and a pkg_resources package within the same namespace under some conditions:
  • both installed normally: works
  • both installed in editable mode: works (this surprises me a bit)
  • native installed in editable mode, pkg_resources normal: works
  • native installed normally, pkg_resources in editable mode: does NOT work.

[gforcada]

With some scripts and the GitHub CLI tool it might be easy to distribute the effort... we can create preparatory branches to switch, so the window of brokenness will not be that big...

One last item that might be a good idea to also add in this massive breakage would be to also rename master to main as default branches 🤞🏾

I will have some days off next week (and without kids!) so I will try to work on it.

I already ported our internal packages at work to native namespaces, and before it bitrots I can re-apply the steps there 👍🏾

[mauritsvanrees]

@gforcada Trying things out, preparing, making tooling, that would be great. But for clarity: please do not yet make the actual move to native namespaces on any package. That is really for Plone 7, so next year.

I guess for a small namespace like borg it would be fine in a Plone 6 minor version already: only borg.localrole exists in this namespace as far as I know. I wanted to add plonetheme there, but I see lots of plonetheme packages on PyPI, so that would be for Plone 7 as well.

I already had a problem after moving zest.releaser to native namespaces, because there is also a zest.pocompile package that I usually install in the same venv or buildout, and this no longer worked until I moved that one over as well.

Indeed we could create a native branch on each package, do the changes there, and on coredev have a PLIP config and on Jenkins a PLIP job to checkouts those branches, before doing anything on a coredev 7.0 branch.

Going from master to main on all Plone repos could be good, but I would make that a separate PLIP. It is something that can be tackled one repo at a time. But: if this makes tooling for the current PLIP easier, then sure, let's combine the work.

[gforcada]

Sure, I was not planning on merging anything 😄 but as you say, tooling needed some love as both zope.testrunner and z3c.dependencychecker were not capable of working with native namespaced distributions.

Fortunately that is fixed now, but there might be other tooling (zc.buildout ?) that might need some help as well, or a transition plan.

The idea of creating a main branch with only those changes (src layout and native namespaces) sounds good, specially if we tie it with periodic (monthly?) rebases and some CI (Jenkins) jobs that ensure things keep working.

Otherwise, if we wait until we want to open the door for Plone 7 changes, and things are not ready, nor tried already (on CI), we might face a stressful point where we stall development while waiting for tooling to get fixed.

See this funny issue in tox: tox-dev/tox#3247 I'm a bit surprised that I was the first one to notice and report it actually, given how widespread is tox and collective.* packages 😅

Anyway: there is a draft PR for borg.localrole and I have a 7.0 branch on buildout.coredev still to be pushed, that just updates borg.localrole branch from master to main 💥 I will push it in the next few days.

Then we will need a Jenkins job to ensure it does work 🤖

[davisagli]

@mauritsvanrees @gforcada In order to avoid the need for new major releases of all packages at the same time, could we do something like this to switch between namespace package implementations based on the Python major version?

(in __init__.py)

import sys
if sys.version_info < (13,):
    __import__("pkg_resources").declare_namespace(__name__)

Ah, no, of course we cannot, because native namespace packages require NOT having __init__.py at all.

[yurj]

https://peps.python.org/pep-0420/#migrating-from-legacy-namespace-packages

[mauritsvanrees]

https://peps.python.org/pep-0420/#migrating-from-legacy-namespace-packages

That is for pkgutil namespaces, which is yet another namespace implementation. This is not used in core Zope and Plone.

[gforcada]

Not sure how much is worth it, but an idea came across:

• create branches for all distributions of a given namespace with only the PEP 420 changes
• make alpha releases of them all
• this allows CI and tooling to be tested with real working code

If we keep this releases parallel to the production ones, we don't have to wait until Plone 7 is open, for then to notice that, oops X and Y tooling/packages/dependencies are broken.

I still need to add the jenkins job to run tests with the borg.localrole PR I made some weeks ago... I'm a bit overwhelmed right now 😵‍💫 😓 maybe this weekend I find some time 🍀

[mauritsvanrees]

Not sure how much is worth it, but an idea came across:

• create branches for all distributions of a given namespace with only the PEP 420 changes
• make alpha releases of them all

The problem with doing this now, is that this gets in the way of other changes that require a major version bump and that we want to include in Plone 6.x.

Take for example plone.app.event:

• Plone 6.0 and 6.1 currently use the master branch and have version 5.1.2.
• You create branch pep-420, do the changes, bump the version, and release 6.0.0a1. (Or you create branch 4.x for use in Plone 6.0 and 6.1, and bump the version on master.)
• Now someone wants to make a major change that would be breaking for 6.0, but that is fine for 6.1 or maybe 6.2. In the case of this package there are plans to use the zoneinfo module from Python 3.9+ instead of pytz. So they want to bump the version to 6.0.0a1 as well, but that is already taken by the pep-420 branch.

If we imagine it really happens this way, it can still be fixed, but it will be confusing:

• You could release 6.0.0a2 with the zoneinfo changes, and then rerelease 6.0.0a1 as 7.0.0a1 with the pep-420 changes. Then within 6.0.0a* you would have one release with native namespaces and another with pkg_resources namespaces.
• Or you abandon 6.0.0.*, release 7.0.0a1 with the zoneinfo changes, and then 8.0.0a1 with the pep-420 changes.

So: I think best not do this.

Crazy idea of my own:

• Create a script that does all the changes for a package.
• Keep those changes locally, do not push them. Well, a pep-420 branch could be pushed I guess, so others can cooperate if further changes are needed.
• Make releases but do not push them to PyPI.
• Put the releases only on dist.plone.org instead, in a folder like /release/pep-420-test-do-not-use-in-production. ;-) Then you can use that url as find-link.
• You could give all packages the version number 0.1a1 to avoid problems when they accidentally get pushed to PyPI, or they end up in your local pip or buildout cache. Actually, no: several will have dependencies with a minimum version specified. You could strip away those minimum (or maximum) versions from setup.py in the script though.

[mauritsvanrees]

Sample PR of moving to a src-layout and the changes needed then: plone/plone.app.event#384
I close that one because it is old and outdated, but as example it works.

[mauritsvanrees]

• When should we move to native namespaces? Until now I was very much inclined to call this a breaking change and wait until Plone 7. But now I want it sooner. It is unknown how long setuptools will keep supporting its own pkg_resources style namespaces. Plone should aim to have its latest release installable and editable/developable on latest pip and setuptools. If we postpone until Plone 7, we may be too late. Plone 6.1 took us two years. Plone 7 may take another two years. I don't want this sword of Damocles hanging above our head.
• So I propose we do this in Plone 6.2. We can start on this during the Alpine City Sprint.
• Of course if for example setuptools 80 drops support, we can pin setuptools to 79 in our constraints, but this is not ideal.
• Moving to native namespaces is not strictly speaking a breaking change. You can install both types of namespaces together, even for the same namespace. But an editable install of both types of namespaces reportedly does not work.

[mauritsvanrees]

@davisagli @tisto If we do this in 6.2, it may be better to rename the current plone.restapi main branch to 9.x.x and let this still keep compatibility with Plone 5.2, 6.0 and 6.1. The main branch will then be version 10 and only support Plone 6.2+.
Does that sound fair?

I want 6.2 fairly soon. The Beethovensprint seems too ambitious, but maybe summer.

[stevepiercy]

It is unknown how long setuptools will keep supporting its own pkg_resources style namespaces.

See pypa/setuptools#3934

There is an issue comment in the setuptools repo: pypa/setuptools#3780 (comment)

Document a recommendation that users run with (setuptools/pkg_resources) deprecation warnings set to error in CI systems.

Do we do that now? If not, I think that would give the PLIP more visibility and urgency.

[davisagli]

@mauritsvanrees +1 for trying to move forward with this sooner rather than later.

You can install both types of namespaces together, even for the same namespace. But an editable install of both types of namespaces reportedly does not work.

I guess this means that we should make sure Plone 6.1 only uses branches that still have setuptools namespaces, and 6.2 only uses branches that have native namespaces? Otherwise it gets hard to test changes to multiple packages together.

If we do this in 6.2, it may be better to rename the current plone.restapi main branch to 9.x.x and let this still keep compatibility with Plone 5.2, 6.0 and 6.1. The main branch will then be version 10 and only support Plone 6.2+.
Does that sound fair?

I was planning to work soon on a new major version of plone.restapi anyway, to drop support for Plone 5.

Document a recommendation that users run with (setuptools/pkg_resources) deprecation warnings set to error in CI systems.

Do we do that now? If not, I think that would give the PLIP more visibility and urgency.

I don't understand the suggestion; it sounds like it would break CI entirely which is probably more urgency than we want before we have a solution ;-)