[BUG] v70.0.0 removes 'packaging' from pkg_resources

[dmick]

setuptools version

setuptools=70.0.0

Python version

3.8

OS

Ubuntu 22.04

Additional environment information

first noticed on readthedocs.org builds today

Description

from pkg_resources import packaging

fails with

ImportError: cannot import name 'packaging' from 'pkg_resources'

Expected behavior

expected the import to succeed as in v69.5.1 and prior

How to Reproduce

python3 -mvenv v
v/bin/pip install pkg_resources
v/bin/python3

from pkg_resources import packaging

Output

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: cannot import name 'packaging' from 'pkg_resources' (/home/dmick/vsetup/lib64/python3.9/site-packages/pkg_resources/__init__.py)

[jeffhataws]

This is also breaking PyTorch 2.1 (not PyTorch 2.2+). Will you consider adding it back?

(aws_neuron_venv_pytorch) ubuntu@ip-10-0-8-190:~$ python -c "from torch.utils import cpp_extension"
Traceback (most recent call last):
  File "<string>", line 1, in <module>
  File "/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.8/site-packages/torch/utils/cpp_extension.py", line 28, in <module>
    from pkg_resources import packaging  # type: ignore[attr-defined]
ImportError: cannot import name 'packaging' from 'pkg_resources' (/home/ubuntu/aws_neuron_venv_pytorch/lib/python3.8/site-packages/pkg_resources/__init__.py)

[abravalheri]

Hi @dmick , please see the explanation for the behaviour you are observing in #4376 (comment).

Tl;DR: the package you are using was relying on an internal implementation detail of pkg_resources and that implementation detail was not part of the public API.

I recommend contacting the maintainers of that package and reporting the bug on their end. The easiest way to fix this is to install and import packaging directly.

The API documentation for pkg_resources can be found in https://setuptools.pypa.io/en/latest/pkg_resources.html.

You can also check which "names" are public in the package by inspecting the __all__ property.

[rsxdalv]

Although the argument is understandable, from pkg_resources import packaging does not seem like an internal API, if it was named _packaging or otherwise it could have been easier to spot. The fact that even PyTorch missed out on this and there are countless broken projects and build pipelines will probably mean a that v70+ ends up underused.

[abravalheri]

@rsxdalv , that is the side effect of imports, and that is why __all__ exists. This is a well-known and documented characteristic of the Python language.

It is unfortunate that so many people missed it. But all the public names in pkg_resources have been documented on the website for years.

If people desire to use packaging, they can include it as a dependency, as it is available on the PyPI.

[rsxdalv]

I agree, once I realized about python imports causing a side-effect where they are automatically exported I was not very happy.
Though, outside the scope of this issue, I would like to ask you this - since this is a potentially widespread issue, would it make sense for new library packages to have an init.py which only imports the things that are to be exported or uses aliases to prevent leaks, such as import packaging as _packaging?

Alternatively, should projects have a deep-lint to check if somebody-somewhere is importing outside of all declaration?

Meanwhile, javascript has some extremely strict linting because every bad pattern (such as importing what you are not supposed to) that can be done will be done.

[abravalheri]

@rsxdalv I believe that every Python developer will have a different answer for that, as there are many styles and workflows for package development 😅¹. I think the best is always to check for the docs of a particular project that you want to use.

You may be able to find some linters that check against "non-exported names". Maybe it is worth asking around if the ruff community has something for that? (update, it is probably under discussion: astral-sh/ruff#167).

Footnotes

1. For example, some projects use __all__, some projects use a specific .api submodule, some projects use _impl subpackage + __init__.py... there are many variations in the ecosystem, it would be difficult to standardize one. ↩