Requested (package) has different version in metadata

[potiuk]

Describe the bug

In Airflow, we are experiencing problems when we are trying to use the PIP released 2 days ago which has the new resolver on by default.

ERROR: Requested oauthlib[signedtoken]>=1.0.0 from https://files.pythonhosted.org/packages/e5/54/4f96c51b171cf3a64a04b8c5167268803205bc5943b5cdf70bd770727b88/oauthlib-1.1.0-1.tar.gz#sha256=0f786c5573248a38efa86c48c59c0c93140ac836ab2a246aeefd8f9039e999ba (from jira->apache-airflow==1.10.13) has different version in metadata: '1.1.0'

How to reproduce

• Setup an empty virtualenv for Python 3.6
• Upgrade to latest pip: pip install --upgrade pip
• pip --version should return 20.3
• Run this command:

pip install "https://github.com/apache/airflow/archive/v1-10-test.tar.gz#egg=apache-airflow[all]" --constraint https://raw.githubusercontent.com/apache/airflow/constraints-1-10/constraints-3.6.txt

• Observe the output. It will keep on finding good dependencies until

Collecting oauthlib[signedtoken]>=1.0.0
  Using cached oauthlib-3.0.2-py2.py3-none-any.whl (143 kB)
  Using cached oauthlib-3.0.1-py2.py3-none-any.whl (142 kB)
  Using cached oauthlib-3.0.0-py2.py3-none-any.whl (142 kB)
  Using cached oauthlib-2.1.0-py2.py3-none-any.whl (121 kB)
  Using cached oauthlib-2.0.7-py2.py3-none-any.whl (124 kB)
  Using cached oauthlib-2.0.6.tar.gz (127 kB)
  Using cached oauthlib-2.0.5.tar.gz (129 kB)
  Using cached oauthlib-2.0.4.tar.gz (127 kB)
  Using cached oauthlib-2.0.3.tar.gz (127 kB)
  Using cached oauthlib-2.0.2.tar.gz (125 kB)
  Using cached oauthlib-2.0.1.tar.gz (122 kB)
  Using cached oauthlib-2.0.0.tar.gz (122 kB)
  Using cached oauthlib-1.1.2.tar.gz (111 kB)
  Using cached oauthlib-1.1.1.tar.gz (108 kB)
  Using cached oauthlib-1.1.0-1.tar.gz (106 kB)
ERROR: Requested oauthlib[signedtoken]>=1.0.0 from https://files.pythonhosted.org/packages/e5/54/4f96c51b171cf3a64a04b8c5167268803205bc5943b5cdf70bd770727b88/oauthlib-1.1.0-1.tar.gz#sha256=0f786c5573248a38efa86c48c59c0c93140ac836ab2a246aeefd8f9039e999ba (from jira->apache-airflow[all]) has different version in metadata: '1.1.0'

Apparently, metadata in published oauthlib 1.1.0-1 is wrong snd points to 1.1.0.

The same command with the legacy resolver works fine:

pip install --use-deprecated legacy-resolver "https://github.com/apache/airflow/archive/v1-10-test.tar.gz#egg=apache-airflow[all]" --constraint https://raw.githubusercontent.com/apache/airflow/constraints-1-10/constraints-3.6.txt

Expected behavior

I expect the resolver does not get broken by broken metadata.

I've opened similar issue oauthlib as I am not sure who can fix it: oauthlib/oauthlib#744

[brainwane]

Hi, @potiuk and thank you for your bug report! I'm sorry you're having trouble right now. Thank you for sharing your report with us. (If you don't mind, please also tell us what could have happened differently so you could have tested and caught and reported this during the resolver beta period.)

As you noted: you need a workaround while the root cause of this problem is being fixed, so you can choose the old resolver behavior using the flag --use-deprecated=legacy-resolver. This will work until we release pip 21.0 (see Deprecation timeline).

I expect the resolver does not get broken by broken metadata.

I understand that this was the previous behavior, but pip's maintainers have decided to make pip stricter and more consistent in dealing with incompatible, inconsistent, and broken combinations of metadata. We are doing this to pave the way for a lot of deeply-needed bugfixes and features, such as warning the user when uninstalling a package that other packages depend on, adding an "upgrade-all" command to pip, properly respecting constraints, and more.

I'll defer to others on diagnosing precisely what needs to be fixed in order to resolve this. Thanks again.

[notatallshaw]

I feel like this is still a bug with the new pip resolver. Reproducing it myself I get the same error (the previous line included):

INFO: pip is looking at multiple versions of oauthlib[signedtoken] to determine which version is compatible with other requirements. This could take a while.
Collecting oauthlib[signedtoken]>=1.0.0
  Downloading oauthlib-3.0.2-py2.py3-none-any.whl (143 kB)
  Downloading oauthlib-3.0.1-py2.py3-none-any.whl (142 kB)
  Downloading oauthlib-2.1.0-py2.py3-none-any.whl (121 kB)
  Downloading oauthlib-2.0.7-py2.py3-none-any.whl (124 kB)
  Downloading oauthlib-2.0.6.tar.gz (127 kB)
  Downloading oauthlib-2.0.5.tar.gz (129 kB)
  Downloading oauthlib-2.0.4.tar.gz (127 kB)
  Downloading oauthlib-2.0.3.tar.gz (127 kB)
  Downloading oauthlib-2.0.2.tar.gz (125 kB)
  Downloading oauthlib-2.0.1.tar.gz (122 kB)
  Downloading oauthlib-2.0.0.tar.gz (122 kB)
  Downloading oauthlib-1.1.2.tar.gz (111 kB)
  Downloading oauthlib-1.1.1.tar.gz (108 kB)
  Downloading oauthlib-1.1.0-1.tar.gz (106 kB)
ERROR: Requested oauthlib[signedtoken]>=1.0.0 from https://files.pythonhosted.org/packages/e5/54/4f96c51b171cf3a64a04b8c5167268803205bc5943b5cdf70bd770727b88/oauthlib-1.1.0-1.tar.gz#sha256=0f786c5573248a38efa86c48c59c0c93140ac836ab2a246aeefd8f9039e999ba (from jira->apache-airflow[all]) has different version in metadata: '1.1.0'

When searching for a version of a package that is compatible with other requirements Pip should not completely error out just because 1 version of that package had broken metadata. Pip should instead discard any version of the package that has broken metadata in the dependency search.

If no version with valid metadata can be found that's what Pip should be reporting, not that a very old version of the package had broken metadata.

[uranusjr]

Pip should not completely error out just because 1 version of that package had broken metadata. Pip should instead discard any version of the package that has broken metadata in the dependency search.

We complete agree.

pip/src/pip/_internal/resolution/resolvelib/candidates.py

Lines 204 to 214 in 30eeb9c

	def _check_metadata_consistency(self, dist):
	# type: (Distribution) -> None
	"""Check for consistency of project name and version of dist."""
	# TODO: (Longer term) Rather than abort, reject this candidate
	# and backtrack. This would need resolvelib support.
	name = canonicalize_name(dist.project_name)
	if self._name is not None and self._name != name:
	raise MetadataInconsistent(self._ireq, "name", dist.project_name)
	version = dist.parsed_version
	if self._version is not None and self._version != version:
	raise MetadataInconsistent(self._ireq, "version", dist.version)

There are, however, still may issues in pip that needed to be worked on, and this particular one is relatively low-impact and have alternative solutions. I understand it may feel urgent when it directly affects you, but there are really only a very small number of packages that have this issue (I believe this is the only one we know served on PyPI so far), and the package maintainer can resolve the issue by yanking the problematic release. The issue is therefore currently considered less urgent.

[potiuk]

Hi, @potiuk and thank you for your bug report! I'm sorry you're having trouble right now. Thank you for sharing your report with us. (If you don't mind, please also tell us what could have happened differently so you could have tested and caught and reported this during the resolver beta period.)

Very good question indeed. The problem in our case that for a long time we had conflicting requirements in Airflow (we have > 420 requirements in total, because Apache Airflow is an orchestrator and we have more than 70 different providers (GCP/AWS/Postgres etc.) - each with its own set of requirements. We are releasing Airflow 2.0 (Release candidate planned for next week) and we've worked hard to get the 2.0 out, and only last week I started to look at our deps to make them conflict-free. I managed to do it (and now we have automated system that keeps the dependencies automatically updated but also pip-checked - those depencies are going to be upgraded every time after they pass all tests and pip-check in our master build. We are using automatically generated set of "good" constraints : https://github.com/apache/airflow/tree/constraints-master - if you look at the history of those commits, you will see that they get automatically updated whenever pip install .[all] --upgrade --upgrade-strategy eager will produce a set fo constratins that:

• all tests pass with them
• the constraints pass the 'pip check` (this works as of last week - previously we had conflicts in those constraints)

We could not test the new resolver before we fixed the constraints, because we had those conflicting constraints. With those, the new resolver went into an endless loop :(. Which is of course totally understandable. Since I just fixed the constraints, that was really the first time we could run the new resolver. And it is quite a coincidence of schedule with the PIP 20.3 release and Airflow 2.0 release as we've already in the past also had some limitation w/regards of PIP for a while and we had to pin it iand update our documentation.

As you noted: you need a workaround while the root cause of this problem is being fixed, so you can choose the old resolver behavior using the flag --use-deprecated=legacy-resolver. This will work until we release pip 21.0 (see Deprecation timeline).

Thanks!. This is what we've done - we updated the documentation and told our users to use this for now (or downgrade to the previous version of PIP). In our official docker images, we pinned PIP to 20.2.4 for now. I would love to use the new resolver thought and wondered if there is a way to disable the strict check :). But i understand from the discussion that it is a design decision for now. I understand that there are more pressing issues, so we can go with that constraint for a while - not ideal and caused us a spike of issues and questions from our users already, so we would prefer if this is fixed of course, also maybe the oauthlib team will fix it (for example by removing the offending version?).

[potiuk]

Another input:

I looked how we can workaround it and I thought we can do it by limiting the oauthlib directly, but it seems we can't:

We already have this in our setup.py oauthlib!=2.0.3,!=2.0.4,!=2.0.5,<3.0.0,>=1.1.2 . It's a bit strange that in this resolver downloads "all versions" of the lib even if we have this limitation (>=1.1.2):

  Downloading oauthlib-2.0.1.tar.gz (122 kB)
  Downloading oauthlib-2.0.0.tar.gz (122 kB)
  Downloading oauthlib-1.1.2.tar.gz (111 kB)
  Downloading oauthlib-1.1.1.tar.gz (108 kB)
  Downloading oauthlib-1.1.0-1.tar.gz (106 kB)
ERROR: Requested oauthlib[signedtoken]>=1.0.0 from https://files.pythonhosted.org/packages/e5/54/4f96c51b171cf3a64a04b8c5167268803205bc5943b5cdf70bd770727b88/oauthlib-1.1.0-1.tar.gz#sha256=0f786c5573248a38efa86c48c59c0c93140ac836ab2a246aeefd8f9039e999ba (from jira->apache-airflow[all]) has different version in metadata: '1.1.0'

[potiuk]

Another question. Would removing or yanking the 1.1.0-1 of oauthlib version help ? I proposed this in oauthlib/oauthlib#744

[uranusjr]

Yes, I’d say yanking is the best solution for now.

[ashb]

Does this error (where a single bad dist causes the resolver to stop) not leave the python ecosystem open to a "left-pad attack" -- a disgruntled package author of something used by lots of transitive deps uploads a purposefully broken dist, 💥 half of pypi becomes uninstallable.

[uranusjr]

With any luck, we’ll be in time to release a solution before it happens. But even if not, I believe the PyPI terms of use give the admins power to remove things uploaded in bad faith. This is slightly different from the left-pad problem since removal is within a user’s rights and the admin cannot restore it without consent. (Disclaimer: IANAL and do not involve in maintaining PyPI.)