linux wheel "link matching" behavior changes when using pip 25.1

[cburroughs]

Pex includes different artifacts when using pip 25.1 than pip 25.0. This is show by the differences in Found vs Skipping links in the examples below.

(I presume to how differences in pip 25.1 colliding with the custom patching, so for anyone else following along the log lines are "from pip", but the pip._internal.models.wheel.Wheel.supported behavior is from pex.)

$ pex3 --version
2.36.1
$ sha256sum ~/bin/pex
1a6c6f39d988a5478bbc1d772a27672984b8d304f03073565c2cfdc00598cc39  /home/ecsb/bin/pex

(So that's the pex-linux-x86_64 release and this is running on x86_64 Linux)

$ pex3 lock create --output=lock-25.0.json --indent=2 --style=universal --pip-version 25.0 --preserve-pip-download-log pex-pip-download-25.0.log --target-system linux --target-system mac --manylinux manylinux2014 --interpreter-constraint 'CPython==3.11.*' psutil==5.9.8
$ grep psutil-5.9.8-cp36-abi3-manylinux_2_12  pex-pip-download-25.0.log 
2025-05-05T16:47:43,788   Found link https://files.pythonhosted.org/packages/b3/bd/28c5f553667116b2598b9cc55908ec435cb7f77a34f2bff3e3ca765b0f78/psutil-5.9.8-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl (from https://pypi.org/simple/psutil/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*), version: 5.9.8
2025-05-05T16:47:43,788   Found link https://files.pythonhosted.org/packages/c5/4f/0e22aaa246f96d6ac87fe5ebb9c5a693fbe8877f537a1022527c47ca43c5/psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (from https://pypi.org/simple/psutil/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*), version: 5.9.8

$ pex3 lock create --output=lock-25.1.json --indent=2 --style=universal --pip-version 25.1 --preserve-pip-download-log pex-pip-download-25.1.log --target-system linux --target-system mac --manylinux manylinux2014 --interpreter-constraint 'CPython==3.11.*' psutil==5.9.8
$ grep psutil-5.9.8-cp36-abi3-manylinux_2_12  pex-pip-download-25.1.log 
2025-05-05T16:48:21,388   Skipping link: none of the wheel's tags (cp36-abi3-manylinux2010_i686, cp36-abi3-manylinux2014_i686, cp36-abi3-manylinux_2_12_i686, cp36-abi3-manylinux_2_17_i686) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/b3/bd/28c5f553667116b2598b9cc55908ec435cb7f77a34f2bff3e3ca765b0f78/psutil-5.9.8-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl (from https://pypi.org/simple/psutil/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*)
2025-05-05T16:48:21,389   Skipping link: none of the wheel's tags (cp36-abi3-manylinux2010_x86_64, cp36-abi3-manylinux2014_x86_64, cp36-abi3-manylinux_2_12_x86_64, cp36-abi3-manylinux_2_17_x86_64) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/c5/4f/0e22aaa246f96d6ac87fe5ebb9c5a693fbe8877f537a1022527c47ca43c5/psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (from https://pypi.org/simple/psutil/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*)

All together as a lockfile diff

$ diff -u lock-25.0.json lock-25.1.json 
--- lock-25.0.json	2025-05-05 16:47:43.886173933 -0400
+++ lock-25.1.json	2025-05-05 16:48:22.359384889 -0400
@@ -23,16 +23,6 @@
             },
             {
               "algorithm": "sha256",
-              "hash": "8cb6403ce6d8e047495a701dc7c5bd788add903f8986d523e3e20b98b733e421",
-              "url": "https://files.pythonhosted.org/packages/b3/bd/28c5f553667116b2598b9cc55908ec435cb7f77a34f2bff3e3ca765b0f78/psutil-5.9.8-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl"
-            },
-            {
-              "algorithm": "sha256",
-              "hash": "d06016f7f8625a1825ba3732081d77c94589dca78b7a3fc072194851e88461a4",
-              "url": "https://files.pythonhosted.org/packages/c5/4f/0e22aaa246f96d6ac87fe5ebb9c5a693fbe8877f537a1022527c47ca43c5/psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl"
-            },
-            {
-              "algorithm": "sha256",
               "hash": "aee678c8720623dc456fa20659af736241f575d79429a0e5e9cf88ae0605cc81",
               "url": "https://files.pythonhosted.org/packages/e7/e3/07ae864a636d70a8a6f58da27cb1179192f1140d5d1da10886ade9405797/psutil-5.9.8-cp36-abi3-macosx_10_9_x86_64.whl"
             }
@@ -57,7 +47,7 @@
   "overridden": [],
   "path_mappings": {},
   "pex_version": "2.36.1",
-  "pip_version": "25.0",
+  "pip_version": "25.1",
   "prefer_older_binary": false,
   "requirements": [
     "psutil==5.9.8"

Comparing vanilla pip behavior (from a git checkout and editable install):

$ git describe --exact-match --tags
25.0
$ TMPDIR=.tmp pip -vvv --disable-pip-version-check --exists-action a --no-input --isolated --log pex-pip-download.log --cache-dir .pip_cache download --dest pip_dest psutil==5.9.8 2>&1 | grep psutil-5.9.8-cp36-abi3-manylinux_2_12
  Skipping link: none of the wheel's tags (cp36-abi3-manylinux2010_i686, cp36-abi3-manylinux2014_i686, cp36-abi3-manylinux_2_12_i686, cp36-abi3-manylinux_2_17_i686) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/b3/bd/28c5f553667116b2598b9cc55908ec435cb7f77a34f2bff3e3ca765b0f78/psutil-5.9.8-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl (from https://pypi.org/simple/psutil/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*)
  Found link https://files.pythonhosted.org/packages/c5/4f/0e22aaa246f96d6ac87fe5ebb9c5a693fbe8877f537a1022527c47ca43c5/psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (from https://pypi.org/simple/psutil/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*), version: 5.9.8
  File was already downloaded /home/ecsb/src/o/pip/pip_dest/psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl

$ git describe --exact-match --tags
25.1
$ TMPDIR=.tmp pip -vvv --disable-pip-version-check --exists-action a --no-input --isolated --log pex-pip-download.log --cache-dir .pip_cache download --dest pip_dest psutil==5.9.8 2>&1 | grep psutil-5.9.8-cp36-abi3-manylinux_2_12
  Skipping link: none of the wheel's tags (cp36-abi3-manylinux2010_i686, cp36-abi3-manylinux2014_i686, cp36-abi3-manylinux_2_12_i686, cp36-abi3-manylinux_2_17_i686) are compatible (run pip debug --verbose to show compatible tags): https://files.pythonhosted.org/packages/b3/bd/28c5f553667116b2598b9cc55908ec435cb7f77a34f2bff3e3ca765b0f78/psutil-5.9.8-cp36-abi3-manylinux_2_12_i686.manylinux2010_i686.manylinux_2_17_i686.manylinux2014_i686.whl (from https://pypi.org/simple/psutil/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*)
  Found link https://files.pythonhosted.org/packages/c5/4f/0e22aaa246f96d6ac87fe5ebb9c5a693fbe8877f537a1022527c47ca43c5/psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (from https://pypi.org/simple/psutil/) (requires-python:>=2.7, !=3.0.*, !=3.1.*, !=3.2.*, !=3.3.*, !=3.4.*, !=3.5.*), version: 5.9.8
  File was already downloaded /home/ecsb/src/o/pip/pip_dest/psutil-5.9.8-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl

So when run from x86_64, pip matches on x86_64 and not i686. Historically, Pex matched on both x86_64 and i686, but now when combined with pip 25.1 matches neither.

[jsirois]

I have a feeling this is dup to #2752, but thanks for the extra data. Like that discussion, I probably won't have a chance to address until ~5/9. I'd suggest not using Pip 25.1 until the next Pex release.

[cburroughs]

Understood, and enjoy your afk time!

[jsirois]

Alright, the fix is out in https://github.com/pex-tool/pex/releases/tag/v2.37.0. Thanks for the review @cburroughs.