PyTorch nightly inconsistency between uv pip install vs uv sync (missing nvidia packages)
#10693
Labels
lock
Related to universal resolution and locking
Comments
|
I'm not surprised these are different. In the first, you set In the latter, you set which only uses that index for those exact packages due to The first also performs single-platform resolution while the latter does a universal resolution. Can you share your lockfile? What platform are you on? Can you reproduce in a Dockerfile? |
|
I'm on linux. Not using docker. Removing uv.lockvs uv pip freeze |
|
Sharing a reproduction with a Dockerfile would be useful for us to isolate the issue. Anyway... I can reproduce this on my x86-64 Linux machine During These dependencies are missing from torch in the lockfile This occurs because pytorch has different metadata depending on the wheel. e.g., the x86-64 wheel we use in But when we lock, we use an arbitrary wheel because it's a universal resolution Frankly, it's really not ideal that they change their dependencies from wheel to wheel. |
|
Very interesting. This is presumably because the nvidia libraries only exist in x64 and not arm64? Maybe I can force x64 resolution in my project file. |
|
Yeah presumably. They could still put the dependencies there with the same platform markers they're using in the other wheel though... 🤷♀ You can force use of the x86-64 wheel with [tool.uv.sources]
torch = [{ url = "https://download.pytorch.org/whl/nightly/cu126/torch-2.7.0.dev20250116%2Bcu126-cp313-cp313t-manylinux_2_28_x86_64.whl" }] |
|
Ick, I was hoping for something like: [tool.uv]
environments = [
"platform_arch == 'x86_64'",
]Let me see if I can start a thread upstream, maybe they're amenable to fixing up the wheels. EDIT: I see now this was the same conclusion reached by @charliermarsh in #10119 (comment) |
The change upstream is pretty trivial, and root cause here may just be an oversight. Funny it appears the cpu linux/arm64 wheels have the cuda dependencies listed. |
tmm1
added a commit
to tmm1/pytorch
that referenced
this issue
Jan 17, 2025
…cies This resolves issues installing torch nightly wheels into a uv venv (astral-sh/uv#10693)
tmm1
added a commit
to tmm1/pytorch
that referenced
this issue
Jan 17, 2025
…cies This resolves issues installing torch nightly wheels into a uv venv (astral-sh/uv#10693)
|
I opened a PR on the pytorch repo to fix the metadata on the nightly wheels. In the mean time, I had to specify the dependencies explicitly to get things working: [project.optional-dependencies]
nvidia = [
# workaround for pytorch/pytorch#145021 and astral-sh/uv#10693
"nvidia-cuda-nvrtc-cu12 == 12.6.77",
"nvidia-cuda-runtime-cu12 == 12.6.77",
"nvidia-cuda-cupti-cu12 == 12.6.80",
"nvidia-cudnn-cu12 == 9.5.1.17",
"nvidia-cublas-cu12 == 12.6.4.1",
"nvidia-cufft-cu12 == 11.3.0.4",
"nvidia-curand-cu12 == 10.3.7.77",
"nvidia-cusolver-cu12 == 11.7.1.2",
"nvidia-cusparse-cu12 == 12.5.4.2",
"nvidia-cusparselt-cu12 == 0.6.3",
"nvidia-nccl-cu12 == 2.21.5",
"nvidia-nvtx-cu12 == 12.6.77",
"nvidia-nvjitlink-cu12 == 12.6.85",
]
torch = [
"torch",
"torchao",
"torchaudio",
"torchvision",
]
[tool.uv.sources]
torch = [{ index = "pytorch-nightly-cu126" }]
torchvision = [{ index = "pytorch-nightly-cu126" }]
torchaudio = [{ index = "pytorch-nightly-cu126" }]
torchao = [{ index = "pytorch-nightly-cu126" }]
nvidia-cuda-nvrtc-cu12 = [{ index = "pytorch-nightly-cu126" }]
nvidia-cuda-runtime-cu12 = [{ index = "pytorch-nightly-cu126" }]
nvidia-cuda-cupti-cu12 = [{ index = "pytorch-nightly-cu126" }]
nvidia-cudnn-cu12 = [{ index = "pytorch-nightly-cu126" }]
nvidia-cublas-cu12 = [{ index = "pytorch-nightly-cu126" }]
nvidia-cufft-cu12 = [{ index = "pytorch-nightly-cu126" }]
nvidia-curand-cu12 = [{ index = "pytorch-nightly-cu126" }]
nvidia-cusolver-cu12 = [{ index = "pytorch-nightly-cu126" }]
nvidia-cusparse-cu12 = [{ index = "pytorch-nightly-cu126" }]
nvidia-cusparselt-cu12 = [{ index = "pytorch-nightly-cu126" }]
nvidia-nccl-cu12 = [{ index = "pytorch-nightly-cu126" }]
nvidia-nvtx-cu12 = [{ index = "pytorch-nightly-cu126" }]
nvidia-nvjitlink-cu12 = [{ index = "pytorch-nightly-cu126" }]
[[tool.uv.index]]
name = "pytorch-nightly-cu126"
url = "https://download.pytorch.org/whl/nightly/cu126"
explicit = true |
|
Nice -- thank you for doing that. |
|
This is also an issue with the released PyTorch 2.6.0 Cuda 12.6 on x86. Maybe it's related to pytorch/pytorch#146679? If so, then it should be resolved for PyTorch 2.6.1. |
pytorch-bot bot
pushed a commit
to pytorch/pytorch
that referenced
this issue
Mar 5, 2025
…cies This resolves issues installing torch nightly wheels into a uv venv (astral-sh/uv#10693)
pytorchmergebot
pushed a commit
to pytorch/pytorch
that referenced
this issue
Mar 10, 2025
…cies (#145021) This resolves issues installing torch nightly wheels into a `uv sync`-generated `.venv` The root cause is that the x64 and arm64 cuda nightly wheels have inconsistent metadata. This can be seen comparing `generated-linux-aarch64-binary-manywheel-nightly.yml` and `generated-linux-binary-manywheel-nightly.yml` `uv` expects consistency: astral-sh/uv#10693 >Frankly, it's really not ideal that they change their dependencies from wheel to wheel. >They could still put the dependencies there with the same platform markers they're using in the other wheel though... 🤷♀ astral-sh/uv#10119 (comment) >I think this is something that basically has to be solved by PyTorch. The issue is that the wheels for `2.6.0.dev20241222+cu126` don't have consistent metadata, and it's a fundamental assumption of uv that the metadata for a given version _is_ consistent. To resolve this, I modified the arm64 nightly build workflow to add two new `PYTORCH_EXTRA_INSTALL_REQUIREMENTS` entries, under `manywheel-py3_11-cuda-aarch64-build` and `manywheel-py3_12-cuda-aarch64-build`. These are based on their equivalents in the x64 workflow for the corresponding python versions. I used the cuda 12.6 dependencies versions for the nvidia packages, to match the `DOCKER_IMAGE: pytorch/manylinuxaarch64-builder:cuda12.6-main` being used by these jobs. (The arm64 workflow file already had several `PYTORCH_EXTRA_INSTALL_REQUIREMENTS` entries, under various cpu wheels. I'm not sure why these are there, but I left them as-is.) Pull Request resolved: #145021 Approved by: https://github.com/seemethere, https://github.com/atalman Co-authored-by: Eli Uriegas <[email protected]> Co-authored-by: Andrey Talman <[email protected]>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
in #2541 (comment),
uv pip installwas fixed for torch-nightly:and in #9651 (comment),
uv syncwas fixed for torch-nightly:however, notice how both produce different results
in the
uv sync/pyproject.tomlversion, the nvidia packages are missing causing runtime failures.cc @charliermarsh #10119 (comment)
The text was updated successfully, but these errors were encountered: