Cannot install scikit-learn from conda-forge channel alongside pytorch from the pytorch channel

[anjos]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

There seems to be no way to have these 2 packages installed together:

• scikit-learn >=1.4 (from this feedstock)
• pytorch >=2.3 (from pytorch channel)

Test command:

mamba create --strict-channel-priority --override-channels -c pytorch -c conda-forge -n test --dry-run 'scikit-learn>=1.4' 'pytorch>=2.3'

The problem seems related to this package's dependency on llvm-openmp. The latter is pinned at build time to the build-time version of that package (using >=), if I understand well (c.f.: #265, conda-forge/openmp-feedstock#126).

It so happens pytorch (on the pytorch channel) has introduced a tight requirement on llvm-openmp (has to be '<16', starting from version 2.3). This introduces a conflict because most versions of scikit-learn that were built in the past months end-up getting a >=16 runtime constraint.

(N.B.: For some reason I do not understand, pytorch from conda-forge channel does not introduce this pin yet. For this reason, version 2.3.0 and 2.3.1 are available from the conda-forge channel and do install correctly. However this does not solve our issues since we need the latest changes in version 2.4.0.)

Concretely, the situation is such that one cannot install later versions of pytorch from the pytorch channel and this package without incurring in resolution problems.

I wonder if a rebuild of this package using llvm-openmp<16 could be considered, or if another solution could be found to allow co-existence with pytorch@pytorch? (I'm asking this here because this package does not have a strict requirement for llvm-openmp whereas pytorch seems to.)

Installed packages

N/A

Environment info

N/A

[jjerphan]

Hi @anjos,

I confirm I can reproduce this issue for osx_arm64 with:

CONDA_SUBDIR=osx-arm64 mamba create --strict-channel-priority --override-channels -c pytorch -c conda-forge -n pskl --dry-run 'scikit-learn>=1.4' 'pytorch>=2.3'                                                                                  

The problem seems related to this package's dependency on llvm-openmp. The latter is pinned at build time to the build-time version of that package (using >=), if I understand well (c.f.: #265, conda-forge/openmp-feedstock#126).

Your analysis is right (LLVM version are forward between major versions, hence the use of >=).

It so happens pytorch (on the pytorch channel) has introduced a tight requirement on llvm-openmp (has to be '<16', starting from version 2.3). This introduces a conflict because most versions of scikit-learn that were built in the past months end-up getting a >=16 runtime constraint.

This is quite constraining LLVM 16 (and thus llvm-openmp) has been used since October 2023 to build packages for macOS.

(N.B.: For some reason I do not understand, pytorch from conda-forge channel does not introduce this pin yet. For this reason, version 2.3.0 and 2.3.1 are available from the conda-forge channel and do install correctly. However this does not solve our issues since we need the latest changes in version 2.4.0.)

I guess it is so that PyTorch stays in line with conda-forge's global pinning constrained linked above; you can open an issue on conda-forge/pytorch-cpu-feedstock on this point.

I wonder if a rebuild of this package using llvm-openmp<16 could be considered, or if another solution could be found to allow co-existence with pytorch@pytorch? (I'm asking this here because this package does not have a strict requirement for llvm-openmp whereas pytorch seems to.)

Unless a really hard constrain explaining the reason for this pin's existence is known, I think that the problem should rather be solved upstream directly than by introducing new build variants of package on conda-forge for llvm 15 (LLVM 17 will soon be used on conda-forge, see conda-forge/conda-forge-pinning-feedstock#4890).

Do you know if the rationale behind this pin documented somewhere?

[anjos]

Right, I forgot that platform constraint (I'm also on osx-arm64).

I have not found any rationale for the tight pin on llvm <= 16, but I have not looked deeply into how those packages are built, simply because IDK where are the recipes. I'll do a little digging and report back.

[jjerphan]

I do not know of equivalents of the feedstock for the pytorch channel, but the recipes are stored within one of the packages' archive as well, so you can manually download one of the pytorch package and inspect it.

[anjos]

Thanks. I was thinking about looking for the build repo and then check if this has not been somewhat reported there. Then, I think I'll open an issue and possibly link this one for a discussion and solution. It may indeed be the case that the pin is just too restrictive w/o a strong reason on the upstream side. If that turns out to be absolutely required, then we should go back and push conda-forge's pytorch version to adopt the same strategy, and think about an overall solution that would work so both packages can co-exist. Do you agree with this strategy?

[anjos]

The restrictive pinning seems to be coming from here.

Which, in turn, links to this open issue on the upstream repository: pytorch/pytorch#99625

[anjos]

@jjerphan: I went around and linked all relevant issues. I also opened a new issue on the build repositories that eventually dump pytorch packages on the pytorch channel of anaconda.org. I hope this is enough to trigger a response for a solution.

[jjerphan]

Thank you.

[hmaarrfk]

Just to chime in to help out the scikit-learn maintainers.

• We do not support mixing conda-forge with any other channel at conda-forge.

the pytorch channel, to my knowledge, is designed for compatibility with the defaults channel which is incompatible with conda-forge.

You can install scikit-learn with pytorch just fine from conda-forge packages:

mamba create --strict-channel-priority --override-channels -c conda-forge -n test --dry-run 'scikit-learn>=1.4' 'pytorch>=2.3'

 $ mamba create --strict-channel-priority --override-channels -c conda-forge -n test --dry-run 'scikit-learn>=1.4' 'pytorch>=2.3'
WARNING: A conda environment already exists at '/home/mark/miniforge3/envs/test'
Remove existing environment (y/[n])? y


Looking for: ["scikit-learn[version='>=1.4']", "pytorch[version='>=2.3']"]

conda-forge/noarch                                  15.9MB @  10.6MB/s  1.7s
conda-forge/linux-64                                36.8MB @  13.7MB/s  3.3s
Transaction

  Prefix: /home/mark/miniforge3/envs/test

  Updating specs:

   - scikit-learn[version='>=1.4']
   - pytorch[version='>=2.3']


  Package                    Version  Build                      Channel           Size
─────────────────────────────────────────────────────────────────────────────────────────
  Install:
─────────────────────────────────────────────────────────────────────────────────────────

  + python_abi                  3.12  4_cp312                    conda-forge     Cached
  + _libgcc_mutex                0.1  conda_forge                conda-forge     Cached
  + ld_impl_linux-64            2.40  hf3520f5_7                 conda-forge     Cached
  + ca-certificates         2024.7.4  hbcca054_0                 conda-forge     Cached
  + libgcc-ng                 14.1.0  h77fa898_0                 conda-forge     Cached
  + libzlib                    1.3.1  h4ab18f5_1                 conda-forge     Cached
  + libstdcxx-ng              14.1.0  hc0a3c3a_0                 conda-forge     Cached
  + zstd                       1.5.6  ha6fb4c9_0                 conda-forge     Cached
  + llvm-openmp               18.1.8  hf5423f3_0                 conda-forge     Cached
  + _openmp_mutex                4.5  2_kmp_llvm                 conda-forge        6kB
  + libiconv                    1.17  hd590300_2                 conda-forge     Cached
  + icu                         75.1  he02047a_0                 conda-forge     Cached
  + libgfortran5              14.1.0  hc5f4f2c_0                 conda-forge     Cached
  + tk                        8.6.13  noxft_h4845f30_101         conda-forge     Cached
  + openssl                    3.3.1  h4bc722e_2                 conda-forge     Cached
  + libxcrypt                 4.4.36  hd590300_1                 conda-forge     Cached
  + libffi                     3.4.2  h7f98852_5                 conda-forge     Cached
  + bzip2                      1.0.8  h4bc722e_7                 conda-forge     Cached
  + ncurses                      6.5  h59595ed_0                 conda-forge     Cached
  + libuuid                   2.38.1  h0b41bf4_0                 conda-forge     Cached
  + libsqlite                 3.46.0  hde9e2c9_0                 conda-forge     Cached
  + libnsl                     2.0.1  hd590300_0                 conda-forge     Cached
  + libexpat                   2.6.2  h59595ed_0                 conda-forge     Cached
  + xz                         5.2.6  h166bdaf_0                 conda-forge     Cached
  + gmp                        6.3.0  hac33072_2                 conda-forge     Cached
  + libabseil             20240116.2  cxx17_he02047a_1           conda-forge     Cached
  + sleef                      3.6.1  h3400bea_1                 conda-forge      998kB
  + libuv                     1.48.0  hd590300_0                 conda-forge     Cached
  + libgfortran-ng            14.1.0  h69a702a_0                 conda-forge     Cached
  + readline                     8.2  h8228510_1                 conda-forge     Cached
  + libxml2                   2.12.7  he7c6b58_4                 conda-forge     Cached
  + mpfr                       4.2.1  h38ae2d0_2                 conda-forge     Cached
  + libprotobuf               4.25.3  h08a7969_0                 conda-forge     Cached
  + libopenblas               0.3.27  pthreads_hac2b453_1        conda-forge     Cached
  + libhwloc                  2.11.1  default_hecaa2ac_1000      conda-forge     Cached
  + mpc                        1.3.1  hfe3b2da_0                 conda-forge     Cached
  + libblas                    3.9.0  23_linux64_openblas        conda-forge     Cached
  + tbb                    2021.12.0  h434a139_3                 conda-forge     Cached
  + liblapack                  3.9.0  23_linux64_openblas        conda-forge     Cached
  + libcblas                   3.9.0  23_linux64_openblas        conda-forge     Cached
  + mkl                     2023.2.0  h84fe81f_50496             conda-forge     Cached
  + cuda-version                12.6  h7480c83_3                 conda-forge     Cached
  + tzdata                     2024a  h0c530f3_0                 conda-forge     Cached
  + cuda-cudart_linux-64     12.6.37  h85509e4_0                 conda-forge     Cached
  + nccl                    2.22.3.1  hbc370b7_1                 conda-forge      112MB
  + libnvjitlink             12.6.20  he02047a_0                 conda-forge     Cached
  + cuda-nvrtc               12.6.20  he02047a_0                 conda-forge     Cached
  + cuda-nvtx                12.6.37  he02047a_0                 conda-forge       31kB
  + libcufft               11.2.6.28  he02047a_0                 conda-forge     Cached
  + libcurand              10.3.7.37  he02047a_0                 conda-forge     Cached
  + python                    3.12.4  h194c7f8_0_cpython         conda-forge     Cached
  + cuda-cudart              12.6.37  he02047a_0                 conda-forge     Cached
  + libcusparse            12.5.2.23  he02047a_0                 conda-forge     Cached
  + libcublas              12.6.0.22  he02047a_0                 conda-forge     Cached
  + libmagma                   2.7.2  h173bb3b_2                 conda-forge     Cached
  + cudnn                   8.9.7.29  h092f7fd_3                 conda-forge     Cached
  + libcusolver            11.6.4.38  he02047a_0                 conda-forge     Cached
  + libmagma_sparse            2.7.2  h173bb3b_3                 conda-forge     Cached
  + libtorch                   2.3.1  cuda120_h2b0da52_300       conda-forge     Cached
  + wheel                     0.44.0  pyhd8ed1ab_0               conda-forge     Cached
  + setuptools                72.1.0  pyhd8ed1ab_0               conda-forge     Cached
  + pip                         24.2  pyhd8ed1ab_0               conda-forge     Cached
  + fsspec                  2024.6.1  pyhff2d567_0               conda-forge     Cached
  + filelock                  3.15.4  pyhd8ed1ab_0               conda-forge     Cached
  + threadpoolctl              3.5.0  pyhc1e730c_0               conda-forge       24kB
  + joblib                     1.4.2  pyhd8ed1ab_0               conda-forge     Cached
  + mpmath                     1.3.0  pyhd8ed1ab_0               conda-forge     Cached
  + typing_extensions         4.12.2  pyha770c72_0               conda-forge     Cached
  + networkx                     3.3  pyhd8ed1ab_1               conda-forge     Cached
  + markupsafe                 2.1.5  py312h98912ed_0            conda-forge     Cached
  + gmpy2                      2.1.5  py312h1d5cde6_1            conda-forge      212kB
  + numpy                      2.0.1  py312h1103770_0            conda-forge     Cached
  + scipy                     1.14.0  py312hc2bc53b_1            conda-forge     Cached
  + scikit-learn               1.5.1  py312h775a589_0            conda-forge       10MB
  + jinja2                     3.1.4  pyhd8ed1ab_0               conda-forge     Cached
  + sympy                     1.13.0  pypyh2585a3b_103           conda-forge     Cached
  + pytorch                    2.3.1  cuda120_py312h26b3cf7_300  conda-forge       25MB

  Summary:

  Install: 77 packages

  Total download: 149MB

─────────────────────────────────────────────────────────────────────────────────────────


Dry run. Exiting.

DryRunExit: Dry run. Exiting.

If you want the latest + latest packages, you can't have pinnings like pytorch put on llvm. I understand that temporarily it is necessary to do so, but in general please try to stick to the the following combination of channels

• conda-forge only (to get support from conda-forge)
• defaults only (to get support from anaconda)
• defaults+pytorch+nividia to (to get support form pytorch upstream -- these are the instructions they provide on their website).

[adrinjalali]

Thanks @hmaarrfk . I think then we can close this one as "wontfix" since it's not really a scikit-learn or a conda-forge issue.