Poetry 1.2.1 install fails to install packages with pip dependency

[SfinxCZ]

• Poetry version: 1.2.1

• Python version: 3.8.12

• OS version and name: macOS 12.6 but same problem observed in docker built with python:3.8 as base image

• pyproject.toml: https://gist.github.com/SfinxCZ/a8a80b14cc19c6631942adb2fb6ca3ff

• I am on the latest stable Poetry version, installed using a recommended method.

• I have searched the issues of this repo and believe that this is not a duplicate.

• I have consulted the FAQ and blog for any relevant entries or release notes.

• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

Issue

After upgrading to poetry 1.2.1 I have encountered an issue where poetry fails to install project linked above (pyproject.toml). From what I've observed, the problem seems to occur, when the project depends on a library that has pip as its dependency (chalice in this case). Then, during the installation poetry tries to upgrade pip to version listed in the poetry.lock file. However, other libraries that are being installed at the same moment require pip to be installed and installation fails - see the log here install.log.

Reproducing this error is tricky as it seems to be a raise condition that depends on the parallelism used in the install step.

Downgrading to poetry 1.2.0 seems to fix the problem.

[neersighted]

cc @radoering @dimbleby

This is related to walking back the use of the virtualenv seed pip -- I think we may need to add a special case for pip and upgrade it first (with no parallelism) when we are manipulating it to avoid this.

[dimbleby]

If that's right then:

• it has long been broken (albeit briefly not broken at 1.2.0): poetry 1.1.x did not prefer the embedded pip
• it's easy to work around (upgrade pip before running poetry)
• a better fix is getting near (Faster build and installation of packages #6205 so that poetry doesn't rely on pip here at all)

suggest that if someone can think of a cheap fix then great, but in that context I submit that it's not worth a lot of effort

[neersighted]

Poetry 1.1.x will not actually manipulate pip at all -- it is effectively blind to pip (try adding pip to a pyproject.toml, then check the lock file using 1.1.15).

This is a new issue, but arguably not a regression as treating pip as just another dependency is a new capability of Poetry 1.2.

Edit: upgrading pip before running Poetry is not really a solution as Poetry may want to change the version (or reinstall if the direct origin URL is different) based on the lock file -- Poetry would need to upgrade pip first before running the concurrent installer on all other dependencies.

[dimbleby]

re upgrading pip: indeed it would be necessary to update to whatever version was written in the lock file. In this case that is the latest but ack that in principle it might not be

[dimbleby]

maybe that poetry 1.1 behaviour suggests the cheap and good-enough-for-now fix is to reinstate pip among the UNSAFE_PACKAGES

That's still broken in principle - what if the project requires a version of pip that's different than the version that's currently in the environment?

But depending on pip is pretty rare, then it's really quite unlikely that "some recent pip" is not good enough - it's probably a sticking plaster that would hold for some time.

[neersighted]

I'm not sure adding pip back to UNSAFE_PACKAGES is the best call here -- I intend to finish #6458 once the next pip version is released. It will solve this issue for every package but Poetry installed in editable mode installing itself (which to be fair, is very rare and specific, but annoying for us people working on Poetry) -- I'd like to try and solve this if it's not too much of a headache as the existing pip-based install path will remain in the codebase for some time.

[beeb]

I think I have this problem with 1.2.0 as well, see the following repro (RE @SfinxCZ saying it wasn't an issue on 1.2.0):

(host) $ docker run -it --platform linux/arm64/v8 python:3.8 /bin/bash
curl -sSL https://install.python-poetry.org | python3 - --version 1.2.0
export PATH="/root/.local/bin:$PATH"
poetry new test && cd test
poetry add matrixprofile

Output:

...
  • Installing protobuf (3.11.2)
  • Installing scipy (1.6.1)
  • Installing matrixprofile (1.1.10): Failed

  CalledProcessError

  Command '['/root/.cache/pypoetry/virtualenvs/test-swbZ_4R8-py3.8/bin/python', '/root/.local/share/pypoetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-22.2.2-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/root/.cache/pypoetry/virtualenvs/test-swbZ_4R8-py3.8', '--no-deps', '/root/.cache/pypoetry/artifacts/ba/03/15/5f3a11f3876d10c9c374fcae1059e5540e3c7aef01678621ee5afe1041/matrixprofile-1.1.10.tar.gz']' returned non-zero exit status 1.

  at /usr/local/lib/python3.8/subprocess.py:516 in run
       512│             # We don't call process.wait() as .__exit__ does that for us.
       513│             raise
       514│         retcode = process.poll()
       515│         if check and retcode:
    →  516│             raise CalledProcessError(retcode, process.args,
       517│                                      output=stdout, stderr=stderr)
       518│     return CompletedProcess(process.args, retcode, stdout, stderr)
       519│ 
       520│ 

The following error occurred when trying to handle this error:


  EnvCommandError

  Command ['/root/.cache/pypoetry/virtualenvs/test-swbZ_4R8-py3.8/bin/python', '/root/.local/share/pypoetry/venv/lib/python3.8/site-packages/virtualenv/seed/wheels/embed/pip-22.2.2-py3-none-any.whl/pip', 'install', '--use-pep517', '--disable-pip-version-check', '--prefix', '/root/.cache/pypoetry/virtualenvs/test-swbZ_4R8-py3.8', '--no-deps', '/root/.cache/pypoetry/artifacts/ba/03/15/5f3a11f3876d10c9c374fcae1059e5540e3c7aef01678621ee5afe1041/matrixprofile-1.1.10.tar.gz'] errored with the following return code 1, and output: 
  Processing /root/.cache/pypoetry/artifacts/ba/03/15/5f3a11f3876d10c9c374fcae1059e5540e3c7aef01678621ee5afe1041/matrixprofile-1.1.10.tar.gz
    Installing build dependencies: started
    Installing build dependencies: finished with status 'done'
    Getting requirements to build wheel: started
    Getting requirements to build wheel: finished with status 'error'
    error: subprocess-exited-with-error
    
    × Getting requirements to build wheel did not run successfully.
    │ exit code: 1
    ╰─> [41 lines of output]
        /root/.cache/pypoetry/virtualenvs/test-swbZ_4R8-py3.8/bin/python: No module named pip
        /tmp/pip-build-env-d64m8ub7/overlay/lib/python3.8/site-packages/setuptools/installer.py:27: SetuptoolsDeprecationWarning: setuptools.installer is deprecated. Requirements should be satisfied by a PEP 517 installer.
          warnings.warn(
        Traceback (most recent call last):
          File "/tmp/pip-build-env-d64m8ub7/overlay/lib/python3.8/site-packages/setuptools/installer.py", line 82, in fetch_build_egg
            subprocess.check_call(cmd)
          File "/usr/local/lib/python3.8/subprocess.py", line 364, in check_call
            raise CalledProcessError(retcode, cmd)
        subprocess.CalledProcessError: Command '['/root/.cache/pypoetry/virtualenvs/test-swbZ_4R8-py3.8/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpgxjzs6b_', '--quiet', 'numpy>=1.16.2']' returned non-zero exit status 1.
        
        The above exception was the direct cause of the following exception:
        
        Traceback (most recent call last):
          File "/tmp/tmp1jbceis0", line 363, in <module>
            main()
          File "/tmp/tmp1jbceis0", line 345, in main
            json_out['return_val'] = hook(**hook_input['kwargs'])
          File "/tmp/tmp1jbceis0", line 130, in get_requires_for_build_wheel
            return hook(config_settings)
          File "/tmp/pip-build-env-d64m8ub7/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 338, in get_requires_for_build_wheel
            return self._get_build_requires(config_settings, requirements=['wheel'])
          File "/tmp/pip-build-env-d64m8ub7/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 320, in _get_build_requires
            self.run_setup()
          File "/tmp/pip-build-env-d64m8ub7/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 482, in run_setup
            super(_BuildMetaLegacyBackend,
          File "/tmp/pip-build-env-d64m8ub7/overlay/lib/python3.8/site-packages/setuptools/build_meta.py", line 335, in run_setup
            exec(code, locals())
          File "<string>", line 4, in <module>
          File "/tmp/pip-build-env-d64m8ub7/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 875, in fetch_build_eggs
            resolved_dists = pkg_resources.working_set.resolve(
          File "/tmp/pip-build-env-d64m8ub7/overlay/lib/python3.8/site-packages/pkg_resources/__init__.py", line 789, in resolve
            dist = best[req.key] = env.best_match(
          File "/tmp/pip-build-env-d64m8ub7/overlay/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1075, in best_match
            return self.obtain(req, installer)
          File "/tmp/pip-build-env-d64m8ub7/overlay/lib/python3.8/site-packages/pkg_resources/__init__.py", line 1087, in obtain
            return installer(requirement)
          File "/tmp/pip-build-env-d64m8ub7/overlay/lib/python3.8/site-packages/setuptools/dist.py", line 945, in fetch_build_egg
            return fetch_build_egg(self, req)
          File "/tmp/pip-build-env-d64m8ub7/overlay/lib/python3.8/site-packages/setuptools/installer.py", line 84, in fetch_build_egg
            raise DistutilsError(str(e)) from e
        distutils.errors.DistutilsError: Command '['/root/.cache/pypoetry/virtualenvs/test-swbZ_4R8-py3.8/bin/python', '-m', 'pip', '--disable-pip-version-check', 'wheel', '--no-deps', '-w', '/tmp/tmpgxjzs6b_', '--quiet', 'numpy>=1.16.2']' returned non-zero exit status 1.
        [end of output]
    
    note: This error originates from a subprocess, and is likely not a problem with pip.
  error: subprocess-exited-with-error
  
  × Getting requirements to build wheel did not run successfully.
  │ exit code: 1
  ╰─> See above for output.
  
  note: This error originates from a subprocess, and is likely not a problem with pip.
  

  at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/env.py:1473 in _run
      1469│                 output = subprocess.check_output(
      1470│                     command, stderr=subprocess.STDOUT, env=env, **kwargs
      1471│                 )
      1472│         except CalledProcessError as e:
    → 1473│             raise EnvCommandError(e, input=input_)
      1474│ 
      1475│         return decode(output)
      1476│ 
      1477│     def execute(self, bin: str, *args: str, **kwargs: Any) -> int:

The following error occurred when trying to handle this error:


  PoetryException

  Failed to install /root/.cache/pypoetry/artifacts/ba/03/15/5f3a11f3876d10c9c374fcae1059e5540e3c7aef01678621ee5afe1041/matrixprofile-1.1.10.tar.gz

  at ~/.local/share/pypoetry/venv/lib/python3.8/site-packages/poetry/utils/pip.py:51 in pip_install
       47│ 
       48│     try:
       49│         return environment.run_pip(*args)
       50│     except EnvCommandError as e:
    →  51│         raise PoetryException(f"Failed to install {path.as_posix()}") from e
       52│ 

The workaround I found:

(host) $ docker run -it --platform linux/arm64/v8 python:3.8 /bin/bash
curl -sSL https://install.python-poetry.org | python3 - --version 1.2.0
export PATH="/root/.local/bin:$PATH"
poetry new test && cd test
poetry run pip install matrixprofile
poetry add matrixprofile

[neersighted]

@beeb That's a issue with the matrixprofile package not building in an isolated build environment and will need to be resolved by that upstream -- it's not the race condition being expressed here.

You can reproduce it easily with pip install --use-pep517 matrixprofile -- anyway, see #6518 or the many similar/duplicate issues if you want to learn more about what is going wrong (and how it's external to Poetry).

[beeb]

Sorry but I thought it was related because of the /root/.cache/pypoetry/virtualenvs/test-swbZ_4R8-py3.8/bin/python: No module named pip in the logs.

How come the build succeeds when invoking poetry run pip install directly? The linked issue doesn't explain that and seems unrelated. I have trouble to understand why I can't install a package that works with pip when using poetry. I highly doubt matrixprofile will update soon since they didn't commit for more than a year, and some of my other dependencies are using it. How am I supposed to make it work finally? Can't poetry provide backwards compatibility with existing packages?