Some packages source installation break in 1.7 due to tar extraction (LinkOutsideDestinationError)

[danni-m]

• Poetry version: 1.7
• Python version: 3.9 and 3.11
• OS version and name: macOS 13.3.1
• pyproject.toml:

[tool.poetry]
name = "test-poetry"
version = "0.1.0"
description = ""
authors = ["blabla"]

[tool.poetry.dependencies]
python = "^3.9"
requests = "2.26.0"
idna = "3.4"

[build-system]
requires = ["poetry-core"]
build-backend = "poetry.core.masonry.api"

poetry.toml:

[virtualenvs]
in-project = false

[installer]
no-binary = [":all:"]

• I am on the latest stable Poetry version, installed using a recommended method.
• I have searched the issues of this repo and believe that this is not a duplicate.
• I have consulted the FAQ and blog for any relevant entries or release notes.
• If an exception occurs when executing a command, I executed it again in debug mode (-vvv option) and have included the output below.

poetry install -vvv
Loading configuration file /private/tmp/test_poetry/poetry.toml
Virtualenv test-poetry-I-n-P1h1-py3.9 already exists.
Using virtualenv: /Users/USER/Library/Caches/pypoetry/virtualenvs/test-poetry-I-n-P1h1-py3.9
Installing dependencies from lock file

Finding the necessary packages for the current system

Package operations: 2 installs, 0 updates, 0 removals, 3 skipped

  • Installing idna (3.4): Pending...
[keyring.backend] Loading KWallet
[keyring.backend] Loading SecretService
[keyring.backend] Loading Windows
[keyring.backend] Loading chainer
[keyring.backend] Loading libsecret
[keyring.backend] Loading macOS
Creating new session for pypi.org
Skipping wheel for idna-3.4-py3-none-any.whl as requested in no binary policy for package (idna)
  • Installing idna (3.4): Preparing...
  • Installing idna (3.4): Failed

  Stack trace:

  13  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/installation/executor.py:269 in _execute_operation
       267│
       268│             try:
     → 269│                 result = self._do_execute_operation(operation)
       270│             except EnvCommandError as e:
       271│                 if e.e.returncode == -2:

  12  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/installation/executor.py:379 in _do_execute_operation
       377│             return 0
       378│
     → 379│         result: int = getattr(self, f"_execute_{method}")(operation)
       380│
       381│         if result != 0:

  11  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/installation/executor.py:504 in _execute_install
       502│
       503│     def _execute_install(self, operation: Install | Update) -> int:
     → 504│         status_code = self._install(operation)
       505│
       506│         self._save_url_reference(operation)

  10  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/installation/executor.py:542 in _install
       540│             archive = self._download_link(operation, Link(package.source_url))
       541│         else:
     → 542│             archive = self._download(operation)
       543│
       544│         operation_message = self.get_operation_message(operation)

   9  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/installation/executor.py:746 in _download
       744│             self._yanked_warnings.append(message)
       745│
     → 746│         return self._download_link(operation, link)
       747│
       748│     def _download_link(self, operation: Install | Update, link: Link) -> Path:

   8  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/installation/executor.py:781 in _download_link
       779│             self._write(operation, message)
       780│
     → 781│             archive = self._chef.prepare(archive, output_dir=original_archive.parent)
       782│
       783│         # Use the original archive to provide the correct hash.

   7  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/installation/chef.py:123 in prepare
       121│             return self._prepare(archive, destination=destination, editable=editable)
       122│
     → 123│         return self._prepare_sdist(archive, destination=output_dir)
       124│
       125│     def _prepare(

   6  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/installation/chef.py:178 in _prepare_sdist
       176│         with temporary_directory() as tmp_dir:
       177│             archive_dir = Path(tmp_dir)
     → 178│             extractall(source=archive, dest=archive_dir, zip=zip)
       179│
       180│             elements = list(archive_dir.glob("*"))

   5  ~/Library/Application Support/pypoetry/venv/lib/python3.9/site-packages/poetry/utils/helpers.py:303 in extractall
       301│         with tarfile.open(source) as archive:
       302│             if hasattr(tarfile, "data_filter"):
     → 303│                 archive.extractall(dest, filter="data")
       304│             else:
       305│                 archive.extractall(dest)

   4  ~/.asdf/installs/python/3.9.17/lib/python3.9/tarfile.py:2235 in extractall
       2233│
       2234│         for member in members:
     → 2235│             tarinfo = self._get_extract_tarinfo(member, filter_function, path)
       2236│             if tarinfo is None:
       2237│                 continue

   3  ~/.asdf/installs/python/3.9.17/lib/python3.9/tarfile.py:2289 in _get_extract_tarinfo
       2287│             tarinfo = filter_function(tarinfo, path)
       2288│         except (OSError, FilterError) as e:
     → 2289│             self._handle_fatal_error(e)
       2290│         except ExtractError as e:
       2291│             self._handle_nonfatal_error(e)

   2  ~/.asdf/installs/python/3.9.17/lib/python3.9/tarfile.py:2287 in _get_extract_tarinfo
       2285│         unfiltered = tarinfo
       2286│         try:
     → 2287│             tarinfo = filter_function(tarinfo, path)
       2288│         except (OSError, FilterError) as e:
       2289│             self._handle_fatal_error(e)

   1  ~/.asdf/installs/python/3.9.17/lib/python3.9/tarfile.py:818 in data_filter
        816│
        817│ def data_filter(member, dest_path):
     →  818│     new_attrs = _get_filtered_attrs(member, dest_path, True)
        819│     if new_attrs:
        820│         return member.replace(**new_attrs, deep=False)

  LinkOutsideDestinationError

  'idna-3.4/tools/intranges.py' would link to '/private/var/folders/xb/xlsw2s1s6h5489tn_l0mjy5h0000gp/T/idna/intranges.py', which is outside the destination

  at ~/.asdf/installs/python/3.9.17/lib/python3.9/tarfile.py:805 in _get_filtered_attrs
       801│             if os.path.isabs(member.linkname):
       802│                 raise AbsoluteLinkError(member)
       803│             target_path = os.path.realpath(os.path.join(dest_path, member.linkname))
       804│             if os.path.commonpath([target_path, dest_path]) != dest_path:
    →  805│                 raise LinkOutsideDestinationError(member, target_path)
       806│     return new_attrs
       807│
       808│ def fully_trusted_filter(member, dest_path):
       809│     return member

Cannot install idna.

Issue

This seems to stem from a change #8544, while fixing deprecation there's a new added filter to tar extract that wasn't used before: https://github.com/python-poetry/poetry/pull/8544/files#diff-26f39fd2c1c2ff93b4e043ed53d6c9be8dd7c44328e15d0d238a068da199e171R281.

I've also verified its possible to install the same source package with pip on the same virtualenv:

poetry run pip install -v --force --no-binary ":all:" idna==3.4
Using pip 23.2.1 from /Users/USER/Library/Caches/pypoetry/virtualenvs/test-poetry-I-n-P1h1-py3.9/lib/python3.9/site-packages/pip (python 3.9)
Collecting idna==3.4
  Using cached idna-3.4-py3-none-any.whl
Installing collected packages: idna
  Attempting uninstall: idna
    Found existing installation: idna 3.4
    Uninstalling idna-3.4:
      Removing file or directory /Users/USER/Library/Caches/pypoetry/virtualenvs/test-poetry-I-n-P1h1-py3.9/lib/python3.9/site-packages/idna-3.4.dist-info/
      Removing file or directory /Users/USER/Library/Caches/pypoetry/virtualenvs/test-poetry-I-n-P1h1-py3.9/lib/python3.9/site-packages/idna/
      Successfully uninstalled idna-3.4
Successfully installed idna-3.4

[notice] A new release of pip is available: 23.2.1 -> 23.3.1
[notice] To update, run: pip install --upgrade pip

[dimbleby]

that's what poetry is supposed to do, per https://packaging.python.org/en/latest/specifications/source-distribution-format/#unpacking-with-the-data-filter

you'll want to raise this with idna - or use their wheel distribution

[danni-m]

@dimbleby, since PEP-721 is only a few months old, can we get some grace time (at least with some configuration option)?
Even if we get a newer version of this package, it might only be compatible with some ecosystem packages people already use in the wild.

If this is acceptable, I can create a PR for this configuration flag.

[dimbleby]

I'd be against that but others might feel differently and their vote counts more.

The wheel installs fine, so you're not blocked (and I expect that's how almost everyone is installing idna anyway).

kjd/idna#129 (comment) suggests that there might be a new idna release in the not-too-distant future: so you're likely just as well off instead submitting a merge request there, to fix their sdist.

[Secrus]

Personally, I am against it. I don't like the idea of backing out of the already implemented way. The more "switches" and "toggles" we add, the more people will abuse this, and packages that don't comply with modern standards won't get fixed (and then break anyway once the grace period ends).

[martinmiglio]

Also experiencing this with llama-cpp-python = "0.2.15":

5.029   LinkOutsideDestinationError
5.029 
5.029   'llama_cpp_python-0.2.15/vendor/llama.cpp/spm-headers/ggml.h' would link to '/tmp/ggml.h', which is outside the destination
5.029 
5.029   at /usr/local/lib/python3.11/tarfile.py:806 in _get_filtered_attrs
5.365        802│             if os.path.isabs(member.linkname):
5.365        803│                 raise AbsoluteLinkError(member)
5.366        804│             target_path = os.path.realpath(os.path.join(dest_path, member.linkname))
5.366        805│             if os.path.commonpath([target_path, dest_path]) != dest_path:
5.366     →  806│                 raise LinkOutsideDestinationError(member, target_path)
5.366        807│     return new_attrs
5.366        808│ 
5.366        809│ def fully_trusted_filter(member, dest_path):
5.366        810│     return member
5.367 
5.367 Cannot install llama-cpp-python.

running in python:3.11-buster with poetry==1.7.0

[dimbleby]

Please report it to llama-cpp-python so that they can fix their sdist.

[dimbleby]

However I suspect the real issue is that python is wrong... python/cpython#107845

Probably something like pypa/build#675 (comment) would be accepted.