Poetry doesn't try public pypi when private pypi included

[damienrj]

• [ X ] I am on the latest Poetry version.
• [ X ] I have searched the issues of this repo and believe that this is not a duplicate.
• [ X ] If an exception occurs when executing a command, I executed it again in debug mode (-vvv option).

• macOS : 11.1
• Poetry version: 1.1.5

Issue

When using a private pypi repo, the public repo is no longer being checked. I can get everything working again by adding

[[tool.poetry.source]]
name = "pypi-public"
url = "https://pypi.org/simple/"

But the wasn't needed in the past.

[sinoroc]

Any details? Is your private repository set as "secondary"? Why do you think the public PyPI is not checked anymore, does your project have dependencies that should be downloaded from the public PyPI?

[damienrj]

Seems to be related to #3306

Yeah, there are packages that are not on our private repository that then fail because it tries to pull from only the private one. Adding the public one as a source is a work around that unblocked me.

[caniko]

Any details? Is your private repository set as "secondary"? Why do you think the public PyPI is not checked anymore, does your project have dependencies that should be downloaded from the public PyPI?

I have this exact issue, you can see my pyproject.toml file here

[damienrj]

The way we have been solving it is by

[[tool.poetry.source]]
name = "private"
url = "https://private/simple"
secondary = true


[[tool.poetry.source]]
name = "pypi-public"
url = "https://pypi.org/simple/"

Otherwise if you look at the lockfile the source for all packages is our private repo. This puts more work on our private repo because there isn't any reason to check it for the public packages since we are not running a full clone of pypi.

[dwyatte]

@damienrj given this is related to #3306, do you want to check to see if the behavior is still present in https://github.com/python-poetry/poetry/releases/tag/1.2.0a1? Seems #3306 (PR #3406) made the release notes in 1.2.0a1 and may be cherry-picked for an upcoming patch release per #4241

[jleclanche]

I'm affected by this. Tried in 1.2.0a2 for good measure… At first glance it looks fixed: poetry.lock no longer updates with the incorrect default repository.

However, it's apparent that poetry still checks the non-default repository for every package even if a valid package has been found on the default one. This might be intended, but it also slows operations down significantly. Common scenario: App depends on a handful of packages in a private repository, and a bunch of packages in pypi. A quick poetry update -vvv will reveal what's going on. In my case i'm using a private gitlab pypi repo and:

This makes poetry update a 35 second operation on a warm cache for my project. If I remove the custom repository, same project, it completes in under 1 second.

[jleclanche]

The above happens even if source="pypi" is specified on every non-private dependency.

[persiyanov]

Experiencing the same issue as @jleclanche has mentioned. I'm on poetry 1.1.8. During poetry update every package seems to be requested from our private PyPI.

I have the following pypi configuration in my pyproject.toml:

[[tool.poetry.source]]
name = "cnstrc_pypi"
url = "https://pypi.cnstrc.com/simple/"
secondary = true

And this is the log piece I'm getting (repeated pattern for every package) for poetry update -vvv:

PyPI: 7 packages found for joblib >=0.14.1
cnstrc_pypi: Response URL https://pypi.org/simple/joblib/ differs from request URL https://pypi.cnstrc.com/simple/joblib/
cnstrc_pypi: 7 packages found for joblib >=0.14.1
PyPI: Getting info for joblib (1.0.1) from PyPI
PyPI: No dependencies found, downloading archives
PyPI: Downloading wheel: joblib-1.0.1-py3-none-any.whl
   1: selecting joblib (1.0.1)

I looks like poetry doesn't respect secondary = true flag in custom pypi configuration.

Changing config to

[[tool.poetry.source]]
name = "cnstrc_pypi"
url = "https://pypi.cnstrc.com/simple/"
secondary = true


[[tool.poetry.source]]
name = "pypi-public"
url = "https://pypi.org/simple/"

as was suggested above doesn't help.

[judahrand]

I'm also finding this is an issue...

[andras-kth]

@jleclanche

The above happens even if source="pypi" is specified on every non-private dependency.

Are you sure? What I observe is that those dependencies where source = "pypi" is specified explicitly are not requested from the alternate repo, but any implicit dependencies (i.e. packages that these explicit dependencies depend on) don't inherit this source designation and those (but only those) are looked up on the alternate index.

BUT, I do think that explicitly specifying the "default" source should NOT be necessary in either case.

I think the culprit is this loop

        packages = []
        for repo in self._repositories:
            packages += repo.find_packages(dependency)

where packages are collected from every registered repo without regard to priorities.
If I'm reading the code correctly self._repositories is already in priority order.
If that's correct, then taking the first match instead might be the desired behavior:

         packages = []
         for repo in self._repositories:
-            packages += repo.find_packages(dependency)
+            packages = repo.find_packages(dependency)
+            if packages:
+                break

Unfortunately, this could produce incorrect results due to what I'd consider a bug
in version matching (#4729) (i.e. in some cases the first "match" doesn't actually match);
so, the issue may, in fact, be somewhat more complicated.