[bug] ERROR: Build-require 'xxx' cannot be found in lockfile (but it's there)

[SSE4]

Environment Details (include every applicable attribute)

• Operating System+version: Linux, docker image conanio/gcc11
• Compiler+version: gcc (Ubuntu 11.1.0-1ubuntu1~20.04) 11.1.0
• Conan version: Conan version 1.44.0
• Python version: Python 3.7.5

Steps to reproduce (Include if Applicable)

run conanio/gcc11 image. execute the following command to create a lockfile:

$ conan lock create --reference gst-plugins-good/1.19.1@ --build -pr:b default -pr:h default

(two profiles are optional, it reproduces in single profile mode as well)

run the command to install:

$ conan install gst-plugins-good/1.19.1@ --lockfile=conan.lock --build missing

it will say:

ERROR: Build-require 'automake' cannot be found in lockfile

but it's clearly there:

$ cat conan.lock  | grep automake
    "ref": "automake/1.16.4",
    "ref": "automake/1.16.4",

(also the full lockfile is attached below)

Logs (Executed commands with output) (Include/Attach if Applicable)

conan install gst-plugins-good/1.19.1@ --lockfile=conan.lock --build missing
Using lockfile: '/home/conan/conan.lock'
Configuration (profile_host):
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=11
os=Linux
os_build=Linux
[options]
[build_requires]
[env]

Configuration (profile_build):
[settings]
arch=x86_64
arch_build=x86_64
build_type=Release
compiler=gcc
compiler.libcxx=libstdc++
compiler.version=11
os=Linux
os_build=Linux
[options]
[build_requires]
[env]

ERROR: Build-require 'automake' cannot be found in lockfile

conan.lock.txt

[SSE4]

@memsharded any ideas what could be wrong here?

[memsharded]

I have had a look.
The problem seems to be when a recipe requires the same package both as a requires and a build_requires. In this case, why is libtool having a regular requires to automake? Is automake some kind of library that that is being linked into libtool?

I have checked that if:

diff --git a/recipes/libtool/all/conanfile.py b/recipes/libtool/all/conanfile.py
index b20875b66..7e254a242 100644
--- a/recipes/libtool/all/conanfile.py
+++ b/recipes/libtool/all/conanfile.py
@@ -43,15 +43,15 @@ class LibtoolConan(ConanFile):
         del self.settings.compiler.libcxx
         del self.settings.compiler.cppstd

-    def requirements(self):
-        self.requires("automake/1.16.4")
-
     @property
     def _settings_build(self):
         return getattr(self, "settings_build", self.settings)

It seems to work past that error (and still fail in a similar one with m4, that I suspect it has the same origin).

I will try to keep investigating why this happen (seems complex to solve, as it is internal to the graph resolution algorithm), I suggest trying to simplify that part of the recipes if possible.

[SSE4]

thanks for taking a look! AFAIK, automake is just a collection of scripts, and has no C++ source code.
there was a related issue: conan-io/conan-center-index#8157
/cc @madebr , maybe he knows if requirement of automake in libtool is redundant

[madebr]

libtool can be used in 2 forms:

• as a requirement providing the libltdl library
• as a build requirement providing scripts for autotools

automake is needed as a build requirement when building automake ==> added to build_requirements

When libtool is used as a build_requirement, it also needs automake (and autoconf) added as a build requirement. ==> added to requirements
We added the automake version to libtool so as to define the automake version in one place only.
This also avoid potential problems of building libtool with automake version X.Y.Z and using libtool with automake version A.B.X.

[Hopobcn]

This issue looks similar to #10345

[SSE4]

to conclude, it's fine if libtool is used in both requires and build_requires simultaneously. and we have many more cases like that, most famous is probably protobuf. so it's not something we can apply workaround for, it deserves a proper fix in the client.

[anton-danielsson]

We are having the same issue. This means we can't use lock files. It's protobuf that is our culprit.

[bukulin]

Similar issue here with cmake. It is not linked, just a tool_requirement. To reproduce the error message I've created a fork of the conan examples and tweaked it a little:
conan-io/examples@master...bukulin:conan_examples:master

libb and app1 both depend on cmake as a tool_requirement. That was enough to have the same error message as we can see in the title of this issue, but I think the reason was slightly different. Shall I open a new issue for that?

[harendt]

I'm observing the same issue. In my case it's protobuf and grpc that are needed both, as requires and tool_requires.

Is there any known workaround? Removing the tool_requires won't work, because the tool_requires package must be x86_64 and the requires package must be aarch64.

[memsharded]

When a tool-requires is missing in a lockfile, the general approach is to make sure that --build=xxx is provided in the conan lock create .... --build=xxxx to make sure that the graph expands and include the tool_requires that will be necessary later for building. Please try that.

[bukulin]

Actually the tool is listed in the lockfile even if I do not add --build=xxx.

Let's suppose that tool requirement is cmake for brevity. If I do conan lock create ... --build=cmake then the only change is that conan lock build-order lists cmake in the first place to build, but further rebuilds still missing cmake from the lockfile.

[technic]

The only workaround I found is installing some random tools like cmake, autoconf etc, and at some point ther problem was gone. But this I can not do in CI