Skip to content

grpc: set gRPC_DOWNLOAD_ARCHIVES to false #23153

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Mar 25, 2025

Conversation

js-nano
Copy link
Contributor

@js-nano js-nano commented Mar 18, 2024

The gRPC CMake configuration step can fail to due to a failure when downloading third party archives. These failures are expected, and the user is instructed to retry. So, if configuration fails, retry it

Also set the gRPC_DOWNLOAD_ARCHIVES CMake flag to False, to avoid downloading these unnecessary archives in the first place, in versions of gRPC that understand the flag

Specify library name and version: grpc/all

This fixes #23094


@conan-center-bot

This comment has been minimized.

@conan-center-bot

This comment has been minimized.

@conan-center-bot conan-center-bot added Failed Missing dependencies Build failed due missing dependencies in Conan Center and removed Service Under Maintenance labels Jun 12, 2024
@conan-center-bot
Copy link
Collaborator

Conan v1 pipeline ✔️

All green in build 4 (0ac619c39a7e119efdfa3a75546aefee41f4113c):

  • grpc/1.50.0:
    All packages built successfully! (All logs)

  • grpc/1.54.3:
    All packages built successfully! (All logs)

  • grpc/1.50.1:
    All packages built successfully! (All logs)

  • grpc/1.48.4:
    All packages built successfully! (All logs)


Conan v2 pipeline ❌

Note: Conan v2 builds are now mandatory. Please read our discussion about it.

The v2 pipeline failed. Please, review the errors and note this is required for pull requests to be merged. In case this recipe is still not ported to Conan 2.x, please, ping @conan-io/barbarians on the PR and we will help you.

See details:

Failure in build 5 (0ac619c39a7e119efdfa3a75546aefee41f4113c):

  • grpc/1.50.1:
    Didn't run or was cancelled before finishing

  • grpc/1.50.0:
    Didn't run or was cancelled before finishing

  • grpc/1.54.3:
    CI failed to create some packages (All logs)

    Logs for packageID 2cf586938382402f8da34d74e46850ff4e56c618:
    [settings]
    arch=x86_64
    build_type=Release
    compiler=gcc
    compiler.cppstd=17
    compiler.libcxx=libstdc++11
    compiler.version=11
    os=Linux
    [options]
    */*:shared=True
    
    [...]
        protobuf/3.21.12#2070373d5f1a2606a4491faef9315a73:10670e90b03eb1607902ed190ae2fecdc52d69bb#eff9a8c0282afb8c3fb04152953185c8 - Download (conan-center)
    Skipped binaries
        abseil/20230125.3, autoconf/2.71, automake/1.16.5, bzip2/1.0.8, c-ares/1.19.1, flex/2.6.4, gnu-config/cci.20210814, gperf/3.1, libcap/2.69, libmount/2.39.2, libselinux/3.6, libsystemd/255, libtool/2.4.7, libxcrypt/4.4.36, lz4/1.9.4, m4/1.4.19, meson/1.2.2, meson/1.4.0, ninja/1.11.1, openssl/3.2.2, pcre2/10.42, pkgconf/2.1.0, pkgconf/2.2.0, re2/20230301, xz_utils/5.4.5, zlib/1.3.1, zstd/1.5.5
    ERROR: Missing binary: grpc/1.54.3:8cc2949c5b31e1cbd41f8dcef63fdab0744e62fa
    
    grpc/1.54.3: WARN: Can't find a 'grpc/1.54.3' package binary '8cc2949c5b31e1cbd41f8dcef63fdab0744e62fa' for the configuration:
    [settings]
    arch=x86_64
    build_type=Release
    compiler=gcc
    compiler.cppstd=17
    compiler.libcxx=libstdc++11
    compiler.version=11
    os=Linux
    [options]
    codegen=True
    cpp_plugin=True
    csharp_ext=False
    csharp_plugin=True
    fPIC=True
    node_plugin=True
    objective_c_plugin=True
    php_plugin=True
    python_plugin=True
    ruby_plugin=True
    shared=False
    [requires]
    abseil/20230125.3.Z
    bzip2/1.0.Z
    c-ares/1.19.Z
    libcap/2.69.Z
    libmount/2.39.Z
    libselinux/3.6.Z
    libsystemd/255.0.Z
    libxcrypt/4.4.Z
    lz4/1.9.Z
    openssl/3.2.Z
    pcre2/10.42.Z
    protobuf/3.21.Z
    re2/20230301.0.Z
    xz_utils/5.4.Z
    zlib/1.3.Z
    zstd/1.5.Z
    
    ERROR: Missing prebuilt package for 'grpc/1.54.3'. You can try:
        - List all available packages using 'conan list grpc/1.54.3:* -r=remote'
        - Explain missing binaries: replace 'conan install ...' with 'conan graph explain ...'
        - This is a **test_package** missing binary. You can use --build (for all dependencies) or --build-test (exclusive for 'test_package' dependencies) to define what can be built from sources
    
    More Info at 'https://docs.conan.io/2/knowledge/faq.html#error-missing-prebuilt-package'
    
  • grpc/1.48.4:
    Didn't run or was cancelled before finishing


Note: To save resources, CI tries to finish as soon as an error is found. For this reason you might find that not all the references have been launched or not all the configurations for a given reference. Also, take into account that we cannot guarantee the order of execution as it depends on CI workload and workers availability.

@@ -174,6 +174,11 @@ def generate(self):
tc.cache_variables["gRPC_BUILD_GRPC_PYTHON_PLUGIN"] = self.options.python_plugin
tc.cache_variables["gRPC_BUILD_GRPC_RUBY_PLUGIN"] = self.options.ruby_plugin

# Never download unnecessary archives
# (supported in gRPC >= 1.62.0)
tc.cache_variables["gRPC_DOWNLOAD_ARCHIVES"] = False
Copy link
Contributor

@samuel-emrys samuel-emrys Jun 15, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not particularly familiar with gRPC - so to confirm, these archives aren't required? I agree that a conan recipe should never attempt to download external files, but if they are required then they should be added to conandata.yml and downloaded/extracted in the source() method.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Based on the issue where the flag was introduced, it seems that they're not required for anything directly related to building or testing gRPC
grpc/grpc#34587 (comment)

In addition, for all but one of the archives, no attempt is made to download the archives -- the CMake script checks for the presence of directories that are (in most cases) already present in the Conan source checkout.

@samuel-emrys
Copy link
Contributor

@RubenRBS looks like the v2 pipeline is failing on a weird error - it can't find itself as a dependency?

@samuel-emrys
Copy link
Contributor

@js-nano from @RubenRBS on slack, to get the conan 2 pipeline passing you'll need to simplify the test_package so that self.tested_reference_str is only requested as a requires(), and not a tool_requires(), i.e.

    def build_requirements(self):
        if not self._is_legacy_one_profile:
            # For the grpc-cpp-plugin executable at build time
-           self.tool_requires(self.tested_reference_str)
            self.tool_requires("protobuf/<host_version>")

This is due to a limitation in the existing CI infrastructure

Copy link
Contributor

This pull request has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

@github-actions github-actions bot added the stale label Mar 22, 2025
js-nano and others added 3 commits March 25, 2025 09:06
The gRPC CMake configuration step can fail to due to a failure when downloading third party archives.  These failures are expected, and the user is instructed to retry.  So, if configuration fails, retry it

Also set the `gRPC_DOWNLOAD_ARCHIVES` CMake flag to `False`, to avoid downloading these unnecessary archives in the first place, in versions of gRPC that understand the flag
@jcar87 jcar87 force-pushed the grpc_retry_configuration branch from 8beb66b to 049e02e Compare March 25, 2025 09:07
@jcar87 jcar87 removed Failed stale Missing dependencies Build failed due missing dependencies in Conan Center labels Mar 25, 2025
@jcar87 jcar87 changed the title grpc: Retry CMake configuration if it fails grpc: set gRPC_DOWNLOAD_ARCHIVES to false Mar 25, 2025
Copy link
Contributor

@jcar87 jcar87 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM - we've only kept the passing of gRPC_DOWNLOAD_ARCHIVES to False

A try/except pattern is not really acceptable (and it really does complicate build logs) - I would strongly advise reporting this upstream if the behaviour is flaky (otherwise it never gets fixed!) - on the other hand, for older versions of grpc where this is still a problem, we would accept a PR that fully inhibits the behaviour and does not attempt to download anything that we know is not needed. But this appears to be addressed for grpc >=1.62.0

@jcar87 jcar87 merged commit 199db04 into conan-io:master Mar 25, 2025
10 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

[package] grpc/all: configuration is designed to possibly fail, but succeed on a retry
6 participants