[ros] install dedicated deb to setup apt gpg key instead of installing keys the old way #19162

mikaelarguedas · 2025-05-31T09:55:52Z

The ROS package GPG keys are now managed by a dedicated package: https://discourse.ros.org/t/ros-signing-key-migration-guide/43937/4

The GPG expires on June 1st so this PR is needed for images to keep building.

mikaelarguedas · 2025-05-31T10:03:33Z

The humble and jazzy CI failures seem due to the official ubuntu apt repo being flaky and not the changes introduced in this PR:

E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/liby/libyaml/libyaml-dev_0.2.5-1build1_amd64.deb  403  Forbidden [IP: 185.125.190.83 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/libu/liburcu/liburcu-dev_0.14.0-3.1build1_amd64.deb  403  Forbidden [IP: 185.125.190.83 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/c/cfitsio/libcfitsio-dev_4.0.0-1_amd64.deb  403  Forbidden [IP: 185.125.190.82 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/c/charls/libcharls2_2.3.4-1_amd64.deb  403  Forbidden [IP: 185.125.190.82 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/c/charls/libcharls-dev_2.3.4-1_amd64.deb  403  Forbidden [IP: 185.125.190.82 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/main/c/colord/libcolord2_1.4.6-1_amd64.deb  403  Forbidden [IP: 185.125.190.82 80]
E: Failed to fetch http://archive.ubuntu.com/ubuntu/pool/universe/d/dav1d/libdav1d-dev_0.9.2-1_amd64.deb  403  Forbidden [IP: 185.125.190.82 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?

tfoote · 2025-05-31T10:04:51Z

Thanks for working with me @mikaelarguedas to get this up update out.

The CI failures here look to be the same intermittent failures of archive.ubuntu.com giving 403 errors. Our internal test builds took a few retires: osrf/docker_images#805 and and we've been seeing this across much of our CI recently. I suspect that a few retires will pass those builds as well.

As this is replacing our expiring gpg key a prompt merge would be appreciated. I know that's tough with the intermittent CI problems.

Signed-off-by: Mikael Arguedas <[email protected]>

mikaelarguedas · 2025-06-02T14:32:45Z

I forced pushed the same content to retrigger CI now that ubuntu repos seem more stable. CI is now green 👍

yosifkit · 2025-06-02T19:31:04Z

# NOTE: How do we break cache and ensure rebuild if that version changes ?
RUN export ROS_APT_SOURCE_VERSION=$(curl -s https://api.github.com/repos/ros-infrastructure/ros-apt-source/releases/latest | grep -F "tag_name" | awk -F\" '{print $4}') ;\
    curl -L -s -o /tmp/ros2-apt-source.deb "https://github.com/ros-infrastructure/ros-apt-source/releases/download/${ROS_APT_SOURCE_VERSION}/ros2-apt-source_${ROS_APT_SOURCE_VERSION}.$(. /etc/os-release && echo $VERSION_CODENAME)_all.deb" \

Downloading and installing a deb without any verification is not acceptable (this is basically curl | bash). This needs to have some sort of verification like a signature and/or a checksum embedded in the Dockerfile (https://github.com/docker-library/official-images/blob/2835b6219fbc7a06fffe5e8ce04db0b9138fc228/README.md#image-build).

Something a little like this:

-# NOTE: How do we break cache and ensure rebuild if that version changes ?
-RUN export ROS_APT_SOURCE_VERSION=$(curl -s https://api.github.com/repos/ros-infrastructure/ros-apt-source/releases/latest | grep -F "tag_name" | awk -F\" '{print $4}') ;\
-    curl -L -s -o /tmp/ros2-apt-source.deb "https://github.com/ros-infrastructure/ros-apt-source/releases/download/${ROS_APT_SOURCE_VERSION}/ros2-apt-source_${ROS_APT_SOURCE_VERSION}.$(. /etc/os-release && echo $VERSION_CODENAME)_all.deb" \
+RUN export ROS_APT_SOURCE_VERSION='1.1.0' \
+    && . /etc/os-release \
+    && curl -L -s -o /tmp/ros2-apt-source.deb "https://github.com/ros-infrastructure/ros-apt-source/releases/download/${ROS_APT_SOURCE_VERSION}/ros2-apt-source_${ROS_APT_SOURCE_VERSION}.${VERSION_CODENAME}_all.deb" \
+    && export SHA256CHECKSUM="ABCDEF1234567890" \
+    && echo "$SHA256CHECKSUM */tmp/ros2-apt-source.deb" | sha256sum --strict --check \

# alternative with semicolons
+RUN set -e; \
+    export ROS_APT_SOURCE_VERSION='1.1.0'; \
+    . /etc/os-release; \
+    curl -L -s -o /tmp/ros2-apt-source.deb "https://github.com/ros-infrastructure/ros-apt-source/releases/download/${ROS_APT_SOURCE_VERSION}/ros2-apt-source_${ROS_APT_SOURCE_VERSION}.${VERSION_CODENAME}_all.deb"; \
+    export SHA256CHECKSUM="ABCDEF1234567890"; \
+    echo "$SHA256CHECKSUM */tmp/ros2-apt-source.deb" | sha256sum --strict --check; \

tianon · 2025-06-02T21:41:48Z

I would also add that we'd prefer if you avoid hitting api.github.com during build -- if you need that, I'd suggest doing that in your own CI and embedding the result via templating in the Dockerfile directly.

ruffsl · 2025-06-02T22:01:38Z

I'd suggest doing that in your own CI and embedding the result via templating in the Dockerfile directly.

Good point! Probably an appropriate location to compute and embed the checksum too.

tfoote · 2025-06-02T23:20:26Z

We'd flagged to add the version in our generator. I've started updating it to embed the version as well as the checksum and I can embed the distro as well while I'm at it.

…alidation to be fixed in the Dockerfile Signed-off-by: Tully Foote <[email protected]>

github-actions · 2025-06-03T05:08:42Z

Diff for a5ce8be:

diff --git a/_bashbrew-cat b/_bashbrew-cat
index 692dfd2..b475388 100644
--- a/_bashbrew-cat
+++ b/_bashbrew-cat
@@ -13,7 +13,7 @@ Directory: ros/humble/ubuntu/jammy/ros-base
 
 Tags: humble-ros-core, humble-ros-core-jammy
 Architectures: amd64, arm64v8
-GitCommit: b525e9ef659ce448db6150fd5407ef62b2c5b265
+GitCommit: eb5634cf92ba079897e44fb7541d3b78aa6cf717
 Directory: ros/humble/ubuntu/jammy/ros-core
 
 Tags: jazzy-perception, jazzy-perception-noble
@@ -28,7 +28,7 @@ Directory: ros/jazzy/ubuntu/noble/ros-base
 
 Tags: jazzy-ros-core, jazzy-ros-core-noble
 Architectures: amd64, arm64v8
-GitCommit: b525e9ef659ce448db6150fd5407ef62b2c5b265
+GitCommit: eb5634cf92ba079897e44fb7541d3b78aa6cf717
 Directory: ros/jazzy/ubuntu/noble/ros-core
 
 Tags: kilted-perception, kilted-perception-noble
@@ -43,7 +43,7 @@ Directory: ros/kilted/ubuntu/noble/ros-base
 
 Tags: kilted-ros-core, kilted-ros-core-noble
 Architectures: amd64, arm64v8
-GitCommit: b835a530495c0b411a0d15db858710a2748ee0a0
+GitCommit: eb5634cf92ba079897e44fb7541d3b78aa6cf717
 Directory: ros/kilted/ubuntu/noble/ros-core
 
 Tags: noetic-perception, noetic-perception-focal
@@ -78,5 +78,5 @@ Directory: ros/rolling/ubuntu/noble/ros-base
 
 Tags: rolling-ros-core, rolling-ros-core-noble
 Architectures: amd64, arm64v8
-GitCommit: 2e9b4e3c02bff2c70271e6f0fd15d4132e8cbcbb
+GitCommit: eb5634cf92ba079897e44fb7541d3b78aa6cf717
 Directory: ros/rolling/ubuntu/noble/ros-core
diff --git a/ros_humble-ros-core-jammy/Dockerfile b/ros_humble-ros-core-jammy/Dockerfile
index 8f6138d..c0e6b9c 100644
--- a/ros_humble-ros-core-jammy/Dockerfile
+++ b/ros_humble-ros-core-jammy/Dockerfile
@@ -11,22 +11,20 @@ RUN echo 'Etc/UTC' > /etc/timezone && \
 
 # install packages
 RUN apt-get update && apt-get install -q -y --no-install-recommends \
+    ca-certificates \
+    curl \
     dirmngr \
     gnupg2 \
     && rm -rf /var/lib/apt/lists/*
 
-# setup keys
-RUN set -eux; \
-       key='C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654'; \
-       export GNUPGHOME="$(mktemp -d)"; \
-       gpg --batch --keyserver keyserver.ubuntu.com --recv-keys "$key"; \
-       mkdir -p /usr/share/keyrings; \
-       gpg --batch --export "$key" > /usr/share/keyrings/ros2-latest-archive-keyring.gpg; \
-       gpgconf --kill all; \
-       rm -rf "$GNUPGHOME"
-
-# setup sources.list
-RUN echo "deb [ signed-by=/usr/share/keyrings/ros2-latest-archive-keyring.gpg ] http://packages.ros.org/ros2/ubuntu jammy main" > /etc/apt/sources.list.d/ros2-latest.list
+
+# Setup ROS Apt sources
+RUN curl -L -s -o /tmp/ros2-apt-source.deb https://github.com/ros-infrastructure/ros-apt-source/releases/download/1.1.0/ros2-apt-source_1.1.0.jammy_all.deb \
+    && echo "1600cb8cc28258a39bffc1736a75bcbf52d1f2db371a4d020c1b187d2a5a083b /tmp/ros2-apt-source.deb" | sha256sum --strict --check \
+    && apt-get update \
+    && apt-get install /tmp/ros2-apt-source.deb \
+    && rm -f /tmp/ros2-apt-source.deb \
+    && rm -rf /var/lib/apt/lists/*
 
 # setup environment
 ENV LANG=C.UTF-8
diff --git a/ros_jazzy-ros-core-noble/Dockerfile b/ros_jazzy-ros-core-noble/Dockerfile
index 397b97a..5e28f03 100644
--- a/ros_jazzy-ros-core-noble/Dockerfile
+++ b/ros_jazzy-ros-core-noble/Dockerfile
@@ -11,22 +11,20 @@ RUN echo 'Etc/UTC' > /etc/timezone && \
 
 # install packages
 RUN apt-get update && apt-get install -q -y --no-install-recommends \
+    ca-certificates \
+    curl \
     dirmngr \
     gnupg2 \
     && rm -rf /var/lib/apt/lists/*
 
-# setup keys
-RUN set -eux; \
-       key='C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654'; \
-       export GNUPGHOME="$(mktemp -d)"; \
-       gpg --batch --keyserver keyserver.ubuntu.com --recv-keys "$key"; \
-       mkdir -p /usr/share/keyrings; \
-       gpg --batch --export "$key" > /usr/share/keyrings/ros2-latest-archive-keyring.gpg; \
-       gpgconf --kill all; \
-       rm -rf "$GNUPGHOME"
-
-# setup sources.list
-RUN echo "deb [ signed-by=/usr/share/keyrings/ros2-latest-archive-keyring.gpg ] http://packages.ros.org/ros2/ubuntu noble main" > /etc/apt/sources.list.d/ros2-latest.list
+
+# Setup ROS Apt sources
+RUN curl -L -s -o /tmp/ros2-apt-source.deb https://github.com/ros-infrastructure/ros-apt-source/releases/download/1.1.0/ros2-apt-source_1.1.0.noble_all.deb \
+    && echo "35441f3092fd05773a3c397fab38661bec466584c7a1f1c05366579997cb5fe7 /tmp/ros2-apt-source.deb" | sha256sum --strict --check \
+    && apt-get update \
+    && apt-get install /tmp/ros2-apt-source.deb \
+    && rm -f /tmp/ros2-apt-source.deb \
+    && rm -rf /var/lib/apt/lists/*
 
 # setup environment
 ENV LANG=C.UTF-8
diff --git a/ros_kilted-ros-core-noble/Dockerfile b/ros_kilted-ros-core-noble/Dockerfile
index b650ebe..b78ebd3 100644
--- a/ros_kilted-ros-core-noble/Dockerfile
+++ b/ros_kilted-ros-core-noble/Dockerfile
@@ -11,22 +11,20 @@ RUN echo 'Etc/UTC' > /etc/timezone && \
 
 # install packages
 RUN apt-get update && apt-get install -q -y --no-install-recommends \
+    ca-certificates \
+    curl \
     dirmngr \
     gnupg2 \
     && rm -rf /var/lib/apt/lists/*
 
-# setup keys
-RUN set -eux; \
-       key='C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654'; \
-       export GNUPGHOME="$(mktemp -d)"; \
-       gpg --batch --keyserver keyserver.ubuntu.com --recv-keys "$key"; \
-       mkdir -p /usr/share/keyrings; \
-       gpg --batch --export "$key" > /usr/share/keyrings/ros2-latest-archive-keyring.gpg; \
-       gpgconf --kill all; \
-       rm -rf "$GNUPGHOME"
-
-# setup sources.list
-RUN echo "deb [ signed-by=/usr/share/keyrings/ros2-latest-archive-keyring.gpg ] http://packages.ros.org/ros2/ubuntu noble main" > /etc/apt/sources.list.d/ros2-latest.list
+
+# Setup ROS Apt sources
+RUN curl -L -s -o /tmp/ros2-apt-source.deb https://github.com/ros-infrastructure/ros-apt-source/releases/download/1.1.0/ros2-apt-source_1.1.0.noble_all.deb \
+    && echo "35441f3092fd05773a3c397fab38661bec466584c7a1f1c05366579997cb5fe7 /tmp/ros2-apt-source.deb" | sha256sum --strict --check \
+    && apt-get update \
+    && apt-get install /tmp/ros2-apt-source.deb \
+    && rm -f /tmp/ros2-apt-source.deb \
+    && rm -rf /var/lib/apt/lists/*
 
 # setup environment
 ENV LANG=C.UTF-8
diff --git a/ros_rolling-ros-core-noble/Dockerfile b/ros_rolling-ros-core-noble/Dockerfile
index d2ed728..e905e61 100644
--- a/ros_rolling-ros-core-noble/Dockerfile
+++ b/ros_rolling-ros-core-noble/Dockerfile
@@ -11,22 +11,20 @@ RUN echo 'Etc/UTC' > /etc/timezone && \
 
 # install packages
 RUN apt-get update && apt-get install -q -y --no-install-recommends \
+    ca-certificates \
+    curl \
     dirmngr \
     gnupg2 \
     && rm -rf /var/lib/apt/lists/*
 
-# setup keys
-RUN set -eux; \
-       key='C1CF6E31E6BADE8868B172B4F42ED6FBAB17C654'; \
-       export GNUPGHOME="$(mktemp -d)"; \
-       gpg --batch --keyserver keyserver.ubuntu.com --recv-keys "$key"; \
-       mkdir -p /usr/share/keyrings; \
-       gpg --batch --export "$key" > /usr/share/keyrings/ros2-latest-archive-keyring.gpg; \
-       gpgconf --kill all; \
-       rm -rf "$GNUPGHOME"
-
-# setup sources.list
-RUN echo "deb [ signed-by=/usr/share/keyrings/ros2-latest-archive-keyring.gpg ] http://packages.ros.org/ros2/ubuntu noble main" > /etc/apt/sources.list.d/ros2-latest.list
+
+# Setup ROS Apt sources
+RUN curl -L -s -o /tmp/ros2-apt-source.deb https://github.com/ros-infrastructure/ros-apt-source/releases/download/1.1.0/ros2-apt-source_1.1.0.noble_all.deb \
+    && echo "35441f3092fd05773a3c397fab38661bec466584c7a1f1c05366579997cb5fe7 /tmp/ros2-apt-source.deb" | sha256sum --strict --check \
+    && apt-get update \
+    && apt-get install /tmp/ros2-apt-source.deb \
+    && rm -f /tmp/ros2-apt-source.deb \
+    && rm -rf /var/lib/apt/lists/*
 
 # setup environment
 ENV LANG=C.UTF-8

Relevant Maintainers:

ros: @tfoote

tfoote · 2025-06-03T05:23:00Z

I've updated the gpg key setup commands to pull the binaries directly by full url and embed the sha256 into the Dockerfile.

Noble example:

# Setup ROS Apt sources
RUN curl -L -s -o /tmp/ros2-apt-source.deb https://github.com/ros-infrastructure/ros-apt-source/releases/download/1.1.0/ros2-apt-source_1.1.0.noble_all.deb \
    && echo "35441f3092fd05773a3c397fab38661bec466584c7a1f1c05366579997cb5fe7 /tmp/ros2-apt-source.deb" | sha256sum --strict --check \
    && apt-get update \
    && apt-get install /tmp/ros2-apt-source.deb \
    && rm -f /tmp/ros2-apt-source.deb \
    && rm -rf /var/lib/apt/lists/*

The failing tests above are our release that's just gone EOL over the weekend with Ubuntu. And we'll be moving it to a snapshot repository for legacy use. I saw the notice here: #19167

We will clear the focal usage out, but I'd like to not block getting the other platforms operational.

The GPG key inside the ROS:noetic images experied on 2025-6-1 This meant the images could not use apt anymore, failing their builds. Hotfix for this was copied over from here: osrf/docker_images#807 The actual fix is being worked on here: docker-library/official-images#19162 This commit can be reverted once the issue is solved upstream.

ruffsl · 2025-06-03T17:25:06Z

@yosifkit or @tianon , from the tag view on Docker Hub it looks like the images have been rebuilt. However, when inspecting the jenkins, the build history looks stale? 7mo old?

https://doi-janky.infosiftr.net/job/multiarch/view/images/view/ros/builds

mikaelarguedas · 2025-06-03T17:46:00Z

However, when inspecting the jenkins, the build history looks stale? 7mo old?

I stumbled upon that recently, they look to now be jobs of a "meta/arch" workflow: https://doi-janky.infosiftr.net/job/meta/job/arm64v8/job/build/76569/

tianon · 2025-06-03T18:50:25Z

Yep, see docker-library/faq#41 (which updated https://github.com/docker-library/faq#an-images-source-changed-in-git-now-what). 👍

mikaelarguedas requested a review from a team as a code owner May 31, 2025 09:55

github-actions bot added the library/ros label May 31, 2025

mikaelarguedas mentioned this pull request May 31, 2025

Update key setup system osrf/docker_images#805

Merged

This comment has been minimized.

Sign in to view

mathias-luedtke mentioned this pull request Jun 1, 2025

ROS signing key expiration ros-industrial/industrial_ci#910

Closed

sea-bass mentioned this pull request Jun 2, 2025

Build failing as GPG key expired sea-bass/pyrobosim#359

Closed

Timple mentioned this pull request Jun 2, 2025

docker images use invalid key (expired 2025-06-01): humble-ros-base, kilted-ros-base, rolling-ros-base, ... osrf/docker_images#807

Closed

rhaschke mentioned this pull request Jun 2, 2025

Pr humble python bindings moveit/moveit2#3487

Open

6 tasks

[ros] use apt key debs instead of installing keys the old way

2aef623

Signed-off-by: Mikael Arguedas <[email protected]>

mikaelarguedas force-pushed the ros_apt_key branch from 8c9861a to 2aef623 Compare June 2, 2025 14:24

This comment has been minimized.

Sign in to view

christophebedard mentioned this pull request Jun 2, 2025

Can't install docker. Key is expired ros2/ros2#1696

Closed

matlabbe mentioned this pull request Jun 2, 2025

CI failing (ROS build): The following signatures were invalid: EXPKEYSIG F42ED6FBAB17C654 introlab/rtabmap#1523

Closed

tfoote mentioned this pull request Jun 2, 2025

Remove Iron as EOL tfoote/ros_interface_proto_builder#16

Merged

tfoote mentioned this pull request Jun 3, 2025

Embedding version hash and distro in string osrf/docker_templates#117

Merged

Reverting noetic change and switching the debian fetch and checksum v…

a5ce8be

…alidation to be fixed in the Dockerfile Signed-off-by: Tully Foote <[email protected]>

GuillaumeLaine mentioned this pull request Jun 3, 2025

fix(tmp): replace expired ros2 apt repo key Auterion/ros-debian-workflow#14

Merged

JJWRoeloffs mentioned this pull request Jun 3, 2025

Hotfix: Fix issue with the ros image not building ci-group/learning_machines_robobo#35

Merged

yosifkit approved these changes Jun 3, 2025

View reviewed changes

yosifkit merged commit 596d9d6 into docker-library:master Jun 3, 2025
8 of 10 checks passed

tfoote deleted the ros_apt_key branch June 3, 2025 16:14

mikaelarguedas mentioned this pull request Jun 3, 2025

Remove noetic from automated build to allow rebuilding all non EOL ROS 2 images osrf/docker_images#815

Merged

mojomex mentioned this pull request Jun 4, 2025

chore: revert "fix(ci): temporary workaround for expired ROS signing key (#331)" tier4/nebula#332

Merged

jmatejcz mentioned this pull request Jun 4, 2025

fix: remove ros2 key fixes RobotecAI/rai#618

Merged

alireza-moayyedi mentioned this pull request Jun 4, 2025

Cannot pull ros docker images: error pulling image configuration osrf/docker_images#816

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

[ros] install dedicated deb to setup apt gpg key instead of installing keys the old way #19162

[ros] install dedicated deb to setup apt gpg key instead of installing keys the old way #19162

Uh oh!

mikaelarguedas commented May 31, 2025

Uh oh!

This comment has been minimized.

mikaelarguedas commented May 31, 2025

Uh oh!

tfoote commented May 31, 2025

Uh oh!

This comment has been minimized.

mikaelarguedas commented Jun 2, 2025

Uh oh!

yosifkit commented Jun 2, 2025

Uh oh!

tianon commented Jun 2, 2025

Uh oh!

ruffsl commented Jun 2, 2025

Uh oh!

tfoote commented Jun 2, 2025

Uh oh!

github-actions bot commented Jun 3, 2025

Uh oh!

tfoote commented Jun 3, 2025

Uh oh!

Uh oh!

ruffsl commented Jun 3, 2025

Uh oh!

mikaelarguedas commented Jun 3, 2025 •

edited

Loading

Uh oh!

tianon commented Jun 3, 2025

Uh oh!

Uh oh!

[ros] install dedicated deb to setup apt gpg key instead of installing keys the old way #19162

[ros] install dedicated deb to setup apt gpg key instead of installing keys the old way #19162

Uh oh!

Conversation

mikaelarguedas commented May 31, 2025

Uh oh!

This comment has been minimized.

mikaelarguedas commented May 31, 2025

Uh oh!

tfoote commented May 31, 2025

Uh oh!

This comment has been minimized.

mikaelarguedas commented Jun 2, 2025

Uh oh!

yosifkit commented Jun 2, 2025

Uh oh!

tianon commented Jun 2, 2025

Uh oh!

ruffsl commented Jun 2, 2025

Uh oh!

tfoote commented Jun 2, 2025

Uh oh!

github-actions bot commented Jun 3, 2025

Uh oh!

tfoote commented Jun 3, 2025

Uh oh!

Uh oh!

ruffsl commented Jun 3, 2025

Uh oh!

mikaelarguedas commented Jun 3, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

tianon commented Jun 3, 2025

Uh oh!

Uh oh!

mikaelarguedas commented Jun 3, 2025 •

edited

Loading