fix(redhat): Also try to find buildinfo in root layer (layer 0) #8924

Romain-Geissler-1A · 2025-05-26T23:15:38Z

Description

This is a followup of #8910 where @knqyf263 rightfully pointed out that we shall not need a default content set for recent RHEL. Indeed there was another bug. This is required for future RHEL 10 support, yet RHEL 10 support is still partial.

Let's consider this Dockerfile:

[root@05f6cdd76f60 rhel10]# cat Dockerfile
FROM registry.access.redhat.com/ubi10/ubi:latest

RUN dnf install -y procps

We can build it, and look at the history of this image like this:

[root@05f6cdd76f60 rhel10]# docker build .                                                                                                                                                                            [+] Building 0.1s (6/6) FINISHED                                                                                                                                                                       docker:default
 => [internal] load build definition from Dockerfile                                                                                                                                                             0.0s
 => => transferring dockerfile: 113B                                                                                                                                                                             0.0s
 => [internal] load metadata for registry.access.redhat.com/ubi10/ubi:latest                                                                                                                                     0.0s
 => [internal] load .dockerignore                                                                                                                                                                                0.0s
 => => transferring context: 2B                                                                                                                                                                                  0.0s
 => [1/2] FROM registry.access.redhat.com/ubi10/ubi:latest                                                                                                                                                       0.0s
 => CACHED [2/2] RUN dnf install -y procps                                                                                                                                                                       0.0s
 => exporting to image                                                                                                                                                                                           0.0s
 => => exporting layers                                                                                                                                                                                          0.0s
 => => writing image sha256:487ca6b14c8ad8efc9e821b85ac153b6a419ae9c9e591a33b23cd4f2a08601a4                                                                                                                     0.0s
 
 
[root@05f6cdd76f60 rhel10]# docker history sha256:487ca6b14c8ad8efc9e821b85ac153b6a419ae9c9e591a33b23cd4f2a08601a4
 IMAGE          CREATED       CREATED BY                                      SIZE      COMMENT
487ca6b14c8a   2 hours ago   RUN /bin/sh -c dnf install -y procps # build…   21.5MB    buildkit.dockerfile.v0
<missing>      12 days ago   /bin/sh -c #(nop) LABEL "build-date"="2025-0…   209MB
<missing>      12 days ago   /bin/sh -c #(nop) COPY file:2c9300aa2a82321b…   0B
<missing>      12 days ago   /bin/sh -c #(nop) CMD ["/bin/bash"]             0B
<missing>      12 days ago   /bin/sh -c #(nop) COPY file:973b743e3299d521…   0B
<missing>      12 days ago   /bin/sh -c #(nop) COPY dir:0458d10eb8f5b6ec8…   0B                                                                                                                                       <missing>      12 days ago   /bin/sh -c #(nop) ENV container oci             0B
<missing>      12 days ago   /bin/sh -c #(nop) LABEL io.openshift.tags="b…   0B
<missing>      12 days ago   /bin/sh -c #(nop) LABEL io.openshift.expose-…   0B
<missing>      12 days ago   /bin/sh -c #(nop) LABEL io.k8s.display-name=…   0B
<missing>      12 days ago   /bin/sh -c #(nop) LABEL io.k8s.description="…   0B
<missing>      12 days ago   /bin/sh -c #(nop) LABEL description="The Uni…   0B
<missing>      12 days ago   /bin/sh -c #(nop) LABEL summary="Provides th…   0B
<missing>      12 days ago   /bin/sh -c #(nop) LABEL com.redhat.license_t…   0B
<missing>      12 days ago   /bin/sh -c #(nop) LABEL com.redhat.component…   0B
<missing>      12 days ago   /bin/sh -c #(nop) LABEL url="https://www.red…   0B
<missing>      12 days ago   /bin/sh -c #(nop) LABEL maintainer="Red Hat,…   0B

We can see the resulting image has 2 "real" layers:

The original RHEL 10 layer from Red Hat (209MB). This one happens to contain a content set, and thus some valid "build info".
The customer layer, which is installing the procps (actually procps-ng) package. This layer has no "build info".

In the logic of lookupBuildInfo in pkg/fanal/applier/docker.go we currently assume that the layer 0 never has any build info, but that layer 1 has. Maybe it was true in the past, but it no longer is the case now. Looking at the logic, in the fina loop trying to find build info for the customer layer, it's ok to try to look for build info in layer 0. At best it will find a valid build info. At worst it will be nil and the loop will exit, returning nil in the final statement. So I believe this change is valid both for new RHEL 10 images and also "old" RHEL images which didn't have any build info in layer 0.

Before the change:

[root@05f6cdd76f60 rhel10]# (cd ../trivy/ && go run github.com/magefile/mage@latest build) && rm -rf /root/.cache/trivy && ../trivy/trivy image sha256:487ca6b14c8ad8efc9e821b85ac153b6a419ae9c9e591a33b23cd4f2a08601a4
2025-05-26T23:12:17Z    WARN    [vulndb] Trivy DB may be corrupted and will be re-downloaded. If you manually downloaded DB - use the `--skip-db-update` flag to skip updating DB. 
2025-05-26T23:12:17Z    INFO    [vulndb] Need to update DB
2025-05-26T23:12:17Z    INFO    [vulndb] Downloading vulnerability DB...
2025-05-26T23:12:17Z    INFO    [vulndb] Downloading artifact...        repo="mirror.gcr.io/aquasec/trivy-db:2"
64.26 MiB / 64.26 MiB [-------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 24.24 MiB p/s 2.9s
2025-05-26T23:12:20Z    INFO    [vulndb] Artifact successfully downloaded       repo="mirror.gcr.io/aquasec/trivy-db:2"
2025-05-26T23:12:20Z    INFO    [vuln] Vulnerability scanning is enabled
2025-05-26T23:12:20Z    INFO    [secret] Secret scanning is enabled
2025-05-26T23:12:20Z    INFO    [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2025-05-26T23:12:20Z    INFO    [secret] Please see also https://trivy.dev/dev/docs/scanner/secret#recommendation for faster secret detection
2025-05-26T23:12:27Z    INFO    [python] Licenses acquired from one or more METADATA files may be subject to additional terms. Use `--debug` flag to see all affected packages.
2025-05-26T23:12:27Z    INFO    Detected OS     family="redhat" version="10.0"
2025-05-26T23:12:27Z    WARN    This OS version is not on the EOL list  family="redhat" version="10"
2025-05-26T23:12:27Z    INFO    [redhat] Detecting RHEL/CentOS vulnerabilities...       os_version="10" pkg_num=173
2025-05-26T23:12:28Z    FATAL   Fatal error     run error: image scan error: scan error: scan failed: scan failed: failed to detect vulnerabilities: unable to scan OS packages: failed vulnerability detection of OS packages: failed detection: redhat vulnerability detection error: failed to get Red Hat advisories: unable to find CPE indices. See https://github.com/aquasecurity/trivy-db/issues/435 for details

After the change:

[root@05f6cdd76f60 rhel10]# (cd ../trivy/ && go run github.com/magefile/mage@latest build) && rm -rf /root/.cache/trivy && ../trivy/trivy image sha256:487ca6b14c8ad8efc9e821b85ac153b6a419ae9c9e591a33b23cd4f2a08601a
4
2025-05-26T23:13:03Z    WARN    [vulndb] Trivy DB may be corrupted and will be re-downloaded. If you manually downloaded DB - use the `--skip-db-update` flag to skip updating DB.
2025-05-26T23:13:03Z    INFO    [vulndb] Need to update DB
2025-05-26T23:13:03Z    INFO    [vulndb] Downloading vulnerability DB...
2025-05-26T23:13:03Z    INFO    [vulndb] Downloading artifact...        repo="mirror.gcr.io/aquasec/trivy-db:2"
64.26 MiB / 64.26 MiB [-------------------------------------------------------------------------------------------------------------------------------------------------------------------] 100.00% 19.64 MiB p/s 3.5s
2025-05-26T23:13:07Z    INFO    [vulndb] Artifact successfully downloaded       repo="mirror.gcr.io/aquasec/trivy-db:2"
2025-05-26T23:13:07Z    INFO    [vuln] Vulnerability scanning is enabled
2025-05-26T23:13:07Z    INFO    [secret] Secret scanning is enabled
2025-05-26T23:13:07Z    INFO    [secret] If your scanning is slow, please try '--scanners vuln' to disable secret scanning
2025-05-26T23:13:07Z    INFO    [secret] Please see also https://trivy.dev/dev/docs/scanner/secret#recommendation for faster secret detection
2025-05-26T23:13:14Z    INFO    [python] Licenses acquired from one or more METADATA files may be subject to additional terms. Use `--debug` flag to see all affected packages.
2025-05-26T23:13:15Z    INFO    Detected OS     family="redhat" version="10.0"
2025-05-26T23:13:15Z    WARN    This OS version is not on the EOL list  family="redhat" version="10"
2025-05-26T23:13:15Z    INFO    [redhat] Detecting RHEL/CentOS vulnerabilities...       os_version="10" pkg_num=173
2025-05-26T23:13:15Z    INFO    Number of language-specific files       num=0

Report Summary

┌─────────────────────────────────────────────────────────────────────────────────┬────────┬─────────────────┬─────────┐
│                                     Target                                      │  Type  │ Vulnerabilities │ Secrets │
├─────────────────────────────────────────────────────────────────────────────────┼────────┼─────────────────┼─────────┤
│ sha256:487ca6b14c8ad8efc9e821b85ac153b6a419ae9c9e591a33b23cd4f2a08601a4 (redhat │ redhat │        0        │    -    │
│ 10.0)                                                                           │        │                 │         │
└─────────────────────────────────────────────────────────────────────────────────┴────────┴─────────────────┴─────────┘
Legend:
- '-': Not scanned
- '0': Clean (no security findings detected)

Checklist

I've read the guidelines for contributing to this repository.
I've followed the conventions in the PR title.
I've added tests that prove my fix is effective or that my feature works.
I've updated the documentation with the relevant information (if needed).
I've added usage information (if the PR introduces new options)
I've included a "before" and "after" example to the description (if the PR is a user interface change).

knqyf263 · 2025-05-27T05:37:36Z

This is what Red Hat shared for the scanner certification. The first layer must not have any buildinfo. If RHEL 10 has buildinfo in the first layer, it's probably their mistake, or they've added a breaking change without notice.

knqyf263 · 2025-05-27T05:41:43Z

However, if that is indeed the case in reality, I think we need to make the correction as you suggested. @DmitriyLewen Could you double-check it?

Romain-Geissler-1A · 2025-05-27T09:20:01Z

Independantly of Red Hat plans, as a RHEL client myself, I do deliver to folks in my company a RHEL base image, which is configured to target our own private mirror of RHEL rpm packages (so it can work without internet access). When I do that, I do build an OCI image which squashes all layers (Red Hat ones + the ones of my company to change the rpm configuration) into a single one. So if you consider this use case legit, this pull request will also "make it work". For me so far it was not seen with RHEL 9 as RHEL 9 had this concept of "default content set" which was actually hidding the problem. Technically the latest RHEL 9 image do have the same layout as the latest RHEL 10 images.

DmitriyLewen · 2025-05-27T09:20:28Z

hm... I didn't find diagram from #8924 (comment).

But i checked some images:

registry.access.redhat.com/ubi7/ubi:latest
registry.access.redhat.com/ubi8/ubi:latest
registry.access.redhat.com/ubi9/ubi:latest
registry.access.redhat.com/ubi10/ubi:latest
These images have only one layer, and that layer contains content sets.

Also i checked test-harness images:

registry.redhat.io/rhel9/python-312@sha256:a4b270f826f5ae08eefada3a73c989fd2cfb712df9d9b86cdce53ee201189a8c
registry.redhat.io/openshift4/ose-console-rhel9@sha256:422c5c78b8fca878dba9d7faaae003e5341741b007fa51646904bc8d0c8a46c6
Both images have 4 layers. 0 layer also contains contents sets.

So looks like we need to check 0 layer for these images.

knqyf263 · 2025-05-28T06:16:07Z

Hmm... That differs from the explanation I received with the diagram. Nevertheless, if that is indeed the actual situation, then we will need to conform to it.

We also need to update the following logic. We may want to share the content set with the second layer only when the first layer doesn't have content set.

trivy/pkg/fanal/applier/docker.go

Lines 63 to 70 in deefd67

    
           // Base layer (layers[0]) is missing content sets 
        
           //   - it needs to be shared from layers[1] 
        
           if index == 0 { 
        
           	if len(layers) > 1 { 
        
           		return layers[1].BuildInfo 
        
           	} 
        
           	return nil 
        
           }

Romain-Geissler-1A · 2025-05-28T07:52:38Z

We may want to share the content set with the second layer only when the first layer doesn't have content set.

It's not clear the way it's coded right now, but it's already the case. The first thing done in this function is to return the layer build info if it exists. So on recent RHEL images, since layer 0 build info exists, we return it immediately, without trying to use layer 1 build info.

DmitriyLewen · 2025-05-28T08:49:09Z

We also need to update the following logic. We may want to share the content set with the second layer only when the first layer doesn't have content set.

I was thinking about this too and trying to find an image for this case.
I tested the image registry.redhat.io/rhel9/rhel-bootc.
This image contains many layers, but only the last layer contains content sets.
But there is one moment - all these layers don't have rpm packages.
So these changes don't make sense for this image.

Since we have confirmation (unlike the case when layer 0 contains set content) that there may be cases when we should take set content for layer 0 from >=layer 2 - I think it is worth sticking to the information you received from RedHat

the worst case that can happen is that we don't return the content for layer 0 (because layer 1 doesn't contain content sets either)
in this case, Trivy will take default content set instead of the content set from >=2 layers.
But I'm not sure I understand how we can be sure that we should take the content set with >=2 layers. We reason this way only based on the information that we should take the content sets from layer 1, but as practice shows, this is not always correct.

Romain-Geissler-1A · 2025-05-28T09:01:40Z

It seems some authoritative Red Hat folks would need to clarify what are the garantees scanners can expect wrt the Red Hat managed images. FYI, yesterday I asked one of my contact point at Red Hat involved in podman & bootc to see if someone at Red Hat can read this discussion and provide more up to date guidelines on what you shall do.

knqyf263 · 2025-05-28T10:20:33Z

So on recent RHEL images, since layer 0 build info exists, we return it immediately, without trying to use layer 1 build info.

Ah, you're right. It already works as desired.

trivy/pkg/fanal/applier/docker.go

Lines 59 to 61 in deefd67

    
           if layers[index].BuildInfo != nil { 
        
           	return layers[index].BuildInfo 
        
           }

knqyf263 · 2025-05-28T10:22:05Z

It seems some authoritative Red Hat folks would need to clarify what are the garantees scanners can expect wrt the Red Hat managed images. FYI, yesterday I asked one of my contact point at Red Hat involved in podman & bootc to see if someone at Red Hat can read this discussion and provide more up to date guidelines on what you shall do.

This may be the ideal approach, but if in reality we need to use the buildinfo from the first layer, I think it's acceptable to make this change while we wait for a response from Red Hat.

DmitriyLewen · 2025-05-28T11:08:36Z

I think it's acceptable to make this change while we wait for a response from Red Hat.

I don't mind

fix(redhat): Also try to find buildinfo in root layer (layer 0)

deefd67

Romain-Geissler-1A requested review from knqyf263 and DmitriyLewen as code owners May 26, 2025 23:15

Romain-Geissler-1A mentioned this pull request May 26, 2025

feat(redhat): Add EOL date for RHEL 10. #8910

Merged

6 tasks

DmitriyLewen approved these changes May 28, 2025

View reviewed changes

knqyf263 added this pull request to the merge queue May 29, 2025

Merged via the queue into aquasecurity:main with commit 906b037 May 29, 2025
13 checks passed

aqua-bot mentioned this pull request May 29, 2025

release: v0.63.0 [main] #8809

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

fix(redhat): Also try to find buildinfo in root layer (layer 0) #8924

fix(redhat): Also try to find buildinfo in root layer (layer 0) #8924

Uh oh!

Romain-Geissler-1A commented May 26, 2025 •

edited

Loading

Uh oh!

knqyf263 commented May 27, 2025

Uh oh!

knqyf263 commented May 27, 2025

Uh oh!

Romain-Geissler-1A commented May 27, 2025

Uh oh!

DmitriyLewen commented May 27, 2025

Uh oh!

knqyf263 commented May 28, 2025

Uh oh!

Romain-Geissler-1A commented May 28, 2025

Uh oh!

DmitriyLewen commented May 28, 2025

Uh oh!

Romain-Geissler-1A commented May 28, 2025

Uh oh!

knqyf263 commented May 28, 2025

Uh oh!

knqyf263 commented May 28, 2025

Uh oh!

DmitriyLewen commented May 28, 2025

Uh oh!

Uh oh!

Uh oh!

fix(redhat): Also try to find buildinfo in root layer (layer 0) #8924

fix(redhat): Also try to find buildinfo in root layer (layer 0) #8924

Uh oh!

Conversation

Romain-Geissler-1A commented May 26, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Checklist

Uh oh!

knqyf263 commented May 27, 2025

Uh oh!

knqyf263 commented May 27, 2025

Uh oh!

Romain-Geissler-1A commented May 27, 2025

Uh oh!

DmitriyLewen commented May 27, 2025

Uh oh!

knqyf263 commented May 28, 2025

Uh oh!

Romain-Geissler-1A commented May 28, 2025

Uh oh!

DmitriyLewen commented May 28, 2025

Uh oh!

Romain-Geissler-1A commented May 28, 2025

Uh oh!

knqyf263 commented May 28, 2025

Uh oh!

knqyf263 commented May 28, 2025

Uh oh!

DmitriyLewen commented May 28, 2025

Uh oh!

Uh oh!

Uh oh!

Romain-Geissler-1A commented May 26, 2025 •

edited

Loading