Feature Request: Batch processing of sbom file with count of components more then 2000 (AWS Inspector limit)

[iriabov]

Description

We catch the limitation error in the AWS Inpector CLI command to scan SBOM files. Error log message:
An error occurred (ValidationException) when calling the ScanSbom operation: Input SBOM failed validation: Max number of components exceeded (2000)

You can find the package limit in AWS documentation:
https://docs.aws.amazon.com/inspector/v2/APIReference/API_scan_ScanSbom.html#API_scan_ScanSbom_RequestSyntax

The JSON file for the SBOM you want to scan. The SBOM must be in CycloneDX 1.5 format. This format limits you to passing 2000 components before throwing a ValidException error.

Expected Behavior

Divide the SBOM file by chunks (with the same metadata and max count of components - 2000). The next step is scanning these files independently. The last step is calculating vulnerabilities from all files.

Actual Behavior

We have an error in the scanning GH action:
An error occurred (ValidationException) when calling the ScanSbom operation: Input SBOM failed validation: Max number of components exceeded (2000)

Steps to Reproduce

Build the image with more the 2000 of components in SBOM file.

Other Information

Please provide any additional information such as logs or screenshots.

[s-kenji]

Hello, thank you for reaching out.
We are confirming receipt of your issue.
We will begin to triage the problem, and will report back when we have pertinent updates.

[bluesentinelsec]

Hello, thank you again for creating this issue.
We understand the 2,000 component limit is preventing you from completing Inspector scans.
Splitting the SBOM into smaller chunks is new functionality that we will have to implement.
We have this feature request in our backlog, but we do not have an ETA as to when work will begin.

I'm going to mark this issue as on-hold until we begin work on the feature.

[bluesentinelsec]

@iriabov Hello, we recently increased the permitted component count to 5,000.
Regarding chunking, that is still being discussed internally as an enhancement.

[iriabov]

Hi @bluesentinelsec
Awesome, thank you for update and thank you so much for your time and effort—I really appreciate it!
If it's alright with you, I could try to implement chunking and create a PR to review.

Thank you!

[bluesentinelsec]

Hi @bluesentinelsec Awesome, thank you for update and thank you so much for your time and effort—I really appreciate it! If it's alright with you, I could try to implement chunking and create a PR to review.

Thank you!

Contributions are certainly welcome. :)
I would actually recommend that you write a stand alone script for this purpose, that way, you don't need to worry about any of the GitHub Actions integration. This is the basic workflow I have in mind:

#1. First generate a large sbom
./inspector-sbomgen container --image large_image:latest -o large_sbom.json

#2. Now scan the SBOM; the program will need to handle chunking the scan into multiple requests,
# and then reassembling the responses into a single JSON document.
./scan_sbom_in_chunks.py --f your_large_sbom.json --profile your_aws_profile

# You'll need to use the AWS SDK to interact with Inspector's ScanSbom API.

Be advised, I think this is a non-trivial program to write; no worries if you need to back out because this is likely a moderate amount of work. Otherwise, feel free to take a stab and we'll provide feedback if you open a pull request.

[raelyz]

@bluesentinelsec any idea roughly when chunking would be implemented? I'm hitting the 5000 limit at the moment

[bluesentinelsec]

@bluesentinelsec any idea roughly when chunking would be implemented? I'm hitting the 5000 limit at the moment

@raelyz Thank you for inquiring about chunking large SBOMs to work around the 5,000 component limit.

Unfortunately we do not have a firm ETA for this feature, as we are currently focusing on improving the performance and accuracy of our inventory collection engine.

To work around the 5,000 component limit, you may consider splitting up your scans into smaller tasks, for example, first scan OS packages, then perform subsequent scans for 3rd party packages. The action provides arguments that can help with this, specifically, "scanners," "skip_scanners," and "skip_files."

[raelyz]

@bluesentinelsec I've attempted to use specific scanners for a java application and I am still hitting the 5000 component limit. Really do hope the team is able to find time to address this issue. I've since disabled the action for several of our applications as a result.

[bluesentinelsec]

@bluesentinelsec I've attempted to use specific scanners for a java application and I am still hitting the 5000 component limit. Really do hope the team is able to find time to address this issue. I've since disabled the action for several of our applications as a result.

@raelyz
Thank you for the feedback. We understand that the 5,000 component limit is preventing you from scanning your artifact as desired. Unfortunately we are not prioritizing this feature at this time. We would be happy to meet with you to discuss your usage and explore alternate solutions. If this is of interest to you, please send us an email at [email protected].

[raelyz]

@bluesentinelsec, sure thing! I've sent you an email we can continue over there.

[fs-david-harrington]

Can you please increase the limit beyond 5,000 - upto 10,000 would be wonderful

[bluesentinelsec]

@fs-david-harrington
Thanks for reaching out. Unfortunately, we are not planning on increasing the 5k component limit.
The Inspector service caps the component limit at 5K for many reasons to include performance and scale.
For example, AWS API Gateway has a hard cap of 10 MB for a payload response, and SBOMs with 10K components would likely exceed that limit.
Also, from a security perspective, there is benefit to minimizing installed packages to reduce the potential attack surface.

One potential solution is for you to push your image to AWS ECR and scan with Inspector (e.g. Inspector Enhanced Scanning for ECR).
Inspector ECR does not have the 5,000 component limit, and you can learn more about it here:
https://docs.aws.amazon.com/inspector/latest/user/scanning-ecr.html

If you have further questions, please feel encouraged to reach out to us at [email protected].

[adam-everest]

Hi @bluesentinelsec, thanks a lot for the update!

To summarise, if someone runs into this issue, the current options are:

1. Exclude some components of the SBOM from scanning to fit under the 5000 component threshold (perhaps with the help of the inputs scanners skip_scanners and skip_files)
2. Implement splitting and chunking the SBOM and then make multiple scans (which then presumably means the price for the full scan will be a multiple of the $0.03 per scan request to Inspector (as of today for us-east-1))
3. Use the ECR Enhanced scanning feature which also uses Amazon Inspector but doesn't have this limit on image size

Please correct me if I'm wrong.