airbyte-ci: generate SBOM on publish

[alafanechere]

What

Closes https://github.com/airbytehq/airbyte-internal-issues/issues/8989
Closes https://github.com/airbytehq/airbyte-internal-issues/issues/8990

We want to expose an url to our connector SBOM in our connector registry.
But we first have to generate them, and the publish is the right place as this can be considered as a connector release artifact.

How

Update our publish pipeline to:

• Generate an SBOM from the published docker image, in SPDX JSON format using syft
• Upload the generated SBOM file to our metadata-service bucket

NB: If the connector was already published we still perform the SBOM generation/upload, this will allow a simple backfill for already published connector version.

Example

I pre-released source-faker from this branch, it uploaded the following SBOM to our metadata service bucket (which has the connectors.airbyte.com CDN):

https://connectors.airbyte.com/files/sbom/airbyte/source-faker/6.2.9-dev.e955ad82c7.spdx.json

User Impact

None, this should be transparent to developers.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		Aug 19, 2024 9:54am

[alafanechere]

• docs: add SBOM url to metadata state in connector documentation #44386
• metadata-service[orchestrator]: set SBOM url in registry #44381
• metadata-service[lib]: add sbom_url to generated registry fields #44379
• airbyte-ci: generate SBOM on publish #44377 👈
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @alafanechere and the rest of your teammates on Graphite

[natikgadzhi]

Looks good!

[natikgadzhi]

I think this is fine, but I would love us to have good discipline about updating image verisons that are referred to in this manner as a const in Python code in airbyte-ci.

How do we make ourselves a reminder to update these at least quarterly?

[alafanechere]

We could centralize all images in a single module with constants. I'm not sure if it's important to update this kind of image which is just a tool and does not takes part in our build process though.

[sentry-io]

Suspect Issues

This pull request was deployed and Sentry observed the following issues:

• ‼️ TransportError: Unexpected response from engine: Server error '502 Bad Gateway' for url 'http://127.0.0.1:33621/q... dagger.client._core in execute View Issue

_{Did you find this useful? React with a 👍 or 👎}