chore: non-dagger publish #59740
chore: non-dagger publish #59740
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
|
…o match current interface.
Co-authored-by: Aaron ("AJ") Steers <[email protected]>
There was a problem hiding this comment.
Made some comments. I think this looks good.
One small request would be if the java-specific shell script could check language on the input connector name and be a no-op with printed warning if so. But in theory, your code shouldn't (🤞) receive any java connector names because of the --java input to the other script. (Aka, don't consider this feedback blocking. Just an idea.)
|
Here is an example of airbyte-ci skipping the image building because the new process already built and uploaded the image when not in In the same example, we confirm the logic to differs does come up empty if the previous commit does not have connector modifications. And the newly published dev-null dest is working fine: https://cloud.airbyte.com/workspaces/a0c2044b-7346-4fd8-ba9b-98d1eb36eb34/connections/20d0a052-fe85-44fd-8a52-8a7b5d5c4bdf/timeline |
|
Here is a run where we are correctly detecting the changed connector from the previous commit, and building the image. |
| # ./build-and-publish-java-connectors-with-tag.sh --publish-option=main-release foo-conn bar-conn | ||
| # | ||
| # 3) Pre-release (dev tag) via JSON pipe | ||
| # echo '{"connector":["foo-conn","bar-conn"]}' | ./build-and-publish-java-connectors-with-tag.sh --publish-option=pre-release |
There was a problem hiding this comment.
nonblocking: I'd mildly prefer to only support this mode, since that's all we need for CI (purely for the sake of having fewer lines of shell script)
There was a problem hiding this comment.
meaning you prefer only json support?
There was a problem hiding this comment.
yeah. Unless that's something you're using for local testing/etc
What
Modify the publishing workflows for JVM to publish images outside of dagger, while relies on airbyte-ci to continue updating registry/metadata.
This relies on the fact that airbyte-ci first checks for a published image, and if the image is published, skips image publishing and only updates metadata.
One edge case here: airbyte-ci will always try to build a pre-release image, regardless of whether docker images already exists. The changes here will build and publish the JVM image before dagger attempts to do so, so dagger attempts will rely on the cache and get the same result. Eventually we want to remove this behaviour from airbyte-ci. I tried to do so, and it's more complicated, so we'll do this later.
How
dockerImageTaginmetadata.yaml.metadata.yamland adding-dev.<git-first-10-digits>.NOTE: we only support publishing destination-dev-null to start to be safe. This is fine since airbyte-ci will continue to build images if it detects the image does not exist.
Review guide
The 3 main files described above.
User Impact
Can this PR be safely reverted and rolled back?