metadata-service[lib]: change metadata upload logic for release candidates #44509
Conversation
|
The latest updates on your projects. Learn more about Vercel for Git ↗︎ 1 Skipped Deployment
|
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @alafanechere and the rest of your teammates on |
alafanechere
changed the title
alafanechere
force-pushed
the
augustin/08-21-metadata-service_lib_introduce_the_Rollout_field_to_metadata_model
branch
from
bbdcb0a to
ba31fd5
Compare
alafanechere
force-pushed
the
augustin/08-21-metadata-service_orchestrator_change_metadata_upload_logic_for_release_candidates
branch
from
fb3fc3b to
80de6c1
Compare
alafanechere
force-pushed
the
augustin/08-21-metadata-service_lib_introduce_the_Rollout_field_to_metadata_model
branch
from
ba31fd5 to
3ccc18d
Compare
alafanechere
force-pushed
the
augustin/08-21-metadata-service_orchestrator_change_metadata_upload_logic_for_release_candidates
branch
from
80de6c1 to
d6a6a60
Compare
alafanechere
force-pushed
the
augustin/08-21-metadata-service_lib_introduce_the_Rollout_field_to_metadata_model
branch
from
3ccc18d to
c678426
Compare
alafanechere
force-pushed
the
augustin/08-21-metadata-service_orchestrator_change_metadata_upload_logic_for_release_candidates
branch
from
d6a6a60 to
2bbf721
Compare
| @@ -144,6 +144,10 @@ def _latest_upload(metadata: ConnectorMetadataDefinitionV0, bucket: storage.buck | |||
| latest_path = get_metadata_remote_file_path(metadata.data.dockerRepository, "latest") | |||
| return upload_file_if_changed(metadata_file_path, bucket, latest_path, disable_cache=True) | |||
|
|
|||
| def _release_candidate_upload(metadata: ConnectorMetadataDefinitionV0, bucket: storage.bucket.Bucket, metadata_file_path: Path) -> Tuple[bool, str]: | |||
There was a problem hiding this comment.
This looks almost identical to _latest_upload on line 143. Should we move "release_candidate" (and "latest", from line 144) to constants, and then have just a helper function _upload that takes it in as a variable, instead of two helper functions?
| is_pre_release = validator_opts.prerelease_tag is not None | ||
| is_release_candidate = metadata.data.release.is_release_candidate |
There was a problem hiding this comment.
Is it worth adding an assertion that is_pre_release and is_release_candidate aren't both True?
There was a problem hiding this comment.
Both can be True. It's determined as a pre-release from options passed to the metadata upload CLI (which originally come from an airbyte-ci connectors publish inputs.
So nothing prevents you from having is_release_candidate set to True, but wanting to release a pre-release for testing (usually from your branch).
This is why I added should_upload_release_candidate and should_upload_latest which would both be falsy if the it's a pre-release.
There was a problem hiding this comment.
Ah okay that makes sense to me. Thanks for clarifying.
| # - the current metadata to the "latest" path | ||
| # - the current doc to the "latest" path | ||
| # - the current inapp doc to the "latest" path | ||
| if should_upload_latest: |
There was a problem hiding this comment.
I think we'll eventually want to move this logic into a helper function so it can be accessed by the final part of the RC publishing pipeline. But all good if you want to leave it here for now.
There was a problem hiding this comment.
I envision the "final part of the RC publishing pipeline" to be orchestrated via an airbyte-ci command which would:
- Delete the
release_candidate/metadata.yamlfile (could be done in this package) - Set the
isReleaseCandidatefield tofalseinmetadata.yaml(could be done in this package) - Push the current version with the
latest(will likely be done directly inairbyte-ci)
So I don't think a change here is required: the update of the metadata.yaml isReleaseCandidate field will change the behavior of the upload function
There was a problem hiding this comment.
Okay cool, that sounds good.
alafanechere
force-pushed
the
augustin/08-21-metadata-service_lib_introduce_the_Rollout_field_to_metadata_model
branch
from
c678426 to
46977fc
Compare
alafanechere
force-pushed
the
augustin/08-21-metadata-service_orchestrator_change_metadata_upload_logic_for_release_candidates
branch
from
2bbf721 to
1e0f886
Compare
alafanechere
force-pushed
the
augustin/08-21-metadata-service_lib_introduce_the_Rollout_field_to_metadata_model
branch
from
46977fc to
c088c5b
Compare
alafanechere
force-pushed
the
augustin/08-21-metadata-service_orchestrator_change_metadata_upload_logic_for_release_candidates
branch
from
1e0f886 to
712ff3f
Compare
| # - the current metadata to the "latest" path | ||
| # - the current doc to the "latest" path | ||
| # - the current inapp doc to the "latest" path | ||
| if should_upload_latest: |
There was a problem hiding this comment.
Okay cool, that sounds good.
Merge activity
|
alafanechere
changed the base branch from
augustin/08-21-metadata-service_lib_introduce_the_Rollout_field_to_metadata_model
to
graphite-base/44509
alafanechere
force-pushed
the
augustin/08-21-metadata-service_orchestrator_change_metadata_upload_logic_for_release_candidates
branch
from
712ff3f to
63e2cfd
Compare
alafanechere
deleted the
augustin/08-21-metadata-service_orchestrator_change_metadata_upload_logic_for_release_candidates
branch
What
Relates to https://github.com/airbytehq/airbyte-internal-issues/issues/9403
When a connector metadata is flagged as a release candidate we should upload it to a dedicated path in GCS, so that it will not be picked up by the registry as the latest version.