Skip to content

Stop publishing java connectors on merge to master #28344

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Jul 15, 2023
Merged

Stop publishing java connectors on merge to master #28344

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
10 commits
Select commit Hold shift + click to select a range
4c7d2d2
process language after modified flag
pedroslopez Jul 15, 2023
cb0588e
modify some connectors
pedroslopez Jul 15, 2023
1a82d3d
empty commit
pedroslopez Jul 15, 2023
77851cd
another bump
pedroslopez Jul 15, 2023
48f1cd5
modified connectors are already filtered
pedroslopez Jul 15, 2023
a7a6a98
change some connectors
pedroslopez Jul 15, 2023
82002c6
change some connectors again
pedroslopez Jul 15, 2023
9393e29
revert to master version
pedroslopez Jul 15, 2023
98602a5
nit
pedroslopez Jul 15, 2023
f88784c
filter for python/lowcode connectors
pedroslopez Jul 15, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion .github/workflows/publish_connectors.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -39,7 +39,7 @@ jobs:
metadata_service_gcs_credentials: ${{ secrets.METADATA_SERVICE_PROD_GCS_CREDENTIALS }}
slack_webhook_url: ${{ secrets.PUBLISH_ON_MERGE_SLACK_WEBHOOK }}
spec_cache_gcs_credentials: ${{ secrets.SPEC_CACHE_SERVICE_ACCOUNT_KEY_PUBLISH }}
subcommand: "connectors --concurrency=1 --execute-timeout=3600 --modified publish --main-release"
subcommand: "connectors --concurrency=1 --execute-timeout=3600 --modified --language=python --language=low-code publish --main-release"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skip java connectors by specifying we only want python and low-code connectors


- name: Publish connectors [manual]
id: publish-connectors
Expand Down
13 changes: 5 additions & 8 deletions tools/ci_connector_ops/ci_connector_ops/pipelines/commands/groups/connectors.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -103,6 +103,8 @@ def connectors(
# and attach modified files to them
selected_connectors_and_files = {connector: modified_connectors_and_files.get(connector, []) for connector in all_connectors}

if modified:
selected_connectors_and_files = modified_connectors_and_files
Comment on lines +106 to +107
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved this to be before the other filters so they can be applied together

if names:
selected_connectors_and_files = {
connector: selected_connectors_and_files[connector]
Expand All @@ -121,8 +123,6 @@ def connectors(
for connector in selected_connectors_and_files
if connector.release_stage in release_stages
}
if modified:
selected_connectors_and_files = modified_connectors_and_files

ctx.obj["selected_connectors_and_files"] = selected_connectors_and_files
ctx.obj["selected_connectors_names"] = [c.technical_name for c in selected_connectors_and_files.keys()]
Expand Down Expand Up @@ -312,12 +312,9 @@ def publish(
"Publishing from a local environment is not recommend and requires to be logged in Airbyte's DockerHub registry, do you want to continue?",
abort=True,
)
if ctx.obj["modified"]:
selected_connectors_and_files = get_modified_connectors(get_modified_metadata_files(ctx.obj["modified_files"]))
selected_connectors_names = [connector.technical_name for connector in selected_connectors_and_files.keys()]
Comment on lines -315 to -317
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Modified connectors were already being filtered out in the previous step, so the selected_connectors_and_files ctx already does this.

This was overwriting the other filters in the publish step

else:
selected_connectors_and_files = ctx.obj["selected_connectors_and_files"]
selected_connectors_names = ctx.obj["selected_connectors_names"]

selected_connectors_and_files = ctx.obj["selected_connectors_and_files"]
selected_connectors_names = ctx.obj["selected_connectors_names"]

main_logger.info(f"Will publish the following connectors: {', '.join(selected_connectors_names)}")
publish_connector_contexts = reorder_contexts(
Expand Down