[airbyte-ci] new commands for migration to base image #30520

alafanechere · 2023-09-18T12:13:53Z

Closes #30238

What

This PR adds useful airbyte-ci commands to make some connector changes:

bump-version: Will edit the changelog and metadata.yaml of the selected connector
airbyte-ci connectors --name=<my-connector> bump-version <{patch|minor|major> <PULL_REQUEST_NUMBER> <my changelog entry>
upgrade-base-image: Will update the baseImage to the latest version in the metadata.yaml
airbyte-ci connectors --name=<my-connector> upgrade-base-image
migrate-to-base-image: Will perform a minor version bump, delete dockerfile / build.gradle and update the connector documentation to share the new build instructions.
airbyte-ci connectors --name=<my-connector> migrate-to-base-image <PULL_REQUEST_NUMBER>

Bonus

Delete the format command

alafanechere · 2023-09-18T12:14:16Z

Current dependencies on/for this PR:

master
- PR connector base image: declare the base image package and implement #30303

This comment was auto-generated by Graphite.

alafanechere · 2023-10-13T17:41:07Z

Tests are not running due to our current rate limiting issue.

bnchrch · 2023-10-13T23:00:27Z

airbyte-ci/connectors/pipelines/pipelines/commands/groups/connectors.py

@@ -509,8 +516,10 @@ def list(


 @connectors.command(name="format", cls=DaggerPipelineCommand, help="Autoformat connector code.")
+@click.option("--commit-and-push", default=False)


❓I dont see this being used?

bnchrch · 2023-10-13T23:01:08Z

airbyte-ci/connectors/pipelines/pipelines/commands/groups/connectors.py

 @click.pass_context
-def format_code(ctx: click.Context) -> bool:
+def format_code(ctx: click.Context, commit_and_push, export_to_host) -> bool:


🤔 What if we just remove this instead of update since were relying on gradlew format atm

Yes, these new options a remainder of an attempt to consolidate these command changing connector code and making them commit and push. I'll revert this. And will open a separate PR to remove format

bnchrch · 2023-10-13T23:02:14Z

airbyte-ci/connectors/pipelines/pipelines/commands/groups/connectors.py

+    """Bump a connector version: update metadata.yaml, changelog and delete legacy files."""
+
+    connectors_contexts = [
+        ConnectorContext(


Do we need a whole connector context?

bnchrch · 2023-10-13T23:08:18Z

airbyte-ci/connectors/pipelines/pipelines/connector_changes/base_image_version_migration.py

+    return context.report
+
+
+async def run_connector_migration_to_base_image_pipeline(context: ConnectorContext, semaphore, pull_request_number: str):


❓Why a pipeline and not a script?

Fine for it to stay as a pipeline but we likely want to add a note in a comment for when we can delete this code.

I'm not sure this is a short lived command, as long as community connector use Dockerfile it has a reason to be considered a pipeline and not adhoc script.

bnchrch · 2023-10-13T23:09:06Z

airbyte-ci/connectors/pipelines/pipelines/consts.py

@@ -38,3 +38,5 @@
 DOCKER_HOST_NAME = "global-docker-host"
 DOCKER_HOST_PORT = 2375
 DOCKER_TMP_VOLUME_NAME = "shared-tmp"
+REPO = git.Repo(search_parent_directories=True)


bnchrch

Few comments but nothing blocking

alafanechere mentioned this pull request Sep 18, 2023

connector base image: declare the base image package and implement #30303

Merged

octavia-squidington-iii added the area/connectors Connector related issues label Sep 18, 2023

alafanechere mentioned this pull request Sep 18, 2023

[airbyte-ci] introduce an autocommit command to update base image version to the latest one #30508

Closed

octavia-squidington-iii added the connectors/source/faker label Sep 18, 2023

alafanechere mentioned this pull request Sep 18, 2023

[airbyte-ci] implement pre/post build hooks #30526

Merged

alafanechere force-pushed the augustin/09-15-make_QA_checks_tolerate_connector_without_dockerfile branch from 9029852 to e6bcea4 Compare September 18, 2023 14:16

alafanechere force-pushed the augustin/09-17/airbyte-ci-new-commands-for-migration branch from 9536d3f to 7b65144 Compare September 18, 2023 14:17

alafanechere force-pushed the augustin/09-15-make_QA_checks_tolerate_connector_without_dockerfile branch from e6bcea4 to fb1a22b Compare September 18, 2023 14:46

alafanechere force-pushed the augustin/09-17/airbyte-ci-new-commands-for-migration branch from 0ff5759 to 1d87e9a Compare September 18, 2023 14:46

This was referenced Sep 18, 2023

[airbyte-ci]: new check on python certified connector to validate their base image use #30527

Merged

Make certified python connectors use our base image #30528

Closed

alafanechere marked this pull request as ready for review September 18, 2023 15:34

octavia-squidington-iii removed the area/connectors Connector related issues label Sep 18, 2023

airbytehq deleted a comment from github-actions bot Sep 18, 2023

alafanechere force-pushed the augustin/09-15-make_QA_checks_tolerate_connector_without_dockerfile branch from fb1a22b to e3a6a77 Compare September 18, 2023 16:02

alafanechere force-pushed the augustin/09-17/airbyte-ci-new-commands-for-migration branch from a1b0d77 to 2d3ee2e Compare September 18, 2023 16:02

alafanechere requested a review from a team September 18, 2023 18:07

alafanechere force-pushed the augustin/09-15-make_QA_checks_tolerate_connector_without_dockerfile branch from e3a6a77 to fc5519a Compare September 19, 2023 15:28

alafanechere force-pushed the augustin/09-17/airbyte-ci-new-commands-for-migration branch from 2d3ee2e to f6e7ab5 Compare September 19, 2023 15:28

alafanechere force-pushed the augustin/09-15-make_QA_checks_tolerate_connector_without_dockerfile branch from fc5519a to 981fc3e Compare September 19, 2023 16:29

alafanechere force-pushed the augustin/09-17/airbyte-ci-new-commands-for-migration branch from f6e7ab5 to 4b23511 Compare September 19, 2023 16:29

alafanechere force-pushed the augustin/declare-first-base-image branch from 1de4960 to 1533e75 Compare September 21, 2023 20:09

alafanechere force-pushed the augustin/09-17/airbyte-ci-new-commands-for-migration branch from c69888e to 0cf2c46 Compare September 21, 2023 20:09

Base automatically changed from augustin/declare-first-base-image to master September 22, 2023 10:26

alafanechere force-pushed the augustin/09-17/airbyte-ci-new-commands-for-migration branch 6 times, most recently from 161d658 to f5f8bd5 Compare October 13, 2023 17:39

alafanechere requested review from bnchrch and erohmensing October 13, 2023 17:39

alafanechere marked this pull request as ready for review October 13, 2023 17:39

alafanechere mentioned this pull request Oct 13, 2023

Remove dockerfiles for certified python connnectors and use the base image #31377

Closed

bnchrch reviewed Oct 13, 2023

View reviewed changes

bnchrch approved these changes Oct 13, 2023

View reviewed changes

alafanechere force-pushed the augustin/09-17/airbyte-ci-new-commands-for-migration branch 2 times, most recently from 51a8fa1 to 16354fe Compare October 14, 2023 13:22

airbyte-ci: new commands for migration to base image

e19a55b

alafanechere force-pushed the augustin/09-17/airbyte-ci-new-commands-for-migration branch from 16354fe to e19a55b Compare October 14, 2023 13:46

alafanechere removed the connectors/source/faker label Oct 14, 2023

alafanechere merged commit 99b9fc9 into master Oct 14, 2023

alafanechere deleted the augustin/09-17/airbyte-ci-new-commands-for-migration branch October 14, 2023 14:06

alafanechere mentioned this pull request Oct 16, 2023

airbyte-ci: fix None docker creds #31455

Merged

ariesgun pushed a commit to ariesgun/airbyte that referenced this pull request Oct 23, 2023

[airbyte-ci] new commands for migration to base image (airbytehq#30520)

5803911

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[airbyte-ci] new commands for migration to base image #30520

[airbyte-ci] new commands for migration to base image #30520

alafanechere commented Sep 18, 2023 •

edited

Loading

alafanechere commented Sep 18, 2023 •

edited

Loading

alafanechere commented Oct 13, 2023

bnchrch Oct 13, 2023

bnchrch Oct 13, 2023

alafanechere Oct 14, 2023

bnchrch Oct 13, 2023

bnchrch Oct 13, 2023

alafanechere Oct 14, 2023

bnchrch Oct 13, 2023

bnchrch left a comment

		@@ -509,8 +516,10 @@ def list(


		@connectors.command(name="format", cls=DaggerPipelineCommand, help="Autoformat connector code.")
		@click.option("--commit-and-push", default=False)

		return context.report


		async def run_connector_migration_to_base_image_pipeline(context: ConnectorContext, semaphore, pull_request_number: str):

[airbyte-ci] new commands for migration to base image #30520

[airbyte-ci] new commands for migration to base image #30520

Conversation

alafanechere commented Sep 18, 2023 • edited Loading

What

Bonus

alafanechere commented Sep 18, 2023 • edited Loading

alafanechere commented Oct 13, 2023

bnchrch Oct 13, 2023

Choose a reason for hiding this comment

bnchrch Oct 13, 2023

Choose a reason for hiding this comment

alafanechere Oct 14, 2023

Choose a reason for hiding this comment

bnchrch Oct 13, 2023

Choose a reason for hiding this comment

bnchrch Oct 13, 2023

Choose a reason for hiding this comment

alafanechere Oct 14, 2023

Choose a reason for hiding this comment

bnchrch Oct 13, 2023

Choose a reason for hiding this comment

bnchrch left a comment

Choose a reason for hiding this comment

alafanechere commented Sep 18, 2023 •

edited

Loading

alafanechere commented Sep 18, 2023 •

edited

Loading