Airbyte CDK: Test via airbyte CI test poetry packages #36497

erohmensing · 2024-03-26T18:39:14Z

Test the python cdk (build, check-ci) via our internal poetry package testing instead of via its own workflow.

Note: this does increase CI runtime for testing the CDK, but improves maintainability as we can treat the cdk as a standard internal python package.

TODO:

Remove Python CDK Tests from required status checks (removed)
Remove Run Airbyte CI Tests from required status checks (renamed from)
Add Internal Poetry Packages CI to required status checks (renamed to)

vercel · 2024-03-26T18:39:20Z

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Ignored Deployment

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	⬜️ Ignored (Inspect)	Visit Preview		May 3, 2024 2:55pm

erohmensing · 2024-03-26T18:39:28Z

Airbyte CDK: Test via airbyte CI test poetry packages #36497 👈
airbyte cdk: simplify datamodel-gen scripts #36530
airbyte ci test to support --extras #36527
Update out of date internal lockfiles #36535
master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @erohmensing and the rest of your teammates on Graphite

install all extras groups add to list of internal poetry packages add to list of internal packages for checking modified check works

natikgadzhi · 2024-03-27T05:29:27Z

What I would also love to do at this stage or later is to document how to run tests on the CDK, and how to build the CDK in the CDK readme — or perhaps in CONTRIBUTING.md, so we can make this flow transparent for newcomers.

girarda · 2024-03-27T19:09:50Z

Adding to Natik's comment:
@erohmensing , how do we test these changes? Is there a new airbyte-ci command to build/assemble the CDK and run tests?

alafanechere

Nice!
We should ask CDK developers to update their branch following the merge of this PR.
Please also bump the pipelines version as you modified the list of internal poetry packages in there.

alafanechere · 2024-03-28T09:58:44Z

.github/workflows/python_cdk_tests.yml

Python CDK Tests is declared as a required check in the branch protection rules settings to merge to master. We should remove it when merging this PR.

The workflow goes away total, right? Can we add the airbyte-ci tests as required branch protection rule?

@natikgadzhi it'll already be required.
CDK tests will run in the workflow testing all our internal packages, which is a required one.

alafanechere · 2024-03-28T09:59:34Z

.github/workflows/airbyte-ci-tests.yml

-    name: Run Airbyte CI tests
+    name: Internal Poetry packages CI


If you rename it you should also change the branch protection rules settings (because Run Airbyte CI tests is currently a required check)

alafanechere · 2024-03-28T10:02:09Z

airbyte-cdk/python/pyproject.toml

+[tool.airbyte_ci]
+optional_poetry_groups = ["dev"]
+poetry_extras = ["file-based", "sphinx-docs", "vector-db-based"]
+poe_tasks = ["build", "check-ci"]


check-ci does not run type checks but they do run in python_cdk_tests right? Is it safe to not run type checks in CI for the CDK? ( cc. @girarda )

Did we give up on running mypy on modified files? It would feel like a regression to me

Yeah, I think we should run type checks still.

The type checks currently run and will continue to run as a separate workflow, run-mypy-on-modified-cdk-files.yml

erohmensing · 2024-05-03T14:53:45Z

how do we test these changes? Is there a new airbyte-ci command to build/assemble the CDK and run tests?

You can still use the same poetry run poe scripts to run tests locally if you prefer not to do them in dagger. There is also the airbyte-ci command: airbyte-ci test -p airbyte-cdk/python - this is what is run in CI. it runs build and check-ci.

octavia-squidington-iii added the CDK Connector Development Kit label Mar 26, 2024

Configure tool.airbyte_ci to test tasks that we use in CI

03d6c42

install all extras groups add to list of internal poetry packages add to list of internal packages for checking modified check works

erohmensing changed the base branch from master to ella/airbyte-ci-test-extras March 26, 2024 23:46

erohmensing force-pushed the ella/cdk-test-in-poetry branch from f362d43 to 253ce0b Compare March 26, 2024 23:46

erohmensing mentioned this pull request Mar 26, 2024

airbyte ci test to support --extras #36527

Merged

erohmensing force-pushed the ella/airbyte-ci-test-extras branch from b3c95d0 to 92c3183 Compare March 26, 2024 23:48

erohmensing force-pushed the ella/cdk-test-in-poetry branch from 253ce0b to 32b2fbc Compare March 26, 2024 23:48

erohmensing changed the base branch from ella/airbyte-ci-test-extras to ella/fix-build-scripts March 27, 2024 00:46

erohmensing force-pushed the ella/cdk-test-in-poetry branch from 32b2fbc to 6a2546c Compare March 27, 2024 00:46

erohmensing mentioned this pull request Mar 27, 2024

airbyte cdk: simplify datamodel-gen scripts #36530

Merged

erohmensing force-pushed the ella/cdk-test-in-poetry branch from d32514f to 1217d71 Compare March 27, 2024 03:36

check build

2048476

erohmensing force-pushed the ella/cdk-test-in-poetry branch 2 times, most recently from 37da8ef to 7ff0920 Compare March 27, 2024 04:02

erohmensing force-pushed the ella/fix-build-scripts branch from 865bead to 1c6fe5c Compare March 27, 2024 04:09

erohmensing force-pushed the ella/cdk-test-in-poetry branch from 7ff0920 to 80c982e Compare March 27, 2024 04:10

erohmensing mentioned this pull request Mar 27, 2024

Update out of date internal lockfiles #36535

Merged

erohmensing force-pushed the ella/fix-build-scripts branch from 1c6fe5c to adebf4e Compare March 27, 2024 04:15

erohmensing force-pushed the ella/cdk-test-in-poetry branch from 80c982e to 987609f Compare March 27, 2024 04:15

erohmensing force-pushed the ella/fix-build-scripts branch from adebf4e to 01c66ae Compare March 27, 2024 04:20

erohmensing force-pushed the ella/cdk-test-in-poetry branch from 987609f to 096b363 Compare March 27, 2024 04:20

remove workflow

f816de3

erohmensing marked this pull request as ready for review March 27, 2024 04:43

erohmensing requested review from a team, alafanechere and girarda March 27, 2024 04:43

erohmensing force-pushed the ella/fix-build-scripts branch from 01c66ae to eab8567 Compare March 27, 2024 04:45

erohmensing force-pushed the ella/cdk-test-in-poetry branch from 7fd08e0 to e611bab Compare March 27, 2024 04:45

rename the job

4f35156

erohmensing force-pushed the ella/fix-build-scripts branch from eab8567 to ba576a6 Compare March 27, 2024 18:17

erohmensing force-pushed the ella/fix-build-scripts branch from ba576a6 to ffa0f3e Compare March 27, 2024 19:15

erohmensing force-pushed the ella/cdk-test-in-poetry branch from 949d92c to 7dc249f Compare March 27, 2024 19:16

Base automatically changed from ella/fix-build-scripts to master March 27, 2024 19:33

erohmensing force-pushed the ella/cdk-test-in-poetry branch from 7dc249f to 8a788e4 Compare March 27, 2024 19:33

alafanechere approved these changes Mar 28, 2024

View reviewed changes

erohmensing force-pushed the ella/cdk-test-in-poetry branch from 8a788e4 to dd15378 Compare May 3, 2024 14:31

erohmensing added 2 commits May 3, 2024 09:49

bump version

632a2a6

update comments

5276b4d

erohmensing force-pushed the ella/cdk-test-in-poetry branch from 5742727 to 5276b4d Compare May 3, 2024 14:55

erohmensing merged commit f23c2e6 into master May 3, 2024
34 checks passed

erohmensing deleted the ella/cdk-test-in-poetry branch May 3, 2024 15:20

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Airbyte CDK: Test via airbyte CI test poetry packages #36497

Airbyte CDK: Test via airbyte CI test poetry packages #36497

erohmensing commented Mar 26, 2024 •

edited

Loading

vercel bot commented Mar 26, 2024 •

edited

Loading

erohmensing commented Mar 26, 2024 •

edited

Loading

natikgadzhi commented Mar 27, 2024

girarda commented Mar 27, 2024

alafanechere left a comment •

edited

Loading

alafanechere Mar 28, 2024

natikgadzhi Apr 2, 2024

alafanechere Apr 2, 2024

alafanechere Mar 28, 2024

erohmensing May 3, 2024

alafanechere Mar 28, 2024

girarda Mar 28, 2024

natikgadzhi Apr 2, 2024

erohmensing May 3, 2024

erohmensing commented May 3, 2024

Airbyte CDK: Test via airbyte CI test poetry packages #36497

Airbyte CDK: Test via airbyte CI test poetry packages #36497

Conversation

erohmensing commented Mar 26, 2024 • edited Loading

vercel bot commented Mar 26, 2024 • edited Loading

erohmensing commented Mar 26, 2024 • edited Loading

natikgadzhi commented Mar 27, 2024

girarda commented Mar 27, 2024

alafanechere left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erohmensing commented May 3, 2024

erohmensing commented Mar 26, 2024 •

edited

Loading

vercel bot commented Mar 26, 2024 •

edited

Loading

erohmensing commented Mar 26, 2024 •

edited

Loading

alafanechere left a comment •

edited

Loading