connectors-ci/cat: validate image under test is the right one #27919

alafanechere · 2023-07-03T15:08:00Z

What

Addresses #27867
This PR aims at validating that the current image under test in CAT is the correct one to alert us explicitly when abusive caching might be happening on our test pipelines

How

On Python connector build: make the pipeline fail if the version label on the docker image does not match the metadata.yaml version
When we pass the connector under test image to CAT tag it with the git revision on not :dev and patch acceptance-test-config to use this git revision tag.
Add a test in CAT that fails if the docker image version label does not match the metadata.yaml version.

evantahler · 2023-07-03T15:41:09Z

tools/ci_connector_ops/ci_connector_ops/pipelines/actions/environments.py

+    if not await connector_container.label("io.airbyte.version") == context.metadata["dockerImageTag"]:
+        raise Exception(
+            "Abusive caching might be happening. The connector container should have been built with the correct version as defined in metadata.yaml"
+        )


evantahler · 2023-07-03T15:42:16Z

...te-integrations/bases/connector-acceptance-test/connector_acceptance_test/tests/test_core.py

+        assert (
+            docker_runner.labels.get("io.airbyte.name") == connector_metadata["data"]["dockerRepository"]
+        ), "io.airbyte.name must be equal to dockerRepository in metadata.yaml"


One day, it would be nice to not need to modify the docker labels at all. I'm curious if we even still need them. I don't think the platform uses them at all...

Agreed! It's just a good enough way for me right now to try to detect and understand the CI problem we might have with caching.
The metadata.yaml is not packaged into the connector image on the Python connectors, if we revamped as planned the build logic for python connector I think we shall package metadata.yaml in the connector image.
Wdyt about a new connector command like metadata that would output the connector metadata?

I like it. Then I can airbybye-ci connectors metadata --name source-foo | jq :D

@evantahler I meant a connector command:
docker run airbyte/pokeapi metadata | jq

alafanechere · 2023-07-03T17:10:22Z

/legacy-test connector=connectors/source-openweather connector-acceptance-test-version=dev

🕑 connectors/source-openweather https://github.com/airbytehq/airbyte/actions/runs/5447005145
✅ connectors/source-openweather https://github.com/airbytehq/airbyte/actions/runs/5447005145
Python tests coverage:

Name                                      Stmts   Miss  Cover
-------------------------------------------------------------
source_openweather/source.py                 22      0   100%
source_openweather/extra_validations.py      22      0   100%
source_openweather/__init__.py                2      0   100%
source_openweather/streams.py                33      5    85%
-------------------------------------------------------------
TOTAL                                        79      5    94%
	 Name                                                        Stmts   Miss  Cover   Missing
	 -----------------------------------------------------------------------------------------
	 connector_acceptance_test/base.py                              12      4    67%   16-19
	 connector_acceptance_test/config.py                           155      8    95%   87, 93, 255, 259, 264-265, 270-271
	 connector_acceptance_test/conftest.py                         236    107    55%   38, 44-46, 51, 56, 61, 84, 90, 96-98, 117, 122-124, 130-132, 138-139, 144-145, 150, 161, 170-179, 185-190, 217, 241, 272, 278, 286-294, 306-325, 333-346, 351-357, 364-375, 382-398, 403
	 connector_acceptance_test/plugin.py                            69     25    64%   22-23, 31, 36, 120-140, 144-148
	 connector_acceptance_test/tests/test_core.py                  572     99    83%   55, 60, 99-110, 115-122, 126-127, 375, 395, 420, 500, 539-544, 553-554, 562-581, 614-625, 629-634, 640, 673-678, 762-769, 908-911, 916, 921, 981-982, 988, 1028-1038, 1055-1056, 1058, 1076-1081
	 connector_acceptance_test/tests/test_incremental.py           162     14    91%   58-65, 70-83, 252
	 connector_acceptance_test/utils/asserts.py                     48      2    96%   78-79
	 connector_acceptance_test/utils/backward_compatibility.py     135      1    99%   209
	 connector_acceptance_test/utils/common.py                      94     10    89%   16-17, 32-38, 72, 75
	 connector_acceptance_test/utils/compare.py                     64     23    64%   21-51, 68, 101-103
	 connector_acceptance_test/utils/connector_runner.py           137     34    75%   30-33, 53-54, 57-61, 64-65, 80-82, 85-87, 90-92, 95-97, 100-102, 132-133, 167-169, 189, 220
	 connector_acceptance_test/utils/json_schema_helper.py         117     13    89%   31-32, 39, 42, 66-69, 97, 121, 210-212
	 -----------------------------------------------------------------------------------------
	 TOTAL                                                        1877    340    82%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:615: The previous and actual discovered catalogs are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:712: This tests currently leads to too much failures. We need to fix the connectors at scale first.
================= 38 passed, 4 skipped, 42 warnings in 30.66s ==================

alafanechere · 2023-07-03T17:26:34Z

/legacy-publish connector=bases/connector-acceptance-test

🕑 Publishing the following connectors:
bases/connector-acceptance-test
https://github.com/airbytehq/airbyte/actions/runs/5447126911

Connector	Version	Did it publish?
bases/connector-acceptance-test		✅

if you have connectors that successfully published but failed definition generation, follow step 4 here ▶️

alafanechere · 2023-07-03T17:59:52Z

/legacy-test connector=connectors/source-openweather

🕑 connectors/source-openweather https://github.com/airbytehq/airbyte/actions/runs/5447338390
✅ connectors/source-openweather https://github.com/airbytehq/airbyte/actions/runs/5447338390
Python tests coverage:

Name                                      Stmts   Miss  Cover
-------------------------------------------------------------
source_openweather/source.py                 22      0   100%
source_openweather/extra_validations.py      22      0   100%
source_openweather/__init__.py                2      0   100%
source_openweather/streams.py                33      5    85%
-------------------------------------------------------------
TOTAL                                        79      5    94%

Build Passed

Test summary info:

=========================== short test summary info ============================
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/plugin.py:63: Skipping TestFullRefresh.test_sequential_reads: not found in the config.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:100: The previous and actual specifications are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:615: The previous and actual discovered catalogs are identical.
SKIPPED [1] ../usr/local/lib/python3.9/site-packages/connector_acceptance_test/tests/test_core.py:712: This tests currently leads to too much failures. We need to fix the connectors at scale first.
================= 38 passed, 4 skipped, 42 warnings in 30.83s ==================

alafanechere added 2 commits July 3, 2023 17:02

cat: add tests to check the image labels match metadata.yaml

c5a04b2

patch acceptance test config to not use dev

0ef1dad

alafanechere marked this pull request as ready for review July 3, 2023 15:19

alafanechere requested review from bnchrch and a team July 3, 2023 15:19

evantahler approved these changes Jul 3, 2023

View reviewed changes

increase test spec timeout

be62ceb

raise and handle DaggerError

2888f05

alafanechere enabled auto-merge (squash) July 3, 2023 18:02

alafanechere merged commit 816e837 into master Jul 3, 2023

alafanechere deleted the augustin/connectors-ci/check-connector-image-under-test-is-the-right-one branch July 3, 2023 18:19

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

connectors-ci/cat: validate image under test is the right one #27919

connectors-ci/cat: validate image under test is the right one #27919

alafanechere commented Jul 3, 2023 •

edited

Loading

evantahler Jul 3, 2023

evantahler Jul 3, 2023

alafanechere Jul 3, 2023

evantahler Jul 3, 2023

alafanechere Jul 3, 2023

alafanechere commented Jul 3, 2023 •

edited by github-actions bot

Loading

alafanechere commented Jul 3, 2023 •

edited by github-actions bot

Loading

alafanechere commented Jul 3, 2023 •

edited by github-actions bot

Loading

connectors-ci/cat: validate image under test is the right one #27919

connectors-ci/cat: validate image under test is the right one #27919

Conversation

alafanechere commented Jul 3, 2023 • edited Loading

What

How

evantahler Jul 3, 2023

Choose a reason for hiding this comment

evantahler Jul 3, 2023

Choose a reason for hiding this comment

alafanechere Jul 3, 2023

Choose a reason for hiding this comment

evantahler Jul 3, 2023

Choose a reason for hiding this comment

alafanechere Jul 3, 2023

Choose a reason for hiding this comment

alafanechere commented Jul 3, 2023 • edited by github-actions bot Loading

Build Passed

alafanechere commented Jul 3, 2023 • edited by github-actions bot Loading

alafanechere commented Jul 3, 2023 • edited by github-actions bot Loading

Build Passed

alafanechere commented Jul 3, 2023 •

edited

Loading

alafanechere commented Jul 3, 2023 •

edited by github-actions bot

Loading

alafanechere commented Jul 3, 2023 •

edited by github-actions bot

Loading

alafanechere commented Jul 3, 2023 •

edited by github-actions bot

Loading