Skip to content

Accelerate snowflake integration test #10374

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 3 commits into from
Feb 16, 2022
Merged

Accelerate snowflake integration test #10374

merged 3 commits into from
Feb 16, 2022

Conversation

tuliren
Copy link
Contributor

@tuliren tuliren commented Feb 16, 2022 •
edited
Loading

  • Turn off dbt doc generation saves 15 minutes for each test run.
  • Fix snowflake source destination test by using a different schema each time.

@github-actions github-actions bot added the area/connectors Connector related issues label Feb 16, 2022
@tuliren
Copy link
Contributor Author

tuliren commented Feb 16, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-snowflake

🕑 connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/1852097758
✅ connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/1852097758
No Python unittests run

@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets February 16, 2022 10:01 Inactive
@tuliren tuliren changed the title Fix connector builds Fix snowflake integration test Feb 16, 2022
@tuliren
Copy link
Contributor Author

tuliren commented Feb 16, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/source-snowflake

🕑 connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/1854732576
✅ connectors/source-snowflake https://github.com/airbytehq/airbyte/actions/runs/1854732576
No Python unittests run

@tuliren
Copy link
Contributor Author

tuliren commented Feb 16, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1854732892
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/1854732892
Python tests coverage:

Name                                                                                                                            Stmts   Miss  Cover
---------------------------------------------------------------------------------------------------------------------------------------------------
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/__init__.py                      2      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/__init__.py               1      0   100%
/actions-runner/_work/airbyte/airbyte/airbyte-integrations/bases/airbyte-protocol/airbyte_protocol/models/airbyte_protocol.py     124      0   100%
normalization/__init__.py                                                                                                           4      0   100%
normalization/destination_type.py                                                                                                  13      0   100%
normalization/transform_catalog/__init__.py                                                                                         2      0   100%
normalization/transform_catalog/catalog_processor.py                                                                              143     77    46%
normalization/transform_catalog/destination_name_transformer.py                                                                   155      8    95%
normalization/transform_catalog/reserved_keywords.py                                                                               13      0   100%
normalization/transform_catalog/stream_processor.py                                                                               520    333    36%
normalization/transform_catalog/table_name_registry.py                                                                            174     34    80%
normalization/transform_catalog/transform.py                                                                                       45     26    42%
normalization/transform_catalog/utils.py                                                                                           33      7    79%
normalization/transform_config/__init__.py                                                                                          2      0   100%
normalization/transform_config/transform.py                                                                                       148     34    77%
---------------------------------------------------------------------------------------------------------------------------------------------------
TOTAL                                                                                                                            1379    519    62%

@tuliren tuliren temporarily deployed to more-secrets February 16, 2022 19:11 Inactive
@tuliren tuliren temporarily deployed to more-secrets February 16, 2022 19:11 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets February 16, 2022 19:12 Inactive
@octavia-squidington-iii octavia-squidington-iii temporarily deployed to more-secrets February 16, 2022 19:12 Inactive
@tuliren tuliren marked this pull request as ready for review February 16, 2022 19:54
@tuliren
Copy link
Contributor Author

tuliren commented Feb 16, 2022 •
edited
Loading

The destination integration test runtime for snowflake decreases from 100 minutes to 30 minutes.

@tuliren tuliren requested a review from subodh1810 February 16, 2022 19:55
@tuliren tuliren changed the title Fix snowflake integration test Accelerate snowflake integration test Feb 16, 2022
@tuliren tuliren temporarily deployed to more-secrets February 16, 2022 20:01 Inactive
@tuliren tuliren temporarily deployed to more-secrets February 16, 2022 20:01 Inactive
@tuliren tuliren merged commit ea41255 into master Feb 16, 2022
@tuliren tuliren deleted the liren/connector-builds branch February 16, 2022 20:16
@octavia-squidington-iii octavia-squidington-iii mentioned this pull request Feb 17, 2022
Merged
@tuliren tuliren linked an issue Feb 23, 2022 that may be closed by this pull request
Snowflake destination acceptance test runtime is too long #10340
Closed
@tuliren tuliren mentioned this pull request Feb 23, 2022
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@subodh1810 subodh1810 Awaiting requested review from subodh1810

Assignees
No one assigned
Labels
area/connectors Connector related issues
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Snowflake destination acceptance test runtime is too long
2 participants
@tuliren @octavia-squidington-iii