Skip to content

Improve check connection performance for all JDBC-related DBs #12820

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 11 commits into from
May 24, 2022
Merged

Improve check connection performance for all JDBC-related DBs #12820

merged 11 commits into from
May 24, 2022

Conversation

alexandr-shegeda
Copy link
Contributor

What

Describe what the change is solving
It helps to add screenshots if it affects the frontend.

How

Optimized check operation by removing the SELECT SCHEMA part.

Recommended reading order

  1. AbstractJdbcDestination.java
  2. CreateDropTableOperation.java
  3. SnowflakeInternalStagingDestination.java

🚨 User Impact 🚨

No user impact. Removed redundant schema retrieve operation in the scope of the check method.

Pre-merge Checklist

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

@alexandr-shegeda alexandr-shegeda requested a review from grishick May 12, 2022 19:31
@github-actions github-actions bot added the area/connectors Connector related issues label May 12, 2022
@alexandr-shegeda alexandr-shegeda mentioned this pull request May 12, 2022
Closed
2 tasks
@alexandr-shegeda alexandr-shegeda linked an issue May 12, 2022 that may be closed by this pull request
Destination Snowflake: improve check connection performance #10376
Closed
1 task
@VitaliiMaltsev VitaliiMaltsev mentioned this pull request May 13, 2022
Closed
@github-actions github-actions bot added the area/documentation Improvements or additions to documentation label May 17, 2022
@alexandr-shegeda alexandr-shegeda temporarily deployed to more-secrets May 17, 2022 17:59 Inactive
@alexandr-shegeda alexandr-shegeda temporarily deployed to more-secrets May 17, 2022 17:59 Inactive
@alexandr-shegeda alexandr-shegeda temporarily deployed to more-secrets May 17, 2022 18:02 Inactive
@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 17, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2340495556
❌ Failed to publish connectors/destination-snowflake
❌ Couldn't auto-bump version for connectors/destination-snowflake

@alexandr-shegeda alexandr-shegeda temporarily deployed to more-secrets May 17, 2022 18:02 Inactive
@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 17, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-snowflake

🕑 connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2340955612
✅ connectors/destination-snowflake https://github.com/airbytehq/airbyte/actions/runs/2340955612
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_config/transform.py                         159     31    81%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 543    352    35%
-------------------------------------------------------------------------------------
TOTAL                                                              1333    559    58%

@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 17, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-azure-blob-storage

🕑 connectors/destination-azure-blob-storage https://github.com/airbytehq/airbyte/actions/runs/2340960074
✅ connectors/destination-azure-blob-storage https://github.com/airbytehq/airbyte/actions/runs/2340960074
No Python unittests run

@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 17, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-clickhouse

🕑 connectors/destination-clickhouse https://github.com/airbytehq/airbyte/actions/runs/2340960710
✅ connectors/destination-clickhouse https://github.com/airbytehq/airbyte/actions/runs/2340960710
No Python unittests run

@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 17, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-clickhouse-strict-encrypt

🕑 connectors/destination-clickhouse-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2340961044
✅ connectors/destination-clickhouse-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2340961044
No Python unittests run

@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 17, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-dynamodb

🕑 connectors/destination-dynamodb https://github.com/airbytehq/airbyte/actions/runs/2340961897
✅ connectors/destination-dynamodb https://github.com/airbytehq/airbyte/actions/runs/2340961897
No Python unittests run

@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 17, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2340963148
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2340963148
No Python unittests run

@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 17, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-mariadb-columnstore

🕑 connectors/destination-mariadb-columnstore https://github.com/airbytehq/airbyte/actions/runs/2340963518

@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 17, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-mssql

🕑 connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/2340964426
❌ connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/2340964426
🐛

@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 17, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-mssql-strict-encrypt

🕑 connectors/destination-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2340964959
❌ connectors/destination-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2340964959
🐛

@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 17, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-mysql

🕑 connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/2340965102
❌ connectors/destination-mysql https://github.com/airbytehq/airbyte/actions/runs/2340965102
🐛

@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 17, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-mysql-strict-encrypt

🕑 connectors/destination-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2340965746
❌ connectors/destination-mysql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2340965746
🐛

@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 17, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-oracle

🕑 connectors/destination-oracle https://github.com/airbytehq/airbyte/actions/runs/2340965974
✅ connectors/destination-oracle https://github.com/airbytehq/airbyte/actions/runs/2340965974
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_config/transform.py                         159     31    81%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 543    352    35%
-------------------------------------------------------------------------------------
TOTAL                                                              1333    559    58%

@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 17, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-oracle-strict-encrypt

🕑 connectors/destination-oracle-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2340966922
❌ connectors/destination-oracle-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2340966922
🐛

@alexandr-shegeda alexandr-shegeda temporarily deployed to more-secrets May 18, 2022 18:49 Inactive
@alexandr-shegeda alexandr-shegeda temporarily deployed to more-secrets May 18, 2022 18:49 Inactive
@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 19, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-oracle-strict-encrypt

🕑 connectors/destination-oracle-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2350318285
❌ Failed to publish connectors/destination-oracle-strict-encrypt
❌ Couldn't auto-bump version for connectors/destination-oracle-strict-encrypt

@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 19, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-oracle

🕑 connectors/destination-oracle https://github.com/airbytehq/airbyte/actions/runs/2350318975
❌ Failed to publish connectors/destination-oracle
❌ Couldn't auto-bump version for connectors/destination-oracle

@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 19, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-mssql-strict-encrypt

🕑 connectors/destination-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2350320963
❌ connectors/destination-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2350320963
🐛 https://gradle.com/s/frzcz3oxj4qis

@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 19, 2022 •
edited by github-actions bot
Loading

/test connector=connectors/destination-mssql

🕑 connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/2350321714
✅ connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/2350321714
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_config/transform.py                         159     31    81%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 543    352    35%
-------------------------------------------------------------------------------------
TOTAL                                                              1333    559    58%

@alexandr-shegeda alexandr-shegeda requested a deployment to more-secrets May 19, 2022 15:30 Abandoned
@alexandr-shegeda alexandr-shegeda temporarily deployed to more-secrets May 19, 2022 15:30 Inactive
@alexandr-shegeda alexandr-shegeda temporarily deployed to more-secrets May 19, 2022 15:30 Inactive
@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 19, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-mssql

🕑 connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/2353082006
❌ Failed to publish connectors/destination-mssql
❌ Couldn't auto-bump version for connectors/destination-mssql

@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 19, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-mssql-strict-encrypt

🕑 connectors/destination-mssql-strict-encrypt https://github.com/airbytehq/airbyte/actions/runs/2353084384
❌ Failed to publish connectors/destination-mssql-strict-encrypt
❌ Couldn't auto-bump version for connectors/destination-mssql-strict-encrypt

@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 19, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-mssql auto-bump-version=false

🕑 connectors/destination-mssql https://github.com/airbytehq/airbyte/actions/runs/2353349802
❌ Failed to publish connectors/destination-mssql

@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 19, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-redshift auto-bump-version=false

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2353399852
🚀 Successfully published connectors/destination-redshift
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2353399852

@alexandertsukanov alexandertsukanov mentioned this pull request May 20, 2022
Merged
14 tasks
@marcosmarxm
Copy link
Member

@alexandr-shegeda can you bump the redshift destination again? I didn't find information about the 35 version and excluded from Dockerhub and publish #12940 with it.

@marcosmarxm
Copy link
Member

Also would be better change the title of your PR. It's doing way more than editing Snowflake destination.

@alexandr-shegeda alexandr-shegeda changed the title 10376 Destination Snowflake: improve check connection performance Improve check connection performance for all JDBC-related DBs May 20, 2022
@alexandr-shegeda
Copy link
Contributor Author

alexandr-shegeda commented May 23, 2022 •
edited by github-actions bot
Loading

/publish connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2371102365
🚀 Successfully published connectors/destination-redshift
❌ Couldn't auto-bump version for connectors/destination-redshift

@alexandr-shegeda alexandr-shegeda temporarily deployed to more-secrets May 23, 2022 12:01 Inactive
@alexandr-shegeda alexandr-shegeda merged commit 44a43f0 into master May 24, 2022
@alexandr-shegeda alexandr-shegeda deleted the osheg/10376-improved-jdbc-check-perf branch May 24, 2022 13:06
@alexandr-shegeda alexandr-shegeda temporarily deployed to more-secrets May 24, 2022 13:06 Inactive
@octavia-squidington-iii octavia-squidington-iii mentioned this pull request May 25, 2022
Merged
jscottpolevault pushed a commit to jscottpolevault/airbyte that referenced this pull request Jun 1, 2022
@alexandr-shegeda
Improve check connection performance for all JDBC-related DBs (airbyt…
c4030fe 
…ehq#12820)

* removed redundant read permission check for destination

* bump docker image versions, update definitions and changelog

* bump docker image versions, update definitions and changelog

* update changelog

* fix for MSSQL and Oracle tests

* fix for MSSQL strict encrypt

* fixed conflicts

* revert versions that were not published
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@grishick grishick grishick approved these changes

Assignees
No one assigned
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Destination Snowflake: improve check connection performance
3 participants
@alexandr-shegeda @marcosmarxm @grishick