Destination snowflake: table casing migration #30068

edgao · 2023-08-31T23:39:44Z

based on top of #30056.

Add the migration, which detects an existing lowercased table and runs a soft reset to create the uppercased table. First commit (7f71d1c) is just renaming a class, and can be reviewed in isolation.

Reminder: This leaves the raw tables and lowercased tables untouched.

(I'm being kind of sloppy; there's now a generic V2TableMigrator class where I guess we can dump all these changes, i.e. the bigquery raw table change + this snowflake casing change? ideally I think we'd want separate migrators per migration type 🤷 but that's a refactor for the future, I think)

Example:
after 3.0.0 sync

after dev image sync

github-actions · 2023-08-31T23:40:06Z

Before Merging a Connector Pull Request

Wow! What a great pull request you have here! 🎉

To merge this PR, ensure the following has been done/considered for each connector added or updated:

PR name follows PR naming conventions
Breaking changes are considered. If a Breaking Change is being introduced, ensure an Airbyte engineer has created a Breaking Change Plan.
Connector version has been incremented in the Dockerfile and metadata.yaml according to our Semantic Versioning for Connectors guidelines
You've updated the connector's metadata.yaml file any other relevant changes, including a breakingChanges entry for major version bumps. See metadata.yaml docs
Secrets in the connector's spec are annotated with airbyte_secret
All documentation files are up to date. (README.md, bootstrap.md, docs.md, etc...)
Changelog updated in docs/integrations/<source or destination>/<name>.md with an entry for the new version. See changelog example
Migration guide updated in docs/integrations/<source or destination>/<name>-migrations.md with an entry for the new version, if the version is a breaking change. See migration guide example
If set, you've ensured the icon is present in the platform-internal repo. (Docs)

If the checklist is complete, but the CI check is failing,

Check for hidden checklists in your PR description
Toggle the github label checklist-action-run on/off to re-run the checklist CI.

github-actions · 2023-09-01T15:26:08Z

destination-snowflake test report (commit `cc4d394286`) - ❌

⏲️ Total pipeline duration: 14mn06s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

github-actions · 2023-09-01T15:40:38Z

destination-bigquery test report (commit `cc4d394286`) - ❌

⏲️ Total pipeline duration: 14mn18s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

edgao · 2023-09-01T16:19:05Z

derp. this needs to check that the uppercase table doesn't yet exist, or else we're just soft resetting on every sync.

github-actions · 2023-09-01T16:47:56Z

destination-bigquery test report (commit `be1a6e8ea2`) - ❌

⏲️ Total pipeline duration: 03mn24s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

github-actions · 2023-09-01T17:03:37Z

destination-snowflake test report (commit `be1a6e8ea2`) - ❌

⏲️ Total pipeline duration: 15mn26s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

edgao · 2023-09-01T18:25:47Z

lemme test if we can skip this migration in overwrite syncs. I think that's the last thing before I call this fully code-complete.

.../io/airbyte/integrations/destination/snowflake/typing_deduping/SnowflakeV2TableMigrator.java

github-actions · 2023-09-01T20:41:26Z

destination-snowflake test report (commit `f5db3aac8b`) - ❌

⏲️ Total pipeline duration: 20mn48s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-snowflake docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-snowflake/metadata.yaml	✅
Connector version semver check	❌
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-snowflake test

github-actions · 2023-09-01T20:54:12Z

destination-bigquery test report (commit `f5db3aac8b`) - ❌

⏲️ Total pipeline duration: 12mn35s

Step	Result
Java Connector Unit Tests	✅
Build connector tar	✅
Build destination-bigquery docker image for platform linux/x86_64	✅
Java Connector Integration Tests	✅
Validate airbyte-integrations/connectors/destination-bigquery/metadata.yaml	✅
Connector version semver check	✅
Connector version increment check	❌
QA checks	✅

🔗 View the logs here

☁️ View runs for commit in Dagger Cloud

Please note that tests are only run on PR ready for review. Please set your PR to draft mode to not flood the CI engine and upstream service on following commits.
You can run the same pipeline locally on this branch with the airbyte-ci tool with the following command

airbyte-ci connectors --name=destination-bigquery test

edgao added 3 commits August 31, 2023 14:28

rename

7f71d1c

add migration code

ca7249d

unused method

e11e693

edgao requested a review from a team as a code owner August 31, 2023 23:39

octavia-squidington-iii added the area/connectors Connector related issues label Aug 31, 2023

edgao mentioned this pull request Aug 31, 2023

🐛 Destination snowflake: Create final tables with uppercase naming #30056

Merged

octavia-squidington-iii added connectors/destination/bigquery connectors/destination/snowflake labels Aug 31, 2023

Automated Commit - Formatting Changes

1a5d600

This comment was marked as outdated.

Sign in to view

edgao and others added 2 commits September 1, 2023 08:09

properly cleanup

cc4d394

Automated Commit - Formatting Changes

397827b

edgao and others added 2 commits September 1, 2023 09:42

improve migration check

be1a6e8

Automated Commit - Formatting Changes

6b179dc

jbfbell approved these changes Sep 1, 2023

View reviewed changes

.../io/airbyte/integrations/destination/snowflake/typing_deduping/SnowflakeV2TableMigrator.java Outdated Show resolved Hide resolved

edgao and others added 4 commits September 1, 2023 13:18

skip migration in overwrite mode

2f5b24a

fix log message

2ac4dba

rename for clarity

f5db3aa

Automated Commit - Formatting Changes

546a590

edgao merged commit abde8fa into edgao/dv2/snowflake/case_sensitivity Sep 1, 2023

edgao deleted the edgao/dv2/snowflake/case_sensitivity_migration branch September 1, 2023 20:35

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Destination snowflake: table casing migration #30068

Destination snowflake: table casing migration #30068

edgao commented Aug 31, 2023 •

edited

Loading

github-actions bot commented Aug 31, 2023

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

edgao commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

edgao commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

Destination snowflake: table casing migration #30068

Destination snowflake: table casing migration #30068

Conversation

edgao commented Aug 31, 2023 • edited Loading

github-actions bot commented Aug 31, 2023

Before Merging a Connector Pull Request

This comment was marked as outdated.

This comment was marked as outdated.

github-actions bot commented Sep 1, 2023

destination-snowflake test report (commit cc4d394286) - ❌

github-actions bot commented Sep 1, 2023

destination-bigquery test report (commit cc4d394286) - ❌

edgao commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

destination-bigquery test report (commit be1a6e8ea2) - ❌

github-actions bot commented Sep 1, 2023

destination-snowflake test report (commit be1a6e8ea2) - ❌

edgao commented Sep 1, 2023

github-actions bot commented Sep 1, 2023

destination-snowflake test report (commit f5db3aac8b) - ❌

github-actions bot commented Sep 1, 2023

destination-bigquery test report (commit f5db3aac8b) - ❌

edgao commented Aug 31, 2023 •

edited

Loading

destination-snowflake test report (commit `cc4d394286`) - ❌

destination-bigquery test report (commit `cc4d394286`) - ❌

destination-bigquery test report (commit `be1a6e8ea2`) - ❌

destination-snowflake test report (commit `be1a6e8ea2`) - ❌

destination-snowflake test report (commit `f5db3aac8b`) - ❌

destination-bigquery test report (commit `f5db3aac8b`) - ❌