MSSQL remove normalization

[jbfbell]

Removes normalization for MSSQL, this is a breaking change

---

	🚀 This PR description was created by Ellipsis for commit ff2e49077f16f6eddc8e74b8b8b3f032ed393a5a.

Summary:

This PR removes normalization for MSSQL, updates various files to reflect this change and the new version, and adds a migration guide.

Key points:

• Removed normalization for MSSQL.
• Updated cdkVersionRequired to '0.30.1' and useLocalCdk to true in build.gradle files of destination-mssql-strict-encrypt and destination-mssql.
• Updated metadata.yaml files of both connectors to reflect new version and breaking changes.
• Updated MssqlStrictEncryptDestinationAcceptanceTest.java and MSSQLDestinationAcceptanceTest.java test files.
• Updated MSSQLDestination.java to reflect changes in destination handler and SQL generator.
• Updated SqlServerOperations.java to reflect changes in insert records function.
• Updated spec.json of destination-mssql to include raw_data_schema property.
• Added new mssql-migrations.md file to document migration process.

---

Generated with ❤️ by ellipsis.dev

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name	Status	Preview	Comments	Updated (UTC)
airbyte-docs	✅ Ready (Inspect)	Visit Preview	💬 Add feedback	Apr 22, 2024 5:45pm

[jbfbell]

• MSSQL remove normalization #36050 👈
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

Join @jbfbell and the rest of your teammates on Graphite

[ellipsis-dev]

👍 Looks good to me!

• Reviewed the entire pull request up to 969707defaa914e4118ccd9180e15b0a90f7b601
• Looked at 507 lines of code in 11 files
• Took 2 minutes and 31 seconds to review

More info

• Skipped 0 files when reviewing.
• Skipped posting 1 additional comments because they didn't meet confidence threshold of 50%.

1. airbyte-integrations/connectors/destination-mssql/src/main/java/io/airbyte/integrations/destination/mssql/MSSQLDestination.java:7:

• Assessed confidence : 10%
• Comment:
  The changes in this file look good. The removal of normalization and the introduction of new features like 's3-destinations' and 'typing-deduping' are correctly implemented.
• Reasoning:
  The PR removes normalization for MSSQL and introduces breaking changes. The changes are well documented in the PR description and in the metadata.yaml files. The code changes in the build.gradle files are updating the cdkVersionRequired and useLocalCdk, which seems fine. The changes in the Java files are mostly related to the removal of normalization and the introduction of new features like 's3-destinations' and 'typing-deduping'. The changes in the spec.json files are adding a new field 'raw_data_schema'. The changes in the test files are mostly refactoring and don't seem to introduce any issues. Overall, the changes look good and seem to be correctly implemented.

Workflow ID: wflow_L1x64FfxXnLixrP1

---

Not what you expected? You can customize the content of the reviews using rules. Learn more here.

[ellipsis-dev]

👍 Looks good to me!

• Performed an incremental review on ff2e49077f16f6eddc8e74b8b8b3f032ed393a5a
• Looked at 521 lines of code in 12 files
• Took 5 minutes and 49 seconds to review

More info

• Skipped 0 files when reviewing.
• Skipped posting 3 additional comments because they didn't meet confidence threshold of 50%.

1. airbyte-integrations/connectors/destination-mssql/src/main/java/io/airbyte/integrations/destination/mssql/MSSQLDestination.java:7:

• Assessed confidence : 0%
• Comment:
  The changes in this file seem to be in line with the intent of the PR, which is to remove normalization for MSSQL. The getDestinationHandler method has been updated to use NoOpJdbcDestinationHandler, and the getMigrations method now returns an empty list. The isV2Destination method now returns true, and the shouldAlwaysDisableTypeDedupe method also returns true. The getSqlGenerator method has been updated to use RawOnlySqlGenerator. These changes seem to be in line with the intent of the PR, and I didn't find any clear violations of best practices, logical bugs, performance bugs, or security bugs.
• Reasoning:
  The changes in this file seem to be in line with the intent of the PR, which is to remove normalization for MSSQL. The getDestinationHandler method has been updated to use NoOpJdbcDestinationHandler, and the getMigrations method now returns an empty list. The isV2Destination method now returns true, and the shouldAlwaysDisableTypeDedupe method also returns true. The getSqlGenerator method has been updated to use RawOnlySqlGenerator. These changes seem to be in line with the intent of the PR, and I didn't find any clear violations of best practices, logical bugs, performance bugs, or security bugs.

2. airbyte-integrations/connectors/destination-mssql/src/main/java/io/airbyte/integrations/destination/mssql/SqlServerOperations.java:4:

• Assessed confidence : 0%
• Comment:
  The changes in this file seem to be in line with the intent of the PR, which is to remove normalization for MSSQL. The insertRecords function has been updated to reflect changes in the insert records function. A new raw_data_schema property has been added to the spec.json of destination-mssql. These changes seem to be in line with the intent of the PR, and I didn't find any clear violations of best practices, logical bugs, performance bugs, or security bugs.
• Reasoning:
  The changes in this file seem to be in line with the intent of the PR, which is to remove normalization for MSSQL. The insertRecords function has been updated to reflect changes in the insert records function. A new raw_data_schema property has been added to the spec.json of destination-mssql. These changes seem to be in line with the intent of the PR, and I didn't find any clear violations of best practices, logical bugs, performance bugs, or security bugs.

3. airbyte-integrations/connectors/destination-mssql/src/main/resources/spec.json:114:

• Assessed confidence : 0%
• Comment:
  The changes in this file seem to be in line with the intent of the PR, which is to remove normalization for MSSQL. The insertRecords function has been updated to reflect changes in the insert records function. A new raw_data_schema property has been added to the spec.json of destination-mssql. These changes seem to be in line with the intent of the PR, and I didn't find any clear violations of best practices, logical bugs, performance bugs, or security bugs.
• Reasoning:
  The changes in this file seem to be in line with the intent of the PR, which is to remove normalization for MSSQL. The insertRecords function has been updated to reflect changes in the insert records function. A new raw_data_schema property has been added to the spec.json of destination-mssql. These changes seem to be in line with the intent of the PR, and I didn't find any clear violations of best practices, logical bugs, performance bugs, or security bugs.

Workflow ID: wflow_PIrj1mAYzVjlYdjm

---

Not what you expected? You can customize the content of the reviews using rules. Learn more here.

[cohenj20]

Am I wrong or does the MSSQL destination now only create raw airbyte tables with an _airbyte_data column containing a JSON blob of all source data? Is there really no way anymore to have airbyte deliver tabular source data to MSSQL?

[evantahler]

Am I wrong or does the MSSQL destination now only create raw airbyte tables with an _airbyte_data column containing a JSON blob of all source data? Is there really no way anymore to have airbyte deliver tabular source data to MSSQL?

Correct - The connector needed to be updated to support a number of new features (record change history, checkpointing etc), and the removal of dbt-based normalization from the airbyte platform. As this destination is not certified, we haven't yet prioritized adding the new Destination V2 final tables.

[cohenj20]

I understand. Do you have a rough estimate on when users could expect this enhancement to be shipped? And does Airbyte have a plan to certify this destination?

[evantahler]

We do not have plans to certify the destination-mssql at this time. That said, as this is a community connector, if you are interested in adding those features, let us know

[brywhi]

This is one of my most heavily used destinations. Normalization is also important for me- I'll have to camp out on an old version for a while it seems.