Skip to content

Commit 511819b

Browse files
marcosmarxmjzcruiseredgaoChristopheDuong
authored
Normalization fix Prefix Tables starting with number (#9301)
* add normalization-clickhouse docker build step * bump normalization version * small changes gradle * fix settings gradle * fix eof file * correct clickhouse normalization * Refactor jinja template for scd (#9278) * merge chris code and regenerate sql files * correct scd post-hook generation for snowflake * fix scd table for snowflake prefix table with number * scd fix for all destinations * use quote * use normalize column for post-hook * change logic to apply quote * add logic to handle prefix for mssql and oracle * run tests * correct unit test * bump normalization version Co-authored-by: James Zhao <[email protected]> Co-authored-by: Edward Gao <[email protected]> Co-authored-by: Christophe Duong <[email protected]>
1 parent 4190dbc commit 511819b

File tree

69 files changed

+354
-708
lines changed

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

69 files changed

+354
-708
lines changed

‎airbyte-integrations/bases/base-normalization/Dockerfile

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -28,5 +28,5 @@ WORKDIR /airbyte
2828
ENV AIRBYTE_ENTRYPOINT "/airbyte/entrypoint.sh"
2929
ENTRYPOINT ["/airbyte/entrypoint.sh"]
3030

31-
LABEL io.airbyte.version=0.1.62
31+
LABEL io.airbyte.version=0.1.63
3232
LABEL io.airbyte.name=airbyte/normalization

‎airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/bigquery/test_nested_streams/models/generated/airbyte_incremental/scd/test_normalization/nested_stream_with_complex_columns_resulting_into_long_names_scd.sql

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
partition_by = {"field": "_airbyte_active_row", "data_type": "int64", "range": {"start": 0, "end": 1, "interval": 1}},
44
unique_key = "_airbyte_unique_key_scd",
55
schema = "test_normalization",
6-
post_hook = ['drop view _airbyte_test_normalization.nested_stream_with_complex_columns_resulting_into_long_names_stg'],
6+
post_hook = ["drop view _airbyte_test_normalization.nested_stream_with_complex_columns_resulting_into_long_names_stg"],
77
tags = [ "top-level" ]
88
) }}
99
-- depends_on: ref('nested_stream_with_complex_columns_resulting_into_long_names_stg')

‎airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/bigquery/test_simple_streams/models/generated/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
partition_by = {"field": "_airbyte_active_row", "data_type": "int64", "range": {"start": 0, "end": 1, "interval": 1}},
44
unique_key = "_airbyte_unique_key_scd",
55
schema = "test_normalization",
6-
post_hook = ['drop view _airbyte_test_normalization.dedup_exchange_rate_stg'],
6+
post_hook = ["drop view _airbyte_test_normalization.dedup_exchange_rate_stg"],
77
tags = [ "top-level" ]
88
) }}
99
-- depends_on: ref('dedup_exchange_rate_stg')

‎airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/bigquery/test_simple_streams/models/generated/sources.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ sources:
66
schema: false
77
identifier: false
88
tables:
9+
- name: _airbyte_raw_1_prefix_startwith_number
910
- name: _airbyte_raw_dedup_cdc_excluded
1011
- name: _airbyte_raw_dedup_exchange_rate
1112
- name: _airbyte_raw_exchange_rate

‎airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/bigquery/test_simple_streams/modified_models/generated/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -3,7 +3,7 @@
33
partition_by = {"field": "_airbyte_active_row", "data_type": "int64", "range": {"start": 0, "end": 1, "interval": 1}},
44
unique_key = "_airbyte_unique_key_scd",
55
schema = "test_normalization",
6-
post_hook = ['drop view _airbyte_test_normalization.dedup_exchange_rate_stg'],
6+
post_hook = ["drop view _airbyte_test_normalization.dedup_exchange_rate_stg"],
77
tags = [ "top-level" ]
88
) }}
99
-- depends_on: ref('dedup_exchange_rate_stg')

‎airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/clickhouse/test_simple_streams/models/generated/airbyte_incremental/scd/test_normalization/dedup_cdc_excluded_scd.sql

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{{ config(
22
unique_key = "_airbyte_unique_key_scd",
33
schema = "test_normalization",
4-
post_hook = ['drop view _airbyte_test_normalization.dedup_cdc_excluded_stg'],
4+
post_hook = ["drop view _airbyte_test_normalization.dedup_cdc_excluded_stg"],
55
tags = [ "top-level" ]
66
) }}
77
-- depends_on: ref('dedup_cdc_excluded_stg')

‎airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/clickhouse/test_simple_streams/models/generated/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{{ config(
22
unique_key = "_airbyte_unique_key_scd",
33
schema = "test_normalization",
4-
post_hook = ['drop view _airbyte_test_normalization.dedup_exchange_rate_stg'],
4+
post_hook = ["drop view _airbyte_test_normalization.dedup_exchange_rate_stg"],
55
tags = [ "top-level" ]
66
) }}
77
-- depends_on: ref('dedup_exchange_rate_stg')

‎airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/clickhouse/test_simple_streams/models/generated/sources.yml

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -6,6 +6,7 @@ sources:
66
schema: false
77
identifier: false
88
tables:
9+
- name: _airbyte_raw_1_prefix_startwith_number
910
- name: _airbyte_raw_dedup_cdc_excluded
1011
- name: _airbyte_raw_dedup_exchange_rate
1112
- name: _airbyte_raw_exchange_rate

‎airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mssql/test_nested_streams/models/generated/airbyte_incremental/scd/test_normalization/nested_stream_with_co__lting_into_long_names_scd.sql

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{{ config(
22
unique_key = "_airbyte_unique_key_scd",
33
schema = "test_normalization",
4-
post_hook = ['drop view _airbyte_test_normalization.nested_stream_with_co__lting_into_long_names_stg'],
4+
post_hook = ["drop view _airbyte_test_normalization.nested_stream_with_co__lting_into_long_names_stg"],
55
tags = [ "top-level" ]
66
) }}
77
-- depends_on: ref('nested_stream_with_co__lting_into_long_names_stg')

‎airbyte-integrations/bases/base-normalization/integration_tests/normalization_test_output/mssql/test_simple_streams/models/generated/airbyte_incremental/scd/test_normalization/dedup_exchange_rate_scd.sql

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,7 @@
11
{{ config(
22
unique_key = "_airbyte_unique_key_scd",
33
schema = "test_normalization",
4-
post_hook = ['drop view _airbyte_test_normalization.dedup_exchange_rate_stg'],
4+
post_hook = ["drop view _airbyte_test_normalization.dedup_exchange_rate_stg"],
55
tags = [ "top-level" ]
66
) }}
77
-- depends_on: ref('dedup_exchange_rate_stg')

0 commit comments

Comments
 (0)