Revert case-senstivity in Snowflake final tables (and don't do it in Redshift)

[evantahler]

Steps:

1. Chang the code to not specify case-sensitivity - remove a lot of quotes.
2. Add code that detects if there is a lower-case table name.
   • Make a new findExistingUpCassedV2Table() like findExistingTable() which will be used within the soft-reset
   • Soft reset (e.g. blow away the loaded_at values in the raw table)

Notes:

• don’t force-uppercase things, implicitly do this by not specifying the case and let the DB handle it
• This migration will leave behind the lower-cased old V2 table... that's OK

Future note:

• We also want to rename the raw tables so they are also case-insensitive, but that's another story because (1) that will slow this down and (2) can't be solved by a soft reset.

[rengler-noodle]

All of our Snowflake destination syncs are experiencing issues that sound related to the title of this issue.

We believe the previous behavior was for airbyte to uppercase all table and column names. Now the syncs are creating new tables whose name and column names appear to match the source casing.

This is leading to some failed syncs and interrupting all downstream data flows.

Is this issue going to be the place to capture updates on this problem we're experiencing?

Just seeing the edit - thank you

[rrittsteiger]

We are experiencing the same issue.

Thank you for your help!

[evantahler]

We will be rolling out this bugfix for Snowflake final tables this weeks

[rengler-noodle]

@evantahler When this change rolled out that this issue is referring to, we started experiencing problems with REPEATED RECORD types (from BigQuery). The data in the these columns has been wiped out with nulls and _AIRBYTE_META reports problems with any field that is a repeated record.

Now I have completely deleted our Snowflake tables and reset and re-synced the connection via airbyte cloud, but these columns still fail to come through.

I'm just beginning to dig into this issue and I'm wondering if you suspect it could be related to the Snowflake destination. I suppose it could be a BigQuery source issue but the syncs were working flawlessly before the change last Friday.

[edgao]

that certainly sounds related to the dv2 rollout! Can you submit a new issue to improve discoverability for other folks who might have the same problem? A few more details that would be useful for investigating:

• if you describe the snowflake table, what type is the repeated column?
• can you show us an example of what the repeated value looks like in the raw data? (feel free to redact sensitive information)

I suspect source-bigquery is misbehaving in some way, which happened to overlap with the old normalization module misbehaving in an opposite way... let's find out together!