🐛 Destination Redshift: Fix DATs blind spot - connection check #13090

alexandertsukanov · 2022-05-23T14:22:52Z

What

Investigating the issue I was able to find that problem appears due to redshiftDataTmpTableMode which was passed into RedshiftCopyS3Destination.java (Deprecated class) and subsequently fixed in the next PR:

https://github.com/airbytehq/airbyte/pull/12002/files

Looks like getSqlOperations() was called in some parent class (abstract class) and this was the root cause of the issue.
Currently, the CopyConsumerFactory approach was replaced with StagingOperations, and Redshift no longer uses CopyConsumerFactory.
(PR https://github.com/airbytehq/airbyte/pull/12601/files#diff-d6c1570221cad1a2b3eeb081801329eb)

How

Locally, I was not able to reproduce the issue with the latest Dest. redshift version. The only thing I have done is remove redshiftDataTmpTableMode as we don't need it anymore. The main purpose of mentioned ENUM was to support multiple
DATA TMP table types (e.g. VARCHAR and SUPER) - this is not required, as we have only SUPER now. Let's follow the KISS principle.

Also, I found that we have the next integration tests:

io/airbyte/integrations/destination/redshift/RedshiftS3StagingInsertDestinationAcceptanceTest.java (Designed to test S3 COPY)
io/airbyte/integrations/destination/redshift/RedshiftInsertDestinationAcceptanceTest.java (Designed to test standard INSERTS)

Both are the child of DestinationAcceptanceTest which contains testCheckConnection() method to verify whether the connection with the DB source was established successfully. ("Test connection button" on UI)

Pre-merge Checklist

Expand the relevant checklist and delete the others.

New Connector

Community member or Airbyter

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the connector is published, connector added to connector index as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub and connector version bumped by running the /publish command described here

Connector Generator

Issue acceptance criteria met
PR name follows PR naming conventions
If adding a new generator, add it to the list of scaffold modules being tested
The generator test modules (all connectors with -scaffold in their name) have been updated with the latest scaffold by running ./gradlew :airbyte-integrations:connector-templates:generator:testScaffoldTemplates then checking in your changes
Documentation which references the generator is updated as needed

Tests

Unit

Put your unit tests output here.

Integration

Put your integration tests output here.

Acceptance

Put your acceptance tests output here.

…um. Code refactoring and clean-up.

…tion_check

…um. Code refactoring and clean-up.

alexandertsukanov · 2022-05-23T14:36:05Z

/test connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2372000504
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2372000504
Python tests coverage:

Name                                                              Stmts   Miss  Cover
-------------------------------------------------------------------------------------
normalization/transform_config/__init__.py                            2      0   100%
normalization/transform_catalog/reserved_keywords.py                 13      0   100%
normalization/transform_catalog/__init__.py                           2      0   100%
normalization/destination_type.py                                    13      0   100%
normalization/__init__.py                                             4      0   100%
normalization/transform_catalog/destination_name_transformer.py     155      8    95%
normalization/transform_config/transform.py                         159     31    81%
normalization/transform_catalog/table_name_registry.py              174     34    80%
normalization/transform_catalog/utils.py                             38      9    76%
normalization/transform_catalog/dbt_macro.py                         22      7    68%
normalization/transform_catalog/catalog_processor.py                147     80    46%
normalization/transform_catalog/transform.py                         61     38    38%
normalization/transform_catalog/stream_processor.py                 543    352    35%
-------------------------------------------------------------------------------------
TOTAL                                                              1333    559    58%

edgao

I think the goal of the ticket was more focused on understanding why DestinationAcceptanceTest#testCheckConnection didn't fail. Were you able to checkout a broken version and investigate that? (the issue says 6ef2f80 was the latest broken version)

alexandertsukanov · 2022-05-24T05:00:19Z

I think the goal of the ticket was more focused on understanding why DestinationAcceptanceTest#testCheckConnection didn't fail. Were you able to checkout a broken version and investigate that? (the issue says 6ef2f80 was the latest broken version)

@edgao
Yep, I had also checked out 6ef2f80eba24675e03a290e6acc368cf64e4c0e8 and I tried to reproduce this issue but without success, on both combinations of options (With S3 Staging and without). All attempts of connection testing were successful.

Also, I think maybe it is somehow related to connection version bumping and the difference in the connector's specification (before the issue the version was 0.3.28). I have tried to download 0.3.28, save the settings, and migrate to 6ef2f80eba24675e03a290e6acc368cf64e4c0e8, but still without any result.

…tion_check

…tion_check # Conflicts: # docs/integrations/destinations/redshift.md

…pot_connection_check' into otsukanov/12459_Fix_DATs_blind_spot_connection_check # Conflicts: # docs/integrations/destinations/redshift.md

alexandertsukanov · 2022-05-25T14:52:30Z

/publish connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2385254820
❌ Failed to publish connectors/destination-redshift
❌ Couldn't auto-bump version for connectors/destination-redshift

alexandertsukanov · 2022-05-25T15:27:53Z

/publish connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2385469786
❌ Failed to publish connectors/destination-redshift
❌ Couldn't auto-bump version for connectors/destination-redshift

alexandertsukanov · 2022-05-25T15:41:46Z

/publish connector=connectors/destination-redshift

🕑 connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2385550096
🚀 Successfully published connectors/destination-redshift
🚀 Auto-bumped version for connectors/destination-redshift
✅ connectors/destination-redshift https://github.com/airbytehq/airbyte/actions/runs/2385550096

edgao · 2022-05-25T23:54:11Z

this was my bad, the issue had the wrong commit hash. I think ae8c4f2b3a should be broken, can you try again with that hash?

alexandertsukanov · 2022-05-26T08:27:39Z

@edgao I have switched to the commit, looks like the problem was here:

airbyte/airbyte-integrations/connectors/destination-jdbc/src/main/java/io/airbyte/integrations/destination/jdbc/copy/CopyDestination.java

Line 60 in ae8c4f2

    
           AbstractJdbcDestination.attemptSQLCreateAndDropTableOperations(outputSchema, database, nameTransformer, getSqlOperations());

and here:

airbyte/airbyte-integrations/connectors/destination-redshift/src/main/java/io/airbyte/integrations/destination/redshift/RedshiftCopyS3Destination.java

Lines 76 to 79 in ae8c4f2

    
           @Override 
        
           public SqlOperations getSqlOperations() { 
        
             throw new UnsupportedOperationException("RedshiftCopyS3Destination.getSqlOperations() without arguments not supported in Redshift destination connector."); 
        
           }

As Destination Redshift operated RedshiftCopyS3Destination.java before, for S3 Staging processing - the reasonable exception was thrown. Also, I have tried RedshiftCopyDestinationAcceptanceTest.testCheckConnection() and test was failed, what is correct behaviour. So, I don't have an idea why this part of the code wasn't caught by the integration test before.

The issue we faced.

edgao · 2022-05-27T00:12:07Z

something might be weird with how the CI tests work - I made a branch corresponding to that commit (#13246), where /test succeeds but running tests locally fails. I think we should reopen #12459 and investigate that behavior.

alexandertsukanov · 2022-05-27T06:27:08Z

@edgao found why this doesn't work on CI. The problem in GCM, the file SECRET_DESTINATION-REDSHIFT__CREDS doesn't contain info about the bucket that's why S3 flow never triggers. 🙂

Here is the log https://github.com/airbytehq/airbyte/runs/6621069306?check_suite_focus=true#step:11:8804
The logic in destination-redshift is pretty straightforward:

airbyte/airbyte-integrations/connectors/destination-redshift/src/main/java/io/airbyte/integrations/destination/redshift/RedshiftDestination.java

Lines 52 to 72 in ae8c4f2

    
           public static DestinationType determineUploadMode(final JsonNode config) { 
        
             final var bucketNode = config.get("s3_bucket_name"); 
        
             final var regionNode = config.get("s3_bucket_region"); 
        
             final var accessKeyIdNode = config.get("access_key_id"); 
        
             final var secretAccessKeyNode = config.get("secret_access_key"); 
        
             // Since region is a Json schema enum with an empty string default, we consider the empty string an 
        
             // unset field. 
        
             final var emptyRegion = regionNode == null || regionNode.asText().equals(""); 
        
             if (bucketNode == null && emptyRegion && accessKeyIdNode == null && secretAccessKeyNode == null) { 
        
               LOGGER.warn("The \"standard\" upload mode is not performant, and is not recommended for production. " + 
        
                           "Please use the Amazon S3 upload mode if you are syncing a large amount of data."); 
        
               return DestinationType.INSERT_WITH_SUPER_TMP_TYPE; 
        
             } 
        
             if (bucketNode == null || regionNode == null || accessKeyIdNode == null || secretAccessKeyNode == null) { 
        
               throw new RuntimeException("Error: Partially missing S3 Configuration."); 
        
             } 
        
             return DestinationType.COPY_S3_WITH_SUPER_TMP_TYPE; 
        
           }

alexandertsukanov · 2022-05-27T06:29:44Z

Updated secrets. Can now switch my job to the detective I think. 😁

edgao · 2022-05-27T20:37:21Z

nice catch! thanks for looking into it 👍

…tehq#13090) * airbyte-12459: Removed RedshiftDataTmpTableMode.java as not needed enum. Code refactoring and clean-up. * airbyte-12459: Removed RedshiftDataTmpTableMode.java as not needed enum. Code refactoring and clean-up. * airbyte-12459: Merged master. * airbyte-12459: Merged master. * auto-bump connector version Co-authored-by: Oleksandr Sheheda <[email protected]> Co-authored-by: Octavia Squidington III <[email protected]>

alexandertsukanov added 2 commits May 23, 2022 16:06

airbyte-12459: Removed RedshiftDataTmpTableMode.java as not needed en…

5df58a4

…um. Code refactoring and clean-up.

Merge branch 'master' into otsukanov/12459_Fix_DATs_blind_spot_connec…

986c532

…tion_check

alexandertsukanov requested review from edgao and alexandr-shegeda May 23, 2022 14:22

github-actions bot added the area/connectors Connector related issues label May 23, 2022

alexandertsukanov linked an issue May 23, 2022 that may be closed by this pull request

Fix DATs blind spot: connection check #12459

Closed

airbyte-12459: Removed RedshiftDataTmpTableMode.java as not needed en…

a687be8

…um. Code refactoring and clean-up.

github-actions bot added the area/documentation Improvements or additions to documentation label May 23, 2022

edgao approved these changes May 23, 2022

View reviewed changes

Merge branch 'master' into otsukanov/12459_Fix_DATs_blind_spot_connec…

593ac76

…tion_check

alexandr-shegeda marked this pull request as ready for review May 24, 2022 13:22

alexandertsukanov added 3 commits May 25, 2022 17:35

Merge branch 'master' into otsukanov/12459_Fix_DATs_blind_spot_connec…

fbcab26

…tion_check # Conflicts: # docs/integrations/destinations/redshift.md

airbyte-12459: Merged master.

70c65b4

Merge remote-tracking branch 'origin/otsukanov/12459_Fix_DATs_blind_s…

fe2b79d

…pot_connection_check' into otsukanov/12459_Fix_DATs_blind_spot_connection_check # Conflicts: # docs/integrations/destinations/redshift.md

alexandr-shegeda approved these changes May 25, 2022

View reviewed changes

airbyte-12459: Merged master.

1ee0746

auto-bump connector version

2ea3c94

octavia-squidington-iii temporarily deployed to more-secrets May 25, 2022 16:13 Inactive

alexandertsukanov merged commit 5371f87 into master May 25, 2022

alexandertsukanov deleted the otsukanov/12459_Fix_DATs_blind_spot_connection_check branch May 25, 2022 16:14

octavia-squidington-iii mentioned this pull request May 25, 2022

Bump Airbyte version from 0.39.1-alpha to 0.39.2-alpha #13188

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🐛 Destination Redshift: Fix DATs blind spot - connection check #13090

🐛 Destination Redshift: Fix DATs blind spot - connection check #13090

alexandertsukanov commented May 23, 2022

alexandertsukanov commented May 23, 2022 •

edited by github-actions bot

Loading

edgao left a comment

alexandertsukanov commented May 24, 2022

alexandertsukanov commented May 25, 2022 •

edited by github-actions bot

Loading

alexandertsukanov commented May 25, 2022 •

edited by github-actions bot

Loading

alexandertsukanov commented May 25, 2022 •

edited by github-actions bot

Loading

edgao commented May 25, 2022

alexandertsukanov commented May 26, 2022

edgao commented May 27, 2022

alexandertsukanov commented May 27, 2022 •

edited

Loading

alexandertsukanov commented May 27, 2022

edgao commented May 27, 2022

🐛 Destination Redshift: Fix DATs blind spot - connection check #13090

🐛 Destination Redshift: Fix DATs blind spot - connection check #13090

Conversation

alexandertsukanov commented May 23, 2022

What

How

Recommended reading order

Pre-merge Checklist

Community member or Airbyter

Airbyter

Community member or Airbyter

Airbyter

Tests

alexandertsukanov commented May 23, 2022 • edited by github-actions bot Loading

edgao left a comment

Choose a reason for hiding this comment

alexandertsukanov commented May 24, 2022

alexandertsukanov commented May 25, 2022 • edited by github-actions bot Loading

alexandertsukanov commented May 25, 2022 • edited by github-actions bot Loading

alexandertsukanov commented May 25, 2022 • edited by github-actions bot Loading

edgao commented May 25, 2022

alexandertsukanov commented May 26, 2022

edgao commented May 27, 2022

alexandertsukanov commented May 27, 2022 • edited Loading

alexandertsukanov commented May 27, 2022

edgao commented May 27, 2022

alexandertsukanov commented May 23, 2022 •

edited by github-actions bot

Loading

alexandertsukanov commented May 25, 2022 •

edited by github-actions bot

Loading

alexandertsukanov commented May 25, 2022 •

edited by github-actions bot

Loading

alexandertsukanov commented May 25, 2022 •

edited by github-actions bot

Loading

alexandertsukanov commented May 27, 2022 •

edited

Loading