Skip to content

tweak snowflake destination to hide INSERT load method #9982

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 3 commits into from
Feb 9, 2022
Merged

tweak snowflake destination to hide INSERT load method #9982

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
1268c2b
tweak snowflake to hide INSERT load method
subodh1810 Feb 1, 2022
0a275e9
Merge branch 'master' into snowflake-insert-hide
subodh1810 Feb 8, 2022
f91cbc8
address review comment
subodh1810 Feb 8, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 2 additions & 10 deletions ...ake/src/main/java/io/airbyte/integrations/destination/snowflake/SnowflakeDestination.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,6 @@ public class SnowflakeDestination extends SwitchingDestination<SnowflakeDestinat
private static final Logger LOGGER = LoggerFactory.getLogger(SnowflakeDestination.class);

enum DestinationType {
INSERT,
COPY_S3,
COPY_GCS,
INTERNAL_STAGING
Expand All @@ -28,7 +27,7 @@ public SnowflakeDestination() {
super(DestinationType.class, SnowflakeDestination::getTypeFromConfig, getTypeToDestination());
}

public static DestinationType getTypeFromConfig(final JsonNode config) {
private static DestinationType getTypeFromConfig(final JsonNode config) {
if (isS3Copy(config)) {
return DestinationType.COPY_S3;
} else if (isGcsCopy(config)) {
Expand All @@ -38,11 +37,6 @@ public static DestinationType getTypeFromConfig(final JsonNode config) {
}
}

public static boolean isInternalStaging(final JsonNode config) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this was unused right?

return config.has("loading_method") && config.get("loading_method").isObject()
&& config.get("loading_method").get("method").asText().equals("Internal Staging");
}

public static boolean isS3Copy(final JsonNode config) {
return config.has("loading_method") && config.get("loading_method").isObject() && config.get("loading_method").has("s3_bucket_name");
}
Expand All @@ -51,14 +45,12 @@ public static boolean isGcsCopy(final JsonNode config) {
return config.has("loading_method") && config.get("loading_method").isObject() && config.get("loading_method").has("project_id");
}

public static Map<DestinationType, Destination> getTypeToDestination() {
final SnowflakeInsertDestination insertDestination = new SnowflakeInsertDestination();
private static Map<DestinationType, Destination> getTypeToDestination() {
final SnowflakeCopyS3Destination copyS3Destination = new SnowflakeCopyS3Destination();
final SnowflakeCopyGcsDestination copyGcsDestination = new SnowflakeCopyGcsDestination();
final SnowflakeInternalStagingDestination internalStagingDestination = new SnowflakeInternalStagingDestination();

return ImmutableMap.of(
DestinationType.INSERT, insertDestination,
DestinationType.COPY_S3, copyS3Destination,
DestinationType.COPY_GCS, copyGcsDestination,
DestinationType.INTERNAL_STAGING, internalStagingDestination);
Expand Down
16 changes: 8 additions & 8 deletions airbyte-integrations/connectors/destination-snowflake/src/main/resources/spec.json
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -81,28 +81,28 @@
"order": 8,
"oneOf": [
{
"title": "[Recommended] Internal Staging",
"title": "Select another option",
"additionalProperties": false,
"description": "Writes large batches of records to a file, uploads the file to Snowflake, then uses <pre>COPY INTO table</pre> to upload the file. Recommended for large production workloads for better speed and scalability.",
"description": "Select another option",
"required": ["method"],
"properties": {
"method": {
"type": "string",
"enum": ["Internal Staging"],
"default": "Internal Staging"
"enum": ["Standard"],
"default": "Standard"
}
}
},
{
"title": "Standard Inserts",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

could you change the order of the options so this is the first in the list? this way it would look like this:
Screen Shot 2022-02-01 at 2 32 40 PM

screenshot obtained using instructions here

"title": "[Recommended] Internal Staging",
"additionalProperties": false,
"description": "Uses <pre>INSERT</pre> statements to send batches of records to Snowflake. Easiest (no setup) but not recommended for large production workloads due to slow speed.",
"description": "Writes large batches of records to a file, uploads the file to Snowflake, then uses <pre>COPY INTO table</pre> to upload the file. Recommended for large production workloads for better speed and scalability.",
"required": ["method"],
"properties": {
"method": {
"type": "string",
"enum": ["Standard"],
"default": "Standard"
"enum": ["Internal Staging"],
"default": "Internal Staging"
}
}
},
Expand Down