Skip to content

Commit 531ed1a

Browse files
authored
update Snowflake destination docs with more info (#10213)
* update Snowflake destination docs with more info * more edits * address review comments
1 parent c27e2a0 commit 531ed1a

File tree

1 file changed

+20
-13
lines changed

1 file changed

+20
-13
lines changed

docs/integrations/destinations/snowflake.md

+20-13
Original file line numberDiff line numberDiff line change
@@ -27,9 +27,14 @@ Note that Airbyte will create **permanent** tables. If you prefer to create tran
2727

2828
## Getting started
2929

30+
### Requirements
31+
32+
1. Active Snowflake warehouse
33+
3034
We recommend creating an Airbyte-specific warehouse, database, schema, user, and role for writing data into Snowflake so it is possible to track costs specifically related to Airbyte \(including the cost of running this warehouse\) and control permissions at a granular level. Since the Airbyte user creates, drops, and alters tables, `OWNERSHIP` permissions are required in Snowflake. If you are not following the recommended script below, please limit the `OWNERSHIP` permissions to only the necessary database and schema for the Airbyte user.
3135

3236
We provide the following script to create these resources. Before running, you must change the password to something secure. You may change the names of the other resources if you desire.
37+
Login into your Snowflake warehouse, copy and paste the following script in a new [worksheet](https://docs.snowflake.com/en/user-guide/ui-worksheet.html). Select the `All Queries` checkbox and then press the `Run` button.
3338

3439
```text
3540
-- set variables (these need to be uppercase)
@@ -106,14 +111,14 @@ commit;
106111

107112
You should now have all the requirements needed to configure Snowflake as a destination in the UI. You'll need the following information to configure the Snowflake destination:
108113

109-
* **Host**
110-
* **Role**
111-
* **Warehouse**
112-
* **Database**
113-
* **Schema**
114-
* **Username**
115-
* **Password**
116-
* **JDBC URL Params** (Optional)
114+
* **[Host](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html)** : The host domain of the snowflake instance (must include the account, region, cloud environment, and end with snowflakecomputing.com). Example - `accountname.us-east-2.aws.snowflakecomputing.com`
115+
* **[Role](https://docs.snowflake.com/en/user-guide/security-access-control-overview.html#roles)** : The role you created for Airbyte to access Snowflake. Example - `AIRBYTE_ROLE`
116+
* **[Warehouse](https://docs.snowflake.com/en/user-guide/warehouses-overview.html#overview-of-warehouses)** : The warehouse you created for Airbyte to sync data into. Example - `AIRBYTE_WAREHOUSE`
117+
* **[Database](https://docs.snowflake.com/en/sql-reference/ddl-database.html#database-schema-share-ddl)** : The database you created for Airbyte to sync data into. Example - `AIRBYTE_DATABASE`
118+
* **[Schema](https://docs.snowflake.com/en/sql-reference/ddl-database.html#database-schema-share-ddl)** : The default schema is used as the target schema for all statements issued from the connection that do not explicitly specify a schema name. Schema name would be transformed to allowed by Snowflake if it not follow [Snowflake Naming Conventions](https://docs.airbyte.io/integrations/destinations/snowflake#notes-about-snowflake-naming-conventions).
119+
* **Username** : The username you created to allow Airbyte to access the database. Example - `AIRBYTE_USER`
120+
* **Password** : The password associated with the username.
121+
* **[JDBC URL Params](https://docs.snowflake.com/en/user-guide/jdbc-parameters.html)** (Optional) : Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3).
117122

118123
## Notes about Snowflake Naming Conventions
119124

@@ -147,17 +152,17 @@ When an identifier is double-quoted, it is stored and resolved exactly as entere
147152

148153
Therefore, Airbyte Snowflake destination will create tables and schemas using the Unquoted identifiers when possible or fallback to Quoted Identifiers if the names are containing special characters.
149154

150-
## Cloud Storage Staging
155+
## Loading Method
151156

152-
By default, Airbyte uses batches of `INSERT` commands to add data to a temporary table before copying it over to the final table in Snowflake. This is too slow for larger/multi-GB replications. For those larger replications we recommend configuring using cloud storage to allow batch writes and loading.
157+
By default, Airbyte uses [INTERNAL STAGING](https://docs.airbyte.com/integrations/destinations/snowflake#internal-staging)
153158

154159
### Internal Staging
155160

156-
Internal named stages are storage location objects within a Snowflake database/schema. Because they are database objects, the same security permissions apply as with any other database objects. No need to provide additional properties for internal staging
161+
Internal named stages are storage location objects within a Snowflake database/schema. Because they are database objects, the same security permissions apply as with any other database objects. No need to provide additional properties for internal staging. This is also the recommended way of using the connector. It doesn't require any external resources and is quick to setup and use.
157162

158163
**Operating on a stage also requires the USAGE privilege on the parent database and schema.**
159164

160-
### AWS S3
165+
### AWS S3 Staging
161166

162167
For AWS S3, you will need to create a bucket and provide credentials to access the bucket. We recommend creating a bucket that is only used for Airbyte to stage data to Snowflake. Airbyte needs read/write access to interact with this bucket.
163168

@@ -180,7 +185,7 @@ Optional parameters:
180185
* Whether to delete the staging files from S3 after completing the sync. Specifically, the connector will create CSV files named `bucketPath/namespace/streamName/syncDate_epochMillis_randomUuid.csv` containing three columns (`ab_id`, `data`, `emitted_at`). Normally these files are deleted after the `COPY` command completes; if you want to keep them for other purposes, set `purge_staging_data` to `false`.
181186

182187

183-
### Google Cloud Storage \(GCS\)
188+
### Google Cloud Storage \(GCS\) Staging
184189

185190
First you will need to create a GCS bucket.
186191

@@ -215,6 +220,8 @@ The final query should show a `STORAGE_GCP_SERVICE_ACCOUNT` property with an ema
215220

216221
Finally, you need to add read/write permissions to your bucket with that email.
217222

223+
## Changelog
224+
218225
| Version | Date | Pull Request | Subject |
219226
|:--------|:-----------| :----- | :------ |
220227
| 0.4.12 | 2022-02-15 | [\#10342](https://github.com/airbytehq/airbyte/pull/10342) | Use connection pool, and fix connection leak. |

0 commit comments

Comments
 (0)