You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/integrations/destinations/snowflake.md
+20-13
Original file line number
Diff line number
Diff line change
@@ -27,9 +27,14 @@ Note that Airbyte will create **permanent** tables. If you prefer to create tran
27
27
28
28
## Getting started
29
29
30
+
### Requirements
31
+
32
+
1. Active Snowflake warehouse
33
+
30
34
We recommend creating an Airbyte-specific warehouse, database, schema, user, and role for writing data into Snowflake so it is possible to track costs specifically related to Airbyte \(including the cost of running this warehouse\) and control permissions at a granular level. Since the Airbyte user creates, drops, and alters tables, `OWNERSHIP` permissions are required in Snowflake. If you are not following the recommended script below, please limit the `OWNERSHIP` permissions to only the necessary database and schema for the Airbyte user.
31
35
32
36
We provide the following script to create these resources. Before running, you must change the password to something secure. You may change the names of the other resources if you desire.
37
+
Login into your Snowflake warehouse, copy and paste the following script in a new [worksheet](https://docs.snowflake.com/en/user-guide/ui-worksheet.html). Select the `All Queries` checkbox and then press the `Run` button.
33
38
34
39
```text
35
40
-- set variables (these need to be uppercase)
@@ -106,14 +111,14 @@ commit;
106
111
107
112
You should now have all the requirements needed to configure Snowflake as a destination in the UI. You'll need the following information to configure the Snowflake destination:
108
113
109
-
***Host**
110
-
***Role**
111
-
***Warehouse**
112
-
***Database**
113
-
***Schema**
114
-
***Username**
115
-
***Password**
116
-
***JDBC URL Params** (Optional)
114
+
***[Host](https://docs.snowflake.com/en/user-guide/admin-account-identifier.html)** : The host domain of the snowflake instance (must include the account, region, cloud environment, and end with snowflakecomputing.com). Example - `accountname.us-east-2.aws.snowflakecomputing.com`
115
+
***[Role](https://docs.snowflake.com/en/user-guide/security-access-control-overview.html#roles)** : The role you created for Airbyte to access Snowflake. Example - `AIRBYTE_ROLE`
116
+
***[Warehouse](https://docs.snowflake.com/en/user-guide/warehouses-overview.html#overview-of-warehouses)** : The warehouse you created for Airbyte to sync data into. Example - `AIRBYTE_WAREHOUSE`
117
+
***[Database](https://docs.snowflake.com/en/sql-reference/ddl-database.html#database-schema-share-ddl)** : The database you created for Airbyte to sync data into. Example - `AIRBYTE_DATABASE`
118
+
***[Schema](https://docs.snowflake.com/en/sql-reference/ddl-database.html#database-schema-share-ddl)** : The default schema is used as the target schema for all statements issued from the connection that do not explicitly specify a schema name. Schema name would be transformed to allowed by Snowflake if it not follow [Snowflake Naming Conventions](https://docs.airbyte.io/integrations/destinations/snowflake#notes-about-snowflake-naming-conventions).
119
+
***Username** : The username you created to allow Airbyte to access the database. Example - `AIRBYTE_USER`
120
+
***Password** : The password associated with the username.
121
+
***[JDBC URL Params](https://docs.snowflake.com/en/user-guide/jdbc-parameters.html)** (Optional) : Additional properties to pass to the JDBC URL string when connecting to the database formatted as 'key=value' pairs separated by the symbol '&'. (example: key1=value1&key2=value2&key3=value3).
117
122
118
123
## Notes about Snowflake Naming Conventions
119
124
@@ -147,17 +152,17 @@ When an identifier is double-quoted, it is stored and resolved exactly as entere
147
152
148
153
Therefore, Airbyte Snowflake destination will create tables and schemas using the Unquoted identifiers when possible or fallback to Quoted Identifiers if the names are containing special characters.
149
154
150
-
## Cloud Storage Staging
155
+
## Loading Method
151
156
152
-
By default, Airbyte uses batches of `INSERT` commands to add data to a temporary table before copying it over to the final table in Snowflake. This is too slow for larger/multi-GB replications. For those larger replications we recommend configuring using cloud storage to allow batch writes and loading.
157
+
By default, Airbyte uses [INTERNAL STAGING](https://docs.airbyte.com/integrations/destinations/snowflake#internal-staging)
153
158
154
159
### Internal Staging
155
160
156
-
Internal named stages are storage location objects within a Snowflake database/schema. Because they are database objects, the same security permissions apply as with any other database objects. No need to provide additional properties for internal staging
161
+
Internal named stages are storage location objects within a Snowflake database/schema. Because they are database objects, the same security permissions apply as with any other database objects. No need to provide additional properties for internal staging. This is also the recommended way of using the connector. It doesn't require any external resources and is quick to setup and use.
157
162
158
163
**Operating on a stage also requires the USAGE privilege on the parent database and schema.**
159
164
160
-
### AWS S3
165
+
### AWS S3 Staging
161
166
162
167
For AWS S3, you will need to create a bucket and provide credentials to access the bucket. We recommend creating a bucket that is only used for Airbyte to stage data to Snowflake. Airbyte needs read/write access to interact with this bucket.
163
168
@@ -180,7 +185,7 @@ Optional parameters:
180
185
* Whether to delete the staging files from S3 after completing the sync. Specifically, the connector will create CSV files named `bucketPath/namespace/streamName/syncDate_epochMillis_randomUuid.csv` containing three columns (`ab_id`, `data`, `emitted_at`). Normally these files are deleted after the `COPY` command completes; if you want to keep them for other purposes, set `purge_staging_data` to `false`.
181
186
182
187
183
-
### Google Cloud Storage \(GCS\)
188
+
### Google Cloud Storage \(GCS\) Staging
184
189
185
190
First you will need to create a GCS bucket.
186
191
@@ -215,6 +220,8 @@ The final query should show a `STORAGE_GCP_SERVICE_ACCOUNT` property with an ema
215
220
216
221
Finally, you need to add read/write permissions to your bucket with that email.
217
222
223
+
## Changelog
224
+
218
225
| Version | Date | Pull Request | Subject |
219
226
|:--------|:-----------| :----- | :------ |
220
227
| 0.4.12 | 2022-02-15 |[\#10342](https://github.com/airbytehq/airbyte/pull/10342)| Use connection pool, and fix connection leak. |
0 commit comments