update Snowflake destination docs with more info #10213
Conversation
github-actions
bot
added
the
area/documentation
| ### Requirements | ||
|
|
||
| 1. Active Snowflake warehouse | ||
| 2. A staging S3 or GCS bucket with credentials \(for the Cloud Storage Staging strategy\). |
There was a problem hiding this comment.
I think we should remove this since everyone should use internal staging
There was a problem hiding this comment.
To clarify, if a user doesn't have an S3 or GCS staging set up, they should be able to move forward simply with an active Snowflake warehouse and not changing the loading method. Is that correct? If so so, agree with Sherif's comment here.
There was a problem hiding this comment.
Yes thats correct @misteryeo
|
|
||
| By default, Airbyte uses batches of `INSERT` commands to add data to a temporary table before copying it over to the final table in Snowflake. This is too slow for larger/multi-GB replications. For those larger replications we recommend configuring using cloud storage to allow batch writes and loading. | ||
| By default, Airbyte uses `INTERNAL STAGING` |
There was a problem hiding this comment.
Can we hyperlink this to the correct section in our docs in case folks want to understand more about this?
| * **Username** | ||
| * **Password** | ||
| * **JDBC URL Params** (Optional) | ||
| * **Host** : The host domain of the snowflake instance (must include the account, region, cloud environment, and end with snowflakecomputing.com). Example - `accountname.us-east-2.aws.snowflakecomputing.com` |
There was a problem hiding this comment.
For this section of descriptions, are external docs from Snowflake that can/should be referenced?
For example, with "Role", this seems like it's appropriate: https://docs.snowflake.com/en/user-guide/security-access-control-overview.html#roles.
|
|
||
| ### Internal Staging | ||
|
|
||
| Internal named stages are storage location objects within a Snowflake database/schema. Because they are database objects, the same security permissions apply as with any other database objects. No need to provide additional properties for internal staging | ||
| Internal named stages are storage location objects within a Snowflake database/schema. Because they are database objects, the same security permissions apply as with any other database objects. No need to provide additional properties for internal staging. This is also the recommended way of using the connector. |
There was a problem hiding this comment.
Is there a reason why we recommend this and why this is default?
| ### Requirements | ||
|
|
||
| 1. Active Snowflake warehouse | ||
| 2. A staging S3 or GCS bucket with credentials \(for the Cloud Storage Staging strategy\). |
There was a problem hiding this comment.
To clarify, if a user doesn't have an S3 or GCS staging set up, they should be able to move forward simply with an active Snowflake warehouse and not changing the loading method. Is that correct? If so so, agree with Sherif's comment here.
| We recommend creating an Airbyte-specific warehouse, database, schema, user, and role for writing data into Snowflake so it is possible to track costs specifically related to Airbyte \(including the cost of running this warehouse\) and control permissions at a granular level. Since the Airbyte user creates, drops, and alters tables, `OWNERSHIP` permissions are required in Snowflake. If you are not following the recommended script below, please limit the `OWNERSHIP` permissions to only the necessary database and schema for the Airbyte user. | ||
|
|
||
| We provide the following script to create these resources. Before running, you must change the password to something secure. You may change the names of the other resources if you desire. | ||
| Login into your Snowflake warehouse, copy and paste the following script in a new [worksheet](https://docs.snowflake.com/en/user-guide/ui-worksheet.html). Select the `All Queries` checkbox and then press the `Run` button. |
There was a problem hiding this comment.
Yes! Love that we're hyperlinking out to Snowflake docs here!
Andy suggested these improvements to our Snowflake documentation