Skip to content

Commit 4a49615

Browse files
authored
Update destinations/gcs.md (#37757)
1 parent deabdbb commit 4a49615

File tree

1 file changed

+31
-28
lines changed
  • docs/integrations/destinations

1 file changed

+31
-28
lines changed

docs/integrations/destinations/gcs.md

+31-28
Original file line numberDiff line numberDiff line change
@@ -17,6 +17,37 @@ The Airbyte GCS destination allows you to sync data to cloud storage buckets. Ea
1717
| Incremental - Append + Deduped || |
1818
| Namespaces || Setting a specific bucket path is equivalent to having separate namespaces. |
1919

20+
21+
22+
## Getting started
23+
24+
### Requirements
25+
26+
1. Allow connections from Airbyte server to your GCS cluster \(if they exist in separate VPCs\).
27+
2. An GCP bucket with credentials \(for the COPY strategy\).
28+
29+
### Setup guide
30+
31+
- Fill up GCS info
32+
- **GCS Bucket Name**
33+
- See [this](https://cloud.google.com/storage/docs/creating-buckets) for instructions on how to create a GCS bucket. The bucket cannot have a retention policy. Set Protection Tools to none or Object versioning.
34+
- **GCS Bucket Region**
35+
- **HMAC Key Access ID**
36+
- See [this](https://cloud.google.com/storage/docs/authentication/managing-hmackeys) on how to generate an access key. For more information on hmac keys please reference the [GCP docs](https://cloud.google.com/storage/docs/authentication/hmackeys)
37+
- We recommend creating an Airbyte-specific user or service account. This user or account will require the following permissions for the bucket:
38+
```
39+
storage.multipartUploads.abort
40+
storage.multipartUploads.create
41+
storage.objects.create
42+
storage.objects.delete
43+
storage.objects.get
44+
storage.objects.list
45+
```
46+
You can set those by going to the permissions tab in the GCS bucket and adding the appropriate the email address of the service account or user and adding the aforementioned permissions.
47+
- **Secret Access Key**
48+
- Corresponding key to the above access ID.
49+
- Make sure your GCS bucket is accessible from the machine running Airbyte. This depends on your networking setup. The easiest way to verify if Airbyte is able to connect to your GCS bucket is via the check connection tool in the UI.
50+
2051
## Configuration
2152
2253
| Parameter | Type | Notes |
@@ -204,34 +235,6 @@ These parameters are related to the `ParquetOutputFormat`. See the [Java doc](ht
204235

205236
Under the hood, an Airbyte data stream in Json schema is first converted to an Avro schema, then the Json object is converted to an Avro record, and finally the Avro record is outputted to the Parquet format. Because the data stream can come from any data source, the Json to Avro conversion process has arbitrary rules and limitations. Learn more about how source data is converted to Avro and the current limitations [here](https://docs.airbyte.com/understanding-airbyte/json-avro-conversion).
206237

207-
## Getting started
208-
209-
### Requirements
210-
211-
1. Allow connections from Airbyte server to your GCS cluster \(if they exist in separate VPCs\).
212-
2. An GCP bucket with credentials \(for the COPY strategy\).
213-
214-
### Setup guide
215-
216-
- Fill up GCS info
217-
- **GCS Bucket Name**
218-
- See [this](https://cloud.google.com/storage/docs/creating-buckets) for instructions on how to create a GCS bucket. The bucket cannot have a retention policy. Set Protection Tools to none or Object versioning.
219-
- **GCS Bucket Region**
220-
- **HMAC Key Access ID**
221-
- See [this](https://cloud.google.com/storage/docs/authentication/managing-hmackeys) on how to generate an access key. For more information on hmac keys please reference the [GCP docs](https://cloud.google.com/storage/docs/authentication/hmackeys)
222-
- We recommend creating an Airbyte-specific user or service account. This user or account will require the following permissions for the bucket:
223-
```
224-
storage.multipartUploads.abort
225-
storage.multipartUploads.create
226-
storage.objects.create
227-
storage.objects.delete
228-
storage.objects.get
229-
storage.objects.list
230-
```
231-
You can set those by going to the permissions tab in the GCS bucket and adding the appropriate the email address of the service account or user and adding the aforementioned permissions.
232-
- **Secret Access Key**
233-
- Corresponding key to the above access ID.
234-
- Make sure your GCS bucket is accessible from the machine running Airbyte. This depends on your networking setup. The easiest way to verify if Airbyte is able to connect to your GCS bucket is via the check connection tool in the UI.
235238

236239
## CHANGELOG
237240

0 commit comments

Comments
 (0)