Skip to content

Commit 39bf96d

Browse files
update docs/specs
1 parent 9ae1167 commit 39bf96d

File tree

2 files changed

+9
-9
lines changed
  • airbyte-integrations/connectors/destination-s3/src/main/resources
  • docs/integrations/destinations

2 files changed

+9
-9
lines changed

airbyte-integrations/connectors/destination-s3/src/main/resources/spec.json

+1-1
Original file line numberDiff line numberDiff line change
@@ -40,7 +40,7 @@
4040
"s3_path_format": {
4141
"description": "Format string on how data will be organized inside the S3 bucket directory",
4242
"type": "string",
43-
"examples": ["${NAMESPACE}/${STREAM_NAME}/"],
43+
"examples": ["${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_${PART_ID}"],
4444
"order": 3
4545
},
4646
"s3_bucket_region": {

docs/integrations/destinations/s3.md

+8-8
Original file line numberDiff line numberDiff line change
@@ -22,28 +22,28 @@ Check out common troubleshooting issues for the S3 destination connector on our
2222
| S3 Endpoint | string | URL to S3, If using AWS S3 just leave blank. |
2323
| S3 Bucket Name | string | Name of the bucket to sync data into. |
2424
| S3 Bucket Path | string | Subdirectory under the above bucket to sync the data into. |
25-
| S3 Bucket Format | string | Additional subdirectories format under S3 Bucket Path. Default value is `${NAMESPACE}/${STREAM_NAME}/` and this can be further customized with variables such as `${YEAR}, ${MONTH}, ${DAY}, ${HOUR} etc` referring to the writing datetime. |
25+
| S3 Bucket Format | string | Additional string format under S3 Bucket Path. Default value is `${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_${PART_ID}`. |
2626
| S3 Region | string | See [here](https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/using-regions-availability-zones.html#concepts-available-regions) for all region codes. |
2727
| Access Key ID | string | AWS/Minio credential. |
2828
| Secret Access Key | string | AWS/Minio credential. |
2929
| Format | object | Format specific configuration. See the [spec](/airbyte-integrations/connectors/destination-s3/src/main/resources/spec.json) for details. |
3030

3131
⚠️ Please note that under "Full Refresh Sync" mode, data in the configured bucket and path will be wiped out before each sync. We recommend you to provision a dedicated S3 resource for this sync to prevent unexpected data deletion from misconfiguration. ⚠️
3232

33-
The full path of the output data with S3 path format `${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}` is:
33+
The full path of the output data with S3 path format `${NAMESPACE}/${STREAM_NAME}/${YEAR}_${MONTH}_${DAY}_${EPOCH}_${PART_ID}` is:
3434

3535
```text
36-
<bucket-name>/<source-namespace-if-exists>/<stream-name>/<upload-date>/<partition-uuid>.<format-extension>
36+
<bucket-name>/<source-namespace-if-exists>/<stream-name>/<upload-date>_<epoch>_<partition-id>.<format-extension>
3737
```
3838

3939
For example:
4040

4141
```text
42-
testing_bucket/data_output_path/public/users/2021_01_01/123e4567-e89b-12d3-a456-426614174000.csv.gz
43-
↑ ↑ ↑ ↑ ↑ ↑
44-
| | | | | | format extension
45-
| | | | | |
46-
| | | | | uuid
42+
testing_bucket/data_output_path/public/users/2021_01_01_1234567890_0.csv.gz
43+
↑ ↑ ↑ ↑ ↑ ↑
44+
| | | | | | | format extension
45+
| | | | | | unique incremental part id
46+
| | | | | milliseconds since epoch
4747
| | | | upload date in YYYY_MM_DD
4848
| | | stream name
4949
| | source namespace (if it exists)

0 commit comments

Comments
 (0)