🎉 GCS destination: use serialized buffer; compress csv & jsonl #11686

tuliren · 2022-04-04T09:15:55Z

What

Cleaned up S3 destination config.
- Add a builder.
- Removed all redundant constructors.
Refactored GCS destination config.
- The GCS config now extends from an S3 config for convenience.
Migrated the GCS destination to use the new serialized buffer strategy to be more scalable.
This PR resolves Apply buffering changes to GCS Destination #11425.
This PR relates to Apply buffering changes to BigQuery Destination when using staging #11203.

Config class

Each destination config class includes a BlobStorageCredentialConfig<ConfigType>
- For S3, it is S3CredentialConfig, whose ConfigType is S3CredentialType.
- For GCS, it is GcsCredentialConfig, whose ConfigType is GcsCredentialType.
S3 has two credential config implementations
- S3AccessKeyCredentialConfig
- S3InstanceProfileCredentialConfig
GCS has one credential config implementation
- GcsHmacKeyCredentialConfig
The GCS credential config can be converted to an S3 credential config.
- This is because currently we always use the S3 client to communicate with GCS.
- This may change in the future.

🚨 User Impact 🚨

The main improvement from this PR is that the GCS destination can handle a large number of streams now. Previously it will throw out-of-memory error whenever there are too many streams (e.g. 50).
🚨 The impact for the users is that CSV and JSONL formats will be automatically compressed with GZIP. For example, if an output file was previously abc.csv, it will become abc.csv.gz after this PR.

Pre-merge Checklist

Expand the relevant checklist and delete the others.

Updating a connector

Community member or Airbyter

Grant edit access to maintainers (instructions)
Secrets in the connector's spec are annotated with airbyte_secret
Unit & integration tests added and passing. Community members, please provide proof of success locally e.g: screenshot or copy-paste unit, integration, and acceptance test output. To run acceptance tests for a Python connector, follow instructions in the README. For java connectors run ./gradlew :airbyte-integrations:connectors:<name>:integrationTest.
Code reviews completed
Documentation updated
- Connector's README.md
- Connector's bootstrap.md. See description and examples
- Changelog updated in docs/integrations/<source or destination>/<name>.md including changelog. See changelog example
PR name follows PR naming conventions

Airbyter

If this is a community PR, the Airbyte engineer reviewing this PR is responsible for the below items.

Create a non-forked branch based on this PR and test the below items on it
Build is successful
If new credentials are required for use in CI, add them to GSM. Instructions.
/test connector=connectors/<name> command is passing
New Connector version released on Dockerhub by running the /publish command described here
After the new connector version is published, connector version bumped in the seed directory as described here
Seed specs have been re-generated by building the platform and committing the changes to the seed spec files, as described here

tuliren · 2022-04-04T10:13:29Z

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2089174172
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2089174172
No Python unittests run

tuliren · 2022-04-04T10:13:33Z

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2089174775
❌ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2089174775
🐛

tuliren · 2022-04-04T10:55:03Z

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2089373624
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2089373624
No Python unittests run

tuliren · 2022-04-04T10:55:10Z

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2089373977
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2089373977
No Python unittests run

tuliren · 2022-04-04T10:57:23Z

@ChristopheDuong, this is the prerequisite for BigQuery changes. It updates many of the GCS classes to be compatible with S3 to directly reuse the S3 code. It also completes the migration for the GCS destination.

...rs/destination-gcs/src/main/java/io/airbyte/integrations/destination/gcs/GcsDestination.java

tuliren · 2022-04-04T18:58:58Z

/test connector=connectors/destination-s3

🕑 connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2091922283
✅ connectors/destination-s3 https://github.com/airbytehq/airbyte/actions/runs/2091922283
No Python unittests run

tuliren · 2022-04-04T18:59:04Z

/test connector=connectors/destination-gcs

🕑 connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2091922489
✅ connectors/destination-gcs https://github.com/airbytehq/airbyte/actions/runs/2091922489
No Python unittests run

tuliren · 2022-04-04T19:55:41Z

Will publish the connector in a follow up PR.

wallies · 2022-04-14T06:47:24Z

Who decided this was a good idea for an ETL tool to output to gz. How are tools using this data afterwards meant to process it

The impact for the users is that CSV and JSONL formats will be automatically compressed with GZIP. For example, if an output file was previously abc.csv, it will become abc.csv.gz after this PR.

tuliren · 2022-04-14T07:10:16Z

@wallies, thank you for raising this question and creating this issue. We made the decision to compress CSV and JSONL formats based on the common use case of these blob storages. People usually use S3 and GCS just to archive their data, and the compression can reduce the storage cost.

I admit that this is not friendly for other use cases. I have created an issue here. Should have a new version that provides an option to not compress these formats by the end of this week or early next week.

wallies · 2022-04-14T07:31:35Z

@wallies, thank you for raising this question and creating this issue. We made the decision to compress CSV and JSONL formats based on the common use case of these blob storages. People usually use S3 and GCS just to archive their data, and the compression can reduce the storage cost.

I admit that this is not friendly for other use cases. I have created an issue here. Should have a new version that provides an option to not compress these formats by the end of this week or early next week.

Much appreciated @tuliren. I also raised this #11872 as we were seeing no file extension at all on any new files

tuliren · 2022-04-22T22:42:28Z

@wallies, the new version with an option to not compress CSV and JSONL files has been published. Please give the new version a try. Thanks~~

tuliren added 6 commits April 3, 2022 18:13

Update gcs destination to s3 consumer factory

abd603c

Fix typo

c04a489

Update gcs config

29c0b42

Remove gcs s3 helper

b8a19d2

Rename enum

c088d22

Clean up s3 config constructors

2ed586b

github-actions bot added the area/connectors Connector related issues label Apr 4, 2022

tuliren added 3 commits April 4, 2022 02:28

Move path format to s3 config

c25484c

Fix acceptance test

5f8376c

Fix unit test

4be508a

tuliren added 5 commits April 4, 2022 03:28

Format code

6642d69

Remove deprecated files

a46ceea

Update comment

c8a7469

Fix jsonl acceptance test

6377b13

Update serialized buffer factory

de338a4

tuliren marked this pull request as ready for review April 4, 2022 10:55

tuliren requested review from edgao and ChristopheDuong April 4, 2022 10:55

Fix redshift stream copier

5c1b38f

ChristopheDuong approved these changes Apr 4, 2022

View reviewed changes

...rs/destination-gcs/src/main/java/io/airbyte/integrations/destination/gcs/GcsDestination.java Outdated Show resolved Hide resolved

tuliren added 5 commits April 4, 2022 08:54

Merge branch 'master' into liren/gcs-migration

97e5a48

Create gcs name transformer

3f463e6

Add constants for suffix

036c1e5

Bump gcs destination version

048824d

Fix typo and unit test

5d9663c

github-actions bot added the area/documentation Improvements or additions to documentation label Apr 4, 2022

tuliren added 2 commits April 4, 2022 10:45

Add unit test for s3 dest config

e3291bb

Fix acceptance test due to bucket object prefix

f267d37

tuliren added 2 commits April 4, 2022 12:01

Update doc

89ea7b1

Mark version as unpublished

4810969

tuliren changed the title ~~🎉 GCS destination: use file serialized buffer~~ 🎉 GCS destination: use serialized buffer; compress csv & jsonl Apr 4, 2022

tuliren merged commit 4eb6d06 into master Apr 4, 2022

tuliren deleted the liren/gcs-migration branch April 4, 2022 19:55

tuliren mentioned this pull request Apr 4, 2022

Publish GCS destination 0.2.0 #11706

Merged

octavia-squidington-iii mentioned this pull request Apr 5, 2022

Bump Airbyte version from 0.35.64-alpha to 0.35.65-alpha #11722

Merged

tuliren mentioned this pull request Apr 23, 2022

🐞 Destination databricks: fix destination config #12299

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

🎉 GCS destination: use serialized buffer; compress csv & jsonl #11686

🎉 GCS destination: use serialized buffer; compress csv & jsonl #11686

tuliren commented Apr 4, 2022 •

edited

Loading

tuliren commented Apr 4, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 4, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 4, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 4, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 4, 2022

tuliren commented Apr 4, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 4, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 4, 2022

wallies commented Apr 14, 2022

tuliren commented Apr 14, 2022

wallies commented Apr 14, 2022

tuliren commented Apr 22, 2022

🎉 GCS destination: use serialized buffer; compress csv & jsonl #11686

🎉 GCS destination: use serialized buffer; compress csv & jsonl #11686

Conversation

tuliren commented Apr 4, 2022 • edited Loading

What

Config class

Recommended reading order

🚨 User Impact 🚨

Pre-merge Checklist

Community member or Airbyter

Airbyter

tuliren commented Apr 4, 2022 • edited by github-actions bot Loading

tuliren commented Apr 4, 2022 • edited by github-actions bot Loading

tuliren commented Apr 4, 2022 • edited by github-actions bot Loading

tuliren commented Apr 4, 2022 • edited by github-actions bot Loading

tuliren commented Apr 4, 2022

tuliren commented Apr 4, 2022 • edited by github-actions bot Loading

tuliren commented Apr 4, 2022 • edited by github-actions bot Loading

tuliren commented Apr 4, 2022

wallies commented Apr 14, 2022

tuliren commented Apr 14, 2022

wallies commented Apr 14, 2022

tuliren commented Apr 22, 2022

tuliren commented Apr 4, 2022 •

edited

Loading

tuliren commented Apr 4, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 4, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 4, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 4, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 4, 2022 •

edited by github-actions bot

Loading

tuliren commented Apr 4, 2022 •

edited by github-actions bot

Loading