Skip to content

Enable staging GCS loads on the BigQuery denormalized destination #8406

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
sherifnada opened this issue Dec 1, 2021 · 2 comments · Fixed by #8788
Closed

Enable staging GCS loads on the BigQuery denormalized destination #8406

sherifnada opened this issue Dec 1, 2021 · 2 comments · Fixed by #8788
Assignees
@DoNotPanicUA
Labels
area/connectors Connector related issues connectors/destination/bigquery-denormalized connectors/destinations-warehouse type/enhancement New feature or request

Comments

@sherifnada
Copy link
Contributor

sherifnada commented Dec 1, 2021 •
edited
Loading

Tell us about the problem you're trying to solve

It's well known that GCS staging loads to bigquery are much more performant (10x or more at times) than INSERT statements. While our Bigquery destination supports this, our Denormalized BigQuery destination does not. This ticket is concerned with supporting GCS staging uploads for the BigQuery Denormalized destination.

Acceptance Criteria

The bigquery denormalized destination should have the option of using GCS staging as the ingestion mechanism, similar to what the standard BQ destination currently offers:
Screen Shot 2021-12-01 at 1 37 30 PM
@sherifnada sherifnada added type/enhancement New feature or request needs-triage area/connectors Connector related issues and removed needs-triage labels Dec 1, 2021
@DoNotPanicUA DoNotPanicUA self-assigned this Dec 9, 2021
@DoNotPanicUA
Copy link
Contributor

Implementing this as part of https://github.com/airbytehq/oncall/issues/27

@ChristopheDuong
Copy link
Contributor

Could you link the PR to the issue when it is up please? That would help on tracking progress too, thanks

@DoNotPanicUA DoNotPanicUA linked a pull request Dec 14, 2021 that will close this issue
BigQuery/BiqQuery denorm Destinations : Add possibility to use different types of GCS files #8788
Merged
@oustynova oustynova moved this to Airbyte Review in GL Roadmap Dec 17, 2021
@alexandr-shegeda alexandr-shegeda moved this from Airbyte Review to Done in GL Roadmap Dec 20, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@DoNotPanicUA DoNotPanicUA

Labels
area/connectors Connector related issues connectors/destination/bigquery-denormalized connectors/destinations-warehouse type/enhancement New feature or request
Projects
No open projects
GL Roadmap
Archived in project
Milestone
No milestone
4 participants
@sherifnada @ChristopheDuong @karinakuz @DoNotPanicUA