Skip to content

destination-snowflake: remove contention on state table #43440

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Conversation

stephane-airbyte
Copy link
Contributor

@stephane-airbyte stephane-airbyte commented Aug 9, 2024

tests are failing in our nightly CI builds. Looking at the logs, it looks like some tests are failing after the call to delete destination_states timeout. That's usually linked to a high contention on the impacted table (in snowflake a DELETE statement will take a full table lock).
I'm alleviating the contention by only deleting once every 100 updates.
This works because the code in JdbcDestinationHandler.getAllDestinationStates() which does a MAX (updated_at) in the client code. So having several rows for the same stream is functionally correct

Copy link

vercel bot commented Aug 9, 2024

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Comments Updated (UTC)
airbyte-docs ✅ Ready (Inspect) Visit Preview 💬 Add feedback Aug 9, 2024 7:19pm

Copy link
Contributor Author

stephane-airbyte commented Aug 9, 2024

@stephane-airbyte stephane-airbyte force-pushed the stephane/08-09-destination-snowflake_remove_contention_on_state_table branch from 22bde92 to de513c8 Compare August 9, 2024 17:51
@stephane-airbyte stephane-airbyte force-pushed the stephane/08-09-destination-snowflake_remove_contention_on_state_table branch 2 times, most recently from 0e50d50 to 50ab70a Compare August 9, 2024 18:06
@octavia-squidington-iii octavia-squidington-iii added the area/documentation Improvements or additions to documentation label Aug 9, 2024
@stephane-airbyte stephane-airbyte force-pushed the stephane/08-09-destination-snowflake_remove_contention_on_state_table branch from 50ab70a to e77120e Compare August 9, 2024 18:08
@stephane-airbyte stephane-airbyte force-pushed the stephane/destination-snowflake-bump-cdk branch from c2b1695 to 7f25aa6 Compare August 9, 2024 18:18
@stephane-airbyte stephane-airbyte force-pushed the stephane/08-09-destination-snowflake_remove_contention_on_state_table branch from e77120e to 2628a7e Compare August 9, 2024 18:19
@stephane-airbyte stephane-airbyte marked this pull request as ready for review August 9, 2024 19:05
@stephane-airbyte stephane-airbyte requested a review from a team as a code owner August 9, 2024 19:05
Copy link
Contributor Author

stephane-airbyte commented Aug 9, 2024

Merge activity

  • Aug 9, 3:05 PM EDT: @stephane-airbyte started a stack merge that includes this pull request via Graphite.
  • Aug 9, 3:08 PM EDT: Graphite rebased this pull request as part of a merge.
  • Aug 9, 3:32 PM EDT: Graphite couldn't merge this PR because it was not satisfying all requirements (Failed CI: 'Connectors CI tests', 'Connectors CI tests on destination-snowflake').
  • Aug 9, 5:50 PM EDT: @stephane-airbyte merged this pull request with Graphite.

@stephane-airbyte stephane-airbyte changed the base branch from stephane/destination-snowflake-bump-cdk to graphite-base/43440 August 9, 2024 19:06
@stephane-airbyte stephane-airbyte changed the base branch from graphite-base/43440 to master August 9, 2024 19:06
@stephane-airbyte stephane-airbyte force-pushed the stephane/08-09-destination-snowflake_remove_contention_on_state_table branch from 2628a7e to 5d6c28c Compare August 9, 2024 19:07
@stephane-airbyte stephane-airbyte force-pushed the stephane/08-09-destination-snowflake_remove_contention_on_state_table branch from 5d6c28c to 9f59129 Compare August 9, 2024 19:13
@stephane-airbyte stephane-airbyte merged commit 2a15fbd into master Aug 9, 2024
34 checks passed
@stephane-airbyte stephane-airbyte deleted the stephane/08-09-destination-snowflake_remove_contention_on_state_table branch August 9, 2024 21:50
LouisAuneau pushed a commit to LouisAuneau/airbyte that referenced this pull request Aug 13, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
area/connectors Connector related issues area/documentation Improvements or additions to documentation connectors/destination/snowflake
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants