-
Notifications
You must be signed in to change notification settings - Fork 4.5k
destination-snowflake: remove contention on state table #43440
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
stephane-airbyte
merged 1 commit into
master
from
stephane/08-09-destination-snowflake_remove_contention_on_state_table
Aug 9, 2024
Merged
destination-snowflake: remove contention on state table #43440
stephane-airbyte
merged 1 commit into
master
from
stephane/08-09-destination-snowflake_remove_contention_on_state_table
Aug 9, 2024
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
This stack of pull requests is managed by Graphite. Learn more about stacking. Join @stephane-airbyte and the rest of your teammates on |
22bde92
to
de513c8
Compare
0e50d50
to
50ab70a
Compare
50ab70a
to
e77120e
Compare
c2b1695
to
7f25aa6
Compare
e77120e
to
2628a7e
Compare
gisripa
approved these changes
Aug 9, 2024
Merge activity
|
2628a7e
to
5d6c28c
Compare
5d6c28c
to
9f59129
Compare
2 tasks
LouisAuneau
pushed a commit
to LouisAuneau/airbyte
that referenced
this pull request
Aug 13, 2024
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Labels
area/connectors
Connector related issues
area/documentation
Improvements or additions to documentation
connectors/destination/snowflake
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
tests are failing in our nightly CI builds. Looking at the logs, it looks like some tests are failing after the call to delete destination_states timeout. That's usually linked to a high contention on the impacted table (in snowflake a DELETE statement will take a full table lock).
I'm alleviating the contention by only deleting once every 100 updates.
This works because the code in JdbcDestinationHandler.getAllDestinationStates() which does a MAX (updated_at) in the client code. So having several rows for the same stream is functionally correct