This repository was archived by the owner on Apr 26, 2024. It is now read-only.
-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Keep track when we try and fail to process a pulled event #13589
Merged
MadLittleMods
merged 34 commits into
develop
from
madlittlemods/keep-track-when-we-tried-to-backfill-an-event
Sep 14, 2022
Merged
Changes from 17 commits
Commits
Show all changes
34 commits
Select commit
Hold shift + click to select a range
e0d7fab
Keep track when we tried to backfill an event
MadLittleMods b8d55d3
Record in some fail spots
MadLittleMods f63d823
Merge branch 'develop' into madlittlemods/keep-track-when-we-tried-to…
MadLittleMods bec26e2
Record and clear attempts
MadLittleMods fee37c3
Add changelog
MadLittleMods d1290be
Remove from when spam checker fails
MadLittleMods f9119d0
Custom upsert to increment
MadLittleMods f5c6fe7
Fix lints
MadLittleMods 16ebec0
Remove extra whitespace
MadLittleMods ce07aa1
Move to correct folder
MadLittleMods 5811ba1
Set the version back
MadLittleMods cf2b093
Fix `TypeError: not all arguments converted during string formatting`
MadLittleMods cbb4194
Add test to make sure failed backfill attempts are recorded
MadLittleMods 621c5d3
Clean up test
MadLittleMods 75c07bb
Fix comments
MadLittleMods 783cce5
Add test for clearing backfill attempts
MadLittleMods 54ef84b
Maybe a better comment
MadLittleMods e4192d7
Update table name with "failed" and include room_id in the primary key
MadLittleMods 7a44932
Rename to record_event_failed_backfill_attempt
MadLittleMods 86d98ca
Merge branch 'develop' into madlittlemods/keep-track-when-we-tried-to…
MadLittleMods 1464101
Add _unsafe_to_upsert_tables check
MadLittleMods 71c7738
Add foreign key references
MadLittleMods df8c76d
Merge branch 'develop' into madlittlemods/keep-track-when-we-tried-to…
MadLittleMods d45b078
Remove reference to event that won't be in the events table
MadLittleMods 33ad64e
Merge branch 'develop' into madlittlemods/keep-track-when-we-tried-to…
MadLittleMods 63bec99
Remove emulated upsert code (all of our dbs no support it)
MadLittleMods 31d7502
Table rename `event_failed_pull_attempts`
MadLittleMods 0b5f1db
Add `last_cause` column
MadLittleMods 4ce7709
Merge branch 'develop' into madlittlemods/keep-track-when-we-tried-to…
MadLittleMods d3a1f84
Merge branch 'develop' into madlittlemods/keep-track-when-we-tried-to…
MadLittleMods 1347686
Update schema version summary
MadLittleMods 57182dc
Remove backfilled check since we plan to go general anyway
MadLittleMods e58bc65
Merge branch 'develop' into madlittlemods/keep-track-when-we-tried-to…
MadLittleMods 70019d2
Move change to latest delta 73
MadLittleMods File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1 @@ | ||
Keep track when we attempt to backfill an event but fail so we can intelligently back-off in the future. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
27 changes: 27 additions & 0 deletions
27
synapse/storage/schema/main/delta/72/04event_backfill_access_time.sql
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,27 @@ | ||
/* Copyright 2022 The Matrix.org Foundation C.I.C | ||
* | ||
* Licensed under the Apache License, Version 2.0 (the "License"); | ||
* you may not use this file except in compliance with the License. | ||
* You may obtain a copy of the License at | ||
* | ||
* http://www.apache.org/licenses/LICENSE-2.0 | ||
* | ||
* Unless required by applicable law or agreed to in writing, software | ||
* distributed under the License is distributed on an "AS IS" BASIS, | ||
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. | ||
* See the License for the specific language governing permissions and | ||
* limitations under the License. | ||
*/ | ||
|
||
|
||
-- Add a table that keeps track of when we last tried to backfill an event. This | ||
-- allows us to be more intelligent when we decide to retry (we don't need to | ||
-- fail over and over) and we can process that event in the background so we | ||
-- don't block on it each time. | ||
CREATE TABLE IF NOT EXISTS event_backfill_attempts( | ||
MadLittleMods marked this conversation as resolved.
Show resolved
Hide resolved
|
||
event_id TEXT NOT NULL, | ||
MadLittleMods marked this conversation as resolved.
Show resolved
Hide resolved
|
||
num_attempts INT NOT NULL, | ||
last_attempt_ts BIGINT NOT NULL | ||
MadLittleMods marked this conversation as resolved.
Show resolved
Hide resolved
|
||
); | ||
MadLittleMods marked this conversation as resolved.
Show resolved
Hide resolved
|
||
|
||
CREATE UNIQUE INDEX IF NOT EXISTS event_backfill_attempts_event_id ON event_backfill_attempts(event_id); | ||
MadLittleMods marked this conversation as resolved.
Show resolved
Hide resolved
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.