MSC2716 leftover todo tasks

[MadLittleMods]

Just trying to keep track of the last remaining little tasks to get MSC2716 good to go.

Todo

• Verify chunk_id_from_query corresponds to an actual insertion event, synapse/rest/client/room_batch.py#L302)
  • -> Verify ?chunk_id actually corresponds to an insertion event that exists (MSC2716) #10776
• Ensure insertion events have unique next_chunk_id before persisting. Ensure (room_id, next_chunk_id) is unique. Avoid cross-talk/conflicts between chunks
  • -> Ensure (room_id, next_batch_id) is unique to avoid cross-talk/conflicts between batches (MSC2716) #10877
  • Reject/soft-fail event over federation
  • Disallow persisting on local server
  • Discussed at Make historical events discoverable from backfill for servers without any scrollback history (MSC2716) (federation) #10245 (comment)
• Split meta events in /batch_send response into their own fields, MSC2716: Incrementally importing history into existing rooms matrix-spec-proposals#2716 (comment)
  • -> Split out /batch_send meta events to their own fields (MSC2716) #10777
• Rename chunk to batch to match /batch_send endpoint, MSC2716: Incrementally importing history into existing rooms matrix-spec-proposals#2716 (comment)
  • -> Rename MSC2716 things from chunk to batch to match /batch_send endpoint #10838
• Rename ?prev_event to ?prev_event_id, MSC2716: Incrementally importing history into existing rooms matrix-spec-proposals#2716 (comment)
  • -> Rename /batch_send query parameter from ?prev_event to more obvious usage with ?prev_event_id (MSC2716) #10839
• Backfill should gracefully handle the case where no one has the complete state for a given event #10764
  • No longer needed because we now use an empty set of prev_events instead of fake prev_events
• Backfill gets clogged up with fake prev_events when using the MSC2716 /batch_send endpoint #11091
• Historical state events from state_events_at_start in the MSC2716 /batch_send endpoint get pushed to an application service /transactions #11241
• Make "marker" events state events, MSC2716: Incrementally importing history into existing rooms matrix-spec-proposals#2716 (comment). When the homeserver sees the marker state change or joins the room again, it can go up the marker event unsigned.replaces_state chain of the to collect all of the m.room.marker events along the way and set as backward extremities on the homeserver.
  • Marker events as state - MSC2716 #12718
  • Marker events as state - MSC2716 complement#371
• Make Element throw away the timeline and re-fetch when it sees the "marker" state change
  • Ask to refresh timeline when historical messages are imported (MSC2716) matrix-react-sdk#8354
  • Timeline needs to refresh when we see a MSC2716 marker event  matrix-js-sdk#2299
• Make Synapse process batches faster
  • -> Draft: Optimize MSC2716 /batch_send endpoint v1 #11778
• Disallow /batch_send usage in a room where the room version doesn't support it or is not the room creator
  • Currently, we just don't persist the insertion, batch, and marker events but we shouldn't allow the /batch_send usage in the first place to make it more obvious
  • -> Disallow /batch_send usage in a room where the room version doesn't support it or is not the room creator #13216
• Add some new background updates to fill in the insertion_events, insertion_event_edges, batch_events, and insertion_event_extremities tables when hs.config.experimental.msc2716_enabled is enabled so homeservers which go from having the config disabled to enabled(or unsupported to supported) can fill in the necessary info to have historical events appear.
  • As discussed at Allow room creator to send MSC2716 related events in existing room versions #10566 (comment)
  • Other PR where I added a background update, Populate rooms.creator field for easy lookup #10697
• Optimize synapse/handlers/federation_event.py#L545(backfill receiving) to be more efficient. Calculate the stream_ordering from newest -> oldest (in the correct order) and persist in the oldest -> newest to get the least missing prev_event fetch thrashing
  • Spawned from Fix historical messages backfilling in random order on remote homeservers (MSC2716) #11114 (comment)
  • Why aren't we sorting topologically when receiving backfill events? See Fix historical messages backfilling in random order on remote homeservers (MSC2716) #11114 (comment)
  • Blocked on a race condition not being able to run the Complement test for this, Resync state after partial-state join #12394 (comment)
  • -> Optimize backfill receiving to have less missing prev_event thrashing (scratch) #13864 and Optimize backfill receiving to have less missing prev_event thrashing (v2) #13970 and No more floating MSC2716 historical batches #13971
• Figure out how we want to elegantly support relations in the MSC2716 API's. How do I reference a message in the same batch which hasn't been persisted yet.
  • MSC2716: Incrementally importing history into existing rooms matrix-spec-proposals#2716 (comment)
• (Element problem) When visiting a link to a specific message (permalink), fix messages not loading before it in the scrollback
• Write MSC2716 test to see if the bridge can join the room later, then import messages before the room was created
• Enable MSC2716 Complement tests for workers

Outstanding PRs that would still be good to merge (even the closed ones):

• Draft: Test that /messages works on remote homeserver and can be backfilled properly after many batches (MSC2716) complement#214
• Properly verify and error on power-level mismatch in /batch_send endpoint #13229
• Draft: Make sure historical state events don't come down /transactions for application services (MSC2716) complement#221

Maybe:

• Add (room_id, next_batch_id) unique constraint to DB, Ensure (room_id, next_batch_id) is unique to avoid cross-talk/conflicts between batches (MSC2716) #10877 (comment)
• Make /messages and /context include the state from any historical messages returned. Currently, if the historical message isn't the first or last (depending on which endpoint), the historical state won't be available for the historical messages in the timeline and the avatar/displayname won't show for them.
  • This is a problem for people who individually import historical messages.
  • And is also a problem for when you import a bunch of history to an existing room at the start. The transition between real messages and historical won't have avatar/displaynames shown. Once you reach the next /messages back, they will show.
  • (also discussed in the #10975 PR description)
  • According to the spec, /messages returns "A list of state events relevant to showing the chunk." so we should be able to make this work and also include any historical state associated with some historical messages in the chunk.
  • According to the spec, /context only returns "The state of the room at the last event returned." which doesn't precisely allow wiggle room for adding some random historical state. But we technically could since it's just a list of state events returned.
  • I don't think we have to worry about /sync since historical messages will supposedly be way off in the distance and really only fetched by paginating /messages. Just to note that /sync returns messages for the first message in the chunk, https://github.com/matrix-org/matrix-doc/issues/687
• Fix error when using invite_room_state on the /batch_send state events, as encountered Fix using MSC2716 batch sending with event persistence workers #11220. This isn't any useful functionality gained but would be nice not to throw a 500 here.
• Allow state events to be sent in the events list
  • The current problem is that the events are persisted in reverse-chronological order so they have the correct stream_ordering while being backfilled which decrements.
  • As an example of why this fails, if we /batch_send -> events: [messageFromA, memberEventB, messageFromB], it will first try to persist messageFromB which will fail because memberEventB hasn't been processed yet.
  • "My hunch is that we want to try and persist the entire batch of events all in one go, rather than doing them one-by-one in reverse order."
  • See Refactor create_new_client_event to use a new parameter, state_event_ids, which accurately describes the usage with MSC2716 instead of abusing auth_event_ids #12083 (comment)
  • Related Idea: Sending multiple events to a single room in a linked batch #12010