Backfill should gracefully handle the case where no one has the complete state for a given event #10764
Description
This issue is spawning from #10566 (comment) and summarizes the problem/conversation there. Thanks @erikjohnston and @richvdh for the insights on this!
We initially used a fake_prev_event_id for MSC2716 to make a chain of events float outside of the normal DAG but have since opted to use empty prev_events=[]. This issue still valid for the general case where no servers have the state for a given event though.
The refactor in #10645 changed the way we handle fetching state for backfills. Before we'd just get the state at the last event, whereas now we try and get the state for each prev_event (including the fake event), which obviously fails. This means that federated homeservers no longer accept the insertion event because it can't fetch the fake_prev_event_id state (Error attempting to resolve state at missing prev_events, ERROR 403: We can't get valid state history.).
The goal of the code that's rejecting the fake event is protecting against an attack where a remote server could arbitrarily replace the servers view of the current state of the room. But the mitigation we use is a sledgehammer and we should replace it with something more intelligent that both mitigates the attack and gracefully handles missing events. (related to #8451)
Potential solutions
To be determined (TBD).
Reproduction case
- Create an event which references a made up
prev_event_id - Try to backfill it from federated homeserver
2021-09-01 20:19:21,560 - synapse.handlers.federation_event - 746 - WARNING - GET-5-$hbHXgulNKlmg9mkGQiBH5SSgq2NL4GO70sIIfHCa-YY - Error attempting to resolve state at missing prev_events
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation_event.py", line 711, in _resolve_state_at_missing_prevs
remote_state = await self._get_state_after_missing_prev_event(
File "/usr/local/lib/python3.8/site-packages/synapse/handlers/federation_event.py", line 777, in _get_state_after_missing_prev_event
) = await self.federation_client.get_room_state_ids(
File "/usr/local/lib/python3.8/site-packages/synapse/federation/federation_client.py", line 370, in get_room_state_ids
result = await self.transport_layer.get_room_state_ids(
File "/usr/local/lib/python3.8/site-packages/synapse/federation/transport/client.py", line 71, in get_room_state_ids
return await self.client.get_json(
File "/usr/local/lib/python3.8/site-packages/synapse/http/matrixfederationclient.py", line 1001, in get_json
response = await self._send_request_with_optional_trailing_slash(
File "/usr/local/lib/python3.8/site-packages/synapse/http/matrixfederationclient.py", line 385, in _send_request_with_optional_trailing_slash
response = await self._send_request(request, **send_request_args)
File "/usr/local/lib/python3.8/site-packages/synapse/http/matrixfederationclient.py", line 631, in _send_request
raise exc
synapse.api.errors.HttpResponseException: 404: Not Found