Background process 'background_updates' threw an exception, after updating to version 1.81.0rc1

[strufkin]

Description

Just updated to 1.81.0rc1 and got below exceptions

Steps to reproduce

• just updated to version 1.81.0rc1
• after restart observing exceptions

Homeserver

my

Synapse Version

1.81.0rc1

Installation Method

pip (from PyPI)

Database

PostgreSQL 12, single, no portdb, not from backup

Workers

Single process

Platform

VM, python3.10, venv

Configuration

No response

Relevant log output

2023-04-04 16:09:47,102 - synapse.storage.background_updates - 431 - INFO - background_updates-0 - Starting update batch on background update 'set_device_id_for_pushers'
2023-04-04 16:09:47,117 - synapse.storage.background_updates - 302 - ERROR - background_updates-0 - Error doing update
Traceback (most recent call last):
  File "/home/<redacted>/synapseenv/env3.10/lib/python3.10/site-packages/synapse/storage/background_updates.py", line 294, in run_background_updates
    result = await self.do_next_background_update(sleep)
  File "/home/<redacted>/synapseenv/env3.10/lib/python3.10/site-packages/synapse/storage/background_updates.py", line 424, in do_next_background_update
    await self._do_background_update(desired_duration_ms)
  File "/home/<redacted>/synapseenv/env3.10/lib/python3.10/site-packages/synapse/storage/background_updates.py", line 467, in _do_background_update
    items_updated = await update_handler(progress, batch_size)
  File "/home/<redacted>/synapseenv/env3.10/lib/python3.10/site-packages/synapse/storage/databases/main/pusher.py", line 570, in _set_device_id_for_pushers
    nb_processed = await self.db_pool.runInteraction(
  File "/home/<redacted>/synapseenv/env3.10/lib/python3.10/site-packages/synapse/storage/database.py", line 902, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "/home/<redacted>/synapseenv/env3.10/lib/python3.10/site-packages/twisted/internet/defer.py", line 1656, in _inlineCallbacks
    result = current_context.run(
  File "/home/<redacted>/synapseenv/env3.10/lib/python3.10/site-packages/twisted/python/failure.py", line 489, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "/home/<redacted>/synapseenv/env3.10/lib/python3.10/site-packages/synapse/storage/database.py", line 868, in _runInteraction
    result = await self.runWithConnection(
  File "/home/<redacted>/synapseenv/env3.10/lib/python3.10/site-packages/synapse/storage/database.py", line 997, in runWithConnection
    return await make_deferred_yieldable(
  File "/home/<redacted>/synapseenv/env3.10/lib/python3.10/site-packages/twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "/home/<redacted>/synapseenv/env3.10/lib/python3.10/site-packages/twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "/home/<redacted>/synapseenv/env3.10/lib/python3.10/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "/home/<redacted>/synapseenv/env3.10/lib/python3.10/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "/home/deployer/projects/synapseenv/env3.10/lib/python3.10/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "/home/<redacted>/synapseenv/env3.10/lib/python3.10/site-packages/synapse/storage/database.py", line 990, in inner_func
    return func(db_conn, *args, **kwargs)
  File "/home/<redacted>/synapseenv/env3.10/lib/python3.10/site-packages/synapse/storage/database.py", line 730, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "/home/<redacted>/synapseenv/env3.10/lib/python3.10/site-packages/synapse/storage/databases/main/pusher.py", line 565, in set_device_id_for_pushers_txn
    txn, "set_device_id_for_pushers", {"pusher_id": rows[-1]["id"]}
KeyError: 'id'

....


KeyError: 'id'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/<redacted>/synapseenv/env3.10/lib/python3.10/site-packages/synapse/metrics/background_process_metrics.py", line 242, in run
    return await func(*args, **kwargs)
  File "/home/<redacted>/synapseenv/env3.10/lib/python3.10/site-packages/synapse/storage/background_updates.py", line 299, in run_background_updates
    raise RuntimeError(
RuntimeError: 5 back-to-back background update failures; aborting.

Anything else that would be useful to know?

No response

[erikjohnston]

I think this was introduced by: #15280, as it changed the name of the row:

5b70f24#diff-62af9acfbf12e3a6c8b8b348ba374f4ce834f7c4e3a1b3d5d6b9242b1714c702R522