Messages not delivered

[head47]

Description

Version: synapse Docker image, latest (v1.122.0)

None of my messages are delivered since the latest update.

Steps to reproduce

• attempt to send any message
• see the client report it as not sent
• check with the recipient that they haven't received the message

Homeserver

my personal homeserver

Synapse Version

1.122.0

Installation Method

Docker (matrixdotorg/synapse)

Database

PostgreSQL 17.2, single server, no portdb, never restored a backup

Workers

Single process

Platform

Arch Linux

Configuration

Presence is on

Relevant log output

2025-01-18 11:22:53,701 - synapse.http.server - 146 - ERROR - PUT-9046 - Failed handle request via 'RoomSendEventRestServlet': <XForwardedForRequest at 0x726483bc63c0 method='PUT' uri='/_matrix/client/v3/rooms/!REDACTED/send/m.room.encrypted/REDACTED' clientproto='HTTP/1.1' site='8008'>
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/twisted/internet/defer.py", line 2010, in _inlineCallbacks
    result = context.run(
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/twisted/python/failure.py", line 549, in throwExceptionIntoGenerator
    return g.throw(self.value.with_traceback(self.tb))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/rest/client/room.py", line 415, in _do
    ) = await self.event_creation_handler.create_and_send_nonmember_event(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/handlers/message.py", line 1025, in create_and_send_nonmember_event
    async with self._worker_lock_handler.acquire_read_write_lock(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/handlers/worker_lock.py", line 224, in __aenter__
    lock = await self.store.try_acquire_read_write_lock(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/lock.py", line 224, in try_acquire_read_write_lock
    lock = await self.db_pool.runInteraction(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/storage/database.py", line 952, in runInteraction
    return await delay_cancellation(_runInteraction())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/twisted/internet/defer.py", line 2010, in _inlineCallbacks
    result = context.run(
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/twisted/python/failure.py", line 549, in throwExceptionIntoGenerator
    return g.throw(self.value.with_traceback(self.tb))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/storage/database.py", line 918, in _runInteraction
    result: R = await self.runWithConnection(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/storage/database.py", line 1047, in runWithConnection
    return await make_deferred_yieldable(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/twisted/python/threadpool.py", line 269, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/twisted/python/threadpool.py", line 285, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/storage/database.py", line 1040, in inner_func
    return func(db_conn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/storage/database.py", line 780, in new_transaction
    r = func(cursor, *args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/lock.py", line 252, in _try_acquire_read_write_lock_txn
    self.db_pool.simple_insert_txn(
  File "/usr/local/lib/python3.12/site-packages/synapse/storage/database.py", line 1098, in simple_insert_txn
    txn.execute(sql, vals)
  File "/usr/local/lib/python3.12/site-packages/synapse/storage/database.py", line 426, in execute
    self._do_execute(self.txn.execute, sql, parameters)
  File "/usr/local/lib/python3.12/site-packages/synapse/storage/database.py", line 488, in _do_execute
    return func(sql, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.UndefinedFile: could not load library "/usr/lib/postgresql/plpgsql.so": /usr/lib/postgresql/plpgsql.so: undefined symbol: ReleaseAllPlanCacheRefsInOwner
2025-01-18 11:22:57,730 - synapse.storage.txn - 787 - WARNING - maybe_backfill_in_the_background-40 - [TXN OPERROR] {try_acquire_read_write_lock-f88f} could not load library "/usr/lib/postgresql/plpgsql.so": /usr/lib/postgresql/plpgsql.so: undefined symbol: ReleaseAllPlanCacheRefsInOwner
 0/5
2025-01-18 11:22:57,730 - synapse.storage.txn - 787 - WARNING - maybe_backfill_in_the_background-40 - [TXN OPERROR] {try_acquire_read_write_lock-f88f} could not load library "/usr/lib/postgresql/plpgsql.so": /usr/lib/postgresql/plpgsql.so: undefined symbol: ReleaseAllPlanCacheRefsInOwner
 1/5
2025-01-18 11:22:57,730 - synapse.storage.txn - 787 - WARNING - maybe_backfill_in_the_background-40 - [TXN OPERROR] {try_acquire_read_write_lock-f88f} could not load library "/usr/lib/postgresql/plpgsql.so": /usr/lib/postgresql/plpgsql.so: undefined symbol: ReleaseAllPlanCacheRefsInOwner
 2/5
2025-01-18 11:22:57,731 - synapse.storage.txn - 787 - WARNING - maybe_backfill_in_the_background-40 - [TXN OPERROR] {try_acquire_read_write_lock-f88f} could not load library "/usr/lib/postgresql/plpgsql.so": /usr/lib/postgresql/plpgsql.so: undefined symbol: ReleaseAllPlanCacheRefsInOwner
 3/5
2025-01-18 11:22:57,731 - synapse.storage.txn - 787 - WARNING - maybe_backfill_in_the_background-40 - [TXN OPERROR] {try_acquire_read_write_lock-f88f} could not load library "/usr/lib/postgresql/plpgsql.so": /usr/lib/postgresql/plpgsql.so: undefined symbol: ReleaseAllPlanCacheRefsInOwner
 4/5
2025-01-18 11:22:57,731 - synapse.storage.txn - 787 - WARNING - maybe_backfill_in_the_background-40 - [TXN OPERROR] {try_acquire_read_write_lock-f88f} could not load library "/usr/lib/postgresql/plpgsql.so": /usr/lib/postgresql/plpgsql.so: undefined symbol: ReleaseAllPlanCacheRefsInOwner
2025-01-18 11:22:57,731 - synapse.metrics.background_process_metrics - 253 - ERROR - maybe_backfill_in_the_background-40 - Background process 'maybe_backfill_in_the_background' threw an exception
Traceback (most recent call last):
  File "/usr/local/lib/python3.12/site-packages/synapse/metrics/background_process_metrics.py", line 251, in run
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/logging/opentracing.py", line 922, in _wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/logging/opentracing.py", line 922, in _wrapper
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/handlers/federation.py", line 235, in maybe_backfill
    async with self._worker_locks.acquire_read_write_lock(
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/handlers/worker_lock.py", line 224, in __aenter__
    lock = await self.store.try_acquire_read_write_lock(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/lock.py", line 224, in try_acquire_read_write_lock
    lock = await self.db_pool.runInteraction(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/storage/database.py", line 952, in runInteraction
    return await delay_cancellation(_runInteraction())
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/twisted/internet/defer.py", line 2010, in _inlineCallbacks
    result = context.run(
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/twisted/python/failure.py", line 549, in throwExceptionIntoGenerator
    return g.throw(self.value.with_traceback(self.tb))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/storage/database.py", line 918, in _runInteraction
    result: R = await self.runWithConnection(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/storage/database.py", line 1047, in runWithConnection
    return await make_deferred_yieldable(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/twisted/python/threadpool.py", line 269, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
             ^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/twisted/python/threadpool.py", line 285, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
                                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
           ^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/storage/database.py", line 1040, in inner_func
    return func(db_conn, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/storage/database.py", line 780, in new_transaction
    r = func(cursor, *args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.12/site-packages/synapse/storage/databases/main/lock.py", line 252, in _try_acquire_read_write_lock_txn
    self.db_pool.simple_insert_txn(
  File "/usr/local/lib/python3.12/site-packages/synapse/storage/database.py", line 1098, in simple_insert_txn
    txn.execute(sql, vals)
  File "/usr/local/lib/python3.12/site-packages/synapse/storage/database.py", line 426, in execute
    self._do_execute(self.txn.execute, sql, parameters)
  File "/usr/local/lib/python3.12/site-packages/synapse/storage/database.py", line 488, in _do_execute
    return func(sql, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
psycopg2.errors.UndefinedFile: could not load library "/usr/lib/postgresql/plpgsql.so": /usr/lib/postgresql/plpgsql.so: undefined symbol: ReleaseAllPlanCacheRefsInOwner

Anything else that would be useful to know?

No response

[head47]

This issue was caused by not upgrading PostgreSQL properly. It is not related to Synapse, and I was able to fix it by following proper PSQL upgrade procedures.