Matrix incident

Degraded performance due to database contention

Minor Resolved View vendor source →

Matrix experienced a minor incident on November 8, 2023 affecting Synapse and Outbound federation, lasting 5h 48m. The incident has been resolved; the full update timeline is below.

Started
Nov 08, 2023, 05:58 PM UTC
Resolved
Nov 08, 2023, 11:47 PM UTC
Duration
5h 48m
Detected by Pingoru
Nov 08, 2023, 05:58 PM UTC

Affected components

SynapseOutbound federation

Update timeline

  1. investigating Nov 08, 2023, 05:58 PM UTC

    We are currently investigating degraded performance across matrix.org Synapse due to increased amounts of database traffic. Encrypted messages may be slow to decrypt as a result.

  2. monitoring Nov 08, 2023, 06:44 PM UTC

    We have identified the problem and have applied a fix. Now monitoring the situation to see whether it improves.

  3. resolved Nov 08, 2023, 11:47 PM UTC

    Service has been nominal over the last hour, and we're now confident that the issue has been resolved.