Matrix incident

Service outage on Matrix.org Synapse

Critical Resolved View vendor source →

Matrix experienced a critical incident on November 12, 2023 affecting Synapse and Outbound federation, lasting 1h 46m. The incident has been resolved; the full update timeline is below.

Started
Nov 12, 2023, 01:28 AM UTC
Resolved
Nov 12, 2023, 03:15 AM UTC
Duration
1h 46m
Detected by Pingoru
Nov 12, 2023, 01:28 AM UTC

Affected components

SynapseOutbound federation

Update timeline

  1. investigating Nov 12, 2023, 01:28 AM UTC

    We're investigating continued outages. Login and sending messages may not work in the meantime.

  2. monitoring Nov 12, 2023, 03:06 AM UTC

    The issue was fixed, which caused a large amount of outgoing federation traffic. This resulted in a similarly large influx of federation traffic, which our systems interpreted as a DDoS and cut inbound federation traffic. We've now allowed traffic to flow again, and currently traffic patterns appear normal. We are monitoring the situation.

  3. resolved Nov 12, 2023, 03:15 AM UTC

    This incident has been resolved.