Matrix had 22 outages in the last 2 years totaling 549h 12m of downtime — averaging 0.9 incidents per month.
There were 22 Matrix outages since August 20, 2024 totaling 549h 12m of downtime. Each is summarised below — incident details, duration, and resolution information.
We are aware of degraded performance on the Matrix.org homeserver, particularly affecting logging in.
This incident has been resolved.
Around 07:00 UTC until 08:00 UTC, performance on the matrix.org homeserver was degraded due to a transient problem preventing database maintenance.
We are investigating a problem sending messages via the matrix.org homeserver
The problem was related to an upgrade to the homeserver on matrix.org. We have rolled back to the previous release, and service has been restored.
The homeserver has been stable since rolling back. The Synapse team will work on a resolution to the bug that caused the outage.
Outbound federation from matrix.org paused for several minutes, and is now catching up.
This incident has been resolved.
We are currently investigating this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Performance on matrix.org is currently slow due to ongoing device cleanup.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
This incident has been resolved.
We are currently investigating this issue.
This incident has been resolved.
Due to a substantial increase in traffic, performance is currently degraded.
This incident has been resolved.
We are currently investigating this issue.
The degraded performance was linked to a harmless error which nevertheless was being reported millions of times per hour, affecting CPU. We've fixed the error and service appears to be stabilising.
This incident has been resolved.
We've identified an issue with the matrix.org database
We are in the process of restoring the matrix.org database from a backup. The matrix.org homeserver will be offline until this has been completed
Restoration is ongoing, we do not yet have a concrete estimate for resolution. Live updates here https://bsky.app/profile/matrix.org/post/3lxuslbzjuc2t
The database has been restore, and verification checks complete. matrix.org's Synapse has been started up again and we are monitoring
We are continuing to monitor for any further issues.
This incident has been resolved.
We are investigating continued problems with inbound federation on the matrix.org homeserver
We are continuing to investigate this issue.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
matrix.org inbound federation was inoperable from roughly 18:55 UTC to 20:00 UTC due to a logic error during a code roll out. The issue is now resolved.
We're investigating reports of broken functionality in federated and unfederated rooms that contain matrix.org users. As a workaround, one can create a new room and continue communicating there. We are working to identify the cause and correct the affected rooms.
We are still working to determine the root cause, which currently appears to only originate on the matrix.org homeserver. In the meantime, we've devised a fix and will be attempting that shortly.
We are still working to restore all affected rooms
We believe we have restored the majority of the affected rooms. We are working on verifying and identifying any remaining rooms that may have been affected.
We believe that all of the affected rooms have been fixed.
We are currently investigating this issue.
The issue is due to degraded DB performance. We've identified what's slowing the DB down and cleaning it up.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We're currently identifying the root cause for degraded performance on matrix.org's Synapse instance.
We are continuing to investigate this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently investigating this issue.
The database performance issue has now been identified and resolved.
We are currently investigating this issue.
This incident has been resolved.
We are currently investigating the cause of this issue.
We are continuing to investigate this issue.
We've identified the cause and are currently implementing a fix.
A fix has been implemented and outbound federation appears to have returned.
Outbound federation is operational.
We are having issue with our email provider. No emails are being sent. Both registration and password reset are impacted.
We are continuing to work on a fix for this issue.
This incident has been resolved.
We are currently aware of an issue which is causing some issues with the creation of new devices in the Sliding Sync Proxy. Some users may find that clients such as Element X may never finish logging in or loading the home screen. Logging out and back in again may be sufficient to work around the problem. Clients that use the standard/old sync mechanism ('Sync v2', /v3/sync) and clients using the new native Simplified Sliding Sync support are not affected.
This incident has been resolved.
Emails are currently not being delivered. This affects registration of new accounts and delivery of email notifications.
This incident has been resolved.