mastodon.social Outage History

This incident has been resolved.

Type: Incident Duration: 21 hours and 15 minutes Affected Components: Website & API, Background queues Apr 20, 10:58:33 GMT+0 - Investigating - We are currently experiencing a DDoS attack. We are actively investigating the issue. Apr 20, 13:05:28 GMT+0 - Monitoring - We have implemented countermeasure against the DDoS attack, and the site is accessible. We are continuing to monitor the situation. Apr 21, 08:13:37 GMT+0 - Resolved - The attacks seem to have stopped, all operations have returned to normal.

Type: Incident Duration: 27 minutes Affected Components: Website & API, Streaming API, Background queues Feb 25, 21:08:32 GMT+0 - Investigating - Hetzner, our cloud provider, appears to be having a significant outage, leaving many of our resources unavailable. We are monitoring the issue. Feb 25, 21:16:30 GMT+0 - Monitoring - Services have been restored and Hetzner connectivity seems to have returned. Will continue to monitor the situation. Feb 25, 21:35:12 GMT+0 - Resolved - This incident has been resolved.

Type: Incident Duration: 12 days, 18 hours and 22 minutes Affected Components: Website & API, Background queues Dec 17, 14:00:00 GMT+0 - Investigating - There appears to be an issue with connection timeouts to our Sidekiq runners, causing some operations to take longer than expected, or require multiple tries. We are currently investigating this issue. Jan 18, 09:22:00 GMT+0 - Resolved - Sidekiq has been stable, so considering this incident resolved. Jan 7, 06:42:57 GMT+0 - Monitoring - We found out the cause for this issue (within a dependency) and we deployed a fix 8 hours ago. So far, it seems to have completely fixed the issue.

Type: Incident Duration: 8 minutes Affected Components: Website & API, Background queues Nov 3, 17:20:27 GMT+0 - Identified - [mastodon.social](http://mastodon.social) is currently unresponsive to a database issue. We identified the problem and are working on fixing it. Nov 3, 17:28:47 GMT+0 - Resolved - This incident has been resolved, everything is now operational again.

Type: Incident Duration: 6 days, 2 hours and 35 minutes Jun 17, 13:50:20 GMT+0 - Identified - We paused sending emails from [mastodon.social](http://mastodon.social) (including password recovery and account confirmation emails) due to an identified issue that sent duplicate emails about our upcoming Terms of Service changes to some users. We are working on fixing this and avoiding more duplicate emails to be sent. Jun 17, 17:42:30 GMT+0 - Monitoring - Email sending has been removed. Servers are working through the backlog, and any pending email should arrive in the next few hours. Jun 18, 05:40:36 GMT+0 - Identified - The backlog of emails has been sent, and emails are reaching out their recipients now. Some emails sent in the last 12 hours are stuck with our email provider and we are working with them to fix this situation. Jun 23, 16:25:19 GMT+0 - Resolved - This incident has been resolved last Thursday, and email have been delivered correctly since then.

Type: Incident Duration: 2 hours and 40 minutes Affected Components: Website & API, Background queues Jun 11, 22:33:26 GMT+0 - Investigating - We are experiencing issues with our redis caching system, as well as kubernetes node errors. We are currently investigating the issue. Jun 11, 22:42:52 GMT+0 - Identified - Website is operative again. Issue seems to be related to kubernetes pods becoming completely unresponsive. Continuing to investigate. Jun 12, 01:13:29 GMT+0 - Resolved - Kubernetes node issues have been resolved, and service is now fully restored.

Type: Maintenance Duration: 16 minutes Affected Components: Background queues, Website & API May 28, 06:30:01 GMT+0 - Identified - Maintenance is now in progress May 28, 06:30:00 GMT+0 - Identified - We will be updating the database to the next minor version, as well as some OS components that will require a restart. Minor downtime of roughly a minute is expected. May 28, 06:46:24 GMT+0 - Completed - Maintenance has completed successfully.

Type: Incident Duration: 11 days, 20 hours and 41 minutes Affected Components: Website & API Jan 29, 16:49:03 GMT+0 - Investigating - We are currently investigating this incident. Jan 29, 17:30:58 GMT+0 - Identified - We've experienced an issue with our recent cache storage upgrade, causing some ephermal data loss (e.g. home feeds losing scrollback or showing the number of active users as "0"). We are working on fixing the underlying issue and restoring/regenerating the lost data. Feb 10, 13:30:27 GMT+0 - Resolved - Data has been restored, and performance issues have been mitigated.

Type: Incident Duration: 30 minutes Affected Components: Website & API Jan 25, 08:43:04 GMT+0 - Resolved - This incident has been resolved. Jan 25, 08:12:40 GMT+0 - Investigating - We are currently investigating this incident.

Type: Incident Duration: 2 hours and 55 minutes Affected Components: Website & API Jan 16, 07:13:58 GMT+0 - Investigating - There appears to be an issue with our nodes connecting to the DB reliably. We are currently investigating the issue. Jan 16, 07:18:39 GMT+0 - Monitoring - DB connections are stable again. Current suspect is a Hetzner networking issue somewhere in the backend. Currently monitoring. Jan 16, 10:09:20 GMT+0 - Resolved - No acknowledgement from Hetzner, but everything remains stable. Considering this closed.

Type: Incident Duration: 18 hours and 31 minutes Affected Components: Streaming API, Background queues, Website & API Nov 13, 16:17:05 GMT+0 - Identified - There was an issue with our cluster autoscaler where it seems to have scaled down the number of nodes we have for our cluster, resulting in strained resources. We are trying to scale back up, but Hetzner is having availability issues in the region we are deployed in. We are working to resolve the issue as quickly as possible. Nov 14, 10:48:00 GMT+0 - Resolved - Capacity issues have been resolved, and the autoscaler has been adjusted.

Type: Incident Duration: 3 hours and 13 minutes Affected Components: Background queues Sep 16, 06:50:00 GMT+0 - Investigating - We are currently unable to send emails to users due to issues with Postmark. We are currently investigating the issue. Sep 16, 10:02:51 GMT+0 - Resolved - Issue was identified as an SSL error on Postmark's side. This has now been resolved, and emails are working again.