Major outage of our compute infrastructure in FRA-2

Update timeline

1. investigating Dec 05, 2024, 01:54 PM UTC

   We're currently experiencing a major outage on our compute infrastructure in our FRA-2 location. We're investigating the issue and will provide additional information soon

2. identified Dec 05, 2024, 02:22 PM UTC

   We've identified the issue and are working on recovering all services ASAP

3. identified Dec 05, 2024, 02:27 PM UTC

   We are currently booting all affected servers again. The services will be back online in the next minutes.

4. monitoring Dec 05, 2024, 06:00 PM UTC

   We brought all services back up and are closely monitoring them

5. monitoring Dec 05, 2024, 10:14 PM UTC

   All services are running as expected again. We will keep monitoring and let the incident open till we publish an official statement tomorrow.

6. resolved Dec 06, 2024, 09:40 AM UTC

   All systems and services have been stable since the recovery and the normal operational state has been fully recovered. The root cause was a power outage in one of our locations. We don't have information about the exact cause yet but are in contact with the datacenter provider to get more information about the incident. To ensure that a similar incident can be preempted or mitigated, we will do an internal review and inform you about our steps as soon as this has been completed.