Exonet experienced a major incident on November 28, 2022 affecting Compute, lasting 3h 31m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 28, 2022, 01:14 PM UTC
We are experiencing a partial outage on our virtualization platform. Engineers are currently investigating the issue. Customers may experience downtime on some services until the issue has been resolved.
- identified Nov 28, 2022, 01:28 PM UTC
Our engineers are currently executing failovers of virtualization nodes. Affected hosts and services are coming back online now.
- identified Nov 28, 2022, 01:59 PM UTC
Multiple virtualization nodes required a manual failover. Engineers are still busy starting all servers.
- identified Nov 28, 2022, 02:21 PM UTC
The node failovers didn't resolve the previous issue because there is another issue with a failed switch, resulting in read-only servers after the node failover. Engineers are currently investigating this issue.
- identified Nov 28, 2022, 02:50 PM UTC
The issue with the switches has been identified and a fix has been applied. Unfortunately it requires another forced failover for all nodes that were affected. Engineers are now starting all the affected servers again.
- identified Nov 28, 2022, 03:20 PM UTC
All servers are back online. We're checking the servers and monitoring for any remaining issues.
- resolved Nov 28, 2022, 04:45 PM UTC
All remaining services with issues have been fully restored. Our apologies for this unexpected outage. A full reason for outage will be emailed tomorrow.
- postmortem Nov 29, 2022, 04:34 PM UTC
All affected customers have received an RFO by email.