Exoscale Outage History

We are experiencing issues with a scheduled NLB maintenance. API is not working, while no traffic drop to existing NLBs is expected. We are working on the fix. Sorry for the inconvenience.

We are investigating a limited number of managed Elastic IPs connectivity issue

The root cause has been identified. We are preparing mitigation.

The issue is only impacting CH-DK-2, AT-VIE-1, DE-FRA-1 and BG-SOF-1. The number of impacted mEIPs is limited.

We are applying mitigation

Updating clusters or nodepools is not possible at the moment.

We observe elevated error rate on most API operations.

Issue escalated to partial outage

All compute related API actions are currently affected. We are currently investigating the cause.

The situation is improving. Less latency or errors are being observed

Situation has mostly recovered back to nominal for most API calls

We are investigating an increased error rates and latencies on the API. We’ll post an update as soon as we have more information.

We are seeing increased latency and timeouts on compute related API endpoints in all zones.

IPv4 is not affected. One of our transit providers is currently no longer advertising our IPv6 prefixes. IPv6 traffic is operating through our remaining transit provider. We have opened a ticket with the affected provider and are awaiting their response. During this time, some users may experience higher-than-normal latency due to traffic being routed through a single transit provider.

Our transit provider has acknowledge the issue and they are investigating.

We are investigating increased error rate with our NLB API in vie2

NLB API operations are currently not available in sof1. Traffic and functionality of existing NLBs is not affected.

Our engineers are actively investigating the issue. They found the root cause of the issue and are working to recover the service.

Our engineers are actively working on recovering the service.

Our engineers are actively working on recovering the service.

We are investigating Increased internet network latencies and packet loss. We’ll post an update as soon as we have more information.

Network latency issue is now resolved.

Network latency issue is now resolved.

We are investigating an issue preventing custom template deletion

We have identified the issue.

We are observing issues while managing SKS entities at BG-SOF-1. We are investigating

We are trying to identify the root-cause of the issue. We will report back as soon as we have more information.

We have identified the root-cause of the problem. A mitigation will be applied soon.

To clarify the full impact of the problem: The Web Portal is unavailable at the moment. IAM is also unavailable. Creation of new SKS clusters is not available at the moment. Existing SKS clusters are not impacted.

Due to a wrong manipulation, we had to stop the SKS service in SOF1 while restoring the database. In the meantime, all Clusters and NodePools stay reachable and work as expected.

We’re experiencing increased error rate on some NLBs in HR-ZAG-1 zone.

We’re continuing to investigate the root cause.

We have identified the root cause and are applying a mitigation.

We are currently experiencing network instability on a link with one of our BGP transit providers. Redundancy measures are actively routing traffic, so we do not expect any impact on service availability.

We are currently experiencing a degradation of network redundancy in AT-VIE-1 due to an ongoing incident affecting one of our transit providers. Following a maintenance operation on their side, the provider encountered a hardware failure on a router serving our connectivity in Vienna. The device could not be recovered remotely and requires a full replacement. At this time, services in the zone remain operational. However, redundancy is reduced, which may increase sensitivity to additional network events. The transit provider has dispatched a replacement device, with an estimated arrival on-site on April 16 at 03:00 CEST. Restoration of full redundancy will occur after the hardware is installed and services are fully recovered. We are closely monitoring the situation and will provide updates as more information becomes available. Impact: Reduced network redundancy in VIE1 (no current service disruption)

We’re seeing degraded performance of our object storage stack in entire DE-MUC-1 zone.

We’re currently applying a mitigation. We’ll update the incident as soon as we have new information.

We’re experiencing increased error rate on some NLBs in CH-DK-2.

We are experiencing increased latency on Object Storage in ch-gva-2

We’re experiencing increased error rate on some NLBs in DE-FRA-1.

We do still experience increased error rate, and continue to investigate