FR-PAR-2 connectivity loss

Update timeline

1. investigating May 22, 2026, 09:21 AM UTC

   We are currently investigating on that issue right now.

2. investigating May 22, 2026, 09:21 AM UTC

   We are continuing to investigate this issue.

3. investigating May 22, 2026, 09:23 AM UTC

   We are continuing to investigate this issue.

4. monitoring May 22, 2026, 09:42 AM UTC

   Most services recovered and our teams are mobilized to recover the rest. Root cause is a power issue in datacenter.

5. monitoring May 22, 2026, 10:24 AM UTC

   Datacenter hosting FR-PAR-2 availability zone occured a complete power failure in operator rooms during a planned datacenter maintenance. Both operator rooms (network connectivity) were powerless for a few minutes leading to a complete isolation of the availability zone during the issue. We are investigating with the datacenter provider to understand the root cause.”

6. monitoring May 22, 2026, 10:28 AM UTC

   Teams are working on the recovery. Here is the status of impacted products: - Instances: you may encounter issues with l_ssd/block snapshots or volumes - Kapsule: Control planes are reachable and in nominal state, you may need to verify that your controllers reconnected properly on the apiservers. You may have experienced node replacement due to the autoheal or autoscaling process, this may delay volume re-attachments. - Databases: snapshots may be blocked, some failovers have been started for HA - Object storage: you may encounter some latencies

7. identified May 22, 2026, 10:31 AM UTC

   Datacenter informs us that we are running on power generators for now until the root cause is identified

8. identified May 22, 2026, 10:44 AM UTC

   As the power failure root cause is not yet understood (datacenter is investigating with its providers), we encourage you to shift workload on alternatives regions/AZ if possible.

9. monitoring May 22, 2026, 11:10 AM UTC

   Updated status for impacted products: - Instances: everything back to normal - Kapsule: same status - Databases: same status - Object Storage: same status - Serveless containers and functions: might face connection timeout issues when contacting their applications

10. monitoring May 22, 2026, 11:25 AM UTC

    Clarification: only network rooms are running on power generators, with enough fuel for a few days

11. monitoring May 22, 2026, 12:26 PM UTC

    Updated status for impacted products: - Kapsule: everything back to normal - Databases: everything back to normal - Object Storage: still some latency issues and timeout on connecting some buckets - Serveless containers and functions: might face connection timeout issues when contacting their applications - File Storage: some instances may encounter troubles with their File Storage, we invite customers to restart their instances/nodes if affected

12. monitoring May 22, 2026, 02:30 PM UTC

    Updated status for impacted products: - Object Storage: still some latency issues and timeout on connecting some buckets - Serveless containers and functions: Everything is back to normal. Situation recovered and stable since 02:40 PM - File Storage: some instances may encounter troubles with their File Storage, we invite customers to restart their instances/nodes if affected

13. monitoring May 22, 2026, 02:57 PM UTC

    Updated status for impacted products: - Object Storage: Everything back to normal - Serveless containers and functions: Everything is back to normal. - File: some instances may encounter troubles with their File Storage, we invite customers to restart their instances/nodes if affected As the failover malfunction has not been diagnosed at this time, our datacenter manager will keep the affected elements under a generator.

14. monitoring May 22, 2026, 08:30 PM UTC

    The datacenter provider has planned an intervention on Saturday at 10:30 CEST to replace batteries on the electrical paths that had issues today. This operation will be done live and should have no impact. Once replaced, old batteries will be inspected and tested to confirm they were the cause of the incident. Power will be kept on generators during this operation and after until the root cause is fully confirmed

15. monitoring May 23, 2026, 09:56 AM UTC

    Batteries replacement is ongoing, 1 of the 4 electrical paths has been done (no impact). Datacenter teams will proceed with the next paths.

16. monitoring May 23, 2026, 11:09 AM UTC

    Batteries replacement has been completed on all 4 electrical paths. Removed batteries are under tests and investigations to confirm their involvement in the initial incident. No more action on production expected for now. We are still on power generators for the time being.

17. monitoring May 23, 2026, 11:41 AM UTC

    Datacenter teams are now confident the batteries were involved in the initial incident, they will attempt to switch back to main power at 14:30 CEST.

18. resolved May 23, 2026, 12:40 PM UTC

    Switchover to main power has been completed successfully. We are back to a nominal state.

19. postmortem Jun 05, 2026, 03:32 PM UTC

    **Correction to Error Report dated June 3, 2026** **FR PAR 2 Connectivity Loss** **Incident Overview** On 22 May 2026, power maintenance in the OpCore PAR5 datacenter led to a total loss of power in the 2 operator rooms. This shutdown caused a global connectivity loss for the FR-PAR-2 availability zone from 11:08 to 11:10 UTC\+2. Service restoration continued until 12:30 UTC\+2. **Incident status links:** [https://status.scaleway.com/incidents/qkj49g6g3ykp](https://status.scaleway.com/incidents/qkj49g6g3ykp) **Regions/AZ Impacted**: FR-PAR-2 **Initial Duration**: 1h50 **Duration by impacted products** : * **Control-Plane:** mainly 20 minutes. * **Data-Plane:** mainly 20 minutes, though some effects continued after connectivity was restored. * **Instance:** 1h50 for instances with block volumes. * **Kapsule:** 1h50, but actions were required on the customer side. \(please check on your end for faulty sockets or connections on your controller or workload. For example, broken SFS sockets\). * **Database:** 1h50, to verify that all databases had returned to a healthy state. * **Object Storage:** TBD \(pending investigation\). Latencies persisted after connectivity was restored. * **File Storage:** 1h50, but actions were required on the customer side. * **Serverless Containers and Functions:** 3h30, due to a Cilium bug affecting recovery after the incident * **Dedibox / Elastic Meta**l: 20 minutes **Primary Impact**: Connectivity Loss **Secondary Impact**: Products with a presence in FR-PAR-2. Mainly Instance, Kapsule, Databases, Object Storage, File Storage, and Serverless Containers & Functions. #### **Impact on Customers** During the incident, all services hosted in FR-PAR-2 were unreachable #### **Root Cause Analysis** **5 whys** * **Why did all networks go down in FR-PAR-2 ?** The datacenter hosting FR-PAR-2 is connected to the backbone through two distinct network paths and equipment, hosted in two distinct rooms in the datacenter. Each of these rooms is powered by two electrical paths. Each electrical path hosts an electrical workshop that converts power \(230 V / 48 V\) to feed the equipment in the room. For a reason that is still unknown, all two electrical workshops encountered an issue triggering network equipment reboot. * **Why did power have an issue ?** Maintenance was planned by the datacenter operator to test the power generators. This is a recurring test, carried out every few weeks to ensure the power generators, batteries, and power switching systems are working correctly. Rectifier batteries failed to support load during switch phase from main to generators leading to 10 seconds power loss. * **Why were software services affected by the power/network issue?** When the availability zone became disconnected from the other AZs, it caused a disconnection of redundant systems \(failover, DBs, data synchronisation, etc.\). When the AZ reconnected, we needed to unlock DBs, reconnect master/slaves, and resync data from the other AZs, which caused some delay in fully restoring services. In addition, many customer requests \(automated or not\) were queued and had to be processed in order to ensure the system remained consistent, causing increased latencies and delays. * **Why were other Availability Zones affected ?** Other availability zones in FR-PAR were partially impacted during the disconnection event. This was mostly caused by failover systems handling the large surge of requests needed to fail over every redundant part of each software service, causing delays and latencies in processing everything. **Conclusion** We will continue to analyse the various events that slowed down some failover events with the other AZs. A series of batteries were confirmed as faulty and have been replaced in the following 24 hours by the provider. The batteries were guaranteed for 12 years and were only at 8 years of service. Investigation is ongoing with the battery provider to understand the underlying cause of early faults. #### **RemediationShort term** **Datacenter provider:** * Battery have been replaced **Mid term** **Scaleway:** * Working on improving our recovery process. **Datacenter provider:** * Pending investigation. **Contact** If you have any further questions or need assistance, please contact our support team. **Impacted services state.** | Service | Mono AZ impact duration | Multi AZ impact duration | Comments | | --- | --- | --- | --- | | Instance | 1 hour and 50 minutes | N/A | Instances on FR-PAR-2 impacted. | | Block storage | 1 hour and 50 minutes | N/A | Availability for some elements stored in FR-PAR-2 \(20 minutes\)After power restarted, some latencies \(10 minutes\) | | Kapsule \(K8S\) | 1 hour and 50 minutes | 10 minutes | Time to converge and respawn assets. | | Object storage | 1 hour and 50 minutes | 0 minutes | Multi AZ not impacted One zone object storage: if the bucket is on FR-PAR-2, this one was unavailable. | | Managed Databases | 1 hour and 50 minutes \(only multi AZ replica impacted\) | N/A | It was not possible to create a multi AZ HA database, or to connect/promote a read replica. | | [api.scaleway.com](http://api.scaleway.com) | N/A | 0 minutes | api is regional no outage. | | [console.scaleway.com](http://console.scaleway.com) | N/A | 1 hour and 50 minutes | failover not working as expected. | | file storage | 1 hour and 50 minutes | N/A | manual intervention was required from customers to recover. | | Serverless Container and Functions | 3 hours and 30 minutes | | Network problems on recovery led to an extended outage. |