Is SENKO down?

We are currently performing an emergency replacement of a failed NVMe drive in this node's RAID array. All customer data is safe and intact. Affected VMs will be back online as soon as the disk replacement is complete and the node has been booted back into the OS. We'll post an update once service is restored.

The affected node is back online and virtual machines are now powering back on. Services should return to normal over the next 15-20 minutes as each VM completes boot.

The VM panel is currently under maintenance, therefore it is not available. Please contact the support team if you require any changes within the VM, such as a OS reinstall or password change. Please note that the maintenance time does not reflect the actual time it will take for the panel to come back online.

Marking as resolved (see post-mortem)

We are investigating the issue on the ams-9950x-odessana node in the Netherlands.

We found the cause of the issue and are working with the data center engineers to diagnose and resolve it as soon as possible.

We are continuing our work on resolving the issue.

After a thorough investigation, we identified the root cause of the incident: a software issue between the AMD platform and IOMMU caused the PCI devices backing the NVMe drives on cluster node ams-9950x-odessana to malfunction on a software level, forcing the system to switch into a read-only mode and render the VMs unoperable. The issue was initially misdiagnosed as a drive failure, which led to an extended delay while we requested remote hands at the data center to replace the suspected faulty drive. Further diagnosis revealed the actual cause. We adjusted the BIOS settings accordingly, then gracefully recovered and resynced the RAID array on the node. All of the VMs are back up since 00:04 AM UTC, and all customer data is safe. However, we strongly recommend running a file system check on any virtual machines hosted on ams-9950x-odessana to rule out any latent issues. We sincerely apologize for the disruption and the extended downtime. We will be providing two extra days of service time to all customer services that were affected by this outage.

We are investigating the issues with the cluster nodes fra-9275f-odessana, fra-9555p-rooki, fra-9375f-budget-2, fra-9334-budget-3, ams-9575f-vulpes, ams-9575f-odessana

During planned maintenance affecting five cluster nodes in our fra-9950X cluster, the data center power phase was brought down as scheduled. Due to the interaction between the motherboard firmware and the redundant power supply configuration used on these servers, the motherboards locked CPU frequency to 600 MHz. This caused severe performance degradation and VM outages for customers hosted on the affected nodes. Affected cluster nodes: fra-9275f-odessana fra-9555p-rooki fra-9375f-budget-2 fra-9334-budget-3 Separately and unrelated to the incident above, our Netherlands location experienced an issue with the VM platform that caused a partial networking malfunction, resulting in outages on: ams-9575f-vulpes ams-9575f-odessana This has already been resolved and no further action is required. ℹ️ This is an unplanned incident. No action is required from customers at this time. We will automatically provide two extra days of service to all impacted customers as compensation for this incident. We sincerely apologize for the inconvenience. This incident was not something we could have anticipated. The issue has already been resolved, and our team has restored full performance across all affected nodes. Thank you for your patience and understanding.