Resolved after 4 hours 56 minutes, on August 31, 2026.
After a thorough investigation, we identified the root cause of the incident: a software issue between the AMD platform and IOMMU caused the PCI devices backing the NVMe drives on cluster node ams-9950x-odessana to malfunction on a software level, forcing the system to switch into a read-only mode and render the VMs unoperable. The issue was initially misdiagnosed as a drive failure, which led to an extended delay while we requested remote hands at the data center to replace the suspected faulty drive. Further diagnosis revealed the actual cause. We adjusted the BIOS settings accordingly, then gracefully recovered and resynced the RAID array on the node. All of the VMs are back up since 00:04 AM UTC, and all customer data is safe. However, we strongly recommend running a file system check on any virtual machines hosted on ams-9950x-odessana to rule out any latent issues. We sincerely apologize for the disruption and the extended downtime. We will be providing two extra days of service time to all customer services that were affected by this outage.
Update timeline
- investigating Aug 31, 2026, 05:55 PM UTC
We are investigating the issue on the ams-9950x-odessana node in the Netherlands.
- identified Aug 31, 2026, 06:56 PM UTC
We found the cause of the issue and are working with the data center engineers to diagnose and resolve it as soon as possible.
- identified Aug 31, 2026, 09:47 PM UTC
We are continuing our work on resolving the issue.
- resolved Sep 01, 2026, 12:22 AM UTC
After a thorough investigation, we identified the root cause of the incident: a software issue between the AMD platform and IOMMU caused the PCI devices backing the NVMe drives on cluster node ams-9950x-odessana to malfunction on a software level, forcing the system to switch into a read-only mode and render the VMs unoperable. The issue was initially misdiagnosed as a drive failure, which led to an extended delay while we requested remote hands at the data center to replace the suspected faulty drive. Further diagnosis revealed the actual cause. We adjusted the BIOS settings accordingly, then gracefully recovered and resynced the RAID array on the node. All of the VMs are back up since 00:04 AM UTC, and all customer data is safe. However, we strongly recommend running a file system check on any virtual machines hosted on ams-9950x-odessana to rule out any latent issues. We sincerely apologize for the disruption and the extended downtime. We will be providing two extra days of service time to all customer services that were affected by this outage.
Your product runs on more than SENKO
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services