IONOS Cloud incident

MK8s - Connectivity Issue

IONOS Cloud is currently experiencing a minor incident affecting Managed Kubernetes, which began 5d ago. The vendor's full update timeline is below.

Started
Jul 18, 2026, 01:33 PM UTC
Resolved
Ongoing
Duration
● 4d 15h
Detected by Pingoru
Jul 18, 2026, 01:33 PM UTC

Affected components

Managed Kubernetes

Update timeline

  1. investigating Jul 18, 2026, 01:33 PM UTC

    We are currently investigating a suspected network connectivity issue influencing the MK8s service. We will keep the status page updated with new information as they become available.

  2. investigating Jul 18, 2026, 01:55 PM UTC

    Initial investigation makes a network connectivity issue unlikely. The team is focusing investigation on Managed Kubernetes side.

  3. identified Jul 18, 2026, 02:26 PM UTC

    We have identified a spike in our provisioning engine queue that is a likely root cause for the issues observed. Our provisioning team is informed and has joined the response.

  4. identified Jul 18, 2026, 02:47 PM UTC

    We have narrowed down the issue and believe we have the culprit identified. We are currently confirming the finding.

  5. identified Jul 18, 2026, 03:03 PM UTC

    Our storage team has confirmed the suspected culprit. We are currently mitigating the issue and will monitor provisioning job execution afterwards.

  6. identified Jul 18, 2026, 03:51 PM UTC

    Storage issue has successfully been resolved, however, job provisioning is currently not progressing successfully. Our provisioning team is investigating.

  7. identified Jul 18, 2026, 04:54 PM UTC

    While provisioning block could be resolved, we are investigating an increased number of storage related errors. We are directing attention to these remaining issues. Customers could still see issues with attaching storages on Kubernetes.

  8. identified Jul 18, 2026, 05:24 PM UTC

    We have found an issue with one storage server belonging to a redundant storage server pair. Our team is implementing a mitigation.

  9. monitoring Jul 18, 2026, 05:54 PM UTC

    The incident should now be mitigated. The affected storage server is currently getting restored. Once this is completed, redundancy will be fully restored. We do not see any remaining provisioning issues any longer. We are monitoring the situation and the restoration and will then set the incident to resolved.

  10. monitoring Jul 18, 2026, 07:27 PM UTC

    Due to the ongoing restoration effort on the storage server customers might still see residual impact when attaching/detaching storage until redundancy is fully restored. We are resetting the impact of the service back to Degraded Performance.

  11. monitoring Jul 18, 2026, 08:24 PM UTC

    The team has encountered a complication during the recovery of the second storage server in the pair. Hardware needs to be replaced. Our datacenter team is working on this task. For customers still affected we are working on implementing a mitigation to unblock storage operations in parallel.

  12. monitoring Jul 18, 2026, 09:31 PM UTC

    Hardware replacement is being conducted. The team continues to mitigate acute storage provisioning issues until hardware replacement and recovery is completed to minimize customer impact.

  13. monitoring Jul 18, 2026, 10:03 PM UTC

    Hardware replacement was completed. Restoration of redundancy is continuing.

  14. monitoring Jul 18, 2026, 10:43 PM UTC

    The first hardware replacement was unsuccessful. Another one is attempted. In the meanwhile a data migration is running to migrate data to another storage target. Due to the amount of data to be migrated the migration is estimated to take several hours.

  15. monitoring Jul 19, 2026, 12:21 PM UTC

    The migration of storages has been completed, which should prevent further issues with attaching storage. Migration of snapshot data is currently ongoing, so snapshot related activities might still fail to complete successfully.