Is IONOS Cloud down?

While phone support will generally be available, our Support capacity is reduced due to multiple cases of sick leave. We want to inform you that our phone support will be limited during the following time slots, leading to increased response times on the phone channel. In these cases, please reach out per email. Thank you.

We have people available now, Support is reachable as usual.

While phone support will generally be available, our capacity is reduced due to multiple cases of sick leave. We want to inform you that our phone support will be limited during the following time slots, leading to increased response times on the phone channel. In these cases, please reach out per email. Thank you.

We still run short on coverage of service, and we are currently doing changes on our ticketing system. So we kindly ask for your understanding if Support responses are not as timely as you are accustomed to. Thank you.

We managed to fully re-establish our service, Support is back. thank you for your patience.

For our partners with subcontracts: We found that subcontracts cannot access the new Frankfurt location "de/fra/1". We are currently working on a fix to make this available, we will keep you informed when it's done.

Wir konnten das Problem beheben, sodass die neue Region nun für alle Verträge verfügbar ist. Wir monitoren, ob es dazu noch weitere Rückmeldungen gibt.

This incident has been resolved.

The DCD is currently unavailable, displaying an endless "Loading DCD..." page. We are currently investigating this issue, and will keep you updated.

The DCD is now available again, and we are monitoring the situation.

We are marking this incident as resolved. Our network team is investigating negative impact on the IAM Services caused by a change rollout. We will update the Status Page once the root cause has been established.

# **Preliminary Root Cause Analysis** This Root Cause Analysis is preliminary as research is still being conducted to determine the technical root cause of the incident. ## **What happened?** On August 31, 2026, between 15:00 UTC and 15:43 ITC, and again between 16:32 UTC and 16:35 UTC, customers were unable to reach IONOS Cloud's Identity and Access Management \(IAM\) service and the Data Center Designer \(DCD\), which is dependent on this service. The disruption affected direct logins, partner and reseller portal access, and associated management consoles. The total customer-facing impact lasted approximately 48 minutes across both intervals. Cloud APIs \([api.ionos.com](http://api.ionos.com)\) remained fully operational throughout the incident. The underlying application services were healthy at all times - the failure was confined to the network edge layer. ## **How was this possible? \(Root Cause\)** During a scheduled maintenance window on August 31, 2026, a network configuration update was applied to the edge network infrastructure with the intent of optimizing routing filters. The configuration was verified as correct prior to and during application. Following the rollout, a routing propagation anomaly emerged on one edge network switch, causing asymmetric routing behavior: incoming TCP connection requests from clients were silently dropped \(black-holed\) at the network edge before reaching the application cluster. Because the application services themselves remained up and healthy, this failure mode was not immediately visible through internal health checks - the services were isolated from receiving inbound public internet traffic rather than failing. The root cause of why this specific switch exhibited asymmetric routing behavior following an otherwise valid configuration change remains under active investigation. Whether this was triggered by a switch platform behavior or a software version-specific bug is being determined through staging environment reproduction. ## **What are we doing to prevent recurrence?** ### **Immediate Actions** * **Configuration Rollback:** Upon identifying the routing anomaly, a full rollback of the network configuration was executed across all affected edge switches. Routing announcements and TCP reachability of the public production IP addresses were verified following the rollback. All affected services - DCD, Partner Portal, Reseller Portal, and IAM - were confirmed fully operational by 16:35 UTC. \(DONE\) ### **Short-term** * **Staging Environment Replication:** Detailed test cases are being executed in the staging environment to reproduce the exact routing propagation behavior under the same switch configuration conditions. The goal is to determine whether the anomaly is attributable to a specific software version bug or a switch platform behavior, so that the precise failure condition can be isolated and addressed before any future rollout. ETA: Within two weeks ### **Mid-term** * **Revised Rollout Strategy:** Based on the findings from staging, a revised rollout approach will be designed to ensure that any future application of these routing filter optimizations can be performed with greater stability guarantees - including more granular validation checkpoints between switch-level changes. ETA: October 2026 ## **Closing remarks** We recognize that loss of access to IAM and DCD carries real operational impact. The fact that the underlying services were healthy throughout is not a mitigation of that impact. We are committed to ensuring that the root cause is fully understood before any re-attempt of the original change, and that the revised rollout strategy addresses the conditions that led to the asymmetric routing behavior. We thank you for your patience while we conclude the investigation.

We are currently investigating increased latency affecting S3 Object Storage in the eu-central-1 region. Some customers may experience slow response times for read and write operations. Our engineering team is actively working on resolving the issue. We will provide updates as more information becomes available.

The issue has been identified and a fix is being implemented.

We see recurring latency spikes affecting our S3 service in eu-central-1. The Object Storage and Network teams are jointly investigating. Although a technical root cause remains undetermined at this time, our highest priority is implementing measures to mitigate the frequency and amplitude of the spikes. We appreciate your patience and will keep you informed.

Our engineering teams have identified a path to remediation. Initial measures have been applied, improving the situation for some services. Customers using Object Storage in eu-central-1 may continue to experience elevated latency, which may vary in severity. Work continues on a comprehensive fix. We will provide further updates

Response times for Object Storage in eu-central-1 have improved significantly and continue to stabilize. We are closely monitoring system performance. We will provide further updates as remediation progresses.

The increased latency affecting S3 Object Storage in eu-central-1 has been resolved. Response times have returned to normal levels. We will continue to monitor the service.

# Root Cause Analysis ## What happened? Starting approximately 19:00 UTC on 24 August 2026, customers accessing S3 Object Storage in the Frankfurt \(FRA4\) data center experienced elevated latency across all operations - uploads, downloads, metadata requests, and deletions. Intermittent HTTP 503 Service Unavailable and 404 Not Found errors were observed on object read requests. The impact was measurable for all customer which had buckets in the both affected datacenters of the region, with some customers experiencing severe degradation depending on their bucket configuration and access patterns. ‌ The incident remained in progress with high priority from 25 August through 3 September 2026. The latency issue was mitigated at approximately 13:40 UTC on 3 September 2026. ## How was this possible? \(Root Cause\) **Primary cause - Software bug in Quality of Service \(QoS\) subsystem** ‌ IONOS S3 Object Storage in the Frankfurt region is using a distributed object storage system. A QoS feature of this service  - implemented via a redis-qos service - applies rate limiting to S3 requests at the cluster level. A bug in the S3 service leads to not well distributed queries to the Redis-QOS service \(which is served by multiple server for high availability\) this caused the Redis-QOS process to reach and sustain 100% CPU utilization, progressively slowing down all S3 request processing at the cluster level. This affected every request passing through the affected nodes, irrespective of the operation type or the specific bucket accessed. ‌ This is an internal defect within the software used. The bug caused the S3 service to consume all available resources before load reached request levels that would normally trigger throttling, meaning the degradation occurred continuously rather than only under peak conditions. ‌ In close cooperation with the software vendor, we disabled the QoS rate-limiting function on 3 September, which fully resolved the latency issue. S3 is currently operating without some QoS features while a permanent fix is prepared by the vendor. ‌ ## What are we doing to prevent recurrence? **Already completed:** ‌ * QoS disabled in the S3 service - fully resolved the latency issue. \(DONE\) * Requesting permanent solution from the software vendor. \(INPROGRESS\) ‌ **Short-term - ETA: within 2 weeks:** ‌ * Permanent fix for the Cloudian QoS bug: IONOS Cloud is in active coordination with the vendor to obtain and deploy a fix for the redis-qos defect. Once the fix is validated, lost QoS features will be re-enabled. * Database partition monitoring: We are implementing monitoring that alerts on partition size growth before any individual partition approaches a problematic threshold. This will allow our team to identify and address bucket layout issues proactively. ‌ **Mid-term - ETA: 1 to 3 months:** ‌ * QoS architecture review: Following the permanent QoS fix, we will review the architectural isolation of the QoS service together with the vendor to ensure that a future resource contention event in the rate-limiting layer cannot propagate to the request path at the same scale. * Monitoring and alerting improvements: We are extending cluster-level monitoring to surface redis-qos CPU saturation and database compaction backlog as first-class incident signals, with automated escalation before customer-visible latency develops. ## Closing remarks An incident of this duration in a core infrastructure service is not acceptable. The high-latency period persisted for nine days, during which customer workloads depending on S3 in the Frankfurt region were degraded. Multiple optimisations and mitigation strategies were implemented during the course of the incident, but could only improve the situation for individual buckets and only to a certain extent. Detecting the underlying QoS bug and developing a mitigation required coordination with the vendor’s engineering team. ‌ While the latency issue is mitigated, we remain in close contact with the vendor. The engineering work to deliver a permanent QoS fix, reduce database partition pressure, and prevent recurrence is in progress. We are also working closely with our technology partner to understand delays in the analysis of the root cause of this incident. We will conduct a joint post mortem to identify areas where collaboration during incidents can be improved. ‌ We recognise the impact this incident caused to your operations. We believe that the listed measures will help us prevent similar error patterns and speed up analysis and recovery for software related issues in the future. ‌ We thank you for your patience during the incident.