IONOS Cloud incident

AI Model Hub Service Degradation and Increased Error Rate

Critical Resolved View vendor source →

IONOS Cloud experienced a critical incident on November 26, 2025 affecting AI Model Hub, lasting 1h 6m. The incident has been resolved; the full update timeline is below.

Started
Nov 26, 2025, 12:40 PM UTC
Resolved
Nov 26, 2025, 01:46 PM UTC
Duration
1h 6m
Detected by Pingoru
Nov 26, 2025, 12:40 PM UTC

Affected components

AI Model Hub

Update timeline

  1. investigating Nov 26, 2025, 12:40 PM UTC

    We are currently investigating increased error rate in our AI Model Hub.

  2. investigating Nov 26, 2025, 12:40 PM UTC

    We are continuing to investigate this issue.

  3. investigating Nov 26, 2025, 12:43 PM UTC

    We are updating the impact to outage. The AI Model Hub Team is currently investigating and is working to restore the service as quickly as possible. We will post the next update here 13:15 UTC latest.

  4. identified Nov 26, 2025, 01:15 PM UTC

    We have identified a connectivity issue preventing communication to the inferencing backends. The Infrastructure Team is currently working on a fix. We will provide the next update by 13:45 UTC.

  5. monitoring Nov 26, 2025, 01:28 PM UTC

    The Infrastructure Team has deployed the fix, and we are seeing improvements in availability. We are monitoring the situation until all metrics return to expected baselines.

  6. resolved Nov 26, 2025, 01:46 PM UTC

    All metrics have returned to baseline. We are marking this incident as resolved. We will publish a Root Cause Analysis here shortly.

  7. postmortem Nov 26, 2025, 01:46 PM UTC

    **What happened:** On November 26, 2025, at 12:19 UTC, we observed a decline in usage-related metrics in our AI Model Hub. This was a result of connectivity issues in the secure connection to our backend inference engines. **What we did:** After resolving the configuration issue the environment recovered. **What we are doing to avoid recurrence:** * To spot a decline in usage metrics more quickly, we have updated our alerting thresholds. * Automatic health checks will be expanded.