Short.io incident
API degraded performance
Resolved after 1 hour 45 minutes, on November 9, 2026.
This incident has been resolved. *Root Cause Analysis:* **Summary** At 18:55 UTC, Limestone Networks, one of our infrastructure providers, lost network connectivity. Short.io runs across several data centers and is designed to keep working when one provider fails. This time it did not: the API was degraded for about 1.5 hours while we restored service manually. **What went wrong** Our customer-facing services were spread across providers, but several internal components they depend on were not. They ran in a single, non-redundant configuration, and some of them were in the data center that went offline: - **Kyverno (cluster policy engine):** Kubernetes asks Kyverno to approve every new pod. With Kyverno down, Kubernetes could not start replacement pods in the healthy data centers, so the cluster could not fail over on its own. - **CoreDNS (internal service discovery):** DNS kept sending internal traffic to pods in the failed data center, so requests that reached healthy infrastructure were still forwarded to backends that could not answer. - **Prometheus (monitoring) and log writer:** Both ran with the same provider and went down with it. We knew about the outage right away, but without metrics and logs we had little visibility into what was failing inside the cluster, which slowed our recovery. Each of these components caused a problem on its own. Together, they meant that losing one provider took down a system that was built to survive it. **What we are doing about it** - We are making sure every part of our system, including internal components like policy enforcement, DNS, monitoring and logging, runs in 3 different data centers at the same time, so losing any one data center does not affect service. - We are reviewing how cluster components respond when a dependency is unavailable, so that one failed internal service cannot stop pods from being rescheduled. - We are making sure our metrics and logs stay available during a data center failure, so we have full visibility while we recover. We apologize for the disruption. We understand how much our customers depend on Short.io, and we are treating these fixes as our top priority.
Update timeline
- investigating Nov 09, 2026, 06:55 PM UTC
We are currently investigating this issue.
- identified Nov 09, 2026, 07:30 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Nov 09, 2026, 08:05 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Nov 09, 2026, 08:40 PM UTC
This incident has been resolved. *Root Cause Analysis:* **Summary** At 18:55 UTC, Limestone Networks, one of our infrastructure providers, lost network connectivity. Short.io runs across several data centers and is designed to keep working when one provider fails. This time it did not: the API was degraded for about 1.5 hours while we restored service manually. **What went wrong** Our customer-facing services were spread across providers, but several internal components they depend on were not. They ran in a single, non-redundant configuration, and some of them were in the data center that went offline: - **Kyverno (cluster policy engine):** Kubernetes asks Kyverno to approve every new pod. With Kyverno down, Kubernetes could not start replacement pods in the healthy data centers, so the cluster could not fail over on its own. - **CoreDNS (internal service discovery):** DNS kept sending internal traffic to pods in the failed data center, so requests that reached healthy infrastructure were still forwarded to backends that could not answer. - **Prometheus (monitoring) and log writer:** Both ran with the same provider and went down with it. We knew about the outage right away, but without metrics and logs we had little visibility into what was failing inside the cluster, which slowed our recovery. Each of these components caused a problem on its own. Together, they meant that losing one provider took down a system that was built to survive it. **What we are doing about it** - We are making sure every part of our system, including internal components like policy enforcement, DNS, monitoring and logging, runs in 3 different data centers at the same time, so losing any one data center does not affect service. - We are reviewing how cluster components respond when a dependency is unavailable, so that one failed internal service cannot stop pods from being rescheduled. - We are making sure our metrics and logs stay available during a data center failure, so we have full visibility while we recover. We apologize for the disruption. We understand how much our customers depend on Short.io, and we are treating these fixes as our top priority.
Your product runs on more than Short.io
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services