Servd experienced a major incident on October 11, 2026, lasting 11h 55m. The incident has been resolved; the full update timeline is below.
Update timeline
- identified Oct 11, 2026, 07:17 PM UTC
A degradation of network connectivity in two data centres is causing timeouts and other connection errors with traffic between project components and out to the public internet. We're liaising with our server provider to identify and rectify the root cause.
- resolved Oct 12, 2026, 07:13 AM UTC
The incident has now been resolved. Two individual problems were discovered and mitigated: - DNS lookup failures in our London based clusters. This was caused by a configuration error with an upstream DNS resolver causing it to reject any dnssec-enabled queries. - Packet loss in us-east-3/4/5 clusters. This was caused by a hardware issue with a hypervisor which caused degraded networking performance with a subset of worker nodes. The fault has been resolved.