Europe API degradation

Update timeline

1. resolved May 27, 2026, 10:03 AM UTC

   On April 30, 2026 around 10:00 UTC, our Europe region experienced a degradation of our API services. Requests coming from customers were reaching our platform, but part of the backend responsible for search operations in Europe stopped responding correctly. Two of the five service instances were still accepting requests but were no longer answering in a timely way. As a result, requests remained “stuck” until they hit technical timeouts. This behavior created congestion on the upstream components that route and secure traffic (API gateway and web application firewall). As connections accumulated and were not released, the network resources became saturated and new requests started to fail or time out. Traffic progressively recovered after we restarted the affected components and the system drained the accumulated requests. The situation went back to normal at 10:30 UTC What was the impact? For customers using our services in Europe during the incident window: - Slower responses and timeouts for API calls, especially search-related operations. - Higher rate of failed or abandoned requests, with some calls never receiving a response within a reasonable time. - Degraded user experience in applications relying on our APIs (delays, retries, and occasional error messages). No data loss was observed; the issue was limited to service availability and latency.

2. postmortem May 27, 2026, 10:04 AM UTC

   ## What happened? On **April 30, 2026 around 10:00 UTC**, our Europe region experienced a degradation of our API services. Requests coming from customers were reaching our platform, but part of the backend responsible for search operations in Europe stopped responding correctly. Two of the five service instances were still accepting requests but were no longer answering in a timely way. As a result, requests remained “stuck” until they hit technical timeouts. This behavior created congestion on the upstream components that route and secure traffic \(API gateway and web application firewall\). As connections accumulated and were not released, the network resources became saturated and new requests started to **fail or time out**. Traffic progressively recovered after we restarted the affected components and the system drained the accumulated requests. The situation went back to normal at **10:30 UTC** ## What was the impact? For customers using our services in Europe during the incident window: * **Slower responses and timeouts** for API calls, especially search-related operations. * **Higher rate of failed or abandoned requests**, with some calls never receiving a response within a reasonable time. * **Degraded user experience** in applications relying on our APIs \(delays, retries, and occasional error messages\). * No data loss was observed; the issue was limited to **service availability and latency**. ## What went wrong, and why? The incident was caused by a combination of a local failure and insufficient protective mechanisms: * Two backend service instances continued to accept requests but **stopped processing them correctly**, keeping connections open until long technical timeouts expired. * These long‑lived connections caused **upstream congestion** in our API gateway and web application firewall, which rely on a shared pool of network resources. * As these resources became exhausted, **new requests could no longer be served reliably**, leading to timeouts and errors for customers. ## How did we respond? Once we detected abnormal error rates and latency in Europe, we: * Identified the two unhealthy search service instances and **restarted them** to immediately restore normal behavior. * Closely monitored API success rates, latency, and network resource usage to confirm that services were recovering and remaining stable. * Reviewed detailed logs and metrics across the Europe API gateway and search services to **reconstruct the sequence of events** and validate the technical root cause. ## How are we preventing this in the future? To prevent similar issues and reduce their impact if they occur, we are implementing the following improvements: * **Stricter timeouts and fail‑fast behavior** Shorten the maximum time a request can stay open in the backend and upstream components, so unhealthy instances are detected quickly and do not block resources for long periods. * **Improved health checks and automatic recovery** Strengthen the health probes on backend instances so that services that stop responding correctly are **automatically removed from traffic and restarted**, instead of remaining partially available. * **Enhanced monitoring and alerting** Add more precise alerts on **error codes, long‑running requests, and network resource usage** in Europe, so that similar patterns are detected earlier and resolved before they become visible to customers. These changes are designed to make our services more resilient and to ensure that, if a component becomes unhealthy, it is isolated quickly and does not impact the overall experience.