WellSaid Labs incident
Increased latency & error rates for API customers
Resolved after 1 day, on March 2, 2024.
## Incident Summary Elevated latency and error rates impacting all API customers. The errors were a result of a single gateway pod serving traffic in a faulty state after it failed to initialize a system component properly. ## Impact Roughly 20% of API customer traffic experienced increased latency and/or error rates for a period of 24 hours. ## Root Cause The source of error rates and latency was pinned down to a single pod responsible for handling API customer traffic. This pod in particular failed to initialize a middleware component properly but continued to unsuccessfully serve traffic. Upon termination of the problematic pod, service was restored. ## Preventative Measures * Improve gateway initialization logic and handling of failure scenarios * Improve logging severity to ensure relevant errors trigger alerts accordingly * Adjust alerting policies around error rates and latency for gateway components
Update timeline
- resolved Mar 05, 2024, 05:12 PM UTC
Elevated latency and error rates impacting all API customers. Incident Start: 2024-03-02 13:48 PT Incident End: 2024-03-03 13:48 PT Incident Duration: 24hrs Impact: Roughly 20% of API Customer traffic experienced increased latency and/or error rates
- postmortem Mar 05, 2024, 05:19 PM UTC
## Incident Summary Elevated latency and error rates impacting all API customers. The errors were a result of a single gateway pod serving traffic in a faulty state after it failed to initialize a system component properly. ## Impact Roughly 20% of API customer traffic experienced increased latency and/or error rates for a period of 24 hours. ## Root Cause The source of error rates and latency was pinned down to a single pod responsible for handling API customer traffic. This pod in particular failed to initialize a middleware component properly but continued to unsuccessfully serve traffic. Upon termination of the problematic pod, service was restored. ## Preventative Measures * Improve gateway initialization logic and handling of failure scenarios * Improve logging severity to ensure relevant errors trigger alerts accordingly * Adjust alerting policies around error rates and latency for gateway components
Your product runs on more than WellSaid Labs
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services