Emailable experienced a major incident on May 26, 2021 affecting Emailable API, lasting 46m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 26, 2021, 07:04 AM UTC
We're experiencing an elevated rate of API errors and are currently investigating this issue.
- resolved May 26, 2021, 07:05 AM UTC
This incident has been resolved.
- postmortem May 26, 2021, 07:06 AM UTC
## Summary At approximately 12:42 AM we detected an elevated number of API requests to our Single Verifier /v1/verify endpoint returning an Internal Error \(500 status code\). Upon investigation, we determined the root cause was due to a connection limit issue introduced by new code with one of our databases. At 1:29 AM we confirmed the connection issue was resolved and the error was no longer being returned. ## Cause Following a new code release we identified a bug that limited connections to one of our databases, mainly reducing the volume of requests we could process at our /v1/verify endpoint. ## Impact Initial requests made to our /v1/verify endpoint started returning the following response. `{ "message": ”Internal Error” }` Due to the way this impacted our system, retrying these requests would return a proper response.