Zonos experienced a minor incident on January 10, 2024, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Jan 10, 2024, 11:05 PM UTC
An issue with an upstream third-party API caused timeouts and exhausted database connections. This caused slow responses and some failures of Landed Cost quotes until the service cycled and database connections became available.
- postmortem Feb 13, 2024, 06:14 PM UTC
**What products were affected and what was the impact?** * Landed Cost Impact: * DEGRADED PERFORMANCE **What timeframe did this issue occur?** | | **Date** | **Time** | | --- | --- | --- | | From: | Jan 10, 2024 | 07:32 MST | | To: | Jan 10, 2024 | 07:54 MST | **How was the issue detected?** We saw an error rate spike within one of our APIs indicating there was an issue with an upstream 3rd party service. **What functionality was affected?** We were unable to generate some landed cost quotes. **What problems did this cause?** The issue caused requests to timeout in one of our APIs. These timed-out requests exhausted the database connection pool until the service was cycled and database connections became available. **What was the resolution of the problem and steps that are being taken for continued follow-up?** We cycled our affected service, releasing database connections. We’re looking at ways to mitigate dependency on this particular 3rd party API to ensure smooth operation of our APIs in the event of a future outage. **What mitigation solutions will we put in place to prevent this issue from occurring in the future?** More resilient handling for upstream 500 errors from 3rd party APIs.