Zonos experienced a critical incident on September 19, 2023 affecting Shopify Duty Tax and Landed Cost API (GraphQL), lasting 28m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 19, 2023, 09:36 PM UTC
We are currently investigating this issue.
- identified Sep 19, 2023, 09:40 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Sep 19, 2023, 09:50 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Sep 19, 2023, 10:04 PM UTC
This incident has been resolved.
- postmortem Sep 21, 2023, 09:32 PM UTC
**What products were affected and what was the impact?** Our Landed Cost APIs \(GraphQL\) were impacted. Impact: MAJOR OUTAGE **What timeframe did this issue occur?** | | **Date** | **Time** | | --- | --- | --- | | From: | Sep 19, 2023 | 15:30 MST | | To: | Sep 19, 2023 | 15:44 MST | **How was the issue detected?** Error rates triggered alarm monitors. **What functionality was affected?** Landed cost quotes for some stores could not be returned. **What problems did this cause?** Some merchants were unable to retrieve landed cost quotes. **What was the resolution of the problem and steps that are being taken for continued follow-up?** We discovered a configuration error in a previous release. Due to caching, scheduled synthetic test failures in the development environment lagged behind the deployment to the production environment. We fixed the configuration error, released the fix, and confirmed issue resolution via manual and automated testing. **What mitigation solutions will we put in place to prevent this issue from occurring in the future?** We will expire caches more quickly in the development environment for synthetic tests to identify potential problems prior to deployment to the production environment.