Zonos incident

2023-08-15 Issue with International Checkout

Minor Resolved View vendor source →

Zonos experienced a minor incident on August 15, 2023 affecting International Checkout, lasting 1h 15m. The incident has been resolved; the full update timeline is below.

Started
Aug 15, 2023, 05:20 PM UTC
Resolved
Aug 15, 2023, 06:36 PM UTC
Duration
1h 15m
Detected by Pingoru
Aug 15, 2023, 05:20 PM UTC

Affected components

International Checkout

Update timeline

  1. investigating Aug 16, 2023, 04:42 AM UTC

    We are currently investigating this issue.

  2. identified Aug 16, 2023, 04:42 AM UTC

    The issue has been identified and a fix is being implemented.

  3. monitoring Aug 16, 2023, 04:43 AM UTC

    A fix has been implemented and we are monitoring the results.

  4. resolved Aug 16, 2023, 04:43 AM UTC

    This incident has been resolved.

  5. postmortem Aug 16, 2023, 04:52 AM UTC

    **What products were affected and what was the impact?** International Checkout Impact: PARTIAL OUTAGE **What timeframe did this issue occur?** | | **Date** | **Time** | | --- | --- | --- | | From: | August 15, 2023 | 11:20 MST | | To: | August 15, 2023 | 11:31 MST | **How was the issue detected?** Synthetic tests and elevated error rates triggered alerts. **What functionality was affected?** About 40% of shipment rating requests from Checkout failed. ‌ **What problems did this cause?** Our internal shipment rating service saw failures that caused some checkouts to be slow, and others to fail. We also saw minimal failures in our catalog service. ‌ **What was the resolution of the problem and steps that are being taken for continued follow-up?** We immediately reverted the change to our catalog service that caused the issue. We are investigating how to fix the issue for the next release. **What mitigation solutions will we put in place to prevent this issue from occurring in the future?** We are exploring opportunities to add more caching to the catalog service, in addition to making architecture and infrastructure improvements.