Zonos incident

2025-01-24 - Shopify tracking number and order import issue

Minor Resolved View vendor source →

Zonos experienced a minor incident on January 24, 2025 affecting Shopify Checkout, lasting 21h 59m. The incident has been resolved; the full update timeline is below.

Started
Jan 24, 2025, 12:30 AM UTC
Resolved
Jan 24, 2025, 10:30 PM UTC
Duration
21h 59m
Detected by Pingoru
Jan 24, 2025, 12:30 AM UTC

Affected components

Shopify Checkout

Update timeline

  1. investigating Feb 13, 2025, 08:52 PM UTC

    We are currently investigating this issue.

  2. resolved Feb 13, 2025, 08:53 PM UTC

    This incident has been resolved.

  3. postmortem Feb 13, 2025, 09:27 PM UTC

    ### What products were affected and what was the impact? Shopify Checkout Partial outage: * Affected some Shopify merchants ### What timeframe did this issue occur? | | **Date** | **Time** | | --- | --- | --- | | From: | Jan 23, 2025 | 05:30 PM MDT | | To: | Jan 24, 2025 | 03:30 PM MDT | ### How was the issue detected? * A discrepancy was identified when some orders in Shopify were missing tracking numbers. * Additionally, merchants using older versions of the Shopify app experienced order import failures. ‌ ### What functionality was affected? * Order fulfillment ‌ ### What problems did this cause? * **Order Processing Issues:** Orders processed without tracking numbers caused confusion and delays for merchants and customers. * **Order Import Failures:** Some merchants experienced missing orders due to import failures. ‌ ### What was the resolution of the problem and steps that are being taken for continued follow-up? * We patched the root cause of the issue, resolving the tracking number and order import failures. * We updated all the affected orders with tracking numbers and manually imported orders that had failed to import. * API headers and logging enhancements were implemented to improve issue detection and response time. ### What mitigation solutions will we put in place to prevent this issue from occurring in the future? * Expand QA coverage to address downstream impacts of refactors, with a focus on comprehensive testing of critical flows, including label creation. * Expand synthetic test suite to validate end-to-end order processing, with particular emphasis on fulfillment workflows. * Establish a consistent testing environment that simulates real-world Shopify store configurations, reducing the risk of silent failures.