Zonos incident

2025-01-24 Issue loading Checkout

Minor Resolved View vendor source →

Zonos experienced a minor incident on January 24, 2025 affecting International Checkout, lasting 2d 19h. The incident has been resolved; the full update timeline is below.

Started
Jan 24, 2025, 07:00 PM UTC
Resolved
Jan 27, 2025, 02:45 PM UTC
Duration
2d 19h
Detected by Pingoru
Jan 24, 2025, 07:00 PM UTC

Affected components

International Checkout

Update timeline

  1. investigating Feb 13, 2025, 09:52 PM UTC

    We are currently investigating this issue.

  2. resolved Feb 13, 2025, 09:53 PM UTC

    This incident has been resolved.

  3. postmortem Feb 13, 2025, 09:56 PM UTC

    **What products were affected and what was the impact?** * Checkout \(Custom Integrations\) Impact: * Major outage for affected merchants **What timeframe did this issue occur?** | | **Date** | **Time** | | --- | --- | --- | | From: | Jan 24, 2025 | 12:00 PM MST | | To: | Jan 27, 2025 | 07:45 AM MST | **How was the issue detected?** Our team was notified that Checkout domains were not being allowed as expected, leading to an immediate investigation by the engineering team. **What functionality was affected?** Online store settings \(allowed store domain\) were not loading properly for all stores using custom integrations. This prevented affected stores from loading checkout. ‌ **What problems did this cause?** Stores relying on custom integrations could not load their checkout correctly. ‌ **What was the resolution of the problem and steps that are being taken for continued follow-up?** * A rollback to a previous version immediately resolved the issue. * A new synthetic test has been implemented to ensure store settings caching is store-specific and does not mix settings between stores. ‌ ‌ **What mitigation solutions will we put in place to prevent this issue from occurring in the future?** * **Improved Testing:** A new synthetic test has been introduced to verify caching behavior and prevent incorrect store settings from being cached. * **Staging Environment:** Setting up a dedicated test environment for custom integrations to catch similar issues earlier.