Zonos incident

2025-01-06 Magento Checkout Issue

Minor Resolved View vendor source →

Zonos experienced a minor incident on January 6, 2025 affecting International Checkout, lasting 1d 6h. The incident has been resolved; the full update timeline is below.

Started
Jan 06, 2025, 09:30 AM UTC
Resolved
Jan 07, 2025, 04:00 PM UTC
Duration
1d 6h
Detected by Pingoru
Jan 06, 2025, 09:30 AM UTC

Affected components

International Checkout

Update timeline

  1. investigating Feb 13, 2025, 08:30 PM UTC

    We are currently investigating this issue.

  2. resolved Feb 13, 2025, 08:31 PM UTC

    This incident has been resolved.

  3. postmortem Feb 13, 2025, 08:38 PM UTC

    **What products were affected and what was the impact?** * All Magento Checkout merchants Impact: * Partial Outage **What timeframe did this issue occur?** | | **Date** | **Time** | | --- | --- | --- | | From: | Jan 6, 2025 | 2:27 PM MST | | To: | Jan 7, 2025 | 09:11 AM MST | ‌ ### How was the issue detected? Our team was notified that carts were not being created for some Magento shoppers, leading to an immediate investigation by the engineering team. ‌ ### What functionality was affected? The checkout process for Magento merchants encountered validation failures due to incorrect UUID handling. This disrupted the checkout flow, impacting merchants using the Magento Checkout. ‌ ### What problems did this cause? * Magento merchants experienced disruptions in checkouts being loaded at times. ‌ ### What was the resolution of the problem and steps that are being taken for continued follow-up? * The issue was identified as stemming from changes made to TSID validation. * A rollback of the problematic changes was performed promptly, restoring checkout functionality. * A synthetic test was created to validate UUID behavior and detect future occurrences of this issue. * Code annotations have been added to document known issues and improve future debugging. ‌ ### What mitigation solutions will we put in place to prevent this issue from occurring in the future? * Establishing alerts detecting deviations in carts-to-checkout sessions. This will help flag potential checkout issues early. * Increasing test coverage to identify gaps in synthetic testing.