Zonos experienced a major incident on January 3, 2025 affecting International Checkout, lasting 5h 44m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 13, 2025, 06:51 PM UTC
We are experiencing an issue with orders not being created despite payments being processed successfully.
- resolved Feb 13, 2025, 06:53 PM UTC
This incident has been resolved.
- postmortem Feb 13, 2025, 06:54 PM UTC
**What products were affected and what was the impact?** * Zonos Checkout Impact: * Major Outage **What timeframe did this issue occur?** | | **Date** | **Time** | | --- | --- | --- | | From: | Jan 3, 2025 | 01:30 PM MST | | To: | Jan 3, 2025 | 07:06 PM MST | ### How was the issue detected? Our team was notified that payments were being processed, but orders were not being created. ### What functionality was affected? * Ability to create orders * Order fulfillment ### What problems did this cause? * Customer frustration due to failed order creation despite successful payment processing. * Potential financial reconciliation issues for merchants and customers. ### What was the resolution of the problem and steps that are being taken for continued follow-up? * The root cause was identified as a backend change, which introduced a bug related to the separation of payment intent and checkout session. * A rollback was implemented to restore functionality, and error logging for Stripe intent failures has now been enabled. ### What mitigation solutions will we put in place to prevent this issue from occurring in the future? * Synthetic tests have been updated to run every 10 minutes to narrow down problematic PRs and detect failures earlier.