Zonos Outage History

Started Sep 04, 2026, 11:56 PM UTC· resolved Sep 05, 2026, 12:56 AM UTC

1. investigating Sep 05, 2026, 12:08 AM UTC

   We are currently investigating this issue.

2. identified Sep 05, 2026, 12:09 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Sep 05, 2026, 12:21 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Sep 05, 2026, 12:56 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 04, 2026, 05:18 PM UTC· resolved Sep 04, 2026, 06:00 PM UTC

1. investigating Sep 04, 2026, 05:29 PM UTC

   We are currently investigating this issue.

2. identified Sep 04, 2026, 05:30 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Sep 04, 2026, 05:30 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Sep 04, 2026, 07:47 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 03, 2026, 09:00 PM UTC· resolved Aug 03, 2026, 10:36 PM UTC

1. investigating Aug 03, 2026, 09:23 PM UTC

   We are currently investigating this issue.

2. identified Aug 03, 2026, 09:24 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified Aug 03, 2026, 09:36 PM UTC

   We are continuing to work on a fix for this issue.

4. identified Aug 03, 2026, 09:57 PM UTC

   We are continuing to work on a fix for this issue.

5. monitoring Aug 03, 2026, 09:57 PM UTC

   A fix has been implemented and we are monitoring the results.

6. monitoring Aug 03, 2026, 10:36 PM UTC

   We are continuing to monitor for any further issues.

7. resolved Aug 03, 2026, 10:36 PM UTC

   This incident has been resolved.

8. postmortem Sep 09, 2026, 08:06 PM UTC

   ## **Incident Summary** On August 3, 2026, Zonos experienced elevated error rates affecting Landed Cost and related services. The incident began at approximately 2:52 PM MDT and was resolved at 3:51 PM MDT, resulting in approximately 59 minutes of degraded service. The incident was a partial outage: requests continued to be processed throughout the event, but a significant portion of requests relying on the affected service returned errors. ## **Impact** The primary affected service experienced an approximately 50% error rate for GraphQL requests during the incident window. As a result, customers using Landed Cost experienced intermittent request failures, while other requests continued to process successfully. Because Landed Cost relies on shared platform services, the underlying issue also resulted in elevated error rates across some related workflows. ## **Root Cause** The incident was triggered by a production deployment that began using a newly introduced data access path within our platform. Under production request volume, this access pattern generated significantly more database activity than anticipated. This exhausted available database connections within a supporting service and caused requests depending on that service to fail. The underlying issue had not manifested previously because the affected functionality had not yet been exercised at production scale. ## **Resolution** Our engineering team initially rolled back a related service deployment while investigating the source of the errors. When this did not resolve the issue, the Landed Cost deployment that activated the problematic access pattern was rolled back. Service recovered immediately following that rollback. A permanent correction was subsequently deployed to optimize the database access pattern and route the affected workload appropriately. ## **Preventive Actions** Following the incident, we reviewed both the immediate failure and how it propagated through dependent services. We are taking several actions to reduce the likelihood and potential impact of similar incidents, including: * Adding additional monitoring and alerting for database connection saturation. * Strengthening safeguards and review practices for database queries against partitioned data. * Expanding testing of newly activated data-access patterns under production-like load. * Reviewing service isolation and failure handling to reduce the impact a degraded dependency can have on other platform services. * Reviewing similar data-access patterns across the platform for the conditions identified during this incident. We apologize for the disruption this incident caused. We will continue using the findings from this event to improve the resilience and observability of the Zonos platform.

Read the full incident report →

Started Jul 08, 2026, 07:15 PM UTC· resolved Jul 08, 2026, 09:38 PM UTC

1. investigating Jul 08, 2026, 07:41 PM UTC

   We are currently investigating this issue.

2. identified Jul 08, 2026, 07:42 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Jul 08, 2026, 07:42 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jul 08, 2026, 09:00 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 26, 2026, 09:27 PM UTC· resolved May 26, 2026, 10:31 PM UTC

1. investigating May 26, 2026, 09:27 PM UTC

   An issue with an authentication provider is currently affecting some users.

2. identified May 26, 2026, 09:29 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring May 26, 2026, 09:41 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring May 26, 2026, 09:54 PM UTC

   We are continuing to see increased error rates and are monitoring for a fix from our authentication provider.

5. monitoring May 26, 2026, 09:59 PM UTC

   We are seen error rates decline and will continue monitoring.

6. resolved May 26, 2026, 10:31 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 11, 2026, 05:42 PM UTC· resolved Mar 11, 2026, 06:06 PM UTC

1. investigating Mar 11, 2026, 09:19 PM UTC

   We are currently investigating this issue.

2. identified Mar 11, 2026, 09:19 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Mar 11, 2026, 09:20 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Mar 11, 2026, 09:21 PM UTC

   This incident has been resolved.

5. postmortem Mar 11, 2026, 09:25 PM UTC

   **What products were affected and what was the impact?** Classify, Country of Origin Impact: DEGRADED PERFORMANCE **What timeframe did this issue occur?** | **Date** | **Time** | | --- | --- | | Mar 11, 2026 | 2026-03-11 10:12 - 2026-03-11 12:06 | **How was the issue detected?** Zonos observed elevated error rates affecting Classify and Country of Origin APIs. ‌ **What functionality was affected?** A portion of Classify and Country of Origin inference requests did not return a valid response. ‌ **What was the root cause?** A firmware issue on a subset of GPU nodes caused network connectivity failures. This caused the nodes to enter an unhealthy state. Our reconnect logic did not properly remove unhealthy nodes from the pool, so they continued to receive requests. ‌ **What was the resolution of the problem, and what steps are being taken to prevent future issues?** Shortly after discovering the issue, we restarted nodes to resolve the connectivity issues and restore service. We then identified the reconnect logic issue, and released a fix to prevent future issues. We have scheduled a firmware upgrade on all nodes to resolve the underlying network issue. We have also refined our alerting to catch errors more quickly in the future.

Read the full incident report →

Started Feb 22, 2026, 06:25 PM UTC· resolved Feb 23, 2026, 04:22 PM UTC

1. investigating Feb 22, 2026, 06:25 PM UTC

   We are currently investigating this issue.

2. investigating Feb 22, 2026, 06:28 PM UTC

   We are continuing to investigate this issue.

3. identified Feb 22, 2026, 07:16 PM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring Feb 23, 2026, 03:45 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Feb 23, 2026, 04:22 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 19, 2026, 09:32 PM UTC· resolved Feb 19, 2026, 10:24 PM UTC

1. investigating Feb 19, 2026, 09:58 PM UTC

   We are currently investigating this issue.

2. identified Feb 19, 2026, 09:59 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Feb 19, 2026, 09:59 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Feb 19, 2026, 10:24 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 19, 2026, 04:29 PM UTC· resolved Feb 19, 2026, 06:57 PM UTC

1. investigating Feb 19, 2026, 04:29 PM UTC

   We are currently investigating this issue.

2. identified Feb 19, 2026, 04:32 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Feb 19, 2026, 04:58 PM UTC

   A fix has been implemented and we are monitoring the results.

4. monitoring Feb 19, 2026, 05:00 PM UTC

   We are continuing to see some issues and are continuing to monitor.

5. monitoring Feb 19, 2026, 05:26 PM UTC

   We are continuing to monitor for any further issues.

6. monitoring Feb 19, 2026, 05:48 PM UTC

   The issue has been resolved and operations have returned to normal. We are actively monitoring to ensure continued stability.

7. resolved Feb 19, 2026, 06:57 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 20, 2025, 08:01 AM UTC· resolved Oct 20, 2025, 08:52 AM UTC

1. investigating Oct 20, 2025, 09:57 PM UTC

   We are currently investigating this issue.

2. identified Oct 20, 2025, 09:57 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Oct 20, 2025, 09:57 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Oct 20, 2025, 09:57 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 04, 2025, 07:59 PM UTC· resolved Sep 04, 2025, 08:17 PM UTC

1. investigating Sep 04, 2025, 07:59 PM UTC

   We are currently investigating this issue.

2. resolved Sep 04, 2025, 08:17 PM UTC

   Users logging in may have experienced a flashing screen. The change was rolled back and the issue has been resolved.

Read the full incident report →

Started Jun 10, 2025, 02:42 PM UTC· resolved Jun 10, 2025, 04:00 PM UTC

1. investigating Jun 10, 2025, 02:54 PM UTC

   An issue with an upstream infrastructure provider is impacting our BigCommerce Plugin

2. identified Jun 10, 2025, 02:54 PM UTC

   The issue has been identified and we are continuing to monitor the impact.

3. identified Jun 10, 2025, 02:55 PM UTC

   We are continuing to work on a fix for this issue.

4. resolved Jun 10, 2025, 04:00 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 24, 2025, 07:00 PM UTC· resolved Jan 27, 2025, 02:45 PM UTC

1. investigating Feb 13, 2025, 09:52 PM UTC

   We are currently investigating this issue.

2. resolved Feb 13, 2025, 09:53 PM UTC

   This incident has been resolved.

3. postmortem Feb 13, 2025, 09:56 PM UTC

   **What products were affected and what was the impact?** * Checkout \(Custom Integrations\) Impact: * Major outage for affected merchants **What timeframe did this issue occur?** | | **Date** | **Time** | | --- | --- | --- | | From: | Jan 24, 2025 | 12:00 PM MST | | To: | Jan 27, 2025 | 07:45 AM MST | **How was the issue detected?** Our team was notified that Checkout domains were not being allowed as expected, leading to an immediate investigation by the engineering team. **What functionality was affected?** Online store settings \(allowed store domain\) were not loading properly for all stores using custom integrations. This prevented affected stores from loading checkout. ‌ **What problems did this cause?** Stores relying on custom integrations could not load their checkout correctly. ‌ **What was the resolution of the problem and steps that are being taken for continued follow-up?** * A rollback to a previous version immediately resolved the issue. * A new synthetic test has been implemented to ensure store settings caching is store-specific and does not mix settings between stores. ‌ ‌ **What mitigation solutions will we put in place to prevent this issue from occurring in the future?** * **Improved Testing:** A new synthetic test has been introduced to verify caching behavior and prevent incorrect store settings from being cached. * **Staging Environment:** Setting up a dedicated test environment for custom integrations to catch similar issues earlier.

Read the full incident report →

Started Jan 24, 2025, 12:30 AM UTC· resolved Jan 24, 2025, 10:30 PM UTC

1. investigating Feb 13, 2025, 08:52 PM UTC

   We are currently investigating this issue.

2. resolved Feb 13, 2025, 08:53 PM UTC

   This incident has been resolved.

3. postmortem Feb 13, 2025, 09:27 PM UTC

   ### What products were affected and what was the impact? Shopify Checkout Partial outage: * Affected some Shopify merchants ### What timeframe did this issue occur? | | **Date** | **Time** | | --- | --- | --- | | From: | Jan 23, 2025 | 05:30 PM MDT | | To: | Jan 24, 2025 | 03:30 PM MDT | ### How was the issue detected? * A discrepancy was identified when some orders in Shopify were missing tracking numbers. * Additionally, merchants using older versions of the Shopify app experienced order import failures. ‌ ### What functionality was affected? * Order fulfillment ‌ ### What problems did this cause? * **Order Processing Issues:** Orders processed without tracking numbers caused confusion and delays for merchants and customers. * **Order Import Failures:** Some merchants experienced missing orders due to import failures. ‌ ### What was the resolution of the problem and steps that are being taken for continued follow-up? * We patched the root cause of the issue, resolving the tracking number and order import failures. * We updated all the affected orders with tracking numbers and manually imported orders that had failed to import. * API headers and logging enhancements were implemented to improve issue detection and response time. ### What mitigation solutions will we put in place to prevent this issue from occurring in the future? * Expand QA coverage to address downstream impacts of refactors, with a focus on comprehensive testing of critical flows, including label creation. * Expand synthetic test suite to validate end-to-end order processing, with particular emphasis on fulfillment workflows. * Establish a consistent testing environment that simulates real-world Shopify store configurations, reducing the risk of silent failures.

Read the full incident report →

Started Jan 06, 2025, 09:30 AM UTC· resolved Jan 07, 2025, 04:00 PM UTC

1. investigating Feb 13, 2025, 08:30 PM UTC

   We are currently investigating this issue.

2. resolved Feb 13, 2025, 08:31 PM UTC

   This incident has been resolved.

3. postmortem Feb 13, 2025, 08:38 PM UTC

   **What products were affected and what was the impact?** * All Magento Checkout merchants Impact: * Partial Outage **What timeframe did this issue occur?** | | **Date** | **Time** | | --- | --- | --- | | From: | Jan 6, 2025 | 2:27 PM MST | | To: | Jan 7, 2025 | 09:11 AM MST | ‌ ### How was the issue detected? Our team was notified that carts were not being created for some Magento shoppers, leading to an immediate investigation by the engineering team. ‌ ### What functionality was affected? The checkout process for Magento merchants encountered validation failures due to incorrect UUID handling. This disrupted the checkout flow, impacting merchants using the Magento Checkout. ‌ ### What problems did this cause? * Magento merchants experienced disruptions in checkouts being loaded at times. ‌ ### What was the resolution of the problem and steps that are being taken for continued follow-up? * The issue was identified as stemming from changes made to TSID validation. * A rollback of the problematic changes was performed promptly, restoring checkout functionality. * A synthetic test was created to validate UUID behavior and detect future occurrences of this issue. * Code annotations have been added to document known issues and improve future debugging. ‌ ### What mitigation solutions will we put in place to prevent this issue from occurring in the future? * Establishing alerts detecting deviations in carts-to-checkout sessions. This will help flag potential checkout issues early. * Increasing test coverage to identify gaps in synthetic testing.

Read the full incident report →

Started Jan 03, 2025, 08:30 AM UTC· resolved Jan 03, 2025, 02:15 PM UTC

1. investigating Feb 13, 2025, 06:51 PM UTC

   We are experiencing an issue with orders not being created despite payments being processed successfully.

2. resolved Feb 13, 2025, 06:53 PM UTC

   This incident has been resolved.

3. postmortem Feb 13, 2025, 06:54 PM UTC

   **What products were affected and what was the impact?** * Zonos Checkout Impact: * Major Outage **What timeframe did this issue occur?** | | **Date** | **Time** | | --- | --- | --- | | From: | Jan 3, 2025 | 01:30 PM MST | | To: | Jan 3, 2025 | 07:06 PM MST | ‌ ### How was the issue detected? Our team was notified that payments were being processed, but orders were not being created. ‌ ### What functionality was affected? * Ability to create orders * Order fulfillment ‌ ### What problems did this cause? * Customer frustration due to failed order creation despite successful payment processing. * Potential financial reconciliation issues for merchants and customers. ‌ ### What was the resolution of the problem and steps that are being taken for continued follow-up? * The root cause was identified as a backend change, which introduced a bug related to the separation of payment intent and checkout session. * A rollback was implemented to restore functionality, and error logging for Stripe intent failures has now been enabled. ‌ ### What mitigation solutions will we put in place to prevent this issue from occurring in the future? * Synthetic tests have been updated to run every 10 minutes to narrow down problematic PRs and detect failures earlier.

Read the full incident report →

Started Nov 12, 2024, 08:27 PM UTC· resolved Nov 12, 2024, 10:20 PM UTC

1. investigating Nov 12, 2024, 08:51 PM UTC

   We are currently investigating elevated error rates.

2. identified Nov 12, 2024, 09:00 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified Nov 12, 2024, 09:39 PM UTC

   We are continuing to work on a fix for this issue.

4. monitoring Nov 12, 2024, 09:55 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Nov 12, 2024, 10:20 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 08, 2024, 04:00 PM UTC· resolved Nov 08, 2024, 05:26 PM UTC

1. investigating Nov 08, 2024, 07:49 PM UTC

   We are currently investigating this issue.

2. identified Nov 08, 2024, 07:49 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Nov 08, 2024, 07:50 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Nov 08, 2024, 07:52 PM UTC

   This incident has been resolved.

5. postmortem Nov 08, 2024, 07:53 PM UTC

   ### What products were affected and what was the impact? Landed Cost API, Checkout Impact: CRITICAL ### What timeframe did this issue occur? | **Date** | **Time** | | --- | --- | | November 8, 2024 | 8:50am - 10:26am MST | ### How was the issue detected? A spike in error logs triggered an alert to our Engineering team, who responded immediately to the issue. ### What functionality was affected? Landed Cost quotes that use our automated item classification service failed. ### What problems did this cause? If an HS Code was not provided in the API request to Landed Cost, and the automatic classification service was enabled, then the landed cost quote would fail. When the landed cost quote fails, shoppers may not be able to place their order. ### What was the resolution of the problem and steps that are being taken for continued follow-up? The root cause of the issue was a deployment issue with the item service used for automatic classification. While the issue was detected immediately, resolution required rebuilding and redeploying services, which took longer than expected. After services were rebuilt and redeployed, the system health was validated and normal operations resumed. ### What mitigation solutions will we put in place to prevent this issue from occurring in the future? We discovered that this issue was due, in part, to a deficiency in our deployment procedures. We are working to update the procedure to prevent any future issues. We are also creating a synthetic test in our lower environments that will catch similar issues before they can be deployed into production.

Read the full incident report →