Commerce Layer Outage History

Started Aug 03, 2026, 05:02 AM UTC· resolved Aug 03, 2026, 05:13 AM UTC

1. resolved Aug 03, 2026, 07:29 AM UTC

   During a routine automated update of the EU database, the procedure couldn't complete and the DB couldn't return to normal operations. Our monitoring systems alerted immediately our team, that could restart the procedure manually and completed it in a few minutes. Root cause analysis currently in progress.

2. postmortem Aug 04, 2026, 08:31 AM UTC

   # Summary On Aug 3rd, 2026, from \`5:02 AM UTC\` till \`5:13 AM UTC\`, a routine automated update process failed to complete, causing a major downtime of our EU Database cluster. Monitoring system alerted immediately our team, that started a manual override of the process, taking it to a successful end. # Leadup While our recurrent automated DB update process was running on the EU database it encountered an unexpected condition, that prevented the process to complete, leaving the cluster in an unconsistent state. # Fault Some database engine items were not in the expected condition for the update to complete correctly and that caused the update to fail. # Detection Moments after the initial failure of the automated process, our monitoring system alerted our infrastructure team. # Root Cause Logical replication feature was not in the proper state after some previous replication activity in the cluster. The current update process couldn't detect this situation in advance to avoid starting at all. # Mitigation and resolution A few minutes after the beginning of the issue, our operators were able to reset the inconsistent parameters and apply the manual update procedure. # Corrective and Preventative Measures We are introducing a more robust update process that, by design, will not be prone to similar issues and will consistently limit the overall risk of errors during to database updates.

Read the full incident report →

Started Nov 18, 2025, 12:20 PM UTC· resolved Nov 19, 2025, 08:34 AM UTC

1. monitoring Nov 18, 2025, 12:20 PM UTC

   The Dashboard was not accessible for a few minutes. This is due to a currently ongoing incident on a third party CDN (https://www.cloudflarestatus.com/incidents/8gmgl950y3h7). They are actively investigating and working on a fix. The dashboard looks accessible in this moment, but we cannot exclude other disservices until the issue is not resolved on their side. We'll keep the situation monitored.

2. monitoring Nov 18, 2025, 02:56 PM UTC

   The CDN provider has confirmed a fix has been implemented. The issue seems to be resolved, but we'll continue to monitor the situation before closing the incident.

3. resolved Nov 19, 2025, 08:34 AM UTC

   The issue has been resolved.

Read the full incident report →

Started May 23, 2025, 05:50 AM UTC· resolved May 24, 2025, 05:50 AM UTC

1. resolved May 26, 2025, 05:11 PM UTC

   After a scheduled deployment, some customers experienced mismatched data responses on their interactions with our services. After detecting a major issue, we initiated the roll-back procedure, reverting our services to the previous code version. Root cause analysis currently in progress.

2. postmortem May 26, 2025, 05:12 PM UTC

   # **Summary** On May 23rd, 2025, from \`5:50 AM UTC\` till \`7:03 AM UTC\`, after a scheduled deployment, some customers experienced mismatched data responses on their interactions with our services. Realizing that it was a major issue, we initiated the roll-back procedure, reverting our services to the previous code version. # **Leadup** At \`5:50 AM UTC\` we concluded a deployment that included an upgrade of a third party provider library. The deployment introduced errors on several sessions. # **Fault** Some sessions with tokens requested during the affected timeframe might have experienced issues with add to cart functionality and checkout. In particular, adding items to cart and placing orders was not always feasible. In addition to this, some placed orders were missing approved\_at and reference data. # **Detection** A few minutes after the release, one of our operators detected an anomaly on the internal metrics dashboard that was confirmed by our clients requests to the support team. # **Root Cause** The upgrade of a third party provider library that manages low level web request wrappers introduced problems in our APIs that have not been detected by our tests in preprod and qa environments. The upgrade was recommended to keep our services up to speed with the latest security patches and compatibility with other components. # **Mitigation and resolution** A few minutes after the release, once we realized the severity of the issue, we decided to roll-back to the previous version of the code in order to immediately restore normal operations and allow a deeper understanding of the root cause offline. In addition to the roll-back, a partial CDN cache purge was performed to reduce the TTL of the wrong data. # **Corrective and Preventative Measures** We are introducing additional automatic tests related to the involved component and other similar patterns.

Read the full incident report →