Ordergroove Outage History

This morning, starting at approximately 5:00 AM CST, we experienced slower than normal order processing rates. The issue was addressed and fully resolved by approximately 12:30 PM CST. Orders are now processing at the normal rate and the system is working through the backlog.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor this issue as the analytics page is still experiencing some instability. There is no data loss in the process since collector is working properly.

We have identified and mitigated the effects of the latency. We are monitoring the performance of the system.

We have identified a secondary cause of latency and addressed the issue. We are monitoring the system and will post an update shortly.

On December 5th, 3:45 AM EST Shopify GraphQL APIs experienced an elevated error rate. The issue was resolved by 4:15 AM EST. During the duration of this issue recurring subscription orders failed to place and went into Rejected status. The root cause of this outage within Shopify was due to a CloudFlare issue which impacted a lot of other service providers that rely on this service. The steps for remediation is to safely retry these rejected orders from our side. We are continuing to run this recovery efforts on all impacted orders and will provide another update as soon as all recurring orders have been retried by our system.

We have identified the issue, applied mitigating steps, and are monitoring

We are continuing to monitor for any further issues.

On November 1st, there was an update to the data being sent over from Shopify which negatively impacted our ability to mark products as in stock in our system. The issue only affects products where inventory is not tracked within Shopify. We have been working since then to implement a short-term solutions and have been rolling it out gradually to all merchants who we found to have been impacted by this change. This short-term fix requires our platform to pull additional product details from Shopify so we have been very gradually rolling it out and measuring system performance. We are going to provide another update as soon as this short-term fix has been rolled out to all stores.

Order Reminder Events 9/15/2025: Yesterday, on 9/15/2025, Order Reminder events were not sent to Klaviyo or delivered through Ordergroove Webhook system What happened We upgraded our queueing system for the application that produces Order Reminder events. During the upgrade, a subset of consumers did not update correctly and were listening to the wrong system, which caused events to be missed. What we’re doing We are carefully replaying all missed Order Reminder events today. To avoid outdated actions, reminders will only be sent if the related order has not already been attempted. We expect the remediation will be fully completed today Next steps Once remediation is complete, we will run an internal post mortem to strengthen monitoring and ensure full coverage of the process going forward.

We have identified an issue related to internal systems that process Shopify events from 03/06/2025 4PM EST and until 03/07/2025 9AM EST. The underlying issue has been resolved and we are actively investigating all impacted areas that require manual intervention. No action needs to be taken, our team will make sure all events will be reprocessed. Areas impacted: - Increase in recurring order placement retry attempts due to connection errors - Delayed delivery of events and updates tied to certain customer actions such as change of frequency or payment updates Next Steps: - We are working diligently to ensure that all delayed events are being processed - We are reviewing all potentially impacted recurring orders to ensure proper placement We will provide further updates throughout the day

All of the services have recovered and message delivery is back to normal. We are in the process of replaying any missed transactions. No action needs to be taken, our team will make sure all events will be reprocessed. We will provide further updates as progress is made.

All valid Shopify events have been replayed successfully. We are now entering the next and final phase of the recovery effort. There is a small number of recurring orders that were impacted by this issue which will be retried over the next 24-48 hour period. We are taking extra precautionary measures to ensure that we only retry orders which were not properly processed by Shopify. No action needs to be taken and we will provide a final update once the task is completed by our team.