Ordergroove had 16 outages in the last 2 years totaling 132h 59m of downtime — averaging 0.7 incidents per month.
There were 16 Ordergroove outages since July 12, 2024 totaling 132h 59m of downtime. Each is summarised below — incident details, duration, and resolution information.
We're investigating reports of the RC3 analytics page experiencing issues while loading metrics. The issue doesn't affect metrics collection. Reporting has not been affected.
The incident is still under investigation.
The Incident has been resolved.
We are currently experiencing a delay in our data pipelines. As a result, Analytics data is not updating at the usual 4 hour latency. At this time, the latest available data reflects activity up to 8:50 AM EST on April 15. Our team is actively working to resolve the issue, and we expect data freshness to return to normal within the next few hours. We will provide another update as soon as more information is available.
The issue causing delays in our data pipelines has been fully resolved. Analytics data is now processing as expected, and users should see data updating within the normal up to 4 hour latency window. We will continue to monitor the system to ensure stability, but no further impact is expected. Thank you for your patience.
We are currently experiencing delays in webhook event delivery. Our team is actively investigating the issue and working to identify the root cause. No events are expected to be lost, but delivery may be delayed. We will provide an update as soon as we have more information or once mitigation steps are in place. Thank you for your patience.
Webhook delivery delays have been fully resolved, and all queues have been successfully cleared. Event processing has returned to normal levels, and all pending events have been delivered, no data was lost. Thank you for your patience while we worked through this.
We are currently investigating a delay with sending subscription data into Ordergroove for some of our Shopify stores. We will provide another update within the next 1 hour.
We have resumed normal operation of sending real-time subscription events into Ordergroove and are now working on delayed events. Another update will be provided within 1 hour.
We are continuing to work on delayed events and will provide another update once all the events have gone through.
The bulk of the delayed subscriptions have been identified and are queued up for delivery. We will provide additional details as they become available.
We are currently investigating response latency and will have an update in about 30 minutes.
We have identified and mitigated the effects of the latency. We are monitoring the performance of the system.
There is an ongoing spike in failed API responses. We are investigating again.
We have identified a secondary cause of latency and addressed the issue. We are monitoring the system and will post an update shortly.
The issue is resolved, and all services are operating normally. We apologize for the inconvenience and will publish a post-mortem within 24 hours which will be available via our CS team.
On December 5th, 3:45 AM EST Shopify GraphQL APIs experienced an elevated error rate. The issue was resolved by 4:15 AM EST. During the duration of this issue recurring subscription orders failed to place and went into Rejected status. The root cause of this outage within Shopify was due to a CloudFlare issue which impacted a lot of other service providers that rely on this service. The steps for remediation is to safely retry these rejected orders from our side. We are continuing to run this recovery efforts on all impacted orders and will provide another update as soon as all recurring orders have been retried by our system.
We have replayed previously rejected orders.
We’re currently experiencing a partial outage affecting SFTP report generation. Some scheduled reports are not being generated or delivered as expected. Our team is actively investigating and working to restore functionality as quickly as possible.
We have identified the issue, applied mitigating steps, and are monitoring
We are continuing to monitor for any further issues.
This incident has been resolved.
We are currently investigation the issue. For some people the Analytics page is not currently loading on the Ordergroove dashboard
The incident has been resolved, and analytics is loading correctly for all accounts now.
On November 1st, there was an update to the data being sent over from Shopify which negatively impacted our ability to mark products as in stock in our system. The issue only affects products where inventory is not tracked within Shopify. We have been working since then to implement a short-term solutions and have been rolling it out gradually to all merchants who we found to have been impacted by this change. This short-term fix requires our platform to pull additional product details from Shopify so we have been very gradually rolling it out and measuring system performance. We are going to provide another update as soon as this short-term fix has been rolled out to all stores.
The short-term fix has been rolled out to all affected merchants and product status is actively being updated in our system back to in stock for products where inventory is not tracked within Shopify.
We're investigating the issue affecting multiple services. We're seeing increased latencies and some requests fail to complete. We'll provide an update by 3:15PM EST.
We’ve identified and resolved the root cause of the issue. Latency levels are steadily improving, and we’re closely monitoring system performance to ensure continued stability. We’ll provide another update by 4 PM EST.
The incident was caused by elevated I/O latency in GCE Persistent Disk within a single region, which impacted our database performance and led to increased application response times. We mitigated the issue by relocating our database instances out of the affected zone. Our team is continuing to work with Google to ensure the underlying issue is fully resolved, but this is no longer impacting Ordergroove services. The incident began at 2:20 PM EST, mitigation was completed by 2:59 PM EST, and full recovery was achieved at 3:18 PM EST.
Order Reminder Events 9/15/2025: Yesterday, on 9/15/2025, Order Reminder events were not sent to Klaviyo or delivered through Ordergroove Webhook system What happened We upgraded our queueing system for the application that produces Order Reminder events. During the upgrade, a subset of consumers did not update correctly and were listening to the wrong system, which caused events to be missed. What we’re doing We are carefully replaying all missed Order Reminder events today. To avoid outdated actions, reminders will only be sent if the related order has not already been attempted. We expect the remediation will be fully completed today Next steps Once remediation is complete, we will run an internal post mortem to strengthen monitoring and ensure full coverage of the process going forward.
The remediation has been successfully completed. All missed Order Reminder events for the September 15 run have been replayed, with safeguards in place to prevent any stale actions. The September 16 run has executed successfully as well, and we expect future runs to proceed smoothly starting tomorrow. We will conduct a post mortem to strengthen monitoring for this process going forward. Thank you for your patience as we resolved this issue.
Between 7:04 AM - 9:05 AM ET we noticed degraded performance in our async workflow for creating Subscriptions. All subscriptions were created successfully. We have identified the root cause and are working on solutions to prevent this from happening again.
Our APIs were experiencing slow response time between 1:50am EST to 2:28am EST
We are experiencing some connectivity issues related to our cloud hosting provider. We are investigating this incident and will provide further updates as soon as they become available. This issue is impacting some internal systems that are responsible for migrating data into Ordergroove and delivering events. This issue also has implications on our Shopify integration. More details regarding this outage can be found here: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW
Although we do not have an exact ETA on recovery, it appears that Google Cloud Platform is starting to recover.
Although we do not have an exact ETA on recovery, it appears that Google Cloud Platform is starting to recover.
Google Cloud Platform has confirmed that the incident is now resolved and we're starting to see recovery across all affected systems. GCP incident: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW
We have identified an issue related to internal systems that process Shopify events from 03/06/2025 4PM EST and until 03/07/2025 9AM EST. The underlying issue has been resolved and we are actively investigating all impacted areas that require manual intervention. No action needs to be taken, our team will make sure all events will be reprocessed. Areas impacted: - Increase in recurring order placement retry attempts due to connection errors - Delayed delivery of events and updates tied to certain customer actions such as change of frequency or payment updates Next Steps: - We are working diligently to ensure that all delayed events are being processed - We are reviewing all potentially impacted recurring orders to ensure proper placement We will provide further updates throughout the day
All of the services have recovered and message delivery is back to normal. We are in the process of replaying any missed transactions. No action needs to be taken, our team will make sure all events will be reprocessed. We will provide further updates as progress is made.
All valid Shopify events have been replayed successfully. We are now entering the next and final phase of the recovery effort. There is a small number of recurring orders that were impacted by this issue which will be retried over the next 24-48 hour period. We are taking extra precautionary measures to ensure that we only retry orders which were not properly processed by Shopify. No action needs to be taken and we will provide a final update once the task is completed by our team.
We have completed the main recovery efforts related to this incident. The small amount of previously affected recurring orders are starting to flow through the placement process.
We have identified a delay in our subscription creation processing due to higher volume than usual. We are looking into this and scaling up our infrastructure in order to process subscriptions in the queue faster.
The issue has been identified and a fix is being implemented.
The incident has been resolved and the backlog has been entirely processed. We apologize for the delay in processing subscription creations, we have identified multiple steps to prevent this incident from happening again and we will prioritize those items to prevent any reoccurrence.