Ordergroove Outage History
Ordergroove is up right nowPingoru has recorded 48 incidents for Ordergroove: 16 in the last 12 months and 5 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 2 | 22 of 720 (3%) |
| May 2026 | 0 | 0 of 744 (0%) |
| June 2026 | 1 | 3 of 720 (0%) |
| July 2026 | 1 | 0 of 744 (0%) |
| August 2026 | 1 | 6 of 744 (1%) |
| September 2026 | 4 | 180 of 720 (25%) |
| October 2026 | 0 | 3 of 511 (1%) |
Incidents, newest first
Minor September 29, 2026 Slightly elevated error rate for Shopify recurring order placement 1d 10h Shopify API & Mobile
Started Sep 29, 2026, 04:50 PM UTC· resolved Oct 01, 2026, 02:56 AM UTC
- investigating Sep 29, 2026, 04:50 PM UTC
We are investigating a slightly elevated error rate for recurring order placement. The issue appears to be related to an update within the Shopify platform and we are working with the Shopify team to help identify and remediate the root cause of this issue.
- resolved Oct 01, 2026, 02:56 AM UTC
The issue has been resolved by the Shopify team.
Notice September 22, 2026 Order Placement was slow 3h 12m Order Placement
Started Sep 22, 2026, 05:47 PM UTC· resolved Sep 22, 2026, 08:59 PM UTC
- monitoring Sep 22, 2026, 05:47 PM UTC
This morning, starting at approximately 5:00 AM CST, we experienced slower than normal order processing rates. The issue was addressed and fully resolved by approximately 12:30 PM CST. Orders are now processing at the normal rate and the system is working through the backlog.
- resolved Sep 22, 2026, 08:59 PM UTC
All pending orders have been successfully processed.
Minor September 2, 2026 Analytics page not loading on Ordergroove Dashboard 6d Analytics & Reporting
Started Sep 02, 2026, 05:20 PM UTC· resolved Sep 08, 2026, 06:12 PM UTC
- investigating Sep 02, 2026, 05:20 PM UTC
We're investigating reports of the RC3 analytics page experiencing issues while loading metrics. The issue doesn't affect metrics collection. Reporting has not been affected.
- monitoring Sep 02, 2026, 09:07 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Sep 03, 2026, 12:44 PM UTC
We are continuing to monitor this issue as the analytics page is still experiencing some instability. There is no data loss in the process since collector is working properly.
- resolved Sep 08, 2026, 06:12 PM UTC
This incident has been resolved.
Minor September 1, 2026 Hosting provider network degradation 30m Subscription Offers, Subscription Creation, Order Placement, APIs and 2 more
Started Sep 01, 2026, 03:49 PM UTC· resolved Sep 01, 2026, 04:19 PM UTC
- investigating Sep 01, 2026, 03:49 PM UTC
Our hosting provider is experiencing a system-wide network degradation which is affecting Ordergroove systems. All systems are still operational but are experiencing additional latency. We are actively working through this incident and will provide further updates shortly.
- resolved Sep 01, 2026, 04:19 PM UTC
Our hosting provider has implemented a network-level mitigation and our systems are back in full operational order. We are going to continue to monitor our systems and provide further updates as needed.
Minor July 6, 2026 Ordergroove Shopify admin application and features unaccessible –
Started Jul 06, 2026, 01:30 PM UTC· resolved Jul 06, 2026, 01:30 PM UTC
- resolved Jul 06, 2026, 03:24 PM UTC
The Ordergroove team performed a scheduled, security related maintenance on our admin backend system. During this upgrade, we temporarily moved all admin traffic to our servers hosted in a different region. After the maintenance was completed, all admin traffic was moved back to servers hosted in our primary region. Unfortunately, some admin users that opened the Ordergroove application or tried to use the login-on-behalf of feature during this cut-over period were denied access due to an app misconfiguration. We addressed the configuration issue and admin users were able to login again. We also noticed that delay in DNS propagation caused some users to continue to experience issues beyond our maintenance window but full access should have been restored by 9:55 AM ET Importantly, this issue impacted merchant staff/admin users and no customer impact has been found. Maintenance window: July 6 9:18AM ET - July 6 9:48AM ET Configuration change: July 6 9:53AM ET
Minor April 15, 2026 Analytics Data Delay 19h 33m Analytics & Reporting
Started Apr 15, 2026, 05:11 PM UTC· resolved Apr 16, 2026, 12:44 PM UTC
- identified Apr 15, 2026, 05:11 PM UTC
We are currently experiencing a delay in our data pipelines. As a result, Analytics data is not updating at the usual 4 hour latency. At this time, the latest available data reflects activity up to 8:50 AM EST on April 15. Our team is actively working to resolve the issue, and we expect data freshness to return to normal within the next few hours. We will provide another update as soon as more information is available.
- resolved Apr 16, 2026, 12:44 PM UTC
The issue causing delays in our data pipelines has been fully resolved. Analytics data is now processing as expected, and users should see data updating within the normal up to 4 hour latency window. We will continue to monitor the system to ensure stability, but no further impact is expected. Thank you for your patience.
Major April 7, 2026 Webhook Delivery Delays 1h 56m
Started Apr 07, 2026, 01:15 AM UTC· resolved Apr 07, 2026, 03:12 AM UTC
- investigating Apr 07, 2026, 01:15 AM UTC
We are currently experiencing delays in webhook event delivery. Our team is actively investigating the issue and working to identify the root cause. No events are expected to be lost, but delivery may be delayed. We will provide an update as soon as we have more information or once mitigation steps are in place. Thank you for your patience.
- resolved Apr 07, 2026, 03:12 AM UTC
Webhook delivery delays have been fully resolved, and all queues have been successfully cleared. Event processing has returned to normal levels, and all pending events have been delivered, no data was lost. Thank you for your patience while we worked through this.
Minor March 11, 2026 Subscription creation delayed 7h 34m Ordergroove App - Subscription Creation & Webhooks
Started Mar 11, 2026, 11:25 AM UTC· resolved Mar 11, 2026, 07:00 PM UTC
- investigating Mar 11, 2026, 11:25 AM UTC
We are currently investigating a delay with sending subscription data into Ordergroove for some of our Shopify stores. We will provide another update within the next 1 hour.
- identified Mar 11, 2026, 12:43 PM UTC
We have resumed normal operation of sending real-time subscription events into Ordergroove and are now working on delayed events. Another update will be provided within 1 hour.
- identified Mar 11, 2026, 02:30 PM UTC
We are continuing to work on delayed events and will provide another update once all the events have gone through.
- resolved Mar 11, 2026, 07:00 PM UTC
The bulk of the delayed subscriptions have been identified and are queued up for delivery. We will provide additional details as they become available.
Major January 21, 2026 We are experiencing a performance degradation in the API response times 1h 56m Subscription Creation, APIs, Admin Dashboard (previously RC3)
Started Jan 21, 2026, 06:22 PM UTC· resolved Jan 21, 2026, 08:18 PM UTC
- investigating Jan 21, 2026, 06:22 PM UTC
We are currently investigating response latency and will have an update in about 30 minutes.
- monitoring Jan 21, 2026, 06:32 PM UTC
We have identified and mitigated the effects of the latency. We are monitoring the performance of the system.
- investigating Jan 21, 2026, 07:05 PM UTC
There is an ongoing spike in failed API responses. We are investigating again.
- monitoring Jan 21, 2026, 07:45 PM UTC
We have identified a secondary cause of latency and addressed the issue. We are monitoring the system and will post an update shortly.
- resolved Jan 21, 2026, 08:18 PM UTC
The issue is resolved, and all services are operating normally. We apologize for the inconvenience and will publish a post-mortem within 24 hours which will be available via our CS team.
Notice December 5, 2025 Shopify Order Placement 5h 12m Shopify API & Mobile
Started Dec 05, 2025, 08:14 PM UTC· resolved Dec 06, 2025, 01:26 AM UTC
- monitoring Dec 05, 2025, 08:14 PM UTC
On December 5th, 3:45 AM EST Shopify GraphQL APIs experienced an elevated error rate. The issue was resolved by 4:15 AM EST. During the duration of this issue recurring subscription orders failed to place and went into Rejected status. The root cause of this outage within Shopify was due to a CloudFlare issue which impacted a lot of other service providers that rely on this service. The steps for remediation is to safely retry these rejected orders from our side. We are continuing to run this recovery efforts on all impacted orders and will provide another update as soon as all recurring orders have been retried by our system.
- resolved Dec 06, 2025, 01:26 AM UTC
We have replayed previously rejected orders.
Major December 1, 2025 Partial Outage: SFTP Report Generation Delays 1d Analytics & Reporting
Started Dec 01, 2025, 08:48 PM UTC· resolved Dec 02, 2025, 09:27 PM UTC
- investigating Dec 01, 2025, 08:48 PM UTC
We’re currently experiencing a partial outage affecting SFTP report generation. Some scheduled reports are not being generated or delivered as expected. Our team is actively investigating and working to restore functionality as quickly as possible.
- monitoring Dec 01, 2025, 10:31 PM UTC
We have identified the issue, applied mitigating steps, and are monitoring
- monitoring Dec 02, 2025, 09:26 PM UTC
We are continuing to monitor for any further issues.
- resolved Dec 02, 2025, 09:27 PM UTC
This incident has been resolved.
Major November 5, 2025 Analytics page not loading consistently on Ordergroove Dashboard 5h 32m Analytics & Reporting
Started Nov 05, 2025, 05:16 PM UTC· resolved Nov 05, 2025, 10:49 PM UTC
- investigating Nov 05, 2025, 05:16 PM UTC
We are currently investigation the issue. For some people the Analytics page is not currently loading on the Ordergroove dashboard
- resolved Nov 05, 2025, 10:49 PM UTC
The incident has been resolved, and analytics is loading correctly for all accounts now.
Notice November 4, 2025 Products showing as out of stock in Ordergroove 7h 9m Ordergroove App - Subscription Creation & Webhooks
Started Nov 04, 2025, 11:44 PM UTC· resolved Nov 05, 2025, 06:54 AM UTC
- monitoring Nov 04, 2025, 11:44 PM UTC
On November 1st, there was an update to the data being sent over from Shopify which negatively impacted our ability to mark products as in stock in our system. The issue only affects products where inventory is not tracked within Shopify. We have been working since then to implement a short-term solutions and have been rolling it out gradually to all merchants who we found to have been impacted by this change. This short-term fix requires our platform to pull additional product details from Shopify so we have been very gradually rolling it out and measuring system performance. We are going to provide another update as soon as this short-term fix has been rolled out to all stores.
- resolved Nov 05, 2025, 06:54 AM UTC
The short-term fix has been rolled out to all affected merchants and product status is actively being updated in our system back to in stock for products where inventory is not tracked within Shopify.
Minor October 29, 2025 Intermittent performance degradation 1h 8m Subscription Offers, Subscription Creation, Order Placement, APIs and 1 more
Started Oct 29, 2025, 06:47 PM UTC· resolved Oct 29, 2025, 07:55 PM UTC
- investigating Oct 29, 2025, 06:47 PM UTC
We're investigating the issue affecting multiple services. We're seeing increased latencies and some requests fail to complete. We'll provide an update by 3:15PM EST.
- investigating Oct 29, 2025, 07:22 PM UTC
We’ve identified and resolved the root cause of the issue. Latency levels are steadily improving, and we’re closely monitoring system performance to ensure continued stability. We’ll provide another update by 4 PM EST.
- resolved Oct 29, 2025, 07:55 PM UTC
The incident was caused by elevated I/O latency in GCE Persistent Disk within a single region, which impacted our database performance and led to increased application response times. We mitigated the issue by relocating our database instances out of the affected zone. Our team is continuing to work with Google to ensure the underlying issue is fully resolved, but this is no longer impacting Ordergroove services. The incident began at 2:20 PM EST, mitigation was completed by 2:59 PM EST, and full recovery was achieved at 3:18 PM EST.
Notice September 16, 2025 Order Reminder Events were missed on 9/15/2025 7h 27m
Started Sep 16, 2025, 07:40 PM UTC· resolved Sep 17, 2025, 03:08 AM UTC
- monitoring Sep 16, 2025, 07:40 PM UTC
Order Reminder Events 9/15/2025: Yesterday, on 9/15/2025, Order Reminder events were not sent to Klaviyo or delivered through Ordergroove Webhook system What happened We upgraded our queueing system for the application that produces Order Reminder events. During the upgrade, a subset of consumers did not update correctly and were listening to the wrong system, which caused events to be missed. What we’re doing We are carefully replaying all missed Order Reminder events today. To avoid outdated actions, reminders will only be sent if the related order has not already been attempted. We expect the remediation will be fully completed today Next steps Once remediation is complete, we will run an internal post mortem to strengthen monitoring and ensure full coverage of the process going forward.
- resolved Sep 17, 2025, 03:08 AM UTC
The remediation has been successfully completed. All missed Order Reminder events for the September 15 run have been replayed, with safeguards in place to prevent any stale actions. The September 16 run has executed successfully as well, and we expect future runs to proceed smoothly starting tomorrow. We will conduct a post mortem to strengthen monitoring for this process going forward. Thank you for your patience as we resolved this issue.
Notice August 21, 2025 Degraded Subscription Creation Performance –
Started Aug 21, 2025, 11:00 AM UTC· resolved Aug 21, 2025, 11:00 AM UTC
- resolved Aug 21, 2025, 04:37 PM UTC
Between 7:04 AM - 9:05 AM ET we noticed degraded performance in our async workflow for creating Subscriptions. All subscriptions were created successfully. We have identified the root cause and are working on solutions to prevent this from happening again.
Notice August 4, 2025 API response time degradation 46m APIs
Started Aug 04, 2025, 06:41 AM UTC· resolved Aug 04, 2025, 07:28 AM UTC
- resolved Aug 04, 2025, 06:41 AM UTC
Our APIs were experiencing slow response time between 1:50am EST to 2:28am EST
Minor June 12, 2025 The google cloud platform is experiencing issues 2h 46m Ordergroove Integration, Order Placement, Ordergroove App - Subscription Creation & Webhooks, APIs and 2 more
Started Jun 12, 2025, 06:21 PM UTC· resolved Jun 12, 2025, 09:07 PM UTC
- investigating Jun 12, 2025, 06:21 PM UTC
We are experiencing some connectivity issues related to our cloud hosting provider. We are investigating this incident and will provide further updates as soon as they become available. This issue is impacting some internal systems that are responsible for migrating data into Ordergroove and delivering events. This issue also has implications on our Shopify integration. More details regarding this outage can be found here: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW
- investigating Jun 12, 2025, 07:10 PM UTC
Although we do not have an exact ETA on recovery, it appears that Google Cloud Platform is starting to recover.
- investigating Jun 12, 2025, 07:18 PM UTC
Although we do not have an exact ETA on recovery, it appears that Google Cloud Platform is starting to recover.
- resolved Jun 12, 2025, 09:07 PM UTC
Google Cloud Platform has confirmed that the incident is now resolved and we're starting to see recovery across all affected systems. GCP incident: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW
Notice March 7, 2025 Partial degradation of service with merchants on the Shopify platform 1d 10h Order Placement, Ordergroove App - Subscription Creation & Webhooks
Started Mar 07, 2025, 04:11 PM UTC· resolved Mar 09, 2025, 02:12 AM UTC
- monitoring Mar 07, 2025, 04:11 PM UTC
We have identified an issue related to internal systems that process Shopify events from 03/06/2025 4PM EST and until 03/07/2025 9AM EST. The underlying issue has been resolved and we are actively investigating all impacted areas that require manual intervention. No action needs to be taken, our team will make sure all events will be reprocessed. Areas impacted: - Increase in recurring order placement retry attempts due to connection errors - Delayed delivery of events and updates tied to certain customer actions such as change of frequency or payment updates Next Steps: - We are working diligently to ensure that all delayed events are being processed - We are reviewing all potentially impacted recurring orders to ensure proper placement We will provide further updates throughout the day
- monitoring Mar 07, 2025, 10:53 PM UTC
All of the services have recovered and message delivery is back to normal. We are in the process of replaying any missed transactions. No action needs to be taken, our team will make sure all events will be reprocessed. We will provide further updates as progress is made.
- monitoring Mar 08, 2025, 10:42 AM UTC
All valid Shopify events have been replayed successfully. We are now entering the next and final phase of the recovery effort. There is a small number of recurring orders that were impacted by this issue which will be retried over the next 24-48 hour period. We are taking extra precautionary measures to ensure that we only retry orders which were not properly processed by Shopify. No action needs to be taken and we will provide a final update once the task is completed by our team.
- resolved Mar 09, 2025, 02:12 AM UTC
We have completed the main recovery efforts related to this incident. The small amount of previously affected recurring orders are starting to flow through the placement process.