Ordergroove Outage History

The issue has been resolved by the Shopify team.

All pending orders have been successfully processed.

This incident has been resolved.

Our hosting provider has implemented a network-level mitigation and our systems are back in full operational order. We are going to continue to monitor our systems and provide further updates as needed.

The Ordergroove team performed a scheduled, security related maintenance on our admin backend system. During this upgrade, we temporarily moved all admin traffic to our servers hosted in a different region. After the maintenance was completed, all admin traffic was moved back to servers hosted in our primary region. Unfortunately, some admin users that opened the Ordergroove application or tried to use the login-on-behalf of feature during this cut-over period were denied access due to an app misconfiguration. We addressed the configuration issue and admin users were able to login again. We also noticed that delay in DNS propagation caused some users to continue to experience issues beyond our maintenance window but full access should have been restored by 9:55 AM ET Importantly, this issue impacted merchant staff/admin users and no customer impact has been found. Maintenance window: July 6 9:18AM ET - July 6 9:48AM ET Configuration change: July 6 9:53AM ET

The issue causing delays in our data pipelines has been fully resolved. Analytics data is now processing as expected, and users should see data updating within the normal up to 4 hour latency window. We will continue to monitor the system to ensure stability, but no further impact is expected. Thank you for your patience.

Webhook delivery delays have been fully resolved, and all queues have been successfully cleared. Event processing has returned to normal levels, and all pending events have been delivered, no data was lost. Thank you for your patience while we worked through this.

The bulk of the delayed subscriptions have been identified and are queued up for delivery. We will provide additional details as they become available.

The issue is resolved, and all services are operating normally. We apologize for the inconvenience and will publish a post-mortem within 24 hours which will be available via our CS team.

We have replayed previously rejected orders.

This incident has been resolved.

The incident has been resolved, and analytics is loading correctly for all accounts now.

The short-term fix has been rolled out to all affected merchants and product status is actively being updated in our system back to in stock for products where inventory is not tracked within Shopify.

The incident was caused by elevated I/O latency in GCE Persistent Disk within a single region, which impacted our database performance and led to increased application response times. We mitigated the issue by relocating our database instances out of the affected zone. Our team is continuing to work with Google to ensure the underlying issue is fully resolved, but this is no longer impacting Ordergroove services. The incident began at 2:20 PM EST, mitigation was completed by 2:59 PM EST, and full recovery was achieved at 3:18 PM EST.