Ordergroove experienced a minor incident on October 29, 2025 affecting Subscription Offers and Subscription Creation and 1 more component, lasting 1h 8m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 29, 2025, 06:47 PM UTC
We're investigating the issue affecting multiple services. We're seeing increased latencies and some requests fail to complete. We'll provide an update by 3:15PM EST.
- investigating Oct 29, 2025, 07:22 PM UTC
We’ve identified and resolved the root cause of the issue. Latency levels are steadily improving, and we’re closely monitoring system performance to ensure continued stability. We’ll provide another update by 4 PM EST.
- resolved Oct 29, 2025, 07:55 PM UTC
The incident was caused by elevated I/O latency in GCE Persistent Disk within a single region, which impacted our database performance and led to increased application response times. We mitigated the issue by relocating our database instances out of the affected zone. Our team is continuing to work with Google to ensure the underlying issue is fully resolved, but this is no longer impacting Ordergroove services. The incident began at 2:20 PM EST, mitigation was completed by 2:59 PM EST, and full recovery was achieved at 3:18 PM EST.