Groove experienced a minor incident on August 30, 2019 affecting Web App and Inbox API, lasting 2h 36m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Aug 30, 2019, 02:03 PM UTC
At 12:30 UTC our background processing queue began backing up jobs. At 14:00 UTC we received a notification of degraded performance in the web app, caused by the growing backlog of unprocessed jobs. We have identified an issue with our search cluster's rebalancing behavior as the source of the backlog and are currently implementing a solution.
- monitoring Aug 30, 2019, 02:52 PM UTC
A fix has been implemented and our search cluster and background processing jobs are recovering. We will continue to monitor the situation as the recovery progresses.
- resolved Aug 30, 2019, 04:40 PM UTC
Our background processing queue is caught up and our search cluster has properly rebalanced. This incident has been resolved.