Workvivo incident

Issue with cache cluster causing downtime

Notice Resolved View vendor source →

Workvivo experienced a notice incident on October 30, 2020 affecting Web Application and Mobile Applications and 1 more component, lasting 2h 5m. The incident has been resolved; the full update timeline is below.

Started
Oct 30, 2020, 12:48 PM UTC
Resolved
Oct 30, 2020, 02:53 PM UTC
Duration
2h 5m
Detected by Pingoru
Oct 30, 2020, 12:48 PM UTC

Affected components

Web ApplicationMobile ApplicationsDesktop ApplicationsAPI ServicesWorker Services

Update timeline

  1. investigating Oct 30, 2020, 12:48 PM UTC

    We are currently experiencing an issue with our cache cluster that is causing the platform to be unresponsive. We are currently working to rectify the issue as quickly as possible.

  2. investigating Oct 30, 2020, 01:14 PM UTC

    We are still working on resolving this issue. The platform is available intermittently, with a percentage of requests failing. We are working to return to being fully operational again as quickly as possible.

  3. monitoring Oct 30, 2020, 01:32 PM UTC

    We have rectified the issue, the platform is now operational again. We are currently monitoring the platform responsiveness to ensure the issue can be considered resolved. Thank you for your patience.

  4. resolved Oct 30, 2020, 02:53 PM UTC

    This incident has now been resolved. The issue was caused by a key failure in our caching infrastructure. We are currently rolling out additional redundancy mechanisms to offer additional protection against this issue in future. Thank you for your patience, and apologies for any inconvenience this incident may have caused.