UserVoice incident

Slowdowns and Errors

Minor Resolved View vendor source →

UserVoice experienced a minor incident on November 14, 2020 affecting Web Portal (subdomain) and Admin Console and 1 more component, lasting 19h 55m. The incident has been resolved; the full update timeline is below.

Started
Nov 14, 2020, 02:15 PM UTC
Resolved
Nov 15, 2020, 10:11 AM UTC
Duration
19h 55m
Detected by Pingoru
Nov 14, 2020, 02:15 PM UTC

Affected components

Web Portal (subdomain)Admin ConsoleUserVoice APIHelpdesk APIWidgetsEmail (incoming)Email (delivery)SearchUser Analytics

Update timeline

  1. identified Nov 14, 2020, 02:15 PM UTC

    We have been notified and identified an issue with our data store that is currently causing some production slowdowns and errors. Engineering are currently working to mitigate and resolve.

  2. identified Nov 14, 2020, 03:14 PM UTC

    Our engineers are continuing to work to repair the data store issue that is causing this issue.

  3. identified Nov 14, 2020, 06:43 PM UTC

    Engineers are continuing to work to resolve corruption in a data store table to fully restore service.

  4. monitoring Nov 14, 2020, 09:19 PM UTC

    Datastore is recovered, though we are operating with some additional latency as we reprocess older data. Engineers are continuing to monitor and attempting to scale up some additional resources to reduce measured latency.

  5. identified Nov 14, 2020, 09:41 PM UTC

    We have identified a linked issue regarding helpdesk ticket creation and are working to resolve.

  6. resolved Nov 15, 2020, 10:11 AM UTC

    All systems should be back and responsive now.