UserVoice incident

DNS Failure Causing Site Availability Issues

Critical Resolved View vendor source →

UserVoice experienced a critical incident on July 17, 2020 affecting Web Portal (subdomain) and Admin Console and 1 more component, lasting 3h 34m. The incident has been resolved; the full update timeline is below.

Started
Jul 17, 2020, 09:37 PM UTC
Resolved
Jul 18, 2020, 01:12 AM UTC
Duration
3h 34m
Detected by Pingoru
Jul 17, 2020, 09:37 PM UTC

Affected components

Web Portal (subdomain)Admin ConsoleUserVoice APIHelpdesk APIWidgetsEmail (incoming)Email (delivery)SearchUser Analytics

Update timeline

  1. identified Jul 17, 2020, 09:37 PM UTC

    Our DNS and DDoS provider, CloudFlare is having a significant issue right now that is impacting our site availability. We are working with them to try and mitigate/resolve the issue.

  2. monitoring Jul 17, 2020, 09:47 PM UTC

    CloudFlare appears to have restored service, resolving our downstream issues.

  3. resolved Jul 18, 2020, 01:12 AM UTC

    Cloudflare believes a bad router announcement caused massive backend failures on their system. This caused DNS to fail for multiple regions (explicitly North America). The issue has been fully resolved and tested.