Happeo incident

Increased error rate

Major Resolved View vendor source →

Happeo experienced a major incident on November 4, 2019, lasting —. The incident has been resolved; the full update timeline is below.

Started
Nov 04, 2019, 11:54 AM UTC
Resolved
Nov 04, 2019, 11:54 AM UTC
Duration
—
Detected by Pingoru
Nov 04, 2019, 11:54 AM UTC

Update timeline

  1. resolved Nov 04, 2019, 11:58 AM UTC

    We identified an increased error rate in our application affecting some parts of Happeo, including login, notifications and the admin panel functions. The error rate increased from 0 to 8,3% on average and peaked at 19,51%. The increased error rates lasted for 24 minutes from 09:59 to 10:23.

  2. postmortem Nov 04, 2019, 12:00 PM UTC

    **Start time:** 2019-11-04 09:59 **Resolution time:**: 2019-10-09 10:23 **Outage time: 23 minutes** **Problem:** Happeo was inaccessible from web and mobile due to the main authentication database being down. **Affected:** All customers, all users. **Root cause:** Google CloudSQL crashed at 08:46:47 and started a recovery at 08:53:09. With Cloud SQL being down our authentication did not work and therefore all requests were stopped in the API Gateway.