Bold Penguin experienced a major incident on September 27, 2019 affecting Authenticator and Partner Engine and 1 more component, lasting 29m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 27, 2019, 04:47 AM UTC
We are currently investigating an issue that has caused a sudden service outage. More details will be posted as they are discovered.
- investigating Sep 27, 2019, 05:15 AM UTC
The cause of the incident was identified as a database failure during a re-indexing operation as part of a production deployment. The database rolled over automatically before any manual intervention needed to happen. The subsequent database rollover caused our monitoring health checks to momentarily fail. Our monitors show there were intermittent failures across the APIs between 12:40am and 12:44am.
- resolved Sep 27, 2019, 05:17 AM UTC
This incident has been resolved.