Populi incident

Populi unavailable for a large subset of users

Notice Resolved View vendor source →

Populi experienced a notice incident on July 3, 2023, lasting —. The incident has been resolved; the full update timeline is below.

Started
Jul 03, 2023, 03:32 PM UTC
Resolved
Jul 03, 2023, 11:30 AM UTC
Duration
Detected by Pingoru
Jul 03, 2023, 03:32 PM UTC

Update timeline

  1. resolved Jul 03, 2023, 03:32 PM UTC

    From roughly 4:20am - 4:50am Pacific time a large subset of Populi users were unable to access the service. This was because a weekly database consistency check started running at roughly 3am and eventually locked up a critical database server, leading to widespread system failure. We eventually managed to restart the database and restore service, and we'll be taking a close look at that check today to prevent it causing issues in the future!