Populi experienced a critical incident on June 11, 2021, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Jun 11, 2021, 03:34 PM UTC
From around midnight to 6:30am Pacific time, a subset of Populi users experienced a number of problems viewing and saving data. A routine data integrity tool that verifies data is replicated properly across our server clusters locked several tables and failed to unlock them as it normally does, which caused many writes to fail. However, because the vast majority of writes were still succeeding, we didn't get notified by our monitoring platforms as we normally do. The issue was reported to us and resolved within 30m. We're very sorry for this disruption to your morning, and are adding safety checks and additional monitoring conditions to make sure this doesn't happen again!