Technolutions experienced a major incident on April 1, 2022 affecting Slate, lasting 2h 6m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Apr 01, 2022, 09:18 PM UTC
We are aware of an issue that is preventing the return of the normal system configuration on multiple databases. Remediation is in progress and we expect to have all databases back to normal operation within the next few minutes.
- monitoring Apr 01, 2022, 09:52 PM UTC
System configurations are returning normal values on all production databases at this time, and these systems have recovered. System configurations for test environments are currently being updated. Environments remain accessible during this refresh.
- resolved Apr 01, 2022, 11:25 PM UTC
On Friday afternoon at approximately 4:20pm Eastern Time, an engineer applied a system configuration update too broadly. This configuration update was intended to update a single configuration value across databases. Due to human error, the value was applied to other unintended configuration values. While Slate databases remained online and operational for end-users and staff that may have already been logged in, new staff and single sign-on logins may have received a "Invalid Security Provider" error message due to this invalid configuration. Login issues were first reported around 4:40pm, as configuration caches expired and more databases picked up incorrect system configurations, and by 5:15pm, remediations were underway to revert the configuration changes. By 5:30pm, databases were restored to the correct and expected configurations. We deeply regret this error, and we have begun an internal audit and review of our protocols to prevent this from happening and to ensure that other such routine configuration updates are always appropriately scoped.