RecoveryLink experienced a major incident on July 7, 2021 affecting Dashboard Analytics, lasting 4h 8m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 07, 2021, 09:04 PM UTC
The Enterprise Dashboard is being taken offline while we investigate an issue with login access.
- investigating Jul 07, 2021, 09:31 PM UTC
We are continuing to investigate the issue associated with login to the Enterprise Dashboard. Operations have ben suspended for maintenance work.
- identified Jul 07, 2021, 10:48 PM UTC
We have identified the source of the problem with the ETL to a reference database table. Applying updates to resolve.
- monitoring Jul 07, 2021, 11:52 PM UTC
A fix has been implemented and is being tested in development environment before release to production for customer access.
- resolved Jul 08, 2021, 01:13 AM UTC
This incident has been resolved and the team will continue monitoring the updates applied for secure operations. An additional protocol has been added to the database ETL update process to protect against changes to references tables that inform the underlying login and permissions controls.