INKY experienced a minor incident on May 30, 2023 affecting Dashboard Services US and Dashboard Services EU, lasting 25m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 30, 2023, 12:39 PM UTC
We are currently investigating an issue preventing users from accessing the Dashboard.
- resolved May 30, 2023, 01:04 PM UTC
Issue is resolved and all users should be able to log into the Dashboard without issue.
- postmortem May 30, 2023, 02:32 PM UTC
# Post incident report: Start: 30-March-2023 1000 UTC End: 30-March-2023 1230 UTC Duration: 2 hour 30 min ## Summary: Users unable to log into Dashboard. ## Root Cause: A system required for authentication failed over to its backup, but during the failover, permissions on it changed preventing login. ## Mitigation Action: We corrected the permissions issue which allowed users to once again log into the Dashboard as expected. ## Customer Impact: For approximately two and a half hours, users were not able to log into the Dashboard or access encrypted emails. ## Follow-up Items and Preventative Measures: * Improving monitoring around the failover process to ensure it is handled smoothly. * Implementing a change to the system itself in how it handles the failover so the permissions follow it regardless of what happens.