SecurID experienced a critical incident on April 11, 2022 affecting na3.access Authentication Service, lasting 9m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Apr 11, 2022, 10:14 AM UTC
SaaS Operations has identified the cause of the issue and is working to implement a fix.
- monitoring Apr 11, 2022, 10:22 AM UTC
The issue affecting SecurID has been corrected. The SaaS Operations team is monitoring the fix. We will post a root cause analysis as soon as it is available.
- resolved Apr 11, 2022, 10:23 AM UTC
After monitoring the fix, SaaS Operations has determined that the incident affecting SecurID has been resolved. We will post a root cause analysis as soon as it is available.
- postmortem Apr 29, 2022, 07:50 PM UTC
Service incidents that occurred on April 10th and 11th caused degraded authentication services for a subset of customers from the North America region hosted on the NA3 deployment. This was caused by a product defect introduced as part of the March release. This defect was triggered by specific use condition that resulted in database connections being exhausted. Customers were rolled back to our February release which resolved the incident on April 11th. **Mitigations :** SecurID is continuously taking steps to improve the SecurID Access service and processes to help ensure such incidents do not occur in the future. This includes, but is not limited to: * The underlying defect has gone through a full analysis by our engineering teams. * SecurID engineering has developed a fix and was deployed to all regions on April 21st * SecurID engineering did a full review of database query plans as it pertained to this defect and optimized plans were added. * Automation added to detect and report database connection leaks