HelloID incident
HelloID login currently not possible for some users
HelloID experienced a major incident on January 25, 2023 affecting Access Management and Access Management and 1 more component, lasting 1d 2h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 25, 2023, 07:14 AM UTC
Currently, it is not possible to log into HelloID for some users. We are investigating the issue.
- identified Jan 25, 2023, 08:18 AM UTC
Microsoft 365 has an outage. It seems this causes problems for HelloID. We keep investigating.
- identified Jan 25, 2023, 08:18 AM UTC
We are continuing to work on a fix for this issue.
- identified Jan 25, 2023, 01:32 PM UTC
Most of the HelloID services recovered, but some are still troublesome. Microsoft is working to recover the impacted services. A part from https://status.azure.com/en-us/status/history/ : Mitigation: We identified a recent change to WAN as the underlying cause and have rolled back this change. Networking telemetry shows recovery from 09:00 UTC onwards across all regions and services, with the final networking equipment recovering at 09:35 UTC. Most impacted Microsoft services automatically recovered once network connectivity was restored, and we worked to recover the remaining impacted services. Go to the URL for the full article.
- monitoring Jan 25, 2023, 02:30 PM UTC
It seems that the remaining services are restored and recovered. We are monitoring the situation. Please reach out to support at your local office if something is not recovered for your service.
- resolved Jan 26, 2023, 09:34 AM UTC
This incident has been resolved. We keep monitoring the situation.
- postmortem Jan 31, 2023, 02:57 PM UTC
**Post Mortem incident 25_th_ of January** On Wednesday the 25_th_ a major outage occurred on the HelloID platform. The outage caused long loading times, multiple redirects or failures to load pages. The outage was caused by an issue at one of our cloud partners Microsoft. A detailed description of the issue can be found at: [Azure status history | Microsoft Azure](https://status.azure.com/en-us/status/history/) **How did we respond?** Tools4ever engineers started to investigate the issue at 6.15 AM UTC. This process was hindered due to loading problems with the administrative interface of the Microsoft cloud environment. Several communication issues between HelloID services were identified and mitigation actions were implemented. At 8 AM UTC Microsoft confirmed that they were experiencing a major incident causing communication issues between Microsoft cloud services. At 10 AM UTC most services were restored. However several databases remained unavailable until 13:45 UTC due to the connection issue in the Microsoft datacenters. At 13:45 UTC all issues were resolved.