IT Portal experienced a major incident on January 11, 2021 affecting US Cloud, lasting 57m. The incident has been resolved; the full update timeline is below.
Affected components
US Cloud
Update timeline
- investigating Jan 11, 2021, 02:11 PM UTC
We are currently investigating this issue.
- identified Jan 11, 2021, 02:21 PM UTC
The issue has been identified and a fix is being implemented
- monitoring Jan 11, 2021, 02:32 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jan 11, 2021, 03:09 PM UTC
This incident has been resolved.
- postmortem Jan 11, 2021, 03:09 PM UTC
We quickly detected high CPU utilization on one of the web servers in our cluster. We determined that the “Host Process for Windows Services” was consuming the resource and we removed it from the cluster. Soon after removal, the performance was back to normal. The web server will remain decommissioned and a new one will be put in its place. We will schedule that soon.