Forticlient Cloud experienced a major incident on November 12, 2024, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Nov 12, 2024, 07:43 PM UTC
This incident is isolated only to the Japan DC. At around 1.39 pm PST, after a batch of scheduled EMS upgrade, 158 nodes were put in "error" state. The issue already fixed by rollback and code change. RCA: The current analysis points to insufficient resources during the upgrade push for customers on Oct 8 which causes the nodes to hit the timeout threshold. As a result of hitting the timeout threshold, it is put in "ERROR" state. There's a faulty logic where if a node is "ASSIGNED" but in "ERROR" state, a new node will be provisioned and assigned. This causes the customer EMS to appear like a new EMS, however without any profiles.