Ion Interactive experienced a major incident on November 1, 2022 affecting ion interactive Platform, lasting 2d 22h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 01, 2022, 03:47 PM UTC
We have identified a behavior in which some customers may have been experiencing difficulties logging in to their console and it is being investigated by our team. The Status Page will be updated once fixed. Thank you for understanding.
- investigating Nov 01, 2022, 04:35 PM UTC
We are continuing to investigate this issue.
- monitoring Nov 01, 2022, 04:55 PM UTC
The situation seems mostly normalized now and the team will keep monitoring. A report should be provided soon.
- resolved Nov 04, 2022, 02:22 PM UTC
This situation has been resolved for a few days.
- postmortem Nov 04, 2022, 06:11 PM UTC
On November 1st, a few customers experienced a brief login and/or downtime situation, due to a communication failure between servers and consoles, caused by a large amount of requests that generated slowness and errors, which made servers temporarily stop services as a protective measurement. Between 12:04 and 2:02pm \(GTM-3\), many users were unable to log in and some consoles were offline for a few minutes at a time, varying among them. To resolve the situation, it was necessary to clear the queue of requests. After that, some consoles were normalized and others required additional recycling of failures to reestablish the communication. Our servers are now configured for automatic cleaning in order to avoid similar events affecting consoles in the future.