Genea experienced a minor incident on January 2, 2023 affecting Mercury, lasting 2h 42m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 27, 2023, 10:05 PM UTC
We have received reports that remote operations on controllers are not working, and there are delays in access logs processing. We are currently investigating the issue.
- identified Nov 27, 2023, 10:07 PM UTC
We started receiving alerts from AWS for high CPU usage of mercury hardware servers. One of the reason we assume due to which high CPU usage occurred is that many controllers got in a loop of offline/online together. We have identified the issue and are deploying a fix
- monitoring Nov 27, 2023, 10:09 PM UTC
We have implemented a fix, and we are currently monitoring the processing of access logs to ensure it is working fine. Additionally, we are checking the stability of the controller connections.
- resolved Nov 27, 2023, 10:10 PM UTC
Controller connection issue has been resolved. Please reach out to Genea Support if you're still experiencing issues.
- postmortem Nov 27, 2023, 10:13 PM UTC
## Root Cause of outage * We started seeing lag in mercury access event processing and also started receiving alerts from AWS for high CPU usage of mercury hardware servers. * Mercury driver instance was not reachable due to high CPU usage. * One of the reason we assume due to which high CPU usage occurred is that many controllers got in a loop of offline/online together. * We are still not sure about the root cause of this issue. We will investigate it moving forward.