Eleos Technologies experienced a major incident on December 15, 2023 affecting API and App Manager and 1 more component, lasting 1h 58m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 15, 2023, 02:01 PM UTC
As of 13:50 UTC the Eleos Platform started experiencing elevated error rates. This effects the platform APIs, and the mobile apps may fall back into offline mode until the issue is resolved.
- investigating Dec 15, 2023, 02:06 PM UTC
As of 14:06 UTC we have temporary disabled new service errors going to the Error Console or the Error Console API.
- investigating Dec 15, 2023, 02:13 PM UTC
As of 14:08 UTC error rates on the Eleos Platform have stabilized with mitigations in place. We are continuing to monitor the system.
- monitoring Dec 15, 2023, 02:24 PM UTC
As of 14:23 UTC we have reenabled service errors being written to the Error Console and the Error Console API. We are continuing to monitor the system.
- monitoring Dec 15, 2023, 02:56 PM UTC
We are working to put in place a fix for the underlying cause of the recent incidents. As part of the deployment of the fix we are going to be temporarily disabling writing service API errors to the Error Console and the Error Console API starting at 15:00 UTC (in about 5 minutes) until about 15:20 UTC. We appreciate your understanding as we work to get this issue completely resolved.
- monitoring Dec 15, 2023, 03:23 PM UTC
We are currently in the process of turning on Error Console logging. Some logs will continue to be missing for about 10 minutes until we bring Error Console logging fully back on. We will post an update when the Error Console is fully available again.
- monitoring Dec 15, 2023, 03:37 PM UTC
We have turned on logging to the Error Console, and are monitoring the stability of the Eleos Platform.
- resolved Dec 15, 2023, 04:00 PM UTC
Error Console logging is currently operating normally, and our system error rates are within our normal operating range. To protect overall system stability, at this time, a percentage of web service errors that occur may not be reflected in the Error Console or the Error Console API during periods of large overall error volume.