Eleos Technologies experienced a major incident on May 22, 2024 affecting Mobile Apps, lasting 4h 5m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 22, 2024, 02:11 PM UTC
We have received reports of push notifications not being sent to Android devices. We are actively investigating.
- investigating May 22, 2024, 02:30 PM UTC
We are continuing to investigate this issue, we will provide another update within the next 30 minutes.
- investigating May 22, 2024, 03:00 PM UTC
We appreciate your patience while we dig into this issue. We are still investigating the root cause and will continue to provide updates as we learn more.
- investigating May 22, 2024, 03:35 PM UTC
We are still investigating the root cause and will continue to provide updates as we learn more. Thank you for your continued patience.
- investigating May 22, 2024, 04:10 PM UTC
As we continue to dig into the root cause, our logging indicates that roughly 10 percent of Android users may not be receiving push notifications. At this point, we still believe iOS to be unaffected.
- identified May 22, 2024, 04:44 PM UTC
We believe we have identified the root cause of these push notification failures and we are working to implement a possible solution. Thank you again for your patience, we apologize for the disruption. We will update once we have confirmed the solution.
- identified May 22, 2024, 05:15 PM UTC
We are continuing to test our potential solution to fix these push notification issues, we will follow up once we have verified this solution works and is in place.
- monitoring May 22, 2024, 05:44 PM UTC
We have implemented a solution across affected customers and are continuing to monitor the push notification issues. If you continue to receive reports from drivers not receiving push notifications, please reach out to us at [email protected]
- resolved May 22, 2024, 06:17 PM UTC
We have completed in rolling out our fix for these Android push notification failures. Since 18:05 UTC, our logging indicates we are no longer encountering issues processing push notifications for Android drivers. During this time period, the push notifications for iOS drivers should have been unaffected. Thank you again for your patience.
- postmortem Jun 06, 2024, 05:12 PM UTC
Between 5/20/2024 19:17 UTC and 5/22/2024 18:05 UTC our systems failed to deliver most push notifications to drivers using Android devices within a subset of Eleos Platform apps. We have directly reached out to each customer who fell under this subset of affected apps, if you have not heard from us directly about this issue, then you and your drivers were unaffected by this incident. During this window, any drivers using this app on an Android device would not have received push notifications for messages they received within the Eleos Platform app. Along with this, any [manual load refresh API](https://dev.eleostech.com/platform/platform.html#operation/postLoadUpdate) calls would not have triggered the driver’s loads to be refreshed automatically. This outage was caused by a vendor we use for delivering push notifications deprecating an API that was setup on a subset of apps. This vendor had notified us of this deprecation, however, they had shut off the API a month earlier than they had scheduled. Any apps set up to use this deprecating/legacy API for push notifications would have been affected. To remedy the issue, we went through and upgraded all affected applications to the vendor’s latest version of this push notification API. We currently lack monitoring around certain push notification failures, which is why this issue lasted as long as it did and why we were not aware of it until a customer notified us. We will be working to build out more robust monitoring around push notifications failures, such that any issues like this going forward will be identified and resolved in a more appropriate amount of time. We are deeply sorry for any disruption we may have caused for you and your drivers.