Intellishift incident
IntelliShift - Telematics Live Feed Delay.
Intellishift experienced a major incident on September 5, 2023 affecting Web Application and Mobile App and 1 more component, lasting 11h 31m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 05, 2023, 06:50 PM UTC
As of 2:45pm Eastern IntelliShift Telematics is experiencing a delay of the live feed. Our team is investigating the issue and are working to resolve this issue as soon as possible. During this time Map View - Live Tracking and Reports may be delayed.
- identified Sep 06, 2023, 01:42 AM UTC
The issue has been identified and a fix is being implemented.
- resolved Sep 06, 2023, 06:21 AM UTC
This incident has been resolved.
- postmortem Sep 07, 2023, 12:36 AM UTC
The IntelliShift telematics live feed experience delays in updating the Map View module if the IntelliShift application. The IntelliShift Product Development & Architecture teams were immediately alerted and worked to find the root case and address the issue immediately. This issue occurred as a result of increased latency and performance degradation on several third-party map, geocoding, and address look up web services that the IntelliShift applications utilize to deliver standard product functionality to users. The increase in response times caused delays in the IntelliShift telematics live feed that is used to update the Map View. The IntelliShift Product Development & Architecture teams worked to address this issue by reviewing all systems logs. After seeing the potential root cause, the team began working with the third-party, while adding other services to reduce the impact of the performance degradation that was occurring on the third-party service. Once stabilized, additional monitoring was added to the service to mitigate the impact this could have in the future. To prevent issue from impacting IntelliShift users in the future, the IntelliShift Product Development & Architecture have added additional services to augment the existing service to reduce the reliance on these services going forward.