Intellishift incident
IntelliShift - Silent Passenger telematics live feed delay.
Intellishift experienced a major incident on October 6, 2022 affecting Web Application, lasting 12h 21m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 06, 2022, 02:54 PM UTC
As of 10:30am Eastern IntelliShift Telematics is experiencing a delay of the live feed. Our team is investigating the issue and are working to resolve this issue as soon as possible. During this time Map View - Live Tracking and Reports may be delayed.
- investigating Oct 06, 2022, 08:58 PM UTC
IntelliShift is experiencing Latency issues that is impacting Web & Mobile applications. Our engineering resources are working to resolve this issue as quickly as possible.
- resolved Oct 07, 2022, 03:15 AM UTC
This incident has been resolved and the IntelliShift Telematics live feed has been restored.
- postmortem Oct 15, 2022, 07:09 PM UTC
The IntelliShift telematics live feed experience delays in updating the Map View module if the IntelliShift application. The IntelliShift Product Development & Architecture teams were immediately alerted and worked to find the root case and address the issue immediately. This issue occurred as a result of increased latency and performance degradation on several third-party map, geocoding, and address look up web services that the IntelliShift applications utilize to deliver standard product functionality to users. The increase in response times caused delays in the IntelliShift telematics live feed that is used to update the Map View. The IntelliShift Product Development & Architecture teams worked to address this issue by reviewing all systems logs. After seeing the potential root cause, the team began working with the third-party, while adding other services to reduce the impact of the performance degradation that was occurring on the third-party service. Once stabilized, additional monitoring was added to the service to mitigate the impact this could have in the future. To prevent issue from impacting IntelliShift users in the future, the IntelliShift Product Development & Architecture have added additional services to augment the existing service to reduce the reliance on these services going forward.