Intellishift incident
IntelliShift - Silent Passenger telematics live feed delay.
Intellishift experienced a major incident on September 14, 2022 affecting Web Application, lasting 14h 11m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 14, 2022, 01:49 PM UTC
As of 9:30am Eastern IntelliShift Telematics is experiencing a delay of the live feed. Our team is investigating the issue and are working to resolve this issue as soon as possible. During this time Map View - Live Tracking and Reports may be delayed.
- investigating Sep 14, 2022, 04:56 PM UTC
IntelliShift continues to investigate this issue and are working diligently to resolve this issue as quickly as possible.
- resolved Sep 15, 2022, 11:53 AM UTC
IntelliShift's - Silent Passenger Live Feed has been restored and assets are back to reporting in real-time. Reports will reflect the activity of the assets during the period of the delay. We are continuing to monitor this very closely. Thank You.
- postmortem Oct 15, 2022, 07:12 PM UTC
The IntelliShift telematics live feed experience delays in updating the Map View module if the IntelliShift application. The IntelliShift Product Development & Architecture teams were immediately alerted and worked to find the root case and address the issue immediately. This issue occurred as a result of increased latency and performance degradation on several third-party map, geocoding, and address look up web services that the IntelliShift applications utilize to deliver standard product functionality to users. The increase in response times caused delays in the IntelliShift telematics live feed that is used to update the Map View. The IntelliShift Product Development & Architecture teams worked to address this issue by reviewing all systems logs. After seeing the potential root cause, the team began working with the third-party, while adding other services to reduce the impact of the performance degradation that was occurring on the third-party service. Once stabilized, additional monitoring was added to the service to mitigate the impact this could have in the future. To prevent issue from impacting IntelliShift users in the future, the IntelliShift Product Development & Architecture have added additional services to augment the existing service to reduce the reliance on these services going forward.