IntelliShift - Telematics live feed delays

Update timeline

1. investigating Sep 12, 2022, 08:51 PM UTC

   We are currently investigating this issue.

2. resolved Sep 13, 2022, 12:24 AM UTC

   This incident has been resolved. Root cause analysis is still in progress.

3. postmortem Oct 15, 2022, 07:12 PM UTC

   The IntelliShift telematics live feed experience delays in updating the Map View module if the IntelliShift application. The IntelliShift Product Development & Architecture teams were immediately alerted and worked to find the root case and address the issue immediately. This issue occurred as a result of increased latency and performance degradation on several third-party map, geocoding, and address look up web services that the IntelliShift applications utilize to deliver standard product functionality to users. The increase in response times caused delays in the IntelliShift telematics live feed that is used to update the Map View. The IntelliShift Product Development & Architecture teams worked to address this issue by reviewing all systems logs. After seeing the potential root cause, the team began working with the third-party, while adding other services to reduce the impact of the performance degradation that was occurring on the third-party service. Once stabilized, additional monitoring was added to the service to mitigate the impact this could have in the future. To prevent issue from impacting IntelliShift users in the future, the IntelliShift Product Development & Architecture have added additional services to augment the existing service to reduce the reliance on these services going forward.