Spot AI incident

Data Processing Degradation - Cameras incorrectly reporting as offline, alerts and cloud-based AI analytics affected.

Major Resolved View vendor source →

Spot AI experienced a major incident on April 5, 2024 affecting Spot Platform, lasting 7h 7m. The incident has been resolved; the full update timeline is below.

Started
Apr 05, 2024, 06:03 PM UTC
Resolved
Apr 06, 2024, 01:10 AM UTC
Duration
7h 7m
Detected by Pingoru
Apr 05, 2024, 06:03 PM UTC

Affected components

Spot Platform

Update timeline

  1. investigating Apr 05, 2024, 10:48 PM UTC

    Our data processing infrastructure is running behind which is causing inaccuracies in camera and appliance online status, as well as delays in alerts and cloud-based AI analytics (such as People Search and Attribute Search) No video data has been lost. There is no impact to video recording or access and no customer footage is currently affected.

  2. resolved Apr 06, 2024, 01:10 AM UTC

    This outage was resolved. Below is an outage summary: Service Outage: Appliances and cameras on the Spot dashboard were incorrectly reported as offline for at most 5 hours and 46 minutes from 11:03 AM PDT to 4:51 PM PDT on 4/5/2024. There was no impact to video recording or access and no customer footage was affected by the outage. Impact: Between 11:03 AM PDT to 4:51 PM PDT on 4/5/2024, - Cameras and appliances were incorrectly reported offline, due to a slowdown in our data processing infrastructure. - Alerts were not sent during this timeframe, due to a slowdown in our data processing infrastructure. - People search and attribute search features cannot be searched during this 6 hour window. There was no impact to any other AI analytics, including people, vehicle, motion detection, LPR, intelligence dashboards, or other AI analytics outside of people and attribute search. Issue Resolution: We apologize for any inconvenience as a result of this service outage and are currently in the process of implementing processes to provide you with the best customer experience. These improvements include: - Increasing compute capabilities and resources to our data processing infrastructure to better handle unexpected data load spikes. - Adding new monitors and alarms on health signals as indicated by our service outage to ensure proactive resolution of data processing slowdowns. Additional Questions: If you have any questions, please get in touch with our support team at [email protected].