VergeSense incident
EN-1 devices not sending reports to cloud (entryway data outage)
VergeSense experienced a major incident on February 9, 2024 affecting API and Webhooks, lasting 12h 23m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 09, 2024, 02:27 AM UTC
We are currently investigating this issue.
- identified Feb 09, 2024, 03:32 AM UTC
We identified a problem with the VergeSense pipeline that was preventing us from ingesting reports from Xovis FW5 devices.
- monitoring Feb 09, 2024, 04:37 AM UTC
We've implemented a fix on our side that should restore report ingestion & metric generation. There is a 7 hour data gap in entryway data that we will attempt to backfill. We are still observing error codes and elevated latency when attempting to access Xovis devices through IBEX. There is an outstanding ticket with Xovis to identify the cause of these issues.
- monitoring Feb 09, 2024, 05:36 AM UTC
All EN-1 devices that were reporting before the outage have resumed reporting. IBEX webUIs are still unavailable for FW5 sensors.
- resolved Feb 09, 2024, 02:50 PM UTC
Sensors are now accessible through IBEX.