Density.io experienced a major incident on September 16, 2024 affecting Atlas and API and 1 more component, lasting 1d 21h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Sep 16, 2024, 04:24 PM UTC
A data pipeline delay from doorways with multiple Entry sensors (s5b and ELR) occurred from September 14 at 10:15 a.m. to September 16 at 11:25 a.m. EST. The data is processing but will come in two rounds. First, you will see inflated counts due to duplicates from multiple sensors, and then the process to remove duplicates will be run. We recommend not using historical data from September 14-16 until this is resolved. Live data (Safe Displays) counts will be impacted by the missing and duplicated data.
- identified Sep 16, 2024, 07:59 PM UTC
The process to correct count duplications between multiple sensors over a single doorway is expected to run overnight (EST). We will provide more information on resolution timing tomorrow, September 17th.
- identified Sep 17, 2024, 02:40 PM UTC
We are continuing to work on a fix for duplicate event removal.
- identified Sep 17, 2024, 08:32 PM UTC
The next round of data corrections from multiple Entry sensors will run on September 18 at 4 a.m. EST to avoid conflicting with peak hours.
- resolved Sep 18, 2024, 01:38 PM UTC
The process to remove duplicate events has been completed. Events have been corrected for September 14 at 10:15 a.m. to September 16 at 11:25 a.m. EST.