Density.io experienced a major incident on December 12, 2023 affecting Atlas and API and 1 more component, lasting 7h 49m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 12, 2023, 02:28 PM UTC
We are currently investigating a large number of errors in Entry data.
- identified Dec 12, 2023, 02:58 PM UTC
Density performed scheduled maintenance this morning at 3 a.m. EST for v2 API changes and fixes to Entry Live (displays). -Between 4-4:30 a.m. EST there was a full outage across the platform due to the v2 cutover. This was resolved at 4:30 a.m. EST. -Between 7-9 a.m. EST a limited number of customers may have experienced v2 API errors. This was resolved at 9 a.m. EST. -At 5:30 a.m. EST, we began to experience issues with Entry data, related to the fixes for Entry Live. This is ongoing; updates to follow.
- identified Dec 12, 2023, 04:34 PM UTC
The Entry count backlog is currently clearing from the earlier outage. Until this is resolved, live data in Displays will be incorrect, and you may experience jumps in numbers. All v2 and v3 APIs are operating normally. If you use APIs to feed a data lake or other record of count data, and received API errors between 4-4:30 a.m. or 7-9 a.m., you will need to backfill your data. Not all customers received error messages, please check your API instance.
- resolved Dec 12, 2023, 10:17 PM UTC
When we cut over the live Entry data to our improved count database, the system ran into unexpected errors. This caused a backup in Entry count, which resulted in incorrect live count for about 20% of spaces. This issue was resolved at 4:30 p.m. EST. The only impact to historical data and APIs was a slight delay in arrival, resolved at 3:45 p.m. EST. There is no incorrect data populating in either location. The legacy Dashboard had a bug with timestamps. If you use the legacy platform for building-level data, this is now available. v2 APIs had a bug with representing timestamps on the response. If you ran into timestamp parsing errors on your end, you may have to re-pull data.