TrackJS experienced a major incident on December 23, 2021 affecting Error Ingestion, lasting 56m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 23, 2021, 02:01 PM UTC
We are investigating some drop or delay in incoming error traffic.
- identified Dec 23, 2021, 02:07 PM UTC
Around 5pm last night (CDT), our systems began adding some valid customer tokens to our block list, believing that they were invalid. The block list has been cleared and we continue to investigate the cause.
- monitoring Dec 23, 2021, 02:26 PM UTC
A network timeout to our infrastructure exposed a bug that blocked the tokens of several active accounts. We have unblocked everything and are working on a resolution to both the bug, as well as pro-active alerting should this sort of over-aggressive blocking be attempted by our system again.
- resolved Dec 23, 2021, 02:58 PM UTC
We are putting in place some new safety checks to prevent this from happening. Data is coming in normally.