PubWise experienced a minor incident on October 6, 2020 affecting Analytics Event Ingestion, lasting 2d 2h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 06, 2020, 12:21 PM UTC
We are currently investigating this issue.
- identified Oct 06, 2020, 12:23 PM UTC
We've identifed the underlying issue for the delayed data and have restarted the service. This resulted in a small amount of data loss.
- monitoring Oct 06, 2020, 12:26 PM UTC
We've determined that improperly formatted data resulted in a database insert failure which led to a retry loop and a continuous retry. The continuous retry led to blocking which led to us determining resetting the service was the best option. We're working on a patch to the system and will mark this resolved when that patch is released.
- resolved Oct 08, 2020, 03:18 PM UTC
This incident is resolved. We've determined the source of the underlying issue was a bug in handling improperly formatted data. The bug has been resolved and the issue should not reoccur.