PubWise experienced a minor incident on August 11, 2020 affecting Analytics Event Ingestion, lasting 13h 58m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Aug 11, 2020, 03:57 PM UTC
A software update yesterday led to loss of data in the data ingestion pipeline for approximately 15 hours for some customers, but not all. There was no effect on ad delivery. The immediate cause of the issue has been identified as a software update. We've rolled back the software update and we are monitoring.
- resolved Aug 11, 2020, 03:59 PM UTC
This issue is resolved. Please let us know if you see any issues.
- postmortem Aug 11, 2020, 03:59 PM UTC
We've identified that the verification process and canary testing process did not appropriately account for site specific failures of the type seen in this release. We're adjusting testing process to keep this from happening in the future.