AdRoll incident

Ads API hard deleting data denoting relationships between AdGroups and Ads

AdRoll experienced a notice incident on January 9, 2024 affecting API, lasting 6d 18h. The incident has been resolved; the full update timeline is below.

Started: Jan 09, 2024, 08:17 PM UTC
Resolved: Jan 16, 2024, 03:14 PM UTC
Duration: 6d 18h
Detected by Pingoru: Jan 09, 2024, 08:17 PM UTC

Affected components

API

Update timeline

identified Jan 09, 2024, 08:17 PM UTC

Issue: Our Ad APIs are having issues where data denoting the relationship between AdGroups and their Ads are being hard deleted from Web Retargeting campaigns' AdGroups instead of soft-deleting them. Impact: Affects Web Retargeting campaign details and reports page where deleted Ads under their parent AdGroups are no longer shown. Scheduled reports are also missing deleted Ad metrics. Next Steps: A fix for this will be deployed in the next couple of hours. Engineering is currently evaluating possible strategies to recover Ad data that has been hard-deleted.
identified Jan 10, 2024, 03:39 PM UTC

A fix for the issue has been deployed to production. In addition, a recovery procedure has also been defined, and we estimate to have a majority of the data denoting the relationship between AdGroups and their Ads restored around Tuesday/Wednesday next week.
resolved Jan 16, 2024, 03:14 PM UTC

The recovery of data denoting the relationship between AdGroups and their Ads is completed.