Campaign Registry incident

Duplicate Campaign Billed Events

Minor Resolved View vendor source →

Campaign Registry experienced a minor incident on March 25, 2022, lasting 1d 5h. The incident has been resolved; the full update timeline is below.

Started
Mar 25, 2022, 02:27 PM UTC
Resolved
Mar 26, 2022, 07:47 PM UTC
Duration
1d 5h
Detected by Pingoru
Mar 25, 2022, 02:27 PM UTC

Update timeline

  1. investigating Mar 26, 2022, 02:22 PM UTC

    We discovered the issue that incorrectly generated duplicate CAMPAIGN_BILLED events for the change introduced in the 3.5.0 release.

  2. identified Mar 26, 2022, 02:23 PM UTC

    The bug affected only the expired campaigns. The root cause has been identified.

  3. identified Mar 26, 2022, 02:24 PM UTC

    The fix is deployed to testing environment and verified.

  4. monitoring Mar 26, 2022, 02:54 PM UTC

    The fix is pushed to production and we're monitoring the results.

  5. resolved Mar 26, 2022, 09:39 PM UTC

    After hours of monitoring, we observed that those duplicate events cease to exist. This incident is resolved.