Campaign Registry incident

Duplicate Campaign Billed Events

Resolved after 1 day 5 hours, on March 25, 2022.

resolved Mar 26, 2022, 09:39 PM UTC

After hours of monitoring, we observed that those duplicate events cease to exist. This incident is resolved.

Started
Mar 25, 2022, 02:27 PM UTC
Resolved
Mar 26, 2022, 07:47 PM UTC
Duration
1d 5h
Detected by Pingoru
Mar 25, 2022, 02:27 PM UTC

Update timeline

  1. investigating Mar 26, 2022, 02:22 PM UTC

    We discovered the issue that incorrectly generated duplicate CAMPAIGN_BILLED events for the change introduced in the 3.5.0 release.

  2. identified Mar 26, 2022, 02:23 PM UTC

    The bug affected only the expired campaigns. The root cause has been identified.

  3. identified Mar 26, 2022, 02:24 PM UTC

    The fix is deployed to testing environment and verified.

  4. monitoring Mar 26, 2022, 02:54 PM UTC

    The fix is pushed to production and we're monitoring the results.

  5. resolved Mar 26, 2022, 09:39 PM UTC

    After hours of monitoring, we observed that those duplicate events cease to exist. This incident is resolved.

Your product runs on more than Campaign Registry

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →