Courier incident

Delayed Automation Execution

Major Resolved View vendor source →

Courier experienced a major incident on February 23, 2023 affecting API, lasting 1h 18m. The incident has been resolved; the full update timeline is below.

Started
Feb 23, 2023, 12:59 AM UTC
Resolved
Feb 23, 2023, 02:18 AM UTC
Duration
1h 18m
Detected by Pingoru
Feb 23, 2023, 12:59 AM UTC

Affected components

API

Update timeline

  1. investigating Feb 23, 2023, 12:59 AM UTC

    We're are currently investigating reports of a delay in resuming execution of Courier Automations while they are in a "waiting" state.

  2. identified Feb 23, 2023, 01:19 AM UTC

    The issue has been identified and a fix is being applied to production systems. Once the fix has been applied, backlogged Automations will begin to clear out. We will post an update in the next 40 minutes.

  3. monitoring Feb 23, 2023, 02:10 AM UTC

    This issue had a broader impact that originally indicated: Instead of only automations in a "WAITING" state, all automations processing was adversely impacted. The applied fix appears to have stabilized automations execution. Backlogged automations are now being processed. The team will continue to monitor.

  4. resolved Feb 23, 2023, 02:18 AM UTC

    The incident has been resolved. All backlogged Automations have been processed.