Courier incident

Automation Logs Delayed

Minor Resolved View vendor source →

Courier experienced a minor incident on July 9, 2024 affecting Web Application and Automations, lasting 3d 3h. The incident has been resolved; the full update timeline is below.

Started
Jul 09, 2024, 07:18 PM UTC
Resolved
Jul 12, 2024, 10:25 PM UTC
Duration
3d 3h
Detected by Pingoru
Jul 09, 2024, 07:18 PM UTC

Affected components

Web ApplicationAutomations

Update timeline

  1. investigating Jul 09, 2024, 07:18 PM UTC

    Courier Automation logs are currently experiencing a delay in showing up. The team is aware of this issue and is mitigating the root cause by relieving the bottleneck of incoming Automation requests.

  2. resolved Jul 12, 2024, 10:25 PM UTC

    This was a symptom of a previous automation issue, where a snowball effect caused a massive backlog of events. It took a long time for the Kinesis stream to catch up. We resolved the issue by temporarily increasing the stream shard count. The issue is now fully resolved.