Wakeo incident

High latency on email delivery and webhooks retries

Minor Resolved View vendor source →

Wakeo experienced a minor incident on February 8, 2024 affecting Webapp and Emails, lasting 8h 57m. The incident has been resolved; the full update timeline is below.

Started
Feb 08, 2024, 11:00 AM UTC
Resolved
Feb 08, 2024, 07:58 PM UTC
Duration
8h 57m
Detected by Pingoru
Feb 08, 2024, 11:00 AM UTC

Affected components

WebappEmails

Update timeline

  1. identified Feb 08, 2024, 05:39 PM UTC

    A release from this morning is causing high latency on the following scopes: - Alert email delivery - Webhooks retries - Position freshness on the homepage's map (caused by our mitigation actions) We have already updated parts of our infrastructure to increase our processing rate in the next couple of hours, and are currently deploying a fix to mitigate the latency. Improvements have already been identified and will be implemented in the next few days.

  2. identified Feb 08, 2024, 05:46 PM UTC

    We are continuing to work on a fix for this issue.

  3. resolved Feb 08, 2024, 08:58 PM UTC

    All delayed tasks have been processed by 8PM CET. All our systems are back to the normal, and we'll deploy improvements to prevent this issue in the next days.