Think Lumo incident

Investigating potential slowdown of alerts

Critical Resolved

Think Lumo experienced a critical incident on January 25, 2024, lasting 17h 49m. The incident has been resolved; the full update timeline is below.

Started
Jan 25, 2024, 09:00 PM UTC
Resolved
Jan 26, 2024, 02:50 PM UTC
Duration
17h 49m
Detected by Pingoru
Jan 25, 2024, 09:00 PM UTC

Update timeline

  1. monitoring Jan 25, 2024, 09:06 PM UTC

    We are investigating a slowdown in alerts. The underlying cause has been traced to the flight information feed from our real time data vendor.

  2. monitoring Jan 25, 2024, 09:27 PM UTC

    Volumes seem to be recovering. We are continuing to monitor.

  3. monitoring Jan 25, 2024, 09:44 PM UTC

    We can confirm that alerts are flowing again. Some alerts may be delayed by up to 15 minutes while services catch up.

  4. monitoring Jan 25, 2024, 09:54 PM UTC

    The real-time data feed from our vendor is down again. We are working with them to identify and resolve the issue. Thank you for your patience.

  5. monitoring Jan 25, 2024, 10:20 PM UTC

    Our vendor has traced the issue to their cloud services provider and are working with them to identify and resolve the issue. We will post an update as soon as we hear more.

  6. monitoring Jan 25, 2024, 11:26 PM UTC

    Our partner is continuing to investigate and resolve with their cloud provider. In the meantime, you may observe spiky alert volumes with sudden bursts of real-time flight alerts followed by periods of inactivity. All other alert types (predictions, waivers, etc.) and subscriptions/registration of flights are unaffected.

  7. monitoring Jan 26, 2024, 06:42 AM UTC

    We have cut over to a different real time data feed, which should cause alerts to start flowing reliably again. Even though service has been restored, this incident will remain open until the original data feed is BAU. We will plan to update you once the original feed is restored. Thank you for your patience.

  8. monitoring Jan 26, 2024, 02:50 PM UTC

    All upstream services have been restored, and we are back at BAU. Thank you for your patience. We will share an findings as soon as they are available. Please reach out to Lumo Support ([email protected]) with any questions about this incident.