Alerts API & FlightStats.com Flight Alerts degraded performance

Update timeline

1. investigating Nov 06, 2025, 12:32 AM UTC

   A subset of Alerts API customers are currently experiencing degraded performance with our Alerting service. Our engineering team is actively investigating the issue and will provide updates as soon as more information becomes available. We will provide the next update on 2025-11-06 at 07:54 UTC. Thank you for your patience and understanding.

2. investigating Nov 06, 2025, 08:12 AM UTC

   Our Engineering Team has identified that the Alert APIs are affecting one specific airline. They are actively investigating the root cause of the issue. Thank you for your patience and understanding.

3. investigating Nov 06, 2025, 03:55 PM UTC

   We have identified an event processing timeout issue that is preventing one airline's flights from triggering Alerts API callbacks. Our engineers are continuing to investigate the cause of these timeouts. A further update will be provided no later than 2025-11-06 23:54 UTC.

4. investigating Nov 06, 2025, 10:39 PM UTC

   The issue was initially limited to some Alerts API and FlightStats.com customers but is now affecting our alerting service for other airlines. Previous mitigation efforts have not resolved the problem. Our development team will continue working on a fix to restore normal service and we will provide the next update at 2025-11-07 06:45 UTC.

5. investigating Nov 06, 2025, 10:50 PM UTC

   We are continuing to investigate this issue.

6. investigating Nov 07, 2025, 05:01 AM UTC

   We’d like to provide you with an update on this case. As per our engineering team, the system has shown stable connectivity over the past six hours, and we continue to monitor its performance closely. The next update will be provided by 13:00 UTC on 2025-11-07, or sooner if there are any further developments.

7. monitoring Nov 07, 2025, 08:13 AM UTC

   The queue size has returned to normal, and alert delivery to customers is operating as usual. The connection timeout issue is stable, and the team is monitoring it. The original issue of missing alerts for several Rule IDs affecting some customers has been resolved, alerts will now be delivered as expected. We will continue to monitor the system and provide updates.

8. investigating Nov 07, 2025, 12:38 PM UTC

   We are continuing to experience latency issues during the processing of flight events. Our engineers are investigating. We will provide a further update no later than 2025-11-07 20:37 UTC. Thank you for your continued patience and understanding.

9. identified Nov 07, 2025, 05:17 PM UTC

   Our engineering team has identified the cause of the latency and are working to clear the alerts queue. We will provide a further update on or before 2025-11-08 01:17 UTC.

10. identified Nov 07, 2025, 08:31 PM UTC

    We are continuing the recovery process here, still no major breakthrough. Database is bloated with huge volume and less memory available. Database Backup recovery is running for a while and we are waiting for the process to complete. We will provide a further update no later than 2025-11-08 5:00 UTC. Thank you for your continued patience and understanding.

11. identified Nov 08, 2025, 01:21 PM UTC

    We are continuing to work on a fix for this issue.

12. monitoring Nov 08, 2025, 07:05 PM UTC

    The database is performing better, but the main delay is due to the large volume of stale data that must be manually cleaned up before normal operations can resume. They are actively working on the cleanup now and anticipate that services will resume in the next update 9th November 1:00 UTC.

13. monitoring Nov 08, 2025, 08:17 PM UTC

    The Flight Alerts API service have been restored and all the operations have been working. Alerts are getting delivered. We are continuously monitoring the system for performance. There could be some latency which is expected due to the heavy backlog. Next update, 2025-11-09 6:30 UTC.

14. monitoring Nov 10, 2025, 12:26 AM UTC

    The performance of the Flight Alerts API looks good and the requests are getting processed. We will continue monitoring the system and the cleanup the backlog and stale data. Next update - 2025-11-10 6:30 UTC

15. monitoring Nov 10, 2025, 06:40 AM UTC

    The performance seems stable now after a temporary spike from 5.30 UTC to 6.20 UTC. The alerts are getting delivered without delays. Next Actions: 1) We are checking why there is a temporary spike during this timeframe 2) Team is working on the Database cleanup to optimize the storage space Next update will be 2025-11-10 10:30 UTC.

16. monitoring Nov 10, 2025, 10:37 AM UTC

    System performance remains stable, with alerts being delivered promptly and no further issues detected. We are continuing to investigate the temporary spike, however have yet discovered the cause. The database clean up is ongoing, and we are monitoring the system closely. A further update will be provided no later than 2025-11-10 14:30 UTC.

17. monitoring Nov 10, 2025, 02:52 PM UTC

    All systems are stable with no further performance issues detected. Root cause investigation is underway. A further update will be provided no later than 2025-11-11 06:30 UTC.

18. monitoring Nov 11, 2025, 06:31 AM UTC

    Alerts performance has been stable over the past 24 hours, with no issues reported for the Alerts API. The root cause was identified as increased database data from sweepers running, which led to SQL query delays. We addressed this by vertically scaling the database and are currently working to clear stale data.

19. resolved Nov 11, 2025, 06:31 AM UTC

    This incident has been resolved.