Sekoia FRA1 Outage History

Started Feb 23, 2026, 02:15 PM UTC· resolved Feb 24, 2026, 08:53 AM UTC

1. investigating Feb 23, 2026, 02:15 PM UTC

   We have identified more than a hundred servers down at once on FRA1. As this is affecting nearly all clusters, we are currently looking into a cloud provider issue. At this time we are not sure about the overall impact, as our team is looking into it.

2. identified Feb 23, 2026, 02:18 PM UTC

   More than a hundred servers are down due to an issue impacting network switches in one of the data centers, causing disruptions to the message bus and other infrastructure components. Our teams are actively investigating, and we are coordinating with the data center provider to determine the estimated time for recovery.

3. identified Feb 23, 2026, 03:06 PM UTC

   This incident continues to impact approximately 80 servers. The event storage cluster is recovering with nodes restarting and data becoming progressively available. Indexation has resumed with some lag still present. Frontend, APIs, automation features, and detection services remain operational. The main residual issues relate to forwarding difficulties caused by four message bus nodes being down, affecting forwarding to search indexing components. Recovery of infrastructure nodes is ongoing. We are still in contact with our cloud provider to ensure all servers come back online as soon as possible.

4. identified Feb 23, 2026, 03:58 PM UTC

   This incident is ongoing with around 80 servers still down. Indexation is operating with less than 10 minutes of lag on the event storage cluster. The data center provider is manually rebooting servers that failed to start correctly, prioritizing critical nodes including message bus nodes. Recovery efforts are ongoing.

5. identified Feb 23, 2026, 05:25 PM UTC

   All event storage cluster servers hosting long-term data are now online with indexes gradually recovering; workers responsible for data retention will be restarted shortly. Remaining event storage cluster servers are being restarted via a custom process and should rejoin shortly.

6. resolved Feb 24, 2026, 08:53 AM UTC

   A few servers are still offline but FRA1 is fully functional since 23/02 at 23:00 CET. At this time we are still expecting a post-mortem from our cloud provider. This incident is now resolved.

Read the full incident report →

Started Feb 16, 2026, 05:17 PM UTC· resolved Feb 17, 2026, 08:41 AM UTC

1. investigating Feb 16, 2026, 05:17 PM UTC

   At 17:50 one of our web-facing provider had a major incident on their load balancer product, resulting in our web UI and APIs being unavailable until 18:05. We have switched over to another provider, the web UI and APIs are now fully available. This does not impact event ingestion in any way. We are still monitoring the situation and opened a case with the faulty cloud provider.

2. resolved Feb 17, 2026, 08:41 AM UTC

   Incident has been resolved at 18:05 as stated in previous update. All remediation actions have been identified to avoid reoccurence. Thank you for your patience during the incident.

Read the full incident report →

Started Feb 12, 2026, 02:16 PM UTC· resolved Feb 12, 2026, 05:01 PM UTC

1. identified Feb 12, 2026, 02:16 PM UTC

   We have identified an issue impacting retrieval of Alerts and Cases through API. The issue is also impacting the Web UI in Alerts listing and Display. Our Engineering team is applying remediation actions to restore the service. We apologize for the inconvenience caused and we thank you for your patience during the resolution.

2. identified Feb 12, 2026, 03:24 PM UTC

   We are continuing to work on a fix for this issue.

3. resolved Feb 12, 2026, 05:01 PM UTC

   The situation has been recovered and the Alerts and Cases functionalities are now fully availaible, through API and the Web UI. We thank you for your patience during the resolution of the incident.

Read the full incident report →

Started Feb 10, 2026, 05:34 AM UTC· resolved Feb 13, 2026, 03:04 PM UTC

1. investigating Feb 10, 2026, 04:02 AM UTC

   We are investigating an ongoing incident on the storage cluster having a major impact.

2. identified Feb 10, 2026, 04:04 AM UTC

   The event storage cluster is partially down and most data are unavailable. The team is trying to stabilize the cluster state. No data is lost but data availability will recover progressively in the next hours.

3. identified Feb 10, 2026, 05:34 AM UTC

   The event storage cluster is partially down and most data are still unavailable. The team is currently trying to reload all data. No data is lost but data availability will recover progressively in the next hours. There is also a huge delay on event processing.

4. identified Feb 10, 2026, 06:37 AM UTC

   The event storage cluster is partially down and most data are still unavailable. The team is still trying to reload all data. No data is lost but data availability will recover progressively in the next hours. There is also a huge delay on event processing.

5. identified Feb 10, 2026, 07:34 AM UTC

   The event storage cluster is partially down and most data are still unavailable. The team is still trying to reload all data. No data is lost but data availability will recover progressively in the next hours. There is also a huge delay on event processing.

6. identified Feb 10, 2026, 09:39 AM UTC

   Update - Our engineering teams have identified the source of the recent instability and are actively progressing through the restoration process. We would like to emphasize upfront that all data remains fully intact and secure. - Data Preservation: All data is preserved and its integrity is guaranteed. - Progressive Recovery: Data reloading is underway, and access will be progressively restored over the coming hours. - Additional mitigation works: in parallel of the data reloading, additional technical solutions are under evaluation to further accelerate recovery - Event Processing: Temporary delays in event processing may be observed while the system returns to its optimal state. Our priority is to restore the cluster to full operational capacity, and we are working diligently to finalize these steps. We appreciate your continued patience and will provide the next update before 12.00 CET.

7. identified Feb 10, 2026, 11:13 AM UTC

   Our engineering team is making steady progress with the restoration. Our efforts have continued and are focused on the following actions - Alert Data Recovery: restoring events linked to alerts to ensure customers can resume immediate investigations. - Search Functionality: re-enable general event searching outside of the alert scope. - Cluster Restoration: full background restoration of the Storage facility for all historical events. Our full technical team is committed to restore the cluster to full operational capacity, and we are working diligently to advance these steps. We appreciate your continued patience and will provide the next update before 14.00 CET.

8. identified Feb 10, 2026, 12:21 PM UTC

   Our engineering team continues to make steady progress restoring the storage facility. What’s working again - You can search events again, - Events linked to older alerts are fully available, so you can investigate historical alert activity as usual. What may still be delayed - Events linked to newer alerts may still appear with some delay. These delays are decreasing and are expected to be fully resolved within the next 2 hours. What’s still impacted - Some functionality remains inconsistent, including SOL queries. Our full technical team remains engaged and is working to restore full operational capacity as quickly as possible. Thank you for your patience, we’ll share the next update before 15:00 CET.

9. identified Feb 10, 2026, 12:22 PM UTC

   We are continuing to work on a fix for this issue.

10. identified Feb 10, 2026, 02:24 PM UTC

    Customers can now investigate their alerts with the associated events in real time. What’s working - Events linked to alerts are fully available, so you can investigate all alert activity as usual. - Global event search is restored, except for search combining eternal and non-eternal events What is still impacted - Some functionalities remain inconsistent, including SOL queries. Our technical team remains fully committed and is working to restore full operational capacity as quickly as possible. Thank you for your patience and understanding. The next update will be shared before 17:00 CET.

11. identified Feb 10, 2026, 03:47 PM UTC

    Our engineering team continues to make steady progress restoring the storage facility. What’s working - Events linked to alerts are fully available, customers can investigate all alert activity as usual. - Global event search is restored, except for search combining eternal and non-eternal events What is still impacted - Some functionalities remain inconsistent, including SOL queries, correlation and anomaly-based alerts. Our technical team remains fully committed and is working to restore full operational capacity as quickly as possible. Thank you for your patience and understanding. The next update will be shared before 18:30 CET.

12. identified Feb 10, 2026, 06:11 PM UTC

    Update - Our engineering team is continuing to work diligently on the full restoration of the storage facility. We are making steady progress, and the platform status remains consistent with our previous update: What’s working - Alert Investigation: Events linked to alerts are fully available. - Event Search: Global event search is available (excluding searches combining eternal and non-eternal events). What is still being restored - Advanced Functionalities: SOL queries, correlation, and anomaly-based alerts remain inconsistent. Our team remains fully mobilized to resolve these remaining items as quickly as possible. Thank you for your continued patience. The next update will be shared before 21:00 CET.

13. identified Feb 10, 2026, 07:48 PM UTC

    Update - Our engineering team is continuing to work diligently on the full restoration of the storage facility. We are making steady progress, and the platform status remains consistent with our previous update: What’s working - Alert Investigation: Events linked to alerts are fully available. - Event Search: Global event search is available (excluding searches combining eternal and non-eternal events). What is still being restored - Advanced Functionalities: SOL queries, correlation, and anomaly-based alerts remain inconsistent. Our team remains fully mobilized to resolve these remaining items as quickly as possible. Thank you for your continued patience.

14. monitoring Feb 11, 2026, 12:09 AM UTC

    Our engineering team is continuing to work diligently on the full restoration of the storage facility. We are now fully in control of the storage platform and are progressively re-enabling its core features. We’ve also entered the last phase of the incident: the lag processing phase to catch up on pending/backlogged processing. What’s working - Alert Investigation: Events linked to alerts are fully available. - Event Search: Global event search is available (excluding searches combining eternal and non-eternal events). What is still being restored - Advanced functionalities: SOL queries, correlation, and anomaly-based alerts may be inconsistent until the backlog is fully processed. Our team remains fully mobilized and committed to resolving the remaining items as quickly as possible. Thank you for your continued patience.

15. monitoring Feb 11, 2026, 09:22 AM UTC

    Our engineering team is continuing to work diligently on the full restoration of the storage facility. We are now fully in control of the storage platform and are progressively re-enabling its core features. The lag processing phase continues to catch up on pending/backlogged processing. We are making steady progress, and the platform status remains consistent with our previous update: What’s working - Alert Investigation: Events linked to alerts are fully available. - Event Search: Global event search is available (excluding searches combining eternal and non-eternal events). What is still being restored - Advanced functionalities: SOL queries, correlation, and anomaly-based alerts may be inconsistent. Our team remains fully mobilized and committed to resolving the remaining items as quickly as possible. Thank you for your continued patience. The next update will be shared before 14:00 CET

16. monitoring Feb 11, 2026, 03:22 PM UTC

    Our engineering team is continuing to work diligently on the full restoration of the storage facility. The lag processing phase continues to catch up on pending/backlogged processing. We are making steady progress, and the platform status remains consistent with our previous update: What’s working Alert Investigation: Events linked to alerts are fully available. Event Search: Global event search is available (excluding searches combining eternal and non-eternal events). What is still being restored Advanced functionalities: SOL queries, correlation, and anomaly-based alerts may be inconsistent. Our team remains fully mobilized and committed to resolving the remaining items as quickly as possible. Thank you for your continued patience. The next update will be shared before 19:00 CET

17. monitoring Feb 11, 2026, 08:53 PM UTC

    Our engineering team is continuing to work diligently on the full restoration of the storage facility. Correlation detection capabilities have been restored. Related alerts will continue raise as the lag is being resolved. The lag processing phase continues to catch up on backlogged processing. What’s working - Alert Investigation: Events linked to alerts are fully available. - Event Search: Global event search is available (excluding searches combining eternal and non-eternal events). What is still being restored - Advanced functionalities: SOL queries, and anomaly-based alerts may be inconsistent. Our team remains fully mobilized and committed to resolving the remaining items as quickly as possible. Thank you for your continued patience.

18. monitoring Feb 12, 2026, 08:54 AM UTC

    All features are now operational and the service is live, including SOL and anomaly detection. We are continuing to process a backlog of older events generated during the incident window. Some specific functionalities remain impacted until processing of the backlog, including - Event Search: searches combining eternal and non-eternal events - SOL requests for events between 10/02 13:00 UTC - 11/02 11:00 UTC Thank you for your patience throughout this incident.

19. monitoring Feb 12, 2026, 05:39 PM UTC

    All features are now operational and the service is live. We are continuing to process a backlog of older events generated during the incident window. Some specific functionalities remain impacted until processing of the backlog, including - SOL requests for events between 10/02 13:00 UTC - 11/02 11:00 UTC Thank you for your patience throughout this incident.

20. resolved Feb 13, 2026, 03:04 PM UTC

    The storage issue impacting our FRA1 region have been fully resolved. All functionalities have been restored. All queued data and events have now been successfully processed. We thank you for your patience throughout this incident.

Read the full incident report →

Started Feb 04, 2026, 07:00 PM UTC· resolved Feb 04, 2026, 09:22 PM UTC

1. investigating Feb 04, 2026, 08:14 PM UTC

   We detected an outage in our alert processing service. We're currenlty investigating the issue. We appreciate your patience and will provide updates as soon as we have more information.

2. identified Feb 04, 2026, 08:15 PM UTC

   We identified the issue and fixed it, and are currently catching the lag of delayed alerts.

3. monitoring Feb 04, 2026, 08:40 PM UTC

   We're currently catching the delay on alert processing and monitoring closely the situation. We appreciate your patience and will provide updates as soon as we have more information.

4. resolved Feb 04, 2026, 09:22 PM UTC

   Alert processing is fully operational, and all queued alerts have been processed. This incident is resolved, and we apologize for any inconvenience this may have caused.

Read the full incident report →

Started Jan 05, 2026, 10:20 AM UTC· resolved Jan 06, 2026, 12:57 PM UTC

1. investigating Jan 05, 2026, 10:20 AM UTC

   We are investigating reports of slow event search.

2. identified Jan 05, 2026, 02:54 PM UTC

   We have identified the issue, which was linked to a specific usage pattern.

3. monitoring Jan 05, 2026, 02:54 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Jan 06, 2026, 12:57 PM UTC

   No slowdowns have been observed since 05/01 13:30 CET. This incident is now resolved.

Read the full incident report →

Started Dec 26, 2025, 10:55 AM UTC· resolved Dec 26, 2025, 02:42 PM UTC

1. investigating Dec 26, 2025, 10:55 AM UTC

   Starting from 7:30, alert retrievals have been delayed due to a failover issue in the sightings cache cluster causing data read inconsistencies. The primary sightings cache instance was manually stopped to force the failover to the secondary instance. Investigation and recovery efforts are ongoing. Customers may experience delays in alert updates during this time.

2. identified Dec 26, 2025, 11:13 AM UTC

   Following the manual failover of the sightings cache cluster primary instance, alert retrieval processing has resumed and backlog processing is underway. The fix has restored normal operation and alerts are being consumed again. Monitoring is ongoing to ensure full recovery.

3. resolved Dec 26, 2025, 02:42 PM UTC

   Alert processing is fully operational, and all queued alerts have been processed. Our investigation found no alert loss. This incident is resolved, and we apologize for any inconvenience this may have caused.

Read the full incident report →

Started Dec 02, 2025, 04:33 PM UTC· resolved Dec 02, 2025, 08:53 PM UTC

1. identified Dec 02, 2025, 04:33 PM UTC

   We are currently experiencing performance degradation on the platform due to high input/output wait on a relational database. Our engineering team has identified the root cause and is implementing a fix. We do not have an ETA on the full resolution of the incident yet, but we expect it to resolve during this evening. We apologize for the inconvenience.

2. resolved Dec 02, 2025, 08:53 PM UTC

   Stability was restored around 18:00 by multiple actions from our team, relieving the database load. The performance issues affecting the platform have been resolved as of 20:45. We are implementing long-term improvements to prevent similar incidents and ensure consistent service quality. We apologize for the disruption and appreciate your understanding.

Read the full incident report →

Started Nov 07, 2025, 04:13 PM UTC· resolved Nov 07, 2025, 04:44 PM UTC

1. identified Nov 07, 2025, 04:13 PM UTC

   We are aware of a temporary unavailability on events related to alerts. Our teams are already working on the issue and applied a fix, events will be available again shortly. We apologize for the inconvenience.

2. resolved Nov 07, 2025, 04:44 PM UTC

   Events are available again. Thank you for your patience.

Read the full incident report →