Sekoia FRA1 Outage History

Today at 12:38 CEST, we have started to progressively deploy our detection engine. Unfortunately, due to a misconfiguration, some events were not indexed into our storage engine, but all events were processed and related alerts were raised. The proportion of non-indexed events increased gradually. This issue was identified at 16:40 CEST and a fix was deployed around 17:00 CEST. We are now working to fix the situation and ensure that all missed events will be correctly pushed to the storage engine. During the issue, the detection engines continued to work as expected and alerts (including their related events) were correctly raised. All events related to alerts were also correctly indexed into our storage engine.

Our team is currently deploying a fix in order to index the missed events into the storage engine. Be aware that the process is expected to cause duplicates of some events that were already pushed. Thanks for your patience.

We are currently experiencing instability in the event storage cluster impacting indexation processes. This causes delays in events showing in events pages. Troubleshooting efforts are ongoing. Detection and alert raising are not impacted. Thanks you for your understanding.

The issue has been identified. Our teams are working on a fix.

The issue has been identified and a fix is being implemented.

We are currently experiencing an incident affecting task execution due to a limitation in the cluster responsible for coordination and scheduling. This issue started yesterday around 18:26 and has prevented new processing tasks from starting. The engineering team has identified the root cause to mitigate the issue. Investigation and remediation efforts are ongoing.

Following a maintenance operation on a shared event storage cluster node, observables are currently unavailable, impacting alert generation and sightings across all regions. This results in alerts not being raised, affecting the product's visibility on security events.

The indexation service in the FRA1 region is currently degraded due to multiple nodes going down in the event storage cluster caused by a hardware outage on our cloud provider. The incident started around 15:00 and prevents writing to several indices. Operations teams are performing recovery operations on the affected indices to restore service.

The issue has been identified and a fix is being implemented.

Event processing delays in the FRA1 region persist following yesterday's cloud provider-related incident. We've identified the issue and we're working towards improving system's stability and reducing processing lags. The situation is actively monitored and being addressed.

More than a hundred servers are down due to an issue impacting network switches in one of the data centers, causing disruptions to the message bus and other infrastructure components. Our teams are actively investigating, and we are coordinating with the data center provider to determine the estimated time for recovery.

This incident continues to impact approximately 80 servers. The event storage cluster is recovering with nodes restarting and data becoming progressively available. Indexation has resumed with some lag still present. Frontend, APIs, automation features, and detection services remain operational. The main residual issues relate to forwarding difficulties caused by four message bus nodes being down, affecting forwarding to search indexing components. Recovery of infrastructure nodes is ongoing. We are still in contact with our cloud provider to ensure all servers come back online as soon as possible.

This incident is ongoing with around 80 servers still down. Indexation is operating with less than 10 minutes of lag on the event storage cluster. The data center provider is manually rebooting servers that failed to start correctly, prioritizing critical nodes including message bus nodes. Recovery efforts are ongoing.

All event storage cluster servers hosting long-term data are now online with indexes gradually recovering; workers responsible for data retention will be restarted shortly. Remaining event storage cluster servers are being restarted via a custom process and should rejoin shortly.

We have identified an issue impacting retrieval of Alerts and Cases through API. The issue is also impacting the Web UI in Alerts listing and Display. Our Engineering team is applying remediation actions to restore the service. We apologize for the inconvenience caused and we thank you for your patience during the resolution.

We are continuing to work on a fix for this issue.

The event storage cluster is partially down and most data are unavailable. The team is trying to stabilize the cluster state. No data is lost but data availability will recover progressively in the next hours.

The event storage cluster is partially down and most data are still unavailable. The team is currently trying to reload all data. No data is lost but data availability will recover progressively in the next hours. There is also a huge delay on event processing.

The event storage cluster is partially down and most data are still unavailable. The team is still trying to reload all data. No data is lost but data availability will recover progressively in the next hours. There is also a huge delay on event processing.

The event storage cluster is partially down and most data are still unavailable. The team is still trying to reload all data. No data is lost but data availability will recover progressively in the next hours. There is also a huge delay on event processing.

Update - Our engineering teams have identified the source of the recent instability and are actively progressing through the restoration process. We would like to emphasize upfront that all data remains fully intact and secure. - Data Preservation: All data is preserved and its integrity is guaranteed. - Progressive Recovery: Data reloading is underway, and access will be progressively restored over the coming hours. - Additional mitigation works: in parallel of the data reloading, additional technical solutions are under evaluation to further accelerate recovery - Event Processing: Temporary delays in event processing may be observed while the system returns to its optimal state. Our priority is to restore the cluster to full operational capacity, and we are working diligently to finalize these steps. We appreciate your continued patience and will provide the next update before 12.00 CET.

Our engineering team is making steady progress with the restoration. Our efforts have continued and are focused on the following actions - Alert Data Recovery: restoring events linked to alerts to ensure customers can resume immediate investigations. - Search Functionality: re-enable general event searching outside of the alert scope. - Cluster Restoration: full background restoration of the Storage facility for all historical events. Our full technical team is committed to restore the cluster to full operational capacity, and we are working diligently to advance these steps. We appreciate your continued patience and will provide the next update before 14.00 CET.

Our engineering team continues to make steady progress restoring the storage facility. What’s working again - You can search events again, - Events linked to older alerts are fully available, so you can investigate historical alert activity as usual. What may still be delayed - Events linked to newer alerts may still appear with some delay. These delays are decreasing and are expected to be fully resolved within the next 2 hours. What’s still impacted - Some functionality remains inconsistent, including SOL queries. Our full technical team remains engaged and is working to restore full operational capacity as quickly as possible. Thank you for your patience, we’ll share the next update before 15:00 CET.

We are continuing to work on a fix for this issue.

Customers can now investigate their alerts with the associated events in real time. What’s working - Events linked to alerts are fully available, so you can investigate all alert activity as usual. - Global event search is restored, except for search combining eternal and non-eternal events What is still impacted - Some functionalities remain inconsistent, including SOL queries. Our technical team remains fully committed and is working to restore full operational capacity as quickly as possible. Thank you for your patience and understanding. The next update will be shared before 17:00 CET.

Our engineering team continues to make steady progress restoring the storage facility. What’s working - Events linked to alerts are fully available, customers can investigate all alert activity as usual. - Global event search is restored, except for search combining eternal and non-eternal events What is still impacted - Some functionalities remain inconsistent, including SOL queries, correlation and anomaly-based alerts. Our technical team remains fully committed and is working to restore full operational capacity as quickly as possible. Thank you for your patience and understanding. The next update will be shared before 18:30 CET.

Update - Our engineering team is continuing to work diligently on the full restoration of the storage facility. We are making steady progress, and the platform status remains consistent with our previous update: What’s working - Alert Investigation: Events linked to alerts are fully available. - Event Search: Global event search is available (excluding searches combining eternal and non-eternal events). What is still being restored - Advanced Functionalities: SOL queries, correlation, and anomaly-based alerts remain inconsistent. Our team remains fully mobilized to resolve these remaining items as quickly as possible. Thank you for your continued patience. The next update will be shared before 21:00 CET.

Update - Our engineering team is continuing to work diligently on the full restoration of the storage facility. We are making steady progress, and the platform status remains consistent with our previous update: What’s working - Alert Investigation: Events linked to alerts are fully available. - Event Search: Global event search is available (excluding searches combining eternal and non-eternal events). What is still being restored - Advanced Functionalities: SOL queries, correlation, and anomaly-based alerts remain inconsistent. Our team remains fully mobilized to resolve these remaining items as quickly as possible. Thank you for your continued patience.

We identified the issue and fixed it, and are currently catching the lag of delayed alerts.