Sekoia FRA1 Outage History

An ongoing incident in the FRA1 region is impacting event indexing and search capabilities for some customers. The event storage is experiencing some failures which causes unstabilities, this results in delayed ingestion and unavailable searches. Engineers are investigating and working on mitigations to restore indexing and search functionality.

An ongoing incident is impacting detection services due to a failure in cache instances. Engineers are performing restarts and recovery steps; service restoration is in progress but may take some time.

The root cause has been identified, and our team is currently applying patches. Event indexing was delayed for the impacted communities, which means some events may appear late on event pages. Indexing is now running and processing the backlog. Search is available again. We will keep you updated.

Engineers identified and isolated the offending playbook and restored real-time processing. A backlog playbook tasks remains and customers may see delayed automated workflows while the backlog is processed. Teams continue to work on completing the backlog.

The major incident affecting automation features in the FRA1 cloud region is currently under recovery. The automation feature for playbooks runs was stopped to mitigate the issue and is now being restarted gradually. Customers may still experience delays in playbook runs, but recovery is in progress.

We have identified the root cause of the issue and a fix is being applied.

We have identified the root cause of the service disruption and applied a fix. Event ingestion, search, and alerting functionalities are now resuming. However, you may experience degraded performance and slower response times as our systems process the backlog of safely stored events.

We are continuing to work on a fix for this issue.

Ingestion is slowly recovering. Additional cluster nodes are being provisioned to improve resilience. Monitoring continues to ensure full restoration.

We are continuing to work on a fix for this issue.

The issue has been identified and a fix is being implemented.

We applied a fix and indexation has now resumed. We're now starting to catch the lag. The incident is actively monitored for stability.

Today at 12:38 CEST, we have started to progressively deploy our detection engine. Unfortunately, due to a misconfiguration, some events were not indexed into our storage engine, but all events were processed and related alerts were raised. The proportion of non-indexed events increased gradually. This issue was identified at 16:40 CEST and a fix was deployed around 17:00 CEST. We are now working to fix the situation and ensure that all missed events will be correctly pushed to the storage engine. During the issue, the detection engines continued to work as expected and alerts (including their related events) were correctly raised. All events related to alerts were also correctly indexed into our storage engine.

Our team is currently deploying a fix in order to index the missed events into the storage engine. Be aware that the process is expected to cause duplicates of some events that were already pushed. Thanks for your patience.

The issue has been identified. Our teams are working on a fix.

The issue has been identified and a fix is being implemented.

We are currently experiencing an incident affecting task execution due to a limitation in the cluster responsible for coordination and scheduling. This issue started yesterday around 18:26 and has prevented new processing tasks from starting. The engineering team has identified the root cause to mitigate the issue. Investigation and remediation efforts are ongoing.

Following a maintenance operation on a shared event storage cluster node, observables are currently unavailable, impacting alert generation and sightings across all regions. This results in alerts not being raised, affecting the product's visibility on security events.

The indexation service in the FRA1 region is currently degraded due to multiple nodes going down in the event storage cluster caused by a hardware outage on our cloud provider. The incident started around 15:00 and prevents writing to several indices. Operations teams are performing recovery operations on the affected indices to restore service.

The issue has been identified and a fix is being implemented.

Event processing delays in the FRA1 region persist following yesterday's cloud provider-related incident. We've identified the issue and we're working towards improving system's stability and reducing processing lags. The situation is actively monitored and being addressed.

More than a hundred servers are down due to an issue impacting network switches in one of the data centers, causing disruptions to the message bus and other infrastructure components. Our teams are actively investigating, and we are coordinating with the data center provider to determine the estimated time for recovery.

This incident continues to impact approximately 80 servers. The event storage cluster is recovering with nodes restarting and data becoming progressively available. Indexation has resumed with some lag still present. Frontend, APIs, automation features, and detection services remain operational. The main residual issues relate to forwarding difficulties caused by four message bus nodes being down, affecting forwarding to search indexing components. Recovery of infrastructure nodes is ongoing. We are still in contact with our cloud provider to ensure all servers come back online as soon as possible.

This incident is ongoing with around 80 servers still down. Indexation is operating with less than 10 minutes of lag on the event storage cluster. The data center provider is manually rebooting servers that failed to start correctly, prioritizing critical nodes including message bus nodes. Recovery efforts are ongoing.

All event storage cluster servers hosting long-term data are now online with indexes gradually recovering; workers responsible for data retention will be restarted shortly. Remaining event storage cluster servers are being restarted via a custom process and should rejoin shortly.

We have identified an issue impacting retrieval of Alerts and Cases through API. The issue is also impacting the Web UI in Alerts listing and Display. Our Engineering team is applying remediation actions to restore the service. We apologize for the inconvenience caused and we thank you for your patience during the resolution.

We are continuing to work on a fix for this issue.