LimaCharlie Outage History

Started Oct 20, 2026, 04:20 PM UTC· resolved Oct 20, 2026, 04:20 PM UTC

1. resolved Oct 20, 2026, 04:20 PM UTC

   Resolved. Between 06:10 and 11:10 UTC on 20 October 2026, a Google Cloud infrastructure incident in the London (europe-west2) region caused intermittent failures for BigQuery-backed features in the LimaCharlie UK region: Replay / historical searches and search cost estimates, which could fail with "failed to get sensor event count: googleapi: Error 400: The job encountered an error during execution ... jobBackendError" The Insight usage overview in the web app Scheduled IOC index refreshes Telemetry ingestion, detection & response, and the UK API endpoints remained available throughout. Google has mitigated the incident (Google Cloud incident Xs2XTHzAVC9YnafnPyzj) and we have seen no further errors since 11:10 UTC. Searches that failed during the window can simply be re-run; scheduled index refreshes resume on their next run.

Read the full incident report →

Started Oct 08, 2026, 01:47 AM UTC· resolved Oct 08, 2026, 05:26 AM UTC

1. investigating Oct 08, 2026, 01:47 AM UTC

   We are aware of the outage, have isolated the issue, and working with our cloud providers to remediate. We will update as soon as we have an ETA.

2. resolved Oct 08, 2026, 05:26 AM UTC

   GCP has rolled back the breaking change, and services are confirmed normalized. We are calling this issue resolved.

Read the full incident report →

Started Sep 19, 2026, 09:50 PM UTC· resolved Sep 19, 2026, 11:16 PM UTC

1. investigating Sep 19, 2026, 09:50 PM UTC

   This incident is being reported retroactively. Starting at approximately 21:50 UTC on 2026-09-19, endpoint sensors connecting to our US datacenter progressively lost their connection to the LimaCharlie cloud. By 22:08 UTC nearly all sensor traffic to the US datacenter was affected. Other datacenters (Canada, Europe, UK, India, Australia) were not impacted. Our connectivity alerting paged the on-call team at 22:17 UTC and engineers began investigating.

2. resolved Sep 19, 2026, 11:16 PM UTC

   Resolved. Sensor connectivity to the US datacenter was restored at 23:13 UTC and traffic was back to normal levels by 23:16 UTC. Some sensors took until approximately 23:25 UTC to reconnect. What happened (all times UTC, 2026-09-19) 20:21 - A scheduled automatic upgrade began replacing the servers that receive sensor connections in our US datacenter, one at a time. ~21:50 - The network load balancer in front of those servers was not updated with the replacement servers and kept sending connections to ones that had been removed. Sensors lost their connection progressively as the upgrade advanced. 22:17 - Our sensor connectivity alert paged the on-call team. 23:13 - We forced the load balancer to resynchronize. Traffic returned to normal by 23:16. Impact US datacenter only, approximately 80 minutes. Sensors kept running on their hosts and buffered telemetry locally while disconnected. That local buffer is bounded, so hosts with higher event volume may show gaps in telemetry for this window. Detection & Response rules did not run on telemetry that was not delivered. The web application, APIs, and non-sensor data sources (adapters, cloud integrations) were not affected. Root cause A load balancer configuration setting (a 60-minute connection drain timeout) caused the component that keeps the load balancer's server list up to date to fall far behind, so the replacement servers were never registered. What we are doing The configuration that caused the delay has been corrected in all datacenters, including the US datacenter (completed 2026-09-22 04:00 UTC). We are moving the sensor load balancer to a newer design that tracks server membership directly. We are adding monitoring that tests the sensor connection path end to end, and tightening existing connectivity alerts so they fire sooner and stay open until service is fully restored. We are connecting that end-to-end check to this status page so the Endpoint Agent component reflects sensor connectivity. We apologize for the disruption.

Read the full incident report →

Started Jun 01, 2026, 11:16 PM UTC· resolved Jun 02, 2026, 12:05 AM UTC

1. investigating Jun 01, 2026, 11:16 PM UTC

   We are currently experience impact on our API endpoints and they are operating in degraded state right now. We are investigating and working to stabilize the issue. We will update as soon as we have more info.

2. resolved Jun 02, 2026, 12:05 AM UTC

   We have stabilized the API endpoints, and seeing normalization across the platform. We have root caused the issue to a logic bug in our processing of extension continuations, and have applied not only a fix, but additional controls to mitigate proliferation in such situations. We are closing the incident but will continue to monitor for the next few hours.

Read the full incident report →

Started Mar 13, 2026, 05:04 PM UTC· resolved Mar 13, 2026, 09:20 PM UTC

1. investigating Mar 13, 2026, 05:04 PM UTC

   We are currently heavily impacted by an outage with GCP Firebase. We are working with GCP support to resolve. Will have updates as soon as we are able.

2. resolved Mar 13, 2026, 09:20 PM UTC

   The underlying issues with GCP Firebase have been resolved, and we have witnessed our system fully recovered. While we will continue to monitor, we are closing the incident/status update at this time.

Read the full incident report →

Started Feb 19, 2026, 08:59 PM UTC· resolved Feb 19, 2026, 08:59 PM UTC

1. resolved Feb 19, 2026, 08:59 PM UTC

   We are currently experiencing a degraded status with our APIs. We are investigating the issue, focused on stabilization. Will update as we have more information. Feb 19, 2026 4:15 PM MST - We are stabilizing, still investigating the root cause of the issue, and still seeing some issues with the API/jwt services. Issues seem related to GCP services. We investigating with GCP on the issue. Feb 19, 2026 6:15PM MST - While we are stable, we are still chasing down some issues with GCP services. We will update as we have more info. Feb 20, 2026 8:45PM MST - All systems nominal and observed stable for some time now. Thanks for your patience. Feb 20, 2026 9:30AM MST - We are reopening this incident as we seeing lingering issues with GCP services that are impacting our service offerings. We are currently engaged with GCP Support and escalating the issue to drive resolution as quickly as possible. Feb 21, 2026 11:30AM MST - We are closing the issue. GCP suffered issues around latency that negatively impacted LC services. We have observed no issue for over 24hrs and are closing the incident. As part of this we have implemented mechanisms to mitigate the impact of issues of this type in case it happens again. Thanks for your patience.

Read the full incident report →

Started Feb 03, 2026, 09:15 PM UTC· resolved Feb 04, 2026, 04:47 PM UTC

1. investigating Feb 03, 2026, 09:15 PM UTC

   We experience unusually high load on one of the internal data systems, as a result some services, including D&R rule suppression, are impacted. Remediation effort on the way, systems began to return to normal operations.

2. resolved Feb 04, 2026, 04:47 PM UTC

   Issues resolved, all systems back to normal.

Read the full incident report →

Started Nov 14, 2025, 05:00 PM UTC· resolved Nov 14, 2025, 07:51 PM UTC

1. investigating Nov 14, 2025, 05:00 PM UTC

   We observed elevated API level that manifests as sensor list not loading for some of our users. Potential problem found, remediation applied, and we continue to monitor the situation as the services are being restored to full capacity.

2. resolved Nov 14, 2025, 07:51 PM UTC

   All systems are good to go. Remediation resolved the issue. Thanks for your time!

Read the full incident report →

Started Sep 30, 2025, 11:43 AM UTC· resolved Sep 30, 2025, 02:06 PM UTC

1. investigating Sep 30, 2025, 11:43 AM UTC

   We are currently experiencing degraded performance of systems in region Europe. We are investigating these issues, and will update as soon as we have more information.

2. resolved Sep 30, 2025, 02:06 PM UTC

   Root cause has been identified, and all systems have normalized.

Read the full incident report →

Started Sep 24, 2025, 01:40 PM UTC· resolved Sep 24, 2025, 06:04 PM UTC

1. investigating Sep 24, 2025, 01:40 PM UTC

   We are experiencing issues which impact sensors within our European data centers. Users may see issues with sensors in this region (e.g. sensors attempting to enroll, sensor telemetry ingestion, sensor tasking, etc.) We are actively investigating the issue.

2. resolved Sep 24, 2025, 06:04 PM UTC

   All systems have returned to normal operation. Thank you for your patience as we worked through this issue.

Read the full incident report →

Started Aug 20, 2025, 11:08 AM UTC· resolved Aug 20, 2025, 11:08 AM UTC

1. resolved Aug 20, 2025, 11:08 AM UTC

   At approx 5:01AM MST, we experienced a short window of outage to Firebase that impacted our web app, and API. System have since recovered. 6:13AM MST - Systems are fully recovered, and nominal.

Read the full incident report →

Started Jun 13, 2025, 03:35 PM UTC· resolved Jun 13, 2025, 03:35 PM UTC

1. resolved Jun 13, 2025, 03:35 PM UTC

   We experienced degraded systems due to residual impact from upstream provider outages from yesterday. We are in a stabilization phase, and are working towards nominal operations. 8:03 AM MST - All systems nominal at this time. Thanks for your patience.

Read the full incident report →

Started Jun 12, 2025, 06:09 PM UTC· resolved Jun 12, 2025, 06:09 PM UTC

1. resolved Jun 12, 2025, 06:09 PM UTC

   We are experiencing high impact from what appear to be outages with various services with Google Cloud. We have not ETA but will update as soon as we have more information. 1:18PM MST - We are seeing some recovery, but not considering this over yet. No official word from GCP yet. We will be heavily monitoring our systems and be working to make the recovery phase as smooth as possible. We will provide more updates as we have them. 2:31PM MST - We are starting to see some recovery of GCP services. Still not fully recovered. We are monitoring and will update as we have more information. 3:42PM MST - GCP is reporting systems normalizing, and we are seeing such on our side. We are still monitoring, but our systems are normalizing and looking solid. 6:48PM MST - Calling a close to this incident as gcp services resolved their residual issues.

Read the full incident report →

Started May 20, 2025, 03:39 PM UTC· resolved May 20, 2025, 03:59 PM UTC

1. investigating May 20, 2025, 03:39 PM UTC

   The Web App can not load. Preliminary investigation indicates possible GCP failure. Investigating more. All API, backend services, and event ingestion are operating normally. This is not impacting any backend services. Only the web app portion of our services. 9:52AM MST - This issue seems to be related to GCP services issues. We are engaging GCP to escalate this issue, and working on other methods to stabilize.

2. resolved May 20, 2025, 03:59 PM UTC

   The issue is identified and resolved.

Read the full incident report →

Started Mar 26, 2025, 04:09 PM UTC· resolved Mar 26, 2025, 07:30 PM UTC

1. investigating Mar 26, 2025, 04:09 PM UTC

   We are investigating API errors which are impacting all LimaCharlie services.

2. resolved Mar 26, 2025, 07:30 PM UTC

   3/26/2025 10:30AM MST - Our API endpoints and some internal systems were heavily stressed by unplanned load from services. We have scaled systems to relieve pressure, and are adjusting some rate limiting to reduce risk exposure. Systems are normalizing but will take some time to process out. 3/26/2025 3:30PM MST - Systems are recovered and we are closing this incident.

Read the full incident report →

Started Mar 19, 2025, 05:24 PM UTC· resolved Mar 19, 2025, 05:24 PM UTC

1. resolved Mar 19, 2025, 05:24 PM UTC

   We experience an unusually high volume of API calls in USA that over-running current scaling, causing some API calls to fail. Investigation and remediation on the way. 12:12PM MST - Systems have stabilized. We are still heavily engaged in monitoring and root causing the issue. More details as we have them. We are not considering this incident over just yet. 6:28PM - Systems have been stable for some time now. Root cause was issues that arose from migration of a backend system. Issues resolved.

Read the full incident report →

Started Dec 11, 2024, 03:18 PM UTC· resolved Dec 11, 2024, 03:18 PM UTC

1. resolved Dec 11, 2024, 03:18 PM UTC

   Summary Root Cause Analysis Report - Post-Mortem Incident Overview: On December 11th, 2024, a few LimaCharlie customers reported Blue Screen of Death (BSOD) issues affecting Windows systems running version 4.32.1 of the LC EDR Agent. The issue only affected a small subset of systems running lc:latest that were under high memory pressure. Engineering quickly identified the issue and rolled back the agent version from 4.32.1 to 4.32.0 to mitigate the problem. The incident response was swift, with engineering identifying and diagnosing the problem within approximately one hour of the first customer report. A community-wide announcement was made and remediation was implemented through a version rollback within 2.5 hours of initial report. Timeline of Events: 2024-12-10 @ 9:41am PT: LC EDR Agent version 4.32.1 is released 2024-12-11 @ 05:02am PT: Multiple customers reported isolated Blue Screen of Death (BSOD) incidents on Windows systems running the LimaCharlie EDR Agent 2024-12-11 @ 06:06am PT: LC engineering diagnosed a kernel driver defect in latest endpoint agent version 4.32.1 2024-12-11 @ 07:18am PT: Announcement posted in LC Community Slack 2024-12-11 @ 07:30am PT: LC Agent latest version rolled back to 4.32.0 Root Causes: The root cause was identified as a kernel driver bug related to improper IRQL handling when accessing paged memory with a spin lock. The issue manifested when the kernel driver (tmp_hbs_acq.sys) attempted to access pageable memory at an interrupt request level (IRQL) that was too high, triggering Windows bug check D1 (DRIVER_IRQL_NOT_LESS_OR_EQUAL). This bug was infrequent on test systems where no memory pressure was present. Actions Taken: Implement stricter IRQL validation checks in the kernel driver code to ensure proper memory access patterns Add comprehensive memory access guards around paged pool operations when holding spin locks Enhance the driver's memory management to properly handle high-pressure scenarios Implement additional automated testing specifically for IRQL-related scenarios in the kernel driver Enhance pre-release testing environment to better simulate high-memory-pressure scenarios and kernel-level interactions across diverse Windows configurations These changes will be implemented and thoroughly tested before being included in a future release. The development team will also review other areas of the kernel driver code for similar potential IRQL issues. Lessons Learned: The critical importance of thorough IRQL validation in kernel driver development to prevent system crashes The need for enhanced pre-release testing environments that better simulate real-world conditions, particularly high-memory-pressure scenarios The value of quick rollback capabilities when dealing with kernel-level components The importance of maintaining detailed version control and change documentation to quickly identify potential causes of issues The effectiveness of rapid customer communication and transparent incident handling in maintaining trust Future Recommendations: Enhance documentation and communication around the intended purpose of lc:latest tag, emphasizing its role as a testing/staging version rather than for production use Develop best practices documentation for rolling out new agent versions across customer environments, including recommended testing procedures and rollback strategies Added “Best Practices” to docs: https://docs.limacharlie.io/docs/endpoint-agent-versioning-and-upgrades#updating-endpoint-agents Update web app to display both latest and stable as current versions on the Deployed Versions page, removing UI indicators that suggest stable to be outdated. Implement a formal notification system for lc:latest version changes to help customers better track and manage testing cycles

Read the full incident report →

Started Nov 15, 2024, 06:49 PM UTC· resolved Nov 15, 2024, 07:03 PM UTC

1. investigating Nov 15, 2024, 06:49 PM UTC

   Users may see performance issues when loading sensor lists. We are aware of the issue and are working on a resolution.

2. resolved Nov 15, 2024, 07:03 PM UTC

   Performance issues should be resolved. We continue to monitor systems to ensure sustained performance.

Read the full incident report →