Alert Logic incident

US-Regions | Customers experiencing multiple issues

Major Resolved View vendor source →

Alert Logic experienced a major incident on December 7, 2021 affecting Extended Endpoint Protection and Extended Endpoint Protection and 1 more component, lasting 21h 39m. The incident has been resolved; the full update timeline is below.

Started
Dec 07, 2021, 03:57 PM UTC
Resolved
Dec 08, 2021, 01:36 PM UTC
Duration
21h 39m
Detected by Pingoru
Dec 07, 2021, 03:57 PM UTC

Affected components

Extended Endpoint ProtectionExtended Endpoint ProtectionExtended Endpoint ProtectionWeb Application FirewallWeb Application FirewallWeb Application FirewallAlert Logic ConsoleAlert Logic ConsoleAlert Logic ConsoleLog Management

Update timeline

  1. investigating Dec 07, 2021, 03:57 PM UTC

    An issue has been discovered that is affecting customers within AWS regions (US-East, US-West & EU-West). Customers in these regions may experience delays in IDS and Log processing and intermittent portal access/functionality.

  2. investigating Dec 07, 2021, 04:01 PM UTC

    We are continuing to investigate this issue.

  3. identified Dec 07, 2021, 05:05 PM UTC

    We are aware of an on-going global issue with AWS. We have performed an impact assessment of Alert Logic services and can confirm that we are collecting and storing data for IDS and Log products. Customers in US regions are not currently processing the collected data for review and escalation by our SOC. Customers in the EU region remain unaffected. Further issues can be seen in US regions when trying to interact with the Alert Logic Console, including search, dashboards and scanning. WAF, Endpoint Protection and Reporting products are currently unaffected.

  4. identified Dec 07, 2021, 05:28 PM UTC

    We are continuing to work on a fix for this issue.

  5. identified Dec 07, 2021, 05:42 PM UTC

    This incident is now confirmed to be isolated to those customers in US regions. EU and UK customers may disregard. We continue to follow service updates from AWS and will provide a further update when we confirm the issue is resolved and have validated internal services.

  6. monitoring Dec 08, 2021, 01:40 AM UTC

    AWS have declared that all impacted services are functional in US regions. Alert Logic Engineers have validated all features and services and can confirm that all systems are operational. For those customers sending IDS and Log data, it has been received and is currently being processed for analysis and escalation. Customers attempting to execute searches for recent log messages or IDS observations may not see the latest results due to the data processing backlog - this is expected to take several hours. If you have any further questions, please contact [email protected].

  7. resolved Dec 08, 2021, 01:36 PM UTC

    Following a period of monitoring, Alert Logic Engineers have confirmed that processing of the data backlog has completed successfully and all services are back to a fully operational state. This Incident is now resolved. If you have any further questions on this incident, please contact [email protected].