Panther Labs incident

Scheduled queries failing for a single customer

Major Resolved View vendor source →

Panther Labs experienced a major incident on September 20, 2021 affecting Detections (Rules and Policies), lasting 5h 15m. The incident has been resolved; the full update timeline is below.

Started
Sep 20, 2021, 03:49 PM UTC
Resolved
Sep 20, 2021, 09:04 PM UTC
Duration
5h 15m
Detected by Pingoru
Sep 20, 2021, 03:49 PM UTC

Affected components

Detections (Rules and Policies)

Update timeline

  1. investigating Sep 20, 2021, 03:49 PM UTC

    We are investigating an issue where Scheduled queries are failing for a single customer. The customer has been notified and we are investigating further.

  2. identified Sep 20, 2021, 04:02 PM UTC

    The issue has been identified and an initial fix is being implemented. The fix will be implemented in a new patch version for the customer in 1.22.3 instead of waiting for 1.24.1.

  3. identified Sep 20, 2021, 07:20 PM UTC

    Fix has been reviewed and merged. The patch release will be published and deployed soon.

  4. monitoring Sep 20, 2021, 08:48 PM UTC

    The fix has been deployed to the one impacted customer. We are monitoring the results.

  5. resolved Sep 20, 2021, 09:04 PM UTC

    This incident has been resolved.