Splunk Observability Cloud JP0 incident

Alerts being sent for some detector rules previously disabled via API

Notice Resolved View vendor source →

Splunk Observability Cloud JP0 experienced a notice incident on February 9, 2024, lasting 4h 17m. The incident has been resolved; the full update timeline is below.

Started
Feb 09, 2024, 08:18 PM UTC
Resolved
Feb 10, 2024, 12:36 AM UTC
Duration
4h 17m
Detected by Pingoru
Feb 09, 2024, 08:18 PM UTC

Update timeline

  1. identified Feb 09, 2024, 08:18 PM UTC

    We have identified a bug that is causing alerts to be sent for detector rules that have been previously disabled via the API. This issue does not impact detectors that have rules disabled via the UI. Starting January 18, some detectors that had rules disabled via the API (such as through Terraform) continued to send alerts. We are currently rolling out a fix to production systems and expect to resolve the issue within the next 3 hours.

  2. resolved Feb 10, 2024, 12:36 AM UTC

    This incident has been resolved.