Splunk OnCall incident

Intermittent timeline event loss affecting approximately 8% of organizations

Minor Resolved View vendor source →

Splunk OnCall experienced a minor incident on May 10, 2024 affecting Web Client (Portal) and Android Client - Mobile App and 1 more component, lasting 12h 4m. The incident has been resolved; the full update timeline is below.

Started
May 10, 2024, 02:54 AM UTC
Resolved
May 10, 2024, 02:58 PM UTC
Duration
12h 4m
Detected by Pingoru
May 10, 2024, 02:54 AM UTC

Affected components

Web Client (Portal)Android Client - Mobile AppiOS Client - Mobile App

Update timeline

  1. investigating May 10, 2024, 02:54 AM UTC

    Splunk On-Call is currently investigating an issue resulting in missing timeline events for approximately 8% of our customer base. These events display ingested alerts and incident actions such as paging and acknowledgments. The actual paging functionality is unaffected. As a workaround, please leverage the Splunk On-Call Response Metrics, On-Call Review, and Incident Frequency reports to obtain your alert event information. We apologize for this inconvenience. Please reach out to On-Call Support with any questions. https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

  2. monitoring May 10, 2024, 04:25 AM UTC

    A fix has been implemented and we are monitoring the results

  3. resolved May 10, 2024, 02:58 PM UTC

    This incident has been resolved. Timeline events are no longer missing, all systems fully operational.