Splunk OnCall incident

Splunk On-Call is experiencing disruptions for Rules Engine functionality and Incident Timeline Data

Minor Resolved View vendor source →

Splunk OnCall experienced a minor incident on May 14, 2024 affecting Web Client (Portal) and Android Client - Mobile App and 1 more component, lasting 6d 23h. The incident has been resolved; the full update timeline is below.

Started
May 14, 2024, 08:56 PM UTC
Resolved
May 21, 2024, 08:18 PM UTC
Duration
6d 23h
Detected by Pingoru
May 14, 2024, 08:56 PM UTC

Affected components

Web Client (Portal)Android Client - Mobile AppiOS Client - Mobile AppAlert Rules Engine

Update timeline

  1. investigating May 14, 2024, 08:56 PM UTC

    Please Note: No disruption to Incident creation and notifications out to on-call end-users have been detected. Impacted Functionality: -Increased RegEx Rules Engine rule timeouts - Disabled Rules Engine Rules may be reenabled to resume functionality. -UI and Mobile App Timeline data (incident cards) - Alert details may still be found via the On-Call API -UI and Mobile App empty Incident alert details - Alert details may still be found via the On-Call API or our Reporting features: Incident Frequency, On-Call Review, and Response Metrics. Please be advised that our engineering teams are working on this issue with urgency. We will provide updates as soon as we learn more information. We apologize for this inconvenience. Please reach out to On-Call Support with any questions: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

  2. investigating May 14, 2024, 08:57 PM UTC

    We are continuing to investigate this issue.

  3. investigating May 15, 2024, 10:03 PM UTC

    We continue to investigate this issue. Please note that only a subset of instances are affected by this issue and no disruption to incident creation and notifications out to on-call end-users have been detected. Apologies for any inconvenience. Please reach out to On-Call Support with any questions: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

  4. investigating May 16, 2024, 10:35 PM UTC

    Please Note: No disruption to Incident creation and notifications out to on-call end-users have been detected. Impacted Functionality: -Increased RegEx Rules Engine rule timeouts - Disabled Rules Engine Rules may be reenabled to resume functionality. -UI and Mobile App Timeline data (incident cards) - Alert details may still be found via the On-Call API -UI and Mobile App empty Incident alert details - Alert details may still be found via the On-Call API or our Reporting features: Incident Frequency, On-Call Review, and Response Metrics. Please be advised that our engineering teams are working urgently on this issue. We will provide updates as soon as we learn more information. We apologize for this inconvenience. Please reach out to On-Call Support with any questions: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

  5. monitoring May 17, 2024, 07:59 PM UTC

    The issue has been identified as upstream from the On-Call platform but impacting our CPU utilization. Mitigation is underway and we are now monitoring the results. Our engineering teams continue to work on this issue with urgency. We will provide updates as soon as we learn more information. Please Note: No disruption to Incident creation and notifications out to on-call end-users have been detected. Stay tuned for further updates as we work towards a complete resolution. We apologize for this inconvenience. Please reach out to On-Call Support with any questions: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/

  6. resolved May 21, 2024, 08:18 PM UTC

    This incident has been resolved. We apologize for this inconvenience. Please reach out to On-Call Support with any questions: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/