Splunk Observability Cloud JP0 incident

Splunk APM Trace Data Being Dropped

Major Resolved View vendor source →

Splunk Observability Cloud JP0 experienced a major incident on October 30, 2023 affecting Splunk APM Monitoring MetricSets and Splunk APM Troubleshooting MetricSets, lasting 55m. The incident has been resolved; the full update timeline is below.

Started
Oct 30, 2023, 06:37 PM UTC
Resolved
Oct 30, 2023, 07:32 PM UTC
Duration
55m
Detected by Pingoru
Oct 30, 2023, 06:37 PM UTC

Affected components

Splunk APM Monitoring MetricSetsSplunk APM Troubleshooting MetricSets

Update timeline

  1. investigating Oct 30, 2023, 06:58 PM UTC

    A degradation of the trace data ingest path for Splunk APM is causing trace spans to be dropped and lost. We are investigating the issue and will provide an update as soon as possible.

  2. investigating Oct 30, 2023, 07:08 PM UTC

    We are continuing to investigate this issue.

  3. identified Oct 30, 2023, 07:18 PM UTC

    The issue has been identified and a fix is being implemented.

  4. monitoring Oct 30, 2023, 07:21 PM UTC

    A fix has been implemented and we are monitoring the results.

  5. resolved Oct 30, 2023, 07:32 PM UTC

    This incident has been resolved.