SignalFx US0 incident

O11y APM Trace Lookups failing between approximately 6/17 5:30am PT - 7:30am PT

Major Resolved View vendor source →

SignalFx US0 experienced a major incident on June 17, 2026 affecting Splunk APM Trace Data, lasting 5h 32m. The incident has been resolved; the full update timeline is below.

Started
Jun 17, 2026, 03:37 PM UTC
Resolved
Jun 17, 2026, 09:09 PM UTC
Duration
5h 32m
Detected by Pingoru
Jun 17, 2026, 03:37 PM UTC

Affected components

Splunk APM Trace Data

Update timeline

  1. investigating Jun 17, 2026, 03:37 PM UTC

    We are investigating why Trace Lookups are failing for traces created between approximately 5:30am PT - 7:30am PT. Monitoring MetricSets and Troubleshooting MetricSets are not impacted. Metrics are still available. We are investigating the issue.

  2. investigating Jun 17, 2026, 04:43 PM UTC

    We are still investigating the issue.

  3. investigating Jun 17, 2026, 05:56 PM UTC

    We are continuing to investigate this issue.

  4. identified Jun 17, 2026, 06:35 PM UTC

    The issue affecting Trace Lookup searchability for traces generated between approximately 5:30 AM and 7:30 AM PT on June 17 has been identified. This occurred during an approved maintenance window associated with repartitioning and scaling activities. Customers may be unable to search for traces generated during this timeframe. Trace ingestion and processing are operating normally, and traces generated outside of the affected window remain searchable. Our team has developed a fix and is currently validating it through internal testing. We expect validation to complete within approximately two hours and will provide another update once the fix is ready for deployment or as additional information becomes available.

  5. identified Jun 17, 2026, 08:07 PM UTC

    We continue to make progress toward resolution. The fix is currently moving through the implementation process and will be rolled out once all deployment steps have been completed. We will provide another update as progress continues.

  6. resolved Jun 17, 2026, 09:09 PM UTC

    The fix has been successfully deployed, and we have confirmed that the issue affecting Trace Lookup searchability for traces generated between approximately 5:30 AM and 7:30 AM PT on June 17 has been resolved. Customers should now be able to search affected traces as expected.