SignalFx US1 experienced a minor incident on March 25, 2026 affecting Splunk APM Monitoring MetricSets and Splunk APM Troubleshooting MetricSets and 1 more component, lasting 6h 49m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 25, 2026, 07:04 PM UTC
A degradation in the performance of the Splunk APM data ingestion pipeline is causing the processing and storage of raw trace data to be delayed by more than fifteen minutes. No data is being lost at this time and MetricSets are not impacted but the most recent data may not be available in trace search results.
- identified Mar 25, 2026, 08:23 PM UTC
We are actively working to mitigate the performance degradation impacting the Splunk APM data ingestion pipeline. Efforts are underway to stabilize the environment and reduce delays in processing raw trace data. Customers may continue to experience delays in trace data availability, while MetricSets remain unaffected. We will provide further updates as mitigation progresses and system performance improves.
- identified Mar 25, 2026, 09:56 PM UTC
We are continuing to work on mitigating the issue and stabilizing the environment. Customers may still experience delays of more than 15 minutes in trace data availability. In addition, Troubleshooting MetricSets (TMS) and Monitoring MetricSets (MMS) are also experiencing similar delays. We will provide further updates as progress continues.
- identified Mar 25, 2026, 11:16 PM UTC
After applying some mitigations, they have not yielded the intended results. We continue to work on mitigating the issue and stabilizing the environment. Customers may still experience delays of more than 15 minutes in trace data availability. In addition, Troubleshooting MetricSets (TMS) and Monitoring MetricSets (MMS) are also experiencing similar delays. We will provide further updates as progress continues.
- identified Mar 25, 2026, 11:48 PM UTC
We are actively mitigating the issue and monitoring as delays improve. Customers may still experience delays of over 15 minutes in trace data, as well as in Troubleshooting and Monitoring MetricSets. We will share further updates as progress is made.
- identified Mar 26, 2026, 12:09 AM UTC
We are continuing to mitigate the issue and are seeing gradual improvement; customers may still experience delays of over 15 minutes in data processing and availability, and we will provide further updates as recovery progresses
- identified Mar 26, 2026, 12:28 AM UTC
We are seeing gradual recovery as mitigation efforts take effect; customers may continue to experience delays in data processing and availability, though performance is steadily improving
- monitoring Mar 26, 2026, 12:46 AM UTC
The latency issues have been resolved, and we are now monitoring the service to ensure continued stability. Thank you for your patience
- resolved Mar 26, 2026, 01:54 AM UTC
The incident has been resolved, and data processing performance is now stable. Thank you for your patience.