SignalFx US0 incident
Charts and Detectors are Intermittently Failing to Load
SignalFx US0 experienced a major incident on March 19, 2026 affecting Splunk APM Interface and Splunk Observability Cloud Web Interface, lasting 4m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 19, 2026, 04:52 PM UTC
Charts and Detectors are intermittently failing. Datapoint ingest is not affected. We are currently investigating.
- identified Mar 19, 2026, 05:28 PM UTC
Charts and Detectors are intermittently failing. Infrastructure monitoring might also see impact. Datapoint Ingest is not impacted. The issue has been identified and a mitigation fix is being implemented.
- identified Mar 19, 2026, 06:08 PM UTC
Engineering teams continue to apply the following mitigation actions: Fine tuning internal query routing to healthy MTS Store Availability Zones with capacity and limiting expensive and high volume customer queries in order to bring general stability and support the success of scale-out operations.
- monitoring Mar 19, 2026, 06:25 PM UTC
The fix has been implemented and we are monitoring the results.
- monitoring Mar 19, 2026, 06:57 PM UTC
We continue to monitor as we slowly reduce the implemented rate limits that partially mitigated the impact to ensure the environment remains stable.
- monitoring Mar 19, 2026, 07:32 PM UTC
We have successfully reverted the rate limits to their previous settings and continue to monitor for any issues.
- resolved Mar 19, 2026, 07:37 PM UTC
This incident has been resolved.