SignalFx EU0 experienced a major incident on October 11, 2024 affecting Splunk Observability Cloud Web Interface and Alerting, lasting 1d 8h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Oct 11, 2024, 06:22 PM UTC
Customers may be experiencing delays in charts and detectors that rely on property and tag updates on metric time series. Datapoint ingest is not affected. We have identified the issue and are actively working on implementing the fix.
- identified Oct 11, 2024, 07:30 PM UTC
We are continuing to make progress on a fix for this issue.
- identified Oct 11, 2024, 09:36 PM UTC
The fix has been implemented and is in the process of being deployed.
- identified Oct 11, 2024, 11:59 PM UTC
We are continuing to deploy the fix and will provide further updates as it starts taking effect.
- identified Oct 12, 2024, 02:09 AM UTC
The fix has been deployed. We are monitoring it and will continue providing updates.
- identified Oct 12, 2024, 04:16 AM UTC
We are in the process of implementing additional fixes and will continue to provide updates
- identified Oct 12, 2024, 06:12 AM UTC
Additional fixes are now implemented and are in the process of being deployed. We will continue to provide updates.
- identified Oct 12, 2024, 09:09 AM UTC
We are continuing to deploy the additional fixes and will provide further updates as they start taking effect
- identified Oct 12, 2024, 11:07 AM UTC
Additional fixes are now deployed and starting to take effect. We will continue to monitor and provide updates
- identified Oct 12, 2024, 01:47 PM UTC
The fixes continue to take effect and we expect the system to recover at a steady pace over the next few hours. We will continue to monitor and provide updates
- identified Oct 12, 2024, 04:42 PM UTC
The recovery is ongoing at a steady pace. We will continue to monitor and provide updates.
- identified Oct 12, 2024, 09:32 PM UTC
The recovery has been progressing as expected and will continue over the next few hours. We will continue to provide updates
- monitoring Oct 13, 2024, 02:07 AM UTC
The system has now recovered, and we are continuing to monitor.
- resolved Oct 13, 2024, 02:33 AM UTC
This incident has been resolved.