SignalFx US1 experienced a minor incident on May 8, 2025 affecting Alerting, lasting 9h 48m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified May 08, 2025, 10:34 PM UTC
Starting at 10:30a PT some small percentage of alerts may have been delayed by up to 2-3 hours. The root cause is known and a fix is being worked on.
- identified May 08, 2025, 11:41 PM UTC
We are continuing to work on a fix for this issue
- identified May 09, 2025, 12:53 AM UTC
We are continuing to work on a fix and will provide updates as more information becomes available
- identified May 09, 2025, 01:53 AM UTC
We're making steady progress on the fix and will keep you informed as more details emerge
- identified May 09, 2025, 02:51 AM UTC
Efforts to resolve the issue are ongoing, and we’ll continue to share updates along the way
- identified May 09, 2025, 03:39 AM UTC
We are continuing to work on the issue. Alert notifications are being sent out on time. A small percentage of the events behind those notifications are delayed being created and are not available to see in the user interface.
- identified May 09, 2025, 04:35 AM UTC
We are actively implementing backend improvements to address delays in event availability. This multi-step process is expected to take several hours. Engineering teams across regions are coordinating to monitor progress and ensure full functionality is restored. Our next update will be provided by 10:00 AM UTC on May 9, 2025
- monitoring May 09, 2025, 08:00 AM UTC
The system has recovered and we are continuing to monitor
- resolved May 09, 2025, 08:23 AM UTC
This incident has been resolved.