Is SignalFx US0 down?

A degradation in the performance of the Splunk APM trace processing pipeline is causing Troubleshooting MetricSets and Monitoring MetricSets to be delayed by more than five minutes. As a result, the APM Troubleshooting experience, service maps and Tag Spotlight do not have access to the most recent data and may fail or be slow to load. The processing of metrics for Business Workflows, which also depends on this pipeline, are equally delayed. Trace data ingest is not impacted at this time; service-level and endpoint-level Monitoring MetricSets and the detectors built from them are also not impacted.

We are continuing to investigate this issue.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

This incident has been resolved.

Customers may be experiencing delays in some charts and detectors. Datapoint ingest is not affected. We are investigating and will provide an update shortly.

Lagging continues and has caused some charts and dashboards to fail to load.

We have scaled up to reduce lag but are are still investigating the root cause.

A degradation in the performance of the Splunk APM metrics processing pipeline is causing Monitoring MetricSets and Troubleshooting MetricSets to be delayed by more than five minutes. Trace data ingest is not impacted, but service, endpoint and workflow dashboards, and other charts and detectors built from Monitoring MetricSets are impacted.

A fix has been implemented and we are monitoring the results.

This incident has been resolved.

We experienced a spike in latency with charts and dashboard metrics starting at 11:51 am PT and it resolved at 12:09 PT. The issue has since been resolved but we are currently monitoring the end user latency.

This incident has been resolved.

Charts and dashboards for a subset of customers may be intermittently failing or loading slowly. Datapoint ingest is not affected. We are investigating and will provide an update shortly.

We are continuing to investigate this issue.

Our teams have identified the cause of the issue and are actively working to resolve the impact. We are currently experiencing high latency in alert notifications, which may delay the delivery of alerts. We appreciate your patience as we work to restore normal service as quickly as possible.

There is no change in the current status. Our teams continue to monitor the situation closely and are actively working to resolve the issue. We appreciate your continued patience and will provide further updates as soon as there are any changes.

Customer impact is no longer observed for the issue causing charts to load slowly. Our teams have implemented mitigation steps and continue to monitor the environment closely. We will provide another update if there are any changes.

Charts are now loading as expected, and customer impact is no longer observed.

Charts for a subset of customers may not be loading. Detectors and Datapoint ingest is not affected. We are actively mitigating it and will provide an update shortly.

Charts for a subset of customers may not be loading. Detectors and Datapoint ingest is not affected. We are in the process of actively resolving the issue. We will provide an update shortly

Charts for a subset of customers may not be loading. Detectors and Datapoint ingest is not affected. We are in the process of actively resolving the issue and should provide an update soon

Mitigation actions have been completed, and we are observing recovery across affected customer environments. Charts should now be loading as expected. Detectors and datapoint ingestion were not impacted during this event. We will continue monitoring to confirm full recovery and stability before declaring the incident resolved.

This incident has been resolved.