SignalFx US1 had 45 outages in the last 2 years totaling 977h 9m of downtime — averaging 1.8 incidents per month.
There were 45 SignalFx US1 outages since May 22, 2025 totaling 977h 9m of downtime. Each is summarised below — incident details, duration, and resolution information.
Rum API Endpoints were not available for one of our IPs, resulting in degraded service or a partial outage at first, and possibly a complete outage towards the end. This issue occurred from 4:22pm PST to 4:57pm PST
Datapoint ingest is affected , some datapoints could be dropped. Team is investigating and will provide an update shortly.
Our engineering team is currently implementing an impact mitigation configuration change. The internal investigation is still ongoing. We will share an additional update as soon as possible.
Our engineering team has completed impact mitigating configuration changes and confirmed services are recovered as of 4:55 PST. We will continue monitoring and share an update as soon as possible.
Our engineering team has confirmed that all the services recovered as of 4:55 PM PDT and remained stable during monitoring. This incident is now resolved.
We are investigating delays in processing Splunk Observability Synthetic Monitoring test results for a small subset of customers. Tests are still running, but metrics may be delayed until the issue is resolved.
We have identified and remediated an issue affecting synthetic test result ingestion for a subset of customers. We apologize for any inconvenience this may have caused and thank you for your patience. Time range of impact: June 1 20:00 UTC - June 2 15:30 UTC
Charts for customers may not be loading. App and UI availability is also impacted. We are investigating and will provide an update shortly.
Charts for customers may not be loading. App and UI availability is also impacted. We are investigating and will provide an update shortly.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the status.
This Incident has been resolved.
A degradation of the ingest path for Splunk Cloud Integration is causing data points to be dropped and lost. We are investigating the issue and will provide an update as soon as possible.
This incident has been resolved.
We are currently investigating this issue.
We are continuing to investigate this issue.
The issue has been identified and a fix is being implemented.
This incident has been resolved.
We are investigating an issue affecting Real User Monitoring (RUM) metrics across all realms. Customers may experience custom metric MTS quota limits being exceeded, resulting in metric data being dropped. RUM metric dimensions that were previously available may be missing.
The issue affecting Real User Monitoring (RUM) metrics has been identified. The fix is currently being implemented. We are actively working to restore full service and will provide further updates as progress continues.
We are continuing to work on the fix. We expect to have further updates within the next hour. We appreciate your patience as we work diligently to resolve this issue and restore full service.
We are currently continuing to test the fix for the Real User Monitoring (RUM) metrics issue. We are thoroughly validating the mitigation to ensure it resolves the issue effectively and maintains system stability. We appreciate your ongoing patience and will provide further updates as testing progresses.
We are currently continuing to test the fix for the Real User Monitoring (RUM) metrics issue. We will share an update once we have further information.
The outage impacting RUM MMS Data Drop has been resolved after internal engineering teams deployed a fix. All systems are operational and stable. Thank you for your patience and understanding.
A degradation in the performance of the Splunk APM metrics processing pipeline is causing Monitoring MetricSets to be delayed by more than five minutes. Trace data ingest is not impacted, but service, endpoint and workflow dashboards, and other charts and detectors built from Monitoring MetricSets are impacted.
We are continuing to investigate this issue.
We are continuing to investigate this issue.
We are continuing to investigate the performance degradation affecting Splunk APM Monitoring MetricSets. While trace data ingestion remains fully operational, users may experience delays of more than five minutes in dashboard, chart, and detector data. Our team is actively working to restore normal processing speeds, and we will provide further updates as more information becomes available
We are actively working to resolve the delay affecting trace monitoring metrics. While trace data ingestion remains unaffected, please note that dashboards, charts, and detectors may experience delays of over five minutes. We are working to restore normal latency and will provide further updates as they become available.
Recovery efforts for trace monitoring metrics in the US1 region are progressing, with system lag steadily decreasing. We continue to monitor the service closely as it returns to normal performance levels. No data loss has occurred.
We are continuing our recovery efforts for trace monitoring metrics in the US1 region. As system lag steadily decreases, we remain engaged in our standard monitoring procedures to ensure a full return to service. No data loss has occurred.
The issue impacting trace monitoring metrics in the US1 region is now resolved. All the APM monitoring sets are now fully operational and performing as expected. We confirm that no data was lost. Thank you for your patience.
We are currently investigating the issue. New APM Data is delayed by approximately 20 minutes. There is no data loss.
Investigation is in progress. Our team is working on mitigation steps and will provide further update as soon as possible
Mitigation steps are in progress, and the situation is improving. We are continuing to monitor and will provide further updates as soon as possible
The issue has been resolved and services have returned to normal operation. Thank you for your patience.
The issue has been identified and is due to a Cloud Provider outage in the Bahrain region.
The issue has been identified as an outage affecting the Bahrain region due to a Cloud Provider outage. We recommend that users temporarily run their Synthetic Monitoring tests from an alternative location while this issue is being resolved by the Cloud Provider.
The issue has been identified as a regional cloud provider outage that is impacting our ability to maintain Synthetics testing in specific areas. Due to the uncertainty regarding the resolution timeline for this issue, we have made the decision to temporarily remove the following locations from our Synthetics testing network: Bahrain Muscat UAE We recommend reviewing your current test configurations and selecting alternative testing locations if necessary to ensure continued coverage. Our engineering team is monitoring the situation closely. We will provide further updates as more information becomes available from the cloud provider. We apologize for any inconvenience this may cause and appreciate your patience as we work to maintain the reliability of our services.
This incident has been resolved.
Cisco is aware of an ongoing cloud service provider outage in the Middle East affecting multiple cloud offers and services. As a result, O11y customers ingesting metrics and/or metadata from impacted regions may experience significant delays and/or interruptions in data delivery. Customers who do not rely on metrics or metadata from these regions are not affected. We will continue to closely monitor this issue as it unfolds.
Cisco is aware of an ongoing cloud service provider outage in the Middle East affecting multiple cloud offers and services. As a result, O11y customers ingesting metrics and/or metadata from impacted regions may experience significant delays and/or interruptions in data delivery. Customers who do not rely on metrics or metadata from these regions are not affected. We will continue to closely monitor this issue as it unfolds.
Cisco is aware of an ongoing cloud service provider outage in the Middle East affecting multiple cloud offers and services. As a result, O11y customers ingesting metrics and/or metadata from impacted regions may experience significant delays and/or interruptions in data delivery. Customers who do not rely on metrics or metadata from these regions are not affected. We will continue to closely monitor this issue as it unfolds.
Cisco is aware of an ongoing cloud service provider outage in the Middle East affecting multiple cloud offers and services. As a result, O11y customers ingesting metrics and/or metadata from impacted regions may experience significant delays and/or interruptions in data delivery. Customers who do not rely on metrics or metadata from these regions are not affected. We will continue to closely monitor this issue as it unfolds.
Monitoring over the past several days has not identified any impact to Splunk Observability Cloud services related to the previously reported cloud service provider disruption in the Middle East. Splunk Observability Cloud services continue to operate normally. We will continue to monitor the situation and will notify customers if conditions change.
We are currently investigating this issue.
We are currently investigating an issue causing failures in analytics jobs and impacting chart and detector functionality in the UI. Customers may experience degraded performance or data loading issues. Team is actively working to identify mitigation to restore full functionality. We will provide further updates as more information becomes available.
We have identified the cause of the issue impacting analytics job processing and UI functionality. Issue was triggered following a ZooKeeper master node restart during scheduled maintenance, which led to service instability and job failures. Recovery actions are in progress, including restarting affected services and stabilizing the environment. Customers may continue to experience degraded performance or data loading issues while mitigation steps are being completed. We will provide further updates as restoration progresses.
We are continuing to make progress restoring analytics job processing and UI functionality. Affected services have been restarted. While recovery is progressing, customers may still experience data delays due to backlog processing. Team is actively working on expediting recovery progress.
We are continuing restoring analytics job processing and overall UI functionality. The majority of affected services have been successfully restarted, and rolling restarts for remaining components are currently underway. Customers may continue to experience data delays as analytics and detector services process backlogs. We will provide additional updates as stability and performance are fully restored.
Recovery is complete. Charts and detectors are now loading with 100% successful displayed data. The team are actively monitoring the situation to ensure of no further issues.
We are continuing to monitor for any further issues.
We have completed the remediation actions and have successfully resolved the incident. At this time, no further action is required of you.
Customers may be experiencing errors while loading Data Management Page.
The issue has been identified and a fix is being implemented.
Our teams are actively working to restore services. Some customers may be seeing improvement at this time. We are committed to resolving the issue as quickly as possible and will provide updates as progress is made.
Restoration is still in progress. Retrying to load the Data Management Page should yield positive results for some customers. Our teams continue to work diligently to resolve the issue and will provide further updates as progress is made.
Restoration is mostly complete. Most customer attempts to load the Data Management Page should be successful. Our teams continue to monitor the situation and appreciate your patience as we finalize the resolution. Further updates will be provided if necessary.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Between 2026-01-23 at 3:30 AM PT and 2026-01-26 at 1:00 AM PT, metrics fetching from AWS Cloudwatch was degraded.
Datapoint ingest for new metric time series was affected between 2:10 AM PST and 2:45 AM PST. There was partial data loss for new metric time series during that time period. This situation is now recovered.
We are currently investigating an issue that may result in the Hosts Infra Navigator View from loading.
We are continuing to investigate this issue. We have confirmed that the issue does not impact Datapoint Ingest at this time.
Engineering teams have identified the root cause and completed recovery actions that caused a degradation of service to the Hosts Infra Navigator View.
The issue started from 10:30 PM PST on January 15, 2026. The root cause is identified and a fix is in development. Only monitoring metric sets and org metrics data ingestion is delayed, there is no data being dropped.
We are continuing to work on a fix for this issue.
This incident has been resolved. On further investigation, the ingestion delay for monitoring metric sets and org metrics was only 2 minutes during the incident.
Users login with Splunk Unified Identity may have issues accessing Splunk Observability and Log Observer.
We are continuing to investigate this issue.
Splunk Unified Identity issue has been identified. This will only affect new login session using Splunk Unified Identity, existing login sessions are unaffected. A fix is in progress.
Teams are continuing to work diligently, and progress is being made in applying the fix.
A fix has been implemented and we are monitoring the results.
The incident has been resolved; however, customers may need to clear their browser cache in order to successfully log in.
0.4% of datapoints dropped overall at peak mostly affecting new MTS creations. The issue is no longer occurring and we are monitoring.
This incident has been resolved.
Customers may be experiencing delays in some charts and detectors. Datapoint ingest is not affected. We are investigating and will provide an update shortly.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.