SignalFx US1 Outage History
SignalFx US1 is up right nowPingoru has recorded 56 incidents for SignalFx US1: 54 in the last 12 months and 15 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 3 | 165 of 720 (23%) |
| May 2026 | 4 | 28 of 744 (4%) |
| June 2026 | 4 | 26 of 720 (4%) |
| July 2026 | 7 | 7 of 744 (1%) |
| August 2026 | 3 | 8 of 744 (1%) |
| September 2026 | 6 | 12 of 720 (2%) |
| October 2026 | 5 | 22 of 588 (4%) |
Incidents, newest first
Notice October 23, 2026 Degraded AWS CloudWatch Metadata Integration 15h 22m Splunk CloudWatch Metric Streams
Started Oct 23, 2026, 05:49 AM UTC· resolved Oct 23, 2026, 09:12 PM UTC
- investigating Oct 23, 2026, 07:33 AM UTC
We are investigating an issue with the AWS CloudWatch integration. Some customers may see missing or outdated resource tags on AWS metrics, and metric stream configuration changes may not be applied. Metric ingestion is not impacted. We will provide an update as soon as more information is available.
- identified Oct 23, 2026, 09:15 AM UTC
We have identified that the degraded AWS CloudWatch integration is caused by an ongoing AWS disruption in the me-central-1 (UAE) region, including AWS CloudWatch and the Resource Groups Tagging API. Customers with AWS resources in this region may see missing or outdated resource tags on AWS metrics, and metric stream configuration changes may not be applied. Metric ingestion is not impacted. AWS has not provided a recovery timeline and recommends migrating to another region.
- resolved Oct 23, 2026, 09:12 PM UTC
Splunk has confirmed that the degraded AWS CloudWatch integration in the me-central-1 (UAE) region is caused by an ongoing AWS regional disruption outside of Splunk's control. AWS has not provided a recovery timeline and recommends migrating affected resources to another region. Metric ingestion is not impacted. We are identifying affected customers and will reach out individually to discuss next steps. If you have immediate concerns, please contact your account team.
Major October 22, 2026 Splunk APM Trace Data Ingestion Delayed 1h 34m Splunk APM Ingest, Splunk APM Trace Data
Started Oct 22, 2026, 08:30 PM UTC· resolved Oct 22, 2026, 10:05 PM UTC
- investigating Oct 22, 2026, 09:56 PM UTC
A degradation in the performance of the Splunk APM data ingestion pipeline is causing the processing and storage of raw trace data to be delayed by more than five minutes. No data is being lost at this time and MetricSets are not impacted but the most recent data may not be available in trace search results.
- identified Oct 22, 2026, 09:58 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Oct 22, 2026, 10:01 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Oct 22, 2026, 10:10 PM UTC
This incident has been resolved.
Minor October 15, 2026 Splunk Observability Cloud Web Interface Degradation 2h 23m Splunk Observability Cloud Web Interface
Started Oct 15, 2026, 06:57 PM UTC· resolved Oct 15, 2026, 09:20 PM UTC
- investigating Oct 15, 2026, 06:57 PM UTC
We are investigating a degradation of the Splunk Observability Cloud web application for some customers. Data ingest is not impacted at this time. We will provide an update as soon as possible.
- investigating Oct 15, 2026, 08:00 PM UTC
Our engineering teams are actively troubleshooting the identified performance problems in the Splunk Observability Web UI, we will share a new update as soon as possible.
- investigating Oct 15, 2026, 09:04 PM UTC
Our engineering teams are actively troubleshooting the identified performance problems in the Splunk Observability Web UI, we will share a new update as soon as possible.
- monitoring Oct 15, 2026, 09:11 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Oct 15, 2026, 09:20 PM UTC
This incident has been resolved.
Major October 11, 2026 Charts not Loading for Some Customers 1h 25m Splunk Observability Cloud Web Interface
Started Oct 11, 2026, 04:09 AM UTC· resolved Oct 11, 2026, 05:34 AM UTC
- investigating Oct 11, 2026, 04:09 AM UTC
Charts for a subset of customers may not be loading. Datapoint ingest is not affected. We are investigating and will provide an update shortly.
- monitoring Oct 11, 2026, 04:24 AM UTC
Charts and dashboards are in a healthy state and we are monitoring the recovery.
- resolved Oct 11, 2026, 05:34 AM UTC
This incident has been resolved.
Minor October 3, 2026 Splunk Observability Synthetic Metric may be missing 57m Splunk Synthetics Monitoring Results processing
Started Oct 03, 2026, 02:20 AM UTC· resolved Oct 03, 2026, 03:18 AM UTC
- investigating Oct 03, 2026, 03:02 AM UTC
We are investigating delays in processing Splunk Observability Synthetic Monitoring test results. Synthetic Monitoring tests are operational but metrics may be missing until the issue is resolved.
- investigating Oct 03, 2026, 03:03 AM UTC
We are continuing to investigate this issue.
- identified Oct 03, 2026, 03:03 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Oct 03, 2026, 03:13 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Oct 03, 2026, 03:18 AM UTC
This incident has been resolved.
Minor September 29, 2026 Charts and Detectors Delayed 3h 4m Splunk Observability Cloud Web Interface
Started Sep 29, 2026, 03:02 PM UTC· resolved Sep 29, 2026, 06:07 PM UTC
- investigating Sep 29, 2026, 03:02 PM UTC
Some customers may be experiencing delays in some charts and detectors. Datapoint ingest is not affected. We are investigating and will provide an update shortly.
- investigating Sep 29, 2026, 03:54 PM UTC
We continue investigating the delays experienced in some charts and detectors. Our engineering teams continue troubleshooting and will share an update shortly.
- monitoring Sep 29, 2026, 04:10 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Sep 29, 2026, 05:26 PM UTC
Service recovery has been confirmed. We will continue to monitor system performance closely and share updates accordingly.
- resolved Sep 29, 2026, 06:07 PM UTC
Our engineering team has confirmed the fix deployed has stabilized impacted services across US1 region. This incident is now considered resolved.
Notice September 27, 2026 Splunk APM Troubleshooting MetricSets Delayed –
Started Sep 27, 2026, 12:45 AM UTC· resolved Sep 27, 2026, 12:45 AM UTC
- resolved Sep 27, 2026, 02:17 AM UTC
From 5:45pm to 6:20pm PT, degradation in the performance of the Splunk APM trace processing pipeline was causing Troubleshooting MetricSets to be delayed by more than five minutes. As a result, the APM Troubleshooting experience, service maps and Tag Spotlight do not have access to the most recent data. The processing of metrics for Business Workflows, which also depends on this pipeline, are equally delayed. Trace data ingest is not impacted at this time; service-level and endpoint-level Monitoring MetricSets and the detectors built from them are also not impacted.
Minor September 18, 2026 Potential Login Issues –
Started Sep 18, 2026, 03:00 PM UTC· resolved Sep 18, 2026, 03:00 PM UTC
- resolved Sep 18, 2026, 03:40 PM UTC
Some of the customer are seeing login failures due to certificate errors. Customers that are already logged are not impacted. The issue has been resolved.
Minor September 15, 2026 Datapoints Being Dropped in Archival tier 6h 46m Datapoint Ingest
Started Sep 15, 2026, 02:41 PM UTC· resolved Sep 15, 2026, 09:28 PM UTC
- identified Sep 15, 2026, 02:41 PM UTC
Beginning at 5:50 AM PDT, datapoint ingestion for the archival tier was impacted, resulting in a small number of datapoints not being accepted. The impact was limited to the archival tier. Detectors and charts were not affected. We have implemented remediation and are actively monitoring the system to ensure continued stability. We will provide updates every 15 minutes.
- identified Sep 15, 2026, 02:52 PM UTC
As of 7:48 AM PDT, we are no longer observing any data being dropped. The system is continuing to recover, and we are actively monitoring it to ensure continued stability.
- monitoring Sep 15, 2026, 02:54 PM UTC
The system is continuing to recover, and we are actively monitoring it to ensure continued stability.
- monitoring Sep 15, 2026, 03:33 PM UTC
The system is showing steady progress in its recovery. Our teams remain actively engaged, closely observing system metrics to ensure continued stability
- monitoring Sep 15, 2026, 04:50 PM UTC
We have successfully restored datapoint ingestion and are seeing steady progress as the data backlog is processed. We are actively monitoring this process to ensure full service functionality returns as quickly as possible.
- monitoring Sep 15, 2026, 05:59 PM UTC
We continue monitoring to ensure full service functionality.
- monitoring Sep 15, 2026, 06:58 PM UTC
We are continuing to monitor the system to ensure ongoing stability. Our teams remain actively engaged in observing performance metrics to confirm that all operations are functioning as expected.
- monitoring Sep 15, 2026, 08:17 PM UTC
To accelerate the recovery process, we are currently adding additional capacity to our clusters. Our teams remain actively engaged and are continuing to monitor system performance as it stabilizes.
- resolved Sep 15, 2026, 09:28 PM UTC
The system has recovered and we ingesting archived data with no delay.
Minor September 12, 2026 Datapoints Being Dropped 1h 5m Datapoint Ingest
Started Sep 12, 2026, 04:32 PM UTC· resolved Sep 12, 2026, 05:37 PM UTC
- investigating Sep 12, 2026, 04:32 PM UTC
Datapoint ingest is affected and we are dropping datapoints some datapoints from 8:52 PDT. We are investigating and will provide an update every 15 mins.
- monitoring Sep 12, 2026, 04:55 PM UTC
Between 8:52 PDT and 9:47 PDT, a small number of data points for new time series were not accepted. As a result, new time series may have experienced a delay before becoming available in charting and detectors. Remediation is in place, and we are continuing to monitor the system to ensure stability.
- resolved Sep 12, 2026, 05:37 PM UTC
The issue has been resolved, and datapoint ingestion is operating normally. We have confirmed that the system remains stable following remediation. This incident is now resolved.
Minor September 9, 2026 Splunk APM Trace Data Being Dropped 37m Splunk APM Ingest, Datapoint Ingest
Started Sep 09, 2026, 12:12 AM UTC· resolved Sep 09, 2026, 12:50 AM UTC
- investigating Sep 09, 2026, 12:12 AM UTC
A degradation of the trace data ingest path for Splunk APM is causing some trace spans to be dropped and lost. We are investigating the issue and will provide an update as soon as possible.
Critical August 28, 2026 Splunk Observability AI Assistant Unavailable 1h 25m AI Assistant
Started Aug 28, 2026, 06:01 PM UTC· resolved Aug 28, 2026, 07:26 PM UTC
- investigating Aug 28, 2026, 06:01 PM UTC
We are investigating a major outage of the Splunk Observability AI Assistant.
- identified Aug 28, 2026, 06:20 PM UTC
The issue has been identified as an Azure OpenAI outage and we are proceeding with a workaround.
- monitoring Aug 28, 2026, 06:25 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Aug 28, 2026, 07:26 PM UTC
Monitoring phase complete. The issue has been resolved.
Minor August 1, 2026 Delayed notifications on us1 3h 45m Alerting
Started Aug 01, 2026, 02:35 PM UTC· resolved Aug 01, 2026, 06:21 PM UTC
- investigating Aug 01, 2026, 02:35 PM UTC
Problems with processing alerts which are leading to delayed notifications. We are currently investigating the issue.
- investigating Aug 01, 2026, 03:33 PM UTC
We are currently investigating elevated notification latency. Customers may experience delays in receiving alert notifications. We are actively working to resolve this issue and will provide updates as we make progress. We appreciate your patience.
- identified Aug 01, 2026, 03:55 PM UTC
We have identified and are actively resolving the issue causing delayed alert notifications. The fix has been implemented and is already showing significant improvement in notification delivery times. While some notifications that were queued before our fix may still experience minor delays as the system catches up, new notifications are being delivered normally. We appreciate your patience and will continue monitoring the situation closely.
- identified Aug 01, 2026, 05:09 PM UTC
We are actively working on resolving the issue. Customers may continue to experience delays in notification delivery. This includes newly generated notifications, which are being processed behind earlier events in the queue. Team is actively investigating the root cause and working to clear the backlog of in-queue notifications. As a temporary mitigation measure, we have disabled image previews in notifications to improve processing speed. Notifications will be text-only for now. We appreciate your patience and will provide further updates as progress is made.
- monitoring Aug 01, 2026, 05:31 PM UTC
We have implemented a fix to address the notification delays. Our monitoring shows significant improvement in notification delivery times, and we are closely tracking performance to ensure the issue is fully resolved. Thank you for your patience as we work to restore full service.
- resolved Aug 01, 2026, 06:21 PM UTC
The notification delivery delays have been fully resolved. All queued notifications have been processed and the system is now operating normally.
Major July 31, 2026 Splunk APM Monitoring MetricSets Dropped 1h 42m Splunk APM Interface, Splunk APM Monitoring MetricSets
Started Jul 31, 2026, 02:51 AM UTC· resolved Jul 31, 2026, 04:33 AM UTC
- investigating Jul 31, 2026, 02:51 AM UTC
A degradation in the performance of the Splunk APM metrics processing pipeline is causing Monitoring MetricSets to be delayed by more than five minutes. Trace data ingest is not impacted, but service, endpoint and workflow dashboards, and other charts and detectors built from Monitoring MetricSets are impacted.
- investigating Jul 31, 2026, 02:54 AM UTC
We are continuing to investigate the issue.
- investigating Jul 31, 2026, 03:24 AM UTC
We are continuing to investigate this issue.
- investigating Jul 31, 2026, 03:44 AM UTC
We are continuing to investigate this issue.
- identified Jul 31, 2026, 04:07 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Jul 31, 2026, 04:08 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jul 31, 2026, 04:33 AM UTC
This incident has been resolved.
Notice July 24, 2026 Small amount of Datapoints Being Dropped 1h 47m
Started Jul 24, 2026, 09:00 PM UTC· resolved Jul 24, 2026, 10:47 PM UTC
- resolved Jul 24, 2026, 09:54 PM UTC
Datapoint ingest for new or reactivated metric time series was affected between 2:28 PM PST to 2:47 PM PST. There was partial data loss for new metric time series during that time period. This situation is now recovered.
Critical July 14, 2026 Splunk Observability Cloud Web Interface Unavailable 1h 13m Splunk Observability Cloud Web Interface
Started Jul 14, 2026, 11:25 AM UTC· resolved Jul 14, 2026, 12:38 PM UTC
- investigating Jul 14, 2026, 11:25 AM UTC
Starting at 4a PT, we are investigating a major outage of the Splunk Observability Cloud web application. Data ingest is not impacted at this time. We will provide an update as soon as possible.
- investigating Jul 14, 2026, 11:58 AM UTC
We are continuing to investigate the issue.
- investigating Jul 14, 2026, 12:13 PM UTC
We are continuing to investigate the issue.
- monitoring Jul 14, 2026, 12:28 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jul 14, 2026, 12:38 PM UTC
This incident has been resolved.
Notice July 8, 2026 GCP Cloud Monitoring metrics not fetched in US1 –
Started Jul 08, 2026, 05:00 AM UTC· resolved Jul 08, 2026, 05:00 AM UTC
- resolved Jul 08, 2026, 10:52 AM UTC
Between 1a PDT and 2:50a PDT, any GCP integration that had the "Sync all (dynamically) listed projects" setting enabled had some of the Cloud Monitoring data from GCP dropped. This issue has been resolved. Customers may see gaps in charts and detectors that use affected GCP Cloud Monitoring metrics. AWS CloudWatch and Azure Monitor metrics are not impacted.
Minor July 4, 2026 MetricTimeSeries Creation Delayed 1h 32m Datapoint Ingest
Started Jul 04, 2026, 05:07 PM UTC· resolved Jul 04, 2026, 06:39 PM UTC
- investigating Jul 04, 2026, 05:07 PM UTC
Customers may experience delays in seeing newly created metric time series. Datapoint ingest is not affected for existing time series. We are investigating and will provide an update shortly.
- resolved Jul 04, 2026, 06:39 PM UTC
The issue causing delays in newly created metric time series has been resolved. Metric time series creation is operating normally. Datapoint ingestion for existing time series was not affected.
Notice July 1, 2026 A partial outage of Splunk APM Tag Spotlight – Splunk APM Tag Spotlight
Started Jul 01, 2026, 03:08 PM UTC· resolved Jul 01, 2026, 03:08 PM UTC
- resolved Jul 01, 2026, 03:08 PM UTC
We have confirmed that the issue affecting Tag Spotlight has been resolved. Customers may have experienced limited functionality within Tag Spotlight beginning at approximately 4:30 PM PT on June 30. Full functionality was restored at 7:20 AM PT on July 1.
Minor June 25, 2026 Logs correlation experience from a APM TraceView page was broken. 17h 15m
Started Jun 25, 2026, 12:00 PM UTC· resolved Jun 26, 2026, 05:15 AM UTC
- resolved Jun 26, 2026, 07:37 AM UTC
Due to a regression in a recent release, users navigating from Logs Related Content on the APM TraceView page to Log Observer were redirected back to APM TraceView instead. This impacted the Logs Correlation experience for customers across all realms, except US3 and Gov. The change has been reverted to restore the expected navigation experience. Product availability for APM and Log Observer was not impacted; the issue was limited to the Logs Correlation workflow from TraceView. Incident Start Time: 05:09 PDT, June 25, 2026 Incident End Time: 22:15 PDT, June 25, 2026
Minor June 11, 2026 RUM API Endpoints Experienced Partial Downtime 2h 15m
Started Jun 11, 2026, 09:00 AM UTC· resolved Jun 11, 2026, 11:15 AM UTC
- resolved Jun 12, 2026, 12:03 AM UTC
Rum API Endpoints were not available for one of our IPs, resulting in degraded service or a partial outage at first, and possibly a complete outage towards the end. This issue occurred from 4:22pm PST to 4:57pm PST
Minor June 3, 2026 Datapoint ingest is affected 2h 18m Datapoint Ingest
Started Jun 03, 2026, 10:49 PM UTC· resolved Jun 04, 2026, 01:08 AM UTC
- investigating Jun 03, 2026, 10:49 PM UTC
Datapoint ingest is affected , some datapoints could be dropped. Team is investigating and will provide an update shortly.
- investigating Jun 03, 2026, 11:25 PM UTC
Our engineering team is currently implementing an impact mitigation configuration change. The internal investigation is still ongoing. We will share an additional update as soon as possible.
- monitoring Jun 04, 2026, 12:05 AM UTC
Our engineering team has completed impact mitigating configuration changes and confirmed services are recovered as of 4:55 PST. We will continue monitoring and share an update as soon as possible.
- resolved Jun 04, 2026, 01:08 AM UTC
Our engineering team has confirmed that all the services recovered as of 4:55 PM PDT and remained stable during monitoring. This incident is now resolved.
Minor June 2, 2026 Splunk Observability Synthetic Monitoring Results Processing delayed 4h Splunk Synthetics Monitoring Results processing
Started Jun 02, 2026, 12:50 PM UTC· resolved Jun 02, 2026, 04:51 PM UTC
- investigating Jun 02, 2026, 12:50 PM UTC
We are investigating delays in processing Splunk Observability Synthetic Monitoring test results for a small subset of customers. Tests are still running, but metrics may be delayed until the issue is resolved.
- resolved Jun 02, 2026, 04:51 PM UTC
We have identified and remediated an issue affecting synthetic test result ingestion for a subset of customers. We apologize for any inconvenience this may have caused and thank you for your patience. Time range of impact: June 1 20:00 UTC - June 2 15:30 UTC
Critical May 29, 2026 Charts not Loading, App and UI availability is also impacted 41m Splunk APM Interface, Splunk Observability Cloud Web Interface, Splunk Observability Cloud API
Started May 29, 2026, 05:24 AM UTC· resolved May 29, 2026, 06:06 AM UTC
- investigating May 29, 2026, 05:24 AM UTC
Charts for customers may not be loading. App and UI availability is also impacted. We are investigating and will provide an update shortly.
- investigating May 29, 2026, 05:40 AM UTC
Charts for customers may not be loading. App and UI availability is also impacted. We are investigating and will provide an update shortly.
- identified May 29, 2026, 05:43 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring May 29, 2026, 05:56 AM UTC
A fix has been implemented and we are monitoring the status.
- resolved May 29, 2026, 06:06 AM UTC
This Incident has been resolved.
Minor May 11, 2026 AWS CloudWatch metrics dropped 17h 4m Cloud Integrations
Started May 11, 2026, 10:40 PM UTC· resolved May 12, 2026, 03:44 PM UTC
- investigating May 12, 2026, 10:38 AM UTC
A degradation of the ingest path for Splunk Cloud Integration is causing data points to be dropped and lost. We are investigating the issue and will provide an update as soon as possible.
- resolved May 12, 2026, 03:44 PM UTC
This incident has been resolved.