SignalFx US0 Outage History

The Scheduled Maintenance for US0 Splunk APM Span Firehose Optimization has been extended while we continue to monitor the improvements.

A fix has been implemented and we are monitoring the results.

We experienced a spike in latency with charts and dashboard metrics starting at 11:51 am PT and it resolved at 12:09 PT. The issue has since been resolved but we are currently monitoring the end user latency.

Customer impact is no longer observed for the issue causing charts to load slowly. Our teams have implemented mitigation steps and continue to monitor the environment closely. We will provide another update if there are any changes.

Mitigation actions have been completed, and we are observing recovery across affected customer environments. Charts should now be loading as expected. Detectors and datapoint ingestion were not impacted during this event. We will continue monitoring to confirm full recovery and stability before declaring the incident resolved.

A fix has been implemented and we are monitoring the results.

A fix has been implemented and we are monitoring the results.

We are continuing to monitor for any further issues.

Apr 5, 12:29 PDT Resolved - This incident has been resolved. Apr 5, 12:14 PDT Monitoring - A fix has been implemented and we are monitoring the results. Apr 5, 11:59 PDT Update - Investigation into the root cause of the Unified Identity issue is ongoing. Apr 5, 11:41 PDT Update - Our team is currently working to resolve an issue impacting Unified Identity services. Apr 5, 11:06 PDT Update - We are continuing to investigate the root cause of this issue. Apr 5, 10:22 PDT Update - We are continuing to investigate this issue. Apr 5, 09:36 PDT Investigating - We are investigating a major outage of Splunk cloud involving logging in via Unified Identity. Ingest is not impacted at this time. We will provide an update as soon as possible. Splunk Log Observer may not be available for unified identity customers.

Mar 19, 17:00 PDT Resolved - This incident has been resolved. Mar 19, 16:12 PDT Update - We are continuing to monitor for any further issues. Mar 19, 15:27 PDT Monitoring - We have successfully addressed an infrastructure issue and the environment is now stable. Our team is continuing to monitor the situation to ensure everything remains fully operational. Mar 19, 14:59 PDT Investigating - Charts and Detectors are intermittently failing. Datapoint ingest is not affected. We are currently investigating.

Mar 16, 18:12 PDT Resolved - This incident has been resolved and the system is stable Mar 16, 18:02 PDT Monitoring - The system has fully recovered and the Operations team is actively monitoring to ensure continued stability and performance Mar 16, 17:05 PDT Identified - We've identified and resolved the underlying issue, the system is recovering and we expect the lag to be fully resolved within an hour. Mar 16, 16:00 PDT Update - The Operations team is currently investigating an issue affecting the rendering of alert modal charts and latency within the alerts tab. Please note that alert creation, delivery, and data ingestion remain fully functional. There is no risk of data loss. Mar 16, 14:37 PDT Investigating - We are currently investigating this issue.

Feb 28, 00:00 PST Completed - The scheduled maintenance has been completed. Feb 9, 00:00 PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Feb 2, 09:42 PST Scheduled - Splunk Synthetic Monitoring will begin updating Google Chrome to version 142 for Browser tests on Monday, February 9th at 12:00 AM PT. We periodically auto-update to newer versions of Google Chrome when available. Due to differences between browser versions, test behavior or timings can sometimes change and may require updates to your test steps. This rollout will be done gradually over two weeks, after which a new runner version for Private Location customers will be made available.

Feb 27, 01:00 PST Completed - The scheduled maintenance has been completed. Feb 27, 00:00 PST In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary. Feb 24, 17:05 PST Update - We will be undergoing scheduled maintenance during this time. Feb 27, 00:00 PST Update - We will be undergoing scheduled maintenance during this time. Feb 27, 00:00 PST Update - We will be undergoing scheduled maintenance during this time. Feb 19, 09:36 PST Scheduled - There will be a planned maintenance window for Splunk APM in Splunk Observability Cloud. During this time, data in the Trace Analyzer, Trace Waterfall and Exemplar features may not be displayed (10 seconds to 2 minutes). All other functionalities, including MMS and TMS-powered charts and graphs, will remain unaffected. No action is needed. Thank you for your understanding.

Feb 20, 08:59 PST Resolved - This incident has been resolved. Feb 20, 08:49 PST Monitoring - A fix has been implemented and we are monitoring the results. Feb 20, 08:33 PST Update - We are continuing to investigate this issue. Feb 20, 08:11 PST Update - We are investigating intermittent failures affecting the Splunk APM and Synthetics landing pages and charts. We will provide an update as soon as possible. Feb 20, 08:10 PST Update - We are investigating intermittent failures affecting the Splunk APM and Synthetics landing pages and charts. We will provide an update as soon as possible. Feb 20, 07:44 PST Investigating - We are investigating the intermittent failures of the APM landing page and charts. We will provide an update as soon as possible.

Feb 19, 21:05 PST Resolved - This incident now stands resolved and no further updates are required. Feb 19, 20:34 PST Update - We are continuing to investigate this issue. Feb 19, 19:51 PST Investigating - We are investigating the intermittent failures of the APM landing page. We will provide an update as soon as possible.

Feb 17, 13:09 PST Resolved - This incident has been resolved. Feb 17, 13:03 PST Monitoring - A fix has been implemented and we are monitoring the results. Feb 17, 12:05 PST Update - Restoration is still in progress. Retrying to load the Data Management Page should yield positive results for some customers. Our teams continue to work diligently to resolve the issue and will provide further updates as progress is made. Feb 17, 11:28 PST Update - Our teams are actively working to restore services. Some customers may be seeing improvement at this time. We are committed to resolving the issue as quickly as possible and will provide updates as progress is made. Feb 17, 10:50 PST Identified - The issue has been identified and a fix is being implemented. Feb 17, 10:09 PST Investigating - Customers may be experiencing errors while loading Data Management Page.

Jan 19, 12:09 PST Resolved - Engineering teams have identified the root cause and completed recovery actions that caused a degradation of service to the Hosts Infra Navigator View. Jan 19, 11:24 PST Update - We are continuing to investigate this issue. We have confirmed that the issue does not impact Datapoint Ingest at this time. Jan 19, 10:45 PST Investigating - We are currently investigating an issue that may result in the Hosts Infra Navigator View from loading.