SolarWinds Observability incident

Degraded Data Ingestion [NA-01, EU-01]

Resolved after 1 day 5 hours, affecting Alerting and Logs, on November 3, 2025.

postmortem Nov 04, 2025, 08:20 PM UTC

**2025-11-05 \(Final Update\)** It was identified today that one of our cloud providers, had a bad network link affecting the path used to send data from the legacy Papertrail infrastructure to the Solarwinds Observability infrastructure. The affected link has since been disabled, and we have confirmed that all impact on our side has been fully resolved. Our cloud provider will perform the necessary repairs with no expected impact to our services. **2025-11-04** During the incident, some customers experienced delays in log ingestion due to an issue affecting our legacy Papertrail infrastructure. This resulted in higher-than-normal log processing latency for a subset of migrated customers. The issue has since resolved, and log ingestion has returned to normal performance. We believe the incident was related to a network issue within our cloud provider’s environment. Our team will continue to work with the provider to confirm the root cause and will publish a post-incident review once the investigation is complete.

Started
Nov 03, 2025, 02:04 PM UTC
Resolved
Nov 04, 2025, 07:10 PM UTC
Duration
1d 5h
Detected by Pingoru
Nov 03, 2025, 02:04 PM UTC

Affected components

AlertingLogs

Update timeline

  1. investigating Nov 03, 2025, 02:04 PM UTC

    We are currently investigating an issue with Logs data ingestion. As a result of this issue, submitting new data through the API might be delayed. We’ll provide an update within the hour.

  2. investigating Nov 03, 2025, 03:04 PM UTC

    We are continuing to investigate this issue.

  3. investigating Nov 03, 2025, 04:02 PM UTC

    We are continuing to investigate this issue.

  4. investigating Nov 03, 2025, 04:58 PM UTC

    We are continuing to investigate this issue.

  5. investigating Nov 03, 2025, 05:59 PM UTC

    We are continuing to investigate this issue.

  6. investigating Nov 03, 2025, 07:00 PM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  7. investigating Nov 03, 2025, 08:00 PM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  8. investigating Nov 03, 2025, 09:00 PM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  9. investigating Nov 03, 2025, 09:59 PM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  10. investigating Nov 03, 2025, 10:59 PM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  11. investigating Nov 03, 2025, 11:59 PM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  12. investigating Nov 04, 2025, 01:06 AM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  13. investigating Nov 04, 2025, 02:01 AM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  14. investigating Nov 04, 2025, 02:59 AM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  15. investigating Nov 04, 2025, 04:00 AM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  16. investigating Nov 04, 2025, 05:02 AM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  17. investigating Nov 04, 2025, 06:02 AM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  18. investigating Nov 04, 2025, 07:01 AM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  19. investigating Nov 04, 2025, 08:04 AM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  20. investigating Nov 04, 2025, 09:05 AM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  21. investigating Nov 04, 2025, 10:03 AM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  22. investigating Nov 04, 2025, 11:04 AM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  23. investigating Nov 04, 2025, 12:11 PM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  24. investigating Nov 04, 2025, 01:11 PM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  25. investigating Nov 04, 2025, 02:09 PM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  26. investigating Nov 04, 2025, 03:09 PM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  27. investigating Nov 04, 2025, 04:12 PM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  28. investigating Nov 04, 2025, 05:14 PM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  29. investigating Nov 04, 2025, 06:12 PM UTC

    We’re currently investigating delay in Logs data ingestion. As a result, New data may not appear in the UI, alerts may be delayed, and some dashboards could show missing or outdated values. We’ll provide another update within the hour.

  30. resolved Nov 04, 2025, 07:10 PM UTC

    This incident has been resolved.

  31. postmortem Nov 04, 2025, 08:20 PM UTC

    **2025-11-05 \(Final Update\)** It was identified today that one of our cloud providers, had a bad network link affecting the path used to send data from the legacy Papertrail infrastructure to the Solarwinds Observability infrastructure. The affected link has since been disabled, and we have confirmed that all impact on our side has been fully resolved. Our cloud provider will perform the necessary repairs with no expected impact to our services. **2025-11-04** During the incident, some customers experienced delays in log ingestion due to an issue affecting our legacy Papertrail infrastructure. This resulted in higher-than-normal log processing latency for a subset of migrated customers. The issue has since resolved, and log ingestion has returned to normal performance. We believe the incident was related to a network issue within our cloud provider’s environment. Our team will continue to work with the provider to confirm the root cause and will publish a post-incident review once the investigation is complete.

Your product runs on more than SolarWinds Observability

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →