SignalFx US1 incident

Datapoints Being Dropped in Archival tier

Resolved after 6 hours 46 minutes, affecting Datapoint Ingest, on September 15, 2026.

monitoring Sep 15, 2026, 08:17 PM UTC

To accelerate the recovery process, we are currently adding additional capacity to our clusters. Our teams remain actively engaged and are continuing to monitor system performance as it stabilizes.

Started
Sep 15, 2026, 02:41 PM UTC
Resolved
Sep 15, 2026, 09:28 PM UTC
Duration
6h 46m
Detected by Pingoru
Sep 15, 2026, 02:41 PM UTC

Affected components

Datapoint Ingest

Update timeline

  1. identified Sep 15, 2026, 02:41 PM UTC

    Beginning at 5:50 AM PDT, datapoint ingestion for the archival tier was impacted, resulting in a small number of datapoints not being accepted. The impact was limited to the archival tier. Detectors and charts were not affected. We have implemented remediation and are actively monitoring the system to ensure continued stability. We will provide updates every 15 minutes.

  2. identified Sep 15, 2026, 02:52 PM UTC

    As of 7:48 AM PDT, we are no longer observing any data being dropped. The system is continuing to recover, and we are actively monitoring it to ensure continued stability.

  3. monitoring Sep 15, 2026, 02:54 PM UTC

    The system is continuing to recover, and we are actively monitoring it to ensure continued stability.

  4. monitoring Sep 15, 2026, 03:33 PM UTC

    The system is showing steady progress in its recovery. Our teams remain actively engaged, closely observing system metrics to ensure continued stability

  5. monitoring Sep 15, 2026, 04:50 PM UTC

    We have successfully restored datapoint ingestion and are seeing steady progress as the data backlog is processed. We are actively monitoring this process to ensure full service functionality returns as quickly as possible.

  6. monitoring Sep 15, 2026, 05:59 PM UTC

    We continue monitoring to ensure full service functionality.

  7. monitoring Sep 15, 2026, 06:58 PM UTC

    We are continuing to monitor the system to ensure ongoing stability. Our teams remain actively engaged in observing performance metrics to confirm that all operations are functioning as expected.

  8. monitoring Sep 15, 2026, 08:17 PM UTC

    To accelerate the recovery process, we are currently adding additional capacity to our clusters. Our teams remain actively engaged and are continuing to monitor system performance as it stabilizes.

  9. resolved Sep 15, 2026, 09:28 PM UTC

    The system has recovered and we ingesting archived data with no delay.

Your product runs on more than SignalFx US1

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →