SignalFx US0 incident

New time series may be delayed in joining detectors in US0

Resolved after 1 day 6 hours, affecting Alerting, on July 10, 2026.

identified Jul 11, 2026, 03:05 PM UTC

We continue to investigate delayed inclusion of newly created time series in detectors in US0. New time series may take up to 10 minutes to be included in detectors, which may delay detector evaluations for those time series. Notification delay mean time remains healthy, and no data loss has been observed. Engineering teams are performing a controlled rollout of affected backend components as part of ongoing mitigation work, and this is not expected to introduce additional customer impact. We will provide another update as validation progresses.

Started
Jul 10, 2026, 12:03 PM UTC
Resolved
Jul 11, 2026, 06:09 PM UTC
Duration
1d 6h
Detected by Pingoru
Jul 10, 2026, 12:03 PM UTC

Affected components

Alerting

Update timeline

  1. investigating Jul 10, 2026, 12:03 PM UTC

    Starting at 1a PDT, a small percentage of detectors may have missed firing. No data has been lost. We are investigating and will provide an update shortly.

  2. monitoring Jul 10, 2026, 12:05 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. monitoring Jul 10, 2026, 12:47 PM UTC

    We are continuing to monitor for any further issues.

  4. monitoring Jul 10, 2026, 01:51 PM UTC

    We are continuing to monitor for any further issues.

  5. investigating Jul 10, 2026, 02:52 PM UTC

    We have seen a return of a small percentage of detectors that may not be firing. No data is lost. We are continuing to investigate.

  6. investigating Jul 10, 2026, 03:52 PM UTC

    We continue to investigate an issue affecting a small percentage of detectors in US0 that may not have fired as expected. No data loss has been observed. Our engineering teams are actively investigating the issue, and we will provide another update as more information becomes available.

  7. investigating Jul 10, 2026, 04:53 PM UTC

    We continue to investigate an issue affecting a small percentage of detectors in US0 that may not have fired as expected. No data loss has been observed. Our engineering teams are actively investigating the issue, and we will provide another update as more information becomes available.

  8. investigating Jul 10, 2026, 06:01 PM UTC

    We continue to investigate an issue affecting a small percentage of detectors in US0 that may not have fired as expected. No data loss has been observed. Our engineering teams are actively investigating the issue, and we will provide another update as more information becomes available.

  9. investigating Jul 10, 2026, 07:02 PM UTC

    We continue to investigate an issue affecting a small percentage of detectors in US0 that may not have fired as expected. No data loss has been observed. Our engineering teams are actively investigating the issue, and we will provide another update as more information becomes available.

  10. identified Jul 10, 2026, 07:39 PM UTC

    A mitigation has been applied for the issue affecting a small percentage of detectors in US0 that may not have fired as expected. We are observing continued recovery and are monitoring the environment to ensure the mitigation is fully effective. No data loss has been observed. We will provide another update once recovery is complete.

  11. identified Jul 10, 2026, 08:53 PM UTC

    We are observing continued recovery and are monitoring the environment to ensure the mitigation is fully effective. No data loss has been observed. We will provide another update once recovery is complete.

  12. monitoring Jul 10, 2026, 10:09 PM UTC

    We are observing continued recovery and are monitoring the environment to ensure the mitigation is fully effective. No data loss has been observed. We will provide another update once recovery is complete.

  13. monitoring Jul 10, 2026, 11:06 PM UTC

    We are observing continued recovery and are monitoring the environment to ensure the mitigation is fully effective. No data loss has been observed. We will provide another update once recovery is complete.

  14. monitoring Jul 11, 2026, 12:16 AM UTC

    We are observing continued recovery and are monitoring the environment to ensure the mitigation is fully effective. No data loss has been observed. We will provide another update once recovery is complete.

  15. identified Jul 11, 2026, 02:15 AM UTC

    We have applied mitigations that reduced the impact, but recovery is not yet complete. A small percentage of detectors in US0 may still not be firing as expected due to delayed processing. No data loss has been observed. Engineering teams continue to investigate towards the additional mitigations. We will provide another update as more information becomes available.

  16. identified Jul 11, 2026, 03:03 AM UTC

    We have identified that newly created time series in US0 may take up to 10 minutes to be included in detectors, which may delay detector evaluations for those new time series. Notification delay mean time is currently healthy, and no data loss has been observed. Engineering teams are continuing mitigation work and validating whether it clears the remaining delays. We will provide another update once validation is complete.

  17. identified Jul 11, 2026, 06:54 AM UTC

    We continue to investigate delayed inclusion of newly created time series in detectors in US0. New time series may take up to 10 minutes to be included in detectors, which may delay detector evaluations for those time series. Notification delay mean time remains healthy, and no data loss has been observed. Engineering teams are performing a controlled rollout of affected backend components as part of ongoing mitigation work, and this is not expected to introduce additional customer impact. We will provide another update as validation progresses.

  18. identified Jul 11, 2026, 11:09 AM UTC

    We continue to investigate delayed inclusion of newly created time series in detectors in US0. New time series may take up to 10 minutes to be included in detectors, which may delay detector evaluations for those time series. Notification delay mean time remains healthy, and no data loss has been observed. Engineering teams are performing a controlled rollout of affected backend components as part of ongoing mitigation work, and this is not expected to introduce additional customer impact. We will provide another update as validation progresses.

  19. identified Jul 11, 2026, 03:05 PM UTC

    We continue to investigate delayed inclusion of newly created time series in detectors in US0. New time series may take up to 10 minutes to be included in detectors, which may delay detector evaluations for those time series. Notification delay mean time remains healthy, and no data loss has been observed. Engineering teams are performing a controlled rollout of affected backend components as part of ongoing mitigation work, and this is not expected to introduce additional customer impact. We will provide another update as validation progresses.

  20. resolved Jul 11, 2026, 06:09 PM UTC

    This incident has been resolved.

Your product runs on more than SignalFx US0

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →