One Identity Starling incident

One Identity Starling - Safeguard remote access (All Regions)

Minor Resolved View vendor source →

One Identity Starling experienced a minor incident on January 16, 2025 affecting Remote Access and Remote Access and 1 more component, lasting 42d 10h. The incident has been resolved; the full update timeline is below.

Started
Jan 16, 2025, 11:34 AM UTC
Resolved
Feb 27, 2025, 10:15 PM UTC
Duration
42d 10h
Detected by Pingoru
Jan 16, 2025, 11:34 AM UTC

Affected components

Remote AccessRemote AccessRemote Access

Update timeline

  1. investigating Jan 16, 2025, 11:34 AM UTC

    We are currently experiencing an partial outage regarding SRA sessions closing unexpectedly after some time. A workaround is possible by downloading the RDP file and connecting locally, rather than starting the session from SRA. We are working on fixing the issue. Further updates will be provided at 05:46 PST.

  2. investigating Jan 16, 2025, 12:51 PM UTC

    We are still investigating this issue. We will provide further information at 08:00 PST

  3. investigating Jan 16, 2025, 03:08 PM UTC

    We are still investigating this issue. Sessions are impacted, however downloading the RDP file and connecting via RDP is still a viable workaround. A further update will be provided at 01:00 PST/09:00 UTC

  4. investigating Jan 17, 2025, 09:03 AM UTC

    We are still investigating this issue. A further update will be provided at 03:00 PST/11:00 UTC

  5. investigating Jan 17, 2025, 11:00 AM UTC

    We are still investigating this issue. A further update will be provided at 05:00 PST/13:00 UTC

  6. investigating Jan 17, 2025, 12:58 PM UTC

    We are still investigating this issue. A further update will be provided at 07:00 PST/15:00 UTC

  7. investigating Jan 17, 2025, 03:08 PM UTC

    We are still investigating this issue. A further update will be provided at 09:00UTC/01:00PST on Monday 2025-01-20

  8. investigating Jan 20, 2025, 09:04 AM UTC

    We are still investigating this issue. A further update will be provided at 05:00 PST/13:00 UTC

  9. investigating Jan 20, 2025, 11:06 AM UTC

    We are still investigating this issue. A further update will be provided at 07:00 PST/15:00 UTC

  10. investigating Jan 20, 2025, 01:09 PM UTC

    We are still investigating this issue. A further update will be provided at 09:00 PST/17:00 UTC

  11. investigating Jan 20, 2025, 04:35 PM UTC

    We are still investigating this issue. A further update will be provided at 09:00UTC/01:00PST on Tuesday 2025-01-21

  12. investigating Jan 21, 2025, 09:06 AM UTC

    We are still investigating this issue. A further update will be provided at 05:00 PST/13:00 UTC

  13. investigating Jan 21, 2025, 11:35 AM UTC

    We are still investigating this issue. A further update will be provided at 07:00 PST/15:00 UTC

  14. investigating Jan 21, 2025, 02:17 PM UTC

    We are still investigating this issue. A further update will be provided at 09:00 PST/17:00 UTC

  15. investigating Jan 21, 2025, 04:08 PM UTC

    We are still investigating this issue. A further update will be provided at 09:00UTC/01:00PST on Wednesday 2025-01-22

  16. investigating Jan 22, 2025, 09:02 AM UTC

    We are still investigating this issue. A further update will be provided at 05:00 PST/13:00 UTC

  17. investigating Jan 22, 2025, 11:24 AM UTC

    We are still investigating this issue. A further update will be provided at 07:00 PST/15:00 UTC

  18. investigating Jan 22, 2025, 05:29 PM UTC

    We are still investigating this issue. A further update will be provided at 09:00UTC/01:00PST on Thursday 2025-01-23

  19. investigating Jan 23, 2025, 09:04 AM UTC

    We are still investigating this issue. A further update will be provided at 05:00 PST/13:00 UTC

  20. investigating Jan 23, 2025, 12:04 PM UTC

    We are still investigating this issue. A further update will be provided at 08:00 PST/16:00 UTC

  21. investigating Jan 23, 2025, 03:03 PM UTC

    We are still investigating this issue. A further update will be provided at 09:00UTC/01:00PST on Friday 2025-01-24

  22. investigating Jan 24, 2025, 10:49 AM UTC

    We are continuing to investigate this issue. A further update will be provided at 13:00 UTC/05:00 PDT.

  23. investigating Jan 24, 2025, 12:59 PM UTC

    We are continuing to investigate this issue. A further update will be provided at 15:00 UTC/07:00 PDT.

  24. investigating Jan 24, 2025, 03:17 PM UTC

    We are still investigating this issue. A further update will be provided at 09:00UTC/01:00PST on Monday 2025-01-27

  25. investigating Jan 27, 2025, 10:01 AM UTC

    We are continuing to investigate this issue. A further update will be provided at 16:00 UTC/08:00 PDT.

  26. investigating Jan 28, 2025, 09:37 AM UTC

    We are continuing to investigate this issue. A further update will be provided at 16:00 UTC/08:00 PDT.

  27. investigating Jan 28, 2025, 11:17 AM UTC

    We are continuing to investigate this issue.

  28. investigating Jan 28, 2025, 11:18 AM UTC

    We are continuing to investigate this issue.

  29. investigating Jan 28, 2025, 11:19 AM UTC

    We are continuing to investigate this issue.

  30. investigating Jan 28, 2025, 11:20 AM UTC

    We are continuing to investigate this issue.

  31. investigating Jan 28, 2025, 04:46 PM UTC

    We are still investigating this issue. A further update will be provided at 09:00UTC/01:00PST on Wednesday 2025-01-29

  32. investigating Jan 29, 2025, 09:27 AM UTC

    We are continuing to investigate this issue. A further update will be provided at 16:00 UTC/08:00 PDT.

  33. investigating Jan 30, 2025, 01:59 PM UTC

    We are still investigating this issue. A further update will be provided at on Wednesday 2025-01-31

  34. investigating Jan 31, 2025, 12:23 PM UTC

    We are continuing to investigate this issue. A further update will be provided at 16:00 UTC/08:00 PDT.

  35. investigating Feb 03, 2025, 10:41 AM UTC

    We are continuing to investigate this issue. A further update will be provided at 16:00 UTC/08:00 PDT

  36. investigating Feb 03, 2025, 04:34 PM UTC

    We are still investigating this issue. A further update will be provided on Tuesday 2025-02-04

  37. investigating Feb 04, 2025, 08:56 AM UTC

    We are continuing to investigate this issue. A further update will be provided at 16:00 UTC/08:00 PDT

  38. investigating Feb 04, 2025, 06:34 PM UTC

    We are still investigating this issue. A further update will be provided on Wednesday 2025-02-05

  39. investigating Feb 05, 2025, 11:13 AM UTC

    We are continuing to investigate this issue. A further update will be provided at 16:00 UTC/08:00 PDT

  40. investigating Feb 07, 2025, 10:21 AM UTC

    We are continuing to investigate this issue. A further update will be provided at 16:00 UTC/08:00 PDT

  41. investigating Feb 07, 2025, 05:05 PM UTC

    We are still investigating this issue. A further update will be provided on Monday 2025-02-10

  42. investigating Feb 10, 2025, 09:38 AM UTC

    We are continuing to investigate this issue. A further update will be provided at 16:00 UTC/08:00 PDT

  43. investigating Feb 11, 2025, 09:43 AM UTC

    We are continuing to investigate this issue. A further update will be provided at 16:00 UTC/08:00 PDT

  44. investigating Feb 13, 2025, 02:35 PM UTC

    We are continuing to investigate this issue. A further update will be provided at 16:00 UTC/08:00 PDT

  45. investigating Feb 13, 2025, 04:10 PM UTC

    We are still investigating this issue. A further update will be provided on Friday 2025-02-14

  46. monitoring Feb 14, 2025, 12:24 PM UTC

    Mitigations are in place and sessions are working normally. We are currently monitoring all sessions in the EU and US to ensure continued operation, and we will follow up further when possible.

  47. monitoring Feb 18, 2025, 12:16 PM UTC

    We continue to monitor the overall performance of our services which are fully functional. We will be deploying a formal fix within the next 7 days which is expected to fully resolve the issue and prevent future occurrences.

  48. monitoring Feb 25, 2025, 02:06 PM UTC

    All services remain fully functional. The formal fix scheduled for deployment will require an additional 48 hours of quality control testing. Status will be updated upon completion.

  49. resolved Feb 27, 2025, 10:15 PM UTC

    The aforementioned fix has been deployed and the issue is now resolved.

  50. postmortem Mar 05, 2025, 07:43 PM UTC

    **What Occurred?** Safeguard Remote Access \(SRA\) sessions were closing unexpectedly in the EU region. ‌ **What went wrong and why?** SRA RDP/SSH sessions in the EU were disconnecting when Azure Kubernetes nodes began running out of WebSockets. In certain scenarios, unused sockets were not being closed correctly resulting in WebSocket exhaustion and subsequent disconnections. ‌ **How are we making incidents like this less likely or less impactful?** We are implementing a more robust WebSocket management solution and increasing SRA's logging to proactively identify and prevent this from recurring. These improvements will increase reliability and mitigate against future occurrences in addition to improving future troubleshooting across the platform.