Secret Server Cloud: Remote Password Changing (RPC) and Heartbeat failures

Update timeline

1. investigating Dec 06, 2025, 11:03 PM UTC

   We are currently investigating reports of customers experiencing pending Remote Password Changing (RPC) tasks and Heartbeat failures. Our engineering team is actively working to identify the root cause and restore normal functionality as quickly as possible. We apologize for the inconvenience and appreciate your patience while we work to resolve the issue. If you have any questions or need assistance, please contact our support team: https://support.delinea.com

2. identified Dec 07, 2025, 12:46 AM UTC

   The issue has been identified. Rollback of the latest release is in progress. Thank you for your patience.

3. monitoring Dec 07, 2025, 02:23 AM UTC

   Rollback is complete. We are seeing successful Remote Password Changing (RPC) tasks and Heartbeats. We will continue to monitor the results.

4. identified Dec 07, 2025, 05:26 AM UTC

   We are experiencing pending Remote Password Changing (RPC) tasks and Heartbeat failures in US and UK regions

5. monitoring Dec 07, 2025, 07:35 AM UTC

   A fix has been implemented and we are monitoring the results.

6. resolved Dec 07, 2025, 07:43 AM UTC

   This incident has been resolved.

7. postmortem Dec 26, 2025, 03:35 PM UTC

   ## Incident Overview ### Summary Secret Server Cloud customers using Distributed Engine \(DE\) for Remote Password Changing \(RPC\) operations experienced RPC requests remaining in a “Pending” state. In some cases, RPC functionality was partially restored before full service was recovered. A follow-on regression occurred during remediation, temporarily impacting a subset of customers again. ### First Occurrence * Start Time: December 6, 2025, 5:44 PM ET * End Time: December 7, 2025, 2:35 AM ET ### Second Occurrence \(Follow-on Regression\) * Start Time: December 14, 2025, 10:53 AM ET * End Time: December 14, 2025, 12:00 PM ET ### Impact * RPC operations initiated through Distributed Engine remained in a pending state and did not complete successfully. * Other Secret Server Cloud functionality was not impacted. ## Root Cause The issue was caused by enhancements introduced as part of the RPC pre-run validation feature, which is tightly integrated with the RPC execution flow. The enhancement required changes to how RPC requests are structured and processed. These changes introduced compatibility issues between the updated Secret Server Cloud service and certain previously installed Distributed Engine versions, preventing RPC requests from being processed correctly. During the initial rollback, additional infrastructure changes included in the same release prevented an automatic rollback from fully completing, which extended the time required to restore full functionality. During the subsequent remediation rollout on Dec 13, 2025, a configuration issue related to message routing caused a temporary regression, impacting a subset of Distributed Engines until the configuration was corrected. ## Preventive Actions To prevent recurrence, we have taken and are taking the following actions: * Improved QA validation to explicitly test upgrades across all supported Distributed Engine versions, not just incremental or phase-based upgrades. * Ensured large, foundational features such as RPC pre-run validation are integrated into QA environments earlier in the release cycle to better reflect production conditions. * Added broader cross-team technical reviews for complex, highly integrated features before release. * Implemented backward compatibility safeguards to ensure newer Secret Server Cloud releases can safely interoperate with older Distributed Engine versions. * Enhanced deployment validation and monitoring to more quickly detect configuration regressions during remediation rollouts.