Rippling experienced a critical incident on October 14, 2024 affecting Single Sign-on (SSO), lasting 11m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 14, 2024, 04:02 PM UTC
We are currently investigating this issue.
- identified Oct 14, 2024, 04:04 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Oct 14, 2024, 04:06 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Oct 14, 2024, 04:13 PM UTC
This incident has been resolved. A post-mortem will be posted on Friday Oct 18.
- postmortem Oct 18, 2024, 11:18 PM UTC
## **Overview** All interactions between Rippling and third-party apps were disrupted for all customers on October 14 between 8:36 AM PDT and 9:06 AM PDT. During this 30 min window, several customers experienced errors with SSO authentication and several scheduled jobs syncing data between Rippling and third-party apps ended prematurely. This issue occurred due to a configuration change that accidentally diverted traffic to a new control flow. The most noticeable impact of this change was the SSO authentication error, which was quickly resolved after the configuration change was reverted at 9:06 AM PDT. The incident led to delays in data synchronization between Rippling and third-party systems; however, all critical transactions were completed by 10:12 PM PDT on October 14, and non-critical updates were finalized by 11:45 AM PDT on October 15. Our customer support team is on standby should you have any questions or see any issues not yet resolved. No unauthorized data access or data loss occurred during or as a result of this incident. To prevent this in the future, we are adding monitors for detecting unusual patterns of traffic based on the status code, especially the most used api’s/critical flows to detect these issues. This effort also includes improving our release process to have additional approvals for configuration changes and written roll-out plan reviewed and approved when rolling out new features. We sincerely apologize for any inconvenience this incident may have caused to our customers. At Rippling, we take responsibility for the stability and reliability of our systems very seriously. Our team is fully committed to conducting a thorough and comprehensive analysis of this incident to understand its root causes and to implement improvements to our systems and processes. We are dedicated to learning from this experience and taking all necessary steps to prevent this type of incident from happening again in the future.