Okta experienced a major incident on December 13, 2024 affecting okta-emea.com cell 1 and okta.com cell 1 and 1 more component, lasting 271d 5h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- resolved Dec 13, 2024, 02:47 PM UTC
Okta is currently experiencing an issue with the service. We are actively investigating and will update this message with more information as soon as we have it.
- resolved Dec 13, 2024, 03:00 PM UTC
Okta teams have become aware of disruption affecting customers in all cells. During this time, customer admins may be unable to access the Okta Help Center, and users may see HTTP 429 errors when attempting to log in. Okta teams are investigating and working to mitigate the issue. We’ll provide an update in 30 minutes or sooner if additional information becomes available.
- resolved Dec 13, 2024, 03:33 PM UTC
Okta Engineering has isolated the issue, and Okta teams have made DNS changes to mitigate the disruption. Customer admins should begin to have access to the Okta Help Center, and users should stop seeing HTTP 429 errors as DNS continues to propagate. We are seeing the error rates subside, and we will continue to monitor the situation. We’ll provide an update in 30 minutes or sooner if additional information becomes available.
- resolved Dec 13, 2024, 03:45 PM UTC
An issue impacting user login has been resolved. Our monitoring shows a return to normal conditions, and we will continue to monitor for any further errors. Additional root cause information will be available within 5 business days.
- resolved Dec 20, 2024, 10:58 PM UTC
We sincerely apologize for any impact this incident has caused to you, your business, and your customers. At Okta trust and transparency are our top priorities. Outlined below are the facts regarding this incident. We are committed to implementing improvements to the service to prevent future occurrences of this incident. Detection and Impact: Beginning on December 13th at 4:47AM PT, Okta became aware of customer reports regarding issues related to authentication and access, at which time the response teams began investigating the incident. During this time, customer admins in all commercial and FedRamp Moderate cells were unable to login to the Okta Help Center via login.okta.com. Additionally, users of embedded and custom browsers may have experienced error messages impacting log in flows in certain cases. Root Cause Summary: Okta Engineering determined that the root cause of the issue was due to a DNS record deletion of a reserved subdomain that was executed as part of automated resource cleanup processes which began at 2:01AM PT. Remediation Steps: At 7:07AM PT, the source of the error was identified and Okta engineering restored the DNS record at 7:15AM PT. At 7:25AM PT all processing had returned to normal. After determining that the clean up process was the source of the deletion, Okta engineering disabled it at 8:42AM PT. Preventative Actions: Certain cleanup operations have been paused until additional protections are put in place, including protecting reserved DNS records. In order to prevent similar incidents from happening again, Okta is addressing the following: Add stronger protections and enhanced monitoring around reserved DNS records in clean-up operations Updates to our monitoring and runbooks for login.okta.com service issues In the event that Okta Help Center is not available, customers can use the relevant support number found at: https://www.okta.com/contact/. When calling in, they will be asked for their Customer Support Account ID for caller verification.