Okta experienced a minor incident on September 23, 2024 affecting okta-emea.com cell 1 and okta.com cell 1 and 1 more component, lasting 351d 21h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- resolved Sep 23, 2024, 10:32 PM UTC
At 2:53pm PT on September 23rd, Okta's Engineering Team became aware of an O365 federation issue on all cells affecting customers federating new domains. During this time, customers federating new domains may experience a '400 Bad Request' error. Okta is documenting a workaround and developing a hotfix release to mitigate the issue. We’ll provide an update in 30 minutes, or sooner if additional information becomes available.
- resolved Sep 23, 2024, 11:21 PM UTC
Okta's Engineering Team is continuing to work on a hotfix release to mitigate the '400 Bad Request' error when federating new domains. The hotfix to resolve this issue will be deployed at approximately 10:00pm US Pacific Time, today, September 23, 2024. In the meantime, the team has documented a workaround, available here: https://support.okta.com/help/s/article/microsoft-o365-federation-issue-september-23-2024 We’ll provide another update in 30 minutes, or sooner if additional information becomes available.
- resolved Sep 23, 2024, 11:51 PM UTC
Okta's Engineering Team is continuing to work on a hotfix release to mitigate the '400 Bad Request error when federating new domains. Our engineering team is running tests and validating the patch to be deployed at approximately 10:00pm US Pacific Time today, September 23, 2024. In the meantime, the team has documented a workaround, available here: https://support.okta.com/help/s/article/microsoft-o365-federation-issue-september-23-2024 We'll provide another update in an hour or sooner if additional information becomes available.
- resolved Sep 24, 2024, 12:26 AM UTC
Okta’s Engineering Team is continuing to work on a hotfix release to mitigate the ’400 Bad Request error when federating new domains. Our engineering team is running tests and validating the patch to be deployed at approximately 2:30 am US Pacific Time, September 24, 2024. In the meantime, the team has documented a workaround, available here: https://support.okta.com/help/s/article/microsoft-o365-federation-issue-september-23-2024 We’ll provide another update in two hours or sooner if additional information becomes available.
- resolved Sep 24, 2024, 02:55 AM UTC
Okta’s Engineering Team is continuing to work on a hotfix release to mitigate the ’400 Bad Request error when federating new domains. Our team is running tests and validating the patch to be deployed at approximately 2:30 a.m. US Pacific Time on September 24, 2024. The team has documented a solution in the tech note below that will need to be applied either before or after the hotfix is deployed: https://support.okta.com/help/s/article/microsoft-o365-federation-issue-september-23-2024 We’ll provide another update in two hours or sooner if additional information becomes available.
- resolved Sep 24, 2024, 05:09 AM UTC
Okta’s Engineering Team is continuing to work on a hotfix release to mitigate the '400 Bad Request' error when federating new domains. Our team is running tests and validating the patch to be deployed at approximately 2:30 a.m. US Pacific Time on September 24, 2024. The team has documented a solution in the tech note below that will need to be applied either before or after the hotfix is deployed: https://support.okta.com/help/s/article/microsoft-o365-federation-issue-september-23-2024 We’ll provide another update in two hours or sooner if additional information becomes available.
- resolved Sep 24, 2024, 07:10 AM UTC
Okta’s Engineering Team is continuing to work on a hotfix release to mitigate the '400 Bad Request' error when federating new domains. Our team is running tests and validating the patch to be deployed at approximately 2:30 a.m. US Pacific Time on September 24, 2024. The team has documented a solution in the tech note below that will need to be applied either before or after the hotfix is deployed: https://support.okta.com/help/s/article/microsoft-o365-federation-issue-september-23-2024 We’ll provide another update in two hours or sooner if additional information becomes available.
- resolved Sep 24, 2024, 09:29 AM UTC
At 2:30 am US Pacific Time on September 24, 2024, Okta’s Engineering Team successfully applied a hotfix release to mitigate the '400 Bad Request' error when federating new domains to all Okta cells except OK7 and OK14. Our Team is continuing to work on deploying the patch in cells OK7 and OK14. The team has documented a solution in the tech note below that will need to be applied either before or after the hotfix is deployed: https://support.okta.com/help/s/article/microsoft-o365-federation-issue-september-23-2024 We’ll provide another update in two hours or sooner if additional information becomes available.
- resolved Sep 24, 2024, 10:33 AM UTC
Okta’s Engineering Team successfully applied a hotfix release to mitigate the '400 Bad Request' error when federating new domains to all Okta cells except OK7. Our Team is continuing to work on deploying the patch in cell OK7. The team has documented a solution in the tech note below that will need to be applied either before or after the hotfix is deployed: https://support.okta.com/help/s/article/microsoft-o365-federation-issue-september-23-2024 We’ll provide another update in two hours or sooner if additional information becomes available.
- resolved Sep 24, 2024, 12:46 PM UTC
Okta’s Engineering Team is continuing to work on deploying the patch in cell OK7 to mitigate the '400 Bad Request' error when federating new domains. The team has documented a solution in the tech note below that will need to be applied either before or after the hotfix is deployed: https://support.okta.com/help/s/article/microsoft-o365-federation-issue-september-23-2024 We’ll provide another update in two hours or sooner if additional information becomes available.
- resolved Sep 24, 2024, 05:00 PM UTC
Okta’s Engineering Team is deploying the patch in cell OK7 to mitigate the '400 Bad Request' error when federating new domains. The team has documented a solution in the tech note below that will need to be applied either before or after the hotfix is deployed: https://support.okta.com/help/s/article/microsoft-o365-federation-issue-september-23-2024 Deployment to OK7 is estimated to be completed in 4 hours. We’ll provide another update in two hours or sooner if additional information becomes available.
- resolved Sep 24, 2024, 06:59 PM UTC
Okta's Engineering Team continues to deploy the patch in cell OK7 to mitigate the '400 Bad Request' error when federating new domains. We expect this deployment to be complete by approximately 1:30 p.m. US Pacific Time on September 24, 2024. The team has documented a solution in the tech note below that will need to be applied either before or after the hotfix is deployed: https://support.okta.com/help/s/article/microsoft-o365-federation-issue-september-23-2024 We'll provide another update in two hours or sooner if additional information becomes available.
- resolved Sep 24, 2024, 08:02 PM UTC
The patch deployment to all cells to mitigate the '400 Bad Request' error when federating new domains has been completed. Okta's Engineering Team has confirmed the resolution. A documented solution in the tech note below will need to be applied if it has not already been: https://support.okta.com/help/s/article/microsoft-o365-federation-issue-september-23-2024 Additional root cause information will be available within 5 Business days.
- resolved Oct 03, 2024, 05:45 PM UTC
We sincerely apologize for any impact this incident has caused to you, your business, and your customers. At Okta trust and transparency are our top priorities. Outlined below are the facts regarding this incident. We are committed to implementing improvements to the service to prevent future occurrences of this incident. Detection and Impact On September 23rd at 10:49am (PT), Okta’s Engineering team became aware of errors for customers attempting to federate new domains to the Office 365 application. Impacted users attempting to perform Service Provided (SP) initiated authentication flows received 400 (“Bad Request”) errors. Identity Provider (IdP) initiated authentication flows worked normally during this time. Root Cause Summary The root cause of the incident is that Okta’s third party provider pushed an unannounced backwards-incompatible update to the WS Fed implementation for Office 365. This caused application Single Sign-on (SSO) errors for newly federated domain users. Remediation Steps Okta provided its customers with a work around to successfully use this service. In parallel, Okta rolled out a hotfix for all cells to adhere to the workaround suggested by the third party. Okta observed that access rates began to improve and, on September 24th at 1:09pm PT, Okta confirmed service functionality was restored. Preventative Actions We are closely working with third party providers to notify Okta and our mutual customers quickly and help to remediate future incidents. Total Duration Total Duration: 12 days, 10 hours, and 2 minutes Actual Time: Sept 13th, 2024 02:53 AM PT - Sep 24th, 2024, 01:09 PM