Okta incident

Issues with Custom Domain in Cell OK8

Major Resolved View vendor source →

Okta experienced a major incident on January 14, 2025 affecting Core Platform, lasting 8d 7h. The incident has been resolved; the full update timeline is below.

Started
Jan 14, 2025, 05:31 PM UTC
Resolved
Jan 23, 2025, 12:32 AM UTC
Duration
8d 7h
Detected by Pingoru
Jan 14, 2025, 05:31 PM UTC

Affected components

Core Platform

Update timeline

  1. resolved Jan 14, 2025, 05:31 PM UTC

    At 1/14/2025 9:16 AM PST, the Core Identity team became aware of an issue with Custom Domains affecting customers in OK8. During this time, users may have experienced issues accessing resources powered by Okta custom domains. This issue has been resolved. Okta took corrective action to resolve the service interruption. The service was fully restored at 9:33 AM PST. Root cause information: We sincerely apologize for any impact this incident has caused to you, your business, and your customers. At Okta trust and transparency are our top priorities. Outlined below are the facts regarding this incident. We are committed to implementing improvements to the service to prevent future occurrences of this incident. Detection and Impact: On January 14th at 9:22AM PT Okta internal monitoring alerts indicated errors in loading custom domains in OK Cell 8. During this time customers who utilize customized domains in OK Cell 8 experienced errors accessing the service. Root Cause Summary: In order to resolve an issue which was caught by our internal monitoring during a change the previous evening, Okta engineering staff was redeploying the edge servers. Due to a bug in the operating procedure used, the service became momentarily unavailable until corrective actions were taken to fully restore the service. Remediation Steps: At 9:24AM PT, Okta Engineering quickly identified the issue and resolved it by placing healthy servers in service. Preventative Actions: Okta has updated the operating procedure and improved the tooling used for managing custom domain services. To ensure this does not happen again, Okta is enhancing current automated testing of this process. Affected cells: okta.com:8

  2. resolved Jan 22, 2025, 11:52 PM UTC

    We sincerely apologize for any impact this incident has caused to you, your business, and your customers. At Okta trust and transparency are our top priorities. Outlined below are the facts regarding this incident. We are committed to implementing improvements to the service to prevent future occurrences of this incident. Detection and Impact: On January 14th at 9:22AM PT Okta internal monitoring alerts indicated errors in loading custom domains in OK Cell 8. During this time customers who utilize customized domains in OK Cell 8 experienced errors accessing the service. Root Cause Summary: In order to resolve an issue which was caught by our internal monitoring during a change the previous evening, Okta engineering staff was redeploying the edge servers. Due to a bug in the operating procedure used, the service became momentarily unavailable until corrective actions were taken to fully restore the service. Remediation Steps: At 9:24AM PT, Okta Engineering quickly identified the issue and resolved it by placing healthy servers in service. Preventative Actions: Okta has updated the operating procedure and improved the tooling used for managing custom domain services. To ensure this does not happen again, Okta is enhancing current automated testing of this process.