Service Degradation on Multiple Cells

Update timeline

1. resolved Jun 12, 2025, 01:09 AM UTC

   At 6:09 PM PST on June 11, 2025, Okta became aware of an issue with our service affecting customers in US Cell 6, APAC Cell 8. This issue has been resolved. Okta took corrective action to resolve the service interruption. The service was fully restored at 6:23 PM PST. Additional root cause information will be available within 5 Business days.

2. resolved Jun 18, 2025, 10:40 PM UTC

   We sincerely apologize for any impact this incident has caused to you, your business, and your customers. At Okta trust and transparency are our top priorities. Outlined below are the facts regarding this incident. We are committed to implementing improvements to the service to prevent future occurrences of this incident. Detection and Impact: On June 11th at 6:07 PM PT, Okta internal monitoring alerted responders regarding a database issue across cells US Cell 6 and APAC Cell 1. Primary databases were deadlocking, resulting in the cell switching to [read only] (https://support.okta.com/help/s/article/What-is-Oktas-Readonly-Mode?language=en_US) mode. During this time, users in Cells US Cell 6 and APAC Cell 1 would have encountered read only mode during this period. Root Cause Summary: Okta had progressively rolled out a configuration change earlier in the day as part of normal scheduled operations. This change exposed an error case which was then observed only in specific environments. Okta determined that MySQL contained a bug which, when a new specific monitoring query was executed, resulted in deadlocking the replica database servers and blocking additional queries from being executed. Okta managed the issue by executing database recovery efforts, thus restoring functionality and availability to normal operating levels. Cleanup procedures for this issue, which entailed rollback of the previously executed configuration change, were executed later in the day in a staggered fashion. During this process, the database primaries for US Cell 6 and APAC Cell 1 unexpectedly experienced a similar deadlock issue. Remediation Steps: During this period US Cell 6 and APAC Cell 1 switched into Read Only mode. Okta engineers executed database failover and recovery operations to return the cell to normal operation. Impact Durations: US Cell 6 - 6:06 PM PT - 6:18 PM - 12 minutes APAC Cell 1 - 6:06 PM PT - 6:24 PM - 18 minutes At 6:24 PM PT all servers had recovered and had returned to normal operations. Preventative Actions: In order to prevent similar incidents from happening again, Okta is currently reviewing the following: Review of database rollback process utilized between replicas and primaries Review sequencing of database configuration updates deployed across environments. These learnings work to prevent similar incidents from happening again.