Jobs Slow to Process

Update timeline

1. resolved Oct 28, 2024, 01:15 PM UTC

   Okta is currently experiencing an issue with the service. We are actively investigating and will update this message with more information as soon as we have it.

2. resolved Oct 28, 2024, 02:59 PM UTC

   Okta's Engineering team has become aware of an import and update issue affecting customers in OK14. During this time, customers in OK14 may experience imports and updates that progress very slowly. Okta's Engineering team is investigating and working to mitigate the issue. We’ll provide an update in 30 minutes or sooner if additional information becomes available.

3. resolved Oct 28, 2024, 03:32 PM UTC

   Okta Engineering continues to investigate and work to mitigate import and update issues affecting customers in OK14. During this time, customers in OK14 may experience imports and updates that progress very slowly and Okta Workflows initiated by Okta events and delegated workflows in FL14 may fail to progress. We’ll provide an update in 30 minutes or sooner if additional information becomes available.

4. resolved Oct 28, 2024, 04:00 PM UTC

   Okta Engineering has identified the potential cause of the issue and continues to work towards mitigating the disruption. Customers in OK14 may experience imports and updates that progress very slowly, and Okta Workflows initiated by Okta events and delegated workflows in FL14 may fail to progress. We’ll provide an update in 30 minutes or sooner if additional information becomes available.

5. resolved Oct 28, 2024, 04:27 PM UTC

   Okta Engineering continues to work towards mitigating the disruption. Customers in OK14 may experience imports and updates that progress very slowly, and Okta Workflows initiated by Okta events and delegated workflows in FL14 may fail to progress. We’ll provide an update in 30 minutes or sooner if additional information becomes available.

6. resolved Oct 28, 2024, 04:46 PM UTC

   Okta continues to monitor the current situation with the import and update delays in OK14. Our monitoring shows that import and update throughput is improving and delays are reducing. Our next update will be in 30 minutes or sooner if additional information becomes available.

7. resolved Oct 28, 2024, 05:18 PM UTC

   An issue impacting import and update performance in OK14 has been addressed. Additional root cause information will be available within 5 Business days.

8. resolved Nov 04, 2024, 11:25 PM UTC

   We sincerely apologize for any impact this incident has caused to you, your business, and your customers. At Okta trust and transparency are our top priorities. Outlined below are the facts regarding this incident. We are committed to implementing improvements to the service to prevent future occurrences of this incident. Detection and Impact On October 28th at 5:06 AM (PT), Okta was alerted that a few customers experienced delays related to imports in US Cell 14. During this time customers may have observed delayed imports as a result, delays in user activation, provisioning, password sync and additional async tasks At 9:38am(PT) task processing in the cell was restored to normal processing times. Root Cause Summary This issue was a result of a build up of both active and queued asynchronous tasks. Under heavy load of these tasks, the scheduler dequeued records too slowly and created a large backlog. This large backlog resulted in scheduling delays of tasks such as imports. Remediation Steps Okta Engineering scaled out for additional processing capacities, and tuned the dequeuing parameters to pull more records from the queue. These actions successfully addressed the issue, and all imports returned to normal processing levels. Preventative Actions Okta Engineering is reviewing and updating alerting, making improvements to address tasks queuing, and updating logging to better identify future issues related to queueing. Total Duration Total Duration (Minutes): 271 Actual Time: 5:07am PT - 9:38am PT