Ocrolus experienced a major incident on January 22, 2024 affecting Document Processing, lasting 9h 17m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 22, 2024, 06:40 PM UTC
We are experiencing performance issues with our human-in-the-loop verification system, which will lead to increased turnaround times and overdue documents for documents that require human verification. The Ocrolus API and Dashboard are not affected by this incident. We are still investigating the root cause.
- identified Jan 22, 2024, 07:30 PM UTC
The performance issues were caused by a spike in human verification tasks for documents that had been uploaded over the weekend as part of a bulk data import job, which we have suspended. We also received an exceptionally high volume of uploads today between 10am and 12pm ET. Both events have contributed to a large backlog of documents that will take some time to work through. Our current estimate is 4 to 5 hours.
- monitoring Jan 22, 2024, 11:23 PM UTC
With the increased volume, the size of the backlog has remained relatively constant throughout the day. Consequently, the average turnaround time for documents that require human verification has risen to above one hour from a norm of under 10 minutes.
- monitoring Jan 23, 2024, 01:56 AM UTC
We have made significant progress on the backlog, and virtually all documents uploaded before 5pm ET have been verified.
- resolved Jan 23, 2024, 03:57 AM UTC
Our human-in-the-loop verification system has been stable for about 6 hours, so we are resolving this incident. Our verifiers will continue working on the backlog until it's cleared.