Veryfi incident

Elevated errors during document processing

Minor Resolved View vendor source →

Veryfi experienced a minor incident on June 24, 2025, lasting 7h 10m. The incident has been resolved; the full update timeline is below.

Started
Jun 24, 2025, 06:35 PM UTC
Resolved
Jun 25, 2025, 01:45 AM UTC
Duration
7h 10m
Detected by Pingoru
Jun 24, 2025, 06:35 PM UTC

Update timeline

  1. investigating Jun 24, 2025, 06:35 PM UTC

    We are experiencing an increased error rate in document processing requests. A small percentage of requests are returning a 504 error response. We are investigating the issue.

  2. investigating Jun 24, 2025, 09:37 PM UTC

    We have reduced error rates further by over provisioning compute resources and spreading requests across more worker nodes. We are continuing work on identifying the root cause of the issue.

  3. investigating Jun 25, 2025, 12:36 AM UTC

    We have identified the root cause as being a newly surfaced edge case scenario in requests being submitted for processing. These particular requests consume an unusually high level of compute resources, which has downstream impact on other requests processed by the same worker processes. We are currently working on a solution to handle these edge cases gracefully.

  4. resolved Jun 25, 2025, 01:45 AM UTC

    We have implemented and validated a fix for the problematic edge case. All systems are back to normal.