Tungsten Automation Americas incident

AP Essentials - Slow Processing of Documents

Minor Resolved View vendor source →

Tungsten Automation Americas experienced a minor incident on September 18, 2025 affecting AP Essentials, lasting 3h 3m. The incident has been resolved; the full update timeline is below.

Started
Sep 18, 2025, 05:22 PM UTC
Resolved
Sep 18, 2025, 08:25 PM UTC
Duration
3h 3m
Detected by Pingoru
Sep 18, 2025, 05:22 PM UTC

Affected components

AP Essentials

Update timeline

  1. identified Sep 18, 2025, 05:22 PM UTC

    We are currently experiencing an incident where document processing is slower than expected. Our engineering teams are actively investigating the root cause and working to restore normal performance as quickly as possible.

  2. identified Sep 18, 2025, 05:22 PM UTC

    We are continuing to work on a fix for this issue.

  3. monitoring Sep 18, 2025, 07:53 PM UTC

    A fix has been applied, and we are seeing improvements in document processing times. We will continue to closely monitor the service to ensure performance fully returns to normal. Thank you for your patience while we worked to resolve this issue.

  4. resolved Sep 18, 2025, 08:25 PM UTC

    The issue causing slowness in document processing has been resolved. Document processing is now functioning as expected. We will continue to monitor the service, but no further impact is expected.

  5. postmortem Oct 07, 2025, 09:40 AM UTC

    Overview On September 18, 2025, customers on the US instance of AP Essentials experienced delays in document processing, where documents remained in the Document Created state and did not progress to OCR. This issue affected multiple partners and customers. ‌ Incident Detail The issue occurred following an internal configuration change related to a future deployment. The change unintentionally modified production environment settings for one of the background processing components, leading to failures in the extract worker service and the buildup of document processing queues. As a result, documents were stalled in the Document Created stage and were not processed further. Upon identification, our Cloud Operations and R&D teams worked jointly to revert the configuration change, which immediately restored normal document processing. All queued documents were successfully processed after the fix was applied. Resolution The configuration change was reverted, restoring the correct system parameters. Once reverted, the extract workers resumed normal operation, and all affected queues were cleared. Processing rates returned to normal, and continuous monitoring confirmed system stability. We have implemented stricter change verification procedures and reinforced internal communication protocols to prevent similar issues in the future.