Capture - Performance degradation

Update timeline

1. investigating Mar 23, 2026, 08:44 AM UTC

   We are currently experiencing an issue in Capture that is affecting the processing of incoming emails. This is causing delays in displaying them in the APA/MediusGo inbox. Our team is actively investigating the issue.

2. identified Mar 23, 2026, 09:34 AM UTC

   We have identified the root cause of the issue affecting the processing of incoming emails in Capture. The disruption was caused by an unexpected infrastructure upgrade on the environment hosting our application. This unplanned activity led to service instability. We are working to stabilize the platform and mitigate the impact.

3. identified Mar 23, 2026, 01:56 PM UTC

   The system is still struggling to return to normal operation. Unfortunately, in the meantime it has become even more unstable, and the performance impact is now more severe. We are attempting to overprovision our infrastructure to mitigate the impact.

4. identified Mar 23, 2026, 03:46 PM UTC

   We are continuing to work on a fix for this issue. The majority of incoming traffic should now be processing, however a subset of it is still blocked. We have requested assistance from our cloud provider to help resolve the remaining problems.

5. identified Mar 23, 2026, 03:49 PM UTC

   We are continuing to work on a fix for this issue.

6. identified Mar 23, 2026, 04:18 PM UTC

   The system has started to show signs of recovery. All invoices sent into the system after 15:30 UTC should now be processed in a timely manner. We have also started processing the backlog of stuck or failed invoices.

7. monitoring Mar 23, 2026, 06:00 PM UTC

   The system has remained stable since the last update regarding the processing of incoming invoices. We are continuing to work through the backlog of invoices affected by the incident.

8. resolved Mar 23, 2026, 09:52 PM UTC

   The backlog of affected invoices has now been fully processed, and the system has remained stable.

9. postmortem Mar 30, 2026, 10:19 AM UTC

   ## What happened On Monday 2026-03-23, invoice processing in the European Capture cluster experienced significant delays lasting ~13 hours. The issue was detected at 08:16 UTC by our operations team, and a status page incident was opened at 08:44 UTC. The team scaled additional capacity and performed targeted recovery actions throughout the day. Processing was fully restored by 16:00 UTC, and all backlogged invoices were cleared by 21:52 UTC. ## Customer impact * Invoice processing through email and API channels experienced delays throughout the day. * During the peak impact period \(approximately 08:00–15:00 UTC\), a significant portion of invoices took longer than normal to appear in customer inboxes. * Some customers experienced timeouts when approving invoices in the correction/validation workflow. * All invoices submitted during the incident were ultimately processed — no data was lost. * A small number of duplicate invoices were created during recovery operations; affected customers have been identified and notified. ## Root cause An unplanned infrastructure maintenance operation by our cloud provider caused internal resource contention on the European processing cluster. The system's resource reserves were insufficient to maintain normal processing throughput while the maintenance was in progress.