Expense, Invoice, Request, Imaging: P1

Update timeline

1. resolved Jul 14, 2026, 09:55 AM UTC

   We are working to determine the user impact and will provide information as it becomes available.

2. resolved Jul 14, 2026, 10:44 AM UTC

   A limited subset of customers in the EU2 Data Center experienced a functional disruption with Invoice, Expense, Request, and Imaging service. When working with Expenses, Requests, and Invoices, customers were unable to attach receipts, upload and retrieve images. On the Imaging side, customers were not seeing attached Images, getting “Receipt not available” error. Service has been restored to a normal operating level, and we have started a mandatory post recovery monitoring period to ensure service performance stability prior to declaring the incident resolved. We will post the next incident communication at or before 11:30 AM UTC on 14 July 2026.

3. resolved Jul 14, 2026, 11:03 AM UTC

   A limited subset of customers in the EU2 Data Center experienced a functional disruption with Invoice, Expense, Request, and Imaging service. When working with Expenses, Requests, and Invoices, customers were unable to attach receipts, upload and retrieve images. On the Imaging side, customers were not seeing attached Images, getting “Receipt not available” error. The Incident Response Team rolled back a recent deployment to restore service. We have verified that service performance is stable and will now resolve this incident. An investigation into the root cause of this incident will now be conducted, and a root cause analysis report provided when that investigation is complete.

4. resolved Jul 23, 2026, 11:13 AM UTC

   Impact as reported: A limited subset of customers in the EU2 Data Center experienced a functional disruption with Invoice, Expense, Request, and Imaging service. When working with Expenses, Requests, and Invoices, customers were unable to attach receipts, upload and retrieve images. On the Imaging side, customers were not seeing attached Images, getting “Receipt not available” error. Root cause: A service deployment introduced a change that validated cache connection authentication on every request instead of performing periodic validation. Although the broader security enhancement being deployed was not enabled and was not the direct cause, this connection-validation behaviour significantly increased load on the shared cache infrastructure. The additional overhead caused CPU saturation on one cache shard in the EU2 environment, leading to degraded cache performance, application errors, HTTP 500 responses, and failed transactions. The issue was not identified during pre-production testing because: Canary deployments did not generate enough sustained load to expose the problem. Performance testing focused on application behaviour rather than infrastructure stress limits. Monitoring did not provide early visibility into cache CPU saturation. The EU2 environment had less capacity headroom than another region where similar changes had been deployed. Corrective Actions: The deployment was rolled back and service restored. The authentication validation has been changed from "every request" to "periodic checks". The cache cluster capacity has been increased in the affected region.