Invoice: P1

Update timeline

1. resolved Apr 15, 2025, 12:14 PM UTC

   We are working to determine the user impact and will provide information as it becomes available.

2. resolved Apr 15, 2025, 12:48 PM UTC

   In the US2, EU2 and APJ1 Data Centers, customers using Invoice Capture is below expected level. Affected users are unable to send the vendor invoices to Concur. Customers should be able to create invoices manually as a workaround. The Incident Response Team (IRT) is currently investigating. Our next scheduled update will be in one hour or sooner should a significant change occur.

3. resolved Apr 15, 2025, 01:42 PM UTC

   In the US2, EU2 and APJ1 Data Centers, customers using Invoice Capture to send vendor invoices to Concur via email is below expected level. Affected users using Invoice Capture unable to send vendor invoices to Concur via email. The invoices are not being successfully delivered or processed by Concur. The Incident Response Team (IRT) continues to investigate with additional resources. Our next scheduled update will be in one hour or sooner should a significant change occur.

4. resolved Apr 15, 2025, 02:44 PM UTC

   In the US2, EU2 and APJ1 Data Centers, sending invoices via email to the Invoice Capture service in SAP Concur is below expected level. Affected users may encounter invoices not being successfully delivered or processed via the Invoice Capture service. A Workaround has not been identified at this time. The Incident Response Team (IRT) is investigating multiple recent changes to identify root cause. Our next scheduled update will be in one hour or sooner should a significant change occur.

5. resolved Apr 15, 2025, 03:27 PM UTC

   In the US2, EU2 and APJ1 Data Centers, sending invoices via email to the Invoice Capture service in SAP Concur was below expected level. Affected users may have encountered invoices not being successfully delivered or processed via the Invoice Capture service. A workaround was not available. Services have returned to normal operating levels. We will continue monitoring performance for the next 15-30 minutes to verify stability. Once verified as stable the incident will be resolved.

6. resolved Apr 15, 2025, 03:50 PM UTC

   In the US2, EU2 and APJ1 Data Centers, sending invoices via email to the Invoice Capture service in SAP Concur was below expected level. Affected users may have encountered invoices not being successfully delivered or processed via the Invoice Capture service. A workaround was not available. The Incident Response Team (IRT) made a configuration change on a business service tier to restore service. The incident's impact has resulted in a Invoice Capture processing backlog. The IRT projects the processing of the backlog to complete later today within the next several hours. Customers should now be able to see emails being delivered successfully. We have verified that service performance is stable and will now resolve the incident. An investigation into the root cause of this incident will be conducted , and a root cause analysis report provided when that investigation is complete.

7. resolved Apr 28, 2025, 07:23 AM UTC

   Impact as Reported: In the US2, EU2 and APJ1 Data Centers, sending invoices via email to the Invoice Capture service in SAP Concur was below expected level. Affected users may have encountered invoices not being successfully delivered or processed via the Invoice Capture service. Root cause: The root cause of the incident has been identified to be a configuration change that inadvertently overwrote the invoice mail service collector authentication parameters preventing it from successfully establishing connections on the internal network. This resulted in inbound emails not being forwarded to the invoice recognition services for extraction and processing and be queued until the issue was identified and addressed. Corrective Actions: The faulty authentication parameters have been fixed to restore service, and the faulty change process is being amended to ensure that any future updates will properly be applied without overwriting relevant code. Additionally, a cooperation between the RnD Service Team and the Incident Response Team has been initiated to improve detection and incident severity categorization for similar scenarios, with the goal to accelerate detection and resolution for potential future incidents.