Invoice: P1

Update timeline

1. resolved Oct 06, 2025, 04:09 AM UTC

   We are working to determine the user impact and will provide information as it becomes available.

2. resolved Oct 06, 2025, 04:42 AM UTC

   In the US2,EU2,APJ1 Data Centers, sending invoices via email to the Invoice Capture service in SAP Concur is below expected level. Affected users may not receive Invoice Capture emails. The Incident Response Team (IRT) is currently investigating. Our next scheduled update will be in one hour or sooner should a significant change occur.

3. resolved Oct 06, 2025, 05:35 AM UTC

   In the US2,EU2,APJ1 Data Centers, sending invoices via email to the Invoice Capture service in SAP Concur is below expected level. Affected users may not receive Invoice Capture emails. The Incident Response Team (IRT) continues investigation into finding out the root cause. Our next scheduled update will be in one hour or sooner should a significant change occur.

4. resolved Oct 06, 2025, 06:41 AM UTC

   In the US2,EU2,APJ1 Data Centers, sending invoices via email to the Invoice Capture service in SAP Concur was below expected level. Affected users may have not received Invoice Capture emails. The Incident Response Team (IRT) has pinpointed the root cause as a DNS issue. They implemented a configuration change as a temporary workaround, which restored the service for now. However, further investigation into the DNS issue is still in progress. We will continue monitoring performance for the next 3 hours to verify stability. Once verified as stable the incident will be resolved.

5. resolved Oct 06, 2025, 08:55 AM UTC

   In the US2,EU2,APJ1 Data Centers, sending invoices via email to the Invoice Capture service in SAP Concur was not available. Affected users may have not received Invoice Capture emails. The Incident Response Team (IRT) reverted the temporary configuration change after the DNS issue had been resolved. We have verified that service performance is stable and will now resolve this incident. An investigation into the root cause of this incident will now be conducted, and a root cause analysis report provided when that investigation is complete.

6. resolved Oct 06, 2025, 05:13 PM UTC

   In the US2,EU2,APJ1 Data Centers, sending invoices via email to the Invoice Capture service in SAP Concur was temporarily unavailable for four hours and forty six minutes. During this incident, affected users may not have received their Invoice Capture emails. All affected emails were securely stored on an internal server during the incident and have now been successfully delivered to their intended recipients. No further action is required.

7. resolved Oct 16, 2025, 12:31 PM UTC

   Impact as Reported: In the US2,EU2,APJ1 Data Centers, sending invoices via email to the Invoice Capture service in SAP Concur was not available. Affected users may have not received Invoice Capture emails. Root cause: Routine automated node patching unexpectedly overwhelmed all DNS nodes in the environment hosting the mail servers receiving and processing invoice capture emails and caused the resolvers to become unresponsive. As they were unable to resolve url's and establish connections to downstream services the mail servers were unable to forward received invoice emails for processing. Service was restored by manually restarting the impacted DNS nodes. Corrective Actions: The following corrective and preventive actions have been identified: The patching automation has been updated to address the issue causing the target nodes to become unresponsive. Monitoring of the DNS nodes has been reviewed and updated to ensure detection of system issues.