Expense, Mobile: P1

Update timeline

1. resolved Feb 11, 2025, 04:10 PM UTC

   We are working to determine the user impact and will provide information as it becomes available.

2. resolved Feb 11, 2025, 04:33 PM UTC

   In the US2 and EU2 Data Centers, GRDC receipt image processing is unavailable via the SAP Concur mobile app. Affected users may experience latency after clicking "Take Photo", followed by "Processing" without advancement in the workflow when attempting to upload a receipt. The Incident Response Team is currently investigating. Our next scheduled update will be in one hour or sooner should a significant change occur.

3. resolved Feb 11, 2025, 05:25 PM UTC

   In the US2 and EU2 Data Centers, GRDC receipt image processing is unavailable via the SAP Concur mobile app. Affected users may experience latency after clicking "Take Photo", followed by "Processing" without advancement in the workflow when attempting to upload a receipt. The Incident Response Team (IRT) has determined the outage is related to a third party vendor outage, and is currently working with the vendor to resolve. Our next scheduled update will be in one hour or sooner should a significant change occur.

4. resolved Feb 11, 2025, 06:25 PM UTC

   In the US2 and EU2 Data Centers, GRDC receipt image processing is unavailable via the SAP Concur mobile app. Affected users may experience latency after clicking "Take Photo", followed by "Processing" without advancement in the workflow when attempting to upload a receipt. The Incident Response Team (IRT) continues their investigation with the third party vendor. Additionally, the team is investigating the potential for internal actions that may help to alleviate impact. Our next scheduled update will be in one hour or sooner should a significant change occur.

5. resolved Feb 11, 2025, 07:15 PM UTC

   In the US2 and EU2 Data Centers, GRDC receipt image processing is unavailable via the SAP Concur mobile app. Affected users may experience latency after clicking "Take Photo", followed by "Processing" without advancement in the workflow when attempting to upload a receipt. The Incident Response Team (IRT) is currently performing a configuration update and testing the update to determine if impact can be resolved via internal actions. If testing should yield positive results, the update will be deployed via a hotfix. Our next scheduled update will be in one hour or sooner should a significant change occur.

6. resolved Feb 11, 2025, 08:13 PM UTC

   In the US2 and EU2 Data Centers, GRDC receipt image processing is unavailable via the SAP Concur mobile app. Affected users may experience latency after clicking "Take Photo", followed by "Processing" without advancement in the workflow when attempting to upload a receipt. The Incident Response Team (IRT) continues their work to complete the configuration change and testing to confirm if this configuration change will resolve impact. Upon completion and validation of successful testing, the team will take action to deploy the hotfix. Our next scheduled update will be in one hour or sooner should a significant change occur.

7. resolved Feb 11, 2025, 08:27 PM UTC

   In the US2 and EU2 Data Centers, GRDC receipt image processing was unavailable via the SAP Concur mobile app. Affected users may have experienced latency after clicking "Take Photo", followed by "Processing" without advancement in the workflow when attempting to upload a receipt. Services have returned to normal operating levels. We will continue monitoring performance for the next 15-30 minutes to verify stability. Once verified as stable the incident will be resolved.

8. resolved Feb 11, 2025, 08:48 PM UTC

   In the US2 and EU2 Data Centers, GRDC receipt image processing is unavailable via the SAP Concur mobile app. Affected users may experience latency after clicking "Take Photo", followed by "Processing" without advancement in the workflow when attempting to upload a receipt. We have observed a new disruption to our services during the monitoring period. The Incident Response Team (IRT) is investigating the issue. Our next scheduled update will be in one hour or sooner should a significant change occur.

9. resolved Feb 11, 2025, 09:58 PM UTC

   In the US2 and EU2 Data Centers, GRDC receipt image processing is unavailable via the SAP Concur mobile app. Affected users may experience latency after clicking "Take Photo", followed by "Processing" without advancement in the workflow when attempting to upload a receipt. The Incident Response Team (IRT) received notification that the third party vendor restored service. After a brief recovery period, impact to the service returned. The IRT continued their mitigation plan completing the configuration change, however testing yielded unsuccessful results. The IRT is engaging additional resources to investigate a potential network issue. Our next scheduled update will be in two hours or sooner should a significant change occur.

10. resolved Feb 11, 2025, 11:45 PM UTC

    In the US2 and EU2 Data Centers, GRDC receipt image processing is unavailable via the SAP Concur mobile app. Affected users may experience latency after clicking "Take Photo", followed by "Processing" without advancement in the workflow when attempting to upload a receipt. The Incident Response Team (IRT) is working on the configuration change and running tests to complete the deployment of the fix, which will restore the service. They have completed the deployment in US2 and are working on completing the fix in EU2. Our next scheduled update will be in two hours or sooner should a significant change occur.

11. resolved Feb 12, 2025, 12:35 AM UTC

    In the US2 and EU2 Data Centers, GRDC receipt image processing was unavailable via the SAP Concur mobile app. Affected users may have experienced latency after clicking "Take Photo", followed by "Processing" without advancement in the workflow when attempting to upload a receipt. Services have returned to normal operating levels. We will continue monitoring performance for the next 15-30 minutes to verify stability. Once verified as stable the incident will be resolved.

12. resolved Feb 12, 2025, 12:37 AM UTC

    In the US2 and EU2 Data Centers, GRDC receipt image processing was unavailable via the SAP Concur mobile app. Affected users may have experienced latency after clicking "Take Photo", followed by "Processing" without advancement in the workflow when attempting to upload a receipt. The Incident Response Team made a configuration change which restored the service. An investigation into the root cause of this incident will now be conducted, and a root cause analysis report provided when that investigation is complete

13. resolved Mar 03, 2025, 11:05 AM UTC

    Impact as Reported: In the US2 and EU2 Data Centers, GRDC receipt image processing was unavailable via the SAP Concur mobile app. Affected users may have experienced latency after clicking "Take Photo", followed by "Processing" without advancement in the workflow when attempting to upload a receipt. Root cause: An outage for the primary third party time stamp service provider used by Concur caused the GRDC receipt image processing service to became unavailable for end users. The incident response team switched to an alternative time stamp service provider to restore service. Corrective Actions: The following corrective actions have been identified: monitoring on the feature is being enhanced to ensure earlier detection of service degradation and prompter response time stamp service providers are being reviewed to ensure stable service offering.