Request: P1

Update timeline

1. resolved Apr 29, 2025, 03:58 AM UTC

   We are working to determine the user impact and will provide information as it becomes available.

2. resolved Apr 29, 2025, 04:26 AM UTC

   In the Concur Cloud for Public Sector Data Center, submitting travel requests through the SAP Concur website and the SAP Concur mobile app are below expected level. Concur Travel and Expense users may experience an intermittent error, “Sorry something went wrong” when attempting to submit travel requests. The Incident Response Team is currently investigating. Our next scheduled update will be in one hour or sooner should a significant change occur.

3. resolved Apr 29, 2025, 05:23 AM UTC

   User impact since the last update is unchanged. In the Concur Cloud for Public Sector Data Center, submitting travel requests through the SAP Concur website and the SAP Concur mobile app are below expected level. Concur Travel and Expense users may experience an intermittent error, “Sorry something went wrong” when attempting to submit travel requests. The Incident Response Team (IRT) continues with the investigation to identify root cause & determine the most effective course of action to resolve it. Our next scheduled update will be in one hour or sooner should a significant change occur.

4. resolved Apr 29, 2025, 06:21 AM UTC

   User impact since the last update is unchanged. In the Concur Cloud for Public Sector Data Center, submitting travel requests through the SAP Concur website and the SAP Concur mobile app are below expected level. Concur Travel and Expense users may experience an intermittent error, “Sorry something went wrong” when attempting to submit travel requests. The Incident Response Team (IRT) has engaged additional resources and is actively analyzing logs to determine the root cause of the incident. Our next scheduled update will be in one hour or sooner should a significant change occur.

5. resolved Apr 29, 2025, 07:21 AM UTC

   User impact since the last update is unchanged. In the Concur Cloud for Public Sector (CCPS) Data Center, submitting travel requests through the SAP Concur website and the SAP Concur mobile app are below expected level. Concur Travel and Expense users may experience an intermittent error, “Sorry, but something went wrong with this task. Try refreshing the page, or sign out and sign in again.” when attempting to submit travel requests. The Incident Response Team (IRT) is continuing its analysis of the issue with additional resources in order to identify the root cause. Our next scheduled update will be in one hour or sooner should a significant change occur.

6. resolved Apr 29, 2025, 08:21 AM UTC

   User impact since the last update is unchanged. In the Concur Cloud for Public Sector (CCPS) Data Center, submitting travel requests through the SAP Concur website and the SAP Concur mobile app are below expected level. Concur Travel and Expense users may experience an intermittent error, “Sorry, but something went wrong with this task. Try refreshing the page, or sign out and sign in again.” when attempting to submit travel requests. The Incident Response Team (IRT) has identified a possible root cause and is currently working on the necessary steps to resolve the issue. Our next scheduled update will be in two hour or sooner should a significant change occur.

7. resolved Apr 29, 2025, 10:21 AM UTC

   User impact since the last update is unchanged. In the Concur Cloud for Public Sector (CCPS) Data Center, submitting travel requests through the SAP Concur website and the SAP Concur mobile app are below expected level. Concur Travel and Expense users may experience an intermittent error, “Sorry, but something went wrong with this task. Try refreshing the page, or sign out and sign in again.” when attempting to submit travel requests. The Incident Response Team (IRT) is continuing to narrow down the identified possible root cause and is close to pinpointing the source of the issue. Our next scheduled update will be in two hours, or sooner if there is a significant change occur.

8. resolved Apr 29, 2025, 12:21 PM UTC

   User impact since the last update is changed. In the Concur Cloud for Public Sector (CCPS) Data Center, submitting travel requests through the SAP Concur website and the SAP Concur mobile app are below expected level. Affected users may intermittently encounter the error message: “Sorry, but something went wrong with this task. Try refreshing the page, or sign out and sign in again.” when attempting to submit travel requests. Users can click the "Close" button and then proceed to submit the travel request successfully. The Incident Response Team (IRT) has engaged additional resources to assist in investigating the identified possible root cause. Our next scheduled update will be in two hours, or sooner if there is a significant change occur.

9. resolved Apr 29, 2025, 12:33 PM UTC

   In the Concur Cloud for Public Sector (CCPS) Data Center, submitting travel requests through the SAP Concur website and the SAP Concur mobile app were below expected level. Affected users may have intermittently encountered the error message: “Sorry, but something went wrong with this task. Try refreshing the page, or sign out and sign in again.” when attempting to submit travel requests. However, users were able to successfully submit their requests after clicking the "Close" button on the error message. Services have returned to normal operating levels. We will continue monitoring performance for the next 15-30 minutes to verify stability. Once verified as stable the incident will be resolved.

10. resolved Apr 29, 2025, 12:56 PM UTC

    In the Concur Cloud for Public Sector (CCPS) Data Center, submitting travel requests through the SAP Concur website and the SAP Concur mobile app were below expected level. Affected users may have intermittently encountered the error message: “Sorry, but something went wrong with this task. Try refreshing the page, or sign out and sign in again.” when attempting to submit travel requests. However, users were able to successfully submit their travel requests after clicking the "Close" button on the error message. The Incident Response Team (IRT) re-deployed the previously implemented change to restore service. We have verified that service performance is stable and will now resolve the incident. An investigation into the root cause of this incident will now be conducted, and a root cause analysis report provided when that investigation is complete.

11. resolved May 12, 2025, 03:22 PM UTC

    In the Concur Cloud for Public Sector (CCPS) Data Center, submitting travel requests through the SAP Concur website and the SAP Concur mobile app were below expected level. Affected users may have intermittently encountered the error message: “Sorry, but something went wrong with this task. Try refreshing the page, or sign out and sign in again.” when attempting to submit travel requests. However, users were able to successfully submit their requests after clicking the "Close" button on the error message. The root cause of the incident has been determined to be a known issue that is being investigated in collaboration with our cloud service provider, where one of more instances of our container infrastructure get stuck in a bad state and the system is not able to terminate them and replaced them with new and healthy instances. This has caused an Imaging deployment to fail and a loop where the deployment automation has tried repeatedly to restart and redeploy, causing intermittent service downtime. The IRT has manually shut down the faulty container instances and rolled back the deployment, resulting in service restoration. While the investigation with our vendor continues, several internal actions are being taken to improve observability, incident response and mitigate the impact of possible future incidents.