SAP Concur incident
Expense, Travel, Invoice, Request, Imaging, Analysis/Intelligence: P1
Resolved after 8 hours 25 minutes, affecting Analysis/Intelligence and Expense and 1 more, on July 7, 2025.
Impact as Reported: In the US2 Data Center, logins through the SAP Concur website and SAP Concur mobile app are below expected levels. Affected users may experience an error, "Sorry something went wrong" when attempting to login. Root cause: A code defect triggered by the selection of specific localization options resulted in an unexpected increase in the payload size for data stored in a cache used by a Next Gen Travel backend service. While the condition persisted the network capacity to the caching layer became saturated and triggered traffic throttling, with a cascading latency impact that eventually resulted in the disruption of search and booking functionalities. Due to a monitoring gap the network saturation was not initially detected by the incident response team, delaying the identification of the root cause of the problem while the investigation was focusing on issues on other parts of the system which were eventually found to be a result of the degradation on the caching layer. Once the actual problem was identified the caching layer was flushed to remove the oversized data sets and the application tier recycled and scaled up to restore service. Corrective Actions The following corrective and preventative actions have been identified: Monitoring and alerting has been enhanced to full visibility of the caching layer, including network metrics The caching infrastructure is undergoing a sharding process to increase resiliency to unexpected abnormal data loads. Data throttling is being implemented at the application layer to reduce the possibility and mitigate the impact of similar code defects. The containers scaling configuration for the service in question is being tuned to better respond to scenarios where the service needs to be restarted under load. The code defect is being remediated.
Affected components
Update timeline
- resolved Jul 07, 2025, 02:07 PM UTC
We are working to determine the user impact and will provide information as it becomes available.
- resolved Jul 07, 2025, 02:30 PM UTC
In the US2 Data Center, logins through the SAP Concur website and SAP Concur mobile app are below expected levels. Affected users may experience an error, "Sorry something went wrong" when attempting to login. The Incident Response Team (IRT) is currently investigating. Our next scheduled update will be in one hour or sooner should a significant change occur.
- resolved Jul 07, 2025, 03:19 PM UTC
User impact remains unchanged. In the US2 Data Center, logins through the SAP Concur website and SAP Concur mobile app are below expected levels. Affected users may experience an error, "Sorry something went wrong" when attempting to login. The Incident Response Team (IRT) took action to increase capacity. However, this action did not provide relief. The IRT is continuing to research a potential database issue to provide resolution. Our next scheduled update will be in one hour or sooner should a significant change occur.
- resolved Jul 07, 2025, 04:18 PM UTC
User impact since last update is unchanged. In the US2 Data Center, logins through the SAP Concur website and SAP Concur mobile app are below expected levels. Affected users may experience an error, "Sorry something went wrong" when attempting to login. After monitoring internal observability tools following the initial capacity increase, the Incident Response Team (IRT) is now taking action to provide additional capacity on an application server tier. After this action is complete, the IRT will continue monitoring to confirm successful results. Our next scheduled update will be in one hour or sooner should a significant change occur
- resolved Jul 07, 2025, 05:38 PM UTC
User impact since last update is unchanged. In the US2 Data Center, logins through the SAP Concur website and SAP Concur mobile app are below expected levels. Affected users may experience an error, "Sorry something went wrong" when attempting to login. The Incident Response Team (IRT) is continuing to investigate. The previous action of increasing capacity has not provided relief. Additional teams have been engaged for further research. Our next scheduled update will be in one hour or sooner should a significant change occur.
- resolved Jul 07, 2025, 06:41 PM UTC
In the US2 Data Center, logins through the SAP Concur website and SAP Concur mobile app are below expected levels. Affected users may experience an error, "Sorry something went wrong" when attempting to login. The Incident Response Team (IRT) has identified a possible root cause and is actively working on a solution that may resolve the issue. Further details regarding completion will be shared as soon as they become available. Our next scheduled update will be in one hour or sooner should a significant change occur.
- resolved Jul 07, 2025, 07:10 PM UTC
In the US2 Data Center, logins through the SAP Concur website and SAP Concur mobile app were below expected levels. Affected users may have experienced an error, "Sorry something went wrong" when attempting to login. Services have returned to normal operating levels. We will continue monitoring performance for the next 15-30 minutes to verify stability. Once verified as stable the incident will be resolved.
- resolved Jul 07, 2025, 08:36 PM UTC
In the US2 Data Center, logins through the SAP Concur website and SAP mobile app was below expected levels. Affected users may have received an error, "Sorry, something went wrong" when attempting to log in. This issue has been resolved. However, we have observed a new degradation to our services. In the US2 Data Center, travel bookings are below expected levels in new Concur Travel through the SAP Concur website and SAP Concur mobile app. Affected users may experience an error, "For some reason something went wrong" when attempting to book air, car, or hotel travel. The Incident Response Team (IRT) is currently investigating. Our next scheduled update will be in one hour or sooner should a significant change occur
- resolved Jul 07, 2025, 09:49 PM UTC
In the US2 Data Center, travel bookings are below expected levels in the New Concur Travel through the SAP Concur website and SAP Concur mobile app. Affected users may experience an error, "For some reason something went wrong" when attempting to book air, car, or hotel travel. The Incident Response Team (IRT) continues investigating to identify the underlying root cause. Our next scheduled update will be in one hour or sooner should a significant change occur.
- resolved Jul 07, 2025, 11:00 PM UTC
In the US2 Data Center, travel bookings are below expected levels in new Concur Travel through the SAP Concur website and SAP Concur mobile app. Affected users may experience an error, "For some reason something went wrong" when attempting to book air, car, or hotel travel. Additionally, some travel configurations are being redirected to the legacy version, where the system may be functioning as expected. The Incident Response Team (IRT) has identified the root cause and is currently working car booking issue to increase server capacity in order to fully resolve the issue. Our next scheduled update will be in one hour or sooner should a significant change occur.
- resolved Jul 07, 2025, 11:54 PM UTC
In the US2 Data Center, travel bookings are below expected levels in new Concur Travel through the SAP Concur website and SAP Concur mobile app. Affected users may experience an error, "For some reason something went wrong" when attempting to book air, car, or hotel travel. The Incident Response Team (IRT) is currently updating an application configuration to restore service. Our next scheduled update will be in one hour or sooner should a significant change occur.
- resolved Jul 08, 2025, 12:42 AM UTC
In the US2 Data Center, travel bookings were below expected levels in new Concur Travel through the SAP Concur website and SAP Concur mobile app. Affected users may have experienced an error, "For some reason something went wrong" when attempting to book air, car, or hotel travel. Additionally, some travel configurations were being redirected to the legacy version, where the system may be functioning as expected. Services have returned to normal operating levels. We will continue monitoring performance for the next 15-30 minutes to verify stability. Once verified as stable the incident will be resolved.
- resolved Jul 08, 2025, 01:01 AM UTC
In the US2 Data Center, travel bookings were below expected levels in new Concur Travel through the SAP Concur website and SAP Concur mobile app. Affected users may have experienced an error, "For some reason something went wrong" when attempting to book air, car, or hotel travel. Additionally, some travel configurations were being redirected to the legacy version, where the system may be functioning as expected. The Incident Response Team (IRT) performed a configuration change to restore service. We have verified that service performance is stable and will now resolve this incident. An investigation into the root cause of this incident will now be conducted, and a root cause analysis report provided when that investigation is complete.
- resolved Jul 22, 2025, 01:17 PM UTC
Impact as Reported: In the US2 Data Center, logins through the SAP Concur website and SAP Concur mobile app were below expected levels. Affected users may have experienced an error, "Sorry something went wrong" when attempting to login. Root cause: An unforeseen increase in requests data volume for a Concur core config service, resulted in its caching layer exhausting network capacity, leading to high latency and/or request failures also affecting several downstream services involved in the authentication workflow and Next Gen Travel search/booking process. What caused this critical increase in requests data volume is still being investigated, more info will be provided as it becomes available. Corrective Actions Monitoring and alerting has been enhanced to provide visibility into network bandwidth utilization metrics for the various micro-services caching layer. The caching infrastructure is undergoing a sharding process to strongly mitigate the blast radius of similar incidents in the future. Additional action items are likely to result from the investigation that is still in progress.
- resolved Aug 05, 2025, 11:57 AM UTC
Impact as Reported: In the US2 Data Center, logins through the SAP Concur website and SAP Concur mobile app are below expected levels. Affected users may experience an error, "Sorry something went wrong" when attempting to login. Root cause: A code defect triggered by the selection of specific localization options resulted in an unexpected increase in the payload size for data stored in a cache used by a Next Gen Travel backend service. While the condition persisted the network capacity to the caching layer became saturated and triggered traffic throttling, with a cascading latency impact that eventually resulted in the disruption of search and booking functionalities. Due to a monitoring gap the network saturation was not initially detected by the incident response team, delaying the identification of the root cause of the problem while the investigation was focusing on issues on other parts of the system which were eventually found to be a result of the degradation on the caching layer. Once the actual problem was identified the caching layer was flushed to remove the oversized data sets and the application tier recycled and scaled up to restore service. Corrective Actions The following corrective and preventative actions have been identified: Monitoring and alerting has been enhanced to full visibility of the caching layer, including network metrics The caching infrastructure is undergoing a sharding process to increase resiliency to unexpected abnormal data loads. Data throttling is being implemented at the application layer to reduce the possibility and mitigate the impact of similar code defects. The containers scaling configuration for the service in question is being tuned to better respond to scenarios where the service needs to be restarted under load. The code defect is being remediated.
Your product runs on more than SAP Concur
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services