Resolved after 2 hours 3 minutes, affecting Japan - Japan Public Transportation (JPT), on June 30, 2025.
Impact as Reported: In the US2 Data Center, creating JPT Route Search entries and adding entries made by Network IC card through the SAP Concur website and SAP Concur mobile app were below expected level. Affected users may have encountered an error when attempting to select a route using the JPT expense type. Additionally, approvers and processors were unable to view or verify the registered route information. Root cause: The incident was triggered by a sudden increase of load on a database induced by a customer initiated large batch processing job. Impact on the database itself was aggravated to the point of rendering the database almost unresponsive by poorly optimized queries that have now been identified; additionally, a circuit breaker mechanism implemented on the JPT service to protect it from load spikes was found to have been incorrectly designed resulting in the service being impacted for all users when triggered by a single unavailable database. Corrective Actions: The following corrective actions have been identified and will be implemented by the end of July 2025: the offending queries have been reviewed and optimized to mitigate the impact of unexpected large batch jobs the JPT circuit breaker has been revised to apply in a targeted manner and not disrupt service for all users the offending customer database entity is being temporarily segregated and monitored
Affected components
Update timeline
- resolved Jul 01, 2025, 01:57 AM UTC
We are working to determine the user impact and will provide information as it becomes available.
- resolved Jul 01, 2025, 02:19 AM UTC
In the US2 Data Center, creating JPT Route Search entries and entries made by Network IC card through the SAP Concur website and SAP Concur mobile app is below expected level. Affected users may encounter an error when attempting to select a route using the JPT expense type. The Incident Response Team (IRT) is currently investigating. Our next scheduled update will be in one hour or sooner should a significant change occur.
- resolved Jul 01, 2025, 02:57 AM UTC
In the US2 Data Center, creating JPT Route Search entries and adding entries made by Network IC card through the SAP Concur website and SAP Concur mobile app were below expected level. Affected users may have encountered an error when attempting to select a route using the JPT expense type. Additionally, approvers and processors were unable to view or verify the registered route information. Services have returned to normal operating levels. We will continue monitoring performance for the next 30-45 minutes to verify stability. Once verified as stable, the incident will be resolved.
- resolved Jul 01, 2025, 03:42 AM UTC
In the US2 Data Center, creating JPT Route Search entries and adding entries made by Network IC card through the SAP Concur website and SAP Concur mobile app were below expected level. Affected users may have encountered an error when attempting to select a route using the JPT expense type. Additionally, approvers and processors were unable to view or verify the registered route information. While the Incident Response Team (IRT) was actively investigating the issue, we observed that the affected service began to recover without manual intervention. We have verified that service performance is stable and will now resolve this incident. An investigation into the root cause of this incident will now be conducted, and a root cause analysis report will be provided when that investigation is complete.
- resolved Jul 22, 2025, 01:15 PM UTC
Impact as Reported: In the US2 Data Center, logins through the SAP Concur website and SAP Concur mobile app were below expected levels. Affected users may have experienced an error, "Sorry something went wrong" when attempting to login. Root cause: An unforeseen increase in requests data volume for a Concur core config service, resulted in its caching layer exhausting network capacity, leading to high latency and/or request failures also affecting several downstream services involved in the authentication workflow and Next Gen Travel search/booking process. What caused this critical increase in requests data volume is still being investigated, more info will be provided as it becomes available. Corrective Actions Monitoring and alerting has been enhanced to provide visibility into network bandwidth utilization metrics for the various micro-services caching layer. The caching infrastructure is undergoing a sharding process to strongly mitigate the blast radius of similar incidents in the future. Additional action items are likely to result from the investigation that is still in progress.
- resolved Jul 24, 2025, 11:07 AM UTC
Impact as Reported: In the US2 Data Center, creating JPT Route Search entries and adding entries made by Network IC card through the SAP Concur website and SAP Concur mobile app were below expected level. Affected users may have encountered an error when attempting to select a route using the JPT expense type. Additionally, approvers and processors were unable to view or verify the registered route information. Root cause: The incident was triggered by a sudden increase of load on a database induced by a customer initiated large batch processing job. Impact on the database itself was aggravated to the point of rendering the database almost unresponsive by poorly optimized queries that have now been identified; additionally, a circuit breaker mechanism implemented on the JPT service to protect it from load spikes was found to have been incorrectly designed resulting in the service being impacted for all users when triggered by a single unavailable database. Corrective Actions: The following corrective actions have been identified and will be implemented by the end of July 2025: the offending queries have been reviewed and optimized to mitigate the impact of unexpected large batch jobs the JPT circuit breaker has been revised to apply in a targeted manner and not disrupt service for all users the offending customer database entity is being temporarily segregated and monitored
Your product runs on more than SAP Concur
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services