SAP Concur incident

Request: P1

Resolved after 11 minutes, affecting Request, on January 9, 2025.

resolved Jan 17, 2025, 01:46 PM UTC

In the EU2 Data Center, submitting Expense and Request reports was below expected level. The affected users were getting an error: “Unexpected System Error” when trying to submit an Expense or Request report through the SAP Concur website or SAP Concur mobile app. Root cause: The root cause of the incident has been identified in a database performance degradation caused by the execution of a resource demanding query with a sub-optimal query plan, which led to 100% CPU utilization on the database server and exhaustion of worker threads. As a result of the database performance issue, the application responsible for Travel Requests processing suffered a requests overflow generating high number of re-try and timeouts, to a point where the application pool was no longer able to process incoming traffic and resulting in the service to be unavailable for several minutes. Corrective Actions: The database entity responsible for the execution of the heavy query has been migrated to a database server with higher capacity; statistics on the affected database have been update to re-optimize the choice of the query plans. Timeouts and re-try values on the application pool are being reviewed and re-configured to ensure a better application response to database performance degradation, with the goal to provide impact mitigation and a better end user experience.

Started
Jan 09, 2025, 09:50 AM UTC
Resolved
Jan 09, 2025, 10:01 AM UTC
Duration
11m
Detected by Pingoru
Jan 09, 2025, 09:50 AM UTC

Affected components

Request

Update timeline

  1. resolved Jan 09, 2025, 11:35 AM UTC

    In the EU2 Data Center, submitting Request reports was below expected level. The affected users were getting an error: “Unexpected System Error” when trying to submit a Request report through the SAP Concur website or SAP Concur mobile app. While the Incident Response Team was assembled, the service self restored. We have verified that service performance is stable and will now resolve the incident. An investigation into the root cause of this incident will be conducted , and a root cause analysis report provided when that investigation is complete.

  2. resolved Jan 17, 2025, 01:46 PM UTC

    In the EU2 Data Center, submitting Expense and Request reports was below expected level. The affected users were getting an error: “Unexpected System Error” when trying to submit an Expense or Request report through the SAP Concur website or SAP Concur mobile app. Root cause: The root cause of the incident has been identified in a database performance degradation caused by the execution of a resource demanding query with a sub-optimal query plan, which led to 100% CPU utilization on the database server and exhaustion of worker threads. As a result of the database performance issue, the application responsible for Travel Requests processing suffered a requests overflow generating high number of re-try and timeouts, to a point where the application pool was no longer able to process incoming traffic and resulting in the service to be unavailable for several minutes. Corrective Actions: The database entity responsible for the execution of the heavy query has been migrated to a database server with higher capacity; statistics on the affected database have been update to re-optimize the choice of the query plans. Timeouts and re-try values on the application pool are being reviewed and re-configured to ensure a better application response to database performance degradation, with the goal to provide impact mitigation and a better end user experience.

Your product runs on more than SAP Concur

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →