Resolved after 27 minutes, affecting Expense, on February 2, 2025.
In the Concur Cloud for Public Sector (CCPS) Data Center, creating and submitting expense reports through the SAP Concur website and SAP Concur mobile app were below expected level. Affected users may have experienced a loading error when attempting to create and submit expense reports. The root cause of the incident has been identified in the degradation of an instance within our container ecosystem that became unresponsive and took about 25 minutes to be terminated, significantly delaying service recovery. The leading theory on why the automatic instance replacement took longer than intended is that a chron-job that was mistakenly set to run every minute instead of once per day, causing significantly higher resource utilization and ultimately leading to resource starvation and failures within the containers management application. Corrective Actions: The offending chron-job has been set to run once per day, additionally, improvements to instance recycling and auto scaling have been implemented to improve the efficiency of resource management.
Affected components
Update timeline
- resolved Feb 02, 2025, 04:23 PM UTC
In the Concur Cloud for Public Sector (CCPS) Data Center, creating and submitting expense reports through the SAP Concur website and SAP Concur mobile app were below expected level. Affected users may have experienced a loading error when attempting to create and submit expense reports. As the Incident Response Team (IRT) was being assembled, the service self restored. We have verified that service performance is stable and will now resolve the incident. An investigation into the root cause of this incident will now be conducted, and a root cause analysis report provided when that investigation is complete.
- resolved Feb 13, 2025, 07:40 PM UTC
In the Concur Cloud for Public Sector (CCPS) Data Center, creating and submitting expense reports through the SAP Concur website and SAP Concur mobile app were below expected level. Affected users may have experienced a loading error when attempting to create and submit expense reports. The root cause of the incident has been identified in the degradation of an instance within our container ecosystem that became unresponsive and took about 25 minutes to be terminated, significantly delaying service recovery. The leading theory on why the automatic instance replacement took longer than intended is that a chron-job that was mistakenly set to run every minute instead of once per day, causing significantly higher resource utilization and ultimately leading to resource starvation and failures within the containers management application. Corrective Actions: The offending chron-job has been set to run once per day, additionally, improvements to instance recycling and auto scaling have been implemented to improve the efficiency of resource management.
Your product runs on more than SAP Concur
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services