Expense, Travel, Request, Imaging, Analysis/Intelligence: P1

Update timeline

1. resolved Jan 24, 2025, 04:57 AM UTC

   We are working to determine the user impact and will provide information as it becomes available.

2. resolved Jan 24, 2025, 05:26 AM UTC

   In the Concur Cloud for Public Sector (CCPS) Data Center, logins through the SAP Concur website and the SAP Concur mobile app is unavailable. Affected users may encounter a "502 Bad Gateway" error when attempting to login. The Incident Response Team (IRT) is currently investigating. Our next scheduled update will be in one hour or sooner should a significant change occur.

3. resolved Jan 24, 2025, 06:00 AM UTC

   In the Concur Cloud for Public Sector (CCPS) Data Center, logins through the SAP Concur website and the SAP Concur mobile app were unavailable. Affected users may have encountered a "502 Bad Gateway" error. Services have returned to normal operating levels. We will continue monitoring performance for the next 15-30 minutes to verify stability. Once verified as stable the incident will be resolved.

4. resolved Jan 24, 2025, 06:18 AM UTC

   In the Concur Cloud for Public Sector (CCPS) Data Center, logins through the SAP Concur website and the SAP Concur mobile app were unavailable. Affected users may have encountered a "502 Bad Gateway" error. The Incident Response Team (IRT) updated the configuration settings on the application server tier to restore service. We have verified that service performance is stable and will now resolve this incident. An investigation into the root cause of this incident will now be conducted, and a root cause analysis report provided when that investigation is complete.

5. resolved Feb 06, 2025, 06:00 PM UTC

   In the Concur Cloud for Public Sector (CCPS) Data Center, logins through the SAP Concur website and the SAP Concur mobile app were unavailable. Affected users may have encountered a "502 Bad Gateway" error. Root cause: During a standard upgrade procedure on the container hosting clusters in the SAP Concur public sector environment an unexpected behaviour by the underlying hyperscaler infrastructure led to two of the target clusters to be left without active nodes and consequently to the unavailability of the services running on those clusters. The incident response team restored service by manually reprovisioning serving capacity on the impacted clusters. Corrective Actions: The following corrective actions have been identified: -The automation used for cluster updates is being updated to address insufficient error handling and improve failure modes, to ensure that serving capacity is always available even in case of unexpected infrastructure issues -The hyperscaler vendor has been engaged to identify and address the cause of the unexpected behavior.