Analysis/Intelligence: P1

Update timeline

1. resolved Apr 16, 2025, 02:08 PM UTC

   We are working to determine the user impact and will provide information as it becomes available.

2. resolved Apr 16, 2025, 02:28 PM UTC

   In the EU2 Data Center, access to the Analysis/Intelligence service through the SAP Concur website is unavailable for 25% of customers. Affected users may encounter an error: "An error has occurred. Please try again", when attempting to access Analysis/Intelligence after clicking the Reporting tab. No workaround has been identified. The Incident Response Team is currently investigating. Our next scheduled update will be in one hour or sooner should a significant change occur.

3. resolved Apr 16, 2025, 02:45 PM UTC

   In the EU2 Data Center, access to the Analysis/Intelligence service through the SAP Concur website was unavailable for a 25% of customers. Affected users may have encountered an error: "An error has occurred. Please try again", when attempting to access Analysis/Intelligence after clicking the Reporting tab. No workaround was identified. To check if your company was affected by this incident, please log onto the Personalized Concur Open website. Services have returned to normal operating levels. We will continue monitoring performance for the next 15-30 minutes to verify stability. Once verified as stable the incident will be resolved.

4. resolved Apr 16, 2025, 02:54 PM UTC

   In the EU2 Data Center, access to the Analysis/Intelligence service through the SAP Concur website was unavailable for a 25% of customers. Affected users may have encountered an error: "An error has occurred. Please try again", when attempting to access Analysis/Intelligence after clicking the Reporting tab. No workaround was identified. To check if your company was affected by this incident, please log onto the Personalized Concur Open website. As the Incident Response Team was investigating, the service self-recovered. We have verified that service performance is stable and will now resolve the incident. An investigation into the root cause of this incident will now be conducted, and a root cause analysis report provided when that investigation is complete.

5. resolved Apr 29, 2025, 01:12 PM UTC

   Impact as Reported: In the EU2 Data Center, access to the Analysis/Intelligence service through the SAP Concur website was unavailable for a subset of customers. Affected users may have encountered an error: "An error has occurred. Please try again", when attempting to access Analysis/Intelligence after clicking the Reporting tab. Root cause: The root cause of the incident has been determined to be the degradation of a storage volume within our cloud service provider infrastructure, resulting in instability for one of our Analytics databases, and ultimately leading to inability for the application tier to establish connections to it. An automated failover for the impacted database was not triggered while the degraded instance was still passing the existing health checks. Once the storage failure was fixed and the database restored to a normal state, the influx of connections previously queued caused application locking and resulted in a second brief outage. Corrective Actions: A work-stream has been initiated to improve monitoring over hyperscaler storage failures, with the goal to shorten fail-over executions, hence shorten the recovery time for similar scenarios. Also, an investigation has been started by the application team, aimed to manage the amount of connections that get attempted and queued during periods of database unavailability.