Travel: P1

Update timeline

1. resolved Apr 28, 2025, 05:16 PM UTC

   We are working to determine the user impact and will provide information as it becomes available.

2. resolved Apr 28, 2025, 05:43 PM UTC

   In the US2 Data Center, car, air and hotel travel searches in the new Concur Travel, using the Sabre and Amadeus GDS services through the SAP Concur website is below expected level. Affected users may experience an error, “For some reason, something went wrong”. when attempting to perform travel searches. The Incident Response Team is currently investigating. Our next scheduled update will be in one hour or sooner should a significant change occur.

3. resolved Apr 28, 2025, 06:29 PM UTC

   User impact since the last update is unchanged. In the US2 Data Center, car, air and hotel travel searches in the new Concur Travel, using the Sabre and Amadeus GDS services through the SAP Concur website is below expected level. Affected users may experience an error, “For some reason, something went wrong”. when attempting to perform travel searches. The Incident Response Team (IRT) continues with the investigation to identify root cause & determine the most effective course of action to resolve it. Our next scheduled update will be in one hour or sooner should a significant change occur.

4. resolved Apr 28, 2025, 07:31 PM UTC

   In the US2 Data Center, air travel searches in the new Concur Travel, using the Sabre and Amadeus GDS services through the SAP Concur website is below expected level. Affected users may experience an error, “For some reason, something went wrong”. when attempting to perform travel searches. New Concur Travel users should now be able to search for car and hotel. The Incident Response Team (IRT) continues to investigate logs. Concurrently, the IRT will restart an business service tier in attempt to restore service. Our next scheduled update will be in one hour or sooner should a significant change occur.

5. resolved Apr 28, 2025, 08:31 PM UTC

   In the US2 Data Center, air travel searches in the new Concur Travel, using the Sabre and Amadeus GDS services through the SAP Concur website is below expected level. Affected users may experience an error, “For some reason, something went wrong”. when attempting to perform travel searches. New Concur Travel users should now be able to search for car and hotel. The restart did not restore service. The Incident Response Team (IRT) is taking action to increase space and capacity on a business service tier. Our next scheduled update will be in one hour or sooner should a significant change occur.

6. resolved Apr 28, 2025, 09:29 PM UTC

   In the US2 Data Center, air travel searches in the new Concur Travel, using the Sabre and Amadeus GDS services through the SAP Concur website is below expected level. Affected users may experience an error, “AR-4000”. when attempting to perform travel searches. New Concur Travel users should now be able to search for car and hotel. The Incident Response Team (IRT) continues in taking action to increase space and capacity on a business service tier. Our next scheduled update will be in one hour or sooner should a significant change occur.

7. resolved Apr 28, 2025, 10:29 PM UTC

   In the US2 Data Center, air travel searches in the new Concur Travel, using the Sabre and Amadeus GDS services through the SAP Concur website was below expected level. Affected users may have experienced an error, “AR-40000” when attempting to perform travel searches. New Concur Travel users should now be able to search for car and hotel. Services have returned to normal operating levels. We will continue monitoring performance for the next 15-30 minutes to verify stability. Once verified as stable the incident will be resolved.

8. resolved Apr 28, 2025, 10:49 PM UTC

   In the US2 Data Center, air travel searches in the new Concur Travel, using the Sabre and Amadeus GDS services through the SAP Concur website was below expected level. Affected users may have experienced an error, “AR-40000” when attempting to perform travel searches. New Concur Travel users should now be able to search for car and hotel. As the Incident Response Team was investigating, Issue service self-recovered. We have verified that service performance is stable and will now resolve the incident. An investigation into the root cause of this incident will now be conducted, and a root cause analysis report provided when that investigation is complete.

9. resolved May 12, 2025, 08:43 AM UTC

   Impact as Reported: In the US2 Data Center, air travel searches in the new Concur Travel, using the Sabre and Amadeus GDS services through the SAP Concur website was below expected level. Affected users may have experienced an error, “AR-40000” when attempting to perform travel searches. New Concur Travel users should now be able to search for car and hotel. Root cause: A travel service experienced connectivity issues to its caching tier. This increased latency to the service, eventually causing dependant services to timeout, and resulting in the air travel service to stop receiving traffic, triggering a capacity scale down. When the initial issue was resolved and traffic restored, the air travel service rapidly scaling up resulted in a sudden spike of connections overwhelming downstream dependencies and causing further disruption. The Incident Response Team throttled traffic to allow services to catch up and fully restore functionality. Corrective Actions: The following corrective and preventative actions were identified: Add data compression to data sent to caching Deploy a new caching cluster dedicated to air searches Increase minimum number of pods for air searches Update service to more gradually scale down when traffic is greatly reduced Add circuit breaker logic to keep air search from having too many requests