Travel: P1

Update timeline

1. resolved May 15, 2025, 01:22 PM UTC

   We are working to determine the user impact and will provide information as it becomes available.

2. resolved May 15, 2025, 01:31 PM UTC

   In the Concur Cloud for Public Sector Data Center (CCPS), air, car and hotel searches and bookings are below expected level. The affected users may experience slowness during travel search and booking process, eventually resulting in error “A duplicate booking attempt from the same user on the same car/date is detected”. The Incident Response Team (IRT) has been engaged and are investigating. Our next scheduled update will be in one hour or sooner, should a significant change occur.

3. resolved May 15, 2025, 02:20 PM UTC

   In the Concur Cloud for Public Sector Data Center, travel bookings are below expected levels. Concur Travel and Expense users may experience latency when attempting to perform travel searches or book travel. Additionally, affected users may experience an error, “We’re sorry. We encountered an issue that prevented us from completing your request” when attempting to perform tasks. ConcurGov users may experience latency when attempting to perform travel searches or book travel. Additionally, affected users may intermittently experience an error, “We’re sorry. We encountered an issue that prevented us from completing your request” when attempting to perform tasks. It may be possible to perform searches and travel bookings by performing multiple attempts. The Incident Response Team (IRT) has engaged additional resources to investigate a potential networking issue. Our next scheduled update will be in one hour or sooner should a significant change occur.

4. resolved May 15, 2025, 03:13 PM UTC

   In the Concur Cloud for Public Sector Data Center, travel bookings are below expected levels. Concur Travel and Expense users may experience latency when attempting to perform travel searches or book travel. Additionally, affected users may experience an error, “We’re sorry. We encountered an issue that prevented us from completing your request” when attempting to perform tasks. ConcurGov users may experience latency when attempting to perform travel searches or book travel. Additionally, affected users may intermittently experience an error, “We’re sorry. We encountered an issue that prevented us from completing your request” when attempting to perform tasks. It may be possible to perform searches and travel bookings by performing multiple attempts. The Incident Response Team (IRT) performed an investigation into a potential networking issue. However, the research into networking issues did not provide a root cause. Further investigation ruled out database errors. The IRT is continuing their research to determine appropriate next steps. Our next scheduled update will be in one hour or sooner should a significant change occur.

5. resolved May 15, 2025, 04:04 PM UTC

   In the Concur Cloud for Public Sector Data Center, travel bookings are below expected levels. Concur Travel and Expense users may experience latency when attempting to perform travel searches or book travel. Additionally, affected users may experience an error, “We’re sorry. We encountered an issue that prevented us from completing your request” when attempting to perform tasks. ConcurGov users may experience latency when attempting to perform travel searches or book travel. Additionally, affected users may intermittently experience an error, “We’re sorry. We encountered an issue that prevented us from completing your request” when attempting to perform tasks. It may be possible to perform searches and travel bookings by performing multiple attempts. The Incident Response Team (IRT) took action to increase capacity on a travel sever tier, but testing confirmed that while this may have provided some relief, errors and latency persisted. The IRT is currently performing a review of internal logs in an effort to determine root cause. To allow sufficient investigation prior to next update, the incident communication timeline will be extended. Our next scheduled update will be in two hours or sooner should a significant change occur.

6. resolved May 15, 2025, 06:23 PM UTC

   In the Concur Cloud for Public Sector Data Center, travel bookings were below expected levels. Concur Travel and Expense users may have experienced latency when attempting to perform travel searches or book travel. Additionally, affected users may have experienced an error, “We’re sorry. We encountered an issue that prevented us from completing your request” when attempting to perform tasks. ConcurGov users may have experienced latency when attempting to perform travel searches or book travel. Additionally, affected users may have intermittently experienced an error, “We’re sorry. We encountered an issue that prevented us from completing your request” when attempting to perform tasks. It may have been possible to perform searches and travel bookings by performing multiple attempts. Services have returned to normal operating levels. We will continue monitoring performance for the next 15-30 minutes to verify stability. Once verified as stable the incident will be resolved.

7. resolved May 15, 2025, 06:38 PM UTC

   In the Concur Cloud for Public Sector Data Center, travel bookings were below expected levels. Concur Travel and Expense users may have experienced latency when attempting to perform travel searches or book travel. Additionally, affected users may have experienced an error, “We’re sorry. We encountered an issue that prevented us from completing your request” when attempting to perform tasks. ConcurGov users may have experienced latency when attempting to perform travel searches or book travel. Additionally, affected users may have intermittently experienced an error, “We’re sorry. We encountered an issue that prevented us from completing your request” when attempting to perform tasks. It may have been possible to perform searches and travel bookings by performing multiple attempts. The Incident Response Team (IRT) increased capacity on a travel service tier to restore service. We have verified that service performance is stable and will now resolve this incident. An investigation into the root cause of this incident will now be conducted, and a root cause analysis report provided when that investigation is complete.

8. resolved May 23, 2025, 10:00 AM UTC

   Impact as Reported: In the Concur Cloud for Public Sector Data Center, travel bookings were below expected levels. Concur Travel and Expense users may have experienced latency when attempting to perform travel searches or book travel. Additionally, affected users may have experienced an error, “We’re sorry. We encountered an issue that prevented us from completing your request” when attempting to perform tasks. ConcurGov users may have experienced latency when attempting to perform travel searches or book travel. Additionally, affected users may have intermittently experienced an error, “We’re sorry. We encountered an issue that prevented us from completing your request” when attempting to perform tasks. It may have been possible to perform searches and travel bookings by performing multiple attempts. Root cause: A scheduled process involved in the completion of travel bookings suffering unexpected performance issues that are intermittently impacting response time for travel searches and bookings. The issue has been mitigated by increasing compute capacity but the incident response team is still investigating root cause to permanently address the issue. We will provide an update once the investigation is complete. Corrective Actions: Corrective actions will be determined once root cause has been fully identified.

9. resolved Jun 13, 2025, 05:24 PM UTC

   In the Concur Cloud for Public Sector Data Center, travel bookings were below expected levels. Concur Travel and Expense users may have experienced latency when attempting to perform travel searches or book travel. Additionally, affected users may have experienced an error, “We’re sorry. We encountered an issue that prevented us from completing your request” when attempting to perform tasks. ConcurGov users may have experienced latency when attempting to perform travel searches or book travel. Additionally, affected users may have intermittently experienced an error, “We’re sorry. We encountered an issue that prevented us from completing your request” when attempting to perform tasks. It may have been possible to perform searches and travel bookings by performing multiple attempts. A scheduled process involved in the completion of travel bookings suffered unexpected performance issues intermittently impacting response time for travel searches and bookings. The issue has been mitigated by increasing compute capacity. Engineering has narrowed the root cause to the code path experiencing issues and timeout failures. The team will work on optimizations to improve performance. Corrective Actions: The following corrective actions were identified: - Increase the minimum number of instances to use for running the scheduled process. - Adjust the auto-scaling of the instances to be more aggressive in scaling up. - Migrated to newer cloud hardware to improve performance of the instances. - The preventative action is to migrate the scheduled jobs to run on their own instances to prevent blocking travel searches.