Expense: P1

Update timeline

1. resolved Apr 30, 2026, 03:53 PM UTC

   We are working to determine the user impact and will provide information as it becomes available.

2. resolved Apr 30, 2026, 04:13 PM UTC

   As of 3:21 PM UTC on 30 April 2026, our internal monitoring indicates a degradation within the Expense Service in the US2 Data Center, affecting the creation and submission of expense reports and related expense task workflows, impacting 100% of customers. The degradation is resulting in an unstable service that is not usable due to poor performance, causing severe delays in service‑wide business operations. When you try to create and submit an expense report or complete expense‑related tasks, you may experience extreme latency and sporadic failures with an error message. Attempted creations, submissions, and tasks are not completing. As a potential workaround, you may be able to try again; however, latency or performance issues may still be experienced. The Incident Response Team (IRT) has gathered, and an investigation is beginning. Additional teams have been engaged, and research is underway into a potential network issue. Our next scheduled update will be provided in one hour or sooner should a significant change occur.

3. resolved Apr 30, 2026, 05:07 PM UTC

   User impact has improved, but some latency and performance issues are continuing. As of 3:21 PM UTC on 30 April 2026, our internal monitoring indicates a degradation within the Expense Service in the US2 Data Center, affecting the creation and submission of expense reports and related expense task workflows, impacting 100% of customers. The degradation is resulting in an unstable service that is not usable due to poor performance, causing severe delays in service‑wide business operations. When you try to create and submit an expense report or complete expense‑related tasks, you may experience extreme latency and sporadic failures with an error message. Attempted creations, submissions, and tasks are not completing. As a potential workaround, you may be able to try again; however, latency or performance issues may still be experienced. The IRT has determined that networking is stable. Upon investigation of internal logs, a restart was performed on pods identified as experiencing impact. This has resulted in some performance improvement,. However, internal monitoring tools are still indicating some performance issues, and internal replication is reflecting that while latency has improved, performance issues are persisting. Our next scheduled update will be provided in one hour or sooner should a significant change occur.

4. resolved Apr 30, 2026, 05:23 PM UTC

   As of 3:21 PM UTC on 30 April 2026, our internal monitoring indicated a degradation within the Expense Service in the US2 Data Center, affecting the creation and submission of expense reports and related expense task worklows, impacting 100% of customers. The degradation resulted in an unstable service that was unusable due to poor performance, causing severe delays in service‑wide business operations. When attempting to create and submit an expense report or complete expense‑related tasks, you experienced extreme latency and sporadic failures with an error message. Attempted creations, submissions, and tasks were not completing. As a potential workaround, you may have been able to try again; however, latency or performance issues may still have been experienced. Services have returned to normal operating levels. We will continue monitoring performance for the next 15-30 minutes to verify stability. Once verified as stable the incident will be resolved.

5. resolved Apr 30, 2026, 05:40 PM UTC

   As of 3:21 PM UTC on 30 April 2026, our internal monitoring indicated a degradation within the Expense Service in the US2 Data Center, affecting the creation and submission of expense reports and related expense task worklows, impacting 100% of customers. The degradation resulted in an unstable service that was unusable due to poor performance, causing severe delays in service‑wide business operations. When attempting to create and submit an expense report or complete expense‑related tasks, you experienced extreme latency and sporadic failures with an error message. Attempted creations, submissions, and tasks were not completing. As a potential workaround, you may have been able to try again; however, latency or performance issues may still have been experienced. The Incident Response Team (IRT) took action to increase capacity in order to restore service. We have verified that service performance is stable and will now resolve the incident. An investigation into the root cause of this incident will be conducted , and a root cause analysis report provided when that investigation is complete.

6. resolved May 14, 2026, 02:42 PM UTC

   Impact as Reported: Our internal monitoring indicated a degradation within the Expense Service in the US2 Data Center, affecting the creation and submission of expense reports and related expense task workflows, impacting 100% of customers. The degradation resulted in an unstable service that was unusable due to poor performance, causing severe delays in service‑wide business operations. When attempting to create and submit an expense report or complete expense‑related tasks, you experienced extreme latency and sporadic failures with an error message. Attempted creations, submissions, and tasks were not completing. Root cause: The Profile Service experienced a significant spike in traffic, which led to performance degradation when users attempted to submit expenses. For expenses involving a travel allowance workflow, users encountered errors. The Incident Response Team identified a code defect in an expense-related service affecting how it handled missing user profiles. Instead of failing gracefully, the service repeatedly attempted to retrieve profiles for users who did not have one, placing excessive load on the Profile Service. This issue affected all dependent services. In particular, the Travel Allowance Service experienced container crashes and returned errors. To resolve the issue, the Incident Response Team stopped the processes that were repeatedly attempting to retrieve non-existent profiles and increased rate limits on the Profile Service. These actions stabilized the system and restored normal latency. Corrective Measures: The following corrective and preventative actions were identified: -Add a limit to number of tasks to process at one time for the impacted service. -Cache the results if user has no profile so that it doesn't keep trying the same user multiple times. -Increased capacity on travel allowance.