Invoice: P1

Update timeline

1. resolved Jul 11, 2025, 03:09 PM UTC

   In the EU2 and US2 Data Centers, customers are encountering issues managing Invoices after submission, creating new Invoices, or viewing Vendor details. For the affected users, Invoice header field values are missing after the submission thus prohibiting Invoice approval and recall. Other users may encounter the following error when trying to create an invoice, "Assigned employee does not have access to the policy." The Incident Response Team has identified root cause and preparing a fix for deployment. Our next scheduled update will be in one hour or sooner should a significant change occur.

2. resolved Jul 11, 2025, 04:09 PM UTC

   In the EU2 and US2 Data Centers, customers are encountering issues managing Invoices after submission, creating new Invoices, or viewing Vendor details. For the affected users, Invoice header field values are missing after the submission thus prohibiting Invoice approval and recall. Other users may encounter the following error when trying to create an invoice, "Assigned employee does not have access to the policy." The Incident Response Team is deploying the fix to an internal environment for testing. Our next scheduled update will be in two hours or sooner should a significant change occur.

3. resolved Jul 11, 2025, 06:12 PM UTC

   In the EU2 and US2 Data Centers, customers are encountering issues managing Invoices after submission, creating new Invoices, or viewing Vendor details. For the affected users, Invoice header field values are missing after the submission thus prohibiting Invoice approval and recall. Other users may encounter the following error when trying to create an invoice, "Assigned employee does not have access to the policy." The fix deployment to the internal environment completed with errors. The IRT will investigate, resolve the errors, and proceed with a fix deploy re-attempt to the internal environment for testing. Our next scheduled update will be in two hours or sooner should a significant change occur.

4. resolved Jul 11, 2025, 08:13 PM UTC

   In the EU2 and US2 Data Centers, customers are encountering issues managing Invoices after submission, creating new Invoices, or viewing Vendor details. For the affected users, Invoice header field values are missing after the submission thus prohibiting Invoice approval and recall. Other users may encounter the following error when trying to create an invoice, "Assigned employee does not have access to the policy." The fix deployment to the internal environment is in progress. Upon completion, testing will be performed to validate the fix. The Incident Response Team will proceed to deploy the fix to production environments if testing yields successful results. Our next scheduled update will be in four hours or sooner should a significant change occur.

5. resolved Jul 12, 2025, 12:15 AM UTC

   In the EU2 and US2 Data Centers, customers are encountering issues managing Invoices after submission, creating new Invoices, or viewing Vendor details. For the affected users, Invoice header field values are missing after the submission thus prohibiting Invoice approval and recall. Other users may encounter the following error when trying to create an invoice, "Assigned employee does not have access to the policy." The fix deployment to production environments is in progress. Upon completion and testing, we will communicate incident recovery and start the monitoring stability phase. Our next scheduled update will be in four hours or sooner should a significant change occur.

6. resolved Jul 12, 2025, 12:59 AM UTC

   In the EU2 and US2 Data Centers, customers encountered issues managing Invoices after submission, creating new Invoices, or viewing Vendor details. For the affected users, Invoice header field values were missing after the submission thus prohibiting Invoice approval and recall. Other users may have encountered the following error when trying to create an invoice, "Assigned employee does not have access to the policy." Services have returned to normal operating levels. We will continue monitoring performance for the next 15-30 minutes to verify stability. Once verified as stable the incident will be resolved.

7. resolved Jul 12, 2025, 01:23 AM UTC

   In the EU2 and US2 Data Centers, customers encountered issues managing Invoices after submission, creating new Invoices, or viewing Vendor details. For the affected users, Invoice header field values were missing after the submission thus prohibiting Invoice approval and recall. Other users may have encountered the following error when trying to create an invoice, "Assigned employee does not have access to the policy." The Incident Response Team deployed a fix that made changes to a previous release to restore service. We have verified that service performance is stable and will now resolve the incident. An investigation into the root cause of this incident will now be conducted, and a root cause analysis report provided when that investigation is complete.

8. resolved Jul 28, 2025, 09:55 AM UTC

   Interim RCA Impact as Reported: In the EU2 and US2 Data Centers, customers encountered issues managing Invoices after submission, creating new Invoices, or viewing Vendor details. For the affected users, Invoice header field values were missing after the submission thus prohibiting Invoice approval and recall. Other users may have encountered the following error when trying to create an invoice, "Assigned employee does not have access to the policy." Root cause: The monthly release made a change to the drop-down list used in Expense and Invoice forms. The change had a defect in the Invoice form list causing the data to not load in the drop-down even though it was available. This had the effect of appearing missing to the users. A hotfix was deployed to remove the change and remediate the issue. We are continuing to review the root cause to identify any further corrective and preventative actions. Corrective Actions: Corrective actions will be determined once root cause has been fully identified.

9. resolved Aug 28, 2025, 01:03 PM UTC

   Impact as Reported: In the EU2 and US2 Data Centers, customers encountered issues managing Invoices after submission, creating new Invoices, or viewing Vendor details. For the affected users, Invoice header field values were missing after the submission thus prohibiting Invoice approval and recall. Other users may have encountered the following error when trying to create an invoice, "Assigned employee does not have access to the policy." Root cause: The monthly service update made a change to the drop-down list used in Expense and Invoice forms. A defect in the code impacting the Invoice form lists resulted in data not loading in its drop-down even though it was available in the system, making it appear as missing to users. A hotfix was deployed to roll back the change and remediate the issue. Corrective Actions: A hotfix was deployed to address the code defect. Code reviewers assignments have been updated for codebase used by multiple services to ensure all use cases are reviewed and tested. Triaging playbooks are being updated to improve code deployments reviews to improve troubleshooting times.