Increased Error Rates Impacting Core Services on FGA US-1

Update timeline

1. investigating Oct 31, 2024, 01:32 PM UTC

   We are currently investigating an incident that impacted our services. Our team is actively working to understand the root cause and mitigate the issue. We will provide updates as we gather more information.

2. identified Oct 31, 2024, 01:42 PM UTC

   Our Engineering team has identified the root cause of this issue and we are reviewing any applicable mitigation actions. We will provide additional updates here in 15 minutes.

3. identified Oct 31, 2024, 01:59 PM UTC

   Our Engineering team is continuing to review all relevant mitigation actions for this issue as we work to resolve this urgently. We will provide additional updates here in 15 minutes.

4. identified Oct 31, 2024, 02:16 PM UTC

   Additional mitigation actions are underway to resolve this issue. We will post additional updates here in approximately 20 minutes.

5. monitoring Oct 31, 2024, 02:19 PM UTC

   Our Engineering team is observing recovery now and services are functioning in the environment as expected.

6. resolved Oct 31, 2024, 03:16 PM UTC

   Services in the environment are functioning as expected and this issue has been resolved. A root cause analysis (RCA) will be posted here within 14 days. We sincerely apologize for the impact this had on you and your users.

7. postmortem Nov 13, 2024, 05:36 PM UTC

   A root-cause analysis for this issue has been performed and is now available at: [https://cdn.auth0.com/website/20241031_Increased_Error_Rates_on_FGA_US1.pdf](https://cdn.auth0.com/website/20241031_Increased_Error_Rates_on_FGA_US1.pdf)