Partial outage for Economy V2 affecting Catalog APIs and Inventory APIs

Update timeline

1. identified May 21, 2025, 11:17 PM UTC

   The issue has been identified, it was due to migrations for service improvements, we are currently working on a fix

2. monitoring May 21, 2025, 11:28 PM UTC

   Issue has been mitigated, we're monitoring for complete recovery

3. resolved May 21, 2025, 11:43 PM UTC

   Migration is completed and incident mitigation is complete

4. postmortem Jun 10, 2025, 10:57 PM UTC

   On May 21st, 2025, between 3:58 PM and 4:25 PM PDT, some customers experienced a partial outage impacting large API groups, including Catalog and Inventory services on PlayFab. The incident was caused by a configuration change during infrastructure migration, which resulted in a connectivity issue for backend databases. The issue was resolved by updating access settings to restore proper connectivity to the affected services. ### Impact During the incident, customers encountered degraded service with failed requests across multiple services reliant on Catalog—including Inventory and Redemption. Service availability dropped significantly below expected levels for approximately 27 minutes. ### Root Cause Analysis The outage was triggered when a migration enabled new access controls on backend databases. This inadvertently blocked necessary connections, leading to failures for dependent services. The infrastructure update caused public access to be disabled, preventing services from communicating as expected. ### Action Items To prevent similar incidents from happening again, we have taken the following actions: · We updated our migration procedures to ensure all required access settings are reviewed and applied before infrastructure changes. · We enhanced our monitoring to detect connectivity issues immediately after infrastructure updates.