PlayFab Outage History

Started Feb 05, 2026, 08:53 PM UTC· resolved Feb 05, 2026, 10:24 PM UTC

1. investigating Feb 05, 2026, 08:53 PM UTC

   We are investigating an issue with PurchaseInventoryItem. Error responses take the form of InvalidRequest errors beginning at 18:20 UTC.

2. investigating Feb 05, 2026, 09:05 PM UTC

   We are continuing to investigate this issue.

3. monitoring Feb 05, 2026, 10:14 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Feb 05, 2026, 10:24 PM UTC

   InvalidRequest errors have subsided.

Read the full incident report →

Started Feb 02, 2026, 06:30 PM UTC· resolved Feb 03, 2026, 10:00 AM UTC

1. identified Feb 02, 2026, 08:14 PM UTC

   Customers may face delays in scaling up Multiplayer Game Servers across multiple regions

2. monitoring Feb 03, 2026, 12:25 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Feb 03, 2026, 03:36 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 28, 2026, 10:35 PM UTC· resolved Jan 29, 2026, 04:06 AM UTC

1. investigating Jan 28, 2026, 10:35 PM UTC

   We are currently investigating an issue with Ban updates causing a delay in our processing and updates to Bans that have changed in the last few day across some titles. This is currently impacting and causing inconsistencies: - Across the following APIs: Admin/UpdateBans, Admin/RevokeBans, Admin/RevokeAllBansForUser, Server/UpdateBans, Server/RevokeBans, Server/RevokeAllBansForUser - Modifying bans via the player bans management page in the developer portal

2. resolved Jan 29, 2026, 04:06 AM UTC

   A fix was deployed, the delays have caught up and bans should be current now.

3. postmortem Feb 17, 2026, 10:59 PM UTC

   On January 28, 2026, between approximately 1:35 PM and 8:10 PM PDT, some customers experienced delays related to ban processing. During this time, our global indexing pipeline for ban-related data fell behind after encountering invalid customer data that caused batch processing failures. We mitigated the issue by deploying a hotfix to skip invalid affected records, allowing the indexing pipeline to catch up and normal processing to resume. ### Impact During the incident window \(about 6 hours and 34 minutes\), ban-related indexing was delayed. As a result, scenarios that rely on timely ban updates may have taken longer than expected to take effect. We did not observe data loss; the primary impact was delayed processing. Any bans enacted during the impact period were eventually applied. ### Root Cause Analysis A customer-created record contained an invalid value that was interpreted incorrectly during decoding \(in this case, an invalid date string\). Because our validation did not sufficiently enforce expected formats for certain fields, the bad data made it into the ban-related data set and caused repeated batch failures in the indexing pipeline. Those failures prevented the pipeline from making forward progress until the problematic records were bypassed. ### Action Items To reduce the likelihood and impact of similar issues, we completed the following actions: * Deployed a hotfix to safely skip invalid records so indexing can continue even when encountering bad data. * Improved error logging in the indexing processor to speed up diagnosis if similar processing failures occur.

Read the full incident report →

Started Jan 19, 2026, 02:15 AM UTC· resolved Jan 19, 2026, 09:45 AM UTC

1. investigating Jan 19, 2026, 02:15 AM UTC

   Billing Summary daily bill estimates may not contain the most recent day's estimate. We are currently investigating this issue, and expect to populate all daily billing estimates in the summary when the issue is resolved

2. monitoring Jan 19, 2026, 04:10 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jan 19, 2026, 04:14 PM UTC

   This incident has been resolved.

4. postmortem Jan 28, 2026, 05:15 PM UTC

   On January 18, 2026, between approximately 6:00 PM and 10:45 PM PT, some customers experienced delayed updates in the PlayFab Billing Summary report. The issue was triggered by a change in an external data query dependency, and service was restored after a hotfix was deployed and the processing pipelines were re-run. ### Impact The daily Billing Summary update \(including the “last updated” timestamp, daily CSV, and refreshed month-to-date estimates\) was delayed by roughly 4.5 hours. Customers viewing Billing Summary during that window may have seen estimates that were older than expected. ### Root Cause Analysis An external partner change introduced a new required parameter for a data update operation used by our billing reporting pipeline. This caused part of the daily aggregation and update flow to fail until we adjusted the operation and reprocessed the affected data. ### Action Items To prevent similar incidents from happening again, we are taking the following actions: * Improving our monitoring process to more quickly detect failures in the end-to-end billing update flow so delays can be identified and addressed sooner. * Investigating improvements to communication with external dependency team for upcoming changes on their end

Read the full incident report →

Started Jan 13, 2026, 11:55 PM UTC· resolved Jan 20, 2026, 09:14 PM UTC

1. identified Jan 13, 2026, 11:55 PM UTC

   We have identified an issue and are working on fixing an issue with delayed scorecard generation for existing PlayFab experiments. Experiments created on or after Jan 13th are working as expected.

2. identified Jan 17, 2026, 12:37 AM UTC

   A mitigation is being implemented for experiments created before Jan 13th to display updated scorecard data in PlayFab.

3. monitoring Jan 19, 2026, 04:31 AM UTC

   A fix has been implemented for experiments created before January 13th, we will continue to validate and monitor health.

4. resolved Jan 20, 2026, 09:14 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 13, 2026, 07:15 PM UTC· resolved Jan 14, 2026, 03:18 AM UTC

1. monitoring Jan 14, 2026, 03:12 AM UTC

   A fix has been implemented and we are monitoring the results.

2. resolved Jan 14, 2026, 03:18 AM UTC

   This incident has been resolved.

3. postmortem Jan 21, 2026, 12:16 AM UTC

   On January 13, 2026, between approximately 11:15 AM and 7:23 PM PST, some customers experienced increased failures when using PlayFab Matchmaking, including errors when calling matchmaking-related APIs \(such as GetMatch\). This was caused by a configuration update that applied more restrictive request limits than intended for some titles. We resolved the issue by rolling back the configuration change and monitoring service health to confirm recovery. ### Impact A subset of titles using the PlayFab Matchmaking Service experienced elevated API failures, which could cause matchmaking flows to fail and lead to a degraded gameplay experience until the rollback was completed. ### Root Cause Analysis A configuration change intended to adjust request limits did not map as expected to existing limits for certain APIs. As a result, some titles were throttled more aggressively than intended, increasing request failures. Because monitoring did not immediately highlight the affected titles, the issue was detected later and then mitigated by reverting the change. ### Action Items To prevent similar incidents from happening again, we have taken the following action: * Improved our throttling telemetry so request-throttling metrics are easier to interpret during investigations.

Read the full incident report →

Started Dec 23, 2025, 07:37 PM UTC· resolved Dec 23, 2025, 10:20 PM UTC

1. resolved Dec 23, 2025, 07:41 PM UTC

   Due to a misconfiguration, the PlayFab service was under-reporting operational data such as API volume by approximately 30%, which resulted in lower usage reported on the dashboards for the Title Overview page. There was no impact on data used for Daily Totals Reports. The issue has since been mitigated (the issue began on December 5 and was resolved on December 22).

2. postmortem Jan 06, 2026, 11:16 PM UTC

   Between December 5, 2025, 11:37 PST and December 22, 2025, 14:20 PST, some PlayFab customers experienced inaccurate metrics reporting for certain dashboards \(which were underreported by as much as 30%\). The issue was caused by an infrastructure misconfiguration. ### Impact Some PlayFab dashboards displayed inaccurate or incomplete data during this period. Metrics such as API call volumes, login counts, and cloudscript processing times were underreported by up to 30%. No customer gameplay or operational functionality was affected, but visibility into service activity and performance was temporarily reduced. ### Root Cause Analysis A component responsible for metrics reporting was misconfigured, leading to the data not being displayed in dashboards at full accuracy. ### Action Items * We updated our infrastructure to ensure metrics are always sent to the correct environment. * We improved monitoring to better detect and alert on unexpected drops in metrics volume.

Read the full incident report →

Started Dec 03, 2025, 10:02 PM UTC· resolved Dec 04, 2025, 05:51 PM UTC

1. identified Dec 03, 2025, 10:02 PM UTC

   The issue has been identified and a fix is being implemented.

2. resolved Dec 04, 2025, 05:51 PM UTC

   This incident has been resolved.

3. postmortem Dec 18, 2025, 11:05 PM UTC

   **Incident Overview** Between October 21st, 2025, and December 3rd, 2025, users were unable to redeem a subset of marketplace purchases for catalog items created during this window. The incident was caused by a misconfiguration affecting a redemption processor that prevented a subset of newly created items from being available for redemption. The issue was resolved by fixing the misconfiguration. **Impact** Users were unable to redeem a subset \(3-4%\) of items that were created during the window. All purchases made during this period have been redeemable since the fix on December 3rd. **Action Items** To prevent similar incidents from happening again, we have taken the following actions: * Additional validation to avoid misconfigurations affecting the redemption processing system * Enhanced alerting for the redemption processor to improve detection of similar issues

Read the full incident report →

Started Nov 26, 2025, 12:53 AM UTC· resolved Nov 26, 2025, 01:18 AM UTC

1. investigating Nov 26, 2025, 12:53 AM UTC

   PlayFab APIs are experiencing intermittent failures in Leaderboard, Lobby and Game Saves services due to failures in an underlying infrastructure dependency. API success rates in these spaces are between 80-100%, with retries leading to success. Failures will manifest as 4XX and 5XX errors.

2. monitoring Nov 26, 2025, 01:06 AM UTC

   A fix has been deployed (4:35PM PST) and failure rates are quickly returning to normal. We are continuing to monitor.

3. resolved Nov 26, 2025, 01:18 AM UTC

   This incident has been resolved.

4. postmortem Jan 07, 2026, 09:14 PM UTC

   On November 25, 2025, between 15:15 PST and 17:00 PST, some customers experienced reduced availability when accessing Leaderboard, Lobby and Game Saves APIs. The incident was caused by a DNS outage in the West US 3 region, which affected communication with our infrastructure. The issue was resolved automatically once the DNS service was restored. ### Impact During the incident, the availability of the FindLobby and FindFriendLobbies APIs dropped to approximately 96%, the availability of all GameSave APIs dropped to an average of 80%, and downstream processing for Leaderboards was delayed. As a result, customers were unable to query for lobbies or upload new game saves using these APIs for about 1 hour and 45 minutes. Leaderboard updates were delayed. Service quality was temporarily below our standard threshold. ### Root Cause Analysis The incident was caused by a DNS outage in the West US 3 region. This outage disrupted communication between our Lobby, GameSave and Leaderboard services and their supporting infrastructure, resulting in reduced API availability. The root cause was related to an external dependency and not due to a code or configuration change within PlayFab. ### Action Items To prevent similar incidents in the future, we have enhanced our monitoring systems to detect and report anomalies in API availability more quickly. We are also reviewing our infrastructure dependencies to improve resiliency against external outages.

Read the full incident report →

Started Nov 24, 2025, 07:37 PM UTC· resolved Nov 24, 2025, 11:25 PM UTC

1. investigating Nov 24, 2025, 07:37 PM UTC

   We are investigating failures causing the Upgrade Account / Plan Recommendation page to not load properly.

2. investigating Nov 24, 2025, 07:37 PM UTC

   We are continuing to investigate this issue.

3. resolved Nov 24, 2025, 11:25 PM UTC

   This incident has been resolved.

4. postmortem Jan 09, 2026, 11:50 PM UTC

   On November 23rd–24th, 2025, some customers experienced errors when using PlayFab’s Plan Recommendation and Pricing Sync features. The incident was caused by an authentication issue in a billing service component, which disrupted several related services. We resolved the issue by updating the connection logic for our data service. ### Impact Customers were unable to access plan recommendations and update workflows. The issue was detected by our monitoring systems and reported by customers. ### Root Cause Analysis The authentication errors were triggered by changes in the underlying .Net version, which affected how a billing service component connected to our data service. ### Resolution We updated the connection logic for the billing service component, restoring normal operation and notifying affected customers. ### Action Items * Improved the connection logic for our data service in the affected billing service component. * Enhanced monitoring to detect authentication issues earlier.

Read the full incident report →

Started Nov 14, 2025, 08:46 PM UTC· resolved Nov 15, 2025, 12:13 AM UTC

1. investigating Nov 14, 2025, 08:46 PM UTC

   We are currently experiencing a delay in bulk segmentation features. Customers may experience delays is seeing player profile updates, segment exports and scheduled task.

2. identified Nov 14, 2025, 11:28 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Nov 14, 2025, 11:59 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Nov 15, 2025, 12:13 AM UTC

   This incident has been resolved.

5. postmortem Nov 18, 2025, 11:39 PM UTC

   On November 14, 2025, between 8:00 AM and 4:54 PM PST, some PlayFab customers experienced delays in Bulk Segmentation features. This resulted in delays in player profile updates, segment exports, and scheduled tasks. The issue was caused by a state in the ADX cluster powering the features, which led to the materialized view becoming unhealthy. We applied manual recovery steps and collaborated with the ADX team to resolve the problem, as the cluster was in a degraded state. ### Impact Customers experienced delays in player profile updates, segment exports, and scheduled tasks due to the degraded health of the player profile materialized view. Bulk Segmentation features were specifically impacted during this period. ### Root Cause Analysis The issue was triggered by an unexpected state in the ADX cluster powering the features, which caused materialized views to stop processing new data. This led to delays for customers and required manual recovery steps to restore normal functionality. ### Action Items · Implemented safeguards to prevent materialized views from entering an unhealthy state in the future.

Read the full incident report →

Started Oct 29, 2025, 04:17 PM UTC· resolved Oct 29, 2025, 11:00 PM UTC

1. investigating Oct 29, 2025, 04:17 PM UTC

   We are currently investigating issues related to connectivity across several services.

2. monitoring Oct 29, 2025, 05:14 PM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring Oct 30, 2025, 04:31 AM UTC

   We are continuing to monitor for any further issues.

4. resolved Oct 30, 2025, 04:37 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 24, 2025, 04:32 AM UTC· resolved Oct 24, 2025, 01:55 PM UTC

1. investigating Oct 24, 2025, 04:32 AM UTC

   We are investigating errors when creating or updating Data Connections definitions in PlayFab.

2. resolved Oct 24, 2025, 01:55 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 15, 2025, 10:30 PM UTC· resolved Oct 15, 2025, 10:45 PM UTC

1. monitoring Oct 15, 2025, 10:30 PM UTC

   We are investigating an increased failure rate for the Client/LoginWithXbox API.

2. resolved Oct 15, 2025, 10:58 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 15, 2025, 10:00 PM UTC· resolved Oct 15, 2025, 10:00 PM UTC

1. resolved Oct 15, 2025, 10:20 PM UTC

   We are investigated an failure rate for the Client/LoginWithXbox API.

Read the full incident report →

Started Oct 09, 2025, 07:54 PM UTC· resolved Oct 09, 2025, 09:40 PM UTC

1. investigating Oct 09, 2025, 07:55 PM UTC

   We are observing increased Error Rates for RedeemPlaystationStoreInventoryItems, ConsumePS5Entitlements, and ConsumePSNEntitlements APIs

2. investigating Oct 09, 2025, 08:00 PM UTC

   We are continuing to investigate this issue.

3. resolved Oct 09, 2025, 09:40 PM UTC

   This incident has been resolved.

4. postmortem Oct 14, 2025, 10:01 PM UTC

   On October 9th, 2025, between 12:20 PM and 2:40 PM PDT, some customers experienced increased failures when accessing the GameClient/ConsumePS5Entitlements and GameClient/ConsumePSNEntitlements APIs, as well as a few related APIs. The incident was caused by an outage in a third-party dependency, which resulted in our APIs receiving increased exceptions and errors. The issue was resolved once the external outage ended and normal service resumed. ### Impact Customers were unable to successfully complete requests through affected entitlements APIs for approximately two hours. This resulted in failed attempts to consume entitlements, impacting the user experience for games relying on these APIs. ### Root Cause Analysis The root cause of the incident was an external issue: a service outage in a third-party dependency. As our APIs depend on calls to the external service for authentication and entitlement verification, the outage led to increased exceptions and errors being returned to our services. ### Action Items To prevent similar incidents from happening again, we have taken the following actions: * Enhanced our monitoring and alerting systems to provide earlier detection of external partner outages affecting entitlements APIs. * Reviewed and improved our exception handling and error messaging to ensure clear communication to customers when third-party outages impact API availability.

Read the full incident report →

Started Sep 17, 2025, 04:01 AM UTC· resolved Sep 17, 2025, 05:35 AM UTC

1. investigating Sep 17, 2025, 04:01 AM UTC

   We're currently experiencing a higher than normal level of Lobby API errors and are investigating the issue.

2. monitoring Sep 17, 2025, 05:27 AM UTC

   We have identified the issue and are currently monitoring the APIs to ensure stability

3. resolved Sep 17, 2025, 05:35 AM UTC

   This incident has been solved

4. postmortem Sep 23, 2025, 09:04 PM UTC

   On September 16, 2025, between 8:20 PM and 10:42 PM PDT, some customers experienced issues with PlayFab Lobbies. During this period, players across all titles using Lobbies were unable to create, join, or update lobbies, although search APIs remained functional. The incident was caused by a maintenance event internal to our Redis infrastructure, which disrupted connectivity between the Lobby service and Redis. We resolved the issue by rebooting the Redis instances, restoring connectivity and service functionality. ### Impact Players were unable to create, join, or update lobbies across all titles that use the PlayFab Lobby system for approximately 2 hours and 22 minutes. While search APIs continued to work, players could not join lobbies they found during this time. ### Root Cause Analysis The root cause of the incident was a maintenance event within our Redis infrastructure that unexpectedly killed connectivity for the Lobby service. While we have the capability to schedule maintenance, this event occurred exactly when the impact started, leading to the service disruption. ### Action Items To prevent similar incidents from happening again, we have taken the following actions: * Updated the Lobby service to better display its status on our service overview dashboard, which will enhance visibility and expedite future detection and mitigation efforts. * Improved our internal processes for scheduling and communicating maintenance events that could impact service connectivity.

Read the full incident report →

Started Sep 09, 2025, 11:28 PM UTC· resolved Sep 10, 2025, 04:24 PM UTC

1. identified Sep 09, 2025, 11:28 PM UTC

   We are currently experiencing an issue with an upstream dependency that is impacting the PlayFab Experimentation service. As a result, scorecard generation is delayed. Our engineering team is actively working with the upstream provider to resolve the issue and restore full functionality. We will continue to provide updates as progress is made.

2. monitoring Sep 09, 2025, 11:54 PM UTC

   The upstream service has rolled back their feature. All impacted scorecards are now being regenerated.

3. resolved Sep 10, 2025, 04:24 PM UTC

   This incident has been resolved. All impacted scorecards are regenerated.

Read the full incident report →

Started Sep 01, 2025, 03:40 AM UTC· resolved Sep 01, 2025, 06:09 PM UTC

1. investigating Sep 01, 2025, 04:56 AM UTC

   We are currently investigating this issue.

2. investigating Sep 01, 2025, 04:57 AM UTC

   The matchmaking feature is experiencing a major degradation. Titles utilizing real-time notifications experienced very low success rates depending upon their scenarios.

3. monitoring Sep 01, 2025, 06:26 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Sep 01, 2025, 06:09 PM UTC

   This incident has been resolved. Impact was observed from 20:40 PDT to 21:30 PDT. Note that this was after the previous related incident.

Read the full incident report →

Started Aug 31, 2025, 07:00 PM UTC· resolved Sep 01, 2025, 03:28 AM UTC

1. investigating Aug 31, 2025, 07:56 PM UTC

   PlayFab Matchmaking APIs are experiencing degraded performance. Approximately 50% of calls to the Matchmaking APIs are returning 500 internal server error. The team is currently investigating the issue.

2. investigating Aug 31, 2025, 07:57 PM UTC

   We are continuing to investigate this issue.

3. identified Sep 01, 2025, 01:18 AM UTC

   The matchmaking feature is experiencing a major degradation. Titles utilizing real-time notifications experienced very low success rates depending upon their scenarios. A mitigation is in place and we are monitoring recovery.

4. monitoring Sep 01, 2025, 02:53 AM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Sep 01, 2025, 03:28 AM UTC

   This incident has been resolved.

6. postmortem Sep 09, 2025, 11:34 PM UTC

   On August 31, 2025, between 11:50 AM and 7:53 PM PDT, some customers experienced a complete outage of PlayFab Matchmaking. The incident was caused by a regression in the authentication library, which led to unexpected high call rates to Entra ID for token acquisition, resulting in service throttling and failures. We resolved the issue by increasing authentication rate limits, updating the authentication code, recycling affected application instances, and isolating impacted titles into dedicated deployments. ### Impact During the outage, matchmaking services were unavailable for all affected titles, preventing players from joining matches. Notification failures also occurred, resulting in players being stuck in matchmaking and not receiving proper updates. ### Root Cause Analysis The root cause of the incident was a regression in the updated authentication library, which failed to cache tokens as expected. This led to a surge in token requests to Entra ID, causing throttling and authentication failures across the service. The updated library created new instances for each authentication attempt, breaking the caching mechanism and increasing network traffic unexpectedly. ### Action Items To prevent similar incidents from happening again, we have taken the following actions: * Implemented service-level caching for authentication tokens to reduce unnecessary external calls. * Enhanced monitoring and diagnostics to detect abnormal network patterns and authentication failures more quickly. * Improved notification handling and recycling processes to prevent players from being stuck in matchmaking.

Read the full incident report →

Started Aug 19, 2025, 04:30 PM UTC· resolved Aug 19, 2025, 08:44 PM UTC

1. identified Aug 19, 2025, 05:10 PM UTC

   We are investigating an increase in errors for Authentication/ValidateToken

2. identified Aug 19, 2025, 05:10 PM UTC

   We are continuing to work on a fix for this issue.

3. resolved Aug 19, 2025, 08:44 PM UTC

   This incident has been resolved.

4. postmortem Aug 26, 2025, 08:35 PM UTC

   On August 19, 2025, between 9:32 AM and 10:50 AM PDT, some customers experienced failures when calling PlayFab APIs due to issues with entity token validation. The incident was caused by a switch in the JSON library used for serialization and deserialization of entity tokens \(for improved security and efficiency\). We resolved the issue by rolling back the serialization change and advising impacted customers to refresh their tokens. ### Impact During the incident, calls to validate entity tokens failed, preventing all further API calls using those tokens. Affected APIs primarily included WriteTelemetryEvents, Catalog search, GetPlayFabIdFrom\*Id, ExecuteCloudScript, and Group-related APIs. This resulted in service disruption for customers relying on these endpoints. ### Root Cause Analysis The root cause was a deployment that switched the JSON library responsible for entity token serialization. After rolling back the change, errors persisted for tokens minted during the incident window, requiring customers to refresh their tokens. ### Action Items To prevent similar incidents from happening again, we have taken the following actions: * Added documentation to the entity token release process to highlight serialization risks and warnings. * Implemented additional backcompat token tests for the entity token to expand test

Read the full incident report →

Started Aug 14, 2025, 09:15 PM UTC· resolved Aug 14, 2025, 10:11 PM UTC

1. investigating Aug 14, 2025, 09:57 PM UTC

   We are currently experiencing an issue in our data delivery pipeline. Customers may notice a delay in events appearing via Data Explorer or in Data Connections. Real-time processing of PlayStream events (e.g. for Automation Rules or WebHooks) is also impacted. No data has been lost and engineers are working to resolve this as soon as possible.

2. identified Aug 14, 2025, 10:00 PM UTC

   We have identified the issue and are working on a fix.

3. monitoring Aug 14, 2025, 10:06 PM UTC

   A fix has been deployed and we are monitoring the results.

4. resolved Aug 14, 2025, 10:11 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 09, 2025, 04:00 AM UTC· resolved Aug 09, 2025, 08:15 AM UTC

1. resolved Aug 12, 2025, 05:52 PM UTC

   The PlayFab Catalog service experienced a drop in availability for about 5 hours. This was due to a misconfigured service. The issue has been resolved and the service is now back to fully operational.

2. postmortem Aug 19, 2025, 08:34 PM UTC

   On August 8, 2025, between 10:55 PM and 1:15 AM PDT, some customers experienced errors and delays when searching or purchasing catalog items through PlayFab’s APIs. The incident was caused by a faulty service configuration intended for experimentation in a single region, which was mistakenly deployed to all regions due to human error. This misconfiguration led to increased internal server errors and service unavailability. The issue was resolved by rolling back the faulty configuration and confirming service recovery through monitoring and health checks. ### Impact Approximately 12,000 players were affected, unable to search or purchase catalog items. During the outage, the latency of Catalog APIs \(particularly SearchItems and GetItems\) increased significantly, and many requests were rejected with a “Service unavailable” status. ### Root Cause Analysis The root cause was a human error in deploying a service configuration intended for limited traffic in a single region. Instead, the configuration update was applied to all regions, increasing resource pressure and causing a spike in internal server errors for Catalog APIs dependent on storage. This led to widespread request failures and degraded service availability. ### Action Items * Rolled back the faulty service configuration to restore normal operation. * Improved deployment validation procedures to ensure experimental changes are restricted to intended regions. * Enhanced monitoring and alerting systems to detect and report anomalies in API error rates and service health more rapidly.

Read the full incident report →

Started Jul 31, 2025, 10:17 PM UTC· resolved Aug 01, 2025, 04:55 AM UTC

1. investigating Jul 31, 2025, 10:17 PM UTC

   We are currently experiencing delayed processing of actions for rule and segment automation. Engineers are working to resolve this as soon as possible.

2. investigating Aug 01, 2025, 12:31 AM UTC

   We are continuing to investigate and testing a potential mitigation to improve action processing throughput.

3. monitoring Aug 01, 2025, 02:01 AM UTC

   A fix has been deployed and we are continuing to monitor as processing catches up.

4. resolved Aug 01, 2025, 04:55 AM UTC

   This incident has been resolved.

5. postmortem Aug 12, 2025, 11:47 PM UTC

   On July 31, 2025, between 10:30 AM and 7:29 PM PDT, some customers experienced significant delays when using PlayStream actions for rules and segments. Action executions were delayed, with a maximum delay of over 500 minutes at peak. The incident was caused by a combination of load and configuration values for the maximum number of records each processor could read at once, which, combined with pod health logic, led to excessive memory usage and processor failures. We resolved the issue by reducing the configuration value, which restored healthy processing across all partitions. ### Impact All PlayFab titles using PlayStream actions for rules and segments were impacted. Action executions were delayed but not dropped; however, the prolonged delay meant that some actions may not have been useful by the time they were processed. ### Root Cause Analysis The incident was caused by a misconfiguration in the number of records each processor attempted to read, combined with a change in the logic for partition allocation per processor. As processor pods failed due to memory exhaustion, the remaining healthy pods became overloaded, leading to a cascading failure and increasing delays in action processing. ### Action Items To prevent similar incidents from happening again, we have taken the following actions: * We reduced the maximum number of records each processor can read at once, improving processor reliability and preventing memory exhaustion. * We improved our monitoring and alerting to detect abnormal processor delays and memory usage earlier.

Read the full incident report →