FatTail experienced a minor incident on April 28, 2025 affecting AdBook+ Order Management, lasting 7d 23h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 28, 2025, 02:28 PM UTC
We are aware of and working urgently to resolve issues where users are unable to either login, or stay logged in to AdBookOMS. We will provide an update as soon as we can. Thank you for your patience while we work to resolve the issue.
- investigating Apr 28, 2025, 08:03 PM UTC
Our team is actively investigating the issue causing users to be intermittently logged out, and we are treating it with the highest priority. We will provide updates as soon as they become available. Thank you for your patience while we work to resolve the issue.
- monitoring Apr 29, 2025, 12:33 PM UTC
At 9PM PST on Monday April 28th we deployed an update to resolve the issue where users could not login or stay logged in to AdBookOMS. We are actively monitoring to ensure all access issues have been addressed. If you have questions or experience ongoing issues with access please reach out via support.fattail.com Thank you for your patience while we worked to identify and resolve this incident. FatTail Support.
- resolved May 06, 2025, 01:53 PM UTC
This issue was resolved with the release of a fix on Monday, April 28, 2025. A detailed postmortem will be shared once our engineering teams have completed a full analysis of the incident. If you have questions in the interim please contact us via support.fattail.com.
- postmortem May 09, 2025, 07:51 PM UTC
### **Summary of Impact** On April 25, 2025 the new User Creation and Login Experience was deployed to all Production environments. The following Monday, April 28, 2025, FatTail was notified by client users that they were experiencing persistent login issues in AdBook. Specifically, users reported encountering an ‘Unexpected Error Has Occurred’ message immediately after entering their credentials from the new sign-in page. Upon successfully logging in, users were initially able to access and navigate through various areas of the system. However, their user sessions were interrupted by unexpected and intermittent logouts. A timeline of events related to the issue is outlined below. * On Friday, April 25, 2025 at 6PM PT the new User Creation and Login Experience was deployed to all Production environments. * On Monday, April 28, 2025 at approximately 6:45AM PT, users began reporting unexpected logouts after logging into AdBook. Upon attempting to login again, users were met with an ‘Unexpected Error Occurred’ message. At 7AM PT, the issue was escalated to FatTail engineering for further review. * At approximately 1:30PM PT FatTail engineering confirmed that the issue did not originate from AdBook, but rather our service provider Microsoft Azure B2C, which handles user identity storage and authentication for users who login to AdBook. * At 4:30PM PT a joint investigation between FatTail and Microsoft engineering determined the issue stemmed from a high volume of users attempting to validate authentication at the same time when logging into the system, which overloaded Azure B2C. * At 5:30PM PT FatTail identified a resolution to the issue to stabilize authentication validation and allow users to login to AdBook, and remained logged in without interruption. * At 9:30PM PT, FatTail engineering deployed the fix to all Production environments. FatTail Client Success verified that users were no longer being intermittently logged out after initial login, and no longer encountered the ‘Unexpected Error Occurred’ issue. ### **Root Cause** With the new AdBook login experience, all user authentication was re-directed through Microsoft Azure B2C. The root cause of the issue stemmed from load-related strain on Microsoft’s authentication process. A high volume of concurrent authentication validation attempts overwhelmed the authentication service, particularly during peak access times throughout the day of April 28, 2025. As a result, the concurrent authentication validation attempts being made across all environments caused users who were logged into the system to be unexpectedly logged out, and encounter an “Unexpected Error Has Occurred” upon the next login attempt. ### **Mitigation & Next Steps** The following steps were taken to mitigate the issue: * The development team implemented a solution that reduced the volume of calls to the identity provider to validate authentication, while still maintaining a strong approach for validating users accessing application resources are authenticated appropriately. FatTail will implement more rigorous load testing solution to mitigate the risk of similar issues occurring in the future.