AdBook+ System Performance

Update timeline

1. investigating May 05, 2025, 04:11 PM UTC

   We are aware of and urgently investigating an issue with AdBook system performance. Users may encounter errors saving lines, running reports or navigating areas of the system. Further updates will be provided here as soon as they are available. Please contact us at support.fattail.com with questions.

2. identified May 05, 2025, 10:24 PM UTC

   We have identified the root cause affecting AdBook+ system performance and are actively working on a resolution. Users may continue to experience latency and longer save times in the meantime. We will provide an update as soon as we have a confirmed timeline for deploying the fix. Please contact us at support.fattail.com with questions.

3. identified May 06, 2025, 01:45 PM UTC

   Our team is continuing the work on a solution to the issue affecting AdBook+ system performance. In the meantime, users may continue to experience latency and longer save times. We will provide our next update in approximately 4 hrs. Please contact us at support.fattail.com with questions.

4. identified May 06, 2025, 05:07 PM UTC

   Our team is continuing the work on a solution to the issue affecting AdBook+ system performance. In the meantime, users may continue to experience latency and longer save times. We will provide our next update in approximately 4 hrs. Please contact us at support.fattail.com with questions.

5. identified May 06, 2025, 08:26 PM UTC

   We will be conducting a deployment today at 5:00 PM ET to address the issue impacting AdBook+ system performance. During this time, the system will be unavailable for approximately 15 to 30 minutes. We appreciate your patience as we work to restore full functionality. Please contact us at support.fattail.com with questions.

6. identified May 06, 2025, 11:25 PM UTC

   We are continuing to monitor AdBook and are working to confirm whether full system performance has been resolved. Our engineering team is making additional optimizations as needed. We will provide additional updates at 11pm ET and again at 8am ET on Wednesday May 7th.

7. identified May 07, 2025, 03:02 AM UTC

   We are continuing to monitor AdBook and are working to confirm whether full system performance has been resolved. Our engineering team is making additional optimizations as needed. We will provide our next update at 8am ET on Wednesday May 7th.

8. monitoring May 07, 2025, 12:03 PM UTC

   We are seeing improved performance and continue to closely monitor system health. Please continue to report any user issues via our helpdesk. Thank you for your patience while we work to fully restore AdBook+. FatTail Support

9. monitoring May 07, 2025, 08:23 PM UTC

   We are continuing to monitor for any further issues.

10. resolved May 09, 2025, 09:20 PM UTC

    This incident has been resolved.

11. postmortem May 14, 2025, 01:08 AM UTC

    ### **Summary of Impact** A timeline of events related to latency within the AdBook\+ environment is outlined below: * On Wednesday April 30th, 2025 FatTail received reports of sporadic latency. These issues were isolated to a few users in only a few client environments, and could not be reproduced. System health indicators were positive at this time. FatTail Engineering teams initiated an investigation and at the time were unable to identify any root cause for the reported slow user experience. * On Monday, May 5th, 2025, at approximately 11AM EST our Support Team received numerous reports of user latency across all Production environments. System monitoring was not indicative of an issue, however, FatTail engineering did identify very long wait times for actions such as search, loading various pages and save actions. Additionally, API requests were timing out, resulting in inconsistent data synchronization through system integrations e.g. Salesforce. FatTail Engineering investigated the issue thoroughly and ruled out any recent software changes as the root cause of the latency. * On Tuesday, May 6th 2025, at approximately 5PM EST FatTail rolled back operating system updates that were suspected to be contributing to the performance issues. Details of these updates are below. Internal testing indicated an immediate improvement in system performance. Status updates were communicated to clients and our teams continued to closely monitor overall system health. ### **Root Cause** FatTail engineering teams are still working to determine the root cause of this incident. We are focused on three likely scenarios: 1. Microsoft system patches were applied on Saturday, April 26th, 2025 as part of our scheduled patching process. It is noteworthy that those patches are applied to systems in a rolling four week cycle, beginning with internal systems, followed by UAT servers, then Sandbox servers, and finally Production. We did not experience any issue with the application of the patches on the lower level systems. 2. Both Microsoft Defender and Webroot Protection applications were running. While our standard configuration includes only Microsoft Defender, we discovered that Webroot was also installed. Here it is noteworthy that both applications had been operating concurrently without issues for over 90 days. 3. The network area storage server was running in an invalid state. Investigation on this is on-going. ### **Mitigation** On Tuesday, May 6th, 2025, FatTail engineering took three steps to mitigate the issue: 1. Rolled-back the Microsoft updates that were installed on April 26th. 2. Uninstalled Webroot. 3. Rebooted the Production servers. At the same time, we re-deployed AdBook v3.95.03, which included additional logging mechanisms designed to capture additional performance data in critical areas of the application. This deployment was not intended to resolve existing issues, but rather to support on-going diagnostics by providing more detailed insights should the above mitigation efforts prove ineffective. ### **Next Steps** FatTail engineering is continuing to monitor system performance, and can confirm that currently, all systems are fully operational and system performance is consistent with typical operating conditions. ‌ Please contact us at [support.fattail.com](http://support.fattail.com) with any follow up questions or concerns.