FatTail Outage History

Started Sep 30, 2026, 03:33 PM UTC· resolved Oct 01, 2026, 09:19 PM UTC

1. investigating Sep 30, 2026, 03:33 PM UTC

   We are aware of and urgently investigating an issue with AdBook system performance. Users may experience errors when trying to access the system. Further updates will be provided here as soon as they are available. Please contact us at support.fattail.com with questions.

2. investigating Sep 30, 2026, 03:39 PM UTC

   We are continuing to investigate this issue.

3. identified Sep 30, 2026, 04:01 PM UTC

   We have identified the issue and AdBook system access has been restored. Users may still experience issues with accessing DQT. Our engineering team is working to resolve that issue. Further updates will be provided here as soon as they are available. Please contact us at support.fattail.com with questions.

4. identified Sep 30, 2026, 04:12 PM UTC

   We have identified the issue and AdBook system access has been restored. Users may still experience issues with accessing DQT and other reporting. Our engineering team is working to resolve that issue. Further updates will be provided here as soon as they are available. Please contact us at support.fattail.com with questions.

5. monitoring Sep 30, 2026, 05:03 PM UTC

   The issue with DQT and other reporting has been fixed. We are continuing to monitor for any further issues.

6. resolved Oct 01, 2026, 09:19 PM UTC

   This issue has been resolved. A postmortem of this incident will be posted once available. Please contact us at support.fattail.com with questions.

7. postmortem Oct 14, 2026, 10:15 PM UTC

   ### **Summary of Impact** On September 30, 2026, at 8:17 AM PT, an AdBook deployment of v4.10.01 intended for the sandbox \(SBX\) environment was unintentionally run against Production. The deployment shut down all environments, and AdBook was unavailable to all customers for 25 minutes. Access was restored at 8:42 AM PT. Some reporting and inventory functions stayed degraded until 2:24 PM PT, when full functionality was restored. No customer data was lost, and no customer action is required. Summary of events: * 8.17 am PT - v4.10.01 deployment begins on Production; all environments shut down * 8.22 am PT - Incident communication protocols begin * 8.24 am PT - v4.09.03 redeployed; application access restored * 8.33 am PT - Customers notified of incident via Status page * 9.30 am PT - DQT \(Data Query Tool\) reporting restored * 1.19 pm PT - Inventory availability validation restored * 2.24 pm PT - Inventory jobs that failed during the incident reprocessed; full functionality restored ### **Root Cause** While deploying v4.10.01 to SBX, the Engineering team hit deployment failures. During troubleshooting, the deployment pipeline's target environment was unintentionally changed from SBX to Production. Two gaps in tooling and process allowed this: * The environment selector's clickable area extends well beyond its visible controls, so a click elsewhere on the page can change the target without any visible indication. * The pipeline has no confirmation step specific to Production, so nothing prompted a final check of the target before services stopped. The extended recovery after the application was rolled back was a result of.v4.10.01 including database changes that aren't backward compatible with v4.09.03. ### **Resolution** The Engineering team redeployed v4.09.03 - the current production version - as the fastest and lowest-risk way to restore access. Application access was restored at 8:42 AM PT. Redeploying the application did not reverse the database changes made by v4.10.01.DQT reporting and inventory availability validation remained affected because database artifacts required by v4.09.03 had been removed or changed during the v4.10.01 deployment. The team restored these objects as they were identified and reprocessed the affected inventory jobs, completing recovery at 2:24 PM PT. ### **Mitigation and Next Steps** To reduce risk of a similar incident we are adding a mandatory production-intent confirmation gate. The gate and subsequent confirmation will prominently identify key deployment parameters \(such as the target environment, release version\), and require explicit acknowledgment before any services are stopped or database changes are applied. Second-person approval will be added as an additional safeguard, taking staff availability and emergency recovery requirements into account.

Read the full incident report →

Started Jun 22, 2026, 09:59 PM UTC· resolved Jul 01, 2026, 08:29 PM UTC

1. investigating Jun 22, 2026, 09:59 PM UTC

   We are aware of and urgently investigating an issue with AdBook system performance. Users may encounter intermittent errors trafficking lines, checking availability, and running reports. Further updates will be provided here as soon as they are available. Thank you for your patience while we work to restore performance. Please contact us at support.fattail.com with questions.

2. monitoring Jun 23, 2026, 03:07 PM UTC

   Our engineering team has identified the issue impacting system performance and completed system optimizations. Performance remains stable at this time. We will leave this incident open for monitoring. Please reach out via support.fattail.com if you have questions or concerns.

3. resolved Jul 01, 2026, 08:29 PM UTC

   This incident has been resolved.

4. postmortem Jul 01, 2026, 09:17 PM UTC

   ### **Summary of Impact** On the afternoon of June 22, 2026, users of [adbook03.fattail.com](http://adbook02.fattail.com/) and [adbook04.fattail.com](http://adbook04.fattail.com) experienced trafficking errors and slow response time on campaign uploads and reporting. Performance degradation was first observed and reported at approx. 4:30pm EST. The FatTail engineering team was able to resolve and restore services at approximately 6:25pm EST. ### **Root Cause** The degradation was caused by elevated request volume that put high pressure on the service caching mechanisms, which caused a slowdown in response time. **Mitigation** There were two steps taken to mitigate the issue: 1. The reporting services were restarted to clear out orphaned \(timed out\) requests that were queued up in the caching service. 2. Cache parameters were updated to provide more resources to handle elevated request volume. ### **Next Steps** The team is implementing monitoring to raise alerts when cache resources are out of normal operating zones.

Read the full incident report →

Started Jun 15, 2026, 03:33 PM UTC· resolved Jun 22, 2026, 09:17 PM UTC

1. identified Jun 15, 2026, 03:33 PM UTC

   We are currently investigating an issue introduced with the Salesforce Summer '26 Release that is impacting the **AdBook Refresh Process** for some clients. As a result, AdBook refreshes may fail and generate errors. Example Error: An internal server error has occurred An error has occurred while processing your request. The Salesforce: The #1 Agentic AI CRM support team has been notified of the problem. If you believe you have additional information that may be of help in reproducing or correcting the error, please contact Salesforce Support. Please indicate the URL of the page you were requesting, any error id shown on this page as well as any other related information. We apologize for the inconvenience. Thank you again for your patience and assistance. And thanks for using Salesforce: The #1 Agentic AI CRM ! Error ID: 2126748694-89894 (251024430) If you encounter this error, please complete the following steps: 1. Navigate to Setup → Apex Jobs in Salesforce. 2. Locate any AdBookBatchRefreshOutOfSync jobs that are currently failing. 3. Abort all failing AdBookBatchRefreshOutOfSync jobs. 4. Once the jobs have been aborted, create a Salesforce Support Case. 5. Include the full error message displayed in the Status Detail field of the failed Apex Job when submitting your case. This information will assist Salesforce in identifying and resolving the issue more quickly. We are actively working with Salesforce to understand the root cause and will provide additional updates as more information becomes available. Thank you for your patience and cooperation. Please contact us via support.fattail.com if you have any questions or concerns.

2. identified Jun 16, 2026, 10:35 PM UTC

   We have identified the cause of the refresh errors as related to the recent Salesforce release. Why Summer '26 Could Have Caused This? Even without any changes to your org or the AdBook package, the Summer '26 platform release (API v67.0, VF runtime v262) may have introduced one or more of the following changes that caused previously valid UDF values to exceed Salesforce limits: 1. Visualforce Runtime Upgrade (v262): Summer '26 shipped a new VF runtime version that changed how POST request parameters and form data are serialized. This can result in longer string representations being passed to underlying managed package methods — even when the underlying data has not changed. 2. SOAP/WS Array Serialization Changes: The failing method accepts a AdBookWS$ArrayOfDynamicPropertyValue parameter — a SOAP-based array type. Platform changes to how these types are marshalled before being passed to the PL/Java layer can cause the serialized string to grow beyond the declared limit. 3. Stricter PL/Java Enforcement: In some platform hardening releases, oversized PL/Java call argument values that were previously tolerated (or silently truncated) are now rejected with a hard exception. This would explain why the same data worked before Summer '26 but fails now. Recommended Next Steps: An updated Managed Package is available (v3.92) to handle longer parameter values as generated by the Summer '26 VF runtime. IF you have been impacted by this issue, please work via support.fattail.com to coordinate the install the new version of the Managed Package in your Salesforce sandbox environment. After completing User Acceptance Testing (UAT) and confirming that everything is functioning as expected, you can proceed with the installation in your production environment.

3. resolved Jun 22, 2026, 09:17 PM UTC

   We have confirmed that this incident can be resolved by upgrading to AdBook Managed Package version 3.92 or later. The updated managed package uses an alternative data point, which prevents the Salesforce internal governor limit from being reached. Additionally, we have verified that the Apex jobs responsible for asynchronous processing will be restored once the impacted jobs are aborted. If you have been impacted by this issue, please work via support.fattail.com to coordinate the install the new version of the Managed Package in your Salesforce sandbox environment. After completing User Acceptance Testing (UAT) and confirming that everything is functioning as expected, you can proceed with the installation in your production environment.

Read the full incident report →

Started Apr 13, 2026, 02:13 PM UTC· resolved Apr 14, 2026, 11:03 AM UTC

1. investigating Apr 13, 2026, 02:13 PM UTC

   We are aware of and urgently investigating an issue with AdBook system performance affecting users of adbook02.fattail.com. Users may experience latency when accessing or navigating within the system. Further updates will be provided here as soon as they are available. Please contact us at support.fattail.com with questions.

2. monitoring Apr 13, 2026, 02:36 PM UTC

   Our engineering teams have performed system maintenance which has resolved the performance issue for clients using adbook02.fattail.com. Users should no longer experience issues logging in to, or navigating AdBook. Please contact us at support.fattail.com with questions.

3. resolved Apr 14, 2026, 11:03 AM UTC

   This issue was fully resolved yesterday and adbook02 health has remained steady since. A postmortem of this incident will be posted once available. Please contact us at support.fattail.com with questions.

4. postmortem Apr 23, 2026, 05:52 PM UTC

   ### **Summary of Impact** On the morning of April 13, 2026, users of [adbook02.fattail.com](http://adbook02.fattail.com) experienced slow performance and difficulty accessing the system. This included issues logging in, running reports, and navigating between pages. Performance degradation was first observed and reported at approx. 5.30am EST. The FatTail engineering team was able to resolve and restore services at approximately 10.25am EST. ### **Root Cause** The slowdown was caused by degradation of one of the reporting services under a period of elevated request volume. The service continued accepting requests but was unable to process them efficiently which impacted overall system performance. Once the affected service was restarted, the backlog cleared and performance returned to normal. **Mitigation and Next Steps** To prevent recurrence, the FatTail team is implementing the following improvements: 1. Improve system protection and traffic management Adjust service throttling and implement additional safeguards to better control concurrent requests and prevent overload during traffic spikes. 1. Strengthen monitoring, alerting, and traffic analysis Enhance monitoring and alerting to detect and address sources of unusually high traffic. Additionally FatTail Support will provide more frequent ticket communication to affected clients while incidents are active.

Read the full incident report →

Started Jan 26, 2026, 04:05 PM UTC· resolved Feb 05, 2026, 05:44 PM UTC

1. investigating Jan 30, 2026, 05:42 PM UTC

   We are aware of and urgently investigating an issue with AdBook system performance. Users may experience latency when accessing or navigating within the system. Further updates will be provided here as soon as they are available. Please contact us at support.fattail.com with questions.

2. investigating Jan 30, 2026, 08:45 PM UTC

   We are continuing to research the cause of the performance issue within AdBook. Additional updates will be provided as soon as possible.

3. investigating Jan 31, 2026, 12:39 AM UTC

   We are continuing to investigate the issue impacting system performance. Additional updates will be provided as soon as possible.

4. monitoring Jan 31, 2026, 01:08 AM UTC

   Following additional maintenance the system is stable. We are actively monitoring performance and will leave this incident open. Please contact us at support.fattail.com with questions.

5. resolved Feb 05, 2026, 05:44 PM UTC

   This incident has been resolved.

6. postmortem Feb 05, 2026, 08:42 PM UTC

   ### **Summary of Impact** Some users experienced slowness and intermittent errors across the platform due to increased API activity. ### **Root Cause** Increase in failed API requests from [Salesforce.com](http://salesforce.com) caused higher-than-expected system load, which impacted overall platform performance. ### **Mitigation** The engineering team adjusted system thresholds to better handle increased request volume and reduce processing delays. Performance returned to normal following these changes. ### **Next Steps** To reduce the risk of similar issues in the future, we are: * Improving how the system handles failed integration requests to minimize impact during traffic spikes * Implementing safeguards to prevent excessive resource usage when errors occur

Read the full incident report →

Started Dec 18, 2025, 09:06 PM UTC· resolved Dec 19, 2025, 01:48 PM UTC

1. investigating Dec 18, 2025, 09:06 PM UTC

   We are aware of and urgently investigating an issue with AdBook system performance. Users may experience latency when accessing or navigating within the system. Further updates will be provided here as soon as they are available. Please contact us at support.fattail.com with questions.

2. monitoring Dec 18, 2025, 09:17 PM UTC

   Performance has been restored and users should no longer experience latency in the system. We are continuing to monitor system health and will publish more information once full details on cause are available.

3. resolved Dec 19, 2025, 01:48 PM UTC

   This incident has been resolved.

4. postmortem Feb 05, 2026, 08:35 PM UTC

   ### **Summary of Impact** On Thursday December 18th, at approximately 3.45pm ET, some users of the [adbook03.fattail.com](http://adbook03.fattail.com) site experienced timeouts and error messages. Our monitoring systems immediately alerted the engineering teams who identified the cause and restored normal service by approximately 4.15pm ET. ### **Root Cause** The issue was caused by an inefficient reporting query that led to increased system load and affected application responsiveness. ### **Resolution** The engineering team updated the affected query, eliminating the performance bottleneck and restoring normal platform behavior. ### **Next Steps** To help prevent similar issues in the future, we are: * Continuing to monitor system performance for similar patterns Reviewing and optimizing reporting queries to reduce the risk of blocking or delays

Read the full incident report →

Started Nov 21, 2025, 06:34 PM UTC· resolved Nov 21, 2025, 07:06 PM UTC

1. investigating Nov 21, 2025, 06:34 PM UTC

   We are aware of and urgently investigating an issue with AdBook system performance. Users may experience issues accessing the system. Further updates will be provided here as soon as they are available. Please contact us at support.fattail.com with questions.

2. resolved Nov 21, 2025, 07:06 PM UTC

   As of 11:00 am PST access to the platform has been restored. We will continue to monitor system performance. Details on the issue will be provided.

3. postmortem Dec 10, 2025, 04:21 PM UTC

   On Friday, November 21, 2025, FatTail experienced a site-wide outage affecting client access to AdBook. Beginning at approximately 1:30PM ET, users were unable to log in to the platform. System access was restored by 2:00PM ET, resulting in an estimated 30-minute service disruption. FatTail's alerting systems immediately identified the issue, prompting the Engineering and Client Success teams to take swift action. The FatTail status page was updated and direct client notifications were sent promptly to ensure transparency during the event. **Summary of Impact** Below is a detailed timeline of the outage: * 1:30PM ET – Internal alerts notified FatTail of login failures within AdBook. Engineering immediately identified the cause of the issue and began working to restore the service. * 2:00PM ET – The issue was fully resolved and all client access to AdBook was restored. ### **Root Cause** The outage was caused by an internal operator error. The engineering team was preparing for a planned maintenance window, and mistakenly issued a command on the production web server that caused the production system to go offline temporarily. FatTail acknowledges this as a preventable error and is taking steps to strengthen operational safeguards. ### **Resolution** Engineering quickly isolated the cause of the outage and restored system functionality by 2:00PM ET. Services became available immediately upon restoration, and Client Success verified successful user login shortly after resolution. Root cause validation and post-incident review began immediately following service restoration. Additional monitoring confirmed system stability and full functionality. Remediation details will be added once Engineering completes their follow-up analysis. ### **Mitigation and Next Steps** FatTail is implementing additional safeguards to prevent this type of incident from recurring. The following actions will be taken: ### Operational Safeguards for Production Access * Implement environment isolation enhancements to reduce the risk of command execution in incorrect contexts. * Introduce updated operational procedures for all team members with domain admin access. ### Engineering Process Improvements * Collaborate with Engineering leadership to define stricter access workflows. * Evaluate tooling options to differentiate local vs. production environments through shell prompts, window themes, or command-line restrictions. * Reinforce internal best practices and training for handling production-access tasks.

Read the full incident report →

Started Sep 29, 2025, 03:32 PM UTC· resolved Oct 01, 2025, 01:22 PM UTC

1. investigating Sep 29, 2025, 03:32 PM UTC

   We are aware of and urgently investigating an issue affecting clients who login via adbook03.fattail.com. Users may encounter errors saving lines, running reports or navigating areas of the system, including sync operations with Salesforce. Further updates will be provided here as soon as they are available. Thank you for your patience while we work to restore performance. Please contact us at support.fattail.com with questions. FatTail Support

2. investigating Sep 29, 2025, 03:57 PM UTC

   We are continuing to investigate this issue.

3. investigating Sep 29, 2025, 04:30 PM UTC

   Performance on adbook03.fattail.com is currently stable. We are continuing to research root cause and are actively monitoring all systems. Further updates will be provided as they become available. Thank you FatTail Support

4. monitoring Sep 29, 2025, 07:41 PM UTC

   Performance on adbook03.fattail.com remains stable. We are continuing to actively monitor all systems.

5. resolved Oct 01, 2025, 01:22 PM UTC

   This incident is being closed. A postmortem will be published as a follow up. Please contact us at support.fattail.com with questions. FatTail Support

Read the full incident report →

Started Jul 28, 2025, 03:31 AM UTC· resolved Jul 28, 2025, 07:42 AM UTC

1. investigating Jul 28, 2025, 03:31 AM UTC

   We are aware of and investigating an issue affecting users accessing AdBookOMS via single-sign on integrations. This includes access via the Salesforce 'Access AdBook' integration. Users are receiving a 502 bad gateway error when attempting to access the system. Users who log directly in to AdBook are not impacted by this incident. FatTail engineering are investigating with urgency. Updates will be provided as soon as they are available. Thank you for your patience while we work to resolve this issue. FatTail Support

2. identified Jul 28, 2025, 07:23 AM UTC

   The issue has been identified and a fix is being implemented.

3. resolved Jul 28, 2025, 07:42 AM UTC

   Users are now able to successfully log in to AdBook via SSO integrations. We will follow up with details on cause of issue once a full report is available. Thank you for your patience while we worked to resolve this issue. FatTail Support.

Read the full incident report →

Started Jun 09, 2025, 03:57 PM UTC· resolved Jun 25, 2025, 12:55 PM UTC

1. investigating Jun 09, 2025, 03:57 PM UTC

   We are aware of and investigating reports of latency across all AdBook+ environments. Our engineering teams are working with urgency to understand issue and restore performance. Thank you for your patience while we investigate. Please contact us at support.fattail.com with questions otherwise updates will be posted as quickly as possible. Thank you FatTail Support

2. investigating Jun 09, 2025, 04:45 PM UTC

   Our engineering team will be performing maintenance at 1.30pm EST which should last ~20 minutes and is designed to restore system performance. Users will not be able to access the system during this maintenance window. Please advise users to save their work ahead of 1.30pm. We will continue to closely monitor system performance and will update this incident further.

3. monitoring Jun 09, 2025, 06:12 PM UTC

   Maintenance has been completed and we are monitoring the results.

4. investigating Jun 09, 2025, 06:58 PM UTC

   We are continuing to research the cause of latency within AdBook. Additional updates will be provided as soon as possible. FatTail Support

5. investigating Jun 09, 2025, 08:08 PM UTC

   Our engineering team will be performing maintenance at 4.15pm EST which should last ~5 minutes and is designed to help with performance issues. Users will not be able to access the system during this maintenance window. We will continue to closely monitor system performance and will update this incident further.

6. monitoring Jun 10, 2025, 12:21 AM UTC

   Following a second maintenance update at approximately 4.30pm EST AdBook+ system performance has returned to normal levels. We continue to closely monitor system health and will leave this incident open. Thank you for your patience while we work to investigate and resolve latency. FatTail Support.

7. resolved Jun 25, 2025, 12:55 PM UTC

   Following a series of updates and monitoring we are closing this incident. A postmortem will be provided with details on cause of performance issues and remediation steps. Thank you. FatTail Support

Read the full incident report →

Started Jun 05, 2025, 04:34 PM UTC· resolved Jun 05, 2025, 09:21 PM UTC

1. investigating Jun 05, 2025, 04:34 PM UTC

   We are aware of and urgently investigating system latency impacting users who login at adbook03.fattail.com. Users may experience intermittent issues or delays loading screens, or saving. Thank you for your patience while we work to investigate and restore performance. Our next update will be provided no later than 4pm EST. Thank you FatTail Support

2. investigating Jun 05, 2025, 07:34 PM UTC

   We are continuing to investigate the AdBook+ performance issues with urgency. Thank you for your patience while we work to restore performance. Our next update will be posted no later than 7pm EST. Thank you FatTail Support

3. resolved Jun 05, 2025, 09:21 PM UTC

   We have identified and resolved the issue affecting system performance. We will publish a postmortem once engineering team have finalized details surrounding the incident. If your users continue to see slow performance please contact us via support.fattail.com. Thanks for your patience while we worked to resolve this issue. FatTail Support

Read the full incident report →

Started May 05, 2025, 04:11 PM UTC· resolved May 09, 2025, 09:20 PM UTC

1. investigating May 05, 2025, 04:11 PM UTC

   We are aware of and urgently investigating an issue with AdBook system performance. Users may encounter errors saving lines, running reports or navigating areas of the system. Further updates will be provided here as soon as they are available. Please contact us at support.fattail.com with questions.

2. identified May 05, 2025, 10:24 PM UTC

   We have identified the root cause affecting AdBook+ system performance and are actively working on a resolution. Users may continue to experience latency and longer save times in the meantime. We will provide an update as soon as we have a confirmed timeline for deploying the fix. Please contact us at support.fattail.com with questions.

3. identified May 06, 2025, 01:45 PM UTC

   Our team is continuing the work on a solution to the issue affecting AdBook+ system performance. In the meantime, users may continue to experience latency and longer save times. We will provide our next update in approximately 4 hrs. Please contact us at support.fattail.com with questions.

4. identified May 06, 2025, 05:07 PM UTC

   Our team is continuing the work on a solution to the issue affecting AdBook+ system performance. In the meantime, users may continue to experience latency and longer save times. We will provide our next update in approximately 4 hrs. Please contact us at support.fattail.com with questions.

5. identified May 06, 2025, 08:26 PM UTC

   We will be conducting a deployment today at 5:00 PM ET to address the issue impacting AdBook+ system performance. During this time, the system will be unavailable for approximately 15 to 30 minutes. We appreciate your patience as we work to restore full functionality. Please contact us at support.fattail.com with questions.

6. identified May 06, 2025, 11:25 PM UTC

   We are continuing to monitor AdBook and are working to confirm whether full system performance has been resolved. Our engineering team is making additional optimizations as needed. We will provide additional updates at 11pm ET and again at 8am ET on Wednesday May 7th.

7. identified May 07, 2025, 03:02 AM UTC

   We are continuing to monitor AdBook and are working to confirm whether full system performance has been resolved. Our engineering team is making additional optimizations as needed. We will provide our next update at 8am ET on Wednesday May 7th.

8. monitoring May 07, 2025, 12:03 PM UTC

   We are seeing improved performance and continue to closely monitor system health. Please continue to report any user issues via our helpdesk. Thank you for your patience while we work to fully restore AdBook+. FatTail Support

9. monitoring May 07, 2025, 08:23 PM UTC

   We are continuing to monitor for any further issues.

10. resolved May 09, 2025, 09:20 PM UTC

    This incident has been resolved.

11. postmortem May 14, 2025, 01:08 AM UTC

    ### **Summary of Impact** A timeline of events related to latency within the AdBook\+ environment is outlined below: * On Wednesday April 30th, 2025 FatTail received reports of sporadic latency. These issues were isolated to a few users in only a few client environments, and could not be reproduced. System health indicators were positive at this time. FatTail Engineering teams initiated an investigation and at the time were unable to identify any root cause for the reported slow user experience. * On Monday, May 5th, 2025, at approximately 11AM EST our Support Team received numerous reports of user latency across all Production environments. System monitoring was not indicative of an issue, however, FatTail engineering did identify very long wait times for actions such as search, loading various pages and save actions. Additionally, API requests were timing out, resulting in inconsistent data synchronization through system integrations e.g. Salesforce. FatTail Engineering investigated the issue thoroughly and ruled out any recent software changes as the root cause of the latency. * On Tuesday, May 6th 2025, at approximately 5PM EST FatTail rolled back operating system updates that were suspected to be contributing to the performance issues. Details of these updates are below. Internal testing indicated an immediate improvement in system performance. Status updates were communicated to clients and our teams continued to closely monitor overall system health. ### **Root Cause** FatTail engineering teams are still working to determine the root cause of this incident. We are focused on three likely scenarios: 1. Microsoft system patches were applied on Saturday, April 26th, 2025 as part of our scheduled patching process. It is noteworthy that those patches are applied to systems in a rolling four week cycle, beginning with internal systems, followed by UAT servers, then Sandbox servers, and finally Production. We did not experience any issue with the application of the patches on the lower level systems. 2. Both Microsoft Defender and Webroot Protection applications were running. While our standard configuration includes only Microsoft Defender, we discovered that Webroot was also installed. Here it is noteworthy that both applications had been operating concurrently without issues for over 90 days. 3. The network area storage server was running in an invalid state. Investigation on this is on-going. ### **Mitigation** On Tuesday, May 6th, 2025, FatTail engineering took three steps to mitigate the issue: 1. Rolled-back the Microsoft updates that were installed on April 26th. 2. Uninstalled Webroot. 3. Rebooted the Production servers. At the same time, we re-deployed AdBook v3.95.03, which included additional logging mechanisms designed to capture additional performance data in critical areas of the application. This deployment was not intended to resolve existing issues, but rather to support on-going diagnostics by providing more detailed insights should the above mitigation efforts prove ineffective. ### **Next Steps** FatTail engineering is continuing to monitor system performance, and can confirm that currently, all systems are fully operational and system performance is consistent with typical operating conditions. ‌ Please contact us at [support.fattail.com](http://support.fattail.com) with any follow up questions or concerns.

Read the full incident report →

Started Apr 28, 2025, 02:28 PM UTC· resolved May 06, 2025, 01:53 PM UTC

1. investigating Apr 28, 2025, 02:28 PM UTC

   We are aware of and working urgently to resolve issues where users are unable to either login, or stay logged in to AdBookOMS. We will provide an update as soon as we can. Thank you for your patience while we work to resolve the issue.

2. investigating Apr 28, 2025, 08:03 PM UTC

   Our team is actively investigating the issue causing users to be intermittently logged out, and we are treating it with the highest priority. We will provide updates as soon as they become available. Thank you for your patience while we work to resolve the issue.

3. monitoring Apr 29, 2025, 12:33 PM UTC

   At 9PM PST on Monday April 28th we deployed an update to resolve the issue where users could not login or stay logged in to AdBookOMS. We are actively monitoring to ensure all access issues have been addressed. If you have questions or experience ongoing issues with access please reach out via support.fattail.com Thank you for your patience while we worked to identify and resolve this incident. FatTail Support.

4. resolved May 06, 2025, 01:53 PM UTC

   This issue was resolved with the release of a fix on Monday, April 28, 2025. A detailed postmortem will be shared once our engineering teams have completed a full analysis of the incident. If you have questions in the interim please contact us via support.fattail.com.

5. postmortem May 09, 2025, 07:51 PM UTC

   ### **Summary of Impact** On April 25, 2025 the new User Creation and Login Experience was deployed to all Production environments. The following Monday, April 28, 2025, FatTail was notified by client users that they were experiencing persistent login issues in AdBook. Specifically, users reported encountering an ‘Unexpected Error Has Occurred’ message immediately after entering their credentials from the new sign-in page. Upon successfully logging in, users were initially able to access and navigate through various areas of the system. However, their user sessions were interrupted by unexpected and intermittent logouts. A timeline of events related to the issue is outlined below. * On Friday, April 25, 2025 at 6PM PT the new User Creation and Login Experience was deployed to all Production environments. * On Monday, April 28, 2025 at approximately 6:45AM PT, users began reporting unexpected logouts after logging into AdBook. Upon attempting to login again, users were met with an ‘Unexpected Error Occurred’ message. At 7AM PT, the issue was escalated to FatTail engineering for further review. * At approximately 1:30PM PT FatTail engineering confirmed that the issue did not originate from AdBook, but rather our service provider Microsoft Azure B2C, which handles user identity storage and authentication for users who login to AdBook. * At 4:30PM PT a joint investigation between FatTail and Microsoft engineering determined the issue stemmed from a high volume of users attempting to validate authentication at the same time when logging into the system, which overloaded Azure B2C. * At 5:30PM PT FatTail identified a resolution to the issue to stabilize authentication validation and allow users to login to AdBook, and remained logged in without interruption. * At 9:30PM PT, FatTail engineering deployed the fix to all Production environments. FatTail Client Success verified that users were no longer being intermittently logged out after initial login, and no longer encountered the ‘Unexpected Error Occurred’ issue. ### **Root Cause** With the new AdBook login experience, all user authentication was re-directed through Microsoft Azure B2C. The root cause of the issue stemmed from load-related strain on Microsoft’s authentication process. A high volume of concurrent authentication validation attempts overwhelmed the authentication service, particularly during peak access times throughout the day of April 28, 2025. As a result, the concurrent authentication validation attempts being made across all environments caused users who were logged into the system to be unexpectedly logged out, and encounter an “Unexpected Error Has Occurred” upon the next login attempt. ### **Mitigation & Next Steps** The following steps were taken to mitigate the issue: * The development team implemented a solution that reduced the volume of calls to the identity provider to validate authentication, while still maintaining a strong approach for validating users accessing application resources are authenticated appropriately. FatTail will implement more rigorous load testing solution to mitigate the risk of similar issues occurring in the future.

Read the full incident report →

Started Apr 27, 2025, 05:37 PM UTC· resolved Jul 14, 2025, 02:36 PM UTC

1. identified Apr 27, 2025, 05:37 PM UTC

   We have identified an issue impacting the 'EditIO' and 'EditDrop' links in OMS email notifications. When users click these links, they are redirected to their AdBook+ landing page instead of the intended Campaign or Drop. This issue does not impact the Portal workflow — email links continue to redirect correctly to specific campaigns within the Portal UI. Our team is actively working on a fix and will provide an update as soon as we have a timeline for the release. We appreciate your patience as we resolve this issue. If you have any questions or concerns, please contact us at support.fattail.com.

2. resolved Jul 14, 2025, 02:36 PM UTC

   This issues was resolved in our 3.98.01 release.

Read the full incident report →

Started Mar 18, 2025, 06:10 PM UTC· resolved Mar 19, 2025, 01:21 PM UTC

1. investigating Mar 18, 2025, 06:10 PM UTC

   We are aware of and investigating an issue affecting all instances of AdBookOMS and Portal. Users may experience slow loading screens and latency while navigating within the application or running reports. Thank you for your patience while we work to restore performance. FatTail Support

2. monitoring Mar 18, 2025, 09:28 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Mar 19, 2025, 01:21 PM UTC

   AdBook+ site performance has been fully resolved. Thank you for your patience while we worked to investigate and resolve this issue. A postmortem will be provided once we have complete details to share on the cause of the issue.

Read the full incident report →