FatTail Outage History

Started Sep 30, 2026, 03:33 PM UTC· resolved Oct 01, 2026, 09:19 PM UTC

1. investigating Sep 30, 2026, 03:33 PM UTC

   We are aware of and urgently investigating an issue with AdBook system performance. Users may experience errors when trying to access the system. Further updates will be provided here as soon as they are available. Please contact us at support.fattail.com with questions.

2. investigating Sep 30, 2026, 03:39 PM UTC

   We are continuing to investigate this issue.

3. identified Sep 30, 2026, 04:01 PM UTC

   We have identified the issue and AdBook system access has been restored. Users may still experience issues with accessing DQT. Our engineering team is working to resolve that issue. Further updates will be provided here as soon as they are available. Please contact us at support.fattail.com with questions.

4. identified Sep 30, 2026, 04:12 PM UTC

   We have identified the issue and AdBook system access has been restored. Users may still experience issues with accessing DQT and other reporting. Our engineering team is working to resolve that issue. Further updates will be provided here as soon as they are available. Please contact us at support.fattail.com with questions.

5. monitoring Sep 30, 2026, 05:03 PM UTC

   The issue with DQT and other reporting has been fixed. We are continuing to monitor for any further issues.

6. resolved Oct 01, 2026, 09:19 PM UTC

   This issue has been resolved. A postmortem of this incident will be posted once available. Please contact us at support.fattail.com with questions.

7. postmortem Oct 14, 2026, 10:15 PM UTC

   ### **Summary of Impact** On September 30, 2026, at 8:17 AM PT, an AdBook deployment of v4.10.01 intended for the sandbox \(SBX\) environment was unintentionally run against Production. The deployment shut down all environments, and AdBook was unavailable to all customers for 25 minutes. Access was restored at 8:42 AM PT. Some reporting and inventory functions stayed degraded until 2:24 PM PT, when full functionality was restored. No customer data was lost, and no customer action is required. Summary of events: * 8.17 am PT - v4.10.01 deployment begins on Production; all environments shut down * 8.22 am PT - Incident communication protocols begin * 8.24 am PT - v4.09.03 redeployed; application access restored * 8.33 am PT - Customers notified of incident via Status page * 9.30 am PT - DQT \(Data Query Tool\) reporting restored * 1.19 pm PT - Inventory availability validation restored * 2.24 pm PT - Inventory jobs that failed during the incident reprocessed; full functionality restored ### **Root Cause** While deploying v4.10.01 to SBX, the Engineering team hit deployment failures. During troubleshooting, the deployment pipeline's target environment was unintentionally changed from SBX to Production. Two gaps in tooling and process allowed this: * The environment selector's clickable area extends well beyond its visible controls, so a click elsewhere on the page can change the target without any visible indication. * The pipeline has no confirmation step specific to Production, so nothing prompted a final check of the target before services stopped. The extended recovery after the application was rolled back was a result of.v4.10.01 including database changes that aren't backward compatible with v4.09.03. ### **Resolution** The Engineering team redeployed v4.09.03 - the current production version - as the fastest and lowest-risk way to restore access. Application access was restored at 8:42 AM PT. Redeploying the application did not reverse the database changes made by v4.10.01.DQT reporting and inventory availability validation remained affected because database artifacts required by v4.09.03 had been removed or changed during the v4.10.01 deployment. The team restored these objects as they were identified and reprocessed the affected inventory jobs, completing recovery at 2:24 PM PT. ### **Mitigation and Next Steps** To reduce risk of a similar incident we are adding a mandatory production-intent confirmation gate. The gate and subsequent confirmation will prominently identify key deployment parameters \(such as the target environment, release version\), and require explicit acknowledgment before any services are stopped or database changes are applied. Second-person approval will be added as an additional safeguard, taking staff availability and emergency recovery requirements into account.

Read the full incident report →

Started Jun 15, 2026, 03:33 PM UTC· resolved Jun 22, 2026, 09:17 PM UTC

1. identified Jun 15, 2026, 03:33 PM UTC

   We are currently investigating an issue introduced with the Salesforce Summer '26 Release that is impacting the **AdBook Refresh Process** for some clients. As a result, AdBook refreshes may fail and generate errors. Example Error: An internal server error has occurred An error has occurred while processing your request. The Salesforce: The #1 Agentic AI CRM support team has been notified of the problem. If you believe you have additional information that may be of help in reproducing or correcting the error, please contact Salesforce Support. Please indicate the URL of the page you were requesting, any error id shown on this page as well as any other related information. We apologize for the inconvenience. Thank you again for your patience and assistance. And thanks for using Salesforce: The #1 Agentic AI CRM ! Error ID: 2126748694-89894 (251024430) If you encounter this error, please complete the following steps: 1. Navigate to Setup → Apex Jobs in Salesforce. 2. Locate any AdBookBatchRefreshOutOfSync jobs that are currently failing. 3. Abort all failing AdBookBatchRefreshOutOfSync jobs. 4. Once the jobs have been aborted, create a Salesforce Support Case. 5. Include the full error message displayed in the Status Detail field of the failed Apex Job when submitting your case. This information will assist Salesforce in identifying and resolving the issue more quickly. We are actively working with Salesforce to understand the root cause and will provide additional updates as more information becomes available. Thank you for your patience and cooperation. Please contact us via support.fattail.com if you have any questions or concerns.

2. identified Jun 16, 2026, 10:35 PM UTC

   We have identified the cause of the refresh errors as related to the recent Salesforce release. Why Summer '26 Could Have Caused This? Even without any changes to your org or the AdBook package, the Summer '26 platform release (API v67.0, VF runtime v262) may have introduced one or more of the following changes that caused previously valid UDF values to exceed Salesforce limits: 1. Visualforce Runtime Upgrade (v262): Summer '26 shipped a new VF runtime version that changed how POST request parameters and form data are serialized. This can result in longer string representations being passed to underlying managed package methods — even when the underlying data has not changed. 2. SOAP/WS Array Serialization Changes: The failing method accepts a AdBookWS$ArrayOfDynamicPropertyValue parameter — a SOAP-based array type. Platform changes to how these types are marshalled before being passed to the PL/Java layer can cause the serialized string to grow beyond the declared limit. 3. Stricter PL/Java Enforcement: In some platform hardening releases, oversized PL/Java call argument values that were previously tolerated (or silently truncated) are now rejected with a hard exception. This would explain why the same data worked before Summer '26 but fails now. Recommended Next Steps: An updated Managed Package is available (v3.92) to handle longer parameter values as generated by the Summer '26 VF runtime. IF you have been impacted by this issue, please via support.fattail.com to coordinate the install the new version of the Managed Package in your Salesforce sandbox environment. After completing User Acceptance Testing (UAT) and confirming that everything is functioning as expected, you can proceed with the installation in your production environment.

3. resolved Jun 22, 2026, 09:17 PM UTC

   We have confirmed that this incident can be resolved by upgrading to AdBook Managed Package version 3.92 or later. The updated managed package uses an alternative data point, which prevents the Salesforce internal governor limit from being reached. Additionally, we have verified that the Apex jobs responsible for asynchronous processing will be restored once the impacted jobs are aborted. If you have been impacted by this issue, please work via support.fattail.com to coordinate the install the new version of the Managed Package in your Salesforce sandbox environment. After completing User Acceptance Testing (UAT) and confirming that everything is functioning as expected, you can proceed with the installation in your production environment.

Read the full incident report →

Started Apr 13, 2026, 02:13 PM UTC· resolved Apr 14, 2026, 11:03 AM UTC

1. investigating Apr 13, 2026, 02:13 PM UTC

   We are aware of and urgently investigating an issue with AdBook system performance affecting users of adbook02.fattail.com. Users may experience latency when accessing or navigating within the system. Further updates will be provided here as soon as they are available. Please contact us at support.fattail.com with questions.

2. monitoring Apr 13, 2026, 02:36 PM UTC

   Our engineering teams have performed system maintenance which has resolved the performance issue for clients using adbook02.fattail.com. Users should no longer experience issues logging in to, or navigating AdBook. Please contact us at support.fattail.com with questions.

3. resolved Apr 14, 2026, 11:03 AM UTC

   This issue was fully resolved yesterday and adbook02 health has remained steady since. A postmortem of this incident will be posted once available. Please contact us at support.fattail.com with questions.

4. postmortem Apr 23, 2026, 05:52 PM UTC

   ### **Summary of Impact** On the morning of April 13, 2026, users of [adbook02.fattail.com](http://adbook02.fattail.com) experienced slow performance and difficulty accessing the system. This included issues logging in, running reports, and navigating between pages. Performance degradation was first observed and reported at approx. 5.30am EST. The FatTail engineering team was able to resolve and restore services at approximately 10.25am EST. ### **Root Cause** The slowdown was caused by degradation of one of the reporting services under a period of elevated request volume. The service continued accepting requests but was unable to process them efficiently which impacted overall system performance. Once the affected service was restarted, the backlog cleared and performance returned to normal. **Mitigation and Next Steps** To prevent recurrence, the FatTail team is implementing the following improvements: 1. Improve system protection and traffic management Adjust service throttling and implement additional safeguards to better control concurrent requests and prevent overload during traffic spikes. 1. Strengthen monitoring, alerting, and traffic analysis Enhance monitoring and alerting to detect and address sources of unusually high traffic. Additionally FatTail Support will provide more frequent ticket communication to affected clients while incidents are active.

Read the full incident report →

Started Sep 29, 2025, 03:32 PM UTC· resolved Oct 01, 2025, 01:22 PM UTC

1. investigating Sep 29, 2025, 03:32 PM UTC

   We are aware of and urgently investigating an issue affecting clients who login via adbook03.fattail.com. Users may encounter errors saving lines, running reports or navigating areas of the system, including sync operations with Salesforce. Further updates will be provided here as soon as they are available. Thank you for your patience while we work to restore performance. Please contact us at support.fattail.com with questions. FatTail Support

2. investigating Sep 29, 2025, 03:57 PM UTC

   We are continuing to investigate this issue.

3. investigating Sep 29, 2025, 04:30 PM UTC

   Performance on adbook03.fattail.com is currently stable. We are continuing to research root cause and are actively monitoring all systems. Further updates will be provided as they become available. Thank you FatTail Support

4. monitoring Sep 29, 2025, 07:41 PM UTC

   Performance on adbook03.fattail.com remains stable. We are continuing to actively monitor all systems.

5. resolved Oct 01, 2025, 01:22 PM UTC

   This incident is being closed. A postmortem will be published as a follow up. Please contact us at support.fattail.com with questions. FatTail Support

Read the full incident report →

Started Jul 28, 2025, 03:31 AM UTC· resolved Jul 28, 2025, 07:42 AM UTC

1. investigating Jul 28, 2025, 03:31 AM UTC

   We are aware of and investigating an issue affecting users accessing AdBookOMS via single-sign on integrations. This includes access via the Salesforce 'Access AdBook' integration. Users are receiving a 502 bad gateway error when attempting to access the system. Users who log directly in to AdBook are not impacted by this incident. FatTail engineering are investigating with urgency. Updates will be provided as soon as they are available. Thank you for your patience while we work to resolve this issue. FatTail Support

2. identified Jul 28, 2025, 07:23 AM UTC

   The issue has been identified and a fix is being implemented.

3. resolved Jul 28, 2025, 07:42 AM UTC

   Users are now able to successfully log in to AdBook via SSO integrations. We will follow up with details on cause of issue once a full report is available. Thank you for your patience while we worked to resolve this issue. FatTail Support.

Read the full incident report →

Started Jun 09, 2025, 03:57 PM UTC· resolved Jun 25, 2025, 12:55 PM UTC

1. investigating Jun 09, 2025, 03:57 PM UTC

   We are aware of and investigating reports of latency across all AdBook+ environments. Our engineering teams are working with urgency to understand issue and restore performance. Thank you for your patience while we investigate. Please contact us at support.fattail.com with questions otherwise updates will be posted as quickly as possible. Thank you FatTail Support

2. investigating Jun 09, 2025, 04:45 PM UTC

   Our engineering team will be performing maintenance at 1.30pm EST which should last ~20 minutes and is designed to restore system performance. Users will not be able to access the system during this maintenance window. Please advise users to save their work ahead of 1.30pm. We will continue to closely monitor system performance and will update this incident further.

3. monitoring Jun 09, 2025, 06:12 PM UTC

   Maintenance has been completed and we are monitoring the results.

4. investigating Jun 09, 2025, 06:58 PM UTC

   We are continuing to research the cause of latency within AdBook. Additional updates will be provided as soon as possible. FatTail Support

5. investigating Jun 09, 2025, 08:08 PM UTC

   Our engineering team will be performing maintenance at 4.15pm EST which should last ~5 minutes and is designed to help with performance issues. Users will not be able to access the system during this maintenance window. We will continue to closely monitor system performance and will update this incident further.

6. monitoring Jun 10, 2025, 12:21 AM UTC

   Following a second maintenance update at approximately 4.30pm EST AdBook+ system performance has returned to normal levels. We continue to closely monitor system health and will leave this incident open. Thank you for your patience while we work to investigate and resolve latency. FatTail Support.

7. resolved Jun 25, 2025, 12:55 PM UTC

   Following a series of updates and monitoring we are closing this incident. A postmortem will be provided with details on cause of performance issues and remediation steps. Thank you. FatTail Support

Read the full incident report →

Started Apr 28, 2025, 02:28 PM UTC· resolved May 06, 2025, 01:53 PM UTC

1. investigating Apr 28, 2025, 02:28 PM UTC

   We are aware of and working urgently to resolve issues where users are unable to either login, or stay logged in to AdBookOMS. We will provide an update as soon as we can. Thank you for your patience while we work to resolve the issue.

2. investigating Apr 28, 2025, 08:03 PM UTC

   Our team is actively investigating the issue causing users to be intermittently logged out, and we are treating it with the highest priority. We will provide updates as soon as they become available. Thank you for your patience while we work to resolve the issue.

3. monitoring Apr 29, 2025, 12:33 PM UTC

   At 9PM PST on Monday April 28th we deployed an update to resolve the issue where users could not login or stay logged in to AdBookOMS. We are actively monitoring to ensure all access issues have been addressed. If you have questions or experience ongoing issues with access please reach out via support.fattail.com Thank you for your patience while we worked to identify and resolve this incident. FatTail Support.

4. resolved May 06, 2025, 01:53 PM UTC

   This issue was resolved with the release of a fix on Monday, April 28, 2025. A detailed postmortem will be shared once our engineering teams have completed a full analysis of the incident. If you have questions in the interim please contact us via support.fattail.com.

5. postmortem May 09, 2025, 07:51 PM UTC

   ### **Summary of Impact** On April 25, 2025 the new User Creation and Login Experience was deployed to all Production environments. The following Monday, April 28, 2025, FatTail was notified by client users that they were experiencing persistent login issues in AdBook. Specifically, users reported encountering an ‘Unexpected Error Has Occurred’ message immediately after entering their credentials from the new sign-in page. Upon successfully logging in, users were initially able to access and navigate through various areas of the system. However, their user sessions were interrupted by unexpected and intermittent logouts. A timeline of events related to the issue is outlined below. * On Friday, April 25, 2025 at 6PM PT the new User Creation and Login Experience was deployed to all Production environments. * On Monday, April 28, 2025 at approximately 6:45AM PT, users began reporting unexpected logouts after logging into AdBook. Upon attempting to login again, users were met with an ‘Unexpected Error Occurred’ message. At 7AM PT, the issue was escalated to FatTail engineering for further review. * At approximately 1:30PM PT FatTail engineering confirmed that the issue did not originate from AdBook, but rather our service provider Microsoft Azure B2C, which handles user identity storage and authentication for users who login to AdBook. * At 4:30PM PT a joint investigation between FatTail and Microsoft engineering determined the issue stemmed from a high volume of users attempting to validate authentication at the same time when logging into the system, which overloaded Azure B2C. * At 5:30PM PT FatTail identified a resolution to the issue to stabilize authentication validation and allow users to login to AdBook, and remained logged in without interruption. * At 9:30PM PT, FatTail engineering deployed the fix to all Production environments. FatTail Client Success verified that users were no longer being intermittently logged out after initial login, and no longer encountered the ‘Unexpected Error Occurred’ issue. ### **Root Cause** With the new AdBook login experience, all user authentication was re-directed through Microsoft Azure B2C. The root cause of the issue stemmed from load-related strain on Microsoft’s authentication process. A high volume of concurrent authentication validation attempts overwhelmed the authentication service, particularly during peak access times throughout the day of April 28, 2025. As a result, the concurrent authentication validation attempts being made across all environments caused users who were logged into the system to be unexpectedly logged out, and encounter an “Unexpected Error Has Occurred” upon the next login attempt. ### **Mitigation & Next Steps** The following steps were taken to mitigate the issue: * The development team implemented a solution that reduced the volume of calls to the identity provider to validate authentication, while still maintaining a strong approach for validating users accessing application resources are authenticated appropriately. FatTail will implement more rigorous load testing solution to mitigate the risk of similar issues occurring in the future.

Read the full incident report →

Started Apr 27, 2025, 05:37 PM UTC· resolved Jul 14, 2025, 02:36 PM UTC

1. identified Apr 27, 2025, 05:37 PM UTC

   We have identified an issue impacting the 'EditIO' and 'EditDrop' links in OMS email notifications. When users click these links, they are redirected to their AdBook+ landing page instead of the intended Campaign or Drop. This issue does not impact the Portal workflow — email links continue to redirect correctly to specific campaigns within the Portal UI. Our team is actively working on a fix and will provide an update as soon as we have a timeline for the release. We appreciate your patience as we resolve this issue. If you have any questions or concerns, please contact us at support.fattail.com.

2. resolved Jul 14, 2025, 02:36 PM UTC

   This issues was resolved in our 3.98.01 release.

Read the full incident report →

Started Mar 18, 2025, 06:10 PM UTC· resolved Mar 19, 2025, 01:21 PM UTC

1. investigating Mar 18, 2025, 06:10 PM UTC

   We are aware of and investigating an issue affecting all instances of AdBookOMS and Portal. Users may experience slow loading screens and latency while navigating within the application or running reports. Thank you for your patience while we work to restore performance. FatTail Support

2. monitoring Mar 18, 2025, 09:28 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Mar 19, 2025, 01:21 PM UTC

   AdBook+ site performance has been fully resolved. Thank you for your patience while we worked to investigate and resolve this issue. A postmortem will be provided once we have complete details to share on the cause of the issue.

Read the full incident report →