FatTail experienced a critical incident on November 21, 2025 affecting AdBook+ System Performance, lasting 31m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 21, 2025, 06:34 PM UTC
We are aware of and urgently investigating an issue with AdBook system performance. Users may experience issues accessing the system. Further updates will be provided here as soon as they are available. Please contact us at support.fattail.com with questions.
- resolved Nov 21, 2025, 07:06 PM UTC
As of 11:00 am PST access to the platform has been restored. We will continue to monitor system performance. Details on the issue will be provided.
- postmortem Dec 10, 2025, 04:21 PM UTC
On Friday, November 21, 2025, FatTail experienced a site-wide outage affecting client access to AdBook. Beginning at approximately 1:30PM ET, users were unable to log in to the platform. System access was restored by 2:00PM ET, resulting in an estimated 30-minute service disruption. FatTail's alerting systems immediately identified the issue, prompting the Engineering and Client Success teams to take swift action. The FatTail status page was updated and direct client notifications were sent promptly to ensure transparency during the event. **Summary of Impact** Below is a detailed timeline of the outage: * 1:30PM ET – Internal alerts notified FatTail of login failures within AdBook. Engineering immediately identified the cause of the issue and began working to restore the service. * 2:00PM ET – The issue was fully resolved and all client access to AdBook was restored. ### **Root Cause** The outage was caused by an internal operator error. The engineering team was preparing for a planned maintenance window, and mistakenly issued a command on the production web server that caused the production system to go offline temporarily. FatTail acknowledges this as a preventable error and is taking steps to strengthen operational safeguards. ### **Resolution** Engineering quickly isolated the cause of the outage and restored system functionality by 2:00PM ET. Services became available immediately upon restoration, and Client Success verified successful user login shortly after resolution. Root cause validation and post-incident review began immediately following service restoration. Additional monitoring confirmed system stability and full functionality. Remediation details will be added once Engineering completes their follow-up analysis. ### **Mitigation and Next Steps** FatTail is implementing additional safeguards to prevent this type of incident from recurring. The following actions will be taken: ### Operational Safeguards for Production Access * Implement environment isolation enhancements to reduce the risk of command execution in incorrect contexts. * Introduce updated operational procedures for all team members with domain admin access. ### Engineering Process Improvements * Collaborate with Engineering leadership to define stricter access workflows. * Evaluate tooling options to differentiate local vs. production environments through shell prompts, window themes, or command-line restrictions. * Reinforce internal best practices and training for handling production-access tasks.