FatTail experienced a notice incident on August 29, 2022 affecting AdBook+ System Performance, lasting 1d 4h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 29, 2022, 11:54 AM UTC
We are aware of and investigating potential performance issues with the AdBook+ site detected by our monitoring service. Currently there has been no reported end user impact and sites are loading consistently, however unexpected errors may surface on campaign load or navigation. We will continue to update this incident as we learn more and resolve the issue. Please contact support.fattail.com if you have questions or concerns.
- investigating Aug 29, 2022, 04:48 PM UTC
1pm ET Update: We are continuing our investigations and sites remain stable at this time. Please contact support.fattail.com if you have questions or concerns.
- identified Aug 29, 2022, 09:54 PM UTC
We have identified the cause of intermittent performance issues and are working urgently on a resolution. In addition to 'unexpected error' messages in the AdBook UI, some clients may see Salesforce errors where campaigns are being created, but AdBook Campaign IDs are not being returned to Salesforce. We will continue to provide updates and timing for a fix when we have more information. Please contact support.fattail.com if you have questions or concerns.
- resolved Aug 30, 2022, 04:31 PM UTC
Our engineering team resolved performance issues at approximately 6pm ET yesterday and we continue to actively monitor systems. We will post a post-incident summary shortly explaining cause of error and remediation steps. Please contact support.fattail.com if you have questions or concerns.
- postmortem Sep 13, 2022, 12:42 AM UTC
Thank you for your patience on August 29th, as our teams worked to resolve performance issues with AdBook\+. While the instability was short lived, we take all service disruptions seriously. We have included details below on the causes of the issue. We continue to monitor all systems closely and are confident that these issues are now fully resolved. As always, please contact us at [support.fattail.com](http://support.fattail.com) with any questions or concerns. ============================================== On August 26th we released version 3.63.01 to AdBook\+ production environments. There was an update included in this version designed to correct an issue where AdBook was not consistently presenting the most current availability forecast number. While the code change resolved the issue with forecast data, it inadvertently created a long running process which resulted in high resource utilization across the application and in turn, slower than expected performance for users and certain integrations. On Monday August 29th, at 6.30am EST our system monitoring alerted us to growing resource utilization. Our engineering teams immediately responded and began triage to identify the source of the issue. At 8.00am EST we posted a notice at [status.fattail.com](http://status.fattail.com) advising users that they may encounter ‘unexpected error’ messages in the UI. Our team continued to work on the issue, and we communicated regularly throughout the day with additional info on potential symptoms which users may experience. At approximately 6pm EST we performed a safe, temporary code change to production environments which returned all environments to expected levels of performance and removed the underlying issue with long running processes. This patch remained in place until Friday September 2nd, when we deployed version 3.63.02 to production, rolling back the code change that caused the issue. We are evaluating the approach to resolving the issue with forecast data and will include a revised fix in an upcoming release.