FatTail experienced a notice incident on January 29, 2024 affecting AdBook+ System Performance and AdBook+ Order Management, lasting 22h 45m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 29, 2024, 03:45 PM UTC
We are aware of and investigating an issue where users are intermittently receiving 'an unexpected error has been encountered' when logging in to, or navigating within AdBook. We are investigating the issue with urgency, and will provide updates as we learn more. Thank you for your patience while we work to resolve. Please contact us at support.fattail.com with questions or concerns. Thank you, FatTail Support
- identified Jan 29, 2024, 07:36 PM UTC
We have identified the cause of this issue and plan to deploy a fix this evening, Monday January 29th, Eastern Standard Time Zone. We will follow up with details on the underlying cause and specific timing for release, shortly. Thank you for your patience while we work to resolve. Please contact us at support.fattail.com with questions or concerns. Thank you, FatTail Support
- identified Jan 30, 2024, 02:07 AM UTC
Our engineering team has identified 2 items which are causing users to hit an error page on an intermittent basis - 1. In our 3.80.01 release on Friday January 26th, we addressed a defect that was causing tasks to not populate on the task list. This resulted in latency on the task list and in the system. On Monday, January 29th, at 9:00pm Pacific Time, we will be deploy an interim release of version 3.80.02 to our non-Asia Pacific environments (adbook01.fattail.com, adbook02.fattail.com and adbook03.fattail.com) where we will be reverting this change and system performance should be restored. On Tuesday, January 30th, at 11:00 am Pacific Time, FatTail will be deploying this interim release of version 3.80.02 to our Asia Pacific environments (adbook03ap.fattail.com). 2. In our 3.80.01 release on Friday January 26th we also made a modification to our AdBook Authentication, which causes issues for users who have bookmarked AdBook+ pages. When attempting to use these bookmarks, users are encountering an error page. To circumvent this issue, we recommend logging into AdBook+ before accessing any bookmarks. Once logged in, your bookmarks should function as expected. A fix for this issue is scheduled to be released on Friday, February 2nd. Thank you for your patience while we work to resolve. Please contact us at support.fattail.com with questions or concerns. Thank you, FatTail Support
- resolved Jan 30, 2024, 02:30 PM UTC
Following last nights release, users should no longer be seeing 'an unexpected error has been encountered' page when logging in or navigating within AdBook. As previously noted, there is an additional fix scheduled to be released on Friday, February 2nd, which will address an issue with attempting to use bookmarked AdBook+ pages to login. To circumvent this issue, users should log into AdBook+ at the main system URL before accessing any bookmarks. Once logged in, your bookmarks should function as expected. Please contact us at support.fattail.com with questions or concerns. Thank you, FatTail Support