FatTail incident
AdBook+ Performance - affecting adbook03.fattail.com
FatTail experienced a notice incident on January 13, 2023 affecting AdBook+ System Performance and AdBook+ Order Management, lasting 6h 43m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 13, 2023, 04:34 PM UTC
We are aware of and investigating an issue affecting only clients who login via adbook03.fattail.com. Users may experience a spinning screen on login, and a delay in the application loading. Thank you for your patience while we work to restore performance. FatTail Support.
- monitoring Jan 13, 2023, 04:40 PM UTC
Site access has been restored and we will continue to monitor to ensure ongoing performance. Our engineering team is investigating cause of issue and we will provide additional details as they are available.
- resolved Jan 13, 2023, 11:17 PM UTC
Performance is stable. A postmortem will be added to this incident once full incident details are available.
- postmortem Jan 20, 2023, 06:52 PM UTC
### **Summary of Impact** At approximately 11.15 am EST on Friday January 13th, users who login to [adbook03.fattail.com](http://adbook03.fattail.com) were unable to access the application. Users in existing sessions experienced an infinite page load. Users attempting to log in to the application could not move past the login screen. In all scenarios users may have eventually seen a “502 Bad Gateway” error. Service was restored at approximately 11.35 am EST following a restart of application services. ### **Root Cause** Our system logging shows that there was a long running process which caused memory usage to increase, ultimately slowing down page load for all users on the adbook03 environment. We have made a couple of refinements to server configurations which may be related; however, our investigations to determine the exact cause have been inconclusive. ### **Mitigation** No specific mitigation steps have been established. ### **Next Steps** As part of the regular course of business our engineering teams will maintain focused on system stability and performance. Any opportunities to refine and improve will be evaluated and executed.