Parade experienced a major incident on October 28, 2022, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Oct 28, 2022, 03:37 PM UTC
Issue Summary We were seeing some pages in the broker portal not loading intermittently for certain users. This primarily affected newly logged in users that logged in after 4:30 AM PST on Oct 22, 2022. Some users logged in with sessions cached before that time were still able to use the portal. Affected pages included every tab in the broker portal except for the Loads and Carriers tabs. Users would have experienced the tabs not loading at all, or not being clickable. This issue was intermittent depending on whether or not third party libraries were able to load at the time of a user visiting the portal, since the issue stemmed from a failed 3rd party CDN. Timeline Issue began when unpkg.com, a popular Javascript Content Delivery Network, started experiencing issues around 4:30 AM PST on October 28, 2022. The Parade team was able to reproduce the issue at 6:28AM PST and identify the root issue. Full resolution was achieved at 8:16 AM PST on October 28, 2022. Users may need to do a hard refresh to get access to the new production fixes, if they were already logged in and were part of the group that experienced these issues. Root Cause Parade was leveraging unpkg.com to manage a few external Javascript libraries. During the morning, unpkg.com experienced issues were certain Javascript packages would not resolve intermittently. This caused unexpected consequences on our production site, which needs those libraries to operate certain pages. Therefore those pages would not load if we were unable to retrieve information from unpkg.com. Technical details of the third party CDN that failed can be found here: https://github.com/mjackson/unpkg/issues/343 This issue affected production sites across many different sites that leveraged unpkg.com as a library management tool. Resolution and recovery The team identified a few libraries that we were using through unpkg.com. We have since replaced these libraries to go through a different CDN network. After this replacement, and a new production hotfix deployment, the team tested and verified that all accounts were able to access the full portal. Corrective and Preventative Measures We have verified that we no longer use ANY libraries through unpgk.com and we will continue to ensure that we do not use unpkg.com in the future. We have also verified that the new version of the broker portal does not contain any usages of The long term plan is for us to self-host certain libraries that are crucial for our application to run, and to use a different production-ready CDN when needed.