Parade experienced a minor incident on September 12, 2022, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Sep 12, 2022, 07:22 PM UTC
Issue Summary We had a brief slowdown of our broker web portal during a short period of time between 6:57 AM to 7:15 AM PST. End users of the portal would have seen a never-ending loading screen, or the inability to add/modify data during this time period. Our statuspage also updated with the proper components affected during this time. Timeline This issue was first reported by our monitoring services at 6:57 AM PST, and automatically escalated to both our support and on-call engineer teams. This issue resolved itself, with no corrective action necessary at 7:15 AM PST. Root Cause This was due to an abnormally larger amount of web requests taking more than 5 seconds to complete, which triggered our monitoring alarms. We have identified a few requests that were taking slightly longer than usual, and blocking resources from handling other requests to our APIs. Resolution and recovery No corrective action was needed by our team, as once the requests finished, the web portal APIs returned to normal functionality. Recovery occurred at 7:15 AM PST, with no data being lost. Corrective and Preventative Measures We are tackling preventative measures from 2 directions, as this issue seems to be a combination of: 1. Surges in traffic: we are working on better identification of our API autoscaling to make sure this aligns with the traffic that we see throughout the day 2. Slow APIs that are affecting the system as a whole: we have identified a few APIs to speed up, so that as we scale, these do not become a growing issue.