AdvicePay experienced a critical incident on April 19, 2023, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Apr 19, 2023, 05:54 PM UTC
At 11:37 AM MDT on 4/19/2023 we received an alert from our uptime monitoring service that the AdvicePay application was unreachable. Functionality was restored as of 11:50 AM MDT. We are investigating the underlying cause.
- postmortem May 18, 2023, 06:59 PM UTC
AWS S3 was having issues with very high latency on the day of the outage. Additionally, AdvicePay was using a default HTTP client without a timeout to request firms' logo images from S3. The combination of those two issues led to a denial of service due to starvation of connections on our servers. AWS has since resolved things on their end. To protect against this issue in the future, AdvicePay has implemented timeouts for all HTTP clients that we use for requesting any resource.