Proxyclick incident
Users may be unable to access the Dashboard and other components
Proxyclick experienced a major incident on June 7, 2022 affecting Dashboard and API, lasting 1h 12m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 07, 2022, 02:23 PM UTC
We are currently investigating this issue.
- monitoring Jun 07, 2022, 02:29 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jun 07, 2022, 02:47 PM UTC
Storage nodes on our internal service bus failed past our load balancing threshold, causing severely degraded performance across all components but primarily affecting our Dashboard and API. Storage was recycled and restored within a few minutes of the initial outage, however, the backlog of requests on the service bus concentrated on only a small subset of our API nodes and caused a secondary overflow to our SQL cluster which extended the incident duration. All storage and API nodes have been recycled and the queues processed, fully restoring service. Our engineering teams will implement improved fail-over and queuing changes to further harden our infrastructure against this type of failure.