Performance degradation due to traffic spike

Update timeline

1. resolved Mar 25, 2024, 04:19 PM UTC

   Status: Resolved Incident Start: 11:13 AM EDT Incident End: 11:33 AM EDT Duration: 20 minutes Impact: During this incident, customers experienced increased latency, timeouts, and errors while accessing our services. This issue significantly affected user experience, hindering the ability to perform operations reliably and efficiently within our platform. Cause: The incident was caused by an unexpected surge in the flow of requests, which exceeded the anticipated thresholds. This sudden increase in demand throttled our application's performance and scaling capabilities. As a result, our services struggled to process incoming requests effectively, leading to the observed issues. Resolution: Our engineering team promptly responded by increasing number of application replicas. Next Steps: To prevent a recurrence of this incident, we are reviewing and adjusting our scaling strategies to ensure they can handle sudden spikes in demand more gracefully. Additionally, we are improving our monitoring systems to detect and respond to unusual patterns of activity more quickly. Our aim is to enhance the resilience and reliability of our services, minimising any impact on your experience.