Slido experienced a major incident on April 22, 2022 affecting Europe API and slido.com and 1 more component, lasting 44m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 22, 2022, 01:08 PM UTC
We are investigating issues that might be affecting some of our customers. Some actions or interactions with Slido or Slido API might be resulting in errors. Our team is on it and trying to identify the root-cause. Please bear with us as we continue to investigate.
- resolved Apr 22, 2022, 01:19 PM UTC
Our team managed to resolve the issue after it was identified and we will take the necessary steps to prevent this from happening again. This is not the experience we are committed to delivering to you and we hope it didn't negatively affect your meeting or event. If you have any other questions, please let us know at [email protected]. Have a wonderful weekend!
- postmortem May 11, 2022, 06:40 AM UTC
## **Summary** On Friday, April 22nd, 2022 at 12:20 UTC, Slido users experienced errors and/or increased latency for around 45 minutes. We’d like to apologize to our customers who were impacted by this disruption. There was no impact to your data and Slido’s security wasn’t breached by this incident. Rest assured that we are taking several actions to improve the resiliency and performance of our service to prevent this type of problem happening again. ## **Root cause** This incident was triggered by continuous, machine-generated requests sent to one of our API endpoints in our EU cluster. The nature and volume of this adverse traffic led to the `max_prepared_stmt_count` limit \[1\] on our MySQL database being reached. As a result, subsequent requests started failing. ## **Remediation** We detected the incident through our monitoring at 12:27 UTC. At 12:48 UTC, we manually blocked the source of the adverse traffic on our firewall. At this stage the value for our `max_prepared_stmt_count` limit \[1\] was increased. At 12:59 UTC, we blocked a second batch of IP addresses and the `max_prepared_stmt_count` value was increased again. The incident was fully resolved at 13:05 UTC. ## **Prevention** We are taking the following actions to ensure this does not happen again: * Control the number of database prepared statements created by our backend service * Improve monitoring and add alerting of the number of prepared statements in our database clusters * Improve rate limiting on our API * Improve tools and procedures for posting external communications during outages – – – – \[1\] [https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar\_max\_prepared\_stmt\_count](https://dev.mysql.com/doc/refman/5.7/en/server-system-variables.html#sysvar_max_prepared_stmt_count)