Tideways incident
MySQL primary database server under extreme load and not responding
Resolved after 40 minutes, affecting Ingest / Data Collection and User Interface, on October 16, 2026.
## **Summary** A maintenance query to delete old entries in a large table was not adequately optimized and required a lot of load. This brought the primary database server under heavy load. Cancelling the query lead to a rollback that also took a while for the system to recover finally. Customers experienced a dataloss of incoming data between 13:42 and 14:28 UTC time on October 16th, 2026. We are very sorry for the inconvenience and you can be assured that we are following up on this incident to make sure a similar mistake does not happen again and is otherwise softened by better automatic recovery and retry mechanisms. ## **Timeline** * **13:30 UTC** - Maintenance query to large table was started. * **13:42 UTC** - Database server load increased to a level that made UI/Ingest unavailable. * **13:49 UTC** - query was killed and started to roll back. * **14:28 UTC** - system recovered and load normalized. ## **Root Cause** An unoptimized maintenance SQL query to cleanup a large table caused high load. ## **Impact** * ~45 minute total data gap across customers \(13:42–14:28 UTC\) * Possible transient UI errors during the incident window ## **Follow-ups** * Current defined best-practices from Ops team were not followed for the maintenance query, so work is being done on reviewing them in the team and potentially surfacing them during code-reviews automatically. * We are investigating how the retry mechanism of daemons from customer machines worked and if we can optimize and improve it for the future.
Affected components
Update timeline
- investigating Oct 16, 2026, 01:53 PM UTC
We are currently investigating this issue.
- investigating Oct 16, 2026, 01:53 PM UTC
We are continuing to investigate this issue.
- identified Oct 16, 2026, 02:10 PM UTC
A maintenance query rolled out during a deployment caused more load than expected, overloading the server. The query was aborted and is now being rolled back, still taking time to process. We hopefully expect the process to be completed in a few minutes.
- resolved Oct 16, 2026, 02:33 PM UTC
Database is responsive again.
- postmortem Oct 16, 2026, 02:53 PM UTC
## **Summary** A maintenance query to delete old entries in a large table was not adequately optimized and required a lot of load. This brought the primary database server under heavy load. Cancelling the query lead to a rollback that also took a while for the system to recover finally. Customers experienced a dataloss of incoming data between 13:42 and 14:28 UTC time on October 16th, 2026. We are very sorry for the inconvenience and you can be assured that we are following up on this incident to make sure a similar mistake does not happen again and is otherwise softened by better automatic recovery and retry mechanisms. ## **Timeline** * **13:30 UTC** - Maintenance query to large table was started. * **13:42 UTC** - Database server load increased to a level that made UI/Ingest unavailable. * **13:49 UTC** - query was killed and started to roll back. * **14:28 UTC** - system recovered and load normalized. ## **Root Cause** An unoptimized maintenance SQL query to cleanup a large table caused high load. ## **Impact** * ~45 minute total data gap across customers \(13:42–14:28 UTC\) * Possible transient UI errors during the incident window ## **Follow-ups** * Current defined best-practices from Ops team were not followed for the maintenance query, so work is being done on reviewing them in the team and potentially surfacing them during code-reviews automatically. * We are investigating how the retry mechanism of daemons from customer machines worked and if we can optimize and improve it for the future.
Your product runs on more than Tideways
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services