DatoCMS experienced a major incident on September 24, 2019, lasting 15h 59m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Sep 24, 2019, 12:00 AM UTC
We are seeing an increased number of calls timing out. When we see a spike of timeouts we can see errors across different endpoints. We are investigating the root cause of the issue and we'll report back as soon as possible.
- identified Sep 24, 2019, 12:00 AM UTC
We have found the root cause in a slow query that was bringing down the database for everyone. We have limited the query duration and the status seems back to normal. We are monitoring and trying to release a stable fix.
- resolved Sep 24, 2019, 03:59 PM UTC
We have identified and improved the query that has caused a snowball error in our database bringing most of the queries to timeouts. The total number of record versions saved in our DB is growing fast and it's causing new challenges for us. We'll monitor the situation in the next few days. Hopefully we'll be faster next time as we have also improved our error reporting from the DB.