DatoCMS incident

High response times on CMA and CDA - Admin interface not loading

Minor Resolved View vendor source →

DatoCMS experienced a minor incident on November 2, 2023, lasting 4d 13h. The incident has been resolved; the full update timeline is below.

Started
Nov 02, 2023, 12:00 AM UTC
Resolved
Nov 06, 2023, 01:56 PM UTC
Duration
4d 13h
Detected by Pingoru
Nov 02, 2023, 12:00 AM UTC

Update timeline

  1. monitoring Nov 02, 2023, 12:00 AM UTC

    We experienced about 10 minutes of high response times on our CMA and CDA endpoints. This means that requests sent to those endpoints (site-api.datocms.com, graphql.datocms.com) started to have high latencies or timeout errors (503). The admin interface was affected too, since it uses the CMA endpoint itself. The cause was as unusual amount of traffic that we are going to investigate further. We are monitoring the situation right now.

  2. monitoring Nov 02, 2023, 12:00 AM UTC

    We noticed another batch of unusual traffic at reported time. We will deploy some countermisures soon.

  3. resolved Nov 02, 2023, 12:00 AM UTC

    We can close the issue now

  4. resolved Nov 06, 2023, 01:56 PM UTC

    Resolved — This incident was caused by a customer with a large user base using a private plugin that went rogue. The plugin started making a lot of API calls and since the user base of this customer is very large the entire platform was hit by a DDoS. This caused generic slow downs as the requests were authenticated as users and were hitting directly our web servers. To avoid this from happening again we have added rate-limitations to JWT tokens, that are used by plugins acting as users, something that we didn't anticipate before. Resolved — We can close the issue now Monitoring — We noticed another batch of unusual traffic at reported time. We will deploy some countermisures soon. Monitoring — We experienced about 10 minutes of high response times on our CMA and CDA endpoints. This means that requests sent to those endpoints (site-api.datocms.com, graphql.datocms.com) started to have high latencies or timeout errors (503). The admin interface was affected too, since it uses the CMA endpoint itself. The cause was as unusual amount of traffic that we are going to investigate further. We are monitoring the situation right now.

  5. resolved Nov 06, 2023, 01:56 PM UTC

    This incident was caused by a customer with a large user base using a private plugin that went rogue. The plugin started making a lot of API calls and since the user base of this customer is very large the entire platform was hit by a DDoS. This caused generic slow downs as the requests were authenticated as users and were hitting directly our web servers. To avoid this from happening again we have added rate-limitations to JWT tokens, that are used by plugins acting as users, something that we didn't anticipate before.