DatoCMS incident

Slow Responses And Intermittent 503 Error Codes

Minor Resolved View vendor source →

DatoCMS experienced a minor incident on April 6, 2021 affecting Content Management API, lasting 15h 3m. The incident has been resolved; the full update timeline is below.

Started
Apr 06, 2021, 12:00 AM UTC
Resolved
Apr 06, 2021, 03:03 PM UTC
Duration
15h 3m
Detected by Pingoru
Apr 06, 2021, 12:00 AM UTC

Affected components

Content Management API

Update timeline

  1. investigating Apr 06, 2021, 12:00 AM UTC

    We’re receiving feedback of slow responses and intermittent 503 error codes on our Content Management API, we’ve doubled up our server resources and investigating in the meantime

  2. investigating Apr 06, 2021, 12:00 AM UTC

    We mitigated the issue, error rate has dramatically lowered but some requests are still returning errors. We are continuing to analyze the traffic.

  3. monitoring Apr 06, 2021, 12:00 AM UTC

    The reason for the problems concerns a sudden spike in the number of concurrent requests. The situation has returned to more or less normal levels, but we temporarily halved the API request rate limits (from 40 reqs/secs -> 20 reqs/sec) to provide better service to the greatest number of customers while the situation stabilizes. We will restore regular rate limits shortly.

  4. monitoring Apr 06, 2021, 12:00 AM UTC

    Regular rate limits have been restored.

  5. monitoring Apr 06, 2021, 12:00 AM UTC

    The situation is back to normal again. We're sorry for any the inconvenience, and we're continuing to analyze the traffic to ensure that system performance continues to remain consistent.

  6. resolved Apr 06, 2021, 03:03 PM UTC

    Closing the issue as resolved