DatoCMS incident

Build trigger and webhooks delay (retrospective)

Major Resolved View vendor source →

DatoCMS experienced a major incident on November 24, 2021, lasting 8h 34m. The incident has been resolved; the full update timeline is below.

Started
Nov 24, 2021, 12:00 AM UTC
Resolved
Nov 24, 2021, 08:34 AM UTC
Duration
8h 34m
Detected by Pingoru
Nov 24, 2021, 12:00 AM UTC

Update timeline

  1. investigating Nov 24, 2021, 12:00 AM UTC

    Between 2021-11-23T21:30:00+00:00 and 2021-11-23T23:20:00+00:00 there was an high increase of latency in the queue that process build triggers and webhooks. This means that deploys and webhooks took more time than usual to being triggered.

  2. identified Nov 24, 2021, 12:00 AM UTC

    The source of the latency was traced down to an abnormal increase of webhook calls in that time frame.

  3. resolved Nov 24, 2021, 08:34 AM UTC

    Build triggers and webhooks are now being processed in two separate queues, so that they do not affect each other anymore.