DRMtoday incident

DRMtoday Production: Region failover in ap-southeast-2

Notice Resolved View vendor source →

DRMtoday experienced a notice incident on March 29, 2018, lasting 48m. The incident has been resolved; the full update timeline is below.

Started
Mar 29, 2018, 09:11 PM UTC
Resolved
Mar 29, 2018, 10:00 PM UTC
Duration
48m
Detected by Pingoru
Mar 29, 2018, 09:11 PM UTC

Update timeline

  1. investigating Mar 29, 2018, 09:11 PM UTC

    We are currently investigating a region failover in ap-southeast-2. Requests were automatically rerouted to other DRMtoday regions. We apologize for increased response times.

  2. monitoring Mar 29, 2018, 09:36 PM UTC

    Since 21:33 UTC requests are being routed to ap-southeast-2 region again.

  3. resolved Mar 29, 2018, 10:00 PM UTC

    Timeline: 20:59 - Health checks failed for region ap-southeast-2. Requests are automatically routed to other regions 20:59 - Operations team alerted 21:11 - Failed status validated. Customer information sent. 21:33 - Health checks ok for region ap-southeast-2. Requests are automatically routed to ap-southeast-2 region again. DRMtoday's internal health checks were triggered before license deliveries were affected. Preliminary root cause: Despite relying on ntp the system clock on one machine was off by more than 10 seconds. This caused time based signatures, which are also used for our internal health checks to be considered invalid. We apologize for increased response times, due to the rerouted traffic.