DRMtoday incident

DRMtoday Production: Key Sync Issues in ap-southeast-2

Notice Resolved View vendor source →

DRMtoday experienced a notice incident on March 8, 2018, lasting 1h 44m. The incident has been resolved; the full update timeline is below.

Started
Mar 08, 2018, 05:31 AM UTC
Resolved
Mar 08, 2018, 07:15 AM UTC
Duration
1h 44m
Detected by Pingoru
Mar 08, 2018, 05:31 AM UTC

Update timeline

  1. investigating Mar 08, 2018, 05:31 AM UTC

    We are currently experiencing issues with key synchronization into our Australian region. Newly ingested keys are not replicated to this region and are not available for delivery.

  2. monitoring Mar 08, 2018, 06:11 AM UTC

    This issue has been resolved.

  3. resolved Mar 08, 2018, 07:15 AM UTC

    This issue has been resolved. Cause: The read replica used for serving licenses in DRMtoday's Sydney region was not receiving key updates (ingests, deletes). Due to a configuration error key synchronization was attempted by a node that was unable to discover its synchronization source. Timeline (all times in UTC) 2018-03-07 11:40 - Application error after an instance failover caused synchronization to read replica to fail. 2018-03-08 04:47 - castLabs notified by customer 2018-03-08 05:05 - issue identified 2018-03-08 06:02 - configuration fix deployed 2018-03-08 06:03 - all keys are up to date in DRMtoday's Sydney region Outlook: In order to avoid similar incidents in the future we will improve alerting. We apologize for the inconvenience.