DRMtoday incident

DRMtoday Production: Issue with frontend servers

Major Resolved View vendor source →

DRMtoday experienced a major incident on October 6, 2020 affecting License Delivery and Key Ingestion and 1 more component, lasting 5h 16m. The incident has been resolved; the full update timeline is below.

Started
Oct 06, 2020, 11:15 AM UTC
Resolved
Oct 06, 2020, 04:31 PM UTC
Duration
5h 16m
Detected by Pingoru
Oct 06, 2020, 11:15 AM UTC

Affected components

License DeliveryKey IngestionDashboard + Config API

Update timeline

  1. investigating Oct 06, 2020, 11:15 AM UTC

    We are currently investigating an issue with our frontend services affecting key ingest, the dashboard, and configuration API.

  2. identified Oct 06, 2020, 12:17 PM UTC

    The issue was caused due to unexpected high load on our ingest API. We're currently scaling up instances to handle the increased load. We are also applying rate limiting for ingest requests for the moment to help the system stabilize.

  3. identified Oct 06, 2020, 01:34 PM UTC

    We have currently disabled the US region for maintenance of the key database. License requests from the area are routed to other regions now.

  4. identified Oct 06, 2020, 02:23 PM UTC

    Maintenance of the key database in the US region has been completed. License requests are processed in the region again.

  5. resolved Oct 06, 2020, 04:31 PM UTC

    This incident has been resolved.