DRMtoday incident

DRMtoday Production: Delivery log issue

Minor Resolved View vendor source →

DRMtoday experienced a minor incident on January 24, 2021 affecting License delivery log + Reporting API, lasting 1d 1h. The incident has been resolved; the full update timeline is below.

Started
Jan 24, 2021, 08:30 AM UTC
Resolved
Jan 25, 2021, 09:37 AM UTC
Duration
1d 1h
Detected by Pingoru
Jan 24, 2021, 08:30 AM UTC

Affected components

License delivery log + Reporting API

Update timeline

  1. investigating Jan 24, 2021, 08:30 AM UTC

    We are investigating an issue with license delivery logs. Please be aware that data shown in the Dashboard or fetched from the reporting API may be incomplete or invalid. License delivery is not affected.

  2. identified Jan 24, 2021, 09:39 AM UTC

    There's an issue with one of the database cluster nodes where license delivery logs are stored. We're currently trying to recover the faulty node.

  3. identified Jan 24, 2021, 12:08 PM UTC

    We've replaced the failed node and started to restore data. Please be aware that data shown in the Dashboard or fetched from the reporting API is still incomplete.

  4. identified Jan 24, 2021, 01:39 PM UTC

    Data until around midnight UTC has been restored from the latest snapshot. We're now indexing the backlog of live-data from today.

  5. resolved Jan 25, 2021, 09:37 AM UTC

    The backlog has been worked off and missing data has been restored. License delivery data in the Dashboard and from the reporting API is now complete and consistent again.