DRMtoday incident

DRMtoday Production: Delayed delivery logs

Minor Resolved View vendor source →

DRMtoday experienced a minor incident on March 29, 2020 affecting License delivery log + Reporting API, lasting 11h 34m. The incident has been resolved; the full update timeline is below.

Started
Mar 29, 2020, 10:54 AM UTC
Resolved
Mar 29, 2020, 10:28 PM UTC
Duration
11h 34m
Detected by Pingoru
Mar 29, 2020, 10:54 AM UTC

Affected components

License delivery log + Reporting API

Update timeline

  1. investigating Mar 29, 2020, 10:54 AM UTC

    We are currently investigating gradually decreasing performance of license delivery log indexing. Please be aware that data starting 2020-03-28 9:00 UTC is currently not available or incomplete when shown in the Dashboard or fetched from the reporting API.

  2. monitoring Mar 29, 2020, 04:01 PM UTC

    We've deployed an update to our indexing strategy in order to mitigate the underlying performance issue and started additional workers to speed up processing to work off the backlog. This may take several hours. We'll keep the issue open on the status page and post updates accordingly. Please be aware that data starting 2020-03-28 9:00 UTC is still not available or incomplete when shown in the Dashboard or fetched from the reporting API.

  3. monitoring Mar 29, 2020, 07:17 PM UTC

    The backlog from our European and US regions has been processed. License delivery data for both regions is now complete and up to date again. We're still indexing data from our Asian and Australian regions. Please be aware that data starting 2020-03-28 21:00 UTC is still not available or incomplete for the Asian and Australian regions when shown in the Dashboard or fetched from the reporting API.

  4. resolved Mar 29, 2020, 10:28 PM UTC

    The backlog has been worked off. License delivery data in the Dashboard and from the reporting API is now complete and consistent again.