Major Incident - Data Provisioning Platform

Update timeline

1. investigating May 28, 2020, 05:58 AM UTC

   We are investigating an incident which relates to the availability of the Data Provisioning Platform. Impact: Since 28.05.20 around 2:30 AM the Data Provisioning Platform experienced an outage, thus produced data cannot be distributed by the system. This incident was classified as a Major Incident. Major Incident Managment and root cause analysis are ongoing. We will provide updates either every 45 minutes or event triggered whichever applies.

2. identified May 28, 2020, 06:37 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring May 28, 2020, 07:19 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved May 28, 2020, 08:16 AM UTC

   This incident has been resolved.

5. postmortem May 29, 2020, 06:28 AM UTC

   Please find below **the root cause analysis and planned actions** to avoid this situation in the future: Due to an **out of storage situation** on the backup system of the data provisioning platform, the production component was blocked and additional data from our Data Routing Platform could not processed any more. ‌ **Resolution of the incident:** * The backup system of the data provisioning platform has been set up with higher storage capacity. ‌ **Planned actions:** * **Improve monitoring** for the Data Provisioning Platform - set alarming thresholds 10% lower than now * The **house keeping jobs** for the Data Provisioning Platform’s backup system **will be improved** regarding the storage management * **Improve the intelligent routing** of the various data the routing platform processes.