Service degradation : Delayed access to West Europe (Production 0) for a subset of customers

Update timeline

1. identified May 27, 2024, 08:33 AM UTC

   We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.

2. identified May 27, 2024, 08:59 AM UTC

   We have identified the issue in regards to the delayed access and have resolved this delay. As a consequence of the delayed access, another issue occurred which resulted in errors when browsing the asset library. Engineers are currently investigating this issue.

3. monitoring May 27, 2024, 09:01 AM UTC

   The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.

4. resolved May 27, 2024, 02:19 PM UTC

   After continuous monitoring, we have confirmed the system is operating normally. The incident has been resolved, and further details will be shared in an upcoming postmortem. We apologize for the impact on affected customers.

5. postmortem May 29, 2024, 12:34 PM UTC

   On May 27, 2024, at 10:06 CET, an issue impacting users in the West Europe \(Production 0\) environment was identified. The issue caused users of West Europe \(Production 0\) to experience errors and slow responses when accessing the application. The engineering team quickly discovered that the problem was caused by a faulty database migration and immediately started to implement a fix. By 10:57 CET, the fix was implemented and deployed to our West Europe \(Production 0\) environment. This resulted in the application working correctly again. After the engineering team closely monitored the application and confirmed that it was working with the desired parameters, the issue was resolved at 16:19 CET. We apologize for the disruption and remain committed to ensuring the reliability and stability of our services.