DRACOON experienced a major incident on February 19, 2024, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Feb 19, 2024, 04:13 PM UTC
We are currently investigating an issue with DRACOON Cloud core service. The issue itself has been resolved after about 1 minute and we are now monitoring the situation to ensure it remains stable. We apologize for any inconvenience this may have caused and appreciate your patience.
- postmortem Sep 18, 2024, 05:06 PM UTC
We experienced an incident on 2024-02-19. What Happened: Our system encountered a critical incident characterized by widespread resource depletion. Why Did This Happen: The incident occurred as a result of a race condition, a scenario where multiple threads or processes compete for shared resources in an unpredictable manner. What We Did: Upon identifying the race condition as the underlying cause of the incident, our incident response team swiftly intervened to mitigate the impact on system performance. We implemented immediate measures to address the resource depletion and restore stability to the system. Furthermore, we conducted a comprehensive review of our codebase to identify and rectify any instances of potential race conditions. What We Can Do to Improve: To prevent similar incidents in the future, we have implemented several proactive measures aimed at enhancing system resilience and stability. Firstly, we have bolstered our monitoring infrastructure to promptly detect any abnormal resource consumption patterns indicative of a race condition. Additionally, we have introduced rigorous edge-case testing to uncover and address potential vulnerabilities within our system's concurrency model. We apologize for any inconvenience caused by this incident and assure our users of our unwavering commitment to delivering reliable and stable services. If you have any further questions or concerns, please do not hesitate to reach out to our support team for assistance.