RESOLVED: SaaS disruption - Performance issues EU1

Update timeline

1. investigating Jun 27, 2025, 06:53 AM UTC

   We are currently experiencing disruptions with EU1 hosting location. As a result your TOPdesk environment may experience Performance issues. We are aware of the problem and are working on a solution. Please contact TOPdesk Support after submitting an incident, so we can store additional logfiles to investigate this. Our apologies for the inconvenience. At the time of writing this we are not able to give you an estimate on when your environment will be available. We aim to update this status page every 30 minutes until the issue has been resolved. E-mail updates will be sent when the issue has been resolved. You can subscribe on the status page (https://status.topdesk.com) for additional updates. Internal major issues with EU1 To inform TOPdesk you are affected by this issue, please visit My TOPdesk.

2. monitoring Jun 27, 2025, 07:37 AM UTC

   Our SaaS engineers have found a problem with our message queue running out of memory. We've taken steps to address the issue, and the queued jobs are being processed gradually. We are monitoring the situation and will update the status page as soon as we have more details.

3. monitoring Jun 27, 2025, 08:50 AM UTC

   We wanted to update you on the current status of our message queue processing. The queue is still being processed at the moment. This should not have an immediate impact on the performance of TOPdesk. Our team will actively monitor the queue throughout the day to ensure it remains functional. If you encounter any urgent issues or require immediate assistance, please feel free to reach out to our support team.

4. resolved Jun 27, 2025, 12:07 PM UTC

   The issues on EU1 have been resolved since 09:30. After monitoring the issue very closely, the build-up messaging queue decreased at the expected rate, and we are able to confirm that the issue has been resolved.

5. postmortem Jul 25, 2025, 10:09 AM UTC

   On June 27th, the EU1 hosting location experienced a significant disruption that impacted the performance and availability of SaaS environments for many customers. Users reported issues such as endless loading when saving cards, and in some cases, the service was entirely unresponsive. We understand how vital our platform is to your daily operations and sincerely apologize for this inconvenience. The root cause of the disruption was a capacity issue with our messaging service at the EU1 hosting location. The messaging service experienced memory issues due to reaching its capacity limits, which resulted in a backlog of requests and contributed to degraded performance and responsiveness in customer environments. After our cloud engineers implemented a fix, the queued up tasks were processed gradually, resulting in customer environments returning to normal. We continued to closely monitor the situation to ensure stability and full recovery for all customers. We take this incident very seriously and have already identified several improvements to prevent similar issues in the future. In addition, we already have an ongoing project to resize the EU1 hosting location so that resources are properly managed and monitored. This will help prevent similar issues with the messaging service running out of memory and improve the overall resilience of our platform. We sincerely apologize for this service interruption and thank you for your patience and understanding. If you have any further questions or concerns, please reach out to our support team.