[App/Intranet] [German System] - Long loading times and slow search

Update timeline

1. investigating May 19, 2025, 01:46 PM UTC

   Some customers may be experiencing longer loading times and delays in receiving search results. Our teams are actively investigating the issue and working to resolve it as quickly as possible. We will share more information soon. You can see on which system your application is hosted by going to the Studio > Settings > API Access and checking the tag next to the title. We apologize for the inconvenience. If you have any questions regarding this topic, please reach out to our support at [email protected].

2. monitoring May 19, 2025, 03:13 PM UTC

   We've implemented a fix, and loading times and search functionality have returned to normal. Our team will continue to monitor the systems closely to ensure the issue is fully resolved.

3. investigating May 20, 2025, 07:41 AM UTC

   We're currently investigating renewed issues with slow search performance. Our team is actively working to identify the root cause and will share updates as we learn more. We apologize for the interruption. In case you have any questions about this topic, please let us know at [email protected].

4. monitoring May 20, 2025, 10:53 AM UTC

   A new fix has been implemented, and both loading times and search functionality have returned to normal. We are continuing to monitor system performance to ensure stability and confirm that the issue is fully resolved. Thank you for your continued patience.

5. resolved May 20, 2025, 03:48 PM UTC

   The issue is resolved. We apologize for the inconvenience this has caused. Please get in touch with us at [email protected] in case you're still having any problems. We’re happy to help!

6. postmortem Jul 07, 2025, 02:24 PM UTC

   **Summary** As our investigation of the incident is now complete, we would like to share more details about the issue that occurred between May 19th 2015, 15:46 CEST, and May 20th 2025, 17:48 CEST. ‌ **What happened** Beginning on May 19th, we observed increased response times and error rates in the backend systems, particularly affecting search functionality. This led to sporadic service degradation throughout the day and impacted the overall performance of our platform. Initial mitigation steps restored stability temporarily. However, performance degraded again on May 20th, resulting in long load times. ‌ **Impact** The issue primarily affected customers hosted on our German system, where customer apps experienced slow response times, and in some cases, unresponsive apps. ‌ **Future Improvements** We are actively working on several long-term improvements to prevent similar incidents in the future: * Refining internal caching and data retrieval mechanisms * Improving configuration consistency and query performance * Evaluating and tuning backend infrastructure * Ensuring future changes undergo deeper validation in staging environments * Monitoring improvements to detect performance anomalies earlier ‌ We are sorry for the trouble this has caused. If you have further questions, please don’t hesitate to reach out to us at [**[email protected]**](mailto:[email protected]).