Delays in Analysis

Update timeline

1. investigating Mar 10, 2025, 12:11 PM UTC

   We are seeing and increase in the number of queue jobs leading to longer analysis results times. We are investigating the source of the delays and implementing mitigation measures.

2. investigating Mar 10, 2025, 02:49 PM UTC

   We are still experiencing challenges in keeping up with the queue of analysis. We are working on mitigation measures to be able to achieve a normal throughput.

3. monitoring Mar 10, 2025, 07:29 PM UTC

   Issues with infrastructure resource management was result in starvation on the analysis workers pool, resulting in less ability to perform concurrent analysis, leading to higher queueing and longer Analysis times. We have implemented mitigation measures and improvements in the way the pool is managed. Analysis times are back to normal and we will keep monitoring the performance of the improvements.

4. monitoring Mar 11, 2025, 11:16 AM UTC

   We've been closely monitoring analysis performance and seeing a clear improvement in resource management and overall availability. We are observing Analysis times within normal ranges.

5. monitoring Mar 11, 2025, 05:37 PM UTC

   Analysis performance has been stable throughout the day and kept within expected levels. We rolled out a few more resource management improvements as a measure to guarantee better stability of the analysis system. We'll keep monitoring how these changes perform.

6. resolved Mar 12, 2025, 05:36 PM UTC

   Analysis performance has been consistently within expected levels and the resource management improvements have proved effective and stable so far. We'll resume normal monitoring of our systems and consider this incident resolved.