Delays in Analysis

Update timeline

1. investigating Feb 25, 2025, 04:13 PM UTC

   We have identified a sudden increase in the number of queued jobs leading to delays in analysis results. We are investigating the source of the delays.

2. investigating Feb 25, 2025, 04:54 PM UTC

   We are still investigating the source of the issue and trying to put in place mitigation measures to bring analysis up to speed. System is struggling to start analysis and most jobs are being heavily impacted.

3. monitoring Feb 25, 2025, 05:31 PM UTC

   Issue was mitigated and system is now catching up backlog of queued analysis. Delays are expectable for a few hours while the system catches up with backed up work. Some analysis might have failed while we deployed mitigation measures and require to be re-analysed. We will keep monitoring the situation.

4. investigating Feb 25, 2025, 05:49 PM UTC

   The mitigation measures were only effective temporarily. Systems is struggling to achieve normal analysis throughput and is back to backing up queued jobs. We are trying to further investigate and identify the source causes and mitigate the issue.

5. investigating Feb 25, 2025, 06:15 PM UTC

   As we keep investigating the source of the issue and try to mitigate it, please remember about our status check bypass in order to not be blocked by any pending analyzes: https://docs.codacy.com/faq/code-analysis/can-i-bypass-codacy-status-check/ . This feature can be used by repository admins to unblock PRs.

6. investigating Feb 25, 2025, 06:17 PM UTC

   Investigating

7. investigating Feb 25, 2025, 09:17 PM UTC

   We continue to investigate the root cause of the outage in our analysis system. As we research and diagnose the problem, we are effectively stopping analyzes. We expect to resume analysis as soon as possible. Customers are advised to bypass analysis gates for the time being.

8. monitoring Feb 25, 2025, 10:45 PM UTC

   Analysis queue is now behaving as expected. It will be monitored and customers should expect some degraded performance as our system runs through the backlog of jobs. Unfortunately, we are not able to fully analyse all dropped commits and pull requests at this time. In order to fully cover pending PR of Codacy analysis we are advising customers to either 1) re-analyze PR 2) re-analyze the head commit or 3) update the PR by adding new commits. As a result and while we are monitoring, proactive SCA will not be ran on 26th of February. We have diagnosed this issue to be related to network and DNS problems. We will be conducting a post-mortem on this issue soon.

9. monitoring Feb 26, 2025, 10:37 AM UTC

   System performance has been able to keep up with demand since last update. We will be slowly ramping back to full performance to cope with the normal daily workload as well as the added effort of having to analyse missed workload from yesterday.

10. monitoring Feb 26, 2025, 01:07 PM UTC

    As we keep ramping back up to full performance we are observing steady and stable performance in our analysis. We will keep monitoring and ramping up to normal performance through the day.

11. resolved Feb 26, 2025, 06:24 PM UTC

    The analysis system's performance and availability are now back to nominal levels. We have finished ramping the system back up and have monitored performance levels throughout the day to ensure that everything is stable.

12. resolved Feb 27, 2025, 09:31 AM UTC

    RCA available at https://codacy.slite.page/p/U1GNMwgpZfcnga/RCA-Lowered-analysis-throughput-leading-to-delays-and-loss-of-analysis-Jobs