Codacy incident

Systems instability due to increased database load

Resolved after 1 hour 2 minutes, on June 26, 2025.

resolved Jun 26, 2025, 01:00 PM UTC

System performance is back to normal values. Analysis delays related to this issue are not to be expected. A large number of concurrent queries exposed inefficiencies on a query that led to increased load and subsequent systems instability and slowness. Team is working on improving the query and preventing this specific issue from re-occurring.

Started
Jun 26, 2025, 11:58 AM UTC
Resolved
Jun 26, 2025, 01:00 PM UTC
Duration
1h 2m
Detected by Pingoru
Jun 26, 2025, 11:58 AM UTC

Update timeline

  1. investigating Jun 26, 2025, 11:58 AM UTC

    We are investigating higher response times for APIs and web app seemingly caused by increased database load.

  2. investigating Jun 26, 2025, 12:10 PM UTC

    As a measure to decrease load in our databases we are going to throttle the number of concurrent analysis. This will result in longer analysis times as analysis will be queued.

  3. identified Jun 26, 2025, 12:26 PM UTC

    Throttling the number of analysis successfully restored health to the affected database. We are gradually bringing up analysis resources to full speed and monitoring the health of the database. Delays in analysis are still to be expected.

  4. monitoring Jun 26, 2025, 12:34 PM UTC

    System is stable now but during the degraded performance period a large backlog of analysis was accumulated. This backlog of analysis is now being tackled and in time analysis times will go back to expected values.

  5. resolved Jun 26, 2025, 01:00 PM UTC

    System performance is back to normal values. Analysis delays related to this issue are not to be expected. A large number of concurrent queries exposed inefficiencies on a query that led to increased load and subsequent systems instability and slowness. Team is working on improving the query and preventing this specific issue from re-occurring.

Your product runs on more than Codacy

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →