Major outage

Update timeline

1. investigating Mar 13, 2025, 03:46 PM UTC

   Investigating a database connection problem

2. investigating Mar 13, 2025, 05:37 PM UTC

   We are investigating the a database failure starting at 15:30 UTC that resulted in cascading unavailability of Codacy's services. Most facets of the App (API, Analysis, Provider webhook detected, WebApp) were affected. We are working with our cloud provider to restore availability of the system with the highest urgency. There's no indication that any historical data has been lost, that any data was leaked or breached. It is highly probable that any provider notification (new PRs to analyse, new Commit) have not been handled by Codacy and therefore won't show up in the WebUI.

3. investigating Mar 13, 2025, 06:06 PM UTC

   We continue solving the database stability problem. As your normal workflow might be heavily impacted by this outage, we suggest disabling the branch protection rule. As an example, in Github, Repository Admins can go to Settings > Branches and then edit the protected branches. In the branch settings there is an area dedicated to Status checks. There should be a “Codacy Production” status check there. It can be disabled by pressing the small x. To re-enabled once service is re-established one can search for “Codacy Production” on the Status checks area search bar and re-enable the check.

4. monitoring Mar 13, 2025, 07:52 PM UTC

   Working with our cloud provider we mitigated the database issue by upgrading it's resources. We are still investigating the sudden spike in db resource consumption and we will be working on RCA to determine it's origin. As of right now we see good performances indicators all across the board. We are consuming the backlogged queue of events but we lost a great number of Provider webhook updates, which results in Commits and PRs not being shown in codacy. If PRs and/or Commits are missing from Codacy you can: a) add a commit to the PR b) redeliver the pull_request.* and push events that failed (https://docs.github.com/en/webhooks/testing-and-troubleshooting-webhooks/redelivering-webhooks) Bearing in mind that we are tackling the existing queue (which should take a couple of hours to fully process), if any PRs in Codacy were not analysed as a result of this incident, you can a) re-analyze PR b) re-analyze the head commit or c) update the PR by adding new commits. If a PR is still blocked, your repo admins can use the escape hatch documented here to bypass Codacy status checks.

5. monitoring Mar 13, 2025, 08:14 PM UTC

   Queue of Analysis job has been processed and normal analysis times have been re-established.

6. resolved Mar 20, 2025, 04:01 PM UTC

   RCA available at https://codacy.slite.page/p/K3snOA21LytzBn/RCA-Database-Failure-March-13-2025