Codacy experienced a minor incident on May 7, 2024, lasting 1d 3h. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating May 07, 2024, 01:31 PM UTC
We have noticed issues detecting repository updates. We are investigating the issue
- investigating May 07, 2024, 01:43 PM UTC
We have confirmed that no new analysis are triggering due to an infrastructure issue. We are working to identify and mitigate the root cause of the issue.
- investigating May 07, 2024, 02:03 PM UTC
We are experiencing problems with our cloud storage layer that is resulting in failure to clone repositories and detect new analysis that need to be performed. We have narrowed down the issue to storage layer but are still investigating.
- monitoring May 07, 2024, 02:35 PM UTC
The issue has been mitigated and analysis will now start being detected and processed. The source of the issue was traced back to a network filesystem failure. Due to back-pressure from queued analysis, analysis times might be longer than normal. During the incident we were unable to process some of the provider webhooks that trigger analysis. If you don't see your PRs and Commits showing up and being analysed in Codacy in the next minutes please amend your latest commit so Codacy detects it.
- identified May 07, 2024, 05:49 PM UTC
The implementation of a hotfix to solve the ongoing issue failed, resulting in the loss of queued jobs and the inability to start register and start new analysis jobs.
- monitoring May 07, 2024, 05:55 PM UTC
Hotfix was reverted. Items to analyse are now being detected and analysis are being started as providers notify codacy. Commits and PRs updated during the afternoon of May 7th before 17:55 UTC were lost and will need new activity (amend, new commit) to be detected by Codacy.
- investigating May 07, 2024, 06:41 PM UTC
We are still investigating issues with our network filesystem that are resulting in slower than usual perfomance when detecting items to analyse and affecting the overall time for analysis.
- monitoring May 07, 2024, 07:15 PM UTC
Performance for repository update detection and analysis seems to be back to nominal values for this time of day. We will keep monitoring for possible spikes.
- resolved May 08, 2024, 05:19 PM UTC
After closely monitoring for the past 24 hours there has been no further issues detected and performance has been completely nominal. We are declaring the incident as closed.