Codacy incident

Major outage

Minor Resolved View vendor source →

Codacy experienced a minor incident on April 15, 2025, lasting 14d 1h. The incident has been resolved; the full update timeline is below.

Started
Apr 15, 2025, 06:06 AM UTC
Resolved
Apr 29, 2025, 07:27 AM UTC
Duration
14d 1h
Detected by Pingoru
Apr 15, 2025, 06:06 AM UTC

Update timeline

  1. investigating Apr 15, 2025, 06:06 AM UTC

    Service is unavailable.

  2. identified Apr 15, 2025, 07:17 AM UTC

    System has been brought back to partial health. App and API access should be partially re-established and we are gradually increasing analysis capacity. Longer than usual analysis times are expected at this stage.

  3. investigating Apr 15, 2025, 07:20 AM UTC

    Scaling up analysis greatly impacted performance. We are continuing our investigations to re-establish service.

  4. monitoring Apr 15, 2025, 07:45 AM UTC

    We are scaling up analysis availability and monitoring performance. App and API are available at this point and analysis is catching up. Monitoring for systems stability.

  5. monitoring Apr 15, 2025, 09:29 AM UTC

    Some delays in older analysis are to be expected as we catch up with the long tail of queued jobs but otherwise systems are back to normal.

  6. monitoring Apr 15, 2025, 09:31 AM UTC

    We are further investigating issues caused by database cleanup mechanisms that cause peaks in memory consumption leading to service instability. Prevention measures had been rolled out throughout last week but these have proved not enough to prevent failures from happening. We are working diligently to better understand and ultimately prevent these failures and disruption to service and customer workflows from happening .

  7. monitoring Apr 15, 2025, 11:19 AM UTC

    We'll be proceeding with configuration changes to our databases to promote better resource management, for both database operations as a well as database background processes, to prevent issue reoccurrence.

  8. investigating Apr 15, 2025, 01:48 PM UTC

    We are observing delays in analysis and pilling up of analysis jobs in queue. We are investigating an issue that seems to be preventing new analysis workers from spawning.

  9. monitoring Apr 15, 2025, 02:50 PM UTC

    Analysis throughput is back to normal performance.

  10. resolved Apr 29, 2025, 07:27 AM UTC

    Resolved