Codecov incident

Intermittent failures

Minor Resolved View vendor source →

Codecov experienced a minor incident on February 5, 2024 affecting Codecov Backend, lasting 5h 17m. The incident has been resolved; the full update timeline is below.

Started
Feb 05, 2024, 12:15 PM UTC
Resolved
Feb 05, 2024, 05:32 PM UTC
Duration
5h 17m
Detected by Pingoru
Feb 05, 2024, 12:15 PM UTC

Affected components

Codecov Backend

Update timeline

  1. investigating Feb 05, 2024, 02:11 PM UTC

    We are currently investigating an issue with our database. High utilization is causing intermittent timeouts and 500s.

  2. monitoring Feb 05, 2024, 04:23 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved Feb 05, 2024, 05:32 PM UTC

    A database query began unexpectedly taking large amounts of time to complete. Our SRE team is focused on improving this response time and working closely with our cloud provider to come to a resolution. We have implemented several fixes to mitigate the issues and are continuing to monitor and improve.