Scrutinizer incident

Intermittent Unavailability of Web Frontend

Critical Resolved View vendor source →

Scrutinizer experienced a critical incident on October 25, 2019 affecting Web, lasting 10h 13m. The incident has been resolved; the full update timeline is below.

Started
Oct 25, 2019, 06:01 AM UTC
Resolved
Oct 25, 2019, 04:14 PM UTC
Duration
10h 13m
Detected by Pingoru
Oct 25, 2019, 06:01 AM UTC

Affected components

Web

Update timeline

  1. investigating Oct 25, 2019, 06:01 AM UTC

    We are currently investigating this issue.

  2. identified Oct 25, 2019, 06:32 AM UTC

    The issue has been identified and a fix is being implemented.

  3. identified Oct 25, 2019, 06:44 AM UTC

    We have identified a degradation in the storage that serves git data for various parts of the platform. A recovery process has been triggered and is currently running.

  4. monitoring Oct 25, 2019, 08:28 AM UTC

    The recovery process is still running. All data is already available, but at a reduced redundancy. Due to the ongoing recovery process and restoring of full data redundancy, we are currently seeing git operations to take longer than usual. This can affect individual pages which display source code or access other git information.

  5. monitoring Oct 25, 2019, 09:25 AM UTC

    We are continuing to monitor for any further issues.

  6. monitoring Oct 25, 2019, 01:15 PM UTC

    The recovery process is still running, and we expect it to take another 2 hours before completing. Certain git operations are still degraded during this time. We will continuing to post status updates here.

  7. resolved Oct 25, 2019, 04:14 PM UTC

    This incident has been resolved.