Percy incident

Intermittent timeouts and delays in Percy builds

Minor Resolved View vendor source →

Percy experienced a minor incident on May 4, 2022 affecting Rendering infrastructure, lasting 4h 6m. The incident has been resolved; the full update timeline is below.

Started
May 04, 2022, 10:34 AM UTC
Resolved
May 04, 2022, 02:40 PM UTC
Duration
4h 6m
Detected by Pingoru
May 04, 2022, 10:34 AM UTC

Affected components

Rendering infrastructure

Update timeline

  1. investigating May 04, 2022, 10:34 AM UTC

    We are currently investigating this issue.

  2. identified May 04, 2022, 11:25 AM UTC

    The issue has been identified and a fix is being implemented.

  3. monitoring May 04, 2022, 12:33 PM UTC

    A fix has been implemented and we are monitoring the results.

  4. monitoring May 04, 2022, 01:00 PM UTC

    We are continuing to monitor for any further issues.

  5. resolved May 04, 2022, 02:40 PM UTC

    We identified an issue introduced as part of a recent OS upgrade that was causing slowdowns in our rendering infrastructure. This has been resolved now, we will conduct internal root cause analysis to make sure this doesn't happen again.