Chef incident

Increased supermarket errors

Minor Resolved View vendor source →

Chef experienced a minor incident on February 6, 2020 affecting Supermarket, lasting 7h 44m. The incident has been resolved; the full update timeline is below.

Started
Feb 06, 2020, 03:36 PM UTC
Resolved
Feb 06, 2020, 11:21 PM UTC
Duration
7h 44m
Detected by Pingoru
Feb 06, 2020, 03:36 PM UTC

Affected components

Supermarket

Update timeline

  1. investigating Feb 06, 2020, 03:36 PM UTC

    We are investigating increased error rates and timeouts with supermarket.chef.io

  2. investigating Feb 06, 2020, 05:56 PM UTC

    We're investigating what is causing the load balancer to remove hosts sporadically.

  3. investigating Feb 06, 2020, 09:26 PM UTC

    The load balancers removing hosts sporadically was a symptom. We're continuing to investigate what appears to be regularly occuring spikes in cache gets at the :00 and :30 minute marks each hour.

  4. monitoring Feb 06, 2020, 10:04 PM UTC

    A fix has been implemented and we are monitoring the results.

  5. resolved Feb 06, 2020, 11:21 PM UTC

    This incident has been resolved.