Cheddar incident

Some API calls returning 500 errors

Minor Resolved View vendor source →

Cheddar experienced a minor incident on June 1, 2018 affecting API, lasting 3h 39m. The incident has been resolved; the full update timeline is below.

Started
Jun 01, 2018, 01:26 PM UTC
Resolved
Jun 01, 2018, 05:06 PM UTC
Duration
3h 39m
Detected by Pingoru
Jun 01, 2018, 01:26 PM UTC

Affected components

API

Update timeline

  1. investigating Jun 01, 2018, 01:26 PM UTC

    API calls are returning 500 errors for some customers. We’re investigating the cause and will provide an update as soon as possible.

  2. monitoring Jun 01, 2018, 02:46 PM UTC

    We identified a database issue that was causing 500 error responses from Cheddar on some API calls and released a fix. The issue should be resolved as of 10:08 EDT. Customers shouldn't be receiving 500 errors at this point. We'll continue to monitor the situation for the next few hours.

  3. resolved Jun 01, 2018, 05:06 PM UTC

    This incident has been resolved, and the API is functioning normally. Our findings suggest that impact from this incident was minimal. Although Cheddar was returning 500 errors for API calls during the incident window, the application was still properly recording data in the platform (e.g. if a customer create call returned a 500, the customer was still created in your Cheddar product account). We’ve implemented fixes that resolve the underlying issues and correct Cheddar’s API responses, but if you have any concerns about specific updates made during that time, please contact us at [email protected].