Flying Sphinx incident

Noticeable number of API timeouts

Notice Resolved View vendor source →

Flying Sphinx experienced a notice incident on June 7, 2015, lasting 32m. The incident has been resolved; the full update timeline is below.

Started
Jun 07, 2015, 01:56 AM UTC
Resolved
Jun 07, 2015, 02:29 AM UTC
Duration
32m
Detected by Pingoru
Jun 07, 2015, 01:56 AM UTC

Update timeline

  1. investigating Jun 07, 2015, 01:56 AM UTC

    There's been a noticeable increase in API requests timing out. Trying to figure out the cause of the problem. Sphinx daemons are unaffected.

  2. monitoring Jun 07, 2015, 02:05 AM UTC

    The issue was isolated to a specific process and Memcache. Clearing the cache and restarting the process, and we're back to normal.

  3. resolved Jun 07, 2015, 02:29 AM UTC

    API's been smooth as silk since the fix, marking this as resolved. Thanks all for your patience.