Noko incident

Mail sending and integrations syncing partially unavailable

Minor Resolved View vendor source →

Noko experienced a minor incident on January 10, 2015, lasting 1h 16m. The incident has been resolved; the full update timeline is below.

Started
Jan 10, 2015, 12:39 PM UTC
Resolved
Jan 10, 2015, 01:55 PM UTC
Duration
1h 16m
Detected by Pingoru
Jan 10, 2015, 12:39 PM UTC

Update timeline

  1. investigating Jan 10, 2015, 12:39 PM UTC

    A few of our background processing queues are not processing tasks, and work is queuing up. We are investigating the cause.

  2. identified Jan 10, 2015, 01:11 PM UTC

    A few queued tasks got stuck when processing due to a timeout when talking to external services not kicking in. We're isolating the affected background worker queues so that further problems with them won't affect other tasks (like mail sending).

  3. monitoring Jan 10, 2015, 01:16 PM UTC

    Background processing is working normally, we'll update when all queued work has been processed.

  4. resolved Jan 10, 2015, 01:55 PM UTC

    All queues are processing normally. There's some backlog on syncing with Basecamp, but we're catching up. If that is the case for your account it should be resolved within the next hour or so.