Honeybadger incident

Bogus check-in alerts

Notice Resolved View vendor source →

Honeybadger experienced a notice incident on March 28, 2019, lasting 4h 55m. The incident has been resolved; the full update timeline is below.

Started
Mar 28, 2019, 03:53 PM UTC
Resolved
Mar 28, 2019, 08:48 PM UTC
Duration
4h 55m
Detected by Pingoru
Mar 28, 2019, 03:53 PM UTC

Update timeline

  1. monitoring Mar 28, 2019, 03:53 PM UTC

    Starting around 6:30 AM UTC on March 28th, we encountered a burst of Lambda invocation errors, which caused a backlog in one of our queues, which caused some of the uptime checks to not be recorded in time to avoid alert conditions, which caused bogus alerts to be sent out. The backlog was cleared within a couple of hours. We're looking at changes we can make to our job queues to prevent this kind of scenario from occurring again.

  2. resolved Mar 28, 2019, 08:48 PM UTC

    This incident has been resolved.