Resolved after 4 hours 55 minutes, on March 28, 2019.
Starting around 6:30 AM UTC on March 28th, we encountered a burst of Lambda invocation errors, which caused a backlog in one of our queues, which caused some of the uptime checks to not be recorded in time to avoid alert conditions, which caused bogus alerts to be sent out. The backlog was cleared within a couple of hours. We're looking at changes we can make to our job queues to prevent this kind of scenario from occurring again.
Update timeline
- monitoring Mar 28, 2019, 03:53 PM UTC
Starting around 6:30 AM UTC on March 28th, we encountered a burst of Lambda invocation errors, which caused a backlog in one of our queues, which caused some of the uptime checks to not be recorded in time to avoid alert conditions, which caused bogus alerts to be sent out. The backlog was cleared within a couple of hours. We're looking at changes we can make to our job queues to prevent this kind of scenario from occurring again.
- resolved Mar 28, 2019, 08:48 PM UTC
This incident has been resolved.
Your product runs on more than Honeybadger
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services