Honeybadger had 10 outages in the last 2 years totaling 13h 33m of downtime — averaging 0.4 incidents per month.
There were 10 Honeybadger outages since July 24, 2024 totaling 13h 33m of downtime. Each is summarised below — incident details, duration, and resolution information.
We had a backlog in our notifications queue that prevented timely delivery of alerts to channels such as Email, Slack, etc. The backlog has been processed as of 19:20 UTC. Our autoscaling rules for this service had a monitoring gap that prevented a scale-out to maintain a low latency on the queue. This monitoring gap has been resolved.
A recent deploy has resulted in our rejecting some check-in API requests. We are in the process of deploying a fix.
This incident has been resolved.
The issue has been identified and a fix is being implemented.
This incident has been resolved.
API ingestion of errors has been interrupted. We are deploying a fix.
The US API has been restored, and the EU API restoration is in process.
The EU API has been restored.
Our error processing pipeline has been impacted by the AWS outage earlier this morning. Errors are being processed, but the end-to-end latency is much higher than normal. We are working to restore the processing to full capacity.
We're continuing to monitor the AWS outage. Errors are being processed, but may be delayed. Uptime checks are also delayed, so you may notice lag with uptime notifications and status page updates.
Processing has been restored and all systems are normal. We'll continue to monitor as AWS continues to restore service. Thanks!
This incident has been resolved.
Our systems were affected by the DynamoDB outage in AWS' us-east-1 region. Impacts included not being able to log use the web app, uptime checks not being executed on time, and check-ins being falsely reported as missing. We are evaluating what steps we can take to mitigate these kinds of impacts from this kind of outage in the future. Additionally, our status page was not updated in a timely manner due to our not having configured various AWS services as dependent components of our status page. We have fixed that issue, so future AWS outages will result in our status page automatically being updated.
We had an interruption in processing of check-in alerts and in-app stats from about 12:30 to about 13:30 UTC. This may have resulted in a failure or a delay in reporting check-ins that went missing during that period.
We have identified and corrected an issue in our data processing pipeline that may have caused delays for some customers. No data has been lost and the system should be caught up shortly.
The backlog has been processed and all systems are running normally.
We are looking into the cause of slowness and timeouts with the web app.
We have identified the cause of the slowdown and performance is back to normal.
This incident has been resolved.
We have identified an issue with Insights event ingestion backlog. We are working on a fix.
We have successfully pushed a fix and are waiting for the backlog to be processed. I also want to give a reminder that this only affects Insights events; error processing was unaffected.
The backlog has been fully processed, so all Insights events should be ingested and available.