Honeycomb.io Outage History

Started Oct 20, 2026, 02:04 PM UTC· resolved Oct 20, 2026, 04:26 PM UTC

1. investigating Oct 20, 2026, 02:04 PM UTC

   We are currently investigating the degradation of various services in our US region.

2. investigating Oct 20, 2026, 03:18 PM UTC

   We've identified and isolated the impacted service and are seeing recovery in query execution. SLO processing remains degraded.

3. resolved Oct 20, 2026, 04:26 PM UTC

   SLO processing has fully recovered. All services are now functional.

4. postmortem Oct 21, 2026, 11:26 PM UTC

   On October 20, Honeycomb experienced about 3 hours of degraded service in the US region. From 13:30 to 14:40 UTC, querying, the Honeycomb UI, the management API, and triggers experienced elevated latency and error rates, with the highest impact between 13:55 and 14:10 UTC. Some trigger alerts may have been late or missed during that time. SLO processing suffered delays between 13:26 and 14:10 UTC. From 13:57 to 14:09 UTC, about half of requests to the event ingest API received 5xx errors. Lesser impact occurred between 13:48 to 13:55 \(1% error rate\) and for approximately minute at 14:15 \(3% error rate\). We’ll publish a more in-depth report on Friday, October 23 which will cover detection, remediation, and outstanding action items.

Read the full incident report →

Started Oct 20, 2026, 01:30 PM UTC· resolved Oct 20, 2026, 02:09 PM UTC

1. resolved Oct 20, 2026, 05:07 PM UTC

   From 13:57 to 14:09 UTC, event ingest in Honeycomb's US production environment was degraded by an concurrent incident. During this window, approximately 53% of incoming events received 5xx errors.

Read the full incident report →

Started Aug 21, 2026, 03:21 PM UTC· resolved Aug 21, 2026, 03:59 PM UTC

1. identified Aug 21, 2026, 03:21 PM UTC

   Canvas is not loading for customers in the US and EU regions and we have identified what we believe to be the cause and are rolling out a fix.

2. monitoring Aug 21, 2026, 03:25 PM UTC

   We have deployed the fix and are seeing recovery of normal Canvas functionality.

3. resolved Aug 21, 2026, 03:59 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 10, 2026, 01:01 PM UTC· resolved Aug 10, 2026, 08:35 PM UTC

1. monitoring Aug 10, 2026, 01:01 PM UTC

   From 11:36UTC to 12:43UTC, queries in US1 that read older data returned errors or timed out. Queries over recent data were un-affected, and no telemetry is being lost. The issue has been identified, and a fix is in place. We are currently monitoring the situation to confirm resolution.

2. resolved Aug 10, 2026, 08:35 PM UTC

   The fix is in place and we've confirmed resolution. Queries and SLOs are processing normally.

Read the full incident report →

Started Jul 31, 2026, 02:30 PM UTC· resolved Jul 31, 2026, 03:00 PM UTC

1. resolved Jul 31, 2026, 03:42 PM UTC

   Between 10:30 and 11:00 AM EDT, we experienced elevated querying failures and slowness. The impact has subsided, and we are currently investigating and monitoring the situation.

2. postmortem Aug 21, 2026, 01:18 PM UTC

   Starting at 9:50 AM Central Time on July 31st we received alerts around querying health and latency. The cause was eventually tracked down to a clash of two different factors. First, we have been introducing Lambda Managed Instances \(LMI\) into our query architecture, which behave similar to classic lambdas in many ways and had been introduced as part of work around querying performance improvements with the goal of a better and faster user experience. Unfortunately, a difference between classic lambdas and LMIs we discovered is the 15 minute hard stop of classic lambdas does not exist on LMIs, which means that certain assumptions built into our querying system no longer held. At the time of the incident, our newer Canvas feature was running batches of queries for a customer which should have a self imposed one minute time out. This timeout was not properly communicated back to the lambda, and while with classic lambdas the task would automatically cut off at 15 minutes, the new LMIs were not enforcing this sort of safety mechanism. As a result, some large queries were running for up to 30 minutes, driving up query latency across the board and resulting in errors for users trying to run fresh queries. The system recovered by 10:10 AM CT, 20 minutes later, when the large queries cleared out, but we continued to investigate the cause of the incident. Once we identified this weakness in our querying architecture we began work to fix it and prevent the same incident from occurring again. We have shipped all associated incident follow ups.

Read the full incident report →

Started Jul 27, 2026, 01:13 PM UTC· resolved Jul 28, 2026, 06:30 PM UTC

1. investigating Jul 27, 2026, 01:13 PM UTC

   We are seeing triggers intermittently failing in the EU region. We are actively investigating the cause.

2. investigating Jul 27, 2026, 01:52 PM UTC

   We are continuing to see issues with triggers and are also noticing issues with querying. We are investigating for the cause of both.

3. monitoring Jul 27, 2026, 04:01 PM UTC

   We have identified the source of the issue, applied a temporary fix, and are working on a permanent fix. We continue to monitor the situation.

4. monitoring Jul 27, 2026, 07:46 PM UTC

   We are seeing a regression in performance with partial degradation of both triggers and querying. We are working to remediate.

5. monitoring Jul 27, 2026, 10:51 PM UTC

   The partial degradation has been resolved for all customers except for those specifically contacted. We are working on mitigation measures so that we can fully restore trigger functionality for everyone.

6. monitoring Jul 28, 2026, 06:21 PM UTC

   We are continuing to monitor for any further issues.

7. resolved Jul 28, 2026, 06:30 PM UTC

   The degradation in querying and triggers has been resolved and we're back to full functionality.

Read the full incident report →

Started Jul 02, 2026, 06:10 PM UTC· resolved Jul 03, 2026, 12:31 AM UTC

1. investigating Jul 02, 2026, 06:10 PM UTC

   We are currently investigating this issue.

2. monitoring Jul 02, 2026, 06:13 PM UTC

   Clearing the CDN cache appears to have solved the issue.

3. resolved Jul 03, 2026, 12:31 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 29, 2026, 10:07 PM UTC· resolved Jun 30, 2026, 12:47 AM UTC

1. monitoring Jun 29, 2026, 10:07 PM UTC

   Starting at 14:24 PT, there was an issue with the Activity Log causing events to stop being processed. You may notice a gap starting at that time. We are slowly backfilling these events and do not expect any data loss to occur. We expect the full data log to be restored around 17:30 PT.

2. resolved Jun 30, 2026, 12:47 AM UTC

   Backfill of the Activity Log events has completed. All events during the affected period should be present.

Read the full incident report →

Started Jun 25, 2026, 06:10 PM UTC· resolved Jun 25, 2026, 08:41 PM UTC

1. investigating Jun 25, 2026, 06:10 PM UTC

   We are investigating an issue with delayed ingest in the EU region. SLOs and Trigger evaluations may also be delayed.

2. monitoring Jun 25, 2026, 06:31 PM UTC

   We are monitoring the situation. Events may have been delayed between 17:53 and 18:04 UTC.

3. resolved Jun 25, 2026, 08:41 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 24, 2026, 06:14 PM UTC· resolved Jun 24, 2026, 07:04 PM UTC

1. investigating Jun 24, 2026, 06:14 PM UTC

   We are investigating an issue with delayed ingest in the US and EU region. Our engineers are rolling back a recent deploy. SLOs and Trigger evaluations may also be delayed.

2. monitoring Jun 24, 2026, 06:22 PM UTC

   We have rolled back the deploy and ingest service has recovered. There has been an ingest outage from 17:55 - 18:14 UTC. We have also observed an delay for Service Maps.

3. monitoring Jun 24, 2026, 06:31 PM UTC

   Triggers, SLOs and Service Maps are recovered. We are continuing to monitoring the Ingest Service

4. resolved Jun 24, 2026, 07:04 PM UTC

   All services have been fully recovered. Impact window: 17:55–18:17 UTC Regions affected: Primary impact in US. The EU region was affected to a much lesser degree. During this period, the following effects may have occurred: Ingest: Some data was not ingested, resulting in complete or partial data loss for events sent during the window. Triggers: Triggers may have failed to fire within the impact window. SLOs: SLI values across the affected window are skewed by the missing data and may show artificial dips or accelerated budget burn. Service Maps: Maps covering the outage window are incomplete. Services and dependencies may be under-counted, as their traces were not ingested. We apologize for the disruption. Please reach out if you have any questions about how this may have affected your data.

5. postmortem Jul 02, 2026, 08:10 PM UTC

   On June 24, we experienced approximately 30 minutes of severe data ingestion degradation in Honeycomb’s US and EU instances. During this degradation, Honeycomb rejected a significant percentage of inbound telemetry, and customers would have seen up to a half-hour gap in ingested telemetry. Additionally, during this same time window, customers would have experienced data processing degradation and service disruption within Anomaly Detection and Service Maps. A deployment containing a change to how services retrieve dataset schemas into local caches caused a cascading set of failures, resulting in failed remote cache retrievals, and subsequently, a MySQL stampede due to several services falling back to retrieving schemas from the database. This stampede resulted in significant memory utilization growth, causing some services to crash loop with out-of-memory exceptions while retrieving schemas. Customers would have seen 5xx errors from our ingestion APIs when this occurred, because ingestion services were among those that were crash looping. This also included crash looping from services responsible for Anomaly Detection and Service Maps. Right after these service crash loops began, alerts fired for the crashing services, and procedures were run to pin all services back to the last known good build before the breaking change landed. Once services were running the previous deploy’s build, the 5xx error responses from ingestion APIs returned to nominal baseline levels, and services that were crash looping recovered and resumed normal operation. The underlying issue centered on how dataset schema cache payloads and metadata were serialized to memcached for remote cache retrieval, and how memcached clients deserialize that data before writing it to a local cache. The code change contained a feature flag to toggle the serialization behavior, and backwards-compatibility safeguards for clients deserializing different schema versions from memcached. However, services’ schema deserialization prior to the feature flag flip on did not behave as intended, treating each backwards-compatible schema read as a hard error rather than as a fallback, which subsequently caused every remote schema cache read to fall through to the database. Subsequent analysis and review of the code identified the issue and we have re-deployed this change with a fix along with additional tests. Services now correctly and successfully deserialize schemas from memcached in a safe, backwards-compatible manner.

Read the full incident report →

Started Jun 19, 2026, 04:39 PM UTC· resolved Jun 19, 2026, 05:21 PM UTC

1. investigating Jun 19, 2026, 04:39 PM UTC

   We are aware of an issue affecting Honeycomb MCP. Write tool functionality may be degraded. Read tools are unaffected.

2. monitoring Jun 19, 2026, 05:18 PM UTC

   We are aware of an issue affecting Honeycomb MCP. Write tool functionality may be degraded. Read tools are unaffected.

3. resolved Jun 19, 2026, 05:21 PM UTC

   MCP Write Tools have been restored. MCP users may need to reconnect to see all available tools.

Read the full incident report →

Started Jun 18, 2026, 07:10 PM UTC· resolved Jun 18, 2026, 07:10 PM UTC

1. resolved Jun 18, 2026, 07:10 PM UTC

   Beginning June 17th, enhance-related requests were failing to complete. We've since identified and deployed a fix, and enhance is now fully operational. Thank you for your patience.

Read the full incident report →

Started Jun 17, 2026, 02:01 PM UTC· resolved Jun 17, 2026, 09:14 PM UTC

1. investigating Jun 17, 2026, 02:01 PM UTC

   We are continuing to investigate intermittent slowness and query failures affecting our Production EU Region. We will provide an update as soon as we have more information.

2. monitoring Jun 17, 2026, 03:29 PM UTC

   Querying in our Production EU Region is fully operational. We will continue to monitor for any abnormal behavior.

3. resolved Jun 17, 2026, 09:14 PM UTC

   We have identified the root cause of the intermittent slowness and query failures affecting our Production EU Region. The issue was triggered by a large volume of dataset deletions that caused elevated database load, resulting in query instability. No data was lost during this incident. Service has been restored. A fix has been applied to help prevent recurrence.

Read the full incident report →

Started Jun 17, 2026, 12:01 PM UTC· resolved Jun 17, 2026, 12:32 PM UTC

1. investigating Jun 17, 2026, 12:01 PM UTC

   We are currently investigating the cause of slowness and query failures in our Production EU Region.

2. resolved Jun 17, 2026, 12:32 PM UTC

   The issue is now resolved and querying is back to fully functional.

Read the full incident report →

Started Jun 11, 2026, 03:30 PM UTC· resolved Jun 11, 2026, 10:24 PM UTC

1. resolved Jun 11, 2026, 11:17 PM UTC

   Requests to Honeycomb's Query Data and Management APIs in the production-eu1 environment encountered increased error rates between approximately 15:35 and 22:24 UTC. Event ingestion was not affected. The issue has been resolved.

Read the full incident report →

Started Jun 10, 2026, 08:58 PM UTC· resolved Jun 10, 2026, 10:02 PM UTC

1. investigating Jun 10, 2026, 08:58 PM UTC

   We are investigating an issue with delayed ingest in the EU region. Received events are still being stored, but there may be a delay in event retrieval in queries. SLOs and Trigger evaluations may also be delayed.

2. investigating Jun 10, 2026, 09:28 PM UTC

   Queries should no longer be delayed. SLOs and Trigger evaluations remain impacted.

3. identified Jun 10, 2026, 09:29 PM UTC

   The issue has been identified and a fix is being implemented.

4. resolved Jun 10, 2026, 10:02 PM UTC

   All services are now healthy.

Read the full incident report →

Started Jun 08, 2026, 04:37 PM UTC· resolved Jun 08, 2026, 05:41 PM UTC

1. investigating Jun 08, 2026, 04:37 PM UTC

   We are currently investigating this issue.

2. resolved Jun 08, 2026, 05:41 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 03, 2026, 08:50 PM UTC· resolved Jun 03, 2026, 08:51 PM UTC

1. identified Jun 03, 2026, 08:50 PM UTC

   Canvas agent cannot load honeycomb internal tools.

2. resolved Jun 03, 2026, 08:51 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 21, 2026, 11:22 PM UTC· resolved May 22, 2026, 04:42 PM UTC

1. investigating May 21, 2026, 11:22 PM UTC

   We have identified and are working to resolve an issue that is causing query results to return inconsistent results.

2. investigating May 22, 2026, 12:49 AM UTC

   Impact from this issue has concluded. For a period of time (different between US and EU instances, noted below), queries spanning data older than the most recent 2 hours with certain GROUP / WHERE clauses returned inconsistent results. Data was never lost, and reruns of the same queries will show the previously missing data. US impact times: 21:50 UTC - 23:20 UTC EU impact times: 20:50 UTC - 23:20 UTC

3. resolved May 22, 2026, 04:42 PM UTC

   This is an administrative update, marking the incident as Resolved. There has been no further impact since 23:20UTC on May 21 2026.

Read the full incident report →

Started May 08, 2026, 08:00 AM UTC· resolved May 11, 2026, 04:37 PM UTC

1. identified May 08, 2026, 08:43 AM UTC

   Activity log delay is rising, and is more than an hour behind due to a MySQL replica failover.

2. monitoring May 08, 2026, 10:13 AM UTC

   Backfilling of the past 2h30m of data is now in progress and should complete shortly.

3. monitoring May 08, 2026, 10:21 AM UTC

   We believe that the activity log should now be caught up.

4. monitoring May 08, 2026, 10:22 AM UTC

   We are continuing to monitor for any further issues.

5. monitoring May 08, 2026, 10:40 AM UTC

   All activity log streams are now caught up, except for the query runs table, which should have all data since the start of the incident backfilled by 1600 PDT.

6. monitoring May 08, 2026, 08:46 PM UTC

   Activity Log recovered at 07:00 UTC-07:00 today (about 6.5h ago). All streams are caught up.

7. resolved May 11, 2026, 04:37 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 01, 2026, 12:45 PM UTC· resolved May 01, 2026, 07:45 PM UTC

1. resolved May 01, 2026, 07:51 PM UTC

   The database failover that caused query issues earlier in the EU (https://status.honeycomb.io/incidents/n855d8kzp32y) has also had knock-on effects on our Activity Log beta feature. Due to recovery issues on the data replication flow used for this mechanism, activity log events will be missing from 12:45 UTC until 19:45 UTC, representing a 5 hour gap in events. We have identified configuration parameters that will be adjusted to reduce the likelihood of such losses in the future.

Read the full incident report →

Started May 01, 2026, 12:30 PM UTC· resolved May 01, 2026, 12:40 PM UTC

1. resolved May 01, 2026, 01:22 PM UTC

   At 12:50, our main database underwent an automated failover. This failover led to issues with some of our query engine connection pools, and caused queries to fail for roughly 10 minutes before self resolving.

Read the full incident report →

Started Apr 30, 2026, 03:03 PM UTC· resolved Apr 30, 2026, 04:37 PM UTC

1. identified Apr 30, 2026, 03:03 PM UTC

   We discovered that links in Slack triggers are not working and have identified the cause and are in process of implementing a fix

2. monitoring Apr 30, 2026, 03:07 PM UTC

   The fix has been implemented and links on Slack triggers are functional again.

3. resolved Apr 30, 2026, 04:37 PM UTC

   We are confident the issue is fully resolved.

Read the full incident report →

Started Apr 20, 2026, 04:56 PM UTC· resolved Apr 20, 2026, 05:02 PM UTC

1. identified Apr 20, 2026, 04:56 PM UTC

   Activity Log events are interrupted, starting from around 15:51UTC. Customers may experience some data lost since the beginning of the incident.

2. resolved Apr 20, 2026, 05:02 PM UTC

   Activity Log is back to operational, and most events should be recovered.

Read the full incident report →

Started Apr 15, 2026, 07:04 PM UTC· resolved Apr 15, 2026, 07:25 PM UTC

1. identified Apr 15, 2026, 07:04 PM UTC

   Activity Log is experiencing a delay in processing events. We are working on remediating the issue.

2. resolved Apr 15, 2026, 07:25 PM UTC

   Activity Log is ingesting events again. We had no loss in data.

Read the full incident report →