Honeycomb.io incident

Data Ingestion lag

Notice Resolved View vendor source →

Honeycomb.io experienced a notice incident on November 15, 2024 affecting ui.honeycomb.io - US1 Querying and ui.honeycomb.io - US1 Trigger & SLO Alerting, lasting 31m. The incident has been resolved; the full update timeline is below.

Started
Nov 15, 2024, 02:49 PM UTC
Resolved
Nov 15, 2024, 03:20 PM UTC
Duration
31m
Detected by Pingoru
Nov 15, 2024, 02:49 PM UTC

Affected components

ui.honeycomb.io - US1 Queryingui.honeycomb.io - US1 Trigger & SLO Alerting

Update timeline

  1. identified Nov 15, 2024, 02:49 PM UTC

    A significant portion of our partitions are currently lagging behind on ingestion. The data is going to be available, but with a delay. Queries may not return recent data just yet, and triggers that run on short cycles are likely to not see the data they are monitoring. SLO alerts may be delayed but will account for the data when they catch up.

  2. monitoring Nov 15, 2024, 03:06 PM UTC

    Our storage engine has caught up with ingest. All public impact should be resolved by now, but we're keeping an eye on the system load while everything internal stabilizes.

  3. resolved Nov 15, 2024, 03:20 PM UTC

    All internal systems are healthy, as well as all public-facing elements.