DFDS IT incident

metrics: kafka-exporter missing data often

Minor Resolved View vendor source →

DFDS IT experienced a minor incident on March 14, 2025 affecting Kafka metrics exporter, lasting 2h 47m. The incident has been resolved; the full update timeline is below.

Started
Mar 14, 2025, 07:53 AM UTC
Resolved
Mar 14, 2025, 10:40 AM UTC
Duration
2h 47m
Detected by Pingoru
Mar 14, 2025, 07:53 AM UTC

Affected components

Kafka metrics exporter

Update timeline

  1. investigating Mar 14, 2025, 07:53 AM UTC

    Currently looking into metrics from kafka-metrics having no data frequently. Currently the most probable cause is a big increase in consumer groups per topic for a few topics leading to much longer scrape times, resulting in a timeout.

  2. monitoring Mar 14, 2025, 09:49 AM UTC

    Issue seem potentially related to slower response times from Confluent Cloud. This seems to be improving now and the exporter can seemingly reliably export metrics within the set scrape threshold. Will continue to monitor the situation.

  3. resolved Mar 14, 2025, 10:40 AM UTC

    Seems stable... for now.