DFDS IT incident

metrics: kafka-exporter missing data frequently

Minor Resolved View vendor source →

DFDS IT experienced a minor incident on April 8, 2025 affecting Kafka metrics exporter, lasting 1d 23h. The incident has been resolved; the full update timeline is below.

Started
Apr 08, 2025, 07:22 AM UTC
Resolved
Apr 10, 2025, 07:06 AM UTC
Duration
1d 23h
Detected by Pingoru
Apr 08, 2025, 07:22 AM UTC

Affected components

Kafka metrics exporter

Update timeline

  1. investigating Apr 08, 2025, 07:22 AM UTC

    Same as the incident a few weeks ago: https://dfdsit.statuspage.io/incidents/vllndmlym8xd Looking into it. Hopefully a more permanent solution this time. If you have alerts that are currently triggering because of the lack of data from this exporter, a temporary measure could be to set "Alert state if no data or all values are null" to "Keep Last State" for those alerts. It's not ideal, but it's much better than muting/silencing the alert itself. See screenshot: https://blob.juna.seq.tf/pub/vrsWnTTVXk37Srs.jpg

  2. monitoring Apr 09, 2025, 04:04 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved Apr 10, 2025, 07:06 AM UTC

    Since fix was implemented not a single scrape has been missed. Will continue to monitor for the next few days If it becomes problematic again, a different exporter is currently being evaluated to replace the current one.