DFDS IT incident

One fluentd pod is frequently terminated due to OOMKilled

Major Resolved View vendor source →

DFDS IT experienced a major incident on June 6, 2023, lasting 17h 18m. The incident has been resolved; the full update timeline is below.

Started
Jun 06, 2023, 03:34 PM UTC
Resolved
Jun 07, 2023, 08:52 AM UTC
Duration
17h 18m
Detected by Pingoru
Jun 06, 2023, 03:34 PM UTC

Update timeline

  1. investigating Jun 06, 2023, 03:34 PM UTC

    Workloads on node ip-10-0-125-129.eu-west-1.compute.internal are affected, and will loose logs when fluentd crashes. Setting the memory limit 30% higher has not solved the issue.

  2. monitoring Jun 07, 2023, 07:06 AM UTC

    Setting the memory limit 260% higher has solved the issue. It seems to be only one problematic workload that was causing the ad hoc high memory requirements.

  3. resolved Jun 07, 2023, 08:52 AM UTC

    This incident has been resolved.