DFDS IT incident

One Kubernetes node is crashed. Multiple pods affected

Major Resolved View vendor source →

DFDS IT experienced a major incident on March 13, 2024 affecting Kubernetes [Hellman] - Capacity/Scheduling, lasting 17h 54m. The incident has been resolved; the full update timeline is below.

Started
Mar 13, 2024, 03:26 PM UTC
Resolved
Mar 14, 2024, 09:20 AM UTC
Duration
17h 54m
Detected by Pingoru
Mar 13, 2024, 03:26 PM UTC

Affected components

Kubernetes [Hellman] - Capacity/Scheduling

Update timeline

  1. identified Mar 13, 2024, 03:26 PM UTC

    Node running out of memory due to high increase of memory of grafana-agent. Also a rogue hangfire process in a dev container seems to contribute to this. A new node has been spawned. The rogue container has been scaled down in order to establish if they both are the root cause of the problem.

  2. monitoring Mar 13, 2024, 04:40 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved Mar 14, 2024, 09:20 AM UTC

    This incident has been resolved.

  4. postmortem Mar 18, 2024, 08:52 AM UTC

    Postmortem published at: [https://github.com/dfds/postmortems/blob/master/PM2024-001 - Two Kubernetes nodes went into NotReady state.md](https://github.com/dfds/postmortems/blob/master/PM2024-001%20-%20Two%20Kubernetes%20nodes%20went%20into%20NotReady%20state.md)