DFDS IT incident

All nodes are down

Critical Resolved View vendor source →

DFDS IT experienced a critical incident on February 14, 2023 affecting Kubernetes [Hellman] - Capacity/Scheduling and Kubernetes [Hellman] - Ingress/Networking, lasting 3h 6m. The incident has been resolved; the full update timeline is below.

Started
Feb 14, 2023, 10:56 AM UTC
Resolved
Feb 14, 2023, 02:02 PM UTC
Duration
3h 6m
Detected by Pingoru
Feb 14, 2023, 10:56 AM UTC

Affected components

Kubernetes [Hellman] - Capacity/SchedulingKubernetes [Hellman] - Ingress/Networking

Update timeline

  1. investigating Feb 14, 2023, 10:56 AM UTC

    We are currently investigating this issue.

  2. monitoring Feb 14, 2023, 11:02 AM UTC

    A fix has been implemented and we are monitoring the results.

  3. monitoring Feb 14, 2023, 01:22 PM UTC

    The new nodes got lost during a node rollover, so a controlled node rollover will be performed now to verify that the problem is fixed.

  4. monitoring Feb 14, 2023, 02:02 PM UTC

    We are continuing to monitor for any further issues.

  5. resolved Feb 14, 2023, 02:02 PM UTC

    This incident has been resolved.