DFDS IT experienced a minor incident on March 3, 2025 affecting Kubernetes [Hellman] - Capacity/Scheduling, lasting 2d 19h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 03, 2025, 01:32 PM UTC
We are currently investigating an issue that is causing the odd Kubernetes worker node to become unresponsive and evict its workloads to other nodes
- monitoring Mar 03, 2025, 06:49 PM UTC
We have implemented a potential fix and are monitirong the results
- resolved Mar 06, 2025, 08:37 AM UTC
Nodes have been stable since the fix was applied so we consider the issue resolved
- postmortem Mar 18, 2025, 08:54 AM UTC
[https://raw.githubusercontent.com/dfds/postmortems/refs/heads/master/PM2025-001 - Node system resource starvation and scheduling capacity](https://raw.githubusercontent.com/dfds/postmortems/refs/heads/master/PM2025-001%20-%20Node%20system%20resource%20starvation%20and%20scheduling%20capacity)