DFDS IT incident

Prometheus unstable due to high memory consumption

Major Resolved View vendor source →

DFDS IT experienced a major incident on May 3, 2022, lasting 4h 53m. The incident has been resolved; the full update timeline is below.

Started
May 03, 2022, 09:16 AM UTC
Resolved
May 03, 2022, 02:10 PM UTC
Duration
4h 53m
Detected by Pingoru
May 03, 2022, 09:16 AM UTC

Update timeline

  1. investigating May 03, 2022, 09:16 AM UTC

    Prometheus currently has a very high memory consumption after the node roll over and has already taken down a couple of nodes, which has already been replaced, but while detaching a node the Prometheus persistant volume did detach correctly, but this has also now been fixed. We are investigating to figure out more.

  2. resolved May 03, 2022, 02:10 PM UTC

    This incident has been resolved.