DFDS IT incident

Prometheus is crash looping

Critical Resolved View vendor source →

DFDS IT experienced a critical incident on March 14, 2023, lasting 1h 20m. The incident has been resolved; the full update timeline is below.

Started
Mar 14, 2023, 11:53 AM UTC
Resolved
Mar 14, 2023, 01:14 PM UTC
Duration
1h 20m
Detected by Pingoru
Mar 14, 2023, 11:53 AM UTC

Update timeline

  1. investigating Mar 14, 2023, 11:53 AM UTC

    We are currently investigating this issue.

  2. monitoring Mar 14, 2023, 12:04 PM UTC

    After restarting Promtheus, it spent about 20 minutes to become ready.

  3. identified Mar 14, 2023, 12:15 PM UTC

    OOMKilled. We will raise the memory allocation for Prometheus.

  4. resolved Mar 14, 2023, 01:14 PM UTC

    3Gi more memory added, and Prometheus placed on dedicated node.