Discord Extreme List incident

Cannot Connect to MongoDB Replicaset

Resolved after 3 hours 59 minutes, affecting Website and API and 1 more, on August 25, 2020.

postmortem Aug 25, 2020, 04:58 PM UTC

When in production, it might be a wise idea to prevent pods \(that can use hostPath volumes\)! I applied some pod security policies to help solve this problem. As many know, eu2 had an unknown issue, which caused it to not reboot, but become un-schedulable, resulting in the policies not applying. The way I fixed this, was to get rid of pod security policies, the resources will still be on the servers, but when time comes to use them again, I will.

Started
Aug 25, 2020, 12:58 PM UTC
Resolved
Aug 25, 2020, 04:58 PM UTC
Duration
3h 59m
Detected by Pingoru
Aug 25, 2020, 12:58 PM UTC

Affected components

WebsiteAPIMongoDB Replicaset

Update timeline

  1. investigating Aug 25, 2020, 12:58 PM UTC

    We are currently investigating this issue.

  2. investigating Aug 25, 2020, 01:08 PM UTC

    We are continuing to investigate this issue.

  3. resolved Aug 25, 2020, 04:58 PM UTC

    This incident has been resolved.

  4. postmortem Aug 25, 2020, 04:58 PM UTC

    When in production, it might be a wise idea to prevent pods \(that can use hostPath volumes\)! I applied some pod security policies to help solve this problem. As many know, eu2 had an unknown issue, which caused it to not reboot, but become un-schedulable, resulting in the policies not applying. The way I fixed this, was to get rid of pod security policies, the resources will still be on the servers, but when time comes to use them again, I will.

Your product runs on more than Discord Extreme List

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →