Remember The Milk incident

Web/desktop and sync are unavailable

Resolved after 1 hour 27 minutes, affecting Web Interface and Push Syncing — Android and 1 more, on February 1, 2024.

postmortem Feb 01, 2024, 02:24 PM UTC

An unexpected failure mode in our database, coupled with a series of unfortunate events, meant that Remember The Milk was in a \(mostly\) read-only mode for roughly 4 hours beginning 2024-02-01 07:49 UTC. Due to the nature of the failure—where the website and various endpoints were up, reads were working, and writes would fail intermittently—our monitoring systems \(5 independent external and internal systems\) didn't alert us to the issue and continued to report that everything was fine, which led to a prolonged downtime. We apologize for the downtime and have taken measures to ensure this scenario won't happen again.

Started
Feb 01, 2024, 12:54 PM UTC
Resolved
Feb 01, 2024, 02:21 PM UTC
Duration
1h 27m
Detected by Pingoru
Feb 01, 2024, 12:54 PM UTC

Affected components

Web InterfacePush Syncing — AndroidPush Syncing — iOS

Update timeline

  1. investigating Feb 01, 2024, 12:54 PM UTC

    We are looking into an issue with the web/desktop app and syncing for the mobile apps. We will update with more details, and are sorry for the inconvenience.

  2. resolved Feb 01, 2024, 02:21 PM UTC

    A new failure mode in our database, coupled with a series of unfortunate events, meant that Remember The Milk was in a (mostly) read-only mode for roughly 4 hours beginning 2024-02-01 07:49 UTC. Due to the nature of the failure—where the website and various endpoints were up, reads were working, and writes would fail intermittently—our monitoring systems (5 independent external and internal systems) didn't alert us to the issue and continued to report that everything was fine, which led to a prolonged downtime. We apologize for the downtime and have taken measures to ensure this scenario won't happen again. Please contact our team if you continue to experience any issues.

  3. postmortem Feb 01, 2024, 02:24 PM UTC

    An unexpected failure mode in our database, coupled with a series of unfortunate events, meant that Remember The Milk was in a \(mostly\) read-only mode for roughly 4 hours beginning 2024-02-01 07:49 UTC. Due to the nature of the failure—where the website and various endpoints were up, reads were working, and writes would fail intermittently—our monitoring systems \(5 independent external and internal systems\) didn't alert us to the issue and continued to report that everything was fine, which led to a prolonged downtime. We apologize for the downtime and have taken measures to ensure this scenario won't happen again.

Your product runs on more than Remember The Milk

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →