Servebolt incident

Elinor-osl sent old emails after a stopped queue had gone unnoticed

Minor Resolved View vendor source →

Servebolt experienced a minor incident on January 6, 2023, lasting —. The incident has been resolved; the full update timeline is below.

Started
Jan 06, 2023, 08:05 PM UTC
Resolved
Jan 06, 2023, 08:05 PM UTC
Duration
Detected by Pingoru
Jan 06, 2023, 08:05 PM UTC

Update timeline

  1. resolved Jan 06, 2023, 08:52 PM UTC

    At 20:34 a stale email queue with cronlog output was restarted when routine maintenance was conducted. The queue wasn't correctly configured in our monitoring systems and had therefore gone unnoticed. Upon restarting it a batch of emails was sent erroneously before the queue was stopped. The problem has been identified and fixed, we apologize for the confusion caused by this. We're implementing further checks and monitoring rules to prevent anything like this from occurring again.