Scalyr incident

Emails not being sent for all users

Minor Resolved View vendor source →

Scalyr experienced a minor incident on January 24, 2023 affecting Main Site, lasting 1h 50m. The incident has been resolved; the full update timeline is below.

Started
Jan 24, 2023, 03:48 PM UTC
Resolved
Jan 24, 2023, 05:38 PM UTC
Duration
1h 50m
Detected by Pingoru
Jan 24, 2023, 03:48 PM UTC

Affected components

Main Site

Update timeline

  1. investigating Jan 24, 2023, 02:34 PM UTC

    Since approximately 12:20 pm UTC, DataSet is currently not sending emails, including alerts and account invitations. These emails are in queue and we expect they will be sent once the issue is resolved

  2. identified Jan 24, 2023, 03:01 PM UTC

    New notifications are now being sent via email. We are investigating whether it will be possible to send out the messages which were queued during the incident.

  3. monitoring Jan 24, 2023, 03:48 PM UTC

    Email notifications have been restored and we're now processing all new email notifications.

  4. resolved Jan 24, 2023, 05:38 PM UTC

    The issue has been resolved. Email notifications generated between 07:19:00 and 09:58:00 EST were dropped and we will not be able to re-send the notifications.