Expel incident

Delayed Notifications

Minor Resolved View vendor source →

Expel experienced a minor incident on October 24, 2024 affecting Email and Slack and 1 more component, lasting 3h 49m. The incident has been resolved; the full update timeline is below.

Started
Oct 24, 2024, 11:15 PM UTC
Resolved
Oct 25, 2024, 03:05 AM UTC
Duration
3h 49m
Detected by Pingoru
Oct 24, 2024, 11:15 PM UTC

Affected components

EmailSlackTeamsPagerDutyOpsgenie

Update timeline

  1. investigating Oct 24, 2024, 11:15 PM UTC

    We're experiencing a delay in delivering some notifications. We are actively investigating the issue and will provide another update by 7:45pm EST.

  2. identified Oct 24, 2024, 11:34 PM UTC

    We have identified the issue and are working with our upstream vendor to address it. We are still experiencing degraded notifications performance. An update will posted no later than 8:30 PM EDT.

  3. identified Oct 25, 2024, 12:07 AM UTC

    Our upstream vender has identified the cause of the issue and they are now rolling out a fix. An update will be posted no later than 8:45PM EDT.

  4. monitoring Oct 25, 2024, 12:38 AM UTC

    Our upstream vendor is still working on rolling out the fix for the delayed messages while we continue to monitor. An update will be posted no later than 9:45PM EDT.

  5. monitoring Oct 25, 2024, 01:36 AM UTC

    Our upstream vendor is rolling out an additional fix for the delayed messages while we continue to monitor the issue. An update will be posted no later than 10:20PM EDT.

  6. monitoring Oct 25, 2024, 02:17 AM UTC

    We are continuing to monitor this issue while the fix takes effect. An update will be posted no later than 11:20 PM EDT.

  7. resolved Oct 25, 2024, 03:05 AM UTC

    This issue has been resolved. All notifications in the backlog have been delivered and new notifications are being delivered as normal.