Expel incident

Alert ingestion delayed for a subset of security devices

Major Resolved View vendor source →

Expel experienced a major incident on April 20, 2026 affecting Alert ingestion, lasting 4h 33m. The incident has been resolved; the full update timeline is below.

Started
Apr 20, 2026, 04:20 PM UTC
Resolved
Apr 20, 2026, 08:54 PM UTC
Duration
4h 33m
Detected by Pingoru
Apr 20, 2026, 04:20 PM UTC

Affected components

Alert ingestion

Update timeline

  1. investigating Apr 20, 2026, 04:20 PM UTC

    At 10:59AM EDT, we noticed that a set of devices have been unhealthy since last week due to invalid credentials. We believe the cause to be a change to our device credentials storage subsystem. We are working to identify the specific set of devices and restore the credentials. Once restored, all delayed alerts will be ingested. We will provide an updated by 1pm EDT.

  2. investigating Apr 20, 2026, 05:00 PM UTC

    We have identified the specific subset of devices that are affected and are working on restoring the credentials for those devices. Once restored, all alerts that have not been ingested for these devices will be ingested. We will provide another update by 1:30pm EDT.

  3. investigating Apr 20, 2026, 05:35 PM UTC

    We have identified the specific subset of devices that are affected and are working on restoring the credentials for those devices. Once restored, all alerts that have not been ingested for these devices will be ingested. We will provide another update by 2:00pm EDT.

  4. investigating Apr 20, 2026, 06:01 PM UTC

    We have identified the specific subset of devices that are affected and are working on restoring the credentials for those devices. Once restored, all alerts that have not been ingested for these devices will be ingested. We will provide another update by 2:30pm EDT.

  5. identified Apr 20, 2026, 06:34 PM UTC

    We have restored the credentials for the affected subset of devices. We are monitoring closely to ensure proper ingestion of delayed alerts. We will provide another update by 3:00pm EDT.

  6. identified Apr 20, 2026, 07:02 PM UTC

    We have restored the credentials for the subset of devices. We are monitoring closely to ensure proper ingestion of delayed alerts. We will provide another update by 3:30pm EDT.

  7. monitoring Apr 20, 2026, 07:34 PM UTC

    We restored the credentials for the subset of devices, and are seeing that the majority of devices have returned to a healthy state. We are investigating the remaining unhealthy devices, which do not seem related to the credentials. We will provide another update by 4:00pm EDT.

  8. monitoring Apr 20, 2026, 08:01 PM UTC

    We restored the credentials for the subset of devices, and are seeing that the majority of devices have returned to a healthy state. We are investigating the remaining unhealthy devices, which do not seem related to credentials. We will provide another update by 4:30pm EDT.

  9. resolved Apr 20, 2026, 08:54 PM UTC

    We restored the credentials for the subset of devices, and are seeing that the majority of devices have returned to a healthy state. We investigated the remaining unhealthy devices, which are not related to the customer secrets vault upgrade. This incident is resolved.