Alert ingestion delayed for a subset of security devices

Update timeline

1. investigating Apr 20, 2026, 04:20 PM UTC

   At 10:59AM EDT, we noticed that a set of devices have been unhealthy since last week due to invalid credentials. We believe the cause to be a change to our device credentials storage subsystem. We are working to identify the specific set of devices and restore the credentials. Once restored, all delayed alerts will be ingested. We will provide an updated by 1pm EDT.

2. investigating Apr 20, 2026, 05:00 PM UTC

   We have identified the specific subset of devices that are affected and are working on restoring the credentials for those devices. Once restored, all alerts that have not been ingested for these devices will be ingested. We will provide another update by 1:30pm EDT.

3. investigating Apr 20, 2026, 05:35 PM UTC

   We have identified the specific subset of devices that are affected and are working on restoring the credentials for those devices. Once restored, all alerts that have not been ingested for these devices will be ingested. We will provide another update by 2:00pm EDT.

4. investigating Apr 20, 2026, 06:01 PM UTC

   We have identified the specific subset of devices that are affected and are working on restoring the credentials for those devices. Once restored, all alerts that have not been ingested for these devices will be ingested. We will provide another update by 2:30pm EDT.

5. identified Apr 20, 2026, 06:34 PM UTC

   We have restored the credentials for the affected subset of devices. We are monitoring closely to ensure proper ingestion of delayed alerts. We will provide another update by 3:00pm EDT.

6. identified Apr 20, 2026, 07:02 PM UTC

   We have restored the credentials for the subset of devices. We are monitoring closely to ensure proper ingestion of delayed alerts. We will provide another update by 3:30pm EDT.

7. monitoring Apr 20, 2026, 07:34 PM UTC

   We restored the credentials for the subset of devices, and are seeing that the majority of devices have returned to a healthy state. We are investigating the remaining unhealthy devices, which do not seem related to the credentials. We will provide another update by 4:00pm EDT.

8. monitoring Apr 20, 2026, 08:01 PM UTC

   We restored the credentials for the subset of devices, and are seeing that the majority of devices have returned to a healthy state. We are investigating the remaining unhealthy devices, which do not seem related to credentials. We will provide another update by 4:30pm EDT.

9. resolved Apr 20, 2026, 08:54 PM UTC

   We restored the credentials for the subset of devices, and are seeing that the majority of devices have returned to a healthy state. We investigated the remaining unhealthy devices, which are not related to the customer secrets vault upgrade. This incident is resolved.