Red Canary incident
Alert Ingestion Delays Causing Failing Status Checks
Red Canary experienced a minor incident on February 10, 2025 affecting SentinelOne and Microsoft and 1 more component, lasting 3h 2m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Feb 10, 2025, 08:00 PM UTC
We have identified an issue causing some alerts to fail ingestion, which may result in failing status checks for certain customers. Our team is actively testing a fix and assessing its impact. No alert data has been lost and all failed alerts will be reprocessed once the fix is implemented. Please note that detections based on these alerts may be delayed until ingestion is fully restored. We appreciate your patience and will provide further updates as they become available.
- monitoring Feb 10, 2025, 08:41 PM UTC
We have successfully deployed a fix for the alert ingestion issue and alerts are now being ingested. Detections may still be delayed while we process the backlog of failed alerts. In addition, status checks that appeared to fail during this incident will automatically resolve; no customer action is required. We will continue to monitor the situation and will close this incident once all backlogged alerts have been ingested.
- monitoring Feb 10, 2025, 08:42 PM UTC
We have successfully deployed a fix for the alert ingestion issue and alerts are now being ingested. Detections may still be delayed while we process the backlog of failed alerts. In addition, status checks that appeared to fail during this incident will automatically resolve; no customer action is required. We will continue to monitor the situation and will close this incident once all backlogged alerts have been ingested.
- resolved Feb 10, 2025, 11:02 PM UTC
This incident has been resolved and all backlogged alerts processed.