Recorded Future experienced a notice incident on January 27, 2025 affecting Alerts, lasting 12h 22m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jan 27, 2025, 02:12 AM UTC
Dear Customer, A subset of our alerts are experiencing a delivery delay. This affects both email alerts and in-app notifications. The affected alerts are currently delayed by up to 2 hours, and the backlog is already decreasing based on our response. Please note that other systems including Recorded Future's web portal, API, collection, and analysis are unaffected. We will follow up when alert delivery returns to normal. Please contact our support team at [email protected] if you have any questions or concerns. Regards, Recorded Future Platform Operations
- resolved Jan 27, 2025, 02:35 PM UTC
Dear Customer, The issue affecting our alert delivery has been resolved, and the domain abuse screenshots have now resumed normal processing. Please contact our support team at [email protected] if you have any questions or concerns.
- postmortem Feb 16, 2025, 07:15 PM UTC
**Issue:** On January 26, 2025 at approximately 9:26 PM EST, our monitoring system had alerted our operations team to a service failure that impacted the system which feeds our alert delivery system. As a result of this incident, Brand Intelligence clients saw delays in delivery of Domain Abuse alerts, including screenshots taken pertaining to the alert. **Cause:** The failure was triggered by an unusually high volume of domain abuse reports, which amounted to over 40x the normal load, for 5 consecutive hours. **Resolution:** To manage that level of increased volume and ensure all information was processed, our operations team engineered the system to slow down the handling of alerts to ensure all messages were processed and alerts sent. After a few hours the system returned to normal, but we formally closed the incident on January 27th, at approximately 9:35 am EST. **Path Forward:** As a result of this incident, we are actively reviewing the autoscaling capacities of our alerting system to better support future spikes of domain abuse reports .