Recorded Future Outage History

Started Sep 29, 2026, 07:23 PM UTC· resolved Sep 29, 2026, 07:26 PM UTC

1. investigating Sep 29, 2026, 07:23 PM UTC

   Dear Customer, We are currently experiencing a service disruption with the Recorded Future application portal. Our Platform Operations team are actively working on restoring accessibility to an optimal level of service. We will continue to provide updates as they become available. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

2. resolved Sep 29, 2026, 07:26 PM UTC

   Dear Customer, We are pleased to inform you that the service disruption affecting the Recorded Future application portal has been resolved. All systems are now operating normally. If you have any remaining questions or continue to experience issues, please contact our support team at [email protected]. Regards, Recorded Future Platform Operations

Read the full incident report →

Started Aug 21, 2026, 06:15 AM UTC· resolved Aug 21, 2026, 07:28 AM UTC

1. investigating Aug 21, 2026, 06:15 AM UTC

   Dear customers, We are currently investigating an issue affecting access to the Attack Surface Intelligence, which may result in errors for some users. Our team is actively working to resolve this as quickly as possible. We will post an update once normal performance is restored. In the meantime, if you have any questions or concerns, please reach out to our Support team at [email protected]. Thank you for your patience, Recorded Future Platform Operations

2. resolved Aug 21, 2026, 07:28 AM UTC

   Dear customers, The issue affecting access to the Attack Surface Intelligence has been resolved, and the service is now operating normally. We apologize for any inconvenience this may have caused. If you continue to experience any issues, please contact our support team at [email protected]. Thank you for your patience, Recorded Future Platform Operations

Read the full incident report →

Started Aug 01, 2026, 01:45 PM UTC· resolved Aug 01, 2026, 04:45 PM UTC

1. resolved Aug 02, 2026, 05:41 PM UTC

   Dear Customer, We recently experienced a service disruption regarding a series of Recorded Future Alerts. Our Platform Operations team recognized the disruption and worked quickly to resolve this incident. Between roughly 13:45-16:45 UTC on August 1st, a series of Alerts could not be generated. We are still investigating the initial cause. While services remain operational, a post-mortem will be updated once our investigation has completed. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

Read the full incident report →

Started Jul 30, 2026, 06:55 AM UTC· resolved Jul 30, 2026, 01:15 PM UTC

1. investigating Jul 30, 2026, 06:55 AM UTC

   Dear Customer, We are currently experiencing a service disruption with the Recorded Future application portal. Our Platform Operations team are actively working on restoring accessibility to an optimal level of service. We will continue to provide updates as they become available. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

2. monitoring Jul 30, 2026, 09:24 AM UTC

   Dear Customer, We have identified the root of instability and believe we have restored performance to optimal levels. We will continue to monitor to ensure services remain fully available and provide an update when the incident is fully resolved. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

3. resolved Jul 30, 2026, 01:15 PM UTC

   Dear Customer, We are pleased to inform you that the service disruption affecting the Recorded Future application portal has been resolved. All systems are now operating normally. If you have any remaining questions or continue to experience issues, please contact our support team at [email protected]. Regards, Recorded Future Platform Operations

4. postmortem Jul 31, 2026, 03:50 AM UTC

   **Issue:** Between 06:30 and 08:30 UTC on July 30, 2026, customers experienced intermittent errors and degraded performance in the Recorded Future Platform. **Cause:** An internal content-processing service became unresponsive. Requests dependent on that service queued and timed out, which affected responsiveness across the Platform more broadly. **Remediation:** The affected service was restored and performance returned to normal. We have increased capacity for this service and are implementing additional safeguards to prevent recurrence and improve detection. This was an availability incident only; no Customer Data was impacted.

Read the full incident report →

Started Jul 09, 2026, 07:34 AM UTC· resolved Jul 09, 2026, 08:01 AM UTC

1. investigating Jul 09, 2026, 07:34 AM UTC

   We are currently experiencing a service disruption with the Recorded Future application portal. Our Platform Operations team are actively working on restoring accessibility to an optimal level of service. We will continue to provide updates as they become available. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

2. investigating Jul 09, 2026, 07:35 AM UTC

   We are continuing to investigate this issue.

3. resolved Jul 09, 2026, 08:01 AM UTC

   Dear Customer, We are pleased to inform you that the service disruption affecting the Recorded Future application portal has been resolved. All systems are now operating normally. If you have any remaining questions or continue to experience issues, please contact our support team at [email protected]. Regards, Recorded Future Platform Operations

4. postmortem Aug 03, 2026, 04:39 AM UTC

   **Issue:** * Between 07:34 and 08:01 UTC on July 9, 2026 customers experienced intermittent ability to log into Recorded Future’s application interface. **Cause:** * A backend service contained a misconfiguration which caused service communication issues. **Remediation / Path Forward:** * The configuration was updated and the service was restarted. * Confirmation that no data or information was compromised during the disruption.

Read the full incident report →

Started May 08, 2026, 12:47 AM UTC· resolved May 08, 2026, 08:17 AM UTC

1. identified May 08, 2026, 12:47 AM UTC

   We are currently experiencing a service disruption with the Recorded Future application portal. Our product operations team have identified the cause are actively working on restoring accessibility to an optimal level of service. We will continue to provide updates as they become available. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

2. identified May 08, 2026, 01:11 AM UTC

   Dear Customer, We are currently experiencing a service disruption with the Recorded Future application portal due to an AWS outage. Our product operations team is aware and will restore accessibility to an optimal service level as soon as the related ASW issues are addressed. We will continue to provide updates as they become available. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

3. monitoring May 08, 2026, 01:56 AM UTC

   Dear Customer, Our product operations team has now implemented measures to restore the platform availability. We will continue to monitor this to ensure optimal operations. We thank you for your patience. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

4. resolved May 08, 2026, 08:17 AM UTC

   Dear Customer, We are pleased to inform you that the service disruption affecting the Recorded Future application portal has been resolved. All systems are now operating normally. If you have any remaining questions or continue to experience issues, please contact our support team at [email protected]. Regards, Recorded Future Platform Operations

5. postmortem Aug 03, 2026, 04:35 AM UTC

   **Issue:** * Between 00:00 and 01:56 UTC on May 8, 2026 customers experienced intermittent access and performance issues with the Recorded Future Platform. **Cause:** * Recorded Future’s cloud partner experienced a hardware issue in one of its availability zones, causing inconsistent performance across systems in that region. * Because the issue caused intermittent degradation rather than a total failure, our automated failover processes did not trigger, and manual failover processes were required **Remediation / Path Forward:** * We redirected traffic and workloads away from the affected availability zone and stabilized service at 01:56 UTC, monitoring until full resolution was confirmed. * We are further automating our failover process to reduce recovery time for similar events. * This was an availability incident only - no Customer Data was lost or compromised.

Read the full incident report →

Started Mar 29, 2026, 01:03 AM UTC· resolved Mar 29, 2026, 08:31 AM UTC

1. investigating Mar 29, 2026, 01:03 AM UTC

   Dear Customer, We are currently experiencing a service disruption with the Recorded Future application portal. Our product operations team are actively working on restoring accessibility to an optimal level of service. We will continue to provide updates as they become available. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

2. monitoring Mar 29, 2026, 01:21 AM UTC

   Dear Client, We have identified the root of instability and believe we have restored performance to optimal levels. We will continue to monitor to ensure services remain fully available and provide an update when the incident is fully resolved. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

3. resolved Mar 29, 2026, 08:31 AM UTC

   Dear Customer, We are pleased to inform you that the service disruption affecting the Recorded Future application portal has been resolved. All systems are now operating normally. If you have any remaining questions or continue to experience issues, please contact our support team at [email protected]. Regards, Recorded Future Platform Operations

Read the full incident report →

Started Mar 05, 2026, 08:30 PM UTC· resolved Mar 05, 2026, 10:47 PM UTC

1. investigating Mar 05, 2026, 08:30 PM UTC

   Dear Customer, We are currently experiencing a service disruption with the Recorded Future Attack Surface Intelligence. Our product operations team are actively working on restoring accessibility to an optimal level of service. We will continue to provide updates as they become available. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

2. investigating Mar 05, 2026, 09:58 PM UTC

   We are continuing to investigate this issue.

3. monitoring Mar 05, 2026, 10:10 PM UTC

   The application is performing as expected. We are continuing to monitor to ensure it remains performant.

4. resolved Mar 05, 2026, 10:47 PM UTC

   The incident has been resolved, and Attack Surface Module has returned to normal performance.

Read the full incident report →

Started Feb 26, 2026, 08:43 PM UTC· resolved Mar 25, 2026, 02:48 AM UTC

1. identified Feb 26, 2026, 08:43 PM UTC

   Dear Customer, Starting today, Recorded Future became aware of a disruption in some of its collection of commits to public repositories, with expected intermittent gaps to customers who rely on ingestion of the "Code Repository" source type within queries or alerts. We are actively working to resolve the collection issue, and are actively backfilling any missing data now. We expect to be fully caught up within the next few days, with relevant alerts being triggered retroactively. We will continue to provide updates as they become available, and when the backfill is fully caught up. Please contact our support team at [email protected], or your Technical Account Manager if you have any questions. Regards, Recorded Future Platform Operations

2. identified Mar 03, 2026, 04:25 PM UTC

   Dear Customer, We wanted to provide an update regarding the disruption and recovery to the code repository collection data. We are still actively working to resolve the collection issue, and expect to be fully caught up within the next 24 hours, with relevant alerts being triggered retroactively. We will continue to provide updates as they become available, and when the backfill is fully caught up. Please contact our support team at [email protected], or your Technical Account Manager if you have any questions. Regards, Recorded Future Platform Operations

3. monitoring Mar 09, 2026, 08:18 PM UTC

   Dear Customer, We wanted to provide an update as to the disruption and recovery to the code repository collection data, as we are still actively working to resolve the collection issue. While conditions have improved dramatically, we are still in a degraded performance state with data coming in slightly delayed. We will continue to provide updates as they become available, and when the backfill is fully caught up. Please contact our support team at [email protected], or your Technical Account Manager if you have any questions. Regards, Recorded Future Platform Operations

4. resolved Mar 25, 2026, 02:48 AM UTC

   Dear Customer, Collection and processing of content is now near current, and systems that process new content within Recorded Future are returning to normal operations. We apologize for the delays experienced earlier today. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

Read the full incident report →

Started Jan 21, 2026, 08:55 AM UTC· resolved Jan 21, 2026, 11:10 AM UTC

1. identified Jan 21, 2026, 08:55 AM UTC

   Dear Customer, We are currently addressing a service disruption with our email delivery provider, SendGrid, which may result in delayed or intermittent delivery of automated email alerts. Our engineering team is working urgently with the provider to restore full functionality. Please note that all alerts remain fully accessible via the Recorded Future Portal. This disruption is limited to email delivery only; your real-time data and monitoring within the platform are unaffected. We will notify you as soon as email services are fully restored. Please contact our support team at [email protected] if you have any questions. Thank you for your patience. Regards, Recorded Future Platform Operations

2. resolved Jan 21, 2026, 11:10 AM UTC

   This incident has been resolved.

3. postmortem Jan 23, 2026, 07:54 PM UTC

   * **Issue:** * Between 08:55 and 11:10 UTC on January 21, 2026, the Recorded Future portal experienced a service disruption affecting outbound email delivery. * **Cause:** * Our outbound mail partner temporarily disabled mail sending from our domain which was promptly resolved. * **Remediation / Path Forward:** * We have worked extensively with our mail partner to ensure that such disabling actions are not taken in the future and we are also investigating adding backup email sending partners to provide additional resilience.

Read the full incident report →

Started Nov 12, 2025, 01:01 AM UTC· resolved Nov 12, 2025, 03:36 AM UTC

1. investigating Nov 12, 2025, 01:01 AM UTC

   Dear Customer, We are currently experiencing a service disruption with the Recorded Future AI. Our product operations team are actively working on restoring accessibility to an optimal level of service. We will continue to provide updates as they become available. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

2. monitoring Nov 12, 2025, 01:53 AM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring Nov 12, 2025, 01:53 AM UTC

   We are continuing to monitor for any further issues.

4. resolved Nov 12, 2025, 03:36 AM UTC

   This incident has been resolved.

5. postmortem Nov 28, 2025, 07:37 PM UTC

   * **Issue:** * Between 20:25 UTC on November 11, 2025 and 01:53 UTC on November 12, 2025, Recorded Future’s AI service experienced slow performance and intermittent outages, affecting the timely delivery and processing of intelligence reports. * **Cause:** * The disruption was caused by a system configuration which did not allow sufficient automatic scaling. * **Remediation / Path Forward:** * Immediate action was taken to update this configuration and we are implementing a comprehensive plan for long-term resilience.

Read the full incident report →

Started Nov 05, 2025, 11:40 AM UTC· resolved Nov 05, 2025, 12:50 PM UTC

1. investigating Nov 05, 2025, 11:40 AM UTC

   Dear Customer, We are currently experiencing a service disruption with the Recorded Future application portal & Mobile Application. Our product operations team are actively working on restoring accessibility to an optimal level of service. We will continue to provide updates as they become available. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

2. investigating Nov 05, 2025, 11:56 AM UTC

   The application portal, API, and Mobile applications are functional, performance is improving but is not yet returned to full speed. We will continue to provide updates as they become available. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

3. resolved Nov 05, 2025, 12:50 PM UTC

   The incident has been resolved, and the User Interface has returned to normal performance. The root cause has been identified, and corrective actions are being implemented.

4. postmortem Dec 03, 2025, 12:57 PM UTC

   * **Issue:** * Between 11:40 UTC and 12:50 UTC on November 5, 2025, Recorded Future Application Portal and Mobile Application experienced performance degradation. From 11:30 - 11:57 UTC, some users experienced access issues. * **Cause:** * The issue was traced to a malfunctioning service that caused an overload of the user interface. * **Remediation / Path Forward:** * The underlying issues has been resolved and the service has been scaled appropriately to prevent recurrence.

Read the full incident report →

Started Oct 27, 2025, 09:56 PM UTC· resolved Oct 27, 2025, 10:08 PM UTC

1. resolved Oct 26, 2025, 10:00 PM UTC

   Dear Customer, Collection and analysis of content is now current and systems that process new content in Recorded Future are operating normally. We apologize for the delays earlier. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

2. postmortem Nov 12, 2025, 02:04 AM UTC

   **Issue:** * Between 21:56 and 22:08 UTC on October 27, 2025, Recorded Future's data processing system experienced a maximum delay of 72 minutes. This delay directly impacted the timeliness of data availability for users. **Cause:** * The root cause of the data processing delay was a loss of synchronization within the message queue cluster, triggered by a network interruption. The unsynchronized state led the message queue cluster to become unresponsive, halting the normal processing of data, resulting in the eventual 72-minute data delay. **Remediation / Path Forward:** * Manually restored cluster integrity. This action successfully restored the cluster integrity and brought the data processing back online.

Read the full incident report →

Started Oct 20, 2025, 07:20 PM UTC· resolved Oct 20, 2025, 08:19 PM UTC

1. investigating Oct 20, 2025, 07:20 PM UTC

   Dear Customer, We are currently experiencing a delay in our content collection and analysis processes exceeding 1 hour in duration as a result of a service disruption with our hosting provider, AWS. This means that there is a delay between when content is published and when it is available in Recorded Future’s web application, API, and alerts. Other systems are running normally, historical data is still available, and we do not anticipate any data loss from this issue. We know the root cause, and are working to mitigate the issue, but do not have an estimate for resolution at this time. We will follow up when the system is back to normal, but please contact our support team at [email protected] if you have any questions or concerns. Regards, Recorded Future Platform Operations

2. identified Oct 20, 2025, 07:42 PM UTC

   Our team has mitigated the issue, and the processing delay is reducing. We will provide an updated estimated resolution time once we are able to confirm the speed of the reduction.

3. resolved Oct 20, 2025, 08:19 PM UTC

   Dear Customer, We are now seeing the analysis processing returning to normal timeframes. While delays may still be present, delay times are within expected ranges. Regards, Recorded Future Platform Operations

4. postmortem Nov 03, 2025, 07:33 PM UTC

   * **Issue:** * On the day of the major [Amazon Web Services \(AWS\) outage](https://aws.amazon.com/message/101925/), our systems experienced a significant delay in data processing. Specifically, at 19:20, a normal surge in data arrived for processing by Recorded Future. However, due to the ongoing AWS outage, neither automatic nor manual scaling of our processing capacity was technically possible. This lack of available compute resources caused the data processing queues to suffer unexpected delays. * **Cause:** * The primary cause was the [AWS regional issue in US-EAST-1](https://aws.amazon.com/message/101925/), which directly inhibited the ability of our Platform to dynamically increase resource capacity \(scaling\) to meet the incoming data load. * **Remediation / Path Forward:** * Once AWS resolved the underlying issue that was preventing scaling, our systems automatically increased capacity and processed the queued data as designed, bringing the queues back to normal levels. * Recorded Future is actively discussing a strategic initiative to migrate our systems to alternative regions toenhance our resilience against future single-region AWS outages.

Read the full incident report →

Started Sep 30, 2025, 03:27 PM UTC· resolved Sep 30, 2025, 03:49 PM UTC

1. investigating Sep 30, 2025, 03:27 PM UTC

   Dear Customer, We are currently experiencing a service disruption with the Recorded Future application portal & Mobile Application. Our product operations team are actively working on restoring accessibility to an optimal level of service. We will continue to provide updates as they become available. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

2. investigating Sep 30, 2025, 03:46 PM UTC

   Dear Customer, We have resolved a service disruption with the Recorded Future Application and APIs. The disruption occurred between 15:27-15:45 UTC on Tuesday September 30, 2025. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

3. resolved Sep 30, 2025, 03:49 PM UTC

   Dear Customer, We have resolved a service disruption with the Recorded Future Application and APIs. The disruption occurred between 15:27-15:45 UTC on Tuesday September 30, 2025. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

Read the full incident report →

Started Aug 13, 2025, 08:45 PM UTC· resolved Aug 13, 2025, 09:03 PM UTC

1. resolved Aug 14, 2025, 01:46 PM UTC

   Dear Customer, We have resolved a service disruption with the Recorded Future Application and APIs. The disruption occurred between 20:45-21:00 UTC on Wednesday August 13, 2025. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

2. postmortem Aug 21, 2025, 08:38 PM UTC

   * **Issue:** * .At 20:45 UTC on August 13th, our automated monitoring system alerted our team to an issue with the Recorded Future Portal and APIs. * **Cause:** * . A configuration change was made containing an incorrect parameter and deployed. * **Remediation / Path Forward:** * The configuration was corrected and deployed at approximately 20:55 UTC, with full recovery confirmed by 21:03 UTC. Going forward, we are adding tests to ensure this type of change is flagged more prominently prior to deployment. .

Read the full incident report →

Started Aug 06, 2025, 02:55 PM UTC· resolved Aug 06, 2025, 05:44 PM UTC

1. investigating Aug 06, 2025, 02:55 PM UTC

   We are currently investigating this issue.

2. identified Aug 06, 2025, 03:52 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified Aug 06, 2025, 04:45 PM UTC

   Dear Customer, We are currently experiencing a service disruption pertaining to the Recorded Future University training platform, which may result in errors trying to log in to https://university.recordedfuture.com/ . We are currently working with our vendor to resolve this problem as soon as possible. Please note that access to the Recorded Future application platform (https://recordedfuture.com/live) is unaffected, and our support site (https://support.recordedfuture.com/hc/en-us) is still available for additional training content. We will follow up when the system performance returns to normal, but please contact our support team at [email protected] if you have any questions or concerns. Regards, Recorded Future Platform Operations

4. resolved Aug 06, 2025, 05:44 PM UTC

   Dear Customer, Access to the Recorded Future University training platform is now restored, and the incident is now considered resolved. Please contact our support team at [email protected] if you have any questions or concerns. Regards, Recorded Future Platform Operations

Read the full incident report →

Started Jul 15, 2025, 06:48 AM UTC· resolved Jul 15, 2025, 07:02 AM UTC

1. investigating Jul 15, 2025, 06:48 AM UTC

   Dear Customer, We are currently experiencing a service disruption to our Sandbox feature, with access currently unavailable on sandbox.recordedfuture.com and private.tria.ge . We're currently reviewing the incident with high urgency, and will continue to provide updates as they become available. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

2. investigating Jul 15, 2025, 07:02 AM UTC

   Dear Recorded Future Customers, We want to inform you that the Sandbox service disruption and degraded performance have now been resolved. System performance and response times are back to normal. Please contact our support team at [email protected] if you have any questions or concerns. Regards, Recorded Future Platform Operations

3. resolved Jul 15, 2025, 07:02 AM UTC

   This incident has been resolved.

4. postmortem Jul 23, 2025, 08:56 PM UTC

   * **Issue:** * At approximately 6:25 UTC on July 15th, we received several reports that the Recorded Future Sandbox \(specifically [sandbox.recordedfuture.com](http://sandbox.recordedfuture.com)\), was unavailable, which was confirmed by our operations team. * **Cause:** * The root cause of the incident was a configuration error. * **Remediation / Path Forward:** * At approximately 6:48 UTC, a formal incident was opened for client communications, with the root cause being identified shortly thereafter. The correct Sandbox configurations were applied, with the incident being marked as resolved at approximately 7:02 UTC. As a path forward, new monitoring checks were added to alert for any future configuration discrepancies.

Read the full incident report →

Started Jul 01, 2025, 04:34 PM UTC· resolved Jul 01, 2025, 11:13 PM UTC

1. investigating Jul 01, 2025, 04:34 PM UTC

   Dear Customer, We are currently experiencing a service disruption to our Sandbox feature, with access currently unavailable on sandbox.recordedfuture.com and private.tria.ge . We're currently reviewing the incident with high urgency, and will continue to provide updates as they become available. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

2. identified Jul 01, 2025, 06:26 PM UTC

   Dear Customer, We are making progress with our recovery of the Sandbox feature to optimal performance levels, with our private.tria.ge site now available and operational. We are still working on restoring full access to the Sandbox on the sandbox.recordedfuture.com site, and will continue to provide updates as they become available. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

3. monitoring Jul 01, 2025, 07:40 PM UTC

   Dear Customer, A fix has been applied for all Sandbox sites, including for sandbox.recordedfuture.com. Our team is monitoring and addressing additional remaining service restoration, but the Sandbox feature should be working normally at this point. We will post an update when we consider the incident fully resolved. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

4. resolved Jul 01, 2025, 11:13 PM UTC

   Dear Recorded Future Customers, We want to inform you that the Sandbox service disruption and degraded performance have now been resolved. System performance and response times are back to normal. Please contact our support team at [email protected] if you have any questions or concerns. Regards, Recorded Future Platform Operations

5. postmortem Jul 03, 2025, 07:47 PM UTC

   ### **Summary** On July 1, 2025, starting at approximately 15:42 UTC, our Recorded Future Sandbox \(excluding US-hosted enterprise users\) services experienced an outage due to a critical incident at Scaleway's nl-ams-1 datacenter. The outage was caused by the datacenter's thermal management solution being unable to cope with extreme temperatures in The Netherlands, leading to a forced shutdown of their services in the affected region. Our team detected the outage via automated alerts and quickly initiated efforts to restore service by provisioning resources in an alternate availability zone. We have reached out to Scaleway and they will create a post-mortem for this incident soon. ### **Impact** The incident resulted in the complete unavailability of several Recorded Future Sandbox environments. Crucially, despite the service unavailability, **no data was lost or compromised**. All data stored in the affected AZ remained encrypted at rest for the entire duration of the incident. ### **Root Cause** The root cause of this outage was an **external infrastructure failure** at Scaleway's nl-ams-1 datacenter. Specifically, their thermal management system failed to maintain operational temperatures, necessitating a shutdown of their services within that datacenter. ### **Timeline of Events** * **July 1, 2025 - 15:42 UTC:** Our internal monitoring systems triggered alerts indicating unavailability of several sandbox-related services. This was also the time Scaleway stated services were shut down. * **July 1, 2025 - Shortly after 15:42 UTC:** Our team immediately began investigating the alerts. Concurrently, we observed an incident posted on the Scaleway status page \([https://status.scaleway.com/incidents/1vz4xfgy2gcl](https://status.scaleway.com/incidents/1vz4xfgy2gcl)\) confirming a widespread issue in their nl-ams-1 datacenter due to thermal management failures and subsequent service shutdowns. * **July 1, 2025 - 15:45 UTC \(Approx.\):** Our team initiated the process of provisioning replacement services in a different Scaleway Availability Zone \(AZ\) to mitigate the impact of the nl-ams-1 outage. * **July 1, 2025 - 19:26 UTC:** Recorded Future Sandbox was successfully restored and became accessible to users. * **July 1, 2025 - 23:17 UTC:** internal pipelines providing data to Recorded Future Platform were successfully restored. * **July 2, 2025 - 00:42 UTC:** Scaleway officially reported the issue in their nl-ams-1 datacenter as resolved. \(Note: Our services were recovered significantly earlier due to our AZ failover strategy\). ### **What Went Well** * **Automated Alerting:** Our monitoring systems effectively detected the service unavailability immediately, allowing for rapid initiation of incident response. * **Quick External Cause Identification:** The prompt update from Scaleway's status page allowed our team to quickly confirm the external nature of the incident and shift focus from internal debugging. * **Effective AZ Failover:** Our team's ability to immediately begin provisioning services in another Availability Zone was crucial in significantly reducing the overall downtime, allowing us to restore service well before Scaleway fully resolved their datacenter issue. ### **Remediations** Following the incident and the restoration of service in the affected Availability Zone: * **Infrastructure Retirement:** After the nl-ams-1 Availability Zone came back online, we retired the old infrastructure that was running in that single AZ. * **Improved Frontend Failover Automation:** We have improved our failover automation for the frontend service to handle entire Availability Zones going down. The essential data and services required to set up frontend deployments now run across multiple AZs.

Read the full incident report →

Started Jun 26, 2025, 11:38 AM UTC· resolved Jun 26, 2025, 01:35 PM UTC

1. investigating Jun 26, 2025, 11:38 AM UTC

   Dear Customer, We are experiencing performance issues with our Browser Extension and our engineers are currently working to resolve this problem as quickly as possible. Please contact our support team at [email protected] if you have any questions or concerns. Regards, Recorded Future Platform Operations

2. resolved Jun 26, 2025, 01:35 PM UTC

   Dear Customer, The performance issues with our Browser Extension have been resolved. Please contact our support team at [email protected] if you have any questions or concerns. Regards, Recorded Future Platform Operations

Read the full incident report →

Started Jun 23, 2025, 11:36 PM UTC· resolved Jun 24, 2025, 03:06 AM UTC

1. investigating Jun 23, 2025, 11:36 PM UTC

   Dear Recorded Future Customers, We want to inform you that our Advanced Query Builder and viewing alerts in the Alerts Dashboard are both currently experiencing degraded performance. Our team is actively investigating the issue and working to restore full functionality as quickly as possible. We apologize for any inconvenience this may cause and appreciate your patience. We will continue to provide updates as more information becomes available.

2. monitoring Jun 24, 2025, 02:40 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jun 24, 2025, 03:06 AM UTC

   Dear Recorded Future Customers, We want to inform you that our Advanced Query Builder and viewing alerts in the Alerts Dashboard degraded performance have been resolved. System performance and response times are back to normal. Please contact our support team at [email protected] if you have any questions or concerns. Regards, Recorded Future Platform Operations

Read the full incident report →

Started Jun 04, 2025, 09:38 AM UTC· resolved Jun 04, 2025, 01:44 PM UTC

1. investigating Jun 04, 2025, 09:38 AM UTC

   Dear Customer, We are currently experiencing a service disruption with the Recorded Future application portal & Mobile Application. Our product operations team are actively working on restoring accessibility to an optimal level of service. We will continue to provide updates as they become available. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

2. investigating Jun 04, 2025, 09:49 AM UTC

   We are continuing to investigate this issue.

3. identified Jun 04, 2025, 10:04 AM UTC

   The issue has been identified, and a fix is being implemented.

4. identified Jun 04, 2025, 10:30 AM UTC

   Dear Customer, We are experiencing a service disruption with our Recorded Future application portal, mobile application and API. These services are currently unavailable. Our engineers are working diligently to resolve this issue as quickly as possible. Please contact our support team at [email protected] if you have any questions or concerns. Regards, Recorded Future Platform Operations

5. monitoring Jun 04, 2025, 10:57 AM UTC

   Dear Customer, The Recorded Future team has resolved the recent system issue affecting the Recorded Future application portal, mobile application, and API, as described in our recent system issue alert email. We understand the impact that a system disruption may have on your business. We have identified the underlying cause of this problem and are currently working on a solution to prevent it from occurring again. You can contact our support team at [email protected] to discuss this issue or any other questions you may have regarding our products. We are always here to help. Regards, Recorded Future Platform Operations

6. monitoring Jun 04, 2025, 11:03 AM UTC

   Dear Customer, We are currently experiencing a delay in our content collection and analysis processes. This means that there is a delay between when content is published and when it is available in Recorded Future’s web application, API, and alerts. Other systems are running normally, historical data is still available, and we do not anticipate any data loss from this issue. We will follow up when the system is back to normal, but please contact our support team at [email protected] if you have any questions or concerns. Regards, Recorded Future Platform Operations

7. resolved Jun 04, 2025, 01:44 PM UTC

   Dear Customer, We can now consider the service disruption with the Recorded Future platform to now be resolved, and a root cause analysis will be produced for this incident. Please contact our support team at [email protected], or your Intelligence Services resource if you have any questions. Regards, Recorded Future Platform Operations

8. postmortem Jun 13, 2025, 01:43 PM UTC

   Dear Customer, Below is the root cause analysis for the Recorded Future service disruption: * **Issue:** * At approximately 9:10 UTC on June 4, 2025, the Recorded Future Portal, APIs and mobile application became unavailable. * **Cause:** * During a scheduled upgrade of an internal database, an error caused authentication requests to fail, resulting in the Recorded Future Portal, API and mobile application becoming unresponsive. * **Remediation / Path Forward:** * Our Platform Operations team recovered the database at approximately 9:45 UTC, and restarted impacted services. At approximately 10:52 UTC, the Recorded Future Portal, API and mobile application were confirmed to be responsive again. After a period of time to allow all relevant processes to operate normally, the incident was marked as fully resolved at 13:45 UTC. * Moving forward, database upgrades include a number of new safety features which should prevent this specific issue from recurring. In addition, automated database rollbacks will be implemented which should speed up recovery in the event of a future database failure.

Read the full incident report →

Started Apr 23, 2025, 10:33 PM UTC· resolved Apr 24, 2025, 01:01 AM UTC

1. investigating Apr 23, 2025, 10:33 PM UTC

   We want to let you know that we are currently experiencing an issue with our alert-email delivery system, SendGrid. As a result, some automated alert messages may be impacted or delayed. Our engineering team is actively working with SendGrid to restore full delivery. We will update you as soon as email alerts are operational again. Thank you.

2. investigating Apr 24, 2025, 12:59 AM UTC

   The SendGrid email delivery issue has been fully resolved. All emails are now sending and new messages are being delivered normally.

3. resolved Apr 24, 2025, 01:01 AM UTC

   Dear Customer, Collection and analysis of content is now current and systems that process new content in Recorded Future are operating normally. We apologize for the delays earlier today. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

Read the full incident report →

Started Apr 15, 2025, 03:10 PM UTC· resolved Apr 15, 2025, 03:48 PM UTC

1. investigating Apr 15, 2025, 03:10 PM UTC

   Dear Customer, We are currently experiencing a service disruption with the Recorded Future application portal and API. Our product operations team are actively working on restoring accessibility to an optimal level of service. We will continue to provide updates as they become available. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

2. investigating Apr 15, 2025, 03:24 PM UTC

   We are continuing to investigate this issue.

3. resolved Apr 15, 2025, 03:48 PM UTC

   Dear Customer, We can now consider the service disruption with our portal and API to be resolved, and a root cause analysis will be produced for this incident. Please contact our support team at [email protected], or your Intelligence Services resource if you have any questions. Regards, Recorded Future Platform Operations

4. postmortem Apr 26, 2025, 07:17 PM UTC

   Dear Customer, Below is our post mortem to the incident in question from April 15, 2025: * **Issue:** * At approximately 14:52 UTC on April 15th, monitoring alerts indicated that both the Portal and API were unresponsive to customers. Following initial investigations, the incident status was escalated to a “Major Outage” for the “Portal” and “API” components on [status.recordedfuture.com](http://status.recordedfuture.com) at around 15:10 UTC. * **Cause:** * The incident was traced to the main authorization service, which is a critical dependency for backend request handling. One instance within the service group entered an unresponsive state but continued to accept and process threads. This led to a situation where most incoming authentication requests to this service appeared to hang. As a result, dependent services—including the Portal and API—were perceived as unresponsive from the end-user perspective for a short period of time. * **Remediation / Path Forward:** * Corrections were made to rebalance traffic and service requests to restore normal service, and the issue was declared fully resolved at approximately 15:13 UTC. * To prevent the recurrence of this issue, the following steps are being undertaken: * Enhancing the authorization service to detect and isolate non-responsive instances automatically. * Introducing safeguards to prevent thread buildup when an instance becomes unresponsive.

Read the full incident report →

Started Feb 07, 2025, 12:53 PM UTC· resolved Feb 07, 2025, 01:39 PM UTC

1. investigating Feb 07, 2025, 12:53 PM UTC

   Access to the Recorded Future application and API is currently impaired, and we are currently investigating.

2. investigating Feb 07, 2025, 01:04 PM UTC

   Dear Client, We are currently experiencing a service disruption which affects the Recorded Future application Portal, API, alerting, and the mobile application. Our product operations team are actively working on identifying the issue and restoring accessibility to an optimal level of service We will continue to provide updates as they become available. Please contact our support team at [email protected] if you have any questions. Regards, Recorded Future Platform Operations

3. resolved Feb 07, 2025, 01:39 PM UTC

   Dear Client, The Recorded Future application Portal, API, alerting, and mobile application services have now been restored. Our teams are actively working to monitor the performance of the platform, and a Root Cause Analysis (RCA) will be posted here once complete. Regards, Recorded Future Platform Operations

4. postmortem Feb 17, 2025, 01:38 AM UTC

   * **Issue:** * Routine database maintenance led to unexpected performance degradation * **Cause:** * A new node was added to the database cluster used for verification of application logins. The process of adding the node used was the same documented process followed for adding nodes to all other database clusters, which is routinely performed \(several times per week\). * It appears that we encountered a rare bug with the database cluster software which caused the new node to be treated as a primary node even though it was added as a backup node, which caused massive latency for the cluster, even blocking reconfiguration commands. * **Remediation / Path Forward:** * We have updated our node addition process to include even more safety measures in order to reduce the risk of similar future occurrences. * We have contacted the database cluster software vendor to notify them of this bug.

Read the full incident report →