Recorded Future experienced a minor incident on November 5, 2024 affecting User Interface and API, lasting 2h 54m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 05, 2024, 01:34 PM UTC
Dear Client, We are currently experiencing degradation of system performance which is impacting accessibility to our playbook alert API, in addition to portal accessibility to such features as alerts and lists. Our product operations team are actively working on restoring accessibility to an optimal level of service We will continue to provide updates as they become available. Please contact our support team at [email protected], or your Intelligence Services representative if you have any questions. Regards, Recorded Future Platform Operations
- monitoring Nov 05, 2024, 02:41 PM UTC
Dear Client, We have identified the root of instability and believe we have restored performance back to optimal levels. We will continue to monitor to ensure services remain fully available, and provide an update when the incident is fully resolved. Please contact our support team at [email protected], or your Intelligence Services representative if you have any questions. Regards, Recorded Future Platform Operations
- resolved Nov 05, 2024, 04:28 PM UTC
Dear Client, We have identified the root of instability and have now restored performance to optimal levels. We will continue to monitor to ensure services remain fully available. Regards, Recorded Future Platform Operations
- postmortem Nov 27, 2024, 09:01 PM UTC
* **Issue:** * On the morning of November 5th \(approximately 7am EST\), Recorded Future’s platform experienced a degradation of system performance which impacted accessibility to various parts of our external API, including our playbook alert API. In addition, this incident negatively impacted end user access to our portal features such as alerts and lists. * **Cause:** * Too much load on a particular querying service was determined to be the root cause of the service disruption, which required an increase of available resources to remediate. * **Remediation / Path Forward:** * Within one hour of incident acknowledgement, the Recorded Future platform operations team engaged with our product development team to collectively increase available resources that would allow for optimal performance in the API and portal. Tentative resolution was communicated at 9:41am EST, with full closure of the incident occurring shortly before 11:30am EST. * Long term, performance improvements are planned for release pertaining to our Threat Intelligence Ransomware service, which is responsible for the majority of our long-running queries that contributed to the impact seen during this incident.