CyberFOX incident
Agents apearing offline/unmanagable in the portal
CyberFOX experienced a minor incident on September 11, 2023 affecting Agent Services and Admin Portal, lasting 1h 18m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 11, 2023, 06:28 PM UTC
We have received reports of agents being unable to be updated in the portal. Those agents are not manageable.
- investigating Sep 11, 2023, 06:29 PM UTC
Agents are reporting to be manageable again.
- identified Sep 11, 2023, 06:33 PM UTC
The issue has been identified as a problem with writing to the database that stores agent commands. The database has recovered and is responding to commands again.
- monitoring Sep 11, 2023, 06:42 PM UTC
Our team is actively working with our upstream provider to identify why agents were unable to write to the database from 2:03 PM to 2:18 PM. We will continue to closely monitor the system for any further issues and report them to you as soon as possible.
- monitoring Sep 11, 2023, 06:56 PM UTC
Our upstream service provider experienced a serious hardware issue that has impacted our databases. They are currently in the process of migrating our databases to a new platform. We anticipate that this migration will be completed soon, but during this time, our services may experience intermittent errors.
- resolved Sep 11, 2023, 07:22 PM UTC
The incident has been resolved. Our upstream provider has confirmed that one of our database servers experienced hardware failure, which triggered the automated recovery process and migrated to a new hardware platform. While the recovery process was ongoing, our application experienced intermittent delays and error messages. However, all services have now returned to normal and are functioning properly.