CyberFOX incident

Data Processing Delays - Agent Reporting Data Impacted

Minor Resolved View vendor source →

CyberFOX experienced a minor incident on February 16, 2024 affecting Agent Services, lasting 3d 3h. The incident has been resolved; the full update timeline is below.

Started
Feb 16, 2024, 01:45 PM UTC
Resolved
Feb 19, 2024, 04:57 PM UTC
Duration
3d 3h
Detected by Pingoru
Feb 16, 2024, 01:45 PM UTC

Affected components

Agent Services

Update timeline

  1. investigating Feb 16, 2024, 02:44 PM UTC

    Our data processing system is delayed, causing some reporting inaccuracies. Our agents can still process all functions and requests for elevations. No data is lost, and the system should catch up soon.

  2. investigating Feb 16, 2024, 04:44 PM UTC

    We are currently investigating the issue of reporting inaccuracies in the agent's metadata and state. New agents will appear in the portal with limited information. The existing agent's information will not be updated. Elevation requests will continue to work, and all other functions will also continue to function normally.

  3. investigating Feb 16, 2024, 07:41 PM UTC

    We are continuing to investigate the issue of reporting inaccuracies in the agent's metadata and state. The development team is reconfiguring the process to allow more resources for it. We have been working with our upstream provider to identify the root cause of the delays and will continue to investigate while this reconfiguration patch is applied. New & existing agents will appear in the portal with limited information, gradually filling in details as processes catch up. Elevation requests will continue to work, and all other functions will continue normally.

  4. investigating Feb 16, 2024, 11:35 PM UTC

    We're still investigating reporting inaccuracies in agent data and have applied an update to improve the data check-in process. We're working with our provider to identify the root cause of delays while this reconfiguration patch is applied. New and existing agents are updating their details gradually, and all functions such as elevation requests are operating normally.

  5. monitoring Feb 17, 2024, 12:18 AM UTC

    We have identified and resolved the issue and are now closely monitoring the results to ensure everything is working properly. Our team has confirmed that all agents are currently checking in, and we have verified that the data is refreshing normally on the portal. We will continue to closely monitor the results over the next few days to ensure that the issue is fully resolved and there are no further complications.

  6. resolved Feb 19, 2024, 04:57 PM UTC

    This incident has been resolved. The development team has corrected the services around the agent status data.