Sensor Data not ingested into the central cloud database

Update timeline

1. resolved Nov 09, 2023, 04:25 PM UTC

   After the cloud upgrade to SysTrack 10.10, a limited set of sensor data was not ingested into the central cloud database. This is a server-side issue only and data was collected and stored on the child systems as normal.

2. postmortem Nov 09, 2023, 04:26 PM UTC

   ## What is the issue? After the cloud upgrade to SysTrack 10.10, a limited set of sensor data was not ingested into the central cloud database. This is a server-side issue only and data was collected and stored on the child systems as normal. Sensor data was not ingested for 75 \(of the 1418\) sensors. The 75 sensors affected are listed at the end of this page. The cloud sensor data is primarily displayed in the SysTrack Prevent module and a limited number of SysTrack dashboards. Resolve, Assist and Visualizer are not impacted. The issue began after the 10.10 cloud update with the schedule listed below: * UAE, Australia, France - 10/20/23 at 21:00 UTC * Canada, Insiders - 10-23-23 at 08:30 UTC * Germany - 10-27-23 at 21:00 UTC * PWD, UK - 10-27-23 at 21:00 UTC * Americas, Optum, AB - 10-28-23 at 10:30 UTC The issue was resolved on the dates/times listed below: * All clouds not listed individually on 11-8-23 at 23:00 UTC ## What was the root cause? A fix for a prior issue was not merged back to the main dev branch and, therefore, it was not included in the 10.10 upgrade. ## What is the prevention strategy? An additional step will be added to the hotfix release process to create a Jira cloud update request. The requester will be responsible for ensuring that the following steps have been taken/completed: * validate that all appropriate Fix Versions are set in the Jira ticket\(s\) * that a pull request is created for all appropriate branches as indicated by the Fix Version\(s\) * that development and QA \(as appropriate\) have completed internal testing * that QA has created a regression test for the issue for future releases \(or a work item to create the regression test has been created\) ### Sensors Impacted * Application Connectivity Problem * Application Crash After Software Change * Application Hang Correlated With High Add-In Load Time * Application Hang Not Waiting For Resources * Application High Latency * Application High Network Utilization * Application Not Responding * Application Restart Failed * Batch Requests per sec * C Drive Low Disk Space * Cache Fault Rate * Commit Ratio * CPU High Limit * CPU Load * CPU Low Limit * CPU Queuing Health Impact * CPU Throttling * Crash On Audit Fail * Critical App Connectivity * Critical Application Crash * Critical Service Not Running * Current Blocked Bandwidth Bytes * Current Connections * Default Gateway Latency Impact * Default Gateway Not Available * Device Manager Status * Disk Problems with Low Demand * Driver Installation Failure * Extended High Application CPU Usage * Extended High Application IOPS Usage * Extended High Application Memory Usage * Extended High CPU Use * Extended High Service CPU Usage * Extended Low Available Memory * Frequent Application Faults * Frequent Application Starts * Frequent Network Disconnects * Frequent Session Disconnects * Frequent System Restarts * GDI Object Leak Detected * Handle Leak Detected * Hard Disk Bad Sector * Hard Drive Health Issues * Head Requests per sec * Health Score Issues * High Disk Latency * High Kernel Mode CPU Use * Kernel Boot Failed to Resume from Hibernate * Kernel Panic Detected * LAN Manager Client Service Stopped * LanmanServer Service Disabled * LanmanServer Service Stopped * Low Pagefile Space * NDIS Driver Failed to Load * Near Real Time CPU Impact * Near Real Time Disk Impact * Near Real Time Latency Impact * Near Real Time Memory Impact * Near Real Time Event Issues * Near Real Time Fault Issues * Near Real Time Hardware Issues * Near Real Time Network Issues * Near Real Time Software Installation Issues * Near Real Time Software Update Issues * Near Real Time Startup Issues * Near Real Time Virtual Machine Issues * Near Real Time Virtual Memory Issues * Non-Paged Pool Leak Detected * Paged Pool Leak Detected * RPC Service Stopped * Thermal Level * Thread Count * Thread Leak Detected * User Object Leak Detected * WiFi Failed to Connect