TrackVia experienced a minor incident on May 26, 2021 affecting Commercial Cloud, lasting 43m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 26, 2021, 01:57 PM UTC
TrackVia is currently investigating a performance issue in our Commercial cloud environment. No data has been lost and the problem should be resolved shortly.
- resolved May 26, 2021, 02:40 PM UTC
This incident has been resolved.
- postmortem Nov 09, 2021, 11:36 PM UTC
# **TrackVia Incident Report** ## **Service Impairment/Outage** ## **May 26,2021** ### Summary On May 26, 2021 from 7:42 AM MT to 8:26 AM MT, TrackVia experienced performance degradation causing a 3 minute outage from 7:52 AM to 7:55 AM MT. TrackVia Operations observed high CPU load affecting application performance. Additional processing capacity was added while TrackVia investigated the issue and full functionality was restored by 8:26 AM MT. ### Root Cause Analysis An investigation of the event concluded optimization efforts conducted on the previous day left the database with a corrupted table cache leading to suboptimal performance. Application processing slowed eventually leading to a small outage. Trackvia Operations flush the table cache restoring database performance. ### Corrective Actions 1. TrackVia Operations initiated an application autoscaling event adding two additional app servers in each Availability Zone. 2. TrackVia Operations forced a database failover 3. TrackVia Operations executed a restart of existing application servers. 4. TrackVia Operations placed account 21227 in maintenance mode to safe guard data integrity. 5. TrackVia Operations executed a flush tables against each database reader which ultimately returned database performance back to normal