TrackVia experienced a minor incident on September 18, 2023 affecting Commercial Cloud and Private Cloud and 1 more component, lasting 8h 58m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Sep 18, 2023, 07:09 PM UTC
[11:57 AM PDT] TrackVia's cloud provider confirmed increased networking latencies and errors affecting multiple Availability Zones in the US-WEST-2 Region. We have identified a potential root cause of the errors and are attempting mitigations. Early signs are this mitigation is reducing error rates and latencies. We continue to work towards a full root case and recovery. [11:43 AM PDT] TrackVia's cloud provider is investigating increased networking latencies and errors affecting multiple Availability Zones in the US-WEST-2 Region.
- identified Sep 18, 2023, 07:10 PM UTC
TrackVia's cloud provider is continuing to work on a fix for this issue.
- identified Sep 18, 2023, 07:46 PM UTC
TrackVia's cloud provider is continuing to investigate increased networking latencies and errors affecting Availability Zones in the US-WEST-2 Region. The issue affects some instances in each of these Availability Zones, where network mappings are not being propagated to the underlying hardware. Any changes to network configurations would see delayed mappings, which will affect network connectivity. For other TrackVia services, such as micro services and backup/recovery, delays in function as well as increased error rates may occur due to this issue. AWS is working to resolve the issue and is seeing improvements as a result of these efforts. We will continue to keep you updated as we make progress towards full recovery.
- identified Sep 18, 2023, 09:33 PM UTC
While TrackVia's cloud provider continues to make progress in addressing the issue, we wanted to provide some more details on the issue. Starting at 10:00 AM PDT this morning, TrackVia's cloud provider has been experiencing a delay in the propagation of network configuration changes to the underlying hardware ensuring network packets can flow between source and destination. The root cause appears to be increased load to the subsystem responsible for the handling of these network mappings. Once load has been reduced on the subsystem responsible for network mapping propagation, we would expect full recovery. We will continue to keep you updated as we make progress towards full recovery.
- identified Sep 18, 2023, 11:31 PM UTC
TrackVia's cloud provider continues to progress toward resolving the increased networking latencies and errors. At this time, they are approximately 50% completed with the update to address resource contention within the subsystem responsible for network mappings propagation. Their current expectation is to have the problem resolved within the next 60 to 90 minutes, and we will continue to provide updates as recovery progresses.
- monitoring Sep 19, 2023, 12:36 AM UTC
TrackVia's cloud provider has successfully applied an update to the subsystem responsible for network mapping propagation to address resource contention. They have seen network mapping propagation times stabilize but they have not yet begun to trend towards normal levels. They expect that to begin over the next 30 minutes, at which time we expect latencies and error rates to improve. We will continue to keep you updated on our progress towards full recovery.
- resolved Sep 19, 2023, 04:07 AM UTC
This incident has been resolved.