Resolved after 38 minutes, on May 7, 2024.
# Incident Report May 10, 2024 ## Summary On May 8 at 8:07 AM MT, TrackVia Support notified the TrackVia Operations team that integration connections were being dropped. An investigation of the issue determined that a TrackVia vendor was reporting issues with their API service.TrackVia’s Vendor first reported an issue at 7:21 MT and resolved the issue. Full functionality was restored by 7:59 MT. ## Root cause TrackVia’s Vendor reports unexpected behavior of the network controller managing load balancers recreated some resources with invalid configurations. ## Resolution and recovery TrackVia’s Vendor corrected the mis-configuration and restored API operations by 7:59 MT. ## Corrective and preventive measures * TrackVia’s Vendor replaced the network controller which caused the mis-configuration. * TrackVia added additional monitoring to identify vendor issues in a more timely manner. ## Incident timeline * May 8, 2023 6:58 MT: Vendor update of a production cluster. * May 8, 2023 7:17 MT: Vendor started receiving alerts for the internal API, Echo API and ActiveDocs Proxy. * May 8, 2023 7:58 MT: Root cause identified and solution deployed. * May 8, 2023 8:00 MT: Service is restored.
Update timeline
- resolved May 07, 2024, 02:48 PM UTC
A TrackVia vendor experienced a service disruption in the ActiveDocs Proxy which may have caused interruption with TrackVia api services. The disruption lasted for approximately 38 minutes and was operational by 7:59 AM MT.
- postmortem May 10, 2024, 05:57 PM UTC
# Incident Report May 10, 2024 ## Summary On May 8 at 8:07 AM MT, TrackVia Support notified the TrackVia Operations team that integration connections were being dropped. An investigation of the issue determined that a TrackVia vendor was reporting issues with their API service.TrackVia’s Vendor first reported an issue at 7:21 MT and resolved the issue. Full functionality was restored by 7:59 MT. ## Root cause TrackVia’s Vendor reports unexpected behavior of the network controller managing load balancers recreated some resources with invalid configurations. ## Resolution and recovery TrackVia’s Vendor corrected the mis-configuration and restored API operations by 7:59 MT. ## Corrective and preventive measures * TrackVia’s Vendor replaced the network controller which caused the mis-configuration. * TrackVia added additional monitoring to identify vendor issues in a more timely manner. ## Incident timeline * May 8, 2023 6:58 MT: Vendor update of a production cluster. * May 8, 2023 7:17 MT: Vendor started receiving alerts for the internal API, Echo API and ActiveDocs Proxy. * May 8, 2023 7:58 MT: Root cause identified and solution deployed. * May 8, 2023 8:00 MT: Service is restored.
Your product runs on more than TrackVia
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services