Viirtue incident
Intermittent Network Connectivity to Las Vegas Datacenter
Viirtue experienced a minor incident on September 25, 2024 affecting LV Core Server and LV QoS Server and 1 more component, lasting 11h 26m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 25, 2024, 03:20 PM UTC
Viirtue has identified intermittent network connectivity issues affecting our Las Vegas Datacenter. Our teams are actively investigating the cause of the disruption and are working to restore stable connectivity as soon as possible. We will provide further updates as more information becomes available. Thank you for your patience.
- identified Sep 25, 2024, 03:55 PM UTC
We have identified the network issue affecting the connection between the New Jersey and Las Vegas data centers and are actively collaborating with our ISP's to resolve it. Please note that this issue is not expected to impact call quality or traffic at either data center.
- identified Sep 25, 2024, 06:10 PM UTC
Virtue Engineers have temporarily "503'd" the Las Vegas core while addressing the network issue between New Jersey and Las Vegas. We do not anticipate any service disruptions at the New Jersey or Florida data centers as a result of this change.
- resolved Sep 26, 2024, 02:47 AM UTC
The network issue between the Las Vegas and New Jersey data centers has been resolved. Additional details will be provided in the postmortem report.
- postmortem Sep 26, 2024, 02:47 AM UTC
A network issue involving a major backbone carrier, Twelve99/Arelion, resulted in significant connectivity disruptions between our New Jersey and Las Vegas data centers. Arelion identified the root cause as "congestion in our backbone between Los Angeles and Ashburn." This delay led to SBUS queuing on both the New Jersey and Las Vegas core systems. To mitigate potential call quality issues caused by the SBUS delays, Viirtue engineers implemented a temporary 503 status on the Las Vegas core server at around 1:30 PM EST. Arelion resolved the congestion at approximately 6:50 PM EST, and Viirtue engineers observed improvements in connectivity shortly thereafter. The 503 status was lifted from the Las Vegas core server at around 10:30 PM EST.