Viirtue experienced a major incident on October 10, 2024 affecting Florida Data Center, lasting 6d 11h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 10, 2024, 01:35 AM UTC
Viirtue engineers are actively investigating a circuit issue at the Florida data center, potentially linked to Hurricane Milton. At this time, the impact is limited to Apollo servers. We advise promptly redirecting any SIP trunk clients currently registered to sip01-fl.5060.cloud to sip01-nj.5060.cloud.
- identified Oct 10, 2024, 04:26 AM UTC
Lumen has confirmed a significant outage within the Tampa metro ring impacting our circuit. Further updates will be provided as they become available.
- identified Oct 10, 2024, 12:41 PM UTC
The latest update from Lumen is below. "2024-10-10 07:33:16 GMT - On October 10, 2024, Lumen discovered a service impact in Florida. Your customer trouble ticket has been related to a network event; all network events are automatically escalated to Lumen leadership."
- identified Oct 10, 2024, 08:45 PM UTC
The latest update from Lumen is below. "Hurricane Milton made landfall as a Category 3 storm during the late overnight hours of October 9th, bringing 8-10 feet of storm surge, sustained hurricane-force winds, isolated tornadoes, and approximately 18 inches of rainfall. Currently, 3,406,056 consumers are without power in Florida. Lumen is closely monitoring Hurricane Milton's path and remains in contact with emergency management teams. Once it is deemed safe and commercial power has resumed, Field Operations will be deployed to assess damages and devise a repair plan."
- identified Oct 11, 2024, 09:37 PM UTC
Lumen Field Operations will be dispatched to work on the outage once emergency personnel deem it safe for travel. We will continue to post updates as we receive them.
- monitoring Oct 14, 2024, 12:57 PM UTC
Viirtue engineers are actively monitoring circuit stability following restoration over the weekend. The circuit has remained stable, and further updates will be provided as necessary.
- monitoring Oct 15, 2024, 05:22 PM UTC
The Viirtue engineering team will conduct a database migration during tonight’s scheduled maintenance window to restore data on the Florida servers. Upon successful completion, the servers will be reintroduced into production.
- resolved Oct 16, 2024, 01:09 PM UTC
The Lumen circuit has stabilized, and the Florida servers have been restored to active production status.