Zeus - Datacenter ISP degradation

Update timeline

1. investigating Jan 15, 2025, 04:37 PM UTC

   We are currently investigating a datacenter ISP carrier outage at the FL datacenter, which is affecting the ZEUS platform, including both the voice platform and management portal. Our team is working closely with the carrier to understand the root cause and resolve the issue as quickly as possible. We will continue to monitor the situation and provide updates as we learn more. Thank you for your patience and understanding during this time.

2. identified Jan 15, 2025, 04:43 PM UTC

   Viirtue Engineers have identified an issue with the Florida Zeus circuit. The web portal and voice traffic have been moved to the New Jersey datacenter. Users may experience issues with mobile app calls. Updates to follow.

3. monitoring Jan 15, 2025, 04:52 PM UTC

   Incident Update: The carrier is currently investigating network issues impacting the Florida region. In response, Viirtue teams have successfully migrated critical service traffic and will continue to monitor the situation throughout the remainder of the day.

4. identified Jan 15, 2025, 04:53 PM UTC

   Lumen has confirmed they are seeing a few outages in the Florida area. While the circuit looks be be stabilizing, traffic will remain pointed to the New Jersey Zeus circuit. Updates to follow.

5. identified Jan 15, 2025, 07:49 PM UTC

   Lumen is working to resolve the outages affecting the Zeus Florida circuit. Updates to follow.

6. identified Jan 15, 2025, 09:09 PM UTC

   Zeus users may experience issues downloading call recordings from the portal, especially for calls older than 4 days. Updates to follow.

7. resolved Jan 16, 2025, 02:07 PM UTC

   Incident has been resolved. Details from carrier: Root Cause: An equipment failure in Tampa, FL impacted services. Fix Action: Services were migrated to an alternate path and the affected card was replaced in Forest Park, GA, thus restoring services to a stable state. This service impact is over. Lumen Network Operations Center (NOC) management will be reviewing this outage for any additional analysis or discovery. In some cases, you may need to reset your services locally at your equipment. If a service issue continues after that reset, please contact the Lumen Repair Center for help.

8. postmortem Jan 16, 2025, 03:44 PM UTC

   The Zeus Florida circuit experienced significant packet loss due to a fiber issue in Lumen's network, which affected customers in the Florida region. Upon identifying the root cause of the issue, the Viirtue team promptly redirected all traffic to the Zeus New Jersey datacenter to mitigate the impact. Lumen resolved the fiber issue early Thursday morning, after which the Viirtue team thoroughly tested the circuit. Once confirmed stable, traffic was successfully redirected back to the Zeus Florida datacenter.