Viirtue Outage History

Viirtue is up right now

Viirtue had 46 outages in the last 2 years totaling 2215h 51m of downtime — averaging 1.9 incidents per month.

There were 46 Viirtue outages since August 7, 2024 totaling 2215h 51m of downtime. Each is summarised below — incident details, duration, and resolution information.

Source: https://status.viirtue.com

Notice May 1, 2025

Zeus - Cluster Portal Access

Detected by Pingoru
May 01, 2025, 12:48 PM UTC
Resolved
May 01, 2025, 12:20 PM UTC
Duration
—
Timeline · 1 update
  1. resolved May 01, 2025, 12:48 PM UTC

    Earlier this morning, prior to business hours, we experienced a brief event impacting the Zeus cluster that caused the customer portal to be temporarily unavailable for approximately 10 minutes. The issue self-recovered without manual intervention. Our engineering team was immediately engaged and is continuing to investigate the root cause. All portal services have since been fully restored and are operating normally.

Read the full incident report →

Notice April 8, 2025

Emergency Network Change in Progress – Las Vegas

Detected by Pingoru
Apr 08, 2025, 12:36 AM UTC
Resolved
Apr 08, 2025, 03:34 AM UTC
Duration
2h 57m
Affected: LV Core ServerLV QoS ServerLV Recording Server
Timeline · 3 updates
  1. investigating Apr 08, 2025, 12:36 AM UTC

    We are currently executing an emergency change to address the network issues affecting our Las Vegas infrastructure. During this change, there may be a brief loss of connectivity to the Las Vegas region, lasting only a few seconds. We expect minimal impact and appreciate your patience as we work to resolve the issue swiftly.

  2. investigating Apr 08, 2025, 02:14 AM UTC

    We will be 503'ing the Las Vegas Apollo core while we perform configuration updates. Updates to follow.

  3. resolved Apr 08, 2025, 03:34 AM UTC

    This incident has been resolved.

Read the full incident report →

Minor March 11, 2025

Incident Report – Core2-NJ NMS Service Interruption

Detected by Pingoru
Mar 11, 2025, 08:12 PM UTC
Resolved
Mar 11, 2025, 07:00 PM UTC
Duration
—
Timeline · 1 update
  1. resolved Mar 11, 2025, 08:12 PM UTC

    Date: March 11, 2025 Time: 14:44:57 EST At approximately 14:44:57 EST on March 11, the NMS service on core2-nj (primary core processing service) experienced a crash, resulting in the temporary disruption of some active calls and device registrations. The service automatically restarted within seconds, and all impacted services recovered immediately. An open ticket is already in place with NetSapiens for further investigation into the root cause. We will provide updates as more information becomes available. Please reach out Support if you have any questions or concerns.

Read the full incident report →

Minor January 28, 2025

Apollo - Portal Issues

Detected by Pingoru
Jan 28, 2025, 03:07 PM UTC
Resolved
Jan 31, 2025, 02:14 PM UTC
Duration
2d 23h
Affected: Manager Portal
Timeline · 4 updates
  1. investigating Jan 28, 2025, 03:07 PM UTC

    Our team is currently investigating reports of outdated voicemails appearing in the portal and completed calls displaying as active. We are working diligently to resolve the issue and will provide an update as soon as possible.

  2. investigating Jan 29, 2025, 03:16 PM UTC

    We are continuing to work with our partners to resolve this issue—updates to follow.

  3. identified Jan 30, 2025, 06:58 PM UTC

    A potential fix will be implemented tonight during scheduled maintenance.

  4. resolved Jan 31, 2025, 02:14 PM UTC

    This incident has been resolved by restarting a few key services during a maintenance window.

Read the full incident report →

Minor January 15, 2025

Zeus - Datacenter ISP degradation

Detected by Pingoru
Jan 15, 2025, 04:37 PM UTC
Resolved
Jan 16, 2025, 02:07 PM UTC
Duration
21h 30m
Affected: APIFlorida Data CenterManager PortalFL Core ServerNJ Core ServerMessaging (SMS/MMS)WebPhoneWebMeetingsProvisioning ServerMobile App - iOSMobile App - AndroidAnalytics QoSRecording-CloudRecording FL
Timeline · 8 updates
  1. investigating Jan 15, 2025, 04:37 PM UTC

    We are currently investigating a datacenter ISP carrier outage at the FL datacenter, which is affecting the ZEUS platform, including both the voice platform and management portal. Our team is working closely with the carrier to understand the root cause and resolve the issue as quickly as possible. We will continue to monitor the situation and provide updates as we learn more. Thank you for your patience and understanding during this time.

  2. identified Jan 15, 2025, 04:43 PM UTC

    Viirtue Engineers have identified an issue with the Florida Zeus circuit. The web portal and voice traffic have been moved to the New Jersey datacenter. Users may experience issues with mobile app calls. Updates to follow.

  3. monitoring Jan 15, 2025, 04:52 PM UTC

    Incident Update: The carrier is currently investigating network issues impacting the Florida region. In response, Viirtue teams have successfully migrated critical service traffic and will continue to monitor the situation throughout the remainder of the day.

  4. identified Jan 15, 2025, 04:53 PM UTC

    Lumen has confirmed they are seeing a few outages in the Florida area. While the circuit looks be be stabilizing, traffic will remain pointed to the New Jersey Zeus circuit. Updates to follow.

  5. identified Jan 15, 2025, 07:49 PM UTC

    Lumen is working to resolve the outages affecting the Zeus Florida circuit. Updates to follow.

  6. identified Jan 15, 2025, 09:09 PM UTC

    Zeus users may experience issues downloading call recordings from the portal, especially for calls older than 4 days. Updates to follow.

  7. resolved Jan 16, 2025, 02:07 PM UTC

    Incident has been resolved. Details from carrier: Root Cause: An equipment failure in Tampa, FL impacted services. Fix Action: Services were migrated to an alternate path and the affected card was replaced in Forest Park, GA, thus restoring services to a stable state. This service impact is over. Lumen Network Operations Center (NOC) management will be reviewing this outage for any additional analysis or discovery. In some cases, you may need to reset your services locally at your equipment. If a service issue continues after that reset, please contact the Lumen Repair Center for help.

  8. postmortem Jan 16, 2025, 03:44 PM UTC

    The Zeus Florida circuit experienced significant packet loss due to a fiber issue in Lumen's network, which affected customers in the Florida region. Upon identifying the root cause of the issue, the Viirtue team promptly redirected all traffic to the Zeus New Jersey datacenter to mitigate the impact. Lumen resolved the fiber issue early Thursday morning, after which the Viirtue team thoroughly tested the circuit. Once confirmed stable, traffic was successfully redirected back to the Zeus Florida datacenter.

Read the full incident report →

Major January 13, 2025

Apollo - Core2-NJ

Detected by Pingoru
Jan 13, 2025, 07:52 PM UTC
Resolved
Jan 13, 2025, 09:37 PM UTC
Duration
1h 44m
Affected: NJ2 Core Server
Timeline · 3 updates
  1. investigating Jan 13, 2025, 07:52 PM UTC

    Viirtue Engineers are currently investigating an issue affecting the Core2-nj Apollo server. Updates to follow.

  2. monitoring Jan 13, 2025, 08:05 PM UTC

    A solution has been implemented, and our engineering team will continue to monitor the system closely. Further updates will be provided as needed.

  3. resolved Jan 13, 2025, 09:37 PM UTC

    This incident has been resolved. More details will be listed in the post-mortem section of this incident.

Read the full incident report →

Notice December 2, 2024

Apollo - Poor QoS scores in the Portal

Detected by Pingoru
Dec 02, 2024, 04:56 PM UTC
Resolved
Dec 02, 2024, 04:56 PM UTC
Duration
—
Affected: Manager PortalAnalytics QoS
Timeline · 1 update
  1. resolved Dec 02, 2024, 04:56 PM UTC

    The QoS score discrepancies observed in the portal at approximately 10:30 AM Eastern have been resolved. Please note that these scores were inaccurate due to an issue with the analytics server and did not reflect actual call quality, which remained unaffected during this time.

Read the full incident report →

Minor October 11, 2024

Apollo - Button Builder

Detected by Pingoru
Oct 11, 2024, 03:39 PM UTC
Resolved
Oct 16, 2024, 01:13 PM UTC
Duration
4d 21h
Affected: Provisioning Server
Timeline · 4 updates
  1. identified Oct 11, 2024, 03:39 PM UTC

    Viirtue's engineering team is actively addressing an issue impacting the button builder within the manager portal. A temporary solution is currently in place as we collaborate with our partners to develop a permanent resolution.

  2. identified Oct 11, 2024, 07:23 PM UTC

    We are continuing to work on a fix for this issue. Updates to follow.

  3. identified Oct 14, 2024, 10:08 PM UTC

    Our engineers are working with partners on a solution and will provide an update once it’s implemented.

  4. resolved Oct 16, 2024, 01:13 PM UTC

    The NDP has been stabilized, and configuration adjustments will be implemented to prevent the recurrence of this issue.

Read the full incident report →

Major October 10, 2024

Apollo - Florida Data center

Detected by Pingoru
Oct 10, 2024, 01:35 AM UTC
Resolved
Oct 16, 2024, 01:09 PM UTC
Duration
6d 11h
Affected: Florida Data Center
Timeline · 8 updates
  1. investigating Oct 10, 2024, 01:35 AM UTC

    Viirtue engineers are actively investigating a circuit issue at the Florida data center, potentially linked to Hurricane Milton. At this time, the impact is limited to Apollo servers. We advise promptly redirecting any SIP trunk clients currently registered to sip01-fl.5060.cloud to sip01-nj.5060.cloud.

  2. identified Oct 10, 2024, 04:26 AM UTC

    Lumen has confirmed a significant outage within the Tampa metro ring impacting our circuit. Further updates will be provided as they become available.

  3. identified Oct 10, 2024, 12:41 PM UTC

    The latest update from Lumen is below. "2024-10-10 07:33:16 GMT - On October 10, 2024, Lumen discovered a service impact in Florida. Your customer trouble ticket has been related to a network event; all network events are automatically escalated to Lumen leadership."

  4. identified Oct 10, 2024, 08:45 PM UTC

    The latest update from Lumen is below. "Hurricane Milton made landfall as a Category 3 storm during the late overnight hours of October 9th, bringing 8-10 feet of storm surge, sustained hurricane-force winds, isolated tornadoes, and approximately 18 inches of rainfall. Currently, 3,406,056 consumers are without power in Florida. Lumen is closely monitoring Hurricane Milton's path and remains in contact with emergency management teams. Once it is deemed safe and commercial power has resumed, Field Operations will be deployed to assess damages and devise a repair plan."

  5. identified Oct 11, 2024, 09:37 PM UTC

    Lumen Field Operations will be dispatched to work on the outage once emergency personnel deem it safe for travel. We will continue to post updates as we receive them.

  6. monitoring Oct 14, 2024, 12:57 PM UTC

    Viirtue engineers are actively monitoring circuit stability following restoration over the weekend. The circuit has remained stable, and further updates will be provided as necessary.

  7. monitoring Oct 15, 2024, 05:22 PM UTC

    The Viirtue engineering team will conduct a database migration during tonight’s scheduled maintenance window to restore data on the Florida servers. Upon successful completion, the servers will be reintroduced into production.

  8. resolved Oct 16, 2024, 01:09 PM UTC

    The Lumen circuit has stabilized, and the Florida servers have been restored to active production status.

Read the full incident report →

Minor September 25, 2024

Intermittent Network Connectivity to Las Vegas Datacenter

Detected by Pingoru
Sep 25, 2024, 03:20 PM UTC
Resolved
Sep 26, 2024, 02:47 AM UTC
Duration
11h 26m
Affected: LV Core ServerLV QoS ServerLV Recording Server
Timeline · 5 updates
  1. investigating Sep 25, 2024, 03:20 PM UTC

    Viirtue has identified intermittent network connectivity issues affecting our Las Vegas Datacenter. Our teams are actively investigating the cause of the disruption and are working to restore stable connectivity as soon as possible. We will provide further updates as more information becomes available. Thank you for your patience.

  2. identified Sep 25, 2024, 03:55 PM UTC

    We have identified the network issue affecting the connection between the New Jersey and Las Vegas data centers and are actively collaborating with our ISP's to resolve it. Please note that this issue is not expected to impact call quality or traffic at either data center.

  3. identified Sep 25, 2024, 06:10 PM UTC

    Virtue Engineers have temporarily "503'd" the Las Vegas core while addressing the network issue between New Jersey and Las Vegas. We do not anticipate any service disruptions at the New Jersey or Florida data centers as a result of this change.

  4. resolved Sep 26, 2024, 02:47 AM UTC

    The network issue between the Las Vegas and New Jersey data centers has been resolved. Additional details will be provided in the postmortem report.

  5. postmortem Sep 26, 2024, 02:47 AM UTC

    A network issue involving a major backbone carrier, Twelve99/Arelion, resulted in significant connectivity disruptions between our New Jersey and Las Vegas data centers. Arelion identified the root cause as "congestion in our backbone between Los Angeles and Ashburn." This delay led to SBUS queuing on both the New Jersey and Las Vegas core systems. To mitigate potential call quality issues caused by the SBUS delays, Viirtue engineers implemented a temporary 503 status on the Las Vegas core server at around 1:30 PM EST. Arelion resolved the congestion at approximately 6:50 PM EST, and Viirtue engineers observed improvements in connectivity shortly thereafter. The 503 status was lifted from the Las Vegas core server at around 10:30 PM EST.

Read the full incident report →

Minor September 6, 2024

Apollo Portal Down

Detected by Pingoru
Sep 06, 2024, 03:48 PM UTC
Resolved
Sep 06, 2024, 04:02 PM UTC
Duration
14m
Affected: Manager Portal
Timeline · 3 updates
  1. monitoring Sep 06, 2024, 03:48 PM UTC

    Viirtue engineers have implemented a fix for the portal issue. The portal is accessible and performance is improving. Updates to follow.

  2. resolved Sep 06, 2024, 04:02 PM UTC

    This incident has been resolved.

  3. postmortem Sep 06, 2024, 04:04 PM UTC

    At approximately 11:35 AM EST, an unexpected spike in Apache workers caused the portal to become unresponsive. The Viirtue engineering team promptly addressed the issue by restarting the Apache service, which successfully released the workers and restored service functionality. Operations began to stabilize shortly after the restart. We are currently conducting a root cause analysis \(RCA\) to prevent similar incidents in the future.

Read the full incident report →

Minor August 29, 2024

Icons Missing in ViiBE

Detected by Pingoru
Aug 29, 2024, 11:42 AM UTC
Resolved
Aug 29, 2024, 11:51 AM UTC
Duration
9m
Affected: ViiBE
Timeline · 3 updates
  1. investigating Aug 29, 2024, 11:42 AM UTC

    We're seeing errors from an upstream font foundry which we use for icons (Font Awesome). We are actively investigating.

  2. resolved Aug 29, 2024, 11:51 AM UTC

    The font issue has been resolved. Post mortem to follow.

  3. postmortem Aug 29, 2024, 11:51 AM UTC

    Parts of ViiBE use FontAwesome \(Font Foundry\) for icon sourcing. On August 29th at 7:42 AM EDT, the engineering team encountered 403 errors from FontAwesome and began investigating. It was quickly identified that Viirtue's allotted monthly page views \(as tracked by FontAwesome\) had been exhausted, necessitating an upgrade to a higher tier. We have increased the allotment by a factor of 100X to support ViiBE's continued growth and ensure uninterrupted adoption.

Read the full incident report →

Minor August 14, 2024

Manager Portal Errors

Detected by Pingoru
Aug 14, 2024, 12:27 PM UTC
Resolved
Aug 14, 2024, 12:34 PM UTC
Duration
7m
Affected: Manager Portal
Timeline · 3 updates
  1. investigating Aug 14, 2024, 12:27 PM UTC

    There is currently an issue causing API logs to appear in the portal. Viirtue engineers are aware of the issue and working on a fix. Updates to follow.

  2. monitoring Aug 14, 2024, 12:30 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved Aug 14, 2024, 12:34 PM UTC

    This incident has been resolved.

Read the full incident report →

Minor August 7, 2024

Apollo Call Recordings

Detected by Pingoru
Aug 07, 2024, 07:31 PM UTC
Resolved
Aug 07, 2024, 10:03 PM UTC
Duration
2h 32m
Affected: Manager Portal
Timeline · 3 updates
  1. identified Aug 07, 2024, 07:31 PM UTC

    The Apollo cluster is experiencing an issue downloading call recording files from the portal. Our engineers are working on the issue and should have a fix soon. Updates to follow.

  2. monitoring Aug 07, 2024, 09:36 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved Aug 07, 2024, 10:03 PM UTC

    This incident has been resolved.

Read the full incident report →