Viirtue experienced a critical incident on July 12, 2025 affecting NJ2 Core Server and Recording NJ, lasting 13h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 12, 2025, 12:24 PM UTC
Our monitoring systems have identified an issue impacting the NJ data center. Our engineering team is actively investigating and working to resolve the incident.
- identified Jul 12, 2025, 12:42 PM UTC
Our engineers have traced the issue to the data center and are coordinating with on-site staff for deeper diagnostics and resolution. The data center reported ongoing preventative UPS maintenance, though it’s not yet confirmed to be related. All services, including the portal, remain fully operational through our redundant Florida data center.
- identified Jul 12, 2025, 12:43 PM UTC
We are continuing to work on a fix for this issue.
- identified Jul 12, 2025, 02:21 PM UTC
Our engineers, in collaboration with on-site data center staff, are working to ensure full service restoration. All possible services have been migrated to our Florida data center, and certificate restoration for white-label partners is underway. All other services remain operational. The data center provider has provided an estimated time of resolution for NJ normalization by 1 PM EST.”
- monitoring Jul 12, 2025, 03:07 PM UTC
All services are fully operational in our secondary data center.
- monitoring Jul 12, 2025, 06:45 PM UTC
We will be performing scheduled maintenance on our NJ infrastructure today at 5 PM EST. All services will remain fully operational and will continue running from our Florida data center during this time. The customer portal will also remain accessible and will continue to be served from our FL data center.
- monitoring Jul 12, 2025, 09:07 PM UTC
starting maintenance
- monitoring Jul 13, 2025, 12:46 AM UTC
All services have been fully restored in our NJ data center. We are currently in the process of normalizing the environment, including migrating DNS records back to NJ.
- resolved Jul 13, 2025, 01:24 AM UTC
All services have been restored and are operational.
- postmortem Jul 22, 2025, 08:56 PM UTC
**Incident Report:** NJ Data Center Maintenance and Outage **Date of Incident:** July 12, 2025 **Prepared by:** Alberto Fernandez, Director of Platform Engineering **Start Time:** 7:53 AM EST **Resolution:** 8:15 PM EST **Impacted Services**: Apollo Services \(NJ Region\) **Customer Impact:** Possible voice degradation, call setup delays, or dropped sessions during rerouting ## Executive Summary At approximately 7:53 AM EST, our monitoring systems detected that several locations were unable to reach our New Jersey data center. Our team immediately initiated an investigation with both our ISP and the data center provider. We discovered that a scheduled maintenance procedure involving the Uninterruptible Power Supply \(UPS\) had not gone as planned. Although the maintenance was expected to have no impact, it resulted in full connectivity loss at the site. To mitigate impact, we quickly rerouted traffic and relocated portal services to our Florida data center. This transition began at 8:05 AM, with the reseller portal fully redirected by 8:15 AM. While the data center had issued a notification about the maintenance, the notice included an incorrect date, August 11, and stated there would be no service interruption. ## Root Cause The outage was directly caused by the data center’s UPS maintenance. While intended to be non-disruptive, the maintenance led to a full outage of inbound and outbound connectivity. We are awaiting a detailed report from the data center. ## Remediation and Preventative Measures * We are working closely with the data center provider to ensure future maintenance notices are timely, accurate, and clearly indicate any potential for impact. * All third-party maintenance affecting core infrastructure will now be reviewed under our internal change management protocols, even when the work is outside of our direct control. ## Customer Communication We understand that even short periods of service instability can disrupt your business. We remain committed to reliability, clear communication, and continuous improvement across our platform. If you have any questions or concerns, or would like to discuss this incident in more detail, please reach out to our support team directly. We value your partnership and thank you for your continued trust.