Viirtue Outage History

Started Sep 02, 2026, 11:48 PM UTC· resolved Sep 03, 2026, 12:57 AM UTC

1. investigating Sep 02, 2026, 11:48 PM UTC

   Starting at approximately 22:46 UTC (6:46 PM ET) on September 2, ViiBE portals are intermittently unreachable. Users may see connection failures or a "no healthy upstream" error. The cause is a production Kubernetes node that is repeatedly going unhealthy, which is disrupting the ingress gateway that serves these sites. We have identified the failing node and are working to move traffic off it. Next update in 30 minutes.

2. investigating Sep 02, 2026, 11:49 PM UTC

   We are continuing to investigate this issue.

3. monitoring Sep 03, 2026, 12:15 AM UTC

   As of 5:12 PM PT, ViiBE services have restarted and the portals are loading again. A production Kubernetes node in the cloud provider failed and repeatedly dropped offline, which took the ingress gateway and several backend services with it. We added capacity, removed the failing node, and moved all workloads to healthy nodes. We are monitoring closely while services finish recovering.

4. resolved Sep 03, 2026, 12:57 AM UTC

   A production Kubernetes node in the cloud provider failed and repeatedly dropped offline, which took the ingress gateway and several backend services with it. We added capacity, removed the failing node, and moved all workloads to healthy nodes.

Read the full incident report →

Started Aug 11, 2026, 02:10 PM UTC· resolved Aug 12, 2026, 05:03 AM UTC

1. investigating Aug 11, 2026, 02:10 PM UTC

   We have identified a incident affecting call queueing on the Apollo Cluster, we are investigating and will provide updates as they are available.

2. identified Aug 11, 2026, 02:19 PM UTC

   We have found that the issue is primarily with queueing on the Florida Core, we are working to resolve the issue with our infrastructure team

3. identified Aug 11, 2026, 02:59 PM UTC

   Our Infrastructure team is actively working to resolve the issue and is treating it as a high priority. We will provide our next update within the hour, or sooner if there are any significant developments.

4. monitoring Aug 11, 2026, 03:37 PM UTC

   After implementing a VPN routing tunnel we've observed improved performance with the server and are monitoring the issue.

5. monitoring Aug 11, 2026, 04:24 PM UTC

   We are still seeing continued issues with the Florida server, our infrastructure team is monitoring and assessing next steps

6. monitoring Aug 11, 2026, 04:38 PM UTC

   We are still monitoring the issue but we have identified that the Vegas server is now affected as it processes the backlog of requests coming from FL. We expect improved performance as this backlog is processed.

7. monitoring Aug 11, 2026, 05:41 PM UTC

   Good afternoon team, Thank you for your patience on this issue. After continued troubleshooting we have identified a long term fix, and we currently are investigating implementation. The issue identified was a network issue locally to the Florida core, which caused backlog queueing. Once the fix is implemented, we will post a further update.

8. monitoring Aug 11, 2026, 06:37 PM UTC

   We are working on implementing another VPN re-route to alleviate the queueing issue. We will update when this has been completed.

9. monitoring Aug 11, 2026, 07:39 PM UTC

   We have successfully implemented the alternate route and are seeing the queuing clear. We will continue to monitor the this afternoon but function should be restored.

10. monitoring Aug 11, 2026, 09:57 PM UTC

    We are actively monitoring the incident but we believe the current effect should be minimal.

11. resolved Aug 12, 2026, 05:03 AM UTC

    This incident has been resolved.

Read the full incident report →

Started Jul 04, 2026, 01:20 AM UTC· resolved Jul 04, 2026, 02:21 AM UTC

1. monitoring Jul 04, 2026, 01:20 AM UTC

   The network congestion event between the New Jersey and Florida data centers has been resolved. During the event, some geo-routed calls failed, and registration statuses were reported incorrectly across the cluster. Our engineering team will continue to closely monitor network health to ensure service remains stable.

2. resolved Jul 04, 2026, 02:21 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 16, 2026, 02:45 AM UTC· resolved Jun 16, 2026, 02:03 PM UTC

1. investigating Jun 16, 2026, 02:45 AM UTC

   We experienced a brief network interruption at our Las Vegas location from 02:07 UTC to 02:12 UTC. Connectivity has been restored and all services are operational. We are monitoring closely and following up with the provider.

2. resolved Jun 16, 2026, 02:03 PM UTC

   The network interruption was caused by a switch port flap in the providers network. This has been monitored and is confirmed stable.

Read the full incident report →

Started Jun 03, 2026, 01:58 PM UTC· resolved Jun 03, 2026, 03:12 PM UTC

1. investigating Jun 03, 2026, 01:58 PM UTC

   Description: We are currently investigating an issue causing UUID errors when attempting to open faxes within the web portal. Updates to follow. Impact: Customers may experience issues with opening faxes. No reports of sending issues at this time. Next Update: We will provide our next update by 11:00AM EST or sooner if new information becomes available.

2. monitoring Jun 03, 2026, 02:29 PM UTC

   We have worked with our fax vendor, identified a root cause, and deployed a solution to the faxes not appearing in the portal. Our team is monitoring the results, and we will continue to track this event.

3. resolved Jun 03, 2026, 03:12 PM UTC

   This incident has been resolved. Please reach out to the Viirtue support team if you have any further questions or concerns.

Read the full incident report →

Started May 16, 2026, 04:14 PM UTC· resolved May 16, 2026, 04:40 PM UTC

1. investigating May 16, 2026, 04:14 PM UTC

   We are currently investigating this issue.

2. identified May 16, 2026, 04:37 PM UTC

   The issue has been identified and a fix is being implemented.

3. resolved May 16, 2026, 04:40 PM UTC

   This incident has been resolved.

4. postmortem May 16, 2026, 04:43 PM UTC

   During a routine Kubernetes cluster upgrade activity, a critical workload dependency impacting RabbitMQ availability failed to initialize successfully. Investigation determined the issue was caused by the removal of an upstream container image from Docker Hub, preventing the cluster from pulling the required image during workload startup. The issue was identified quickly, and service was restored by updating the workload configuration to reference an alternate trusted image source containing the same image artifact. Total impact duration was minimal, and no data loss occurred. This incident highlighted a broader software supply chain dependency risk associated with externally hosted container images. As a result, we are reviewing additional safeguards around image provenance, availability, and internal artifact mirroring to further reduce exposure to upstream registry changes.

Read the full incident report →

Started Apr 20, 2026, 03:01 AM UTC· resolved Apr 20, 2026, 04:43 AM UTC

1. identified Apr 20, 2026, 03:01 AM UTC

   We are aware of an issue causing MobileConnect devices to register with an incorrect suffix, and we have escalated it with our partners. In the meantime, for any call queue users utilizing MobileConnect, the “m” device must be added to the queue to ensure they can receive queue calls until the issue is resolved.

2. resolved Apr 20, 2026, 04:43 AM UTC

   The incident has been resolved. If you experience any issues receiving calls on the MobileConnect app, please log out of the app and then log back in.

Read the full incident report →

Started Apr 08, 2026, 02:51 PM UTC· resolved Apr 08, 2026, 02:51 PM UTC

1. resolved Apr 08, 2026, 02:51 PM UTC

   The Core2-NJ portal became unavailable due to an invalid Apache certificate. The certificate was promptly removed, and service was restored within minutes.

Read the full incident report →

Started Dec 02, 2025, 08:12 PM UTC· resolved Dec 02, 2025, 08:12 PM UTC

1. resolved Dec 02, 2025, 08:12 PM UTC

   At approximately 2:46 PM EST, the core1-TPA server experienced a crash but recovered quickly and is now fully operational. Any calls handled by this server at the time of the crash would have failed.

Read the full incident report →

Started Nov 21, 2025, 03:05 PM UTC· resolved Nov 21, 2025, 04:03 PM UTC

1. investigating Nov 21, 2025, 03:05 PM UTC

   We are aware of an issue affecting new registrations and outbound calls for Mobile Connect users. Our engineers are actively working to resolve it.

2. identified Nov 21, 2025, 03:26 PM UTC

   We discovered an issue in the Mobile Connect middleware update deployed last night. To prevent any disruption to your service, we are rolling back to the last known stable version. Our team is actively monitoring the process and will share updates as soon as they are available.

3. monitoring Nov 21, 2025, 03:37 PM UTC

   The rollback has been completed and registrations for mobile connect are starting to show as valid. We are monitoring progress. Please open and close the mobile connect application if you experience an issue to force re-registration.

4. resolved Nov 21, 2025, 04:03 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 10, 2025, 09:02 PM UTC· resolved Nov 11, 2025, 12:38 AM UTC

1. investigating Nov 10, 2025, 09:02 PM UTC

   At about 3:40 pm est our NJ and LV core crashed for a unknown reason, and we dropped about 1k live calls. The server auto recovered and is operational again. We are currently investigating the cause for this crash and will take necessary corrective action.

2. resolved Nov 11, 2025, 12:38 AM UTC

   This incident has been resolved.

3. postmortem Nov 11, 2025, 12:39 AM UTC

   The crash condition observed has been identified as a bug by NetSapiens related to v44. They have tagged it with Jira NMS-3558 in which a rare condition causes the sipbx service to crash. We are actively working with the vendor to implement a patch as soon as one is available.

Read the full incident report →

Started Oct 13, 2025, 03:29 PM UTC· resolved Oct 13, 2025, 03:37 PM UTC

1. investigating Oct 13, 2025, 03:29 PM UTC

   We are currently investigating the issue and working to restore service

2. resolved Oct 13, 2025, 03:37 PM UTC

   Portal service should be restored at this time.

Read the full incident report →

Started Sep 29, 2025, 08:14 PM UTC· resolved Sep 29, 2025, 08:29 PM UTC

1. investigating Sep 29, 2025, 08:14 PM UTC

   We are currently investigating reports of portal degradation, or slowness.

2. monitoring Sep 29, 2025, 08:18 PM UTC

   We have identified a load spike at 4PM, we have it contained, and we are monitoring system health.

3. resolved Sep 29, 2025, 08:29 PM UTC

   We are currently investigating the source of the spike at 4PM. At this time, we are monitoring all systems. Services are restored, and no further impact is expected.

Read the full incident report →

Started Sep 27, 2025, 11:46 AM UTC· resolved Sep 27, 2025, 05:30 PM UTC

1. investigating Sep 27, 2025, 11:46 AM UTC

   We are currently investing a portal outage occurring with our NJ server.

2. identified Sep 27, 2025, 11:51 AM UTC

   Our NJ Data Center is experiencing an outage. We are following up with the data center on status and an ETR.

3. identified Sep 27, 2025, 12:30 PM UTC

   Our rack in the data center is experiencing a power outage and we are being told that worst case scenario on restoration is 1PM EST. We are diverting portal traffic to Florida and will provide the next update at 1PM or sooner if we receive word from the DC that power has been restored.

4. identified Sep 27, 2025, 12:51 PM UTC

   Portal traffic has been fully diverted to Florida and the portal should be available now. Further updates on this incident will be provided by 1PM or sooner.

5. identified Sep 27, 2025, 03:51 PM UTC

   Our engineers are still working on restoring service and have confirmed that the portal, web phone, and SIP trunks should be operational at this time. They are still seeing Mobile Connect down and are working to restore that as soon as possible. Further updates regarding Mobile Connect will be posted as soon as they become available.

6. monitoring Sep 27, 2025, 04:27 PM UTC

   All services have been restored. If you are still having issues with the web phone, please refresh the page or log out and back in. Complete RCA will be forthcoming within 5-7 business days.

7. resolved Sep 27, 2025, 05:30 PM UTC

   all services have been restore

Read the full incident report →

Started Sep 18, 2025, 05:05 PM UTC· resolved Sep 18, 2025, 07:50 PM UTC

1. investigating Sep 18, 2025, 05:05 PM UTC

   We are currently observing a degradation of portal service we are actively investigating and working on a solution

2. investigating Sep 18, 2025, 05:23 PM UTC

   We are actively pursuing the cause of the incident and have engaged our engineering partners. We are actively working to stabilize the portal, and the NJ core. updates to follow as we continue investigation.

3. investigating Sep 18, 2025, 05:42 PM UTC

   We are working on redirecting traffic to improve service

4. investigating Sep 18, 2025, 05:48 PM UTC

   We are seeing the system recovering and performance should be improving

5. monitoring Sep 18, 2025, 05:54 PM UTC

   System performance should be improved we we are continuing to monitor

6. monitoring Sep 18, 2025, 06:50 PM UTC

   We are still investigating the cause of the incident. Service should be restored at this time

7. resolved Sep 18, 2025, 07:50 PM UTC

   We are continuing to investigate the cause of the issue and continuing to monitor portal performance. We will be publishing RCA once we have completed our investigation

Read the full incident report →

Started Aug 19, 2025, 04:23 PM UTC· resolved Aug 19, 2025, 04:52 PM UTC

1. investigating Aug 19, 2025, 04:23 PM UTC

   We are currently experiencing an issue effecting SBUS queueing relating to both NJ and FL cores. Our team is actively working to resolve this issue. We are engaging with our Platform Engineering team to diagnose. We will give more informed updates as information becomes available. Impact: Customers may experience delays with CDR's, VM's, Email issues, and possible QOS issues if GEO-Calling is present.

2. identified Aug 19, 2025, 04:37 PM UTC

   Team, We have identified the structural issue, and after testing, the VM function is working as intended. Please re-test the VM functionality and update your tickets as needed.

3. resolved Aug 19, 2025, 04:52 PM UTC

   Good afternoon Everyone, After Platform Engineering implemented structural solutions, this issue has been resolved. Please notify all customers impacted, and we will post a more in-depth Post-Modem to the status page.

4. postmortem Aug 29, 2025, 04:22 PM UTC

   Our engineering team identified a configuration error in the dial translation for the Cloud PBX voicemail feature, which prevented calls from being routed to the correct location. The issue was quickly corrected, and additional safeguards are being implemented to prevent this from occurring in the future. During this time, regular SBUS queueing also occurred and quickly subsided, though no correlation between the two issues has been determined.

Read the full incident report →

Started Aug 13, 2025, 02:21 PM UTC· resolved Aug 14, 2025, 12:17 PM UTC

1. investigating Aug 13, 2025, 02:21 PM UTC

   Our zeus portal is currently unresponsive, we are currently investigating and re-routing

2. investigating Aug 13, 2025, 02:33 PM UTC

   Update: We have re-routed traffic to NJ to restore services to the Zeus portal Internal Reference: Jira Ticket: KIT-47 Next Update: We are actively investigating the cause of the issue and have implemented a temporary solution

3. monitoring Aug 13, 2025, 02:44 PM UTC

   Update: We have re-routed traffic to NJ to restore services to the Zeus portal Internal Reference: Jira Ticket: KIT-47 Next Update: We are continuing to investigate and will provide a RCA when we have completed our investigation

4. resolved Aug 14, 2025, 12:17 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 06, 2025, 07:04 PM UTC· resolved Aug 06, 2025, 07:40 PM UTC

1. investigating Aug 06, 2025, 07:04 PM UTC

   We have noticed connectivity micro cut in Florida and it has recovered. We are still monitoring.

2. monitoring Aug 06, 2025, 07:06 PM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring Aug 06, 2025, 07:08 PM UTC

   We are continuing to monitor for any further issues.

4. resolved Aug 06, 2025, 07:40 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 04, 2025, 06:04 PM UTC· resolved Aug 05, 2025, 12:35 AM UTC

1. investigating Aug 04, 2025, 06:04 PM UTC

   We are currently investigating an issue affecting the Apollo Florida Lumen circuit. Updates to follow.

2. monitoring Aug 04, 2025, 06:18 PM UTC

   The Lumen circuit has stabilized, and traffic has returned to the Florida datacenter. We will continue to monitor the circuit and provide updates as needed.

3. resolved Aug 05, 2025, 12:35 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Jul 24, 2025, 04:53 PM UTC· resolved Jul 24, 2025, 05:09 PM UTC

1. investigating Jul 24, 2025, 04:53 PM UTC

   We have identified an issue with Florida server. The engineers are actively investigating at the moments. More updates will follow.

2. resolved Jul 24, 2025, 05:09 PM UTC

   A power issue affected the Core2-FL server but has since been resolved. Traffic automatically failed over to the Las Vegas core shortly after the outage occurred. Service has now been fully restored on Core2-FL. A detailed postmortem will be provided at a later time.

Read the full incident report →

Started Jul 22, 2025, 08:49 PM UTC· resolved Jul 24, 2025, 06:47 PM UTC

1. identified Jul 22, 2025, 08:49 PM UTC

   Devices registered to the core2-fl Apollo core will not capture recordings, unless the happens to be a GEO call. We have escalated the issue with Netsapiens and will provide an update soon.

2. identified Jul 22, 2025, 11:35 PM UTC

   We will be implementing changes on our recording servers based on recommendations from NetSapiens. Further updates will be provided once the changes have been applied and tested.

3. monitoring Jul 23, 2025, 01:21 AM UTC

   We’ve implemented configuration changes based on NetSapiens’ recommendations and are actively monitoring the system to assess their impact.

4. monitoring Jul 23, 2025, 01:28 PM UTC

   We are seeing calls successfully recording on Core2-FL. We will continue to monitor and provide updates throughout the day.

5. monitoring Jul 23, 2025, 07:04 PM UTC

   We are seeing significant improvements and will continue to monitor the issue. Updates to follow.

6. monitoring Jul 24, 2025, 02:40 PM UTC

   We are continuing to see improvements in call recording conversion speed and reliability. Updates to follow.

7. resolved Jul 24, 2025, 06:47 PM UTC

   The issue has been resolved. Calls on Core2-FL are now being recorded reliably and within standard timing expectations.

Read the full incident report →

Started Jul 22, 2025, 05:52 PM UTC· resolved Jul 22, 2025, 08:46 PM UTC

1. investigating Jul 22, 2025, 05:52 PM UTC

   Our engineers are currently working on an issue we are experiencing with unassigned numbers hosted in Bandwidth. Updates to follow.

2. monitoring Jul 22, 2025, 06:24 PM UTC

   We are implementing a change at the carrier level that should resolve the issues we are seeing with unassigned numbers in Bandwidth. Updates to follow.

3. monitoring Jul 22, 2025, 07:22 PM UTC

   We are continuing to monitor the impact of the recent changes. Updates to follow.

4. resolved Jul 22, 2025, 08:46 PM UTC

   Bandwidth implemented changes to our trunk to reduce forced retries for inactive or disabled numbers. We expect to see continued improvements moving forward.

Read the full incident report →

Started Jul 21, 2025, 03:21 PM UTC· resolved Jul 23, 2025, 01:25 AM UTC

1. investigating Jul 21, 2025, 03:21 PM UTC

   Following the maintenance performed on Friday night, we observed significant improvement in the performance of recording-LV. However, we’ve recently identified that the service has begun exhibiting intermittent issues again. This will mostly impact calls processed in core2-FL & LV. Our engineering team is actively investigating the root cause with the highest priority to restore full stability as quickly as possible. We appreciate your patience and will continue to provide updates as we learn more.

2. resolved Jul 23, 2025, 01:25 AM UTC

   This issue is related to the ongoing Florida Recording incident. We are closing this incident, and all updates will be provided under the active incident record.

Read the full incident report →

Started Jul 14, 2025, 03:06 PM UTC· resolved Jul 14, 2025, 06:09 PM UTC

1. investigating Jul 14, 2025, 03:06 PM UTC

   We are currently investigating an issue causing the manager portal to load slowly. Updates to follow.

2. investigating Jul 14, 2025, 03:06 PM UTC

   We are continuing to investigate this issue.

3. investigating Jul 14, 2025, 03:12 PM UTC

   Due to ongoing connectivity issues at our NJ data center, we are proactively returning HTTP 503 responses to divert traffic away as a preventative measure. Services remain operational via our redundant infrastructure.

4. investigating Jul 14, 2025, 03:22 PM UTC

   All voice and web portal traffic directed to the core2-nj core has been redirected to our other cores. Service should improve while we work on finding the root cause.

5. investigating Jul 14, 2025, 04:35 PM UTC

   We are aware of the audio issue affecting a subset of calls on the Las Vegas core and are investigating the issue.

6. monitoring Jul 14, 2025, 04:51 PM UTC

   The 503 has been removed from core2-nj, and voice traffic has been redirected to this core.

7. monitoring Jul 14, 2025, 05:26 PM UTC

   We’re continuing to work through service stabilization efforts across both regions. Teams remain engaged and progress is ongoing.

8. monitoring Jul 14, 2025, 05:50 PM UTC

   We’ve pinpointed the root cause and are currently validating the fix.

9. resolved Jul 14, 2025, 06:09 PM UTC

   All services have been fully restored and are now operational. An incident report will be shared later today, with a full root cause analysis to follow.

10. postmortem Jul 15, 2025, 04:03 PM UTC

    ## Update #### Date: July 18th, 2025 1:09PM EDT A complete Root Cause Analysis \(RCA\) has been prepared and will be provided to all partners and customers with active support tickets related to this incident. If you do not currently have an open ticket but would like to request a copy of the RCA, please contact your account manager or submit a support ticket. ## Incident Report: NJ Data Center Network Event & Related Service Impact #### Date of Incident: July 14, 2025 #### Prepared by: Alberto Fernandez \(Director of Platform Engineering\) #### Start Time: 10:53 AM EST #### Partial Restoration Time 11:10 AM EST #### Full Resolution Time: 2:09 PM EST #### Services Impacted: Apollo services across NJ, FL, and Las Vegas data centers #### Customer Impact: Potential voice degradation, call setup delays, or dropped sessions for some users during routing transitions. ### Executive Summary At 10:53 AM EST, our monitoring systems detected a sharp and unusual increase in SBUS queue activity within our NJ data center. This triggered an immediate alert to our Network Operations and Engineering teams, who launched a rapid response investigation. As a precautionary and proactive measure, we rerouted all real-time traffic away from NJ to our geographically redundant data centers in Las Vegas and Florida. This action ensured continuity of service while we investigated the root cause of the anomaly in NJ. During the transition period, we identified two additional, unrelated issues affecting the Vegas and Florida clusters. Each was diagnosed and mitigated independently. All systems are now fully operational and stable across all locations. Further details on these issues will be provided in the final RCA. ### Potential Root Cause The incident was triggered by a transient physical network cut in our NJ data center’s upstream routing path. While the disruption lasted approximately 5 minutes, it had downstream effects on signaling queues \(SBUS\) and session routing. The system behaved as designed by rerouting traffic through redundant sites, but the rapid load redistribution temporarily stressed configuration limits which temporarily impacted service quality. ### Initial Remediation * NJ Network Cut Mitigation: Root cause confirmed with upstream provider. Cable rerouted and physical safeguards enhanced. * SBUS Monitoring: Sensitivity tuned for improved queue behavior detection. ### Customer Communication We recognize that service interruptions, even brief ones, can be disruptive to your operations. We remain committed to transparency, reliability, and continuous improvement. ### Next Steps * Perform full internal post-incident review. * Continue monitoring all three data centers at increased granularity for the next 72 hours. * Deliver a final Root Cause Analysis \(RCA\) document within 7 business days. If you have any further questions or would like to discuss the incident in more detail, please contact our support team. Thank you for your continued trust.

Read the full incident report →

Started Jul 12, 2025, 12:24 PM UTC· resolved Jul 13, 2025, 01:24 AM UTC

1. investigating Jul 12, 2025, 12:24 PM UTC

   Our monitoring systems have identified an issue impacting the NJ data center. Our engineering team is actively investigating and working to resolve the incident.

2. identified Jul 12, 2025, 12:42 PM UTC

   Our engineers have traced the issue to the data center and are coordinating with on-site staff for deeper diagnostics and resolution. The data center reported ongoing preventative UPS maintenance, though it’s not yet confirmed to be related. All services, including the portal, remain fully operational through our redundant Florida data center.

3. identified Jul 12, 2025, 12:43 PM UTC

   We are continuing to work on a fix for this issue.

4. identified Jul 12, 2025, 02:21 PM UTC

   Our engineers, in collaboration with on-site data center staff, are working to ensure full service restoration. All possible services have been migrated to our Florida data center, and certificate restoration for white-label partners is underway. All other services remain operational. The data center provider has provided an estimated time of resolution for NJ normalization by 1 PM EST.”

5. monitoring Jul 12, 2025, 03:07 PM UTC

   All services are fully operational in our secondary data center.

6. monitoring Jul 12, 2025, 06:45 PM UTC

   We will be performing scheduled maintenance on our NJ infrastructure today at 5 PM EST. All services will remain fully operational and will continue running from our Florida data center during this time. The customer portal will also remain accessible and will continue to be served from our FL data center.

7. monitoring Jul 12, 2025, 09:07 PM UTC

   starting maintenance

8. monitoring Jul 13, 2025, 12:46 AM UTC

   All services have been fully restored in our NJ data center. We are currently in the process of normalizing the environment, including migrating DNS records back to NJ.

9. resolved Jul 13, 2025, 01:24 AM UTC

   All services have been restored and are operational.

10. postmortem Jul 22, 2025, 08:56 PM UTC

    **Incident Report:** NJ Data Center Maintenance and Outage **Date of Incident:** July 12, 2025 **Prepared by:** Alberto Fernandez, Director of Platform Engineering **Start Time:** 7:53 AM EST **Resolution:** 8:15 PM EST **Impacted Services**: Apollo Services \(NJ Region\) **Customer Impact:** Possible voice degradation, call setup delays, or dropped sessions during rerouting ## Executive Summary At approximately 7:53 AM EST, our monitoring systems detected that several locations were unable to reach our New Jersey data center. Our team immediately initiated an investigation with both our ISP and the data center provider. We discovered that a scheduled maintenance procedure involving the Uninterruptible Power Supply \(UPS\) had not gone as planned. Although the maintenance was expected to have no impact, it resulted in full connectivity loss at the site. To mitigate impact, we quickly rerouted traffic and relocated portal services to our Florida data center. This transition began at 8:05 AM, with the reseller portal fully redirected by 8:15 AM. While the data center had issued a notification about the maintenance, the notice included an incorrect date, August 11, and stated there would be no service interruption. ## Root Cause The outage was directly caused by the data center’s UPS maintenance. While intended to be non-disruptive, the maintenance led to a full outage of inbound and outbound connectivity. We are awaiting a detailed report from the data center. ## Remediation and Preventative Measures * We are working closely with the data center provider to ensure future maintenance notices are timely, accurate, and clearly indicate any potential for impact. * All third-party maintenance affecting core infrastructure will now be reviewed under our internal change management protocols, even when the work is outside of our direct control. ## Customer Communication We understand that even short periods of service instability can disrupt your business. We remain committed to reliability, clear communication, and continuous improvement across our platform. If you have any questions or concerns, or would like to discuss this incident in more detail, please reach out to our support team directly. We value your partnership and thank you for your continued trust.

Read the full incident report →