NUSO experienced a major incident on July 16, 2025 affecting UCaaS - Green Platform, lasting 9d. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 16, 2025, 12:33 PM UTC
We are currently experiencing an issue with the X3 instance on our Green platform and are actively working to resolve it. All other instances remain fully operational. Resellers who manage their own DNS records may need to update them to route traffic to available instances during this outage.
- identified Jul 16, 2025, 04:39 PM UTC
We have identified an issue impacting the X3 instance on our Green platform. Our internal systems remain unaffected due to built-in redundancy, and no disruption is being observed across other instances. Resellers who manage their own DNS and have not yet updated their records away from the X3 instance may experience service interruptions. All other services continue to operate normally.
- identified Jul 17, 2025, 07:32 PM UTC
Due to the nature of the issue with the X3 instance, it needs to be rebuilt in coordination with the manufacturer. This work will be performed over the weekend with the goal of restoring service Monday morning. All other instances continue to operate normally. We will provide updates once the rebuild is completed. Thank you for your patience during this extended maintenance period.
- monitoring Jul 21, 2025, 03:01 PM UTC
The rebuild is now fully complete, and all systems are ready. However, final data replication is taking a bit longer than expected. To ensure a smooth transition and avoid any potential disruption, we’re holding off on shifting live traffic until overnight tonight. We expect everything to be fully live by early tomorrow morning.
- monitoring Jul 21, 2025, 03:01 PM UTC
We are continuing to monitor for any further issues.
- monitoring Jul 22, 2025, 07:21 PM UTC
All rebuild work remains complete, and systems are fully prepared for cutover. However, final data synchronization is still in progress and requires additional time to ensure consistency across environments. To avoid any service impact, we are extending the transition window. The go-live is now targeted for the end of this week. We’ll continue to monitor progress closely and will provide another update ahead of the final cutover.
- resolved Jul 25, 2025, 01:13 PM UTC
All rebuild activities are fully complete, and the systems have been online and handling traffic without issues since last night. No further work is required, and all services remain fully operational. This incident is now resolved.