[Resolved]: UK: Partial Outage across the UK Deployment

Update timeline

1. resolved Oct 23, 2025, 07:50 PM UTC

   Incident details: Investigating: We have multiple internal alerts related to Device to Communication that are currently under investigation. Our internal monitoring and Microsoft's monitoring have reported degraded IoT Hubs. Start Time: 19:07 UTC 23-10-2025 Incident Scope: UK (UK Deployment) Description: Under investigation within the operations team. Next Update: The next update will be in 30 minutes or as information becomes available

2. investigating Oct 23, 2025, 07:50 PM UTC

   Status changed to Investigating

3. investigating Oct 23, 2025, 08:22 PM UTC

   Hello Everyone, We are still seeing a degradation on our IoT Hubs at this time. The cause is still under investigation, but we are applying manual mitigations to help improve the recovery. Next update will be in 30 minutes or as information becomes available. Kind Regards Online Operations Team

4. investigating Oct 23, 2025, 08:24 PM UTC

   Hello Everyone, Devices should remain functioning under our current emergency mode. Kind Regards Online Operations Team

5. investigating Oct 23, 2025, 08:41 PM UTC

   **Update**: Microsoft is working to restore functionality to the IOT service. Mitigation efforts are underway; we are working with the Microsoft engineering team to have these in place as quickly as possible. **Next Update**: The next update will be in 30

6. identified Oct 23, 2025, 08:50 PM UTC

   Status changed to Identified

7. identified Oct 23, 2025, 09:21 PM UTC

   **Update**: Recovery efforts are being hampered by ongoing issues with the Microsoft Azure IOT platform. We are working closely with Microsoft engineering to rectify this issue. **Next Update**: The next update will be in 30 minutes

8. identified Oct 23, 2025, 09:55 PM UTC

   **Update**: Investigation by Microsoft is ongoing by their engineering team. **Next Update**: The next update will be in 30

9. identified Oct 23, 2025, 10:33 PM UTC

   **Update**: Microsoft is continuing their investigation with the backend engineering teams. IOT functionality and fail over DR services remain in a failing state and they are continuing to investigate this incident. **Please note:** - Emergency Mode automatically engaged at the start of this incident. - Login, Print and scanning should remain operational in a limited capacity. - Customers with 'Zero Trust Printing' will be directly impacted by this incident. Internal testing is continuing, NT-ware Operations are closely monitoring the situation. **Next Update**: The next update will be in 30

10. identified Oct 23, 2025, 11:29 PM UTC

    **Update**: - Microsoft investigation is continuing within their support and engineering teams. - Clarification on the previous update. Zero trust customers who are utilising the 'Universal Output Queue' cannot print at present. - Internal testing has confirmed Secure print via cloud is still operational with the Emergency Mode enabled. NT-ware Operations is continuing to monitor the situation and working on remediations options. **Next Update:** The next update will be in 1 hour

11. identified Oct 24, 2025, 12:41 AM UTC

    **Update**: No further updated from Microsoft, NT-ware Operations will continue to monitor the incident. **Next Update**: The next update will be in 1 Hour

12. identified Oct 24, 2025, 01:45 AM UTC

    **Update**: No further updated from Microsoft, incident has been escalated. **Next Update**: The next update will be in 1 Hour

13. identified Oct 24, 2025, 02:59 AM UTC

    **Update**: No further updates from Microsoft, NT-ware Operations will continue to monitor the incident. **Next Update**: The next update will be in 1 Hour.

14. identified Oct 24, 2025, 03:59 AM UTC

    **Update**: No further updates from Microsoft, NT-ware Operations will continue to monitor the incident. **Next Update**: The next update will be in 1 Hour.

15. identified Oct 24, 2025, 05:00 AM UTC

    Update: Some changes were made by Microsoft against our resources. We are currently testing these changes. We will continue to keep you updated with any further changes in information. **Next Update**: The next update will be in 1 Hour.

16. identified Oct 24, 2025, 05:18 AM UTC

    **Update**: Initial testing is looking promising. Our metrics are displaying connected devices and our SmartClients have been reconnecting to the IoT hub. **Next Update**: The next update will be in 1 Hour.

17. identified Oct 24, 2025, 06:13 AM UTC

    **Update**: From our testing and metrics, the issue appears to be resolved. NT-ware operations will continue to monitor this to ensure it stays on track. **Next Update**: The next update will be in 1 Hour.

18. monitoring Oct 24, 2025, 07:01 AM UTC

    **Update**: We are seeing device connection levels returning to normal. We have moved this incident into Monitoring as the operations teams monitors the start of day load increase over the next 1-2 hours. **Next Update**: The next update will be in 2 Hours

19. monitoring Oct 24, 2025, 07:01 AM UTC

    Status changed to Monitoring

20. resolved Oct 24, 2025, 09:01 AM UTC

    Hello Everyone, Update: Incident Resolved. Date/Time: October 24th, 06:30 UTC The connected devices level has been back to the scale of the previous days. We will review the findings and collected information from this incident to further improve our online services. There will be a post-mortem published withing 20 working days following an internal investigation and review with Microsoft. We are sorry for the inconvenience this has caused. Kind Regards Online Operations Team

21. resolved Oct 24, 2025, 09:01 AM UTC

    Status changed to Resolved

22. resolved Nov 09, 2025, 09:02 PM UTC

    **User Impact  ** With communication to the IoT disconnected, the following impacts occurred to the end users: - Users configured to print in either spool file local/job information local would revert back to Emergency Mode. - Users configured to print using Zero Trust Universal Output Queue would not be able to print and would have to use a backup print method. - Local Scan Agents would not function correctly. - Job retrieval from uniFLOW servers may have been impacted. - Users printing or scanning in any other way would not be affected. **Scope of Impact: ** UK Deployment **Incident Start Date and Time ** October 23, 2025, 18:52 UTC **Incident End Date and Time** October 24, 2025, 06:30 UTC **Root Cause: ** Our alerts and telemetry identified that there was a drop in connections to the IoT hub within the UK. After reviewing the metrics, we determined that the IoT hub devices were not recovering. Our Operations Team performed a failover of the IoT hub to our geo-redundant location. Unfortunately, this process failed and the IoT hub was stuck in a non-active state. Following a joint investigation with the Microsoft engineering team it was confirmed a configuration change to the UK South firewall made by Microsoft caused the failure. This was reverted by Microsoft to restore functionality and recover the IOT into a working state. **How did we respond: ** October 23 2025 – 18:52 UTC – Metrics can be seen that devices have dropped from our IoT hub. Devices where applicable would have entered Emergency Mode. October 23 2025 – 19:07 UTC – Our alerts triggered a notification to the Operations Team of this issue. October 23 2025 – 19:20 UTC – As metrics were not showing devices as returning, a failover was attempted to the UK West. October 23 2025 – 19:34 UTC – The attempt to failover errored leaving the IOT in a ‘Non-Active’ another failover attempt was initiated which also failed. October 23 2025 – 20:35 UTC – A ticket was raised with Microsoft for the issue as metrics were showing 0 connections and the IoT hub was in a non-active state. October 23 2025 – 21:18 UTC – First response from Microsoft. Discussion began between NT-ware Operations and Microsoft. Microsoft engineers began their investigation with back-end technical teams. October 23 2025 – 22:03 UTC – Further testing of IoT hubs (not in use) triggered errors upon failover within the same region – thus pinpointing it to an issue with the failover. October 24 2025 – 04:03 UTC – Microsoft responded with changes that had been made in their back end restoring original firewall configuration. NT-ware confirmed that total connected devices were recovering, reflecting changes that had allowed the fail-over to complete and recover the system to a working state. October 24 2025 – 4:40 UTC – Operations’ test tenants were confirmed to be showing SmartClient and device connections to the IoT hub. Print jobs also demonstrated connectivity. October 24 2025 – 05:03 UTC – Metrics were reflected as normal – now purely a monitoring phase. October 24 2025 – 06:30 UTC – Status page incident closed. **How are we making incidents like this less likely or less impactful? ** We apologize for the impact on the customers affected. We are continuously taking steps to improve the uniFLOW Online Platform and our processes to help ensure such incidents do not occur in the future. In this case, this includes (but is not limited to): - The Operations Team have reviewed their processes and handling of this incident. Findings have been raised in Operations and with Management for improvement opportunities. - NT-ware has received confirmation from Microsoft that the configuration change procedure has been reviewed to lessen such a situation in the future **Was this incident related to previous incidents? ** No. **How can customers make incidents like this less impactful? ** Emergency Mode was already enabled by default, making this incident less impactful.