Tungsten Automation EMEA Outage History

**Overview** On Sep 10 from 06:40 to 16:30UTC the processing of inbound mails to the EU hosted AP Essentials service stopped for some documents for some customers. This was caused by a service failure in one of the responsible servers and triggered by automated security updates. Once restarted the pending items were processed successfully. **Incident Detail** Once alerted to the issue, our technical team identified the failed service and restarted this to resume the processing of inbound documents. Queued items were successfully processed and customers advised on what action to take should any duplicates be reported. **Resolution** The immediate resolution was to restart the failed services, additional monitoring has been added to identify this type of failure and speed up detection in future should the same situation arise. Controls around the security updates are also reviewed to ensure such updates no longer interfere with routine operation.

_**Overview**_ On Sep 6 following Printix 2025.3 deployment, some customers could not sign-in on HP devices running the HP Go embedded application where devices were using the HP OXPd embedded platform. As part of a newly released feature, an additional database field was added but not populated for all existing HP device registrations, causing this failure. _**Resolution**_ Following reports of this failure, immediate workaround guidance for customers with small number of devices was given and steps were taken to refresh all existing device registrations were refreshed and completed by Sep 08, 2025 - 14:03 UTC. _**Action Items**_ Additional steps are being taken to ensure test environments are more representative to such conditions in future regression tests. Additional checks are being implemented to ensure production data updates are not missed during update cycles.

**Overview** On Aug 29 from 10:00 to 10:39UTC users of Printix lost existing connections and were unable to start new connections, API access became restricted at the same time. This was caused by a failure of the Microsoft Azure load balancing service that presents out the Printix service and loss of IP connectivity. **Incident Detail** Although our technical teams were immediately alerted, the Azure service automatically recovered from this failure and users were able to reconnect to Printix in the following 30 minutes. **Resolution** Service was restored without intervention from the Cloud Services team. A case was raised with Microsoft requesting investigation of the incident.