Intermittent connectivity issues to AdvaPACS tenants in the Canada (can1) region

Update timeline

1. resolved Jun 11, 2026, 11:32 PM UTC

   AdvaPACS users with tenants located in the Canada (can1) region will have experienced intermittent connection timeouts when trying to access AdvaPACS.

2. postmortem Jun 12, 2026, 08:20 AM UTC

   ### Summary Between 14:32 UTC and 15:14 UTC, AdvaPACS users in the Canada region experienced intermittent connection timeouts when accessing the platform. During this window, image uploads, downloads, and viewing functionality were unavailable or unreliable. The issue was caused by a failure in a core infrastructure routing component responsible for directing incoming traffic to the appropriate AdvaPACS services. The component exhausted its available memory under heavy concurrent upload load, causing it to crash and restart repeatedly. Service was fully restored at 15:14 UTC and has remained stable since. ### What Happened Due to the routing component being deprecated by the organization that supported it, AdvaPACS has been migrating to a newer and more capable traffic routing layer. This new component had been successfully deployed and validated across multiple smaller regions over several weeks prior to being promoted to larger regions, including Canada. Under the higher volume and concurrency of the Canada region, the routing component's in-memory request buffering exceeded its allocated resources. This caused repeated crashes, during which all in-flight requests were dropped and users experienced connection timeouts. ### Impact * Users in the Canada region were unable to reliably access AdvaPACS for approximately 42 minutes. * Image uploads, downloads, and viewing were affected. * No data loss occurred as a result of this incident. ### Resolution Once the issue was identified, the engineering team determined the root cause was memory exhaustion in the routing layer. Resource allocations were increased to stabilise the component, and service was restored. ### Preventive Measures We have identified the following actions to prevent recurrence: 1. **Enhanced monitoring and alerting** — Dedicated monitoring and automated alerting are being implemented for the routing component to ensure rapid detection and response to any future resource pressure. 2. **Disk-based request buffering** — The routing component is being tuned to offload more request buffering to disk rather than holding entire request bodies in memory. This eliminates the risk of memory exhaustion under heavy upload concurrency. ### Current Status All services in the Canada region are operating normally. The preventive measures listed above are actively being implemented and are expected to be completed shortly.