Network Change Propagation Delay - US-E region

Update timeline

1. investigating Mar 14, 2025, 02:02 PM UTC

   We are currently investigating delays with Network change propagation to devices in the US-E region. Changes made in our UI or via our API are taking longer than expected to push to all devices in this region. Engineers are aware of this issue and are working to resolve it as quickly as possible.

2. investigating Mar 14, 2025, 02:27 PM UTC

   Engineers have identified this issue with our update orchestrator. We are working diligently to resolve this as quickly as possible.

3. investigating Mar 14, 2025, 02:35 PM UTC

   We have confirmed that updates are delayed to a single node in the US-E region. This node has been taken out of service to prevent impact to customer traffic, however, updates will appear to remain queued for the US-E region in general even though all other nodes are accepting and completing updates normally. Because we have taken the affected node out of service, customer traffic is not affected by this issue. This is now essentially a display issue within our UI, albeit an annoying one. Please accept our apologies for the continued delay in resolving this and rest assured that all changes are otherwise completing successfully across our platform despite potentially incorrect device status and/or completion messages for the US-E region. As soon as we have further updates, we will post them here. We anticipate this issue to be fully resolved within another 30 minutes.

4. investigating Mar 14, 2025, 03:28 PM UTC

   Engineers have implemented a fix and we are now waiting for all pending updates to clear the queue for this node in the US-E region. We will update this case as soon as we have confirmed all queued updates have completed successfully and everything has returned to normal.

5. resolved Mar 14, 2025, 03:52 PM UTC

   This issue has now been fully resolved. All updates are successfully processing to all US-E nodes at this time.