Flowroute experienced a minor incident on April 11, 2022 affecting Outbound Voice, lasting 3h 41m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 11, 2022, 03:50 PM UTC
We are currently investigating reports of outbound packet loss at our East NJ PoP.
- investigating Apr 11, 2022, 04:03 PM UTC
We are continuing to investigate this issue.
- resolved Apr 11, 2022, 07:32 PM UTC
Our engineering team has done extensive testing and has not been able to reproduce the reported packet loss at the East NJ PoP. The customers that had issues with this PoP had their issues resolved when switching to the East VA PoP.
- postmortem Apr 15, 2022, 05:25 PM UTC
**Investigation results:** On April 14, 2022 we were provided with the official investigation results from a Senior Interop Engineer at 3CX: > After looking into this we found that the 147.75.65.0/24 range had been added in our global blacklist. We have now removed that value but it will take approximately 6 hours for all systems to receive the change. Normally by now all affected systems must have switched to one of the other IP addresses so it should be OK. If any more customers have issues please let us know. > ---Your 3CX Interop Team **Scope of impact:** This incident affected some 3CX PBX systems that were directly connected to the New Jersey Edge proxy \([us-east-nj.sip.flowroute.com](http://us-east-nj.sip.flowroute.com), 147.75.65.192/28\) by IP address or had cached DNS records. **Mitigation plan:** 1. Immediately escalate incidents to all involved parties \(3CX Support, Flowroute Support\) 2. Whitelist all IP ranges listed in [Firewall Requirements](https://manage.flowroute.com/accounts/interconnection/setup/) 3. Avoid using direct IP connections and/or cashed DNS records when it’s possible