YourCloudTelco experienced a minor incident on February 26, 2021 affecting YourCloudTelco Calling Platform (Network), lasting 1d 8h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Feb 26, 2021, 11:00 PM UTC
We have isolated yesterday's call quality issues to customers connecting to us over Megaport IX. Our network engineers are working to resolve this and will update shortly.
- identified Feb 27, 2021, 04:20 AM UTC
We are making a change to the network in the next 30mins, during this time there will be an outage for a maximum of 2 minutes.
- identified Feb 27, 2021, 05:22 AM UTC
Megaport engineers report a layer1 fiber issue between our two companies' routers which in turn has caused the mega-IX BGP route to flap. We have now isolated our side and can replicate identical stats from our router back to Megaport. To advance we and raised the fault back to Equinix and disabled our Mega-IX BGP route which will stop the BGP flapping. As a side note, it’s rare that fiber randomly fails like this, generally, the cause is human error.
- resolved Feb 28, 2021, 07:57 AM UTC
Equinix identified and resolved a fault within the cross-connect between our racks and the Megaport-IX. After monitoring the fix for the previous 24 hrs we have now reinstated the BGP-route and will close this incident.
- postmortem Feb 28, 2021, 07:57 AM UTC
Over the past year, we have invested heavily in our DevOps group, a big part of which are the monitoring systems. Last Friday 26 Feb at around 11:00 am our CSRs reported consistent voice quality complaints. Within a few minutes, we identified the issue as outside of our network. In this case, while we diagnosed the symptom \(the Meg-IX BGP route was flapping\) unravelling the underlying fault required an outage which I'm always reluctant to do during work hours. In summary, the fault was a physical fibre fault within the Equinix network. Mike Johnstone