TPP Wholesale experienced a minor incident on June 3, 2021 affecting International and National, lasting 3h 15m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 03, 2021, 02:13 AM UTC
We are aware of degraded performance for customers who are using the PartnerConsole DNS system. Our engineers are currently investigating as a matter of urgency. Further updates will be provided as soon as possible.
- monitoring Jun 03, 2021, 04:44 AM UTC
Our Platform Engineers have identified & amended the issue with the PartnerConsole DNS Platform. We will continue to monitor the situation and will provide updates accordingly.
- resolved Jun 03, 2021, 05:28 AM UTC
We have now confirmed the fix has been applied and services are operating as expected. Apologies again for any inconvenience caused - we appreciate everyone’s patience while we resolved this isolated issue. Please contact support if you need anything at all.
- postmortem Jun 09, 2021, 02:22 AM UTC
**Which customers were affected?** Any customer domain for which the domain name\(s\) nameserver was delegated to: * [ns1.partnerconsole.net](http://ns1.partnerconsole.net). * [ns2.partnerconsole.net](http://ns2.partnerconsole.net). * [ns3.partnerconsole.net](http://ns3.partnerconsole.net). **Why did this happen?** It is our belief that the initial incident was triggered due to a DDoS directed at the ParnerConsole Nameserver Range which caused internal systems to fail. Engineering at WebCentral then had to run the restart process manually to bring these services back online. **How will we prevent this from happening in the future?** As this platform is currently managed by WebCentral, we are not in a position to state which processes in the recovery or failover failed, but we can confirm that we are moving away from WebCentral's DNS into the CentralNic controlled AnyCast network which will avoid such outages in future.