TPP Wholesale incident

PartnerConsole DNS Outage

Resolved after 3 hours 15 minutes, affecting International and National, on June 3, 2021.

postmortem Jun 09, 2021, 02:22 AM UTC

**Which customers were affected?** Any customer domain for which the domain name\(s\) nameserver was delegated to: * [ns1.partnerconsole.net](http://ns1.partnerconsole.net). * [ns2.partnerconsole.net](http://ns2.partnerconsole.net). * [ns3.partnerconsole.net](http://ns3.partnerconsole.net). **Why did this happen?** It is our belief that the initial incident was triggered due to a DDoS directed at the ParnerConsole Nameserver Range which caused internal systems to fail. Engineering at WebCentral then had to run the restart process manually to bring these services back online. **How will we prevent this from happening in the future?** As this platform is currently managed by WebCentral, we are not in a position to state which processes in the recovery or failover failed, but we can confirm that we are moving away from WebCentral's DNS into the CentralNic controlled AnyCast network which will avoid such outages in future.

Started
Jun 03, 2021, 02:13 AM UTC
Resolved
Jun 03, 2021, 05:28 AM UTC
Duration
3h 15m
Detected by Pingoru
Jun 03, 2021, 02:13 AM UTC

Affected components

InternationalNational

Update timeline

  1. investigating Jun 03, 2021, 02:13 AM UTC

    We are aware of degraded performance for customers who are using the PartnerConsole DNS system. Our engineers are currently investigating as a matter of urgency. Further updates will be provided as soon as possible.

  2. monitoring Jun 03, 2021, 04:44 AM UTC

    Our Platform Engineers have identified & amended the issue with the PartnerConsole DNS Platform. We will continue to monitor the situation and will provide updates accordingly.

  3. resolved Jun 03, 2021, 05:28 AM UTC

    We have now confirmed the fix has been applied and services are operating as expected. Apologies again for any inconvenience caused - we appreciate everyone’s patience while we resolved this isolated issue. Please contact support if you need anything at all.

  4. postmortem Jun 09, 2021, 02:22 AM UTC

    **Which customers were affected?** Any customer domain for which the domain name\(s\) nameserver was delegated to: * [ns1.partnerconsole.net](http://ns1.partnerconsole.net). * [ns2.partnerconsole.net](http://ns2.partnerconsole.net). * [ns3.partnerconsole.net](http://ns3.partnerconsole.net). **Why did this happen?** It is our belief that the initial incident was triggered due to a DDoS directed at the ParnerConsole Nameserver Range which caused internal systems to fail. Engineering at WebCentral then had to run the restart process manually to bring these services back online. **How will we prevent this from happening in the future?** As this platform is currently managed by WebCentral, we are not in a position to state which processes in the recovery or failover failed, but we can confirm that we are moving away from WebCentral's DNS into the CentralNic controlled AnyCast network which will avoid such outages in future.

Your product runs on more than TPP Wholesale

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →