Cradlepoint experienced a minor incident on November 21, 2021, lasting 1d 22h. The incident has been resolved; the full update timeline is below.
Update timeline
- identified Nov 21, 2021, 11:45 PM UTC
NetCloud Perimeter is experiencing a performance degradation causing clients to reconnect slowly. During this period, all active clients will continue to remain operational.
- monitoring Nov 22, 2021, 12:38 AM UTC
We are aware of the performance degradation causing a potential delay with clients reconnecting. We are monitoring and during this period all active clients will continue to remain operational.
- monitoring Nov 22, 2021, 05:55 PM UTC
NetCloud Perimeter service is fully functional but we are continuing to monitor the system.
- monitoring Nov 22, 2021, 07:52 PM UTC
NetCloud Perimeter service is operational but we are continuing to monitor.
- identified Nov 22, 2021, 08:41 PM UTC
We have identified an issue that the team is addressing. The team is making some updates on the backend and you may see a disruption in your network connectivity over the next couple of hours. The engineering team is aware of this issue and working as quickly as possible to address. We apologize for any inconvenience this has caused you.
- identified Nov 22, 2021, 11:34 PM UTC
Cradlepoint engineering has been actively clearing IP table rules on NCP nodes in order to improve system load capacity. The team has been successful in reducing the number of database deadlocks as the root cause of the service degradation affecting client status updates. Engineering will continue to slowly apply updates and monitor until all cases of deadlock conditions cease and the system is fully stable. Potential mitigation tactics, if the user power cycles/reboots the Cradlepoint device it could resolve issues involved with NCP failing to reconnect.
- monitoring Nov 23, 2021, 02:00 AM UTC
Cradlepoint engineering is continuing to clear IP table rules on NCP nodes with positive improvement in system load capacity. Over the next several hours Engineering will slowly apply more updates to nodes and monitor until the system is fully stable. Potential mitigation tactics, if the user power cycles/reboots the Cradlepoint device it could resolve issues involved with NCP failing to reconnect. For MS Windows or other clients that may be experiencing connection issues, we recommend re-authenticating and applying the most recent NCP client update if possible.
- monitoring Nov 23, 2021, 04:26 AM UTC
Cradlepoint engineering has completed clearing IP table rules and will continue monitoring. If any devices have not reconnected, the user may power cycle/reboot the Cradlepoint device. For MS Windows or other clients that may be experiencing issues reconnecting, we recommend re-authenticating and applying the most recent NCP client update if possible.
- monitoring Nov 23, 2021, 05:49 PM UTC
NetCloud Perimeter remains operational and stable. Cradlepoint engineering will continue monitoring through the day. If any devices have not reconnected, the user may power cycle/reboot the Cradlepoint device. For MS Windows or other clients that may be experiencing issues reconnecting, we recommend re-authenticating and applying the most recent NCP client update if possible.
- resolved Nov 23, 2021, 10:22 PM UTC
NetCloud Perimeter remains operational and stable so we are resolving this incident.
- postmortem Dec 01, 2021, 11:55 PM UTC
Cradlepoint’s engineering team has completed a Root Cause Analysis \(RCA\) of the service disruption that occurred on November 19 2021. Please follow the link below to view the RCA in our Customer Connect Community. Any questions or concerns please contact Cradlepoint Support. [https://customer.cradlepoint.com/s/article/Service-Degradation-RCA-Nov-19th-2021](https://customer.cradlepoint.com/s/article/Service-Degradation-RCA-Nov-19th-2021)