ClearlyIP Outage History

At approximately 11:27AM ET on November 3, ClearlyIP's NOC team began to receive system alerts indicating excess session activity within its US-East trunking systems. The team's investigation identified unauthenticated INVITE messages being received from unknown IPs as the source of the excess traffic, as well as a customer generating unexpected levels of outbound calls. Steps were taken to mitigate the traffic's impact, including restarting affected services to clear sessions and restore normal status. By approximately 11:55AM ET, the US-East situation was stabilized and session traffic had returned to normal levels. The team has identified a number of improvements and traffic management strategies to help avoid similar incidents which will be implemented in future maintenance events.

At approximately 4:10 PM ET, ClearlyIP's monitoring systems detected registration issues within US-Central trunking. Within a few minutes, it was determined that one of the two primary trunking servers was maintaining its active calls \(2.us-central\) and that its counterpart \(1.us-central\) had stalled and was not accepting traffic. It was quickly determined that the issue stemmed from a database cluster deadlock within the US-Central footprint. Upon analysis, the deadlock's cause was identified to be related to unsuccessful credential change activity. A series of actions were taken during the next 30 minutes which restored 2.us-central to service. Following that, maintenance was performed to resolve version differences on the database node not currently in production before restoring it \(and 1.us-central\) to normal service shortly after 6PM ET.

During the early afternoon of Friday May 16, our engineering team detected a potential issue and proactively moved production SIP services onto a secondary redundancy system in an abundance of caution in order to prevent a widespread service disruption. For approximately an hour, the secondary system operated with limited BLF functionality but processed calls and registrations normally. There may have been very brief issues with call setup or registration during the failover and failback process. BLF features were restored to normal functionality at the conclusion of the failback operation.