ClearlyIP Outage History

Started Sep 11, 2026, 02:30 PM UTC· resolved Sep 11, 2026, 03:22 PM UTC

1. investigating Sep 11, 2026, 03:08 PM UTC

   We are receiving reports of outbound call failures with Clearly Cloud USA, but it does not appear to be affecting all users. Engineers are investigating.

2. resolved Sep 11, 2026, 03:22 PM UTC

   This issue has been resolved since 9:45 CDT

Read the full incident report →

Started Sep 01, 2026, 11:00 PM UTC· resolved Sep 03, 2026, 11:54 AM UTC

1. investigating Sep 02, 2026, 03:15 PM UTC

   We are current investigating an issue where outbound calls to a small percentage of rate centers are failing for customers. Note that this only affects customers directly connecting through our XCast platform.

2. identified Sep 02, 2026, 03:17 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Sep 02, 2026, 05:45 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Sep 03, 2026, 11:54 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 25, 2026, 08:51 PM UTC· resolved Aug 27, 2026, 02:18 PM UTC

1. investigating Aug 25, 2026, 08:51 PM UTC

   We are seeing an increase in mail delivery delays for messages originating from our platforms towards Microsoft mail services.

2. monitoring Aug 25, 2026, 08:55 PM UTC

   Queue processing continues successfully, though slowly as Microsoft accepts mail from our platform(s). We expect to see acceptance rates increase over time as well. Engineering continues to monitor activity and delivery improvements.

3. monitoring Aug 26, 2026, 02:24 AM UTC

   We continue to monitor for improved mail delivery rates to Microsoft mail services.

4. monitoring Aug 26, 2026, 02:07 PM UTC

   While we are still seeing delays in delivering mail to Microsoft mail services, acceptance rates are continually improving and the current backlog has been almost completely processed. We are continuing to monitor for performance and will close this out once we are satisfied with consistent delivery rates.

5. resolved Aug 27, 2026, 02:18 PM UTC

   Mail continues to flow normally and at expected deliverability rates without delays.

Read the full incident report →

Started Aug 21, 2026, 01:46 PM UTC· resolved Aug 21, 2026, 03:07 PM UTC

1. investigating Aug 21, 2026, 01:46 PM UTC

   We are current investigating intermittent call quality issues with inbound calls with our upstream carriers

2. resolved Aug 21, 2026, 03:07 PM UTC

   Upstream carrier has resolved the issues at approximately 9:13am CST

3. postmortem Sep 15, 2026, 08:14 PM UTC

   Several reports of poor audio quality on inbound calls were received beginning around 6:30 AM CDT. Investigation revealed the issue was region-specific, involving an upstream carrier who was promptly notified. We observed the issue to be fully resolved sometime shortly after 9AM CDT.

Read the full incident report →

Started Aug 18, 2026, 03:45 PM UTC· resolved Aug 18, 2026, 04:34 PM UTC

1. investigating Aug 18, 2026, 04:21 PM UTC

   We are currently investigating an issue with US-East trunking servers not accepting registrations.

Read the full incident report →

Started Jun 22, 2026, 04:59 PM UTC· resolved Jun 22, 2026, 07:38 PM UTC

1. investigating Jun 22, 2026, 04:59 PM UTC

   We are currently investigating issues with our US East trunking servers that is causing completion issues and audio issues.

2. monitoring Jun 22, 2026, 06:59 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jun 22, 2026, 07:38 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 15, 2026, 12:53 PM UTC· resolved Jun 15, 2026, 05:48 PM UTC

1. identified Jun 15, 2026, 12:53 PM UTC

   We are currently investigating an issue with our CA-East trunks and have taken them offline at this time.

2. monitoring Jun 15, 2026, 04:32 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jun 15, 2026, 05:48 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 09, 2026, 07:22 PM UTC· resolved Jun 10, 2026, 01:16 AM UTC

1. investigating Jun 09, 2026, 07:22 PM UTC

   IS NOC is reviewing reports of connectivity issues for some Pulsar360 hosted PBXs in the Toronto region.

2. investigating Jun 09, 2026, 08:44 PM UTC

   Work continues to isolate and find the root cause for connectivity interruptions in the Toronto region.

3. monitoring Jun 09, 2026, 09:04 PM UTC

   IS NOC has identified and resolved the issue(s) responsible for connectivity interruption. We are seeing alarms clearing at this time and will continue to monitor.

4. resolved Jun 10, 2026, 01:16 AM UTC

   All services remain stable and operational, free from the previously experienced issues.

Read the full incident report →

Started Jun 09, 2026, 02:27 PM UTC· resolved Jun 09, 2026, 05:24 PM UTC

1. investigating Jun 09, 2026, 02:27 PM UTC

   We are currently investigating an issue which is affecting chat/sms on Clearly Cloud USA - w

2. monitoring Jun 09, 2026, 02:50 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Jun 09, 2026, 05:24 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jun 01, 2026, 01:54 PM UTC· resolved Jun 01, 2026, 08:13 PM UTC

1. monitoring Jun 01, 2026, 01:54 PM UTC

   We are seeing some isolated connectivity issues in our US-Central region facilities. They seem to be transient in nature and quick to assert/resolve. We are working with our upstream provider(s) and monitoring closely.

2. resolved Jun 01, 2026, 08:13 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 31, 2026, 06:03 PM UTC· resolved Jun 01, 2026, 11:54 AM UTC

1. investigating May 31, 2026, 06:03 PM UTC

   Our US-Central datacenter is experiencing routing issues at this time. Intermittent problems are resulting from this.

2. identified May 31, 2026, 06:22 PM UTC

   Our upstream connectivity provider(s) are experiencing degraded network performance which is affecting ClearlyIP/14IP services. They are working diligently to resolve this issue.

3. identified May 31, 2026, 07:48 PM UTC

   Work continues by our upstream provider(s) to isolate and resolve the root connectivity issues.

4. identified May 31, 2026, 09:31 PM UTC

   Work continues by our upstream provider(s) to isolate and resolve an apparent DDoS attack affecting multiple customers at the US-Central facilities.

5. monitoring Jun 01, 2026, 12:40 AM UTC

   We are seeing recovery but are continuing to monitor the situation while our upstream provider(s) work towards a full resolution.

6. identified Jun 01, 2026, 01:06 AM UTC

   We are seeing degraded connectivity once again, affecting services in US-Central. We are awaiting updates and progress reports from our upstream provider(s).

7. monitoring Jun 01, 2026, 01:57 AM UTC

   We are once again seeing recovery and are continuing to monitor the situation. Our upstream provider(s) have confirmed positive progress towards resolution of this ongoing DDoS event.

8. monitoring Jun 01, 2026, 03:35 AM UTC

   Latest updates from our upstream provider(s) indicate a full restoration of all services and connectivity. We are confirming the same and continue to see clean network operation at this time. We will continue to monitor.

9. resolved Jun 01, 2026, 11:54 AM UTC

   Connectivity has remained stable.

10. postmortem Jun 10, 2026, 08:52 PM UTC

    **Summary / Impact** On the afternoon of May 31st, 2026, 14IP's network operations group began to observe connectivity issues with resources located at its US-Central \(MKE\) data center. The team began working with their counterparts at the data center facility to understand the nature of the problem. It was quickly determined that a distributed denial of service \(DDoS\) attack targeted upstream was affecting all tenants of the data center. Significant malicious traffic \(exceeding 450gbps\) was measured upstream during the peak of the attack which overwhelmed equipment as well as primary, secondary, and tertiary links. This traffic was mitigated through coordinated, escalating filtering with upstream transit providers, as well as internal engineering to stabilize routing. During the incident, connectivity to systems and services primarily located in this data center were impacted significantly for periods of time between approximately 2 PM ET and 9:30 PM ET. This included Clearly Cloud, ComXchange hosted, Xivea, Business Connect, EPlatform, and several other systems. \(US-East trunking services continued to operate during the US-Central incident.\) **Follow-Up** Several facility-level network and system improvements have been implemented following the incident which should reduce the risk of future occurrences. This includes the deployment of additional DDoS scrubbing capacity, improved granularity for detecting attacks, additional fiber capacity upstream, as well as core network enhancements. 14IP plans additional improvements to its systems and operations to improve platform resilience and flexibility when responding to future incidents.

Read the full incident report →

Started May 28, 2026, 05:16 PM UTC· resolved May 28, 2026, 06:50 PM UTC

1. identified May 28, 2026, 05:16 PM UTC

   An upstream Canadian carrier is currently having issues. This may affect incoming calls routed through that carrier, but is limited to ONLY Canadian DIDs through that carrier. Outbound should not be affected. At this time, we have no information from the carrier on ETA.

2. monitoring May 28, 2026, 06:00 PM UTC

   The upstream carrier is now routing properly. We are monitoring and waiting on more information related to the outage.

3. resolved May 28, 2026, 06:50 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 27, 2026, 06:26 PM UTC· resolved Apr 27, 2026, 06:51 PM UTC

1. investigating Apr 27, 2026, 06:26 PM UTC

   The team is investigating Call Panel on Clearly Cloud USA displaying a bad certificate.

2. resolved Apr 27, 2026, 06:51 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 03, 2026, 08:32 PM UTC· resolved May 08, 2026, 07:22 PM UTC

1. investigating Apr 03, 2026, 08:32 PM UTC

   We are seeing issues with provisioning and registration of some Grandstream devices. The team is actively reviewing the situation

2. monitoring Apr 05, 2026, 12:01 AM UTC

   We have implemented a fix. If you are still having issues with your devices, please contact us by emailing [email protected] and we can assist you further.

3. resolved May 08, 2026, 07:22 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 19, 2026, 02:00 PM UTC· resolved Mar 19, 2026, 03:38 PM UTC

1. identified Mar 19, 2026, 02:55 PM UTC

   The team has identified the issue and is currently working on a fix.

2. resolved Mar 19, 2026, 03:38 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 13, 2026, 11:52 PM UTC· resolved Mar 14, 2026, 01:19 AM UTC

1. investigating Mar 13, 2026, 11:52 PM UTC

   We are currently investigating this issue.

2. identified Mar 14, 2026, 12:41 AM UTC

   The issue has been identified and a fix is being implemented.

3. resolved Mar 14, 2026, 01:19 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 19, 2026, 02:54 PM UTC· resolved Feb 19, 2026, 05:03 PM UTC

1. investigating Feb 19, 2026, 02:54 PM UTC

   We are currently investigating an issue with Call Panel on Clearly Cloud Canada.

2. investigating Feb 19, 2026, 04:03 PM UTC

   We are continuing to investigate this issue.

3. monitoring Feb 19, 2026, 04:04 PM UTC

   A fix is being implemented. If you still are experiencing issues, please open a support ticket and the team will assist.

4. resolved Feb 19, 2026, 05:03 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 15, 2025, 06:41 PM UTC· resolved Dec 15, 2025, 07:00 PM UTC

1. investigating Dec 15, 2025, 06:41 PM UTC

   The team is currently investigating an issue with Clearly Cloud USA

2. identified Dec 15, 2025, 06:42 PM UTC

   The team has identified the issue and should be resolved quickly.

3. monitoring Dec 15, 2025, 06:43 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Dec 15, 2025, 07:00 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 04, 2025, 06:57 PM UTC· resolved Dec 04, 2025, 07:24 PM UTC

1. investigating Dec 04, 2025, 06:57 PM UTC

   We are currently investigating this issue.

2. monitoring Dec 04, 2025, 07:03 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Dec 04, 2025, 07:24 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Dec 02, 2025, 04:45 PM UTC· resolved Dec 02, 2025, 07:46 PM UTC

1. investigating Dec 02, 2025, 04:45 PM UTC

   We are currently investigating inbound and outbound calling issues with Clearly Cloud US platform.

2. monitoring Dec 02, 2025, 04:52 PM UTC

   A fix has been implemented and we are monitoring the results.

3. identified Dec 02, 2025, 05:09 PM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring Dec 02, 2025, 05:30 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Dec 02, 2025, 07:46 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 11, 2025, 11:15 AM UTC· resolved Nov 11, 2025, 11:15 AM UTC

1. resolved Nov 11, 2025, 07:04 PM UTC

   SIP2 and Proxy55 are not responding. Calls routing to failover trunks

Read the full incident report →

Started Nov 03, 2025, 04:49 PM UTC· resolved Nov 03, 2025, 06:10 PM UTC

1. investigating Nov 03, 2025, 04:49 PM UTC

   We are currently investigating this issue.

2. identified Nov 03, 2025, 04:57 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Nov 03, 2025, 05:29 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Nov 03, 2025, 06:10 PM UTC

   This incident has been resolved.

5. postmortem Nov 04, 2025, 05:13 PM UTC

   At approximately 11:27AM ET on November 3, ClearlyIP's NOC team began to receive system alerts indicating excess session activity within its US-East trunking systems. The team's investigation identified unauthenticated INVITE messages being received from unknown IPs as the source of the excess traffic, as well as a customer generating unexpected levels of outbound calls. Steps were taken to mitigate the traffic's impact, including restarting affected services to clear sessions and restore normal status. By approximately 11:55AM ET, the US-East situation was stabilized and session traffic had returned to normal levels. The team has identified a number of improvements and traffic management strategies to help avoid similar incidents which will be implemented in future maintenance events.

Read the full incident report →

Started Oct 30, 2025, 08:21 PM UTC· resolved Oct 30, 2025, 09:13 PM UTC

1. investigating Oct 30, 2025, 08:21 PM UTC

   We are currently investigating this issue.

2. monitoring Oct 30, 2025, 08:51 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Oct 30, 2025, 09:13 PM UTC

   This incident has been resolved.

4. postmortem Nov 03, 2025, 08:39 PM UTC

   At approximately 4:10 PM ET, ClearlyIP's monitoring systems detected registration issues within US-Central trunking. Within a few minutes, it was determined that one of the two primary trunking servers was maintaining its active calls \(2.us-central\) and that its counterpart \(1.us-central\) had stalled and was not accepting traffic. It was quickly determined that the issue stemmed from a database cluster deadlock within the US-Central footprint. Upon analysis, the deadlock's cause was identified to be related to unsuccessful credential change activity. A series of actions were taken during the next 30 minutes which restored 2.us-central to service. Following that, maintenance was performed to resolve version differences on the database node not currently in production before restoring it \(and 1.us-central\) to normal service shortly after 6PM ET.

Read the full incident report →

Started Oct 30, 2025, 11:05 AM UTC· resolved Oct 30, 2025, 02:15 PM UTC

1. identified Oct 30, 2025, 12:05 PM UTC

   The issue has been identified and a fix is being implemented.

2. monitoring Oct 30, 2025, 01:20 PM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring Oct 30, 2025, 01:21 PM UTC

   We are continuing to monitor for any further issues.

4. resolved Oct 30, 2025, 02:15 PM UTC

   This incident has been resolved.

5. postmortem Nov 03, 2025, 07:47 PM UTC

   A scheduled data center facility network maintenance event occurred during the evening of 10/29, which caused a variety of anticipated monitoring alarms during its completion. The data center facility provided expected updates on the maintenance activity and no issues were immediately noted following its conclusion. At approximately 8AM ET on 10/30, a customer reported that trunk registrations to the ClearlyIP US-East trunking servers were failing, while US-Central trunks were operating normally. The issue was escalated by support to engineering, as it indicated potential issues with inbound calling mapped to the US-East region and would also affect customers who hadn't configured redundancy with ClearlyIP US-Central trunking. After analysis, some database connectivity issues within the affected data center were identified as likely to be responsible. The engineering team conducted a full review of the affected equipment and began work to restore access to the unresponsive database servers. This required a careful review of replication and synchronization states before steps to re-activate each of the affected servers were taken. By approximately 8:45 AM ET, the team had determined data integrity was intact and started operations to restore synchronization between the US-East database notes. At approximately 9 AM ET, US-East trunking servers were reopened to customer traffic. Systems monitoring and review occurred immediately following US-East reactivation, and support teams were advised to monitor customer equipment for proper re-registration with US-East servers. During final analysis, several process and monitoring improvements were identified for implementation which should help avoid similar monitoring ambiguity \(and potential disruption\) during service provider maintenance events.

Read the full incident report →

Started Oct 29, 2025, 06:37 PM UTC· resolved Oct 29, 2025, 07:17 PM UTC

1. investigating Oct 29, 2025, 06:37 PM UTC

   Clearly Cloud USA is currently down and the team is investigating.

2. identified Oct 29, 2025, 06:56 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Oct 29, 2025, 07:02 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Oct 29, 2025, 07:17 PM UTC

   This incident has been resolved.

5. postmortem Nov 03, 2025, 04:09 PM UTC

   At approximately 2:30 PM ET on October 29, alerts began to be received by ClearlyIP's Network Operations team indicating problems accessing the database cluster used by Clearly Cloud for most US-based customers. This was interfering with call routing for a portion of customers and prevented access to Call Panel and other system components. The team immediately began investigating and observed the problem related to a protective mechanism which had detected large amounts of database request traffic and proactively restarted services to prevent data loss or corruption. The source of the requests was quickly identified, stopped, and determined to be legitimate. The team then checked each server in the cluster before determining it was safe to restore normal service by approximately 2:45 PM ET. A postmortem analysis on the source of the database traffic identified it to be related to delivering an unusual backlog of SMS and email messages from the platform. Temporary measures were taken to prevent a recurrence of the same issue while long-term improvements are scheduled.

Read the full incident report →