ClearlyIP Outage History

Started Oct 29, 2025, 04:51 PM UTC· resolved Oct 29, 2025, 06:37 PM UTC

1. investigating Oct 29, 2025, 04:51 PM UTC

   We are seeing some issues which are affecting the send/receiving of SMS as well as emails from Clearly Cloud USA. The team is currently investigating the issue.

2. identified Oct 29, 2025, 05:18 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Oct 29, 2025, 06:22 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Oct 29, 2025, 06:37 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 20, 2025, 04:45 PM UTC· resolved Oct 20, 2025, 07:12 PM UTC

1. investigating Oct 20, 2025, 05:55 PM UTC

   The team is investigating reported intermittent 1-way audio issues.

2. investigating Oct 20, 2025, 06:12 PM UTC

   We are continuing to investigate this issue.

3. monitoring Oct 20, 2025, 06:34 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Oct 20, 2025, 07:12 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Sep 19, 2025, 07:55 PM UTC· resolved Sep 19, 2025, 08:17 PM UTC

1. identified Sep 19, 2025, 07:55 PM UTC

   An upstream vendor is having an outage that can affect managing products in our portals. In addition, support may be inaccessible, however phone service and support is still functioning.

2. resolved Sep 19, 2025, 08:17 PM UTC

   Upstream vendor issue has been resolved.

Read the full incident report →

Started Sep 17, 2025, 07:44 PM UTC· resolved Sep 17, 2025, 08:11 PM UTC

1. investigating Sep 17, 2025, 07:44 PM UTC

   We are currently investigating this issue.

2. identified Sep 17, 2025, 07:44 PM UTC

   The issue has been identified and we are working on resolution.

3. resolved Sep 17, 2025, 08:11 PM UTC

   This incident has been resolved.

4. postmortem Sep 19, 2025, 07:32 PM UTC

   On 9/16 at approximately 1:50 PM CDT, ClearlyIP began to receive indications of connectivity problems affecting the Pulsar360 Denver data center. Infrastructure and network teams were immediately notified and began troubleshooting. The team quickly ruled out internal infrastructure issues, confirming that links to all Pulsar360 equipment were working normally. Within 15 minutes, the team had identified the source of the issue as a regional Internet routing problem caused by actions of an unrelated third-party fiber provider. The issue originated within a route \(BGP\) advertisement that erroneously included large sections of IP addresses--among them addresses used by Pulsar360's Denver location. That bad route information caused many ISPs to misdirect traffic away from the Denver data center location. ClearlyIP's engineering team pursued parallel efforts to resolve the underlying issue and explore workarounds that could be executed more rapidly. While contact was still being made with the third-party company through upstream contacts, a solution involving forced advertisement of smaller network segments \(counteracting the large mis-advertisement\) was tested successfully. The approach was approved quickly and the team immediately began executing it for all of the Pulsar360 Denver network segments during the next 30 minutes, restoring connectivity. Stable connectivity had been achieved across the entire Denver network approximately one hour after the incident had been reported.

Read the full incident report →

Started Aug 13, 2025, 10:20 PM UTC· resolved Aug 14, 2025, 01:51 AM UTC

1. investigating Aug 13, 2025, 11:47 PM UTC

   We are currently investigating this issue.

2. investigating Aug 13, 2025, 11:49 PM UTC

   The team is investigating an issue that is causing inbound and outbound calls to fail occasionally. BLF lights are also affected by this.

3. identified Aug 14, 2025, 12:17 AM UTC

   The issue has been identified and a fix is being implemented.

4. monitoring Aug 14, 2025, 12:51 AM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Aug 14, 2025, 01:51 AM UTC

   This incident has been resolved.

6. postmortem Aug 15, 2025, 02:06 AM UTC

   At approximately 6:18 PM ET, an internal alarm detected a component restart within the Clearly Cloud US platform. The engineering team quickly determined that the component successfully failed over to a redundant system while restarting, but this resulted in a few moments of disruption for dialing and status notifications \(BLF, MWI, etc.\) The issue was immediately escalated to key members of the infrastructure and software groups who were conferenced in to analyze the situation. Approximately ten minutes later, another restart \(and failover\) occurred, causing another brief disruption to new calls and status updates. The team identified the issue to be triggered by data integrity problems with several contact records. Research confirmed that this issue had never been encountered before, occurring while a routine system task purges expired contact records every 10 minutes. Several solutions were explored, and ultimately a repair of the affected data table was performed at approximately 7:40 PM ET, resolving the underlying problem. Additional cleanup tasks were implemented at approximately 8:00 PM as the escalation team monitored the platform. After recording several task cycles without further incident, the team reviewed potential improvements and documented the incident before marking it resolved at approximately 8:15 PM ET. While every part of Clearly Cloud has redundancies and most of these fail over gracefully without impacting operations, the component in question requires a few moments for backup systems to take over. During this timeframe--between a few seconds and a minute--established calls would be unaffected, but new calls may fail to complete and status messages for BLF, MWI, parking, and other notifications may fail to be sent. Although the issue's impact was limited due to the time of day, multiple brief failovers in a short timeframe caused disruption, which was noticed by several users ClearlyIP's engineering team will incorporate what was learned about this incident within upcoming platform improvements.

Read the full incident report →

Started Jul 24, 2025, 03:45 PM UTC· resolved Jul 24, 2025, 06:23 PM UTC

1. investigating Jul 24, 2025, 04:41 PM UTC

   We are currently investigating an issue with the XCast Webphone, attendant console and acd monitor having a spinning red circle and not connecting. This is NOT affecting calls.

2. identified Jul 24, 2025, 05:57 PM UTC

   Web Phone and part of the attendant console are up, still working on the ACD Monitor and Conference Monitor.

3. identified Jul 24, 2025, 06:05 PM UTC

   ACD Monitor is online.

4. resolved Jul 24, 2025, 06:23 PM UTC

   All web services are back online.

Read the full incident report →

Started Jun 27, 2025, 04:04 PM UTC· resolved Jun 27, 2025, 06:55 PM UTC

1. investigating Jun 27, 2025, 04:04 PM UTC

   We are currently investigating reports of intermittent audio issues, choppy or pixilated audio, on our Clearly Cloud USA infrastructure.

2. monitoring Jun 27, 2025, 04:35 PM UTC

   We are no longer seeing issues with call quality, however the team is continuing to monitor.

3. resolved Jun 27, 2025, 06:55 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 16, 2025, 05:04 PM UTC· resolved May 16, 2025, 07:25 PM UTC

1. investigating May 16, 2025, 05:04 PM UTC

   The team is actively working on an issue where some users are losing registration with Clearly Cloud US.

2. monitoring May 16, 2025, 05:31 PM UTC

   A fix has been implemented and we are monitoring the results.

3. identified May 16, 2025, 05:38 PM UTC

   The issue has been identified and a fix is being implemented.

4. identified May 16, 2025, 05:57 PM UTC

   We are continuing to work on a fix for this issue.

5. identified May 16, 2025, 06:00 PM UTC

   Calls and registrations are working as expected, however BLF lights, including parking and extension status, are not functioning at this time.

6. identified May 16, 2025, 07:24 PM UTC

   We are continuing to work on a fix for this issue.

7. resolved May 16, 2025, 07:25 PM UTC

   This incident has been resolved.

8. postmortem May 16, 2025, 10:36 PM UTC

   During the early afternoon of Friday May 16, our engineering team detected a potential issue and proactively moved production SIP services onto a secondary redundancy system in an abundance of caution in order to prevent a widespread service disruption. For approximately an hour, the secondary system operated with limited BLF functionality but processed calls and registrations normally. There may have been very brief issues with call setup or registration during the failover and failback process. BLF features were restored to normal functionality at the conclusion of the failback operation.

Read the full incident report →

Started May 08, 2025, 03:15 PM UTC· resolved May 08, 2025, 09:30 PM UTC

1. investigating May 08, 2025, 03:53 PM UTC

   We are currently investigating reports of odd call issues from a subset of users.

2. investigating May 08, 2025, 03:55 PM UTC

   Multiple users have reported issues with no audio on calls, unable to dial out and other call oddities.

3. monitoring May 08, 2025, 04:15 PM UTC

   After receiving several customer reports of issues relating to registration and call processing, we have been unable to replicate or confirm any problems. There have been no more reports of issues for the past 20 minutes. Our teams will continue to monitor.

4. monitoring May 08, 2025, 04:39 PM UTC

   We are continuing to monitor for any further issues.

5. resolved May 09, 2025, 03:21 PM UTC

   This incident has been resolved.

6. postmortem May 12, 2025, 04:07 PM UTC

   Over the course of approximately two hours, several customers in specific regions reported intermittent Call Panel connection status messages and registration timeouts with some phones. Our support team investigated the reports and enlisted assistance from engineering while attempting to reproduce. The problem was not definitively identified while issues were reported and may be attributable to connectivity issues or other factors outside ClearlyIP's network, but our teams will continue monitoring the situation while awaiting additional reports.

Read the full incident report →

Started May 02, 2025, 07:28 PM UTC· resolved May 02, 2025, 08:14 PM UTC

1. investigating May 02, 2025, 08:01 PM UTC

   We are currently investigating this issue with calling with our CA-East region.

2. identified May 02, 2025, 08:02 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring May 02, 2025, 08:02 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved May 02, 2025, 08:14 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 29, 2025, 05:49 PM UTC· resolved Apr 29, 2025, 06:10 PM UTC

1. investigating Apr 29, 2025, 05:49 PM UTC

   We are currently investigating this issue.

2. monitoring Apr 29, 2025, 05:54 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Apr 29, 2025, 06:10 PM UTC

   This incident has been resolved.

4. postmortem Apr 30, 2025, 07:36 PM UTC

   An issue occurred on April 29th during a failover operation between elements of Clearly Cloud's core infrastructure. This impacted Canada customers' ability to register phones and place calls for approximately 4 minutes. Most elements of Clearly Cloud's infrastructure are designed with several redundancies and failover occurring seamlessly without impact to customer operations. In this case, the initial failover operation completed as expected, but the backup components did not handle the traffic they received as intended. Both the cause of the failover and the issue which caused the backup components to properly handle traffic have been identified. Improvements have been planned which should prevent similar failures.

Read the full incident report →

Started Apr 28, 2025, 03:02 PM UTC· resolved Apr 28, 2025, 04:08 PM UTC

1. investigating Apr 28, 2025, 03:02 PM UTC

   We are currently investigating this issue.

2. identified Apr 28, 2025, 03:03 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Apr 28, 2025, 03:12 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Apr 28, 2025, 04:08 PM UTC

   This incident has been resolved.

5. postmortem Apr 30, 2025, 07:35 PM UTC

   For approximately 14 minutes during the morning of April 28th, some US customers experienced intermittent issues with phone registration and call setup. The issue resulted from a high level of database load and was resolved rapidly by the engineering team. The causes of the database traffic in question have been identified and measures are being implemented which should prevent recurrence under similar scenarios.

Read the full incident report →

Started Apr 13, 2025, 05:12 AM UTC· resolved Apr 13, 2025, 06:02 AM UTC

1. identified Apr 13, 2025, 05:12 AM UTC

   Clearly IP NOC personnel have identified an issue impacting connectivity to US East SIP Trunking services by an upstream vendor. All other SIP Trunking locations/regions remain unaffected.

2. resolved Apr 13, 2025, 06:02 AM UTC

   Impacts to connectivity at the upstream vendor have been resolved.

Read the full incident report →

Started Apr 08, 2025, 02:29 PM UTC· resolved Apr 08, 2025, 06:17 PM UTC

1. investigating Apr 08, 2025, 02:52 PM UTC

   Call Panel and Management interface offline on Clearly Cloud USA

2. identified Apr 08, 2025, 02:53 PM UTC

   The team has identified the issue and is working on resolution.

3. identified Apr 08, 2025, 03:26 PM UTC

   The team is still working through checks.

4. identified Apr 08, 2025, 03:28 PM UTC

   Clearly Cloud is now resolved.

5. identified Apr 08, 2025, 04:16 PM UTC

   We are continuing to work on a fix for this issue.

6. monitoring Apr 08, 2025, 04:18 PM UTC

   All system checks have been completed and issues are resolved.

7. resolved Apr 08, 2025, 06:17 PM UTC

   This incident has been resolved.

8. postmortem Apr 10, 2025, 01:02 PM UTC

   Our response began immediately when availability monitors detected the possibility of an issue with Clearly Cloud operations. It was confirmed that some Clearly Cloud operations were impacted and the NOC team worked to restore full functionality as quickly as possible. As service was restored, teams began to identify the sequence of events which caused the description and root causes. ‌ Ultimately, it was determined that a core network switch malfunctioned for a very brief period, automatically triggering protective high-availability systems. This led to a number of resources shifting to work around the perceived issue, leading to manual intervention due to the nature of the actual equipment failure. ClearlyIP has initiated a proactive replacement of all switches of the same type, revised incident response protocols, and improved processes which will both reduce the likelihood of recurrence and significantly reduce recovery times for other situations.

Read the full incident report →

Started Apr 01, 2025, 06:15 PM UTC· resolved Apr 01, 2025, 10:06 PM UTC

1. investigating Apr 01, 2025, 09:17 PM UTC

   We are currently investigating an issue with ComXchange licensing servers which may result in an invalid license status.

2. identified Apr 01, 2025, 10:00 PM UTC

   The issue has been identified and a fix is being implemented.

3. resolved Apr 01, 2025, 10:06 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 06, 2025, 03:00 PM UTC· resolved Mar 07, 2025, 03:15 AM UTC

1. investigating Mar 06, 2025, 03:00 PM UTC

   We are currently investigating a data center internet issue that is leading to high delays on inbound and outbound calling.

2. identified Mar 06, 2025, 03:32 PM UTC

   We are seeing alerts closing and believe the issue may be resolved. The team is actively checking all systems to verify.

3. monitoring Mar 06, 2025, 04:43 PM UTC

   Update: Data center connectivity is restoring, and systems should be coming back online now. If you are still experiencing issues, please open a ticket at support.clearlyip.com. Our team is monitoring and verifying full system recovery.

4. identified Mar 06, 2025, 05:09 PM UTC

   Clearly Cloud USA is still reporting some issues.

5. monitoring Mar 06, 2025, 05:22 PM UTC

   Data center connectivity is restoring, and systems should be coming back online now. If you are still experiencing issues, please open a ticket at support.clearlyip.com. Our team is monitoring and verifying full system recovery.

6. monitoring Mar 06, 2025, 06:58 PM UTC

   We are still seeing intermittent issues with incoming and outgoing calls, primarily with our Clearly Cloud US platform. The team is still working on this.

7. identified Mar 06, 2025, 07:29 PM UTC

   The issue has been identified and a fix is being implemented.

8. monitoring Mar 06, 2025, 08:30 PM UTC

   The team has stabilized the system. If you are still experiencing issues, please open a ticket at support.clearlyip.com. Our team is monitoring and verifying full system recovery.

9. monitoring Mar 06, 2025, 08:30 PM UTC

   We are continuing to monitor for any further issues.

10. resolved Mar 07, 2025, 03:15 AM UTC

    The issue was resolved at approximately 16:00 CST. Details about the cause of the issue and its scope have been emailed to all customers and partners. If you have not received this message, please email your account representative or contact support.

11. postmortem Mar 07, 2025, 08:44 PM UTC

    The issue was resolved at approximately 16:00 CST. Details about the cause of the issue and its scope have been emailed to all customers and partners. If you have not received this message, please email your account representative or contact support.

Read the full incident report →

Started Mar 04, 2025, 11:45 AM UTC· resolved Mar 04, 2025, 01:09 PM UTC

1. investigating Mar 04, 2025, 02:48 PM UTC

   We are currently investigating an issue with Clearly Cloud USA Admin GUI and Call Panel.

2. resolved Mar 04, 2025, 02:50 PM UTC

   Web access has been fully restored.

3. postmortem Mar 08, 2025, 07:31 PM UTC

   At approximately 05:39 CST, our web service began restarting irregularly. This did not affect inbound or outbound calling or registration, but did result in the Management and Call Panel web properties to fail to load. At 07:59 CST, the service was taken completely offline, and restarted cleanly, and service resumed.

Read the full incident report →

Started Feb 25, 2025, 07:24 PM UTC· resolved Feb 25, 2025, 10:24 PM UTC

1. investigating Feb 25, 2025, 07:24 PM UTC

   We are investigating a possible network issue.

2. monitoring Feb 25, 2025, 08:31 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Feb 25, 2025, 10:24 PM UTC

   This incident has been resolved.

4. postmortem Feb 26, 2025, 04:36 PM UTC

   This was a partial outage which affected customers utilizing our Clearly Cloud USA Cloud PBX product. Customers may have intermittent registration and call issues on our Clearly Cloud USA Cloud PBX product related to an extreme CPU load change. The root cause of this was due to internal API calls that were abnormally high. As a result of this issue, the engineering team has adjusted the rate limits on the affected APIs to better deal with the situation that occurred. This helped bring our load down and back to normal levels, resolving the issues.

Read the full incident report →

Started Feb 25, 2025, 03:22 AM UTC· resolved Feb 25, 2025, 04:02 AM UTC

1. investigating Feb 25, 2025, 03:22 AM UTC

   We are currently investigating this issue.

2. monitoring Feb 25, 2025, 03:29 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Feb 25, 2025, 04:02 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Feb 10, 2025, 03:20 PM UTC· resolved Feb 10, 2025, 10:20 PM UTC

1. investigating Feb 10, 2025, 03:20 PM UTC

   Some users are experiencing issues with queue calls not making it to agents. The team is investigating.

2. monitoring Feb 10, 2025, 03:38 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Feb 10, 2025, 10:20 PM UTC

   This incident has been resolved.

4. postmortem Feb 10, 2025, 10:21 PM UTC

   One of the queue nodes began behaving erratically, so it was taken offline and a different node brought up to replace it.

Read the full incident report →

Started Feb 10, 2025, 04:48 AM UTC· resolved Feb 10, 2025, 06:24 AM UTC

1. identified Feb 10, 2025, 04:48 AM UTC

   We will be undergoing maintenance on a backend storage server that may cause intermittent calling issues with our Clearly Cloud USA servers.

2. resolved Feb 10, 2025, 06:24 AM UTC

   This incident has been resolved.

3. postmortem Feb 10, 2025, 06:24 AM UTC

   Related to Feb 9, 2025 issue, the storage server was decommissioned, which required maintenance to Clearly Cloud USA to complete.

Read the full incident report →

Started Feb 08, 2025, 07:14 PM UTC· resolved Feb 08, 2025, 07:18 PM UTC

1. identified Feb 08, 2025, 07:14 PM UTC

   The team has identified the issue and is taking steps to correct.

2. resolved Feb 08, 2025, 07:18 PM UTC

   This incident has been resolved.

3. postmortem Feb 08, 2025, 07:33 PM UTC

   A back end storage server is experiencing issues. Our monitoring detected the issues and our team forced failovers to mitigate the storage server issue.

Read the full incident report →

Started Feb 06, 2025, 06:23 PM UTC· resolved Feb 06, 2025, 07:38 PM UTC

1. identified Feb 06, 2025, 06:23 PM UTC

   We have identified an issue with our US Central servers and our team is investigating. If you are setup with multiple trunking servers, there should be no issue. If you are using only our central servers, you will want to change your registration to other servers or the time being.

2. resolved Feb 06, 2025, 07:38 PM UTC

   This incident has been resolved.

3. postmortem Feb 08, 2025, 07:31 PM UTC

   Our monitoring systems detected an inconsistency in our database servers, causing our central trunking servers to fail to take calls. NOC put the central servers into maintenance mode, causing all calls to be routed though different trunking servers \(depending on individual customer setups\) and began resolving the database issue. Once resolved, the central trunking servers were brought back online.

Read the full incident report →

Started Feb 05, 2025, 07:46 PM UTC· resolved Feb 05, 2025, 09:56 PM UTC

1. investigating Feb 05, 2025, 07:46 PM UTC

   We are currently investigating this issue.

2. investigating Feb 05, 2025, 07:48 PM UTC

   We are currently investigating an issue causing call disruption, registration issues, and website access.

3. identified Feb 05, 2025, 08:20 PM UTC

   We are currently investigating, however this appears to be related to Cloud Flare issue. See https://www.cloudflarestatus.com/ for more info on their issue. For customers using Cloud Flare as their DNS (1.1.1.1 or 2.2.2.2), updating your DNS appears to resolve the issue. We use Cloud Flare DNS as well as other DNS providers, and as a precautionary measure, we are removing Cloud Flare DNS from critical areas to limit the delays that may be caused by Cloud Flare.

4. monitoring Feb 05, 2025, 09:07 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved Feb 05, 2025, 09:56 PM UTC

   This incident has been resolved.

6. postmortem Feb 08, 2025, 07:28 PM UTC

   Cloud Flare DNS was having issues with Chicago location \(See [https://www.cloudflarestatus.com/](https://www.cloudflarestatus.com/) for more info on their issue.\) For customers using Cloud Flare as their DNS \(1.1.1.1 or 2.2.2.2\), updating your DNS appears to resolve the issue. We use Cloud Flare DNS as well as other DNS providers, and as a precautionary measure, we removed Cloud Flare DNS from critical areas to limit the delays that may be caused by Cloud Flare.

Read the full incident report →

Started Dec 05, 2024, 07:43 AM UTC· resolved Dec 05, 2024, 07:59 AM UTC

1. identified Dec 05, 2024, 07:43 AM UTC

   The issue has been identified and a fix is being implemented.

2. monitoring Dec 05, 2024, 07:49 AM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring Dec 05, 2024, 07:50 AM UTC

   We are continuing to monitor for any further issues.

4. resolved Dec 05, 2024, 07:59 AM UTC

   This incident has been resolved.

Read the full incident report →