Is Cloudli down?

Dear clients, We are currently investigating an issue with the Cloudli Connect service. This may impact some clients ability make and receive calls. We will update you upon further discovery. *** Chers clients, Nous enquêtons actuellement sur un problème affectant le service Cloudli Connect. Cela pourrait affecter la capacité de certains clients à passer et recevoir des appels. Nous vous tiendrons informés dès que nous aurons plus d'informations.

The issue has been identified and fix is being implemented. *** Le problème a été identifié et le correctif est en cours de mise en œuvre.

A fix has been implemented and we are monitoring the results. *** Un correctif a été mis en œuvre et nous surveillons les résultats.

Additional improvements have been implemented, and we are continuing to monitor the results. *** D'autres améliorations ont été apportées et nous continuons de surveiller les résultats.

This incident has been resolved. We will provide an RFO no later than three (3) business days from today. *** Cet incident a été résolu. Nous fournirons un RFO au plus tard dans trois (3) jours ouvrables à partir d'aujourd'hui.

## Summary of Events: On July 29, 2026, beginning at approximately 9:07 AM EDT, Cloudli was notified of system alarms affecting portions of the Cloudli Connect voice platform. Two client hosts supporting registrar workloads experienced kernel panics caused by Media service modules operating in kernel mode. The incident affected SIP registrations and carrier connectivity and resulted in failed calls, including 603 “All Routes Exhausted” responses, as well as isolated calls with no audio issues. ## Incident Analysis and Mitigation Measures: Initial alarms were received at 9:07 AM EDT, and troubleshooting began at 9:10 AM. The unresponsive client host was rebooted at 9:15 AM. Cloudli posted investigating and identified updates at 9:21 AM and 9:23 AM, respectively. Test calls succeeded by 9:30 AM, and monitoring began at 9:33 AM, marking the system “Operational.” At this time Cloudli NOC continued monitoring the systems proactively as we began root cause analysis to implement a resolution. Cloudli issued a monitoring update at 12:47 PM notifying partners and customers of additional measures taken to mitigate the issue. The incident was formally resolved at 2:26 PM EDT. Root cause analysis determined that client hosts experienced kernel panics associated with media service modules running in kernel mode for improved performance. The host failures disrupted registrar services and carrier routing-data access. Although cluster jobs were distributed across separate servers and datacenters, each registrar cluster relies on a single caching database instance, by design, for carrier database access. When the carrier cluster could not access the caching database, calls returned 603 “All Routes Exhausted” responses. The media service modules instability also caused intermittent audio failures. Service was restored by rebooting the affected client hosts and restarting the media service module instances associated with audio failures. Engineering will continue investigating media service modules’ kernel-mode panic behavior and implement options that preserve performance and host stability. Cloudli will also complete additional redundancy work, including removal of the single-instance caching database dependency within each registrar cluster. ## Final Remarks: At Cloudli, we take any interruption of service very seriously and are continuously evaluating new processes and mitigation measures that can be proactively implemented to ensure service continuity. When service interruptions do occur, our incident management procedure prioritizes prompt and clear notification, and timely status and resolution updates to our customers and partners. We thank you for your continued support. Please feel free to reach out if you would like to discuss the particulars of this incident report further.

We are currently investigating an API issue with the virtual fax product. This issue could have an impact on services for users using the API - our team is analyzing the situation. Regular virtual fax customers not using the API are not affected. *** Nous enquêtons actuellement sur un problème d'API concernant le service de fax virtuel. Ce problème pourrait impacter les services pour les utilisateurs de l'API ; notre équipe analyse la situation. Les clients utilisant le service de fax virtuel sans l'API ne sont pas concernés.

A fix has been implemented and we are monitoring the results. *** Un correctif a été mis en œuvre et nous surveillons les résultats.

This incident has been resolved. *** Cet incident a été résolu.

## Executive Summary Virtual Fax API communications were disrupted. This impacted the provisioning of Virtual Fax accounts, fax transmission through both the API and Portal, and retrieval of faxes through the API and Portal. Fax2Email and Email2Fax services remained fully operational throughout the incident. ## Root Cause Analysis On June 13, 2026, at 23:17 \(EST\), the 3rd-Party Fax Backend LetsEncrypt certificate was automatically renewed. As part of the certificate renewal process, a new intermediate signing certificate was introduced. The server responsible for Cloudli Virtual Fax API integrations did not have this intermediate certificate present in its trusted certificate store and was therefore unable to validate the renewed 3rd-Party Fax Backend certificate. ## Incident Analysis and Mitigation Measures: Because the underlying server infrastructure and messaging services remained operational, existing monitoring systems did not detect the issue. The first customer-reported indication of the problem occurred on June 14, 2026, at 23:57 EDT when ProcessFusion opened a support ticket reporting fax failures through Portal. The NOC team was notified on June 15, 2026, at 08:45 EDT and immediately began investigating. ## Conclusion The issue was resolved at 10:30 EST on June 15, 2026, after the missing intermediate signing certificate was imported into the server trusted certificate store. Additional monitoring was implemented later that day to proactively detect similar failures and automatically notify the on-call team. ## Final Remarks: At Cloudli, we take any interruption of service very seriously and are continuously evaluating new processes and mitigation measures that can be proactively implemented to ensure service continuity. When service interruptions do occur, our incident management procedure prioritizes prompt and clear notification, along with timely status updates and resolution communications to our customers and partners. We thank you for your continued support. Please feel free to reach out if you would like to discuss the particulars of this incident report further.

One Canadian Telco provider is currently experiencing an outage that affects inbound calls to some DIDs at Windsor, ON and Chatham, ON. Impacts: This could potentially cause abnormal delays or an inability to receive calls in those regions. ********** Un fournisseur de télécommunications canadien rencontre actuellement une panne qui affecte les appels entrants vers certains numéros de Windsor, ON et de Chatham, ON . Conséquences : Cette panne pourrait entraîner des retards anormaux ou l’impossibilité de recevoir des appels dans ces régions.

The Canadian Telco is conducting detailed troubleshooting and further analysis. We will keep you updated with more information once it becomes available. ******** Le fournisseur de télécommunications canadien effectue un dépannage détaillé et une analyse approfondie. Nous vous tiendrons informés avec davantage d’informations dès qu’elles seront disponibles.

The issue has been escalated to our carrier's upstream provider, where the underlying network issue has been identified. We will continue to monitor the situation closely and share updates as they become available. ********* Le problème a été transmis au fournisseur en amont de notre transporteur, où la cause sous-jacente du problème de réseau a été identifiée. Nous continuerons à surveiller la situation de près et à communiquer des mises à jour dès qu’elles seront disponibles.

According to local ILEC of the affected regions, the estimate time of repair (ETR) is 2026/06/03 04:00 EST. We continue to monitor the situation and will update you accordingly. ********** Selon l'opérateur local des regions affectés, l'heure de réparation estimée est le 3 juin 2026 à 4 h 00 HNE. Nous continuons de suivre la situation et vous tiendrons informés.

According to affected local ILEC, they are still working on it. We will provide you more update once we have it. **** D'après l'opérateurs local concerné, Ils y travaillent encore.. Nous vous tiendrons informés dès que nous aurons plus d'informations.

This issue is related to a major outage in the area, and the local ILEC transport team is actively working to resolve it. We will keep you updated with more information once it becomes available. **** Ce problème est lié à une panne majeure dans la région, et l'équipe de transport de l'opérateur local travaille activement à sa résolution. Nous vous tiendrons informés dès que nous aurons plus d'informations.

Our upstream local carrier has confirmed a major outage caused by a cable cut and a fault within the ring infrastructure. Their network team is actively reconfiguring affected circuits to restore service. An estimated time to restoration (ETR) is not yet available. The incident remains a high priority, and we will provide further updates as they become available. **************** Notre transporteur local en amont a confirmé une panne majeure causée par une coupure de câble et une défaillance de l’infrastructure en anneau. Son équipe réseau procède actuellement à la reconfiguration des circuits touchés afin de rétablir les services. Aucun délai estimé de rétablissement (ETR) n’est disponible pour le moment. L’incident demeure une priorité élevée et nous communiquerons toute nouvelle mise à jour dès qu’elle sera disponible.

This incident has been resolved. We got notification from the affected local carrier that the transport issue on previously reported rate centers have been resolved. Calls to the affected rate centers are now successfully completing without any issues. *** Cet incident a été résolu. Nous avons reçu une notification de l'opérateur local concerné indiquant que le problème de transport précédemment signalé aux centres de tarification a été résolu. Les appels vers les centres de tarification concernés aboutissent désormais sans problème.

One Canadian Telco provider is currently experiencing an outage that affects inbound calls to different regions in Ontario and Quebec (Canada). Impacts: This could potentially cause abnormal delays or an inability to receive calls in those regions. ********** Un fournisseur de télécommunications canadien rencontre actuellement une panne qui affecte les appels entrants vers certains numéros dans différentes régions de l'Ontario et du Québec (Canada). Conséquences : Cette panne pourrait entraîner des retards anormaux ou l’impossibilité de recevoir des appels dans ces régions.

The Canadian Telco provider has resolved the issue, and inbound calls have been working fine since 13:40 EDT. ********** Le fournisseur de télécommunications canadien a résolu le problème et les appels entrants fonctionnent correctement depuis 13h40 HAE.

The Canadian Telco provider resolved the issue since 13:40 EDT. ********** Le fournisseur de télécommunications canadien a résolu le problème depuis 13h40 HAE.