ClearlyIP experienced a minor incident on April 8, 2025 affecting Hosted PBX and Clearly Cloud USA - w, lasting 3h 47m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 08, 2025, 02:52 PM UTC
Call Panel and Management interface offline on Clearly Cloud USA
- identified Apr 08, 2025, 02:53 PM UTC
The team has identified the issue and is working on resolution.
- identified Apr 08, 2025, 03:26 PM UTC
The team is still working through checks.
- identified Apr 08, 2025, 03:28 PM UTC
Clearly Cloud is now resolved.
- identified Apr 08, 2025, 04:16 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Apr 08, 2025, 04:18 PM UTC
All system checks have been completed and issues are resolved.
- resolved Apr 08, 2025, 06:17 PM UTC
This incident has been resolved.
- postmortem Apr 10, 2025, 01:02 PM UTC
Our response began immediately when availability monitors detected the possibility of an issue with Clearly Cloud operations. It was confirmed that some Clearly Cloud operations were impacted and the NOC team worked to restore full functionality as quickly as possible. As service was restored, teams began to identify the sequence of events which caused the description and root causes. Ultimately, it was determined that a core network switch malfunctioned for a very brief period, automatically triggering protective high-availability systems. This led to a number of resources shifting to work around the perceived issue, leading to manual intervention due to the nature of the actual equipment failure. ClearlyIP has initiated a proactive replacement of all switches of the same type, revised incident response protocols, and improved processes which will both reduce the likelihood of recurrence and significantly reduce recovery times for other situations.