ClearlyIP experienced a minor incident on October 30, 2025 affecting US Central, lasting 51m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 30, 2025, 08:21 PM UTC
We are currently investigating this issue.
- monitoring Oct 30, 2025, 08:51 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Oct 30, 2025, 09:13 PM UTC
This incident has been resolved.
- postmortem Nov 03, 2025, 08:39 PM UTC
At approximately 4:10 PM ET, ClearlyIP's monitoring systems detected registration issues within US-Central trunking. Within a few minutes, it was determined that one of the two primary trunking servers was maintaining its active calls \(2.us-central\) and that its counterpart \(1.us-central\) had stalled and was not accepting traffic. It was quickly determined that the issue stemmed from a database cluster deadlock within the US-Central footprint. Upon analysis, the deadlock's cause was identified to be related to unsuccessful credential change activity. A series of actions were taken during the next 30 minutes which restored 2.us-central to service. Following that, maintenance was performed to resolve version differences on the database node not currently in production before restoring it \(and 1.us-central\) to normal service shortly after 6PM ET.