NUACOM incident
Investigating issues with Cloud PBX ,Severity: major
NUACOM experienced a major incident on August 26, 2024, lasting 41m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Aug 26, 2024, 01:19 PM UTC
We are investigating an incident affecting Cloud PBX. We will provide updates via email and Statuspage shortly.
- monitoring Aug 26, 2024, 01:34 PM UTC
A fix has been implemented and we are monitoring the system
- resolved Aug 26, 2024, 04:30 PM UTC
The monitoring system hasn't detected any further issues, so we're closing this incident for now.
- postmortem Aug 26, 2024, 05:14 PM UTC
**Summary:** One of our database clusters became overloaded, leading to an outage that impacted multiple accounts. During this period, affected users were unable to make or receive calls. **Root Cause:** The outage was caused by a significant overload on one of the database cluster hosts. This overload resulted in a bottleneck that prevented the database from processing requests efficiently, disrupting user service. **Resolution:** Our engineering team promptly identified the issue and re-routed all heavy queries to other hosts within the same cluster. This action successfully alleviated the load on the affected host, restoring normal operations. **Preventive Measures:** We will review and optimize query distribution across the cluster to prevent future overloads. Additional monitoring and alerts will be implemented to detect and address similar issues more quickly. **Next Steps:** Conduct a thorough analysis of the affected queries to identify optimization opportunities. Consider expanding the cluster capacity or implementing more granular load balancing. We apologize for the inconvenience this outage caused and are committed to preventing similar incidents in the future.