NUACOM Outage History
NUACOM is up right nowPingoru has recorded 51 incidents for NUACOM: 16 in the last 12 months and 4 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 1 | 1 of 720 (0%) |
| May 2026 | 1 | 3 of 744 (0%) |
| June 2026 | 1 | 0 of 720 (0%) |
| July 2026 | 3 | 20 of 744 (3%) |
| August 2026 | 1 | 4 of 744 (1%) |
| September 2026 | 2 | 32 of 720 (4%) |
| October 2026 | 1 | 1 of 512 (0%) |
Incidents, newest first
Minor October 16, 2026 Slow performance of Management Portal and API 59m Public API, Management Portal
Started Oct 16, 2026, 10:50 AM UTC· resolved Oct 16, 2026, 11:50 AM UTC
- investigating Oct 16, 2026, 10:50 AM UTC
We are aware that some customers are experiencing slow performance and occasional errors when using the Management Portal and related services. Our team is investigating and we will post an update as soon as we have more information.
- identified Oct 16, 2026, 11:39 AM UTC
We have identified the cause of the slow performance affecting the Management Portal and related services and a fix has been applied. Performance has been improving since around 12:20 Irish time and we are continuing to monitor. We will post a further update shortly.
- resolved Oct 16, 2026, 11:50 AM UTC
This incident has been resolved. Between approximately 11:10 and 11:50 Irish time, some customers experienced slow performance and errors in the Management Portal and related services, and some new calls may have failed to connect or taken longer to set up. Phones that were already registered stayed connected. Service has returned to normal and we will continue to monitor. We apologise for the disruption.
Notice September 14, 2026 Virgin Media Internet Connectivity Issues Affecting Customers 22m
Started Sep 14, 2026, 03:45 PM UTC· resolved Sep 14, 2026, 04:07 PM UTC
- investigating Sep 14, 2026, 03:45 PM UTC
We are aware that some customers using Virgin Media for their internet connectivity are currently experiencing service disruptions, which is impacting their ability to make and receive calls over the Nuacom platform. This is an issue with Virgin Media's network, not with Nuacom's infrastructure. Affected customers should contact Virgin Media directly to resolve their connectivity issues. Our platform and voice services are operating normally. If you are experiencing call issues and are NOT a Virgin Media customer, or if your Virgin Media connection is stable but you are still unable to make/receive calls, please contact our support team.
- resolved Sep 14, 2026, 04:07 PM UTC
The brief inbound call disruption reported over the last 30 minutes has been resolved. Calls are now being received normally. We do not yet have confirmation of the root cause, but it may have been related to the earlier Virgin Media connectivity issues affecting some customers. If you continue to experience call delivery problems, please contact our support team. Thank you for your patience.
Minor September 1, 2026 Yealink handsets showing "No Service" / failing to register 1d 7h Cloud PBX
Started Sep 01, 2026, 09:49 AM UTC· resolved Sep 02, 2026, 05:38 PM UTC
- identified Sep 01, 2026, 09:49 AM UTC
We're aware that some Yealink desk phones are showing "No Service" or "Invalid Line" and failing to register. Our team has identified the cause and is working through affected accounts to apply the fix. If your Yealink handset is showing this error, please contact our support team — in the cases we've seen so far this has resolved the issue immediately. We'll update this page as remaining accounts are resolved.
- monitoring Sep 01, 2026, 10:18 AM UTC
A fix has been implemented. We're now monitoring affected accounts to confirm Yealink handsets are registering normally.
- monitoring Sep 01, 2026, 10:19 AM UTC
A fix has been implemented. We're now monitoring affected accounts to confirm Yealink handsets are registering normally. We'll post a final update once this is confirmed resolved.
- resolved Sep 02, 2026, 05:38 PM UTC
This incident is now resolved. The configuration issue that caused some Yealink desk phones to show "No Service" or "Invalid Line" was fixed on the morning of 1 September, and monitoring since then shows affected handsets registering normally with no further reports. A permanent configuration change will follow as part of our normal maintenance and will not affect service. If any handset at your site is still showing this error, please restart it and contact our support team if it does not recover.
Major August 4, 2026 Intermittent call failures affecting some accounts 4h 3m Incoming Calls, Outbound Calls
Started Aug 04, 2026, 09:50 AM UTC· resolved Aug 04, 2026, 01:54 PM UTC
- investigating Aug 04, 2026, 09:50 AM UTC
We are investigating reports of intermittent call failures affecting inbound and outbound calls on a subset of customer accounts.
- identified Aug 04, 2026, 09:56 AM UTC
We have identified the cause of the incident as an issue affecting one of our platform databases, which is impacting call handling for a subset of accounts. Our engineering team is actively working on a fix. We will update this incident once the issue is resolved and services are fully restored.
- identified Aug 04, 2026, 10:17 AM UTC
We are continuing to work on a fix for this issue.
- monitoring Aug 04, 2026, 10:18 AM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Aug 04, 2026, 10:18 AM UTC
We are continuing to monitor for any further issues.
- monitoring Aug 04, 2026, 10:30 AM UTC
We are continuing to monitor for any further issues.
- resolved Aug 04, 2026, 01:54 PM UTC
This incident has been resolved.
Minor July 21, 2026 Intermittent one-way audio on some inbound PSTN calls (upstream carrier) 17h 14m Incoming Calls
Started Jul 21, 2026, 03:51 PM UTC· resolved Jul 22, 2026, 09:06 AM UTC
- investigating Jul 21, 2026, 03:51 PM UTC
We are investigating reports of one-way audio affecting a subset of inbound calls, where the caller cannot be heard by the answering agent.
- investigating Jul 21, 2026, 03:51 PM UTC
Analysis indicates the issue originates with our upstream telephony carrier rather than the Nuacom platform, and it affects individual calls intermittently rather than all traffic. Outbound calls and the majority of inbound calls are unaffected. We have raised a fault with the carrier and will update as we learn more.
- identified Jul 21, 2026, 03:52 PM UTC
The issue has been identified and the issue is raised with upstream provider.
- identified Jul 21, 2026, 03:52 PM UTC
We are continuing to work on a fix for this issue.
- resolved Jul 22, 2026, 09:06 AM UTC
This issue was resolved last night 11:30PM by the upstream provider.
Notice July 21, 2026 Intermittent one-way audio on some inbound calls (from upstream carrier) –
Started Jul 21, 2026, 08:00 AM UTC· resolved Jul 21, 2026, 08:00 AM UTC
- resolved Jul 21, 2026, 03:50 PM UTC
We are investigating reports of one-way audio affecting a subset of inbound calls, where the caller cannot be heard by the answering agent.
Minor July 6, 2026 Issues with incoming audio on calls 2h 32m Incoming Calls
Started Jul 06, 2026, 08:34 AM UTC· resolved Jul 06, 2026, 11:07 AM UTC
- investigating Jul 06, 2026, 08:34 AM UTC
Small number of accounts may experience issues with incoming audio
- identified Jul 06, 2026, 08:36 AM UTC
The issue has been identified and a fix is currently being applied. We will post a further update once the service has been fully restored.
- monitoring Jul 06, 2026, 09:34 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jul 06, 2026, 11:07 AM UTC
Resolved — The outgoing call issue affecting select accounts has been fully resolved. The root cause has been identified and an internal investigation is ongoing to prevent recurrence. We apologise for the disruption caused.
Major June 11, 2026 Service Notice — Outgoing Call Issues Affecting Select Accounts 19m Outbound Calls
Started Jun 11, 2026, 01:50 PM UTC· resolved Jun 11, 2026, 02:10 PM UTC
- investigating Jun 11, 2026, 01:50 PM UTC
We wanted to let you know that we are currently seeing an issue with outgoing calls affecting a small number of accounts. Our team is on it and working to resolve this as quickly as possible. Incoming calls are working as normal. We will keep you updated and apologise for any disruption caused.
- identified Jun 11, 2026, 02:09 PM UTC
Our team has identified the cause of the outgoing call issue. We are working on a fix right now and aim to have everything back to normal as soon as possible. We will update you again once it is resolved.
- monitoring Jun 11, 2026, 02:09 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jun 11, 2026, 02:10 PM UTC
We are pleased to confirm that the outgoing call issue affecting a select number of accounts has been fully resolved.
Minor May 6, 2026 Issue with incoming calls 3h 10m Incoming Calls
Started May 06, 2026, 09:45 AM UTC· resolved May 06, 2026, 12:55 PM UTC
- investigating May 06, 2026, 09:45 AM UTC
We are currently experiencing an issue affecting incoming calls. Some customers may find that inbound calls are not being received. Our team is actively investigating the cause. Further updates will be posted here shortly.
- identified May 06, 2026, 09:53 AM UTC
We have identified the cause of the issue affecting incoming calls. Our engineers are currently monitoring the situation closely. We appreciate your patience and will continue to provide updates.
- monitoring May 06, 2026, 10:03 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved May 06, 2026, 12:55 PM UTC
This incident has been resolved.
Major April 7, 2026 Mobile app degraded performance, you may have issue to make and receive calls 32m Mobile App, Incoming Calls, Outbound Calls
Started Apr 07, 2026, 12:49 PM UTC· resolved Apr 07, 2026, 01:22 PM UTC
- investigating Apr 07, 2026, 12:49 PM UTC
We are investigating the issue
- identified Apr 07, 2026, 01:16 PM UTC
The issue has been identified and a fix is being implemented.
- resolved Apr 07, 2026, 01:22 PM UTC
The issue has now been resolved. To apply the changes, please log out of the mobile app and log back in.
Minor February 19, 2026 Issues making and receiving calls 34m Desktop App, Mobile App, Cloud PBX, Incoming Calls and 1 more
Started Feb 19, 2026, 09:56 AM UTC· resolved Feb 19, 2026, 10:31 AM UTC
- identified Feb 19, 2026, 09:56 AM UTC
We have identified an issue with a recent update that is affecting our database, slowing it down. We are rolling back the changes and will come with an update soon.
- monitoring Feb 19, 2026, 10:16 AM UTC
The rollback has been implemented at 10:04 GMT, and we are monitoring the systems.
- resolved Feb 19, 2026, 10:31 AM UTC
The issue has been resolved. Apologies for the inconvenience caused.
Minor January 23, 2026 Public API server error –
Started Jan 23, 2026, 02:00 AM UTC· resolved Jan 23, 2026, 02:00 AM UTC
- resolved Jan 26, 2026, 11:25 AM UTC
API 500 Server Error
- postmortem Jan 26, 2026, 11:27 AM UTC
## Service Incident Summary – January 23, 2026 On January 23, 2026, Nuacom experienced a service disruption that affected access to the platform for a period of time. ### What happened Between 02:30 and 07:41 \(Ireland time\), some customers experienced difficulties accessing Nuacom services. This included issues with the web portal, apps, and API-dependent features. A small number of customers also experienced temporary visibility issues with SMS history. ### Impact * Some users were unable to access services during the incident window * Core SMS delivery continued to function as normal * The issue was resolved the same morning and all services were restored ### Resolution Our engineering team identified the cause of the disruption and applied corrective actions to restore full service. The system has been stable since resolution, and we continue to monitor performance closely. ### Data and records No customer configuration data was lost. In cases where call or message history was temporarily unavailable during the incident, visibility has now been restored where applicable. ### Preventive actions We have reviewed our deployment and monitoring processes and are implementing improvements to: * Detect similar issues earlier * Reduce the likelihood of recurrence * Improve communication during incidents ### Our apology We apologise for the inconvenience caused and appreciate your patience and understanding. If you have any questions or notice anything unusual, please contact our support team.
Minor January 19, 2026 Contact name look-up, Portal Reports and Call Logs service are temporarily unavailable 19h 58m Public API, Desktop App, Mobile App, 3rd Party Integrations
Started Jan 19, 2026, 11:30 AM UTC· resolved Jan 20, 2026, 07:28 AM UTC
- identified Jan 19, 2026, 11:30 AM UTC
Investigating / Mitigation To reduce system load, we have temporarily disabled the non-critical Contact Name Look-up, Portal Reports and Call Logs service. We are actively working to resolve the load issue and will restore service as soon as possible.
- resolved Jan 20, 2026, 07:28 AM UTC
Our team has identified the issue related to database cluster degradation. During the maintenance throughout the night, the team has restored enough database capacity to cover the core elements of the service. We are continuously monitoring and ready to respond in case of new issues. Apologies to all the clients and partners affected by this incident.
Minor January 19, 2026 Phone system degraded performance 1d 1h Management Portal, Cloud PBX, Incoming Calls, Outbound Calls
Started Jan 19, 2026, 10:48 AM UTC· resolved Jan 20, 2026, 12:39 PM UTC
- investigating Jan 19, 2026, 10:48 AM UTC
We are investigating this issue
- identified Jan 19, 2026, 12:20 PM UTC
We are currently experiencing degraded system performance, which may result in slower response times. Our engineers have identified the issue and are actively implementing a solution. Thank you for your patience while we work to restore full performance.
- identified Jan 19, 2026, 04:38 PM UTC
Apologies for the delay in providing an update on the system status. Our team is still working toward a full resolution of the issue. The system remains operational; however, you may experience delays when calls are connecting.
- monitoring Jan 20, 2026, 07:28 AM UTC
Our team has identified the issue related to database cluster degradation. During the maintenance throughout the night, the team has restored enough database capacity to cover the core elements of the service. We are continuously monitoring and ready to respond in case of new issues. Apologies to all the clients and partners affected by this incident.
- resolved Jan 20, 2026, 12:39 PM UTC
The incident has been resolved. If you are still experiencing any issues, please don't hesitate to contact our customer care team. Apologies for any inconvenience caused,
Minor December 3, 2025 Issue with call audio using desktop app 35m Desktop App
Started Dec 03, 2025, 11:51 AM UTC· resolved Dec 03, 2025, 12:27 PM UTC
- investigating Dec 03, 2025, 11:51 AM UTC
We are currently investigating the issue.
- resolved Dec 03, 2025, 12:27 PM UTC
The issue with the calls using the desktop app is now resolved, apologies for the inconvenience caused. The team is investigating the root cause.
Minor October 23, 2025 IOS app degraded performance, you may have issue to receive incoming calls 1h 5m Mobile App, Incoming Calls
Started Oct 23, 2025, 02:36 PM UTC· resolved Oct 23, 2025, 03:42 PM UTC
- investigating Oct 23, 2025, 02:36 PM UTC
We are currently investigating this issue.
- resolved Oct 23, 2025, 03:42 PM UTC
The incident is now resoved, sorry for the inconvenience caused, the team is continuing the investigation into the root cause.
Minor September 16, 2025 Degraded performance on the Web applications 20m Public API, Management Portal, Desktop App, Mobile App and 2 more
Started Sep 16, 2025, 08:41 AM UTC· resolved Sep 16, 2025, 09:02 AM UTC
- investigating Sep 16, 2025, 08:41 AM UTC
We are observing issues with the API being slow or unavailable, we are investigating the cause of the issue.
- monitoring Sep 16, 2025, 08:53 AM UTC
The issue was identified and resolved. The NOC team is monitoring the performance.
- resolved Sep 16, 2025, 09:02 AM UTC
The issue is resolved, and the systems are back operational. Apologies for the problems caused.
Notice July 23, 2025 Desktop app connection 2h 49m Desktop App
Started Jul 23, 2025, 01:50 PM UTC· resolved Jul 23, 2025, 04:39 PM UTC
- investigating Jul 23, 2025, 01:50 PM UTC
Wea re currently investigating the this issue
- investigating Jul 23, 2025, 02:02 PM UTC
Please reload the desktop app by clicking on View → Reload (located at the top left corner of the app). This will make the app operational again. We apologise for the inconvenience caused.
- resolved Jul 23, 2025, 04:39 PM UTC
This incident has been resolved, all systems are fully operational. please reload your desktop app if stilll affected; How to reload the app: 1. Click on “View” in the top-left menu of the app. 2. Select “Reload” from the dropdown.
Minor July 16, 2025 Delay in the connecting of the calls 6h 4m Cloud PBX, Incoming Calls, Outbound Calls
Started Jul 16, 2025, 10:54 AM UTC· resolved Jul 16, 2025, 04:59 PM UTC
- investigating Jul 16, 2025, 10:54 AM UTC
We are currently investigating this issue.
- identified Jul 16, 2025, 11:14 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Jul 16, 2025, 11:15 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jul 16, 2025, 04:59 PM UTC
This incident has been resolved.
Minor April 16, 2025 Some calls processing may be affected for select accounts 3h 33m Management Portal, Cloud PBX, Incoming Calls, Outbound Calls
Started Apr 16, 2025, 08:45 AM UTC· resolved Apr 16, 2025, 12:18 PM UTC
- investigating Apr 16, 2025, 08:45 AM UTC
We are investigating this issue.
- investigating Apr 16, 2025, 08:46 AM UTC
We are currently investigating an issue that may impact call handling or performance for a limited number of accounts. Our team is actively working to identify and resolve the cause. We’ll provide further updates as soon as more information becomes available. Thank you for your patience.
- identified Apr 16, 2025, 09:14 AM UTC
The issue has been identified, and our team is currently working on implementing a resolution. We’ll provide further updates as progress is made. Thank you for your continued patience.
- identified Apr 16, 2025, 10:06 AM UTC
Our engineers are actively working on resolving the issue. We are focused on restoring full service as quickly as possible. Thank you for your continued patience and understanding.
- monitoring Apr 16, 2025, 10:41 AM UTC
We’ve implemented a fix and are currently monitoring the system closely to ensure everything remains stable. We sincerely apologise for the disruption this has caused and truly appreciate your patience and understanding as we worked to resolve the issue. Further updates will follow as needed.
- monitoring Apr 16, 2025, 11:07 AM UTC
We’re aware that the call log history may not be available for some accounts on the portal and mobile apps. Our team is currently working to restore full access. We apologise for any inconvenience this may cause and appreciate your continued patience. Updates will follow as we progress.
- monitoring Apr 16, 2025, 11:07 AM UTC
We’re aware that the call log history may not be available for some accounts on the portal and mobile apps. We are continuing to monitor for any further issues.
- monitoring Apr 16, 2025, 12:15 PM UTC
We are continuing to monitor for any further issues.
- resolved Apr 16, 2025, 12:18 PM UTC
The call logs are now available within the management portal, all the systems are now fully operational.
Notice April 7, 2025 We have experienced an outage on one of the gateways affecting some clients between 04:30 am and 09:40 IST –
Started Apr 07, 2025, 08:52 AM UTC· resolved Apr 07, 2025, 08:52 AM UTC
- resolved Apr 07, 2025, 08:57 AM UTC
One of our SIP gateways (k1) received many requests from the Internet, and it could not process all of them. Some IP handsets have experienced the error "SIP Registration Fail". We found the source of the issue and have resolved the issue. We are increasing the capacity of the system so it can deal with larger volumes of traffic.
Notice April 3, 2025 We have experienced a short outage on one of the gateways affecting some clients between 15:34 and 15:49 IST 1h 19m
Started Apr 03, 2025, 03:10 PM UTC· resolved Apr 03, 2025, 04:30 PM UTC
- resolved Apr 03, 2025, 03:10 PM UTC
One of our SIP gateways (k1) received a large amount of requests from the Internet Due to the high load, it was not able to process all requests. Some IP handsets have experienced the error "SIP Registration Fail". For about 10 min, the IP handsets which have an extension board, would not see the button's indication light working correctly.
Minor March 21, 2025 Reported Issue with BLF indications/function of the phones. 1h 29m Incoming Calls, Outbound Calls
Started Mar 21, 2025, 12:24 PM UTC· resolved Mar 21, 2025, 01:54 PM UTC
- investigating Mar 21, 2025, 12:24 PM UTC
We are currently investigating a reported issue affecting the BLF (Busy Lamp Field) functionality on some IP phones. Users may experience delays or failure in BLF status updates, such as incorrect or missing presence indicators (e.g., busy/available/idle). This may affect the call transfer, call pick up in some situations. Our engineering team is actively working to identify the root cause and restore full functionality as soon as possible.
- resolved Mar 21, 2025, 01:54 PM UTC
The issue with the BLF (IP handsets expansion panel key lights) has been resolved. The root cause of the issue was the sheer amount of Subscribe requests received from the handsets. We are planning maintenance windows over the weekend to improve the infrastructure configuration and avoid these issues happening again. Sincere apologies to anyone affected.
Minor March 13, 2025 Issue affecting incoming calls resolved. BLF buttons are affected for some clients 13h 46m Cloud PBX, Incoming Calls, Outbound Calls
Started Mar 13, 2025, 11:36 AM UTC· resolved Mar 14, 2025, 01:22 AM UTC
- investigating Mar 13, 2025, 11:36 AM UTC
We are investigating this issue
- investigating Mar 13, 2025, 11:37 AM UTC
We are continuing to investigate this issue.
- identified Mar 13, 2025, 12:09 PM UTC
The issue was identified. We were getting flooded with large amounts of traffic. We have implemented a temporary fix by blocking specific types of traffic with the side effect of not having the VoIP handsets extension panels "Busy light indicators" working. This should allow the phones to successfully register and make/receive calls. Further updates will follow. Sincere apologies to the customers affected.
- monitoring Mar 13, 2025, 02:39 PM UTC
Here's an update on the recent call disruption that affected some clients. A high volume of incoming traffic caused excessive load, leading to intermittent call failures. We have now blocked the specific traffic sources, which has stabilized service. **Some clients will not have their IP phone's side panel button lights working (BFL) until later today when we manage to enable that feature safely.** We will perform a maintenance window tonight to add further safeguards and prevent similar issues from recurring. While we aim to minimize any impact, brief interruptions may occur. We sincerely appreciate your patience and understanding as we work to improve our services. If you have any questions, please feel free to reach out at [email protected].
- resolved Mar 14, 2025, 01:22 AM UTC
This incident has been resolved.
Major February 27, 2025 Investigating issues with Cloud PBX ,Severity: major 4h 46m Cloud PBX
Started Feb 27, 2025, 01:52 PM UTC· resolved Feb 27, 2025, 06:39 PM UTC
- investigating Feb 27, 2025, 01:52 PM UTC
We are investigating an incident affecting Cloud PBX. We will provide updates via email and Statuspage shortly.
- resolved Feb 27, 2025, 06:39 PM UTC
All services are restored, and the NOC team closely monitors the system. The root cause was detected, and a future mitigation plan is put in place along with the incident report. Please accept our sincere apologies for the issues caused to your business.
- postmortem Mar 09, 2025, 10:02 PM UTC
## Incident Summary * **Date and Time \(Europe/Dublin\):** 27th Feb 2025, 13:50 – 17:00 * **Duration:** 3 hours 10 minutes * **Affected Services:** Calling system ## Incident Description **What Happened** At approximately 13:50, a critical hardware malfunction occurred on a data center server. The error message—“A PCI parity error was detected on a component”—pointed to a defective network card. As a result, the host machine became inoperable, and all virtual machines on this host went offline. Complicating matters, this server was the designated failover server for the primary machine that had experienced a similar malfunction earlier in the day. Because the failover server was still recovering and performing a file system check, it was not fully prepared to handle the additional load. Consequently, our operations team had to manually redistribute services to other servers, resulting in extended downtime. **Conclusion Summary** This was the second hardware failure to occur on the same day—an exceptional occurrence given our strong track record of zero server faults over the past five years. Moving forward, we are focusing our efforts on faster automated recoveries and more robust load and stress testing to prevent similar incidents. **Assurance to Our Customers** We sincerely apologize for any inconvenience caused by this incident We recognize the inconvenience this second outage caused and are committed to preventing similar incidents. Our team is actively refining both automated and manual failover processes to ensure quicker recovery times. Through enhanced monitoring, comprehensive testing, and ongoing infrastructure improvements, we will continue delivering reliable, high-quality service. If you have any concerns or questions, please reach out to our support team at any time. ## Detection and Response **Detection** Our monitoring systems detected multiple connectivity alerts originating from the same physical host, immediately notifying our Network Operations Center \(NOC\). **Response Actions Taken** * **13:50** – The hardware fault occurred, and the NOC team was alerted. * **13:55** – Automated failover processes reassigned most affected clients, mitigating the impact for a majority of customers. * **14:00** – Diagnostics confirmed the malfunction was hardware-related. * **14:10** – Manual redistribution of client accounts to alternative hosts began. * **17:00** – Full redistribution completed, restoring all services. ## Impact Assessment **User Impact** * Approximately 7% of clients experienced a complete outage of PBX and calling services while the redistribution was underway. While the automated failover partially succeeded, the event highlighted critical areas where additional failover capacity and faster manual procedures can further minimize future downtime. ## Lessons Learned 1. **Automation Enhancement** – We need improved tooling for situations where automated failover cannot manage the entire load, allowing for quicker manual redistribution. 2. **Improved Response Times** – Regular drills and simulations will enable our team to respond more efficiently to unexpected hardware faults. ## Preventative Measures **Short-Term Actions** * Verify that all services are fully operational following this incident. * Maintain heightened system monitoring to detect potential hardware anomalies earlier. * Schedule additional Disaster Recovery Testing to confirm failover resilience. **Long-Term Actions** * Conduct regular redundancy testing to ensure minimal downtime in the event of hardware failures. * Refine failover testing procedures to maintain consistent high availability. * Implement load and stress testing to prepare systems to handle recovery processes more seamlessly.