SharpenCX experienced a major incident on May 14, 2025 affecting Inbound Voice, lasting 7h 1m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 14, 2025, 03:42 PM UTC
We are currently investigating an issue with inbound calls dropping or resulting in dead air.
- investigating May 14, 2025, 04:28 PM UTC
We have isolated the suspected fault point and are taking corrective action.
- monitoring May 14, 2025, 04:51 PM UTC
A fix has been implemented and we're monitoring recovery.
- investigating May 14, 2025, 05:00 PM UTC
We are seeing a resurgence of the symptoms at a low rate. We are resuming investigation to identify additional corrective actions.
- investigating May 14, 2025, 05:01 PM UTC
Current symptoms recognized are as follows. The investigation continues. 1. Dead air for a portion of inbound calls (variable rate of failure) 2. Unexpected disconnect from Sharpen Q phone 3. Unexpected drop of connected call at 2 minutes.
- investigating May 14, 2025, 05:21 PM UTC
We are continuing our investigation but have been able to identify that the current expected failure rate is ~20%. All necessary resources are engaged to resolve this issue as soon as possible.
- investigating May 14, 2025, 05:45 PM UTC
We have identified another corrective action and are coordinating steps to implement.
- identified May 14, 2025, 06:08 PM UTC
Current symptoms recognized are as follows. The investigation continues to correct the last symptom, unexpected drops from Sharpen Q phone. 1. Dead air for a portion of inbound calls (variable rate of failure) <- Corrected as of 1:54 PM ET 2. Unexpected disconnect from Sharpen Q phone 3. Unexpected drop of connected call at 2 minutes. <- Corrected as of 12:30 PM ET
- identified May 14, 2025, 06:54 PM UTC
Some WebRTC connections are still failing to connect and we're continuing our investigation. If a user has an active connection, it is recommended to not refresh to potentially lose the connection.
- identified May 14, 2025, 07:38 PM UTC
We are continuing our investigation. We've been able to identify and correct a portion of the resources supporting the Sharpen Q Phone WebRTC collective and are continuing efforts toward full restoration
- identified May 14, 2025, 07:53 PM UTC
We are observing disruptions to outbound calls and inbound call alerts for some agents. All necessary subject matter experts continue to investigate.
- identified May 14, 2025, 08:15 PM UTC
We are performing additional corrective actions on remaining WebRTC hosts to extend recovery. There may be a brief interruption during the change.
- identified May 14, 2025, 08:27 PM UTC
We are now observing a disruption to outbound Sharpen Q initiated calls.
- identified May 14, 2025, 08:40 PM UTC
Outbound calls are now working as expected.
- identified May 14, 2025, 08:45 PM UTC
The majority of WebRTC resources are now functioning as expected and most users should be seeing recovery. We are correcting a small portion of remaining infrastructure and will update when all necessary resources are fully restored.
- monitoring May 14, 2025, 09:00 PM UTC
All WebRTC resources are fully restored and confirmed to be working. All previously observed impacts have subsided and we're monitoring recovery.
- resolved May 14, 2025, 10:43 PM UTC
The primary service incident that began earlier today is resolved. We understand this has been an extended disruption and sincerely apologize for the impact this had on your operations throughout the day. We appreciate your patience and understanding. While the main issue is resolved, there is a remaining defect where some call recordings are not immediately accessible through the Insights user interface or the contact card. Our team is actively working on the final remediation steps for this. We anticipate that all call recordings will be fully accessible by the end of today. Should you require immediate access to a specific call recording before it becomes available in the UI, please open a request with our Care team. They will be able to retrieve the recording for you. Thank you again for your continued patience.