Some Fuse applications are experiencing degradation

Update timeline

1. resolved Jul 26, 2024, 04:03 PM UTC

   Type: Incident Duration: 4 hours and 28 minutes Affected Components: PCI Fuse, FUSE Jul 26, 16:03:00 GMT+0 - Investigating - A subset of Fuse applications may be experiencing degradation when handling input prompts in IVR flows. We have all necessary resources engaged and will restore service as soon as possible. Jul 26, 16:42:01 GMT+0 - Identified - We have successfully isolated the fault zone and are working through additional troubleshooting steps to identify the source of the disruption. All resources continue to be engaged until this is identified and resolved. Jul 26, 16:45:35 GMT+0 - Identified - We have identified the issue and are working to introduce a fix. Our target is within the next 15 minutes. Jul 26, 17:18:57 GMT+0 - Identified - We have encountered a disruption to our restoration method and are pursuing alternative options. Jul 26, 16:57:25 GMT+0 - Identified - We are introducing a fix across a collective of supporting resources. Incremental recovery will be experienced over the next 30-45 minutes. In parallel we are pursuing options to accelerate our full restoration time. Jul 26, 17:39:48 GMT+0 - Identified - We have identified the fault zone, have tested a fix in our preproduction environment, and are coordinating the best option to release to production. Jul 26, 18:06:48 GMT+0 - Identified - We are continuing our investigation to determine the most appropriate means of applying an identified fix. Jul 26, 18:29:10 GMT+0 - Identified - We have determined the best course of action to fix this issue. This approach is estimated to take 2 hours to implement. We are investigating alternative options with quicker resolution times in the interim. Jul 26, 18:58:50 GMT+0 - Identified - We are continuing to work on a fix for this incident. Jul 26, 19:22:27 GMT+0 - Identified - We are implementing a fix and will be monitoring recovery shortly. Jul 26, 19:26:54 GMT+0 - Monitoring - We have observed recovery from the fix and are monitoring. Jul 26, 20:04:44 GMT+0 - Identified - Our previously released fix resolved the majority of user experiences, but some configurations were still encountering errors. We have identified the remaining issue and are implementing a fix. Jul 26, 20:13:01 GMT+0 - Monitoring - We have implemented a secondary fix and are seeing recovery. We will continue to monitor the results. Jul 26, 20:30:48 GMT+0 - Resolved - This issues has been resolved. We have observed sustained recovery since our latest fix.