Level365 experienced a major incident on May 21, 2024 affecting Call Recording, lasting 8h 27m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 21, 2024, 12:52 PM UTC
We are currently investigating reports of call recordings not showing up in Call History.
- identified May 21, 2024, 01:33 PM UTC
We have identified the cause of the issue and are working towards a resolution as quickly as we can.
- monitoring May 21, 2024, 04:14 PM UTC
We have implemented a fix and call recordings are starting to populate in Call History. There may be some call recordings from this morning that do not populate and we are working to get those restored.
- resolved May 21, 2024, 09:20 PM UTC
There have not been any new issues reported regarding call recordings since recording services were restored this afternoon. We consider this incident to be resolved. We are aware there are some recordings from this morning that are not showing up in Call History and we will continue to work towards restoring those call recordings.
- postmortem May 23, 2024, 12:22 PM UTC
_Complications occurred during the 5/21/24 maintenance update, which prevented the recording server from returning online. Personnel were dispatched to the data center to apply a fix. During this time the secondary recording server was still recording calls, they were just inaccessible via our Web App._ _At approximately noon ET, our primary recording server connection was restored and recordings that occurred before our maintenance as well as new recordings were once again accessible. However, call recordings during the downtime were still inaccessible from our secondary recording server. The configuration of the secondary recording server was then updated, which restored all recordings, and will help prevent this type of incident in the future._