YourCloudTelco experienced a minor incident on September 5, 2021 affecting YourCloudTelco Calling Platform (Network), lasting 6h 50m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Sep 05, 2021, 11:56 PM UTC
We have investigated an issue with voicemail, the voicemail message is playing but the caller is unable to leave a message.
- identified Sep 06, 2021, 12:09 AM UTC
We have identified the issue and our team is implementing a fix.
- monitoring Sep 06, 2021, 04:44 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Sep 06, 2021, 06:47 AM UTC
This incident has been resolved.
- postmortem Sep 10, 2021, 12:24 AM UTC
The cause of this outage was the loss of a physical disk array holding call recordings requiring a remote hands call out to reset the device physically. While the entire process, including rack visit and remount, lasted one hour, the re-sync took another hour. October 2, 2021: 2:00am Over the following days, we will advise an upgrade that will deprecate this array, ultimately moving call recordings over to its replacement CEPH distributed file storage cluster. We apologise for any inconvenience.