I.T Communications Limited incident
SoftSwitch PBX CL1 - Server 02 and 03 Issues
I.T Communications Limited experienced a major incident on November 5, 2021 affecting Soft-Switch PBX - pbx-cl1-01.soft-switch-pbx.uk and Soft-Switch PBX - pbx-cl1-02.soft-switch-pbx.uk and 1 more component, lasting 1h 14m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Nov 05, 2021, 04:40 PM UTC
We found a fault with the Databases which stores all the data for Soft-Swtich Cluster 01 and waiting for the databass to catch up with each other before the service will return back to normal.
- monitoring Nov 05, 2021, 04:41 PM UTC
We are continuing to monitor for any further issues.
- monitoring Nov 05, 2021, 04:59 PM UTC
Soft-Switch Cluster 1 - stores its data on 3 seperate databases. Last night we tried to introduce a 4th Database which failed to sync. A fix was applied however casued the other databases replication to crash. The resync is taking its time due to the size of the databases. We continue to work on this until the service is restored. Updates to follow.
- monitoring Nov 05, 2021, 05:02 PM UTC
We are now in the process of failing the servers over to the good working database so service can be resumed quicker. This will allow us to work on the other databases without continued loss of service.
- monitoring Nov 05, 2021, 05:11 PM UTC
PBX-03 is now back up and working.
- monitoring Nov 05, 2021, 05:17 PM UTC
PBX-02 is now Operational
- monitoring Nov 05, 2021, 05:37 PM UTC
Pbx01 now back in service.
- resolved Nov 05, 2021, 05:54 PM UTC
Server 4 is now added to cluster for WebRTC Full service restored