Probado experienced a major incident on February 8, 2024 affecting VoIP (Voice over IP) and Data Services (Hosting), lasting 21d. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Feb 08, 2024, 10:13 AM UTC
We've identified the issue affecting performance with client services and are actively addressing the issue. Depending on Services Client's may be contacted individually for additional information.
- identified Feb 08, 2024, 11:38 AM UTC
While we address the main issue, the team is continuing to work on mitigating the impact for clients. We've begun migrating services to additional hardware we have on standby.
- identified Feb 08, 2024, 11:38 AM UTC
We are continuing to work on a fix for this issue.
- identified Feb 08, 2024, 03:21 PM UTC
The team are continuing to work on the performance impact and migrations to alleviate the excess overhead on the degraded hardware. Meanwhile we've had confirmation a replacement component will be collected and shipped to the Datacentre tomorrow with an engineer being available on-site to install. we greatly appreciate your patience and understanding.
- identified Feb 09, 2024, 12:21 PM UTC
Our engineer is on site and has replaced the faulty hardware - we're awaiting for the units to pair up again. While they are in this syncing client's may experience slight performance issues. We expect performance to stabilize within the next 30 minutes.
- monitoring Feb 09, 2024, 12:37 PM UTC
The installation is now complete and we're seeing network traffic be automatically redistributed as expected. We'll continue to monitor performance throughout the infrastructure.
- resolved Feb 29, 2024, 10:47 AM UTC
performance has returned to normal and traffic is rebalanced across all hardware.
- postmortem Feb 29, 2024, 10:47 AM UTC
**Scenario:** Our monitoring service flagged that one of our modules within the datacentre's networking architecture was showing indications of failure. During sourcing a replacement component from our suppliers, the component failed and network traffic was handed over to its counterpart \(which placed additional load on that unit\). This impacted performance across our main core network and we took steps to mitigate the hit on user’s experience by offloading services where appropriate. **Resolution:** As above a replacement part was sourced and one of our engineers was tasked to collect and swap out the faulty unit. Once this was done it took a couple of hours for traffic to redistribute across the network evenly and monitoring showed traffic returning to normal. **Conclusion:** While the infrastructure operated in a fault tolerant manner and failed over to the secondary component exactly as designed \(keeping 100% of services up and running\), We recognise that performance was impacted and both noticeable and not ideal for clients. This will be further reviewed and tackled in one of our upcoming meetings to see if any point the process can be optimised to reduce the impact. We appreciate your patience while we worked through this issue.