5C AI experienced a critical incident on April 7, 2023 affecting Network and Virtual Instances and 1 more component, lasting 3d 2h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 07, 2023, 08:21 AM UTC
We are currently investigating a criticsl network issue affecting the QC2 region. We will provide update as soon as we have a better assessment of the scope of the impact. The hypertec Cloud team.
- investigating Apr 07, 2023, 09:39 AM UTC
We have detected 4 pieces of network equipment which are not working. We are investigating the specifics of each situation. We have also detected several power problems which cause impact in a limited fashion. This will be addressed at the same time. We will provide an update once the contingency plan is established.
- identified Apr 07, 2023, 10:14 AM UTC
We have identified the problem regarding networking. 95% of the network is back online. What issues remains are caused by power disruptions which we are still adressing. We will provide an update at resolution or in one hour to notify of our progress. The Hypertec Cloud team
- identified Apr 07, 2023, 11:35 AM UTC
We are still troubleshooting fallout effects from the network outage. We are roughly in the same state than an hour ago. However, we recovered internal tools which will allow us to track and address remaining issues.
- monitoring Apr 07, 2023, 02:56 PM UTC
The Hypertec Cloud team has now recovered most of the infrastructure. Any physical work left is non-impacting. Due to this event, it is possible that your devices would have ended up in a stopped state. We would invite to login to your portal and validate their state, starting them if needed. We will continue to monitor the situation and ensure stability post event. If you still have issues, please contact us via support.hypertec.cloud or by writting at [email protected]. Thank you for you paticience through this incident. The Hypertec Cloud team,
- identified Apr 07, 2023, 06:52 PM UTC
The Hypertec Cloud team has identified that in the wake of the network outage, it's management plaform now suffers from slowness that causes regular start up tasks to timeout. We are actively troubleshooting this situation as this is preventing impacted workload from recovering.
- resolved Apr 10, 2023, 10:56 AM UTC
We have been fixing lingering issues and monitoring the situation over the weekend. We have confirmed platform stability and are considering this incident as resolved. Thank you for your patience thorugh this incident. The Hypertec Cloud team.