5C AI experienced a critical incident on September 6, 2023 affecting Network and API, lasting 16h 44m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 06, 2023, 06:04 PM UTC
Dear Customer, We are currently investigating a situation affecting the QC2 region where the connectivity to the outside internet is not working. We will provide an update shortly on the status of the situation. We apologize for the problems this causes. The Hypertec Cloud team
- identified Sep 06, 2023, 06:30 PM UTC
Hypertec Cloud's network team has identified a piece of network equipment that is not handing requests properly. We have dispatched onsite hands to attempt to correct the problem. In the meantime we are looking at other solutions to move traffic away from the equipement in problem. We will provide an update once either of the two paths provide improvements. The Hypertec team,
- identified Sep 06, 2023, 07:17 PM UTC
Hypertec Cloud's personnel is now onsite. We will be executing physical changes momentarilly. Expect follow up soon. The Hypertec Cloud team.
- identified Sep 06, 2023, 07:46 PM UTC
We are continuing our troubleshooting of the affected devices. So far, onsite manipulations only yield partial internal connectivity restores. Hence further troubleshooting is needed. Our network team is actively looking for a solution. The Hypertec Cloud team,
- identified Sep 06, 2023, 09:26 PM UTC
We are continuing to work on a fix to the problem. Our network engineers are looking at a routing problem affecting internet traffic. We are commited to reaching resolution as soon as possible. The hypertec Cloud team
- identified Sep 06, 2023, 10:27 PM UTC
Hypertec Cloud's network team will be implementing a new configuration within the next 30 minutes to resolve the routing issue. We are expecting this configuration to restore all connectivity issues. We will provide an update when the configuration is in place. The Hypertec Cloud team
- identified Sep 06, 2023, 11:40 PM UTC
Hypertec's network team is still attempting to correct routing issues that are preventing internet connection to go out of our MTL2 point of service. Troubleshooting is continuing to correct the routing. The Hypertec Cloud team,
- monitoring Sep 07, 2023, 12:23 AM UTC
Hypertec's network team have restored the routing configuration. Connectivity to the internet is now up at MTL2. We will monitor the situation to ensure that it is stable. We thank you for your patience through this incident. If you are still experiencing issues, please contact support by replying to this e-mail. The Hypertec Cloud team
- resolved Sep 07, 2023, 10:48 AM UTC
Hypertec Cloud now considers this incident resolved.