h8lio incident

node7 out from kubernetes network

Minor Resolved View vendor source →

h8lio experienced a minor incident on February 13, 2024 affecting kube.h8l.io and ci.h8l.io and 1 more component, lasting 13h 6m. The incident has been resolved; the full update timeline is below.

Started
Feb 13, 2024, 02:23 PM UTC
Resolved
Feb 14, 2024, 03:30 AM UTC
Duration
13h 6m
Detected by Pingoru
Feb 13, 2024, 02:23 PM UTC

Affected components

kube.h8l.ioci.h8l.iomonitoring.h8l.io

Update timeline

  1. identified Feb 13, 2024, 02:23 PM UTC

    This is the follow up of the incidents https://h8lio.statuspage.io/incidents/5l7c7jdgv57z We are still waiting for the provider to put back the node7 on the private network (Kubernetes network). Based on their last feedback (1h ago), they should do it soon. We are also setting up a backup plan in case of a delayed intervention. We will keep you posted...

  2. identified Feb 13, 2024, 03:37 PM UTC

    We just have a feedback from our hosting provider support, the intervention should start soon. Note, the node7 is one of our new strong CI build agent. So you could experience delayed builds.

  3. identified Feb 13, 2024, 04:31 PM UTC

    We juste received a new powerful bare metal server ordered this morning to replace node7 while is in maintenance (as we still don't know when the intervention will be scheduled). The installation of the new node8 is in progress and we let you know as soon is available.

  4. monitoring Feb 13, 2024, 06:26 PM UTC

    The new node (node8) has been installed and is taking the load of the node7.

  5. monitoring Feb 13, 2024, 06:48 PM UTC

    We are continuing to monitor for any further issues.

  6. monitoring Feb 14, 2024, 01:46 AM UTC

    The node7 is back on the Kubernetes network! We are going to do some sanity checks before to make it public.

  7. resolved Feb 14, 2024, 03:30 AM UTC

    This incident has been resolved.