RackNerd Outage History

Started Dec 26, 2024, 01:31 AM UTC· resolved Dec 26, 2024, 01:31 AM UTC

1. resolved Dec 26, 2024, 01:31 AM UTC

   We are aware that CHI114KVM is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: This is now resolved. If you are still experiencing any issues please open a support ticket and we'd be happy to assist.

Read the full incident report →

Started Dec 25, 2024, 10:40 PM UTC· resolved Dec 25, 2024, 10:40 PM UTC

1. resolved Dec 25, 2024, 10:40 PM UTC

   We are aware that NYRN105KVM is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: This is now resolved. If you are still experiencing any issues please open a support ticket and we'd be happy to assist.

Read the full incident report →

Started Dec 25, 2024, 08:34 PM UTC· resolved Dec 25, 2024, 08:34 PM UTC

1. resolved Dec 25, 2024, 08:34 PM UTC

   We are aware that ATL132KVM is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: This is now resolved. If you are still experiencing any issues please open a support ticket and we'd be happy to assist.

Read the full incident report →

Started Dec 25, 2024, 11:57 AM UTC· resolved Dec 25, 2024, 11:57 AM UTC

1. resolved Dec 25, 2024, 11:57 AM UTC

   We are aware that NJ108KVM is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: This is now resolved. If you are still experiencing any issues please open a support ticket and we'd be happy to assist.

Read the full incident report →

Started Dec 25, 2024, 05:30 AM UTC· resolved Dec 25, 2024, 05:30 AM UTC

1. resolved Dec 25, 2024, 05:30 AM UTC

   We are aware that FIBER8 Shared Hosting Server is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: This is now resolved. If you are still experiencing any issues please open a support ticket and we'd be happy to assist.

Read the full incident report →

Started Dec 23, 2024, 12:46 PM UTC· resolved Dec 23, 2024, 12:46 PM UTC

1. resolved Dec 23, 2024, 12:46 PM UTC

   We are aware that DAL104KVM is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: This is now resolved. If you are still experiencing any issues please open a support ticket and we'd be happy to assist. UPDATE: We are seeing a recurrence of the previous issues with DAL104KVM. Our team is investigating this further. UPDATE: This is now resolved again. We are keeping a close eye on this. If you experience any issues, please open a support ticket and we'll assist immediately.

Read the full incident report →

Started Dec 23, 2024, 04:05 AM UTC· resolved Dec 23, 2024, 04:05 AM UTC

1. resolved Dec 23, 2024, 04:05 AM UTC

   We are aware that LAXSSD4051nerd6DC02 is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: This is now resolved. If you are still experiencing any issues please open a support ticket and we'd be happy to assist.

Read the full incident report →

Started Dec 22, 2024, 06:33 AM UTC· resolved Dec 22, 2024, 06:33 AM UTC

1. resolved Dec 22, 2024, 06:33 AM UTC

   We are aware that multiple host-nodes of San Jose " SJ111KVM SJ110KVM SJ108KVM SJ156KVM SJ176KVM SJ175KVM SJ177KVM" is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE : These nodes are back online - SJ111KVM SJ110KVM SJ108KVM SJ175KVM SJ177KVM UPDATE : This particular node also back online - SJ156KVM UPDATE : This is now resolved for all the affected host-nodes. If you are still experiencing any issues please open a support ticket and we'd be happy to assist.

Read the full incident report →

Started Dec 21, 2024, 08:07 AM UTC· resolved Dec 21, 2024, 08:07 AM UTC

1. resolved Dec 21, 2024, 08:07 AM UTC

   We are aware that SEA108KVM is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: This is now resolved. If you are still experiencing any issues please open a support ticket and we'd be happy to assist.

Read the full incident report →

Started Dec 20, 2024, 03:26 PM UTC· resolved Dec 20, 2024, 03:26 PM UTC

1. resolved Dec 20, 2024, 03:26 PM UTC

   We are aware that CHI158KVM is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: This is now resolved. If you are still experiencing any issues please open a support ticket and we'd be happy to assist.

Read the full incident report →

Started Dec 19, 2024, 01:23 PM UTC· resolved Dec 19, 2024, 01:23 PM UTC

1. resolved Dec 19, 2024, 01:23 PM UTC

   We are aware that SJ182KVM is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: This is now resolved. If you are still experiencing any issues please open a support ticket and we'd be happy to assist.

Read the full incident report →

Started Dec 19, 2024, 10:25 AM UTC· resolved Dec 19, 2024, 10:25 AM UTC

1. resolved Dec 19, 2024, 10:25 AM UTC

   We are aware that 7950XSJ300 is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: This is now resolved. If you are still experiencing any issues please open a support ticket and we'd be happy to assist.

Read the full incident report →

Started Dec 18, 2024, 02:38 PM UTC· resolved Dec 18, 2024, 02:38 PM UTC

1. resolved Dec 18, 2024, 02:38 PM UTC

   We are aware that MIA2000 & MIA3000 Shared Hosting Server are currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: We have identified a network routing loop issue with the subnet that these two servers reside on. The issue has been escalated to our network's upstream provider and this will be resolved shortly. UPDATE: The routing issue has mostly been resolved at this time. The cause was due to an issue with Lumen/Level3 not properly announcing the prefix. Our upstream is working to permanently address this with Lumen and to ensure this situation will never repeat again. While the issue has been mostly resolved, we are still investigating a few routes that are experiencing issues. We will update this status incident once it is fully resolved. UPDATE: We are pleased to report that the network routing issue affecting MIA2000 and MIA3000 Shared Hosting Servers has now been fully resolved. The root cause was identified as a misconfiguration with Lumen/Level3 not properly announcing the subnet prefix. Our upstream provider has worked closely with Lumen to resolve the issue and implemented measures to prevent a recurrence of this situation. We sincerely apologize for the disruption this caused to your services and for the impact on your operations during this time. We appreciate your patience and understanding as we worked with our upstream provider to resolve the issue. If you continue to experience any connectivity problems, please reach out to our support team for assistance.

Read the full incident report →

Started Dec 18, 2024, 12:33 PM UTC· resolved Dec 18, 2024, 12:33 PM UTC

1. resolved Dec 18, 2024, 12:33 PM UTC

   We are aware that NYRN101KVM is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: This is now resolved. If you are still experiencing any issues please open a support ticket and we'd be happy to assist.

Read the full incident report →

Started Dec 18, 2024, 05:18 AM UTC· resolved Dec 18, 2024, 05:18 AM UTC

1. resolved Dec 18, 2024, 05:18 AM UTC

   We are aware that CHI114KVM is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: This is now resolved. If you are still experiencing any issues please open a support ticket and we'd be happy to assist.

Read the full incident report →

Started Dec 18, 2024, 03:45 AM UTC· resolved Dec 18, 2024, 03:45 AM UTC

1. resolved Dec 18, 2024, 03:45 AM UTC

   As communicated via email to customers on node: 7950XDAL303 (one of our Ryzen nodes in Dallas), all VM's hosted on this node are currently being migrated to another hypervisor, in order to avoid extended downtime of troubleshooting the hardware issue with the node. We will update this status incident, once migrations have been complete. All data and VM's will be migrated intact and no IP changes will be necessary. UPDATE: This is now completed successfully.

Read the full incident report →

Started Dec 18, 2024, 02:18 AM UTC· resolved Dec 18, 2024, 02:18 AM UTC

1. resolved Dec 18, 2024, 02:18 AM UTC

   We are aware that DE4000 Shared Hosting Server is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: This is now resolved. If you are still experiencing any issues please open a support ticket and we'd be happy to assist.

Read the full incident report →

Started Dec 17, 2024, 07:45 PM UTC· resolved Dec 17, 2024, 07:45 PM UTC

1. resolved Dec 17, 2024, 07:45 PM UTC

   We are aware that SJ240KVM is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: This is now resolved. If you are still experiencing any issues please open a support ticket and we'd be happy to assist.

Read the full incident report →

Started Dec 17, 2024, 04:09 PM UTC· resolved Dec 17, 2024, 04:09 PM UTC

1. resolved Dec 17, 2024, 04:09 PM UTC

   We are aware that SJ126KVM is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: This is now resolved. If you are still experiencing any issues please open a support ticket and we'd be happy to assist.

Read the full incident report →

Started Dec 17, 2024, 06:50 AM UTC· resolved Dec 17, 2024, 06:50 AM UTC

1. resolved Dec 17, 2024, 06:50 AM UTC

   We are aware that SJ240KVM is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: This is now resolved. If you are still experiencing any issues please open a support ticket and we'd be happy to assist.

Read the full incident report →

Started Dec 16, 2024, 08:17 PM UTC· resolved Dec 16, 2024, 08:17 PM UTC

1. resolved Dec 16, 2024, 08:17 PM UTC

   We are aware that 7950XDAL303 is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: This is now resolved. If you are still experiencing any issues please open a support ticket and we'd be happy to assist.

Read the full incident report →

Started Dec 16, 2024, 04:15 PM UTC· resolved Dec 16, 2024, 04:15 PM UTC

1. resolved Dec 16, 2024, 04:15 PM UTC

   As a follow-up to the scheduled maintenance event we sent to affected customers via email on December 10, 2024 - please be aware that there is currently a scheduled maintenance event which is now ongoing. This affects VPS's hosted on nodes with the word "nerd5" within the node name. A web version of the maintenance notice that was sent can be found here for your reference: https://www.racknerd.com/nerd5-vps-maintenance.html We will update this status incident once the maintenance window has concluded. UPDATE 10:50 AM PST: Batch 1 of 3 has been completed, we are on track. We are about to begin on the second batch. We will update once additional information is available. UPDATE 11:45 AM PST: Batch 2 of 3 has been completed, we are ahead of schedule. We are about to begin on the third and final batch for the Los Angeles DC-02 (nerd5) maintenance. We will update once additional information is available. UPDATE 12:30 PM PST: Batch 3 of 3 is completed. At this time, all Los Angeles DC-02 (nerd5) hypervisors are back online and fully operational, with the exception of a single hypervisor "LAXSSD2008nerd5DC02". LAXSSD2008nerd5DC02 might have an unrelated hardware issue that was identified during the boot process, and this is currently being looked into. We will update once LAXSSD2008nerd5DC02 is back online. In the meantime, all other nerd5 hypervisors/nodes are back online. UPDATE 2:45 PM PST: LAXSSD2008nerd5DC02 is back online, at this time this maintenance event for Los Angeles DC-02 (nerd5 section) is fully concluded and finished.

Read the full incident report →

Started Dec 16, 2024, 01:17 PM UTC· resolved Dec 16, 2024, 01:17 PM UTC

1. resolved Dec 16, 2024, 01:17 PM UTC

   As a follow-up to the scheduled maintenance event we sent to affected customers via email on December 10, 2024 - please be aware that there is currently a scheduled maintenance event in our Los Angeles DC-03 location, which is now ongoing. A web version of the maintenance notice that was sent can be found here for your reference: https://www.racknerd.com/dc03-maintenance.html We will update this status incident with additional updates as we make progress. Thank You for your patience and understanding during this time. UPDATE 11:00 AM PST: Progress is being made and servers are actively being unracked and transported to the new facility. We will update once additional information is available. UPDATE 5:15 PM PST: Excellent progress is continuing to be made. We will share a more detailed update later this evening. We are continuing to work on this. UPDATE 6:24 PM PST: Over 70% of our Los Angeles DC-03 footprint is now back online (at the new facility). This status incident is now being downgraded to a Partial Outage status, as we continue to work through the list of devices that are still offline and/or still need to be racked/cabled. UPDATE 8:15 PM PST: We are pleased to report that over 90% of our Los Angeles DC-03 footprint is now back online and operational at the new facility. We have shifted our focus to reviewing still-offline devices on an individual server basis, as we continue to go through the list of servers that are still offline. Each server is being thoroughly reviewed to ensure it receives the necessary attention and proper handling. We are actively making progress and will share further updates once the situation is more situated. Thank you for your continued patience. UPDATE 9:58 PM PST: We are nearing the finish line here, we are continuing to work on going through each server, one by one within all of our cabinets in this location, and ensuring that each server is showing positive power-on indicators. We will share another status update shortly. UPDATE 10:10 PM PST: We are very much ahead of schedule, and we are pleased to report that we have fully completed this facility migration maintenance at this time. Please note that following a facility-wide migration of this nature, a few devices here and there may require manual attention. We have made every effort to verify and ensure that as many servers as possible have been powered on. Additionally, we have physically checked all servers in our cabinets to confirm they are showing power-on indicators. If you are still experiencing issues or if your server in Los Angeles DC-03 remains inaccessible as of now, we recommend first checking SSH to ensure your software stacks are properly started. If SSH is not accessible, please verify the server status via IPMI. If you're unable to resolve the issue, please submit a support ticket, and we will investigate further on a case-by-case basis. Our support team is available 24x7 to assist with any issues you may encounter.

Read the full incident report →

Started Dec 16, 2024, 02:48 AM UTC· resolved Dec 16, 2024, 02:48 AM UTC

1. resolved Dec 16, 2024, 02:48 AM UTC

   We are aware that FIBER10 Shared Hosting Server is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: This is now resolved. If you are still experiencing any issues please open a support ticket and we'd be happy to assist.

Read the full incident report →

Started Dec 15, 2024, 03:01 PM UTC· resolved Dec 15, 2024, 03:01 PM UTC

1. resolved Dec 15, 2024, 03:01 PM UTC

   We are aware that SEA137KVM is currently experiencing connectivity issues. We are currently looking into this node further and will provide additional updates on this matter once we have more information. UPDATE: This is now resolved. If you are still experiencing any issues please open a support ticket and we'd be happy to assist.

Read the full incident report →