I.T Communications Limited experienced a critical incident on March 1, 2019 affecting Volta Core Juniper 10Gb Port Switch 4, lasting 1h 56m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Mar 01, 2019, 03:37 PM UTC
We are aware of a switch fault at one of our Data Centre locations. We have requested the Data Centre staff to reboot the switch. More info to follow shortly.
- identified Mar 01, 2019, 03:39 PM UTC
We are continuing to work on a fix for this issue.
- identified Mar 01, 2019, 03:52 PM UTC
an engineer is now assigned to this fault and is onsite.
- identified Mar 01, 2019, 03:58 PM UTC
Engineer now at the affected Server Rack on 4th floor. The switch has been powered off/on and waiting for it to fully boot up.
- monitoring Mar 01, 2019, 04:02 PM UTC
The Swich is now back up and full service restored. We are now investigating the cause of the switch to go down.
- monitoring Mar 01, 2019, 04:21 PM UTC
Steve one of our network guys is looking into the issue with the switch and to understand what went wrong with it. Failover, We operate 2 independant switches in each Server Rack. the current failover setup relies only on the link status that the network adapter provides. This option detects failures such as removed cables and physical switch power failures. Inlight of this issue today. we are exploring Beacon probing which sends out and listens for beacon probes on all NICs on the servers, and uses this information, in addition to link status, to determine link failure and sends beacon packets every second. If no reply is received - the NIC which connects to the Switch is automatically removed from service and serivce is resumed by the other switch.
- resolved Mar 01, 2019, 05:34 PM UTC
This incident has been resolved.