AHS-7569 - Problems with top of the rack switch in nbg6-nn (2023-03-17)

Update timeline

1. investigating Mar 17, 2023, 04:30 PM UTC

   We are currently see a problem with a device in nbg6-nn

2. identified Mar 17, 2023, 04:48 PM UTC

   The faulted device was rebooted.

3. monitoring Mar 17, 2023, 05:01 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Mar 20, 2023, 07:56 AM UTC

   This incident has been resolved.

5. postmortem Mar 20, 2023, 07:56 AM UTC

   # Background Last week Thursday planned switch updates were done in datacenter location nbg4-he. After the update when monitoring the systems we saw an increase in log actions with the new firmware version. The log content looks like debug messages from a low level plane. We were doing the update together with external network consultants and therefore we analyzed the behavior together. Unfortunately nothing related to what we saw could be found in the manufacturers tech documents nor on the internet. Log files in general are not problematic but sometimes when the amount of log data written gets too much it could damage the storage device of the network switch because often they are not made for excessive writes. So we opened a case with the manufacturer. The case itself was initially treated as not so important \(P3\) because there was not downtime. ADESTIS and the external consultants still were concerned about the writes because a few years earlier we had issues with a version which killed the disk of the switches due to heavy writes. Therefore we decided to delete the log file. And it seems to work because the file itself was not recreated. So this “solution” was rolled out to all devices which had the same firmware version also in other datacenters. # Issues reported A few hours later our monitoring noticed that the log partition of some of those network devices got filled up. Still together with the external consultants we analyzed the issue. It seems as the deletion of the file just removed the connection to the directory but the file handle must have been still open and data was written to the disk. Since the files were deleted no log rotation occurred and the partition was fully occupied. We directly contacted the support and requested a priority increase to P1 \(approx. 17:30 CET, 2023-03-16\). About an hour later we had a call with the manufacturer. We provided additional information and we were told to get a feedback 30 minutes later. The problem seems also be complicated for manufacturer because we received the first feedback over two hours later. Unfortunately only further information were requested and no solution was provided. This was ongoing until approx. 23:30 CET. On the next day we asked several times for a solution but we only received suggestions which were either not working or which were incompatible with the platform. Our own investigations also continued and we found a command on the line card management layer which showed us that the file logging which creates these file was enabled. We checked several devices and the setting was active on all devices with the same version. Further checks in that control plane showed that there was also a command to disable the logging. We tested this command on a device in a test environment and it seemed to work. So we informed the manufacturer about the command and asked whether it is ok and safe to use in production environment. # Incident We got the feedback from the support that it is safe to use this command and that it does what we need. Since we have already tested it on a device successfully and we had the go from the support we chose a switch and send the command \(approx. 17:15 CET, 2023-03-17\). The switch became unresponsive for a few minutes which was not the case with the other switch we tested. We immediately took actions to isolate the device from the rest of the network. When the switch was responsible again we saw that the process which is responsible to exchange information between the control plane and the chipset has crashed and was restarted. Nevertheless the switch still seemed to affect the rest of the network therefore we decided to restart the device \(around 17:45\). With the restart the problems disappeared and we focused on recovering affected workload. # Investigations and Actions We called out for the support again and provided the core dump of the failed process for further investigations. We were fixing smaller issues until 02:30 CET on 2023-03-18. The devices were carefully monitored since then and we are now waiting for a solution from the manufacturer. We expect to see a new service release / bugfix for the problems we had.