Telenor CDN experienced a notice incident on December 24, 2020, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Dec 24, 2020, 12:25 PM UTC
We experienced a disk subsystem issue on a central cache from 06:50 to 12:00 CET.
- postmortem Jan 07, 2021, 03:44 PM UTC
# More details regarding the disk subsystem issue We would like to provide some more information about this issue. We received quality warnings and alerts for some of the content that was beeing handled by this server from our monitoring systems, and we also received disk alerts indicating possible disk issues. Our guard on duty was unable to fix the issue, and had to escalate the issue to employees on Christmas holiday. As soon as we were able to pinpoint the problem, the server was taken out of production and the issue was resolved. * The issue is related to a firwmare fault in both the SSD’s and the disk controller on some of our central cache servers with a specific hardware specification. * The issue only occurs sporadically after a long running time / power on hours. * Only some of the cached content was affected, and only playout to Norway. As of 06.01.2021 we have received firmware updates from the hardware vendor, and have performed the relevant upgrades on all of the central caches with the specific hardware specification. We apologize for the inconvenience this may have caused.