SCORM Cloud experienced a minor incident on June 24, 2022 affecting SCORM Cloud Website and SCORM Cloud API, lasting 2h 24m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 24, 2022, 01:10 PM UTC
We're experiencing an elevated level of errors for course imports where some files are incorrectly flagged by our automated virus scan tool. We are currently investigating this issue.
- investigating Jun 24, 2022, 01:48 PM UTC
We are continuing to investigate this issue.
- identified Jun 24, 2022, 02:00 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Jun 24, 2022, 02:54 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jun 24, 2022, 03:34 PM UTC
This incident has been resolved.
- postmortem Jun 24, 2022, 04:07 PM UTC
## Problem Description & Scope **Affected time range** \(times are CDT\): June 24, 4:00 AM to June 24 10:30 AM **Description**: An update to the ClamAV malware database introduced a signature that flagged any zip file as a virus. Therefore, any attempted course import, whether through the API or the UI, of a zipped package failed due to a failed virus scan check. ## Root Cause As part of our safety checks before we import a provided course, we use ClamAV to scan the file for any viruses. ClamAV maintains a database of known malware signatures which is [automatically updated](https://docs.clamav.net/manual/Usage/SignatureManagement.html) on a regular basis. This morning at approximately 4 AM CDT, a new update to this database was pushed out which contained a test signature that flagged any zip files that were scanned. We began to receive tickets about these import issues from customers using both the UI and the API and getting the same result: a message that their uploaded course failed our virus scan. Therefore we could narrow the issue down to the virus scanner itself, where we then located the signature that was generating all of these failures. We tested this theory by using one of our own sample courses, and it too returned the same error message. Upon further investigation, we noticed other people using the ClamAV tool posting in GitHub about the very same signature. They too had tested with known good zip files that were still being flagged based on this signature. We therefore concluded that this particular signature was most likely a false positive and not likely to indicate that the flagged file contained malware. ## Corrective Action We added the signature in question to our whitelist and, after verifying that it solved the issue and allowed course imports, pushed it out to our production environment. After confirming with customers that reported import issues that their courses could now be successfully imported, we concluded that this issue was resolved. ## Future Action As part of our corrective action, we notified ClamAV of the potential false positive through their online portal. Once we are notified that the database has been updated and the offending signature has been changed, we can safely remove it from our whitelist. ## Timeline Below is a timeline of events occurring on June 24, 2022 \(times in CDT\): * 4:02 AM - We started to see an elevated number of imports that were flagged by our automated virus scan. * 6:28 AM - An increase in tickets began to flow in as normal business hours started and customers noticed that their imports were failing. * 7:05 AM - We were provided with our first sample courses that had failed the check, and we started to investigate whether there was something in the course that was causing this error. * 7:59 AM - At this point, we had received a number of tickets about courses from multiple authoring tools failing. We tested the import with our own sample courses which threw the same error, so we determined that the issue had to lie with ClamAV. * 8:10 AM - The incident report was opened and published to our status page. * 8:37 AM - We identified the signature that was causing our sample course to be flagged by ClamAV, and verified that the same signature was causing the sample courses from customers to fail on import as well. * 8:56 AM - After finding information that other ClamAV users were having issues with this signature, we added it to our whitelist and tested that the scan would still catch and identify other malicious files in the course zip. * 9:04 AM - A hotfix was built and prepared for release. * 9:53 AM - The fix was published to our production environment, and we verified that we could import courses again. * 10:34 AM - After confirmation from our customers that their imports were working as expected again, we marked the issue as resolved.