Labflow incident
Labflow Outage Incident (Partial) - file uploads and file viewing
Labflow experienced a major incident on January 20, 2023 affecting File Conversions and Filestack API, lasting 4h 10m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 20, 2023, 05:42 PM UTC
Labflow Outage Incident (Partial) Status: In-Progress At 8:30AM (CST) on 1/20/23, Students started experiencing intermittent failure of file uploads and file viewing. Time Frame: Starting at: 8:30AM (CST) on 1/20/23 Ending at: (In-Progress) Duration: (In-Progress) Root Cause: (TBD) Labflow’s external service that processes file uploads and file viewing, FileStack, is experiencing intermittent issues. Remedies: Labflow is working with the vendor to remediate this issue as quickly as possible. Labflow needs to temporarily disable the ability to upload files within reports. Student Impact: The intermittent failure of file uploads and file viewing is creating a frustrating experience for students where they may encounter an error preventing them from uploading a file one time, but are then successful on other attempts, or situations where 1 or 2 of their files do not appear to be uploaded but others appear successful.
- investigating Jan 20, 2023, 06:00 PM UTC
We are continuing to investigate this issue.
- identified Jan 20, 2023, 07:25 PM UTC
The root cause of the issue has been identified. The external service used for file uploads and viewing, Filestack, is now testing solutions.
- identified Jan 20, 2023, 08:41 PM UTC
We are continuing to monitor the external service's progress. No additional information is available at this time.
- resolved Jan 20, 2023, 09:52 PM UTC
Status: Resolved At 8:30AM (CST) on 1/20/23, Students started experiencing intermittent failure of file uploads and file viewing. This impacted report assignments that required file uploads such as images or PDFs. Time Frame: Starting at: 8:30AM (CST) on 1/20/23 Ending at: 3:20PM (CST) on 1/20/23 Duration: 6 hours 50 minutes Root Cause: Labflow’s vendor that processes file uploads and file viewing, FileStack, was experiencing intermittent issues. Labflow is expecting a follow-up root cause analysis report to understand their future remediation strategies. Remediation: Labflow has re-enabled the file uploads and file viewing functionality as of 3:20PM (CST). Student Impact: The intermittent failure of file uploads and file viewing created a frustrating experience for students where they may encounter an error preventing them from uploading a file one time, but are then successful on other attempts, or situations where 1 or 2 of their files do not appear to be uploaded but others appear successful.