Some Bulk Uploads are failing

Update timeline

1. identified Sep 20, 2024, 02:49 AM UTC

   We have been notified of an issue where some Bulk Upload Sessions are failing. We are currently investigating the issue.

2. monitoring Sep 20, 2024, 03:02 AM UTC

   A fix has now been implemented. We are monitoring new Bulk Uploads that are submitted to check if any issues still persists.

3. resolved Sep 20, 2024, 03:35 AM UTC

   This incident has been resolved. A postmortem will be posted within 2 working days of the incident after we are able to gather the relevant information. Thanks for your patience

4. postmortem Sep 20, 2024, 03:39 AM UTC

   ### **Postmortem: Some Bulk Uploads are failing** **Incident Date:** 19/09/2024, 9:30 PM **Duration:** 1 day hour 17 hours and 30 minutes **Summary:** On 20/09/2024 at 11:40am, it was bought to our attention that multiple bulk upload sessions across the system had been failing. We began investigating the issue and could see it was intermittent across customer sites. Most sessions had completed without error, but there had been multiple failures since our last deployment at the 9:30PM on the 19th September NZT. As part of the release _4.22 Expire Collection Members via API_, we added better exception handling and error prevention to our data layer to avoid unnecessary calls to the database. The issue arose when specific settings were selected during a bulk upload, and the bulk upload file contained a particular mix of new, and existing users. The new exception handler triggered an error in the bulk upload function, revealing a unknown bug that would have previously failed silently. We quickly patched the bug and deployed the fix to production. **Root Cause:** An existing bug in the Bulk Upload function was revealed when better error handling was added to the system in a recent update. **Impact:** The outage affected some customers' ability to bulk upload users using a CSV, depending on the settings they had selected, and the specific mix of new and existing users in their site. We have detected that 3 sites were affected across 11 attempted bulk upload sessions. This disruption occurred across 2 days until we were alerted, and patched the site with the fix. **Conclusion:** The incident was caused by a previously unknown bug in the bulk upload functionality, which was exposed by the improved exception handling introduced in release 4.22. While the issue impacted a small number of customers, it resulted in failures across multiple bulk upload sessions. We responded quickly, identified the root cause, and deployed a patch to resolve the issue. To prevent similar issues occurring in future updates, we will improve our testing procedures wherever feasible. We are also working to optimize our monitoring systems to ensure that potential issues are identified and brought to our attention quicker. We apologize for any inconvenience caused and remain committed to ensuring the stability and performance of our system. If you have any questions or concerns, please don’t hesitate to [reach out to our support team](mailto:[email protected]). **The Intuto Team**