Resolved after 5 hours 30 minutes, affecting US Firstup Platform and EU Firstup Platform, on November 14, 2025.
**Summary:** On November 14, a large file uploaded by one customer program caused the system responsible for processing user imports to slow down significantly. Because the system processes import jobs in order, this slow-running file created a backlog that delayed other customers’ imports during the incident. Once the issue was identified, the problematic file was stopped and the system was restored, allowing imports to resume normal processing. **Impact:** * CSV files uploaded during the affected timeframe experienced delays. * No data loss occurred, and all delayed imports completed successfully once the system recovered. **Root Cause:** A large CSV file uploaded by one customer program contained a high volume of duplicate entries, which triggered recursive processing behavior. This caused the import system to spend an unusually long time processing that single file, slowing down the overall import queue. **Mitigation:** An incident was declared as soon as the scope was understood. After identifying the slow-running upload as the source of the delays, the team stopped the problematic job to prevent it from continuing to block the queue. Performance was then restored by restarting the processing workers and performing maintenance to improve lookup speeds. These steps allowed the system to clear the backlog and return to normal operation. All delayed import jobs completed successfully once the processing queue began moving again. **Recurrence Prevention:** The following actions have been identified as follow-up actions to commit to as a part of this process: * Adding monitoring that alerts us if import jobs begin taking unusually long or if the queue grows too quickly. * Improving how we validate uploaded user data. * Enhancing our import processing logic so that slow-running jobs can be isolated without delaying other customers.
Affected components
Update timeline
- investigating Nov 14, 2025, 06:53 PM UTC
We're currently investigating an issue where user imports become stuck in Created status.
- identified Nov 14, 2025, 08:01 PM UTC
The issue has been identified and work to mitigate any outstanding stuck imports is underway.
- identified Nov 14, 2025, 08:21 PM UTC
Imports are no longer in a 'stuck' state and the cause is understood. We're continuing to monitor to ensure a steady state.
- monitoring Nov 14, 2025, 08:40 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Nov 15, 2025, 12:24 AM UTC
No further issues have been noted since the fix was implemented; the incident is resolved.
- postmortem Dec 02, 2025, 12:14 AM UTC
**Summary:** On November 14, a large file uploaded by one customer program caused the system responsible for processing user imports to slow down significantly. Because the system processes import jobs in order, this slow-running file created a backlog that delayed other customers’ imports during the incident. Once the issue was identified, the problematic file was stopped and the system was restored, allowing imports to resume normal processing. **Impact:** * CSV files uploaded during the affected timeframe experienced delays. * No data loss occurred, and all delayed imports completed successfully once the system recovered. **Root Cause:** A large CSV file uploaded by one customer program contained a high volume of duplicate entries, which triggered recursive processing behavior. This caused the import system to spend an unusually long time processing that single file, slowing down the overall import queue. **Mitigation:** An incident was declared as soon as the scope was understood. After identifying the slow-running upload as the source of the delays, the team stopped the problematic job to prevent it from continuing to block the queue. Performance was then restored by restarting the processing workers and performing maintenance to improve lookup speeds. These steps allowed the system to clear the backlog and return to normal operation. All delayed import jobs completed successfully once the processing queue began moving again. **Recurrence Prevention:** The following actions have been identified as follow-up actions to commit to as a part of this process: * Adding monitoring that alerts us if import jobs begin taking unusually long or if the queue grows too quickly. * Improving how we validate uploaded user data. * Enhancing our import processing logic so that slow-running jobs can be isolated without delaying other customers.
Your product runs on more than Firstup
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services