Resolved after 17 hours 12 minutes, affecting User Sync, on August 15, 2025.
**Summary**: On August 14th, 2025, at approximately 8:00 PM PT, we began experiencing elevated error rates in user sync jobs across several customer programs. The issue was due to a change introduced during a recent hotfix, which impacted how user identifier matching was processed. As a result, many user sync jobs failed with higher-than-normal error rates. A platform incident was declared on August 15th, and our engineering teams immediately worked to restore normal service. By the evening of August 15th, all impacted jobs had been successfully reprocessed, and user sync services were fully restored. **Impact**: 1\. Users in affected programs were not updated correctly during the incident window 2\. More than 260 sync jobs required reprocessing. 3\. Data inconsistencies persisted until the reprocessing was complete. **Root Cause**: A hotfix deployed on August 14th introduced a defect in user sync processing logic. Specifically, the changes altered the handling of user identifier matching resulting in widespread “two different users found” errors. **Mitigation**: 1\. The defective hotfix was rolled back. 2\. Stuck jobs were cleared to allow downstream jobs to process. 3\. All impacted jobs were reprocessed to ensure data consistency. 4\. Additional monitoring was added to identify elevated error rates more quickly in the future. **Recurrence Prevention**: The following actions have been taken or have been identified as follow-up actions to commit to as a part of the formal RCA \(Root Cause Assessment\) process: 1\. Adjusting monitoring thresholds to catch sync job error spikes earlier. 2\. Improving detection of long-running or blocked jobs. 3\. Enhancing hotfix testing to better cover user sync workflows. 4\. Strengthening post-hotfix monitoring to validate stability and prevent recurrence.
Affected components
Update timeline
- investigating Aug 15, 2025, 02:46 PM UTC
We are investigating an issue affecting User Sync (User File via SFTP processing). This is resulting in user sync jobs encountering elevated error rates. You may see the user sync email showing that many records were not processed due to errors. Our next update will be in 30 minutes.
- monitoring Aug 15, 2025, 03:22 PM UTC
We have applied a fix for the issue affecting user sync processing. We are reprocessing errored jobs and closely monitoring the results.
- resolved Aug 16, 2025, 07:59 AM UTC
This incident has been resolved. All previously errored user sync files have now been successfully reprocessed, and summary emails were sent as expected. We have been monitoring the user sync service overnight and no further issues have been observed. Any new errors that appear will be valid user sync processing errors and can be addressed using the guidance in our Help Centre article: https://support.firstup.io/hc/en-us/articles/13072635044503-SFTP-File-Upload-Errors-and-User-Sync-Results If you have any questions or encounter further issues, please reach out to our Support team at support.firstup.io.
- postmortem Sep 11, 2025, 11:53 PM UTC
**Summary**: On August 14th, 2025, at approximately 8:00 PM PT, we began experiencing elevated error rates in user sync jobs across several customer programs. The issue was due to a change introduced during a recent hotfix, which impacted how user identifier matching was processed. As a result, many user sync jobs failed with higher-than-normal error rates. A platform incident was declared on August 15th, and our engineering teams immediately worked to restore normal service. By the evening of August 15th, all impacted jobs had been successfully reprocessed, and user sync services were fully restored. **Impact**: 1\. Users in affected programs were not updated correctly during the incident window 2\. More than 260 sync jobs required reprocessing. 3\. Data inconsistencies persisted until the reprocessing was complete. **Root Cause**: A hotfix deployed on August 14th introduced a defect in user sync processing logic. Specifically, the changes altered the handling of user identifier matching resulting in widespread “two different users found” errors. **Mitigation**: 1\. The defective hotfix was rolled back. 2\. Stuck jobs were cleared to allow downstream jobs to process. 3\. All impacted jobs were reprocessed to ensure data consistency. 4\. Additional monitoring was added to identify elevated error rates more quickly in the future. **Recurrence Prevention**: The following actions have been taken or have been identified as follow-up actions to commit to as a part of the formal RCA \(Root Cause Assessment\) process: 1\. Adjusting monitoring thresholds to catch sync job error spikes earlier. 2\. Improving detection of long-running or blocked jobs. 3\. Enhancing hotfix testing to better cover user sync workflows. 4\. Strengthening post-hotfix monitoring to validate stability and prevent recurrence.
Your product runs on more than Firstup
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services