Alpaca experienced a notice incident on November 11, 2025 affecting Account API, lasting 25m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 11, 2025, 07:25 AM UTC
We are currently investigating this issue.
- resolved Nov 11, 2025, 07:50 AM UTC
The incident is resolved.
- postmortem Nov 19, 2025, 01:41 PM UTC
We are reporting on a service interruption that affected the activation of new accounts for several partners, starting on or around **November 10–11, 2025**. ### **What Happened** Our automated system responsible for moving newly onboarded accounts from the **"Approved"** status to **"Active"** encountered a technical limit within our database environment. Specifically, the process was attempting to handle too many accounts in a single operation. Once this database limit was hit, the activation job stalled, resulting in a large backlog of accounts that were approved but failed to receive the final **account\_activated** event. This prevented users from fully utilizing their newly opened accounts. ### **Impact** * **Scope:** Tens of thousands of accounts across several partners were temporarily stuck in the "Approved" status. * **User Experience:** Affected users experienced delays in the final activation of their brokerage accounts, preventing full use of their new accounts. * **Partner Impact:** Partners observed a halt in the emission of the expected **account\_activated** events and faced increased support inquiries regarding the delay. * **Total Resolution Time:** The issue was fully identified and resolved within **80 minutes**. ### **Resolution** The issue was identified through partner reports of missing activation events and internal monitoring showing an unusual build-up of accounts in the intermediate "Approved" status. Our engineering team took the following steps: 1. We isolated the affected accounts and safely re-entered them into the activation pipeline. 2. We **restarted the activation flow** in smaller, controlled batches to ensure adherence to database limits and prevent recurrence. 3. We verified that the accounts successfully progressed to **"Active,"** and the **account\_activated** events are now being emitted reliably. **The account activation pipeline is now functioning normally.** ### **Preventative Measures** To prevent recurrence and strengthen our onboarding pipeline, we are implementing the following fixes: * **Process Adjustment:** Modifying the account activation jobs to automatically process work in batches that respect database constraints. * **Enhanced Monitoring:** Adding new monitoring and alerts specifically designed to detect: * Unusual growth in accounts stuck in intermediate statuses \(e.g., Approved but not Active\). * Any unexpected drop in the volume of **account\_activated** events. * **Operational Readiness:** Updating internal runbooks to enable faster and safer remediation of large-scale account backlogs, should a similar issue arise. We apologize for the delay in account opening experienced by your users and the resulting operational impact on your teams. We appreciate your patience as we continue to enhance the stability of our critical services.