Alpaca Outage History

Started Dec 29, 2025, 05:07 AM UTC· resolved Dec 29, 2025, 05:21 AM UTC

1. identified Dec 29, 2025, 05:07 AM UTC

   Our support forum (https://forum.alpaca.markets/) is currently read only. Please reach out to [email protected] if you have any issues.

2. resolved Dec 29, 2025, 05:21 AM UTC

   Payment issue resolved. The forum is working normally.

Read the full incident report →

Started Dec 15, 2025, 07:01 PM UTC· resolved Dec 16, 2025, 01:05 AM UTC

1. investigating Dec 15, 2025, 07:01 PM UTC

   We observed intermittent failures on Broker API and Live Trading API due to database connection saturation. Services have recovered, but the underlying issue is still being mitigated. Engineering is tuning connection usage and limits to prevent recurrence. Continuous monitoring is in place.

2. resolved Dec 16, 2025, 01:05 AM UTC

   at 4:01PM The team deployed a fix resolving the incident.

3. postmortem Dec 16, 2025, 04:49 PM UTC

   ## **Intermittent Broker & Trading API Failures \(December 15, 2025\)** We sincerely apologize for the service degradation experienced by our partners today, December 15, 2025. This incident primarily affected the Broker API and Live Trading API, causing intermittent transaction rejections and data processing delays. ‌ ### **What Happened** Our platform experienced a temporary resource exhaustion event within our core data processing system. Specifically, we observed a rapid and unexpected spike in the number of concurrent connections to our primary brokerage database. This increase was caused by a combination of factors: 1. System Scaling Changes: Recent scaling updates to our Order Management System \(OMS\) unintentionally increased the potential demand for database connections. 2. Account Data Warm-up Issue: A critical internal job responsible for proactively loading \(or "warming up"\) partner account data failed early in the day due to a minor bug related to a system test account. 3. High Traffic Volume: When normal market traffic began, the OMS was forced to load account data on demand for a large volume of requests, with each request opening a new database connection. This high-volume, on-demand loading quickly exceeded the database's maximum connection capacity, leading to the system's saturation. This resource contention resulted in a cascading failure, where key services—such as those responsible for processing new orders and managing account state—could not access the database, causing service slowdowns and intermittent failures. ### **Impact** The primary impact was experienced during periods of high trading volume on December 15. * Trading: Partners experienced intermittent rejection of new order submissions and delays in receiving order status updates across several brokerage endpoints. * Account Operations: There were delays and failures in certain account-related operations, such as creating new accounts or processing large batches of internal jobs \(e.g., dividend payments\). * Data Integrity: Crucially, we can confirm that no client funds or trading data were lost or compromised. The system's design ensures that all transactions are safely logged and persisted. The issue was a temporary access and processing delay, not a data loss event. The issue affected several of our B2B partners, including those relying on real-time order submission and account updates. ### **Resolution** Our engineering teams were immediately engaged and implemented several mitigating actions to restore full service: 1. Immediate Connection Rebalancing: We temporarily scaled down non-critical services in our test and worker environments to free up immediate database connections for the core trading platform. 2. Job Fix Deployment: We deployed an emergency fix for the bug that prevented the critical account data warm-up job from running, ensuring this process now operates correctly. 3. Connection Limit Tuning: We immediately began the process of reducing the maximum allowed connections for specific, known-idle microservices that were holding an unnecessarily high number of open connections. As of **4:01 PM EST**, the platform has fully recovered, and all core services are operating within normal performance parameters. ### **Preventative Measures** We are implementing a robust set of follow-up actions to prevent any recurrence and strengthen the overall resilience of our platform. These efforts are organized into three key themes: * Database Connection Optimization: We will establish and enforce firm-wide connection budgets for every application that accesses the core brokerage database. This will ensure predictable resource usage and prevent any single service from overwhelming the system. * System Resilience & Monitoring: We are enhancing our monitoring and alerting to proactively detect unexpected spikes in database connection usage and account loading activity. This includes deeper investigation into the specific mechanisms that cause a high volume of connections for processes like “trading controls” to implement necessary caching layers. * Process Optimization: We are reviewing and adjusting the operational schedule for large-scale, internal processes \(such as cash dividend processing\) to ensure they do not conflict with or impact peak trading hours for our partners. We appreciate your patience and trust as we continue to invest in our platform's reliability. Our commitment to providing a stable, high-performance financial services platform remains our top priority.

Read the full incident report →

Started Dec 15, 2025, 01:11 PM UTC· resolved Dec 15, 2025, 06:26 PM UTC

1. monitoring Dec 15, 2025, 01:11 PM UTC

   Current Status: Monitoring & Verification Our engineering team has implemented a service restart and is actively monitoring all relevant metrics to confirm service stability and full recovery. Summary Starting at approximately 7:24 AM EST (12:24 UTC), clients attempting to use the v1/instant_funding API may have experienced elevated error rates, specifically HTTP 500 errors or Gateway Timeout messages. This issue was caused by a service timeout in our Payment Service when communicating with the GoBroker API. Impact Users attempting Instant Funding transactions were unable to complete them. This primarily affected partners utilizing the v1/instant_funding endpoint. Update The Payment Service and its worker components were restarted between 12:34 UTC and 12:40 UTC. Following the restart, service metrics indicate a return to normal request flow, and error rates have significantly decreased. We are currently monitoring the system and awaiting confirmation of full resolution from affected partners.

2. monitoring Dec 15, 2025, 01:53 PM UTC

   we are processing large batches of Memo post , it can take few minutes please expect some delay for now. We are actively monitoring for any issue and will keep you posted.

3. monitoring Dec 15, 2025, 02:17 PM UTC

   We are currently accepting new memo posts, but processing is slower than usual. Our team is actively monitoring the system for any issues, and we will provide updates as we make progress.

4. monitoring Dec 15, 2025, 02:31 PM UTC

   Memo posts are currently flowing correctly, and we continue to monitor the system as a few items are still pending.

5. monitoring Dec 15, 2025, 02:54 PM UTC

   We encountered a minor database issue while processing a large batch of memo posts. That issue has been fixed, and we are currently monitoring the system as the previously-stuck transfers are now being sequentially processed.

6. monitoring Dec 15, 2025, 03:57 PM UTC

   All Memoposts have been processed and normal operations are resumed. we're sill actively monitoring instant funding activity to check for new issues or leftovers.

7. resolved Dec 15, 2025, 06:26 PM UTC

   As we didnt observe any issue with respect to memopost since our last update hence marking it resolved here.

8. postmortem Dec 16, 2025, 04:05 PM UTC

   ## **Degraded Instant Funding API Performance \(December 15, 2025\)** ‌ ### **What Happened** Starting at approximately 7:24 AM EST \(12:24 UTC\) on December 15, 2025, clients using the Instant Funding API experienced an increase in transaction failures and gateway timeouts. Our initial investigation identified a technical issue where our internal payment processing service experienced significant delays when communicating with our core transaction engine. This was further complicated by an unexpected behavior in the internal ledger system that led to transaction processing issues, causing a backlog of pending funding requests. These concurrent issues collectively resulted in slow processing times and the eventual timeout errors visible to our partners. ### **Impact** We understand this event caused operational delays and deeply regret the inconvenience. * **Service Affected:** Instant Funding API \(specifically the `v1/instant_funding` endpoint\). * **User Experience:** Affected users attempting Instant Funding transactions were unable to complete them, resulting in visible HTTP 500 errors or gateway timeouts. * **Scope:** The issue primarily impacted partners using the Instant Funding API. At its peak, approximately **8,300 funding transfers** across multiple partners were delayed and queued for processing. * **Data Integrity:** **Crucially, we can confirm that no client funds or transaction data were lost or compromised.** All affected requests were successfully retained in a pending state and processed upon resolution. ### **Resolution** Our engineering teams were immediately mobilized and worked continuously to resolve the issue and process the accumulated backlog. 1. **Immediate Restoration:** The primary service involved in the funding flow was restarted to terminate open connections and locks held in wrongful states, which immediately mitigated the initial timeouts and allowed new requests to flow correctly. 2. **Addressing Core Instability:** Our internal core transaction engine was identified as contributing to the issue, and we immediately performed a controlled rollback to a previously stable version, which resolved the transaction processing errors. 3. **Backlog Clearance:** To clear the substantial backlog of pending transfers in a stable manner, we temporarily paused automated processing and executed the transfers in smaller, controlled batches. 4. **Full Recovery:** All service components returned to normal processing capacity, and the entire backlog of approximately 8,300 transfers was successfully processed by 10:14 AM EST \(15:14 UTC\). Normal API response times and throughput have been fully restored. ### **Commitment & Preventative Measures** We are committed to maintaining the highest level of service reliability. Our team has initiated a detailed post-incident review and is prioritizing the following themes to prevent recurrence: * **System Resilience & Stability:** We are immediately prioritizing and implementing a planned architectural update to the payment service to allow it to handle large batches of pending transfers more robustly and efficiently. * **Monitoring and Alerting:** We will enhance our database monitoring systems to more quickly detect and alert on internal system resource contention \(e.g., database transaction processing issues\), allowing for faster intervention before customer-facing impact occurs. * **Infrastructure Optimization:** We will conduct a comprehensive capacity review of the relevant service components and transaction databases to ensure they can scale effectively under high-volume load and prevent bottlenecks that can lead to system-wide lockups. We appreciate your patience and partnership as we continue to improve the reliability of our platform.

Read the full incident report →

Started Dec 15, 2025, 01:08 PM UTC· resolved Dec 15, 2025, 03:58 PM UTC

1. investigating Dec 15, 2025, 01:08 PM UTC

   We are currently investigating the JNLC issue, as it appears to be stuck. Our engineering team is already looking into it, and we will provide an update shortly.

2. identified Dec 15, 2025, 01:23 PM UTC

   Our engineering team has identified the issue and is implementing mitigations to reduce the impact. We are also monitoring transaction throughput and processing times closely to ensure all pending items continue to move through the system.

3. monitoring Dec 15, 2025, 01:30 PM UTC

   Journal processing is back to normal. We’re actively monitoring to ensure stability.

4. resolved Dec 15, 2025, 03:58 PM UTC

   As we didn't observe any issue since last update, we are marking this resolve.

5. postmortem Dec 16, 2025, 04:03 PM UTC

   ## **Transaction Processing Delays and Restoration \(December 15, 2025\)** We sincerely apologize for the service degradation experienced on December 15, 2025, which primarily affected the timely processing of transactions and related account updates for our B2B partners. We understand the impact this can have on your operations and appreciate your patience as our teams worked to restore full stability. ### **What Happened** The incident began with our core ledger system experiencing database connection processing issues. The root cause was identified as a code change deployed on Dec 10th, internal system update intended for performance monitoring. This update required additional database connections, which under high system load ended up consuming all available transactional database connections and created a bottleneck. Although the release was deployed on Wednesday, we did not observe this issue on Wednesday, Thursday, or Friday.The update inadvertently exhausted the available connections to our transactional database, creating a severe bottleneck. This issue first manifested as a rapid slowdown in our internal job queue, preventing new account and transaction events \(journals and memo posts\) from being processed promptly. We deployed an initial quick fix to address the connection issue; however, it did not fully resolve the problem. As a result, we ultimately rolled back to the prior stable version of the core service. ### **Impact** The degradation resulted in short delays and a build-up of transaction volume within our systems. * **Journal Processing:** We observed a queue of approximately 2,500 journal entries awaiting processing, leading to short delays in real-time account updates. This queue was successfully cleared and processed during the initial mitigation. * **Memo Post Processing:** A significant backlog of nearly 9,000 memo posts becoming stuck in a pending state, causing delays in related services like instant funding. * **Data Integrity:** Crucially, we can confirm that no client data was lost, and all funds and transactions remain secure. The system was successfully rolled back, and all backed-up transactions were eventually processed correctly. ### **Resolution** The system is now fully stable and operational, and all pending transaction volume has been processed. 1. **Immediate Hotfix:** Our team initially deployed a fix to disable the problematic metric collection feature, which cleared the initial journal processing queue. 2. **Emergency Rollback:** Due to a subsequent critical error, an immediate emergency rollback of the core ledger service to a stable version was executed, which successfully restored core system stability. 3. **Backlog Clearance:** Following the rollback, all pending memo posts and other transactions were successfully processed in batches to ensure full and accurate booking. Normal transaction processing and system throughput have been fully restored. ### **Preventative Measures** We are committed to preventing recurrence and enhancing the resilience of our platform. To address the core issue of resource exhaustion, our immediate focus is on strengthening our testing and validation process. We will implement rigorous, high-load **stress testing** for all major service deployments, specifically targeting database connection pooling and resource consumption. This action will ensure system stability under maximum transactional volume and prevent future capacity issues. We are proactively enhancing our database observability by implementing granular telemetry and refined alerting thresholds. This framework will provide deeper visibility into connection health, allowing our engineering teams to identify and mitigate potential saturation risks before they impact service availability. ‌ We are committed to maintaining a highly reliable and transparent service for our partners and will provide an update once these key follow-up items are completed.

Read the full incident report →

Started Dec 05, 2025, 12:26 PM UTC· resolved Dec 05, 2025, 12:26 PM UTC

1. resolved Dec 05, 2025, 12:26 PM UTC

   Due to a brief Cloudflare outage (~16-24 minutes), some Alpaca services including our public documentation site, Broker dash, and website were temporarily unavailable. Our APIs (including Trading API and Broker API) were not affected. Cloudflare has resolved the issue and all services have been restored. Further Details here https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q

Read the full incident report →

Started Dec 02, 2025, 04:52 AM UTC· resolved Dec 02, 2025, 05:15 AM UTC

1. investigating Dec 02, 2025, 04:52 AM UTC

   We are currently investigating an incident impacting our API services. Clients may be experiencing issues with API requests, including failures or unexpected service restarts. This is due to the failure of several k8s nodes.

2. investigating Dec 02, 2025, 04:54 AM UTC

   We see errors have dropped to normal levels.

3. resolved Dec 02, 2025, 05:15 AM UTC

   The root cause has been identified as an issue with the cluster nodes. Our team will be conducting internal investigation into how we can make the system more resilient to these types of failures in the future.

4. postmortem Dec 02, 2025, 06:23 PM UTC

   We are committed to providing a reliable and stable platform for our B2B partners. We sincerely apologize for the disruption experienced on December 1, 2025, which resulted in intermittent API service degradation. We have fully resolved the issue and restored all services to normal operation. ‌ ### **What Happened** At approximately 10:37pm ET on December 1, 2025, our internal monitoring systems detected an elevated error rate in our primary production API services. The disruption was triggered by the unexpected failure of several core internal compute resources \(nodes\) within our cloud infrastructure. This failure led to the abrupt restart of various underlying financial services and associated processing jobs, which in turn caused the elevated error rate and connectivity issues for our clients. Our engineering teams immediately mobilised to isolate the failure and restore system health. We quickly identified the cause as a resource scheduling constraint preventing new nodes from scaling up to replace the failed ones. ### **Impact** The incident was mitigated within 17 minutes. * **API Connectivity:** Clients experienced brief connection failures and elevated 5xx error responses when making API requests. * **Transaction and Data Impact:** Approximately **56,400 API requests** failed during the brief period of the incident. * **Crucially, we can confirm that no client data was lost, and all funds and transactions remain secure and properly accounted for.** Any critical service components that experienced downtime were successfully restarted and recovered without data corruption. ### **Resolution** Our primary focus was on rapidly mitigating the service impact: 1. **Stabilisation and Service Restoration:** The affected systems automatically restarted after new compute resources became available. 2. **Full Service Verification:** We confirmed the complete resolution of the underlying node issues and verified the full functionality of our core API and trading systems. ### **Preventative Measures and Commitment** To prevent a recurrence of this issue and strengthen the resilience of our platform, our teams are prioritizing the following action items: * **System Resilience & Monitoring:** We will conduct a thorough review of the replication and redundancy settings for affected systems and dependencies. We are increasing the number of instances for some services to reduce the impact of similar failures. * **Service Logging and Observability:** We identified several logs that could be adjusted to accelerate diagnosing outages in the future and are making changes to them. * **Minor Bug Fixes:** We identified a few minor improvements in the way our services handled the failure and are updating those services, which should accelerate recovery in the future. We are committed to maintaining the highest standards of reliability and transparency. We appreciate your patience and understanding as we continue to invest in our platform's stability

Read the full incident report →

Started Nov 25, 2025, 10:15 AM UTC· resolved Nov 25, 2025, 01:16 PM UTC

1. investigating Nov 25, 2025, 10:15 AM UTC

   The Nov 24, 2025 beginning-of-day job completed, but some accounts failed to update correctly. Our engineering team is actively working to correct the affected balances.

2. investigating Nov 25, 2025, 10:52 AM UTC

   Our teams have narrowed down the issue, and we are currently evaluating potential solutions. We will provide an update shortly.

3. identified Nov 25, 2025, 11:06 AM UTC

   The job responsible for updating balances was failing due to database timeout issues. We’ve increased the timeout settings and are rerunning the job. We’ll keep you posted on the progress.

4. identified Nov 25, 2025, 11:35 AM UTC

   Our database is experiencing some increased load, but the balance-update job is still running without any issues. We're actively monitoring it and will keep you updated as it progresses.

5. monitoring Nov 25, 2025, 11:56 AM UTC

   We’ve completed remediation and all affected accounts now reflect correct beginning-of-day balances. No action is required on your side. We are continuing to monitor to ensure there are no further issues. Thank you for your patience while we worked through this.

6. resolved Nov 25, 2025, 01:16 PM UTC

   As we are no longer seeing any issue reported with balances so marking this as resolved.

7. postmortem Nov 25, 2025, 07:20 PM UTC

   We understand the importance of timely and accurate balance information. This update provides details regarding a delay in the synchronization of beginning-of-day balances for some partner accounts that occurred on the morning of **November 25, 2025**. ### **What Happened** Our beginning-of-day \(BOD\) job, a critical internal process responsible for calculating and updating client cash and position balances, experienced a series of failures starting early this morning. This failure was caused by an issue in our transaction processing system. The job, which handles a large volume of data, encountered database query timeouts during the reconciliation of account balance data from the previous day. This high-load query was performing inefficiently, leading to connection closures and the subsequent failure of the balance synchronization job. Repeated manual attempts to rerun the job were also unsuccessful due to these performance issues. ### **Impact** The core impact was a delay in the timely synchronization of balances for a subset of our B2B partner accounts. * Affected Data: The primary impact was on the display and availability of the Beginning-of-Day \(BOD\) balances for November 25, 2025. * Scope: Approximately 680,000 accounts across specific correspondents were initially impacted by the delay in updated balances. * Crucially, we can confirm that NO client funds or transaction records were lost, compromised, or incorrectly executed due to this issue. All underlying trade data and financial records remained secure and intact. ### **Resolution** The issue is now fully resolved, and normal service has been restored. Our engineering teams successfully mitigated the problem by increasing the database query transaction timeout and reducing the batch size for the balance processing job. This change allowed the job to complete the heavy data reconciliation process without timing out. Additionally, we performed targeted manual updates for a critical set of partner accounts to minimize disruption while the main job completed. The balance synchronization job completed successfully, and all affected accounts now reflect the correct Beginning-of-Day balances for November 25, 2025. ### **Preventative Measures and Future Commitment** We are committed to enhancing the stability of our data processing pipeline to prevent recurrence. Our immediate follow-up actions are focused on three core themes: 1. System Resiliency & Optimization: * Permanent Database Query Refactoring: Implement the identified, optimized query logic to reduce the execution time of the critical balance calculation query from minutes to under 30 seconds. This is the long-term fix for the root cause. * Persistent Configuration: Permanently adjust the transaction timeout and batch size configurations for the balance processing job to ensure reliable execution, regardless of data volume fluctuations. 2. Enhanced Monitoring & Observability: * Load and Performance Profiling: Integrate advanced profiling tools into the balance job to better identify and troubleshoot slow queries under real-time production load in the future. * Connection Management Review: Investigate and address an observed increase in idle database connections to safeguard resource availability for critical jobs. 3. Partner Communication Enhancements: * Streamlined Eventing: We are finalizing the design to publish more granular system events to partners upon the completion of key processes, such as the End-of-Day \(EOD\) and Beginning-of-Day \(BOD\) balance updates, improving transparency and integration capabilities. We apologize for the inconvenience this delay caused and thank you for your patience and understanding. Maintaining the highest level of system reliability and transparency is our top priority. * The Alpaca Team

Read the full incident report →

Started Nov 21, 2025, 09:47 AM UTC· resolved Nov 21, 2025, 11:17 AM UTC

1. identified Nov 21, 2025, 09:47 AM UTC

   There is a delay in batch data processing. Users may experience delayed memo post credits to their accounts, proceeds from previous business day sales may be unavailable for withdrawal for 1.5h.

2. identified Nov 21, 2025, 10:12 AM UTC

   The BOD job is running smoothly. It should complete within the next hour.

3. monitoring Nov 21, 2025, 10:48 AM UTC

   The BOD job completed successfully. We are monitoring for any additional issues.

4. resolved Nov 21, 2025, 11:17 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 18, 2025, 12:50 PM UTC· resolved Nov 18, 2025, 08:04 PM UTC

1. investigating Nov 18, 2025, 12:50 PM UTC

   Cloudflare is having an outage which is affecting Alpaca's UI. https://www.cloudflarestatus.com/incidents/8gmgl950y3h7 Our APIs are available.

2. monitoring Nov 18, 2025, 01:56 PM UTC

   We have bypassed Cloudflare after regaining access to the Cloudflare dashboard for now while they continue having issues.

3. resolved Nov 18, 2025, 08:04 PM UTC

   Cloudflare has resolved their incident and we have migrated our domains back to Cloudflare.

Read the full incident report →

Started Nov 18, 2025, 11:31 AM UTC· resolved Nov 18, 2025, 01:17 PM UTC

1. investigating Nov 18, 2025, 11:31 AM UTC

   JNLCs are getting delayed due to the high volume of batches being processed in our ledger system. We are continuously monitoring the system during this period.

2. monitoring Nov 18, 2025, 12:48 PM UTC

   We have processed the large batches and we are back to normal. All backlogs are cleared.

3. resolved Nov 18, 2025, 01:17 PM UTC

   Manual batches are cleared and system back to normal

4. postmortem Nov 19, 2025, 01:45 PM UTC

   We are providing an update on a service interruption that impacted transaction processing on **November 18, 2025**. ### **What Happened** During a period of heavy transaction load, a specific internal maintenance process temporarily blocked data synchronization between our core accounting and reconciliation systems. This blockage caused a backlog, temporarily delaying the completion status for a segment of executed financial transactions \(journals\). ### **Impact** * Approximately **40,000 transactions** were processed but temporarily showed an intermediary, "stuck" status. * **Partners who rely on real-time transaction status updates or derived balance fields may have experienced brief delays or temporary inconsistencies in reporting.** * **No customer funds or executed transactions were lost.** All transactions were successfully processed by our core ledger system. ### **Resolution** Our engineering team identified the degradation in our database synchronization immediately. We performed a controlled remediation process to reconcile the data, manually updating the status of the backlogged transactions and verifying consistency across all affected systems. **The issue has been fully resolved, and normal transaction processing is fully restored.** ### **Preventative Measures** We have identified the root cause—a long-running internal job that held a database resource for too long—and are implementing immediate fixes to ensure this does not recur: * **Process Optimization:** Hardening the internal job to prevent future resource contention. * **Resilience and Monitoring:** Enhancing our tools for faster recovery and strengthening alerts around transaction backlogs and database health to detect similar issues instantly. We appreciate your understanding and thank you for your patience. We remain committed to the highest standards of reliability.

Read the full incident report →

Started Nov 14, 2025, 08:41 AM UTC· resolved Nov 14, 2025, 08:41 AM UTC

1. resolved Nov 14, 2025, 08:41 AM UTC

   This incident has been resolved

Read the full incident report →

Started Nov 13, 2025, 06:31 PM UTC· resolved Nov 13, 2025, 09:42 PM UTC

1. investigating Nov 13, 2025, 06:31 PM UTC

   We’ve identified that journals are stuck in the "Sent_to_Clearing" status, and the team is actively investigating the issue.

2. resolved Nov 13, 2025, 09:42 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Nov 13, 2025, 06:13 PM UTC· resolved Nov 13, 2025, 06:17 PM UTC

1. investigating Nov 13, 2025, 06:13 PM UTC

   We are currently investigating this issue.

2. identified Nov 13, 2025, 06:14 PM UTC

   We have identified that dividend payouts for asset AAPL were duplicated while processing

3. monitoring Nov 13, 2025, 06:16 PM UTC

   The duplicated dividends entries are cancelled and balances are adjusted. This would have caused change in balances. SSE events are also being pushed for cancellation.

4. resolved Nov 13, 2025, 06:17 PM UTC

   This incident has been resolved.

5. postmortem Nov 19, 2025, 01:43 PM UTC

   We are issuing an update regarding a brief service issue that occurred on **November 13, 2025**, which temporarily caused an incorrect display of cash and buying power for certain accounts. ### **What Happened** A small number of accounts received **duplicate cash dividend credits** related to a recent **AAPL** corporate action. This occurred due to a combination of factors: 1. An operational error where a dividend processing team member attempted to execute the action a second time after receiving a system timeout on the first attempt \(which had, in fact, been successful\). 2. Although the duplicate credit was swiftly corrected by a **reversal entry**, a temporary **system synchronization delay** prevented this correction from immediately appearing in all partner and client-facing systems. This delay resulted in the temporary **overstatement of cash balances and buying power** in affected accounts. ### **Impact** * A subset of accounts temporarily displayed **incorrect, inflated cash balances and buying power.** * The primary risk was confined to the brief window where clients _could_ have attempted to withdraw or trade against these temporarily overstated balances. * Some of our partners experienced **increased support inquiries** and operational effort while we worked to align all system balances. * **No actual financial loss** occurred as all balances were fully reconciled. ### **Resolution** The issue was first raised by our partners and quickly confirmed by our internal monitoring, which detected unexpected changes in account balances. The **overstated cash and buying power balances were fully rectified and aligned** across all systems during our **end-of-day reconciliation process**. All client balances are now correct and reflect the single, accurate dividend credit. ### **Preventative Measures** We are implementing several key changes to prevent this type of incident and improve the resilience of our corporate actions processing: * **Operational Protocol Update:** Reviewing and updating internal procedures for processing and reversing large transaction batches, especially when system errors or timeouts occur, to ensure single execution. * **System Capacity Review:** We are working to enhance the handling capacity of our core processing systems to better manage large events and prevent timeouts or misleading errors that can lead to manual duplication. * **Workflow Improvements:** Developing more robust, systematized controls and workflows in our user interfaces to reduce reliance on manual processing steps. We sincerely apologize for the operational noise and inconvenience this caused our partners and their clients. We appreciate your partnership as we reinforce our systems.

Read the full incident report →

Started Nov 12, 2025, 12:52 PM UTC· resolved Nov 12, 2025, 04:24 PM UTC

1. investigating Nov 12, 2025, 12:52 PM UTC

   There is currently a display issue impacting the cash withdrawable amount for accounts that executed trades on 11/11. Our team is investigating and are working to fix.

2. monitoring Nov 12, 2025, 12:58 PM UTC

   The team is performing a rollback of a recent change, which may temporarily affect pre-market trading until the rollback process is fully completed.

3. monitoring Nov 12, 2025, 12:59 PM UTC

   we are seeing the issue recovering and the team is monitoring.

4. monitoring Nov 12, 2025, 01:20 PM UTC

   We are continuing to monitor for any further issues.

5. resolved Nov 12, 2025, 04:24 PM UTC

   Everything is now stable hence marking this incident as resolved.

6. postmortem Nov 19, 2025, 01:42 PM UTC

   We are reporting on a service issue that temporarily displayed incorrect **Cash Withdrawable** balances for a subset of users, starting late on **November 10, 2025**, and resolving after our Beginning of Day \(BOD\) process on **November 12, 2025**. ### **What Happened** A recent code change, which was deployed to enhance security around money-out validation \(preventing the use of unsettled funds\), unfortunately introduced a regression in our balance calculation logic. In specific scenarios, this faulty logic mistakenly flagged funds that were already **fully settled** as unavailable for withdrawal. This caused the **Cash Withdrawable** balance to show as **$0** or significantly lower than the available **Buying Power**, despite the user having sufficient settled funds. ### **Impact** * Some users saw an **incorrect Cash Withdrawable balance**, leading to confusion and unnecessary support inquiries. * **Withdrawal attempts** for correctly settled funds may have been temporarily **blocked or failed** until the issue was resolved. * Our partners experienced **elevated support volume** while assisting users with the discrepancy between Buying Power and Cash Withdrawable. * **Importantly, the underlying account balances and Buying Power were correct; only the Cash Withdrawable field was miscalculated.** ### **Resolution** The issue was detected following reports from multiple partners and confirmed by internal teams. Our engineering team took immediate steps: 1. We isolated the issue to the recently deployed validation logic. 2. We successfully **rolled back or updated the faulty logic** to restore the correct treatment of settled funds. 3. The correct Cash Withdrawable balances were fully restored and verified across all affected accounts after the subsequent Beginning of Day process. **All affected balances are now displaying correctly, and withdrawal functionality has been fully restored.** ### **Preventative Measures** To ensure the integrity of our balance calculations and prevent similar incidents, we are implementing the following immediate actions: * **Expanded Testing:** Significantly increasing test coverage for all money-out, settlement, and holiday-related timeline logic to proactively catch these scenarios. * **Targeted Monitoring:** Adding new system alerts specifically designed to detect any mismatch or discrepancy between Buying Power and Cash Withdrawable balances. * **Deployment Safeguards:** Implementing stricter change-management and rollout procedures for any code changes affecting core balance logic before they are deployed to production. We apologize for the confusion and operational burden this incident placed on our partners and their clients. We are committed to maintaining accurate balance display across all systems.

Read the full incident report →

Started Nov 11, 2025, 07:25 AM UTC· resolved Nov 11, 2025, 07:50 AM UTC

1. investigating Nov 11, 2025, 07:25 AM UTC

   We are currently investigating this issue.

2. resolved Nov 11, 2025, 07:50 AM UTC

   The incident is resolved.

3. postmortem Nov 19, 2025, 01:41 PM UTC

   We are reporting on a service interruption that affected the activation of new accounts for several partners, starting on or around **November 10–11, 2025**. ### **What Happened** Our automated system responsible for moving newly onboarded accounts from the **"Approved"** status to **"Active"** encountered a technical limit within our database environment. Specifically, the process was attempting to handle too many accounts in a single operation. Once this database limit was hit, the activation job stalled, resulting in a large backlog of accounts that were approved but failed to receive the final **account\_activated** event. This prevented users from fully utilizing their newly opened accounts. ### **Impact** * **Scope:** Tens of thousands of accounts across several partners were temporarily stuck in the "Approved" status. * **User Experience:** Affected users experienced delays in the final activation of their brokerage accounts, preventing full use of their new accounts. * **Partner Impact:** Partners observed a halt in the emission of the expected **account\_activated** events and faced increased support inquiries regarding the delay. * **Total Resolution Time:** The issue was fully identified and resolved within **80 minutes**. ### **Resolution** The issue was identified through partner reports of missing activation events and internal monitoring showing an unusual build-up of accounts in the intermediate "Approved" status. Our engineering team took the following steps: 1. We isolated the affected accounts and safely re-entered them into the activation pipeline. 2. We **restarted the activation flow** in smaller, controlled batches to ensure adherence to database limits and prevent recurrence. 3. We verified that the accounts successfully progressed to **"Active,"** and the **account\_activated** events are now being emitted reliably. **The account activation pipeline is now functioning normally.** ### **Preventative Measures** To prevent recurrence and strengthen our onboarding pipeline, we are implementing the following fixes: * **Process Adjustment:** Modifying the account activation jobs to automatically process work in batches that respect database constraints. * **Enhanced Monitoring:** Adding new monitoring and alerts specifically designed to detect: * Unusual growth in accounts stuck in intermediate statuses \(e.g., Approved but not Active\). * Any unexpected drop in the volume of **account\_activated** events. * **Operational Readiness:** Updating internal runbooks to enable faster and safer remediation of large-scale account backlogs, should a similar issue arise. We apologize for the delay in account opening experienced by your users and the resulting operational impact on your teams. We appreciate your patience as we continue to enhance the stability of our critical services.

Read the full incident report →

Started Nov 04, 2025, 03:11 PM UTC· resolved Nov 05, 2025, 04:19 PM UTC

1. investigating Nov 04, 2025, 08:39 AM UTC

   Our end of day job is running longer than expected. This will impact the balance and position snapshot for the 3-NOV, 2025. Account activation will be delayed.

2. identified Nov 04, 2025, 11:23 AM UTC

   We are rerunning the job after rollback

3. identified Nov 04, 2025, 12:08 PM UTC

   Position update completed. Balance calculation still running.

4. identified Nov 04, 2025, 02:39 PM UTC

   Our team is still actively investigating the issue impacting cash withdrawals and Buying Power accuracy. Impacted Functions Remain: 1. Inability to withdraw cash. 2. Stale or inaccurate Buying Power information. We understand the severity of this issue and are working with the highest urgency to isolate the root cause and deploy a fix.

5. identified Nov 04, 2025, 03:11 PM UTC

   We are prioritizing the completion of a necessary process to restore the ability to withdraw cash. Important Note: While we work to resolve the withdrawal issue, the following figures will temporarily remain unreliable or outdated: 1. Day Trading Buying Power 2. Pattern Day Trading limits 3. Margin Multiplier calculations. 4. Displayed Interest Rates. Thank you for your continued patience.

6. identified Nov 04, 2025, 04:03 PM UTC

   The necessary position sync is complete, and we confirm that cash is now withdrawable for all clients. Our engineering team is now focused on resolving the remaining issues that impact financial calculations. We are working to restore accuracy for: 1. Day Trading and Pattern Day Trading Buying Power. 2. Margin Multiplier. 3. Up-to-date Interest Rates. Thank you for your continued patience.

7. identified Nov 04, 2025, 05:42 PM UTC

   We confirm that withdrawable cash and settled cash functionality have been fully resolved and are working normally. Our engineering team is still focused on the final dependent issues related to account calculations. We are actively working to restore the accuracy and availability of the following figures: 1. Day Trading Buying Power 2. Pattern Day Trading limits 3. Margin Multiplier calculations 4. Displayed Interest Rates We appreciate your continued patience as we work through these final fixes.

8. identified Nov 04, 2025, 07:57 PM UTC

   Our engineering team has successfully identified and validated the final fix required to restore accurate balances and Buying Power figures. Action Plan: To avoid any interruption to live trading, we will initiate the final data synchronization process after the close of trading hours today. This process is confirmed to restore accuracy for account balances and all Buying Power figures. Temporary Impact During Sync: Please note that Account Creation will be temporarily queued (for approximately one hour) while the system performs the final sync. All other core services will remain operational. We will update this page immediately once the sync is complete and all figures are fully restored.

9. identified Nov 04, 2025, 09:35 PM UTC

   The critical system resync to fix client balances and Buying Power is now running. We anticipate a temporary impact on account creation, but all new requests are being queued and will be processed immediately once the sync is complete.

10. monitoring Nov 04, 2025, 10:43 PM UTC

    Full Balance and Buying Power Accuracy Restored We are happy to confirm that the final system resynchronization process has successfully completed. All client balances, Buying Power, Margin Multiplier, and Interest Rate calculations are now accurate and fully restored. We sincerely appreciate your patience throughout this extended incident. Our team is now actively monitoring all affected systems to ensure stability and to prevent recurrence.

11. resolved Nov 05, 2025, 04:19 PM UTC

    Following the final system resynchronisation process yesterday, all related jobs ran successfully, and no further issues were observed. We are marking this incident as resolved and will share a detailed root cause analysis and postmortem shortly.

12. postmortem Nov 07, 2025, 12:14 AM UTC

    ## Incident Summary: The issue originated early that morning with a delay in our end-of-day job, which impacted balance and position snapshots and led to inaccuracies in several financial calculations. Crucially, this was a display issue only; there was no material impact to the source data or customers' actual funds or positions. ## Resolution: Our team worked in stages to restore services, first resolving all functionality for withdrawable cash and settled cash. This restored functionality while also enhancing operating levels. Concurrently with the remediation, we proactively upgraded the database to significantly boost capacity, stability, and speed. This not only restored functionality but also resulted in enhanced operational levels. We can now confirm that the final data synchronization process, initiated after the close of trading hours on November 4, **completed successfully**. This has restored full accuracy to all account balances and the following figures: * Day Trading Buying Power * Pattern Day Trading limits * Margin Multiplier calculations * Displayed Interest Rates All services are now operating normally. We sincerely apologize for the disruption this has caused to your business and clients. We appreciate your patience as we worked to complete this resolution.

Read the full incident report →

Started Oct 29, 2025, 08:50 PM UTC· resolved Oct 30, 2025, 01:46 AM UTC

1. identified Oct 29, 2025, 08:50 PM UTC

   Our options market data provider, Exegy, is experiencing a system data feed outage at this time. We are currently working with our vendor to restore the system as soon as possible.

2. resolved Oct 30, 2025, 01:46 AM UTC

   The issue has been resolved. Live data resumed normal operation around 3:15 PM EST, and historical data was backfilled and corrected shortly after. The Exegy team has confirmed resolution and will implement additional monitoring and safeguards to prevent recurrence.

Read the full incident report →

Started Oct 27, 2025, 04:07 PM UTC· resolved Oct 27, 2025, 05:49 PM UTC

1. investigating Oct 27, 2025, 04:07 PM UTC

   We are able to see high increase in database connection usage

2. investigating Oct 27, 2025, 04:36 PM UTC

   We are working on it

3. resolved Oct 27, 2025, 05:49 PM UTC

   Identified the issue and is fixed

Read the full incident report →

Started Oct 23, 2025, 06:05 AM UTC· resolved Oct 23, 2025, 03:45 PM UTC

1. investigating Oct 23, 2025, 06:05 AM UTC

   Client using SSE journal steam via API /v2beta1/events/journals/status will not be receiving the journal status due to ongoing issue.

2. identified Oct 23, 2025, 06:06 AM UTC

   We have identified the issue. This has affected Prod and Sandbox. Recovery in sandbox is in progress

3. identified Oct 23, 2025, 06:35 AM UTC

   We are still working on the recovery

4. identified Oct 23, 2025, 07:30 AM UTC

   Sandbox events have resumed and system functionality is restored.

5. monitoring Oct 23, 2025, 07:58 AM UTC

   Both Production and Sandbox environments have recovered and event flow is stable. We identified a gap of approximately 7 hours of journal events during the incident period. The missing data has been captured from another source and will be reconciled separately. We’ll continue to monitor to ensure no further impact.

6. monitoring Oct 23, 2025, 01:35 PM UTC

   We have identified the issue is reoccurring again

7. monitoring Oct 23, 2025, 03:44 PM UTC

   We are continuing to monitor for any further issues.

8. resolved Oct 23, 2025, 03:45 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 22, 2025, 06:28 PM UTC· resolved Oct 22, 2025, 07:31 PM UTC

1. monitoring Oct 22, 2025, 06:28 PM UTC

   Our options market data provider, Exegy, experienced two intermittent data feed outages today during the following approximate times: 1:12 – 1:29 PM ET 1:39 – 1:50 PM ET During these periods, you may have noticed option data values showing as 0. We apologize for any inconvenience this may have caused. The data is now back online and functioning normally. We will continue to monitor the situation and provide updates if necessary.

2. resolved Oct 22, 2025, 07:31 PM UTC

   Our market data vendor has confirmed that OPRA data is now back online and operating normally. We will continue to monitor the situation and issue a new update if any additional outages occur.

Read the full incident report →

Started Oct 22, 2025, 12:44 AM UTC· resolved Oct 22, 2025, 12:44 AM UTC

1. investigating Oct 22, 2025, 12:44 AM UTC

   Brokerage trading partners were unable to login to portal (https://broker-app.alpaca.markets) for a short period of time.

2. resolved Oct 22, 2025, 12:44 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 20, 2025, 08:37 AM UTC· resolved Oct 20, 2025, 11:53 AM UTC

1. identified Oct 20, 2025, 08:37 AM UTC

   We're currently experiencing a login issue due to a widespread outage affecting Amazon Web Services (AWS), which is impacting our authentication systems. Our team is actively monitoring the situation, and we will provide an update as soon as services are restored.

2. monitoring Oct 20, 2025, 09:39 AM UTC

   As per AWS update - they have applied initial mitigations and observing early signs of recovery for some impacted AWS Services. During this time, requests may continue to fail as we work toward full resolution. We recommend customers retry failed requests. While requests begin succeeding, there may be additional latency and some services will have a backlog of work to work through, which may take additional time to fully process. We will continue to monitor and provide updates.

3. monitoring Oct 20, 2025, 11:00 AM UTC

   According to the latest update from AWS, the underlying DNS issue has been fully resolved, and most AWS services are now operating normally. Trading API users can now log in via https://app.alpaca.markets/account/login without any issues.

4. resolved Oct 20, 2025, 11:53 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 18, 2025, 06:42 AM UTC· resolved Oct 18, 2025, 12:52 PM UTC

1. investigating Oct 18, 2025, 06:42 AM UTC

   There was a end of day job failure causing delay in balance and position computation availability. System is recovering and expected to be completed by 6:30 AM ET.

2. monitoring Oct 18, 2025, 06:42 AM UTC

   A fix has been implemented and we are monitoring the results.

3. monitoring Oct 18, 2025, 07:19 AM UTC

   Rerun of job is in progress

4. monitoring Oct 18, 2025, 09:36 AM UTC

   Rerun of job is in progress

5. resolved Oct 18, 2025, 12:52 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Oct 16, 2025, 02:05 PM UTC· resolved Oct 16, 2025, 02:32 PM UTC

1. investigating Oct 16, 2025, 02:05 PM UTC

   The team has identified some delays in processing some journals

2. monitoring Oct 16, 2025, 02:23 PM UTC

   Our systems are stabilized now and our engineers are monitoring the performance

3. resolved Oct 16, 2025, 02:32 PM UTC

   Systems are stabilized

Read the full incident report →

Started Oct 16, 2025, 07:06 AM UTC· resolved Oct 16, 2025, 10:44 AM UTC

1. investigating Oct 16, 2025, 07:06 AM UTC

   This long running job would reconcile end of day balances and positions for 15-OCT-2025. This would also delay new account activation, margin call and end of day position and balance data.

2. investigating Oct 16, 2025, 07:46 AM UTC

   We are monitoring

3. identified Oct 16, 2025, 08:41 AM UTC

   We are creating the end of day balances and positions as of 15-OCT-2025

4. resolved Oct 16, 2025, 10:44 AM UTC

   We’ve resolved the earlier delay with our End-of-Day (EOD) processing for October 15th. All jobs have now completed successfully, end-of-day balances and positions are now available, and funds from yesterday's sales are also now available for withdrawal. The delay was caused by a query that unintentionally blocked the EOD process. Once identified, it was safely terminated to allow processing to continue. We’re now reviewing the root cause and taking steps to help prevent similar issues in the future.

Read the full incident report →