Alpaca Outage History

Started May 23, 2026, 02:06 AM UTC· resolved May 23, 2026, 02:06 AM UTC

1. resolved May 23, 2026, 04:52 AM UTC

   On May 22, 2026, at 10:06 PM ET, Alpaca experienced a partial outage of Trading API services for approximately five minutes. Services were restored quickly and our team is continuing to monitor Trading API services to ensure they remain operational.

Read the full incident report →

Started May 21, 2026, 02:33 PM UTC· resolved May 21, 2026, 08:36 PM UTC

1. investigating May 21, 2026, 02:33 PM UTC

   We are currently investigating a technical issue that caused some orders to receive duplicate executions following a restart of the principal engine. This may result in some customers temporarily seeing incorrect filled quantities or affected buying power. Our engineering team is actively working to resolve this We will provide an update once the corrections are complete.

2. identified May 21, 2026, 02:40 PM UTC

   All affected orders have been identified. Corrections are in progress and we will confirm once fully resolved.

3. monitoring May 21, 2026, 03:22 PM UTC

   We have completed our corrections and the accurate filled quantities should now be reflected in customer accounts. Our teams are now finalising any outstanding data clean up and will provide a formal trade event notification and final update once this is completed.

4. monitoring May 21, 2026, 03:46 PM UTC

   We are now preparing to send the corrected trade events and will confirm once they have been processed.

5. monitoring May 21, 2026, 04:44 PM UTC

   Our team is continuing to work on the resolution. We will provide another update once the corrected trade events have been sent.

6. monitoring May 21, 2026, 05:04 PM UTC

   We have identified a buying power issue affecting accounts impacted by the earlier duplicate execution. Our team is actively working on a fix as a high priority. We will update you once resolved.

7. investigating May 21, 2026, 05:15 PM UTC

   We are continuing to investigate the buying power issue affecting accounts impacted by the earlier duplicate execution. Our team is actively working on a fix as a high priority. We will continue to update you as we investigate.

8. investigating May 21, 2026, 05:25 PM UTC

   We are continuing to investigate the buying power issue affecting accounts impacted by the earlier duplicate execution. Our team is actively working on a fix as a high priority. We will continue to update you as we investigate.

9. identified May 21, 2026, 05:43 PM UTC

   We have identified the issue which affected buying power. Our team is actively working on a fix for buying power and resolving duplicate orders as a high priority. We will continue to update you as we investigate.

10. identified May 21, 2026, 05:53 PM UTC

    We have identified the issue which affected buying power. Our team is actively working on a fix for buying power and resolving duplicate orders as a high priority. We will continue to update you as we investigate.

11. identified May 21, 2026, 06:06 PM UTC

    We have identified the issue which affected buying power. Our team is actively working on a fix for buying power and resolving duplicate orders as a high priority. We will continue to update you as we investigate.

12. identified May 21, 2026, 06:17 PM UTC

    Our team is actively working on a fix for buying power and resolving duplicate orders as a high priority. We will continue to update you as we work towards a resolution.

13. identified May 21, 2026, 06:27 PM UTC

    Our team is actively working on a fix for buying power and resolving duplicate orders as a high priority. We will continue to update you as we work towards a resolution. We greatly appreciate your patience.

14. identified May 21, 2026, 06:41 PM UTC

    Our team is actively working on a fix for buying power and resolving duplicate orders as a high priority. We will continue to update you as we work towards a resolution. We greatly appreciate your patience.

15. monitoring May 21, 2026, 06:57 PM UTC

    We anticipate a fix to be complete shortly. We will continue to update you as we progress.

16. monitoring May 21, 2026, 07:15 PM UTC

    Our team is continuing to work on resolving this as quickly as possible. We will continue to update you as we progress. Thank you for your patience.

17. monitoring May 21, 2026, 07:39 PM UTC

    Our team team has implemented a fix and we are monitoring to ensure services are restored. We will continue to update you as we monitor. Thank you for your patience.

18. monitoring May 21, 2026, 08:30 PM UTC

    We are continuing to monitor to ensure the issue is resolved. We greatly appreciate your patience.

19. resolved May 21, 2026, 08:36 PM UTC

    All duplicate orders have been sorted and we are seeing accounts are restored to normal. We will consider this issue as resolved. Thank you for your patience and understanding.

20. postmortem May 22, 2026, 08:03 PM UTC

    On May 21, 2026, our platform experienced an issue where a subset of overnight fractional limit orders received duplicate executions. The issue affected orders placed between 8:00 PM ET on May 20, 2026 and approximately 8:58 AM ET on May 21, 2026. All affected orders were identified and corrected. ‌ Our engineering teams monitored the platform overnight process. The incident is now closed. ‌ We know reliable order execution is critical to your operations, and we thank you for your understanding as we restored services.

Read the full incident report →

Started May 19, 2026, 07:42 AM UTC· resolved May 19, 2026, 09:13 AM UTC

1. investigating May 19, 2026, 07:42 AM UTC

   We are investigating above issue. Updates will follow.

2. investigating May 19, 2026, 07:46 AM UTC

   Gobroker BOD is running longer than expected today , You should see correct Balances already , position data is delayed.

3. investigating May 19, 2026, 07:55 AM UTC

   End of day positions returned in /v1/reporting/eod/positions are delayed. Endpoint API Doc: https://docs.alpaca.markets/us/reference/get-v1-reporting-eod-positions

4. investigating May 19, 2026, 08:18 AM UTC

   Our team is actively monitoring the job and investigating the delay. We will provide another update as soon as we have more information. Thank you for your patience.

5. identified May 19, 2026, 08:43 AM UTC

   We are continuing to monitor the situation. The job is still in progress, and our team remains actively engaged. We will share an update as soon as there is further progress.

6. monitoring May 19, 2026, 09:04 AM UTC

   The position data processing job has now completed successfully. Position data should be fully up to date and reflected correctly. If you experience any discrepancies or have questions, please don't hesitate to reach out to us. Thank you for your patience.

7. resolved May 19, 2026, 09:13 AM UTC

   The position data processing job has completed successfully. This incident is now resolved. If you notice any remaining discrepancies, please don't hesitate to reach out. Thank you for your patience during this time

Read the full incident report →

Started May 16, 2026, 09:55 AM UTC· resolved May 16, 2026, 10:48 AM UTC

1. identified May 16, 2026, 09:55 AM UTC

   End Of Day (EOD) position snapshots and balances for 15-May-2026 are still being processed. The team has identified the issue, and the rerun is currently in progress.

2. identified May 16, 2026, 10:10 AM UTC

   EOD Balance processing has completed. We are syncing position snapshots and expect completion before 7:00 AM EDT.

3. resolved May 16, 2026, 10:48 AM UTC

   EOD balance and position batch processing has completed successfully.

Read the full incident report →

Started May 15, 2026, 05:14 AM UTC· resolved May 15, 2026, 06:30 AM UTC

1. investigating May 15, 2026, 05:14 AM UTC

   We have identified an issue where certain securities available for overnight trading are incorrectly marked as restricted. Impact Clients may be unable to place trades for specific newly listed overnight-tradable securities. Regular trading hours and other asset classes remain unaffected.

2. identified May 15, 2026, 05:26 AM UTC

   We identified a likely root cause. The team is restoring the correct configuration. Investigation is ongoing, and we will share another update once recovery is confirmed.

3. monitoring May 15, 2026, 06:02 AM UTC

   Mitigation measures have been implemented, and services are showing signs of recovery. We are currently validating the overnight trading symbol set and transitioning into a monitoring phase while final verification checks are completed.

4. resolved May 15, 2026, 06:30 AM UTC

   After monitoring we are no longer seeing issues. Overnight trading symbol eligibility appears to be operating normally.

5. postmortem May 15, 2026, 06:06 PM UTC

   On May 15, we identified an issue affecting overnight trading. Some securities that are generally eligible for overnight trading were temporarily ineligible for overnight trading, preventing clients from placing or modifying orders for those symbols during the overnight session. ‌ Our engineering team identified an issue resulting from receiving an erroneous source file form our execution venue and manually updated the system restoring overnight trading for the impacted symbols... Regular market-hours trading, other asset classes, account balances, positions, and prior transactions are not impacted. ‌ Thank you for your patience and understanding as we worked to restore services.

Read the full incident report →

Started May 12, 2026, 03:43 PM UTC· resolved May 13, 2026, 03:28 PM UTC

1. investigating May 12, 2026, 03:43 PM UTC

   We are currently investigating an issue causing slowness and queuing of journals. Our engineering team is actively working on resolving this.

2. identified May 12, 2026, 03:59 PM UTC

   We have identified the root cause and the engineering team is actively working on a mitigation plan. We will continue to provide updates as we progress.

3. identified May 12, 2026, 04:06 PM UTC

   We are continuing to work on a fix for this issue.

4. identified May 12, 2026, 04:18 PM UTC

   Mitigation Plans are being evaluated currently for any unknown impact,We will continue to provide updates as we progress.

5. identified May 12, 2026, 04:36 PM UTC

   We are seeing journals getting processed faster.

6. identified May 12, 2026, 04:56 PM UTC

   Currently we are not seeing journals getting stuck, they are getting processed without any issue.

7. monitoring May 12, 2026, 05:35 PM UTC

   We are continuing to monitor to ensure journals are processing as normal.

8. monitoring May 12, 2026, 05:38 PM UTC

   Journal processing seems to be normal at this moment. Our team is working on fix and being tested in lower environment.

9. monitoring May 12, 2026, 06:22 PM UTC

   Testing is still under progress. Journal processing seems to be normal at this moment.

10. monitoring May 12, 2026, 07:18 PM UTC

    Journal processing seems to be normal at this moment. We are continuing to monitor.

11. monitoring May 13, 2026, 08:34 AM UTC

    Remediation measures were applied yesterday and journals have been flowing correctly since then. We will monitor today's market open to validate the changes and address any unforeseen issues. A final confirmation will follow, after which we will close this incident. Thank you for your patience.

12. resolved May 13, 2026, 03:28 PM UTC

    Journals have been processing normally through today's market open with no further issues observed. We consider this incident fully resolved and are closing it out. Thank you for your patience.

13. postmortem May 14, 2026, 04:26 PM UTC

    On May 12, 2026, our platform experienced a processing delay affecting journal transfers. The issue originated from an unexpected surge in volume. However, no transactions were lost, rejected, or duplicated and there was no impact to funds or positions. Our engineering teams deployed a fix and continued to monitor the situation for 24 hours post deployment. There have been no issues since and the incident is now closed. We know timely processing is critical to your operations, and we thank you for your understanding as we restored services..

Read the full incident report →

Started May 12, 2026, 04:21 AM UTC· resolved May 12, 2026, 07:01 AM UTC

1. investigating May 12, 2026, 04:21 AM UTC

   We are investigating above issue. Updates will follow.

Read the full incident report →

Started May 05, 2026, 06:36 AM UTC· resolved May 05, 2026, 12:20 PM UTC

1. identified May 05, 2026, 06:36 AM UTC

   We are currently experiencing a delay in our Beginning of Day (BOD) processing. As a result, settled cash and downstream updates, such as EOD positions, are expected to be available closer to 6:00 AM ET today (typically available around 4:00 AM ET). We are actively monitoring the process and will provide update once processing is complete.

2. identified May 05, 2026, 07:08 AM UTC

   We've identified a long-running query as the source of this issue. Our engineering team is actively working on a resolution, and we'll provide an update once the fix is in place.

3. monitoring May 05, 2026, 07:31 AM UTC

   We have identified and addressed the underlying cause of the issue. A corrective fix has been applied and the affected process has been restarted. Our team continues to monitor the situation, and we will provide further updates as needed.

4. monitoring May 05, 2026, 07:54 AM UTC

   We are continuing to investigate the EOD delay. The affected process has been rerun, and our team is closely monitoring logs and system metrics to assess the effectiveness of the applied fix. We will share an update once we have more clarity.

5. monitoring May 05, 2026, 08:07 AM UTC

   EOD Job is still running and we are monitoring the logs for any issue.

6. monitoring May 05, 2026, 08:28 AM UTC

   A critical component of the end-of-day processing has completed successfully. The remaining processes are in progress and advancing as expected. We will continue to provide updates as the situation develops.

7. monitoring May 05, 2026, 08:50 AM UTC

   We are continuously monitoring the final phase of the end-of-day processing and will provide an update upon completion.

8. monitoring May 05, 2026, 08:58 AM UTC

   All phases of end-of-day processing have been completed successfully. Beginning-of-day processing is now underway, and we expect settled cash balances to be updated by 6:30 AM EDT. We will provide a further update once this is confirmed.

9. monitoring May 05, 2026, 09:48 AM UTC

   We have encountered an issue during the beginning-of-day processing and are currently working with our engineering team to investigate. We will provide an update shortly.

10. monitoring May 05, 2026, 10:04 AM UTC

    we are still investigating this BOD job failure.

11. monitoring May 05, 2026, 10:24 AM UTC

    We have taken corrective measures and re-run the B.O.D. job. We are continuing to monitor for any further issues.

12. monitoring May 05, 2026, 10:46 AM UTC

    B.O.D. job has resumed and is progressing without issues. Our team continues to monitor closely and will share an update once complete.

13. monitoring May 05, 2026, 11:12 AM UTC

    Following the applied fix, the beginning-of-day job has progressed past the previously failed stage and is now running through the remaining phases. We will continue to monitor and provide updates as processing progresses.

14. monitoring May 05, 2026, 11:24 AM UTC

    The Beginning-of-day job is running smoothly and we expect it to complete by 8:00 AM EDT. We will confirm once processing is finished.

15. monitoring May 05, 2026, 11:58 AM UTC

    Settled cash has been updated across all accounts. Position data is expected to be updated within the next 15 minutes.

16. resolved May 05, 2026, 12:20 PM UTC

    All beginning-of-day processing, including position updates, has been completed successfully.We are marking this incident resolved here now.

17. postmortem May 06, 2026, 07:32 PM UTC

    On the night of May 4 our overnight batch processing cycle was delayed due to issues affecting 2 critical jobs. As a result, settled cash availability was four hours later than their normal window. We deployed two code fixes mitigating the incident at 4:58 AM ET and fully resolving the issues at 9:55 AM ET. No impact to live trading, order execution, or market connectivity occurred, and no customer funds, positions, or orders were lost or misreported. We understand that delays in file processing can impact your workflows and downstream processes, and we appreciate your patience and understanding as our teams worked to identify, mitigate, and fully resolve the issue as quickly as possible.

Read the full incident report →

Started Apr 27, 2026, 12:13 PM UTC· resolved Apr 27, 2026, 07:54 PM UTC

1. investigating Apr 27, 2026, 12:13 PM UTC

   We are currently experiencing issues with SMS two-factor authentication for the Trading Dashboard. If you cannot log in, please contact support. This does not affect our APIs or Broker Dashboard.

2. identified Apr 27, 2026, 12:14 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Apr 27, 2026, 07:20 PM UTC

   We believe the issue with SMS based two-factor authentication for the Trading Dashboard is resolved. We are currently monitoring the service to validate.

4. resolved Apr 27, 2026, 07:54 PM UTC

   Login for Trading Dashboard is working normally again..

Read the full incident report →

Started Apr 07, 2026, 02:45 AM UTC· resolved Apr 07, 2026, 03:30 AM UTC

1. investigating Apr 07, 2026, 02:45 AM UTC

   Just In Time reports are not generated

2. investigating Apr 07, 2026, 03:09 AM UTC

   Team is still working to identify the root cause

3. identified Apr 07, 2026, 03:22 AM UTC

   We have identified the underlying cause. We are working on the recovery.

4. resolved Apr 07, 2026, 03:30 AM UTC

   We have fixed the issue and reports are now available

Read the full incident report →

Started Mar 31, 2026, 01:25 PM UTC· resolved Mar 31, 2026, 02:10 PM UTC

1. investigating Mar 31, 2026, 01:25 PM UTC

   We are currently investigating a service degradation that is impacting account information access and transaction processing. Client Impact: Users may experience intermittent timeouts or delays when attempting to access their account details. Current Status: Our engineering teams are actively engaged and working to address the underlying issues related to processing capacity and system delays. We are prioritising a resolution and will provide an update when more information is available.

2. identified Mar 31, 2026, 01:34 PM UTC

   We are seeing improvement in API response times, and journals are no longer getting stuck. We are continuing to monitor the situation and will post further updates as needed.

3. monitoring Mar 31, 2026, 01:43 PM UTC

   The service has been restored to normal operation. We are no longer seeing issues with any API endpoints or journals. We will continue monitoring to ensure stability.

4. monitoring Mar 31, 2026, 01:56 PM UTC

   All services are operating normally with no issues detected. We are continuing to monitor and will provide updates if anything changes.

5. resolved Mar 31, 2026, 02:10 PM UTC

   The issue has been fully resolved and we are marking this incident as closed. If you observe any further issues, please don't hesitate to reach out to us.

Read the full incident report →

Started Mar 23, 2026, 07:45 PM UTC· resolved Mar 23, 2026, 08:10 PM UTC

1. investigating Mar 23, 2026, 07:45 PM UTC

   We are seeing issue with journal processing due to redpanda.

2. identified Mar 23, 2026, 07:48 PM UTC

   We have identified an issue with redpanda cluster and we are in the process of restart.

3. monitoring Mar 23, 2026, 07:55 PM UTC

   The cluster is up and journals are getting processed

4. resolved Mar 23, 2026, 08:10 PM UTC

   Journal processing are back to normal

Read the full incident report →

Started Mar 17, 2026, 04:03 PM UTC· resolved Mar 17, 2026, 04:17 PM UTC

1. investigating Mar 17, 2026, 04:03 PM UTC

   We are investigating above issue. Updates will follow.

2. resolved Mar 17, 2026, 04:17 PM UTC

   Website is back up and there was no impact.

Read the full incident report →

Started Mar 12, 2026, 04:54 PM UTC· resolved Mar 12, 2026, 08:18 PM UTC

1. investigating Mar 12, 2026, 04:54 PM UTC

   We are currently seeing intermittent issues with orders being filled. Our team is actively working on the issue with full attention.

2. investigating Mar 12, 2026, 04:55 PM UTC

   We are continuing to investigate this issue.

3. identified Mar 12, 2026, 05:05 PM UTC

   We are still looking into the issue ,will provide next update shortly.

4. identified Mar 12, 2026, 05:13 PM UTC

   This is due to a crash in our main messaging system (RabbitMQ). We have restarted the messaging system and are now restarting the associated programs to re-establish stable connections. We are actively monitoring the system to ensure it is back to normal as quickly as possible.

5. identified Mar 12, 2026, 05:25 PM UTC

   While we were able to restart the main messaging infrastructure, we are currently managing some recurring connection instability and have identified a backlog of orders that have not yet been processed. Our engineering team is fully engaged and focused on clearing this backlog and addressing the underlying causes of the intermittent system disruptions to ensure a complete and stable resolution. We will provide another update once the system has fully stabilised.

6. identified Mar 12, 2026, 05:38 PM UTC

   We are currently experiencing intermittent delays in order processing due to a system connectivity issue. Our team is actively working to resolve this. Here's what's happening: 1.Orders that are stuck in "pending" status are being manually processed and re-routed. 2.For any orders pending cancellation, we are coordinating directly with our trading partners to ensure they are properly canceled. 3.Crypto order processing is being restored as part of our ongoing recovery efforts. Our engineering team has identified the underlying cause and is working on a permanent fix. We will share another update once the issue is fully resolved. We apologize for the inconvenience and appreciate your patience.

7. identified Mar 12, 2026, 05:48 PM UTC

   The root cause of the instability has been traced to a specific node within our messaging cluster. A targeted restart of that component is being prepared and will be executed shortly. In the meantime, the order backlog is actively being cleared , pending orders are decreasing and filled orders are increasing as manual re-routing continues. The crypto exchange component restart has been completed successfully. The next step to restore full system stability is the execution of this remaining component restart. We will confirm once it is complete.

8. identified Mar 12, 2026, 06:01 PM UTC

   After restart of problematic component in messaging cluster The messaging system has been stabilised and key trading services are back online and operating normally.A final reconnection step for our crypto trading service is currently being completed to ensure all systems are fully restored. We expect full recovery shortly and will confirm once everything is back to normal. Thank you for your patience.

9. identified Mar 12, 2026, 06:13 PM UTC

   The majority of services have been restored. Our team has identified one remaining synchronisation issue within the messaging infrastructure and is applying a targeted fix. As a precaution, affected services have been temporarily scaled down during the repair. A small number of orders that did not reach their intended venues are being re-processed. We will confirm resolution once this final fix is verified.

10. identified Mar 12, 2026, 06:26 PM UTC

    The fix for the primary affected queue has been successfully applied , messages are flowing and the backlog is cleared. During this process, two additional queues were found to have similar synchronisation issues. Our team is now applying the same fix to these remaining components to restore full stability. In parallel, we are evaluating longer-term improvements to our messaging infrastructure to prevent recurrence. We will provide an update once all components are fully restored.

11. identified Mar 12, 2026, 06:37 PM UTC

    We are still working on the issue and monitoring system closely. We will write next update soon.

12. identified Mar 12, 2026, 06:56 PM UTC

    Our messaging system is operating normally as of now. Our team is now focused on cleaning up the remaining orders that were impacted during the disruption. We appreciate your continued patience.

13. identified Mar 12, 2026, 07:10 PM UTC

    We are still doing cleanup of backlog of all pending cancel and pending replace orders.

14. identified Mar 12, 2026, 07:22 PM UTC

    The current phase is operational cleanup and client impact mitigation: normal order flow is working, and the focus is on systematically clearing pending replace/pending cancel states (including crypto) while continuing elevated monitoring.

15. identified Mar 12, 2026, 07:40 PM UTC

    Operational cleanup of pending_cancel is still going on.

16. resolved Mar 12, 2026, 08:18 PM UTC

    This incident has been resolved.

17. postmortem Mar 13, 2026, 04:40 PM UTC

    **Impact:** On March 12, 2026, our trading platform experienced intermittent connectivity to multiple trading venues. Consequently, orders were either not successfully delivered to venues or remained in a pending status. **Incident overview and remediation:** Our investigation found instability within our core queueing system and a crash-loop in our order consumer components within routing service, causing cascading latency across the platform. To resolve the issue, we restarted the queueing system, restoring connectivity and clearing the order backlog. Any remaining "stuck" orders were manually reconciled by our operations team. **Next steps** * We have extended this weekend's maintenance window \(March 14–15, 2026\) to two hours to upgrade system capacity, restart key services for stability, and apply critical infrastructure updates both internally and across one of our third party vendors. * We’re validating our remediation plan against the root cause to prevent recurrence. * We are maintaining heightened surveillance of platform stability and will provide updates via our status page. If you have questions or believe your orders were impacted, please contact your account manager or our support team through your usual support channel.

Read the full incident report →

Started Mar 09, 2026, 03:23 AM UTC· resolved Mar 09, 2026, 04:15 AM UTC

1. investigating Mar 09, 2026, 03:23 AM UTC

   We are currently investigating this issue.

2. identified Mar 09, 2026, 03:30 AM UTC

   We have identified the issue with one of the infrastructure component causing replication to fail. Team is currently working on it.

3. identified Mar 09, 2026, 03:33 AM UTC

   Team have fixed the underlying root cause and new journals are getting executed. We are working on the backfill.

4. identified Mar 09, 2026, 03:34 AM UTC

   We are continuing to work on a fix for this issue.

5. identified Mar 09, 2026, 03:47 AM UTC

   Team is still working on the backfill

6. identified Mar 09, 2026, 04:00 AM UTC

   Team is still working on backfilling the journals

7. identified Mar 09, 2026, 04:13 AM UTC

   Journals backfilled completed.

8. monitoring Mar 09, 2026, 04:13 AM UTC

   A fix has been implemented and we are monitoring the results.

9. resolved Mar 09, 2026, 04:15 AM UTC

   This incident has been resolved.

10. postmortem Mar 11, 2026, 02:55 PM UTC

    ## **Service Status Update: Transaction Processing Delay \(March 8-9, 2026\)** ## **Root Cause** On the evening of March 8, 2026, our platform experienced a disruption in the processing of journal entries \(internal account transfers\). The issue was triggered by a high-volume internal data job related to annual tax reporting. This specific job generated a very large volume of database writes, which caused WAL \(Write-Ahead Log\) growth to reach the configured limit for replication slots. Once this threshold was reached, our database marked the affected logical replication slots as lost, which interrupted our Change Data Capture \(CDC\) pipeline which is needed for advancing journal processing from “sent to clearing” to “executed. ## **Impact** * **Data Integrity:** We can confirm that **no data was lost**, and all funds remain fully accounted for. The issue was limited strictly to a delay in processing time, not the accuracy of the transactions. * **Affected Partners:** The delay impacted 31 correspondent firms, with the majority of the volume concentrated in specific high-activity accounts. * **Current Status:** All backlogged transactions were successfully processed and finalized by 11.10pm ET on 8th March ## **Resolution** Once the delay was identified, our engineering team took the following immediate actions: * **Terminated the Data Job:** The heavy internal tax processing task was halted to relieve pressure on the database. * **Recreated Replication Slots:** The affected logical replication slots were recreated. * **Backfilled CDC Data:** Engineers performed a **backfill** to restore the replication pipeline and synchronize the missing changes ‌ ## **Commitment to Reliability** To prevent a recurrence, we are implementing the following safeguards: 1. **Job Optimization:** Future large-scale data tasks \(such as tax reporting\) will be broken into smaller batches to prevent system saturation. 2. **Increased Capacity:** We are reviewing our synchronization storage limits to provide a larger buffer during peak periods of internal data activity. 3. **Enhanced Monitoring:** We have updated our internal playbooks to ensure faster automated alerting if data synchronization lags in the future. We apologize for any inconvenience this delay may have caused to your operations.

Read the full incident report →

Started Feb 09, 2026, 05:01 PM UTC· resolved Feb 09, 2026, 05:36 PM UTC

1. investigating Feb 09, 2026, 05:01 PM UTC

   We are currently investigating this issue.

2. investigating Feb 09, 2026, 05:16 PM UTC

   We are continuing to investigate this issue.

3. resolved Feb 09, 2026, 05:36 PM UTC

   This issue has been resolved.

Read the full incident report →

Started Feb 09, 2026, 03:51 PM UTC· resolved Feb 09, 2026, 04:04 PM UTC

1. investigating Feb 09, 2026, 03:51 PM UTC

   INDW reported issues

2. resolved Feb 09, 2026, 04:04 PM UTC

   Issue is resolved

Read the full incident report →

Started Feb 06, 2026, 03:37 PM UTC· resolved Feb 06, 2026, 04:46 PM UTC

1. investigating Feb 06, 2026, 03:37 PM UTC

   investigating issue

2. investigating Feb 06, 2026, 03:42 PM UTC

   We are currently seeing time outs in our accounts_ID endpoint. Our team is currently looking into the issue

3. identified Feb 06, 2026, 03:48 PM UTC

   We identified the issue causing the time outs and addressing it

4. identified Feb 06, 2026, 04:02 PM UTC

   We are implementing a fix now

5. monitoring Feb 06, 2026, 04:24 PM UTC

   a fix has been implemented and we are continuing to monitor

6. resolved Feb 06, 2026, 04:46 PM UTC

   The endpoint is now stabilized and we are not seeing and more errors

Read the full incident report →

Started Feb 04, 2026, 07:01 PM UTC· resolved Feb 04, 2026, 07:59 PM UTC

1. investigating Feb 04, 2026, 07:01 PM UTC

   We are currently investigating this issue.

2. investigating Feb 04, 2026, 07:01 PM UTC

   We are continuing to investigate this issue.

3. investigating Feb 04, 2026, 07:06 PM UTC

   We are continuing to investigate this issue.

4. identified Feb 04, 2026, 07:24 PM UTC

   There is an invalid message that is pushed to the queue causing the service crash.

5. identified Feb 04, 2026, 07:30 PM UTC

   Invalid messages are removed from the queue. We are monitoring the queue and service

6. monitoring Feb 04, 2026, 07:30 PM UTC

   A fix has been implemented and we are monitoring the results.

7. resolved Feb 04, 2026, 07:59 PM UTC

   The issue is fixed and service is back to normal

Read the full incident report →

Started Jan 30, 2026, 05:05 PM UTC· resolved Jan 30, 2026, 05:09 PM UTC

1. investigating Jan 30, 2026, 05:05 PM UTC

   We are running a planned automation test to validate incident workflows. No services are impacted and no customer action is required.

2. investigating Jan 30, 2026, 05:08 PM UTC

   Closing the test incident.

3. resolved Jan 30, 2026, 05:09 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Jan 23, 2026, 02:12 PM UTC· resolved Jan 23, 2026, 02:49 PM UTC

1. investigating Jan 23, 2026, 02:12 PM UTC

   We are currently experiencing increased response times due to elevated database connection usage. Our engineering team is actively investigating the issue, and we will keep you posted with updates here.

2. identified Jan 23, 2026, 02:21 PM UTC

   Our engineering team has identified a few overloaded backend components and has restarted them and cleared a blocking database process to relieve pressure.

3. monitoring Jan 23, 2026, 02:25 PM UTC

   After clearing the blocking database process, Overall response times are improving and moving back toward what we normally see.

4. monitoring Jan 23, 2026, 02:33 PM UTC

   We are continuously monitoring and things are normal as of now.

5. resolved Jan 23, 2026, 02:49 PM UTC

   As everything is fine now , Marking this incident resolved here.

Read the full incident report →

Started Jan 20, 2026, 09:20 AM UTC· resolved Jan 20, 2026, 10:26 AM UTC

1. investigating Jan 20, 2026, 09:20 AM UTC

   API response for orders and JNLC were delayed intermittently

2. investigating Jan 20, 2026, 09:32 AM UTC

   Accounts are taking longer than expected to fully load, which is occasionally causing some users to see an error (a timeout) when they try to check their journals or orders. We are actively working to quickly finish loading these accounts and stabilize the system.

3. identified Jan 20, 2026, 09:43 AM UTC

   Core databases and replication appear healthy; impact is limited to specific journal /account operations and is intermittent. Engineering is actively working to stabilize the affected service and reduce latency; we will provide another update as we learn more.

4. investigating Jan 20, 2026, 09:46 AM UTC

   Our team is identified issues with one of the service being stale to refresh data and we are working on it.

5. monitoring Jan 20, 2026, 09:51 AM UTC

   Error rates and latency are improving after remedial actions, we are continuously monitoring for any issue.

6. monitoring Jan 20, 2026, 10:01 AM UTC

   overall performance has improved for all affected endpoints. We will keep monitoring closely before marking this incident as resolved.

7. monitoring Jan 20, 2026, 10:11 AM UTC

   After We have restarted the affected backend services then pending account/journal work has cleared. Error rates and timeouts are now very low, and requests are succeeding again. We are continuing to monitor closely for any other issue.

8. monitoring Jan 20, 2026, 10:20 AM UTC

   Errors are very low and limited to brief blips during restarts; we will keep monitoring closely before marking this incident as resolved.

9. resolved Jan 20, 2026, 10:26 AM UTC

   We are no longer observing any issues across our APIs. This incident is now marked as resolved. A root cause analysis will be published by tomorrow..

10. postmortem Jan 20, 2026, 09:41 PM UTC

    # **Service update — Order processing & account sync** **January 20, 2026** ‌ **What happened \(short\):**A background database process created by a price-update job held an open transaction for an extended period. That blocked the database from cleaning up a very busy internal table, causing worker backlogs. As a result some orders and account updates were slower or intermittently timed out. **Impact:** * Slower order responses and occasional timeouts for a subset of customers. * Delays in account balance refreshes and transaction visibility. * No data was lost — all funds, transactions and records remained secure. **What we did:** * Stopped the offending background work, killed the long-running transaction, and manually cleaned the affected table. * Performed a rolling restart of worker and order-processing services to clear stuck processes. * Restored normal processing and cleared the backlog. Services were healthy again later that morning \(all backlog processed\). ‌ **Why this won’t happen again \(actions\):** **Immediate** * Added transaction timeouts for database access at the application level. **Short/medium term** * Paused a non-critical job which had resulted in resource contentions; this will be re-enabled only once this issue is resolved. * Improve alerting/handling for long-open transactions and failed database cleanups so we can act earlier. **Long term** * Review and tune database maintenance settings so write-heavy tables are protected. **Our commitment:**We apologize for the disruption. We’ve taken steps to fix the immediate problem and are implementing monitoring and architectural changes to reduce the chance of recurrence. If you’d like additional detail or a technical appendix, we’re happy to share it.

Read the full incident report →

Started Jan 12, 2026, 02:58 PM UTC· resolved Jan 12, 2026, 06:50 PM UTC

1. investigating Jan 12, 2026, 02:58 PM UTC

   We are currently investigating elevated error rates on multiple API. Some requests may fail with HTTP 500 responses. Our engineering team is actively working to identify and resolve the root cause.

2. identified Jan 12, 2026, 03:08 PM UTC

   Our team is actively addressing the issue, clients may still experience intermittent timeouts on order placement while we fix the issue.

3. monitoring Jan 12, 2026, 03:14 PM UTC

   The team identified and addressed the underlying issue. We are seeing our API's recovering. The team is continuing to monitor performance. We will continue to provide updates

4. monitoring Jan 12, 2026, 03:20 PM UTC

   We are no longer observing 500 errors across our API endpoints. All services are now accessible, and we are continuing to monitor for any further issues.

5. monitoring Jan 12, 2026, 03:33 PM UTC

   We have identified another increase in 5xx errors. We are actively addressing and will provide more updates. clients may still experience intermittent timeouts on order placement while we fix the issue.

6. monitoring Jan 12, 2026, 03:50 PM UTC

   We are continuing to address the issue. Clients may still experience intermittent 5xx errors

7. monitoring Jan 12, 2026, 04:02 PM UTC

   We are continuing to address the issue. Clients may still experience intermittent 5xx errors.

8. monitoring Jan 12, 2026, 04:22 PM UTC

   The system has largely stabilised and the majority of requests are completing successfully. We are still observing intermittent timeouts (~3-5%) affecting position updates. Our engineering team is actively investigating the root cause related to high-volume data requests and working toward a full resolution. We will continue to provide updates as we make progress.

9. monitoring Jan 12, 2026, 04:45 PM UTC

   We continue to see intermittent timeouts on an order-handling service, occasionally impacting orders and account/position lookups—though most requests are completing successfully. Our engineers are actively isolating affected pods, capturing diagnostic data, and restarting them to restore stability. We are also investigating database behavior and traffic patterns to identify the underlying root cause.

10. monitoring Jan 12, 2026, 04:55 PM UTC

    Our investigation is progressing. We have identified several contributing factors and are actively analyzing traffic patterns, system behavior, and recent changes to determine the root cause. The team is working diligently toward a full resolution and will continue to provide updates.

11. monitoring Jan 12, 2026, 05:08 PM UTC

    Our investigation continues to make progress. We have analysed system behavior during the affected periods and have narrowed our focus to connection management, which we believe may be contributing to the intermittent issues. We are encouraged that most impacted requests are ultimately completing successfully. The team remains fully engaged and is working toward a resolution. We appreciate your patience and will keep you informed as we learn more.

12. monitoring Jan 12, 2026, 05:20 PM UTC

    All API endpoints are operating normally. We are continuing to investigate the root cause and will share findings once available. Thank you for your patience.

13. monitoring Jan 12, 2026, 05:35 PM UTC

    we are still monitoring the system.

14. monitoring Jan 12, 2026, 05:48 PM UTC

    The system has stabilized and no issues are currently being observed. Our team continues to work on identifying the root cause and will provide an update once we have more information.

15. monitoring Jan 12, 2026, 06:05 PM UTC

    We have not observed any abnormalities since the system stabilised. We will close this incident after 45 minutes. Root cause analysis will continue, and findings will be shared separately. Thank you for your patience.

16. resolved Jan 12, 2026, 06:50 PM UTC

    Our team has mitigated the incident. All systems are operational and customer impact has ceased. We are continuing our root cause analysis to prevent recurrence.

17. postmortem Jan 12, 2026, 10:12 PM UTC

    # January 13th update **Follow-up: Resolution of Monday Market Open Incident** As a follow-up to our communication regarding Monday’s service instability, we are providing a summary of our findings and the corrective actions taken. **Root Cause Analysis** Over the weekend, a planned change was implemented which included the rollout of Istio into our production network. Following this deployment, we observed intermittent connectivity issues that resulted in the instability seen on Monday. Our investigation confirmed that the Istio layer was not stable in establishing connections between services and components over the network. This issue was exacerbated during the market open, where the high traffic footprint led to significant latency and slowness. While this configuration had been present in our staging environments for some time, the issue only manifested in production due to the unique load requirements of the live market open. **System Impact** We specifically investigated why a connection and memory issue within specific pods impacted critical trading functions. The analysis showed that the connection instability caused an excessive backlog of concurrent queries. This led to a significant memory spike that exceeded typical thresholds, creating a cascading effect on service responsiveness during peak traffic. **Remediation** To address this, we have removed the Istio plane across all critical services. All impacted services were restarted following this change to ensure a clean state. **Current Status** Since the removal of the Istio layer, system performance has returned to its baseline and connections remain stable. We are continuing to monitor the environment closely and are utilizing enhanced load testing to ensure our infrastructure remains resilient during peak traffic. -------------------------------------------------------------------------- ### **What Happened** Shortly after market open on January 12, 2026 \(approx. 9:36 AM EST\), our monitoring systems detected a significant degradation in performance across our core APIs. This resulted in elevated error rates and latency for incoming API requests. Our engineering team identified that the issue was caused by resource contention within our system. A combination of high market-open traffic and underlying system abnormalities triggered a significant memory spike in a single pod. This exhaustion of resources caused internal services to become unresponsive, resulting in connection timeouts and creating a bottleneck between our API gateway and the database services. ### **Impact** We understand that reliability is paramount for your operations. Below is a summary of the impact observed during the incident window \(9:36 AM – 11:20 AM EST\): * **API Availability:** Partners experienced intermittent `500` and `504` error responses on Order, Account, and Position endpoints. * **Order Processing:** A subset of orders experienced processing delays. In some cases, orders that timed out on the API response were successfully processed in the background. * **Data Latency:** There were short delays in position updates and trade confirmation events \(SSE\) for executed orders. * **Critical Data Integrity:** **No data was lost during this incident.** All funds and positions remain safe and secure. All transactions that appeared to time out but were executed have been reconciled. ### **Resolution** Our team executed a series of mitigation strategies to restore stability. Immediate action was taken to isolate and restart the affected service instances to clear the connection backlog. We deployed a hotfix to eliminate redundant metrics processing, which reduces unnecessary overhead and helps lower overall memory consumption. The system was fully stabilized by 11:20 AM EST. We have confirmed that error rates have returned to nominal levels and all backlog queues have been processed. We will continue to investigate any abnormalities. ### **Preventative Measures** We are committed to learning from every incident to strengthen our platform. We are prioritizing the following actions: * **Hands-on Monitoring:** We will have our team monitor the market opens all this week to ensure quick intervention. * **System Capacity Review:** We are auditing our resource allocation thresholds to ensure our services can handle "perfect storm" scenarios where high volume coincides with complex queries. * **Deployment Process Optimization:** We are revising our release procedures to ensure that non-critical background processes \(such as metrics collection\) cannot impact core transaction performance during peak market hours. * **Enhanced Monitoring:** We are implementing stricter alerts regarding database connection locking to detect and auto-remediate similar contention issues faster in the future.

Read the full incident report →

Started Jan 06, 2026, 08:56 AM UTC· resolved Jan 06, 2026, 11:59 AM UTC

1. investigating Jan 06, 2026, 08:56 AM UTC

   The BOD process delay is caused by a GoBroker BOD pod being stuck in a Pending state due to resource constraints. This is not a system outage. The engineering team is actively investigating the underlying resource and node scheduling issue.

2. investigating Jan 06, 2026, 09:17 AM UTC

   BOD process is running: The resource constraint issues have been resolved, and the Beginning-of-Day (BOD) process is now running. Estimated time: The estimated completion time is 2 hours from now

3. identified Jan 06, 2026, 09:18 AM UTC

   We have identified the issue with resources on the node and it has been rectified. BOD process is running. ETA is 90 Mins by now.

4. monitoring Jan 06, 2026, 10:11 AM UTC

   B.O.D. Job failed due to timeout , Job is rerunning . We will keep you posted for updates.

5. monitoring Jan 06, 2026, 10:25 AM UTC

   Job is running without any issue as of now, we are actively monitoring and will keep posting status here.

6. monitoring Jan 06, 2026, 10:48 AM UTC

   The rerun has been progressing without any errors since it was initiated. Our team is actively monitoring the job, and we'll promptly share any further updates as they become available.

7. monitoring Jan 06, 2026, 11:25 AM UTC

   We are still monitoring the job ,will keep you posted here.

8. monitoring Jan 06, 2026, 11:28 AM UTC

   All balances have now been successfully and accurately updated.

9. monitoring Jan 06, 2026, 11:31 AM UTC

   We are continuing to monitor for any further issues.

10. monitoring Jan 06, 2026, 11:50 AM UTC

    The BOD job has now completed successfully, and all account data (balances, positions, activities ) should be up to date.

11. resolved Jan 06, 2026, 11:59 AM UTC

    Since we are no longer seeing issue ,Marking this resolved.

12. postmortem Jan 07, 2026, 04:55 PM UTC

    ## **Service Status Update: Account Data Processing Delay \(January 6, 2026\)** ### **What Happened** On the morning of January 6, 2026, our platform experienced a delay in the standard start-of-day data synchronisation process. This was caused by a temporary resource constraint within our processing environment, which prevented the internal jobs responsible for updating account balances and positions from initialising at their scheduled time. Upon identifying the capacity issue, our engineering teams manually adjusted system resources to allow the processing to begin. However, the initial recovery attempt encountered a synchronisation timeout, requiring a secondary re-run of the update job to ensure complete data accuracy. The process was successfully completed, and all systems returned to a healthy state by 06:44 AM ET. ### **Impact** The delay primarily affected the visibility of up-to-date account information for our partners and their end-users. * **Data Accuracy:** Users may have seen delayed updates to their account balances, holdings, and recent transaction history. * **Transaction Processing:** Some users may have encountered "insufficient balance" errors during fund transfer attempts, as the system was awaiting the finalised balance updates from the previous business day. * **Data Integrity:** We can confirm that **no data was lost** and **all funds remain entirely secure**. This was a processing delay only, and all account activities have since been reconciled correctly. ### **Resolution** Our technical team took the following immediate actions to resolve the incident: * **Capacity Expansion:** At 04:10 AM ET, we manually increased the available system capacity to bypass the resource bottleneck and allow the pending tasks to execute. * **Job Restoration:** After the initial job timed out, we initiated a controlled re-run of the entire start-of-day sequence to ensure every account was accurately updated. * **Final Verification:** We conducted a full audit of account balances and positions to confirm that all data was synchronized and the service was fully restored by 06:44 AM ET. ### **Preventative Measures** We are committed to maintaining the highest levels of reliability for our partners. To prevent a recurrence, we are implementing the following improvements: * **System Capacity Review:** We are auditing our resource allocation policies to ensure that core financial processing tasks always have priority and cannot be delayed by auxiliary or non-critical system activities. * **Resilience & Monitoring:** We are enhancing our automated alerting for "pending" internal tasks to identify and resolve resource constraints before they impact the start of the business day. **Process Optimization:** Our engineering team is optimizing the timeout thresholds and retry logic for long-running data jobs to ensure more resilient performance during peak processing windows.

Read the full incident report →

Started Jan 01, 2026, 11:59 AM UTC· resolved Jan 01, 2026, 12:10 PM UTC

1. monitoring Jan 01, 2026, 11:59 AM UTC

   An issue occurred during the Beginning-of-day process, resulting in a delay of approximately two hours in balance updates. Balances are now being updated. As a result, there was a delay in updating cash withdrawable, settled cash, and trade cash. Some accounts may continue to display stale or inaccurate balances until processing is complete.

2. monitoring Jan 01, 2026, 12:00 PM UTC

   We are continuing to monitor for any further issues.

3. monitoring Jan 01, 2026, 12:09 PM UTC

   All balances have now been successfully and accurately updated.

4. resolved Jan 01, 2026, 12:10 PM UTC

   We are marking this resolved as all the balances have been updated.

Read the full incident report →