postmortem Mar 11, 2026, 02:55 PM UTC
## **Service Status Update: Transaction Processing Delay \(March 8-9, 2026\)** ## **Root Cause** On the evening of March 8, 2026, our platform experienced a disruption in the processing of journal entries \(internal account transfers\). The issue was triggered by a high-volume internal data job related to annual tax reporting. This specific job generated a very large volume of database writes, which caused WAL \(Write-Ahead Log\) growth to reach the configured limit for replication slots. Once this threshold was reached, our database marked the affected logical replication slots as lost, which interrupted our Change Data Capture \(CDC\) pipeline which is needed for advancing journal processing from “sent to clearing” to “executed. ## **Impact** * **Data Integrity:** We can confirm that **no data was lost**, and all funds remain fully accounted for. The issue was limited strictly to a delay in processing time, not the accuracy of the transactions. * **Affected Partners:** The delay impacted 31 correspondent firms, with the majority of the volume concentrated in specific high-activity accounts. * **Current Status:** All backlogged transactions were successfully processed and finalized by 11.10pm ET on 8th March ## **Resolution** Once the delay was identified, our engineering team took the following immediate actions: * **Terminated the Data Job:** The heavy internal tax processing task was halted to relieve pressure on the database. * **Recreated Replication Slots:** The affected logical replication slots were recreated. * **Backfilled CDC Data:** Engineers performed a **backfill** to restore the replication pipeline and synchronize the missing changes ## **Commitment to Reliability** To prevent a recurrence, we are implementing the following safeguards: 1. **Job Optimization:** Future large-scale data tasks \(such as tax reporting\) will be broken into smaller batches to prevent system saturation. 2. **Increased Capacity:** We are reviewing our synchronization storage limits to provide a larger buffer during peak periods of internal data activity. 3. **Enhanced Monitoring:** We have updated our internal playbooks to ensure faster automated alerting if data synchronization lags in the future. We apologize for any inconvenience this delay may have caused to your operations.
