URGENT: LoanPro Service Interruption due to AWS Regional Outage

Update timeline

1. investigating Oct 20, 2025, 07:30 AM UTC

   LoanPro in the US is currently experiencing a service interruption. The root cause has been confirmed as a significant AWS (Amazon Web Services) regional outage that is impacting a wide range of services and companies globally. Due to this external event, all access to the LoanPro platform and associated API connections are currently affected. Our operations team is fully engaged and treating this as our highest priority. We are actively monitoring the situation in real-time, working with the AWS status updates. You can monitor AWS Status as well here (https://health.aws.amazon.com/health/status) We understand the critical nature of this disruption to your business, and we sincerely apologize for the inconvenience. We will provide our next status update via email and on our official status page within the next 30 minutes. Thank you for your patience and understanding as we work to restore full service.

2. identified Oct 20, 2025, 08:05 AM UTC

   UPDATE: LoanPro Service Interruption (AWS US-EAST-1 Outage) We have confirmed this is part of a widespread AWS outage in the US-EAST-1 (N. Virginia) region. AWS is reporting "Increased Error Rates and Latencies" for multiple services that are critical to LoanPro's operation. The AWS team is actively working on a resolution. In parallel, our own engineering team is closely monitoring the situation and reviewing all possible mitigation steps. We will continue to provide updates as new information becomes available. Thank you for your continued patience.

3. identified Oct 20, 2025, 09:01 AM UTC

   UPDATE: LoanPro Service Interruption - Regional Status Unaffected Regions: Services hosted in our Australia (AU) and Canada (CA) regions are online and operational. Payment Processing: While core services in AU and CA are up, you may experience delays in payment processing globally due to dependencies on the affected US region. AWS Status: AWS has confirmed significant error rates with their DynamoDB service, which is a primary contributor to the outage. Our team continues to monitor the situation. We appreciate your patience and will provide another update as soon as one is available.

4. monitoring Oct 20, 2025, 09:28 AM UTC

   UPDATE: Signs of Recovery - LoanPro Systems Coming Online We are seeing the first positive signs of recovery. AWS has applied initial mitigations and is reporting "early signs of recovery" for their services in the US-EAST-1 region. As a result, we are observing that core LoanPro systems are becoming accessible, and some users may now be able to log in. However, please be aware that the AWS issue is not fully resolved. You will likely experience significant slowness, high latency, and intermittent failures as the services work through their backlog. Our team is actively monitoring the platform's stability as AWS continues its full restoration.

5. monitoring Oct 20, 2025, 09:28 AM UTC

   We are continuing to monitor for any further issues.

6. monitoring Oct 20, 2025, 10:16 AM UTC

   LoanPro Recovery Update: Systems Restored, Processing Delays Expected We are pleased to confirm that LoanPro services are back online and stabilizing as the AWS platform recovers. Core functionality has been restored. However, the outage has created a processing backlog, and we want to provide clear expectations: Daily Maintenance (DM): The service interruption occurred in the middle of our DM cycle. Our engineering team is now manually initiating all DM processes to ensure system integrity. While processing throughput is normal, please anticipate delays in DM completion times. Queued Payments & Actions: All actions attempted during the outage (such as payments, AutoPays, rule executions, webhooks, etc.) were safely queued. The system is now working through this significant backlog, which will cause delays in the posting and completion of these items. Our team is monitoring all systems closely to clear this backlog as efficiently as possible. We will provide a final update once all processing times have returned to normal. Thank you for your continued patience and understanding.

7. monitoring Oct 20, 2025, 10:53 AM UTC

   Update: DM Processing Delayed for Some Tenants, Waiting on AWS Resource Allocation AWS has confirmed they have mitigated the root cause (a DNS issue), and most of their core services are now operating normally. This means the LoanPro platform is back online and accessible. However, for some tenants, our Daily Maintenance (DM) process is now experiencing a significant delay. While the vast majority of tenants have already seem their DM process completed, tenants whose DM process had not started before the outage are seeing delays. As a direct result of the widespread outage, there is now a massive, global backlog of requests for autoscaling resources. The DM process is queued, but we are dependent on AWS to allocate the server resources required to execute it. Per the last update (https://health.aws.amazon.com/health/status) AWS is throttling these queues so as not to overwhelm their data centers but expect full functionality to return soon. We apologize for this extended delay and appreciate your continued patience.

8. monitoring Oct 20, 2025, 12:11 PM UTC

   We are continuing to monitor the recovery of the AWS US-EAST-1 region. While the LoanPro platform and core services are back online, we are experiencing significant delays in all background processing. This includes: - Daily Maintenance (DM) - Autopay and Payment processing - Rules Engine execution - Webhook deliveries - OpenSearch Indexing - All other queued jobs Root Cause of Current Delays: The latest AWS updates confirm they are still experiencing two major issues that directly impact our processing: 1. AWS Lambda Delays: AWS is mitigating "Lambda's polling delays for... SQS." In simple terms, the AWS service responsible for picking new jobs (like a DM process or a payment) from our queue is not checking the queue frequently enough. This means new jobs are not being started on time. 2. AWS EC2 Capacity Issues: AWS is still recovering its ability to launch new servers (EC2 instances). Our systems are online and ready to process, but we are in a system-wide queue waiting for AWS to work through its backlog and allocate these critical Lambda and EC2 resources. We will continue to post updates as soon as AWS shows progress in clearing this processing backlog.

9. monitoring Oct 20, 2025, 03:37 PM UTC

   We are continuing to actively monitor the ongoing service degradation in the AWS US-EAST-1 region. The latest updates from AWS confirm the root cause is an internal network connectivity issue within their EC2 server infrastructure. This is preventing the reliable launch of new servers, which directly impacts our auto-scaling capabilities and the execution of background processes like Daily Maintenance (DM). While we are pleased to report that the majority of our tenants' DM processes have been successfully completed, a subset of tenants were ejected from the AWS processing queue and have been unable to complete DM. We are acutely aware of the difficulty this can cause for our valued clients and our engineering team is actively exploring all possible avenues to accelerate the resolution for these specific accounts. For now, the remaining DM jobs for affected tenants are queued, awaiting the necessary resources to become available as AWS works toward a full recovery. We will post another update as soon as we have more information. Thank you for your continued patience.

10. monitoring Oct 20, 2025, 05:16 PM UTC

    The LoanPro platform remains functional, but services in our US region are still impacted by the ongoing AWS (US-EAST-1) instability. As we've noted, this is causing unpredictable, intermittent errors and significant delays in background processing (like DM, payments, and autopays). We have a new update from AWS: They have confirmed that the root cause is impacting their EC2 (server) and Lambda (processing) services. They have announced they are "in the process of validating a fix" and will deploy it as soon as it is confirmed safe. (https://health.aws.amazon.com/health/status) We are hopeful this will resolve our processing queues. Our systems are ready and we are monitoring the situation in close contact with AWS support. Thank you for your continued patience.

11. monitoring Oct 20, 2025, 07:23 PM UTC

    We are seeing positive signs of recovery, aligning with the latest AWS update. AWS reports "significant improvements" to Lambda (processing) and "increased launches" of EC2 (server) instances. While this is a step forward, our background processing (including DM for affected tenants) remains blocked. We are still encountering residual network connectivity issues from the outage, which are preventing these new servers from successfully communicating with the database (RDS). We hope these signs of recovery will lead to full resolution soon.

12. monitoring Oct 20, 2025, 07:34 PM UTC

    We have good news. We are seeing a significant, positive step toward full recovery. All Daily Maintenance (DM) queues are now moving and actively processing. We are continuing to work on payment and NACHA file generation queues, but this is a good sign. Please note, the underlying AWS (US-EAST-1) environment is still resource-constrained, so throughput is slower than normal. Our engineering team is closely monitoring all jobs and will continue to add server resources as AWS makes them available. This is a strong sign of recovery, and we will continue to monitor the situation until all systems and backlogs are back to 100%.

13. monitoring Oct 20, 2025, 09:37 PM UTC

    We have a significant positive update. As a direct result of improving conditions at AWS, all Daily Maintenance (DM) processing for affected tenants is now actively clearing. We expect DM to be fully complete for all tenants within the next few minutes. This aligns with the latest AWS update, which now reports "full recovery for Lambda invocations" (the service that powers our background jobs) and that EC2 server launches are returning to "pre-event levels." AWS is now working through its general backlog of queued events. This is a strong, positive sign of recovery. We will continue to monitor until all system backlogs are cleared and performance returns to normal. We intend to wait until AWS returns to normal operations before we fully mark this event as resolved.

14. resolved Oct 20, 2025, 10:32 PM UTC

    We are pleased to confirm that this incident is now resolved. AWS has successfully restored all services to their normal, pre-event levels, including the EC2 and Lambda functions that were impacting our platform. All LoanPro services are now fully operational. All Daily Maintenance (DM) processes have successfully completed, and all background processing queues (including payments and autopays) are clear. We are aware of the significant disruption this global AWS outage caused to your operations and we deeply appreciate your patience and understanding as our team worked to manage the situation and restore full service. We will of course continue monitoring AWS and all LoanPro infrastructure, but this will be the final update for this incident unless AWS provides additional details that we feel need to be shared.

15. postmortem Oct 20, 2025, 10:56 PM UTC

    AWS has provided a detailed explanation of the cascading event. We are sharing it here in full for transparency:‌ > _**Oct 20 3:53 PM PDT** Between 11:49 PM PDT on October 19 and 2:24 AM PDT on October 20, we experienced increased error rates and latencies for AWS Services in the US-EAST-1 Region. Additionally, services or features that rely on US-EAST-1 endpoints such as IAM and DynamoDB Global Tables also experienced issues during this time. At 12:26 AM on October 20, we identified the trigger of the event as DNS resolution issues for the regional DynamoDB service endpoints. After resolving the DynamoDB DNS issue at 2:24 AM, services began recovering but we had a subsequent impairment in the internal subsystem of EC2 that is responsible for launching EC2 instances due to its dependency on DynamoDB. As we continued to work through EC2 instance launch impairments, Network Load Balancer health checks also became impaired, resulting in network connectivity issues in multiple services such as Lambda, DynamoDB, and CloudWatch. We recovered the Network Load Balancer health checks at 9:38 AM. As part of the recovery effort, we temporarily throttled some operations such as EC2 instance launches, processing of SQS queues via Lambda Event Source Mappings, and asynchronous Lambda invocations. Over time we reduced throttling of operations and worked in parallel to resolve network connectivity issues until the services fully recovered. By 3:01 PM, all AWS services returned to normal operations. Some services such as AWS Config, Redshift, and Connect continue to have a backlog of messages that they will finish processing over the next few hours. We will share a detailed AWS post-event summary._ ‌Considering the complexity, scope, and global impact of today’s AWS outages, I am proud of LoanPro's team, system, and clients for how each handled this unexpected event. Thank you for your patience and partnership. This will be the final update for this incident.