Temporary API Connection Errors (HTTP Socket Exhaustion)

Update timeline

1. identified Jul 31, 2025, 11:14 PM UTC

   Since our Azure migration last Sunday, we've observed intermittent API connection errors caused by unclosed HTTP connections leading to socket exhaustion. While we are actively monitoring the core application for internal impacts, API calls to Intuto might occasionally return errors. Failed API requests should be retried. A resolution is in progress, and we expect to deploy a fix on Wednesday night, August 6th, NZST.

2. monitoring Aug 06, 2025, 08:18 AM UTC

   A fix has been implemented and we are monitoring the results.

3. investigating Aug 06, 2025, 11:09 PM UTC

   We've identified that the previous fix deployed yesterday, has not fully resolved our socket exhaustion errors. We are actively investigating the root cause and working on a revised solution. We appreciate your patience. Further updates will be provided as new information becomes available.

4. investigating Aug 10, 2025, 08:36 PM UTC

   We've updated this incident to reflect that the ongoing socket exhaustion issue is now affecting all components across the system. While our fixes to date have helped reduce the impact, they haven't entirely resolved the problem. We're continuing to research new causes and are actively consulting with external experts in the field to assist in our investigation. We appreciate your patience as we work towards a full resolution.

5. monitoring Aug 19, 2025, 10:51 PM UTC

   A fix has been implemented and we are monitoring the results.

6. resolved Aug 20, 2025, 02:49 AM UTC

   This incident has now been resolved. A post mortem will be provided within 5 working days

7. postmortem Sep 03, 2025, 09:59 PM UTC

   ### **Postmortem: Temporary API Connection Errors \(HTTP Socket Exhaustion\)** **Incident Date:** August 1st, 2025 – August 20th, 2025 **Author:** Aaron Leggett ### **Incident Summary** Between August 1st and August 20th, 2025, our core application and API experienced intermittent connectivity issues, characterized by `System.InvalidOperationException: Handle is already used by another Socket` errors in our logs. This issue caused a widespread degradation of service, affecting any system component that relied on an outbound HTTP connection, including our **webhooks**, external **integrations** \(such as **emails**\), and certain **API** endpoints. The problem stemmed from unclosed connections that led to **socket exhaustion**. This incident occurred during our platform's migration to a more secure and scalable environment to benefit our future growth. ### **Impact** The incident **impacted all functionality that required an HTTP connection**, leading to a degradation of service across multiple components. * **User-Facing:** Users experienced intermittent failures when making API calls to our service. While some bugs were reported, the impact was largely mitigated by automated retry mechanisms in our webhooks and other systems, which reduced overall customer-facing failures for most integrations. The primary visible failure was **unsent emails**, which occurred because the underlying code never reached the point where the emails would have been sent or reported on. * **System-Facing:** The exception effectively halted any thread where it occurred, which prevented subsequent steps in the process, such as data synchronization or outbound email sends. These interruptions were generally temporary, as most systems have built-in retry logic to handle transient failures. However, this did lead to delays in data processing and a higher volume of retry attempts. * **Business Impact:** The incident required a significant allocation of resources. Our **entire team was dedicated to resolving this single issue**, with support from an external contracted team and direct assistance from Microsoft. To manage the problem, our team constantly monitored the system and performed manual restarts when it reached a state of exhaustion. The most severe impact occurred during our **peak times of 1-5 PM NZT on weekdays**, which required close, continuous monitoring. ### **Timeline of Events** * **August 1st:** The issue was first identified after our platform's migration to Microsoft Azure. We discovered several potential improvements to reduce the possibility of socket exhaustion and identified that the root cause was related to the new environment. Our previous Windows VM never reached its socket limit, as it could use all available sockets within IIS. In contrast, the Linux App Services environment had a significantly lower and more restrictive limit. * **August 3rd:** A third-party contractor that we frequently work with was brought in to assist with the investigation and troubleshooting. * **August 6th:** A range of **mitigating fixes** were implemented to help reduce the number of sockets being used. This included implementing a range of **asynchronous updates** and other optimizations tailored to the new Linux environment's resource management constraints. * **August 7th:** We identified that the deployed fixes had not fully resolved the socket exhaustion errors. With all internal options exhausted, we began **direct talks with Microsoft** for support. * **August 19th:** With the help of Microsoft support, the root cause was finally discovered in the application's codebase. * **August 20th:** A patch containing the definitive fix was released to production in the morning. Monitoring over the afternoon peak time showed that the issue had been fully resolved. ### **Root Cause** The root cause of the incident was a specific code bug involving the creation and disposal of `SafeFileHandle` objects using `IntPtr.Zero`. * **Code Implementation:** This was a piece of **legacy code** that had not been refactored in some time. It used a pattern of creating `SafeFileHandle(IntPtr.Zero)` instances to represent disposable resources. This practice was once a **standard recommended by Microsoft** but was **removed from their documentation at the start of this year** due to evolving platform best practices. * **Platform-Specific Behavior:** While this code did not cause issues in our previous Windows environment, it led to critical problems on the Linux platform. On Unix-like systems, `IntPtr.Zero` corresponds to the standard input \(stdin\) file descriptor, which is designated as file descriptor number zero. The `SafeFileHandle`'s `Dispose()` method attempts to close this handle, which is a key system resource. * **Resulting Exception:** By unintentionally closing stdin, the application disrupted the underlying networking stack \(Kestrel\), which led to the `Handle is already used by another Socket` exception. This bug effectively caused our application to leak and prematurely close socket connections, leading to the observed socket exhaustion and service degradation. This behavior was also exacerbated in the new Linux environment due to the **significantly lower socket limit** compared to our previous Windows VM setup. * **Pre-migration testing:** This issue was not caught in our pre-migration testing. While our testing was extensive and covered a wide variety of techniques and all system functionality individually, it was the sheer number of concurrent requests—a condition only exposed during stress testing—that caused the socket exhaustion. ### **Corrective Actions** * **Immediate Actions Taken:** A definitive patch was released to production on August 20th. This fix involved refactoring a legacy async class to remove the problematic `SafeFileHandle(IntPtr.Zero)` call, which was causing the underlying error. * **Long-Term Actions Taken:** We have taken the following steps to prevent recurrence and strengthen our platform: * **Refactored legacy code** to adopt modern .NET practices like `IHttpClientFactory` for efficient connection pooling. * **Implemented new monitoring** for key system metrics, such as open file descriptors, to proactively detect resource leaks. * **Updated our development standards** and conducted a comprehensive code review to prevent similar issues. We also provided formal training on .NET Core best practices for our Linux environment. * **Future Actions:** * **Investigate and implement system stress testing** to identify similar resource-related issues before they reach production. ‌ We sincerely apologize for the service disruption and the frustration this incident may have caused. We are committed to providing a stable and reliable platform. We deeply appreciate our customers for their patience and understanding as we worked to resolve this complex issue.