Partial Outage

Update timeline

1. investigating Feb 11, 2025, 11:11 PM UTC

   Jeenie's services had significantly increased response times and increased error rates for approximately 30 minutes. Users would have experienced significant delays, or failures when attempting to log in, and when initiating calls.

2. resolved Feb 11, 2025, 11:13 PM UTC

   We identified the source of the increased response times and error rates, and made adjustments to our resource allocations to address the latency.

3. postmortem Feb 12, 2025, 10:15 PM UTC

   **Impact description** The operations team had been monitoring a period of intermittent degraded performance from earlier in the day, and began receiving a significantly increased rate of alerts about internal service restarts. It quickly became clear that the affected service was not automatically recovering as it typically would. At this time, the team was still unable to see any clear clear signs of resource constraints that would explain the inability of the service to recover. Recent changes to infrastructure, or service code, had also already been ruled out earlier in the day, as there were no significant changes of relevance. The team was ultimately able to correlate the increase in service restarts with an increase in log messages from the affected service that demonstrated an internal failure to communicate with the primary database service. That was initially confusing because it did not align with any clearly visible resource constraints that would explain it, as the upstream database service was performing nominally. **Initial response** Understanding that there must be a resource constraint somewhere that would impair the service’s ability to communicate with the primary database, the team opted to make some immediate infrastructure adjustments to mitigate the situation: * doubled the number of service instances available * doubled the amount of memory, and memory limits, allocated to each service instance The combinations of these updates allowed the service to have enough operating headroom to recover once it was redeployed. After deploying these changes, service returned to normal within a couple of minutes. During this time, customers would have experienced widespread slowness, and frequent errors preventing logins and creating calls on the platform. While some calls were still being fulfilled during this time, many customers would have experienced a significant impact. **Root cause** With the service returned to normal operations, the team continued to investigate possible root causes. After a very thorough review, the team ultimately determined the following: * the maximum number of database connections internally allowed per service instance had become an invisible bottleneck that triggered a cascading series of failures * as one service instance exhausted the number of database connections it was allowed to have, traffic would begin to rapidly pile up on that instance, causing a very quick spike in memory usage. This spike was often too quick to clearly distinguish in our monitoring systems, and so it was not clear from monitoring tools that this was a recurring cause for service restarts * the infrastructure would then respond as expected and restart the affected service instance, as it was exceeding the amount of memory it was allowed to use * as the service instance was restarting, the traffic it was handling would be redistributed onto the other service instances * the immediate increased load in the other service instances would then trigger the same saturation of database connections, and resulting memory use, causing the infrastructure to restart those instances as well * as individual service instances restarted, they would very quickly become overloaded due to traffic being redistributed from other overloaded instances, thus leading to a loop of service restarts **Response plan** The team has a number of near term changes planned to mitigate the risk of this type of failure in the future. Those upcoming changes include: * Significant increase in number of active database connections allowed per service instance * Reconfigure internal services that manage the connection between the affected service and the primary database, which should reduce latency between the two and be simpler to monitor * Make a series of updates to the database that will improve the performance of frequently used queries * Review areas where improved internal instrumentation and monitoring could provide a leading indicator for this type of issue in the future These changes, when taken together, will significantly reduce the risk of a similar situation in the future.