Technolutions Outage History

The interruption in availability was caused by a loss of quorum in the failover cluster. We are continuing to investigate periodic network interruptions that have occurred following recent upgrades to the database infrastructure and operating systems. In the meantime, we have further hardened the cluster configuration to prevent unintended failovers and interruptions.

There have been no further issues. We will continue to operate with manual failovers for a bit longer as we validate the efficacy of the automatic failovers. We will also continue to monitor for any other conditions that may result from the database server upgrades.

There have been no continued impacts. The incident, which primarily affected databases on the LIMA cluster, was caused by an automated failover to secondary infrastructure. A routine servicing update to the third-party database engine introduced a behavioral change that, under specific conditions, resulted in an exhaustion of worker threads. Upon failing over, some databases were slower to recover as a result of a related exhaustion of worker threads on this secondary node. We have disabled the functionality that resulted in this behavioral change, as advised by the vendor, and have increased the ceiling for worker threads to reduce the potential for reoccurrence. Everything remains stable at this time, and we will continue to monitor for any further issues.

At approximately 3:08pm Eastern Time, connectivity to databases hosted on the LUNA cluster was interrupted as a result of a database server stack dump. While connectivity usually is restored within a matter of seconds, some databases on the LUNA cluster did not see an automatic restoration of connectivity. A manual failover was therefore initiated, and by 3:38pm Eastern Time, connectivity was restored to all databases on the LUNA cluster. Databases on other clusters were not affected.

Earlier this morning, at approximately 2:51am Eastern Time, AWS experienced a service interruption which impacted certain specific components of Slate. While most Slate services remained online, the Slate Reader, PDF generation, and other services using the AWS DynamoDB database experienced elevated error rates. We observed full resolution of the AWS event by approximately 5:19am Eastern Time.

Earlier this morning, between approximately 8am and 10am Eastern Time, the HTML message bodies on a subset of scheduled mailings may have appeared blank when sending. A configuration update applied to background worker nodes temporarily resulted in some of these HTML messages failing to render at send time. All background worker nodes are rendering all HTML message bodies properly at this time. No transactional messages (e.g., form or event confirmation messages) were affected but, due to the specific and isolated nature of this issue, we are unfortunately unable to determine which specific HTML bodies may have failed to render properly at send time. We have put in place additional safeguards in order to help prevent this type of an issue from re-occurring during configuration updates in the future.

No further instances of latency have been observed on the ARES cluster. As a precautionary measure, we'll reset any other infrastructure that has recently experienced similar conditions which may have contributed to the elevated latency observed here.

On Wednesday afternoon, beginning around 3:40pm Eastern Time, a routine load balancer re-synchronization resulted in an unbalanced traffic distribution that increased the latency of some connections and returned a TCP RST status for some others, while still successfully processing and handling yet other connections depending upon their original node affinity. We have determined that this unbalanced traffic distribution was the result of an issue in the load balancers that reset node statuses during a re-synchronization. We have resolved the issue with the load balancers to prevent this from reoccurring. By 4:00pm Eastern Time, traffic had been fully redistributed across nodes and connection performance had returned to expected levels.