Is Technolutions down?

We are investigating an issue that is affecting the availability of certain databases. We will provide updates as we have them.

The affected databases are online and available again. We are continuing to investigate the cause of this interruption.

The interruption in availability was caused by a loss of quorum in the failover cluster. We are continuing to investigate periodic network interruptions that have occurred following recent upgrades to the database infrastructure and operating systems. In the meantime, we have further hardened the cluster configuration to prevent unintended failovers and interruptions.

Databases on LUNA were unavailable earlier this hour as the high availability service initiated an unexpected failover and got caught in a failover loop. Database servers were upgraded this weekend to an updated version, and we're investigating what role that may be playing. We are disabling the automatic failover temporarily as a protective measure as we investigate further.

There have been no further issues. We will continue to operate with manual failovers for a bit longer as we validate the efficacy of the automatic failovers. We will also continue to monitor for any other conditions that may result from the database server upgrades.

We are investigating the intermittent availability of certain databases in the US region. Some databases are still recovering following a failover to secondary infrastructure and may be unavailable during the completion of the failover. We will provide updates shortly.

All affected databases recovered approximately a half-hour ago. We're continuing to address some internal connectivity issues, but we're not seeing any observable impacts at this time. There may be brief connection interruptions as failbacks complete later.

We are continuing to monitor for any further issues.

There have been no continued impacts. The incident, which primarily affected databases on the LIMA cluster, was caused by an automated failover to secondary infrastructure. A routine servicing update to the third-party database engine introduced a behavioral change that, under specific conditions, resulted in an exhaustion of worker threads. Upon failing over, some databases were slower to recover as a result of a related exhaustion of worker threads on this secondary node. We have disabled the functionality that resulted in this behavioral change, as advised by the vendor, and have increased the ceiling for worker threads to reduce the potential for reoccurrence. Everything remains stable at this time, and we will continue to monitor for any further issues.