Technolutions Outage History

Started May 14, 2026, 04:29 PM UTC· resolved May 15, 2026, 03:30 AM UTC

1. investigating May 14, 2026, 04:29 PM UTC

   We're observing increased disk latency for databases hosted on the MORA cluster. We will be initiating a failover to secondary infrastructure as we investigate further.

2. monitoring May 14, 2026, 05:20 PM UTC

   We initiated a failover at 12:30pm Eastern Time, the failover went smoothly, and databases are operating within expected performance ranges. We will initiate a failback later this evening following a restart of the primary service infrastructure. We will continue to monitor to ensure that performance levels remain within expected ranges.

3. resolved May 15, 2026, 03:30 AM UTC

   This incident has been resolved.

Read the full incident report →

Started May 04, 2026, 05:14 AM UTC· resolved May 04, 2026, 06:53 AM UTC

1. investigating May 04, 2026, 05:14 AM UTC

   We are investigating an issue that is affecting the availability of certain databases. We will provide updates as we have them.

2. investigating May 04, 2026, 05:16 AM UTC

   The affected databases are online and available again. We are continuing to investigate the cause of this interruption.

3. resolved May 04, 2026, 06:53 AM UTC

   The interruption in availability was caused by a loss of quorum in the failover cluster. We are continuing to investigate periodic network interruptions that have occurred following recent upgrades to the database infrastructure and operating systems. In the meantime, we have further hardened the cluster configuration to prevent unintended failovers and interruptions.

Read the full incident report →

Started Apr 27, 2026, 05:43 PM UTC· resolved Apr 27, 2026, 09:15 PM UTC

1. monitoring Apr 27, 2026, 05:43 PM UTC

   Databases on LUNA were unavailable earlier this hour as the high availability service initiated an unexpected failover and got caught in a failover loop. Database servers were upgraded this weekend to an updated version, and we're investigating what role that may be playing. We are disabling the automatic failover temporarily as a protective measure as we investigate further.

2. resolved Apr 27, 2026, 09:15 PM UTC

   There have been no further issues. We will continue to operate with manual failovers for a bit longer as we validate the efficacy of the automatic failovers. We will also continue to monitor for any other conditions that may result from the database server upgrades.

Read the full incident report →

Started Apr 26, 2026, 01:13 AM UTC· resolved Apr 26, 2026, 05:44 AM UTC

1. investigating Apr 26, 2026, 01:13 AM UTC

   We are investigating the intermittent availability of certain databases in the US region. Some databases are still recovering following a failover to secondary infrastructure and may be unavailable during the completion of the failover. We will provide updates shortly.

2. monitoring Apr 26, 2026, 01:39 AM UTC

   All affected databases recovered approximately a half-hour ago. We're continuing to address some internal connectivity issues, but we're not seeing any observable impacts at this time. There may be brief connection interruptions as failbacks complete later.

3. monitoring Apr 26, 2026, 01:39 AM UTC

   We are continuing to monitor for any further issues.

4. resolved Apr 26, 2026, 05:44 AM UTC

   There have been no continued impacts. The incident, which primarily affected databases on the LIMA cluster, was caused by an automated failover to secondary infrastructure. A routine servicing update to the third-party database engine introduced a behavioral change that, under specific conditions, resulted in an exhaustion of worker threads. Upon failing over, some databases were slower to recover as a result of a related exhaustion of worker threads on this secondary node. We have disabled the functionality that resulted in this behavioral change, as advised by the vendor, and have increased the ceiling for worker threads to reduce the potential for reoccurrence. Everything remains stable at this time, and we will continue to monitor for any further issues.

Read the full incident report →

Started Dec 05, 2025, 08:58 PM UTC· resolved Dec 11, 2025, 08:29 PM UTC

1. identified Dec 05, 2025, 08:58 PM UTC

   We are currently investigating an intermittent issue that has affected the processing of certain exports and imports. We believe that this is due to an issue with an updated AWS SDK that interacts with a background storage layer. We have temporarily reverted to the previous version of this third-party tool. Imports that previously failed are being reprocessed now.

2. resolved Dec 11, 2025, 08:29 PM UTC

   We have completed our review of the incident that affected the processing of certain exports and imports on 12/5. Slate uses a cluster of worker nodes to process background jobs. These nodes are periodically refreshed with the latest version of Slate, and on 12/5 they were updated to a build that included a newer version of a third-party AWS SDK. Worker nodes employ a shadow-copying mechanism that creates local cached copies of binary dependencies to avoid locking files on shared storage. This mechanism refreshes cached assemblies only when it detects a change in the assembly’s identity, which typically occurs during breaking or versioned updates. The updated AWS SDK introduced an internal structural change: a property was moved from a derived class to its base class. Although no public API signature changed, the compiled IL binding did. Because the assembly’s identity did not change, the shadow-copy cache was not refreshed. As a result, the worker nodes attempted to execute IL referencing the property on the derived class against the older cached version, causing a runtime “method not found” error. On the afternoon of 12/5, we reverted the worker nodes to the earlier AWS SDK version, restoring normal operation. We have since implemented enhancements to our worker-node architecture to provide stronger cache isolation and to ensure that updates to the AWS SDK and other dependencies are correctly recognized, even when internal changes occur without a corresponding change in assembly identity. Web nodes use a different shadow-copying mechanism and were not affected.

Read the full incident report →

Started Nov 11, 2025, 08:08 PM UTC· resolved Nov 11, 2025, 08:38 PM UTC

1. resolved Nov 11, 2025, 09:00 PM UTC

   At approximately 3:08pm Eastern Time, connectivity to databases hosted on the LUNA cluster was interrupted as a result of a database server stack dump. While connectivity usually is restored within a matter of seconds, some databases on the LUNA cluster did not see an automatic restoration of connectivity. A manual failover was therefore initiated, and by 3:38pm Eastern Time, connectivity was restored to all databases on the LUNA cluster. Databases on other clusters were not affected.

Read the full incident report →

Started Oct 20, 2025, 06:51 AM UTC· resolved Oct 20, 2025, 09:19 AM UTC

1. resolved Oct 20, 2025, 06:51 AM UTC

   Earlier this morning, at approximately 2:51am Eastern Time, AWS experienced a service interruption which impacted certain specific components of Slate. While most Slate services remained online, the Slate Reader, PDF generation, and other services using the AWS DynamoDB database experienced elevated error rates. We observed full resolution of the AWS event by approximately 5:19am Eastern Time.

Read the full incident report →

Started Jun 12, 2025, 03:05 PM UTC· resolved Jun 12, 2025, 03:05 PM UTC

1. resolved Jun 12, 2025, 03:05 PM UTC

   Earlier this morning, between approximately 8am and 10am Eastern Time, the HTML message bodies on a subset of scheduled mailings may have appeared blank when sending. A configuration update applied to background worker nodes temporarily resulted in some of these HTML messages failing to render at send time. All background worker nodes are rendering all HTML message bodies properly at this time. No transactional messages (e.g., form or event confirmation messages) were affected but, due to the specific and isolated nature of this issue, we are unfortunately unable to determine which specific HTML bodies may have failed to render properly at send time. We have put in place additional safeguards in order to help prevent this type of an issue from re-occurring during configuration updates in the future.

Read the full incident report →

Started Mar 12, 2025, 02:01 PM UTC· resolved Mar 12, 2025, 08:04 PM UTC

1. investigating Mar 12, 2025, 02:01 PM UTC

   We're observing increased latency on the ARES cluster and will initiate a service failover. Connections to databases on the ARES cluster may be briefly affected as they're rerouted.

2. monitoring Mar 12, 2025, 02:27 PM UTC

   The failover completed successfully and services are operating within normal parameters at this time. We'll continue to monitor this cluster to ensure continued performance within normal operating parameters.

3. resolved Mar 12, 2025, 08:04 PM UTC

   No further instances of latency have been observed on the ARES cluster. As a precautionary measure, we'll reset any other infrastructure that has recently experienced similar conditions which may have contributed to the elevated latency observed here.

Read the full incident report →

Started Jan 29, 2025, 08:40 PM UTC· resolved Jan 29, 2025, 09:00 PM UTC

1. resolved Jan 29, 2025, 09:00 PM UTC

   On Wednesday afternoon, beginning around 3:40pm Eastern Time, a routine load balancer re-synchronization resulted in an unbalanced traffic distribution that increased the latency of some connections and returned a TCP RST status for some others, while still successfully processing and handling yet other connections depending upon their original node affinity. We have determined that this unbalanced traffic distribution was the result of an issue in the load balancers that reset node statuses during a re-synchronization. We have resolved the issue with the load balancers to prevent this from reoccurring. By 4:00pm Eastern Time, traffic had been fully redistributed across nodes and connection performance had returned to expected levels.

Read the full incident report →