ServiceChannel Outage History

**Incident Report: Login Failures due to port exhaustion** **Date of Incident:** 02/27/2026 **Time/Date Incident Started:** 02/27, 5:00 PM EST **Time/Date Stability Restored:** 02/27, 5:06 PM EST **Time/Date Incident Resolved:** 02/27, 5:15 PM EST **Users Impacted:** Some **Frequency:** Continuous **Impact:** Major **Incident description:** On February 27, 2026, the Service Channel platform experienced an issue with the Login and Authentication services, resulting in some users being unable to log in. Additionally, a subset of users who were already logged in encountered authentication errors, which caused their sessions to become invalid. Monitoring systems alerted the SRE team at 5:00 PM EST. The SRE team initiated incident triage and identified the underlying issue. Service restarts and connection resets restored stability by 5:06 PM EST. All servers and services were fully verified and confirmed operational by 5:15 PM EST. **Root Cause Analysis:** The incident was caused by a sudden increase in concurrent connections to the Login and Authentication services, which resulted in port exhaustion on impacted nodes. As available ports were depleted, the system was unable to establish new connections and intermittently failed to maintain existing authentication sessions. This condition led to users being unable to log in, while a subset of users who were already authenticated experienced session validation failures that caused authentication interruptions. The issue was resolved by restarting the affected nodes, which reset the exhausted ports and restored normal connection handling. **Actions Taken:** * Monitoring alerts were received by the SRE team indicating authentication service degradation. * The SRE team initiated incident triage and investigated the login and authentication failures. * Port exhaustion was identified on impacted nodes during troubleshooting. * Recovery actions, including restarting affected nodes and resetting connections, were performed. * Services returned to normal operation following these recovery actions. * Platform functionality was fully verified after stability was restored.

**Incident Report: Cloud Provider issue cause Service Channel Platform Interruption** **Date of Incident:** 02/26/2026 **Time/Date Incident Started:** 02/26, 9:42 AM EST **Time/Date Stability Restored:** 02/26, 10:06 AM EST **Time/Date Incident Resolved:** 02/26, 10:15 AM EST **Users Impacted:** All U.S. **Frequency:** Continuous **Impact:** Major **Incident description:** On February 26, 2026, the Service Channel platform experienced multiple service interruptions due to an outage at a cloud hosting provider. The outage impacted several platform services that rely on the provider’s managed infrastructure. Our monitoring systems alerted the SRE team at 9:42 AM EST. The cloud hosting provider identified and resolved the underlying issue, after which service stability was restored by 10:06 AM EST. All servers and services were fully verified and confirmed operational by 10:15 AM EST. **Root Cause Analysis:** The cloud hosting provider experienced an issue with their managed SQL instances, which disrupted communication between several web applications and the database layer. This resulted in partial service interruptions across the platform. The cloud provider implemented a corrective action to resolve the SQL service issue, which restored connectivity and platform functionality. **Actions Taken:** * Monitoring alerts were received by the SRE team indicating service degradation. * Investigation identified connectivity issues between certain application modules and the database servers. * During incident triage, multiple recovery actions were attempted across impacted components; the issue persisted throughout these efforts * Services returned to normal operation once the cloud provider resolved the SQL issue