MindTouch incident

MindTouch degraded performance: Slow load time

Minor Resolved View vendor source →

MindTouch experienced a minor incident on November 25, 2024 affecting Application (General Service) and Search and 1 more component, lasting 1h 49m. The incident has been resolved; the full update timeline is below.

Started
Nov 25, 2024, 05:35 PM UTC
Resolved
Nov 25, 2024, 07:24 PM UTC
Duration
1h 49m
Detected by Pingoru
Nov 25, 2024, 05:35 PM UTC

Affected components

Application (General Service)SearchIn-Product Contextual HelpEmail ServicesMindTouch Success CenterAnalytics

Update timeline

  1. investigating Nov 25, 2024, 05:35 PM UTC

    MindTouch Engineering is looking into issues related to slow load on MindTouch sites.

  2. monitoring Nov 25, 2024, 05:56 PM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved Nov 25, 2024, 07:24 PM UTC

    This incident has been resolved.

  4. postmortem Dec 04, 2024, 02:59 PM UTC

    **Incident Details:** **Date and Time:**` `Nov 25, 17:35 UTC **Impact:**` ` The unavailability of IP addresses caused interruptions in network functionality and scalability within the environment. **Root Cause:**` ` The failure to scale down instances from the old release after the migration led to an excessive consumption of IP addresses in the existing subnets, ultimately exhausting the available pool. **Resolution:** 1. **Remediation Actions:** 1. Added additional subnets to the network to increase the available IP address pool. 1. Introduced the new subnets into the scaling groups to ensure consistent network operations. 1. **Incident Closure:**` ` The system has been stabilized, and no further issues have been observed since the remedial actions were implemented.