KnowledgeOwl Outage History

**What happened** We successfully completed a critical transition away from a legacy dependency; however, the deployment introduced a dormant performance regression. This issue remained hidden during low-traffic periods, only impacting database health once subjected to the full weight of production loads. **How we resolved it** Our monitoring systems detected the issue and alerted our team. Our engineering team placed all services into maintenance mode during investigation and troubleshooting. Once a successful fix was in place, all services were brought back online. **What we're doing to prevent this** We are evaluating improvements to our testing process to better simulate real-world traffic conditions before deploying infrastructure changes. We apologize for the downtime and appreciate everyone’s support, kind words, and patience during and after the incident.

Thank you for your patience through yesterday’s incident. We know performance impacts to our app and your live knowledge bases are never pleasant, and we appreciate your patience and grace while we worked through things. Here’s a brief postmortem of what happened and our steps to remediate: An unexpected spike in traffic led to intermittent 504 errors. To maintain core system availability, we temporarily restricted computationally heavy operations while scaling our infrastructure. Once that was complete, we returned back to normal operations. We’ve noticed no significant spikes or errors since then.

Microsoft Edge users experienced security warnings when accessing knowledge base files due to a SmartScreen flag on our file hosting domain. No data was compromised. We have deployed additional safeguards to prevent recurrence.

A database cluster node experienced degraded performance, temporarily impacting overall app performance until automatic failover mechanisms restored normal operations.

**What happened** A recent security-related library upgrade introduced unforeseen CPU inefficiencies at the same time we saw a significant increase in legitimate traffic. The combination temporarily overwhelmed our resources before the system could scale as expected. **What we are doing about it** We are actively working to stabilize performance and reduce the likelihood of this happening again, including short-term mitigations while we address the underlying CPU issues. **Our apologies** Thank you for your patience and understanding, and as always, if you notice anything unexpected, please reach out. Our apologies again for the disruption!