CyberFOX experienced a major incident on December 3, 2025, lasting 11h 45m. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Dec 03, 2025, 02:43 PM UTC
Yesterday, our automated SSL certificate import process encountered an unexpected failure. As a result, an expired certificate remained briefly active in our Content Delivery Service for the Password Boss services. The issue was promptly identified and addressed. Within minutes of discovery, the updated certificate was manually installed, restoring full functionality. All services are now operating normally with no residual impact. We are conducting a thorough review of our certificate automation workflows to strengthen reliability and prevent similar occurrences in the future. Enhancements will include additional monitoring, validation checks, and failover safeguards to ensure seamless certificate updates going forward.
- postmortem Dec 03, 2025, 02:50 PM UTC
# SSL Certificate Automation Failure ## Summary On **December 3, 2025 at 00:00 UTC \(December 2, 2025 at 8:00 PM EST\)**, our automated SSL certificate import process experienced a failure. As a result, an expired certificate briefly remained active on our Content Delivery Service. ## Impact * **Duration:** The issue began at 00:00 UTC / 8:00 PM EST. * **Identification:** The incident was identified at **7:30 AM EST** on December 3. * **Resolution:** The updated certificate was manually installed by **7:45 AM EST**, restoring secure connections. * **Scope:** Limited to the Content Delivery Service for the Password Boss services. * **Customer Experience:** Some customers may have seen browser or application warnings about insecure connections. No data was compromised, and service availability was maintained. ## Resolution The issue was detected by our operations team at 7:30 AM EST and resolved within 15 minutes through manual certificate installation. All services are now fully operational. ## Preventive Actions We are implementing several improvements to strengthen reliability and prevent similar incidents: * **Automation Review:** Comprehensive audit of all certificate automation processes. * **Validation Checks:** Adding success verification steps to confirm certificates are properly imported. * **Real‑Time Alerts:** Enhancing monitoring to notify our team immediately if a failure occurs. * **SSL Monitoring Enhancements:** Expanding monitoring of SSL certificate services to identify and alert even faster at the moment of failure. * **Resilience Enhancements:** Introducing fallback mechanisms to ensure certificates are applied even if automation fails. ## Commitment We take our customers' trust and security seriously. While this incident was short‑lived and contained, it highlighted the importance of robust certificate lifecycle management. The corrective actions underway will reduce the likelihood of recurrence and further strengthen the reliability of our services.