Users unable to log in.

Update timeline

1. investigating Oct 29, 2025, 04:11 PM UTC

   We are currently investigating this issue.

2. identified Oct 29, 2025, 04:18 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Oct 29, 2025, 04:31 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Oct 29, 2025, 04:40 PM UTC

   This incident has been resolved.

5. postmortem Oct 29, 2025, 08:00 PM UTC

   Database Storage Capacity Incident Incident Overview: * **Date/Time:** 2025-10-29, 16:11-16:40 UTC * **Duration:** 29 minutes * **Impact:** Users unable to log in across all platforms \(Website, Mobile Apps for iOS and Android, Chrome Extension\) * **Suspected Cause:** Database disk storage reached capacity limit * **Resolution Method:** Increased database storage capacity limits Incident Details: At approximately 16:11 UTC, multiple user reports indicated authentication failures across all platforms. Initial testing confirmed the issue, with staff members also unable to log in to the system. Investigation revealed that the MariaDB database had reached its 100GB storage capacity limit, causing it to enter a special Amazon state for full disks. The database was originally provisioned with a 100GB limit when initially created, which had been sufficient until today. Technical analysis suggests that database indexes, which enable fast data retrieval, likely occupy a significant portion of the storage space rather than the actual data itself. Response Actions: * **16:11 UTC:** Initial reports received about users unable to log in; investigation began * **16:18 UTC:** Issue identified as database storage capacity reached its limit * **16:18-16:31 UTC:** Remediation implemented by doubling the database storage limits * **16:31 UTC:** Fix implemented and monitoring began * **16:40 UTC:** Incident resolved; all systems confirmed operational The response team quickly identified the root cause and implemented an immediate solution by increasing the database storage capacity, which promptly restored system availability. Retrospective & Recommendations: **Findings:** * The MariaDB database reached its 100GB storage capacity limit without prior warning * No proactive monitoring was in place to alert the team before reaching critical storage thresholds * The timing was fortunate as it occurred during business hours rather than during a critical deployment period **Challenges:** * Initial concern that the issue might be related to ongoing development work caused momentary panic * Current alerting system focuses on activity metrics and traffic spikes rather than infrastructure capacity * Existing alerts are not well-tuned, resulting in notification fatigue and unclear prioritization **Recommendations:** * Configure disk space threshold alerts at 80% capacity for all database instances * Review and increase storage limits for other database instances to prevent similar incidents * Refine the current alerting system to reduce noise and prioritize critical notifications * Implement a more comprehensive monitoring strategy for infrastructure resources **Follow-Up:** * Team to implement disk space monitoring alerts as a priority task * Review storage requirements for all database instances * Evaluate and improve the current alerting system to ensure important notifications receive appropriate attention Additional Notes: This incident was resolved quickly due to prompt identification of the root cause. The timing was fortunate as it occurred before a planned deployment rather than during or after it, which could have complicated diagnosis.