Service Disruption Due to Unexpected Memory Exhaustion

Update timeline

1. investigating Mar 06, 2026, 02:08 PM UTC

   At approximately 05:00 UTC on March 6, our service experienced a temporary disruption caused by a sudden surge of abnormal requests that led to excessive memory usage and exhausted available system memory. This resulted in the service becoming unavailable for a short period. As an immediate mitigation, we enabled zram-based swap on the affected hosts to provide additional compressed memory buffering and improve resilience against sudden memory spikes. The service has since been restored and is operating normally. Our team is continuing to investigate the root cause of the abnormal request pattern and will implement further safeguards once the underlying issue is identified. We will provide additional updates if new information becomes available.

2. resolved Mar 06, 2026, 02:09 PM UTC

   This incident has been resolved.