Voicenotes Outage History

We experienced a production API outage after a database schema migration on a large MySQL table caused increased database load and connection buildup. The RDS instance reached its maximum connection limit, resulting in Too many connections errors and 500 responses across the API. Application and background worker services were temporarily stopped to prevent new database connections. Existing connections cleared, services were restarted, and the API returned to normal operation. We are reviewing our production migration process and database safeguards to reduce the risk of similar incidents.

On July 24, 2026, our services experienced a minor disruption affecting users in Western North America due to a network performance incident with our upstream provider, Cloudflare. The incident occurred between approximately 11:24 UTC and 11:59 UTC. Cloudflare identified instability involving an infrastructure partner, implemented a fix, and subsequently confirmed the incident was resolved. Our services have been operating normally since the resolution. We apologize for any inconvenience caused. https://www.cloudflarestatus.com/incidents/p2pvdgpmw0ft

The API service experienced degraded performance due to a significant buildup in internal processing queues. The high number of pending requests caused increased response times, which in turn led to elevated 504 Gateway Timeout errors. The backlog has now been cleared, and API response times have returned to normal levels. We are continuing to monitor the system to ensure stability and to prevent recurrence.

We are resolving this incident, as we have now observed approximately 30 minutes of stable service availability and normal customer traffic patterns. Cloudflare has fully resolved the global outage, as reflected on their official status page. We have cleared the previously accumulated queues and verified that all services are functioning as expected.