ChatDaddy incident
Issue in Receiving Live Inbox Events + Accessing Message Flows
ChatDaddy experienced a major incident on August 8, 2024 affecting Authentication Service and Flow service, lasting 3h 30m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Aug 08, 2024, 06:53 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Aug 08, 2024, 07:41 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Aug 08, 2024, 10:23 AM UTC
This incident has been resolved.
- postmortem Dec 01, 2024, 01:58 PM UTC
Our events & flows connections to the database unexpectedly started failing. It seems to have been caused by an unusually high number of connections that were created by the aforementioned services. Though despite the reduction in connections & other measures, this problem persisted – we’re still not entirely sure of the root cause. Since then, we've updated our infrastructure to prevent the services from ever breaching their quota of connections