ChatDaddy Outage History

We found the issue to be in an internal update relating to Meta AI chats. The issue has been resolved & an alarm put in place to catch similar issues quicker.

We're in the process of improving the efficiency of the queries put to the DB, and expect to complete the process during the first week of Dec 24.

One of our deployments, in a series of deployments, to fix a bug in channel access failed – thus causing this incident. We’ve rectified the order of deployments to prevent this happening in the future. More integration tests are also needed to better test for this before deployment to production, we will be undertaking this in 2025.

This was the same issue as the one on Oct 31. We finally managed to track the source of the delay, that being the incorrect reading of “high memory usage” in our queuing system that prevented more events from being added. We’ve added more memory to counteract this as well as increase the threshold to high memory usage. Furthermore, the internal alarm setup on Oct 31 helped catch this incident early & resolve it much faster than the incident on the 31st.

We’ve not been able to identify the exact issue that caused this problem, it seems to be an issue with the queuing system we have in place. We have some suspected sources that we will continue to investigate. However, an internal alarm has also been setup to catch this issue early & resolve it quickly to reduce disruption.

We had an unexpected load to the authentication service denied further requests, this was a precaution to not further overload the system. We’ve thus increased the capacity of the service.