Freshchat experienced a minor incident on November 14, 2021 affecting Freddy AI Agent, lasting 4h 15m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 14, 2021, 04:45 AM UTC
Degraded performance in bots for few customers in INDIA DC. We are investigating the errors causing the issue.
- investigating Nov 14, 2021, 07:35 AM UTC
We have addressed the issue partially by fixing an unhealthy server. However , major part of the issue is still getting investigated. We are working on the issue at database level.
- investigating Nov 14, 2021, 07:40 AM UTC
The issue is due to database taking longer to read and write data. We are working with our cloud provider to bring this under control.
- monitoring Nov 14, 2021, 08:42 AM UTC
The issue has been brought under control, bot should be working fine now. The database received sudden surge in inflow of data that seem to have triggered this series of events that caused clogging in API response. We'll continue to monitor the situation. We'll also continue to investigate the query that triggered the issue and arrive at preventive measures.
- resolved Nov 14, 2021, 09:00 AM UTC
We have gained control of the lag in writing to database (latency). It is coming down and all are normal now. We are closing the incident. We’ll continue to monitor the system.