Freshcaller experienced a minor incident on November 9, 2021 affecting Search and Sign in / Sign up and 1 more component, lasting 5h 36m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 09, 2021, 10:01 PM UTC
Due to an ongoing outage in AWS, some of the Freshdesk services may experience degraded performance and latency. AWS and its engineers are working towards mitigating the outage on priority and Freshworks is following up on the same. More updates to follow
- investigating Nov 09, 2021, 10:53 PM UTC
AWS Team have identified the S3 subsystem responsible for increased 5xx error rates for the S3 PUT APIs, and are working to isolate the root cause within this subsystem. Customers may also be experiencing increased latency when performing PUT operations. During this time, we recommend customers retry any failed requests.
- investigating Nov 10, 2021, 01:05 AM UTC
We are starting to see some improvement in the 5xx error rates and latencies for S3 API requests, in particular S3 PUT API calls. The issue affected a subsystem stores routing metadata used by Amazon S3 to map API requests to the storage nodes. A recent update caused increased load within this subsystem, which led to increased error rates and latencies for the S3 APIs. We have now successfully mitigated this increased load within this subsystem and are seeing early signs of recovery. As the sub-system processes the backlog of requests, S3 API error rates and latencies will continue to improve.
- investigating Nov 10, 2021, 02:05 AM UTC
We continue to see a gradual improvement in error rates as we process the backlog of mappings between request metadata and data storage in the sub-system affected by the increased load. AWS is currently working on mitigations to speed up the processing of the backlog during this event. Once the backlog is resolved, we expect that the error rate will fully recover. The vast majority of requests to S3 APIs continue to operate normally.
- investigating Nov 10, 2021, 02:18 AM UTC
Our Cloud provider continues to process the backlog of mappings between request metadata and data storage in the sub-system affected by the increased load. They have implemented two parallel mitigations to improve the speed of processing. Both mitigations are in process of deployment. Once the backlog is resolved, we expect that the error rate will fully recover. The vast majority of requests to S3 APIs continue to operate normally.
- resolved Nov 10, 2021, 03:37 AM UTC
Our cloud service provider has completed the mitigation to accelerate processing of the mappings between S3 API request metadata and storage. The backlog has been fully processed and S3 API errors and latencies have returned to normal levels. The issue has been resolved and the service is operating normally.