Increased error rate and partial unavailability in APAC region

Update timeline

1. resolved Aug 23, 2023, 01:55 PM UTC

   Between august 1st and august 9th, FeedbackFruits experienced issues that caused an increased error rate and partial unavailability in the APAC region. This incident caused different aspects of our product to behave abnormally. Although only a small percentage of requests was affected, this may have caused noticeable disruption to a selection of end users.

2. postmortem Aug 23, 2023, 02:02 PM UTC

   Between august 1st and august 9th, FeedbackFruits experienced an incident that caused an increased error rate and partial unavailability in the APAC region. This incident caused different aspects of our product to behave abnormally. Although only a small percentage of requests was affected, this still caused noticeable disruption to certain end users. **Mitigation** The issue was first raised internally on the 1st of august, and an engineer started looking into it the same day. However, after investigation, the root cause of the issue could not be found. After several attempts to determine the cause of the issue, we started applying mitigation. Several workarounds were introduced in the days following the first report. Unfortunately, these workarounds did not have the desired affect. **Cause of the Issue** ‌The issues were caused by intermittent connectivity issues with a caching database. The connection with the instance dropped at times, causing our servers to be unable to access it and return errors. After contacting our infrastructure provider Microsoft Azure, we worked together with them on a solution. We realized that the cache consumed a lot more memory than we expected. This should not have led to connection issues though, and the team at Microsoft was able to recover the instance. We rolled out a fix for the memory consumption simultaneously, which will avoid the instance from being at capacity in the future. **Communication** Because of the low percentage of requests that was affected, impact was believed to be minimal, and the incident was not reported publicly on our statuspage. This does not comply with the communication and transparency we aim to offer our users. An incident like this should be publicly communicated, which is why we decided to backfill the incident. **Steps Ahead** As we move forward, we're taking the following actions to prevent such incidents: 1. Process optimization: Our team is actively fine-tuning the resource consumption and monitoring to ensure it performs well. 2. Enhanced communication: We're committed to providing clearer and timely updates during incidents. **Our Apologies** We sincerely apologize for any trouble this incident may have caused you. We acknowledge the importance of reliable services and are working diligently to prevent similar occurrences.