Increased background job latency in APAC region

Update timeline

1. investigating Aug 25, 2023, 07:51 AM UTC

   We are seeing increased processing times for progress calculations, course and group synchronizations and grade publishing.

2. identified Aug 25, 2023, 08:05 AM UTC

   We are scaling up capacity to decrease the latency of background jobs. This should clear the current queue shortly.

3. monitoring Aug 25, 2023, 09:42 AM UTC

   The latency of background jobs is back to normal. We are monitoring the situation and investigating the cause.

4. resolved Aug 25, 2023, 11:02 AM UTC

   This incident has been resolved.

5. postmortem Sep 05, 2023, 02:04 PM UTC

   On August 25th, FeedbackFruits experienced an incident that caused an increased latency in background job processing. **Mitigation** The issue was first raised internally by our support staff based on internal monitoring systems and reports from users about delays in progress calculations and sync jobs. It took several hours for our engineers to become aware of the issue due to timezone differences. After becoming aware of the problem, an engineer noticed the scaling was not working correctly and manually scaled up the capacity of our background processing to attempt to deal with the backlog of jobs waiting to be processed. This was able to resolve the delay and allow our engineers to start investigating the root cause and apply a fix. **Cause of the issue** The issues were caused by a oversight in the logic that is responsible for ensuring the progress calculation happens efficiently, which caused progress to continuously be recalculated, resulting in a buildup of background jobs and corresponding delays. Our engineers developed an alternative solution and applied it the same day to solve the root cause. **Steps ahead** To prevent similar incidents from happening in the future, we are taking the following steps: 1. Separation of progress calculation and synchronization jobs: When things go wrong, the impact of the problem should be minimal. In this case, although the root cause did not involve synchronization jobs, they were still impacted by the delays. To prevent this in the future, we have isolated them from progress calculation. 2. More responsive scaling: Similarly, the impact of the problem could have been reduced with better scaling sooner. Our engineers are working on making this scaling mechanism more responsive to prevent our users from being impacted by these kinds of incidents. We are committed to providing reliable tools that scale with our users' needs, and will continue to improve our products as we encounter these kinds of challenges.