IncentFit incident

502/504 Errors on Web and Mobile

Resolved after 3 hours, affecting Web Application and Mobile Application, on November 2, 2022.

postmortem Nov 07, 2022, 04:10 PM UTC

Our team has analyzed the outage on 11/2/2022 and determined the root cause. We have also determined and implemented a solution to prevent this occurrence in the future. IncentFit uses AWS’s auto-scaling architecture to dynamically scale our systems based on user demand on our site and application. On 11/2 we had an extremely brief, but high spike in traffic and the system that automatically scales the servers did not properly handle it. Our team responded within minutes to the problem; unfortunately it took about 2 hours to fully diagnose the problem and an additional hour to fix it. Our Product team has already implemented a solution to the automatic scaling logic so that if this type of activity were to happen in the future, it would not affect user experience.

Started
Nov 02, 2022, 02:32 PM UTC
Resolved
Nov 02, 2022, 05:33 PM UTC
Duration
3h
Detected by Pingoru
Nov 02, 2022, 02:32 PM UTC

Affected components

Web ApplicationMobile Application

Update timeline

  1. investigating Nov 02, 2022, 02:32 PM UTC

    We are currently investigating this issue. Users may experience increased loading times on website and app, or be unable to access the web portal or app. If you cannot access the mobile app for a gym check-in please email our support team for further assistance. All other fitness tracker data will be imported once this issue is resolved. Please subscribe for updates.

  2. resolved Nov 02, 2022, 05:33 PM UTC

    This incident has been resolved.

  3. postmortem Nov 07, 2022, 04:10 PM UTC

    Our team has analyzed the outage on 11/2/2022 and determined the root cause. We have also determined and implemented a solution to prevent this occurrence in the future. IncentFit uses AWS’s auto-scaling architecture to dynamically scale our systems based on user demand on our site and application. On 11/2 we had an extremely brief, but high spike in traffic and the system that automatically scales the servers did not properly handle it. Our team responded within minutes to the problem; unfortunately it took about 2 hours to fully diagnose the problem and an additional hour to fix it. Our Product team has already implemented a solution to the automatic scaling logic so that if this type of activity were to happen in the future, it would not affect user experience.

Your product runs on more than IncentFit

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →