IncentFit experienced a critical incident on February 14, 2023 affecting Web Application and Mobile Application and 1 more component, lasting 1h 8m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Feb 14, 2023, 08:41 PM UTC
We've identified the issue and are deploying a fix.-
- monitoring Feb 14, 2023, 09:04 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Feb 14, 2023, 09:49 PM UTC
This incident has been resolved.
- postmortem Feb 14, 2023, 10:05 PM UTC
~2:30PM: A new release is deployed. It had many different changes to various bugs throughout the system, including some updates to a new feature. ~2:55pm: Release succeeds. 3:00pm: Operations team begins to notice general slowness throughout site. 3:04pm: Issue is escalated, 502's begin appearing on webapp ~3:10pm: Initial investigation determines that a bad database update was made. Problematic entry is removed servers are restarted. ~3:15pm The 502 Errors continue. 3:38pm: It is determined that a new piece of code that looks up data for the aforementioned new feature is slowing down the site. 3:40pm: New Deploy is sent. Some site functionality is restored. 4:16pm: We determine that some of this bad code is still live running in production. 4:18pm: We stop all processes for this code on all servers. 4:24pm: Deploy is successful