IncentFit experienced a major incident on July 23, 2019 affecting Activity Rewards & Reimbusements and Mobile Application, lasting 1d 19h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 23, 2019, 10:07 PM UTC
We are currently investigating reports of users experiencing issues loading portions of the version 3.9.3 of the iOS mobile app. Users may open the menu and not see menu items, or may not see the map when trying to check into a gym. We also have received reports of users being unable to compete check-ins due to these issues as the mobile app may time out after portions of it do not load. These issues are limited so far to iOS users on 3.9.3
- investigating Jul 24, 2019, 03:26 PM UTC
We are continuing to work on this issue and it is our top priority. We have currently identified 2 separate, but related, issues on the newest version of the app (3.9.3) on iOS *only*. All Android users are unaffected. The issues are: - Users are unable to access sub-menu items like Challenges, or other pages for recording activities, including geolocation check-ins - Users will see portions of the app not render (images do not load) or images and menu items are slow to load For all of these cases, users should still be able to access the web portal and record activities there. All activities and earnings are saved in our system and this issue does not affect prior earnings. If employees have had issues checking into their gym they can contact our support team for courtesy credit at support(at)incentfit(dot)com. We will post another update shortly when the issue has been fully identified. We expect this to be resolved by end of day.
- identified Jul 24, 2019, 04:53 PM UTC
The issue has been identified and we expect a resolution by end of day.
- identified Jul 24, 2019, 07:57 PM UTC
A updated app which will resolve all issues has been submitted to Apple and is pending approval by Apple App Store. We've asked for expedited review by the Apple team and are hopeful they will review very soon.
- monitoring Jul 24, 2019, 11:55 PM UTC
A new app update has been released. Users may now reinstall the iOS app from the Apple store to receive the latest update (3.9.4). We expect this should resolve all issues and will continue to monitor the situation.
- resolved Jul 25, 2019, 06:03 PM UTC
We have confirmed that the latest version of the iOS app (3.9.4) resolves all outstanding issues, including: - login issues for first-time and returning users - geolocation issues - timeout issues - loading issues on certain pages - functionality issues on sub-menus (challenges, add activity, etc.) Our team will be publishing a postmortem shortly.
- postmortem Jul 26, 2019, 06:37 PM UTC
## Summary of Incident Version 3.9.3 of the iOS app shipped with various errors breaking basic functionality. These included: * Non-functional buttons * Geolocation and QR code scanning issues * Missing graphical assets * Occasional timeout errors * Missing/delayed menu loading * SSO failures The functionality and add activity issues affected all users who upgraded to version 3.9.3 of the iOS app or who installed IncentFit for the first time. SSO issues affected users on all iOS versions of the app whose organizations use SSO for authentication or who were logged out of the app and tried to log back in. Users who did not update or did not log out of the iOS app would not have seen these SSO issues. ## Root Causes In line with our release policies, our development team completed significant testing on Android and Apple versions of the app prior to the initial release of 3.9.3. Upon submission of the Apple version of the app, Apple notified us that a change in their policies required us to upgrade one of our app dependencies. They also prohibited us from re-leasing \(or rolling back to\) a previous version of the app since prior versions also conflicted with their new policy. We updated this dependency in accord with the policy and resubmitted for approval. The root cause of the issue was our failure to complete a full-fledged round of testing when Apple requested that we upgrade this dependency. In general, this secondary round of testing is not something we do unless we change portions of the code. Since this was a dependency update, versus a code alteration, we did not complete a full set of tests after the dependency change. As a result, we did not catch that navigation and other issues were non-functional prior to release. Normally we immediately release a prior and working version of the app when we discover large scale issues, but could not in this case because of the aforementioned prohibition from Apple on rolling back. ## Learnings/Next Steps Moving forward we will change our policies and procedures to ensure that the integration test plan is completed prior to every release; even if the only changes to the app are small, or seemingly innocuous ones like a dependency upgrade. Moreover, we will improve our integration test plan itself to check for more scenarios, including SSO logins using client test credentials.