Virtuagym experienced a major incident on April 1, 2022 affecting Website and Mobile Apps and 1 more component, lasting 56d 3h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 01, 2022, 10:16 AM UTC
We're currently investigating a credit assignment issue where, in some cases, the credit validity of the credits handed out for March was increased and the next credit handout date was also changed to the same date as the validity. The cause of the incident is being investigated by the entire credit team so that we get to a solution as quickly as possible. We will continue to update the status page throughout. Our apologies for the inconvenience.
- investigating Apr 01, 2022, 03:53 PM UTC
We continue with our investigation phase and, although we weren't able to completely identify the root cause yet, we are working on identifying all the cases that were affected by this issue so that we can issue a correction for those (both for the incorrect credit handouts in the past as well as for the missed handouts). Although rare, some cases might require a manual credit assignment of the affected credit type on the client's profile (if you resort to this remediation measure you should keep track of those situations so that you can correct the credit balance once we issue the fix). Next week we'll update you with the next steps to pursue. Again, we apologize for the inconvenience this occurrence might bring you.
- investigating Apr 05, 2022, 03:18 PM UTC
At this point we tested a correction for a single member that we identified manually and the fix for this member alone was successful. Now we're proceeding to fully automate the identification of all the cases that were affected by this issue so that once we deploy the fix it can cover all cases.
- identified Apr 08, 2022, 03:27 PM UTC
At this point we finalized a script that allows us to calculate what should be the credit assignment/handout moments in any given membership. Additionally, after some improvements we have a first version of a way to compare those instances with the ones that actually happened (credit assignments) to track the differences between them (regarding frequency, validity and if the credit assignment happened or not) and we will test it in one full club to check if we get the expected results. This will help us to understand if we still need to make some improvements on this matching script. After this is done we’ll be able to move significantly quicker on the actual corrections as we already made progress on that part previously as we reported. We’ll give more updates beginning of next week about the status of the described steps. Thanks once again for your patience.
- identified Apr 12, 2022, 03:43 PM UTC
We continue to work on this issue, which is causing some of our customers a lot of inconvenience. For more clarity we'll describe the steps being taken on this process: 1. Independently monitor the assignment process (separate program) 2. Compare credit assignment results with what our independent monitoring expects (theoretical credit assignment for a membership) 3. When differences are identified we flag them for review. Flagged differences are added to the monitor if they are correct (they are actual differences) 4. Repeat steps 2-3 until a given club has no more differences flagged 5. At that point we turn on the modification feature on our independent monitor, which effectively corrects the issue. At this point we stand on step 4, close to its conclusion, after which we'll move to step 5 and bringing this long issue to a close.
- identified Apr 13, 2022, 04:12 PM UTC
We are currently running a process that identifies all the incorrect/missing credit assignments and we'll get the results tomorrow morning. After that, we'll start implementing the fix that we expect to be concluded by end of day on Friday (15/04).
- monitoring Apr 15, 2022, 06:26 PM UTC
A fix has been implemented and we have confirmed that the identified cases were corrected. The credit assignments that had incorrect dates were corrected and the missing assignments were added with the correct credit validity dates as a new row in the applicable credit overview table with today's date (15/04/2022) as the creation date. We'll continue some tests to identify other cases that could have been missed by the identification process so that we can also correct those. We don't expect any additional issues in the future. We apologize for the inconvenience that this issue might have caused.
- resolved May 27, 2022, 01:24 PM UTC
We have monitored the previous cases and have verified that they were fully resolved. If you still have difficulties related to this issue please reach out to us.