Viirtue experienced a minor incident on August 19, 2024 affecting Manager Portal, lasting 1d 22h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 19, 2024, 02:54 PM UTC
We are aware of an issue affecting a segment of call history data in the portal. Engineers are investigating the issue, updates to follow.
- identified Aug 19, 2024, 03:55 PM UTC
The affected date ranges for missing call history data are August 3rd to August 16th. Engineers are working on restoring the missing data now. Updates to follow.
- identified Aug 19, 2024, 06:05 PM UTC
Engineers have isolated the problem tables in the database and are working on replicating the data from the other cores. Updates to follow.
- identified Aug 19, 2024, 08:55 PM UTC
Engineers have isolated the problem tables in the database and are working on replicating the data from the other cores. Please reach out to support if you need to grab an urgent recording. Updates to follow.
- identified Aug 20, 2024, 01:07 PM UTC
Engineers have isolated the problem tables in the database and will make changes to address the issue during a maintenance window tonight. Please reach out to support if you need to grab an urgent recording. Updates to follow.
- identified Aug 20, 2024, 09:06 PM UTC
Engineers have isolated the problem tables in the database and will make changes to address the issue during a maintenance window tonight. Please reach out to support if you need to grab an urgent recording. Updates to follow.
- monitoring Aug 21, 2024, 04:05 AM UTC
Engineers are currently migrating the lost data to Core2-NJ. Call history data should start to appear in the portal over the next few hours. Updates to follow.
- resolved Aug 21, 2024, 01:48 PM UTC
This incident has been resolved.
- postmortem Aug 21, 2024, 01:48 PM UTC
While Engineers restored a large portion of the missing data in the call history tab, call traces for calls involving core2-nj were unable to be restored. Any calls that originated or terminated through core2-nj before August 17th will not have a call trace. Calls that took place on the Florida and Las Vegas cores will have call traces before the 17th. We apologize for the inconvenience.