Viirtue experienced a notice incident on December 2, 2024 affecting ViiBE, lasting 7h 21m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Dec 02, 2024, 09:27 PM UTC
The Netsapiens PBX Usage Report which is accessible through ViiBE is missing both seat counts and mobile connect counts. We will trigger a reprocessing this evening to ensure those values are populated. We will follow up with more details once the data has been populated. Existing data for November may be inaccessible until the process completes. Updates to follow.
- resolved Dec 03, 2024, 04:49 AM UTC
The Netsapiens PBX Usage Report for the month of November 2024 should now how all values populated correctly. We thank you for your patience.
- postmortem Dec 03, 2024, 04:49 AM UTC
**Incident Overview:** Following recovery efforts from Hurricane Milton, a database restoration inadvertently overwrote a critical configuration required for the NetSapiens PBX Usage Report. **Root Cause Analysis:** The restoration process performed by our technology partner did not include steps to preserve or verify the specific configuration relied upon by the NetSapiens PBX Usage Report. This oversight resulted in the configuration being overwritten during the recovery. **Actions Taken:** 1. **Collaboration with Technology Partner:** * The incident was reviewed with our technology partner to ensure mutual understanding of the issue and its impact. 2. **Update to Standard Operating Procedures \(SOPs\):** * The technology partner has updated their SOPs to include a specific step for preserving and restoring the NetSapiens PBX Usage Report configuration during any future database restoration. 3. **Confirmation of SOP Updates:** * We have received confirmation from the technology partner that these SOP updates have been implemented effectively. **Future Mitigation:** In the unlikely event that a similar database restoration is required, the updated SOPs ensure that the configuration critical to the NetSapiens PBX Usage Report will not be missed, significantly reducing the risk of recurrence. This incident underscores the importance of robust documentation and verification in restoration procedures, and we are confident that the corrective actions taken will prevent similar issues in the future.