Voxtelesys incident
Investigating SMS and Portal Connectivity Issues
Voxtelesys experienced a minor incident on February 27, 2024 affecting SMS Messaging and MMS Messaging and 1 more component, lasting 1h 41m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 27, 2024, 08:31 PM UTC
Voxtelesys NOC has received reports of intermittent issues with SMS and reaching the portal. Voxtelesys network engineering is currently investigating.
- monitoring Feb 27, 2024, 08:52 PM UTC
A temporary fix has been implemented. If you are still seeing issues, please flush any DNS resolver cache. If the issue persists please contact [email protected].
- resolved Feb 27, 2024, 10:13 PM UTC
At 14:16 CST, DNS Records that had been published to Cloudflare for several months were mysteriously removed, as indicated by the logs provided by Cloudflare. At 14:17:22 CST, Voxtelesys monitoring systems detected Origin Errors being reported by Cloudflare across various API endpoints and portal.voxtelesys.net. At 14:33 CST, Voxtelesys engineering pushed a mass DNS record change to the voxtelesys.net domain. By 14:36 CST, the DNS change had propagated to mainstream public DNS resolvers. This entire event lasted 20 minutes. We have engaged Cloudflare's Enterprise Support and are awaiting a much-needed explanation to the logs provided to us. We apologize for the inconvenience and will continue to monitor closely for any further issues.