Resolved after 1 hour 41 minutes, affecting SMS Messaging and MMS Messaging and 1 more, on February 27, 2024.
At 14:16 CST, DNS Records that had been published to Cloudflare for several months were mysteriously removed, as indicated by the logs provided by Cloudflare. At 14:17:22 CST, Voxtelesys monitoring systems detected Origin Errors being reported by Cloudflare across various API endpoints and portal.voxtelesys.net. At 14:33 CST, Voxtelesys engineering pushed a mass DNS record change to the voxtelesys.net domain. By 14:36 CST, the DNS change had propagated to mainstream public DNS resolvers. This entire event lasted 20 minutes. We have engaged Cloudflare's Enterprise Support and are awaiting a much-needed explanation to the logs provided to us. We apologize for the inconvenience and will continue to monitor closely for any further issues.
Affected components
Update timeline
- investigating Feb 27, 2024, 08:31 PM UTC
Voxtelesys NOC has received reports of intermittent issues with SMS and reaching the portal. Voxtelesys network engineering is currently investigating.
- monitoring Feb 27, 2024, 08:52 PM UTC
A temporary fix has been implemented. If you are still seeing issues, please flush any DNS resolver cache. If the issue persists please contact [email protected].
- resolved Feb 27, 2024, 10:13 PM UTC
At 14:16 CST, DNS Records that had been published to Cloudflare for several months were mysteriously removed, as indicated by the logs provided by Cloudflare. At 14:17:22 CST, Voxtelesys monitoring systems detected Origin Errors being reported by Cloudflare across various API endpoints and portal.voxtelesys.net. At 14:33 CST, Voxtelesys engineering pushed a mass DNS record change to the voxtelesys.net domain. By 14:36 CST, the DNS change had propagated to mainstream public DNS resolvers. This entire event lasted 20 minutes. We have engaged Cloudflare's Enterprise Support and are awaiting a much-needed explanation to the logs provided to us. We apologize for the inconvenience and will continue to monitor closely for any further issues.
Your product runs on more than Voxtelesys
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services