Splunk OnCall incident
Our notification provider is experiencing US Short code SMS delivery delays
Splunk OnCall experienced a minor incident on May 4, 2024 affecting Notifications - SMS, lasting 1d 15h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring May 04, 2024, 09:59 PM UTC
The issue has been identified. Our provider is working with carrier partners to resolve the issue. In the meantime, please diversify your paging policies to include WhatsApp, Voice (phone call), and Push notifications to avoid a single point of failure. Documentation is found here: Paging Policies: https://help.victorops.com/knowledge-base/paging-policy-setup/ Notification Types: https://help.victorops.com/knowledge-base/notification-types/ We apologize for this inconvenience. Please reach out to On-Call Support with any questions! https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/
- monitoring May 05, 2024, 12:59 AM UTC
Update: This issue affects shortcode SMS to the domestic US (AMER). Our provider continues to work with carrier partners to resolve this issue. You may track the progress toward the resolution of this issue here: https://status.twilio.com/incidents/p6jvzrw06911
- monitoring May 05, 2024, 06:02 PM UTC
The Splunk On-Call notification provider continues to experience SMS delivery delays when sending messages to the Proximus Mobility network in the US (AMER) over a subset of shortcodes. Our provider is working with the carrier partner to resolve the issue. Splunk On-Call will continue to monitor and provide updates as soon as more information becomes available. As mentioned, please diversify your paging policies to include WhatsApp, Voice (phone call), and Push notifications to avoid single points of notification failure. Documentation is found here: Paging Policies: https://help.victorops.com/knowledge-base/paging-policy-setup/ Notification Types: https://help.victorops.com/knowledge-base/notification-types/ We apologize for this inconvenience. Please reach out to On-Call Support with any questions! https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/
- resolved May 06, 2024, 01:33 PM UTC
Our provider is no longer experiencing SMS delivery delays when sending messages to the Proximus Mobility network in the US (AMER) over a subset of shortcodes. This incident has been resolved. All systems are operational. If you're still experiencing issues with SMS please reach out to On-Call Support: https://help.victorops.com/knowledge-base/how-to-contact-splunk-on-call-support/ We apologize for this inconvenience.