Splunk OnCall incident
We are currently investigating a problem regarding the performance of SMS and Voice notification delivery.
Splunk OnCall experienced a major incident on October 20, 2025 affecting Notifications - SMS and Notifications - Phone and 1 more component, lasting 17h 49m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 20, 2025, 08:40 AM UTC
Please bear with us while we troubleshoot this issue — updates to follow. Where possible, please adjust your personal paging policies to Push notification and email. If you are affected by this reporting feature, please reach out to On-Call Support. Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support
- identified Oct 20, 2025, 09:47 AM UTC
The issue has been identified and is related to an ongoing AWS incident affecting multiple services: https://health.aws.amazon.com/health/status Please continue to diversify your paging policies to incorporate Push and email. ------------- Current AWS status: Increased Error Rates and Latencies Oct 20 2:27 AM PDT We are seeing significant signs of recovery. Most requests should now be succeeding. We continue to work through a backlog of queued requests. We will continue to provide additional information. Oct 20 2:22 AM PDT We have applied initial mitigations and we are observing early signs of recovery for some impacted AWS Services. During this time, requests may continue to fail as we work toward full resolution. We recommend customers retry failed requests. While requests begin succeeding, there may be additional latency and some services will have a backlog of work to work through, which may take additional time to fully process. We will continue to provide updates as we have more information to share, or by 3:15 AM.
- identified Oct 20, 2025, 10:42 AM UTC
Our cloud telephony partner (SMS and Voice) is seeing partial recovery and is scaling their systems to handle the influx of traffic and processing of notifications. ----- AWS update: Increased Error Rates and Latencies Oct 20 3:35 AM PDT The underlying DNS issue has been fully mitigated, and most AWS Service operations are succeeding normally now. Some requests may be throttled as they work toward full resolution.
- monitoring Oct 20, 2025, 11:01 AM UTC
AWS fix has been implemented. We are seeing the successful delivery of SMS and Voice through our notification provider. However, there still may be a backlog of notifications to be processed. Moving incident status to Monitoring. Where possible, to avoid single points of failure, please take action to diversify your On-Call personal paging policies to include all forms of notification: SMS, Voice, Push, WhatsApp Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support
- monitoring Oct 20, 2025, 04:02 PM UTC
We are continuing to montior the situation. Currently SMS notifications are delivering as normal. However, Voice/Call notification are deeply backlogged and you may experience repeated calls for incidents that have already been actioned in your On-Call instance. We are working directly with our notification provider to help migitate this isse as soon as possible. Where possible, to avoid single points of failure, please take action to diversify your On-Call personal paging policies to include all forms of notification: SMS, Voice, Push, WhatsApp Thank you for your contnued patience while we work to resolve this issue. Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support
- monitoring Oct 20, 2025, 08:17 PM UTC
Call/Voice queues have now cleared and we are seeing voice notifications being delivered in real time again. Currently, all notification types are operational and working as expected. We are continuing to monitor notification rates and will provide further updates on this Status Page Incident as the ongoing AWS incident progresses. Thank you for your continued patience while we work to resolve this issue. Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support
- monitoring Oct 20, 2025, 09:57 PM UTC
We are continuing to monitor for any further issues.
- resolved Oct 21, 2025, 02:30 AM UTC
We are no longer experiencing any issues involving delayed or failed SMS or Phone call notification delivery. Please contact Splunk On-Call support if you encounter any further problems. Contact Us: https://help.splunk.com/en/splunk-enterprise/alert-and-respond/splunk-on-call/introduction-to-splunk-on-call/getting-started-guide-for-splunk-on-call-admins/contact-splunk-on-call-support