Splunk OnCall incident
VictorOps Portal Unavailable on Web and Mobile - Still Ingesting Alerts & Notifications
Splunk OnCall experienced a minor incident on July 2, 2019 affecting Web Client (Portal) and Android Client - Mobile App and 1 more component, lasting 1h 11m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jul 02, 2019, 02:00 PM UTC
The VictorOps portal is currently unavailable via web and mobile. We are ingesting alerts, processing incidents, and delivering notifications to users as normal. You can ack or resolve incidents by responding to phone call and SMS notifications. Cloudflare is facing network issues (which is in turn having downstream effects on those who utilize its services, including VictorOps) and is working towards a resolution. Please see https://www.cloudflarestatus.com/ We are actively working to resolve this issue as quickly as possible. Use the Subscribe to Updates option or follow @VOSupport on Twitter for updates. If you have any immediate questions or concerns, our support team is standing by to respond via email [email protected] or submit the form on our Contact Support page (https://victorops.com/contact-support/).
- identified Jul 02, 2019, 02:02 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Jul 02, 2019, 02:11 PM UTC
Portal access has been restored.
- resolved Jul 02, 2019, 03:12 PM UTC
We've been actively monitoring the implemented fix and the overall situation at the moment. Based on positive technical observations, we're moving this incident from Monitoring to Resolved. Thank you for your patience during this time. Again, we sincerely apologize for any inconvenience this issue may have caused. As a follow-up to this issue, our teams will be conducting a Post Incident Review and will follow-up with our findings on this StatusPage. If you have any immediate questions in reference to this incident, please don't hesitate to contact our Support Team with an email to [email protected] or submit the form on our Contact Support page (https://victorops.com/contact-support/).
- postmortem Jul 02, 2019, 08:12 PM UTC
We detected the issue due to the Cloudflare outage at 7:49am MDT \(UTC -6\). The VictorOps Technical Support Team was notified at 7:53am MDT. After verifying the issue and acknowledging Cloudflare’s status we created our StatusPage incident at 8am MDT. Cloudflare offered resolution at 8:11am MDT and VictorOps became fully operational. We remained in a monitoring state until 9:12am MDT, upon that time we resolved the incident. Please see the following link to the Cloudflare StatusPage for the 7/2/19 incident: [https://www.cloudflarestatus.com/incidents/9wjyx63y2xsy](https://www.cloudflarestatus.com/incidents/9wjyx63y2xsy) And please find the most recent explanation from Cloudflare here: [[https://blog.cloudflare.com/cloudflare-outage/](https://blog.cloudflare.com/cloudflare-outage/)](https://blog.cloudflare.com/cloudflare-outage) As always, we welcome and encourage you to reach out to the VictorOps Support team with any questions: [[email protected]](mailto:[email protected])