Services not responding

Update timeline

1. investigating Nov 05, 2024, 10:56 PM UTC

   🚨 We’re currently investigating an issue causing Intuto to be unresponsive. Our team is working to identify the cause, and we appreciate your patience while we resolve it. Thank you for bearing with us! 🙏

2. investigating Nov 05, 2024, 11:09 PM UTC

   ✅ Our services are now back online and responding to our monitoring systems. We are still investigating the root cause of the outage to ensure stability.

3. resolved Nov 06, 2024, 02:51 AM UTC

   🔄 We’re confident the issue has been resolved, and are actively working with our supplier to identify the root cause. We continue to closely monitor the application, and will provide a post-mortem in the coming days.

4. postmortem Nov 21, 2024, 10:22 PM UTC

   ### **Postmortem: Services not responding** **Incident 1 Date:** 06/11/2024, 11:56 AM **Duration:** 12 minutes **Incident 2 Date:** 06/11/2024, 07:40 PM **Duration:** 10 minutes **Summary:** On 06/11/2024 at 11:56am, our monitoring software notified us that the Intuto site was not responding. Our team began investigating and found that we had lost all access to our server infrastructure, and that none of our hosted sites were responding to requests. We immediately contacted our infrastructure provider, and while trying to pinpoint the issue, the systems bought themselves back online, and our monitoring systems reported that all systems were running. Our provider continued to investigate the issue, without seeing any obvious issues in their reporting. This outage occurred for 12 minutes in total. At 7:40PM on the same day, our monitoring systems alerted us that the issue was occurring again. We resumed contact with our infrastructure provider, who continued to investigate. The second outage occurred for a total of 8 minutes. Our provider continued investigations into the night. Noting below > Investigation on customer VDOMs showed no issues. BGP and routing have been up for over 2 weeks \(since last FlexVM FortiGate upgrade\). No system, HA or routing error logs on the customer VDOMs. IaaS platform investigation also showing no errors or issues, including physical switches. At 12:30am on 07/11/2024, our provider opened a ticket with Fortinet support. At 9:55am Fortinet support confirmed that there was a global issue with FortiGuard servers causing FlexVM FortiGate firewalls to be unable to validate licenses from FortiGuard servers. This was resolved by the Fortinet support team at 03:00 NZDT. **Root Cause:** FlexVM FortiGate firewalls were unable to validate licenses from FortiGuard servers for a period of up to 22 minutes on multiple occasions. FlexVM FortiGate firewalls are designed to drop traffic when they are unable to validate the license from FortiGuard servers. This caused an outage for all affected customer services passing through the FortiGate for multiple 10-20 minute periods. **Impact:** The outage directly affected the availability of all services hosted on the impacted infrastructure, rendering them inaccessible to end users for a combined total of 20 minutes across two incidents. While the issue was resolved without any loss of data or lasting impact, the incidents underscore the dependency on third-party infrastructure and the cascading effects of their global issues. **Conclusion:** The incident was caused by a global issue with Fortinet’s FortiGuard servers that prevented the FlexVM FortiGate firewalls from validating their licenses. This highlighted a critical single point of failure in the system’s dependency on FortiGuard servers for firewall operation. We are confident this was a one-off incident. We apologize for any inconvenience caused and remain committed to ensuring the stability and performance of our system. If you have any questions or concerns, please don’t hesitate to [reach out to our support team](mailto:[email protected]). **The Intuto Team**