Intuto experienced a major incident on July 31, 2024 affecting API V2 and Login, Signup, and Single Sign-On (SSO) and 1 more component, lasting 3h 7m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 31, 2024, 09:15 PM UTC
We have identified an upstream provider is causing an outage on Intuto. Long running tasks, such as publishing courses, and background jobs such as issuing certificates and reminder emails are affected. We are monitoring the issue with the provider and will update customers when we know more
- investigating Jul 31, 2024, 09:23 PM UTC
Our upstream provider has identified they are experiencing issues related to their Microsoft 365 services. This can be followed on the Microsoft Status Page https://x.com/MSFT365Status
- monitoring Jul 31, 2024, 09:51 PM UTC
This incident seems to have been resolved. We are awaiting a notification from our provider
- resolved Aug 01, 2024, 12:23 AM UTC
This incident has been resolved.
- postmortem Aug 12, 2024, 09:30 PM UTC
### **Postmortem: Loss of Connectivity to Microsoft Services** **Incident Date:** 01/08/2024, 8:23 AM **Duration:** 1 hour 20 minutes **Summary:** On 01/08/2024, our monitoring systems detected an outage, which we quickly identified as a network-related issue. Upon investigation, Intuto immediately contacted our infrastructure provider. They informed us that they had been receiving reports of connectivity problems from customers since 8:23 AM. The issue specifically impacted network traffic to Microsoft services and resulted in a nationwide outage across New Zealand. To mitigate the disruption, our infrastructure provider established a temporary communication bridge at 9:43 AM, rerouting all Microsoft traffic through Australia. This workaround allowed us to bypass the issue while Microsoft investigated the root cause. By 12:35 PM, Microsoft had successfully re-routed all traffic through alternative paths and devices, fully restoring services. **Root Cause:** Microsoft have provided the following root cause summary: A Microsoft-managed networking device within the Auckland peering location experienced a hardware issue. As a result, this device started to drop network packets received from external service providers and Azure ExpressRoute circuits. Microsoft are currently working with the networking device manufacturer to isolate the cause of the hardware issue. **Impact:** The outage affected some customers' ability to access our services, though the impact varied across our user base. This disruption lasted for 1 hour and 20 minutes until our provider implemented the temporary communication bridge. **Conclusion:** This incident highlights the critical need for resilient infrastructure and swift response measures. While the impact was not uniform, we understand the inconvenience it caused and are dedicated to ongoing improvement. We are actively working with our infrastructure provider and Microsoft to prevent similar issues in the future. Our goal is to ensure a reliable service experience, and we thank you for your patience during this incident. If you have any questions or concerns, please don’t hesitate to [reach out to our support team](mailto:[email protected]). **The Intuto Team**