IronTree Network Outage

Update timeline

1. identified Dec 02, 2024, 08:22 AM UTC

   We have identified an issue similar to last week, we have deployed Technicians to the DC. All services currently unavailable.

2. identified Dec 02, 2024, 09:12 AM UTC

   We have got the system operational again, however we are adding additional redundancy at the Data Centre. This was scheduled for Thursday, but due to this issue persisting we will be doing it now.

3. monitoring Dec 02, 2024, 09:19 AM UTC

   We are busy monitoring the system, while we add more redundancy. Connections may be intermittent while we do the change over.

4. monitoring Dec 02, 2024, 09:44 AM UTC

   We have replaced the affected cables and added more redundancy. RCA will be sent out this week.

5. identified Dec 02, 2024, 12:30 PM UTC

   After replacing the Cables all systems were operational, systems have gone down again and we are sending technicians back to the Data Centre. This is our top priority.

6. identified Dec 02, 2024, 12:56 PM UTC

   Technicians are onsite and busy with the issue.

7. identified Dec 02, 2024, 01:23 PM UTC

   We are working on the network actively to get services operational.

8. monitoring Dec 02, 2024, 01:49 PM UTC

   We are busy testing, currently systems are up.

9. monitoring Dec 02, 2024, 01:53 PM UTC

   Busy rebooting switches

10. monitoring Dec 02, 2024, 02:08 PM UTC

    Systems are back up, we are monitoring

11. monitoring Dec 02, 2024, 03:25 PM UTC

    There will be intermittent connectivity while we test redundancy.

12. monitoring Dec 02, 2024, 05:08 PM UTC

    Various network changes have been made to help prevent this issue going forward. System's are fully functional at the moment, we will continue monitoring and have technicians onsite at the Data Centre in the morning. We sincerely apologise for the affect this has had on your business operations.

13. monitoring Dec 06, 2024, 02:57 PM UTC

    We will continue to monitor our systems over the weekend. All systems are currently stable.

14. resolved Dec 10, 2024, 10:11 AM UTC

    We have been monitoring this incident for 8 days and the systems have remained stable. We are resolving this incident for now. Please see link to RCA: https://everfiles.metrofilecloud.com/public-link/browse/kvXvJ4EIlH