Adeptcore experienced a critical incident on May 28, 2025 affecting ACP - Network, lasting 22h 32m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 28, 2025, 04:14 PM UTC
We have received reports of VM's becoming unresponsive and are working to identify and resolve this issue with datacenter engineers as quickly as possible. We will be posting updates as we get them.
- identified May 28, 2025, 04:30 PM UTC
We have identified the issue and are working with the datacenter to restore networking services.
- identified May 28, 2025, 04:41 PM UTC
We are seeing networking being restored at this time and are waiting for everything to finish coming back to begin verifying functionality.
- identified May 28, 2025, 05:30 PM UTC
We are actively investigating this issue. Currently, we’re experiencing difficulties re-establishing communication between the hosts and storage devices. Our team is working closely with the datacenter to resolve this as swiftly as possible.
- identified May 28, 2025, 06:45 PM UTC
The datacenter team has resolved all outstanding issues on their end. We have also resolved an additional issue and are in the process of bringing the affected virtual machines back online. As virtual machines come back online and are confirmed to be operational, we will be issuing another status update.
- identified May 28, 2025, 07:45 PM UTC
All services are in the process of being restored. Virtual machines should be coming up and a portion of them are already up. Networking issues have all be resolved, part of this issue was internal networking between management and production clusters and our storage network. The team is engaged on bringing up all servers in the proper order and confirming logins.
- monitoring May 28, 2025, 09:26 PM UTC
At this time, all services appear to have been restored and we have received no further reports of ongoing issues. If you or your clients are still continuing to experience any issues, please submit a support ticket or reach out to us via phone to receive support on a case by case basis.
- resolved May 29, 2025, 02:47 PM UTC
We have continued to monitor the environment through the evening into this morning and have confirmed that no further issues are occurring. We will be marking this incident as resolved at this time. We expect to provide a postmortem and additional details in regards to this outage in the next two weeks.