MX Connect experienced a major incident on October 19, 2023 affecting MX Connect, lasting 22h 39m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 19, 2023, 04:31 PM UTC
We are investigating reports of latency with MX Connect (www.mxconnect.com). We are actively researching and will provide more information shortly.
- investigating Oct 19, 2023, 04:36 PM UTC
We are continuing to investigate this issue.
- investigating Oct 19, 2023, 06:00 PM UTC
We are still continuing to investigate this issue. Thank you for your patience
- investigating Oct 19, 2023, 06:09 PM UTC
cluster service is failing. Attempts are being made to restore the cluster service
- investigating Oct 19, 2023, 06:38 PM UTC
team has raised the incident with Microsoft and in contact with a technical advisor to find a solution
- investigating Oct 19, 2023, 07:06 PM UTC
We are in the process of conducting ongoing investigations and attempting to implement potential fixes. Thank you for continued patience.
- investigating Oct 19, 2023, 07:46 PM UTC
working on a fix, ems is being uninstalled from all the sql nodes
- investigating Oct 19, 2023, 08:04 PM UTC
cluster config was corrupted, team is attempting to recover backup
- investigating Oct 19, 2023, 08:49 PM UTC
We are continuing to investigate this issue. Still attempting to restore Atlanta cluster
- identified Oct 19, 2023, 08:50 PM UTC
Our dedicated technology team is still actively engaged in ongoing efforts to implement a solution.
- identified Oct 19, 2023, 09:38 PM UTC
Our dedicated technology team is still actively engaged in ongoing efforts to implement a solution. team is still trying to restore databases with recovery
- monitoring Oct 19, 2023, 11:19 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Oct 20, 2023, 05:55 AM UTC
Services have been restored but are still being monitored for further issues
- resolved Oct 20, 2023, 04:39 PM UTC
This incident has been resolved.