MB connect line experienced a major incident on April 30, 2024 affecting mbCONNECT24 Europe V2, lasting 7h 34m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 30, 2024, 04:30 AM UTC
We are experiencing temporary connectivity issues with our system. The R&D team is informed and the situation is being investigated. Users of the system might have problems connecting to the system. We will keep you updated
- monitoring Apr 30, 2024, 07:45 AM UTC
A fix has been implemented and we are monitoring the results. The devices are currently reconnecting to the portal.
- resolved Apr 30, 2024, 12:04 PM UTC
This incident has been resolved.
- postmortem Apr 30, 2024, 12:32 PM UTC
# Timeline of events * Around 01:00am CEST our monitoring system started recording devices not being able to reconnect to the server when they lost connection. * Early morning, about 06:30am CEST, investigation started by the technical team. At that time, only around 40% of devices were still connected. * As the connections kept dropping, a reboot was initiated around 07:30am CEST. * This reboot did not solve the issues, and our technical team kept searching for the root cause. * The problem was found in an unoptimized configured service and hot fixed at around 08:20am CEST. * A second reboot was initiated at 08:30am CEST. * Devices restored their connections over the next hours. * At around 1:30pm CEST normal operation was restored. # Impact This issue was confined to the RL/mbCONNECT24 EU V2 Server. # Follow-up actions Possible side effects of the hot fix are being tested and monitored. Afterward, the changes will be implemented into the next RSP release.