BitLaunch experienced a critical incident on March 13, 2023 affecting AMS1, lasting 15h 53m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 13, 2023, 11:06 PM UTC
Engineers are currently looking into this issue.
- identified Mar 13, 2023, 11:35 PM UTC
The DC is performing emergency power maintenance. Unfortunately this was not scheduled. We will continue to update as we receive them
- identified Mar 14, 2023, 12:59 AM UTC
Emergency power maintenance is still on-going. We apologise for this inconvenience.
- identified Mar 14, 2023, 08:58 AM UTC
Emergency power maintenance is still on-going. We are seeking an ETA on resolution. We sincerely apologise for the inconvenience this has caused.
- monitoring Mar 14, 2023, 10:21 AM UTC
Network connectivity has been restored in the DC. We will be monitoring for further issues.
- investigating Mar 14, 2023, 12:15 PM UTC
We are investigating a secondary outage which we believe to be on-going power issues.
- identified Mar 14, 2023, 01:04 PM UTC
Datacenter and Upstream engineers are working on this issue as priority. We will continue to post updates as we get them.
- monitoring Mar 14, 2023, 02:10 PM UTC
Upstream networking engineers have completed their fix. We will continue to monitor until we mark this issue as resolved.
- resolved Mar 14, 2023, 02:59 PM UTC
Connectivity has been restored in this region. A post mortem will follow. We apologize for the inconvenience caused.
- postmortem Mar 14, 2023, 03:00 PM UTC
We would like to provide a more detailed explanation of the recent networking outage that affected our services and customers. Our network experienced a significant outage due to a sudden power loss at the Amsterdam datacenter. This caused our upstream providers network infrastructure to go down. Our hardware remained operational during this time. Once power was restored, this caused a scenario which unfortunately led to a split-brain issue on upstream routing engines. This continued to cause issues with the connectivity of our services. Engineers worked to get this resolved as soon as possible. We appreciate your patience during this time. If you continue to experience any issues, please open a support ticket. Thank you.