CloudFactory experienced a major incident on April 17, 2025 affecting ICEcloud, lasting 13h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 17, 2025, 05:17 PM UTC
The front end webserver is down.
- monitoring Apr 17, 2025, 05:23 PM UTC
In some regions the apps are live again.
- monitoring Apr 17, 2025, 05:36 PM UTC
The applications on the tablets are live. Icecloud.be is still down but we are keeping this state until the ice cream vans end their day shift. We'll continue working after 22:00.
- monitoring Apr 17, 2025, 07:44 PM UTC
A few systemfiles of the front-end server got corrupted. We were only able to repair a part of the system. We are creating a new server and recovering backups this night to avoid problems in the near future. There will be no loss of data, the data is secured on separate servers with duplicates in another datacenter.
- monitoring Apr 17, 2025, 09:39 PM UTC
The new server is up and running. We are restoring backups and bringing all websites back up one by one.
- monitoring Apr 18, 2025, 05:44 AM UTC
The backup is completed. We are checking the systems to see if everything has recovered successfully.
- resolved Apr 18, 2025, 06:18 AM UTC
This incident has been resolved.