Gurobi Optimization experienced a major incident on February 1, 2021 affecting Gurobi Cloud Billing, lasting 18m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 01, 2021, 05:50 AM UTC
New license sessions cannot be started. We are investigating.
- resolved Feb 01, 2021, 06:08 AM UTC
The license key system is fixed, and all systems are back to normal.
- postmortem Feb 01, 2021, 10:34 PM UTC
Late on Sunday January 31, we received several reports of failures to start Gurobi cloud sessions. Since these were urgent, two of Gurobi’s most senior people investigated immediately, even though it was late in the day for them. The root cause was the tool that generates license key files on the Gurobi website. This is used both by traditional Gurobi licenses and for Gurobi cloud sessions. For reasons of reliability and security, this tool must be upgraded periodically. We recently made major upgrades to the server that issues license keys, and in the process, we inadvertently replaced this license key generation program with an old version. Once we installed the latest version on the server, all systems returned to normal. We have a system in place to monitor this tool and notify us when the replacement date is approaching. However, this also failed. Later this week, we will investigate the monitoring system to prevent this type of failure in the future. The recent server upgrades were designed to _increase_ reliability, and it is unfortunate that it triggered this outage. We apologize for any inconvenience. Fixing the monitoring system should avoid a similar failure in the future.