Outage in Chicago

Update timeline

1. investigating Jan 14, 2024, 11:44 PM UTC

   We are currently investigating this issue.

2. identified Jan 15, 2024, 12:20 AM UTC

   An issue has arisen at the datacenter that houses hardware in this location. Technicians are currently working to bring servers back online.

3. identified Jan 15, 2024, 01:24 AM UTC

   We're still awaiting further information from the Datacenter about the exact cause of this issue and next course of action. We have no reason to believe that customer data is impacted and expect servers to be online as normal once the issue is resolved.

4. monitoring Jan 15, 2024, 07:08 AM UTC

   Servers are back online. Currently work is continuing at the datacenter to improve stability. Details to follow.

5. monitoring Jan 15, 2024, 09:09 AM UTC

   Severe weather conditions in the Chicago area contributed to the initial failure of HVAC system within the datacenter. This resulted in a temperature rise causing unsafe operating conditions for hardware situated within. DC Critical Facility Engineers are deploying all available resources, including floor fans and additional cooling equipment, to mitigate the temperature rise. Our services are still operational at this point in time and we expect this to be fully resolved in the coming hours.

6. resolved Jan 16, 2024, 12:59 PM UTC

   This incident has been resolved.

7. postmortem Jan 16, 2024, 01:01 PM UTC

   Severe weather conditions in the Chicago area contributed to the initial failure of HVAC system within the datacenter. This resulted in a temperature rise causing unsafe operating conditions for hardware situated within. DC Critical Facility Engineers worked to deploy all available resources, including floor fans and additional cooling equipment, to mitigate the temperature rise, resulting in our equipment being powered on several hours after initial shutdown.