Concentric down - Google Cloud outage (cooling related failure)

Update timeline

1. investigating Jul 19, 2022, 04:36 PM UTC

   The Concentric application is currently down. This appears to be an issue with GCP but we are investigating and will update when we better understand the issue or when the issue is resolved.

2. investigating Jul 19, 2022, 04:49 PM UTC

   Google Cloud incident status: https://status.cloud.google.com/incidents/fmEL9i2fArADKawkZAa2 No workaround provided by Google Cloud currently. We apologise for the ongoing disruption.

3. investigating Jul 19, 2022, 05:52 PM UTC

   Some further details regarding likely underlying issue (cooling related failure) released by Google Cloud: https://status.cloud.google.com/incidents/XVq5om2XEDSqLtJZUvcH No workaround currently for our affected services. We apologise for the ongoing disruption.

4. investigating Jul 19, 2022, 07:49 PM UTC

   Google Cloud have stated that mitigation work is currently underway but no estimated time has been provided as yet for resolution. Relevant Google Cloud incident report: https://status.cloud.google.com/incidents/fmEL9i2fArADKawkZAa2

5. identified Jul 19, 2022, 10:50 PM UTC

   Cooling system restoration is expected by Jul 20, 00:30 BST. Once the cooling system is restored, the Google Cloud team will work on restoring the services, including the services impacting Concentric. We continue to monitor the situation during this unprecedented outage – linked to record temperatures in London today – and appreciate users' patience whilst restoration work continues.

6. identified Jul 20, 2022, 04:24 AM UTC

   This incident has been resolved.

7. resolved Jul 20, 2022, 04:32 AM UTC

   This incident has been resolved.