Galaxy Project incident

Galaxy Europe jobs not being processed

Resolved after 15 minutes, on March 12, 2019.

postmortem Mar 14, 2019, 06:49 AM UTC

We experienced extremely high disk load due to the web server processes processing large amounts of data. This caused our job handlers to also become non-responsive. We restarted the handlers, but due to high disk load they could not restart in a reasonable amount of time. Given the situation, and due to an ongoing training that was being impacted by this outage, we took the opportunity to restart the main web processes resulting in a web downtime of 15 minutes. We apologise for any inconvenience this has caused Helena Rasche[ ](http://RascheUseGalaxy.eu)UseGalaxy.eu Sysadmin

Started
Mar 12, 2019, 08:00 AM UTC
Resolved
Mar 12, 2019, 08:15 AM UTC
Duration
15m
Detected by Pingoru
Mar 12, 2019, 08:00 AM UTC

Update timeline

  1. resolved Mar 13, 2019, 05:56 PM UTC

    Due to datacenter OpenStack issue, our compute nodes were unavailable, and no jobs were processed.

  2. postmortem Mar 14, 2019, 06:49 AM UTC

    We experienced extremely high disk load due to the web server processes processing large amounts of data. This caused our job handlers to also become non-responsive. We restarted the handlers, but due to high disk load they could not restart in a reasonable amount of time. Given the situation, and due to an ongoing training that was being impacted by this outage, we took the opportunity to restart the main web processes resulting in a web downtime of 15 minutes. We apologise for any inconvenience this has caused Helena Rasche[ ](http://RascheUseGalaxy.eu)UseGalaxy.eu Sysadmin

Your product runs on more than Galaxy Project

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →