Gorilla Experiment Builder experienced a minor incident on November 15, 2021 affecting Gorilla Platform, lasting 3h 34m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Nov 15, 2021, 03:25 PM UTC
Some participants may experience temporary issues accessing experiments due to DNS propagation issues relating to the research.sc domain. This should be resolved shortly as the domain propagates more completely, and will not impact researchers currently editing or building their experiments
- monitoring Nov 15, 2021, 03:25 PM UTC
We are continuing to monitor for any further issues.
- resolved Nov 15, 2021, 09:55 PM UTC
DNS propagation appears to be complete worldwide and has been for a few hours, so we're marking this as resolved. We will post more details as to the cause in due course.
- postmortem Nov 19, 2021, 12:35 PM UTC
On Tuesday November 15th, at approximately 14:45 GMT, our [research.sc](http://research.sc) domain stopped responding to web traffic. We quickly realised that this was because the domain had been moved to a different billing package by our registrar, and in the process of moving it, had reset it to its default settings, and so all our DNS settings had been lost. We restored the correct DNS settings to reconnect the domain to our servers, which in turn re-established all of our systems. This was completed by 15:12 GMT. As DNS can take some time to propagate, we continued to monitor the situation for the rest of the afternoon and evening. By 21:50 GMT, we were satisfied that the DNS changes had fully propagated worldwide, and we had received no new reports of problems for several hours, and so marked the issue as resolved. The direct cause of this was that our DNS provider had retired an unrelated product that we were using, and was therefore removing it from our subscription. We were well aware of this and had already long migrated from the obsolete product. However, they inexplicably cancelled the entire contract - not simply the product - and the contract also contained the [research.sc](http://research.sc) domain. Realising this, we contacted them and asked them to move the [research.sc](http://research.sc) domain to one of our live contracts. They then did so, but in the process, reset the domain settings to their defaults, wiping out all of our DNS settings. That the settings got wiped at all, and additionally without warning us first, is and was startling, and we will be taking the issue up with them in due course. The domain was offline for approximately 28 minutes before everything was re-established, with DNS taking a few hours after that to propagate for most users \(a process we have no control over\). We will be talking to our registrar about the incident, and now that the domain is on an evergreen subscription, do not anticipate that it can happen again.