BlueFox Host experienced a critical incident on March 22, 2021, lasting 17m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Mar 22, 2021, 09:43 PM UTC
We are currently investigating this issue.
- identified Mar 22, 2021, 09:43 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Mar 22, 2021, 09:50 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 22, 2021, 10:01 PM UTC
This incident has been resolved.
- postmortem Mar 22, 2021, 10:11 PM UTC
## **Leadup** Nothing out of the ordinary, prior events unrelated to affected services. ## **Fault** php7.3-fpm failure due to bad configuration. ## **Detection** Elastic APM reported site connection failures. ## **Root causes** While working to configure APM and APM traces, we had to reconfigure php7.3-fpm to allow the elastic-php plugin to be utilised. When attempting to do this, there was a slight misconfiguration between our documentation and The Elastic Company’s documentation, this lead to a typo in the configuration which attempted to load an extension that didn’t exist. Thanks to this, php7.3-fpm failed to respond to requests correctly, leading to gateway issues forwarded to Cloudflare. ## **Lessons Learnt** * We could’ve identified the issue earlier and verified this in a non-production environment * Ensure that internal documentation is up-to-date relating to services we deploy- this is not common and is a new service we are deploying. We do not expect this to happen again.