eCatholic experienced a notice incident on April 29, 2016, lasting 23h 30m. The incident has been resolved; the full update timeline is below.
Update timeline
- identified May 02, 2016, 04:23 PM UTC
We were fixing a bug this afternoon related to photo albums and the fix caused some problems with our file servers that is now causing some sites to load with missing images or styling problems. We are addressing the issue and hope to have it resolved soon. We do not currently have a timeline for a resolution, but we will continue to post updates
- monitoring May 02, 2016, 04:24 PM UTC
We are currently restoring site images and styling to all sites affected by the file server issues that we reported late this afternoon. The full restoration will probably take through the night, but sites will be restored incrementally, so most sites will probably be restored sooner than morning.
- investigating May 02, 2016, 04:24 PM UTC
We are still restoring site images to all sites affected by the file server issues that we reported late Friday afternoon. The full restoration is taking longer than expected because hundreds of sites were affected. We do not currently have an ETA of when the restoration will complete, but it could likely be another 24-48 hours before all site images are restored. We are terribly sorry for the inconvenience that this issue has caused and we appreciate your patience and prayers as we continue to work around the clock to resolve the issue. In the meantime, all affected sites should work fine other than the missing images that are being restored.
- monitoring May 02, 2016, 04:25 PM UTC
While we continue to work on restoring the files and site images, you may experience some intermittent issues while uploading files, documents, and bulletins. We will notify everyone via email as soon as the issues are completely resolved and continue to post updates to this page.
- resolved May 02, 2016, 04:25 PM UTC
Your website should now be back up and running as normal. You're prayers and patience have been very much appreciated. All site files and images have been completely restored. However, if you uploaded any documents or images on Friday, April 29 between 1 AM - 4 PM CDT, you will need to re-upload those files. Again, we are so sorry for the inconvenience that this issue has caused. Please notify us of any additional issues with your website. Our team will meet on Monday and we'll post a recap with more details about the issues from this weekend.
- postmortem Aug 01, 2018, 07:38 PM UTC
I am very, very sorry for the disruption that the file server issue had on your website over the weekend. ###### Here’s what happened… * Early last week, we released an update for a bug related to photo albums. The update fixed the original bug, but there were still orphaned photo album files that would need to be deleted. * On Friday, our development team deployed a second update to clean up (delete) the orphaned photo album files to avoid future problems. The second update was approved by our development team, tested on our BETA servers with no problems, and then deployed to the rest of our servers. * Shortly after releasing the update, our team noticed unanticipated problems with some site files (images and documents) being deleted. By the time we discovered the issue, hundreds of sites had been affected. * Normally, we would have restored from a redundant server with little to no downtime. However, this process failed due to a problem with the redundant file server. * We then had to start restoring from slower secondary backups. In this situation, and because hundreds of sites were affected, the restoration took many hours over the weekend. ###### Here’s what we are doing moving forward… * Firstly, we are currently analyzing this particular situation and we will be meeting as a team throughout this week to discuss how we can try to avoid situations like this in the future. Our goal is always ZERO downtime, and we are taking this rare mishap very seriously. * Secondly, we are going to improve our communication and transparency during these situations to keep you better informed about the issue and how we are responding. This [new STATUS page](http://status.eCatholic.com) is the first step in that direction. (You may need to REFRESH) If you ever have a concern about a possible outage or issue with your website that might be a global problem, you can check the status of our servers on [this page](http://status.eCatholic.com). ###### THANK YOU…. Thanks again for your understanding, patience, and prayers! As always, our team will continue to strive to provide you with excellent service, and be even more diligent in finding ways to avoid rare instances such as this one. ######-Josh Simmons, Founder & CEO