Ekklesia 360 experienced a major incident on October 7, 2016, lasting 2d 10h. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Oct 07, 2016, 03:25 PM UTC
The team is currently looking into resolving an interruption in API service. During this time, recently changed content may not display on sites until the issue is resolved.
- identified Oct 07, 2016, 03:48 PM UTC
The team discovered an issue where DNS may not be routing requests properly. We're temporarily disabling access to some servers while we work to roll out a solution as soon as possible.
- identified Oct 07, 2016, 04:38 PM UTC
Engineers continue to work to restore DNS resolution for the CMS and for client sites. Thanks for your patience!
- identified Oct 07, 2016, 05:26 PM UTC
The team continues to work to restore service. Unfortunately there will be some delays for email and phone support. We're really sorry for the disruption today!
- identified Oct 07, 2016, 05:54 PM UTC
The team is nearly finished putting a workaround in place to resolve DNS.
- identified Oct 07, 2016, 07:12 PM UTC
The Ekklesia 360 (MonkCMS) platform has been brought back online, and the team continues to work out issues with DNS to serve new content. Thank you!
- identified Oct 07, 2016, 07:53 PM UTC
API services have been restored, and shortly we will be transitioning to another DNS host. During that time, there may be some intermittent delays or interruptions in service again until the issue is fully resolved.
- monitoring Oct 07, 2016, 08:15 PM UTC
DNS is functioning as normal again, and we're continuing to monitor the systems affected.
- monitoring Oct 07, 2016, 10:33 PM UTC
Core services remain accessible, however some reports of intermittent DNS issues have been made. The team is investigating any remaining cases.
- monitoring Oct 10, 2016, 02:14 AM UTC
DNS has been confirmed up for our core services. Thank you!
- resolved Oct 10, 2016, 02:14 AM UTC
This incident has been resolved.