Ekklesia 360 incident

Switching Loop Error in Data Center

Major Resolved View vendor source →

Ekklesia 360 experienced a major incident on May 27, 2016, lasting 10h 13m. The incident has been resolved; the full update timeline is below.

Started
May 27, 2016, 03:23 AM UTC
Resolved
May 27, 2016, 01:37 PM UTC
Duration
10h 13m
Detected by Pingoru
May 27, 2016, 03:23 AM UTC

Update timeline

  1. identified May 27, 2016, 03:23 AM UTC

    Our engineers were alerted to a switching loop occurring in one of data center providers networks. Our team is engaged with their Systems Engineers and we are working to resolve the issue. During this time, Customers may be unable to access the CMS and website pages may load slowly or be missing content.

  2. monitoring May 27, 2016, 03:52 AM UTC

    Our engineers continue to work on correcting a networking issue affecting CMS and website accessibility. Most systems are now accessible, and we're continuing to monitor the situation as connectivity is restored quickly.

  3. monitoring May 27, 2016, 05:06 AM UTC

    This issue has been mitigated significantly and tests are being run to determine resolution. During this time, website pages may load slowly or be missing content.

  4. monitoring May 27, 2016, 08:00 AM UTC

    Our hosting engineers continue to work to correct the effects of the earlier switching failure. Currently, most Monk systems are online, however we are still seeing the effects of poor API connectivity which powers website content. During this time, website pages may load slowly or appear to be missing content intermittently.

  5. resolved May 27, 2016, 01:37 PM UTC

    This incident has been resolved.