Tenant Turner incident

RESOLVED - Intermittent Phone and Website Access Issue

Notice Resolved View vendor source →

Tenant Turner experienced a notice incident on November 6, 2019 affecting Tenant Turner, lasting 2d 7h. The incident has been resolved; the full update timeline is below.

Started
Nov 06, 2019, 12:22 PM UTC
Resolved
Nov 08, 2019, 07:44 PM UTC
Duration
2d 7h
Detected by Pingoru
Nov 06, 2019, 12:22 PM UTC

Affected components

Tenant Turner

Update timeline

  1. monitoring Nov 07, 2019, 06:04 PM UTC

    When did the issue occur? Wednesday, 11/6, from 7:22am EST to 5:26pm EST What Tenant Turner functionality was impacted? When leads called your Tenant Turner phone number they received an application error. Some leads were able to get through and receive text messages, but most did not. A small number of Tenant Turner users also had an issue logging in to Tenant Turner and viewing their Tenant Turner listings on their website. What occurred? On Tuesday, 11/5, we upgraded our web servers to improve the performance and speed of Tenant Turner. On Wednesday, 11/6, at 7:22am EST we started receiving errors in our phone system logs. Our team immediately started investigating the issue and reached out to several different providers for assistance. We found that the errors were originating from OpenDNS* whom had partially blocked one of our subdomains causing the issue to occur. We were able to rebuild a new web server to direct traffic there by 5:26pm EST, which solved the issue. OpenDNS also ended up unblocking our domain early Thursday morning (by 4:30a EST). What are we doing now? OpenDNS has yet to provide a reason for the partial block but it appears that they did so in error. We're working with OpenDNS to understand why the block occurred and ensure they don't block us again. We're also actively monitoring our error logs to know if anything were to happen again. If we see anything in those logs, an update will be posted here. *OpenDNS Global Network is a Cisco subsidiary that provides content filtering for mostly enterprise businesses. OpenDNS is not associated with Tenant Turner nor any of our infrastructure providers, but they are part of the backbone of the Internet. They process an estimated 100 billion internet traffic requests daily from 85 million users through 25 data centers worldwide, which is why our phone systems were impacted without warning.

  2. monitoring Nov 07, 2019, 06:05 PM UTC

    Website and phone systems are fully operational now. We are continuing to monitor for any further issues.

  3. resolved Nov 08, 2019, 07:44 PM UTC

    This incident has been resolved.