Retreaver incident

There's an issue with Call Handling at Twilio

Major Resolved View vendor source →

Retreaver experienced a major incident on November 5, 2018 affecting Call Handling and Twilio Incoming Phone Calls and 1 more component, lasting 3h 33m. The incident has been resolved; the full update timeline is below.

Started
Nov 05, 2018, 04:50 PM UTC
Resolved
Nov 05, 2018, 08:24 PM UTC
Duration
3h 33m
Detected by Pingoru
Nov 05, 2018, 04:50 PM UTC

Affected components

Call HandlingTwilio Incoming Phone CallsTwilio Incoming SIP CallsTwilio Outgoing Phone CallsTwilio Outgoing SIP Calls

Update timeline

  1. identified Nov 05, 2018, 04:50 PM UTC

    Some calls may fail to go through. We're waiting on Twilio to fix their issue, historically they have had an incredibly reliable service. Updates will follow shortly.

  2. monitoring Nov 05, 2018, 04:59 PM UTC

    The last error was seen by our engineers around 11:52 AM EDT, so we're assuming that Twilio is now fully operational. We continue to monitor this situation.

  3. monitoring Nov 05, 2018, 05:32 PM UTC

    Twilio is back down. We’re continuing to monitor.

  4. monitoring Nov 05, 2018, 05:44 PM UTC

    We haven't seen any errors in the past 5 minutes. We're continuing to monitor the situation.

  5. monitoring Nov 05, 2018, 05:50 PM UTC

    Twilio is reporting that the issue has been identified and their services have recovered. We're continuing to monitor for issues.

  6. monitoring Nov 05, 2018, 06:06 PM UTC

    We're seeing errors again. Some calls may intermittently fail as we wait for Twilio to get their services under control. Our apologies to our users for this.

  7. monitoring Nov 05, 2018, 06:15 PM UTC

    This morning and afternoon, our telephony provider, Twilio, experienced degraded service in 3 separate periods, from 10:58 AM until 11:54 AM (EST), 12:15 PM - 12:40 PM, and 1:00 PM - 1:04 PM. During this time, some calls were negatively impacted, resulting in error messages being played to the caller. We're continuing to monitor the situation and will provide further updates soon.

  8. monitoring Nov 05, 2018, 06:58 PM UTC

    We're continuing to see small, intermittent bursts of errors from our telephony provider, Twilio (from 1:51 to 1:54 pm EST). This outage is affecting their entire platform. We're continuing to monitor the situation and sincerely apologize to our customers for the issue.

  9. resolved Nov 05, 2018, 08:24 PM UTC

    Twilio has issued the all clear, and systems seem to be operating normally. We haven't seen any additional errors in the past hour. We sincerely regret the impact of this issue on our customers.