ProcedureFlow incident

Automated emails not being sent

Minor Resolved View vendor source →

ProcedureFlow experienced a minor incident on November 6, 2015, lasting 57m. The incident has been resolved; the full update timeline is below.

Started
Nov 06, 2015, 08:26 PM UTC
Resolved
Nov 06, 2015, 09:24 PM UTC
Duration
57m
Detected by Pingoru
Nov 06, 2015, 08:26 PM UTC

Update timeline

  1. investigating Nov 06, 2015, 08:26 PM UTC

    We're investigating problems sending automated emails from ProcedureFlow (including invitations and password resets).

  2. identified Nov 06, 2015, 08:36 PM UTC

    Our email provider's DNS host name (Amazon SES) is failing to resolve which is preventing emails from being sent. We are attempting to create a workaround. From Amazon: 12:29 PM PST We are currently investigating elevated error rates for our email-sending API in the US-EAST-1 Region for the SMTP endpoint only.

  3. monitoring Nov 06, 2015, 09:11 PM UTC

    Amazon has resolved the DNS problems and automated emails are working again. Please resend any invitations/forgot password requests that did not receive an email. From Amazon: 1:00 PM PST We've identified root cause and are seeing recovery for elevated error rates to our email-sending API in the US-EAST-1 Region for the SMTP endpoint. HTTP endpoints continue to operate normally.

  4. resolved Nov 06, 2015, 09:24 PM UTC

    This incident has been resolved. From Amazon: 1:19 PM PST Between 11:38 AM and 12:45 PM PST we experienced elevated error rates for our email-sending API to the SMTP endpoint in the US-EAST-1 Region. This did not affect the SendEmail/SendRawEmail HTTP APIs. The issue has been resolved and the service is operating normally.