Cheddar incident

Deliverability issues with default SMTP provider

Minor Resolved View vendor source →

Cheddar experienced a minor incident on July 31, 2018 affecting Default SMTP Provider, lasting 22h 35m. The incident has been resolved; the full update timeline is below.

Started
Jul 31, 2018, 03:20 PM UTC
Resolved
Aug 01, 2018, 01:56 PM UTC
Duration
22h 35m
Detected by Pingoru
Jul 31, 2018, 03:20 PM UTC

Affected components

Default SMTP Provider

Update timeline

  1. investigating Jul 31, 2018, 03:20 PM UTC

    We are currently experiencing compromised deliverability of emails via our default SMTP service. Outgoing customer communication transactional emails from Cheddar product accounts using the default SMTP configuration may not be delivered until the issue is resolved. Customers using their own SMTP service are unaffected. To see if your Cheddar product is using our default SMTP configuration, please visit the customer communications page of your Cheddar dashboard here: https://www.getcheddar.com/admin/emails.

  2. investigating Jul 31, 2018, 04:27 PM UTC

    Sending has been re-enabled for our default SMTP provider. However, Google may be erroneously marking some of these emails as spam, so some recipients still might not be receiving emails that are sent using our default SMTP configuration. We're working to resolve this issue as soon as possible, and will keep you updated on our progress.

  3. identified Jul 31, 2018, 06:42 PM UTC

    We've submitted a request to Google and they're in the process of reviewing it now. As soon as they send us an update, we'll let you know.

  4. monitoring Aug 01, 2018, 12:15 AM UTC

    We've implemented a fix that appears to have resolved the issue. We'll continue to monitor the situation, but transactional emails from Cheddar product accounts are being successfully delivered and are no longer being flagged by Google as spam.

  5. resolved Aug 01, 2018, 01:56 PM UTC

    Our monitoring shows that emails continue to be delivered successfully, so we believe the issue is now resolved. If you have any questions or concerns, get in touch with our support team at support.getcheddar.com.