Chameleon.io incident

Email delivery delays

Major Resolved View vendor source →

Chameleon.io experienced a major incident on December 16, 2022 affecting Transactional emails, lasting 4h 30m. The incident has been resolved; the full update timeline is below.

Started
Dec 16, 2022, 10:34 PM UTC
Resolved
Dec 17, 2022, 03:05 AM UTC
Duration
4h 30m
Detected by Pingoru
Dec 16, 2022, 10:34 PM UTC

Affected components

Transactional emails

Update timeline

  1. investigating Dec 16, 2022, 08:23 PM UTC

    Transactional emails (Magic links, Data export completion, Invites, etc.) are not being delivered at the moment. We've identified an issue with out email provider and are in contact with them and are following along here => https://status.postmarkapp.com/ which also appears to be having issues. Support email and everything else Chameleon is not affected by this -- reach out to us via [email protected] with anything that needs our urgent attention.

  2. identified Dec 16, 2022, 09:28 PM UTC

    Postmark confirmed it's upstream from them at the deliverability service -- emails will be sent as the upstream provider comes back from this incident.

  3. identified Dec 16, 2022, 10:34 PM UTC

    The Postmark team continues to work on this => update here https://ea5d5bbc.sorryapp.com/ We have seen some transactional emails being delivered but not yet back to 100%.

  4. resolved Dec 18, 2022, 05:09 AM UTC

    As of 7PM pacific time Friday night transactional email services was restored and the pending deliveries were delivered albeit delayed.