Zipline incident

Inbound email processing issue

Minor Resolved View vendor source →

Zipline experienced a minor incident on April 21, 2023 affecting Email Handling, lasting 3d 19h. The incident has been resolved; the full update timeline is below.

Started
Apr 21, 2023, 10:51 PM UTC
Resolved
Apr 25, 2023, 06:03 PM UTC
Duration
3d 19h
Detected by Pingoru
Apr 21, 2023, 10:51 PM UTC

Affected components

Email Handling

Update timeline

  1. investigating Apr 21, 2023, 10:51 PM UTC

    We are investigating an issue where inbound emails are not being processed as expected

  2. investigating Apr 21, 2023, 11:57 PM UTC

    We are still investigating this issue.

  3. identified Apr 22, 2023, 12:52 AM UTC

    We have identified the root cause for the issue, and we are currently remedying the cause.

  4. identified Apr 22, 2023, 01:08 AM UTC

    We have tested and deployed the fix for the issue, and are now focussing on processing emails that had previously failed. We do not anticipate any loss of data or emails.

  5. identified Apr 22, 2023, 03:46 AM UTC

    We're continuing the work to reprocess emails that initially did not process.

  6. resolved Apr 25, 2023, 06:03 PM UTC

    We have added additional handling for when our up-stream email service fails, to ensure the issue will not recur. All emails that could be re-processed have been done at this point.