Vero incident
Delays in email processing - newsletter and automated emails
Vero experienced a major incident on October 20, 2021 affecting Transactional emails and Vero 1.0: Newsletter processing and 1 more component, lasting 6h 35m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 20, 2021, 04:56 AM UTC
We are currently investigating delays in queuing of newsletter campaigns
- investigating Oct 20, 2021, 05:45 AM UTC
We are continuing to investigate this issue - newsletters are sending but noting we are seeing queuing delays of 30+ minutes
- investigating Oct 20, 2021, 06:55 AM UTC
We are currently experiencing delays in processing emails which our team is actively working to resolve. This includes both newsletters and now automated campaigns. If you have questions or want to discuss potential or observed impact on your account please email us at [email protected].
- identified Oct 20, 2021, 07:27 AM UTC
The issue has been identified and a fix is being implemented
- monitoring Oct 20, 2021, 08:42 AM UTC
A fix has been implemented and we are currently monitoring. We expect there will be delays in processing emails whilst we process a backlog of requests over the next few hours.
- monitoring Oct 20, 2021, 10:09 AM UTC
We are continuing to monitor for further issues but we anticipate that our system backlog should be processed in approximately another hour, we will update our status components back to Operational as those workloads are real-time again.
- monitoring Oct 20, 2021, 10:45 AM UTC
At 02:00 UTC today we identified processing delays with some of our workers. After several hours of investigation and intermittent reliability issues we made the decision to pause our workers temporarily in order to determine the root cause. As a result we were able to pinpoint the cause: a regression in a code deploy made earlier today causing worker configuration issues and resulting in workers consistently failing before completing their jobs. Once fixed we were able to redeploy our worker fleet with services resuming at 10:12 UTC. Since this time we have been working through the backlog of accumulated API calls and automations for the Vero Workflows product. Please see a summary of the affected service timeline below. No data was lost during this incident. **Vero Newsletters** 02:00 UTC - 07:50 UTC: unreliable queueing of newsletters. 07:50 UTC - 10:00 UTC: not able to send newsletters. **Vero Workflows** 02:00 UTC - 07:50 UTC: unreliable queueing of newsletters (within Vero Workflows). 07:50 UTC - 10:00 UTC: - Not able to send newsletters. - No automated processing. - No API call processing (though all data was safely collected). 10:00+ UTC: working through API and automation backlog (API real time as of 9.15pm). -- We will continue to monitor and will resolve this incident once all services are real time and operating as normal. If you have any questions, please email us at [email protected]. Thank you for your patience!
- resolved Oct 20, 2021, 11:32 AM UTC
This incident has been resolved, all services are real time and operating as normal. If you have questions or want to discuss potential or observed impact on your account please email us at [email protected].