Vero experienced a minor incident on May 16, 2022 affecting Reports data availability (Vero default and Mailgun integrations) and Transactional emails and 1 more component, lasting 3d 6h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 16, 2022, 11:03 PM UTC
We are currently seeing a degradation in performance which has predominantly affected automated email processing. Further impacted systems that have seen intermittent delays are calculating segments, processing newsletters as well as the campaign statistics, logs and webhooks that result from sending emails. Our team has been working to mitigate these issues and we will provide further updates shortly - we will ofcourse write a full post-mortem once all is resolved. Please don't hesitate to contact [email protected] if you have any questions.
- monitoring May 17, 2022, 06:12 AM UTC
All services are now operational and we are working to process a backlog of requests. It's possible we will still experience some intermittent delays over the next 24 hours, as we continue to solve the root cause. Until then we'll keep components marked as degraded. Our support team will be reaching out to customers whose newsletters have been affected. Please email [email protected] if you have any questions regarding your account.
- monitoring May 18, 2022, 07:02 AM UTC
All has been processing as expected for several hours and we're now confident the issue is resolved. We'll follow up shortly with a full post-mortem.
- resolved May 20, 2022, 05:52 AM UTC
On Monday 16th of May at 9am UTC we experienced performance degradation in our event stream processing service. As we process events at a significant volume this most notably led to delays in processing emails and statistics whilst we restored performance. Once the issue was solved it took some time to process the backlog which completed on Tuesday 17th of May at 8:12am UTC.