Vero experienced a minor incident on April 17, 2021 affecting Reports data availability (Vero default and Mailgun integrations) and General UI access and speed and 1 more component, lasting 11h 45m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Apr 17, 2021, 02:03 AM UTC
We are have been experiencing intermittent issues causing parts of Vero's UI to be slow or unresponsive, we are currently working to resolve the issue.
- identified Apr 17, 2021, 03:07 AM UTC
We are continuing to work on a fix for this issue
- identified Apr 17, 2021, 06:09 AM UTC
We are continuing to work on a fix for this issue.
- identified Apr 17, 2021, 08:55 AM UTC
We are continuing to work on a fix for this issue.
- monitoring Apr 17, 2021, 11:53 AM UTC
All parts of the Vero UI have been reliably available again over the past hour and we will continue to monitor. There will be some delays to accessing recent campaign statistics and logs data in certain sections of the Vero UI but we expect any backlog to be processed over the next few hours and will update our status components accordingly.
- monitoring Apr 17, 2021, 01:24 PM UTC
We are continuing to monitor for any further issues.
- resolved Apr 17, 2021, 01:49 PM UTC
This incident has now been resolved. We are continuing to monitor for any further issues.
- postmortem Apr 19, 2021, 01:53 AM UTC
At 12:31 UTC 17 April we were alerted to intermittent issues with Vero's UI. By 01:57 UTC we had identified the root cause: a recent deploy to the search cluster powering elements of our UI had led to unforeseen load on that cluster, causing it to fail. We rolled the changes back and, unfortunately, had to initiate a full recovery on the cluster involved. UI access was restored at approximately 05:30 UTC 17 April, with significant delays in the Logs and Customers functionality whilst the restore continued. The recovery process completed at approximately 09:45 UTC 17 April. During the period 12:31-09:45 UTC, **only** UI access was interrupted. Backend processing **continued as normal**, with no effect on newsletters, workflows or other functions. We are continuing to monitor to ensure the new cluster is performing as expected and there are no further interruptions. If you have any further questions, please email us at [[email protected]](mailto:[email protected]).