Vero experienced a minor incident on March 24, 2021 affecting Reports data availability (Vero default and Mailgun integrations) and Logs page activity and 1 more component, lasting 4d 22h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 24, 2021, 06:57 AM UTC
On Tuesday 23rd March at approximately 6:30pm UTC we observed delays in processing automated messages (behavioural, transactional and workflows). We identified an issue with our round-robin system. We debugged and fixed the root cause by approximately 9.30pm UTC. During the period 6:30pm-9.30PM UTC automated messages were operating at a severely reduced throughput, resulting in a backlog of automated messages. The backlog of automated messages was fully processed and running in realtime by 11pm UTC. Newsletters sent between 9.30pm-12:30am UTC (Wednesday) were sent at a reduced rate whilst this backlog was processed, resulting in some delays. We've been posting updates to the various system components on this status page over the last 12 hours to reflect the above delays as they occurred. -- Since 12:30am UTC we have identified a second issue with some of the raw hardware provided by our cloud provider. **It appears this is a hardware issue and is not related to the earlier incident**. Messages **are** currently sending in realtime but we are operating at an overall reduced capacity with some delays in aggregating conversion metrics and sending webhooks. We're working with our cloud provider to resolve this issue, at which time we'll be able to scale to full capacity. In the interim we're monitoring and will post updates if the situation changes and message send speeds are affected. If you have any questions, please email us at [email protected].
- identified Mar 24, 2021, 11:05 PM UTC
We are still processing at a reduced rate. Reports and Logs are particularly affected, as are webhooks. We've updated the components below to reflect this. Message throughput has been affected periodically since 1:30pm UTC (Wednesday) and we have been updating the affected components to reflect this as it occurs. We are making progress on this issue and apologise for the extended impact.
- monitoring Mar 25, 2021, 06:12 AM UTC
All messages, logs, reports and webhooks are processing realtime. Conversion statistics will be updating in realtime within the next 12 hours. We'll continue to monitor and provide updates as they occur.
- monitoring Mar 26, 2021, 04:57 AM UTC
Messages, logs, reports and webhooks continue to be processed in realtime. We are still processing the backlog of conversion statistics which we anticipate will be complete within 6-12 hours. If you have any questions about your account please email us at [email protected].
- resolved Mar 29, 2021, 05:05 AM UTC
Conversion results have been updating in realtime for the last 48 hours and we can confirm this incident is now resolved! 🎉