Vero experienced a major incident on August 15, 2021 affecting Vero 1.0: Ingestion API and Transactional emails and 1 more component, lasting 1d. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 15, 2021, 02:28 AM UTC
We're experiencing an elevated level of API errors and are currently looking into the issue.
- identified Aug 15, 2021, 02:50 AM UTC
We have identified an issue with a database that has failed and we are currently working our service provider to have our services resumed as soon as possible.
- monitoring Aug 15, 2021, 03:43 AM UTC
We have recovered the affected services and are processing through the backlog. We estimate this will take around three hours, at which time all API processing should return to normal. We will post an update once we confirm these timelines and/or if there are further updates. Whilst we are processing the backlog there will be delays to processing Workflows and other automated campaigns.
- monitoring Aug 15, 2021, 05:08 AM UTC
The backlog of API and campaign requests have been cleared and processing times have now returned to normal. We are continuing to monitor for further issues and are also performing related maintenance activities.
- monitoring Aug 15, 2021, 11:06 AM UTC
We are continuing to monitor for further issues and are also performing related maintenance activities.
- resolved Aug 16, 2021, 01:22 AM UTC
This incident has been resolved.