Vero experienced a minor incident on September 1, 2022 affecting Vero 1.0: Ingestion API and General UI access and speed and 1 more component, lasting 4h 46m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 01, 2022, 09:10 AM UTC
We're currently seeing delays in processing API requests and emails as well as loading the UI. Our team has been working to mitigate these issues and we will provide further updates as soon as possible. Please don't hesitate to contact [email protected] if you have any questions.
- identified Sep 01, 2022, 09:30 AM UTC
The UI is now accessible! We've also identified the root cause and should be able to process API requests and emails shortly.
- identified Sep 01, 2022, 09:45 AM UTC
Emails are now processing. We expect there will still be some delays in sending emails whilst we process the current backlog. If you have any questions regarding your campaigns please don't hesitate to email [email protected].
- monitoring Sep 01, 2022, 10:57 AM UTC
API requests are now processing 🎉. All services are now operational and we are working to process a backlog of requests. Once all data is processing real time we will resolve the issue and follow up with additional details. As always, please don't hesitate to email [email protected] if you have any questions
- resolved Sep 01, 2022, 01:57 PM UTC
This incident has been resolved. All API requests and emails are being processed in real time. If you experience any further issues please email [email protected] and we will investigate ASAP.
- postmortem Sep 01, 2022, 11:19 PM UTC
Overnight, shortly before 8:00AM UTC, our internal monitoring system alerted us to issues with our API processing. Upon investigation we realized that API processing, email processing, and several other services \(including access to the UI\) were being impacted. Investigation revealed that an unusual pattern of heavy API traffic was the cause. Actions were taken to mitigate the issue, and we were able to restore functionality to our services, and begin processing the backlog. At approximately 1:57PM UTC, the backlog had been fully processed and all API requests and emails began to be processed in real time. We'll be looking into how to optimize and guard against this particular pattern of API traffic in the future.