Envoy experienced a minor incident on April 1, 2025 affecting Analytics, lasting 5h 23m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 01, 2025, 11:21 PM UTC
Due to an internal process temporarily impacting system performance, we've paused the tasks that update analytics in real time. These processes will resume after office hours, and we expect analytics functionality to be fully restored by 9 PM PST. Until then, dashboards may show stale data. We appreciate your understanding. For any questions/concerns, please reach out to Envoy Support via [email protected]
- resolved Apr 02, 2025, 04:44 AM UTC
The underlying task has been completed, and analytics data is now up to date. We appreciate your patience while the functionality was being restored.
- postmortem Apr 03, 2025, 05:26 PM UTC
On Tuesday morning, we noticed delays in analytics job completion, which resulted in stale data appearing on customer dashboards. Some infrastructure components also experienced intermittent restarts, leading to brief service disruptions. Our engineering team investigated the issue in collaboration with AWS and identified a degraded task within the underlying cloud infrastructure as a contributing factor. As a precaution, we paused data processing, reverted recent changes, and initiated a controlled restart process around 4 PM PT, completing full recovery by approximately 10 PM PT. AWS later indicated that the behavior was due to a rare condition in their storage subsystem, which can affect performance when certain transactions run concurrently. Upon the internal corrective actions, all systems have since returned to normal. If you have any concerns or require additional information, please don't hesitate to contact our Customer Support team via [[email protected]](mailto:[email protected]) or contact your dedicated account manager.