Ambra experienced a notice incident on September 14, 2023 affecting Web Services and Image Processing and 1 more component, lasting 48m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 14, 2023, 05:25 PM UTC
We have received reports of issues on the Ambra platform. Engineering teams are currently investigating. Additional information will be provided as soon as it is available.
- investigating Sep 14, 2023, 05:56 PM UTC
Our Engineering teams are actively investigating and working to identify the root cause. We understand the urgency and we appreciate your patience as we work to address the issue.
- resolved Sep 14, 2023, 06:13 PM UTC
The incident has been fully resolved and service is back to normal levels. Our team will be conducting a root cause analysis and sharing as soon as possible. We will continue to monitor the situation to ensure there are no further issues.
- postmortem Sep 18, 2023, 04:39 PM UTC
During the previous 9/12 performance issues we switched new ingestion to a separate storage cluster to allow new uploads to complete while we continued working on the contention issues with the previous cluster. As load increased on the second cluster, it ran into AWS resource limits on a filesystem used for temporary storage. This was corrected by provisioning additional throughput to allow the system to handle full levels of production traffic.