Ambra experienced a notice incident on April 5, 2024 affecting Web Services and Image Processing and 1 more component, lasting 1h 7m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Apr 05, 2024, 02:37 PM UTC
We have identified a major issue affecting Ambra. Our Engineering teams have been mobilized to address the issue, and we will provide frequent updates to keep you informed.
- investigating Apr 05, 2024, 03:01 PM UTC
Our Engineering teams have identified an issue with transcoding services impacting the viewing of images. We are currently adding nodes to the Transcoding services to mitigate the issue. We understand the urgency and we appreciate your patience as we work to address the issue.
- resolved Apr 05, 2024, 03:44 PM UTC
The incident has been fully resolved and service is back to normal levels. Our team will be conducting a root cause analysis and sharing as soon as possible. We will continue to monitor the situation to ensure there are no further issues.
- postmortem May 08, 2024, 07:32 PM UTC
We have observed that the transcoding nodes were utilizing an unusually high amount of resources while processing certain DICOM files that did not conform to expected formats. This led to a significant increase in resource consumption, reaching the limits of our autoscaling capabilities. Consequently, this resulted in the termination of some processes due to insufficient memory. To address this, we have applied a corrective update to our autoscaling process as of April 18th. For a detailed explanation of the incident and the steps we've taken, please refer to the postmortem of the incident dated April 18th.