Ambra experienced a minor incident on January 14, 2025 affecting Web Services and Image Processing and 1 more component, lasting 1h 50m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 14, 2025, 05:05 PM UTC
We have received reports of issues on the InteleShare platform. Engineering teams are currently investigating. Additional information will be provided as soon as it is available.
- identified Jan 14, 2025, 05:23 PM UTC
Our engineering team has identified a services queue backlog that has affected the ingestion of studies to InteleShare. Studies are now ingesting, but it may take time to catch up on the backlog. The engineering team continues to investigate the issue and we will provide further updates as they are available.
- investigating Jan 14, 2025, 05:48 PM UTC
Our Engineering teams continue to investigate and work to identify a resolution. We understand the urgency and we appreciate your patience as we work to address the issue.
- monitoring Jan 14, 2025, 06:31 PM UTC
At this time, the services backlog has improved and studies should now be ingesting, however we continue to monitor the situation to ensure this progress continues as expected.
- resolved Jan 14, 2025, 06:55 PM UTC
The incident has been resolved and service is back to normal levels. Our team will be conducting a root cause analysis and sharing as soon as possible. We will continue to monitor the situation to ensure there are no further issues.
- postmortem Feb 11, 2025, 05:17 PM UTC
While investigating delays in uploads becoming available, we determined that an InteleShare internal queue was backlogged and that not all queue workers were being efficiently utilized. After tuning some queue parameters and restarting queue workers, the rate of queue processing dramatically improved and the system was able to catch up and complete all pending activities. We have improved our existing monitoring so that queue alerts are routed more quickly to our operations team, and with highest possible priority. R&D will investigate possible improvements to the queue processing logic for future InteleShare releases.