Ambra Outage History

Started Jul 20, 2026, 01:11 PM UTC· resolved Jul 20, 2026, 03:00 PM UTC

1. investigating Jul 20, 2026, 01:11 PM UTC

   We have received reports of log in issues on the InteleShare platform. Users appear to be unable to log in, or initial log in is slow. Engineering teams are currently investigating. Additional information will be provided as soon as it is available.

2. investigating Jul 20, 2026, 01:41 PM UTC

   Our Engineering teams are actively investigating and working to identify the root cause. We understand the urgency and we appreciate your patience as we work to address the issue.

3. investigating Jul 20, 2026, 02:04 PM UTC

   Our engineering teams have not yet identified the root cause. We are continuing to investigate and will provide further updates as soon as possible.

4. monitoring Jul 20, 2026, 02:29 PM UTC

   At this time, our engineering team has made a configuration change to our system backend to improve stability. Since implementing this change, the website’s frontend has been operating reliably. The root cause has been identified, and we will continue to monitor system performance.

5. resolved Jul 20, 2026, 03:00 PM UTC

   The incident has been fully resolved and service is back to normal levels. Our team will be conducting a root cause analysis and sharing as soon as possible. We will continue to monitor the situation to ensure there are no further issues.

6. postmortem Jul 21, 2026, 08:44 PM UTC

   **What Changed** As part of the scheduled weekend maintenance window, the InteleShare infrastructure was switched to a more modern autoscaling and container-scheduling approach using Karpenter. Karpenter selects the most cost-efficient node/instance type for the current cluster load, replacing a prior approach that effectively over-provisioned compute without the team realizing it. The intent was more efficient and cost-effective use of cloud resources, faster scaling in response to traffic, and access to a wider variety of instance types to avoid AWS capacity constraints. **What Failed** InteleShare workload has relatively low CPU and memory requirements but handles a very high volume of concurrent network connections. Once the workload moved to Karpenter's "right-sized" scheduling, several instances of that workload's container were placed on smaller nodes than before \(moved from a 2XL instance down to an XL, roughly half the CPU\). The Linux kernel's connection-tracking table sizes its maximum entries based on the node's CPU and memory. Because the smaller nodes had a much lower ceiling, and because multiple instances of the high-connection workload were scheduled onto the same nodes, the combined connection count exceeded the conntrack limit once Monday-morning production load hit. Once the limit was reached, new connections on the affected nodes experienced delays or timeouts - and because the limit is enforced at the OS level, it affected other containers co-located on the same nodes as well. **Why It Wasn't Caught Earlier** The change had already been running in UAT and all other pre-production environments, which the team could observe before rollout, and it behaved correctly there. The issue only manifested under the connection volume and concurrency pattern of full U.S. production traffic - UAT volume is orders of magnitude lower and does not reproduce the same ordering/concurrency of requests. Simulating true production-scale concurrent-connection load in a lower environment is difficult and, at current test capacity, was not something the team could reasonably reproduce ahead of time. **Impact** Clients experienced degraded performance - delayed or timed-out connections - for approximately two to two and a half hours on Monday morning \(roughly 9:37 a.m. Eastern into the 10-11 a.m. hour\), concentrated around peak login/usage volume. **Resolution** The on-call platform engineer was paged automatically when the platform-wide Code Blue was triggered. The team identified the affected application tier and reverted that specific component back to the prior scheduling configuration; the rest of the Karpenter rollout \(roughly 85%\) remains in place. The revert was performed before the underlying root cause \(the conntrack ceiling\) was fully understood - root cause was confirmed afterward through investigation.

Read the full incident report →

Started Apr 24, 2025, 04:27 PM UTC· resolved Apr 24, 2025, 07:37 PM UTC

1. investigating Apr 24, 2025, 04:27 PM UTC

   We have received reports of issues on the InteleShare platform with Gateways not pinging. Engineering teams are currently investigating. Additional information will be provided as soon as it is available. Please place a case with Service Now or call support for any issues with this service.

2. resolved Apr 24, 2025, 07:37 PM UTC

   The incident has been fully resolved and service is back to normal levels. We will continue to monitor the situation to ensure there are no further issues.

3. postmortem Apr 28, 2025, 08:45 PM UTC

   On Thursday, April 24th, some InteleShare customer gateways were temporarily unable to connect to InteleShare due to third-party files remaining in use after a service stop was requested. To prevent this from happening in the future, the next Gateway release will include versioning of third-party libraries to avoid such conflicts. We appreciate your patience and understanding.

Read the full incident report →

Started Feb 05, 2025, 12:36 AM UTC· resolved Feb 05, 2025, 01:17 AM UTC

1. investigating Feb 05, 2025, 12:36 AM UTC

   We have identified an issue affecting InteleShare. Our Engineering teams have been mobilized to address the issue, and we will provide frequent updates to keep you informed.

2. monitoring Feb 05, 2025, 12:42 AM UTC

   A fix has been implemented and the platform is available now. We are monitoring closely.

3. resolved Feb 05, 2025, 01:17 AM UTC

   The incident has been fully resolved and service is back to normal levels. Our team will be conducting a root cause analysis and sharing. We will continue to monitor the situation to ensure there are no further issues.

4. postmortem Feb 11, 2025, 05:11 PM UTC

   While implementing a minor change to the InteleShare database schema, a database update process was able to acquire some locks it needed to implement the schema change but hung while waiting to acquire the remaining locks. Since it had already acquired some locks, the update process blocked certain normal database activity, causing UI and API errors. We were able to identify and cancel a long-running background process which had not properly released its locks and which was preventing the database update. Our R&D and operations teams will investigate future improvements to our update processes, by implementing or adjusting timeouts to ensure schema updates can fail and retry as needed without affecting normal system activity.

Read the full incident report →

Started Jan 22, 2025, 11:20 PM UTC· resolved Jan 23, 2025, 12:08 AM UTC

1. identified Jan 22, 2025, 11:20 PM UTC

   We have identified an issue affecting InteleShare customers who utilize single sign on (SSO) to access the InteleShare platform. Only customers utilizing our vendor, PingOne, to manage these certificates are currently affected. These users may experience errors logging in with SSO at this time. We have engaged PingOne for critical support at this time, and we will provide further updates on our status page.

2. investigating Jan 22, 2025, 11:46 PM UTC

   We are continuing to work on the situation with PingOne and a possible workaround at this time. In the meantime if your users are able to log into the system without using SSO, please have them use their InteleShare account. Further updates will continue to be posted here.

3. resolved Jan 23, 2025, 12:08 AM UTC

   At this time, customers affected by this issue are now able to sign back into InteleShare with SSO. We will post an RCA on our status page as soon as one is made available.

4. postmortem Jan 23, 2025, 12:19 AM UTC

   The incident was traced to an unexpected change in the status of our SSO integration which led to authentication requests failing, preventing user access. Once the issue was identified, our team engaged with support for Ping Identity and were able to restore normal functionality

Read the full incident report →

Started Jan 22, 2025, 01:45 AM UTC· resolved Mar 06, 2025, 09:30 PM UTC

1. investigating Jan 22, 2025, 01:45 AM UTC

   We would like to inform you of a recently discovered security vulnerability in SimpleHelp (version 5.5.7 and earlier), a remote support and desktop management software used to support many InteleShare clients. More information about the vulnerability can be found here: https://simple-help.com/kb---security-vulnerabilities-01-2025#security-vulnerabilities-in-simplehelp-5-5-7-and-earlier We highly recommend that you share the information posted in the following knowledge article with your security team. https://serviceportal.intelerad.com/csm?id=kb_article_view&sysparm_article=KB0032213

2. investigating Jan 22, 2025, 01:47 AM UTC

   We are continuing to investigate this issue.

3. investigating Jan 22, 2025, 02:01 AM UTC

   We are continuing to investigate this issue.

4. investigating Jan 22, 2025, 02:45 PM UTC

   We are continuing to investigate this issue.

5. identified Jan 22, 2025, 05:14 PM UTC

   The Intelerad team is continuing to analyze the situation. Please make sure you are monitoring for the Indicators of Compromise (IOCs) already provided. Please note we have updated the list and removed a false positive from the list (vmtoolsd.exe). Should you notice any IOCs present on your systems our recommendation is to consult with your Security Team immediately regarding device isolation. You can also open a case in the Intelerad Service Portal and we're happy to support you as best we can. Link to IOCs: https://serviceportal.intelerad.com/csm?id=kb_article_view&sysparm_article=KB0032213

6. identified Jan 24, 2025, 08:35 PM UTC

   Please note we have updated the list of Indicators of Compromise (IOC). Should you notice any IOC present on your systems our recommendation is to consult with your Security Team immediately regarding device isolation. You can also open a case in the Intelerad Service Portal and we're happy to support you as best we can. Link to IOCs: https://serviceportal.intelerad.com/csm?id=kb_article_view&sysparm_article=KB0032213

7. identified Jan 29, 2025, 07:14 PM UTC

   We have deployed additional security monitoring of our client’s InteleShare environments. If we detect any activity in your environment related to the SimpleHelp vulnerability that we believe is a potential indicator of compromise, we will contact you immediately. If your team detects a compromise, please notify Intelerad Support immediately through the Intelerad Service Portal. Additionally, we will not be using SimpleHelp in conjunction with any of our Intelerad products going forward. Please view these articles for additional information: SimpleHelp Security Vulnerability - Config File Verification - https://serviceportal.intelerad.com/csm?id=kb_article_view&sysparm_article=KB0032228 SimpleHelp Security Vulnerability IOC & FAQ- https://serviceportal.intelerad.com/csm?id=kb_article_view&sysparm_article=KB0032213

8. identified Feb 05, 2025, 07:48 PM UTC

   As of February 4, the investigation, which is being conducted with the assistance of outside forensics experts, is beginning to reach its conclusion. Our investigation to date has identified no evidence of data access or exfiltration within the Intelerad corporate network, and our third-party experts have confirmed that the incident is contained. While we do not anticipate any substantive updates before the close of the investigation, we are continuing to support individual clients who have specific questions or concerns. Once our investigation has reached a close, we will provide a final update. If you have further questions, please reach out to our support team.

9. resolved Mar 06, 2025, 09:30 PM UTC

   We are pleased to share that our SimpleHelp incident investigation is now complete. Most importantly, the forensic analysis found no evidence of compromise to any records (including unauthorized access, data staging or theft) within the Intelerad environment, and the incident is fully contained. While this investigation is now closed and we do not anticipate further updates, our security team is available to answer any specific questions your security team may still have. Please don’t hesitate to reach out.

Read the full incident report →

Started Nov 06, 2024, 12:21 AM UTC· resolved Nov 06, 2024, 05:46 PM UTC

1. investigating Nov 06, 2024, 12:21 AM UTC

   We have received reports of issues on the Ambra platform related to Gateways. Engineering teams are currently investigating. Additional information will be provided as soon as it is available.

2. investigating Nov 06, 2024, 12:50 AM UTC

   Our Engineering teams are actively investigating and working to identify the root cause. We understand the urgency and we appreciate your patience as we work to address the issue.

3. investigating Nov 06, 2024, 01:37 AM UTC

   This evening a gateway update that only impacted gateway logging configuration was pushed out to customers. In some cases the package was not fully applied which left some Gateway devices in a state which they couldn't function. Please note this is a package installation issue only. If you would like to perform the fix yourself, please see the KB0031682 - https://serviceportal.intelerad.com/csm?id=kb_article_view&sysparm_article=KB0031682. Please call support at 888-315-0790 if you need assistance with the installation.

4. investigating Nov 06, 2024, 02:19 AM UTC

   We continue to work to identify the root cause of the install issue. In the interium we have identified how to manually fix the install issue. If you would like to perform the fix yourself, please see the KB0031682 in the ISP - https://serviceportal.intelerad.com/csm?id=kb_article_view&sysparm_article=KB0031682 or call support at 888-315-0790 if you need assistance with the new installation.

5. investigating Nov 06, 2024, 02:56 AM UTC

   We are continuing to investigate this issue.

6. investigating Nov 06, 2024, 03:32 AM UTC

   We are continuing to investigate this issue.

7. investigating Nov 06, 2024, 04:21 AM UTC

   We continue to work to identify the root cause of the install issue. In the interim we have identified how to manually fix the install issue. If you would like to perform the fix yourself, please see the KB0031682 in the ISP - https://serviceportal.intelerad.com/csm?id=kb_article_view&sysparm_article=KB0031682 or call support at 888-315-0790 if you need assistance with the new installation. The next update will be in 3 hours.

8. investigating Nov 06, 2024, 06:42 AM UTC

   We continue to work to identify the root cause of the install issue. In the interim we have identified how to manually fix the install issue. If you would like to perform the fix yourself, please see the KB0031682 in the ISP - https://serviceportal.intelerad.com/csm?id=kb_article_view&sysparm_article=KB0031682 or call support at 888-315-0790 if you need assistance with the new installation.

9. investigating Nov 06, 2024, 09:50 AM UTC

   We continue to work to identify the root cause of the install issue. In the interim we have identified how to manually fix the install issue. If you would like to perform the fix yourself, please see the KB0031682 in the ISP - https://serviceportal.intelerad.com/csm?id=kb_article_view&sysparm_article=KB0031682 or call support at 888-315-0790 if you need assistance with the new installation.

10. investigating Nov 06, 2024, 12:50 PM UTC

    We continue to work to identify the root cause of the install issue. In the interim we have identified how to manually fix the install issue. If you would like to perform the fix yourself, please see the KB0031682 in the ISP - https://serviceportal.intelerad.com/csm?id=kb_article_view&sysparm_article=KB0031682 or call support at 888-315-0790 if you need assistance with the new installation.

11. resolved Nov 06, 2024, 05:46 PM UTC

    The incident has been closed. Our team will be conducting a root cause analysis. We will continue to monitor the situation to ensure there are no further issues.

12. postmortem Dec 12, 2024, 10:26 PM UTC

    Issue: After the recent InteleShare release, a small percent of transcoding requests began failing intermittently, in a way that caused the thread that was handling that request to enter an infinite loop waiting for additional data that would never arrive. As these stuck threads accumulated, their combined resources eventually led to performance degradation, and finally to errors or timeouts when the maximum thread pool limit was reached. Root Cause: The recent release of InteleShare included updates to a client library used for internal network communication. The new library improved overall performance, but had different timeout behavior which could sometimes cause slow connections to be closed but without passing the error through to other components. Resolution: We have adjusted our configuration settings so that the new library behaves similarly to the previous library and the system is now stable.

Read the full incident report →