iSalus Healthcare incident

Unable to access OfficeEMR/Billing Issues

Resolved after 1 day 3 hours, on March 2, 2025.

postmortem Mar 07, 2025, 09:49 PM UTC

**Post Incident Report** iSalus | External Distribution | March 7, 2025 **Incident Overview** On March 2, 2025, from approximately 1:45 AM to 2:32 PM ET, some customers experienced intermittent performance degradation and connectivity issues within their cloud environments. Our team immediately began investigating and identified that the disruption was caused by an unexpected issue within the cloud networking infrastructure. Services were fully restored following corrective actions, and we have taken steps to prevent a recurrence. **Timeline of Events \(All Times ET\)** • March 2, 2025, 1:25 AM – Automated monitoring systems detected degraded performance and connectivity issues. • March 2, 2025, 7:51 AM – Customers began reporting service interruptions. • March 2, 2025, 8:03 AM – Additional technical resources were engaged to assess the issue and implement corrective actions. • March 2, 2025, 1:45 PM – Root cause analysis identified an issue related to cloud networking components. • March 2, 2025, 5:32 PM – A final remediation step was completed, resolving the service impact. • March 3, 2025, 8:10 AM – Stability was confirmed, and we marked the incident as fully resolved. • March 6, 2025, 10:00 PM – Vendor technicians rolled back temporary network routing from secondary back to primary. **Root Cause Analysis** The issue was caused by an unexpected condition within the cloud networking infrastructure that resulted in excessive resource utilization. This led to degraded system performance and connectivity interruptions. Once identified, our team took corrective measures to restore normal operations. **Corrective and Preventative Actions** To prevent similar issues in the future, we are implementing the following measures: • Proactive Monitoring Enhancements – Additional safeguards have been introduced to detect and respond to early signs of resource strain. • Process Improvements – We are reviewing and refining our network management procedures to prevent recurrence. • Ongoing Analysis – Our team continues to evaluate the incident to identify further optimizations and improvements. ‌ We sincerely apologize for any inconvenience this incident may have caused. We appreciate your patience and trust as we continuously work to enhance the reliability of our services. If you have any further questions, please contact our support team.

Started
Mar 02, 2025, 12:35 PM UTC
Resolved
Mar 03, 2025, 04:24 PM UTC
Duration
1d 3h
Detected by Pingoru
Mar 02, 2025, 12:35 PM UTC

Update timeline

  1. investigating Mar 02, 2025, 12:35 PM UTC

    We are aware that there is an issue with accessing the Officemd.net site and OfficeEMR Mobile application affecting all of our customers at this time. Please know our development team is working diligently towards a resolution, and we will provide an update as soon as possible.

  2. investigating Mar 02, 2025, 12:51 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will provide a final update as soon as possible. We thank you for your patience.

  3. investigating Mar 02, 2025, 01:08 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will provide a final update as soon as possible. We thank you for your patience.

  4. investigating Mar 02, 2025, 01:23 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will provide a final update as soon as possible. We thank you for your patience.

  5. investigating Mar 02, 2025, 01:38 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will provide a final update as soon as possible. We thank you for your patience.

  6. investigating Mar 02, 2025, 01:53 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will provide a final update as soon as possible. We thank you for your patience.

  7. investigating Mar 02, 2025, 02:08 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will provide a final update as soon as possible. We thank you for your patience.

  8. investigating Mar 02, 2025, 02:24 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will provide a final update as soon as possible. We thank you for your patience.

  9. investigating Mar 02, 2025, 02:39 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will provide a final update as soon as possible. We thank you for your patience.

  10. investigating Mar 02, 2025, 02:55 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will provide a final update as soon as possible. We thank you for your patience.

  11. investigating Mar 02, 2025, 03:10 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will provide a final update as soon as possible. We thank you for your patience.

  12. investigating Mar 02, 2025, 03:26 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will provide a final update as soon as possible. We thank you for your patience.

  13. investigating Mar 02, 2025, 03:46 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will provide a final update as soon as possible. We thank you for your patience.

  14. investigating Mar 02, 2025, 04:01 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will provide a final update as soon as possible. We thank you for your patience.

  15. investigating Mar 02, 2025, 04:16 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will provide a final update as soon as possible. We thank you for your patience.

  16. investigating Mar 02, 2025, 04:36 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will provide a final update as soon as possible. We thank you for your patience.

  17. investigating Mar 02, 2025, 04:58 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will provide a final update as soon as possible. We thank you for your patience.

  18. investigating Mar 02, 2025, 05:15 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will continue to provide updates every 30 minutes. We thank you for your patience.

  19. investigating Mar 02, 2025, 05:47 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will continue to provide updates every 30 minutes. We thank you for your patience.

  20. investigating Mar 02, 2025, 06:18 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will continue to provide updates every 30 minutes. We thank you for your patience.

  21. investigating Mar 02, 2025, 06:47 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will continue to provide updates every 30 minutes. We thank you for your patience.

  22. investigating Mar 02, 2025, 07:18 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will continue to provide updates every 30 minutes. We thank you for your patience.

  23. investigating Mar 02, 2025, 07:49 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will continue to provide updates every 30 minutes. We thank you for your patience.

  24. investigating Mar 02, 2025, 08:19 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing officemd.net. We will continue to provide updates every 30 minutes. We thank you for your patience.

  25. identified Mar 02, 2025, 08:54 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing OfficeEMR. We will provide a final update as soon as possible. We thank you for your patience.

  26. identified Mar 02, 2025, 09:25 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing OfficeEMR. We will provide a final update as soon as possible. We thank you for your patience.

  27. identified Mar 02, 2025, 10:00 PM UTC

    Please know our teams are still actively working to resolve your current issue with accessing OfficeEMR. We will provide a final update as soon as possible. We thank you for your patience.

  28. identified Mar 02, 2025, 10:31 PM UTC

    Please know our teams are still actively working to fully resolve your current issue with OfficeEMR. Limited functionality may now be available, and we will provide a final update as soon as possible. We thank you for your continued patience.

  29. monitoring Mar 02, 2025, 11:07 PM UTC

    Please know the issue with accessing OfficeEMR has currently been resolved. We will continue to monitor this and provide a final update shortly. We thank you for your patience!

  30. monitoring Mar 03, 2025, 02:42 AM UTC

    Please know the issue with general OfficeEMR access and EMR functionality has been resolved. OUTSTANDING ISSUE: Our teams continue to work with our partners to reestablish connections for all billing processes. Claim processing, ERAs, and eligibility will be unavailable at this time. We will continue to provide updates as soon as possible. We thank you for your patience.

  31. monitoring Mar 03, 2025, 12:12 PM UTC

    Please know the issue with general OfficeEMR access and EMR functionality has been resolved. OUTSTANDING ISSUE: Our teams continue to work with our partners to reestablish connections for all billing processes. Claim processing, ERAs, and eligibility will be unavailable at this time. We will continue to provide updates as soon as possible. We thank you for your patience!

  32. monitoring Mar 03, 2025, 01:49 PM UTC

    OUTSTANDING ISSUE: Our teams continue to work with our partners to reestablish connections for all billing processes. Claim processing, ERAs, and eligibility will be unavailable at this time. We will continue to provide updates as soon as possible. We thank you for your patience!

  33. monitoring Mar 03, 2025, 02:54 PM UTC

    Please know the issue with billing connections for claims submissions, ERAs, and eligibility has currently been resolved. We will continue to monitor this and provide a final update shortly. We thank you for your patience!

  34. resolved Mar 03, 2025, 04:24 PM UTC

    The issues with OfficeEMR access and billing processes have been fully resolved. We appreciate your patience during this time. Please reach out to support for any further questions/issues.

  35. postmortem Mar 07, 2025, 09:49 PM UTC

    **Post Incident Report** iSalus | External Distribution | March 7, 2025 **Incident Overview** On March 2, 2025, from approximately 1:45 AM to 2:32 PM ET, some customers experienced intermittent performance degradation and connectivity issues within their cloud environments. Our team immediately began investigating and identified that the disruption was caused by an unexpected issue within the cloud networking infrastructure. Services were fully restored following corrective actions, and we have taken steps to prevent a recurrence. **Timeline of Events \(All Times ET\)** • March 2, 2025, 1:25 AM – Automated monitoring systems detected degraded performance and connectivity issues. • March 2, 2025, 7:51 AM – Customers began reporting service interruptions. • March 2, 2025, 8:03 AM – Additional technical resources were engaged to assess the issue and implement corrective actions. • March 2, 2025, 1:45 PM – Root cause analysis identified an issue related to cloud networking components. • March 2, 2025, 5:32 PM – A final remediation step was completed, resolving the service impact. • March 3, 2025, 8:10 AM – Stability was confirmed, and we marked the incident as fully resolved. • March 6, 2025, 10:00 PM – Vendor technicians rolled back temporary network routing from secondary back to primary. **Root Cause Analysis** The issue was caused by an unexpected condition within the cloud networking infrastructure that resulted in excessive resource utilization. This led to degraded system performance and connectivity interruptions. Once identified, our team took corrective measures to restore normal operations. **Corrective and Preventative Actions** To prevent similar issues in the future, we are implementing the following measures: • Proactive Monitoring Enhancements – Additional safeguards have been introduced to detect and respond to early signs of resource strain. • Process Improvements – We are reviewing and refining our network management procedures to prevent recurrence. • Ongoing Analysis – Our team continues to evaluate the incident to identify further optimizations and improvements. ‌ We sincerely apologize for any inconvenience this incident may have caused. We appreciate your patience and trust as we continuously work to enhance the reliability of our services. If you have any further questions, please contact our support team.

Your product runs on more than iSalus Healthcare

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →