Is MendMD Down Right Now? Live MendMD Status & Outages | IsDown incident
Portal Delays
Is MendMD Down Right Now? Live MendMD Status & Outages | IsDown experienced a minor incident on November 1, 2022 affecting Video Sessions and SMS Text Message Notifications and 1 more component, lasting 3h 2m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 01, 2022, 05:08 PM UTC
Our development team is currently investigating reports of delays occurring in the Mend portal, and we hope to have everything resolved as quickly as possible. We appreciate your patience.
- investigating Nov 01, 2022, 05:14 PM UTC
Our development team has implemented a fix to resolve the Mend portal delays and we are monitoring the results. We appreciate your patience.
- monitoring Nov 01, 2022, 05:15 PM UTC
Our development team has implemented a fix to resolve the Mend portal delays and we are monitoring the results. We appreciate your patience.
- resolved Nov 01, 2022, 08:11 PM UTC
This incident has now been fully resolved.
- postmortem Nov 02, 2022, 07:00 PM UTC
_**What happened?**_ * On Tuesday, November 1st, at approximately 1:04 pm EST, the Mend Development team discovered a large spike in video connection polling that caused additional requests to the Mend API to be delayed. This resulted in a system outage that lasted until 1:08pm EST. _**Why it happened?**_ * Mend Video utilizes a polling process that controls the online connectivity indicator for appointments. This polling process periodically updates to ensure that the users are still connected. Due to the timing and volume of the API calls received for this functionality, the process responsible for updating this indicator failed under the increased load. _**What did we do to fix the issue?**_ * At approximately 1:06pm EST on November 1st, the Mend team manually throttled API calls to this update endpoint in order to allow the system to recover. **How will we prevent this from happening in the future?** * Over the past several months, the Mend scaling team has been focused on re-working how this polling process works to remove the individual polling completely. This work has been staged for release, but was not yet deployed to production as of November 1st. We have expedited the release of these changes and deployed it in our production environment at 6am EST on November 2nd. The Mend team will continue to monitor this process, however, initial data has shown a drastic improvement in overall portal responsiveness and stability.