Welkin Health incident
Welkin not operational: Cannot Login to Care, Admin & Designer
Welkin Health experienced a critical incident on June 21, 2022 affecting Care and Designer and 1 more component, lasting 2h 13m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jun 21, 2022, 11:10 PM UTC
On Tuesday June 21, 2022 , beginning at around time 3:45 PM PDT Welkin’s engineers could not Login into Welkin and ran into "Unknown errors" in the Login Page. We are currently working on identifying the root cause of the issue. We sincerely apologize for this disruption, and thank you for your patience. The Welkin Team
- identified Jun 21, 2022, 11:30 PM UTC
We have identified the root cause of the problem & are working on a solution. The estimated resolution time is 2 hours & 30 minutes. We sincerely apologize for this disruption, and thank you for your patience. The Welkin Team
- identified Jun 22, 2022, 12:08 AM UTC
Engineering has identified that root cause to be an issue with the code release pipeline & will provide a detailed postmortem this week. They have proposed to move forward to a later release 2022.6.4 instead of restoring the current release 2022.5.6.1. This might cause minor changes to configuration & improvements in PFA and few other fixes. This according to Engineering is the optimal solution to restore services with least impact. Please let us know if you have any questions or concerns. Thank you, The Welkin Team
- resolved Jun 22, 2022, 01:24 AM UTC
The issue is resolved & all systems are back up and running as of 5:27 PM PDT,Tuesday June 21st. Release notes will be published here shortly: https://release-notes.welkinhealth.com/ Please report any incidents on our support site: https://welkinhealth.atlassian.net/servicedesk/customer/portal/1 We sincerely apologize for this disruption, and thank you for your patience. The Welkin Team
- postmortem Jun 28, 2022, 05:06 PM UTC
Starting Tuesday, June 21st, 2022 beginning around 4:00 PM PST Welkin Health issued a partial release for a release scheduled for Friday, June 24, 2022. This error caused Welkin customers to experience performance degradation and interruptions.The service incident was resolved on June 21st at 05:27 PM PST. Users have since been able to resume normal activity on Welkin and no further issues related to this incident have persisted.**Service Performance Incident Summary:** * During release preparation, we had a human error that accidentally promoted release 2022.6.5 only partially and the release was stuck in a non functional way * Once that was identified, the best mitigation forward was to promote the full next release, 2022.6.6 that also contained a bug fix to mitigate the issue **Steps being taken to ensure this doesn’t happen in the future:** * Simplify release process and add additional fail safes. * Avoid rushed releases that require overtime and that stress critical resources On behalf of our team here at Welkin, we apologize for the service impact that your team may have experienced. We strive to deliver an exceptional experience for our customers and continue to implement changes in order to meet that standard. Don’t hesitate to reach out with any questions or concerns. The Welkin Team