Welkin Health incident
L2 Partial Outage: January 08 2021 Welkin Health API Outage
Welkin Health experienced a minor incident on January 8, 2021 affecting Welkin API, lasting 1d. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jan 08, 2021, 06:04 PM UTC
On January 08, 2021, beginning at around 6:00 AM, Welkin’s customer's began experiencing API issues. This partially impacted some of our components that used API in our Coach Portal & Workshop. This could have impacted custom code which uses the API such as Web hooks. The root cause has been identified and we are working on an out of band release as soon as we are able. We sincerely apologize for this disruption, and thank you for your patience. The Welkin Team
- identified Jan 08, 2021, 07:36 PM UTC
We’re still working on a solution and will continue over the weekend if necessary.
- identified Jan 08, 2021, 08:09 PM UTC
We currently have found that this is impacting only a few customers that use app messages within the Welkin UI. We have confirmed that all other Welkin API & Webhooks are functioning normally. We continue to work on a solution to the issue. Additionally we are changing the subject message to downgrade it to a partial L2 outage.
- identified Jan 09, 2021, 12:33 AM UTC
Engineering is working on a solution & current ETA for release is January 9th 11:00AM PDT
- resolved Jan 09, 2021, 06:40 PM UTC
The partial outage is resolved & the service has been fully restored!
- postmortem Jan 12, 2021, 03:22 AM UTC
Engineering work on our SMS and App Messaging features in Coach Portal introduced a defect into the platform on Thursday, Jan 7, 6am Pacific time. When users attempted to send App Messages from the Welkin Coach Portal, they may have received an error message and the App Message would not be sent. The problem did not affect App Messages sent through Welkin’s APIs, only from the UI. To resolve the issue, Welkin Engineering team created a fix for the issue & executed an emergency out of band hot-fix release. This issue was resolved on Saturday, Jan 9, 7am Pacific time. Welkin plans to review & improve the quality of hot fixes and make appropriate changes to prevent or minimize this occurrence in future. On behalf of our team here at Welkin, please accept our sincere apology for the service impact that your team likely experienced. We strive to deliver an exceptional experience for our customers and continue to implement changes in order to meet that standard. Please do not hesitate to reach out with any additional questions or feedback regarding this outage.