Kickserv experienced a major incident on July 27, 2021 affecting Web Application (app.kickserv.com) and iOS App and 1 more component, lasting 21h 19m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jul 27, 2021, 04:17 PM UTC
We are currently seeing errors attempting to add charges to a job. This is related to a minor permissions change we just added to our API. The fix is in progress and shouldn't take long. Thanks for bearing with us.
- identified Jul 27, 2021, 04:19 PM UTC
This issue currently affects only users in the staff or tech roles. Admins and owners are unaffected. The fix is minor and is in progress.
- monitoring Jul 28, 2021, 01:27 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jul 28, 2021, 01:37 PM UTC
Staff and technician users are once again able to add job charges in the web and mobile apps. Thanks for your patience.
- postmortem Jul 28, 2021, 01:37 PM UTC
On Tuesday, July 27, our engineers deployed an update to permissions on our items API. This particular API call is is normally used to list products and services, and should only be accessible to users who have a role of Owner or Admin. However, after deployment, we received reports of problems adding job charges. After investigating, we found that the same API call used in the item list view was also being used to search for an item while adding new job charges. This meant that Staff and Tech users were no longer able to add a job charge without running into an error. As usual, the remedy for this sort of bug involves two steps: 1\) stop the immediate symptoms; and 2\) address the underlying architectural issue. We took care of step one by putting permissions back the way they were, and job charges are now functioning properly again. For step two, we’ll need to make the API work the way it always should’ve worked: use a different API call when searching for an item to add as a job charge. This is a very different use case than showing the main list of items: one is an administrative function, and the other is used in the normal job workflow. They should be treated differently. We’ll also be reviewing our internal system documentation to come up with a more reliable way of tracking what API calls are used in what places. The Kickserv codebase is large and constantly changing, and some parts of it are close to ten years old—which is a lot of lines of code for a developer to have to keep track of! We’re always looking for ways to improve quality of life for our engineers as well as our customers. We regret that this happened, and thanks for bearing with us.