Printful incident
Sync variants couldn't be edited for brief time.
Printful experienced a minor incident on August 3, 2022, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Aug 03, 2022, 01:07 PM UTC
We saw an issue where sync variants in store could not be edited, this was a regression from another launch earlier today. Bugfix for this issue was deployed live in around an hour and sync variant editing in store is currenty working again.
- postmortem Aug 03, 2022, 01:38 PM UTC
@martins.viksna: **Post mortem** - Edit sync variant incident on Aug 3 **Incident summary** * Between 12:47 and 14:47 users could not edit sync variants. Modal just didn’t open. * The event was triggered by a minor fix for Feature Flag to not load independently from other designmaker Feature Flags * The bug was caused by trying to access vuex store when it was not yet initialised * Issue was detected by sentry, and less than 60 users got affected. At 14:18 our dev team received the first report from CS team. I immediately knew where the possible issue could be and reverted the changes. **Leadup** I added The Feature Flag fix as a part of different issue. Unfortunately, I didn’t understand correctly the scope of this minor change and that lead to insufficient tests to catch the issue. **Fault** The issue was that there was a flow that called “inside label color group” Feature Flag before generator and vuex module was initialised. \(All flags for designmaker are called there in a bundle and then stored in vuex\) **Impact** Affected less than 60 users according to sentry. **Detection** The incident was reported to our dev team on Aug 3, 2022, 14:18 GMT\+0200. We immediately looked into it and soon found a cause. **Recovery** Fastest way to fix the issue was to revert the changes. Because the previous variant, when not finding cached Feature Flag, made a request to get it from DB. After that, the fix was confirmed locally. deployed to production and confirmed in the production. **Timeline** Aug 3, 2022, 12:47 GMT\+0200 fix for the inside label native templates was deployed. Aug 3, 2022, 14:18 GMT\+0200 the Account Specialist reported the issue Aug 3, 2022, 14:47 GMT\+0200 issue was fixed in the production **Lessons learned** * The scope of the changes was bigger than expected * Always look at sentry after deploying a fix **Corrective actions** * If change might affect more than one flow, that should be noted and tested * Always keep sentry open after a bug fix