Printful incident

Sync variants couldn't be edited for brief time.

Minor Resolved View vendor source →

Printful experienced a minor incident on August 3, 2022, lasting —. The incident has been resolved; the full update timeline is below.

Started
Aug 03, 2022, 02:00 PM UTC
Resolved
Aug 03, 2022, 02:00 PM UTC
Duration
Detected by Pingoru
Aug 03, 2022, 02:00 PM UTC

Update timeline

  1. resolved Aug 03, 2022, 01:07 PM UTC

    We saw an issue where sync variants in store could not be edited, this was a regression from another launch earlier today. Bugfix for this issue was deployed live in around an hour and sync variant editing in store is currenty working again.

  2. postmortem Aug 03, 2022, 01:38 PM UTC

    @martins.viksna: **Post mortem** - Edit sync variant incident on Aug 3 **Incident summary** * Between 12:47 and 14:47 users could not edit sync variants. Modal just didn’t open. * The event was triggered by a minor fix for Feature Flag to not load independently from other designmaker Feature Flags * The bug was caused by trying to access vuex store when it was not yet initialised * Issue was detected by sentry, and less than 60 users got affected. At 14:18 our dev team received the first report from CS team. I immediately knew where the possible issue could be and reverted the changes. **Leadup** I added The Feature Flag fix as a part of different issue. Unfortunately, I didn’t understand correctly the scope of this minor change and that lead to insufficient tests to catch the issue. **Fault** The issue was that there was a flow that called “inside label color group” Feature Flag before generator and vuex module was initialised. \(All flags for designmaker are called there in a bundle and then stored in vuex\) **Impact** Affected less than 60 users according to sentry. **Detection** The incident was reported to our dev team on Aug 3, 2022, 14:18 GMT\+0200. We immediately looked into it and soon found a cause. **Recovery** Fastest way to fix the issue was to revert the changes. Because the previous variant, when not finding cached Feature Flag, made a request to get it from DB. After that, the fix was confirmed locally. deployed to production and confirmed in the production. **Timeline** Aug 3, 2022, 12:47 GMT\+0200 fix for the inside label native templates was deployed. Aug 3, 2022, 14:18 GMT\+0200 the Account Specialist reported the issue Aug 3, 2022, 14:47 GMT\+0200 issue was fixed in the production **Lessons learned** * The scope of the changes was bigger than expected * Always look at sentry after deploying a fix **Corrective actions** * If change might affect more than one flow, that should be noted and tested * Always keep sentry open after a bug fix