FlagSmith incident

Erroneous flag values

Minor Resolved View vendor source →

FlagSmith experienced a minor incident on July 4, 2023, lasting —. The incident has been resolved; the full update timeline is below.

Started
Jul 04, 2023, 10:30 AM UTC
Resolved
Jul 04, 2023, 10:30 AM UTC
Duration
Detected by Pingoru
Jul 04, 2023, 10:30 AM UTC

Update timeline

  1. resolved Jul 05, 2023, 08:43 AM UTC

    Customer reports of erroneous flag values being served in local evaluation mode.

  2. postmortem Jul 05, 2023, 10:02 AM UTC

    ## Summary of the issue Following a release of the Core API at 10:35 UTC, a regression was introduced which meant that the generated environment document contained erroneous flag values for those flags which had recent change requests that had not been committed \(and potentially deleted\). Since the environment document is used to generate the flags for the Edge API and SDKs running in local evaluation this meant that certain customers using these methods to evaluate their flags would have received erroneous flag values. In this situation, the flag values served were those that were included in the uncommitted change requests. ## Resolution steps At 17:20 UTC we were notified of this issue by a customer that was affected by the erroneous values. At 17:59 UTC the issue was identified and a fix was being developed. This fix was fully developed and released by 19:10 UTC and all affected environments were regenerated by 19:30 UTC. The PR for the fix can be found [here](https://github.com/Flagsmith/flagsmith/pull/2378) for those interested in reviewing further. ## Next steps / preventative measures In order to prevent these steps in the future we plan to expand our end-to-end testing suite to further cover our change requests workflows so that we can identify these issues earlier.