Retrospective: Redis connection errors occurring for some projects

Update timeline

1. investigating Aug 28, 2026, 09:06 AM UTC

   We have noted increased error rates on a subset of projects caused by connectivity issues with projects' redis instances.

2. resolved Aug 28, 2026, 09:28 AM UTC

   The issue has now been fully resolved. The issue was caused by a deployment misconfiguration which prevented PHP from being able to communicate with redis. We are currently in the process of refactoring the use of Redis (which has changed is licensing to prevent commercial use) within projects deployed to the Servd platform. The first step of this change is to create a set of feature gates within our deployment mechanism which allow us to flip a project between Redis and the proposed replacement (currently DragonflyDB). Yesterday we released these feature gates to all projects' deployment mechanisms with the default still set to use Redis. We were therefore expecting no change to any projects at this time, but would have the ability to flip the feature gates on a per project basis in the future. This was tested on multiple internal projects before rolling out generally. At 01:30 BST we were alerted to elevated 500 errors from multiple projects. Upon inspection these projects seemed to be having issues communicating with their Redis instances. We assumed this must be related to the feature gate changes that had rolled out earlier in the day. These projects had all had their subscriptions renewed over the past 30 minutes which was the apparent trigger. During subscription renewals we perform a 'partial sync' on projects - syncing updates related to changes to project subscriptions without deploying any settings changes made in the Servd dashboard. This relies on taking a snapshot of the project's previous sync's deployment values and merging those with any changes due to the subscription. This resulted in the new feature gates having no default value set - the sync was executed with them as empty values. Our deployment scripts also contained logic that checked for the value of the feature gates to direct traffic to either redis or its replacement. This logic failed due to the empty value of the feature gate (it expected a specific boolean value) which led to the bad configuration being deployed. To fix the problem we ran through all impacted projects and executed a 'full sync' on each of them. This ensured a non-empty value was applied for the feature gates and allowed a correct configuration to be deployed. We also subsequently updated the deployment scripts to handle the empty value elegantly to ensure the problem does not repeat for further project renewals. The total impact on projects varied in time as we ran through them from start to finish. We have also adjusted our testing methodology for any deployment script updates which include new values, to include testing with partial syncs to ensure they work as expected.