Templafy experienced a major incident on March 31, 2025 affecting Account Management and Library & Dynamics, lasting 5h 37m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Mar 31, 2025, 07:53 AM UTC
A limited number of customers in our West Europe (Production 0) cluster are experiencing issues where feature flags are not enabling the intended functionality. Further updates will be posted here soon.
- monitoring Mar 31, 2025, 09:27 AM UTC
The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. We are observing the systems to prevent any further disruptions.
- resolved Mar 31, 2025, 01:31 PM UTC
The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.
- postmortem Apr 02, 2025, 11:52 AM UTC
Incident Initiation At 9:26 AM CET on March 31, 2025, engineers detected an issue affecting global feature flag functionality. Certain features controlled by feature flags, such as the collapsible task pane, became temporarily unavailable for a small subset of users in West Europe \(Production 0\). Investigation Between 9:30 AM and 10:10 AM, engineers conducted an investigation to identify the root cause. It was determined that the newly deployed Foundation package version contained a bug that disrupted global feature flag functionality. The impact was confined to West Europe \(Production 0\) systems that had upgraded to this version. Mitigation and Resolution At 10:10 AM, engineers initiated mitigation efforts by rolling back the faulty Foundation package version to the previous stable release. This rollback was applied across all affected services. At 11:21 AM, the final fix was successfully deployed to production, restoring feature flag functionality. To prevent further impact, production releases were temporarily blocked until a full fix was delivered across all systems. Impact and Scope Scope: The incident was limited to a subset of customer on our West Europe \(Production 0\) cluster who were using certain feature flags. User Experience: Feature flag-controlled functionalities, including the collapsible task pane, were temporarily unavailable. Post-Incident Actions To prevent similar incidents in the future, the engineering team will review the Foundation package update process and strengthen automated testing for feature flag dependencies before deployment.