Templafy incident
Service degradation - Email signature synchronizations taking longer then usual
Templafy experienced a minor incident on July 23, 2024 affecting Email Signature, lasting 7h 14m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jul 23, 2024, 09:13 AM UTC
We have identified an issue that affects a subset of customers and are working towards a resolution. Further updates will be posted here soon.
- monitoring Jul 23, 2024, 11:07 AM UTC
The incident has been successfully mitigated, and our team is actively monitoring the situation to ensure ongoing stability and performance. Signature processing is working within the desired parameters now, and prior synchronization requests are being picked up and processed. We are observing the systems to prevent any further disruptions.
- resolved Jul 23, 2024, 04:27 PM UTC
The incident has been resolved, and further information will be provided in a postmortem shortly. We apologize for the impact to affected customers.
- postmortem Jul 26, 2024, 12:55 PM UTC
On July 22, 2024, at 11:04 AM CET, an incident was initiated when the email signature synchronization process took longer than usual to complete, causing delays in distributing email signature updates to end users. The engineering team quickly initiated an investigation and identified that the issue was caused by message processing. The problem arose due to a recent migration to a new messaging framework, which began to exhibit performance issues. To mitigate the problem, the team reverted to the previous messaging framework, confirmed that the synchronization was running within the desired parameters, and by 1:07 PM CET, managed to deploy the fixed version to all production environments. The team continued to monitor system performance and, after ensuring everything was operating at the desired performance metrics, marked the incident as resolved at 6:27 PM CET. The engineering team is currently conducting extensive tests to identify the root cause of the performance issues in the new messaging framework. Additionally, they are focusing on further quality automation and performance checks to prevent similar incidents in the future.