Resolved on January 7, 2025.
## **What Happened** Earlier this year, our team identified that the previous method of synchronizing Template Project entries could, in some cases, cause **database deadlocks**. These deadlocks occurred during Template Project page loads, when the system attempted to copy content from the Library entry each time the page was opened. Because of the database deadlock, some synchronizations failed. To eliminate this risk and improve stability, we updated the synchronization process in our October 15, 2025 release so that updates now occur through an **asynchronous background task** whenever a Library entry is modified. This successfully resolved the deadlock issue. However, Template Project entries that had already failed to synchronize earlier \(because of those deadlocks\) were **not automatically updated** by the new process. As a result, some entries in your account remained out of sync with their Library sources. ## **Customer Impact** This meant that in certain cases, content in Template Project entries did not reflect the latest updates from the corresponding Library entry. In some situations, footnotes and attachments were also affected. ## **How We Resolved It** To fully address the issue: * We performed targeted data updates to correct affected entries by synchronizing them with their Library sources. * We completed a full **backfill** of content, including footnotes and attachments. * We introduced a new automated **sanity check** that identifies unsynchronized entries going forward. * We added a **repair tool** that allows us to quickly fix any inconsistencies that are detected. ## **How We’re Preventing This Going Forward** We’ve updated our internal engineering process to ensure that when we move logic from real-time \(synchronous\) updates to background \(asynchronous\) tasks, we explicitly assess whether a **data backfill** is required. This helps prevent similar inconsistencies and ensures existing records remain aligned with new system behaviour.
Update timeline
- resolved Nov 25, 2025, 10:39 PM UTC
The issue has subsided as of 10:01am EST. The Loopio Engineering team is continuing to monitor our systems.
- postmortem Dec 10, 2025, 04:47 PM UTC
## **What Happened** Earlier this year, our team identified that the previous method of synchronizing Template Project entries could, in some cases, cause **database deadlocks**. These deadlocks occurred during Template Project page loads, when the system attempted to copy content from the Library entry each time the page was opened. Because of the database deadlock, some synchronizations failed. To eliminate this risk and improve stability, we updated the synchronization process in our October 15, 2025 release so that updates now occur through an **asynchronous background task** whenever a Library entry is modified. This successfully resolved the deadlock issue. However, Template Project entries that had already failed to synchronize earlier \(because of those deadlocks\) were **not automatically updated** by the new process. As a result, some entries in your account remained out of sync with their Library sources. ## **Customer Impact** This meant that in certain cases, content in Template Project entries did not reflect the latest updates from the corresponding Library entry. In some situations, footnotes and attachments were also affected. ## **How We Resolved It** To fully address the issue: * We performed targeted data updates to correct affected entries by synchronizing them with their Library sources. * We completed a full **backfill** of content, including footnotes and attachments. * We introduced a new automated **sanity check** that identifies unsynchronized entries going forward. * We added a **repair tool** that allows us to quickly fix any inconsistencies that are detected. ## **How We’re Preventing This Going Forward** We’ve updated our internal engineering process to ensure that when we move logic from real-time \(synchronous\) updates to background \(asynchronous\) tasks, we explicitly assess whether a **data backfill** is required. This helps prevent similar inconsistencies and ensures existing records remain aligned with new system behaviour.
Your product runs on more than Loopio
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services