Resolved after 23 hours 52 minutes, on March 24, 2023.
"TFS4JIRA Azure DevOps integration" app experienced partial outage between 2:30 UTC and 8:50 UTC on 24th March 2023. During this time, user interface of our app inside Jira was unavailable for all customers. Synchronization of Azure work items and Jira issues was working correctly. Root cause TFS4JIRA uses a library called "Liquibase" to maintain a database schema. The library sets a lock in the DATABASECHANGELOGLOCK table for a few seconds while the application is starting. This is to avoid two instances of TFS4JIRA upgrading the database schema at the same time. The lock was set at 02:22 UTC and was never released. This can happen e.g. if the application server is killed right after setting the lock but before it is released. The stale lock blocked new application servers from starting. Those servers were responsible for handling user interface related requests. Post mortem & lessons learned The stale lock has been released manually and this immediately unlocked all the awaiting application servers which started to handle requests properly. Our team is going to work on permanent solution which will allow our app to automatically recover from such problem in the future.
Update timeline
- resolved Mar 24, 2023, 11:22 AM UTC
"TFS4JIRA Azure DevOps integration" app experienced partial outage between 2:30 UTC and 8:50 UTC on 24th March 2023. During this time, user interface of our app inside Jira was unavailable for all customers. Synchronization of Azure work items and Jira issues was working correctly. Root cause TFS4JIRA uses a library called "Liquibase" to maintain a database schema. The library sets a lock in the DATABASECHANGELOGLOCK table for a few seconds while the application is starting. This is to avoid two instances of TFS4JIRA upgrading the database schema at the same time. The lock was set at 02:22 UTC and was never released. This can happen e.g. if the application server is killed right after setting the lock but before it is released. The stale lock blocked new application servers from starting. Those servers were responsible for handling user interface related requests. Post mortem & lessons learned The stale lock has been released manually and this immediately unlocked all the awaiting application servers which started to handle requests properly. Our team is going to work on permanent solution which will allow our app to automatically recover from such problem in the future.
Your product runs on more than Appfire
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services