Git Integration for Jira incident

Potentially Degraded Performance

Major Resolved View vendor source →

Git Integration for Jira experienced a major incident on April 18, 2020 affecting Git Integration for Jira Cloud - Global Region, lasting 16d 8h. The incident has been resolved; the full update timeline is below.

Started
Apr 18, 2020, 10:21 AM UTC
Resolved
May 04, 2020, 07:06 PM UTC
Duration
16d 8h
Detected by Pingoru
Apr 18, 2020, 10:21 AM UTC

Affected components

Git Integration for Jira Cloud - Global Region

Update timeline

  1. investigating Apr 18, 2020, 10:21 AM UTC

    Over the past ~5 hours we have been noticing degraded performance as well as timeout issues. We are actively investigating the root cause at the moment. The app might be a bit slow while we figure things out. If you encounter an issue, please open a support ticket.

  2. investigating Apr 18, 2020, 06:29 PM UTC

    We are continuing to investigate the issue. We will be disabling and enabling indexing (including Smart Commits) to assist with diagnosing the issue. Expect some disruption over the next few hours. We apologize for this interruption in service.

  3. investigating Apr 19, 2020, 04:47 AM UTC

    We are continuing to investigate the issue, and have deployed a few hotfixes for additional logging and debugging to better understand what's going on. Indexing of repos (including Smart Commits) is enabled and running, and should remain so for the next several hours. Albeit in a throttled fashion. Indexing of git tags is temporarily disabled. We apologize for this interruption in service.

  4. identified Apr 20, 2020, 05:05 AM UTC

    We believe we have identified the issue, and stabilized all services. Full post-mortem will be available after this incident is resolved. For most customers - the application and indexing is restored to normal. For customers using Azure DevOps or VSTS OAuth integrations - a Jira administrator will need to "Reconnect" the integration. This can be done by clicking on the integration name https://d.pr/i/pnxGmK then on the "Reconnect" button https://d.pr/i/5mo2Zo. For some customers - indexing is working through repositories and should finish over the next 24 hours. Tag indexing is paused generally and will be restored before this incident is resolved.

  5. monitoring Apr 21, 2020, 02:10 AM UTC

    Current status: - Indexing for new repositories (and the majority of customers) is working normally. - Only a few large indexing runs remain to catch up. - Some Azure DevOps/VSTS customers will still need to reconnect (see below). Tag indexing is paused for some customers and will be restored before this incident is resolved. Please contact [email protected] if you have any questions. -------- For customers using Azure DevOps or VSTS OAuth integrations - a Jira administrator will need to "Reconnect" the integration. This can be done by clicking on the integration name https://d.pr/i/pnxGmK then on the "Reconnect" button https://d.pr/i/5mo2Zo.

  6. monitoring Apr 22, 2020, 07:43 PM UTC

    Current status is similar to last post. This weekend we will be making additional changes related to the incident. - Indexing for new repositories (and the majority of customers) is working normally. - Only a few large indexing runs remain to catch up. - Some Azure DevOps/VSTS customers will still need to reconnect (see below). Tag indexing is paused for some customers and will be restored before this incident is resolved. Please contact [email protected] if you have any questions. -------- For customers using Azure DevOps or VSTS OAuth integrations - a Jira administrator will need to "Reconnect" the integration. This can be done by clicking on the integration name https://d.pr/i/pnxGmK then on the "Reconnect" button https://d.pr/i/5mo2Zo.

  7. monitoring Apr 24, 2020, 07:04 PM UTC

    Things are slowly returning back to normal. Current status: - Indexing should be pretty much back to normal, with the exception of Tag indexing, which is still paused for some customers. - There are a handful of repos that are not indexed yet due to performance reasons - Some Azure DevOps/VSTS customers will still need to reconnect (see below). We are releasing some bug fixes tonight, and will perform some additional maintenance tonight, Friday, April 24th from 11PM EST to 1AM EST. Please expect some downtime during this window. Please contact [email protected] if you have any questions. -------- For customers using Azure DevOps or VSTS OAuth integrations - a Jira administrator will need to "Reconnect" the integration. This can be done by clicking on the integration name https://d.pr/i/pnxGmK then on the "Reconnect" button https://d.pr/i/5mo2Zo.

  8. resolved May 04, 2020, 07:06 PM UTC

    The incident is resolved. Notes: 1. Tags processing is disabled for a very few number of customers. We aim to re-enable tags processing for these customers in a few weeks. 2. Customers using the Azure DevOps/VSTS OAuth integration must reconnect before reindexing will continue. Please contact [email protected] with any questions.