Bitbucket Cloud web, api, and Pipelines service outage

Update timeline

1. investigating Jan 21, 2025, 03:30 PM UTC

   We are investigating an issue with Bitbucket Web and Git operations that is impacting Atlassian Bitbucket Cloud customers. We will provide more details within the next hour.

2. investigating Jan 21, 2025, 03:58 PM UTC

   We are still investigating an issue with Bitbucket Web and Git operations that is impacting Atlassian Bitbucket Cloud customers. We will provide more details within the next hour.

3. investigating Jan 21, 2025, 05:07 PM UTC

   We are investigating an issue with saturated Bitbucket database that impacts all Bitbucket operations. We will provide more details within the next hour.

4. investigating Jan 21, 2025, 05:49 PM UTC

   We are investigating an issue with saturated Bitbucket database that impacts all Bitbucket operations. We will provide more details within the next 30 minutes.

5. monitoring Jan 21, 2025, 06:02 PM UTC

   We have identified the root cause of the database issue and have mitigated the problem. We are now monitoring closely.

6. monitoring Jan 21, 2025, 07:28 PM UTC

   We have identified the root cause of the database issue that impacted Bitbucket website and Git operations; this has been mitigated now. We are experiencing Pipelines degradation that we are working to resolve.

7. monitoring Jan 21, 2025, 08:08 PM UTC

   All Git, Web, API and Pipelines services are now operational. We are continuing to monitor database and Pipelines reliability.

8. resolved Jan 22, 2025, 12:08 AM UTC

   Earlier we experienced database contention on high traffic tables, which resulted in website, API, and Pipelines outages. All Bitbucket services are now operational. A full post mortem will be published.

9. postmortem Jan 28, 2025, 06:13 PM UTC

   ### Summary On January 21, 2025, between 14:02 and 17:49 UTC, Atlassian customers using Bitbucket Cloud were unable to use the website, API, or Pipelines. The event was triggered by write contention in a high traffic database table. The incident was detected within eight minutes. We then worked to both throttle traffic and improve query performance, which allowed services to recover. The total time to resolution was about three hours and 47 minutes. ### **IMPACT** The overall impact was between 14:02 and 17:49 UTC, affecting Bitbucket Cloud. This impacted customers globally, and they were unable to use the website, APIs, or Pipelines services. Git hosting \(SSH\) was unaffected. ### **ROOT CAUSE** The issue was caused by an increase in API traffic triggering write contention on a high-traffic table, resulting in increased CPU usage and degraded database performance. This ultimately impacted the availability of core services \(web, API, and Pipelines\). ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. While we have several testing and preventative processes in place, this specific issue wasn’t identified because the code path being triggered does not commonly experience this type of traffic. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Running additional maintenance on core database tables * Added throttling on write-heavy operations To improve service resilience and recovery time for our environments, we will implement additional preventative measures such as: * Improving database observability to isolate failures * Continuing to shard data to better distribute traffic load We apologize to customers whose services were impacted by this incident and are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support