Charts are not loading on Atlassian Analytics

Update timeline

1. investigating Nov 11, 2023, 10:02 AM UTC

   We are investigating an issue with charts that is impacting Atlassian Analytics customers. We will provide more details within the next hour.

2. resolved Nov 11, 2023, 10:09 AM UTC

   Simple Between 09:05 UTC to 10:00 UTC, we experienced charts failing to load for Atlassian Analytics. The issue has been resolved and the service is operating normally. Detailed Between 09:05 UTC to 10:00 UTC, all customers experienced charts failing to load for Atlassian Analytics. The root cause was a upstream update that caused database connectivity problems. There is no impact to the Atlassian Data Lake. We have deployed a fix to mitigate the issue and have verified that the services have recovered. The conditions that cause the bug have been addressed and we’re actively working on a permanent fix. The issue has been resolved and the service is operating normally.

3. postmortem Nov 22, 2023, 05:12 PM UTC

   ### **SUMMARY** On November 11, 2023, between 9:07 and 10:03 UTC, customers using Atlassian Analytics were unable to view charts on their dashboards. The event was triggered by a routine upgrade of a database cluster used by the service responsible for querying the data used in Atlassian Analytics charts. The incident was detected within 10 minutes by our monitoring system and mitigated by redeploying the impacted services which put affected Atlassian systems into a known good state. The total time to resolution was about 56 minutes. ### **IMPACT** The overall impact was between 9:07 UTC and 10:03 UTC on November 11, 2023. The incident caused a service disruption for all customers using Atlassian Analytics during the outage window. More specifically, the service responsible for querying data to populate charts experienced errors leading to failures when viewing and creating charts. ### **ROOT CAUSE** The issue was triggered by a routine upgrade of a database cluster which caused an unexpected loss of connectivity with the service responsible for issuing queries for Atlassian Analytics. Investigation during the incident identified that this was due to the database cluster changing the IP address after the upgrade. The service did not react to this change in an expected manner ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We know that outages impact your productivity. While we design our services to be highly available and resilient to most failures, this scenario was not handled as expected. We are prioritizing the following actions to avoid repeating this type of incident: * Implement more robust automatic database failure recovery. * Improve existing and implement new monitoring processes in order to detect similar issues faster. * Improve our internal documentation to more quickly resolve similar issues. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support