Asana Outage History

Incident: A gradual rollout changing Asana’s network routing resulted in certain internal requests for our non-North American clusters being misrouted. This caused sporadic errors loading information for users whose Asana domains are hosted in non-US clusters. ‌ Impact: Users with SAML SSO configured saw intermittent errors when attempting to login to Asana between November 18, 11:02 UTC and November 19, 01:27 UTC. ‌ Moving forward: We are improving our internal monitoring to promptly identify issues with future networking changes and working to make our login flow more resilient to data loading errors.Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.

Incident: An experimental flag was enabled that changed initial page loading behavior. With the flag enabled, users were inadvertently directed to load a code version that was not servable by all groups of servers. Subsequent requests would fail when requesting this incompatible version, including requests to report the page load errors. ‌ Impact: From 16:52 to 18:59 UTC, 10% of users were unable to load the Asana application in the desktop app or web browser. Existing sessions, as well as mobile and API traffic, were not impacted. ‌ Moving forward: We reverted the configuration change and identified gaps in our monitoring. We are adding monitoring and alerting for version mismatches during page load in order to more quickly identify similar regressions in the future. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.

**Incident**: On June 28, 2024 between 0334 and 0356 UTC, network maintenance was performed to remap internal endpoints, with unforecasted negative effects on communication between internal services. This caused a large amount of requests to Asana during the maintenance period to either drop or time out. **Impact**: App usability was degraded for US-based users, resulting in load times of up to a minute. Users of the mobile app in our US, European, Australian, and Japanese data centers were also affected to a lesser extent by elevated error rates. **Moving forward**: We've identified process gaps during internal retrospective. Moving forward, we'll perform internal endpoint remappings more incrementally to avoid app disruption, and will apply a more thorough approval process for similar maintenance procedures. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.

Incident: An off-peak routine update of our service handling new page loads caused us to deploy code that depended on an artifact that wasn't yet available. The net effect was that all traffic attempting to do a fresh page load was broken for 48 minutes from June 28 5:52 pm PT to June 28 6:39 pm PT. Impact: Asana was unavailable to new page loads, but existing sessions continued to work as normal. Moving forward: This incident involved interaction across teams and systems. Our postmortem of this event identified operational and tooling changes to reduce the likelihood of incidents -- and decrease the time to resolution. We have an ongoing [5 Whys](https://wavelength.asana.com/workstyle-ask-5-whys-to-get-to-the-root-of-any-problem/) analysis for this incident.