Resolved after 35 minutes, affecting App and API, on February 5, 2025.
Incident: Starting at 2025-02-05 21:05 UTC, a configuration change caused a large increase in server logs, overloading logging infrastructure and causing server restarts. This caused cascade failure as other servers were overloaded. The configuration change was reverted, and our application recovered after the servers restarted, with recovery by around 2025-02-05 21:30. Impact: Users saw errors loading our application and loading data within the application. No customer data was lost. Moving forward: We have removed the source of logs which resulted in overload, and are improving monitoring and resilience of the system which triggered failure. We will also be moving to staged configuration rollouts to help reduce or avoid customer impact for similar issues. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
Affected components
Update timeline
- investigating Feb 05, 2025, 09:19 PM UTC
We are seeing a high rate of errors for the Asana application in the US region.
- investigating Feb 05, 2025, 09:22 PM UTC
We are continuing to investigate this issue.
- investigating Feb 05, 2025, 09:34 PM UTC
We have made configuration changes and are monitoring for recovery.
- monitoring Feb 05, 2025, 09:35 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Feb 05, 2025, 09:54 PM UTC
This incident has been resolved.
- postmortem Feb 07, 2025, 10:46 PM UTC
Incident: Starting at 2025-02-05 21:05 UTC, a configuration change caused a large increase in server logs, overloading logging infrastructure and causing server restarts. This caused cascade failure as other servers were overloaded. The configuration change was reverted, and our application recovered after the servers restarted, with recovery by around 2025-02-05 21:30. Impact: Users saw errors loading our application and loading data within the application. No customer data was lost. Moving forward: We have removed the source of logs which resulted in overload, and are improving monitoring and resilience of the system which triggered failure. We will also be moving to staged configuration rollouts to help reduce or avoid customer impact for similar issues. Our metric considers a weighted average of uptime experienced by users at each data center. The number of minutes of downtime shown reflects this weighted average.
Your product runs on more than Asana
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services