Asana experienced a notice incident on August 1, 2024 affecting App and App and 1 more component, lasting 4h 27m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Aug 01, 2024, 03:21 PM UTC
We are investigating application issues related to overload, and will provide updates.
- identified Aug 01, 2024, 04:04 PM UTC
We are continuing to work on a fix for this issue.
- identified Aug 01, 2024, 04:34 PM UTC
We are continuing to work on a fix for this issue.
- identified Aug 01, 2024, 04:37 PM UTC
We are continuing to work on a fix for this issue.
- identified Aug 01, 2024, 04:58 PM UTC
We are continuing to work on a fix for this issue.
- identified Aug 01, 2024, 05:03 PM UTC
We are continuing to work on a fix for this issue.
- identified Aug 01, 2024, 05:05 PM UTC
We are continuing to work on a fix for this issue.
- identified Aug 01, 2024, 05:33 PM UTC
We are continuing to work on a fix for this issue.
- identified Aug 01, 2024, 06:13 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Aug 01, 2024, 06:35 PM UTC
We've identified the root cause and are monitoring our systems
- resolved Aug 01, 2024, 07:49 PM UTC
This incident has been resolved.
- postmortem Aug 02, 2024, 12:16 AM UTC
**Incident:** On 2024-08-01 at 14:09 UTC, an increased connection rate on a central database saturated the database’s ability to accept new connections. As a result, Asana became unavailable to users in all regions. In response, Asana engineers throttled traffic for automations, webhooks, and API, and were able to restore primary functionality of the Asana application at 15:01 UTC. The application continued to run with sporadic unavailability until 18:22 UTC, when the underlying cause of the increased connections was determined and disabled. At this point the application was fully operational, but asynchronous workloads such as automations and notifications were delayed due to the work enqueued while traffic was throttled. The incident was resolved when asynchronous work queues fully caught up at 19:39 UTC. **Impact:** From 14:09 UTC to 15:01 UTC, users were unable to access Asana through the web, mobile, or API. From 15:01 UTC to 18:22 UTC users could access Asana through the web and mobile, but experienced sporadic unavailability. Access to the API was restored gradually between 15:15 UTC and 15:58 UTC. From 15:01 UTC to 19:39 UTC users experienced delayed automations, background actions, webhooks, and event streams. No customer data was lost. **Moving forward:** We have reverted the change that caused the increased connections. We are adding monitoring to detect changes in database connection rates, so we can identify potential problems earlier. Beyond this, we will be replicating data from the central database to other databases to reduce the impact of any single database being overloaded.