Service Outage

Update timeline

1. investigating Oct 28, 2025, 07:10 PM UTC

   We are currently investigating this incident..

2. identified Oct 28, 2025, 07:49 PM UTC

   AWS ECS is experiencing high task launch error rates and increased number of ECS tasks failing to start in the US-EAST-1 Region. .

3. identified Oct 28, 2025, 09:27 PM UTC

   **Current Status:** We are experiencing intermittent service disruptions affecting a subset of customer requests. Some requests are completing successfully, while others are failing due to capacity constraints. **What's Happening:** Our systems are currently operating at capacity, causing some incoming requests to fail. While many requests continue to process normally, users may experience: * Failed API requests * Timeout errors * Intermittent connection issues **Root Cause:** We have identified the need to scale our server infrastructure to handle the current load. However, our cloud provider is experiencing an outage that is preventing us from provisioning additional servers at this time. **Next Steps:** Our team is actively monitoring the cloud provider's status and will scale our infrastructure immediately once their systems are restored. We are also exploring alternative mitigation strategies in the interim. **Timeline:** Resolution is dependent on our cloud provider restoring service. We will provide updates as soon as we have new information..

4. monitoring Oct 29, 2025, 03:11 AM UTC

   The service degradation has been mitigated and requests are now flowing through normally. All systems are operational. **Resolution:** We have successfully resolved the capacity issues and restored full service functionality. Customer requests are processing as expected, and error rates have returned to normal levels. Our team continues to closely monitor system performance and capacity to ensure stability..

5. resolved Oct 29, 2025, 04:39 AM UTC

   This incident has been resolved..