Nanonets incident
High Response Times for Non Instant learning models in US region
Nanonets experienced a major incident on February 21, 2025 affecting API, lasting 48m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 21, 2025, 02:19 PM UTC
We are currently investigating this issue.
- monitoring Feb 21, 2025, 03:04 PM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Feb 21, 2025, 03:04 PM UTC
We are continuing to monitor for any further issues.
- resolved Feb 21, 2025, 03:07 PM UTC
This incident has been resolved.
- postmortem Feb 21, 2025, 03:27 PM UTC
On 21st Feb 2025, from 6:00 PM IST to 08:30 PM IST, non-instant learning models on [app.nanonets.com](http://app.nanonets.com) experienced high response times and timeouts due to extreme load on our system. Although auto-scaling was triggered on time, one of our services choked under the increased throughput. Our engineers quickly identified the bottleneck and increased the service's throughput, normalizing the response times. We apologize for the inconvenience and we are implementing measures to enhance system resilience and better handle future traffic spikes.