Nanonets incident
Delayed file processing for all ILM models in app.nanonets.com
Nanonets experienced a major incident on January 28, 2026 affecting API, lasting 1h 50m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jan 28, 2026, 03:09 PM UTC
We are currently investigating this issue.
- identified Jan 28, 2026, 03:35 PM UTC
We have identified the issue. A secondary database migration impacted indexes on a few core tables, leading to the current disruption. Our engineering team is actively working on restoring the database and we expect processing to resume once recovery is complete. We apologize for this unanticipated issue and will share an update as soon as services are fully restored.
- identified Jan 28, 2026, 04:20 PM UTC
We have recovered the db. Processing has resumed and we are working on clearing the backlog.
- monitoring Jan 28, 2026, 04:39 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jan 28, 2026, 04:59 PM UTC
This incident has been resolved.
- postmortem Jan 30, 2026, 05:46 AM UTC
**Incident Summary** On 28th Jan 03:00 PM UTC, a secondary database migration unintentionally impacted indexes on a small number of core database tables. This led to degraded performance and temporary disruption of processing. **Impact** During this period, some prediction requests were delayed or unable to complete successfully until the database was stabilized. **Root Cause** A database schema change applied as part of a migration caused unintended modifications to critical indexes. While the migration itself completed, the index impact affected normal query performance on core tables because of high traffic volume. **Resolution** Our engineering team promptly identified the issue and initiated recovery actions, including restoring affected indexes and stabilizing database performance. Once recovery was complete, normal processing resumed. **Preventive Measures** To avoid similar incidents in the future, we are implementing the following improvements: * Database migrations will no longer be executed during peak usage hours for secondary database as well. * All future migrations will be scheduled during low-traffic windows with defined rollback plans. * Additional pre-deployment checks and safeguards will be added to detect potential impact on core database structures. We sincerely apologize for the inconvenience this incident may have caused. We understand the importance of reliability and take full responsibility for the disruption. Please be assured that we are taking concrete steps to ensure safer deployments and uninterrupted service going forward.