Nanonets Outage History

From 22:00 UTC to 23:15 UTC on May 14th, we observed performance degradation in the oneshot service affecting LLM Nano models. The issue was primarily caused by a network issue with our GPU service provider. Our team worked with the provider to stabilize the service and performance has since recovered. As an additional preventive measure, we are introducing stronger fallback mechanisms to route requests to more accurate backup models during such events. We apologize for the inconvenience caused and appreciate your patience.

This incident has been resolved.

Between 7:19 UTC and 7:29 UTC, a subset of users experienced intermittent 503 errors when accessing app.nanonets.com. A backend node went down and was replaced by a new node. Due to a DNS caching issue, traffic for some users could not be routed to the new node, resulting in failed requests. Other users were unaffected. The issue was identified and resolved by updating the load balancer configuration to ensure proper traffic routing. Preventive measures have been put in place to avoid recurrence.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

At 1:00 AM UTC on September 4th, we posted an incident titled “Zero learning model file processing failures.” Upon further investigation, we identified that the issue was isolated to a single model and was not platform-wide. No other models or regions were impacted. The previously posted incident has been removed. We would like to clarify that there has been no platform-wide downtime in any region. Thank you for your understanding.

During a planned migration activity, users were unexpectedly logged out of the platform between 11:50 UTC and 12:05 UTC. The disruption occurred because database reads failed for old pods during the migration, while new pods came up late, leading to temporary session validation failures. The issue was resolved once the new pods were fully operational. We sincerely apologize for the inconvenience caused and will ensure future migrations are executed with better coordination to avoid service disruption.

This incident has been resolved.

Between 15:05 UTC and 15:40 UTC on July 23, 2025, a small subset of users using the Sync API for Instant Learning models on app.nanonets.com may have experienced intermittent prediction failures. We identified an edge case in our system that triggered this issue under specific conditions. The issue was resolved promptly with a fix, and the system has been stable since. All other regions and services remained unaffected and continue to operate normally. We apologize for the inconvenience caused and appreciate your understanding.

On July 22nd, 2025, users of app.nanonets.com may have experienced intermittent slowness during the periods 08:45–08:56 UTC and 10:03–10:11 UTC. This issue was caused by an unusually high load on our primary database. Our engineering team promptly identified the root cause and deployed a fix. The system has since stabilized and is functioning normally. We appreciate your patience during this time. All other regions remained fully operational without any downtime.

Between 13:33 UTC and 13:44 UTC on June 24th 2025, we observed intermittent failures in the Prediction API for our EU region due to a critical infrastructure issue. Our alerting systems flagged the anomaly, and our engineers quickly identified and resolved the problem. Services in the EU region were fully restored by 13:44 UTC. Other regions remained stable and were not impacted during this period. We appreciate your patience and understanding.

This incident has been resolved.