Nanonets Outage History

Our database was running at capacity due to disk usage surge, this lasted for an hour before we recognized the problem and scaled the database.

We observed processing delay on tasks in Agents Platform for a couple of hours caused by architectural changes introduced by our team to increase observability of our platform.

Between 04:21 AM UTC and 04:25 AM UTC on June 29th, app.nanonets.com experienced a brief service disruption, during which the application was unavailable. The issue was caused by an unexpected error encountered during backend infrastructure operations, resulting in a temporary interruption of service. The issue was identified and resolved promptly, and all services have been fully restored. We apologize for the inconvenience and appreciate your patience.

Frontend resources taking time to load causing UI to freeze in loading state. Appropriate CDN fixes were implemented to fix the issue.

Between 12:53 AM IST and 01:00 AM IST on June 1, app.nanonets.com experienced a brief service disruption, during which the application was unavailable. The issue was caused by an unexpected error encountered during backend infrastructure operations, resulting in a temporary interruption of service. The issue was identified and resolved promptly, and all services have been fully restored. We are reviewing the incident and strengthening our deployment and validation processes to further reduce the likelihood of similar occurrences. We apologize for the inconvenience and appreciate your patience.

From 22:00 UTC to 23:15 UTC on May 14th, we observed performance degradation in the oneshot service affecting LLM Nano models. The issue was primarily caused by a network issue with our GPU service provider. Our team worked with the provider to stabilize the service and performance has since recovered. As an additional preventive measure, we are introducing stronger fallback mechanisms to route requests to more accurate backup models during such events. We apologize for the inconvenience caused and appreciate your patience.

This incident has been resolved.

Between 7:19 UTC and 7:29 UTC, a subset of users experienced intermittent 503 errors when accessing app.nanonets.com. A backend node went down and was replaced by a new node. Due to a DNS caching issue, traffic for some users could not be routed to the new node, resulting in failed requests. Other users were unaffected. The issue was identified and resolved by updating the load balancer configuration to ensure proper traffic routing. Preventive measures have been put in place to avoid recurrence.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

This incident has been resolved.

At 1:00 AM UTC on September 4th, we posted an incident titled “Zero learning model file processing failures.” Upon further investigation, we identified that the issue was isolated to a single model and was not platform-wide. No other models or regions were impacted. The previously posted incident has been removed. We would like to clarify that there has been no platform-wide downtime in any region. Thank you for your understanding.

During a planned migration activity, users were unexpectedly logged out of the platform between 11:50 UTC and 12:05 UTC. The disruption occurred because database reads failed for old pods during the migration, while new pods came up late, leading to temporary session validation failures. The issue was resolved once the new pods were fully operational. We sincerely apologize for the inconvenience caused and will ensure future migrations are executed with better coordination to avoid service disruption.