AI Model Hub: Increased Error Rate

Update timeline

1. investigating Mar 09, 2026, 08:26 AM UTC

   Our Model Hub Team is currently working on resolving errors related to an instance running the llama 405b model.

2. identified Mar 09, 2026, 11:52 AM UTC

   The team has identified the root cause: hardware degradation affecting this model's hosting environment is causing backend instability. We are currently implementing a fix.

3. monitoring Mar 09, 2026, 06:53 PM UTC

   Our AI Model Hub Team has mitigated the incident. While the underlying root cause is not yet fully established or resolved, the model service should be stable. We are monitoring the situation while the investigation is ongoing

4. resolved Mar 11, 2026, 07:20 PM UTC

   We are marking this incident as resolved. The incident was caused by capacity constraints following a hardware failure. While capacity has been restored, we still see some usage‑specific constraints with the Llama 3.1 405B Instruct model. Our AI ModelHub team will deploy optimizations to the model to increase performance and reliability. We recommend that users still experiencing issues with the model check GPT‑OSS 120B as a potential (temporary) replacement.