BentoML experienced a major incident on October 30, 2025 affecting BentoCloud Deployment Services, lasting 23h 17m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 30, 2025, 12:45 PM UTC
We are investigating a new GKE update to containerd which is affecting new GPUs scaling up. This is applies to GCP deployments when autoscaling is triggered for GPU based services
- identified Oct 30, 2025, 02:32 PM UTC
A patch update that automatically rolled in by GKE has affected our custom containerd configurations. This has brought down some critical services in some nodes and has affected some deployments. We have identified a pathway to fix and is currently implementing it
- identified Oct 30, 2025, 04:16 PM UTC
A hotfix is being patched. Working on a long term fix at the moment.
- monitoring Oct 31, 2025, 12:04 AM UTC
The issue should be resolved. Continuing investigation and monitoring for further issues
- resolved Oct 31, 2025, 12:03 PM UTC
No further issues found in the last 18 hours. This issue is resolved by rolling back a minor GKE version.