Resolved after 3 hours 36 minutes, affecting Amp Agent, on September 8, 2026.
This morning's incident left 369 user connections to threads in a degraded state. They are now restored and working.
Affected components
Update timeline
- investigating Sep 08, 2026, 11:18 AM UTC
We're seeing elevated error rates and delays when creating and connecting to threads.
- monitoring Sep 08, 2026, 11:30 AM UTC
Thread error rates are back to normal, and we'll continue to monitor.
- monitoring Sep 08, 2026, 11:41 AM UTC
Prometheus autoscaling caused database pod eviction A Google managed Prometheus collector autoscaler increased the collectors’ resource requests and restarted collectors across the cluster. The collector pods had a wrongly configured too-high priority, higher than our FoundationDB storage pods. Kubernetes preempted FoundationDB pods to make room for the collectors. Replacement storage pods then had to wait for new nodes compatible with their storage and compute requirements. This took long enough for FoundationDB to begin data recovery, which extended the outage. The service recovered and was fully available again after about 20 minutes. We are changing the scheduling and capacity configuration so that monitoring workloads cannot evict database storage pods again.
- resolved Sep 08, 2026, 12:01 PM UTC
Database recovered, limits adjusted. What we're going to do: • Audit all of our resource priority classes • Introduce more spare capacity for storage nodes so that we don't have to wait for new nodes to come online • Revise alerts so it's faster than 5min for us to pinpoint where the failure comes from
- investigating Sep 08, 2026, 12:05 PM UTC
Some threads fail to connect sporadically
- monitoring Sep 08, 2026, 12:12 PM UTC
Updating the priority classes for our storage nodes lead to a re-deployment at the same time as another deployment was going on. We're disabling auto-deployments now while incidents are going on.
- resolved Sep 08, 2026, 12:33 PM UTC
Fully recovered, monitoring, and rolling out fixes for the problems we found
- investigating Sep 08, 2026, 02:31 PM UTC
We are experiencing database issues again and are investigating.
- resolved Sep 08, 2026, 02:55 PM UTC
This morning's incident left 369 user connections to threads in a degraded state. They are now restored and working.
Your product runs on more than Amp Code
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services