Braintrust Outage History
Braintrust is up right nowPingoru has recorded 35 incidents for Braintrust: 31 in the last 12 months and 7 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 3 | 1 of 720 (0%) |
| May 2026 | 3 | 24 of 744 (3%) |
| June 2026 | 2 | 0 of 720 (0%) |
| July 2026 | 1 | 1 of 744 (0%) |
| August 2026 | 5 | 17 of 744 (2%) |
| September 2026 | 3 | 3 of 720 (0%) |
| October 2026 | 0 | 0 of 744 (0%) |
| November 2026 | 0 | 0 of 123 (0%) |
Incidents, newest first
Major September 30, 2026 Elevated errors and latency on the US data plane API 1h 18m Braintrust-Hosted Data Plane (US) (API), Web UI / Control Plane
Started Sep 30, 2026, 11:36 AM UTC· resolved Sep 30, 2026, 12:54 PM UTC
- investigating Sep 30, 2026, 11:36 AM UTC
We are investigating elevated errors affecting the Braintrust control plane APIs. Some users may experience failures when accessing projects, prompts, settings, or creating new resources.
- investigating Sep 30, 2026, 11:50 AM UTC
We are investigating elevated error rates and increased latency on the US centrally-hosted data plane. Some requests, including function invocations, BTQL queries and loading logs or experiments in the UI, may fail or time out.
- monitoring Sep 30, 2026, 12:35 PM UTC
We have applied a fix and are monitoring the results.
- resolved Sep 30, 2026, 12:54 PM UTC
The issue has been resolved.
Major September 9, 2026 Slow or failed data loading in US region 39m Braintrust-Hosted Data Plane (US) (API)
Started Sep 09, 2026, 09:30 PM UTC· resolved Sep 09, 2026, 10:10 PM UTC
- investigating Sep 09, 2026, 09:30 PM UTC
We're investigating slow or failed data loading affecting some customers on our US-hosted service. Affected pages may load slowly, hang, or show errors.
- monitoring Sep 09, 2026, 09:34 PM UTC
We've applied a mitigation for the data-loading issue affecting our US-hosted service and are monitoring performance. We'll share another update once we've confirmed stability.
- resolved Sep 09, 2026, 10:10 PM UTC
The issue causing slow or failed data loading in our US-hosted service has been resolved. We're sorry for the disruption.
Major September 4, 2026 Elevated 5XXs on Control Plane API 32m Web UI / Control Plane
Started Sep 04, 2026, 03:19 PM UTC· resolved Sep 04, 2026, 03:52 PM UTC
- investigating Sep 04, 2026, 03:19 PM UTC
We are investigating elevated errors affecting the Braintrust control plane APIs. Some users may experience failures when accessing projects, prompts, settings, or creating new resources.
- monitoring Sep 04, 2026, 03:34 PM UTC
We have mitigated the issue causing elevated errors in the Braintrust control plane APIs, we are continuing to monitor closely.
- resolved Sep 04, 2026, 03:52 PM UTC
The issue causing elevated errors in the Braintrust control plane APIs has been resolved.
Major August 28, 2026 Gateway 5xxs 13m AI Gateway
Started Aug 28, 2026, 06:41 PM UTC· resolved Aug 28, 2026, 06:54 PM UTC
- investigating Aug 28, 2026, 06:41 PM UTC
The gateway encountered a load spike leading to 5xx errors.
- resolved Aug 28, 2026, 06:54 PM UTC
The load spike has subsided and 5xxs are mitigated
Major August 28, 2026 Gateway 5xxs 8m AI Gateway
Started Aug 28, 2026, 03:47 PM UTC· resolved Aug 28, 2026, 03:55 PM UTC
- investigating Aug 28, 2026, 03:47 PM UTC
A fresh gateway deploy was attempted and led to 5xx errors, and then was immediately rolled back.
- resolved Aug 28, 2026, 03:55 PM UTC
The deploy was rolled back and gateway 5xxs were mitigated.
Major August 27, 2026 Gateway 5xxs 11h 25m AI Gateway
Started Aug 27, 2026, 01:50 AM UTC· resolved Aug 27, 2026, 01:15 PM UTC
- investigating Aug 27, 2026, 01:50 AM UTC
A deployment caused the gateway to begin returning 5xxs for a portion of traffic.
- resolved Aug 27, 2026, 01:15 PM UTC
The faulty deploy was rolled back and 5xxs were mitigated.
Minor August 19, 2026 Upstream package mirror is degraded, Brainstore is not currently affected 4h 33m
Started Aug 19, 2026, 06:01 PM UTC· resolved Aug 19, 2026, 10:35 PM UTC
- monitoring Aug 19, 2026, 06:01 PM UTC
The upstream APT package mirror on AWS us-east-1 is degraded. There is currently no downtime and we are looking into alternative package mirrors in the meantime. Please note that for both the Braintrust-hosted data plane and self-hosted data planes: EC2 instance replacement due to healthcheck failures or deploys may result in failed instance initialization and downtime until the upstream issue is resolved.
- resolved Aug 19, 2026, 10:35 PM UTC
The upstream APT package mirror in AWS us-east-1 has recovered and we're no longer seeing impact when starting new brainstore instances.
Minor August 3, 2026 Gateway 5XXs 27m AI Gateway
Started Aug 03, 2026, 07:52 PM UTC· resolved Aug 03, 2026, 08:19 PM UTC
- identified Aug 03, 2026, 07:52 PM UTC
We identified an issue causing 5XXs introduced by a recent Gateway deployment and are currently rolling it back.
- monitoring Aug 03, 2026, 08:06 PM UTC
The issue has been resolved. We are monitoring the service to confirm normal operation.
- resolved Aug 03, 2026, 08:19 PM UTC
The issue has been resolved and service has remained stable following the rollback. We will continue to monitor internally, but this incident is now considered resolved.
Critical July 16, 2026 High 504 error rate in EU data plane 28m Braintrust-Hosted Data Plane (US) (API), Braintrust-Hosted Data Plane (EU) (API)
Started Jul 16, 2026, 08:32 AM UTC· resolved Jul 16, 2026, 09:01 AM UTC
- investigating Jul 16, 2026, 08:32 AM UTC
The hosted EU data plane is currently experiencing a very high rate of 504 errors.
- resolved Jul 16, 2026, 09:01 AM UTC
We have reverted to an older configuration to avoid an AWS outage with Cloudfront.
Critical June 12, 2026 US Data plane outage 2m AI Gateway, Web UI / Control Plane
Started Jun 12, 2026, 06:42 PM UTC· resolved Jun 12, 2026, 06:44 PM UTC
- investigating Jun 12, 2026, 06:16 PM UTC
Our US data plane is encountering issues.
- resolved Jun 12, 2026, 06:26 PM UTC
A configuration change was identified as the root cause and reverted.
Major June 4, 2026 Issues loading logs page 16m Web UI / Control Plane
Started Jun 04, 2026, 10:51 PM UTC· resolved Jun 04, 2026, 11:07 PM UTC
- investigating Jun 04, 2026, 10:51 PM UTC
We're experiencing issues loading logs and some other pages in Braintrust. Our team is actively investigating and working to resolve.
- resolved Jun 04, 2026, 11:07 PM UTC
We have pushed a fix and the loading issues have been remediated.
Major May 26, 2026 Errors logging into Braintrust 37m Web UI / Control Plane
Started May 26, 2026, 09:33 PM UTC· resolved May 26, 2026, 10:10 PM UTC
- investigating May 26, 2026, 09:33 PM UTC
Our upstream auth provider is having issues. Fresh logins to Braintrust will fail with an error.
- resolved May 26, 2026, 10:11 PM UTC
Our auth provider has recovered.
Minor May 15, 2026 Automation degradation 22h 50m AI Gateway, Web UI / Control Plane
Started May 15, 2026, 08:00 PM UTC· resolved May 16, 2026, 06:50 PM UTC
- investigating May 15, 2026, 08:00 PM UTC
On May 15, some automations stopped running which impacted scorer executions.
- resolved May 16, 2026, 06:50 PM UTC
We determined the issue was caused by a role used by the automation service losing required permissions following a configuration change. The issue was reported on May 15 at approximately 1:00 PM PT and resolved on May 16 at 11:50 AM PT. To help prevent similar incidents, we have improved our alerting and added audit logging for role and permission changes.
Major May 5, 2026 High rate of errors on control plane 47m Web UI / Control Plane
Started May 05, 2026, 11:35 PM UTC· resolved May 06, 2026, 12:23 AM UTC
- investigating May 05, 2026, 11:35 PM UTC
We're observing a high rate of errors on the production control plane which is causing errors on braintrust.dev and data plane operations.
- identified May 06, 2026, 12:08 AM UTC
We have identified the cause and are working with an upstream provider on a fix.
- monitoring May 06, 2026, 12:10 AM UTC
We are actively monitoring the fix.
- resolved May 06, 2026, 12:23 AM UTC
We are no longer seeing issues.
- resolved May 06, 2026, 12:41 AM UTC
The issue was caused by our firewall provider unexpectedly blocking traffic despite safeguards we had in place to prevent this behavior. We are actively working with the vendor on additional mitigation measures to reduce the likelihood of this occurring again.
Critical April 26, 2026 Control plane outage 25m Web UI / Control Plane
Started Apr 26, 2026, 06:52 AM UTC· resolved Apr 26, 2026, 07:17 AM UTC
- investigating Apr 26, 2026, 07:17 AM UTC
Our upstream auth provider is having an outage. Users unable to login to Braintrust.
- resolved Apr 26, 2026, 07:23 AM UTC
Provider has recovered. Error levels back to zero.
Critical April 20, 2026 Control Plane Database Connectivity Issues 15m AI Gateway, Web UI / Control Plane
Started Apr 20, 2026, 03:33 AM UTC· resolved Apr 20, 2026, 03:49 AM UTC
- identified Apr 20, 2026, 03:33 AM UTC
We're currently experiencing an issue with database connectivity. We're currently attempting to mitigate by connecting through an alternative method.
- monitoring Apr 20, 2026, 03:46 AM UTC
We've updated our database connection method and are observing recovery. We're monitoring to look for remaining impact.
- resolved Apr 20, 2026, 03:49 AM UTC
We're no longer seeing impact from the connectivity issues and we are still investigating the root cause.
Critical April 13, 2026 Control plane database degradation 21m AI Gateway, Web UI / Control Plane
Started Apr 13, 2026, 09:30 PM UTC· resolved Apr 13, 2026, 09:51 PM UTC
- investigating Apr 13, 2026, 09:38 PM UTC
We experienced a brief period of degraded performance on the Braintrust hosted control plane due to elevated database load. The issue was caused by an internal traffic spike to a specific query pattern.
- resolved Apr 13, 2026, 09:51 PM UTC
Performance quickly returned to normal when the problematic query was identified and remediated, and we are implementing safeguards to prevent similar future issues.
Major March 31, 2026 Elevated control plane errors 12m Web UI / Control Plane
Started Mar 31, 2026, 05:38 PM UTC· resolved Mar 31, 2026, 05:50 PM UTC
- investigating Mar 31, 2026, 05:52 PM UTC
We're seeing a high rate of 5xx errors
- resolved Mar 31, 2026, 06:05 PM UTC
Error rates have dropped and functionality is restored.
Critical March 20, 2026 Database load issues with control plane 1h 8m Web UI / Control Plane
Started Mar 20, 2026, 11:07 PM UTC· resolved Mar 21, 2026, 12:15 AM UTC
- investigating Mar 20, 2026, 11:07 PM UTC
The control plane is experiencing very high load.
- monitoring Mar 20, 2026, 11:14 PM UTC
Error rates have recovered by database usage is still high.
- monitoring Mar 21, 2026, 12:09 AM UTC
Database size is being increased. This will cause a short outage.
- monitoring Mar 21, 2026, 12:15 AM UTC
DB upgrade complete. CPU usage appears to be at normal levels. Monitoring for now.
- resolved Mar 21, 2026, 12:23 AM UTC
Load issues have subsided
Major March 20, 2026 Authorization service in degraded state 29m Web UI / Control Plane
Started Mar 20, 2026, 09:52 AM UTC· resolved Mar 20, 2026, 10:21 AM UTC
- investigating Mar 20, 2026, 09:53 AM UTC
Degraded performance of our authorization provider is causing issues with accessing the control plane.
- monitoring Mar 20, 2026, 10:21 AM UTC
The authorization provider has applied a mitigation and we're seeing signs of recovery. We're continuing to monitor until all errors have subsided.
- resolved Mar 20, 2026, 10:32 AM UTC
The authorization provider has completed mitigation work and is reporting a full recovery in their systems. We are no longer observing impact to users.
Critical March 10, 2026 Control Plane Errors 11s Web UI / Control Plane
Started Mar 10, 2026, 07:25 PM UTC· resolved Mar 10, 2026, 07:25 PM UTC
- investigating Mar 10, 2026, 07:03 PM UTC
An upstream provider had a brief outage that caused high errors in our control plane
- resolved Mar 10, 2026, 07:11 PM UTC
Provider recovered
Major March 10, 2026 Authorization service in degraded state 19m Web UI / Control Plane
Started Mar 10, 2026, 04:18 PM UTC· resolved Mar 10, 2026, 04:38 PM UTC
- investigating Mar 10, 2026, 04:18 PM UTC
Degraded performance of our authorization provider is causing issues with accessing the control plane. The service provider is investigating the root issue.
- monitoring Mar 10, 2026, 04:35 PM UTC
The authorization provider has applied a mitigation and we're seeing signs of recovery. We're continuing to monitor until all errors have subsided.
- resolved Mar 10, 2026, 04:38 PM UTC
The authorization provider has completed mitigation work and is reporting a full recovery in their systems. We are no longer observing impact to users.
Major March 3, 2026 S3 Export Automations failing 33m Web UI / Control Plane
Started Mar 03, 2026, 03:50 PM UTC· resolved Mar 03, 2026, 04:24 PM UTC
- investigating Mar 03, 2026, 03:50 PM UTC
S3 automations have stopped running on the expected schedule they were set to.
- resolved Mar 03, 2026, 04:24 PM UTC
Affected a small subset of users.
Minor February 19, 2026 Authorization service in degraded state 1h 18m Web UI / Control Plane
Started Feb 19, 2026, 04:34 PM UTC· resolved Feb 19, 2026, 05:53 PM UTC
- investigating Feb 19, 2026, 04:34 PM UTC
Degraded performance of our authorization vendor is causing issues with user authorization accessing the control plane. The vendor has identified the issue and is working on a fix.
- investigating Feb 19, 2026, 05:20 PM UTC
We've observed improvement in authorization errors but our authorization provider is still reporting a degraded state. We're continuing to monitor.
- monitoring Feb 19, 2026, 05:53 PM UTC
Impact for Braintrust users appears to have been resolved for the past 30 minutes. We'll continue to monitor until our authorization provider reports full resolution.
- resolved Feb 19, 2026, 06:35 PM UTC
We have not observed any recurrence of issues and our authorization provider has reported full recovery and stable performance across their entire system.
Minor February 18, 2026 Control Plane UI Hanging 3m Web UI / Control Plane
Started Feb 18, 2026, 09:32 PM UTC· resolved Feb 18, 2026, 09:35 PM UTC
- investigating Feb 18, 2026, 09:32 PM UTC
At the current moment in time we have reports of hanging in the UI particularly in dataset or trace views. We are currently narrowing down the code change causing the issue. Please stand by we are aware and currently working to resolve.
- resolved Feb 18, 2026, 09:35 PM UTC
The issue has been resolved and you should be seeing no slowness in the Braintrust Control Plane UI. Please let us know if you have any issues.