Writer Outage History

The issue affecting WRITER Agent has been resolved. All systems are now operating normally. We apologize for any inconvenience this may have caused.

In WA after authenticating and selecting the connector to retrieve context some users but not all are prompted to configure the connector in a loop until it fails to content.

TemplateHighP95Latency

On 2026-03-10, Writer Agent experienced a SEV1 full outage driven by database lock contention during deployment. An in-band schema migration waited on long-lived idle-in-transaction sessions, causing query queueing on core.threads and rapid connection pool exhaustion across pods. Service recovered after terminating blocking sessions and stabilizing rollout; follow-up actions are in progress to harden migration safety and transaction lifecycle controls.

Envoy pod q7fjq tripped its default ext_authz circuit breaker (max_connections: 1,024) after absorbing 57% of traffic due to a rollout without readiness probes, then getting hit by a GKE node removal reconnection storm. The circuit breaker locked for 17 minutes, returning HTTP 500 for ~62% of auth requests on that pod. 9.4% of all user-facing requests failed (5,203 of 55,449). The system self-recovered when the auth connection latency tail cleared naturally. The other two envoy pods were completely healthy throughout. A skynet-frontend restart at 17:41:15 was coincidental (old pods still alive when CB cleared at 17:42:00).

Palmyra vision is having issues when serviced through public api requests.

Outage on Baseten embedding models

Writer experienced a service disruption on from 2:30 PM–2:40 PM PT (22:30–22:40 UTC), during which users received 403 errors. This was caused by planned maintenance that temporarily affected inbound traffic. Service has been restored.