LLMOPS North Europe Data Ingestion had issues Between 12:00 AM and 12:45 AM UTC on April 4, 2026

Update timeline

1. resolved Apr 04, 2026, 01:26 AM UTC

   Between 12:00 AM and 12:45 AM UTC on April 4, 2026, a database outage in north Europe region caused an interruption to trace and audit ingestion. Data ingestion might have impacted for services using LLMOps: Telemetry trace ingestion might have failed LLM inputs and outputs were not recorded in audit logs during the outage. Currently service has been restored and healthy. No Ingestion issues observed.

2. postmortem May 06, 2026, 07:08 PM UTC

   ### **Customer Impact** Between April 4, 2026 at 1:00 am UTC and April 4, 2026 at 1:45 am UTC, a subset of customers in the Europe region experienced transient failures and latency in trace and audit data ingestion for LLM products. During this 45-minute window, affected customers were unable to save LLM call inputs and outputs. The impact was limited to customers in the Europe region relying on trace and audit logging for large language model operations. Other regions and services were not affected. ### **Root Cause** A scheduled maintenance job caused an increase in DB workers percentage. Due to this sudden increase in DB workers, the database became unreachable for ingesting spans and audits. ### **Detection** Automated monitoring detected the issue at 12:56 am UTC on April 4, 2026—approximately four minutes before the start of the customer impact window. A critical alert fired when database worker utilization exceeded its threshold, and our engineering team acknowledged the alert within one minute. A coordinated response effort was immediately established. ### **Response** Upon receiving the alert, our engineering team began investigating within one minute. The team quickly determined that the trace and audit ingestion failures coincided with a recent scheduled maintenance activity. We could also see an increase in database worker utilization. The maintenance job was terminated and the database was bumped to 8 vcores, hence allowing the DB workers to restore and the ingestion to be restored ### **Follow-up** We sincerely apologize for the disruption during this window. To prevent recurrence, we are implementing the following improvements: - * **Stricter maintenance procedures:** We are updating our maintenance procedures to run in low-traffic periods and with increased resources.