Is Trigger.dev down?

There has been a security incident at Trigger.dev. We've blocked the exploit used in this attack but our investigation has identified exposure across a subset of organizations. The attacker gained unauthorized access to environment variables for these organizations. We have emailed all impacted account owners with steps to mitigate. We will follow up with more details.

We’ve published our report on the November 1, 2026 security incident: https://trigger.dev/blog/security-incident-report-nov-2026

Between 12:21 and 15:09 UTC, deploying with the Vercel integration overwrote some environment variables with empty values in Trigger.dev. Projects with affected variables may see failing deploys or runs that error because of missing credentials. You are only affected if all of these are true: - Your project uses the Vercel integration with environment variable syncing enabled - The project has environment variables marked as Sensitive in Vercel - That project deployed between 12:21 and 15:09 UTC today If any of these don't apply, your project was not affected and you don't need to do anything. Only the environments that deployed in that window are affected. The cause has been fixed and no new variables are being cleared. Variables that were already cleared are still empty: redeploying will not restore them. We are working on restoring them, and we are contacting affected customers directly with the list of impacted projects and variables. If you can't wait, re-entering the values in the Trigger.dev dashboard under Environment Variables fixes the affected project right away.

We stopped the issue at 15:10 UTC and no environment variables have been cleared since. A permanent fix is in progress. Variables that were cleared between 12:21 and 15:10 UTC are still empty and will not come back on redeploy. We have contacted every affected customer directly. Re-entering them in the Trigger.dev dashboard under Environment Variables fixes the affected project right away. We're still working with affected customers to restore their values, so if you need help or have questions, reach out to us at support or reply to our email.

We're investigating run logs being dropped (from your run when you do console.log/logger.log). This doesn't impact run operation but does impact observability. This isn't due to a code or configuration change. More information soon.

Logs are being fully ingested again. The issue was with the internal workers in our shared OpenTelemetry service. They became unhealthy but weren't rolled. We're looking into why and how to prevent this from happening again.

The insert delay for the runs list and logs is longer than usual. This impacts the dashboard and APIs associated with listing runs and viewing run logs. This is being caused by higher load and slow auto-scaling in our ClickHouse cluster.

We have identified the root cause of the issue and are working towards mitigating the issue. Currently the run list dashboard and API are fine, but we aren't able to insert new run logs and trace data.

Run logs/traces are inserting and the ingestion delays for are coming down. We have fixed the underlying issue. We will resolve this issue with more details when the ingestion delays are back to normal.

The ingestion delay is down to normal levels. A post-mortem will follow.

Postmortem: Missing task logs and traces Date: 2026-09-03 Duration: ~67 minutes Impact: New logs, spans and trace data for some runs were not stored and cannot be recovered Summary Between approximately 14:35 and 15:43 UTC on 2026-09-03, batches of new task events (the logs, spans and span events that make up a run's trace) were rejected by ClickHouse, the database behind the trace view, logs view and event search. Each batch was retried three times over about 1.5 seconds and then dropped. The events in those batches are permanently lost. Run execution, run status and the runs list were unaffected throughout, as were logs and traces that had already been stored. We are sorry for the missing data and for making affected runs harder to understand. What customers experienced Runs that were emitting logs or spans during the window may have a trace that is empty or missing spans, depending on which of their batches happened to land. The dashboard did not return errors: reads succeeded with fewer or zero rows, so this appeared as missing content rather than a failure. In a sample of dashboard trace queries during the window, about 41% returned no rows, versus about 2% normally. Separately, the OpenTelemetry ingestion endpoint returned some HTTP 500s and became slow, so tasks emitting a lot of telemetry may have logged export warnings. Runs outside the window were not affected. Root cause A release included two ClickHouse schema migrations run back to back on the task events table. The first dropped some unused indexes, which scheduled a background mutation over historical data. The second added a new input-only column. ClickHouse applies alters to a table in order, so the metadata change for the new column waited behind the index-drop mutation on the writer replicas. Our migration runner reported both as successful in under two seconds, because an ALTER returns once the replica it is connected to has accepted it, and our deploy pipeline treats that exit code as "the schema is ready". It does not check that the column exists on every writer replica or attempt a trial insert. The same release changed the task-event writer to include the new column in every insert, with no flag, fallback or separate schema-first release. Once the new code was running against replicas that did not yet serve the column, every task-event insert failed. The flush scheduler retries a failed batch three times and then drops it with no durable replay, which turned a temporary schema mismatch into permanent loss. We cancelled the background mutation and rolled the application back, but the failures stopped when ClickHouse Cloud autoscaled our writer replicas under memory pressure from the mutation, and the new replicas came up with the column visible. The rollback would have stopped the loss roughly twenty minutes after it was merged had that not happened. What we've done Bounded the queues on the OpenTelemetry collectors that receive telemetry from task runners, in all regions, so a stalled database no longer causes them to run out of memory and return errors. Re-rolled the release the following morning with the column visible on every replica, which went out cleanly. Fixed a collector alert for dropped telemetry that had been configured in a way that meant it could never fire. Ongoing Alert ownership and severity: routing alerts to the owner of the in-flight deploy, and making sustained production insert failures page. A migration gate that checks capability rather than exit code: wait for outstanding mutations on the affected table to finish, confirm required state, and fail-closed. A durable buffer for task-event batches that exhaust their retries, so a ClickHouse incompatibility or writer outage becomes a delay we can replay rather than a loss, with metrics that count what was actually dropped. Ingestion lag monitoring: add alerts on how long it has been since a task event last landed in the database, and on how long merges and schema operations have been outstanding, so stalled or silently failing ingestion pages us regardless of the cause. We treated "the migration ran" as "the database is ready". The changes above are about making the release process prove that before a change goes out. We apologise for the missing logs and traces.

# Postmortem: Missing task logs and traces Date: 2026-09-03 Duration: ~67 minutes Impact: New logs, spans and trace data for some runs were not stored and cannot be recovered ## Summary Between approximately 14:35 and 15:43 UTC on 2026-09-03, batches of new task events (the logs, spans and span events that make up a run's trace) were rejected by ClickHouse, the database behind the trace view, logs view and event search. Each batch was retried three times over about 1.5 seconds and then dropped. The events in those batches are permanently lost. Run execution, run status and the runs list were unaffected throughout, as were logs and traces that had already been stored. We are sorry for the missing data and for making affected runs harder to understand. ## What customers experienced Runs that were emitting logs or spans during the window may have a trace that is empty or missing spans, depending on which of their batches happened to land. The dashboard did not return errors: reads succeeded with fewer or zero rows, so this appeared as missing content rather than a failure. In a sample of dashboard trace queries during the window, about 41% returned no rows, versus about 2% normally. Separately, the OpenTelemetry ingestion endpoint returned some HTTP 500s and became slow, so tasks emitting a lot of telemetry may have logged export warnings. Runs outside the window were not affected. ## Root cause A release included two ClickHouse schema migrations run back to back on the task events table. The first dropped some unused indexes, which scheduled a background mutation over historical data. The second added a new input-only column. ClickHouse applies alters to a table in order, so the metadata change for the new column waited behind the index-drop mutation on the writer replicas. Our migration runner reported both as successful in under two seconds, because an ALTER returns once the replica it is connected to has accepted it, and our deploy pipeline treats that exit code as "the schema is ready". It does not check that the column exists on every writer replica or attempt a trial insert. The same release changed the task-event writer to include the new column in every insert, with no flag, fallback or separate schema-first release. Once the new code was running against replicas that did not yet serve the column, every task-event insert failed. The flush scheduler retries a failed batch three times and then drops it with no durable replay, which turned a temporary schema mismatch into permanent loss. We cancelled the background mutation and rolled the application back, but the failures stopped when ClickHouse Cloud autoscaled our writer replicas under memory pressure from the mutation, and the new replicas came up with the column visible. The rollback would have stopped the loss roughly twenty minutes after it was merged had that not happened. ## What we've done - Bounded the queues on the OpenTelemetry collectors that receive telemetry from task runners, in all regions, so a stalled database no longer causes them to run out of memory and return errors. - Re-rolled the release the following morning with the column visible on every replica, which went out cleanly. - Fixed a collector alert for dropped telemetry that had been configured in a way that meant it could never fire. ## Ongoing - Alert ownership and severity: routing alerts to the owner of the in-flight deploy, and making sustained production insert failures page. - A migration gate that checks capability rather than exit code: wait for outstanding mutations on the affected table to finish, confirm required state, and fail-closed. - A durable buffer for task-event batches that exhaust their retries, so a ClickHouse incompatibility or writer outage becomes a delay we can replay rather than a loss, with metrics that count what was actually dropped. - Ingestion lag monitoring: add alerts on how long it has been since a task event last landed in the database, and on how long merges and schema operations have been outstanding, so stalled or silently failing ingestion pages us regardless of the cause. We treated "the migration ran" as "the database is ready". The changes above are about making the release process prove that before a change goes out. We apologise for the missing logs and traces.

The runs list dashboard and API is degraded leading to increased latency and/or timeouts.We're investigating and attempting to bring this back online.

We have identified the root cause of the issue and are working on a fix. The runs list API and dashboard are currently healthy but the underlying root cause is still unpatched.

We are still working on the root cause patch and it will rollout as soon as its tested and ready. We're resolving this status update for now as the run list API and dashboard have been running in good health for the last 30 minutes