Trigger.dev Outage History

Started Nov 01, 2026, 10:50 PM UTC· resolved Nov 01, 2026, 10:50 PM UTC

1. resolved Nov 01, 2026, 10:50 PM UTC

   There has been a security incident at Trigger.dev. We've blocked the exploit used in this attack but our investigation has identified exposure across a subset of organizations. The attacker gained unauthorized access to environment variables for these organizations. We have emailed all impacted account owners with steps to mitigate. We will follow up with more details.

Read the full incident report →

Started Oct 06, 2026, 12:21 PM UTC· resolved Oct 06, 2026, 03:10 PM UTC

1. investigating Oct 06, 2026, 12:21 PM UTC

   Between 12:21 and 15:09 UTC, deploying with the Vercel integration overwrote some environment variables with empty values in Trigger.dev. Projects with affected variables may see failing deploys or runs that error because of missing credentials. You are only affected if all of these are true: - Your project uses the Vercel integration with environment variable syncing enabled - The project has environment variables marked as Sensitive in Vercel - That project deployed between 12:21 and 15:09 UTC today If any of these don't apply, your project was not affected and you don't need to do anything. Only the environments that deployed in that window are affected. The cause has been fixed and no new variables are being cleared. Variables that were already cleared are still empty: redeploying will not restore them. We are working on restoring them, and we are contacting affected customers directly with the list of impacted projects and variables. If you can't wait, re-entering the values in the Trigger.dev dashboard under Environment Variables fixes the affected project right away.

2. resolved Oct 06, 2026, 03:10 PM UTC

   We stopped the issue at 15:10 UTC and no environment variables have been cleared since. A permanent fix is in progress. Variables that were cleared between 12:21 and 15:10 UTC are still empty and will not come back on redeploy. We have contacted every affected customer directly. Re-entering them in the Trigger.dev dashboard under Environment Variables fixes the affected project right away. We're still working with affected customers to restore their values, so if you need help or have questions, reach out to us at support or reply to our email.

Read the full incident report →

Started Sep 28, 2026, 07:20 AM UTC· resolved Sep 28, 2026, 07:50 AM UTC

1. investigating Sep 28, 2026, 07:20 AM UTC

   We're investigating run logs being dropped (from your run when you do console.log/logger.log). This doesn't impact run operation but does impact observability. This isn't due to a code or configuration change. More information soon.

2. resolved Sep 28, 2026, 07:50 AM UTC

   Logs are being fully ingested again. The issue was with the internal workers in our shared OpenTelemetry service. They became unhealthy but weren't rolled. We're looking into why and how to prevent this from happening again.

Read the full incident report →

Started Sep 03, 2026, 02:56 PM UTC· resolved Sep 04, 2026, 12:55 PM UTC

1. investigating Sep 03, 2026, 02:56 PM UTC

   The insert delay for the runs list and logs is longer than usual. This impacts the dashboard and APIs associated with listing runs and viewing run logs. This is being caused by higher load and slow auto-scaling in our ClickHouse cluster.

2. investigating Sep 03, 2026, 03:28 PM UTC

   We have identified the root cause of the issue and are working towards mitigating the issue. Currently the run list dashboard and API are fine, but we aren't able to insert new run logs and trace data.

3. investigating Sep 03, 2026, 03:50 PM UTC

   Run logs/traces are inserting and the ingestion delays for are coming down. We have fixed the underlying issue. We will resolve this issue with more details when the ingestion delays are back to normal.

4. investigating Sep 03, 2026, 04:20 PM UTC

   The ingestion delay is down to normal levels. A post-mortem will follow.

5. resolved Sep 04, 2026, 12:55 PM UTC

   Postmortem: Missing task logs and traces Date: 2026-09-03 Duration: ~67 minutes Impact: New logs, spans and trace data for some runs were not stored and cannot be recovered Summary Between approximately 14:35 and 15:43 UTC on 2026-09-03, batches of new task events (the logs, spans and span events that make up a run's trace) were rejected by ClickHouse, the database behind the trace view, logs view and event search. Each batch was retried three times over about 1.5 seconds and then dropped. The events in those batches are permanently lost. Run execution, run status and the runs list were unaffected throughout, as were logs and traces that had already been stored. We are sorry for the missing data and for making affected runs harder to understand. What customers experienced Runs that were emitting logs or spans during the window may have a trace that is empty or missing spans, depending on which of their batches happened to land. The dashboard did not return errors: reads succeeded with fewer or zero rows, so this appeared as missing content rather than a failure. In a sample of dashboard trace queries during the window, about 41% returned no rows, versus about 2% normally. Separately, the OpenTelemetry ingestion endpoint returned some HTTP 500s and became slow, so tasks emitting a lot of telemetry may have logged export warnings. Runs outside the window were not affected. Root cause A release included two ClickHouse schema migrations run back to back on the task events table. The first dropped some unused indexes, which scheduled a background mutation over historical data. The second added a new input-only column. ClickHouse applies alters to a table in order, so the metadata change for the new column waited behind the index-drop mutation on the writer replicas. Our migration runner reported both as successful in under two seconds, because an ALTER returns once the replica it is connected to has accepted it, and our deploy pipeline treats that exit code as "the schema is ready". It does not check that the column exists on every writer replica or attempt a trial insert. The same release changed the task-event writer to include the new column in every insert, with no flag, fallback or separate schema-first release. Once the new code was running against replicas that did not yet serve the column, every task-event insert failed. The flush scheduler retries a failed batch three times and then drops it with no durable replay, which turned a temporary schema mismatch into permanent loss. We cancelled the background mutation and rolled the application back, but the failures stopped when ClickHouse Cloud autoscaled our writer replicas under memory pressure from the mutation, and the new replicas came up with the column visible. The rollback would have stopped the loss roughly twenty minutes after it was merged had that not happened. What we've done Bounded the queues on the OpenTelemetry collectors that receive telemetry from task runners, in all regions, so a stalled database no longer causes them to run out of memory and return errors. Re-rolled the release the following morning with the column visible on every replica, which went out cleanly. Fixed a collector alert for dropped telemetry that had been configured in a way that meant it could never fire. Ongoing Alert ownership and severity: routing alerts to the owner of the in-flight deploy, and making sustained production insert failures page. A migration gate that checks capability rather than exit code: wait for outstanding mutations on the affected table to finish, confirm required state, and fail-closed. A durable buffer for task-event batches that exhaust their retries, so a ClickHouse incompatibility or writer outage becomes a delay we can replay rather than a loss, with metrics that count what was actually dropped. Ingestion lag monitoring: add alerts on how long it has been since a task event last landed in the database, and on how long merges and schema operations have been outstanding, so stalled or silently failing ingestion pages us regardless of the cause. We treated "the migration ran" as "the database is ready". The changes above are about making the release process prove that before a change goes out. We apologise for the missing logs and traces.

6. resolved Sep 04, 2026, 12:55 PM UTC

   # Postmortem: Missing task logs and traces Date: 2026-09-03 Duration: ~67 minutes Impact: New logs, spans and trace data for some runs were not stored and cannot be recovered ## Summary Between approximately 14:35 and 15:43 UTC on 2026-09-03, batches of new task events (the logs, spans and span events that make up a run's trace) were rejected by ClickHouse, the database behind the trace view, logs view and event search. Each batch was retried three times over about 1.5 seconds and then dropped. The events in those batches are permanently lost. Run execution, run status and the runs list were unaffected throughout, as were logs and traces that had already been stored. We are sorry for the missing data and for making affected runs harder to understand. ## What customers experienced Runs that were emitting logs or spans during the window may have a trace that is empty or missing spans, depending on which of their batches happened to land. The dashboard did not return errors: reads succeeded with fewer or zero rows, so this appeared as missing content rather than a failure. In a sample of dashboard trace queries during the window, about 41% returned no rows, versus about 2% normally. Separately, the OpenTelemetry ingestion endpoint returned some HTTP 500s and became slow, so tasks emitting a lot of telemetry may have logged export warnings. Runs outside the window were not affected. ## Root cause A release included two ClickHouse schema migrations run back to back on the task events table. The first dropped some unused indexes, which scheduled a background mutation over historical data. The second added a new input-only column. ClickHouse applies alters to a table in order, so the metadata change for the new column waited behind the index-drop mutation on the writer replicas. Our migration runner reported both as successful in under two seconds, because an ALTER returns once the replica it is connected to has accepted it, and our deploy pipeline treats that exit code as "the schema is ready". It does not check that the column exists on every writer replica or attempt a trial insert. The same release changed the task-event writer to include the new column in every insert, with no flag, fallback or separate schema-first release. Once the new code was running against replicas that did not yet serve the column, every task-event insert failed. The flush scheduler retries a failed batch three times and then drops it with no durable replay, which turned a temporary schema mismatch into permanent loss. We cancelled the background mutation and rolled the application back, but the failures stopped when ClickHouse Cloud autoscaled our writer replicas under memory pressure from the mutation, and the new replicas came up with the column visible. The rollback would have stopped the loss roughly twenty minutes after it was merged had that not happened. ## What we've done - Bounded the queues on the OpenTelemetry collectors that receive telemetry from task runners, in all regions, so a stalled database no longer causes them to run out of memory and return errors. - Re-rolled the release the following morning with the column visible on every replica, which went out cleanly. - Fixed a collector alert for dropped telemetry that had been configured in a way that meant it could never fire. ## Ongoing - Alert ownership and severity: routing alerts to the owner of the in-flight deploy, and making sustained production insert failures page. - A migration gate that checks capability rather than exit code: wait for outstanding mutations on the affected table to finish, confirm required state, and fail-closed. - A durable buffer for task-event batches that exhaust their retries, so a ClickHouse incompatibility or writer outage becomes a delay we can replay rather than a loss, with metrics that count what was actually dropped. - Ingestion lag monitoring: add alerts on how long it has been since a task event last landed in the database, and on how long merges and schema operations have been outstanding, so stalled or silently failing ingestion pages us regardless of the cause. We treated "the migration ran" as "the database is ready". The changes above are about making the release process prove that before a change goes out. We apologise for the missing logs and traces.

Read the full incident report →

Started Aug 24, 2026, 02:25 PM UTC· resolved Aug 24, 2026, 03:34 PM UTC

1. investigating Aug 24, 2026, 02:25 PM UTC

   The runs list dashboard and API is degraded leading to increased latency and/or timeouts.We're investigating and attempting to bring this back online.

2. investigating Aug 24, 2026, 02:56 PM UTC

   We have identified the root cause of the issue and are working on a fix. The runs list API and dashboard are currently healthy but the underlying root cause is still unpatched.

3. resolved Aug 24, 2026, 03:34 PM UTC

   We are still working on the root cause patch and it will rollout as soon as its tested and ready. We're resolving this status update for now as the run list API and dashboard have been running in good health for the last 30 minutes

Read the full incident report →

Started Aug 13, 2026, 12:21 PM UTC· resolved Aug 13, 2026, 12:21 PM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Aug 05, 2026, 08:28 PM UTC· resolved Aug 05, 2026, 08:34 PM UTC

1. investigating Aug 05, 2026, 08:28 PM UTC

   Realtime streams are currently degraded, we're working with our partner to bring full performance back as quickly as possible.

2. resolved Aug 05, 2026, 08:34 PM UTC

   A small percentage of streams experienced issues but it's resolved now. The root cause was some server crashes for our provider which they're working on a permanent fix for now.

Read the full incident report →

Started Jul 31, 2026, 04:35 PM UTC· resolved Jul 31, 2026, 04:35 PM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Jul 16, 2026, 04:19 PM UTC· resolved Jul 16, 2026, 04:26 PM UTC

1. investigating Jul 16, 2026, 04:19 PM UTC

   The dashboard is currently unavailable for all users. A fix has been identified and is being applied. Task runs are not impacted by this issue.

2. resolved Jul 16, 2026, 04:26 PM UTC

   The dashboard is now available. If you are still experiencing issues please hard reload the dashboard application.

Read the full incident report →

Started Jul 16, 2026, 01:27 PM UTC· resolved Jul 16, 2026, 02:16 PM UTC

1. investigating Jul 16, 2026, 01:27 PM UTC

   Listing runs on the dashboard and via the API is currently degraded. A fix is being rolled out now. Run operations are unaffected.

2. investigating Jul 16, 2026, 01:49 PM UTC

   Runs list is recovering. Continuing to investigate and monitoring the situation. Run operations continue unimpacted.

3. resolved Jul 16, 2026, 02:16 PM UTC

   The issue is now resolved, the dashboard and API are operating normally.

Read the full incident report →

Started Jul 16, 2026, 09:38 AM UTC· resolved Jul 16, 2026, 09:38 AM UTC

1. resolved Jul 16, 2026, 09:38 AM UTC

   SSO login access has been restored.

Read the full incident report →

Started Jul 15, 2026, 11:57 AM UTC· resolved Jul 15, 2026, 11:57 AM UTC

1. resolved Jul 15, 2026, 11:57 AM UTC

   Root cause identified and a fix applied. We're provisioning additional ClickHouse resources to prevent this occurring again.

Read the full incident report →

Started Jul 01, 2026, 02:12 AM UTC· resolved Jul 01, 2026, 02:12 AM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Jul 01, 2026, 01:16 AM UTC· resolved Jul 01, 2026, 01:36 AM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Jun 30, 2026, 09:40 AM UTC· resolved Jul 01, 2026, 11:22 AM UTC

1. investigating Jun 30, 2026, 09:40 AM UTC

   Runs are executing normally. The replication of runs to ClickHouse which powers the Runs list are impacted. We're investigating.

2. investigating Jun 30, 2026, 10:11 AM UTC

   The runs.list() SDK function and Runs dashboard page are degraded. This is due to slow replication into our ClickHouse cluster. We are investigating.

3. investigating Jun 30, 2026, 10:19 AM UTC

   Spans and logs are also degraded. This is due to slow replication into our ClickHouse cluster. We are investigating.

4. investigating Jun 30, 2026, 11:06 AM UTC

   We believe slow ClickHouse replication is due to long running merges with deeply nested output objects. We have vertically scaled and are working with ClickHouse support on mitigations.

5. investigating Jun 30, 2026, 11:41 AM UTC

   The root cause has been identified and remediation work is taking place

6. investigating Jun 30, 2026, 12:36 PM UTC

   Runs are triggering and executing normally. The runs list (runs.list API + dashboard) is slow and may be delayed. Run logs and spans may be delayed. Cause identified, remediation in progress. Next update in 30 mins.

7. investigating Jun 30, 2026, 01:17 PM UTC

   Runs are triggering and executing normally. The runs list (runs.list API + dashboard) performance is now improving. Run logs and spans are also improving. We're actively monitoring the rollout of the remediation. Next update in 30 mins.

8. investigating Jun 30, 2026, 02:08 PM UTC

   We're continuing to monitor API response times, which are steadily improving. Our mitigation efforts remain ongoing.

9. investigating Jun 30, 2026, 03:43 PM UTC

   First, the important part: your runs are safe. Runs execute and are stored in Postgres as normal. This issue is isolated to ClickHouse, which we use as a read-optimised view store for querying runs. Triggering, execution, and storage have been unaffected throughout. This is a visibility issue, not a data loss one. The fix is downgrading ClickHouse to 25.12. That rollback runs at the ClickHouse Cloud infrastructure level, so we're coordinating with them to apply it. It's agreed and queued. What you're seeing While the view was degraded, the runs list, filtering, bulk actions, the query API, and all error monitoring and alerting were slow or unavailable, and dashboard charts, traces, logs, and span detail were blank or incomplete for affected runs. Run execution, billing, and your Postgres-stored run data were unaffected throughout. Dashboard and API queries against run data are slow or failing. There was a window this morning where these reads were down entirely. A small percentage of runs created since June 17 aren't yet in the correct state in the view. The runs ran and are stored. The view of them is catching up. Run logs and spans have recovered. Run views are still catching up. What's causing it ClickHouse moved from 25.12 to 26.2 on June 17. The new version added a cap on JSON column complexity (input_format_binary_max_type_complexity, defaulting to 1000), measured across the combined outputs in each batched insert. Once a batch went over, those inserts stopped landing in the view. Then this morning, June 30, memory and CPU on the instance climbed sharply as the backlog of unmerged data caught up with it, driving reads down until it ran out of memory and caused today's outage. The 25.12 rollback resolves the root cause. What we've done so far Cleared the data that couldn't merge so healthy merges can continue. Run logs and spans recovered quickly; run views are slower and still catching up. Scaled the cluster horizontally to add headroom. Lined up the 25.12 rollback with ClickHouse Cloud.

10. investigating Jun 30, 2026, 03:54 PM UTC

    This incident has been resolved. We are continuing to monitor and a post-mortem will be posted soon.

11. resolved Jul 01, 2026, 11:22 AM UTC

    Update: the full postmortem is now published: https://trigger.dev/blog/incident-report-jun-30-2026

Read the full incident report →

Started Jun 24, 2026, 09:14 AM UTC· resolved Jun 24, 2026, 09:14 AM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Jun 24, 2026, 07:35 AM UTC· resolved Jun 24, 2026, 07:35 AM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Jun 24, 2026, 06:22 AM UTC· resolved Jun 24, 2026, 06:22 AM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Jun 24, 2026, 05:24 AM UTC· resolved Jun 24, 2026, 05:24 AM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Jun 24, 2026, 04:39 AM UTC· resolved Jun 24, 2026, 04:39 AM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Jun 24, 2026, 02:50 AM UTC· resolved Jun 24, 2026, 02:50 AM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Jun 24, 2026, 02:00 AM UTC· resolved Jun 24, 2026, 02:00 AM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Jun 24, 2026, 01:30 AM UTC· resolved Jun 24, 2026, 01:30 AM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Jun 24, 2026, 12:14 AM UTC· resolved Jun 24, 2026, 12:14 AM UTC

No vendor updates were posted for this incident.

Read the full incident report →

Started Jun 23, 2026, 07:39 PM UTC· resolved Jun 23, 2026, 08:07 PM UTC

No vendor updates were posted for this incident.

Read the full incident report →