TRMNL Outage History

TRMNL is up right now

TRMNL had 12 outages in the last 2 years totaling 43h 4m of downtime — averaging 0.5 incidents per month.

There were 12 TRMNL outages since June 5, 2025 totaling 43h 4m of downtime. Each is summarised below — incident details, duration, and resolution information.

Source: https://trmnl.statuspage.io

Notice May 19, 2026

Dashboard and API interruption

Detected by Pingoru
May 19, 2026, 12:07 AM UTC
Resolved
May 19, 2026, 01:15 AM UTC
Duration
1h 8m
Affected: Display API
Timeline · 3 updates
  1. investigating May 19, 2026, 12:07 AM UTC

    The Dashboard and device API are currently not responding. The CPU usage on these two servers is higher than normal. We are investigating the issue.

  2. monitoring May 19, 2026, 12:21 AM UTC

    The Dashboard and API have been restored. We have disabled some features to lower server CPU usage. We are looking further into the initial cause of the outage.

  3. resolved May 19, 2026, 01:15 AM UTC

    This incident has been resolved.

Read the full incident report →

Minor May 5, 2026

TRMNL dashboard downtime

Detected by Pingoru
May 05, 2026, 01:21 PM UTC
Resolved
May 05, 2026, 01:55 PM UTC
Duration
33m
Timeline · 3 updates
  1. investigating May 05, 2026, 01:21 PM UTC

    The TRMNL dashboard application is intermittently down. The API is still up, so device refresh isn't affected. We're looking into the issue.

  2. monitoring May 05, 2026, 01:41 PM UTC

    We discovered that an expensive analytics query was running, consuming a large amount of the web server CPU and memory. The query was killed and the server became responsive again. The situation: monitored.

  3. resolved May 05, 2026, 01:55 PM UTC

    This incident has been resolved.

Read the full incident report →

Major April 28, 2026

Redis cache timeouts

Detected by Pingoru
Apr 28, 2026, 06:47 PM UTC
Resolved
Apr 28, 2026, 07:34 PM UTC
Duration
47m
Affected: Display APIFirmware APIScreen Generation Workers
Timeline · 4 updates
  1. investigating Apr 28, 2026, 06:46 PM UTC

    Under investigation

  2. investigating Apr 28, 2026, 06:47 PM UTC

    We have started to investigate this issue.

  3. identified Apr 28, 2026, 06:52 PM UTC

    We've identified a commit that failed worker deployment, and a second that affected Redis. These have been reverted and we continue to find the root cause.

  4. resolved Apr 28, 2026, 07:34 PM UTC

    We've manually redeployed workers and systems continue to be stable.

Read the full incident report →

Minor April 13, 2026

DNS "Hostname not known" Errors

Detected by Pingoru
Apr 13, 2026, 09:17 PM UTC
Resolved
Apr 14, 2026, 12:49 AM UTC
Duration
3h 32m
Affected: Screen Generation Workers
Timeline · 3 updates
  1. investigating Apr 13, 2026, 09:17 PM UTC

    Over the weekend we noticed a growing surge of "Hostname not known" errors where usually there are nominal, causing plugins to not be able to reach endpoints to refresh data. This is a transient issue and is under investigation.

  2. investigating Apr 14, 2026, 12:18 AM UTC

    We continue to investigate the issue. It is not related to any recent code changes.

  3. resolved Apr 14, 2026, 12:49 AM UTC

    We've resolved an issue with DNS resolution errors on all worker machines.

Read the full incident report →

Notice March 30, 2026

Some plugins are not rendering

Detected by Pingoru
Mar 30, 2026, 05:16 PM UTC
Resolved
Mar 30, 2026, 08:50 PM UTC
Duration
3h 33m
Affected: Screen Generation Workers
Timeline · 4 updates
  1. investigating Mar 30, 2026, 05:16 PM UTC

    Some users are reporting that some of their plugins have not updated since about 14:00 UTC. We are investigating.

  2. investigating Mar 30, 2026, 06:09 PM UTC

    On our background worker nodes, we have found that Firefox times out during plugin rendering. We are continuing to investigate the cause.

  3. monitoring Mar 30, 2026, 08:05 PM UTC

    We have deployed a fix and worker health has improved. We are continuing to monitor the situation.

  4. resolved Mar 30, 2026, 08:50 PM UTC

    We identified that a change made to the Google Calendar plugin resulted in very long render times for that particular plugin. This delay quickly exhausted our pool of Firefox workers and prevented other plugins from getting a chance to generate. The offending change has been rolled back.

Read the full incident report →

Major March 26, 2026

DB Replica lag affecting site responsiveness

Detected by Pingoru
Mar 26, 2026, 02:22 PM UTC
Resolved
Mar 26, 2026, 07:58 PM UTC
Duration
5h 36m
Affected: Display APIFirmware APIScreen Generation Workers
Timeline · 5 updates
  1. investigating Mar 26, 2026, 02:58 PM UTC

    We are currently investigating the issue as the main database has 32GB of replica lag.

  2. monitoring Mar 26, 2026, 03:33 PM UTC

    We've identified the issue and restored site functionality. We are still monitoring the system as we identify the underlying cause. Screen generation will still see lag due to the backlog and one worker being taken offline.

  3. monitoring Mar 26, 2026, 05:45 PM UTC

    Screen generation workers continue to work through the backlog. This can result in: - Existing plugins not showing up-to-date information (old screeens) - New devices showing "You're all caught up!" due to no screens yet generated

  4. monitoring Mar 26, 2026, 06:22 PM UTC

    All workers are caught up, but one worker is currently still offline. While this is technically degraded, the existing workers can handle the ongoing load.

  5. resolved Mar 26, 2026, 07:58 PM UTC

    All screen generation workers and APIs have been restored. Internal infrastructure analysis and adjustments will continue with no impact on customers or operations.

Read the full incident report →

Critical January 28, 2026

API non-responsiveness

Detected by Pingoru
Jan 28, 2026, 06:19 PM UTC
Resolved
Jan 28, 2026, 07:21 PM UTC
Duration
1h 1m
Affected: Display API
Timeline · 3 updates
  1. investigating Jan 28, 2026, 06:19 PM UTC

    Today at 12:46pm ET we deployed a domain switchover and our services are catching up.

  2. identified Jan 28, 2026, 06:48 PM UTC

    CPU usage on our API server began spiking at 9a ET for approximately 4 hours and our DNS changes may have exacerbated the latency. We've disabled some logging and are seeing a return to normal levels.

  3. resolved Jan 28, 2026, 07:21 PM UTC

    An infrastructure change to our underlying cache system was made around 9 AM Eastern time, which caused one of our cache entries to start growing without bounds. Eventually this entry got so big it ate up all the CPU time just serializing this value and writing it back out to the cache. The cache for this value was disabled and performance has returned to normal.

Read the full incident report →

Minor November 23, 2025

Database overloaded

Detected by Pingoru
Nov 23, 2025, 01:07 AM UTC
Resolved
Nov 23, 2025, 04:05 PM UTC
Duration
14h 57m
Affected: Display API
Timeline · 2 updates
  1. investigating Nov 23, 2025, 01:07 AM UTC

    We’re investigating a database-related issue that prevents our API from responding to some device requests for new content. We have temporarily reduced webhook rate limits to 5 per hour, down from 12 and 30 per hour (TRMNL+ subscribers).

  2. resolved Nov 23, 2025, 04:05 PM UTC

    A long-running database process (WalSender - Write Ahead Log Sender) was the root cause of extra memory usage and has since finished. We also up-sized the database permanently and removed temporary webhook rate limit reductions.

Read the full incident report →

Notice October 23, 2025

Failed database migration

Detected by Pingoru
Oct 23, 2025, 09:22 PM UTC
Resolved
Oct 23, 2025, 05:30 AM UTC
Duration
—
Timeline · 1 update
  1. resolved Oct 23, 2025, 09:22 PM UTC

    During non-peak hours we deployed a migration that drops a legacy table, "Playlist Groups." This was responsible for scheduling different plugins through the day, but has since been replaced by the Scheduler, which was deployed to all accounts on September 8, 2025. Database CPU quickly climbed to 100%, preventing us from connecting + debugging. Engineers upsized the database, connected, and discovered the table being dropped (+ foreign key being migrated) had legacy records with a composite index. This composite index was used in the service invoked by GET requests to /api/display, which caused some devices to render a "Technical Difficulties" image. After fixing + adding back the composite index, and modifying the Playlist query that is triggered by GET requests to /api/display, service returned to normal.

Read the full incident report →

Major July 10, 2025

Browser pool timeout

Detected by Pingoru
Jul 10, 2025, 04:45 AM UTC
Resolved
Jul 10, 2025, 02:30 PM UTC
Duration
9h 44m
Affected: Screen Generation Workers
Timeline · 3 updates
  1. investigating Jul 10, 2025, 02:36 AM UTC

    Google Calendar, Private Plugins, and Recipes leverage our Firefox browser pool, which is timing out during screen generation. Investigating.

  2. monitoring Jul 10, 2025, 11:44 AM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved Jul 10, 2025, 06:49 PM UTC

    Incident resolved. On July 9 at 18:13 EDT we deployed major changes to the Framework design system. This included an asset deduplication strategy that created a bottleneck in our image generation pipeline for calendar plugins and private plugins. Within 30 minutes the issue was identified, and within 3 hours patched. Additional edge cases were discovered on July 10 between 06:00-10:10 EDT and also patched.

Read the full incident report →

Critical June 6, 2025

Server timeouts (again) leading to device boot looping

Detected by Pingoru
Jun 06, 2025, 01:17 AM UTC
Resolved
Jun 06, 2025, 02:50 AM UTC
Duration
1h 32m
Affected: Display API
Timeline · 3 updates
  1. investigating Jun 06, 2025, 01:17 AM UTC

    Investigating. Feel free to turn your device off to preserve power, as it may drain your battery while retrying for new content.

  2. monitoring Jun 06, 2025, 02:31 AM UTC

    A fix has been implemented and we are monitoring the results.

  3. resolved Jun 06, 2025, 02:50 AM UTC

    Brief post mortem here for wider distribution: https://www.reddit.com/r/trmnl/comments/1l4hs9z/trmnls_first_outage_now_resolved/

Read the full incident report →

Major June 5, 2025

Server timeouts leading to device boot looping

Detected by Pingoru
Jun 05, 2025, 11:58 PM UTC
Resolved
Jun 06, 2025, 12:34 AM UTC
Duration
36m
Affected: Display API
Timeline · 4 updates
  1. investigating Jun 05, 2025, 11:58 PM UTC

    Investigating, will share more details soon.

  2. investigating Jun 05, 2025, 11:58 PM UTC

    We are continuing to investigate this issue.

  3. investigating Jun 06, 2025, 12:34 AM UTC

    Still figuring out the cause, but increased latency on our API servers led to repeated timeouts. If your device is not already showing content again, turn it off for 10 seconds to clear the board's buffer and then back on.

  4. resolved Jun 06, 2025, 12:34 AM UTC

    This incident has been resolved.

Read the full incident report →