TRMNL Outage History
TRMNL degraded View live status →Pingoru has recorded 13 incidents for TRMNL: 10 in the last 12 months and 1 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 2 | 4 of 720 (1%) |
| May 2026 | 2 | 2 of 744 (0%) |
| June 2026 | 0 | 0 of 720 (0%) |
| July 2026 | 0 | 0 of 744 (0%) |
| August 2026 | 1 | 0 of 744 (0%) |
| September 2026 | 0 | 0 of 720 (0%) |
| October 2026 | 0 | 0 of 520 (0%) |
Incidents, newest first
Major August 19, 2026 Bad gateway 8m
Started Aug 19, 2026, 10:49 PM UTC· resolved Aug 19, 2026, 10:58 PM UTC
- investigating Aug 19, 2026, 10:49 PM UTC
Currently investigating outage to trmnl.com
- investigating Aug 19, 2026, 10:49 PM UTC
We are continuing to investigate this issue.
- resolved Aug 19, 2026, 10:58 PM UTC
Our Digital Ocean droplet required manual intervention. PUMA crashes. Will investigate how it got into this state.
Notice May 19, 2026 Dashboard and API interruption 1h 8m Display API
Started May 19, 2026, 12:07 AM UTC· resolved May 19, 2026, 01:15 AM UTC
- investigating May 19, 2026, 12:07 AM UTC
The Dashboard and device API are currently not responding. The CPU usage on these two servers is higher than normal. We are investigating the issue.
- monitoring May 19, 2026, 12:21 AM UTC
The Dashboard and API have been restored. We have disabled some features to lower server CPU usage. We are looking further into the initial cause of the outage.
- resolved May 19, 2026, 01:15 AM UTC
This incident has been resolved.
Minor May 5, 2026 TRMNL dashboard downtime 33m
Started May 05, 2026, 01:21 PM UTC· resolved May 05, 2026, 01:55 PM UTC
- investigating May 05, 2026, 01:21 PM UTC
The TRMNL dashboard application is intermittently down. The API is still up, so device refresh isn't affected. We're looking into the issue.
- monitoring May 05, 2026, 01:41 PM UTC
We discovered that an expensive analytics query was running, consuming a large amount of the web server CPU and memory. The query was killed and the server became responsive again. The situation: monitored.
- resolved May 05, 2026, 01:55 PM UTC
This incident has been resolved.
Major April 28, 2026 Redis cache timeouts 48m Display API, Firmware API, Screen Generation Workers
Started Apr 28, 2026, 06:46 PM UTC· resolved Apr 28, 2026, 07:34 PM UTC
- investigating Apr 28, 2026, 06:46 PM UTC
Under investigation
- investigating Apr 28, 2026, 06:47 PM UTC
We have started to investigate this issue.
- identified Apr 28, 2026, 06:52 PM UTC
We've identified a commit that failed worker deployment, and a second that affected Redis. These have been reverted and we continue to find the root cause.
- resolved Apr 28, 2026, 07:34 PM UTC
We've manually redeployed workers and systems continue to be stable.
Minor April 13, 2026 DNS "Hostname not known" Errors 3h 32m Screen Generation Workers
Started Apr 13, 2026, 09:17 PM UTC· resolved Apr 14, 2026, 12:49 AM UTC
- investigating Apr 13, 2026, 09:17 PM UTC
Over the weekend we noticed a growing surge of "Hostname not known" errors where usually there are nominal, causing plugins to not be able to reach endpoints to refresh data. This is a transient issue and is under investigation.
- investigating Apr 14, 2026, 12:18 AM UTC
We continue to investigate the issue. It is not related to any recent code changes.
- resolved Apr 14, 2026, 12:49 AM UTC
We've resolved an issue with DNS resolution errors on all worker machines.
Notice March 30, 2026 Some plugins are not rendering 3h 33m Screen Generation Workers
Started Mar 30, 2026, 05:16 PM UTC· resolved Mar 30, 2026, 08:50 PM UTC
- investigating Mar 30, 2026, 05:16 PM UTC
Some users are reporting that some of their plugins have not updated since about 14:00 UTC. We are investigating.
- investigating Mar 30, 2026, 06:09 PM UTC
On our background worker nodes, we have found that Firefox times out during plugin rendering. We are continuing to investigate the cause.
- monitoring Mar 30, 2026, 08:05 PM UTC
We have deployed a fix and worker health has improved. We are continuing to monitor the situation.
- resolved Mar 30, 2026, 08:50 PM UTC
We identified that a change made to the Google Calendar plugin resulted in very long render times for that particular plugin. This delay quickly exhausted our pool of Firefox workers and prevented other plugins from getting a chance to generate. The offending change has been rolled back.
Major March 26, 2026 DB Replica lag affecting site responsiveness 5h 36m Display API, Firmware API, Screen Generation Workers
Started Mar 26, 2026, 02:22 PM UTC· resolved Mar 26, 2026, 07:58 PM UTC
- investigating Mar 26, 2026, 02:58 PM UTC
We are currently investigating the issue as the main database has 32GB of replica lag.
- monitoring Mar 26, 2026, 03:33 PM UTC
We've identified the issue and restored site functionality. We are still monitoring the system as we identify the underlying cause. Screen generation will still see lag due to the backlog and one worker being taken offline.
- monitoring Mar 26, 2026, 05:45 PM UTC
Screen generation workers continue to work through the backlog. This can result in: - Existing plugins not showing up-to-date information (old screeens) - New devices showing "You're all caught up!" due to no screens yet generated
- monitoring Mar 26, 2026, 06:22 PM UTC
All workers are caught up, but one worker is currently still offline. While this is technically degraded, the existing workers can handle the ongoing load.
- resolved Mar 26, 2026, 07:58 PM UTC
All screen generation workers and APIs have been restored. Internal infrastructure analysis and adjustments will continue with no impact on customers or operations.
Critical January 28, 2026 API non-responsiveness 1h 1m Display API
Started Jan 28, 2026, 06:19 PM UTC· resolved Jan 28, 2026, 07:21 PM UTC
- investigating Jan 28, 2026, 06:19 PM UTC
Today at 12:46pm ET we deployed a domain switchover and our services are catching up.
- identified Jan 28, 2026, 06:48 PM UTC
CPU usage on our API server began spiking at 9a ET for approximately 4 hours and our DNS changes may have exacerbated the latency. We've disabled some logging and are seeing a return to normal levels.
- resolved Jan 28, 2026, 07:21 PM UTC
An infrastructure change to our underlying cache system was made around 9 AM Eastern time, which caused one of our cache entries to start growing without bounds. Eventually this entry got so big it ate up all the CPU time just serializing this value and writing it back out to the cache. The cache for this value was disabled and performance has returned to normal.
Minor November 23, 2025 Database overloaded 14h 57m Display API
Started Nov 23, 2025, 01:07 AM UTC· resolved Nov 23, 2025, 04:05 PM UTC
- investigating Nov 23, 2025, 01:07 AM UTC
We’re investigating a database-related issue that prevents our API from responding to some device requests for new content. We have temporarily reduced webhook rate limits to 5 per hour, down from 12 and 30 per hour (TRMNL+ subscribers).
- resolved Nov 23, 2025, 04:05 PM UTC
A long-running database process (WalSender - Write Ahead Log Sender) was the root cause of extra memory usage and has since finished. We also up-sized the database permanently and removed temporary webhook rate limit reductions.
Notice October 23, 2025 Failed database migration –
Started Oct 23, 2025, 05:30 AM UTC· resolved Oct 23, 2025, 05:30 AM UTC
- resolved Oct 23, 2025, 09:22 PM UTC
During non-peak hours we deployed a migration that drops a legacy table, "Playlist Groups." This was responsible for scheduling different plugins through the day, but has since been replaced by the Scheduler, which was deployed to all accounts on September 8, 2025. Database CPU quickly climbed to 100%, preventing us from connecting + debugging. Engineers upsized the database, connected, and discovered the table being dropped (+ foreign key being migrated) had legacy records with a composite index. This composite index was used in the service invoked by GET requests to /api/display, which caused some devices to render a "Technical Difficulties" image. After fixing + adding back the composite index, and modifying the Playlist query that is triggered by GET requests to /api/display, service returned to normal.
Major July 10, 2025 Browser pool timeout 11h 53m Screen Generation Workers
Started Jul 10, 2025, 02:36 AM UTC· resolved Jul 10, 2025, 02:30 PM UTC
- investigating Jul 10, 2025, 02:36 AM UTC
Google Calendar, Private Plugins, and Recipes leverage our Firefox browser pool, which is timing out during screen generation. Investigating.
- monitoring Jul 10, 2025, 11:44 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jul 10, 2025, 06:49 PM UTC
Incident resolved. On July 9 at 18:13 EDT we deployed major changes to the Framework design system. This included an asset deduplication strategy that created a bottleneck in our image generation pipeline for calendar plugins and private plugins. Within 30 minutes the issue was identified, and within 3 hours patched. Additional edge cases were discovered on July 10 between 06:00-10:10 EDT and also patched.
Critical June 6, 2025 Server timeouts (again) leading to device boot looping 1h 32m Display API
Started Jun 06, 2025, 01:17 AM UTC· resolved Jun 06, 2025, 02:50 AM UTC
- investigating Jun 06, 2025, 01:17 AM UTC
Investigating. Feel free to turn your device off to preserve power, as it may drain your battery while retrying for new content.
- monitoring Jun 06, 2025, 02:31 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Jun 06, 2025, 02:50 AM UTC
Brief post mortem here for wider distribution: https://www.reddit.com/r/trmnl/comments/1l4hs9z/trmnls_first_outage_now_resolved/
Major June 5, 2025 Server timeouts leading to device boot looping 36m Display API
Started Jun 05, 2025, 11:58 PM UTC· resolved Jun 06, 2025, 12:34 AM UTC
- investigating Jun 05, 2025, 11:58 PM UTC
Investigating, will share more details soon.
- investigating Jun 05, 2025, 11:58 PM UTC
We are continuing to investigate this issue.
- investigating Jun 06, 2025, 12:34 AM UTC
Still figuring out the cause, but increased latency on our API servers led to repeated timeouts. If your device is not already showing content again, turn it off for 10 seconds to clear the board's buffer and then back on.
- resolved Jun 06, 2025, 12:34 AM UTC
This incident has been resolved.