TRMNL had 10 outages in the last 2 years totaling 31h 19m of downtime — averaging 0.4 incidents per month.
There were 10 TRMNL outages since October 23, 2025 totaling 31h 19m of downtime. Each is summarised below — incident details, duration, and resolution information.
Currently investigating outage to trmnl.com
We are continuing to investigate this issue.
Our Digital Ocean droplet required manual intervention. PUMA crashes. Will investigate how it got into this state.
The Dashboard and device API are currently not responding. The CPU usage on these two servers is higher than normal. We are investigating the issue.
The Dashboard and API have been restored. We have disabled some features to lower server CPU usage. We are looking further into the initial cause of the outage.
This incident has been resolved.
The TRMNL dashboard application is intermittently down. The API is still up, so device refresh isn't affected. We're looking into the issue.
We discovered that an expensive analytics query was running, consuming a large amount of the web server CPU and memory. The query was killed and the server became responsive again. The situation: monitored.
This incident has been resolved.
Under investigation
We have started to investigate this issue.
We've identified a commit that failed worker deployment, and a second that affected Redis. These have been reverted and we continue to find the root cause.
We've manually redeployed workers and systems continue to be stable.
Over the weekend we noticed a growing surge of "Hostname not known" errors where usually there are nominal, causing plugins to not be able to reach endpoints to refresh data. This is a transient issue and is under investigation.
We continue to investigate the issue. It is not related to any recent code changes.
We've resolved an issue with DNS resolution errors on all worker machines.
Some users are reporting that some of their plugins have not updated since about 14:00 UTC. We are investigating.
On our background worker nodes, we have found that Firefox times out during plugin rendering. We are continuing to investigate the cause.
We have deployed a fix and worker health has improved. We are continuing to monitor the situation.
We identified that a change made to the Google Calendar plugin resulted in very long render times for that particular plugin. This delay quickly exhausted our pool of Firefox workers and prevented other plugins from getting a chance to generate. The offending change has been rolled back.
We are currently investigating the issue as the main database has 32GB of replica lag.
We've identified the issue and restored site functionality. We are still monitoring the system as we identify the underlying cause. Screen generation will still see lag due to the backlog and one worker being taken offline.
Screen generation workers continue to work through the backlog. This can result in: - Existing plugins not showing up-to-date information (old screeens) - New devices showing "You're all caught up!" due to no screens yet generated
All workers are caught up, but one worker is currently still offline. While this is technically degraded, the existing workers can handle the ongoing load.
All screen generation workers and APIs have been restored. Internal infrastructure analysis and adjustments will continue with no impact on customers or operations.
Today at 12:46pm ET we deployed a domain switchover and our services are catching up.
CPU usage on our API server began spiking at 9a ET for approximately 4 hours and our DNS changes may have exacerbated the latency. We've disabled some logging and are seeing a return to normal levels.
An infrastructure change to our underlying cache system was made around 9 AM Eastern time, which caused one of our cache entries to start growing without bounds. Eventually this entry got so big it ate up all the CPU time just serializing this value and writing it back out to the cache. The cache for this value was disabled and performance has returned to normal.
We’re investigating a database-related issue that prevents our API from responding to some device requests for new content. We have temporarily reduced webhook rate limits to 5 per hour, down from 12 and 30 per hour (TRMNL+ subscribers).
A long-running database process (WalSender - Write Ahead Log Sender) was the root cause of extra memory usage and has since finished. We also up-sized the database permanently and removed temporary webhook rate limit reductions.
During non-peak hours we deployed a migration that drops a legacy table, "Playlist Groups." This was responsible for scheduling different plugins through the day, but has since been replaced by the Scheduler, which was deployed to all accounts on September 8, 2025. Database CPU quickly climbed to 100%, preventing us from connecting + debugging. Engineers upsized the database, connected, and discovered the table being dropped (+ foreign key being migrated) had legacy records with a composite index. This composite index was used in the service invoked by GET requests to /api/display, which caused some devices to render a "Technical Difficulties" image. After fixing + adding back the composite index, and modifying the Playlist query that is triggered by GET requests to /api/display, service returned to normal.