Treefort Outage History
Treefort is up right nowPingoru has recorded 35 incidents for Treefort: 24 in the last 12 months and 7 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 3 | 2 of 720 (0%) |
| May 2026 | 1 | 1 of 744 (0%) |
| June 2026 | 2 | 4 of 720 (1%) |
| July 2026 | 0 | 0 of 744 (0%) |
| August 2026 | 1 | 13 of 744 (2%) |
| September 2026 | 2 | 2 of 720 (0%) |
| October 2026 | 4 | 12 of 744 (2%) |
| November 2026 | 0 | 0 of 229 (0%) |
Incidents, newest first
Minor October 27, 2026 Auth outage 1h API, Apps
Started Oct 27, 2026, 06:00 AM UTC· resolved Oct 27, 2026, 07:00 AM UTC
- resolved Oct 27, 2026, 06:00 AM UTC
Minor October 16, 2026 Web apps briefly unavailable 8h 37m Apps
Started Oct 16, 2026, 07:19 PM UTC· resolved Oct 17, 2026, 03:56 AM UTC
- identified Oct 16, 2026, 07:19 PM UTC
Our upstream hosting provider is restarting host machines across the fleet over the next several hours. This may result in brief periods of downtime for web apps as traffic is routed to other machines.
- identified Oct 17, 2026, 12:42 AM UTC
Our upstream hosting provider is continuing to apply a security patch for a critical vulnerability to their host machines, which requires restarting the machine. We are continuing to see brief outages among apps and the admin dashboard as infrastructure is restarted, but downtime affecting users has remained very brief thanks to cross-region redundancy. We expect full stability to return within a few hours as our hosting provider finishes applying updates.
- resolved Oct 17, 2026, 03:56 AM UTC
Our hosting provider reports that all machines have been restarted, and we expect no further disruptions at this time.
Minor October 15, 2026 eBook processing errors 1h 46m Admin Dashboard
Started Oct 15, 2026, 03:48 PM UTC· resolved Oct 15, 2026, 05:34 PM UTC
- investigating Oct 15, 2026, 03:48 PM UTC
We're seeing some failures in eBook processing jobs. We're currently investigating the problem.
- resolved Oct 15, 2026, 05:34 PM UTC
We've identified the problem and put a mitigation in place while we continue to work on a permanent fix. All eBooks are now processing normally.
Minor October 13, 2026 Partial auth outage 7m Apps
Started Oct 13, 2026, 10:37 PM UTC· resolved Oct 13, 2026, 10:44 PM UTC
- investigating Oct 13, 2026, 10:37 PM UTC
- monitoring Oct 13, 2026, 10:42 PM UTC
- resolved Oct 13, 2026, 10:44 PM UTC
Minor September 28, 2026 Admin dashboard pages intermittently not loading 1h 10m Admin Dashboard
Started Sep 28, 2026, 05:15 PM UTC· resolved Sep 28, 2026, 06:26 PM UTC
- investigating Sep 28, 2026, 05:15 PM UTC
Some pages on the admin dashboard are intermittently not loading. Refreshing the page sometimes works to load the affected page. We're investigating the problem.
- monitoring Sep 28, 2026, 05:42 PM UTC
We've identified a faulty API machine instance which was causing some network requests to hang indefinitely. We've replaced the instance and are continuing to monitor performance.
- resolved Sep 28, 2026, 06:26 PM UTC
Minor September 1, 2026 Authentication system errors 1h 9m Apps
Started Sep 01, 2026, 12:02 PM UTC· resolved Sep 01, 2026, 01:12 PM UTC
- investigating Sep 01, 2026, 12:02 PM UTC
Our authentication system is intermittently returning errors when users try to sign in. We're investigating the problem.
- monitoring Sep 01, 2026, 12:28 PM UTC
One of the authentication app servers encountered a fault. We've replaced the virtual machine with a new one and are seeing errors subside.
- resolved Sep 01, 2026, 01:12 PM UTC
Minor August 25, 2026 File upload outage 13h
Started Aug 25, 2026, 03:00 AM UTC· resolved Aug 25, 2026, 04:00 PM UTC
- resolved Aug 25, 2026, 03:00 AM UTC
File uploads were broken in the admin dashboard.
Minor June 17, 2026 Some tags/collections not loading 30m Admin Dashboard, Apps
Started Jun 17, 2026, 09:39 PM UTC· resolved Jun 17, 2026, 10:10 PM UTC
- investigating Jun 17, 2026, 09:39 PM UTC
The Tags tab of the admin dashboard is currently not display tags, and some collections are not being populated in apps. We're actively investigating the issue.
- identified Jun 17, 2026, 09:57 PM UTC
We're currently testing and deploying a fix for this issue.
- resolved Jun 17, 2026, 10:10 PM UTC
The fix is live and full app and admin dashboard functionality is restored.
Minor June 8, 2026 Treefort Analytics down 3h 32m API, Admin Dashboard, Apps
Started Jun 08, 2026, 03:00 PM UTC· resolved Jun 08, 2026, 06:32 PM UTC
- investigating Jun 08, 2026, 03:00 PM UTC
We're seeing heightened error rates in our API and some requests are intermittently failing. Dashboard analytics are affected and some content requests are loading slowly or not at all. We're currently investigating.
- investigating Jun 08, 2026, 03:47 PM UTC
We've identified an issue with our analytics system. We are isolating the issue so that other aspects of the platform can fully recover while we work on a fix for analytics.
- identified Jun 08, 2026, 04:16 PM UTC
We are working on a fix for the analytics system. In the meantime analytics are still unavailable via the admin dashboard or API. We have finished processing webhooks that we backed up due to this issue and general access to the admin dashboard has been restored.
- resolved Jun 08, 2026, 06:32 PM UTC
All systems have recovered. We will be sending out a post-mortem to customers shortly.
Minor May 28, 2026 Network connectivity issues 1h Admin Dashboard, Apps
Started May 28, 2026, 09:33 PM UTC· resolved May 28, 2026, 10:33 PM UTC
- investigating May 28, 2026, 09:35 PM UTC
We are investigating network connectivity issues with an upstream provider. The Treefort admin dashboard is currently unavailable for some users.
- investigating May 28, 2026, 09:45 PM UTC
The connectivity issues are affecting apps as well. Users on the west coast may experience slow load times. The upstream provider is investigating the issue
- monitoring May 28, 2026, 10:14 PM UTC
A fix is being implemented. In the meantime we are continuing to see sporadically slow load times.
- resolved May 28, 2026, 10:40 PM UTC
The fix is complete and network connectivity has been fully restored.
Minor April 20, 2026 Bad deployment 47m Admin Dashboard, Apps
Started Apr 20, 2026, 04:55 PM UTC· resolved Apr 20, 2026, 05:42 PM UTC
- identified Apr 20, 2026, 04:55 PM UTC
We've identified a bad deployment that is causing articles to fail to load in apps and video series to fail to load in the admin dashboard.
- resolved Apr 20, 2026, 05:42 PM UTC
Both issues have been resolved. App downtime for articles was ~33 minutes. Admin dashboard downtime for video series was ~43 minutes.
Minor April 17, 2026 Partial outage 45m Apps
Started Apr 17, 2026, 01:48 PM UTC· resolved Apr 17, 2026, 02:34 PM UTC
- monitoring Apr 17, 2026, 01:48 PM UTC
We are monitoring lingering issues with several apps related to the previous outage.
- resolved May 12, 2026, 11:19 PM UTC
Minor April 17, 2026 Outage 18m Admin Dashboard, Apps
Started Apr 17, 2026, 12:51 PM UTC· resolved Apr 17, 2026, 01:10 PM UTC
- investigating Apr 17, 2026, 12:57 PM UTC
We are investigating a major platform outage.
- identified Apr 17, 2026, 01:08 PM UTC
Our upstream hosting provider is addressing an issue related to handling secure network traffic. We are seeing most services start to recover.
- resolved Apr 17, 2026, 01:15 PM UTC
All services have recovered.
Minor March 27, 2026 Authentication intermittently unavailable 1h API
Started Mar 27, 2026, 10:04 PM UTC· resolved Mar 27, 2026, 11:04 PM UTC
- investigating Mar 27, 2026, 09:09 PM UTC
We're seeing a spike in failed requests to our API and are actively investigating. Some thumbnails and content may not load, and user authentication may be slow or encounter errors.
- monitoring Mar 27, 2026, 09:19 PM UTC
An upstream provider is experiencing capacity issues resulting in intermittent network failures. Services are recovering at this time and error rates have dropped significantly. We are continuing to monitor apps to ensure full recovery.
- identified Mar 27, 2026, 11:08 PM UTC
Our upstream provider is having hardware issues with the machines our authentication service runs on. We are working to provision additional machines to restore full service. Some actions that require authentication may intermittently fail in the meantime.
- monitoring Mar 27, 2026, 11:25 PM UTC
We've performed a database failover to a new primary node and are seeing services recover. We're continuing to monitor.
- resolved Mar 28, 2026, 04:48 AM UTC
Minor March 27, 2026 Admin dashboard and user authentication problems 49m API, Admin Dashboard
Started Mar 27, 2026, 05:02 PM UTC· resolved Mar 27, 2026, 05:52 PM UTC
- investigating Mar 27, 2026, 05:02 PM UTC
We're investigating problems with our authentication service and admin dashboard.
- identified Mar 27, 2026, 05:10 PM UTC
We've identified unusually high load on one of our authentication database servers. We've restarted the database cluster to restore baseline service but response times may continue to be slow as the database catches up.
- monitoring Mar 27, 2026, 05:23 PM UTC
- resolved Mar 27, 2026, 05:53 PM UTC
Minor March 3, 2026 User logins intermittently failing 19m API
Started Mar 03, 2026, 11:15 PM UTC· resolved Mar 03, 2026, 11:35 PM UTC
- investigating Mar 03, 2026, 11:15 PM UTC
We're seeing a spike in authentication errors in the apps and are actively investigating.
- monitoring Mar 03, 2026, 11:24 PM UTC
We've identified and addressed the problem and are now monitoring the results. User logins should be working normally now.
- resolved Mar 03, 2026, 11:35 PM UTC
Authentication systems are now stable and working normally again.
Minor February 20, 2026 Slow authentication responses 1h 36m API
Started Feb 20, 2026, 07:45 PM UTC· resolved Feb 20, 2026, 09:22 PM UTC
- investigating Feb 20, 2026, 07:45 PM UTC
Logging in/logging out of apps and the admin dashboard is currently slow and may time out waiting for a response.
- identified Feb 20, 2026, 07:50 PM UTC
Authentication servers are experiencing unusual load, and we are scaling up machines to accommodate it.
- monitoring Feb 20, 2026, 07:56 PM UTC
Authentication response times have returned to normal and we're continuing to monitor.
- resolved Feb 20, 2026, 09:22 PM UTC
Minor February 20, 2026 Intermittent app unavailability 19m API, Apps
Started Feb 20, 2026, 04:49 PM UTC· resolved Feb 20, 2026, 05:09 PM UTC
- investigating Feb 20, 2026, 04:49 PM UTC
We're seeing issues with network requests to our API and are currently investigating.
- investigating Feb 20, 2026, 04:59 PM UTC
Our upstream provider is having continued issues in one of its datacenters. We are identifying affected machines and adding capacity to work around the problem. Apps are functional at this time but may continue to see performance issues.
- resolved Feb 20, 2026, 05:09 PM UTC
We've successfully mitigated the problems, and apps are responding normally. We're continuing to monitor app performance.
Minor February 19, 2026 Network connectivity problems 59m
Started Feb 19, 2026, 09:20 PM UTC· resolved Feb 19, 2026, 10:19 PM UTC
- investigating Feb 19, 2026, 09:20 PM UTC
We're seeing significant disruption in network traffic in our upstream provider's datacenters, causing major app degradation and outages. We're actively investigating.
- investigating Feb 19, 2026, 09:31 PM UTC
We've localized the issue to a problem with our API servers and are adding additional machines to compensate for increased network traffic. Apps are functional at this time but may continue to see degraded performance as we continue to work on the issue.
- monitoring Feb 19, 2026, 09:53 PM UTC
Our upstream provider has addressed their issue and our API servers have stabilized. All services are operating normally at present, but we are continuing to monitor.
- resolved Feb 19, 2026, 10:19 PM UTC
Servers remain stable and apps are functioning normally.
Minor February 3, 2026 Auth service degraded performance 16m API
Started Feb 03, 2026, 02:00 AM UTC· resolved Feb 03, 2026, 02:17 AM UTC
- investigating Feb 03, 2026, 02:13 AM UTC
We are investigating failed network requests to our authentication service. Some users may experience trouble signing in or out of Treefort apps or our admin dashboard.
- investigating Feb 03, 2026, 02:34 AM UTC
We provisioned new auth servers and performance has stabilized. We are continuing to investigate the root cause.
- resolved Feb 03, 2026, 02:55 AM UTC
Auth performance has returned to normal. Actual disruption to users was minimal.
Minor January 30, 2026 Intermittent login problems 25m
Started Jan 30, 2026, 07:35 PM UTC· resolved Jan 30, 2026, 08:00 PM UTC
- investigating Jan 30, 2026, 07:35 PM UTC
We're seeing increased error rates in our authentication service. Some users may be unable to log in at this time. We're currently investigating the problem.
- monitoring Jan 30, 2026, 07:49 PM UTC
One of the machines in our authentication service cluster was unhealthy and causing requests to fail. We've restarted the machine and error rates are now dropping. We're continuing to monitor to ensure service is fully restored.
- resolved Jan 30, 2026, 08:00 PM UTC
The authentication service is now stable and operating normally.
Minor December 17, 2025 Image uploads not working 19m Admin Dashboard
Started Dec 17, 2025, 07:55 PM UTC· resolved Dec 17, 2025, 08:14 PM UTC
- investigating Dec 17, 2025, 07:55 PM UTC
Admins are currently experiencing errors when trying to upload artwork on the admin dashboard. You may see an error that says "Something went wrong while uploading." We're actively investigating and will post an update when we know more.
- resolved Dec 17, 2025, 08:14 PM UTC
We identified the issue as a failing credential on our cloud storage. We rotated the credential and uploads are now working normally again.
Major November 18, 2025 Apps and admin dashboard intermittently unavailable 4h 45m Admin Dashboard, Apps
Started Nov 18, 2025, 11:45 AM UTC· resolved Nov 18, 2025, 04:30 PM UTC
- identified Nov 18, 2025, 11:45 AM UTC
One of the CDN providers we rely on is having global network issues, resulting in app errors and content failing to load. See https://www.cloudflarestatus.com/incidents/8gmgl950y3h7 for more details as they become available.
- monitoring Nov 18, 2025, 03:23 PM UTC
CDN issues appear to be resolved and services are back to normal.
- resolved Nov 18, 2025, 04:30 PM UTC
Minor November 14, 2025 Upstream network issues 27m Admin Dashboard, Apps
Started Nov 14, 2025, 06:38 PM UTC· resolved Nov 14, 2025, 07:05 PM UTC
- resolved Nov 14, 2025, 06:57 PM UTC
The underlying issue appears to have been resolved. The network was unreliable for a period of roughly 30 minutes. End user impact appears to have been minimal.
Minor November 3, 2025 Degraded performance 34m Admin Dashboard, Apps
Started Nov 03, 2025, 07:52 PM UTC· resolved Nov 03, 2025, 08:27 PM UTC
- investigating Nov 03, 2025, 08:19 PM UTC
We are seeing slow network responses across apps and the admin dashboard due to high CPU usage in our primary database. We are investigating the issue.
- identified Nov 03, 2025, 08:29 PM UTC
We've identified the issue as under provisioned database hardware. We are working on scaling the hardware.
- resolved Nov 03, 2025, 08:37 PM UTC
Hardware has been scaled and response times have returned to normal.