Treefort incident

Authentication intermittently unavailable

Minor Resolved

Treefort experienced a minor incident on March 27, 2026, lasting 1h. The incident has been resolved; the full update timeline is below.

Started
Mar 27, 2026, 10:04 PM UTC
Resolved
Mar 27, 2026, 11:04 PM UTC
Duration
1h
Detected by Pingoru
Mar 27, 2026, 10:04 PM UTC

Update timeline

  1. investigating Mar 27, 2026, 09:09 PM UTC

    We're seeing a spike in failed requests to our API and are actively investigating. Some thumbnails and content may not load, and user authentication may be slow or encounter errors.

  2. monitoring Mar 27, 2026, 09:19 PM UTC

    An upstream provider is experiencing capacity issues resulting in intermittent network failures. Services are recovering at this time and error rates have dropped significantly. We are continuing to monitor apps to ensure full recovery.

  3. identified Mar 27, 2026, 11:08 PM UTC

    Our upstream provider is having hardware issues with the machines our authentication service runs on. We are working to provision additional machines to restore full service. Some actions that require authentication may intermittently fail in the meantime.

  4. monitoring Mar 27, 2026, 11:25 PM UTC

    We've performed a database failover to a new primary node and are seeing services recover. We're continuing to monitor.

  5. resolved Mar 28, 2026, 04:48 AM UTC