DB Replica lag affecting site responsiveness

Update timeline

1. investigating Mar 26, 2026, 02:58 PM UTC

   We are currently investigating the issue as the main database has 32GB of replica lag.

2. monitoring Mar 26, 2026, 03:33 PM UTC

   We've identified the issue and restored site functionality. We are still monitoring the system as we identify the underlying cause. Screen generation will still see lag due to the backlog and one worker being taken offline.

3. monitoring Mar 26, 2026, 05:45 PM UTC

   Screen generation workers continue to work through the backlog. This can result in: - Existing plugins not showing up-to-date information (old screeens) - New devices showing "You're all caught up!" due to no screens yet generated

4. monitoring Mar 26, 2026, 06:22 PM UTC

   All workers are caught up, but one worker is currently still offline. While this is technically degraded, the existing workers can handle the ongoing load.

5. resolved Mar 26, 2026, 07:58 PM UTC

   All screen generation workers and APIs have been restored. Internal infrastructure analysis and adjustments will continue with no impact on customers or operations.