Read Only History Lag

Update timeline

1. identified Mar 09, 2023, 05:21 AM UTC

   We are seeing some history lag on our read only database again. All data is caught up on our primary write database, just desktop and mobile apps will show a lag. Working on making more configuration changes to help the issue.

2. monitoring Mar 09, 2023, 05:55 AM UTC

   We have implemented a fix and data is catching up again. We are working to identify the root cause and will post as soon as we have a permanent solution.

3. identified Mar 09, 2023, 09:48 AM UTC

   We are reconfiguring much of the read only database. You may see lag in historical data for the current month, callout log, and alarm events on the desktop and mobile clients. All data is flowing into our primary write database and realtime data and alarms are operating normally. The read only database exists to reduce load on the primary database but is having issues. Our team is working hard on the issue.

4. identified Mar 09, 2023, 11:07 AM UTC

   We have bypassed the read only database for now while we work on a permanent fix. We have to manually fill some data and perform more configuration but it should help us resolve the issue much faster. Realtime data, alarms and notifications remain fully operational.

5. monitoring Mar 09, 2023, 12:10 PM UTC

   We continue to backfill some calculated flow, callout log, alarm event, and historical data. The system is stable. Disruption has been lessened by our ability to keep realtime data, alarming, and many other aspects of the system up. Most of the data left to backfill is used in caching/performance and is not critical to operations. We will post when all data has been restored. We thank you for your patience.

6. monitoring Mar 09, 2023, 05:34 PM UTC

   Historical data has been backfilled for a few hours. We are still working on some calculated flow, callout log, and historical alarm event data.

7. monitoring Mar 09, 2023, 06:36 PM UTC

   Historical alarm event data is now current, without lag. There is still a gap in the last 24 hours we are backfilling, all historical data prior is still loading as it has been.

8. monitoring Mar 09, 2023, 09:04 PM UTC

   As part of our updates we are restarting the main service. You may see values go in and out for the next 5-10 minutes.

9. monitoring Mar 09, 2023, 09:07 PM UTC

   The main service has recovered.

10. monitoring Mar 09, 2023, 11:45 PM UTC

    Calculated flow data has been backfilled. We still have a small amount of callout log and alarm event data to backfill and will post a final update when complete.

11. monitoring Mar 10, 2023, 05:54 PM UTC

    The system is still functioning well, and has been been since early the morning of 03/09. We have more scale and performance improvements coming. We are still working on a very small amount of historical callout log and alarm event data that needs to be backfilled but should have that resolved in a matter of hours. We will leave this incident open for visibility but the system continues to be fully operational.

12. monitoring Mar 10, 2023, 06:38 PM UTC

    The small amount of remaining alarm event data has been backfilled.

13. resolved Mar 10, 2023, 08:35 PM UTC

    The final few thousand missing CalloutLogEntries have been backfilled. Thank you for your patience. We are going to resolve this incident. Contact [email protected] with any questions or concerns.