TankLogix incident

Historical Lag

Minor Resolved View vendor source →

TankLogix experienced a minor incident on March 8, 2023 affecting Historical Data, lasting 6h 29m. The incident has been resolved; the full update timeline is below.

Started
Mar 08, 2023, 08:36 PM UTC
Resolved
Mar 09, 2023, 03:05 AM UTC
Duration
6h 29m
Detected by Pingoru
Mar 08, 2023, 08:36 PM UTC

Affected components

Historical Data

Update timeline

  1. investigating Mar 08, 2023, 08:36 PM UTC

    We are monitoring some lag recording data to our historical database. This could cause historical data and alarm notification logs to be behind on the desktop and mobile apps.

  2. identified Mar 08, 2023, 09:15 PM UTC

    We have identified the issue and data is catching up.

  3. identified Mar 08, 2023, 09:20 PM UTC

    We are restarting the historical data Ignition gateway, you may see system interruption for 5-10 minutes.

  4. monitoring Mar 08, 2023, 10:05 PM UTC

    Data is caught up, we will be manually backfilling some data and will post again when complete.

  5. monitoring Mar 09, 2023, 12:02 AM UTC

    We are still seeing some lag on our read only database (used for clients and mobile apps) and continue to monitor the situation closely

  6. monitoring Mar 09, 2023, 12:33 AM UTC

    We are performing some maintenance to our read only database, realtime values are working, historical data is safe. You may see the most recent month of historical data become unavailable while we apply a fix.

  7. monitoring Mar 09, 2023, 01:17 AM UTC

    Current month data should be catching up again now. Thank you for your patience.

  8. monitoring Mar 09, 2023, 02:10 AM UTC

    Everything appears caught up. We will continue to monitor closely.

  9. resolved Mar 09, 2023, 03:05 AM UTC

    This incident has been resolved, we will continue monitoring closely.