Print Tracker Pro incident

Cluster Node Down

Major Resolved View vendor source →

Print Tracker Pro experienced a major incident on January 16, 2023 affecting DCA Server, lasting 1h 8m. The incident has been resolved; the full update timeline is below.

Started
Jan 16, 2023, 10:43 PM UTC
Resolved
Jan 16, 2023, 11:51 PM UTC
Duration
1h 8m
Detected by Pingoru
Jan 16, 2023, 10:43 PM UTC

Affected components

DCA Server

Update timeline

  1. identified Jan 16, 2023, 10:43 PM UTC

    A change to our production environment resulted in excessive resource usage targetted at a specific node in our Kubernetes cluster. This node has gone unresponsive and we're in the process of replacing it. We anticipate some downtime in the connections between the installs and our server which may result in delays when sending jobs to the installs.

  2. identified Jan 16, 2023, 10:46 PM UTC

    We are continuing to work on a fix for this issue.

  3. identified Jan 16, 2023, 11:10 PM UTC

    Installs are still unable to connect to the server, however, the webadmin remains fully operational and users should feel free to continue to use Print Tracker and make changes to their fleets.

  4. monitoring Jan 16, 2023, 11:42 PM UTC

    The failing cluster node is fully operational and installs should be re-connecting with the servers. We will be monitoring the system closely to ensure that the system is stable.

  5. resolved Jan 16, 2023, 11:51 PM UTC

    This incident has been resolved.