SpringboardVR incident

Time sync failing, causing some Monitor and Station functions to fail

Notice Resolved View vendor source →

SpringboardVR experienced a notice incident on September 4, 2019, lasting 32m. The incident has been resolved; the full update timeline is below.

Started
Sep 04, 2019, 04:15 PM UTC
Resolved
Sep 04, 2019, 04:48 PM UTC
Duration
32m
Detected by Pingoru
Sep 04, 2019, 04:15 PM UTC

Update timeline

  1. investigating Sep 04, 2019, 04:15 PM UTC

    We are currently investigating this issue.

  2. resolved Sep 04, 2019, 04:48 PM UTC

    The issue was Google's Firebase cloud functions. You can see their issue report here. https://status.firebase.google.com/incident/Hosting/19014 We use this to synchronize time between the monitor and station and to avoid time drift. We request the current time from the this service every two minutes and when it fails the system will get confused and prevents some remote actions from firing. We've already identified a way for us to 'fallback' to client machine time if this happens again and will patch a fix soon. Thanks for your patience, Jamie