SpringboardVR experienced a critical incident on March 2, 2019 affecting Operator Panel and VR Launcher and Services, lasting 5h 58m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Mar 02, 2019, 08:38 PM UTC
We are experiencing issues with write access to our databases. This is causing things to appear in the Monitor etc. properly but is preventing you from creating or editing reservations.
- monitoring Mar 02, 2019, 08:40 PM UTC
Write access is restored and we are monitoring the situation.
- identified Mar 02, 2019, 09:37 PM UTC
We've noticed a few more connection issues and are currently rebuilding some of our database read replicas. There may be degraded performance for a few minutes while the scaling occurs.
- identified Mar 02, 2019, 10:06 PM UTC
We're continuing to see database sync errors between our Read and Write servers. We are recreating our entire database stack now.
- monitoring Mar 02, 2019, 10:28 PM UTC
We've entirely rebuilt our database from a backup and everything appears to be stable now. We're continuing to monitor the situation but things appear stable now.
- resolved Mar 03, 2019, 02:36 AM UTC
After our database rebuild everything has returned to being stable and operating at full capacity