Probado incident
Devices not displaying correct settings / allowing connections to servers
Probado experienced a major incident on June 15, 2023 affecting Provisioning Infrastructure, lasting 1h 7m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Jun 15, 2023, 08:55 AM UTC
We've identified an issue with our provisioning infrastructure preventing users devices from being able to collect their settings after being powered on. Our team has taken remedial action and users should again start to see their devices collecting the appropriate configuration settings. We're continuing to monitor devices to ensure they are collecting their configurations in a timely manner. we recommend powering off and on your device if you are still having issues ( this is applicable to both Phones and Terminals)
- monitoring Jun 15, 2023, 09:02 AM UTC
We are continuing to monitor for any further issues.
- resolved Jun 15, 2023, 09:28 AM UTC
All Devices continue to collect their configs as normal. Those users who are experiencing issues please try powering off and on your device before contacting the helpdesk.
- postmortem Jun 15, 2023, 09:41 AM UTC
**Incident:** out of hours maintenance was carried out on one of our Infrastructure servers. These particular types of servers manage and deploy configurations to devices so they those devices are able to cater for clients needs. During the AM Support staff identified that we had lost communication with one particular server and the server in question was not handing out configuration settings when they were requested to do so by client devices. **Resolution:** Upon identifying this issue the support team performed a power cycle of the affected server which restored all connectivity and a flurry of requests began to come through on our monitoring logs indicating clients were again successfully collecting their configs. We continued to directly monitor this servers behaviour to ensure it remained stable and performed during the high volume of traffic coming in. **Take away:** We’re investigating additional strategies for how these clusters of servers operate including:- * transitioning to faster more performant hardware * adding further \(additional\) cluster resilient strategies to these types of servers. We apologise for this outage and rest assured we will endeavour to deploy the appropriate strategies as soon as humanly possible.