Storyous experienced a notice incident on October 24, 2021 affecting Pokladní systém, tablety (POS) and Administration (admin.storyous.com), lasting 3d 15h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 24, 2021, 05:43 PM UTC
A part of our users is not able to use Admin and POS due to an error in our infrastructure.
- identified Oct 24, 2021, 05:47 PM UTC
The issue has been identified and a fix is being implemented.
- identified Oct 24, 2021, 10:08 PM UTC
We are forced to leave the affected users offline and wait until morning for our infrastructure provider support which can help us with complete data recovery.
- monitoring Oct 25, 2021, 08:15 AM UTC
We are trying to prevent data loss and waiting for our infrastructure provider support to reboot the system. The expected fix will be around 11 AM.
- monitoring Oct 25, 2021, 10:41 AM UTC
Unfortunately, the fix did not help and we are trying to solve the problem another way.
- monitoring Oct 25, 2021, 01:18 PM UTC
Our infrastructure provider is investigating our case with the highest priority. Next update in 1 hour.
- monitoring Oct 25, 2021, 03:21 PM UTC
We are still waiting for our infrastructure provider to fix the issue.
- monitoring Oct 25, 2021, 06:22 PM UTC
If our infrastructure provider will not come up with the fix, we have some workaround with another company, it is already in progress, but it will take some time to prepare it. Delivery time is estimated for tomorrow morning.
- monitoring Oct 26, 2021, 08:23 AM UTC
Our infrastructure provider is not able to help us, we are waiting for another company to finish creating the dump of the database. They started working on it yesterday in the evening and it can take 12 to 14 hours to prepare it. Unfortunately, we can not say the exact time when it will be done. Next update in 1 hour.
- monitoring Oct 26, 2021, 09:17 AM UTC
We are still waiting to be contacted when the dump of the database will be successfully created. Next update in 1 hour.
- monitoring Oct 26, 2021, 10:18 AM UTC
We are still waiting for the message that the dump of the database is ready.
- monitoring Oct 26, 2021, 11:36 AM UTC
We got the message that the dump of the database is ready and they started with import on the server. Next update in 1 hour.
- monitoring Oct 26, 2021, 12:45 PM UTC
We are progressing in resolving the issue and it is going in the right direction. It can still take up to several hours to have a fully working environment.
- monitoring Oct 26, 2021, 01:44 PM UTC
Based on current progress we expect that service will be restored today in the evening between 21-24.
- monitoring Oct 26, 2021, 07:29 PM UTC
We have almost good news. We need to do some last processing during this night, which will finish about 3 am, but we are quite sure this is the last part necessary. At 7 am we start with the deployment process and between 8-9 am we should be online. Then we expect it will take about 30-45 minutes to fully sync all bills from all devices. Our main goal is to be fully operational before 10 am. Due to a data corruption bug, there can be some data loss for Sunday and Monday unfortunately, although we tried to keep it as low as possible and we expect a maximum of a few bills per place (if any).
- monitoring Oct 27, 2021, 07:18 AM UTC
Everything is processing as expected, admin login should work for most of the merchants, historical data are being processed in the background. Most users should be able to log in to admin. We expect that service can be slower than usual or it can take a while to upload all historical data from all ePOS in the next 60 minutes.
- monitoring Oct 27, 2021, 06:02 PM UTC
The administration system and ePOS should work without any problem. However, we are still monitoring the situation.
- resolved Oct 28, 2021, 09:37 AM UTC
This incident has been resolved.