SMRTs experienced a minor incident on March 7, 2019 affecting Reporting / KPI System, lasting 18h 57m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 07, 2019, 03:49 PM UTC
We're currently investigating an issue where many Reports & KPIs are failing to load.
- identified Mar 07, 2019, 04:35 PM UTC
Our provider Amazon Web Services has confirmed one of their datacenters are experiencing an outage, we're currently redirecting traffic to other datacenters to bring the system back to normal. While reports/KPIs are experiencing issues, mission critical components such as POS and Assembly should operate normally.
- identified Mar 07, 2019, 06:24 PM UTC
KPI/Reports are still experiencing issues, we are in constant contact with AWS to resolve the datacenter issues.
- identified Mar 07, 2019, 07:10 PM UTC
We’re currently directing traffic to another data center to resolve issue, ETA is 30min.
- identified Mar 07, 2019, 10:00 PM UTC
All systems are 100% operational. KPI’s and Reports are generating with new information; however, a re-sync of the database is required to restore historical data to the reports. We will work with Amazon Web Services to resolve and update you once the re-sync is complete.
- identified Mar 07, 2019, 10:02 PM UTC
We are continuing to work on a fix for this issue.
- monitoring Mar 08, 2019, 04:23 AM UTC
The original database has been restored via a previous snapshot and we've synced all data was added after that point. To further prevent any future outages we've also configured our KPI database to exist over multiple data-centers. We will continue to monitor the results.
- resolved Mar 08, 2019, 10:46 AM UTC
Together with the AWS operations team, we've monitored the reports / KPI database through the night without issues. Processes are being implemented to ensure report / KPI availability during future AWS datacenter outages. Please do not hesitate to reach to reach out to the SMRT support team if you have any questions or concerns regarding the report / KPI system.