Fourth incident
High Priority Incident - Engage & Connected Apps - Slowness and Errors
Fourth experienced a critical incident on July 29, 2024 affecting Analytics and Engage and 1 more component, lasting 1d 8h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 29, 2024, 07:47 AM UTC
We are aware of a high priority incident that may be affecting some of our customers. This is incident is currently being investigated to determine the root cause of the issue. Status and updates will continue here every 60 minutes until the incident has been resolved. We apologise for this disruption to your service.
- investigating Jul 29, 2024, 08:00 AM UTC
We are aware of a critical incident that may be affecting some of our customers. This is incident is currently being investigated to determine the root cause of the issue. Status and updates will continue here every 30 minutes until the incident has been resolved. We apologise for this disruption to your service.
- investigating Jul 29, 2024, 08:30 AM UTC
We are aware of a critical incident that may be affecting some of our customers. We are continuing to investigate to determine the root cause of the issue. Status and updates will continue here every 30 minutes until the incident has been resolved. We apologise for this disruption to your service.
- investigating Jul 29, 2024, 09:01 AM UTC
We are aware of a critical incident that may be affecting some of our customers. We are continuing to investigate to determine the root cause of the issue. Status and updates will continue here every 30 minutes until the incident has been resolved. We apologise for this disruption to your service.
- investigating Jul 29, 2024, 09:33 AM UTC
We are aware of a critical incident that may be affecting some of our customers. We are continuing to investigate to determine the root cause of the issue. Status and updates will continue here every 30 minutes until the incident has been resolved. We apologise for this disruption to your service.
- investigating Jul 29, 2024, 10:03 AM UTC
UPDATE FOR CUSTOMERS USING OUR INVENTORY SOLUTIONS: We strongly advise customers to invoke any disaster recovery procedures they have in place to ensure suppliers receive orders in a timely fashion. We are continuing to investigate to determine the root cause of the issue. Status and updates will continue here every 30 minutes until the incident has been resolved. We apologise for this disruption to your service.
- investigating Jul 29, 2024, 10:39 AM UTC
FOR ALL CUSTOMERS: Given the nature and impact of this incident, we must advise that the following services may be impacted. • POS data submissions • Reporting data • Updates to Single Sign On users • Creation of new SSO users • Updating information of existing employees within our Workforce Management Solutions • Data imports for RME FOR CUSTOMERS USING OUR INVENTORY SOLUTIONS: We strongly advise customers to invoke any disaster recovery procedures they have in place to ensure suppliers receive orders in a timely fashion. We are continuing to investigate to determine the root cause of the issue. Status and updates will continue here every 30 minutes until the incident has been resolved. We apologise for this disruption to your service.
- investigating Jul 29, 2024, 11:09 AM UTC
Our technical teams have implemented corrective actions that we believe have improved the situation with all affected services. Customers should see immediate improvements when logging in through SSO. However, it will take some time for the systems to process the backlog of data. Please be aware that our teams are continuing to investigate the root cause and any additional remedial actions that may be required.
- investigating Jul 29, 2024, 11:37 AM UTC
Our technical teams have implemented corrective measures that we believe have improved the situation with all affected services. Customers should notice immediate improvements when logging in through SSO. However, please note that it will take some time for the systems to process the backlog of data. We are continuing to investigate the root cause and determine any additional remedial actions that may be required, which could include the resubmission of POS and T&A data submitted during the affected period.
- investigating Jul 29, 2024, 11:59 AM UTC
Our technical teams have implemented corrective measures that we believe have improved the situation with all affected services. Customers should notice immediate improvements when logging in through SSO. However, please note that it will take some time for the systems to process the backlog of data. We are continuing to investigate the root cause and determine any additional remedial actions that may be required, which could include the resubmission of POS and T&A data submitted during the affected period. The next update will be provided when further information is available.
- monitoring Jul 29, 2024, 01:51 PM UTC
We are confident that the actions taken previously have returned our Core API and Single Sign On to full functionality. These systems are currently working through the backlog which could take up to 24hrs for this to clear and bring the data in the core solutions up to date. This includes but is not limited to: • POS data submissions • Reporting data • Updates to Single Sign On users • Creation of new SSO users • Updating information of existing employees within our Workforce Management Solutions • Data imports for RME Although there are in-built retry mechanisms in place, if you believe that your platform is missing any data, please raise a ticket via our help centre and we will do all we can to assist. We apologise for this disruption to your service.
- monitoring Jul 30, 2024, 08:58 AM UTC
Dear Customer, We are pleased to inform you that our proactive monitoring efforts have successfully restored our solutions to full service. We have cleared all backlogs and updated all data to ensure accuracy. Rest assured, we will continue to closely monitor performance throughout the day to ensure ongoing reliability. If you happen to notice any data discrepancies on your platform, please don't hesitate to reach out to us via our help center, and we will promptly assist you. Thank you for your understanding and support during this time.
- resolved Jul 30, 2024, 04:30 PM UTC
After monitoring throughout the day, we are confident that the issue is resolved for all customers. This will conclude the updates to the status page regarding this outage however a formal Incident Report will be posted here with the next 5 business days. We greatly apologise for this disruption.