Fourth incident
High Priority Incident - Tradesimple - Delays in Document Processing
Fourth experienced a major incident on May 17, 2024 affecting FnB (Stock Live) and Inventory (Stock R9) and 1 more component, lasting 10d 10h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 17, 2024, 10:10 PM UTC
Dear Customer, We regret to inform you that we are currently experiencing an issue that is causing delays in processing documents on Tradesimple, including orders. Our team is working tirelessly to resolve this matter, however to ensure that orders do not miss any cutoffs, we recommend placing them well in advance, preferably at least 30 minutes before the deadline. We sincerely appreciate your understanding and patience as we strive to resolve this issue as quickly as possible. Your continued support is greatly appreciated. An update will be provided no later than Saturday morning.
- investigating May 17, 2024, 10:15 PM UTC
We are continuing to investigate this issue.
- investigating May 18, 2024, 08:18 AM UTC
Dear Customer, We regret to inform you that we are currently experiencing an issue that is causing delays in processing documents on Tradesimple, including orders. Our team is continuing to work tirelessly to resolve this matter, however to ensure that orders do not miss any cutoffs, we recommend placing them well in advance, preferably at least 30 minutes before the deadline. We sincerely appreciate your understanding and patience as we strive to resolve this issue as quickly as possible. Your continued support is greatly appreciated.
- investigating May 19, 2024, 08:14 PM UTC
Dear Valued customer, . Throughout the weekend, our teams have been working tirelessly to resolve this issue. Unfortunately, despite making progress in the investigation, the issue remains unresolved. . Current situation: We are experiencing intermittent delays in processing documents through Tradesimple. Our current hyper care actions ensures that the majority of documents, including orders, are processed with minimal delays. In the rare event that a document fails, we receive alerts that are immediately acted upon. As an extra measure, we recommend that customers place orders promptly to minimize the risk of missing any cut-off times. We advise all sites to place orders at least 30 minutes in advance. Please be assured that our team is treating this issue with the utmost priority and continues to work to resolve it as a matter of urgency. We sincerely apologize for the disruption to your service and assure you that we are doing everything we can to resolve this.
- investigating May 20, 2024, 08:42 AM UTC
Dear Valued Customer, Our teams are continuing the investigation to resolve this problematic issue. Currently, we are still seeing intermittent delays in processing orders. Although our fail-safes are still in place, when delays are experienced, they are considerable and have the potential to be business-impacting. To ensure smooth operations, we recommend making alternate arrangements to ensure that suppliers receive any orders with pressing importance.
- investigating May 20, 2024, 11:31 AM UTC
Dear Valued Customer, Our teams have performed actions in an attempt to improve the processing of orders and we can confirm that we are seeing a significant improvement. Further actions are being implemented to bring the system back to expected levels of performance and our hyper-care is ongoing in order to keep customer impact to a minimum. We sincerely apologize for the disruption to your service and assure you that we are doing everything we can to resolve this.
- investigating May 20, 2024, 04:13 PM UTC
Dear Valued Customer, We have continued to monitor performance levels throughout the day and are satisfied that actions taken have had a direct impact on improvement in the situation. We will continue to actively monitor throughout the evening and into tomorrow. We greatly appreciate your patience and understanding on this issue.
- investigating May 21, 2024, 08:46 AM UTC
Dear Valued Customer, We are pleased to advise that the solutions have remained stable throughout the evening and into this morning. We have continued to monitor performance levels and have seen no signs of the issues that have affected stability. We plan to continue with a period of active monitoring for several days to ensure the systems remain stable and fit for purpose as we approach the bank holiday weekend. We greatly appreciate your patience and understanding on this issue.
- investigating May 23, 2024, 10:13 AM UTC
Dear Valued Customer, We wanted to inform you that we conducted maintenance from 00:30 to 03:21 UTC as part of our commitment to providing you with the best possible service. We noticed a slight delay in processing messages, however, our team has rectified this. Thank you for your understanding and patience as we work to improve your experience.
- monitoring May 23, 2024, 04:41 PM UTC
Dear Valued Customer, We have now implemented all actions to resolve these issues and are currently monitoring for completeness. The next update will be provided upon confirmation of resolution. We apologise for this disruption to your service.
- resolved May 28, 2024, 08:32 AM UTC
We are confident that our solution has resolved this incident. This will conclude the updates to the status page regarding this outage however a formal Incident Report will be posted here with the next 5 business days. We greatly apologise for this disruption.