Tilroy experienced a notice incident on August 8, 2024, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Aug 08, 2024, 03:08 PM UTC
Between 12:50 and 12:30 we received a high number of errors and error reports while scanning products. This problems was identified and fix implemented.
- postmortem Aug 08, 2024, 03:14 PM UTC
#### Incident Overview On 8/8 at approximately 12:15 , our production environment experienced a problem due to a human error made during an urgent change in the system \(hot fix\). This incident resulted in a high number of errors, leading to an interruption of services for many users. We deeply regret any inconvenience this caused and want to provide a detailed explanation of what occurred, the immediate steps taken to resolve the issue, and the long-term measures we are implementing to prevent a recurrence. #### What Happened During an urgent update to our system, a developer made a critical change that caused errors in our production environment. Although we have processes in place to handle testing, changes and updates, these processes were not fully secure, leading to the issue. Additionally, our communication about the high number of errors was insufficient at the moment of the incident. #### Immediate Actions Taken 1. **Identification and implement fix** : As soon as the issue was identified, our team implemented a fix and stabilized the system. 2. **Error Monitoring**: Detailed monitoring to ensure all errors were identified and addressed in real-time. To prevent such incidents from occurring in the future, we are reviewing the current guidelines to make changes during retail hours. We will enhance our communication to ensure timely and effective information to our users in the event of any system issues, also when less staff is available like holiday periods. #### Conclusion We sincerely apologize for the disruption of services caused by this incident. Our commitment to providing a reliable and high-quality service to our users remains our top priority. We appreciate your understanding and patience as we work to improve our systems and processes to prevent such occurrences in the future.