Tilroy incident
We register a higher number of errors about prices.
Tilroy experienced a minor incident on November 24, 2025 affecting Point Of Sale and E-commerce and 1 more component, lasting 1h 42m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 24, 2025, 08:53 AM UTC
We receive more calls as usual about errors on prices. This is under investigation.
- investigating Nov 24, 2025, 08:53 AM UTC
We are continuing to investigate this issue.
- identified Nov 24, 2025, 09:00 AM UTC
The issue has been identified and a fix is being implemented. We recommend to use the offline client.
- monitoring Nov 24, 2025, 09:26 AM UTC
A fix has been implemented and we are monitoring the results.
- identified Nov 24, 2025, 09:38 AM UTC
The issue has not been solved, we have identified the issue and are implementing a fix. It is recommended to use the offline client.
- monitoring Nov 24, 2025, 09:51 AM UTC
A fix has been implemented and we are monitoring the results.
- resolved Nov 24, 2025, 10:35 AM UTC
This incident has been resolved.
- postmortem Nov 24, 2025, 11:00 PM UTC
On 25/11 at 09:50, the core Point-of-Sale \(POS\) and backend API services experienced a severe performance degradation, leading to transactional timeouts and a critical slowdown across retail operations. The incident was triggered by a corruption of a primary index on the MongoDB collection. This caused queries to bypass the fast index and perform costly full collection scans, which rapidly consumed all available CPU and I/O resources. The resulting database congestion caused the application to have timeouts. The issue was resolved by rebuilding the index, restoring full service. Because of the size and load this took more than half an hour. We sincerely apologize for the disruption and the resulting inability to process transactions efficiently for the duration of the incident. We understand that peak performance is critical to your daily retail operations, and this incident fell short of the reliability standard you expect from our product. We confirm that customers utilizing the offline functionality were able to continue processing sales locally. All transactions recorded offline during the database degradation period were successfully synced to the central database immediately after the service was restored, ensuring no loss of sales data. Thank you for your patience during this incident.