Rebuy experienced a major incident on November 13, 2024 affecting Smart Cart and Checkout Extensions Widgets and 1 more component, lasting 4h 29m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 13, 2024, 05:19 PM UTC
We're experiencing an unexpected issue causing Rebuy to be temporarily unavailable for some stores. Our team is working to restore full functionality as quickly as possible. We apologize for the inconvenience and appreciate your patience. Please check back soon for updates, and thank you for being a valued Rebuy user.
- identified Nov 13, 2024, 05:22 PM UTC
The issue has been identified and a fix is being implemented.
- identified Nov 13, 2024, 05:31 PM UTC
We are continuing to work on a fix for this issue.
- identified Nov 13, 2024, 05:41 PM UTC
Issues are beginning to resolve for some customers, and the list of restored functionality continues to grow. We're actively working to complete the fix for all users. Thank you for your patience.
- identified Nov 13, 2024, 05:58 PM UTC
We're seeing intermittent success with Rebuy loading, and our team is actively working on a full resolution. Thank you for your patience.
- identified Nov 13, 2024, 06:15 PM UTC
We're implementing additional measures to expedite the resolution for the impacted sites.
- identified Nov 13, 2024, 07:13 PM UTC
The additional changes have been released, and we are restarting services to ensure they take effect.
- identified Nov 13, 2024, 07:56 PM UTC
Restarts and remediation efforts are ongoing as we work toward full resolution. Thank you for your patience.
- monitoring Nov 13, 2024, 08:27 PM UTC
A fix has been implemented, and we’re seeing resolution across most features. Our team is still working on Smart Cart and continues to test and monitor to ensure stability for all other features. Thank you for your patience!
- monitoring Nov 13, 2024, 09:39 PM UTC
A fix has been published for Smart Cart and the issue appears resolved. We are continuing to monitor.
- resolved Nov 13, 2024, 09:48 PM UTC
The issue has been resolved. We'll be conducting a thorough review and will share a post-mortem with insights on what occurred and the steps we're taking to prevent it in the future. Thank you for your patience!
- postmortem Nov 15, 2024, 01:39 PM UTC
**Summary:** An unexpected issue caused a cache-clearing function to trigger unintentionally, leading to an overload that ultimately locked a database. Customers experienced intermittent issues, though some were unaffected, others saw various symptoms. To address this, we disabled key functions suspected to be the cause and implemented a hotfix to prevent further database locks. Once these measures were implemented and services were restarted, the issue was fully resolved. **Next Steps:** As a preventative measure, we will conduct a thorough code review of the cache-clearing function to identify and address any vulnerabilities that could lead to unintended triggers.