Rebuy Outage History

Started May 29, 2025, 06:44 PM UTC· resolved May 29, 2025, 08:21 PM UTC

1. investigating May 29, 2025, 06:44 PM UTC

   We are currently investigating the issue.

2. investigating May 29, 2025, 06:46 PM UTC

   We are investigating the issue

3. identified May 29, 2025, 06:54 PM UTC

   The issue has been identified, and a fix is being implemented.

4. monitoring May 29, 2025, 06:59 PM UTC

   A fix has been implemented and we are monitoring the results.

5. resolved May 29, 2025, 08:21 PM UTC

   The issue involving Smart Search loading has been fully resolved, and everything is functioning as expected.An update included an improperly formatted file, which conflicted with smart search. The file was reformatted to correct the issue.

Read the full incident report →

Started May 16, 2025, 04:34 PM UTC· resolved May 16, 2025, 04:55 PM UTC

1. investigating May 16, 2025, 04:34 PM UTC

   We are currently seeing issues with checkout widgets and content blocks. The team is actively engaged.

2. investigating May 16, 2025, 04:36 PM UTC

   We are continuing to investigate this issue.

3. identified May 16, 2025, 04:49 PM UTC

   The issue has been identified and a fix is being implemented.

4. resolved May 16, 2025, 04:55 PM UTC

   This incident has been resolved.

5. postmortem May 19, 2025, 01:56 PM UTC

   **Issue:** Some Shopify merchants experienced issues with widgets not displaying correctly in their checkouts. This was linked to a new feature we recently introduced within the checkout extension framework. The disruption was traced back to a logic issue that caused valid widget IDs to be misinterpreted as invalid, impacting the visibility of the widget during checkout. ‌ **Actions Taken:** * **Rollback to Stable Version:** As soon as the issue was confirmed, we rolled back to the last known-good version of the extension to limit further impact. * **Root Cause Fix:** Once identified, the root cause was addressed with a hot-fix to ensure valid widget IDs were recognized appropriately.

Read the full incident report →

Started Apr 22, 2025, 08:01 AM UTC· resolved Apr 22, 2025, 08:41 AM UTC

1. investigating Apr 22, 2025, 08:01 AM UTC

   We are currently investigating reports of login and navigation difficulties within the Rebuy administrative portal. These issues appear to be affecting user access and navigation, but all on-site features, such as widgets, Smart Cart, Smart Search, and other core functionalities, remain fully operational. Our team is working to resolve the issue as quickly as possible, and we will provide further updates soon.

2. monitoring Apr 22, 2025, 08:30 AM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Apr 22, 2025, 08:41 AM UTC

   The issues have been resolved following the recent changes applied. We will conduct a post-mortem analysis and share the findings.

Read the full incident report →

Started Apr 01, 2025, 02:05 AM UTC· resolved Apr 01, 2025, 07:20 AM UTC

1. investigating Apr 01, 2025, 02:05 AM UTC

   We’re currently investigating multiple reports of sync issues that are intermittently affecting reporting features, on-site widgets, and the smart cart. This issue appears to be sporadic and does not impact all clients. Our engineering team is actively working to identify and resolve the problem. We’ll provide updates as we learn more.

2. investigating Apr 01, 2025, 02:18 AM UTC

   We are continuing to investigate this issue.

3. investigating Apr 01, 2025, 02:33 AM UTC

   A change is being implemented to help alleviate the issue.

4. investigating Apr 01, 2025, 02:51 AM UTC

   We're beginning to see some errors resolving and signs of improvement, but the issue is not yet fully resolved. Our team continues to actively troubleshoot and assess next steps to fully address the underlying problem.

5. investigating Apr 01, 2025, 03:22 AM UTC

   We are continuing to see improvements and are implementing an additional change to further alleviate the issue while we continue to investigate root cause.

6. investigating Apr 01, 2025, 04:01 AM UTC

   Additional improvements are being observed; however, sporadic issues persist across some stores, primarily related to product synchronization. Our team is continuing to investigate the situation.

7. investigating Apr 01, 2025, 04:57 AM UTC

   Product images in the widgets are still exhibiting inconsistent behavior, but the carts and checkout functionality continue to be working properly. Our team is working on identifying the root cause.

8. identified Apr 01, 2025, 05:50 AM UTC

   The issue has been identified, and we are actively deploying a fix.

9. monitoring Apr 01, 2025, 07:14 AM UTC

   A fix has been implemented and we are monitoring the results.

10. resolved Apr 01, 2025, 07:20 AM UTC

    The issue has been resolved. Our team is preparing a formal RCA and will post it once complete.

11. postmortem Apr 04, 2025, 10:24 PM UTC

    **Issue:** Customers experienced degraded service due to issues with product images, links, and webhooks, impacting both the admin panel and certain on-site features such as widgets and Smart Cart. The problem was identified after a scheduled webhook health check triggered which led to processing issues. **Root Cause:** The issue stemmed from the update to our APIs to align with Shopify's required version, which included changes to webhook handling and product/variant data formats. After the update, malformed data started to appear during the webhook health check. This caused increased processing load on Bigtable, leading to service degradation across other systems. The malformed data also caused errors in the Smart Cart service and issues with product images and links in the admin panel and widgets. The data corruption was traced to conflicts between new and old webhooks attempting to process similar items at the same time. **Actions Taken:** * **Bigtable Load Reduction:** Immediate actions were taken to reduce the load on Bigtable, including manual resource updates, which helped stabilize the system temporarily. * **Smart Cart Hotfix:** The Smart Cart team released hotfixes to handle the malformed data, resolving the errors in the Smart Cart service. * **Workaround for Inventory Webhooks:** A temporary solution was implemented by ignoring webhooks related to inventory levels to prevent further processing of malformed data. * **Data Resync:** We initiated the process of resyncing all affected products to remove the malformed data and prevent further issues. **Next Steps:** We are currently working on fully resolving the webhook issue related to product inventory levels. This will allow us to rollback the workaround for inventory webhooks currently in place and help to prevent similar issues from occurring in the future. **Update:** All related tasks to the post mortem have been completed and confirmed successful.

Read the full incident report →

Started Mar 25, 2025, 03:32 AM UTC· resolved Mar 25, 2025, 04:49 AM UTC

1. investigating Mar 25, 2025, 03:32 AM UTC

   We are currently experiencing a high volume of 503 errors, which is impacting service availability. Our team is actively investigating and working to resolve the issue as quickly as possible. Thank you for your patience.

2. identified Mar 25, 2025, 03:43 AM UTC

   We have identified a potential root cause of the issue and are taking the necessary actions to resolve it.

3. identified Mar 25, 2025, 04:04 AM UTC

   We are continuing to work on resolving the issue. Thank you for your patience as we address this.

4. identified Mar 25, 2025, 04:34 AM UTC

   A fix has been implemented, and we are beginning to see a decrease in errors. More updates to follow.

5. monitoring Mar 25, 2025, 04:35 AM UTC

   A fix has been implemented and we are monitoring the results.

6. monitoring Mar 25, 2025, 04:43 AM UTC

   We are continuing to monitor for any further issues.

7. resolved Mar 25, 2025, 04:49 AM UTC

   The issue has been resolved. Our team will prepare a formal Root Cause Analysis and post it on the status page incident. Thank you for your patience throughout this process.

8. postmortem Mar 25, 2025, 06:12 PM UTC

   **Issue:** Customers experienced 503 Service Unavailable errors, impacting On-site functionality. The issue was identified during routine alert monitoring and in response to reports of degraded service. **Root Cause:** The issue stemmed from the Ingress configuration not being properly applied after an update to the NGINX controller. Specifically, the upgrade to NGINX version 1.12.1 as part of a critical security vulnerability update led to a misapplication of the configuration. The security evaluation process on the configuration snippets caused delays in loading, resulting in traffic being directed to default configurations, which led to service disruptions. **Actions Taken:** * **Stage Environment Rollback:** To resolve the issue, the rollback plan was initiated, downgrading from NGINX version 1.12.1 to 1.12.0 in the staging environment. While this did not immediately restore full functionality, it was a key first step in addressing the issue. * **Configuration Investigation and Fix:** Further investigation revealed that the Ingress configurations were not properly loaded due to the security evaluation process and an additional contributing factor. Once identified, configuration changes were made to resolve the snippet evaluation issue. The controller configurations were purged and redeployed, correcting the “default” routing behavior and restoring service. * **Production Environment Fix:** This same resolution was applied to the production environment \([Rebuyengine.com](http://Rebuyengine.com)\), which was fully restored to normal operation. **Next Steps:** All necessary actions to address the vulnerability have been completed, and no further steps are required at this time. Moving forward, we’ve enhanced our testing process to ensure that if this error is encountered, we can take appropriate actions prior to any impact to the Rebuy services.

Read the full incident report →

Started Feb 12, 2025, 03:07 PM UTC· resolved Feb 12, 2025, 03:43 PM UTC

1. investigating Feb 12, 2025, 03:07 PM UTC

   In some cases, the post-purchase offer (PPO) doesn’t appear, and users are redirected straight to the Thank You page. In other instances, the PPO does render, but users can't add items due to an error, though they can still decline the offer(s) and proceed to the Thank You page as expected. This issue isn't occurring on all sites, but our team has identified some errors and is actively working on a fix.

2. identified Feb 12, 2025, 03:24 PM UTC

   The issue has been identified, and the team is currently implementing a fix.

3. monitoring Feb 12, 2025, 03:37 PM UTC

   The fix has been successfully implemented, and we're seeing positive results. The team is continuing to monitor the situation.

4. resolved Feb 12, 2025, 03:43 PM UTC

   The issue has been fully resolved, and the fix has been implemented successfully. We are now seeing the expected behavior. The team will continue to monitor the situation to ensure everything remains stable. Additionally, we will investigate the root cause to prevent similar issues in the future. Please reach out if any further issues arise.

5. postmortem Feb 14, 2025, 08:25 PM UTC

   We recently made an update to our infrastructure to improve efficiency and performance. Following the release, some merchants experienced issues with Widgets and Post Purchase Offers due to CORS-related errors. Our team quickly identified the cause and implemented the necessary changes to resolve these issues. To prevent similar occurrences in the future, we’ve made improvements to our testing processes to catch the specific errors that were seen during this maintenance. We appreciate your patience and understanding as we continue to optimize our systems. If you have any questions or need assistance, our support team is here to help.

Read the full incident report →

Started Feb 11, 2025, 09:25 PM UTC· resolved Feb 11, 2025, 09:57 PM UTC

1. investigating Feb 11, 2025, 09:25 PM UTC

   We're aware of an issue causing widgets to not load on the site and are actively working to resolve it. Thank you for your patience.

2. identified Feb 11, 2025, 09:37 PM UTC

   We've identified the issue and implemented a fix. Widgets should now begin loading properly.

3. monitoring Feb 11, 2025, 09:47 PM UTC

   The issue appears to be resolved after the implemented fix. We are actively monitoring the situation to ensure stability and will provide updates as needed.

4. resolved Feb 11, 2025, 09:57 PM UTC

   The issue has been fully resolved, and all systems are stable. Testing has confirmed functionality, and we will be conducting a post-mortem review to analyze the root cause and identify any improvements for the future. Thank you for your patience, and we appreciate your support. Please reach out if you experience any further issues.

5. postmortem Feb 14, 2025, 08:19 PM UTC

   We recently made an update to our infrastructure to improve efficiency and performance. Following the release, some merchants experienced issues with Widgets and Post Purchase Offers due to CORS-related errors. Our team quickly identified the cause and implemented the necessary changes to resolve these issues. To prevent similar occurrences in the future, we’ve made improvements to our testing processes to catch the specific errors that were seen during this maintenance. We appreciate your patience and understanding as we continue to optimize our systems. If you have any questions or need assistance, our support team is here to help.

Read the full incident report →

Started Jan 23, 2025, 06:44 PM UTC· resolved Jan 23, 2025, 07:08 PM UTC

1. investigating Jan 23, 2025, 06:44 PM UTC

   We are currently investigating an intermittent issue affecting the loading of Smart Search on-site and within admin settings for some users. Our team is working to identify the root cause and resolve this as quickly as possible. Thank you for your patience.

2. monitoring Jan 23, 2025, 06:57 PM UTC

   A fix has been implemented to address the intermittent issue with Smart Search loading on-site and in admin settings. We are seeing positive results, and continuing to monitor.

3. monitoring Jan 23, 2025, 07:02 PM UTC

   The fix implemented for the intermittent Smart Search loading issue continues to show positive results. We are monitoring the situation closely and can confirm that the issue remains resolved at this time.

4. resolved Jan 23, 2025, 07:08 PM UTC

   The issue with Smart Search loading has been fully resolved, and everything is functioning as expected. We will conduct a root cause analysis and take appropriate steps to prevent similar occurrences in the future. Thank you for your patience.

5. postmortem Jan 28, 2025, 06:46 PM UTC

   **What Happened** During a migration a configuration setting was unintentionally missed. This setting controlled rate limits for one of our key services, causing errors that disrupted functionality. ‌ **What We Did to Resolve It** After identifying the root cause, we implemented a temporary fix by adjusting the configuration to restore functionality. We also worked to port the missing configuration to V2 to prevent further disruptions. ‌ #### **Next Steps** We’ve now updated our processes to ensure all necessary configurations are accounted for in future migrations. This includes adding safeguards and testing mechanisms to catch similar issues before they impact functionality.

Read the full incident report →

Started Dec 20, 2024, 09:29 PM UTC· resolved Dec 20, 2024, 09:29 PM UTC

1. resolved Dec 20, 2024, 09:29 PM UTC

   When attempting to add an item to the cart, the cart remained empty, and the Smart Cart does not load upon refreshing the page. The issue has since been resolved and our teams are investigating root cause.

Read the full incident report →

Started Dec 18, 2024, 12:20 PM UTC· resolved Dec 18, 2024, 01:17 PM UTC

1. investigating Dec 18, 2024, 12:20 PM UTC

   We are currently receiving reports that Smart Cart v2 is experiencing intermittent loading issues for some customers. Legacy carts continue to function as expected, while v2 carts have temporarily reverted to the native cart. Our team is actively investigating the situation. Thank you for your patience.

2. monitoring Dec 18, 2024, 01:10 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Dec 18, 2024, 01:17 PM UTC

   The issue has been resolved, and our engineering team will conduct a root cause analysis. We will provide an update in this outage with the post-mortem once it is complete.

4. postmortem Dec 19, 2024, 09:07 PM UTC

   #### **What Happened** A high-demand process coincided with a scheduled daily job, leading to excessive resource usage in our database. This caused an overload, which resulted in an interruption of the **SmartCartV2 service**. ‌ **Initial Actions to Resolve the Outage** We adjusted the memory allocation for operations by increasing it from the default value. This change was applied dynamically without needing a full MySQL restart, minimizing disruption. ‌ #### **What We’re Doing Going Forward** 1. **Improved Resource Management:** * We’ve restructured how **SmartCartV2** data is processed to prevent resource conflicts between different tasks, such as the one that occurred in this instance. 2. **Update System Logic:** * Our applications will be updated to work seamlessly with this new data process, ensuring smooth and efficient performance with the new model.

Read the full incident report →

Started Nov 29, 2024, 04:19 PM UTC· resolved Nov 29, 2024, 07:24 PM UTC

1. identified Nov 29, 2024, 04:19 PM UTC

   We are aware of reports from some customers experiencing delays with Rebuy loading and are actively working on it.

2. identified Nov 29, 2024, 04:36 PM UTC

   We’re seeing latency begin to level out and performance improving. Our teams are continuing to monitor the situation closely and actively work on ensuring everything returns to optimal performance.

3. identified Nov 29, 2024, 05:10 PM UTC

   We are continuing to see improving latency across the board. However, we are currently running behind on reporting analytics and Klaviyo events. Our teams are actively addressing these delays and working to catch up as quickly as possible.

4. identified Nov 29, 2024, 06:04 PM UTC

   We are seeing continued stability across the board and are closely monitoring the situation to take proactive measures and prevent a recurrence of the latency experienced earlier today. We will continue to share updates as they become available.

5. monitoring Nov 29, 2024, 06:42 PM UTC

   The system remains stable, and we are seeing consistent performance. Our teams will continue to closely monitor the situation to address any potential issues promptly.

6. resolved Nov 29, 2024, 07:24 PM UTC

   We are now observing normal utilization levels with no unexpected spikes. The system is performing well, and we’re confident that things are stable. Our team will continue to monitor closely for any potential issues.

Read the full incident report →

Started Nov 13, 2024, 05:19 PM UTC· resolved Nov 13, 2024, 09:48 PM UTC

1. investigating Nov 13, 2024, 05:19 PM UTC

   We're experiencing an unexpected issue causing Rebuy to be temporarily unavailable for some stores. Our team is working to restore full functionality as quickly as possible. We apologize for the inconvenience and appreciate your patience. Please check back soon for updates, and thank you for being a valued Rebuy user.

2. identified Nov 13, 2024, 05:22 PM UTC

   The issue has been identified and a fix is being implemented.

3. identified Nov 13, 2024, 05:31 PM UTC

   We are continuing to work on a fix for this issue.

4. identified Nov 13, 2024, 05:41 PM UTC

   Issues are beginning to resolve for some customers, and the list of restored functionality continues to grow. We're actively working to complete the fix for all users. Thank you for your patience.

5. identified Nov 13, 2024, 05:58 PM UTC

   We're seeing intermittent success with Rebuy loading, and our team is actively working on a full resolution. Thank you for your patience.

6. identified Nov 13, 2024, 06:15 PM UTC

   We're implementing additional measures to expedite the resolution for the impacted sites.

7. identified Nov 13, 2024, 07:13 PM UTC

   The additional changes have been released, and we are restarting services to ensure they take effect.

8. identified Nov 13, 2024, 07:56 PM UTC

   Restarts and remediation efforts are ongoing as we work toward full resolution. Thank you for your patience.

9. monitoring Nov 13, 2024, 08:27 PM UTC

   A fix has been implemented, and we’re seeing resolution across most features. Our team is still working on Smart Cart and continues to test and monitor to ensure stability for all other features. Thank you for your patience!

10. monitoring Nov 13, 2024, 09:39 PM UTC

    A fix has been published for Smart Cart and the issue appears resolved. We are continuing to monitor.

11. resolved Nov 13, 2024, 09:48 PM UTC

    The issue has been resolved. We'll be conducting a thorough review and will share a post-mortem with insights on what occurred and the steps we're taking to prevent it in the future. Thank you for your patience!

12. postmortem Nov 15, 2024, 01:39 PM UTC

    **Summary:** An unexpected issue caused a cache-clearing function to trigger unintentionally, leading to an overload that ultimately locked a database. Customers experienced intermittent issues, though some were unaffected, others saw various symptoms. To address this, we disabled key functions suspected to be the cause and implemented a hotfix to prevent further database locks. Once these measures were implemented and services were restarted, the issue was fully resolved. ‌ **Next Steps:** As a preventative measure, we will conduct a thorough code review of the cache-clearing function to identify and address any vulnerabilities that could lead to unintended triggers.

Read the full incident report →

Started Oct 29, 2024, 05:33 PM UTC· resolved Oct 29, 2024, 06:29 PM UTC

1. investigating Oct 29, 2024, 05:33 PM UTC

   We're currently experiencing technical difficulties. Our team is investigating the issue and will provide updates soon.

2. identified Oct 29, 2024, 05:35 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Oct 29, 2024, 06:15 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Oct 29, 2024, 06:29 PM UTC

   This incident has been resolved.

5. postmortem Oct 30, 2024, 01:57 PM UTC

   **Post-Mortem: Service Disruption** We recently experienced an issue that temporarily impacted one of our core services, while other systems continued running normally. This occurred during a routine system update, where one service didn’t adapt to new settings as expected. We’re actively reviewing our processes and related integrations to understand the cause and ensure smoother transitions in the future.

Read the full incident report →