Edit order Outage History

Started Oct 22, 2026, 06:44 PM UTC· resolved Oct 22, 2026, 06:44 PM UTC

1. resolved Oct 22, 2026, 06:44 PM UTC

   **Shopify checkout attack**. Since October 10, several third-party Shopify apps have had malicious code added to their storefront settings. The code redirected shoppers to fake checkout pages that captured their payment details. **Cleverific was not affected**. Even so, we took every precaution our security and engineering policies require. - Cleverific doesn’t load code onto your storefront through theme app extensions or app embeds. - Our app has no setting that runs custom code on your storefront. - We audited our systems and access controls. All internal access requires hardware security keys and two-factor authentication. *PHP vulnerabilities (CVE-2026-109618, CVE-2026-109619)*. Cleverific doesn’t run PHP, so these vulnerabilities don’t affect our systems. We also audited our vendors to confirm that each one either doesn’t use PHP or has taken appropriate steps to mitigate these vulnerabilities. *Compromised certificate authority intermediates (eMudhra/emSign)*. On October 17, eight intermediate certificates from the certificate authority eMudhra were found to have weak private keys. Browsers and operating systems are revoking trust in them. Cleverific doesn’t use certificates from the affected certificate authority. As a precaution, we updated our base images and language runtimes so our systems no longer trust the affected certificates. *What we recommend for your store*: 1. In your theme editor, open App embeds and review every embed that’s switched on. 2. Check any app that has a “custom JavaScript” or “custom HTML” setting, and make sure you recognize everything in it. 3. Remove apps you don’t use. 4. Test checkout in a fresh private window and confirm the address stays on your store’s domain. If you have questions about this incident or our security practices, contact us at [email protected]

Read the full incident report →

Started Nov 28, 2025, 08:20 PM UTC· resolved Nov 28, 2025, 10:20 PM UTC

1. resolved Nov 28, 2025, 08:30 PM UTC

   On November 28, 2025, we experienced a service disruption affecting merchants using the Liquid email link snippet in their order notification emails. End customers clicking these "Edit Order" email links received 404 errors. The issue was introduced during a customization modification on or about November 21, 2025, reported on November 28 at 3:20 PM EST, and resolved by 5:20 PM EST (approximately 2 hours from report to resolution).

2. postmortem Dec 05, 2025, 04:46 PM UTC

   ## What Happened On November 28, 2025, we experienced a service disruption affecting merchants using the Liquid email link snippet in their order notification emails. End customers clicking these "Edit Order" email links received 404 errors. The issue was introduced during a customization modification on or about November 21, 2025, reported on November 28 at 3:20 PM EST, and resolved by 5:20 PM EST \(approximately 2 hours from report to resolution\). ### Timeline * **~November 21, 2025**: Issue introduced during customization modification * **November 28, 2025 3:20 PM EST \(8:20 PM UTC\)**: Issue reported - email links returning 404 errors * **November 28, 2025 5:20 PM EST \(10:20 PM UTC\)**: Fix deployed and service restored * **Detection to Resolution**: ~2 hours * **Issue Present**: ~7 days \(undetected\) ### Impact **Who Was Affected:** * Merchants using the Liquid email link snippet in their order notification emails * End customers clicking "Edit Order" email links received 404 errors * **Not Affected**: Customers using the "View Order" status page link continued to work normally * **Not Affected**: Merchants not using the Liquid email link snippet * **Not Affected**: Enterprise customers on dedicated infrastructure **What Was Affected:** * Email-based customer portal access via Liquid snippet only * Alternative access methods remained fully operational throughout the incident --- ## Root Cause During implementation of a customization for a specific merchant, an internal routing parameter was inadvertently changed without updating the corresponding authentication logic. This created a mismatch between the URL structure in email links and what our authentication system expected, affecting all merchants using the Liquid email link snippet. **Technical Details:** Our email links embed the order ID in the URL path \(e.g., `/orders/12345/edit/d`\). The authentication code was updated to look for a different parameter name, causing it to not find the order ID and reject valid authentication attempts. --- ## Resolution We deployed a backward-compatible fix that accepts both the old and new parameter names, ensuring email links work regardless of which format they use. This approach: 1. **Immediately restored service** for all affected customers 2. **Maintains compatibility** with any emails already sent 3. **Prevents similar issues** if parameter names change in the future --- ## What We're Doing to Prevent This We've taken the following actions to prevent similar incidents: ### Immediate Actions \(Completed\) * ✅ **Added comprehensive end-to-end testing** for the email link authentication flow * ✅ **Implemented backward compatibility** to handle both parameter formats * ✅ **Enhanced monitoring** to detect authentication failures more quickly ### Short-term Improvements \(In Progress\) * **Expanded test coverage** for all customer portal access methods * **Improved change review process** to identify impacts on shared authentication code * **Enhanced error monitoring** to alert our team of unusual 404 patterns ### Long-term Improvements \(Planned\) * **Standardized routing conventions** across the customer portal * **Additional automated testing** for critical user authentication flows --- ## Our Commitment This incident highlighted a gap in our testing coverage for a critical customer-facing feature. While the impact was limited and quickly resolved, we take any service disruption seriously. We're committed to: * **Transparency**: Sharing what went wrong and how we fixed it * **Continuous improvement**: Learning from incidents to strengthen our systems * **Reliability**: Investing in testing and monitoring to prevent future issues If you experienced issues during this incident or have questions about this report, please contact our support team. --- ## Technical Appendix _For customers interested in the technical details:_ ### Authentication Flow The customer portal email link authentication uses HMAC-SHA256 signatures to verify that links are legitimate and haven't been tampered with. The signature is computed over the customer ID and order ID, ensuring each link is unique and secure. ### What Changed * **Before**: Authentication code read order ID from the `id` URL parameter * **Incident**: Code was updated to read from `order_id` parameter, but URL structure didn't change * **Fix**: Code now accepts both `id` and `order_id` parameters for maximum compatibility ### Detection Delay The issue went undetected for approximately 7 days between deployment \(~November 21\) and the first customer report \(November 28\). This delay highlights the importance of: * **Proactive monitoring** - We have since enhanced error monitoring to detect similar authentication failures * **Usage analytics** - The affected feature \(Liquid email link snippet\) is used by a subset of merchants, which contributed to the delayed detection * **E2E testing** - Post-incident, we added comprehensive end-to-end tests for this authentication flow ### Response Time Once reported, the issue was resolved within 2 hours. Our team opted for a forward fix \(backward compatibility\) rather than a rollback, as it provided better long-term resilience and ensured any emails already sent would continue to work. ### Alternative Access Methods The customer portal can be accessed through multiple methods: 1. **Liquid email link snippet** - Direct links with embedded authentication in email templates \(affected by this incident\) 2. **Status page "View Order" links** - Links from order status pages \(unaffected\) 3. **Custom email integrations** - Merchants using custom email implementations \(unaffected\) This redundancy helped minimize impact during the incident, as only merchants using the specific Liquid email link snippet were affected.

Read the full incident report →

Started Oct 20, 2025, 04:17 PM UTC· resolved Oct 20, 2025, 06:28 PM UTC

1. identified Oct 20, 2025, 04:17 PM UTC

   The third-party vendor has acknowledged the recurrence of the issue and has a fix ready to be deployed. They are also adding additional capacity on their end to handle a high inflow of requests. This update should go out in about by 9:20 am PDT

2. identified Oct 20, 2025, 04:31 PM UTC

   We are continuing to monitor a resolution for this issue

3. monitoring Oct 20, 2025, 05:18 PM UTC

   We’ve regained access to the Heroku dashboard, although performance remains unstable. Metrics show that requests are starting to come back into the system. We’re continuing to monitor closely and will share another update as soon as we get confirmation for full resolution.

4. resolved Oct 20, 2025, 06:28 PM UTC

   Our upstream service provider has restored operations, and we’ve confirmed that our application services are back in operations. We’ll continue to monitor closely to ensure everything remains stable and available.

Read the full incident report →

Started Aug 21, 2025, 06:54 PM UTC· resolved Aug 21, 2025, 08:30 PM UTC

1. monitoring Aug 21, 2025, 06:54 PM UTC

   We've noticed a widespread outage in the Shopify GraphQL API that is causing Cleverific services to only be intermittently available. This issue is affecting apps across the Shopify ecosystem. We are monitoring our comms with Shopify for resolution.

2. monitoring Aug 21, 2025, 07:39 PM UTC

   Shopify has acknowledged the issue and is investigating.

3. monitoring Aug 21, 2025, 08:13 PM UTC

   It has been reported that this may be due to an incident with an upstream infrastructure provider which has potentially been resolved. We are seeing server errors settle to zero and have not seen the errors that alerted us to this issue in the last 15 minutes. We are downgrading the incident from partial outage to degraded performance as we monitor our systems for the next 15 minutes

4. monitoring Aug 21, 2025, 08:29 PM UTC

   No new errors have been logged or reported. We are validating that this issue has been resolved, and all systems are now operational.

5. resolved Aug 21, 2025, 08:30 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Aug 06, 2025, 06:12 PM UTC· resolved Aug 06, 2025, 06:34 PM UTC

1. identified Aug 06, 2025, 06:12 PM UTC

   Shopify is currently dropping necessary information from the "More Actions" menu link that Cleverific needs to load merchants' orders. We're currently in contact with Shopify.

2. resolved Aug 06, 2025, 06:34 PM UTC

   Shopify has implemented a solution. Orders and draft orders should now load as expected in Cleverific.

Read the full incident report →

Started Jun 10, 2025, 03:22 PM UTC· resolved Jun 10, 2025, 10:16 PM UTC

1. investigating Jun 10, 2025, 03:22 PM UTC

   Our service provider is experiencing a major outage, and we are waiting for their engineers to respond.

2. investigating Jun 10, 2025, 03:30 PM UTC

   We are continuing to investigate this issue.

3. investigating Jun 10, 2025, 03:50 PM UTC

   Engineers at our platform service provider, Heroku, are continuing to investigate the service outage issue. We have seen some evidence that services are starting to recover but are still subject to major outages. https://status.heroku.com/incidents/2822

4. investigating Jun 10, 2025, 04:21 PM UTC

   Update from Heroku: "We suspect a recent configuration change as the likely trigger and are working to revert it. In parallel, we’re developing an expedited rollout strategy and safeguards to prevent the recurrence of this config change."

5. identified Jun 10, 2025, 04:48 PM UTC

   Progress from Heroku: Investigations are still ongoing. A permanent fix is in progress, and it will require disabling the configuration change that reintroduces impact.

6. identified Jun 10, 2025, 05:39 PM UTC

   Update from Heroku: "We continue to work on the permanent fix and explore alternate paths to resolution. Once the fix is developed, an estimated time to resolution will be shared."

7. identified Jun 10, 2025, 06:31 PM UTC

   Progress from Heroku: "We’ve identified the likely trigger of the issue and are making strong progress toward a resolution. A recent automated update disrupted network connectivity on affected instances. We're implementing a fix to prevent this from recurring, and a permanent solution is currently in development."

8. identified Jun 10, 2025, 07:50 PM UTC

   Promising update from our platform service provider: "We are observing positive results in our initial testing on pre-production environments. We’re continuing to test the fix before applying it to production environments and will provide a resolution timeline once determined."

9. monitoring Jun 10, 2025, 09:11 PM UTC

   We are seeing incremental improvements in platform services coming back online. However, Cleverific's core services are still impacted by the platform outage. We will continue to monitor as services come back online.

10. resolved Jun 10, 2025, 10:16 PM UTC

    Our platform provider has resolved the issue. Cleverific's core services are all operational. We will continue to monitor the platform for any issues and will follow up with our colleagues at our platform provider, Salesforce/Heroku, for a detailed post mortem.

Read the full incident report →