Rivo Outage History

Started Sep 19, 2025, 04:09 PM UTC· resolved Sep 19, 2025, 11:10 PM UTC

1. investigating Sep 19, 2025, 04:09 PM UTC

   On the morning of September 19, 2025, Shopify’s automatic migration to their new developer platform introduced a bug that automatically deleted checkout extensions with names exceeding a character limit. This deletion occurred without notice or Rivo intervention. This was not triggered by any deployment or changes from Rivo’s side. We’re investigating further and resolving..

2. identified Sep 19, 2025, 05:32 PM UTC

   We’ve confirmed that a Shopify platform bug during their developer migration caused several checkout extensions to be automatically deleted. Impacted extensions include reward redemption, potential points earned, and Rewards automations by VIP Tier. Our team is actively re-enabling these extensions for affected merchants.

3. monitoring Sep 19, 2025, 08:33 PM UTC

   Rivo checkout extensions that we're removed by Shopify are being re-enabled across affected merchants. Your Rivo loyalty program has remained fully functional throughout, customers have continued earning points, redeeming rewards, and engaging as normal. The only impact was to specific checkout redemption methods (Reward Redemption, Potential Points Earned display, and VIP Tier automations), which may have been removed and require re-adding. We're monitoring closely to ensure all affected accounts are restored and stable. Shopify is also addressing the root cause on their side to prevent recurrence.

4. resolved Sep 19, 2025, 11:10 PM UTC

   The checkout extensions that were removed by Shopify’s platform migration bug have now been fully restored. Impacted extensions included Reward Redemption, Potential Points Earned display, and VIP Tier reward automations. The Rivo platform fully functional throughout this incident, customers continued earning points, redeeming rewards, and engaging as normal. The only impact was to some checkout redemption methods, which Shopify has now restored. Thank you for your patience while we worked through this.

5. postmortem Sep 19, 2025, 11:11 PM UTC

   ### **What happened?** On September 19, 2025, [Shopify’s migration to their new developer platform](https://shopify.dev/changelog/next-gen-dev-platform-ga) introduced a bug that deleted Rivo checkout extensions with names exceeding a character limit. This was not caused by any Rivo deployment, and the Rivo loyalty platform itself remained fully functional during this incident. Customers continued earning and redeeming rewards as normal. The only impact was to checkout redemption methods, including Reward Redemption, Potential Points Earned display, and VIP Tier automations, which were temporarily removed from checkout until restored. The incident was detected early in the morning and declared a P0 incident internally. Customer Success and Engineering teams worked quickly to re-enable extensions for merchants. By early afternoon, communications were sent out to merchants and functionality had been fully restored and stability confirmed. ### **What’s Next?** To prevent recurrence we’ve implemented internal tracking of 24-hour usage for every checkout extension. This monitoring allows us to instantly identify which merchants rely on specific extensions and respond faster if a similar Shopify issue occurs. We’re also adding automated alerts for sudden drops in extension usage and engaging Shopify for improved communication.

Read the full incident report →

Started May 19, 2025, 09:30 AM UTC· resolved May 19, 2025, 09:51 PM UTC

1. investigating May 19, 2025, 09:49 PM UTC

   We're observing elevated login errors affecting users on multiple shops using Rivo Accounts. This issue was traced to an interruption with our third-party email provider, Mailgun. Users attempting to log in during this window may have seen an “An error occurred during login” message.

2. resolved May 19, 2025, 09:51 PM UTC

   This incident has been resolved.

3. postmortem May 19, 2025, 10:06 PM UTC

   ### **What happened?** Between 2:25 PM and 2:50 PM PDT, some Rivo Accounts customers may have experienced errors while signing in. The error stemmed from Mailgun, our third-party transactional email provider. Mailgun experienced degraded performance tied to a datastore issue, which interrupted the communication flow for login email verification. This resulted in user-facing errors like: > “An error occurred during login. Please try again.” [_An overview of that incident from Mailgun can be seen here._](https://status.mailgun.com/incidents/tqpy8vllgtys) Rivo’s core account services remained operational, but specifically email login actions dependent on Mailgun failed intermittently. We rolled out backup authentication methods last week that will be available if a customer has troubles signing in. ### **What’s Next?** We’ve added backup services like Shop Pay login for better resiliency when a service disruption occurs.

Read the full incident report →

Started Feb 14, 2025, 10:16 PM UTC· resolved Feb 14, 2025, 10:39 PM UTC

1. investigating Feb 14, 2025, 10:16 PM UTC

   We're currently investigating an issue with delay in sending authentication code emails from a third party email provider. We're sorry for the disruption.

2. identified Feb 14, 2025, 10:24 PM UTC

   We've identified there was a delay in in authentication codes from our external email provider. There was approximately a 15 minute delay in sending codes between 1:57pm - 2:10pm PST. Email sending has resumed after a minor disruption.

3. monitoring Feb 14, 2025, 10:29 PM UTC

   We've switched to a fallback service, monitoring sending services for stability.

4. resolved Feb 14, 2025, 10:39 PM UTC

   We resolved authentication code sending and emails should be smoothly sending for everyone once again. Thanks for your patience!

5. postmortem Feb 15, 2025, 09:46 PM UTC

   ### **What happened?** On **February 14, 2025 at approximately 1:30pm PST**, a Shopify update introduced an unexpected issue in `shops_event_listener.js`, which affected how `CustomEvent` was handled. This impacted composed events, causing disruptions for some Shadow DOM-based themes, leading to login errors for some customers. As a result, a small number of users may have seen the message: _"An error occurred during login, please try again."_ This issue affected a small percentage of authentication emails, causing them to be sent incorrectly as `0`. This Shopify update would have also introduced unexpected issues that may have affected the way that thousands of themes that use shadow DOM with composed events functioned. To ensure stability, we deployed a hotfix as soon as the issue was identified on **February 14th at 11:55pm** as Shopify worked on their resolution, which was [deployed](https://x.com/madmath/status/1890775821184319962) on the morning of **February 15th 2025**. ### **What’s Next?** Now that everything is back to normal, we’ve taken extra steps to improve proactive monitoring and system resilience to catch similar issues from external providers faster in the future. Thanks for your patience—and we’re sorry for any disruption here.

Read the full incident report →

Started Dec 27, 2024, 10:00 PM UTC· resolved Dec 27, 2024, 10:00 PM UTC

1. resolved Dec 28, 2024, 02:42 PM UTC

   From 2PM - 7PM on 27th Dec 2024 our 3rd party email validation service was experiencing intermittent issues. At 7:22PM we switched to an alternative backup provider to resolve the issue. To prevent this from happening in the future, we've implemented automatic failover if any external service around email login should fail in the future.

2. postmortem Dec 28, 2024, 02:50 PM UTC

   To prevent this from happening in the future, we've implemented automatic failover if any external service around email login should fail in the future.

Read the full incident report →