Partial degradation of service with merchants on the Shopify platform

Update timeline

1. monitoring Mar 07, 2025, 04:11 PM UTC

   We have identified an issue related to internal systems that process Shopify events from 03/06/2025 4PM EST and until 03/07/2025 9AM EST. The underlying issue has been resolved and we are actively investigating all impacted areas that require manual intervention. No action needs to be taken, our team will make sure all events will be reprocessed. Areas impacted: - Increase in recurring order placement retry attempts due to connection errors - Delayed delivery of events and updates tied to certain customer actions such as change of frequency or payment updates Next Steps: - We are working diligently to ensure that all delayed events are being processed - We are reviewing all potentially impacted recurring orders to ensure proper placement We will provide further updates throughout the day

2. monitoring Mar 07, 2025, 10:53 PM UTC

   All of the services have recovered and message delivery is back to normal. We are in the process of replaying any missed transactions. No action needs to be taken, our team will make sure all events will be reprocessed. We will provide further updates as progress is made.

3. monitoring Mar 08, 2025, 10:42 AM UTC

   All valid Shopify events have been replayed successfully. We are now entering the next and final phase of the recovery effort. There is a small number of recurring orders that were impacted by this issue which will be retried over the next 24-48 hour period. We are taking extra precautionary measures to ensure that we only retry orders which were not properly processed by Shopify. No action needs to be taken and we will provide a final update once the task is completed by our team.

4. resolved Mar 09, 2025, 02:12 AM UTC

   We have completed the main recovery efforts related to this incident. The small amount of previously affected recurring orders are starting to flow through the placement process.