Subscribe Pro experienced a notice incident on March 2, 2018, lasting 2h 2m. The incident has been resolved; the full update timeline is below.
Update timeline
- investigating Mar 02, 2018, 02:06 PM UTC
We are experiencing a platform outage and are actively working to restore service.
- resolved Mar 02, 2018, 04:08 PM UTC
We have identified and resolved the root cause of the service outage. Please stay tuned for a full report of what went wrong.
- postmortem Jul 31, 2018, 05:06 PM UTC
Unfortunately this morning Subscribe Pro had an extended period of intermittent downtime for our subscriptions platform. We do sincerely apologize to all our clients for this for this outage. Historically, since our launch 4 years ago, we have maintained extraordinarily high availability. Although today tarnished that record a bit, we don't intend to let anything similar happen in the foreseeable future. What Went Wrong ---- 1. We deployed updates to our software this morning around 8:05am eastern time. These updates included some code changes and an additional index in our database. 2. Within 10 minutes after the deployment, we started noticing performance issues with our database server. 3. We began investigating the issue immediately. Initially we traced the performance issues to the creation of a the new database index. 4. We instituted emergency procedures to add more resources to our production environment and continued investigating the root cause. 5. Eventually we were able to determine the root cause was related to a code change in the software. And to determine that the database index change was a red herring. Moving Forward --- We understand how important 24x7x365 availability is to an eCommerce business and we do put in a lot of effort to make sure your sites achieve that. Today we were not successful in that, but we are using the experience to learn what we can do better in the future.