Subscribe Pro incident

Partial Outage of the Merchant UI and API

Major Resolved View vendor source →

Subscribe Pro experienced a major incident on September 24, 2019 affecting API | api.subscribepro.com and Management Portal | platform.subscribepro.com, lasting 1h 18m. The incident has been resolved; the full update timeline is below.

Started
Sep 24, 2019, 03:51 PM UTC
Resolved
Sep 24, 2019, 05:10 PM UTC
Duration
1h 18m
Detected by Pingoru
Sep 24, 2019, 03:51 PM UTC

Affected components

API | api.subscribepro.comManagement Portal | platform.subscribepro.com

Update timeline

  1. investigating Sep 24, 2019, 03:51 PM UTC

    We are investigating a partial outage of our Merchant UI and API

  2. monitoring Sep 24, 2019, 04:00 PM UTC

    A fix is in place and we are monitoring.

  3. resolved Sep 24, 2019, 05:10 PM UTC

    This incident has been resolved.

  4. postmortem Sep 24, 2019, 06:12 PM UTC

    Today our API and web UI were down for close to 15 minutes. The root cause was a long running database update which was necessary as part of an upgrade. When we estimated the length of time the update would need we were off by 1 order of a magnitude. Consequently the update took close to 15 minutes instead of a little over a minute and translated into downtime for the API and UI. These sort of database updates are something we do rarely. Never-the-less, we have identified methods we can use to do more extensive testing in the future, for similar updates. This would allow us to plan alternate, incremental ways to update the database and schedule those at off times.