Subscribe Pro incident

Slow Platform Response / Partial Outage

Critical Resolved View vendor source →

Subscribe Pro experienced a critical incident on July 30, 2018 affecting API | api.subscribepro.com and Management Portal | platform.subscribepro.com, lasting 1h 56m. The incident has been resolved; the full update timeline is below.

Started
Jul 30, 2018, 07:38 PM UTC
Resolved
Jul 30, 2018, 09:35 PM UTC
Duration
1h 56m
Detected by Pingoru
Jul 30, 2018, 07:38 PM UTC

Affected components

API | api.subscribepro.comManagement Portal | platform.subscribepro.com

Update timeline

  1. investigating Jul 30, 2018, 07:38 PM UTC

    We are currently investigating a partial outage of the Subscribe Pro Web UI and API.

  2. monitoring Jul 30, 2018, 08:01 PM UTC

    We believe the issue is resolved. We will continue monitoring and provide a post-mortem once we fully understand the issue.

  3. resolved Jul 30, 2018, 09:35 PM UTC

    This issue causing the partial outage of our Web UI and API has been resolved. The issue was related to our tabular reporting, which has been temporarily disabled until the root issue is addressed.

  4. postmortem Aug 01, 2018, 09:14 PM UTC

    We investigated the root cause on Monday’s partial outage. In short some long running reporting queries were able to overwhelm database connections and make the platform unresponsive. Today we have implemented a system to stop long running queries and prevent them from impacting overall system performance. With this new system, an outage similar to Monday’s will no longer be possible.