Subscribe Pro experienced a major incident on July 19, 2019 affecting API | api.subscribepro.com and Management Portal | platform.subscribepro.com, lasting 2h 34m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 19, 2019, 10:50 AM UTC
We are currently investigating an outage of our platform UI and API.
- monitoring Jul 19, 2019, 11:20 AM UTC
A fix has been implemented and we are monitoring.
- resolved Jul 19, 2019, 01:24 PM UTC
This outage has been resolved. Please contact [email protected] if you experience any further issues accessing our platform web UI or API.
- postmortem Jul 19, 2019, 06:22 PM UTC
We had an outage this morning that affected all Subscribe Pro customers. The root cause was a disk that filled up on our transactional database system. There were two other systems failures which caused this issue to lead to over an hour of downtime: * Our designated on-call personnel failed to respond as promptly as required. * We were operating with a mis-understanding of the auto-scaling settings around the disk that filled up. We have taken action to address both of these system issues: * We have fine tuned our procedures for on-call personnel and increased our training frequency. * We have adjusted and tested the relevant auto-scaling settings for database disk. We take any down time very seriously and have used this as opportunity to improve our systems and processes so that this type of issue can’t happen again. Thank you.