Payment experienced a critical incident on August 31, 2019 affecting Proxy API, lasting 4h 28m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 31, 2019, 01:53 PM UTC
Heroku is experiencing widespread platform issues that is disallowing our proxy api to restart.
- investigating Aug 31, 2019, 05:52 PM UTC
We want to give you more information on progress at this point, and what we know about the event. At 4:33 AM PDT one of 10 datacenters in one of the 6 Availability Zones in the US-EAST-1 Region saw a failure of utility power. Backup generators came online immediately, but for reasons we are still investigating, began quickly failing at around 6:00 AM PDT. This resulted in 7.5% of all instances in that Availability Zone failing by 6:10 AM PDT. Over the last few hours we have recovered most instances but still have 1.5% of the instances in that Availability Zone remaining to be recovered. Similar impact existed to EBS and we continue to recover volumes within EBS. New instance launches in this zone continue to work without issue.
- resolved Aug 31, 2019, 06:21 PM UTC
This incident has been resolved.