Ghost Inspector incident
AWS sa-east-1 (São Paulo, Brazil) Geolocation Offline
Ghost Inspector experienced a minor incident on October 10, 2020 affecting Test Runner, lasting 1h 2m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Oct 10, 2020, 09:53 PM UTC
It appears that the AWS sa-east-1 data center which hosts our São Paulo, Brazil geolocation is currently down. We have confirmed that other AWS customers are experiencing the outage. We will be monitoring the status of the region and will post updates.
- monitoring Oct 10, 2020, 10:07 PM UTC
The AWS sa-east-1 (São Paulo, Brazil) region appears to be back online and test traffic is flowing properly. We will continue to monitor the issue.
- resolved Oct 10, 2020, 10:55 PM UTC
The AWS sa-east-1 data center appears to be stable again. We are closing this incident.
- postmortem Aug 17, 2021, 03:33 PM UTC
The root cause of this incident was an outage at the AWS level. Because the outage was region-wide, no remediation was possible on our end. We needed to wait for AWS to bring the region back online. Because this only affected traffic for tests running through the São Paulo, Brazil geolocation, the impact was minimal. After this incident, we considered the idea of rolling traffic over to a different geolocation in this situation, but determined that it is best to let the tests fail and make it clear that the geolocation is having issues. We believe that’s a more useful course of action than redirecting traffic in an unexpected way.