Badgr experienced a critical incident on August 31, 2021 affecting Badgr Pro (badgr.com) and Badgr API (api.badgr.io), lasting 4h 26m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 31, 2021, 06:45 PM UTC
The infrastructure provider is experiencing network issues that are affecting the site.
- investigating Aug 31, 2021, 07:42 PM UTC
This is being caused by an issue with AWS, you can follow their status page at https://status.aws.amazon.com/. We are looking for a workaround.
- investigating Aug 31, 2021, 08:58 PM UTC
AWS has localized the issue to packet loss within an availability zone in the us-west-2 region. We are pursuing a workaround, and we will have an update as soon as more information is available.
- investigating Aug 31, 2021, 10:51 PM UTC
We have isolated the issue and are working around it.
- investigating Aug 31, 2021, 10:56 PM UTC
We have successfully worked around the AWS infrastructure issue and are back online.
- monitoring Aug 31, 2021, 10:57 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Aug 31, 2021, 11:11 PM UTC
This incident has been resolved.
- postmortem Aug 31, 2021, 11:12 PM UTC
This outage was due to an AWS Availability Zone \(usw2-az2\) experiencing packet loss. This caused intermittent issues communicating with critical services required for Badgr to function. A work around was put in place after AWS identified the source of the issue.