Maitai experienced a major incident on August 27, 2024 affecting API, lasting 18m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Aug 27, 2024, 07:20 AM UTC
The issue has been identified and a fix is being implemented.
- resolved Aug 27, 2024, 07:39 AM UTC
This incident has been resolved.
- postmortem Aug 27, 2024, 07:39 AM UTC
At 11:40PM PDT, a handful of requests failed, and reporting DNS resolution failures. Starting at 12:00AM PDT, alerts started coming in about DNS resolution failure from our external monitoring service, specifically in our `aws-us-west-2` region. Upon further investigation, this issue was sourced to a configuration change that had been been made earlier in the night. Initially thought to be a major issue, the impact appears to have been minimal. Further investigation of our traffic numbers confirms a limited impact of this issue, with no discernible decrease in traffic during the night. Please let us know if you did see any issues, and we apologize for any disruptions that we may have caused.