PowerOutage.us experienced a major incident on September 26, 2022 affecting Power Outage Site and Power Outage API, lasting 6h 38m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Sep 26, 2022, 11:08 PM UTC
We are continuing to see increased web request error rates due to extremely high abnormal traffic which is circumventing our page caching. We are working on implementing an emergency mitigation.
- monitoring Sep 27, 2022, 12:16 AM UTC
We implemented a few additional abnormal traffic protections, and added an additional web cluster of servers to deal with the excessive abnormal load. We have seen two more abnormal spikes in traffic that have not resulted in additional 50x errors. Currently we are waiting for full DNS propagation and monitoring server health. In total there have been about 11 minutes where portions of incoming requests were being dropped by overloaded servers.
- monitoring Sep 27, 2022, 01:07 AM UTC
A few API requests from customers were getting blocked by the new traffic filtering rules. We have made some modifications to prevent authorized clients with valid API keys from get caught in the filters. We continue to monitor server health as the DNS changes propagate.
- resolved Sep 27, 2022, 05:47 AM UTC
DNS changes have been successfully propagated to core DNS servers. No additional issues have been found. Issue is being closed, but we will continue to monitor server health in detail. Feel free to reach out to [email protected] or join our discord discord.gg/PRrqwe7dpD if you have any questions!