PowerOutage.us incident
Investigating increased web request error rates
PowerOutage.us experienced a minor incident on September 26, 2022 affecting Power Outage Site and Power Outage API, lasting 50m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 26, 2022, 09:25 PM UTC
We are investigating increased web request error rates.
- resolved Sep 26, 2022, 10:16 PM UTC
We experienced a brief increase in web request errors for about two minutes. This was caused by a sudden increase in traffic which was seemingly designed to circumvent our first line of page caching. In total our servers served back 165k 503 errors, and successfully handled 302k requests, all in the two-minute span. Before this surge we were serving about 61k web requests an hour, about 1k a minute. Our server auto-scaling for handling increase loads cannot respond that fast. In this case there is not much we can do in the short term to prevent this type of abuse, how they circumvented the cache forcing our web servers to have to build pages for each request can't be fixed quickly, but it is now on our radar for future development. In the meantime, we have left our web cluster at a higher than normal scaling to help handle other similar request storms if that occur.