Scrapfly incident
Cluster upgrade required to fix our browser network issue
Scrapfly experienced a notice incident on August 13, 2022 affecting Website and Web Scraping API, lasting 2d 1h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Aug 13, 2022, 08:53 AM UTC
The browser pool is still starving due to the network IP allocation hanging. We are affected by this issue https://github.com/cloudnativelabs/kube-router/issues/383. In order to fix correctly the issue we will proceed to a cluster upgrade
- identified Aug 13, 2022, 09:37 AM UTC
All critical node part has been upgraded - We are rolling out workers nodes (~150 - expect 15-30min before full upgrade) and throttled incoming traffic to 50k/rpm
- identified Aug 13, 2022, 09:48 AM UTC
Browsers pools are back online and addressable - rolling upgrade still pending
- monitoring Aug 13, 2022, 09:55 AM UTC
A fix has been implemented and we are monitoring the results.
- monitoring Aug 13, 2022, 10:11 AM UTC
Migration is now completed. We will keep the monitoring window until Monday. The service is currently back to normal and traffic throttled has been removed.
- resolved Aug 15, 2022, 09:58 AM UTC
This incident has been resolved.