BrightData incident
70% sporadic degradation due to servers overload
BrightData experienced a minor incident on September 22, 2021 affecting Bright Data Network, lasting 2h 18m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Sep 22, 2021, 05:24 PM UTC
[RESOLVED] Service disruption in the Residential, Mobile, Datacenter and ISP networks On 22-Sep, from 17:24 to 19:34 UTC, the Residential, Mobile, Datacenter and ISP networks experienced intermittent service disruptions due to an overloading of internal DB servers. All issues were fixed and resolved. Timeline 22-Sep-2021 (UTC): 16:19:53 - Internal DB server received high spike of operations, causing it to respond slowly 17:24:26 - Secondary DB server takes over but crashed due to high load during switch-over while no space left on its device, causing an outage in the Residential and Mobile networks 18:20:31 - Relevant residential network DB servers were upgraded, to handle all incoming data, Residential and Mobile networks starts to recover 18:39:00 - Different DB server crashed due to high load, causing an outage in the Datacenter and ISP networks 19:20:00 - Datacenter and ISP networks DB servers were upgraded, Datacenter and ISP networks start to recover 19:24:00 - R&D perform further optimizations 19:34:39 - Datacenter and ISP networks reach usual BW rate. The issue has been resolved and the service is operating normally. Beside the immediate actions which were taken, we also enhanced the infrastructure to handle such high spikes in the future.