Gini experienced a critical incident on March 27, 2020 affecting User-Center API, lasting 1d. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 27, 2020, 03:11 PM UTC
API availability is impacted. We are in contact with our datacenter provider to understand what is happening. The problem seems to be BGP related.
- monitoring Mar 27, 2020, 05:56 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Mar 28, 2020, 03:21 PM UTC
This incident has been resolved.
- postmortem Mar 30, 2020, 04:13 PM UTC
Starting Friday around 15:00 we received alerts about reduced API availability and around 15:30 our entire external network was down. Our investigations quickly revealed that this outage is out of our immediate control and we contacted our datacenter provider for additional assistance. Looking from the outside it became clear that our network was no longer announced via BGP or that the routes kept changing. As we learned during the outage this was done by our provider as part of a DDoS mitigation which we fell victim to. Neither Gini nor our provider were the actual target but we were still affected and had to take countermeasures. After it became apparent, that redistributing the traffic alone will not stop the attack, our provider decided to use a number of scrubbing centres to remove the unwanted traffic and gain back control. The attack continued until Sunday morning but our services regained full stability from Saturday 03:30 onwards. We discussed additional options to reduce the time it took to fend off the attack and we and our datacenter provider will continue to optimise the procedures and tools.