CSIU experienced a critical incident on February 16, 2021 affecting Financial Information System (FIS), lasting 10d 5h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 16, 2021, 03:43 PM UTC
Please be advised we have discovered an issue between us and one of our ISPs beginning at approximately 10:09 AM that is causing some clients to lose connection to our hosted services. We have a ticket open with our ISP and hope to get more details on what is causing this as soon as possible.
- identified Feb 16, 2021, 11:00 PM UTC
We apologize for the length of time it has taken to resolve this issue. Our ISP has finally determined that our websites have been targeted by a Distributed Denial-of-Service (DDoS) attack. A DDoS attack event does NOT compromise data. The intent of a DDoS attack is to disrupt Internet Service. We have been working all day with our ISP as they implement potential fixes to address this issue. We apologize for the inconvenience this has caused, and assure you that we are working around the clock with them to restore your services.
- monitoring Feb 16, 2021, 11:28 PM UTC
A fix has been implemented by our ISP and we are monitoring the results. The results are promising at this point.
- identified Feb 16, 2021, 11:52 PM UTC
The fix that was implemented resolved the initial issues, but we are again seeing issues with connectivity from one of our ISPs.
- identified Feb 16, 2021, 11:53 PM UTC
The fix that was implemented resolved the initial issues, but we are again seeing issues with connectivity from one of our ISPs.
- monitoring Feb 17, 2021, 01:07 AM UTC
The attackers pivoted and the ISP had to re-evaluate their fix. We are once again back to a working state.
- monitoring Feb 17, 2021, 02:51 PM UTC
We are aware of recurring connectivity issues to our hosted services. We are working with our one ISP to investigate.
- investigating Feb 17, 2021, 02:53 PM UTC
We are aware of recurring connectivity issues to our hosted services. We are working with our one ISP to investigate.
- monitoring Feb 17, 2021, 04:19 PM UTC
We continue to dialog with our ISP as they work to keep this attack at bay. They have once again stopped the attack and services have been restored again. We will leave this incident in a monitoring status until we are confident this issue is no longer recurring.
- investigating Feb 18, 2021, 02:42 PM UTC
We have been alerted once again about connectivity issues occurring. We are already in contact with our ISP to investigate. We will update as soon as we have more details.
- monitoring Feb 18, 2021, 02:57 PM UTC
Connectivity to services has been restored. We continue to work with our ISP to investigate this attack. We will leave this incident in a monitoring status until we are confident this issue is no longer recurring.
- investigating Feb 18, 2021, 03:06 PM UTC
Connectivity issues are occurring again. We remain in contact with our ISP, investigating this attack. We will update as soon as we have more details.
- monitoring Feb 18, 2021, 03:22 PM UTC
Connectivity to services has been restored. We continue to work with our ISP to investigate these attacks. We will leave this incident in a monitoring status until we are confident this issue is no longer recurring.
- investigating Feb 18, 2021, 03:37 PM UTC
Connectivity issues are occurring again. We are actively in contact with our ISP, investigating these attacks. We will update as soon as we have more details.
- monitoring Feb 18, 2021, 03:42 PM UTC
Connectivity to services has been restored. We continue to actively work with our ISP to investigate these attacks. We will leave this incident in a monitoring status until we are confident this issue is no longer recurring.
- investigating Feb 22, 2021, 03:27 PM UTC
Connectivity issues are occurring again. We are in contact with our ISP, who are investigating three active current attacks. We will update as soon as we have more details.
- monitoring Feb 23, 2021, 02:57 AM UTC
We continue to work with our ISP to put steps in place that have lessened the impacts of the attacks, but they continue as multiple coordinated attacks originating from thousands of locations around the world. The attacks have reached speeds of up to 20Gbps when our bandwidth is considerably less. We continue to work with our provider for short term, as well as long term solutions and have filed an incident report with the FBI.
- monitoring Feb 23, 2021, 09:55 PM UTC
The DDoS attack mitigation efforts put in place by our ISP appear to have worked well today. We did not receive any new reports of connectivity issues. We continue to work with our ISP on implementing additional mitigation strategies, and we continue to monitor for any signs of malicious activity. We again apologize for the inconvenience this is causing and appreciate your understanding.
- monitoring Feb 24, 2021, 06:28 PM UTC
We have seen no additional recurrences since our last update. We will continue to monitor ongoing for changes and alert you as necessary.
- monitoring Feb 25, 2021, 09:37 PM UTC
We have had no issues affecting connectivity to our systems today. We continue to monitor for any new problems. We are engaged in ongoing discussions with the FBI and our ISP regarding this incident, and we are following up on additional short- and long-term mitigation efforts. Thank you for your continued support, patience, and understanding.
- resolved Feb 26, 2021, 09:01 PM UTC
There have been no new connectivity issues reported for several days, so we are closing the incident. We will continue to monitor for any signs of recurrence, and will open a new incident and keep you informed if any new issues arise.
- postmortem Apr 22, 2021, 12:55 PM UTC
As we shared in February, we experienced a denial-of-service attack \(DDoS\) that resulted in a temporary outage of some services. We sincerely regret the difficulty and inconvenience it caused. While frustrating, DDoS attacks – including the one we experienced – do not involve improper access to data. These types of attacks flood networks and servers with an excessive amount of anonymous traffic, causing disruptions and outages. We want to confirm for you that this incident did not result in a breach of our systems nor any unauthorized third-party access to data. We know this incident raised questions and, as a valued partner, we want you to know that we took immediate action, alerted authorities, and enlisted cybersecurity experts who also confirmed that network security was not impacted due to this incident. In addition, we worked with our local service provider who has since improved their DDoS mitigation device and we have increased network capacity to help defend against similar attacks in the future. Thank you for your patience and understanding throughout this incident. We understand the trust you place in our systems, and frustrations when they are inaccessible. Thanks for being a part of the CSIU software family!