DonorDrive experienced a notice incident on October 23, 2019 affecting Admin and Export API and 1 more component, lasting 1d 2h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Oct 23, 2019, 02:57 PM UTC
We have identified an issue causing DonorDrive admin and public sites to be intermittently inaccessible. Our team is working to resolve the issue and we will provide an update. Please contact [email protected] with any questions.
- identified Oct 23, 2019, 04:56 PM UTC
Our Internet Service Provider, CenturyLink, has identified an error in their data center that has rendered DonorDrive sites inaccessible. They are working to resolve this issue as quickly as possible. We are awaiting confirmation that they have corrected their errors and DonorDrive sites are accessible once more. Please contact [email protected] with any questions.
- identified Oct 23, 2019, 07:05 PM UTC
We are continuing to work on a fix for this issue.
- identified Oct 23, 2019, 09:41 PM UTC
DonorDrive admin and public sites are intermittently accessible. We are continuing work to ensure sites are fully accessible. Please contact [email protected] with any questions.
- identified Oct 24, 2019, 12:53 AM UTC
DonorDrive admin and public sites are accessible after refreshing. Our team is still working to resolve the issue causing them to appear inaccessible initially. Please contact [email protected] with any questions.
- monitoring Oct 24, 2019, 10:32 AM UTC
DonorDrive admin and public sites have been restored and are accessible. Please note there may be a small subset of users still encountering occasional load issues. Our Internet Service Provider and our team continue working to resolve any remaining issues. Please contact [email protected] with any questions.
- resolved Oct 24, 2019, 05:31 PM UTC
DonorDrive admin and public sites are fully accessible. We will post a detailed postmortem of the incident shortly. Please contact [email protected] with any questions.
- postmortem Oct 25, 2019, 01:25 PM UTC
From Wednesday, October 23rd at 10:35 AM EDT to Thursday, October 24th at 5:43 AM EDT, DonorDrive was down or intermittently inaccessible due to a distributed denial of service \(DDoS\) incident. DonorDrive employs DDoS mitigation services with our network provider to prevent these types of events and, unfortunately, those services failed to prevent the downtime and the impact in this scenario. We have and continue to take aggressive steps to improve the mechanisms in place to mitigate these types of scenarios in the future, including deployment of incremental hardware and software solutions designed to mitigate the impact of any future incidents.