The DOI Foundation Outage History

Started Oct 12, 2026, 03:07 AM UTC· resolved Oct 13, 2026, 05:17 AM UTC

1. identified Oct 12, 2026, 03:07 AM UTC

   Due to a sudden large spike in traffic -- either a DDoS attempt or a unfriendly bulk DOI resolution -- some DOI resolution would have failed on Sunday evening / early Monday morning. Affected regular users were mostly automated traffic, but a smaller amount of human browser use of doi.org would also have been affected. We are taking steps to identify and control the traffic.

2. resolved Oct 13, 2026, 05:17 AM UTC

   This incident has been resolved.

3. postmortem Oct 13, 2026, 05:18 AM UTC

   On October 11th and 12th [doi.org](http://doi.org) experienced three large traffic spikes: from 22:30 to 22:40, 23:51 to 00:04, and 01:47 to 02:33. In one particular region traffic was 20x to 25x normal during the incident. Users who accessed [doi.org](http://doi.org) in other regions were unaffected. In the affected region, we believe load shedding mostly kept the system stable for non-automated users. But many automated users would have had legitimate requests rejected during the incident. A smaller number of non-automated users would have hit an error, which probably was resolved simply by reloading the page. Considering the largest part of the spike 02:00 to 02:35, out of 4M requests not from the source of the traffic spike, 172k were automated requests that were rejected by the load shedder as part of maintaining system stability, and almost 4k were automated or non-automated requests that failed either due to the load shedder or transient instability. It’s often not entirely clear whether a traffic spike represents an attempted DDoS attack or simply an overzealous bulk DOI resolution activity. In this case, looking again at the 02:00 to 02:35 time range, there were 18.8M requests apparently from the source of the traffic spike, of which 6M were rejected by load shedding; of the remaining 12.8M, 11.1M were DOI Not Found. That does seem that it may indicate an attempted DDoS attack. Although load shedding would eventually give way to just scaling to absorb the traffic, in this incident we decided to use Cloudflare facilities to challenge the apparent source of the traffic. This would allow non-automated traffic \(a human at a browser\) but would generally not allow automated traffic. We believe that “legitimate” automated traffic is unaffected by this. The attack, if that is what it is, has continued, but Cloudflare is absorbing it – at a rate of over 9M requests per minute shortly after we instituted the challenge, but averaging closer to 1M requests per minute over the past day. We will continue to monitor this. If you have automated traffic to doi.org that you believe is being adversely affected by the current configuration, please send a message to [email protected].

Read the full incident report →

Started Mar 17, 2026, 09:14 AM UTC· resolved Mar 17, 2026, 02:32 PM UTC

1. identified Mar 17, 2026, 09:14 AM UTC

   A surge in traffic has affected some of the backend infrastructure of doi.org. Most DOI resolutions are failing. We are working on a fix.

2. monitoring Mar 17, 2026, 09:57 AM UTC

   Service is restored. We continue to monitor. Downtime was roughly from 6:30 UTC to 9:30 UTC.

3. resolved Mar 17, 2026, 02:32 PM UTC

   Monitoring shows traffic flowing normally again. This incident has been resolved.

4. postmortem Mar 18, 2026, 03:58 AM UTC

   At 06:30 UTC on 2026-03-17, after over an hour of substantially elevated traffic, four different servers in the backend of [doi.org](http://doi.org) became unstable approximately simultaneously. These servers collectively provide handle resolution for DOIs maintained by Crossref, the DOI Registration Agency which handles the largest resolution load. [doi.org](http://doi.org) itself continued to function, but resolutions of DOIs maintained by Crossref began to give errors. We received our first notification that something was amiss at 06:38 UTC, but investigation did not begin until a second notification at 07:39 UTC. Restarting the servers did not succeed in removing the instability. There was enough traffic that the servers fell over more or less immediately after restarting. We quickly tightened [doi.org](http://doi.org) rate limiting \(at Cloudflare\) in an attempt to control the traffic, but it did not make enough difference; servers continued to become unstable quickly after restarting. We suspect that the rate limiting might have prevented the issue had it been in place before the traffic surge, with the servers already running, but it didn’t decrease the traffic enough to allow bringing the servers up. We next considered increased resources for servers. This also did not help, at least without further exploration of configuration or software changes. The bottleneck was not directly memory or CPU, but the ability of a each single machine to handle a sufficiently large number of network connections. So we set in motion increasing the number of servers. While waiting on that process, we noticed, belatedly, that the source of the traffic surge identified itself by User-Agent. We blocked that User-Agent string and that reduced the traffic enough to bring the servers online. By 09:30 UTC service was back to normal. **Remaining questions and lessons learned** It’s still not clear exactly what in the nature of the traffic at 06:30 UTC caused such instability. The traffic might have been 5x typical, but that level of traffic happens with some frequency. Some sort of failure cascade seems likely – one machine failed, and that increased the traffic to other machines enough that one more failed, and so on. We have increased the number of backend servers and have made it easier to increase this number again in the future. In hindsight we could have resolved the issue much earlier simply by noticing that the source of increased traffic was easy to block. This is not always the case, but it was the case this time and downtime was increased by failure to take advantage of this quickly. Also, there are other levers at Cloudflare for controlling traffic dramatically, such as to deal with a DDoS attack. It would be unfortunate if some legitimate automatic use of [doi.org](http://doi.org) was temporarily blocked, but it might be preferable to an extended downtime. We have put much effort over the years in increasing the reliability and scalability of the higher levels of [doi.org](http://doi.org) infrastructure, with great success. In that time the backend handle servers have always been rock solid, but it is no surprise that eventually instability due to increased traffic has come for them as well. In the short and medium term we expect that simply having more backend servers will prevent a repeat of this, but we will continue to explore other options as traffic continues to increase.

Read the full incident report →

Started Nov 18, 2025, 12:55 PM UTC· resolved Nov 18, 2025, 05:23 PM UTC

1. investigating Nov 18, 2025, 12:55 PM UTC

   Cloudflare is experiencing a global network issue which is affecting many internet services, including DOI resolutions on doi.org. Services should return to normal once the issue at Cloudflare is resolved. For more information and updates on the Cloudflare outage, see https://www.cloudflarestatus.com/incidents/8gmgl950y3h7

2. investigating Nov 18, 2025, 01:22 PM UTC

   We have bypassed Cloudflare as a presumably temporary measure. Users may not see the effect of this immediately depending on DNS propagation and caching, but if Cloudflare's incident continues, this should help to allow more DOI resolution.

3. investigating Nov 18, 2025, 01:23 PM UTC

   We are continuing to investigate this issue.

4. investigating Nov 18, 2025, 02:47 PM UTC

   Cloudflare reports that their incident is resolved. We will continue to bypass Cloudflare for the moment while they monitor their fix. We note though, as an unfortunate limitation of bypass, that IPv6 access to doi.org is currently not working.

5. resolved Nov 18, 2025, 05:23 PM UTC

   Cloudflare reports that things have returned to normal. We have restored use of Cloudflare for DOI resolution at doi.org. IPv6 access to doi.org should resume for any impacted IPv6 users as soon as DNS propagation and cache expiration is complete.

6. postmortem Nov 20, 2025, 05:14 PM UTC

   DOI resolutions start at Cloudflare, which provides a number of functions, notably including DDoS protection. The Cloudflare outage on Tuesday is discussed at length at [https://blog.cloudflare.com/18-november-2025-outage/](https://blog.cloudflare.com/18-november-2025-outage/) . DOI resolutions began to be partially impacted as early as 11:30 UTC, although some DOI resolutions continued to succeed for most of the outage. Our internal monitoring alerted us to the problem by 11:50 UTC. Once it was clear that the Cloudflare outage was not going to be resolved quickly, we decided to bypass Cloudflare. This required access to Cloudflare functionality which was itself affected by the outage, but by 13:10 UTC we were able to finalize the bypass. The bypass allowed DOI resolutions to resume within less than 2 hours of the initial issues, instead of the full 3 hour outage experienced by Cloudflare generally. The fact that many of the resources that DOIs point to were also offline, however, meant that the restart of successful resolution may not have been noticeable to all users, as DOI resolution may have succeeded in redirecting them to a document they were still unable to access. Additionally, we found that the bypass did not work for IPv6 traffic; we have resolved that issue should we decide to bypass Cloudflare again in the future. We will continue to monitor DOI use of network services such as Cloudflare and AWS, as we have done for some years now, configuring them to optimize the DOI system. Cloudflare, which is on the whole quite reliable, will continue to be our DDoS mitigation solution for the immediate future, but we will continue to monitor that specific issue in combination with other alternatives.

Read the full incident report →

Started Oct 17, 2025, 03:08 AM UTC· resolved Oct 17, 2025, 03:32 AM UTC

1. resolved Oct 17, 2025, 04:49 AM UTC

   At 2025-10-17 03:08Z lasting for a minute or so, there was a surge of resolution traffic at doi.org at maybe 150x normal traffic. We suspect this may have been a DDoS attempt or a test of a possible DDoS technique. Resolution was impacted for ordinary users of doi.org; as many as 80000 resolution requests (approximately 5% of the apparently legitimate traffic during this time) received various errors, most in the first few minutes but some as late as 03:32Z. Currently resolution is normal but we continue to monitor.

Read the full incident report →

Started Oct 12, 2025, 05:20 PM UTC· resolved Oct 12, 2025, 09:56 PM UTC

1. investigating Oct 12, 2025, 05:20 PM UTC

   Overnight there was an unusual series of traffic spikes on the doi.org resolution servers. The spikes were large (100x normal traffic) but lasted for only seconds each time and happened roughly once an hour, so scaling in new resolution servers in response to traffic was not helpful. Our "load shedding" infrastructure successfully kicked in and prevented any instability, at the cost of a small number of "friendly" requests potentially receiving either a 429 or 503 response. I estimate that up to 2000-3000 friendly requests might have gotten such responses, many of which are normally friendly bots, but some of which might have been ordinary users. If it was a human clicking a link, they almost certainly would have succeeded on trying a second time. If there are any user complaints about doi.org resolution service early today (2025-10-12) I would be interested to know about it. We are monitoring the situation to see whether these spikes resume, and considering ways to prevent this kind of traffic from impacting the use of the system by other users in the future.

2. resolved Oct 12, 2025, 09:56 PM UTC

   The traffic spikes have stopped. We will continue considering ways to prevent this kind of traffic from impacting the use of the system by other users in the future.

Read the full incident report →