DataCite had 11 outages in the last 2 years totaling 98h 34m of downtime — averaging 0.5 incidents per month.
There were 11 DataCite outages since October 30, 2024 totaling 98h 34m of downtime. Each is summarised below — incident details, duration, and resolution information.
Requests to the MDS API are returning errors. We've identified the issue and are working on a fix.
The MDS API is now operational.
We are investigating an issue where new DOI records and DOI record updates may be taking longer than usual to appear in services like Fabrica, Commons, and the REST API. API requests are still being received and processed successfully.
We have identified and resolved the issue. New records and updates should now appear as expected in services like Fabrica, Commons, and the REST API, and queued changes have been processed.
We are investigating degraded performance across our services. This is manifesting as occasional error messages or slow response times for API requests and/or frontend services.
Performance has stabilized. We are continuing to monitor.
Performance of all services has returned to normal levels.
We are investigating intermittent performance issues affecting our services, which may result in slow response times for API requests and frontend applications.
Intermittent performance issues have stabilized. We will continue to monitor.
We're experiencing some timeouts adding to our search index cluster, this will mostly be noticeable in DOIs that are registered might not be findable via certain API endpoints and search/management interfaces. We believe this is a disk space issue due to increased demand and are investigating a remedy. Note no actual data loss as this is just the indexing side, data storage is unaffected and therefore DOI registration is operational.
A temporary fix has been put in place and things should be working now, but we may still have a period of time where some things have not been indexed. We are working on a longer term fix for increased demand and also add any missing items (i.e. DOIs) to our search indices.
We've added a further fix and all search indexing and related services should be operational. We are now monitoring and looking to clean up any missing data in the indices.
All systems are operational and data unavailable in search index has been restored.
We've identified an issue that may affect access to the DataCite homepage at https://datacite.org and the Make Data Count homepage at https://makedatacount.org. Other DataCite services and APIs are not affected. The issue has been isolated, and we are working on a resolution.
The DataCite homepage at https://datacite.org and the Make Data Count homepage at https://makedatacount.org are now available. We will continue to monitor.
The test services that are powered by our OpenSearch cluster are currently not accepting new index writes. This means data is still being saved to the test database, but new records and record updates (for DOIs, members, repositories, etc.) are not appearing in REST API list responses, Fabrica list views, and other places that rely on the OpenSearch index We believe we already know the issue but are currently discussing about how best to resolve. Production services are not affected and are operating as normal.
Test services are restored.
We're currently experiencing increased load on our services and are investigating the underlying cause and mitigation that we can take. This mostly appears to be affecting our search cluster, and therefore has a knock-on effect to queries and retrieval through DataCite Commons and some functionality in Fabrica. DOI Registration services should be unaffected, although may experience timeouts viewing content after registration.
Mitigation strategies have been deployed to help with some anonymous traffic that was causing increased load on our search cluster. Services appear to be resuming normal function but with potentially still some degraded performance, we will continue monitoring.
This incident has been resolved.
We're currently experiencing increased load on our services, we're investigating mitigation strategies. This mostly appears to be affecting our search cluster. Registration services should be unaffected, though may experience timeouts viewing content through frontend services / api search interfaces.
We are continuing to investigate this issue.
Services appear stable, we will continue monitoring.
This incident has been resolved.
We've noticed an increase load on our services which is having an affect on returning search results in different places. We are investigating possible causes and attempting to compensate.
Yesterday we made some changes to how traffic was handled by our services, this combined with overall load reducing, everything should be working as normal now.