Crossref incident

Metadata indexing failure and instability on the REST API (Starting Friday 12th 20:00 UTC and continuing) affecting several other tools

Minor Resolved View vendor source →

Crossref experienced a minor incident on September 15, 2025 affecting Public REST API and Polite REST API and 1 more component, lasting 17d 6h. The incident has been resolved; the full update timeline is below.

Started
Sep 15, 2025, 09:13 AM UTC
Resolved
Oct 02, 2025, 03:31 PM UTC
Duration
17d 6h
Detected by Pingoru
Sep 15, 2025, 09:13 AM UTC

Affected components

Public REST APIPolite REST APIMetadata searchParticipation reportsCrossmark dialog server

Update timeline

  1. investigating Sep 15, 2025, 09:13 AM UTC

    We have found that some records failed to be indexed by the REST API on Friday 12th September, and continuing. Our technical team is investigating.

  2. investigating Sep 15, 2025, 04:33 PM UTC

    We are continuing to investigate this issue.

  3. investigating Sep 16, 2025, 02:14 PM UTC

    Our technical team is re-pushing all of the records that have failed to be indexed since Friday, 12 September. Those records should be indexed in the REST API in the next 24 hours. -IF

  4. monitoring Sep 17, 2025, 12:48 PM UTC

    A fix has been implemented and we are monitoring the results. Do to the heavy workload of indexing a large volume of records yesterday, we have observed longer response times in the Public and Polite pools of the REST API. We believe those response times should improve throughout the day. -IF

  5. monitoring Sep 19, 2025, 10:15 AM UTC

    The public and polite pools are still showing as unstable with long response times and timeouts occurring. We are still looking into this as a priority and will post with more information as we know it.

  6. monitoring Sep 22, 2025, 07:08 PM UTC

    We are continuing to monitor for any further issues.

  7. identified Sep 23, 2025, 01:26 PM UTC

    The public and polite pools are still experiencing some instability, which is affecting downstream tools such as Participation Reports, Search, and Crossmark. We're working on a fix incl. infrastructure upgrades that should help long-term. -GH

  8. identified Sep 26, 2025, 01:59 PM UTC

    We've identified and fixed one issue which was affecting citation counts. We're still working on two additional tasks: upgrading Elasticsearch in the REST API and updating the service that synchronises new and updated metadata records with the REST API. -GH

  9. resolved Oct 02, 2025, 03:31 PM UTC

    We have updated the service that synchronises new and updated metadata records with the REST API; all three pools of the REST API are performing well as a result (as are the services that run on it). Thus, we are resolving this incident. -IF