Failed delivery of notification emails for very large query submissions and some forward-link alerts

Update timeline

1. investigating Jul 10, 2025, 03:19 AM UTC

   As a result of our recent AWS migration (https://status.crossref.org/incidents/s92nm4xn1w84), Metadata Manager submissions are currently returning 'TypeError' error messages and our depositor reports are returning a 404 'Not Found' status code error message. Our technical team is working on a fix. -IF

2. identified Jul 10, 2025, 02:08 PM UTC

   The issue has been identified and we continue to work on a fix for Metadata Manager registrations that are failing with the 'TypeError' error messages. We have deployed a fix for the depositor reports. That issue is resolved (e.g., https://data.crossref.org/depositorreport?pubid=J458872). -IF

3. identified Jul 10, 2025, 07:21 PM UTC

   We are continuing to work on a fix for the errors present with Metadata Manager registrations. We have also identified errors with reference registrations via the Simple Text Query form (https://apps.crossref.org/SimpleTextQuery/) and errors in the display of Crossmark dialog boxes (e.g., https://crossmark.crossref.org/dialog/?doi=10.3233/JIFS-179309&domain=pdf). -IF

4. identified Jul 10, 2025, 08:36 PM UTC

   Errors to Crossmark dialog boxes have been fixed, as you can see with this example: https://crossmark.crossref.org/dialog/?doi=10.36676/girt.v12.i1.109&domain=pdf We're continuing to work on Metadata Manager and the Simple Text Query form. -IF

5. identified Jul 11, 2025, 02:08 PM UTC

   We'll have an emergency, unplanned release (https://status.crossref.org/incidents/1c42q5j29ds2) in ~30 minutes to address the ongoing issues with Metadata Manager and newly reported errors with delivery of notification emails after registrations. During that release window - from 1430 to 1630 today - our members may experience delays in deposit system admin tool (doi.crossref.org) processing. Deposits submitted during this time will be stored and processed after the emergency maintenance has been completed.

6. identified Jul 11, 2025, 02:08 PM UTC

   We are continuing to work on a fix for this issue.

7. monitoring Jul 11, 2025, 03:22 PM UTC

   A fix has been implemented and we are monitoring the results.

8. monitoring Jul 11, 2025, 09:16 PM UTC

   Submissions from Metadata Manager and the Simple Text Query form have been restored. Unfortunately, delivery of email notifications of submission logs following a registration (for Simple Text Query form, web deposit form, and XML submissions) do appear to be delayed. We believe this is due to the volume of notification emails in the system, and the result of quite a backlog there. We are monitoring. -IF

9. identified Jul 12, 2025, 03:20 PM UTC

   Delivery of email notifications of submission logs following a registration (for Simple Text Query form, web deposit form, and XML submissions) began failing again late yesterday. Our technical team continues to work on a fix. -IF

10. identified Jul 14, 2025, 05:42 PM UTC

    Delivery of email notifications of submission logs following a registration (for Simple Text Query form, web deposit form, and XML submissions) has been restored for many members and their submissions. We are working to remedy the other outstanding errors that are causing some email notifications to go undelivered. -IF

11. identified Jul 15, 2025, 08:51 PM UTC

    We are continuing to work on a fix for this issue.

12. identified Jul 16, 2025, 02:40 PM UTC

    We'll have an emergency, unplanned release (https://status.crossref.org/incidents/c5nyp41k6vx5) in ~20 minutes to address the ongoing errors with delivery of notification emails after registrations. During that release window - from 1500 to 1700 UTC today - our members may experience delays in deposit system admin tool (doi.crossref.org) processing. Deposits submitted during this time will be stored and processed after the emergency maintenance has been completed.

13. identified Jul 17, 2025, 04:29 PM UTC

    Yesterday's release did fix delivery of notification emails for many submissions, but we still have a small minority of notification emails that are going undelivered. Our technical team continues to work on a fix. -IF

14. identified Jul 21, 2025, 12:42 PM UTC

    We are continuing to work on a fix for this issue.

15. identified Jul 22, 2025, 07:39 PM UTC

    We have resolved the problems for the vast majority of members and those submissions, but we know that a fraction of notifications are failing for very large submissions. We are continuing to work on a fix for this issue. -IF

16. identified Jul 23, 2025, 08:20 PM UTC

    We are continuing to work on a fix for this issue.

17. identified Jul 25, 2025, 08:41 PM UTC

    We are continuing to work on a fix for this issue.

18. identified Jul 30, 2025, 04:07 PM UTC

    We are continuing to work on a fix for this issue.

19. identified Aug 06, 2025, 06:27 PM UTC

    We are continuing to work on a fix for this issue.

20. identified Aug 08, 2025, 05:23 PM UTC

    We have confirmed that all notifications for registrations are being delivered and have been since 22 July. A small fraction of query submission email notifications (for very large queries) and forward-link alerts are going undelivered. Our technical team continues to work on a fix. -IF

21. monitoring Aug 26, 2025, 03:00 PM UTC

    A fix has been implemented and we are monitoring the results.

22. resolved Aug 27, 2025, 01:16 PM UTC

    This incident has been resolved. -IF