Resolved after 10 minutes, affecting Cadmus AU Student and Teachers API and Cadmus AU Data API and Reports, on July 24, 2025.
# What Happened * A database maintenance period triggered a migration on the database that was meant to tighten constraints for data consistency. For example, old text columns that were supposed to be UUIDs were updated. * The change created incompatibilities between certain kinds of API requests and the database, returning errors. For example LTI launch requests from LMSs were affected. Not all requests were affected. # How it Happened * Database maintenance periods are normally configured outside busy hours per Cadmus region. A configuration change updated this erroneously and went unnoticed. * Cadmus APIs are always tested to be backwards compatible over at-least 2 versions. However this type of change created a missed inconsistency that should have been tested. # How it was fixed * New API versions were immediately deployed to replace the older versions to ensure the database and API were consistent. The deployed replaced the in-compatible APIs. # Future Mitigation * Database upgrade schedules were manually checked and configured. The central infrastructure configuration now enforces the maintenance periods correctly. * A new release process of testing data-consistency type migrations is now in-place. We intent to perform more such migrations as we cleanup old-data and old-APIs with the new process.
Affected components
Update timeline
- identified Jul 24, 2025, 01:30 AM UTC
The issue has been identified and a fix is being implemented.
- monitoring Jul 24, 2025, 01:30 AM UTC
A fix has been deployed and the APIs are expected to be back online soon.
- resolved Jul 24, 2025, 01:31 AM UTC
This incident has been resolved.
- postmortem Jul 28, 2025, 02:56 AM UTC
# What Happened * A database maintenance period triggered a migration on the database that was meant to tighten constraints for data consistency. For example, old text columns that were supposed to be UUIDs were updated. * The change created incompatibilities between certain kinds of API requests and the database, returning errors. For example LTI launch requests from LMSs were affected. Not all requests were affected. # How it Happened * Database maintenance periods are normally configured outside busy hours per Cadmus region. A configuration change updated this erroneously and went unnoticed. * Cadmus APIs are always tested to be backwards compatible over at-least 2 versions. However this type of change created a missed inconsistency that should have been tested. # How it was fixed * New API versions were immediately deployed to replace the older versions to ensure the database and API were consistent. The deployed replaced the in-compatible APIs. # Future Mitigation * Database upgrade schedules were manually checked and configured. The central infrastructure configuration now enforces the maintenance periods correctly. * A new release process of testing data-consistency type migrations is now in-place. We intent to perform more such migrations as we cleanup old-data and old-APIs with the new process.
Your product runs on more than Cadmus
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services