4Schools incident
Emails (including login codes) failing to send from 4Schools products
4Schools experienced a major incident on December 19, 2024 affecting Application and Application and 1 more component, lasting 7h 23m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 19, 2024, 05:25 PM UTC
We are currently investigating this issue.
- identified Dec 19, 2024, 06:19 PM UTC
The service we rely on for email delivery, Mailchimp, is down. Unfortunately, they are not acknowledging the problem on their status page. However, extensive testing by our team, peers in other industries, and some chatter on social media have convinced us that Mailchimp's transactional email service is the problem. We are looking into alternatives.
- identified Dec 19, 2024, 08:24 PM UTC
Our developers are updating affected products now, to move them away from Mailchimp services to a more reliable provider. ⚠ Please note, some messages that used to come from `4schools.net` will now come from `idm.4schools.net` or `udm.4schools.net`. We will study the effects of this change and respond if needed to ensure delivery is reliable. Current status: ✅ Student Services login code emails are now working ⬛ CMS4Schools login codes fix is being deployed ⬛ Student Services reports and communication email fix is being developed ⬛ CMS4Schools notification email fix is being developed
- identified Dec 19, 2024, 09:28 PM UTC
Current status: ✅ Student Services login code emails are now working ⬛ CMS4Schools login codes fix is being deployed ✅ Student Services reports and communication emails are now working ⬛ CMS4Schools notification email fix is being developed
- monitoring Dec 19, 2024, 11:54 PM UTC
Email delivery is now working reliably. We are monitoring email queues and will take corrective action if problems reoccur.
- resolved Dec 20, 2024, 12:48 AM UTC
Email delivery continues to work reliably from 4Schools applications. Users may notice old messages getting delivered as the mail provider catches up with some queued messages. If these messages are no longer relevant (for example, an expired login access code), please disregard the message. Thank you for your patience today as we worked hard to overcome this email outage!