Resolved after 8 hours 18 minutes, affecting Campaign Delivery, on December 19, 2025.
**Summary:** On December 19 at 10:58 UTC, customers using campaign email delivery in the EU region experienced delays in email sending. Emails were successfully queued during the incident and delivered once the issue was resolved. An incident was declared after it was identified that campaign emails were not being sent as expected. The engineering team determined that an internal cleanup job caused unexpected load on the system, slowing email processing. A targeted hotfix was deployed to mitigate the issue, and normal operations fully resumed the same day, at 16:10 UTC. **Impact:** * Campaign emails for EU customers were significantly delayed. * No emails were lost; all queued emails were delivered after recovery. **Root Cause:** An internal cleanup job was run during normal peak hours, generating a large volume of backend operations. This coincided with high email traffic and triggered inefficient processing behavior, which delayed campaign email delivery across the EU region. **Mitigation:** Once the incident was declared, engineers identified an inefficient database query contributing to the slowdown. A targeted hotfix was deployed to bypass the slow operation, allowing email processing to recover. All queued emails were successfully delivered after the fix was applied. **Recurrence Prevention:** To minimize the likelihood of similar issues in the future, Firstup will take the following actions: * Move the EU email processing workload to a dedicated database cluster, aligning it with other regions. * Ensure internal cleanup jobs are restricted to off-peak hours. * Improve monitoring and alerting to provide earlier, high-signal detection of email processing delays.
Affected components
Update timeline
- investigating Dec 19, 2025, 02:32 PM UTC
We are investigating a service disruption affecting email delivery on EU communities (US communities are not affected) This may be causing delays in email delivery from campaigns. Our next update will be in one hour.
- investigating Dec 19, 2025, 03:37 PM UTC
We are continuing to investigate this issue and are currently testing a potential solution. Campaigns are sending; however, delivery delays are still occurring.
- monitoring Dec 19, 2025, 04:15 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Dec 19, 2025, 10:50 PM UTC
No further indications of email deliverability issues have been observed or reported. This incident is resolved.
- postmortem Dec 30, 2025, 11:14 PM UTC
**Summary:** On December 19 at 10:58 UTC, customers using campaign email delivery in the EU region experienced delays in email sending. Emails were successfully queued during the incident and delivered once the issue was resolved. An incident was declared after it was identified that campaign emails were not being sent as expected. The engineering team determined that an internal cleanup job caused unexpected load on the system, slowing email processing. A targeted hotfix was deployed to mitigate the issue, and normal operations fully resumed the same day, at 16:10 UTC. **Impact:** * Campaign emails for EU customers were significantly delayed. * No emails were lost; all queued emails were delivered after recovery. **Root Cause:** An internal cleanup job was run during normal peak hours, generating a large volume of backend operations. This coincided with high email traffic and triggered inefficient processing behavior, which delayed campaign email delivery across the EU region. **Mitigation:** Once the incident was declared, engineers identified an inefficient database query contributing to the slowdown. A targeted hotfix was deployed to bypass the slow operation, allowing email processing to recover. All queued emails were successfully delivered after the fix was applied. **Recurrence Prevention:** To minimize the likelihood of similar issues in the future, Firstup will take the following actions: * Move the EU email processing workload to a dedicated database cluster, aligning it with other regions. * Ensure internal cleanup jobs are restricted to off-peak hours. * Improve monitoring and alerting to provide earlier, high-signal detection of email processing delays.
Your product runs on more than Firstup
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services