Cornerstone incident
Spamhaus Temporarily Blacklisted UK Email IP
Cornerstone experienced a major incident on October 6, 2025 affecting Response Time and Response Time and 1 more component, lasting 15d 19h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 06, 2025, 11:49 PM UTC
Cornerstone UK email sending IPs have been temporarily blacklisted by Spamhaus, a leading global email reputation authority, due to poor email hygiene detected in email activity originating from certain customer accounts. During this time, emails may fail to deliver. This is our top priority and we are working closely with the vendor to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.
- investigating Oct 07, 2025, 01:35 AM UTC
We are continuing to investigate this issue as our top priority and will provide regular updates until it is fully resolved. Thank you for your continued patience and understanding.
- identified Oct 07, 2025, 03:39 AM UTC
We have identified the cause of the issue and have implemented internal corrective measures. We are working to restore normal operations. We will update you further in the next two hours.
- identified Oct 07, 2025, 05:31 AM UTC
Email delivery success rate is improving. We will update you further in the next two hours.
- identified Oct 07, 2025, 07:36 AM UTC
We are continuing to work on the fix for this issue. The next update will follow within 2 hours or as soon as we receive further information.
- identified Oct 07, 2025, 09:35 AM UTC
We are currently awaiting a response from Spamhaus regarding the delisting request and will provide an update as soon as we receive any information.
- identified Oct 07, 2025, 11:45 AM UTC
Cornerstone IPs remain listed on the Spamhaus blocklist, which means email deliveries may continue to be impacted, depending on the client’s Domain Reputation Checker policies for Spamhaus. We are still awaiting further updates from Spamhaus regarding the delisting request.
- identified Oct 07, 2025, 02:55 PM UTC
We are currently awaiting a response from Spamhaus regarding the delisting request and will provide an update as soon as we receive any information.
- identified Oct 07, 2025, 07:00 PM UTC
We are still awaiting a response from Spamhaus regarding the delisting request and will provide an update as soon as we receive any information.
- identified Oct 07, 2025, 08:56 PM UTC
We continue to work closely with Spamhaus as we await confirmation of our delisting request. We will share regular updates as new information becomes available.
- identified Oct 07, 2025, 11:01 PM UTC
We continue to work closely with Spamhaus as we await confirmation of our delisting request. We will share regular updates as new information becomes available.
- identified Oct 08, 2025, 01:00 AM UTC
We continue to work closely with Spamhaus as we await confirmation of our delisting request. We will share regular updates as new information becomes available.
- identified Oct 08, 2025, 03:08 AM UTC
We continue to work closely with Spamhaus as we await confirmation of our delisting request. We will share regular updates as new information becomes available.
- identified Oct 08, 2025, 05:01 AM UTC
We continue to work closely with Spamhaus as we await confirmation of our delisting request. We will share regular updates as new information becomes available.
- identified Oct 08, 2025, 07:07 AM UTC
We continue to work closely with Spamhaus as we await confirmation of our delisting request. We will share regular updates as new information becomes available.
- identified Oct 08, 2025, 09:07 AM UTC
We continue to work closely with Spamhaus as we await confirmation of our delisting request. We will share regular updates as new information becomes available.
- identified Oct 08, 2025, 12:47 PM UTC
We continue to work closely with Spamhaus as we await confirmation of our delisting request. We will share regular updates as new information becomes available.
- identified Oct 08, 2025, 03:03 PM UTC
We continue to work closely with Spamhaus as we await confirmation of our delisting request. We will share regular updates as new information becomes available.
- identified Oct 08, 2025, 05:00 PM UTC
We continue to work closely with Spamhaus as we await confirmation of our delisting request. We will share regular updates as new information becomes available.
- identified Oct 08, 2025, 08:33 PM UTC
We are in ongoing communication with Spamhaus and awaiting confirmation of our delisting request. We will continue to provide regular updates until this is resolved.
- identified Oct 08, 2025, 10:44 PM UTC
We continue to communicate with SpamHaus and are awaiting a response to our delisting request. Regular updates will be shared until this is resolved.
- identified Oct 09, 2025, 12:40 AM UTC
We are in ongoing communication with SpamHaus and are currently awaiting their response to our delisting request. We’ll continue to provide regular updates until the issue is fully resolved.
- identified Oct 09, 2025, 02:41 AM UTC
We’re actively engaging with SpamHaus and awaiting their reply regarding our delisting request. Regular updates will be provided until the matter is completely resolved.
- identified Oct 09, 2025, 04:39 AM UTC
We’re actively engaging with SpamHaus and awaiting their reply regarding our delisting request. Regular updates will be provided until the matter is completely resolved.
- identified Oct 09, 2025, 07:01 AM UTC
We’re actively engaging with SpamHaus and awaiting their reply regarding our delisting request. Regular updates will be provided until the matter is completely resolved.
- identified Oct 09, 2025, 09:45 AM UTC
We’re actively engaging with SpamHaus and awaiting their reply regarding our delisting request. Regular updates will be provided until the matter is completely resolved.
- identified Oct 09, 2025, 12:03 PM UTC
We’re actively engaging with SpamHaus and awaiting their reply regarding our delisting request. The next update will be provided in 4 hours or sooner if any feedback from SpamHouse is available before that time.
- identified Oct 09, 2025, 04:12 PM UTC
We’re actively engaging with SpamHaus and still waiting for their reply regarding our delisting request. The next update will be provided in 4 hours or sooner if any feedback from SpamHouse is available before that time.
- identified Oct 09, 2025, 08:31 PM UTC
Communication with Spamhaus is ongoing as we await confirmation of the delisting request. We will continue to share updates on a regular basis.
- identified Oct 09, 2025, 11:02 PM UTC
Communication with Spamhaus is ongoing as we await confirmation of the delisting request. We will continue to share updates on a regular basis.
- identified Oct 10, 2025, 02:11 AM UTC
We remain in contact with Spamhaus while awaiting confirmation of the delisting. Further updates will be provided in due course.
- identified Oct 10, 2025, 04:06 AM UTC
We remain in contact with Spamhaus while awaiting confirmation of the delisting. Further updates will be provided in due course.
- identified Oct 10, 2025, 06:11 AM UTC
We’re actively engaging with SpamHaus and awaiting their reply regarding our delisting request. Regular updates will be provided until the matter is completely resolved.
- identified Oct 10, 2025, 08:41 AM UTC
We remain in contact with Spamhaus while awaiting confirmation of the delisting. Further updates will be provided in due course.
- identified Oct 10, 2025, 10:47 AM UTC
We remain in contact with Spamhaus while awaiting confirmation of the delisting. Further updates will be provided in due course.
- monitoring Oct 10, 2025, 10:57 AM UTC
The CSOD Technology Team has successfully updated our email configuration to route traffic through an IP address in good standing. Beginning at 1200 AM Pacific Time on October 10th, UK Swimlane customers should no longer experience issues with email delivery due to Spamhaus blacklisting. All emails from Cornerstone are now expected to be delivered successfully. We will continue to closely monitor the situation for the next 48 hours before closing this incident. Background: The email delivery disruption affecting UK Swimlane customers was due to our primary email IP being blacklisted by Spamhaus, starting at approximately 3:30 PM Pacific Time on October 6th. During this period, some customers may have experienced intermittent email delivery failures, depending on their organization’s IT security policies. We appreciate your patience and will provide further updates as needed.
- monitoring Oct 11, 2025, 11:01 AM UTC
Our monitoring confirms that email delivery has remained fully stable since the IP configuration change on October 10th. No new delivery failures have been observed due to Spamhaus blacklisting, and emails are being successfully delivered. Everything continues to look good at this time. We will continue to monitor until the 48-hour window is complete. We will update you again in the next 24 hours.
- monitoring Oct 12, 2025, 11:03 AM UTC
Email delivery remains stable following the IP configuration change on October 10th. No new delivery failures have been detected, and all emails continue to be successfully delivered. We will continue to monitor the situation closely and provide an update after 24 hours.
- monitoring Oct 12, 2025, 11:04 AM UTC
Our monitoring confirms that email delivery has remained fully stable since the IP configuration change on October 10th. No new delivery failures have been observed due to the Spamhaus blacklisting, and all emails are being successfully delivered. Everything continues to look good at this time. We are continuing to monitor the system and will provide another update within the next 24 hours as we complete the 48-hour observation window.
- monitoring Oct 13, 2025, 11:19 AM UTC
Our monitoring confirms that email delivery has remained fully stable since the IP configuration change on October 10th. No new delivery failures have been observed due to the Spamhaus blacklisting, and all emails are being successfully delivered.
- monitoring Oct 21, 2025, 04:27 AM UTC
Email delivery has remained stable since October 10, with no new delivery failures related to the Spamhaus blacklisting. All emails continue to be successfully delivered, and we are maintaining close monitoring to ensure ongoing stability.
- resolved Oct 22, 2025, 07:41 PM UTC
After an extended period of monitoring, this incident is now considered resolved. Our efforts do not stop here. Cornerstone teams continue to work closely with Spamhaus and our customers to strengthen email hygiene practices, while also enhancing our email infrastructure and application safeguards to help prevent similar incidents in the future. A detailed Root Cause Analysis (RCA) will be shared within 7 to 10 business days. For additional details, please refer to the related Known Issue article: ( https://cornerstoneondemand.my.site.com/s/articles/Known-Issue-Email-Delivery-failures-for-UK-Swimlanes ).
- postmortem Nov 03, 2025, 01:49 AM UTC
**Incident Summary** On October 6, 2025, at approximately 3:15 PM PT, email delivery was disrupted for multiple customers hosted in UK swimlanes across PRD, PIL, and STG environments. The disruption occurred when the primary IP and dedicated proxy used for outbound email traffic were blacklisted by Spamhaus, resulting in delivery delays and failures for emails routed through the affected infrastructure. **Root Cause** Spamhaus identified that certain emails originating from Cornerstone’s infrastructure had reached their spam traps, indicating that outbound traffic contained unverified or inactive recipient addresses. This led to the IP and proxy being blacklisted by Spamhaus. **Resolution** Normal email delivery was restored on October 10, 2025, after traffic was rerouted to a new IP address. Additional corrective measures taken: * Implemented client-level sharding to isolate traffic and reduce the likelihood of future blacklisting. * Blocked problematic or invalid email addresses detected in outbound traffic. * Advised affected customers to review and update their mailing lists to improve email hygiene. **Planned Improvements** To strengthen email reliability and mitigate future risks: * The email infrastructure is being enhanced with sharded, multi-proxy routing to improve isolation and failover capability. * Customer segmentation and compliance controls are being reinforced to ensure that only verified and SPF/DKIM-aligned traffic is processed through the sharded infrastructure, minimizing the risk of email service disruption for customers adhering to mandatory email requirements. * Direct coordination with Spamhaus has been established, and real-time sender reputation monitoring has been implemented to proactively detect and prevent similar issues.