Cornerstone experienced a notice incident on July 22, 2025 affecting Response Time and Response Time and 1 more component, lasting 17d 19h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jul 22, 2025, 12:06 AM UTC
The UK region's production, pilot, and stage swimlanes are currently experiencing some issues with the email server resulting in email delivery failures. This is our top priority and we are working closely with the vendor to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.
- identified Jul 22, 2025, 02:19 AM UTC
We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.
- identified Jul 22, 2025, 04:10 AM UTC
We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.
- identified Jul 22, 2025, 06:47 AM UTC
We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.
- identified Jul 22, 2025, 12:10 PM UTC
We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.
- monitoring Jul 22, 2025, 03:32 PM UTC
Email delivery is now restored and any newly triggered emails are being delivered successfully. We will continue closely monitoring activity to ensure stability. Please also refer to the related Known Issue page, "Known Issue: Email Issues for UK Swimlanes" ( https://cornerstoneondemand.my.site.com/s/articles/Known-Issue-Email-Issues-for-UK-Swimlanes ) for additional details.
- monitoring Jul 22, 2025, 04:57 PM UTC
We are continuing to monitor for any further issues.
- identified Jul 24, 2025, 10:58 AM UTC
While the overall email delivery service for the UK region's production, pilot, and stage swimlanes customers remain unimpacted, customers sending emails specifically to the Microsoft Email Domains, may experience some issues with the email deliveries. We are closely working with the Microsoft Teams to address the email delivery delay issues. Please also refer to the related Known Issue page, "Known Issue: Email Issues for UK Swimlanes" ( https://cornerstoneondemand.my.site.com/s/articles/Known-Issue-Email-Issues-for-UK-Swimlanes ) for additional details. This is our top priority and we are working closely with the vendor to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.
- identified Jul 24, 2025, 01:24 PM UTC
We are continuing to work closely with the vendor to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available.
- identified Jul 24, 2025, 03:53 PM UTC
We are continuing to work closely with Microsoft to address the email delivery delays. We will continue to share regular updates until this is resolved. This is our top priority. We appreciate your continued patience.
- monitoring Jul 24, 2025, 05:42 PM UTC
Emails are flowing successfully again. Microsoft lifted the throttling limitation and the queues have cleared. We will continue closely monitoring activity to ensure stability. Please also refer to the related Known Issue page, "Known Issue: Email Issues for UK Swimlanes" ( https://cornerstoneondemand.my.site.com/s/articles/Known-Issue-Email-Issues-for-UK-Swimlanes ) for additional details.
- monitoring Jul 31, 2025, 01:13 AM UTC
Email delivery continues to flow successfully, and we are closely monitoring activity to ensure sustained stability.
- resolved Aug 08, 2025, 07:38 PM UTC
After an extended period of monitoring, this incident is now considered resolved. That said, our efforts do not stop here. We’ve implemented enhanced, around-the-clock monitoring alerts to proactively detect potential issues. In parallel, we continue working closely with customers to improve email hygiene practices and are actively enhancing our email infrastructure and application to help safeguard against similar incidents in the future. A full RCA will be shared next week.
- postmortem Aug 16, 2025, 05:32 AM UTC
**Issue Summary:** On 19 July 2025, internal monitoring tools alerted Cornerstone engineers to email delivery issues impacting the UK AWS environment. Outbound emails were being sent without complete or correct SPF \(Sender Policy Framework\) record configurations. This misconfiguration resulted in failed authentication checks, triggering Spamhaus to list one of our primary sending IP address. The listing caused email delivery failures for UK recipients, impacting customer communication and business operations. **Root Cause:** Missing or incomplete SPF record configuration for outbound email servers in the UK environment, leaving the sending IP vulnerable to blacklisting by Spamhaus. **Corrective Action:** The incident team removed the affected IP from active sending pools. Immediate remediation steps included sanitizing recipient lists, adding missing IPs to SPF records, and blocking high-volume senders to mitigate further impact. **Preventive Measures:** To avoid recurrence, Cornerstone has initiated the following: SPF Record Compliance – Add all active sending IPs to their respective SPF records and verify configurations across environments. Email Verification – Cornerstone will work on enforcing verification of email addresses on self-registration pages to reduce the risk of invalid or malicious sign-ups impacting deliverability.