Cornerstone experienced a notice incident on July 19, 2025 affecting Response Time and Response Time and 1 more component, lasting 2d 5h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Jul 19, 2025, 01:38 PM UTC
The UK region's production, pilot, and stage swimlanes are currently experiencing some issues with the email server resulting in the email delivery failures. This is our top priority and we are working to resolve the problem as soon as possible. Please check back periodically for additional updates, which will be posted as they become available. Issue start time: 4:50 AM PST.
- investigating Jul 19, 2025, 03:29 PM UTC
We are continuing to investigate this issue as our top priority. We will continue to share regular updates until the issue is fully resolved. Thank you for your continued patience and understanding.
- investigating Jul 19, 2025, 05:22 PM UTC
We are continuing to investigate this issue as our top priority. We will continue to share regular updates until the issue is fully resolved. Thank you for your continued patience and understanding.
- investigating Jul 19, 2025, 07:39 PM UTC
We are continuing to investigate this issue as our top priority. We will continue to share regular updates until the issue is fully resolved. Thank you for your continued patience and understanding.
- investigating Jul 19, 2025, 09:33 PM UTC
We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.
- investigating Jul 19, 2025, 11:32 PM UTC
We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.
- investigating Jul 20, 2025, 01:31 AM UTC
We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.
- investigating Jul 20, 2025, 03:41 AM UTC
We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.
- investigating Jul 20, 2025, 05:44 AM UTC
We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.
- investigating Jul 20, 2025, 08:31 AM UTC
We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.
- investigating Jul 20, 2025, 10:47 AM UTC
We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.
- investigating Jul 20, 2025, 12:35 PM UTC
We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.
- investigating Jul 20, 2025, 02:45 PM UTC
We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.
- investigating Jul 20, 2025, 04:56 PM UTC
We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.
- investigating Jul 21, 2025, 01:08 AM UTC
We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.
- investigating Jul 21, 2025, 03:09 AM UTC
We are continuing to investigate this issue as our top priority and will continue to share regular updates until it is fully resolved. Thank you for your continued patience and understanding.
- investigating Jul 21, 2025, 05:12 AM UTC
This issue remains our top priority, and we are actively investigating it. We expect an update from our vendor within the next ~4 hours and will continue to provide regular updates until the matter is fully resolved. We sincerely appreciate your continued patience and understanding.
- investigating Jul 21, 2025, 07:03 AM UTC
This issue remains our top priority, and we continue to actively monitor the situation. We are currently awaiting further input from our vendor, who has not yet provided an expected update. We understand the importance of this matter and will share any new information as soon as it becomes available. We appreciate your continued patience and understanding.
- resolved Jul 21, 2025, 07:17 PM UTC
Email delivery is now fully restored and any newly triggered emails are being delivered successfully. We will continue closely monitoring activity to ensure stability. A detailed RCA will be provided within 7 to 10 business days. In the meantime, please refer to the related Known Issue page, "Known Issue: Email Issues for UK Swimlanes" ( https://cornerstoneondemand.my.site.com/s/articles/Known-Issue-Email-Issues-for-UK-Swimlanes ) for additional details.
- postmortem Aug 16, 2025, 05:31 AM UTC
**Issue Summary:** On 19 July 2025, internal monitoring tools alerted Cornerstone engineers to email delivery issues impacting the UK AWS environment. Outbound emails were being sent without complete or correct SPF \(Sender Policy Framework\) record configurations. This misconfiguration resulted in failed authentication checks, triggering Spamhaus to list one of our primary sending IP address. The listing caused email delivery failures for UK recipients, impacting customer communication and business operations. **Root Cause:** Missing or incomplete SPF record configuration for outbound email servers in the UK environment, leaving the sending IP vulnerable to blacklisting by Spamhaus. **Corrective Action:** The incident team removed the affected IP from active sending pools. Immediate remediation steps included sanitizing recipient lists, adding missing IPs to SPF records, and blocking high-volume senders to mitigate further impact. **Preventive Measures:** To avoid recurrence, Cornerstone has initiated the following: SPF Record Compliance – Add all active sending IPs to their respective SPF records and verify configurations across environments. Email Verification – Cornerstone will work on enforcing verification of email addresses on self-registration pages to reduce the risk of invalid or malicious sign-ups impacting deliverability.