Jira Product Discovery experienced a notice incident on October 24, 2023 affecting Open projects and Create and delete fields and 1 more component, lasting 1d 5h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 24, 2023, 11:27 PM UTC
We are investigating an issue with emails that have been sent out to admins stating domain verification is failing. Affected users may see emails sent from Atlassian's default domain instead of the custom domain they have configured We will provide more details within the next hour.
- identified Oct 25, 2023, 12:37 AM UTC
We continue to work on resolving the issues with emails that have been sent out to admins stating domain verification is failing for Jira Work Management, Jira Service Management, Jira Software, and Jira Product Discovery. We have identified the root cause and expect recovery shortly.
- monitoring Oct 25, 2023, 01:46 AM UTC
We have identified the root cause of the issues with emails that have been sent out to admins stating domain verification is failing for Jira Work Management, Jira Service Management, Jira Software, and Jira Product Discovery. A fix is currently being implemented, and we expect this issue to be resolved shortly.
- monitoring Oct 25, 2023, 06:02 AM UTC
The fix for the issue has been rolled out and is expected to be complete by 3 am UTC, 26 October. In the meantime, we have refreshed all affected domains and the service is expected to be working normally. We will continue to monitor this issue for 24 hours to ensure it has been resolved and will share an update once confirmed.
- resolved Oct 26, 2023, 04:37 AM UTC
Between 10-23-2023 21:00 UTC and 10-26-2023 03:00 UTC, some customers received emails stating that the domain verification was failing for Jira Work Management, Jira Service Management, Jira Software, and Jira Product Discovery. The issue has been resolved and the service is operating normally. Additional details about the issue will be shared in our Post Incident Review
- postmortem Nov 02, 2023, 06:05 AM UTC
### **SUMMARY** From 9pm UTC on October 23, 2023, until 8pm UTC on October 24, 2023, Atlassian customers using the Jira family products that have configured notifications from custom email domains experienced a degradation as email notifications were being sent from the default [atlassian.net](http://atlassian.net) domain instead of a custom domain. The incident was detected by Atlassian customers as they started to receive notifications from the [atlassian.net](http://atlassian.net) domain instead of previously configured custom domains. It was mitigated by excluding redundant validations from the custom domain verification job which put Atlassian systems into a known good state. The total time to resolution was about 23 hours. ### **IMPACT** The overall impact was between October 23, 2023 9pm UTC and October 24 8pm UTC on Jira family products, including Jira Service Management, Jira Software and Jira Work Management. The incident caused service disruption to customers who have notifications from custom email domains enabled for their Jira products as email notifications were being sent from the default [atlassian.net](http://atlassian.net) domain instead of the custom domain configured by the customer. ### **ROOT CAUSE** The issue was caused by a change to the service responsible for custom domain email notifications lifecycle that inadvertently enabled SPF record validation as part of a daily validation job. As a result, previously working custom domain configuration transitioned to an unverified state for some Atlassian customers, resulting in notifications from Jira products being sent from [atlassian.net](http://atlassian.net) domain instead of a previously configured custom domain. ### **REMEDIAL ACTIONS PLAN & NEXT STEPS** We are continuously improving our system's resiliency. We are prioritizing the following improvement actions to avoid repeating this type of incident: * Improve monitoring and alerting for custom domains lifecycle to detect such regressions before they reach production. We apologize to customers whose services were impacted during this incident; we are taking immediate steps to improve the platform’s performance and availability. Thanks, Atlassian Customer Support