SEDNA Outage History

An initial investigation has been completed into the outage that affected some Sedna customers on our Dublin 1 and Dublin 2 clusters earlier today. The disruption was traced to a capacity issue that prevented impacted users from logging in or accessing the platform. As a precautionary measure, we have adjusted scaling policies on both clusters and temporarily disabled a non-essential feature to reduce load. Our engineering team is continuing the root cause analysis and will implement further improvements to prevent recurrence.

An application outage was identified affecting some Sedna customers on our Dublin 2 cluster. During this time, impacted users were unable to log in or access the platform. The issue was treated as an emergency by our engineering team and has now been resolved. A root cause analysis is currently underway, and Sedna’s engineering team is actively investigating to determine the underlying cause and prevent recurrence.

This morning (between approximately 10:10 and 10:40 UTC), we experienced a performance degradation of email ingress and egress in our Dublin2 and Dublin3 clusters. This issue was brought to our attention by automated alerting built on our internal observability tooling. Application scaling policies increased capacity during this period, reducing its overall length. This was a repeated issue, seen previously on 16th March, caused by abnormally expensive processing of a set of malformed emails sent to a large number of recipients. Mitigation steps we have taken so far have unfortunately not been sufficient to avoid a recurrence, and as such, we will continue to investigate to improve resilience.

The issue causing elevated CPU load in the Dublin2 cluster has been identified and mitigated. We have implemented measures to prevent recurrence. All services have resumed and should now be operating normally.

Correct measures was put in place once delay was noted by our engineers. This has been monitored and we can confirm that service has been restored and is working as expected.

Earlier today (from 11:31 to 12:09 UTC) we experienced a performance degradation of email ingress and egress globally across tenants in our public clusters. This was caused by abnormally expensive processing of a set of malformed emails sent to a large number of recipients. We will continue to monitor and investigate how our systems can be updated to improve resilience.

This incident has been resolved, and all back logged messages have been processed