Mail not being sent or received in some SEDNA tenants (Virginia 1 Cluster)

Update timeline

1. investigating Jun 28, 2023, 06:38 PM UTC

   We are currently experiencing an issue affecting some mail being received into or sent from some SEDNA tenants in the Virginia1 Cluster region. Our Engineering team is actively working to resolve this and all pending mail will be sent and received once the incident is resolved.

2. investigating Jun 28, 2023, 06:54 PM UTC

   We are continuing to investigate this issue.

3. investigating Jun 28, 2023, 07:44 PM UTC

   We are continuing to investigate this issue. We apologize for any inconvenience this may cause and thank you for your patience while we work on getting this resolved.

4. identified Jun 28, 2023, 08:25 PM UTC

   We've identified a potential cause for this issue and are working to rectify.

5. identified Jun 28, 2023, 08:58 PM UTC

   The issue has been identified and a fix is in progress.

6. monitoring Jun 28, 2023, 09:20 PM UTC

   A fix has been implemented and we are monitoring the results.

7. resolved Jun 28, 2023, 09:35 PM UTC

   This incident has been resolved. A post-incident report will be shared in due course.

8. postmortem Jul 04, 2023, 07:21 PM UTC

   # Summary On the 28th June, 2023, the SEDNA Platform experienced a partial loss of service. A number of customers served by the North American server infrastructure were not able to view their recently sent messages or see new messages in their Inbox. We deeply regret the inconvenience caused during the five-hour and four-minute duration of this issue. ‌ We want to assure you that no messages were lost during this incident. However, there was a delay in processing both inbound and outbound messages as some sent messages were temporarily held in a staging queue, which were then delivered correctly at the end of the incident window. ‌ # Remediation and Prevention ‌ An incident of this nature receives Sedna’s highest level of scrutiny to ensure we can provide our customers with full confidence in the system. Following the incident the team conducted a series of follow ups: ‌ * Engineering Retro - documentation of the specific actions which caused the incident, the remediation, and the alerts through out to identify where / how improvements are needed * Engineering / CS Coordination Retro - confirmation of coordination between Support and Engineering to ensure we were providing the best possible support to our customers given the difficult situation * Leadership Team Retro - review with Sedna leadership on the issue, and specific actions which will be taken to ensure it is not repeated ‌ As a result of the reto’s listed above. SEDNA has taken specific action to ensure we reduce the likelihood of incidents like this in the future, and has developed a full plan of action Those actions include: ‌ * A full detailed code review of the issue in question to identify the offending sections, how the code was developed, and what requirements were being targeted which produced such an error * A full detailed review of the QA process for the offending code to determine if additional measures could / should have been taken to ensure such a bug cannot be introduced in the future * A full review of the queueing and alerting systems used to identify similar issues * A review of potential rollback procedures to include analysis of queue impacts so that we might faster resolve any similar issues in the future * A scheduled review of our autotagging specific development process to reduce the risk of inappropriate tagging and improved testing procedures * Consideration of enhancements to improve the queue processing approach that impacts this area of service ‌ This is only an initial list of actions taken. As we work through the above we expect additional specific actions to be identified and remedied. ‌ # What you can expect from SEDNA ‌ We understand the critical nature of the services SEDNA provides your business. We will continue to communicate with customers to answer any questions and ensure we do our best to provide a seamless customer experience. We apologize for any issues these events may have caused. ‌ Please reach out directly to SEDNA Support \([[email protected]](mailto:[email protected])\) with any questions.