Mono Solutions experienced a major incident on March 15, 2023 affecting Third Party Products, lasting 55d 22h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Mar 15, 2023, 10:06 AM UTC
On 13.03.23 there has been an issue with our email provider. The issue has been solved but we are aware that some email accounts are full because of that incident and we are in active contact with the email provider to further investigate. It is our priority to solve the issue.
- monitoring Mar 20, 2023, 08:26 AM UTC
We are still monitoring for any possible issues with emails. This is an update from the email provider: "On Monday, 13/03/2023 we initiated a migration on our prodB environment. For 99% of our users the migration has been completed and you should expect a return to normal operating. We have received reports of users describing issues with duplicated email messages on their IMAP clients. This is related to the means in which IMAP clients determine whether a message has already been downloaded locally. The quickest way to address this issue is to completely remove and re-add the email account on the mail client. This ensures that any incorrect caching information kept by the mail client is discarded, and presents a consistent view of mailbox contents."
- monitoring Mar 21, 2023, 01:16 PM UTC
There are issues at the moment with the webmail login and we are waiting for an update from the email provider. We will keep monitoring the issue.
- monitoring Mar 21, 2023, 02:11 PM UTC
The issue with the webmail login has been resolved. We are still monitoring the general email issue.
- monitoring Mar 24, 2023, 01:18 PM UTC
The issues with emails should now be resolved. If the issues related to Outlook (or other email client) inbox still persists, the quickest way to solve it is to completely remove and re-add the email account on the mail client. We will keep monitoring with the email provider for any possible further issue.
- investigating Mar 30, 2023, 06:58 PM UTC
Update from the email provider: We are investigating an issue preventing some users on Cluster B from logging into Webmail. Our engineering team has been engaged. We will provide an update once we have additional information.
- monitoring Mar 30, 2023, 07:54 PM UTC
Update from the email provider: Incident resolved The engineering team identified some of the VMs along with IMAPs were identified in the stopped state and started them back up. the affected users on prod B should be able to log in with no issues as of 19:06 UTC. We are closing the incident. Incident Start Time: 03-30-2023 17:45:00 UTC Incident End Time:03-30-2023 19:06:00 UTC Total Duration:1 hour, 21 minutes We will keep monitoring
- resolved May 10, 2023, 08:31 AM UTC
We are resolving this incident. We have monitored issues related to the email provider migration and it seems that everything is operational.