CodeTwo experienced a minor incident on December 5, 2022 affecting Mail flow and Signature adding (rules processing), lasting 56m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Dec 05, 2022, 05:33 PM UTC
We are currently investigating a problem with Azure CosmosDB availability in North Central US. Some users may experience delays in email delivery. We are working on identifying the problem and fixing it ASAP.
- identified Dec 05, 2022, 05:46 PM UTC
The issue has been identified and a fix is being implemented.
- monitoring Dec 05, 2022, 05:58 PM UTC
A fix has been implemented and we are seeing signs of recovery. The problem should be mitigated soon. We are monitoring the results.
- resolved Dec 05, 2022, 06:30 PM UTC
This incident is now resolved. All emails have been delivered but a small percentage might have been bereft of signatures due to database availability issues (if this concerns your tenant, we will notify you via email with more details within 48 hours). We will also provide a public RCA within 48 hours after we fully analyze the problem. The RCA will be provided as an update of this incident here. Please accept our apologies for all inconveniences.
- postmortem Dec 08, 2022, 11:12 AM UTC
**What happened?** Between 17:55 UTC and 18:40 UTC on 05 Dec 2022, a subset of customers in North Central US may have experienced issues with emails not getting signatures or being slightly delayed. This did not impact email deliverability in any way. **What went wrong and how we responded** We observed a problem with Microsoft’s SQL database availability in North Central US, which affected automatic scaling. We immediately switched to secondary services in the region but that did not fully remediate the issue, so we manually rescaled the SQL database. **What are the next steps?** We are investigating the database availability problem with Microsoft to make sure we’re no longer affected by such incidents on their side. We’ll also deploy custom scaling policies for the SQL database to replace standard mechanisms and ensure similar issues don’t happen in the future.