Datto incident

INKY - Internally sent emails are not getting delivered and emails to external recipients are getting delayed

Major Resolved View vendor source →

Datto experienced a major incident on June 19, 2026 affecting EU, lasting 3h 19m. The incident has been resolved; the full update timeline is below.

Started
Jun 19, 2026, 10:03 AM UTC
Resolved
Jun 19, 2026, 01:22 PM UTC
Duration
3h 19m
Detected by Pingoru
Jun 19, 2026, 10:03 AM UTC

Affected components

EU

Update timeline

  1. investigating Jun 19, 2026, 10:03 AM UTC

    The INKY R&D Team is investigating the issue where emails to internal recipients filtered through INKY are not showing up in INKY and not getting delivered to recipients. Emails to external recipients, while also not appearing in INKY, successfully get delivered but with a noticeable delay. Please subscribe to this incident to receive update notifications as they become available.

  2. identified Jun 19, 2026, 10:22 AM UTC

    The issue has been identified and a fix is being implemented.

  3. monitoring Jun 19, 2026, 10:30 AM UTC

    A fix has been implemented and we are monitoring the results.

  4. monitoring Jun 19, 2026, 12:33 PM UTC

    The service has caught up with processing through items queued in INKY, and the R&D team is reviewing individual examples of residual issues . Please work with our support team and provide them with message IDs for review if the issue appears to still persist.

  5. resolved Jun 19, 2026, 01:22 PM UTC

    This incident has been resolved.

  6. postmortem Jul 01, 2026, 03:53 PM UTC

    **Inky Public Root Cause Analysis** **EU Central Email Processing Delay** **Summary** On the date of the incident, Inky experienced a disruption to email processing within the EU Central region. Increased inbound email volume coincided with a failure of infrastructure autoscaling, preventing additional processing capacity from being automatically provisioned. As a result, email messages queued for processing and experienced delivery delays. Service was restored after infrastructure capacity was manually increased, allowing queued messages to be processed and normal throughput to resume. **Customer Impact** Customers whose email traffic was processed through the EU Central region experienced delays in email analysis and delivery. **Impact Details** * Service affected: Email Processing * Region affected: EU Central * Impact start: 08:15 UTC * Service restoration: 10:00 UTC * Total duration: Approximately 105 minutes * Maximum observed delay: Up to approximately 90 minutes for some messages No customer data was lost during the incident. Queued messages were retained and processed once capacity was restored. **What Happened** During a normal increase in email traffic, the platform required additional processing capacity. The automatic scaling mechanism responsible for adding resources did not successfully provision additional servers due to a configuration issue within the scaling configuration. Because additional capacity was not added as expected, processing resources became constrained and email messages began accumulating in processing queues, resulting in delivery delays. **Resolution** The operations team investigated the processing delays and determined that additional processing capacity was required. Infrastructure resources in the affected region were manually scaled to meet demand. As the additional resources came online, queue depth decreased and processing throughput returned to normal levels. All queued messages were subsequently processed and delivered. **Root Cause** The incident was caused by a configuration condition within the autoscaling setup that prevented automatic expansion of processing capacity when demand increased. A previous capacity adjustment increased the desired resource level but did not include a corresponding increase to the maximum scaling limit. When additional capacity was required, the autoscaling process was unable to complete the scale-up operation, preventing new resources from being added automatically. **Preventative Actions** To reduce the likelihood of similar incidents, the following corrective actions are being implemented: 1. **Improve autoscaling configuration safeguards** 1. Infrastructure templates are being updated so scaling limits are automatically aligned with required capacity settings, reducing the possibility of configuration mismatches. 1. **Enhance monitoring and alerting** 1. New monitoring will directly detect and alert on autoscaling failures, enabling faster identification and response when scaling operations do not complete successfully. 1. **Accelerate operational response** 1. Alerting and operational procedures will be updated to provide faster, more direct mitigation guidance when autoscaling failures occur. **Final Status** The issue has been fully resolved, and normal email processing operations have been restored. Additional monitoring and infrastructure improvements are being implemented to strengthen resiliency and improve detection of similar conditions in the future. **Incident Duration:** 08:15 UTC – 10:00 UTC \(105 minutes\)` `**Affected Service:** Inky Email Processing \(EU Central Region\)