INKY incident

Currently experiencing delays in processing affecting Outbound Mail Protection and Internal Mail Protection

Notice Resolved View vendor source →

INKY experienced a notice incident on February 6, 2024 affecting Inky Region 1 - Southeast US and Inky Region 2 - Eastern US and 1 more component, lasting 1d 1h. The incident has been resolved; the full update timeline is below.

Started
Feb 06, 2024, 07:39 PM UTC
Resolved
Feb 07, 2024, 08:45 PM UTC
Duration
1d 1h
Detected by Pingoru
Feb 06, 2024, 07:39 PM UTC

Affected components

Inky Region 1 - Southeast USInky Region 2 - Eastern USInky Region 3 - Northwest US

Update timeline

  1. investigating Feb 06, 2024, 07:39 PM UTC

    Currently investigating processing delays impacting Outbound Mail protection and Internal Mail Protection

  2. investigating Feb 06, 2024, 09:45 PM UTC

    We are continuing to investigate the issue. If you are impacted by the delay, you can disable the IPW Outbound Rule (the Priority 0 Rule) in your transport rules to bypass Inky for the outbound emails.

  3. investigating Feb 06, 2024, 10:31 PM UTC

    We have reached out to Microsoft to attempt to determine what is causing what appears to be an issue affecting mail sending from Office 365. Multiple services outside Inky appear to be affected by connection problems from Office 365 that are resulting in delays of message sending

  4. monitoring Feb 06, 2024, 11:13 PM UTC

    Error rates have returned to normal. We continue to seek an update from Microsoft on what led to the increase in connection failures from the Exchange Online system.

  5. monitoring Feb 07, 2024, 12:07 PM UTC

    We will continue to closely monitor the errors from Microsoft servers throughout the day while we seek an explanation from them for the connection issues they were experiencing.

  6. monitoring Feb 07, 2024, 02:34 PM UTC

    Stability has continued from Microsoft since the errors stopped yesterday. Out of an abundance of caution, we are continuing to monitor throughout the day. We do feel that any team that had disabled the IPW Outbound transport rule should re-enable the rule to resume the Banner Stripping and Outbound Protection features of INKY.

  7. resolved Feb 07, 2024, 08:45 PM UTC

    Not seeing any reoccurrence of the errors we were seeing with Microsoft. Delivery has been steady throughout the day without issue. Closing out this incident.