INKY experienced a notice incident on December 23, 2025 affecting Inky Region 1 - Southeast US and Inky Region 2 - Eastern US and 1 more component, lasting 7h 9m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Dec 23, 2025, 02:14 PM UTC
Around 12:50 AM EST, INKY's outbound servers in one region experienced errors that made it impossible for them to accept outbound mail. During this time any attempts to hit this set of servers would result in a deferral where the sending server would be asked to try again. The retry would hit servers that were not experiencing issues, and the email would be processed normally but with a delay of a few minutes - dependent on how often the sending server was retrying. At ~7:40 AM EST, more servers hit these same errors reducing the number of servers able to accept the outbound mail to a much smaller number resulting in the possibility of outbound delays at this time going up until the issue was resolved across all servers at 8:20 AM EST. During this time, we also uncovered an issue with the alerting tied to this particular error with the outbound servers that was remedied as well. We are still actively monitoring this issue to ensure everything is running as expected.
- monitoring Dec 23, 2025, 02:38 PM UTC
The initial fix we put in place did not remedy the issue and it is possible that some mail delays could still occur. Our engineers are working diligently to address the issue. We will continue to update as necessary.
- resolved Dec 23, 2025, 09:24 PM UTC
Everything running smoothly as expected and has been have we monitored throughout the day. Fixes now in place to correct issues tied to alerting and to prevent similar failures from occurring again.