INKY experienced a minor incident on November 3, 2022 affecting Inky Region 1 - Southeast US and Inky Region 2 - Eastern US and 1 more component, lasting 53m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Nov 03, 2022, 01:38 PM UTC
Inky is investigating a issue resulting in delays in processing messages resulting in an increase in message deferals and possible delivery delays beginning at 13:00 UTC
- monitoring Nov 03, 2022, 01:47 PM UTC
Mail processing times are returning to normal. Any mail that was deferred due to the slow down will process normally.
- monitoring Nov 03, 2022, 02:00 PM UTC
Normal error rates have returned and Inky engineers are insuring everything is properly functioning.
- resolved Nov 03, 2022, 02:32 PM UTC
Engineers have verified that the error rates are normal and all systems are functioning as expected.
- postmortem Nov 14, 2022, 06:32 PM UTC
# Post incident report: Start: 3-November-2022 1300 UTC End: 3-November-2022 1400 UTC Duration: 1 hour ## Summary: Delay in processing incoming mail for Office 365 customers for a short time due to a configuration error introduced in a fix for a previous issue. ## Root Cause: On November 2nd, we applied a fix to correct an increase in false positives for some realtime link analysis. This fix contained a configuration error that decreased our processing capacity which impacted the incoming mail during the typical morning email spike in traffic. This decreased capacity resulted in a slowdown in processing for Office 365 customers. This resulted in a delay for the final delivery of the messages for the morning of November 3rd. ## Mitigation Action: We corrected the configuration error introduced in the previous fix. Shortly after fixing the error, we saw the processing rates return to normal. ## Customer Impact: On the morning of November 3rd, some Office 365 customers had a period of approximately 30 minutes where email delivery was delayed.