INKY experienced a minor incident on March 3, 2023 affecting Inky Region 1 - Southeast US and Inky Region 2 - Eastern US and 1 more component, lasting 59m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Mar 03, 2023, 12:38 AM UTC
We are currently investigating an issue that may be causing a delay with some outbound email.
- resolved Mar 03, 2023, 01:38 AM UTC
Discovered an issue with the servers in one region that were not updating correctly causing deferrals. We fixed the issue and ensured all the deferred email was fully delivered.
- postmortem Mar 03, 2023, 03:15 PM UTC
# Post incident report: Start: 2-March-2023 2100 UTC End: 3-March-2023 0140 UTC Duration: 4 hour 40 min ## Summary: Outgoing mail to one region experienced delays due to a configuration error. ## Root Cause: On March 2nd, a configuration change was not successfully applied to one region, leading to delayed delivery of outgoing mail passing through that region. ## Mitigation Action: We resolved the configuration issue in the affected region, enabling mail to be delivered as expected. ## Customer Impact: On March 2nd, some of our customers experienced a delay of approximately five hours in receiving their email. ## Follow-up Items and Preventative Measures: To prevent similar incidents in the future, we are taking the following actions: * Increasing and fine-tuning monitoring on the outgoing servers to quickly detect and alert our teams to any issues with outgoing mail. * Improving the automation of the configuration change processes to minimize the risk of manual errors.