Mailgun experienced a minor incident on September 27, 2014, lasting 3d 15h. The incident has been resolved; the full update timeline is below.
Update timeline
- scheduled Sep 27, 2014, 02:26 AM UTC
Rackspace, our parent company and hosting provider, will be performing datacenter maintenance that may impact Mailgun's infrastructure between Sunday 9/28, 11:00AM UTC through Monday, 9/29 11:00AM UTC. We do not expect the API or SMTP endpoints to be affected. Mail delivery should be unaffected as well. Services such as events/domains/validation APIs may experience periodic network disruption during the maintenance window. Please follow this maintenance notification. We will keep you informed through our status page (http://status.mailgun.com) and Twitter (@mail_gun).
- scheduled Sep 28, 2014, 10:00 AM UTC
Maintenance will begin as scheduled in 60 minutes.
- in progress Sep 28, 2014, 11:00 AM UTC
Scheduled maintenance is currently in progress. We will provide updates as necessary.
- in progress Sep 28, 2014, 08:50 PM UTC
We are currently seeing intermittent connectivity on the API endpoint. All hands on deck working to resolve.
- in progress Sep 28, 2014, 09:52 PM UTC
We've made a few adjustments to resolve the API connectivity issues. We're continuing to monitor. Thanks for your patience.
- in progress Sep 28, 2014, 11:03 PM UTC
We're seeing higher failure rates on the API. Hang in there, we're digging in.
- in progress Sep 28, 2014, 11:49 PM UTC
API connectivity has been restored. Thanks for your patience.
- in progress Sep 29, 2014, 01:27 AM UTC
Rackspace is now beginning rolling reboots of Cloud instances throughout the ORD datacenter, where a majority of Mailgun infrastructure resides. We planned to fail over to our alternate location to minimize impact, however, we have been unable to do so. As a result, you may see connectivity issues to Mailgun's API and SMTP endpoints. We have all hands on deck to recover services immediately. We are truly sorry for the inconvenience this will cause. Please reach out to us if we can help in any way.
- in progress Sep 29, 2014, 02:00 AM UTC
We are currently aware of failures accessing Mailgun's Control Panel from Rackspace's Cloud Control Panel. We do not have an ETA on resolution. We can provide an emergency workaround if you email [email protected].
- in progress Sep 29, 2014, 03:20 AM UTC
We're speedily recovering nodes that have been rebooted. Several nodes are remaining. If a message is queued, it will be delivered. However, some delays may be present. We'll work to crunch any queues through the night. Thanks for your patience.
- in progress Sep 29, 2014, 03:56 AM UTC
Between 3:00AM UTC - 3:12AM UTC, you may see a lack of events on the Events API and Logs Control Panel tab. Messages related to the events were unaffected in their normal delivery process. We do not anticipate being able to recover these events. However, we'll keep you updated.
- in progress Sep 29, 2014, 05:34 AM UTC
A small percentage of customers with dedicated IPs may currently see messages Accepted, but not yet Delivered, within their logs. Those messages remain queued and will be released shortly. Thanks for your patience.
- in progress Sep 29, 2014, 06:47 AM UTC
The API and SMTP endpoints have been stable for an hour. We are still working through the previously mentioned dedicated IPs with messages queued. For clarity, if you have a dedicated IP in the 209.61.151.0/24 subnet, we are working on releasing those messages as quickly as we can. All other messages are delivering promptly.
- in progress Sep 29, 2014, 10:40 AM UTC
We have identified TLS connectivity issues. These are caused by errorneous F5 LB behavior of unidentified nature, resulting in randomly blocked packets causing connection timeouts. We are continuing our research.
- in progress Sep 29, 2014, 06:24 PM UTC
On Friday, Sep 26th at 4pm PDT, we were informed that Rackspace would be performing reboots of their performance cloud instances (https://status.rackspace.com/). We planned to fail over to our alternate locations to minimize the impact. We have subsequently noticed connectivity issues caused by erroneous F5 LB behavior. This is resulting in blocked packets causing connection timeouts for some customers. We are currently working with Rackspace and F5 to resolve this issue.
- in progress Sep 29, 2014, 06:42 PM UTC
SSO from Rackspace's Cloud Control Panel to Mailgun's Control Panel has been resolved. If you continue seeing issues with this functionality, please contact [email protected].
- in progress Sep 29, 2014, 07:14 PM UTC
We believe we've made substantial progress on resolving the majority of SMTP connectivity issues. We're continuing to monitor.
- in progress Sep 29, 2014, 07:40 PM UTC
Some domains utilizing an IP within the block 209.61.151.0/24, with messages queued between Sun, 28 Sep 2014 18:50:00 GMT - Mon, 29 Sep 2014 08:00:0 GMT, may experience duplicate email deliveries. We are trying to minimize this as much as possible. Thanks for your patience!
- in progress Sep 29, 2014, 10:01 PM UTC
We are continuing to work with customers reporting SMTP connectivity issues. If you're still having trouble connecting, please ensure you have a ticket open with us, so we can work with you on debugging.
- in progress Sep 30, 2014, 01:22 AM UTC
In 10 minutes, we will be failing over to an alternate load balancer. For those with persistent connections, they will be terminated, but can be re-established immediately. All other connections should continue on to the alternate load balancer without interruption.
- in progress Sep 30, 2014, 04:06 AM UTC
We've failed over Load Balancers, made additional adjustments, and are seeing success with SMTP connections. If you're still experiencing issues, please open or update your ticket, so we can gather additional details from you. Thanks!
- monitoring Sep 30, 2014, 05:05 AM UTC
At this time, we've made additional adjustments and have seen SMTP connections improve. We will be moving this status to Verifying/Monitoring to collect any late reports. In addition, we are marking all services as fully operational. We've received several requests for an RCA and will provide one within 72 hours of closing the incident. Please subscribe to the Status Page alerts to receive a notification when the RCA is published.
- monitoring Sep 30, 2014, 05:07 PM UTC
We are still seeing a very small subset of users experiencing connectivity issues, if you are still seeing these issues, please contact us.
- completed Sep 30, 2014, 06:13 PM UTC
We have resolved the final connectivity issues to the SMTP endpoint. Again, we'll prepare a full RCA within 72 hours of this point. Thanks for your patience throughout this incident!
- postmortem Aug 03, 2018, 05:30 PM UTC
A postmortem has been published on our blog. http://blog.mailgun.com/mailgun-post-mortem-september-2014/