Mailgun experienced a major incident on February 27, 2015, lasting 6d 19h. The incident has been resolved; the full update timeline is below.
Update timeline
- scheduled Feb 27, 2015, 11:30 PM UTC
Rackspace, our parent company, will be performing critical maintenance for its Cloud infrastructure starting next week: https://community.rackspace.com/general/f/53/p/4978/11947#11947. Parts of Mailgun infrastructure are deployed in Rackspace's Next Gen Cloud so will be affected by this maintenance. To avoid service interruption we will be switching traffic over completely to another datacenter. The maintenance for the affected region, ORD, is scheduled for Wednesday/Thursday next week so we will perform the switch no later than Monday. We anticipate our core services (sending/receiving, batch sending, mailing lists) to remain fully functional; some auxiliary services (events, stats, etc.), however, may be glitchy. During the maintenance we'll be providing updates via StatusPage.
- scheduled Mar 03, 2015, 12:27 AM UTC
We successfully completed our traffic migration in anticipation of the upcoming maintenance earlier today. The majority of customers' traffic is now flowing through our second datacenter in IAD while the rest of the traffic is slowly bleeding off of ORD. All systems are currently operating normally, if you are experiencing any issues, please feel free to reach out to Mailgun support anytime. We'll be providing further updates via StatusPage as they become available.
- in progress Mar 05, 2015, 05:55 AM UTC
Scheduled maintenance is currently in progress. We will provide updates as necessary.
- in progress Mar 05, 2015, 11:21 AM UTC
We are currently investigating issues with logs in the control panel.
- in progress Mar 05, 2015, 01:05 PM UTC
We are currently investigating issues with domains tab, logs tab, campaigns tab and suppressions tab in the control panel.
- in progress Mar 05, 2015, 03:09 PM UTC
We are still investigating the reported issues in the Control panel and are looking into issues with some of our API end points failing. At this time message sending is not affected.
- in progress Mar 05, 2015, 03:40 PM UTC
We have resolved the reported control panel issues but will monitor for any changes while we continue work on the issues with some of the API endpoints failing.
- in progress Mar 05, 2015, 07:22 PM UTC
We are currently experiencing issues with logs and counters being delayed. This means that events may not show up for a while in the logs tab of the control panel and event counters are similarly lagging behind. We are still monitoring the control panel for any changes.
- in progress Mar 05, 2015, 11:50 PM UTC
We are currently investigating an issue with the Logs tab and Events API reporting errors.
- monitoring Mar 06, 2015, 01:08 AM UTC
The issue with the Logs tab and Events API have been resolved. Logs and counters should also be processing at a normal speed as well. We will continue monitoring for any lingering issues.
- completed Mar 06, 2015, 06:54 PM UTC
The scheduled maintenance has been completed.