Spreedly experienced a notice incident on August 8, 2014, lasting 31m. The incident has been resolved; the full update timeline is below.
Update timeline
- scheduled Aug 08, 2014, 02:38 PM UTC
We are going to do a planned migration of our outbound IP - the IP we use to talk to external API's (e.g. gateways and receivers) - to a new firewall. The IP will not change - that's the reason we're migrating - so no gateway whitelists will need to be updated. The goal is for this change to have no visible impact on customers following our recommendations for timeouts (https://docs.spreedly.com/reference/api/v1/timeouts/). To this end we will pause inbound connections, allow established connections to drain, move the IP between our firewalls, and resume connections. Ideally this will complete well within the 60 second standard TCP timeout. Even though we've done multiple dry runs of this process, it's of course possible for things to not go as planned, in which case we have rollback plans already queued up. If you have any questions, please don't hesitate to contact us at [email protected].
- completed Aug 08, 2014, 03:09 PM UTC
There was about 2-3 minutes of failed outbound gateway traffic due to a missed step in our migration - something we'll address next time - but the migration is now complete. We apologize for any disruption, and thank you for your patience with us as we continue to improve our infrastructure!
- postmortem Aug 03, 2018, 05:30 PM UTC
We completed the majority of the change needed - reconfiguring the receiving firewall to handle the handed off IP address block - within our target window of 60 seconds. However, there was a necessary change to our app server routing which - while prepped and queued - got missed, which meant that outbound connections to gateways began hanging. **There were no established gateway connections interrupted, however.** Gateway interaction requests will have received an "Unable to communicate with the gateway" or similar message back from us for the 2-3 minute window before we figured out why our status tools were telling us connections weren't going through and ran the missed command. The biggest takeaway from this issue was something we already knew but haven't been able to address yet: we have one key piece of our infrastructure that uses a web GUI for configuration, and we need to get it replaced with a properly scriptable solution. In this case it bit us since we couldn't script the whole of today's migration end to end - ironically, the piece that got missed was the one thing we could script, since we were so focused on getting through the GUI changes. We already know how we will replace this piece of the infrastructure, and that work is queued up with a high priority.