Fat Zebra incident

Live: System migration to new data centre

Resolved after 9 days 6 hours, on March 3, 2016.

completed Mar 12, 2016, 05:52 PM UTC

All services are now restored and monitoring indicates everything is working as expected. Search indexing has been kicked off and should be completed in about 3 hours. If any merchants are experiencing issues following this change please contact [email protected] for assistance.

Started
Mar 03, 2016, 11:44 AM UTC
Resolved
Mar 12, 2016, 05:52 PM UTC
Duration
9d 6h
Detected by Pingoru
Mar 03, 2016, 11:44 AM UTC

Update timeline

  1. scheduled Mar 03, 2016, 11:44 AM UTC

    As a part of our system upgrades Fat Zebra are migrating Live services into a new data centre which will improve capacity, speed and reliability. In order to make this change we will be transitioning in two stages: * Migration of the Sandbox environment and verification - this has been completed and problems discovered during testing have been remediated. * Migration of the Live environment and verification - detailed in this notice. These migrations will involve a database migration which will result in downtime of approximately 1 hour. Following this indexes for searching in the Merchant Dashboard may also be out of date, however where required we will reindex newest records first moving backwards 3 months at a time. Following these changes the public IP addresses of Fat Zebra Live endpoints will be updated to direct traffic to the new servers. If your system whitelists IP addresses please contact [email protected] for the new addresses prior to this change. Further details around service interruption times will be announced during the change window. If there are any questions with regards to this change please contact [email protected].

  2. scheduled Mar 12, 2016, 12:00 PM UTC

    Maintenance will begin as scheduled in 60 minutes.

  3. scheduled Mar 12, 2016, 12:29 PM UTC

    Our change begins in approximately 30 minutes. Estimates from dress rehearsals during the week indicate that the data migration may take up to 2.5 hours which means that Gateway services will be unavailable for between 2 and 3 hours during this period. We will announce when the Gateway is being shutdown via this status page and Twitter.

  4. in progress Mar 12, 2016, 01:00 PM UTC

    Scheduled maintenance is currently in progress. We will provide updates as necessary.

  5. in progress Mar 12, 2016, 01:03 PM UTC

    The Gateway will be switched into maintenance mode while we begin migrating our databases to the new servers. Following this the Gateway will remain in maintenance mode until we switch across to our new data centre. Updates to follow.

  6. in progress Mar 12, 2016, 03:05 PM UTC

    The migration is progressing as expected. Currently we are verifying database indexes and testing bank connectivity prior to re-enabling the gateway services. Further updates to follow.

  7. monitoring Mar 12, 2016, 04:48 PM UTC

    Gateway services have been restored and we are currently performing additional verification and finishing changes required to restore services such as the merchant dashboard, hosted payment pages etc. Further updates to follow.

  8. completed Mar 12, 2016, 05:52 PM UTC

    All services are now restored and monitoring indicates everything is working as expected. Search indexing has been kicked off and should be completed in about 3 hours. If any merchants are experiencing issues following this change please contact [email protected] for assistance.

Your product runs on more than Fat Zebra

Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.

Add your stack →