Aptoma incident

Urgent critical service affecting maintenance for DrFront V3, DrMobile, DrPublish and DrVideo

Minor Resolved View vendor source →

Aptoma experienced a minor incident on October 1, 2014, lasting 11h 29m. The incident has been resolved; the full update timeline is below.

Started
Oct 01, 2014, 12:45 PM UTC
Resolved
Oct 02, 2014, 12:15 AM UTC
Duration
11h 29m
Detected by Pingoru
Oct 01, 2014, 12:45 PM UTC

Update timeline

  1. scheduled Oct 01, 2014, 12:45 PM UTC

    During this urgent maintenance window DrFront V3, DrMobile, DrPublish and DrVideo is expected to become unavailable for content production for about 10 to 20 minutes. DrFront V2 will not be affected. Distribution of content to end users will not be impacted, but production of new content will not be possible. It is expected that this situation will occur some time after 18:00 CEST (Oslo time), and we will get back with more specific information about scheduled time for start. We apologize for the inconvenience this may cause you and the late notice. Security is one of our primary concerns and we thank you for your understanding and patience during this important maintenance. Progress and status updates will be posted to Twitter at https://twitter.com/aptomaops and at http://status.aptoma.no/ . Questions? Contact us at http://aptoma.com/support .

  2. scheduled Oct 01, 2014, 02:40 PM UTC

    Background information: A critical security flaw has recently been identified in the Xen Hypervisor technology, which powers cloud providers like Amazon, Rackspace and IBM Softlayer. To patch and apply this update a system reboot for the host server which runs the virtual cloud servers is necessary. Amazon and Rackspace have executed necessary actions in the previous days, and today, on a very short notice, we got informed that IBM Softlayer is also performing the same actions. Unfortunately IBM Softlayer support couldn’t provide exact time for impact or if only specific servers are impacted or everyone. We are therefore informing about all potential affected services. All we know as of know is that work is scheduled to be finalized region for region, starting in their Singapore location at 17:00 CEST (Oslo time), followed by Toronto and then Amsterdam, which is the location for the affected Aptoma services. We will communicate status changes as soon as new information is at hand.

  3. scheduled Oct 01, 2014, 04:15 PM UTC

    All services are still functioning normally. The provider has revised the maintenance schedule. Based on this information it is estimated that Aptoma services could be affected at earliest 22:30 CEST (Oslo time). We will get back with more specific estimates as soon as new information is at hand, at latest at 20:00 CEST.

  4. scheduled Oct 01, 2014, 06:04 PM UTC

    All services are still functioning normally. Current information at hand indicates that Aptoma services could become impacted some time between approximately 22:00 CEST and 23:59 CEST. We will get back with more specific estimates at 21:00 CEST.

  5. scheduled Oct 01, 2014, 07:10 PM UTC

    All services are still functioning normally. Previously issued estimates are still valid. Next update at 22:15 CEST.

  6. scheduled Oct 01, 2014, 08:06 PM UTC

    All services are still functioning normally. We expect the scheduled maintenance to impact Aptoma services some time after 23:00 CEST. Next update at 23:00 CEST.

  7. in progress Oct 01, 2014, 09:02 PM UTC

    The service affecting maintenance have now started. Distribution of content to end users will not be impacted, but content production systems will soon become inaccessible. We will post status updates every 30 minutes and when new relevant information is at hand.

  8. in progress Oct 01, 2014, 09:34 PM UTC

    Services are currently affected. We will post status updates every 20 minutes from now.

  9. in progress Oct 01, 2014, 09:44 PM UTC

    Services are still affected. We are monitoring closely. Next update follows in 20 minutes.

  10. in progress Oct 01, 2014, 10:05 PM UTC

    Services are recovering. Next update follows in 10 minutes.

  11. monitoring Oct 01, 2014, 10:22 PM UTC

    Services have recovered. We will continue to monitor the situation.

  12. in progress Oct 01, 2014, 10:36 PM UTC

    DrMobile and DrFront V3 is currently affected again. DrPublish and DrVideo are functioning normally.

  13. in progress Oct 01, 2014, 10:51 PM UTC

    DrMobile and DrFront V3 are still impacted. Next update at 01:10 CEST.

  14. in progress Oct 01, 2014, 11:17 PM UTC

    DrMobile and DrFront V3 are still impacted. Next update at 01:30 CEST or when new information is at hand.

  15. monitoring Oct 01, 2014, 11:30 PM UTC

    DrMobile and DrFront V3 are recovering now. Next update at 01:40 CEST.

  16. monitoring Oct 01, 2014, 11:44 PM UTC

    All services have recovered. We will continue to monitor the situation.

  17. completed Oct 02, 2014, 12:15 AM UTC

    Update Thursday October 2nd at 09:54: We have posted a statement about this at http://blog.aptoma.com/2014/10/02/service-outages-29-09-2014-and-01-10-2014/ . Original update at Thursday October 2nd at 02:15: All affected services are verified to be functioning normally again. We are deeply sorry about the inconvenience this urgent security maintenance caused you. If you have any questions, feedback or other things in mind, let us know at http://aptoma.com/support .