Aptoma experienced a major incident on June 10, 2021 affecting DrPublish - GUI and DrPublish - API /io and 1 more component, lasting 3h 39m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- identified Jun 10, 2021, 08:43 PM UTC
Services are in the process of transitioning onto alternate datacenters. Some customers may experience aborted requests or connectivity issues while the services are recovering. We expect services to recover within 30 minutes.
- identified Jun 10, 2021, 09:12 PM UTC
We have received the following update from our cloud provider AWS: > 1:55 PM PDT We can confirm increased API error rates and latencies for the EC2 APIs and connectivity issues for instances within a single Availability Zone (euc1-az1) within the EU-CENTRAL-1 Region, caused by an increase in ambient temperature within a subsection of the affected Availability Zone. Other Availability Zones within the EU-CENTRAL-1 Region are not affected by the issue and we continue to work towards resolving the issue. The consequences for some of our customers currently are: * Certain customers can't search for articles in DrP Newsroom * The same customers can't upload and insert assets into DrPublish articles
- identified Jun 10, 2021, 09:14 PM UTC
We are continuing to work on a fix for this issue.
- identified Jun 10, 2021, 09:43 PM UTC
We notice intermittent image editing issues in DrEdition Front Page does, but these issues are now resolved. The following issues still persists for some customers: * Can't search for articles in DrP Newsroom * Can't upload and insert assets into DrPublish articles Please note that for these customers it's still possible to create and publish DrPublish articles. We are still investigating.
- identified Jun 10, 2021, 10:18 PM UTC
We have received reports from AWS that temperatures at the affected datacenter(s) are returning to normal levels, but not all of our services have returned to normal operations yet. The following issues still persists for some of our customers: * Can't search for articles in DrP Newsroom * Can't upload and insert assets into DrPublish articles
- identified Jun 10, 2021, 10:49 PM UTC
AWS reports that "temperatures continue to return to normal levels" but that their engineers are still not able to physicallay access the affected region to restore normal services. The same subset of customers are still affected with the following * Can't search for articles in DrP Newsroom * Can't upload and insert assets into DrPublish articles We are currently making plans for what to do if AWS can not re-establish normal services in their affected datacenters.
- identified Jun 10, 2021, 11:28 PM UTC
We have received reports that AWS engineers are still locked out of their affected datacenters due to environmental conditions, and we are initiating our own fallback plans in parallell with their ongoing efforts. We expect our plans to take several hours to complete. We will abort our plans if AWS engineers are successful before our own fallback plan completion. We will post an update at 06:00 at the latest.
- monitoring Jun 10, 2021, 11:37 PM UTC
All services have recovered. We are monitoring and double checking.
- resolved Jun 11, 2021, 12:22 AM UTC
We have been monitoring and double checking, and we conclude that the incident has been resolved. You should expect everything to work as normal.
- postmortem Jun 11, 2021, 11:23 AM UTC
An AWS availability zone \(AZ\) in Frankfurt got shut down due to overheating. Some features in our technology services \(AMP services\) still rely on specific AZs. While most affected AMP services recovered fully with 20 minutes with only partial outages during this short period, two features in DrPublish and one in DrEdition Print Automation did not recover until approx 01:34 Friday 11th June CEST, namely DrPublish Newsroom search, DrPublish image and graphics uploads, and DrEdition Print Automation content inbox. Some customers were not affected, due to dependence on a separate AZ. For the customers set up for these features on the affected AZ, consequences were severely limiting for production in this timeframe \(approx Thursday 2021-06-10 22:20 CEST to Friday 2021-06-11 01:30 CEST\). Duration 3h 10min. We have for some time worked to remove the weakness of being dependent to one single AZ at AWS Frankfurt for these specific features, but we are not yet finished. This work will continue at top priority. We expect to be finished during 2021. We are sorry for the problems caused, and will do what we can to make sure similar outages at AWS cannot affect our services this way in the future.