Shorthand incident

Publishing and story preview intermittent partial page load failure

Minor Resolved View vendor source →

Shorthand experienced a minor incident on December 7, 2021 affecting Shorthand and AWS EC2 and 1 more component, lasting 4h 20m. The incident has been resolved; the full update timeline is below.

Started
Dec 07, 2021, 09:59 PM UTC
Resolved
Dec 08, 2021, 02:19 AM UTC
Duration
4h 20m
Detected by Pingoru
Dec 07, 2021, 09:59 PM UTC

Affected components

ShorthandAWS EC2Marketing

Update timeline

  1. identified Dec 07, 2021, 09:59 PM UTC

    AWS has reported elevated error rates in their EC2 APIs resulting in some Shorthand service issues in publishing and previewing stories. Retrying publishing or reloading previews works intermittently. We are monitoring the situation and working with AWS to address the problem.

  2. monitoring Dec 07, 2021, 11:35 PM UTC

    All affected AWS underlying infrastructure is back to normal, except for Shorthand Analytics which relies on an AWS service still being fully restored (API Gateway). Story publishing, previewing and all other core functionality is back to normal. We will continue to monitor service availability until AWS has fully closed the incident.

  3. resolved Dec 08, 2021, 02:19 AM UTC

    All services are now back to full operation, including the slower to resolve Shorthand Analytics service.