Labrador CMS experienced a critical incident on July 16, 2026 affecting Labrador Editor, lasting 1h 32m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- monitoring Jul 16, 2026, 09:27 AM UTC
We are currently experiencing issues with publishing and editing articles. The root cause has been identified as an issue with our upstream cloud provider (AWS CloudFront) affecting the delivery of published content. Our team is actively monitoring the situation. We will post updates here as soon as we know more. We apologize for the inconvenience. Issue reported by AWS: https://health.console.aws.amazon.com/health/home#/account/dashboard/open-issues?eventID=arn:aws:health:global::event/CLOUDFRONT/AWS_CLOUDFRONT_OPERATIONAL_ISSUE/AWS_CLOUDFRONT_OPERATIONAL_ISSUE_EDF4C_83808542A4E&eventTab=details
- monitoring Jul 16, 2026, 09:29 AM UTC
We are continuing to monitor for any further issues.
- resolved Jul 16, 2026, 10:59 AM UTC
We have deployed a fix that restores publishing and editing of articles. The issue was caused by a fault at our cloud provider, and we have rerouted traffic around the affected component. Publishing should now work as normal, though it may be slightly slower than usual while the temporary route is in place. We are continuing to monitor closely and apologize for the disruption.
- postmortem Jul 30, 2026, 08:47 AM UTC
**Publishing outage — CloudFront VPC origin failure** Date: 2026-07-16 Duration: ~3h \(09:53–12:53 CEST, restored region by region\) Impact: Publishing hung for 60\+ seconds for all customers and surfaced an error in the editor "SyntaxError: Unexpected token '<'". **Background** Published content reaches the public websites through the **labupdate service.** The CMS pushes every published article to it, and it writes it to the front environment. It is the only path by which published content reaches the websites. This push travels over the public internet and enters AWS through a **CloudFront distribution with a VPC origin**, a setup chosen for low latency for distant regions while keeping the labupdate service private. **Root cause** On the morning of the incident, a global AWS-side fault in CloudFront VPC origins caused all requests through that path to hang. The CMS pusher, which sends each publish through the CloudFront distribution to labupdate, never received a response. It waited out its 60-second timeout and retried up to three times, keeping the publish request running long past the CMS nginx timeout. Nginx then aborted the request with a 504 error page, which surfaced the "**SyntaxError: Unexpected token '<'**" error editors saw. Report from AWS: [https://health.aws.amazon.com/health/status?eventID=arn:aws:health:global::event/CLOUDFRONT/AWS\_CLOUDFRONT\_OPERATIONAL\_ISSUE/AWS\_CLOUDFRONT\_OPERATIONAL\_ISSUE\_EDF4C\_83808542A4E](https://health.aws.amazon.com/health/status?eventID=arn:aws:health:global::event/CLOUDFRONT/AWS_CLOUDFRONT_OPERATIONAL_ISSUE/AWS_CLOUDFRONT_OPERATIONAL_ISSUE_EDF4C_83808542A4E) **Resolution** We routed around the CloudFront distribution in front of the labupdate service. An internet-facing network load balancer \(NLB\) was deployed to each region, serving traffic to the same destination as the CloudFront distribution. Besides this bypass, no existing infrastructure or authentication setup was touched. The bypass was verified in ap-northeast-2 before DNS cutover. Eu-north-1 and us-east-2 followed with the verified design. Same evening, AWS resolved the VPC origin fault. After verifying the CloudFront path end-to-end, DNS was flipped back the next day \(17-07-2026\), with the NLBs remaining as warm standbys in case the issue arose again. Articles published 09:53–12:53 required manual re-publishing \(publishes were CMS-side successful but never synced to front\). **Timeline \(CEST\)** * 09:53 — Last successful publish; failures begin * 10:26 — First customer reports * 10:44 — Root cause identified: CloudFront VPC origins failing * ~12:00 — NLB bypass deployed to apne2, verified end-to-end * 12:15 — Asia \(apne2\) restored — DNS cut over to NLB * 12:44 — Europe \(eun1\) restored * 12:53 — US \(use2\) restored * Later same day — AWS resolved VPC origin fault **Follow-up** * Documented a switchover procedure so we can quickly reroute around CloudFront if this happens again. * Setting up alerts around the affected resources so this class of failure can be detected faster. * Further improving routines for updating the status-page to keep customers informed about ongoing issues.