Labrador CMS Outage History
Labrador CMS is up right nowPingoru has recorded 27 incidents for Labrador CMS: 6 in the last 12 months and 2 in the last 90 days.
Show as a table
| Month | Incidents | Hours with an open incident |
|---|---|---|
| April 2026 | 1 | 1 of 720 (0%) |
| May 2026 | 0 | 0 of 744 (0%) |
| June 2026 | 0 | 0 of 720 (0%) |
| July 2026 | 1 | 2 of 744 (0%) |
| August 2026 | 0 | 0 of 744 (0%) |
| September 2026 | 0 | 0 of 720 (0%) |
| October 2026 | 2 | 8 of 511 (2%) |
Incidents, newest first
Major October 21, 2026 Security upgrade 6h 26m Labrador Editor
Started Oct 21, 2026, 01:50 PM UTC· resolved Oct 21, 2026, 08:17 PM UTC
- investigating Oct 21, 2026, 01:50 PM UTC
Due to a worldwide active exploit in part of the software we use in our infrastructure we have turned on an extra level of security on all CMS instances. This will require an extra user/password for all users of Labrador. This will not affect readers or external services using our front APIs, but it WILL be needed for all requests to your CMS, including CMS-APIs. We will share the required credentials to all customers, but if you have not received it yet, send an email to [email protected]
- investigating Oct 21, 2026, 01:51 PM UTC
We are continuing to investigate this issue.
- identified Oct 21, 2026, 04:07 PM UTC
We have implemented the patch and will roll it out to all customers now. When all customers have received the patch we will remove the extra layer of security.
- monitoring Oct 21, 2026, 07:21 PM UTC
All customers have been upgraded, we are rolling out removal of the extra security layer now.
- resolved Oct 21, 2026, 08:17 PM UTC
All affected services have now been patched and the extra security layer has been removed.
Major October 20, 2026 Degraded API performance 1h 21m Labrador Frontend
Started Oct 20, 2026, 11:51 AM UTC· resolved Oct 20, 2026, 01:13 PM UTC
- investigating Oct 20, 2026, 11:51 AM UTC
We are experiencing an increased level of 503s on our APIs and are currently investigating. This may affect internal API-requests for automatic fronts and similar.
- identified Oct 20, 2026, 12:32 PM UTC
We have identified the issue and are currently working on a fix.
- monitoring Oct 20, 2026, 01:06 PM UTC
The fix has been deployed and we are monitoring the situation. Services should be going back to normal again.
- resolved Oct 20, 2026, 01:13 PM UTC
The incident has been resolved. "Most read"-lists may take some time to come back due to cache, but everything will be back to normal shortly.
Critical July 16, 2026 Publishing and updating articles 1h 32m Labrador Editor
Started Jul 16, 2026, 09:27 AM UTC· resolved Jul 16, 2026, 10:59 AM UTC
- monitoring Jul 16, 2026, 09:27 AM UTC
We are currently experiencing issues with publishing and editing articles. The root cause has been identified as an issue with our upstream cloud provider (AWS CloudFront) affecting the delivery of published content. Our team is actively monitoring the situation. We will post updates here as soon as we know more. We apologize for the inconvenience. Issue reported by AWS: https://health.console.aws.amazon.com/health/home#/account/dashboard/open-issues?eventID=arn:aws:health:global::event/CLOUDFRONT/AWS_CLOUDFRONT_OPERATIONAL_ISSUE/AWS_CLOUDFRONT_OPERATIONAL_ISSUE_EDF4C_83808542A4E&eventTab=details
- monitoring Jul 16, 2026, 09:29 AM UTC
We are continuing to monitor for any further issues.
- resolved Jul 16, 2026, 10:59 AM UTC
We have deployed a fix that restores publishing and editing of articles. The issue was caused by a fault at our cloud provider, and we have rerouted traffic around the affected component. Publishing should now work as normal, though it may be slightly slower than usual while the temporary route is in place. We are continuing to monitor closely and apologize for the disruption.
- postmortem Jul 30, 2026, 08:47 AM UTC
**Publishing outage — CloudFront VPC origin failure** Date: 2026-07-16 Duration: ~3h \(09:53–12:53 CEST, restored region by region\) Impact: Publishing hung for 60\+ seconds for all customers and surfaced an error in the editor "SyntaxError: Unexpected token '<'". **Background** Published content reaches the public websites through the **labupdate service.** The CMS pushes every published article to it, and it writes it to the front environment. It is the only path by which published content reaches the websites. This push travels over the public internet and enters AWS through a **CloudFront distribution with a VPC origin**, a setup chosen for low latency for distant regions while keeping the labupdate service private. **Root cause** On the morning of the incident, a global AWS-side fault in CloudFront VPC origins caused all requests through that path to hang. The CMS pusher, which sends each publish through the CloudFront distribution to labupdate, never received a response. It waited out its 60-second timeout and retried up to three times, keeping the publish request running long past the CMS nginx timeout. Nginx then aborted the request with a 504 error page, which surfaced the "**SyntaxError: Unexpected token '<'**" error editors saw. Report from AWS: [https://health.aws.amazon.com/health/status?eventID=arn:aws:health:global::event/CLOUDFRONT/AWS\_CLOUDFRONT\_OPERATIONAL\_ISSUE/AWS\_CLOUDFRONT\_OPERATIONAL\_ISSUE\_EDF4C\_83808542A4E](https://health.aws.amazon.com/health/status?eventID=arn:aws:health:global::event/CLOUDFRONT/AWS_CLOUDFRONT_OPERATIONAL_ISSUE/AWS_CLOUDFRONT_OPERATIONAL_ISSUE_EDF4C_83808542A4E) **Resolution** We routed around the CloudFront distribution in front of the labupdate service. An internet-facing network load balancer \(NLB\) was deployed to each region, serving traffic to the same destination as the CloudFront distribution. Besides this bypass, no existing infrastructure or authentication setup was touched. The bypass was verified in ap-northeast-2 before DNS cutover. Eu-north-1 and us-east-2 followed with the verified design. Same evening, AWS resolved the VPC origin fault. After verifying the CloudFront path end-to-end, DNS was flipped back the next day \(17-07-2026\), with the NLBs remaining as warm standbys in case the issue arose again. Articles published 09:53–12:53 required manual re-publishing \(publishes were CMS-side successful but never synced to front\). **Timeline \(CEST\)** * 09:53 — Last successful publish; failures begin * 10:26 — First customer reports * 10:44 — Root cause identified: CloudFront VPC origins failing * ~12:00 — NLB bypass deployed to apne2, verified end-to-end * 12:15 — Asia \(apne2\) restored — DNS cut over to NLB * 12:44 — Europe \(eun1\) restored * 12:53 — US \(use2\) restored * Later same day — AWS resolved VPC origin fault **Follow-up** * Documented a switchover procedure so we can quickly reroute around CloudFront if this happens again. * Setting up alerts around the affected resources so this class of failure can be detected faster. * Further improving routines for updating the status-page to keep customers informed about ongoing issues.
Critical April 22, 2026 Labrador development environment down 27m Labdevs Development
Started Apr 22, 2026, 06:52 AM UTC· resolved Apr 22, 2026, 07:19 AM UTC
- identified Apr 22, 2026, 06:52 AM UTC
The labrador development environment is currently down due to a full disk. We are in the process of removing data and rebooting the server.
- monitoring Apr 22, 2026, 07:06 AM UTC
The dev environment should now be accessible and clusters are slowly coming back up.
- resolved Apr 22, 2026, 07:19 AM UTC
The development environment is back up and running with all clusters stable.
Major March 30, 2026 Major outage on infrastructure 12m Labrador Editor, Labrador Frontend
Started Mar 30, 2026, 07:22 PM UTC· resolved Mar 30, 2026, 07:35 PM UTC
- monitoring Mar 31, 2026, 09:27 AM UTC
Our infrastructure at OVH experienced issues for some of their Virtual Private Servers instances which affected our connections to the editor/cms for all customers and for some of our customers that are using OVH also for their frontend. We experienced issues from 21.22 CET to 21.35 CET, 30.03.26. For more info, read incident report "[GRA6][Virtual Private Servers] - Some instances incident notification" at OVH "...issue affecting our Virtual Private Servers offer on the specific region GRA6." Incident info: https://bare-metal-servers.status-ovhcloud.com/incidents/s06688dvxskr We are still monitoring the incident
- monitoring Mar 31, 2026, 10:42 AM UTC
We are continuing to monitor for any further issues.
- resolved Mar 31, 2026, 12:10 PM UTC
No new issues has been reported beyond initial outage
Notice January 20, 2026 Network issues 25m
Started Jan 20, 2026, 08:30 PM UTC· resolved Jan 20, 2026, 08:55 PM UTC
- resolved Jan 21, 2026, 09:05 AM UTC
From 21:45 - 21:55 CET our infrastructure partner OVH experienced network issues. This led to instability for some of our customers, both on CMS and Front. The issues were solved by OVH and systems were back to normal operation at 21:55 CET.
Major September 24, 2025 Labrador Front service disruptions 2h 19m
Started Sep 24, 2025, 08:30 PM UTC· resolved Sep 24, 2025, 10:49 PM UTC
- resolved Sep 25, 2025, 11:34 AM UTC
At 24.09.2025 between 22:22-22:49 GMT+2 we experienced service disruptions for Labrador Front, resulting in slower response times and intermittent HTTP errors. The issue was caused by an edge case in our caching layer where, under the right circumstances, the cache would get overwhelmed, causing requests to queue up. We have identified the cause of this issue and have taken immediate action to ensure it will not reoccur.
Major May 27, 2025 Degraded front performance 43m Labrador Editor, Labrador Frontend
Started May 27, 2025, 08:20 PM UTC· resolved May 27, 2025, 09:03 PM UTC
- investigating May 27, 2025, 08:20 PM UTC
We are currently experiencing service disruptions. We are looking into the issue and will update you as soon as we know more.
- identified May 27, 2025, 08:45 PM UTC
The issue is identified to be caused by network issues at our server provider. They are actively working on resolving the issues, more information can be found here: https://network.status-ovhcloud.com/incidents/3y7hl8330d7q
- monitoring May 27, 2025, 08:51 PM UTC
The network is stabilizing, but we are still experiencing slower response times. We will continue monitoring.
- resolved May 27, 2025, 09:03 PM UTC
The incident is resolved and all systems are back to normal operation for now.
- postmortem Jun 02, 2025, 01:14 PM UTC
## Summary On Tuesday 27.05.2025 between 22:02 - 22:50 CEST and Wednesday 28.05.2025 between 18:50 - 19:27 CEST one of our infrastructure providers, OVH, experienced network disruptions affecting two of our three data centers. This resulted in a partial or complete service outage for traffic destined to both Labrador CMS and Labrador Front. Following the incident on Wednesday a subset of our customers experienced further service degradation, as some Labrador components were stuck in an unhealthy state and needed manual intervention. At 20:30 CEST all Labrador services returned to a fully operational state. ## Details Our internal monitoring systems reported unavailable services and sites on Tuesday 22:03 CEST. Initial investigation revealed that a network outage was ongoing at OVH in several data centers, causing connectivity disruptions between our services. Networks returned to normal at 22:50 CEST. This incident reoccurred on Wednesday between 18:50 - 19:27 CEST. Once network operation was restored, internal monitoring and customer reports indicated that a subset of our clients still had service degradation, in the form of slower responses or issues publishing content. Further diagnostics revealed that one of our servers was stuck in an unhealthy state following the network incident. The affected server was pulled from our cluster, and Labrador services returned to a fully healthy state at 20:30 CEST. ## Impacted services Services affected by this incident are specified in the table below. | **Service name** | **Minutes** | **Time from — to** | | --- | --- | --- | | Labrador CMS | 48 | 22:02 — 22:50 | | Labrador CMS | 100 | 18:50 — 20:30 | | Labrador Front | 48 | 22:02 — 22:50 | | Labrador Front | 100 | 18:50 — 20:30 | ## Incident timeline Following is a timeline that describes the entire incident handling process. **Tuesday:** * `2025.05.27 22:02` Initial service outage alerts registered * `2025.05.27 22:15` Large scale network outage confirmed * `2025.05.27 22:36` Network incident confirmed by OVH * `2025.05.27 22:45` Root cause resolved by OVH, network declared healthy * `2025.05.27 22:50` All Labrador services healthy **Wednesday:** * `2025.05.28 18:53` Initial service outage alerts registered * `2025.05.28 19:00` Large scale network outage confirmed * `2025.05.28 19:03` Network incident confirmed by OVH * `2025.05.28 19:24` Root cause resolved by OVH, network declared healthy * `2025.05.28 19:45` Reports that some customers still experienced degradation * `2025.05.28 20:15` Service degradation root cause identified * `2025.05.28 20:30` Affected services restarted, fully operational ## Root cause The root cause of the incidents was determined to be backbone networking malfunctions, resulting in network disruptions at OVH. * [Tuesday: OVH incident summary](https://network.status-ovhcloud.com/incidents/3y7hl8330d7q) * [Wednesday: OVH incident summary](https://network.status-ovhcloud.com/incidents/j292y56ckyfq) ## Planned actions We will be assessing how these two incidents affect our SLA coverage. We are continuously working on improving and decentralizing our infrastructure so that we are less vulnerable to large scale data center network outages. One of our current largest efforts in this regard is moving the Labrador CMS and Front infrastructure to the cloud, reducing our exposure to various OVH outages and increasing our geographical presence and flexibility. At the moment around 25% of our customers have Labrador Front migrated fully to our AWS cloud. These customer were much less impacted by this incident, as only their Labrador CMS Editor was affected. We expect to migrate most of our remaining customers to AWS in the following weeks, and the larger or more complex customers over the summer.
Major April 28, 2025 Service disruptions 30m Labrador Editor, Labrador Frontend
Started Apr 28, 2025, 11:49 AM UTC· resolved Apr 28, 2025, 12:19 PM UTC
- investigating Apr 28, 2025, 11:49 AM UTC
Labrador CMS is currently experiencing partial service disruptions. We are looking into this issue and will update you as soon as we know more.
- monitoring Apr 28, 2025, 12:14 PM UTC
The problem has been identified. Too aggressive use of our import APIs resulted in a subset of customers to experience problems with database connections. The API-usage has been dialed down and operations should shortly return to normal. We will continue to monitor performance.
- resolved Apr 28, 2025, 12:19 PM UTC
This incident has been resolved.
Notice February 13, 2025 Network incident AWS Stockholm 1h 15m Labrador Frontend
Started Feb 13, 2025, 11:10 PM UTC· resolved Feb 14, 2025, 12:25 AM UTC
- monitoring Feb 14, 2025, 01:26 AM UTC
Customers hosted within our AWS Stockholm region experienced service disruptions this night between 00:10-01:25 CET caused by a networking incident at AWS. A fix has been implemented, all services are operational again and we are monitoring the situation. More details: https://health.aws.amazon.com/health/status?eventID=arn:aws:health:eu-north-1::event/MULTIPLE_SERVICES/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE/AWS_MULTIPLE_SERVICES_OPERATIONAL_ISSUE_6337A_0CF62E5431F
- resolved Feb 14, 2025, 07:53 AM UTC
This incident has been resolved.
Notice January 19, 2025 Apex redirect issues – Labrador Frontend
Started Jan 19, 2025, 10:14 PM UTC· resolved Jan 19, 2025, 10:14 PM UTC
- resolved Jan 19, 2025, 10:14 PM UTC
Some customers experienced an issue where readers would not be redirected to www.domainname.tld when trying to access domainname.tld. The error has been identified and systems are back to normal operation again.
Major October 30, 2024 Service outage due to network issues 44m Labrador Editor, Labdevs Development, Labrador Frontend
Started Oct 30, 2024, 01:32 PM UTC· resolved Oct 30, 2024, 02:17 PM UTC
- investigating Oct 30, 2024, 01:32 PM UTC
We are experiencing issues across all services and are actively investigating.
- monitoring Oct 30, 2024, 01:46 PM UTC
The problem was caused by issues at one of our infrastructure providers. Services are back online now, but we are actively monitoring. A more thorough description of the issues will follow when we have more information from our server provider.
- resolved Oct 30, 2024, 02:17 PM UTC
Our infrastructure provider reports that the issue has been identified and fixed. The incident is resolved and we will update with more details when we have them.
- postmortem Oct 31, 2024, 06:18 AM UTC
## Summary On Wednesday 30.10.2024 between 13:23 - 13:40 UTC one of our primary infrastructure providers, OVHcloud, experienced network disruptions across multiple data centers. This resulted in a partial or complete service outage for a large part of the traffic destined to both Labrador CMS and Labrador Front. ## Details Our internal monitoring systems reported the first unavailable services and sites at 13:25 UTC. Initial investigation revealed that a network outage was ongoing at one of our infrastructure providers, causing connectivity disruptions for all their services. Network services started to return at 13:40 UTC and all systems came back online. At 13:45 UTC, all systems were confirmed to be operational. ## Impacted services Services affected by this incident are specified in the table below. All Labrador CMS customers were affected to varying degrees. CMS access was down, but most customers without their own Varnish cache layers were still available for readers of cached pages. | **Service name** | **Minutes** | **Time from — to** | | --- | --- | --- | | Labrador CMS | 20 | 13:25 — 13:45 | | Labrador Front | 20 | 13:25 — 13:45 | ## Incident timeline Following is a timeline that describes the entire incident handling process. All times UTC. * `2024.10.30 13:25` Initial service outage alerts registered * `2024.10.30 13:27` Large scale network outage confirmed * `2024.10.30 13:32` Statuspage updated and customers notified * `2024.10.30 13:40` Network back up again. * `2024.10.30 13:45` All services confirmed operational and customers notified. ## Root cause The root cause of the incident was determined to be network disruptions at our infrastructure provider, caused by one of their pairing partners pushing a faulty network update. ## Planned actions We are continuously working on improving and decentralizing our infrastructure so that we are less vulnerable to these large scale network outages. One of our current largest efforts in this regard is moving more parts of the Labrador CMS and Front infrastructure to the cloud. Currently storage, image rendering and Varnish caching has been moved to AWS, with the rest of Labrador Front following in the coming months. ## Additional reading For more information on the incident, the OVHcloud incident report can be found here: [https://network.status-ovhcloud.com/incidents/qgb1ynp8x0c4](https://network.status-ovhcloud.com/incidents/qgb1ynp8x0c4) In addition, Cloudflare has an interesting blog post with some more details here: [https://blog.cloudflare.com/cloudflare-perspective-of-the-october-30-2024-ovhcloud-outage/](https://blog.cloudflare.com/cloudflare-perspective-of-the-october-30-2024-ovhcloud-outage/)