Servd Outage History

Started Nov 02, 2026, 03:20 PM UTC· resolved Nov 02, 2026, 05:57 PM UTC

1. investigating Nov 02, 2026, 03:20 PM UTC

   Projects attempting to build bundles from GitHub repositories are currently failing and throwing the following errors: fatal: could not read Username for 'https://github.com': No such device or address fatal: expected flush after ref listing We're currently investigating and will report updates as soon as we have them.

2. investigating Nov 02, 2026, 03:54 PM UTC

   We're seeing customers are able to get around the issue by temporarily [switching to the "Manual" repository connection type](https://servd.host/docs/change-repository-or-connection-type). **Note:** This requires the SSH deploy key found in the project's "Build & Deploy" page to be registered with the repository. GitHub have recently changed their requirements with regards to the length of SSH deploy keys, so if you require your project's deploy key to be regenerated, reach out to support via our chat widget or at [email protected].

3. monitoring Nov 02, 2026, 05:25 PM UTC

   We've identified the change in behaviour with regards to GitHub repository clone authentication and have deployed a fix. We're currently monitoring to verify the fix.

4. resolved Nov 02, 2026, 05:57 PM UTC

   We've now resolved the incident, but please reach out if you continue to see issues. We believe the underlying issues was an undocumented change in how GitHub authenticates certain OAuth2 repository clone URLs. A fix was deployed that adjusted how our task runner cloned repositories to a mechanism that correctly authenticated.

Read the full incident report →

Started Oct 21, 2026, 09:12 AM UTC· resolved Oct 23, 2026, 12:43 PM UTC

1. identified Oct 21, 2026, 09:12 AM UTC

   Due to the PHP critical vulnerability due to be disclosed today we have blocked all multipart upload requests on all projects and on servd.host. This will create 403 errors for image and other file uploads. We are working on restoring normal functionality, however due to the critical nature and trivial exploitation of the bug, this is the safest course of action for the time being.

2. identified Oct 21, 2026, 12:50 PM UTC

   We are currently waiting for PHP to release fixes which is due at 14:00 UTC. At this time we will prep them for use in Servd projects which we are expecting to take 20-30 minutes. Once these have been confirmed as effective at mitigating the issue we will release them for all projects. We will update this status page once they have been released. Once released - any new bundles created in our dashboard which are using PHP 8.2 + will contain the fix. Deploying these will automatically re-enable multipart requests for your project. This timeline may change in two scenarios: 1. PHP do not release their updates on time or it is delayed for another reason 2. The fixes within the PHP releases do not actually address the attack we have been seeing today (there has been no official confirmation that the two are related, but there is a very strong likelihood that they are the same thing).

3. identified Oct 21, 2026, 01:55 PM UTC

   We've received details of the vulnerability and are awaiting official patched PHP images. As soon as they're available, we'll be publishing new base bundles, and allowing customers to deploy them to their environments. In the meantime, we're shoring up our defensive edge rules based on the released info.

4. identified Oct 21, 2026, 02:54 PM UTC

   We're currently blocked on releasing patched PHP versions, due to delays in the release of the official PHP docker images. We'll be exporting base bundles as soon as they're made available.

5. identified Oct 21, 2026, 03:48 PM UTC

   The patched official PHP docker images have been published. We'll have patched PHP versions available shortly.

6. identified Oct 21, 2026, 04:16 PM UTC

   Patched official PHP images have been published, but we're not yet seeing any exported image layers to them. This is blocking our efforts to release patched PHP versions to Servd. We're regularly monitoring for changes.

7. identified Oct 21, 2026, 05:12 PM UTC

   The official PHP docker images have completed publishing, we're now in the process of making new PHP versions available.

8. identified Oct 21, 2026, 05:33 PM UTC

   We've now released patched PHP versions for 8.5.12, 8.4.27, 8.3.36 or 8.2.35. Please proceed to build and sync new bundles to all your environments. When a safe bundle is deployed to an environment, our platform will automatically disable the multipart upload block. If you need assistance doing so, please reach out to us via [email protected]. For projects running PHP 8.1 or lower, we recommend updating your sites to PHP 8.2 or higher as soon as possible. For these projects, the multipart upload block will remain in place for the foreseeable future but will be automatically lifted when a safe bundle is deployed.

9. identified Oct 22, 2026, 09:54 AM UTC

   We are beginning to automatically rebuild deployed bundles for all unpatched project environments running PHP 8.2 or higher, replacing them with builds running a safe version of PHP. However, the quickest and safest means to secure your projects is to rebuild your deployed bundles manually as soon as you can. We've also just published a post regarding this incident in our blog https://servd.host/blog/mitigation-of-critical-php-cves We'll be keeping it updated with additional information as it becomes available.

10. identified Oct 22, 2026, 05:54 PM UTC

    We are currently rolling out updates to all eligible projects that have not yet manually updated. We are expecting this to be completed within the next few hours. We are also working on backporting the patches for projects using PHP 7.4, 8.0 and 8.1. We are hoping to roll out those updated tomorrow. Some projects are not currently able to be updated due to a loss of connection between Servd and the project's git repo - we will be emailing the owners of these projects once we have completed updates for other projects.

11. identified Oct 23, 2026, 08:23 AM UTC

    Backported patches for PHP 7.4, 8.0 and 8.1 have been created and we plan to roll these out to projects today. Once complete, we will be in contact with the owners of any projects that **have not** been able to be updated for any reason with further steps to take. At that time we will also close off this incident.

12. resolved Oct 23, 2026, 12:43 PM UTC

    We have automatically updated all projects that can be updated from our side. We'll reach out to the owners of those that remain. Thanks for your patience while we worked through getting everyone patched!

Read the full incident report →

Started Oct 16, 2026, 03:44 PM UTC· resolved Oct 18, 2026, 04:13 PM UTC

1. investigating Oct 16, 2026, 03:44 PM UTC

   We've investigating signs of degraded CPU performance in our CA-EAST-1 cluster, which is impacting response times.

2. investigating Oct 16, 2026, 05:45 PM UTC

   We're seeing performance improving, however are still investigating the underlying cause.

3. resolved Oct 18, 2026, 04:13 PM UTC

   Performance has returned to normal levels, thanks for your patience.

Read the full incident report →

Started Oct 11, 2026, 07:17 PM UTC· resolved Oct 12, 2026, 07:13 AM UTC

1. identified Oct 11, 2026, 07:17 PM UTC

   A degradation of network connectivity in two data centres is causing timeouts and other connection errors with traffic between project components and out to the public internet. We're liaising with our server provider to identify and rectify the root cause.

2. resolved Oct 12, 2026, 07:13 AM UTC

   The incident has now been resolved. Two individual problems were discovered and mitigated: - DNS lookup failures in our London based clusters. This was caused by a configuration error with an upstream DNS resolver causing it to reject any dnssec-enabled queries. - Packet loss in us-east-3/4/5 clusters. This was caused by a hardware issue with a hypervisor which caused degraded networking performance with a subset of worker nodes. The fault has been resolved.

Read the full incident report →

Started Sep 14, 2026, 12:56 PM UTC· resolved Sep 14, 2026, 03:54 PM UTC

1. investigating Sep 14, 2026, 12:56 PM UTC

   We are in the process of cycline them out. Some project components may be unavailable for a few seconds while we do so.

2. resolved Sep 14, 2026, 03:54 PM UTC

   We've now resolved the incident. Thanks for your patience.

Read the full incident report →

Started Sep 02, 2026, 09:36 AM UTC· resolved Sep 02, 2026, 09:40 AM UTC

1. resolved Sep 02, 2026, 03:54 PM UTC

   Between 09:36 and 09:40 UTC, our US-EAST-6 cluster received a large-scale DDoS attack which briefly caused the cluster's load balancers to return 502 Bad Gateway responses to a subset of projects hosted in US-EAST-6. At 09:37 UTC, our automated Cloudflare-backed DDoS protections activated and began blocking malicious traffic. Meanwhile, the clusters load balancers scaled up and error rates dropped back to normal levels. In response, we've permanently increased the baseline capacity of the clusters load balancer and are prioritising measures to better absorb the initial shock of similar DDoS attacks in the future. Apologies for the interruption.

Read the full incident report →

Started Aug 28, 2026, 12:30 AM UTC· resolved Aug 28, 2026, 09:28 AM UTC

1. investigating Aug 28, 2026, 09:06 AM UTC

   We have noted increased error rates on a subset of projects caused by connectivity issues with projects' redis instances.

2. resolved Aug 28, 2026, 09:28 AM UTC

   The issue has now been fully resolved. The issue was caused by a deployment misconfiguration which prevented PHP from being able to communicate with redis. We are currently in the process of refactoring the use of Redis (which has changed is licensing to prevent commercial use) within projects deployed to the Servd platform. The first step of this change is to create a set of feature gates within our deployment mechanism which allow us to flip a project between Redis and the proposed replacement (currently DragonflyDB). Yesterday we released these feature gates to all projects' deployment mechanisms with the default still set to use Redis. We were therefore expecting no change to any projects at this time, but would have the ability to flip the feature gates on a per project basis in the future. This was tested on multiple internal projects before rolling out generally. At 01:30 BST we were alerted to elevated 500 errors from multiple projects. Upon inspection these projects seemed to be having issues communicating with their Redis instances. We assumed this must be related to the feature gate changes that had rolled out earlier in the day. These projects had all had their subscriptions renewed over the past 30 minutes which was the apparent trigger. During subscription renewals we perform a 'partial sync' on projects - syncing updates related to changes to project subscriptions without deploying any settings changes made in the Servd dashboard. This relies on taking a snapshot of the project's previous sync's deployment values and merging those with any changes due to the subscription. This resulted in the new feature gates having no default value set - the sync was executed with them as empty values. Our deployment scripts also contained logic that checked for the value of the feature gates to direct traffic to either redis or its replacement. This logic failed due to the empty value of the feature gate (it expected a specific boolean value) which led to the bad configuration being deployed. To fix the problem we ran through all impacted projects and executed a 'full sync' on each of them. This ensured a non-empty value was applied for the feature gates and allowed a correct configuration to be deployed. We also subsequently updated the deployment scripts to handle the empty value elegantly to ensure the problem does not repeat for further project renewals. The total impact on projects varied in time as we ran through them from start to finish. We have also adjusted our testing methodology for any deployment script updates which include new values, to include testing with partial syncs to ensure they work as expected.

Read the full incident report →

Started Jul 08, 2026, 12:57 AM UTC· resolved Jul 08, 2026, 12:58 AM UTC

1. resolved Jul 08, 2026, 10:37 AM UTC

   At 00:57 UTC, Servd's US-EAST-1 cluster received a large DDoS attack which lasted 30 seconds. At which point our Cloudflare-backed DDoS protections activated and ended the attack. In the interim, we saw elevated 502 responses as our infrastructure quickly scaled up to handle the load. Apologies for any interruptions to your projects. For those looking to insulate themselves from such attacks we recommend employing [Servd's Private Load Balancer addon](https://servd.host/docs/private-load-balancer-addon).

Read the full incident report →

Started Jun 12, 2026, 01:53 PM UTC· resolved Jun 13, 2026, 08:49 AM UTC

1. resolved Jun 13, 2026, 08:49 AM UTC

   We're confident the issue has fully resolved. Thanks for your patience.

Read the full incident report →

Started Jun 03, 2026, 10:54 AM UTC· resolved Jun 03, 2026, 01:06 PM UTC

1. investigating Jun 03, 2026, 10:54 AM UTC

   We're seeing new workloads (backup tasks and Syncs) failing to start up as expected in our uk-london-5 cluster, resulting in stuck deployments. We're investigating currently.

2. investigating Jun 03, 2026, 12:52 PM UTC

   We've fixed the core issue, and are waiting for things to recover.

3. resolved Jun 03, 2026, 01:06 PM UTC

   We've now resolved the incident. Thanks for your patience.

Read the full incident report →

Started May 20, 2026, 01:10 AM UTC· resolved May 20, 2026, 07:47 AM UTC

1. investigating May 20, 2026, 01:10 AM UTC

   There is an ongoing issue with availability of a small number of projects in our us-east-6 and us-east-7 clusters. We have tracked this down to a problem with database data for these projects failing to attach to nodes as expected. We are communicating with our upstream providers currently to rectify.

2. investigating May 20, 2026, 01:20 AM UTC

   The underlying issue has been resolved and we're monitoring to ensure full recovery of all projects

3. resolved May 20, 2026, 07:47 AM UTC

   We've now resolved the incident. Thanks for your patience.

Read the full incident report →

Started Apr 23, 2026, 07:37 AM UTC· resolved May 02, 2026, 11:35 PM UTC

1. investigating Apr 23, 2026, 07:58 AM UTC

   Things seem to be recovering. We will monitor for any reoccurrence.

2. resolved May 02, 2026, 11:35 PM UTC

   We've now resolved the incident. Thanks for your patience.

Read the full incident report →

Started Apr 02, 2026, 12:35 PM UTC· resolved Apr 23, 2026, 07:37 AM UTC

1. investigating Apr 02, 2026, 01:01 PM UTC

   We've observed requests to Bitbucket have started going through again. Our current working theory is a limited intermittent outage with Bitbucket's API, however nothing has been reported on their status page as of yet.

2. resolved Apr 23, 2026, 07:37 AM UTC

   We've now resolved the incident. Thanks for your patience.

Read the full incident report →

Started Feb 20, 2026, 06:39 PM UTC· resolved Feb 20, 2026, 07:26 PM UTC

1. resolved Feb 22, 2026, 04:28 PM UTC

   Cloudflare has published a post-mortem blog post about the issue which prompted this disruption to service on Servd. You can read more about it here [https://blog.cloudflare.com/cloudflare-outage-february-20-2026/](https://blog.cloudflare.com/cloudflare-outage-february-20-2026/)

Read the full incident report →

Started Dec 05, 2025, 08:57 AM UTC· resolved Dec 05, 2025, 09:26 AM UTC

1. resolved Dec 05, 2025, 09:26 AM UTC

   Cloudflare have marked their as resolved https://www.cloudflarestatus.com/incidents/lfrm31y6sw9q We're seeing service back to normal, thanks for your patience.

Read the full incident report →

Started Nov 26, 2025, 08:59 PM UTC· resolved Nov 26, 2025, 09:17 PM UTC

1. resolved Nov 26, 2025, 09:42 PM UTC

   Between 20:59 and 21:17 UTC there were two drops in connectivity to the US-EAST-7 cluster due to a large scale DDoS attack. The attack was mitigated by our Ingress Platform's DDoS defences, however some attack traffic was able to reach the cluster before the defences automatically activated. We're exploring strategies to better deal with attack traffic prior to our DDoS protections activating.

Read the full incident report →

Started Nov 18, 2025, 11:48 AM UTC· resolved Nov 18, 2025, 06:15 PM UTC

1. resolved Nov 18, 2025, 06:15 PM UTC

   Cloudflare have marked their incident as fully resolved, and we're no longer seeing any issues across Servd, so are marking this incident as resolved too. Apologies for the disruption.

Read the full incident report →

Started Oct 20, 2025, 07:34 AM UTC· resolved Oct 22, 2025, 02:32 PM UTC

1. resolved Oct 22, 2025, 02:32 PM UTC

   The upstream AWS issue is now fully resolved.

Read the full incident report →

Started Sep 26, 2025, 02:55 PM UTC· resolved Sep 26, 2025, 03:21 PM UTC

1. resolved Sep 26, 2025, 03:21 PM UTC

   SSL validation appears to be going through normally now. Thanks for your patience.

Read the full incident report →

Started Sep 24, 2025, 11:09 PM UTC· resolved Sep 25, 2025, 01:09 AM UTC

1. resolved Sep 25, 2025, 05:44 AM UTC

   Docker Hub (Servd's bundle storage provider) [suffered an incident](https://www.dockerstatus.com/pages/incident/533c6539221ae15e3f000031/68d47a2f93c09e05486d93a9) which prevented Servd's task runner from exporting new bundles during bundles builds and pulling existing bundles during syncs. The upstream incident was fixed by Docker Hub at 1:09 UTC, and all Servd systems are operational once again.

Read the full incident report →

Started Sep 23, 2025, 02:04 PM UTC· resolved Sep 23, 2025, 02:32 PM UTC

1. resolved Sep 23, 2025, 03:17 PM UTC

   Between 14:04 UTC and 14:32, project requests routing to 74.220.21.86 experienced time outs. The impact was confined to projects that were previously located in our decommissioned UK-LONDON-1 and UK-LONDON-3 clusters that were then migrated over to our active UK-LONDON-4 cluster. These requests utilise our deprecated legacy ingress. To avoid similar issues in the future, we strongly recommend upgrading your impacted projects to our Ingress Platform. https://servd.host/docs/upgrading-to-new-ingress

Read the full incident report →

Started Aug 21, 2025, 02:52 PM UTC· resolved Aug 21, 2025, 03:07 PM UTC

1. resolved Aug 21, 2025, 03:07 PM UTC

   We've now resolved the incident. Thanks for your patience.

Read the full incident report →

Started Aug 19, 2025, 03:42 PM UTC· resolved Aug 19, 2025, 04:11 PM UTC

1. resolved Aug 19, 2025, 04:11 PM UTC

   The issue has been resolved with the upstream provider and builds are now progressing as expected.

Read the full incident report →

Started Jul 11, 2025, 10:39 AM UTC· resolved Jul 11, 2025, 11:54 AM UTC

1. resolved Jul 11, 2025, 03:33 PM UTC

   We've had confirmation from our server provider (DigitalOcean) that the issue originated with their infrastructure. Specifically they state they detected an issue with one of their European carriers which caused outbound traffic from a subset of their locations in London, Amsterdam and Frankfurt to fail. They're working with their carrier to ensure the issue does not reoccur. If you'd like to add resilience against these kinds of incidents to your project, drop a message to [email protected] and we'd be happy to advise.

Read the full incident report →

Started Jun 06, 2025, 07:10 AM UTC· resolved Jun 06, 2025, 08:51 AM UTC

1. resolved Jun 06, 2025, 08:51 AM UTC

   We've now resolved the incident. Thanks for your patience.

Read the full incident report →