Is Servd down?

Due to the PHP critical vulnerability due to be disclosed today we have blocked all multipart upload requests on all projects and on servd.host. This will create 403 errors for image and other file uploads. We are working on restoring normal functionality, however due to the critical nature and trivial exploitation of the bug, this is the safest course of action for the time being.

We are currently waiting for PHP to release fixes which is due at 14:00 UTC. At this time we will prep them for use in Servd projects which we are expecting to take 20-30 minutes. Once these have been confirmed as effective at mitigating the issue we will release them for all projects. We will update this status page once they have been released. Once released - any new bundles created in our dashboard which are using PHP 8.2 + will contain the fix. Deploying these will automatically re-enable multipart requests for your project. This timeline may change in two scenarios: 1. PHP do not release their updates on time or it is delayed for another reason 2. The fixes within the PHP releases do not actually address the attack we have been seeing today (there has been no official confirmation that the two are related, but there is a very strong likelihood that they are the same thing).

We've received details of the vulnerability and are awaiting official patched PHP images. As soon as they're available, we'll be publishing new base bundles, and allowing customers to deploy them to their environments. In the meantime, we're shoring up our defensive edge rules based on the released info.

We're currently blocked on releasing patched PHP versions, due to delays in the release of the official PHP docker images. We'll be exporting base bundles as soon as they're made available.

The patched official PHP docker images have been published. We'll have patched PHP versions available shortly.

Patched official PHP images have been published, but we're not yet seeing any exported image layers to them. This is blocking our efforts to release patched PHP versions to Servd. We're regularly monitoring for changes.

The official PHP docker images have completed publishing, we're now in the process of making new PHP versions available.

We've now released patched PHP versions for 8.5.12, 8.4.27, 8.3.36 or 8.2.35. Please proceed to build and sync new bundles to all your environments. When a safe bundle is deployed to an environment, our platform will automatically disable the multipart upload block. If you need assistance doing so, please reach out to us via [email protected]. For projects running PHP 8.1 or lower, we recommend updating your sites to PHP 8.2 or higher as soon as possible. For these projects, the multipart upload block will remain in place for the foreseeable future but will be automatically lifted when a safe bundle is deployed.

We are beginning to automatically rebuild deployed bundles for all unpatched project environments running PHP 8.2 or higher, replacing them with builds running a safe version of PHP. However, the quickest and safest means to secure your projects is to rebuild your deployed bundles manually as soon as you can. We've also just published a post regarding this incident in our blog https://servd.host/blog/mitigation-of-critical-php-cves We'll be keeping it updated with additional information as it becomes available.

We are currently rolling out updates to all eligible projects that have not yet manually updated. We are expecting this to be completed within the next few hours. We are also working on backporting the patches for projects using PHP 7.4, 8.0 and 8.1. We are hoping to roll out those updated tomorrow. Some projects are not currently able to be updated due to a loss of connection between Servd and the project's git repo - we will be emailing the owners of these projects once we have completed updates for other projects.

Backported patches for PHP 7.4, 8.0 and 8.1 have been created and we plan to roll these out to projects today. Once complete, we will be in contact with the owners of any projects that **have not** been able to be updated for any reason with further steps to take. At that time we will also close off this incident.

We have automatically updated all projects that can be updated from our side. We'll reach out to the owners of those that remain. Thanks for your patience while we worked through getting everyone patched!

We've investigating signs of degraded CPU performance in our CA-EAST-1 cluster, which is impacting response times.

We're seeing performance improving, however are still investigating the underlying cause.

Performance has returned to normal levels, thanks for your patience.

A degradation of network connectivity in two data centres is causing timeouts and other connection errors with traffic between project components and out to the public internet. We're liaising with our server provider to identify and rectify the root cause.

The incident has now been resolved. Two individual problems were discovered and mitigated: - DNS lookup failures in our London based clusters. This was caused by a configuration error with an upstream DNS resolver causing it to reject any dnssec-enabled queries. - Packet loss in us-east-3/4/5 clusters. This was caused by a hardware issue with a hypervisor which caused degraded networking performance with a subset of worker nodes. The fault has been resolved.

We are in the process of cycline them out. Some project components may be unavailable for a few seconds while we do so.

We've now resolved the incident. Thanks for your patience.

Between 09:36 and 09:40 UTC, our US-EAST-6 cluster received a large-scale DDoS attack which briefly caused the cluster's load balancers to return 502 Bad Gateway responses to a subset of projects hosted in US-EAST-6. At 09:37 UTC, our automated Cloudflare-backed DDoS protections activated and began blocking malicious traffic. Meanwhile, the clusters load balancers scaled up and error rates dropped back to normal levels. In response, we've permanently increased the baseline capacity of the clusters load balancer and are prioritising measures to better absorb the initial shock of similar DDoS attacks in the future. Apologies for the interruption.