Uscreen experienced a minor incident on October 12, 2024 affecting Storefront, lasting 5h 16m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Oct 12, 2024, 06:04 PM UTC
Our team is currently investigating an issue with our 3rd Party Server Partners regarding some web pages being unavailable. We will continue to update this incident as our investigation proceeds.
- investigating Oct 12, 2024, 06:46 PM UTC
Our team continues to investigate this issue with DigitalOcean, our 3rd party server partner. In the meantime, you can switch your DNS to our other servers. Please note that DNS changes may take some time to propagate, depending on your DNS host, and in some cases, this can take several hours. Here are the steps to do that: 1. Go to your Domain Settings. 2. Click "Use a different domain" and select a Uscreen subdomain. 3. Repeat the process, but this time select "Custom domain" to generate a new set of DNS records. This will begin the migration to our newer servers.
- identified Oct 12, 2024, 08:03 PM UTC
If you are affected by the issue with DigitalOcean, please follow these steps: 1. Go to your Domain Settings. 2. Click "Use a different domain" and select a Uscreen subdomain. 3. Repeat the process, but this time select "Custom domain" to generate a new set of DNS records.
- resolved Oct 12, 2024, 10:49 PM UTC
Our 3rd Party Server Partner, DigitalOcean, has restored service. Web Pages should be back up and running now. If your site was impacted by this outage, we highly recommend following our Migration guide and moving your site to our more reliable servers. Please visit https://help.uscreen.tv/en/articles/9492966 for more information.
- postmortem Oct 15, 2024, 02:38 PM UTC
**Incident Summary** On October 12, 2024, users with domains connected to our legacy DigitalOcean load balancer experienced outages due to a large-scale attack targeting DigitalOcean servers. **Impact** Customers using the older infrastructure suffered service disruptions as the load balancer was temporarily disabled. **Root Cause** The legacy DigitalOcean infrastructure, which lacks modern DDoS protection, became vulnerable. **Resolution** We strongly recommend updating domain records to point to our new Google Cloud \(GCP\) load balancers. GCP’s built-in Cloud Armor provides robust DDoS protection, ensuring higher reliability and security. **Next Steps** To prevent future issues, we urge all customers to move to our GCP infrastructure, which offers far superior performance and protection compared to older systems on DigitalOcean. This transition ensures the platform’s resilience against future DDoS attacks and other risks. [For more information regarding this migration please review our help article](https://help.uscreen.tv/en/articles/9492966-migrating-your-domain-to-our-new-servers).