Daily Outage History

We've identified an issue that's causing some users to fail to join calls. The daily-js library has to download a bundle of additional JavaScript as part of joining a call. This bundle is downloaded from c.daily.co, which is using Amazon's CloudFront CDN. They aren't reporting issues yet, but we're engaging with their support to figure out why this is happening.

We're receiving reports of a few other regions experiencing similar connection issues. If you're monitoring client errors and you see messages that start with "Failed to load call object bundle https://c.daily.co/....", you're being affected by this. We're working with AWS to get to the bottom of this.

We've confirmed through several different end users that this is a DNS issue. Affected users from multiple regions have been able to join calls by pointing DNS to Google or CloudFlare, using IPs like 1.1.1.1 or 8.8.8.8 for DNS. Obviously, this solution doesn't scale; we're working with AWS to identify the root cause of the DNS resolution issue for c.daily.co.

Cloudflare has posted an incident concerning intermittent DNS failures for .co domains: https://www.cloudflarestatus.com/incidents/z3b5zxjtp6g1 This aligns with the troubleshooting we've done so far. Some internet discussions suggest that it can be somewhat ISP-dependent, but this is unconfirmed. If this is indeed related to the .co domain itself, it would mean that it affects participants' ability to join calls, but it could also affect the ability to make API requests to api.daily.co. Updates to follow.

The .co registry appears to be experiencing issues. Cloudflare's recursive DNS is affected (and they've acknowledged it). Many regional ISPs, such as AT&T in the southeast US, rely on Cloudflare's DNS to power their own DNS. Unlike Cloudflare, other DNS providers, like Google and Quad9, are serving .co records from stale cache. If they stop doing this, then anyone using those DNS services will start to experience these same failures. This won't be fully resolved until the .co registry comes back online. In the meantime, different DNS providers may see problems come and go. All of Daily's services remain online and functional, so if your users can resolve .co hostnames, they can use our services without problems.

Cloudflare has posted that they've implemented a fix, and they are monitoring the results. We're still not certain that Cloudflare is the actual root cause of 100% of our affected users, but we'll be monitoring error rates.

Cloudflare has resolved their status incident, but we're unsure if that means the underlying issue is resolved. We're continuing to monitor our own error rates and health checks.

The rate of bundle download failures has decreased, but it's also tracking our overall usage volume through the day. The number of affected users appears to be very small, but we still have tests that can replicate the failure. Unfortunately this is completely out of our control. If you have users that are continuing to be affected, you can suggest that they use Google's or Quad9's DNS servers at 1.1.1.1 or 9.9.9.9.

Our tests just confirmed that DNS resolution is no longer returning errors for .co domains in our tests. This issue has been resolved!

Our upstream carrier partner has identified the root cause of the issue impacting SIP/PSTN dial-in (inbound) calling and has implemented a fix. Service is beginning to recover, though some degradation may still be present while the fix fully propagates. We are actively monitoring and working to confirm full restoration of inbound calling. Outbound (dial-out) calling remains unaffected.

We've seen an increase in "Failed to load call object bundle" errors within the past hour. This happens when the browser is unable to download our JavaScript bundle from the CDN. If this happens to you or your users, you should be able to refresh the page and join your call successfully. We're working with our CDN provider to identify the issue.