Is Daily down?

We're seeing a recurrence of the problems with RegistryDNS.co and AT&T DNS from Monday. Users trying to join calls from affected regions using ISP DNS may get "Unable to join call" errors. We know specifically that AT&T DNS servers 68.94.157.1 and 68.94.156.1 are affected. We've been working hard on solving this problem since the incident on Monday. We're releasing daily-js 0.91.0 shortly to address this.

We expect to have daily-js 0.91.0 available within the next 60 to 90 minutes. The update includes automated failover to `dailywebrtc.com` or `dailywebrtc.net` if `daily.co` is unreachable. If you haven't updated daily-js in a while, you can do preliminary testing with 0.90.0 in order to be ready to release an update with 0.91 as soon as it's ready. 0.91 will only contain this feature, as well as a few dependency version updates for security. If you have customers with restrictive networks, they may need to add `*.dailywebrtc.com` and `*.dailywebrtc.net` to their firewall/security allowlists. These domains are relatively new, so they may be flagged by Cisco app firewalls and similar security appliances.

daily-js 0.91.0 is now available on npm and GitHub. If your app uses call object mode, you can update your dependency to that version and redeploy, and this issue should be resolved for you. We're publishing an update to our networking guide soon, but in the meantime, anything that mentions a `.daily.co` hostname in that doc is also available at `.dailywebrtc.com` and `.dailywebrtc.net`. We'll have updates for embedded Prebuilt and direct link customers soon. We'll leave this incident open while RegistryDNS.co remains offline.

The RegistryDNS.co servers continue to be intermittently unavailable, but AT&T's DNS servers seem to be more responsive. We encourage you to update to daily-js 0.91.0 as soon as possible to avoid future issues for your users.

We've identified an issue that's preventing some users from being able to join calls, resulting in errors like "Unable to join call" in the browser. This is being caused by failures from the DNS servers that serve the .co TLD itself. Several third-party DNS providers like Google, Cloudflare, and Quad9 are working around this in order to continue to make .co domains available. But users that are using their ISP's DNS service (specifically AT&T users in the southeast US and/or Texas) are experiencing intermittent problems accessing any .co domain, including daily.co. Unfortunately, the only workaround is to encourage your users to use a third-party DNS provider, like Cloudflare or Quad9. We're working on addressing this long-term.

We are working on implementing a fallback to a .com domain so that users on ISPs that are not fixing this will not see any issues starting calls.

CentralNIC confirmed that there was indeed an issue with the .co nameservers, and they resolved it at approximately 06:30 UTC today. Our monitoring of both their DNS servers and downstream providers confirmed this. We're continuing to work urgently on updating our infrastructure to make our services available on alternate hostnames. We'll post updates to our Networking Guide when we do: https://docs.daily.co/guides/privacy-and-security/corporate-firewalls-nats-allowed-ip-list We'll also provide a root cause analysis for this incident in the next few days.

tester

Resolved by chad

test

We're receiving reports of some users having problems connecting to Daily calls. It seems to be localized to the Texas area. We're investigating.

We've identified an issue that's causing some users to fail to join calls. The daily-js library has to download a bundle of additional JavaScript as part of joining a call. This bundle is downloaded from c.daily.co, which is using Amazon's CloudFront CDN. They aren't reporting issues yet, but we're engaging with their support to figure out why this is happening.

We're receiving reports of a few other regions experiencing similar connection issues. If you're monitoring client errors and you see messages that start with "Failed to load call object bundle https://c.daily.co/....", you're being affected by this. We're working with AWS to get to the bottom of this.

We've confirmed through several different end users that this is a DNS issue. Affected users from multiple regions have been able to join calls by pointing DNS to Google or CloudFlare, using IPs like 1.1.1.1 or 8.8.8.8 for DNS. Obviously, this solution doesn't scale; we're working with AWS to identify the root cause of the DNS resolution issue for c.daily.co.

Cloudflare has posted an incident concerning intermittent DNS failures for .co domains: https://www.cloudflarestatus.com/incidents/z3b5zxjtp6g1 This aligns with the troubleshooting we've done so far. Some internet discussions suggest that it can be somewhat ISP-dependent, but this is unconfirmed. If this is indeed related to the .co domain itself, it would mean that it affects participants' ability to join calls, but it could also affect the ability to make API requests to api.daily.co. Updates to follow.

The .co registry appears to be experiencing issues. Cloudflare's recursive DNS is affected (and they've acknowledged it). Many regional ISPs, such as AT&T in the southeast US, rely on Cloudflare's DNS to power their own DNS. Unlike Cloudflare, other DNS providers, like Google and Quad9, are serving .co records from stale cache. If they stop doing this, then anyone using those DNS services will start to experience these same failures. This won't be fully resolved until the .co registry comes back online. In the meantime, different DNS providers may see problems come and go. All of Daily's services remain online and functional, so if your users can resolve .co hostnames, they can use our services without problems.

Cloudflare has posted that they've implemented a fix, and they are monitoring the results. We're still not certain that Cloudflare is the actual root cause of 100% of our affected users, but we'll be monitoring error rates.

Cloudflare has resolved their status incident, but we're unsure if that means the underlying issue is resolved. We're continuing to monitor our own error rates and health checks.

The rate of bundle download failures has decreased, but it's also tracking our overall usage volume through the day. The number of affected users appears to be very small, but we still have tests that can replicate the failure. Unfortunately this is completely out of our control. If you have users that are continuing to be affected, you can suggest that they use Google's or Quad9's DNS servers at 1.1.1.1 or 9.9.9.9.

Our tests just confirmed that DNS resolution is no longer returning errors for .co domains in our tests. This issue has been resolved!

This incident has been resolved.