Daily Outage History

We're seeing a recurrence of the problems with RegistryDNS.co and AT&T DNS from Monday. Users trying to join calls from affected regions using ISP DNS may get "Unable to join call" errors. We know specifically that AT&T DNS servers 68.94.157.1 and 68.94.156.1 are affected. We've been working hard on solving this problem since the incident on Monday. We're releasing daily-js 0.91.0 shortly to address this.

We expect to have daily-js 0.91.0 available within the next 60 to 90 minutes. The update includes automated failover to `dailywebrtc.com` or `dailywebrtc.net` if `daily.co` is unreachable. If you haven't updated daily-js in a while, you can do preliminary testing with 0.90.0 in order to be ready to release an update with 0.91 as soon as it's ready. 0.91 will only contain this feature, as well as a few dependency version updates for security. If you have customers with restrictive networks, they may need to add `*.dailywebrtc.com` and `*.dailywebrtc.net` to their firewall/security allowlists. These domains are relatively new, so they may be flagged by Cisco app firewalls and similar security appliances.

We've identified an issue that's preventing some users from being able to join calls, resulting in errors like "Unable to join call" in the browser. This is being caused by failures from the DNS servers that serve the .co TLD itself. Several third-party DNS providers like Google, Cloudflare, and Quad9 are working around this in order to continue to make .co domains available. But users that are using their ISP's DNS service (specifically AT&T users in the southeast US and/or Texas) are experiencing intermittent problems accessing any .co domain, including daily.co. Unfortunately, the only workaround is to encourage your users to use a third-party DNS provider, like Cloudflare or Quad9. We're working on addressing this long-term.

We are working on implementing a fallback to a .com domain so that users on ISPs that are not fixing this will not see any issues starting calls.

Our upstream carrier partner has identified the root cause of the issue impacting SIP/PSTN dial-in (inbound) calling and has implemented a fix. Service is beginning to recover, though some degradation may still be present while the fix fully propagates. We are actively monitoring and working to confirm full restoration of inbound calling. Outbound (dial-out) calling remains unaffected.

We've seen an increase in "Failed to load call object bundle" errors within the past hour. This happens when the browser is unable to download our JavaScript bundle from the CDN. If this happens to you or your users, you should be able to refresh the page and join your call successfully. We're working with our CDN provider to identify the issue.

We're tracking a Deepgram incident that is causing problems with transcription events, as well as captions in Daily Prebuilt: https://status.deepgram.com/incidents/rp0jzltym9kw

We're continuing to see very brief occurrences of slow or missing transcription data from the Deepgram streaming endpoint. We're talking to Deepgram engineers and continuing to monitor the situation.

We’ve identified the cause of the issue. While a backlog of webhooks remains, we’re actively working to reduce it. Webhook deliveries will continue to be delayed for some time, but no data is being lost.

We're continuing to work through a backlog of webhook events, and the delay is starting to decrease. It will still take some time to get fully caught up.

We're seeing an increased rate of errors when dialing out SIP/PSTN calls.

We've rolled back the release that was in process when we started seeing the errors, which seems to have resolved the issue. We'll be monitoring for any regressions.

AWS is addressing an issue with DynamoDB that is causing problems with many of their services in us-east-1. We have seen issues loading the dashboard and docs sites, and for a few hours we've been unable to log in to Statuspage to update this site. We're sorry about that delay. Up until a few minutes ago, our metrics suggested that core call functionality wasn't being impacted. We just saw a small number of call failures in us-east-1, though. We're continuing to monitor the situation and do what we can to minimize this outage's impact on you and your customers. We'll post more information here as we have it.

AWS is continuing to address issues in us-east-1. We're still seeing normal activity levels across our APIs and call servers for this time of day, so we have no reason to believe that any your customers' calls are actually being affected. Part of AWS's mitigation efforts involve limiting the rate at which we can start new call servers. Out of an abundance of caution, we're temporarily removing us-east-1 from our list of available regions for new call sessions. This means your users will likely connect to a call through us-west-2 instead. This failover happens automatically and transparently, and you don't need to do anything. We've already scaled up call servers in us-west-2 to handle the increase in sessions. We'll keep you posted as we continue to monitor the situation.

We're in the process of rolling back a release to address this issue.

This appears to be related to the Google Cloud outage posted here: https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW We'll post here as we learn more.

We're seeing some increases in response time from our internal API endpoint that returns connection information to daily-js. This may be causing "room lookup timed out" errors for users attempting to join calls. We believe this is related to the ongoing GCP incident.

We had a brief spike in response time, but platform operations appear normal since approximately 20:05 UTC. We're continuing to monitor for any follow-on effects, and we'll leave this status post open until various upstream providers close their respective status incidents.

SignalWire has confirmed that they made an unannounced change that's causing the problem, and they're working to revert it. We're also working on a quick production update to work around the issue if they can't revert quickly enough.

We've identified an issue causing degraded ("broken up" or "choppy") audio between some Daily sessions and SignalWire SIP/PSTN endpoints. PSTN dialout and PIN dialin do not seem to be affected. SIP dialout seems to be moderately affected. PIN-less PSTN dialin and SIP dialin seem to be experiencing a much higher proportion of affected calls.

We've identified an issue with how Daily and SignalWire are negotiating audio codecs for incoming SIP/PSTN calls. We're working on an update that will change the default audio codec used between Daily and SignalWire to Opus. Once this is deployed, you may still experience audio issues if you've explicitly set your SIP audio codec to PCMU as described here: https://docs.daily.co/guides/products/dial-in-dial-out/sip#sip-dial-in-audio-and-video

We're getting reports of delayed audio from some customers using SIP dialin and dialout. When the phone user joins the call, they can talk and others will hear them, but the phone user won't hear any audio from other call participants (bot or human) for the first 20-30 seconds. We're continuing to troubleshoot the issue, and we'll post here as soon as we have more info.

We've identified the issue, and we're testing a fix in our staging infrastructure. We'll post another update when we've deployed the fix.

We've temporarily disabled the component that sends webhooks.

We're addressing an issue with an internal database that's causing problems with existing meetings, as well as starting new ones. Your users are likely seeing some failures when trying to join meeting sessions, and users in ongoing sessions are seeing occasional meeting moves. We'll post more information as soon as soon as it's available.