Dubber had 27 outages in the last 2 years totaling 691h 5m of downtime — averaging 1.1 incidents per month.
There were 27 Dubber outages since July 19, 2024 totaling 691h 5m of downtime. Each is summarised below — incident details, duration, and resolution information.
We are currently investigating reports of errors being returned from the US API. Recording & portal availability is not impacted.
We have identified an issue and are working on a resolution.
The issue has now been resolved, and we will continue to monitor.
We are currently investigating an issue affecting Dubber's Microsoft Teams Pause & Resume App. The issue is not impacting the desktop client however the browser version is unavailable. Call recording is NOT impacted by this incident, only the Pause & Resume app.
A fix has been deployed to all regions, the browser version of the Pause and Resume App is now functioning as expected.
Dubber are aware of a Cloudflare incident which is impacting access to the Dubber portal. Cloudflare are aware of the issues and are working to resolve the problems. Dubber will periodically provide updates on the Cloudflare issues which can also be seen here: https://www.cloudflarestatus.com/
Dubber has continued to monitor our portal interfaces and we are seeing no issues accessing Dubber portals and we are satisfied that Cloudflare has resolved the issue and this issue is being closed.
We are aware of an issue where Dubber recording bots are appearing as participants in Microsoft Teams meetings. Our investigation confirms this behaviour is not caused by Dubber. We have reported the issue to Microsoft, who are currently investigating the cause. Recording and playback functionality remain fully operational and are not impacted.
This incident has been resolved.
Dubber are aware of an ongoing AWS incident, see https://health.aws.amazon.com/health/status for details. This may be impacting Dubber services in the US-East region, and we therefore recommend that partners in this region with redundancy in place failover to another region. Dubber are continuing to monitor the situation closely.
Dubber are continuing to monitor the ongoing AWS incident.
AWS have confirmed their incident to be resolved.
Dubber is investigating an issue with recording playback for the US East Coast BroadWorks accounts. Will investigate and provide updates as they are obtained. We are aware of this problem and are working to identify and resolve this problem.
Dubber is continuing to investigate the issues regarding call playback for BroadWorks customers in the US East.
We continue to review and troubleshoot. We are aware that Playback AND recording download are currently impacted.
As we continue working this issue we understand the impact is being felt in the US Dubber portal generally and not limited in scope to BroadWorks accounts. We are aware of the issues downloading and playing recordings back. We are identifying the root cause and taking actions to correct this issue. Please continue to monitor the status page for updates on this issue.
Dubber has identified the issue causing the delayed processing of recording files and has taken corrective actions to clear this backlog which has already reduced by 33%. Dubber is continuing to review ways to expedite the clearing of the backlog.
This incident has been resolved.
Dubber is investigating an issue with our technology partner relating to an API outage. We are following updates from our partner and will provide more details as they are given.
Dubber's customer facing API services are currently being impacted by an outage on our partner's platform. Dubber is monitoring the situation and will provide updates and guidance as we receive them.
The issue we are experiencing at the moment is an interruption with our API gateway causing issues accessing Dubber platform via api.dubber.net. Recording is not impacted and recording playback is fine via the Dubber portal itself. Recordings are not impact We will continue to work with our partner and will provide updates as we receive them.
Our API Partner has identified their issue and is implementing a fix, we will closely monitor their progress and provide updates.
Our partner has implemented a fix and is seeing recovery in some regions. They are continuing to work to mitigate the API issues for all impacted regions. Dubber confirms we can see items recovering and backlogs processing. We will continue to monitor for updates.
Our API partner has resolved their issue and have resumed normal service. Dubber systems are processing through a backlog of items but we also confirm that everything is operational with this partner from our perspective. This issue is resolved and Dubber will continue to monitor our partner's performance. Recordings were not impacted in any way. API functions have returned to normal.
Dubber are aware of a problem that may be impacting the ability to process calls and payments. We are investigating as a matter of urgency and will provide regular updates.
The issue has been identified and a fix is being implemented.
A fix has been implemented and the issue looks to be resolved. We will be closely monitoring for at least the next 30 minutes and will provide further updates in due course.
This incident has been resolved.
We are investigating reports that recordings have stopped updating in the US Dubber portal. We will provide more detail when this has been identified.
Dubber is continuing to work to identify the issues impacting the US Dubber portal accounts. We will identify and remedy the issues as swiftly as possible
We found the problem and have taken corrective action. We can see call recordings resuming in the US and we will continue to monitor the nodes to verify this issue has been rectified.
This incident has been resolved.
We are aware that some customers in the US Region are experiencing delaying in call recordings being available in the Dubber portal. We are investigating this incident and will provide further details as we have them.
We are continuing to investigate this incident. We will provide further updates when we have them.
The issue has been identified and Dubber is working towards a resolution. We can confirm that all call records have been captured, there have been no loss of call recordings.
We are continuing to work on a fix for this issue.
This incident has been resolved.
We are currently investigating an issue accessing the Insights dashboard in the US Dubber portal. We are aware of this and will be identifying and correcting the issue. Will update. This appears to be limited to the US Region only.
We had a minor issue that limited access to our Insights dashboard in the US region briefly and this has been corrected and is now back in full operation.
We are aware of a slow down or degraded performance in the postings of call recordings to user accounts and are investigating at this time. There is no evidence that this is impacting call recording itself it is just a delayed posting of recordings at this time. We will update with further information as we investigate this.
Update, we have confirmed recordings continue to be processed normally however we are working on the backlog of recordings that has built up. We are working to expedite the processing of these queued up recordings to catch back up to near real time. We will update as this progresses
Update, we are working to improve the performance of Dubber portal to bring the backlog back under control. Our team is engaged and working on this. Recordings continue to happen without impact based on all available feedback and we see the backlog of recordings and are monitoring those queues and our server performance. Thanks for your patience
We have implemented some fixes to resolve the problem and we are watching the backlog of recordings trend downwards. This issue will resolve given time. We are continuing to monitor the performance and we are taking additional steps to expedite this recovery. Thanks again for your patience. Note, it is possible during this recovery process that recordings might appear in user accounts but for a short period might not be play-able this is a normal process and will also correct itself during this recovery period.
We continue to monitor the recovering of the platform and the backlog of pending recordings is moving downwards as expected. This issue will be all caught up in the coming hours and we will continue to keep an eye on it until the backlog is completely resolved.
We have restored service and recordings are appearing in the portal for playback. A small number of customers may still experience slight delays for the next few hours.
Dubber is aware of an issue intermittently impacting MS Teams call initiation which we are currently investigating. Please note recording is not impacted.
We are continuing to investigate this issue.
This incident is now resolved and MS Teams calls are now initiating as expected.
After investigation, and liaising with Microsoft, Microsoft confirmed they experienced a power event in the North Europe region, which impacted their services in this region. Microsoft have confirmed that power and services were fully restored.
Dubber is aware of an SSO issue impacting portal access which we're currently investigating. Please note recording of calls is not impacted.
We are continuing to investigate this issue.
This incident has been resolved.
Dubber is investigating delays of call recordings appearing in the Dubber portal in our US and CA regions. We are continuing to capture recordings.
We have resolved the incident with call recordings now appearing in the portal within usual timeframes. If you continue to experience issues please contact Dubber Support.
We are aware that some customers using Dubber on Microsoft Teams are experiencing delays in accessing recordings. Please be assured that calls are being recorded and all recordings will be available once the incident is resolved. We are working to resolve the incident as soon as possible and will provide regular updates.
Dubber has restored service with calls appearing in the Dubber portal as expected. If you continue to experience service difficulties please contact Dubber support for assistance.
We have had reports of call recording issues between MS Teams and Dubber. We are exploring this issue with priority and will work to resolve this in the shortest time possible.
We are continuing to investigate the issue. This has been escalated and the required engineering teams are enguaged.
We have taken action to address the incident and currently monitoring the systems.
The team are continuing to investigate with the highest priority.
Our monitoring is indicating the incident is resolved. We are continuing to monitor systems.
We are aware that some customers are experiencing issues accessing the Dubber portal. Please be assured that this incident is impacting portal access only and that calls are being recorded. We are working to resolve the incident as soon as possible and will provide regular updates.
This incident has been resolved.
We are investigating an incident that is impacting the Payment Services for some customer. We are working to resolve the incident as soon as possible and will provide regular updates.
We are continuing to investigate this issue.
Corrective actions as been completed and we believed the incident is resolved. We are continuing to monitor verify payments are working as expected.
We are aware that some customers are experiencing an issue with the playback of recordings in the Dubber portal. Please be assured that this incident is impacting call playback only and calls are being recorded. All calls will be available for playback when this incident is resolved. We are working to resolve the incident as soon as possible and will provide regular updates.
This incident is now resolved. Playback is now operational.
We are aware that some customers are experiencing issues accessing the Dubber portal when using single sign on. Please be assured that this incident is impacting portal access only. Recording is not impacted by this incident. We are working to resolve the incident as soon as possible and will provide regular updates.
Dubber Engineering has resolved the problem. Users can access the portal through single sign on (SSO).
One of our suppliers has informed us of a service degrading issue, which resulted in an impact to some customers ability to hit Dubber's API service. The supplier has informed us that the root cause has been identified, and a mitigation has been put in place. The issue is therefore believed to be resolved, however Dubber are continuing to monitor. Call recording remains fully functional and is unaffected.
Our supplier identified the root cause of the issue to be an invalid DNS Start Of Authority (SOA), and applied mitigating action at ~14:06 UTC (Oct 01, 2024). The changes have mostly propagated and the issue is resolved for most customers, however the exact recovery time will vary depending on client's DNS provider, and how the cache is managed.
Our supplier has now classified this issue as fully resolved.
We are aware that some customers are experiencing delays in call recordings being available for playback in the Dubber portal. Please be assured that this incident is impacting call processing only and calls are being recorded. All calls will be available for playback when this incident is resolved. We are working to resolve the incident as soon as possible and will provide regular updates.
We are aware that some customers are experiencing delays in call recordings being available for playback in the Dubber portal. As we continue to work on this issue, be assured that this incident is impacting call processing only and calls are being recorded. All calls will be available for playback when this incident is resolved. We are working to resolve the incident as soon as possible and will provide regular updates.
We are aware that some customers are experiencing delays in call recordings being available for playback in the Dubber portal. Please be assured that this incident is impacting call processing only and calls are being recorded. All calls will be available for playback when this incident is resolved. We estimate that this issue will be resolved this afternoon. We will continue to provide regular updates until this issue is solved.
The cause of the incident has been identified and corrective action is being completed. Delayed call recordings are being processed into the portal and will be available for playback in the coming hours. We will continue to provide regular updates.
Dubber is working hard to resolve an issue that some customers are experiencing when trying to access their recordings. Though there is a delay in call recordings being available for playback, this incident is only affecting processing as call are being recorded. Once this issue is resolved, all calls will be available for playback. We are working to resolve the incident as soon as possible and will provide regular updates.
Dubber acknowledges that some of our US Dubber Portal customers are experiencing delays in call recordings ingestion. This incident is impacting call processing only and calls are still being recorded. All calls will be available for playback when this incident is resolved. We are working to resolve the incident as soon as possible and will provide regular updates.
This incident has been resolved.
Some customers have reported issues with logging in to the UK portal. Dubber is currently investigating. There is no impact on the capture of call recordings.
The issue has been identified and Dubber is working on resolution.
A fix has been implemented and we are monitoring the results.
The issue has been resolved.