Updox experienced a notice incident on May 28, 2025 affecting Updox and Video Chat and 1 more component, lasting 7d 7h. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating May 28, 2025, 12:59 PM UTC
Updox is experiencing a disruption in service. Users may not be able to access the application. We are investigating the issue and will post updates every 30 minutes.
- identified May 28, 2025, 01:15 PM UTC
Our team has determined the cause of the service interruption within Updox and is working on a resolution. Thank you for your patience.
- monitoring May 28, 2025, 01:34 PM UTC
We have implemented a fix for the disruption in service and the issue is resolved. All systems are operational at this time. We will continue to monitor.
- investigating May 28, 2025, 04:01 PM UTC
Updox is experiencing a disruption in service. Users may not be able to access the application. We are investigating the issue and will post updates every 30 minutes.
- identified May 28, 2025, 04:36 PM UTC
Our team has determined the cause of the service interruption within Updox and is working on a resolution. Thank you for your patience.
- monitoring May 28, 2025, 04:56 PM UTC
We have implemented a fix for the disruption in service and the issue is resolved. All systems are operational at this time. We will continue to monitor.
- investigating May 29, 2025, 01:01 PM UTC
We are currently investigating this issue.
- monitoring May 29, 2025, 01:54 PM UTC
We have implemented a fix for the disruption in service and the issue is resolved. All systems are operational at this time. We will continue to monitor. We recognize the same issue has occurred several times this week and apologize for the inconvenience. We will share more information and timelines shortly but are working to replace network hardware causing issues as fast as possible.
- monitoring May 29, 2025, 04:46 PM UTC
We’re aware of the temporary outage that has been occurring every few hours since yesterday. This downtime is due to a faulty hardware component, and our team is addressing it with urgency. We understand how important uptime is to you and apologize for the inconvenience.
- monitoring May 30, 2025, 06:02 PM UTC
Thank you for your patience. To permanently resolve the intermittent outages caused by the faulty hardware, we will be performing a hardware upgrade during a scheduled maintenance window. Maintenance Window: May 30th 10.00PM ET to 30th May 12.00AM ET Expected Impact: Temporary service unavailability Our team is working to complete this upgrade as quickly and safely as possible to restore full reliability. We appreciate your continued understanding and will provide updates once the maintenance is complete.
- monitoring May 31, 2025, 04:06 AM UTC
We have completed the planned maintenance and experienced a 3 minute temporary outage during the cutover. Thank you for your patience. The team continues to monitor performance and networking prior to marking the incident resolved.
- monitoring Jun 02, 2025, 07:45 PM UTC
Our team continues to monitor the network upgrade that was performed late Friday night. All systems are observed to be operational, however we are seeing a slight delay in inbound and outbound faxes for one fax carrier. We are fine tuning configuration and working with the fax carrier to resolve the slight delay.
- monitoring Jun 02, 2025, 11:00 PM UTC
The fax delay is resolved and all systems are observed to be operational. The team continues to monitor performance and networking prior to marking the incident resolved. We appreciate your patience and understanding.
- monitoring Jun 03, 2025, 06:34 PM UTC
Our team continues to monitor the network upgrade that was performed late Friday night. All systems are observed to be operational, however we are seeing a slight delay in inbound and outbound faxes for one fax carrier. We are fine tuning configuration and working with the fax carrier to resolve the slight delay.
- monitoring Jun 03, 2025, 09:45 PM UTC
Incident Overview From May 27 through May 30, 2025, Updox experienced service disruptions due to recurring firewall failures at one of our data centers. Updox has redundant firewalls to ensure continued operations but both firewalls experienced failures intermittently. These failures impacted platform availability and caused cascading effects across multiple services. How We Responded Our engineering team worked diligently to identify and resolve the root cause of the disruptions. On May 27, the earliest signs of the issue emerged when faxing services were disrupted due to network storage problems triggered by firewall instability. As the disruptions continued, we identified recurring failures in the firewall hardware at one of our data centers as the root cause. These failures intermittently interrupted network traffic, impacting both platform availability and dependent services. Throughout the incident, our team collaborated closely with the data center to restart the firewalls during each occurrence. We also made additional adjustments to stabilize the platform and closely monitored system performance to minimize user impact. To fully resolve the issue, we replaced the failing firewall hardware with new equipment during a planned maintenance window on May 30 between 10:00 PM and 12:00 AM. Corrective and Preventative Actions To ensure these issues do not happen again, we are taking the following steps: • Firewall Replacement: The faulty firewall hardware was replaced during a planned maintenance window on May 30, 2025, between 10:00 PM and 12:00 AM. This upgrade restored stable network performance and resolved the root cause of the disruptions. • Post-Cutover Monitoring: We are actively monitoring network and system performance metrics following the firewall replacement to ensure continued stability and responsiveness. During this monitoring phase, we are partnering with our fax carriers to fine tune our network peering together to resolve temporary fax delays. We know that many of you rely on Updox every day to support your operations. We sincerely apologize for this disruption and are committed to strengthening our systems to prevent it from happening again. Thank you for your patience and continued trust.
- resolved Jun 04, 2025, 08:01 PM UTC
Over the last several days, the team has monitored diligently and in partnership with our fax carriers made several adjustments to resolve intermittent fax delays. Today, in partnership with our fax carriers, we have seen normal fax delivery times through morning and peak hours and consider the incident resolved. Please see the earlier update for the RCA and should you have questions or additional issues to report, please contact [email protected].