ChargeOver experienced a notice incident on July 8, 2025, lasting —. The incident has been resolved; the full update timeline is below.
Update timeline
- resolved Jul 08, 2025, 03:22 PM UTC
ChargeOver was inaccessible from approx 3:27 CDT, to 4:57 CDT. This was a networking-system related outage. We are still investigating root cause. A postmortem will follow.
- postmortem Jul 28, 2025, 03:04 PM UTC
# Incident details On July 5th, a firewall-related issue caused ChargeOver to be inaccessible for approximately 90 minutes. Customers were unable to log in or access the ChargeOver application during this time frame. # Root cause At approximately 3:27pm CT, one of the high-availability firewalls experienced an error which caused it to stop passing traffic into/out-of ChargeOver’s internal network. Although the firewall was configured in a way which should trigger automatic fail-over \(we use industry standard Netgate firewalls, which use CARP to share IP and network status information and monitor for fail-over\) to a secondary firewall, the automatic fail-over did not occur. ChargeOver staff were immediately and automatically notified, and after troubleshooting were able to resolve the outage by power-cycling the affected firewall. # Incident timeline * 3:27pm CT - Firewall experiences an error, which stops some inbound/outbound traffic. ChargeOver staff are immediately alerted. * 3:28pm CT - ChargeOver staff respond and begin troubleshooting. * 4:57pm CT - Problem resolved after power-cycling the affected firewall. # Remediation plan We are still investigating what caused the error which took the firewall offline, as well as why the firewall did not fail over correctly to a secondary firewall when the error occurred. As we investigate further, we’ll have a better idea of whether this requires replacement or reconfiguration of the firewall. We are improving documentation, to enable our engineering team to troubleshoot quicker/isolate root cause quicker for future incidents. We are also working with data center staff to establish some better monitoring tools to help us troubleshoot quicker/isolate root cause quicker.