Some customers are having trouble accessing ChargeOver

Update timeline

1. investigating Nov 12, 2024, 05:05 PM UTC

   We are receiving reports that some customers are having trouble accessing the ChargeOver app. We are investigating further. All ChargeOver services appear to be operating normally, but it looks like some web traffic may be having problems accessing ChargeOver services.

2. investigating Nov 12, 2024, 05:35 PM UTC

   All ChargeOver services are operating normally. Some customers are having trouble accessing ChargeOver. At this time, we believe this is due to a carrier network routing issue outside of ChargeOver. We are still investigating and working with our data center providers to get more information.

3. monitoring Nov 12, 2024, 05:50 PM UTC

   All ChargeOver services are operating normally, and are accessible to everyone now. A networking or connectivity incident occurred with our data center provider (Flexential) which prevented some customers from accessing ChargeOver. We are still waiting for full details, and then will post an update. We continue to monitor connectivity. We believe any impact is now resolved, and everyone should be able to access ChargeOver at this time. An update will follow with more details.

4. resolved Nov 12, 2024, 06:10 PM UTC

   All connectivity issues are resolved now. Our primary data center (Flexential - Chaska) experienced a networking issue which impacted the ability to connect to ChargeOver services for some customers.

5. postmortem Dec 02, 2024, 05:29 PM UTC

   ## Incident details On November 12th, some customers had trouble accessing ChargeOver. This was a networking issue at the primary data center ChargeOver is hosted within. ## Root cause On 12 November at 10:26 CT, ChargeOver began receiving automated reports of connectivity issues. The issue was tracked down to a networking issue at the data center ChargeOver’s primary location is within \(Flexential in Chaska, MN\). The problem was immediately escalated to Flexential engineers who identified an internet circuit at Flexential’s Chicago POP as the source of the issue. Flexential opened a ticket with the circuit provider who confirmed there was an issue with packets over a certain size traversing the device due to an MTU misconfiguration. The circuit provider confirmed that they fixed the issue, however Flexential kept production traffic off the link while more investigation is performed. Total impact to service was around 41 minutes. ## Incident timeline * Nov 12 - 10:26am CT - ChargeOver’s automated uptime monitoring identifies and alerts our engineering team of an issue * Nov 12 - 10:53am CT - ChargeOver identifies the issue as a bandwidth/networking issue outside of the ChargeOver network, and escalates the issue to Flexential engineering * Nov 12 - 11:05am CT - Flexential resolves the issue ## Remediation plan ChargeOver is working with our data center provider Flexential to improve monitoring and connectivity.