Zaptec incident

Zaptec Portal slow/unresponsive

Major Resolved View vendor source →

Zaptec experienced a major incident on February 17, 2025 affecting Portal, lasting 2h 31m. The incident has been resolved; the full update timeline is below.

Started
Feb 17, 2025, 12:13 PM UTC
Resolved
Feb 17, 2025, 02:45 PM UTC
Duration
2h 31m
Detected by Pingoru
Feb 17, 2025, 12:13 PM UTC

Affected components

Portal

Update timeline

  1. investigating Feb 17, 2025, 12:13 PM UTC

    We are currently investigating this issue.

  2. resolved Feb 17, 2025, 02:45 PM UTC

    This incident has been resolved.

  3. postmortem Feb 18, 2025, 12:04 PM UTC

    `Due to increased load on one of the critical endpoints from a third party, we experienced saturation on the database level. At February the 17th 12:56 CET, we saw degradation of the API related to reads (OCPP and load balancing was not affected). We identified the root cause and introduced mitigating actions on the offending third party. At 14:55, the system stabilised and at 19:12, the third party had been rate limited on the problematic endpoint. We know the problematic part of the critical endpoint and will plan to address it accordingly.`