Zaptec experienced a major incident on February 14, 2025 affecting API, lasting 2h 5m. The incident has been resolved; the full update timeline is below.
Affected components
Update timeline
- investigating Feb 14, 2025, 12:35 PM UTC
We are currently investigating this issue.
- monitoring Feb 14, 2025, 01:11 PM UTC
A fix has been implemented and we are monitoring the results.
- resolved Feb 14, 2025, 02:41 PM UTC
This incident has been resolved.
- postmortem Feb 17, 2025, 09:02 AM UTC
On Friday 14th of February at 12:35 CET the cloud-based Sense service, which provides Automatic Power Management \(APM\) functionality to Zaptec installations, began running below normal capacity, leading to a disruption of functionality. The degradation was identified at 12:37 CET. Our engineers started investigating logs and metrics for the service. We identified the root cause as an overlooked manual configuration change. This was restored to original state and the service was again operational at 13:55 CET. We have updated our monitoring to identify similar problems much faster in the future, and are reviewing our processes to prevent manual configurations from persisting in our production environment.