Seravo incident

Availability issues in cluster fi-perko

Critical Resolved View vendor source →
Started
Mar 12, 2026, 03:10 AM UTC
Resolved
Mar 12, 2026, 07:47 AM UTC
Duration
4h 36m
Detected by Pingoru
Mar 12, 2026, 03:10 AM UTC

Affected components

fi-perko cluster

Update timeline

  1. investigating Mar 12, 2026, 03:10 AM UTC

    We are currently investigating this issue.

  2. identified Mar 12, 2026, 04:33 AM UTC

    The issue has been identified and a fix is being implemented.

  3. monitoring Mar 12, 2026, 06:21 AM UTC

    A fix has been implemented and we are monitoring the results.

  4. resolved Mar 12, 2026, 07:47 AM UTC

    This incident has been resolved.

  5. postmortem Mar 13, 2026, 09:37 AM UTC

    # Notice of a Server Outage on March 12, 2026 On Thursday March 12, 2026, a failure related to the server resource usage caused an outage in the cluster fi-perko. The outage was caused by a newly implemented security feature which led to increased resource usage. Monitoring failed to notice and pinpoint the increased resource usage in time, which led to the outage. The outage was aggravated by an unrelated networking change which led to delay in recovery. This caused an outage to all sites in the fi-perko cluster. The problem was noticed immediately by our on-call monitoring processes, and measures were immediately implemented to begin recovery. The disruption first began at 04:00 \(UTC\+2\), escalated at 05:00 and ended at 08:19 \(UTC\+2\). We apologise for any inconvenience caused by the disruption. ## Timeline \(all timestamps UTC\+2 \(EET\)\) * 12.3.2026 03:47 Partial outage * 12.3.2026 03:52 On-call performs initial recovery measures * 12.3.2026 04:00 Recovered from partial outage * 12.3.2026 05:00 Incident escalated again * 12.3.2026 05:10 On-call attempts recovery measures * 12.3.2026 05:15 Full outage * 12.3.2026 05:27 On-call performs recovery measures * 12.3.2026 06:17 Recovered from the outage - begin site recovery * 12.3.2026 08:19 All affected sites were back online ## Follow-Up Action As a result of the incident, we at Seravo have identified the need for the following measures: * Testing new features with extended data-sets to be able to catch issues caused by the sheer volume of processed data * Improvements to system resource monitoring

Looking to track Seravo downtime and outages?

Pingoru polls Seravo's status page every 5 minutes and alerts you the moment it reports an issue — before your customers do.

  • Real-time alerts when Seravo reports an incident
  • Email, Slack, Discord, Microsoft Teams, and webhook notifications
  • Track Seravo alongside 5,000+ providers in one dashboard
  • Component-level filtering
  • Notification groups + maintenance calendar
Start monitoring Seravo for free

5 free monitors · No credit card required