Degraded Performance & Database Unavailability

Update timeline

1. identified Apr 16, 2026, 07:47 AM UTC

   We are currently facing degraded performance due to high database loads on our production infrastructure

2. identified Apr 16, 2026, 07:58 AM UTC

   We have identified the issue and are currently deploying a fix

3. monitoring Apr 16, 2026, 08:22 AM UTC

   We have deployed a fix and are monitoring the impact

4. identified Apr 16, 2026, 08:35 AM UTC

   We continue working on our database perforamce. Degraded performances might be possible

5. identified Apr 16, 2026, 08:47 AM UTC

   Due to infrastructure changes that we are currently making, the database is currently not available

6. identified Apr 16, 2026, 09:07 AM UTC

   We are facing infrastructure problems with our database being unresponsive on Microsoft Azure's side. We have already contacted Microsoft Azure support and are collaborating closely with them to resolve this issue as soon as possible

7. identified Apr 16, 2026, 09:20 AM UTC

   We are awaiting response from Microsoft Azure support. We are expecting a response until 12:00 at the latest, as this a Severity A issue on MS Azure's side.

8. identified Apr 16, 2026, 10:05 AM UTC

   We just got the information from Microsoft Azure support that our containers went unhealthy with no fault on our part and they are working on getting them available as soon as possible. In the meanwhile we are already working on setting up a fall-back database instance.

9. monitoring Apr 16, 2026, 10:25 AM UTC

   While the issue on Microsoft Azure's side has not been resolved yet, we updated up our infrastructure to use a different database instance. All services are functional again. We continue verifying.

10. resolved Apr 16, 2026, 10:31 AM UTC

    While we still have no response from Microsoft Azure, we have verified that our system is functional again.

11. postmortem Apr 16, 2026, 10:42 AM UTC

    # **Incident: Degraded Performance & Database Unavailability** **Duration:** April 16, 2026, 09:30 - 12:31 CEST \(~3 hours\) **Severity:** Major **Affected components:** All \(Authentication & User Management, Core API\) ### Summary On April 16, 2026, the NetCero production system experienced degraded performance followed by a full outage. The initial trigger was high database load caused by a software update. Recovery was significantly delayed by an infrastructure failure at Microsoft Azure, where a database scaling operation stalled and rendered the database unreachable for over an hour. ### Timeline * **09:30** - High database load detected on production infrastructure. * **09:58** - Root cause identified; fix deployed. * **10:22** - Fix deployed; monitoring impact. * **10:35** - Continued work on database performance; degraded performance persists. * **10:47** - Infrastructure scaling initiated; database becomes unavailable during the process. * **11:07** - Database unresponsive on Microsoft Azure's side. Azure support contacted. * **11:20** - Awaiting response from Microsoft Azure support \(Severity A ticket\). * **12:05** - Microsoft Azure confirms containers went unhealthy with no fault on our part. Fallback database instance setup begins. * **12:25** - Infrastructure switched to a new database instance. All services functional. Monitoring continues. * **12:31** - Incident resolved. System verified as fully operational. Microsoft Azure is still working on restoring the initial database. ### Root cause A software update caused increased database load. The subsequent attempt to scale the database infrastructure at Microsoft Azure failed unexpectedly, rendering the database unreachable for over an hour. Microsoft Azure confirmed that the failure was on their side. The combination of the initial performance issue and the Azure-side infrastructure failure caused the extended outage. ### Actions taken * Software performance optimizations have been implemented and deployed * Database migrated to a new, dedicated instance * Database capacity and configuration improved * Automated checks are being added to our development pipeline to detect similar performance issues early * Further infrastructure improvements are planned to increase resilience We apologize for the inconvenience. ## Update \(17.04.2026\) We have received a Root Cause Analysis from Microsoft Azure regarding the infrastructure failure that extended the outage . **Azure confirmed that the issue was entirely on their side and not caused by any customer configuration or action on our part.** ### **Root Cause \(Azure Infrastructure\)** When we initiated the database scaling operation, the PostgreSQL engine - which runs as a managed container within Azure's platform - entered an unhealthy state. This prevented the scaling process from completing, leaving the server stuck in an "Updating" state and making it completely unavailable. In simpler terms: Azure's own internal process for resizing our database failed mid-operation, and their system was unable to recover automatically. The database was locked in a broken state until Azure's support team intervened. **Key findings from Azure's analysis:** * The failure originated in Azure's platform-managed container orchestration, not in our database or configuration * Azure's built-in health checks detected the problem but the automated recovery workflows were insufficient to resolve it in a timely manner * Manual intervention by Azure's backend team was required to restore service ### **Mitigation** Azure has reinitialized the affected infrastructure through their internal recovery process. We have since migrated to a new, dedicated database instance to avoid any residual risk from the affected server. We are pursuing SLA service credits with Microsoft Azure, as the infrastructure-side downtime exceeded the guaranteed availability commitment.