Auvik Outage History

Started Jul 27, 2025, 05:21 PM UTC· resolved Jul 31, 2025, 01:40 PM UTC

1. identified Jul 27, 2025, 05:21 PM UTC

   Auvik will undergo emergency maintenance. During this time, you may not be able to log into Auvik. There may also be interruptions to your network monitoring.

2. identified Jul 27, 2025, 08:22 PM UTC

   Most sites are now operational. However, some users are still experiencing issues with site loading or logging in. Our team is actively working to resolve these remaining problems. Monitoring is not impacted. We appreciate your patience.

3. identified Jul 27, 2025, 10:55 PM UTC

   We are continuing to verify sites and restore access. Some users may still experience issues with data not loading on their sites. Monitoring is not impacted.

4. investigating Jul 28, 2025, 02:09 PM UTC

   Affected Services: Services based on hierarchical data based on user roles (sub-sites, user roles, etc.) Cluster(s): Still determining. More than just US4 We are investigating reports of degraded performance. Our team is working diligently to identify the root cause and restore optimal performance. Impact: While we address the situation, users may experience access to some functionality with sites and sub-sites Services: Monitoring and alerting are currently working as expected Next Steps: We will provide updates as we learn more. We appreciate your patience as we work to resolve this issue.

5. investigating Jul 28, 2025, 03:07 PM UTC

   Affected Services: Services based on hierarchical data based on user roles (sub-sites, user roles, etc.) Cluster(s): Appears to include several sites We are investigating reports of degraded performance. Our team is working diligently to identify the root cause and restore optimal performance. Impact: While we address the situation, users may experience access to some functionality with sites and sub-sites Services: Monitoring and alerting are currently working as expected Next Steps: We will provide updates as we learn more. We appreciate your patience as we work to resolve this issue.

6. identified Jul 28, 2025, 04:11 PM UTC

   Affected Services: Services based on hierarchical data, based on user roles (sub-sites, user roles, shared collectors) Cluster(s): Several Cluster Our team has identified the root cause of the degraded performance and is currently investigating a solution to restore normal service levels. Impact: While we work on the resolution, users may continue to experience performance issues. Correction to the prior post, sites with disconnected shared collector issues are not currently being monitored. Next Steps: We will provide updates as the situation progresses. Your patience is greatly appreciated, and we regret any inconvenience you may be experiencing.

7. identified Jul 28, 2025, 06:50 PM UTC

   Affected Services: Services based on hierarchical data, based on user roles (sub-sites, user roles, shared collectors) Cluster(s): Several Cluster Our team has identified the root cause of the degraded performance and is currently investigating a solution to restore normal service levels. Impact: While we work on the resolution, users may continue to experience performance issues. Issues with email alerts, shared clusters, and login issues on US5 are being addressed. Data display or data displaying incorrectly is still under investigation Next Steps: We will provide updates as the situation progresses. Your patience is greatly appreciated, and we regret any inconvenience you may be experiencing.

8. monitoring Jul 28, 2025, 09:41 PM UTC

   Our team has implemented fixes for the ongoing issues. We are in recovery mode, but we are still addressing a few outstanding problems. Impact: Users now have access to their client in the US5 cluster. Shared collectors now have connectivity. There is still an outstanding issue regarding site hierarchy and user access to sites. Alerting and monitoring are working for those sites. Next Steps: We will continue to work on these issues and will provide updates as they are relevant. We appreciate your patience as we work through this issue.

9. monitoring Jul 29, 2025, 08:41 AM UTC

   Our team has implemented fixes for the ongoing issues. We are in recovery mode, but we are still addressing a few outstanding problems. Impact: There is still an outstanding issue regarding site hierarchy and user access to sites. This is a rendering issue. The sites are not disassociated from their parent site in the back-end tables. The All Sites pages are not loading data for some clients. Alerts and monitoring are working for both of these ongoing issues. Next Steps: We will continue to work on these issues and will provide updates as they are relevant. We expect continued progress throughout the day today We appreciate your patience as we work through this issue.

10. monitoring Jul 29, 2025, 03:46 PM UTC

    Our team has implemented fixes for the ongoing issues. We are in recovery mode, but we are still addressing a few outstanding problems. Impact: There is still an outstanding issue regarding site hierarchy and user access to sites. This is a rendering issue. The sites are not disassociated from their parent site in the back-end tables. The All Sites pages are not loading data for some clients. Alerts and monitoring are working for both of these ongoing issues. Next Steps: We will continue to work on these issues and will provide updates as they are relevant. We expect continued progress throughout the day today. We appreciate your patience as we work through this issue.

11. monitoring Jul 29, 2025, 08:58 PM UTC

    Our team has implemented fixes for the ongoing issues. We are in recovery mode, but we are still addressing a few outstanding problems. Impact: There is still an outstanding issue regarding site hierarchy and user access to sites. This is a rendering issue. The sites are not disassociated from their parent site in the back-end tables. The All Sites pages are not loading data for some clients. Alerts and monitoring are working for both of these ongoing issues. Next Steps: We will continue to work on these issues and will provide updates as they are relevant. We expect continued progress throughout the evening. We appreciate your patience as we work through this issue.

12. monitoring Jul 30, 2025, 09:03 AM UTC

    Our team has implemented fixes for the ongoing issues. We are in recovery mode, but we are still addressing a few outstanding problems. Impact: There is still an outstanding issue regarding site hierarchy and user access to sites. This is a rendering issue. The sites are not disassociated from their parent site in the back-end tables. The All Sites pages are not loading data for some clients. Alerts and monitoring are working for both of these ongoing issues. Next Steps: We will continue to work on these issues and will provide updates as they are relevant. We expect continued progress throughout the morning. We appreciate your patience as we work through this issue.

13. monitoring Jul 30, 2025, 01:48 PM UTC

    Our team has implemented fixes for the ongoing issues. We are in recovery mode, but we are still addressing a few outstanding problems. Impact: There is still an outstanding issue regarding site hierarchy and user access to sites. This is a rendering issue. The sites are not disassociated from their parent site in the back-end tables. The All Sites pages are not loading data for some clients. Significant progress has been made overnight in fixing the sites affected. Alerts and monitoring are working for both of these ongoing issues. Next Steps: We will continue to work on these issues and will provide updates as they are relevant. We expect continued progress throughout the morning. We appreciate your continued patience as we work through this issue.

14. monitoring Jul 30, 2025, 05:52 PM UTC

    Our team has implemented fixes for the ongoing issues. We are still in recovery mode. Impact: The issues regarding site hierarchy and user access to sites, along with sites disassociated from their parent site, have been resolved. We continue to address the ancillary problems from this incident. Next Steps: We request that if you are still experiencing issues with your tenant(s), contact support. We appreciate your continued patience as we work through this issue.

15. monitoring Jul 30, 2025, 11:07 PM UTC

    Our team has implemented fixes for the ongoing issues. We are still in recovery mode. Impact: We continue to address the ancillary problems from this incident. We are continuing to address issues to ensure stability. Next Steps: We request that if you are still experiencing issues with your tenant(s), contact support. We appreciate your continued patience as we work through this issue.

16. monitoring Jul 31, 2025, 10:15 AM UTC

    Our team has implemented fixes for the ongoing issues. We are still in recovery mode. Impact: We continue to address the ancillary problems from this incident. We are continuing to address issues to ensure stability. Next Steps: We request that if you are still experiencing issues with your tenant(s), contact support. We appreciate your continued patience as we work through this issue.

17. resolved Jul 31, 2025, 01:40 PM UTC

    The incident has been fully resolved. Regular service has been restored, and all systems are operating as expected. Impact: Users should no longer experience any issues related to this incident. Service has been fully restored. We apologize for the degradation in services. We thank you for your understanding. If you continue to experience issues, please don't hesitate to contact our support team. We will post an RCA after an internal investigation.

Read the full incident report →

Started Jul 26, 2025, 04:43 PM UTC· resolved Jul 27, 2025, 11:51 AM UTC

1. investigating Jul 26, 2025, 04:43 PM UTC

   We are currently experiencing a service disruption. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Impact: Users may experience that sites are not loading properly and intermittent disruption in monitoring. Next Steps: We will update this information as more details become available. We appreciate your patience as we work to restore full functionality.

2. investigating Jul 26, 2025, 05:20 PM UTC

   We are continuing to investigate. We are rebooting all services in us4. Services are partially available in us1 and au1.

3. identified Jul 26, 2025, 07:42 PM UTC

   We’ve identified a potential cause of the issue and are actively working on a fix. Some sites are beginning to load again, but intermittent issues may still persist for some users.

4. identified Jul 26, 2025, 08:12 PM UTC

   us1 is now operational. We are continuing to work through partial issues in au1 and us4, where some sites are still experiencing service disruptions.

5. identified Jul 26, 2025, 10:09 PM UTC

   au1 is now operational. Recovery efforts remain ongoing in region us4, where a maintenance window was initiated at 09:38 PM UTC to support remediation. We’ll continue to share updates as we make progress.

6. monitoring Jul 27, 2025, 03:36 AM UTC

   Our team has implemented a fix for the disruption in us4, and the services are returning to normal. We continue to monitor the situation to ensure stability and confirm that the service remains fully functional.

7. resolved Jul 27, 2025, 11:51 AM UTC

   The incident has been fully resolved. Regular service has been restored, and all systems operate as expected. Impact: Users should no longer experience any issues related to this service disruption. [RCA] We thank you for your understanding. If you continue to experience issues, please don't hesitate to contact our support team. We will post an RCA after an internal investigation.

8. postmortem Aug 11, 2025, 03:33 PM UTC

   # Service Disruption - Hierarchical Data Display Issues ## Root Cause Analysis ### Duration of incident Discovered: July 26, 2025 14:00 UTC Resolved: July 31, 2025 12:45 UTC ### Cause Following a core system upgrade on July 26, 2025, multiple clusters experienced degraded performance due to unexpected issues during service initialization. This led to corruption in certain hierarchical data structures, which in turn impacted various user experiences. A core infrastructure failure caused an overload in internal system processes, preventing certain backend services from initializing properly. As a result, critical hierarchical user-role data and related settings failed to load or loaded incorrectly across several environments. ### Effect The disruption impacted customers across multiple regions. Effects included: * Missing custom settings such as alert preferences and interface configurations. * Alert notifications being sent to incorrect recipients. * Inability to access dashboards or view accurate site data. * Site maps failing to render correctly. * Login issues for end-users in some environments. * Inconsistent or missing hierarchical relationships in site selectors. * Temporary loss of monitoring due to disassociated shared collectors. * These issues collectively degraded service functionality, limited access for internal support teams, and disrupted monitoring workflows. ### Action taken _All times are in UTC_ **07/26/2025** **11:00** – Core upgrade initiated. **13:27** – Initial service failures observed. Engineering begins recovery processes. **14:19** – Backend query failures reported. Engineering continues to recover and stabilize services. **16:24** – Incident response team mobilized. **07/27/2025** _\(Services restored throughout the day\)_ Hierarchy services replayed and clusters rebooted to stabilize services. Alerting system functionality restored. **7/28/2025** _\(Services restored throughout the day\)_ Shared monitoring agents reassociated. Back-end service migrations initiated for affected tenants. **7/28/2025** _\(Services restored throughout the day\)_ Repair scripts run to reset affected data and restore processing pipelines. **7/30/2025-07/31/2025** _\(Services restored throughout the day\)_ Corrupted tenants identified and corrected. All affected clusters verified for service integrity. **07/31/2025** **12:45** - Incident considered resolved. ### Future consideration\(s\) * Improve validation checks during post-upgrade procedures to avoid cascading service impacts. * Temporarily pause specific background services \(e.g., data cleaners and processors\) during upgrades until core services are stable. * Implement automated detection for corrupted tenant hierarchies or missing role-based configurations. * Revisit default alert notification behavior to avoid unintended mass-notifications.

Read the full incident report →

Started Jul 25, 2025, 07:24 PM UTC· resolved Jul 25, 2025, 09:13 PM UTC

1. investigating Jul 25, 2025, 07:24 PM UTC

   Collectors have been disconnected in us3 since 14:35 ET (18:35 UTC). We are investigating the issue.

2. monitoring Jul 25, 2025, 08:27 PM UTC

   We have restarted the affected systems, and collectors are beginning to reconnect. We are monitoring the recovery closely.

3. resolved Jul 25, 2025, 09:13 PM UTC

   The incident has been resolved.

4. postmortem Jul 30, 2025, 04:27 PM UTC

   # Service Disruption - Collectors offline for clients on the US3 cluster ## Root Cause Analysis ### Duration of incident Discovered: Jul 25, 2025 14:45 - UTC Resolved: Jul 25, 2025 17:54 - UTC ### Cause Backend nodes were removed from the US3 cluster during a routine cleanup effort intended to optimize efficiencies.. This removal unintentionally included the backend hosting the root tenant, leading to disconnected collectors within our US3 cluster. ### Effect Collectors in the US3 cluster lost connectivity with customer sites, resulting in disruptions to data collection and monitoring services. This caused temporary gaps in visibility across affected environments. ### Action taken _All times are in UTC_ **07/25/2026** **14:45** - Engineering notices that collector connections are beginning to fail. **18:28** – Tenants not loading observed by the team. **18:35** – Outage reports increase. **18:40** – SEV declared, and the root cause investigation begins. **18:48** – Backends re-added to balance load. **19:00** – Alternate issues ruled out. **19:23** – Root tenant backend identified as missing. **19:25** – Cluster restart initiated. **20:18** – Services begin recovery. **21:07** – Incident resolved. ### Future consideration\(s\) * Strengthen the backend removal process to confirm the root tenant is excluded.

Read the full incident report →

Started Jul 03, 2025, 05:14 PM UTC· resolved Jul 03, 2025, 06:44 PM UTC

1. investigating Jul 03, 2025, 05:14 PM UTC

   Affected Services: Access to part of the UI Service not impacted: Monitoring and alerting functionality Description: We are currently experiencing degraded performance with access to the user Interface. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Impact: Users may experience issues with their user interface. Next Steps: We will update this information as more details become available. Thank you for your patience as we work to restore full functionality.

2. identified Jul 03, 2025, 05:35 PM UTC

   Affected Services: Access to part of the UI Service not impacted: Monitoring and alerting functionality Description: We are currently experiencing degraded performance with access to the user Interface, and are currently investigating a solution to restore normal service levels. Impact: While we work on the resolution, users may continue to experience interruptions in the User interface. Next Steps: We will provide updates as the situation progresses. Your patience is greatly appreciated, and we regret any inconvenience you may be experiencing.

3. monitoring Jul 03, 2025, 06:25 PM UTC

   User Affected Services: Access to part of the UI Cluster(s): US3 Description: Our team has implemented a fix for the performance issues affecting the User interface. We monitor the system to ensure stability and confirm that performance remains at expected levels. Impact: System performance should be restored to normal. We will continue to monitor for any irregularities. Services [services not affected] are not impacted. Next Steps: A final update will be provided once we confirm the issue is resolved. We appreciate your patience as we work through this issue.

4. resolved Jul 03, 2025, 06:44 PM UTC

   Affected Services: Access to part of the UI Cluster(s): US3 Description: The performance issue affecting access to the UI has been fully resolved, and normal operations have resumed. All systems are functioning as expected. Impact: Users should no longer experience any performance-related issues. Next Steps: Service has been restored. We apologize for the disruption and appreciate your continued patience. If you continue to experience issues, please contact our support team.

5. postmortem Jul 29, 2025, 05:50 PM UTC

   # Service Degraded - API and UI interruption on the US3 Cluster ## Root Cause Analysis ### Duration of incident Discovered: Jul 3, 2025, 17:10 UTC Resolved: Jul 3, 2025, 18:30 UTC ### Cause A service component in the US3 region experienced a critical failure that triggered a crash loop, rendering an internal service inoperable. ### Effect Users experienced disruption to both the user interface and API functions in the US3 environment. ### Action taken _All times are in UTC_ **07/03/2025** **17:10** Deployment of user interface \(UI\) change. **17:30** Engineering alerted to issues with the UI and APIs on US3. **17:40** Attempted rollback of the most recent deployment version. **17:50** Crash loop persisted despite rollback, indicating the issue was not caused by regression. **17:52** Engineering alerted to issues with the UI and APIs on US3. Introduced more detailed diagnostic logging and restarted the affected services. **18:12** Comparison against other environments revealed the same error had occurred without incident elsewhere, suggesting environmental context was a key factor. **18:30** Services stabilized and returned to normal. ### Future consideration\(s\) * Review environmental differences that contributed to varying behavior between clusters. * Consider adding a safe-fail mechanism for deployments to prevent full-service crash loops.

Read the full incident report →

Started Jun 11, 2025, 12:00 AM UTC· resolved Jun 11, 2025, 12:00 PM UTC

1. resolved Jun 11, 2025, 12:19 PM UTC

   The processing of legacy alerts for clients on the US1 cluster was delayed on May 11, 2025, from 00:00 to 12:00 UTC. The service has been restored, and alerts have been processed through the system with the proper time codes. No other services or clusters were affected. Auvik apologizes for the delay in alerts and will post an RCA after conducting an internal analysis.

2. postmortem Jun 11, 2025, 07:49 PM UTC

   # Service Disruption - Legacy alerts for sites on the US1 cluster are delayed ## Root Cause Analysis ### Duration of incident Discovered: Jun 11, 2025 00:00 - UTC Resolved: Jun 11, 2025 12:00 - UTC ### Cause The job manager that processes legacy alerts restarted due to memory issues and did not recover properly. ### Effect This resulted in a backlog of posting legacy alerts for customers on the US1 cluster. ### Action taken _All times are in UTC_ **06/11/2025** **00:00** The processing job for legacy alerts fails on the US1 cluster. **10:00** Auvik Support receives a support ticket regarding missing alerts since the previous evening. Engineering begins the investigation. **10:00-11:00** Engineering determines the cause for the alerts not processing. The legacy alerting service is reinitiated, and additional memory is allocated to the work. Alerts begin to flow. **12:00** The delayed legacy alerts are fully processed, and the system is reported up to date. ### Future consideration\(s\) * Auvik will enhance its internal monitoring to detect and respond to this failure more effectively.

Read the full incident report →

Started Jun 04, 2025, 01:42 AM UTC· resolved Jun 04, 2025, 02:16 AM UTC

1. investigating Jun 04, 2025, 01:42 AM UTC

   Affected Services: Auvik Dashboard Cluster(s): AU1 Description: We are currently experiencing degraded performance loading the Auvik Dashboard. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Impact: Users may experience slower load times of the Auvik Dashboard. Monitoring services are not impacted. Next Steps: We will update as more information becomes available. Thank you for your patience as we work to restore full functionality.

2. investigating Jun 04, 2025, 01:43 AM UTC

   We are continuing to investigate this issue.

3. resolved Jun 04, 2025, 02:16 AM UTC

   This incident has been resolved.

4. postmortem Jun 11, 2025, 02:19 PM UTC

   # Service Disruption - AU1 Cluster Performance Degradation ## Root Cause Analysis ### Duration of incident Discovered: Jun 3, 2025 – 00:10 UTC Resolved: Jun 3, 2025 – 02:16 UTC ### Cause The performance degradation was caused by resource limitations in the database system supporting the AU1 cluster. These limitations temporarily prevented the system from efficiently cleaning up and processing data. This led to slower load times until the underlying resources were increased. ### Effect Customers connected to the AU1 cluster experienced slow performance when accessing the Auvik Web UI. Pages were taking longer than usual to load, and in some cases, data within the interface appeared delayed or incomplete. While monitoring systems remained unaffected, the responsiveness was degraded during the incident window. ### Action taken _All times are in UTC_ **06/03/2025** **00:10** Customer Support escalated the slowness to Engineering. Investigation began immediately. **00:25** Signs of performance issues in the backend database were detected. **01:00** Resource contention was identified as the source of the slowdown. **02:07** Resources allocated to the database were increased. **02:13** UI responsiveness returned to normal. **02:16** Incident declared resolved and public status page updated. ### Future consideration\(s\) * Implement database health monitoring to identify issues proactively. * Tune database cleanup and optimization settings to match usage patterns better.

Read the full incident report →

Started May 10, 2025, 02:00 PM UTC· resolved May 10, 2025, 02:00 PM UTC

1. resolved May 12, 2025, 02:01 PM UTC

   Towards the end of Auvik's scheduled maintenance window, on 5/10/2025, Engineering noticed some loading issues with sites on several clusters. Upon investigation, it was determined there was an issue with the data flow between systems. This interruption required Auvik to extend its maintenance window. Auvik was able to bring each cluster's tenants up throughout the process. This work was considered completed at 21:00 EDT. Auvik will furnish an RCA after an internal review has been completed.

2. postmortem May 16, 2025, 02:26 PM UTC

   # Service Disruption - Sites are not available after maintenance ## Root Cause Analysis ### Duration of incident Discovered: May 10, 2025 13:04 - UTC Resolved: May 11, 2025 01:00 - UTC ### Cause A scheduled upgrade of the system failed to complete successfully. ### Effect Auvik functionality was impacted after the upgrade was implemented. This began a cascade of product functionality failures that required reimplementing the upgraded version using a stepped restart of Auvik. ### Action taken _All times are in UTC_ **04/10/2025** **11:00** Upgrade process begins on core components. **12:45** An issue is detected affecting data replication, and some clusters experience connectivity problems. **13:05** Engineering begins active investigation into the connectivity issue. **13:24** Recovery actions initiated for affected clusters. **13:49** Maintenance window extended to address ongoing issues. **14:00-14:05** Impacted clusters begin recovering. **14:21** Post-upgrade validation reveals a new issue affecting dashboard display in most regions. **14:35** Further analysis confirms the issue affects multiple clusters. **15:00** Deeper technical investigation begins to isolate the root cause, which is suspected to involve backend services. **17:04** Root cause identified as an issue with a core data processing component. **17:20** Mitigation strategies explored; decision made to re-attempt the upgrade with a modified approach. **18:30-20:17** Second upgrade process begins; similar issues surface in specific regions. **21:00-21:25** Recovery actions for affected clusters show positive results; services begin to stabilize. **21:30-21:40** Core services successfully rolled out to additional clusters with improved configuration. **23:47** One final cluster exhibits recovery issues, addressed through targeted intervention. **05/11/2025** **00:00-01:00** Final recovery actions completed; all services return to normal. **01:00** Complete system restoration is confirmed. ### Future consideration\(s\) * Implement additional alerting to monitor bandwidth issues on the backend systems more effectively and proactively to prevent bottlenecks. * Complete the improvements that are already in progress. * Mitigate the load placed on all backend systems simultaneously after a maintenance window. * Remove several single-point failure configurations with more scalable configurations.

Read the full incident report →

Started Apr 14, 2025, 10:25 PM UTC· resolved Apr 15, 2025, 02:25 AM UTC

1. investigating Apr 14, 2025, 10:25 PM UTC

   Affected Services: Site availability Cluster(s): US4 Description: We are currently experiencing degraded performance with sites running on the US4 cluster. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Impact: Users may experience connectivity issues with their tenants. Services: None of the other clusters and services are affected. Next Steps: We will provide updates as more information becomes available or within the next hour. Thank you for your patience as we work to restore full functionality.

2. identified Apr 14, 2025, 10:55 PM UTC

   Affected Services: Site availability Cluster(s): US4 Description: Our team has identified the root cause of the degraded performance affecting client site availability in the US4 cluster. We are currently investigating a solution to restore normal service levels. Impact: While we work on the resolution, users may experience connectivity issues as sites become available again. Services: None of the other clusters and services are affected. Next Steps: Our team is actively working to resolve the issue and will provide updates as progress is made or by 23:00 UTC Thank you for your patience as we work to restore full functionality.

3. monitoring Apr 14, 2025, 11:40 PM UTC

   Affected Services: Site availability Cluster(s): US4 Description: Our team has implemented a fix for the issue affecting site connectivity on the US4 cluster. We are waiting for the rest of the sites to be available online. We monitor the situation to ensure stability and confirm that the service remains fully functional. Impact: Services should be operating normally, except for the remaining site, which we are waiting for to become fully available. Services: None of the other clusters and services are affected.. Next Steps: We will provide a final update once all issues are resolved. Thank you for your patience, and we apologize for any inconvenience caused.

4. monitoring Apr 15, 2025, 12:39 AM UTC

   Affected Services: Site availability Cluster(s): US4 Description: Our team has implemented a fix for the issue affecting site connectivity on the US4 cluster. We are waiting for the rest of the sites to be available online. We monitor the situation to ensure stability and confirm that the service remains fully functional. Impact: Services should operate normally, except for the remaining sites, which we are continuing to work to make fully available. Services: None of the other clusters and services are affected.. Next Steps: We will provide a final update once all issues are resolved. Thank you for your patience, and we apologize for any inconvenience caused.

5. monitoring Apr 15, 2025, 01:40 AM UTC

   Affected Services: Site availability Cluster(s): US4 Description: Our team has implemented a fix for the issue affecting site connectivity on the US4 cluster. We are waiting for the rest of the sites to be available online. We monitor the situation to ensure stability and confirm that the service remains fully functional. Impact: Services should operate normally, except for the remaining sites, which we continue working to make fully available. Services: None of the other clusters and services are affected.. Next Steps: We will provide a final update once all issues are resolved. Thank you for your patience, and we apologize for any inconvenience caused.

6. resolved Apr 15, 2025, 02:25 AM UTC

   Affected Services: Site availability Cluster(s): US4 Description: The issue affecting site availability has been fully resolved. Regular service has been restored, and all systems are now operating as expected. Impact: Users should no longer experience issues related to this incident except for select clients we have communicated with. Next Steps: We are preparing a detailed Root Cause Analysis (RCA) report to provide further insights into the incident and preventive measures. Thank you for your patience, and we apologize for any inconvenience caused.

7. postmortem Apr 17, 2025, 03:54 PM UTC

   # Service Disruption - Over 50% of clients on the US4 cluster experienced service interruptions. ## Root Cause Analysis ### Duration of incident Discovered: Apr 14, 2025 19:45 UTC Resolved: Apr 15, 2025 04:05 UTC ### Cause A configuration change related to Meraki Devices. ### Effect About 55% of tenants in US4 became inaccessible due to increased traffic and system load. Action taken _All times are in UTC_ **04/14/2025** **19:45** - Auvik receives internal alerts for abnormal CPU usage on its backend systems for the US4 cluster. **19:50** - Engineering begins an investigation into the issue, actively taking measures to stabilize the system. **20:42** - A large number of sites become inaccessible, and Auvik implements its incident response. **20:42-21:45** - Engineering continues to investigate. **21:45** - A possible root cause of the issue is identified, and Engineering begins recovering sites. **04/14/25-04/15/25** **21:45 - 00:10** - Engineering continues to bring most of the affected sites back online. **04/15/25** **00:10** - All sites, except one client, are back up and accessible. **00:10-01:00** - Auvik continues to work on bringing the last client tenants online and getting them up and running. **01:00** - A root cause is determined for the cause of the incident. Engineering creates mitigation steps. **01:00-03:05** - Mitigation steps are implemented, and the remaining sites of the last client are brought online and accessible. ### Future consideration\(s\) * Auvik has implemented safeguards to prevent a recurrence.

Read the full incident report →

Started Mar 31, 2025, 06:15 PM UTC· resolved Apr 01, 2025, 02:52 PM UTC

1. investigating Mar 31, 2025, 06:15 PM UTC

   Affected Services: Internet Connection Service Cluster(s):US3 Description: We are currently experiencing degraded performance with the internet connection check ping service. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Impact: Users may experience false internet connection disconnects. Services, including other monitoring services, are not impacted. Next Steps: We will update you as more information becomes available or within the next hour. Thank you for your patience as we work to restore full functionality.

2. investigating Mar 31, 2025, 06:54 PM UTC

   Affected Services: Internet Connection Service Cluster(s):US3 Description: We are currently experiencing degraded performance with the internet connection check ping service. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Some clients may see the alerts associated with the Internet connection service missing from the alert dashboard. This is temporary, and these alerts will be restored when the internet connection check service is restored. Impact: Users may experience false internet connection disconnects. Services, including other monitoring services, are not impacted. Next Steps: We will update you as more information becomes available or by 20:00 UTC. Thank you for your patience as we work to restore full functionality.

3. investigating Mar 31, 2025, 08:00 PM UTC

   Affected Services: Internet Connection Service Cluster(s):US3 Description: We are currently experiencing degraded performance with the internet connection check ping service. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Some clients may see the alerts associated with the Internet connection service missing from the alert dashboard. This is temporary, and these alerts will be restored when the internet connection check service is restored. Impact: Users may experience false internet connection disconnects. Services, including other monitoring services, are not impacted. Next Steps: We will update you as more information becomes available or by 21:00 UTC. Thank you for your patience as we work to restore full functionality.

4. investigating Mar 31, 2025, 09:02 PM UTC

   Affected Services: Internet Connection Service Cluster(s):US3 Description: We are currently experiencing degraded performance with the internet connection check ping service. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Some clients may see the alerts associated with the Internet connection service missing from the alert dashboard. This is temporary, and these alerts will be restored when the internet connection check service is restored. Impact: Users may experience false internet connection disconnects. Services, including other monitoring services, are not impacted. Next Steps: We will update you as more information becomes available or by 22:00 UTC. Thank you for your patience as we work to restore full functionality.

5. identified Mar 31, 2025, 09:57 PM UTC

   Affected Services: Internet Connection Service Cluster(s):US3 Description: We are currently experiencing degraded performance with the internet connection check ping service. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Some clients may see the alerts associated with the Internet connection service missing from the alert dashboard. This is temporary, and these alerts will be restored when the internet connection check service is restored. Impact: Users may experience false internet connection disconnects. Services, including other monitoring services, are not impacted. Next Steps: Auvik will disable alerts for clients on the US3 cluster for up to 1 hour starting at 22:00 UTC. This is a preventative measure as we work through false alerts for the internet connection checks. We apologize for the late notice. Thank you for your patience as we work to restore full functionality.

6. identified Mar 31, 2025, 10:19 PM UTC

   Affected Services: Internet Connection Service Cluster(s):US3 Description: We are currently experiencing degraded performance with the internet connection check ping service. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Some clients may see the alerts associated with the Internet connection service missing from the alert dashboard. This is temporary, and these alerts will be restored when the internet connection check service is restored. Impact: Users may experience false internet connection disconnects. Services, including other monitoring services, are not impacted. Next Steps: Auvik will disable alerts for clients on the US3 cluster for up to 1 hour starting at 22:00 UTC. This is a preventative measure as we work through false alerts for the internet connection checks. Clients may experience a slowed UI response time during this work. This UI slowness should be very short if it is noticeable at all. We apologize for the late notice. Thank you for your patience as we work to restore full functionality.

7. identified Mar 31, 2025, 10:59 PM UTC

   Affected Services: Internet Connection Service Cluster(s):US3 Description: We are currently experiencing degraded performance with the internet connection check ping service. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Some clients may see the alerts associated with the Internet connection service missing from the alert dashboard. This is temporary, and these alerts will be restored when the internet connection check service is restored. Impact: Users may experience false internet connection disconnects. Services, including other monitoring services, are not impacted. Next Steps: Auvik has disabled alerts for clients on the US3 cluster. This action will continue for an additional hour until 23:00 UTC. This is a preventative measure as we work through false alerts for the internet connection checks. Clients may experience a slowed UI response time during this work. Any UI slowness should be very short, if noticeable at all. We apologize for the extended window for this action. Thank you for your patience as we work to restore full functionality.

8. identified Mar 31, 2025, 11:48 PM UTC

   Affected Services: Internet Connection Service Cluster(s):All Clusters Description: We are currently experiencing degraded performance with the internet connection check ping service. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Some clients may see the alerts associated with the Internet connection service missing from the alert dashboard. This is temporary, and these alerts will be restored when the internet connection check service is restored. Impact: Users may experience false internet connection disconnects. Services, including other monitoring services, are not impacted. Next Steps: Auvik will perform an emergency cluster restart on US3 tenants at 00:00, which will take approximately 1.5 hours to complete. At this time, Auvik will also perform a 20-minute maintenance window to allow for a restart of the Internet connection service for all of Auvik. We sincerely apologize for the extended window for this action. Thank you for your patience as we work to restore full functionality.

9. identified Mar 31, 2025, 11:49 PM UTC

   We are continuing to work on a fix for this issue.

10. identified Apr 01, 2025, 12:22 AM UTC

    Affected Services: Internet Connection Service Cluster(s):US3 Description: We are currently experiencing degraded performance with the internet connection check ping service. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Some clients may see the alerts associated with the Internet connection service missing from the alert dashboard. This is temporary, and these alerts will be restored when the internet connection check service is restored. Impact: The 20-minute maintenance window for the internet connection service for all clusters has been completed. Services, including other monitoring services, are not impacted. Next Steps: The US3 cluster is still going through its restart process. We sincerely apologize for the extended window for this action. Thank you for your patience as we work to restore full functionality.

11. monitoring Apr 01, 2025, 01:11 AM UTC

    Affected Services: Internet Connection Service Cluster(s):US3 Description: Our team has implemented a fix for the issue affecting the Internet connection ping check for the tenants on the US3 cluster, and performance is returning to normal. We are currently monitoring the situation to ensure stability and confirm that the service remains fully functional. Impact: Services should be operating normally; however, we continue monitoring for irregularities. Next Steps: Tenants on the US3 cluster are still recovering and look healthy. Thank you for your patience, and we apologize for any inconvenience caused.

12. monitoring Apr 01, 2025, 01:25 AM UTC

    Affected Services: Internet Connection Service Cluster(s):US3 Description: Our team has implemented a fix for the issue affecting the Internet connection ping check for the tenants on the US3 cluster, and performance is returning to normal. We are monitoring the situation to ensure stability and confirm that the service remains fully functional. Impact: Services should be operating normally; however, we continue monitoring for irregularities. Next Steps: Tenants on the US3 cluster are still recovering and look healthy. We will continue to monitor the status of the tenants on US3 overnight and report back in the morning. Thank you for your patience, and we apologize for any inconvenience caused.

13. monitoring Apr 01, 2025, 01:54 PM UTC

    Affected Services: Internet Connection Service Cluster(s):US3 Description: Our team has implemented a fix for the issue affecting the Internet connection ping check for the tenants on the US3 cluster, and performance is returning to normal. We monitor the situation to ensure stability and confirm that the service remains fully functional. Impact: Services are operating normally for most sites. We do continue monitoring for irregularities with a few sites that have been contacted Next Steps: Tenants on the US3 cluster are still recovering and look healthy. We are attending to a few sites to regain full functionality. Thank you for your patience, and we apologize for any inconvenience caused.

14. resolved Apr 01, 2025, 02:52 PM UTC

    Affected Services: Internet Connection Service Cluster(s):US3 Description: The issue affecting Internet Connection Ping Checks has been fully resolved. Regular service has been restored, and all systems are operating as expected. Impact: Users should no longer experience any issues related to this incident. Next Steps: We are preparing a detailed Root Cause Analysis (RCA) report to provide further insights into the incident and preventive measures. Thank you for your patience, and we apologize for any inconvenience caused.

15. postmortem Apr 09, 2025, 05:25 PM UTC

    # Service Disruption - Cloud Ping Checks create false alerts on the US3 cluster. ## Root Cause Analysis ### Duration of incident Discovered: Mar 31, 2025, 13:15 - UTC Resolved: Apr 01, 2025, 15:52 - UTC ### Cause The performance of the ping server service on the US3 cluster degraded and produced invalid data. ### Effect The ping server service sent incorrect data based on the internet connection checks to the alerting service, which created large batches of false alerts sent to customers on the US3 cluster. ### Action taken _All times are in UTC_ **03/31/2025** **17:10 -** Ping Server started showing symptoms of degradation. **17:15 -** Internet Connections are marked offline. Customers experience excessive false alert reports based on the cloud ping check service on the US3 cluster. **17:20 -** The Auvik engineering team begins its investigation. **17:20-20:00 -** Auvik continues its investigations and disables the cloud ping service for several large customers on the US3 cluster to prevent excessive alerting once the service is restored. **20:00 -** Auvik resets the ping server service on the US3 cluster. Ping services fail over to the backup primary ping server service. **22:25 -** The primary ping server service load rises to a level that begins impacting customers on other clusters. **04/01/25** **00:00 -** The US3 cluster is restarted to revert cloud ping checks to the US3 cluster ping server services. Auvik notifies the customer where the cloud ping checks were disabled that the service will remain down until engineering can confirm they can be enabled without causing excessive alerting. **01:00-01:25 -** The US3 cluster fully restarts successfully. Functionality is restored for most clients on the US3 cluster. **12:00-15:30 -** Engineering reviews the disabled configurations and disables the responses to the cloud ping check-based alerts. **15:30-15:52 -** Auvik validates that all cloud ping check services and alerts are enabled for all customers on the US3 cluster. Additional clean-up commences. The incident is concluded. ### Future consideration\(s\) * Auvik is building a new cloud ping check server service for the product. This new server service will be rolled out gradually and is expected to be fully deployed into production over the next month. * Our error handling in the service that processes the cloud ping server data has been improved to identify and ignore invalid data. * Addresses will no longer be considered offline when invalid data is received.

Read the full incident report →

Started Feb 24, 2025, 05:45 PM UTC· resolved Feb 24, 2025, 07:32 PM UTC

1. investigating Feb 24, 2025, 05:45 PM UTC

   Affected Services: Clients on the US4 cluster Service not impacted: Clients other clusters Description: We are experiencing degraded performance with tenants on the US4 cluster. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Impact: Users will experience issues with connectivity to their tenants Services: Other clusters are not experiencing issues Next Steps: We will provide updates as more information becomes available or by 18:30 UTC. Thank you for your patience as we work to restore full functionality.

2. identified Feb 24, 2025, 05:49 PM UTC

   Affected Services: Clients on the US4 cluster Service not impacted: Clients other clusters Description: Our team has identified the root cause of the degraded performance affecting tenants on the US4 cluster and is currently investigating a solution to restore normal service levels. Impact: Users will experience issues with connectivity to their tenants Services: Other clusters are not experiencing issues Next Steps: Our team is actively working to resolve the issue and will provide updates as progress is made or by 18:30 UTC Thank you for your patience as we work to restore full functionality.

3. identified Feb 24, 2025, 06:29 PM UTC

   Affected Services: Clients on the US4 cluster Service not impacted: Clients other clusters Description: Our team has identified the root cause of the degraded performance with tenants on the US4 cluster. We are seeing tenants becoming available to normal service levels. Impact: While we work on the resolution, users start to see their tenants become responsive, Services: Other clusters are not impacted. Next Steps: Our team is actively working to resolve the issue and will provide updates as progress is made or by 19:30 -UTC. Thank you for your patience as we work to restore full functionality.

4. monitoring Feb 24, 2025, 07:00 PM UTC

   Affected Services: Clients on the US4 cluster Service not impacted: Clients other clusters Description: Our team has fixed the issue affecting tenants' inaccessibility on the US4 cluster. The remaining tenants are recovering. We are monitoring the situation to ensure stability and confirm that the service remains fully functional. Impact: Service should operate normally; some tenant sites are still becoming accessible. Services: sites on other clusters are not affected Next Steps: We will provide a final update once the issue is resolved. Thank you for your patience, and we apologize for any inconvenience caused.

5. resolved Feb 24, 2025, 07:32 PM UTC

   Affected Services: Tenants on US4 Services not impacted: Tenants on all other clusters Description: The issue affecting tenant inaccessibility on the US4 cluster has been fully resolved. Regular service has been restored, and all systems are now operating as expected. Impact: Users should no longer experience any issues related to this incident. Next Steps: We are preparing a detailed Root Cause Analysis (RCA) report to provide further insights into the incident and preventive measures. Thank you for your patience, and we apologize for any inconvenience caused.

6. postmortem Mar 11, 2025, 02:00 PM UTC

   # Service Disruption - Clients on the US4 Cluster Unreachable ## Root Cause Analysis ### Duration of incident Discovered: Feb 28, 2025 Time - 16:32 - UTC Resolved: Feb 28, 2025 Time - 19:30- UTC ### Cause Overload of backend resources for services on the US4 cluster. ### Effect Tenants on the US4 cluster became inaccessible. ### Action taken _All times in UTC_ **02/28/2025** **16:32** - Auvik Engineering discovers several non-responsive backends on the US4 cluster, which causes some tenants to be unresponsive. Engineering begins investigating. **17:00** - Attempts are made to revive the non-responsive backends. **17:28** - Cluster is in distress, with more backends starting to fail. **17:45** - Engineering restarts the entire cluster. **18:10-19:30** - The cluster is observed as it restarts and monitored as it comes up to full functionality. The incident is declared resolved. ### Future consideration\(s\) * Auvik is currently improving backend monitoring and stability within the product and infrastructure. These improvements aim to help mitigate potential issues proactively in the future.

Read the full incident report →

Started Feb 19, 2025, 05:14 PM UTC· resolved Feb 19, 2025, 08:27 PM UTC

1. investigating Feb 19, 2025, 05:14 PM UTC

   Affected Services: Cloud Ping Check Cluster(s): US3 Description: We are currently experiencing degraded performance with the Cloud Ping Service check. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Impact: Users may experience excessive false alerts. Services: All other monitoring, alerting, maps and integrations are not impacted. Next Steps: We will provide updates as more information becomes available or within the next hour Thank you for your patience as we work to restore full functionality.

2. investigating Feb 19, 2025, 05:28 PM UTC

   Affected Services: Cloud Ping Check Cluster(s): US3 Description: We are currently experiencing degraded performance with the Cloud Ping Service check. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Impact: Users may experience excessive false alerts. Services: All other monitoring, alerting, maps and integrations are not impacted. Auvik recommends you disable your Cloud Ping Check and any customized Cloud Ping Check alerts until the problem is resolved, Next Steps: We will provide updates as more information becomes available or by 18:00 UTC Thank you for your patience as we work to restore full functionality.

3. identified Feb 19, 2025, 06:00 PM UTC

   Affected Services: Cloud Ping Check Cluster(s): All Clusters Description: We are currently experiencing degraded performance with the Cloud Ping Service check. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Impact: Users may experience excessive false alerts. Resources failing over from US3 may affect alerting in other clusters. Services: All other monitoring, alerting, maps and integrations are not impacted. Next Steps: Our team is actively working to resolve the issue and will provide updates as progress is made or by 19:00 UTC Thank you for your patience as we work to restore full functionality.

4. identified Feb 19, 2025, 06:44 PM UTC

   Affected Services: All alerts Cluster(s): All Clusters Auvik is posting an emergency maintenance window to disable alerts starting at 19:00 UTC. Alerts are scheduled to be re-enabled by 20:00 UTC Thank you for your patience as we work to restore full functionality.

5. identified Feb 19, 2025, 07:21 PM UTC

   Affected Services: All alerts Cluster(s): All Clusters The alerting maintenance window has been ended. alerts will now flow as intended Thank you for your patience as we work to restore full functionality.

6. monitoring Feb 19, 2025, 07:28 PM UTC

   Affected Services: Cloud Ping Check Cluster(s): All Clusters Description: We are currently experiencing degraded performance with the Cloud Ping Service check. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Impact: Service should operate normally; however, we continue monitoring for any irregularities. Services: All other monitoring, alerting, maps and integrations are not impacted. Next Steps: We will provide a final update once we confirm the issue is fully resolved. Thank you for your patience, and we apologize for any inconvenience caused.

7. resolved Feb 19, 2025, 08:27 PM UTC

   Affected Services: Cloud Ping Check Cluster(s): All Clusters Description: We are currently experiencing degraded performance with the Cloud Ping Service check. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Impact: Users should no longer experience any issues related to this incident. Next Steps: We are preparing a detailed Root Cause Analysis (RCA) report to provide further insights into the incident and preventive measures. Thank you for your patience, and we apologize for any inconvenience caused.

8. postmortem Mar 11, 2025, 01:52 PM UTC

   # Service Degraded - Cloud Ping Services Check Failing Intermittently on the US3 Cluster ## Root Cause Analysis ### Duration of incident Discovered: Feb 19, 2025 14:18 - UTC Resolved: Mar 01, 2025 15:00 - UTC ### Cause The Cloud Ping service became unstable due to a large number of clients running ping checks at a 5-second interval, leading to widespread ping check failures. ### Effect Clients received excessive Cloud ping check alerts corresponding to failed pings. ### Action taken _All times in UTC_ **02/13/2025-02/19/2025** Auvik started receiving complaints about an unusually high number of internet connection failures. A general investigation begins with customers reporting these issues. **02/19/2025** **14:18** - Auvik Engineering ascertains that the US3 cluster has several clients with a high number of internet connection checks set to the 5-second setting. An internal investigation then begins. **17:42** - Auvik disables Cloud Ping alerts in the US3 cluster for those affected. **17:53-18:44** - Auvik Engineering decides to restart the ping service to help clear the lag and re-stabilize it. A maintenance window is required to perform this action. **19:00** - A one-hour maintenance window is started. **19:21** - The work required under the maintenance window concludes early, and the services are back up and running. Cloud Ping alerts are restored for all clients. **02/24/2025** It’s noted that while the ping service is behaving normally for most clients, there continue to be intermittent problems. It is determined that a complete cluster restart is required. To minimize the impact on all customers, a decision is made to do maintenance on 03/01/2025 **03/01/2025** **12:00-15:00** - Auvik undergoes maintenance, during which US3 is safely restarted to restore the health of all services. ### Future consideration\(s\) * Auvik has worked with several clients who have set up a 5-second ping check to regulate the flow and prevent system overload. * Auvik will remove clients' ability to perform a 5-second cloud ping check and default the check frequency to one minute. The timing of this change will follow in future Auvik release notes.

Read the full incident report →

Started Feb 15, 2025, 11:02 PM UTC· resolved Feb 16, 2025, 03:21 AM UTC

1. identified Feb 15, 2025, 11:02 PM UTC

   Affected clusters: US3 Description: We are currently experiencing an outage on US3. Our team is actively working to restore sites on this cluster as quickly as possible. Impact: Sites hosted on US3 are currently unreachable. Next Steps: We will provide updates as more information becomes available or within the next hour. Thank you for your patience as we work to restore full functionality.

2. monitoring Feb 15, 2025, 11:30 PM UTC

   Affected clusters: US3 Description: Our team has restarted US3. Tenants are starting and we are continuing to monitor recovery of all tenants. Impact: Some tenants hosted on US3 may still be starting and unreachable at this time. Next Steps: We will provide a final update once we confirm the issue is fully resolved. Thank you for your patience, and we apologize for any inconvenience caused.

3. monitoring Feb 16, 2025, 12:25 AM UTC

   Affected clusters: US3 Description: The majority of tenants have been restored and we are performing final cluster checks. Impact: N/A Next Steps: We will provide updates as more information becomes available or within the next hour. Thank you for your patience as we work to restore full functionality.

4. monitoring Feb 16, 2025, 12:51 AM UTC

   Affected clusters: US3 Description: We have encountered an issue during monitoring of the cluster. The cluster is non-operational at this time. Our team is actively working to restore sites on this cluster as quickly as possible. Impact: Tenants hosted on US3 are not accessible at this time. Next Steps: We will provide updates as more information becomes available or within the next hour. Thank you for your patience as we work to restore full functionality.

5. resolved Feb 16, 2025, 03:21 AM UTC

   Affected clusters: US3 Description: The issue affecting US3 has been fully resolved. Normal service has been restored, and all systems are now operating as expected. Impact: Users should no longer experience any issues related to this incident. Next Steps: We are preparing a detailed Root Cause Analysis (RCA) report to provide further insights into the incident and preventive measures. Thank you for your patience, and we apologize for any inconvenience caused.

6. postmortem Mar 11, 2025, 01:42 PM UTC

   # Service Disruption - Clients on the US3 Cluster Unreachable ## Root Cause Analysis ### Duration of incident Discovered: Feb 14, 2025 Time - 22:42 - UTC Resolved: Feb 17, 2025 Time - 16:30 - UTC ### Cause Auvik made changes to its system to address issues with third-party integration for a client on US3 to process information that was not working as expected. ### Effect This change exposed a bug in the code that caused the backend systems to become overloaded. This caused data corruption in the hierarchical tables, which caused more instability in the system for clients on the US3 cluster. ### Action taken _All times in UTC_ **02/14/2025** **22:42** - First signs of increased backend pressure on the systems on the US3 cluster. **2/15/2025** **15:40** - Backend pressure on the US3 cluster increases. Engineering begins to monitor its systems for performance issues. **16:00-23:00** - Engineering attempts several interventions to reduce backend pressure. Success is intermittent. Ultimately, the root cause is identified as an abnormally growing dataset due to a bug. **23:00** - The tenant associated with the data table is disabled. However, several backends in the cluster are in severe distress, requiring a complete reboot of the cluster. A reboot is initiated. **02/16/2025** **00:00** - Most tenants are observed to be functional. **00:23** - Steps are initiated to re-enable the offending tenant. Unforeseen issues during this step create a cascading failure that results in another cluster reboot. **00:42** - The offending tenant is disabled again, and US3 is rebooted. **03:15** - Cluster is deemed stable. **16:00** - Engineering diagnoses a further root cause of the instability arising from the offending tenant. **17:00-21:30** - This tenant’s data is cleaned up manually. Finally, the tenant is restarted successfully. **02/17/25** **16:30** - Engineering notes that some hierarchical datasets have been corrupted, causing some tenants' alert notifications to be set to default values. Engineering initiates a cleanup of all such occurrences. **03/01/25** - A fix is applied to prevent further occurrences of such issues. All clusters are upgraded with this bug fix. ### Future consideration\(s\) * The bug that caused the instability has been addressed. * A review of how Auvik processes data in its hierarchical tables is under review. * Improved internal processes have been implemented to diagnose the cause of similar issues more quickly should they occur.

Read the full incident report →

Started Feb 03, 2025, 07:23 PM UTC· resolved Feb 03, 2025, 09:56 PM UTC

1. investigating Feb 03, 2025, 07:23 PM UTC

   Affected Services: Discovery Consolidation Cluster(s): US6 Description: We are currently experiencing degraded performance with the consolidation of devices. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Impact: Users may experience issues with new device discovery and consolidation. Services: Alerting is not impacted. Next Steps: We will provide updates as more information becomes available or by 19:30 UTC. Thank you for your patience as we work to restore full functionality.

2. investigating Feb 03, 2025, 07:42 PM UTC

   Affected Services: Discovery Consolidation Cluster(s): US6 Description: We are currently experiencing degraded performance with the consolidation of devices. Our team is still actively investigating the root cause and working to resolve the issue as quickly as possible. Impact: Users may experience issues with new device discovery and consolidation. Services: Alerting is not impacted. Next Steps: We will provide updates as more information becomes available or by 20:30 UTC. Thank you for your patience as we work to restore full functionality.

3. identified Feb 03, 2025, 08:31 PM UTC

   Affected Services: Discovery Consolidation Cluster(s): US6 Description: Our team has identified the root cause of the degraded performance affecting new device discovery consolidation. Cluster(s): US6. We are currently investigating a solution to restore normal service levels. Impact: While we work on the resolution, users will continue to experience device discovery and consolidation. Services: Alerting is not impacted. Next Steps: Our team is actively working to resolve the issue and will provide updates as progress is made or within the next hour Thank you for your patience as we work to restore full functionality.

4. monitoring Feb 03, 2025, 08:42 PM UTC

   Affected Services: Discovery Consolidation Cluster(s): US6 Description: Our team has implemented a fix for the issue affecting the consolidation of devices, and the performance consolidation of devices is returning to normal. We are monitoring the situation to ensure stability and confirm that the service remains fully functional. Impact: Service is returning to normal; however, we continue monitoring for irregularities. Services Alerting was not impacted. Next Steps: We will provide a final update once we confirm the issue is fully resolved. Thank you for your patience, and we apologize for any inconvenience caused.

5. resolved Feb 03, 2025, 09:56 PM UTC

   Affected Services: Discovery Consolidation Cluster(s): US6 Description: The issue affecting Discovery Consolidation has been fully resolved. Normal service has been restored, and all systems are now operating as expected. Impact: Users should no longer experience any issues related to this incident. Next Steps: We are preparing a detailed Root Cause Analysis (RCA) report to provide further insights into the incident and preventive measures. Thank you for your patience, and we apologize for any inconvenience caused.

6. postmortem Feb 10, 2025, 03:27 PM UTC

   # **Service Degraded - Newly discovered devices and consolidation are not working for clients on the US6 cluster.** ## **Root Cause Analysis** ### Duration of incident Discovered: Feb 02, 2023 17:00 - UTC Resolved: Feb 02, 2023 21:30 - UTC ### Cause A reorganization of engineering caused a permission change for tenant migrations. ### Effect This change caused permission issues with a tenant migration to another cluster, which, in turn, also caused problems with consolidation on the same cluster. ### Action taken _All times in UTC_ **02/03/2025** **16:58 –** A tenant is migrated off of the US6 cluster. **18:00 –** Engineering is aware of consolidation issues for clients on the US6 cluster and begins investigating. **20:28 –** The initial cause for the interruption is determined. Engineering disables the migration service. **20:34 –** The tenant migration that caused the issues is identified. **20:47 –** The root cause of the interruption of services is identified. **21:15 –** The underlying issues that caused the service interruption are fixed. **21:45 –** Tenant migration is re-enabled tenant migrations in the consolidation service. **22:01 –** The problematic tenant is successfully migrated. **22:39 –** All services are confirmed to be running as intended. ### Future consideration\(s\) * Auvik is reviewing permission changes that have occurred and validating tests of the blast radius of the changes. * Any changes will have full comments and documentation created to follow the changes better. * Auvik will set up a test migration regularly to validate tenant migration functionality.

Read the full incident report →

Started Feb 03, 2025, 10:23 AM UTC· resolved Feb 03, 2025, 11:30 AM UTC

1. investigating Feb 03, 2025, 10:23 AM UTC

   Affected Services: All clients are currently not accessible Service not impacted: NA Description: Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Impact: Users are experiencing no access to their tenants Next Steps: We will provide updates as more information becomes available or within the next at 11:00 UTC. Thank you for your patience as we work to restore full functionality.

2. identified Feb 03, 2025, 10:32 AM UTC

   Affected Services: All clients are currently not accessible Description: Our team has identified the root cause of the site down. We are currently investigating a solution to restore normal service levels. Impact: While we work on the resolution, users may experience slower load times and intermittent connectivity issues, Next Steps: Our team is actively working to resolve the issue and will provide updates as progress is made or by 11:00 UTC Thank you for your patience as we work to restore full functionality.

3. monitoring Feb 03, 2025, 11:01 AM UTC

   Affected Services: Clients on US4 Cluster Description: Our team has implemented a fix for the issue, and tenants are in the process of becoming fully accessible. We are monitoring the situation to ensure stability and confirm that the service remains fully functional. Impact: Services should be operating normally; with a few client sites still in the process of starting up. We continue to monitor for any irregularities. Next Steps: We will provide a final update once we confirm the issue is fully resolved. Thank you for your patience, and we apologize for any inconvenience caused.

4. resolved Feb 03, 2025, 11:30 AM UTC

   Affected Services: clients in US4 are now accessible. Description: The issue affecting US4 tenants has been resolved. Regular service has been restored, and all systems are operating as expected. Impact: Users should no longer experience any issues related to this incident. Next Steps: We are preparing a detailed Root Cause Analysis (RCA) report to provide further insights into the incident and preventive measures. Thank you for your patience, and we apologize for any inconvenience caused.

5. postmortem Feb 18, 2025, 04:34 PM UTC

   # Service Disruption - Clients on US4 are not accessible ‌ ## Root Cause Analysis ### Duration of incident Discovered: Feb 03, 2023 Time - 09:27 - UTC Resolved: Feb 03, 2023 Time - 11:55 - UTC ### Cause Overload of backend resources for services on the US4 cluster. ### Effect Tenants on the US4 cluster became inaccessible. ### Action taken _All times in UTC_ **02/03/2025** **09:27** - Engineering receives alerts that tenants on the US4 cluster are not accessible. **09:33** - Engineering reacts to the outage and begins its investigation. **09:53** - Engineering restarts US4 cluster backends to address its non-responsiveness. **9:53- 11:55** - The cluster is observed as it restarted and monitored as it comes up to full functionality. The incident is declared resolved. ### Future consideration\(s\) * Auvik is currently improving backend monitoring and stability within the product and infrastructure. These improvements are aimed to assist in proactively mitigating potential issues in the future.

Read the full incident report →

Started Dec 13, 2024, 05:18 PM UTC· resolved Dec 13, 2024, 06:23 PM UTC

1. investigating Dec 13, 2024, 05:18 PM UTC

   Affected Services: US4 Cluster Description: We are currently experiencing an outage on tenants hosted on our US4 cluster. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Impact: Users will not be able to reach their tenants hosted in US4. Next Steps: We will provide updates as more information becomes available or within the next hour. Thank you for your patience as we work to restore full functionality.

2. investigating Dec 13, 2024, 05:51 PM UTC

   We are continuing to investigate this issue.

3. monitoring Dec 13, 2024, 05:59 PM UTC

   Affected Services: US4 cluster Description: Our team has implemented a fix for the issue affecting the US4 cluster. Tenants are being restored and we are continuing to monitor the recovery progress. Impact: Any unreachable tenant is queued to be started and will be reachable within approximately 1 hour. Next Steps: We will provide a final update once we confirm the issue is fully resolved. Thank you for your patience, and we apologize for any inconvenience caused.

4. resolved Dec 13, 2024, 06:23 PM UTC

   Affected Services: US4 Cluster Description: The issue affecting US4 has been addressed and the system has recovered. Impact: Users should now be able to access their tenants on US4. Next Steps: We are preparing a detailed Root Cause Analysis (RCA) report to provide further insights into the incident and preventive measures. Thank you for your patience, and we apologize for any inconvenience caused.

5. postmortem Jan 10, 2025, 03:29 PM UTC

   # Service Disruption - Cluster US4 is unreachable for customers ## Root Cause Analysis ### Duration of incident Discovered: Dec 13, 2024 17:03 - UTC Resolved: Dec 13, 2024 18:23 - UTC ### Cause Routine maintenance tasks in preparation for the upcoming weekend's maintenance cause an unexpected load to the system. Effect The backend systems overwhelmed the systems on the US4 cluster, which caused a communication interruption with the tenants. ### Action taken _All times in UTC_ **12/13/2024** **16:57 -** Steps to prepare the system for the next day’s maintenance performed. **17:03 -** Tenants on the US4 cluster become unreachable. **17:09 -** The Auvik engineering team assembles stakeholders to investigate the service interruption. **17:25 -** The backend systems on the US4 cluster begin to recover independently. **17:39 -** Tenants begin to become reachable internally. **17:40 -** Tenants become visible in the UI. **17:57** - Engineering addressed tenants that are not coming back up gracefully. **18:23 -** Tenants on US4 have recovered. ### Future consideration\(s\) * Auvik has altered its preparation for scheduled maintenance, eliminating processes that could affect system performance in the future.

Read the full incident report →

Started Dec 06, 2024, 04:16 PM UTC· resolved Dec 06, 2024, 04:48 PM UTC

1. investigating Dec 06, 2024, 04:16 PM UTC

   Affected Services: Alert Integrations Cluster(s): All Description: We are currently experiencing degraded performance with alert integrations. Our team is actively investigating the root cause and working to resolve the issue as quickly as possible. Impact: Users may experience alerts that are not posted to their integrated systems. Services: Monitoring and UI are not impacted. Next Steps: We will provide updates as more information becomes available or within the next hour. Thank you for your patience as we work to restore full functionality.

2. identified Dec 06, 2024, 04:34 PM UTC

   Affected Services: Alert Integrations Cluster(s): All Description: Our team has identified the root cause of the degraded performance affecting alert integrations and is currently investigating a solution to restore normal service levels. Impact: While we work on the resolution, users may continue to experience alerts that are not posted to their integrated systems. Services: Monitoring and UI are not impacted. Next Steps: Our team is actively working to resolve the issue and will provide updates as progress is made or within the next hour. Thank you for your patience as we work to restore full functionality.

3. monitoring Dec 06, 2024, 04:44 PM UTC

   Affected Services: Alert Integrations Cluster(s): All Description: Our team has implemented a fix for the alert integration issue, and performance has returned to normal. We are currently monitoring the situation to ensure stability and confirm that the service remains fully functional. Impact: Service should operate normally; however, we continue monitoring for any irregularities. Services Monitoring and UI are not impacted. Next Steps: We will provide a final update once we confirm the issue is fully resolved. Thank you for your patience, and we apologize for any inconvenience caused.

4. resolved Dec 06, 2024, 04:48 PM UTC

   Affected Services: Alert Integrations Cluster(s): All Description: The issue affecting Alert Integrations has been fully resolved. Normal service has been restored, and all systems are now operating as expected. Impact: Users should no longer experience any issues related to this incident. Next Steps: We are preparing a detailed Root Cause Analysis (RCA) report to provide further insights into the incident and preventive measures. Thank you for your patience, and we apologize for any inconvenience caused.

Read the full incident report →

Started Oct 26, 2024, 02:51 PM UTC· resolved Oct 26, 2024, 05:57 PM UTC

1. investigating Oct 26, 2024, 02:51 PM UTC

   We’re experiencing disruption with network topology and select UI data not populating in the Auvik UI. Some data may be unavailable. We will continue to provide updates as they become available.

2. identified Oct 26, 2024, 03:32 PM UTC

   We’ve identified the source of the service disruption with maps and other UI elements. We will be restarting the system to recover from this incident.

3. monitoring Oct 26, 2024, 04:50 PM UTC

   The system has been restarted and services are coming back online. Tenants are beginning to start and become available at this time. We are monitoring the system and will provide an update in 30 minutes.

4. monitoring Oct 26, 2024, 05:23 PM UTC

   We are continuing to monitor the system as tenants are becoming available. We estimate that all tenants will be fully operational within the hour and will provide an update accordingly.

5. resolved Oct 26, 2024, 05:57 PM UTC

   The source of the disruption has been resolved. Services have been fully restored. A Root Cause Analysis (RCA) will follow after a full review has been completed.

6. postmortem Nov 01, 2024, 04:00 PM UTC

   # Service Disruption - Maps and data not being populated in the UI ## Root Cause Analysis ### Duration of incident Discovered: Oct 26, 2024 13:13 - UTC Resolved: Oct 26, 2024 17:59 - UTC ### Cause Requests to the permission cache prevented data from syncing properly across clusters. ### Effect Key product features, like map functionality, became unavailable to users. ### Action taken _All times in UTC_ **10/26/2024** **11:00** Planned upgrade of backend data started. **13:13** Issues were noted with permissions that affected UI components. **13:30** Confirmed issues with Maps during the post-upgrade check. **14:24** Attempt to restart services to enable UI updates. **15:30** Decision was made to restart services from the beginning to flush out any lingering issues. **15:47** Backend services restarted with an additional emphasis on enabling a clean restart. **17:59** Service fully restored. ### Future consideration\(s\) * Parallelize investigations and timebox responses for efficiency to avoid prolonged troubleshooting when a complete restart could resolve issues. * Improve upgrade and detection protocols to catch errors earlier.

Read the full incident report →