Northpass had 18 outages in the last 2 years totaling 1337h 2m of downtime — averaging 0.7 incidents per month.
There were 18 Northpass outages since June 25, 2024 totaling 1337h 2m of downtime. Each is summarised below — incident details, duration, and resolution information.
We are investigating unavailability of the AWS app
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are tracking an active Microsoft Azure incident in the East US region that is causing intermittent connection and DNS errors across our platform. Some users may experience delayed actions, failed webhook deliveries, or transient errors while we ride out the upstream issue.
We are continuing to monitor the upstream service provider for status updates.
This incident has been resolved.
Time: 06:47–06:55 EST (8 minutes) Impact: AWS only application unavailable/slow due to query timeouts. Cause: Database changes increased load, causing timeouts on application queries. Resolution: Service restored after mitigating the database changes and stabilising load.
We're experiencing an elevated level of API errors on Azure platform and are currently looking into the issue.
We are currently experiencing a full Azure platform outage. Initial investigation indicates the issue is related to Azure Front Door (our global load balancer) and originates on the Azure side. We are monitoring the situation closely and are in contact with Microsoft Azure support.
We have confirmed that the outage is caused by an ongoing issue with Azure Front Door (our global load balancer) on the Microsoft Azure side. This issue is impacting traffic routing to our backend services, resulting in a full platform outage. Our team remains in contact with Microsoft Azure support and is monitoring their mitigation progress.
The outage remains in progress. Microsoft Azure has confirmed an active incident affecting Azure Front Door (our global load balancer), which continues to disrupt traffic routing to our backend services. Our platform remains fully unavailable while we wait for a fix from the cloud provider. We are in continuous contact with Microsoft Azure support and will restore services as soon as the issue is resolved on their side.
The outage remains in progress. Microsoft Azure has confirmed an active incident affecting Azure Front Door (our global load balancer), which continues to disrupt traffic routing to our backend services. Our platform remains fully unavailable. However, a fix is currently being implemented by Microsoft Azure and we are awaiting the result. We are in continuous contact with Microsoft Azure support and will restore services as soon as the issue is resolved on their side.
As a result of the fix implemented by Microsoft Azure, the outage is now recovered. We observed recovery on our end at 20:20 UTC as our platform is now available and operational. However, we continue monitoring recovery status and will provide updates within the hour.
We are continuing to monitor for any further issues.
This incident has been resolved.
Start time: 07:10 UTC Impact: Partial degradation of integrations and data synchronization We are currently experiencing degraded performance caused by an upstream service provider. As a result, users may notice delays in: • Event synchronization between HubSpot and our platform • Delivery of Analytics Data in dashboards and reports • Data sync between Customer Education and Gainsight Our team has identified that the issue originates from one of our upstream partners, and we are actively monitoring their status while working on mitigation steps to minimize impact.
We continue to work with the third-party vendor in an effort to resolve the ongoing issue. Users may continue to notice delays in: • Event synchronization between HubSpot and our platform • Delivery of Analytics Data in dashboards and reports • Data sync between Customer Education and Gainsight The vendor has acknowledged and is actively working to resolve this.
We continue to work with the third-party vendor in an effort to resolve the ongoing issue. Users may continue to notice delays in: • Event synchronization between HubSpot and our platform • Delivery of Analytics Data in dashboards and reports • Data sync between Customer Education and Gainsight The vendor has acknowledged and is actively working to resolve this.
Start time: 07:10 UTC Impact: Partial degradation of integrations and data synchronization We are continuing to monitor the situation closely as our cloud provider works through mitigation efforts. Users may still notice delays in the following areas at this time: • Event synchronization between HubSpot and our platform • Delivery of Analytics Data in dashboards and reports • Data sync between Customer Education and Gainsight
Our cloud service provider has restored functionality. Customers may continue to experience delays in the below areas as we work through the backlog. • Event synchronization between HubSpot and our platform • Delivery of Analytics Data in dashboards and reports • Data sync between Customer Education and Gainsight
We are continuing to monitor as we work through the backlog.
This incident has been resolved.
We are currently investigating this issue.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
Between 14:22 and 14:46 EDT today, our entire application was unavailable due to an unexpected memory exhaustion in our backend layer. Root Cause: A high-volume enrolment process caused backend cache errors leading to an "Out of Memory" condition. Preventative Measures Script Throttling: We'll introduce rate-limiting for large-scale data loads. Capacity Expansion: Backend cache memory limits were increased.
We are currently investigating this issue
We are continuing to investigate this issue.
The issue has been identified by the Engineering team and fix is being implemented...
We are continuing to work on the fix for this issue, thank you for your patience and understanding.
A fix has been implemented and we are monitoring the results, thank you for your patience.
This issue has been resolved. Thank you for you patience and understanding
A recent deployment of new functionality introduced a bug in learner enrollments that affected some of our customers. As for future avoidance we enhanced tests covering sign-in and enrollments that will prevent from introducing similar errors moving forward.
We are currently investigating issue with performance
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
We are continuing to monitor for any further issues.
This incident has been resolved.
We are currently investigating issue with Course Viewer that is preventing learners from using parts of the course viewer
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
Time Detected: 09:30 Status: Resolved Time Resolved: 10:45 Summary: The login issues affecting customers using Single Sign-On (SSO) have been resolved. The issue was traced to an abnormal behaviour in a firewall rule following a recent component update. Resolution Steps: The engineering team identified the cause and reverted the changes to the component update, we will continue monitoring to ensure stable access for all customers. Thank you for your patience and understanding.
Incident Duration: 04:30 - 04:36 (6 minutes) Incident Description: On 05.10.2024, from 04:30 to 04:36, our application experienced an outage due to a scheduled database maintenance operation by our cloud provider. During this maintenance window, database connections were temporarily unavailable, which caused the application to be inaccessible for a brief period. Impact: - All users were unable to access the application during the downtime. - No data loss occurred, and full service was restored once the maintenance was completed. We apologize for any inconvenience caused and appreciate your understanding. If you have any further questions, please don't hesitate to reach out to our support team.
Incident Start: 14:33 EDT Incident End: 16:08 EDT Impact: During the incident period, customers experienced increased latency. This impacted the UI responsiveness, causing inconvenience to our users. Cause: The root cause of the incident was identified as abnormal behavior with rate limiting. This anomaly disrupted the normal operation of our network infrastructure, resulting in the issues. Resolution: Our engineering team quickly identified the problem and applied a fix. Once the fix was completed problem was resolved. Next Steps: We are conducting a thorough review of our update procedures and tests to prevent similar incidents in the future.
We are currently investigating issue with increased error rate and timeouts.
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.
We are currently facing increase latency due to Azure global outage.
Azure team status: We have implemented networking configuration changes and have performed failovers to alternate networking paths to provide relief. Monitoring telemetry shows improvement in service availability, and we are continuing to monitor to ensure full recovery.
This incident has been resolved.
We are investigating issue with failures of loading the navigation only in admin panel. Learners are not affected
The issue has been identified and a fix is being implemented.
A fix has been implemented and we are monitoring the results.
This incident has been resolved.