Resolved after 13 hours 3 minutes, affecting View Employees and Run Reports and 1 more, on April 24, 2026.
On Friday, April 24, 2026, the Netchex platform experienced extended, intermittent service disruption throughout the day, including failed sign-ins, slow or failed page loads, and difficulty reaching key parts of the system. We sincerely apologize for the impact this had on you and your employees. Your trust matters to us. Below is what happened, how we worked through it, and the steps we’re taking to reduce the chance of a repeat. What Happened Customers experienced intermittent disruption from approximately 8:12 AM CT until full resolution at 9:15 PM CT, in two main waves. The first wave began at approximately 8:12 AM CT, with performance issues and intermittent failures on employee and timecard pages, later extending to platform navigation menus. Mitigations rolled out during the morning, and conditions partially improved through the early afternoon as our cloud provider deployed their fix. A second, broader wave began at approximately 4:13 PM CT as the upstream condition re-emerged, affecting sign-in, navigation, and overall site responsiveness. Conditions began returning to normal at approximately 7:29 PM CT, and we posted the all-clear at 9:15 PM CT after close monitoring. No customer data was lost, altered, or exposed, and there was no compromise to authentication or account security. Failed requests produced error pages and sign-in failures, not partial or incorrect data. What Caused It The disruption was caused by a regional incident in Microsoft Azure’s East US region, where portions of the Netchex platform are hosted. A prior update to core Azure regional infrastructure contained a latent defect that surfaced under load Friday morning. As Microsoft executed a staged rollback, the same condition appeared in turn across all three availability zones in the region. Several Netchex services are deployed redundantly across these zones so that any single-zone failure does not affect customers. Because all three zones were impacted at different points, that protection was not sufficient on its own. The morning and late-afternoon waves were two symptoms of the same regional event — first appearing in one zone, then re-appearing in another during Microsoft’s recovery. How We Fixed It Our Engineering and SRE teams launched a coordinated response as soon as the morning impact was identified, restarting affected services and deploying targeted mitigations to clear stuck connections and restore capacity while the upstream condition persisted. After Microsoft acknowledged the regional incident around midday, we began failover procedures for impacted services and prepared a larger contingency to move affected services onto infrastructure outside the impacted region. When the second wave began in the late afternoon, the team continued operational mitigations and stayed ready to execute the contingency. Microsoft completed their rollback before that was needed; conditions began returning to normal at approximately 7:29 PM CT, and we validated end-to-end functionality before posting the all-clear at 9:15 PM CT. Our Commitment to Improvement We fell short of the reliability standard you expect from us. While the originating fault was in our cloud provider’s infrastructure, we own the customer experience on our platform, and there is meaningful work we can do to limit how much an upstream regional event reaches you. The following actions are underway or being accelerated as a direct result of this incident. Granular per-service regional failover. Our business continuity plan supports a full-site cross-region failover, and that capability remains in place — but this incident showed it is the wrong tool for a partial regional impairment. We are accelerating in-flight work to enable per-service failover at a much finer granularity, redirecting only the specific services affected to a healthy region without moving the entire platform. This will materially reduce both the scope and duration of customer impact during regional events. Faster detection and triage. We are reviewing how the morning impact was detected and how quickly customer communication began, and tightening internal alerting on cloud dependency health so early signals reach our on-call engineers — and our status page — faster than they did on Friday. Improved handling of unhealthy service instances. During the morning impact, a small number of instances entered a degraded state and kept receiving a share of customer traffic before being recycled. We are tightening the health-check and pool-eviction settings so persistently unhealthy instances are removed from rotation automatically rather than continuing to serve requests. We are committed to providing a platform you can depend on. These actions are how we learn from this incident and earn back your full confidence. Thank you for your patience and continued partnership.
Affected components
Update timeline
- investigating Apr 24, 2026, 01:12 PM UTC
Our top engineering and SRE teams are actively investigating the issue. We’re treating this with the highest urgency and working to identify the root cause. Thank you for your patience as we work quickly toward resolution. Known Impact / Preliminary Findings: Performance issues one employee and timecard pages
- investigating Apr 24, 2026, 01:37 PM UTC
A possible fix is being deployed across affected systems. If successful, services should begin returning to normal shortly. All hands are on deck and we’re closely monitoring the rollout to confirm full recovery. Known Impact / Preliminary Findings: We have identified the cause of the performance issue, a mitigation is being rolled out and services should begin restoring shortly
- investigating Apr 24, 2026, 04:14 PM UTC
Unfortunately, the initial fix did not fully resolve the issue. Our engineering and SRE teams are analyzing new data and actively pursuing alternative solutions. This remains our highest priority, and we are committed to resolving it as quickly and effectively as possible. Known Impact / Preliminary Findings: Ongoing issues in navigation menus
- investigating Apr 24, 2026, 05:03 PM UTC
Unfortunately, the initial fix did not fully resolve the issue. Our engineering and SRE teams are analyzing new data and actively pursuing alternative solutions. This remains our highest priority, and we are committed to resolving it as quickly and effectively as possible. Known Impact / Preliminary Findings: Microsoft Azure has acknowledged a service outage impacting our region. We are beginning failover procedures for impacted services: https://azure.status.microsoft/en-us/status
- investigating Apr 24, 2026, 06:42 PM UTC
A possible fix is being deployed across affected systems. If successful, services should begin returning to normal shortly. All hands are on deck and we’re closely monitoring the rollout to confirm full recovery. Known Impact / Preliminary Findings: Azure is reporting a fix being rolled out to our region. We are seeing services recover as well. Our team is continuing to actively monitor: https://azure.status.microsoft/en-us/status
- investigating Apr 24, 2026, 08:37 PM UTC
The issue has been resolved and all systems are currently stable. Our teams will continue to monitor performance closely to ensure full recovery holds. We appreciate your patience and trust—resolving this was a top priority across our organization. More technical details on the issue and resolution will be shared shortly.
- investigating Apr 24, 2026, 09:13 PM UTC
Our top engineering and SRE teams are actively investigating the issue. We’re treating this with the highest urgency and working to identify the root cause. Thank you for your patience as we work quickly toward resolution. Known Impact / Preliminary Findings: Seeing additional reports of peformance issues. Azure is still reporting impacts in our region: https://azure.status.microsoft/en-us/status
- investigating Apr 25, 2026, 12:29 AM UTC
A possible fix is being deployed across affected systems. If successful, services should begin returning to normal shortly. All hands are on deck and we’re closely monitoring the rollout to confirm full recovery. Known Impact / Preliminary Findings: Azure is reporting restoration of services again. We are seeing conditions in Netchex return to normal. Our team is continuing to monitor at this time.
- resolved Apr 25, 2026, 02:15 AM UTC
The issue has been resolved and all systems are currently stable. Our teams will continue to monitor performance closely to ensure full recovery holds. We appreciate your patience and trust—resolving this was a top priority across our organization. More technical details on the issue and resolution will be shared shortly.
- postmortem May 04, 2026, 11:13 PM UTC
On Friday, April 24, 2026, the Netchex platform experienced extended, intermittent service disruption throughout the day, including failed sign-ins, slow or failed page loads, and difficulty reaching key parts of the system. We sincerely apologize for the impact this had on you and your employees. Your trust matters to us. Below is what happened, how we worked through it, and the steps we’re taking to reduce the chance of a repeat. What Happened Customers experienced intermittent disruption from approximately 8:12 AM CT until full resolution at 9:15 PM CT, in two main waves. The first wave began at approximately 8:12 AM CT, with performance issues and intermittent failures on employee and timecard pages, later extending to platform navigation menus. Mitigations rolled out during the morning, and conditions partially improved through the early afternoon as our cloud provider deployed their fix. A second, broader wave began at approximately 4:13 PM CT as the upstream condition re-emerged, affecting sign-in, navigation, and overall site responsiveness. Conditions began returning to normal at approximately 7:29 PM CT, and we posted the all-clear at 9:15 PM CT after close monitoring. No customer data was lost, altered, or exposed, and there was no compromise to authentication or account security. Failed requests produced error pages and sign-in failures, not partial or incorrect data. What Caused It The disruption was caused by a regional incident in Microsoft Azure’s East US region, where portions of the Netchex platform are hosted. A prior update to core Azure regional infrastructure contained a latent defect that surfaced under load Friday morning. As Microsoft executed a staged rollback, the same condition appeared in turn across all three availability zones in the region. Several Netchex services are deployed redundantly across these zones so that any single-zone failure does not affect customers. Because all three zones were impacted at different points, that protection was not sufficient on its own. The morning and late-afternoon waves were two symptoms of the same regional event — first appearing in one zone, then re-appearing in another during Microsoft’s recovery. How We Fixed It Our Engineering and SRE teams launched a coordinated response as soon as the morning impact was identified, restarting affected services and deploying targeted mitigations to clear stuck connections and restore capacity while the upstream condition persisted. After Microsoft acknowledged the regional incident around midday, we began failover procedures for impacted services and prepared a larger contingency to move affected services onto infrastructure outside the impacted region. When the second wave began in the late afternoon, the team continued operational mitigations and stayed ready to execute the contingency. Microsoft completed their rollback before that was needed; conditions began returning to normal at approximately 7:29 PM CT, and we validated end-to-end functionality before posting the all-clear at 9:15 PM CT. Our Commitment to Improvement We fell short of the reliability standard you expect from us. While the originating fault was in our cloud provider’s infrastructure, we own the customer experience on our platform, and there is meaningful work we can do to limit how much an upstream regional event reaches you. The following actions are underway or being accelerated as a direct result of this incident. Granular per-service regional failover. Our business continuity plan supports a full-site cross-region failover, and that capability remains in place — but this incident showed it is the wrong tool for a partial regional impairment. We are accelerating in-flight work to enable per-service failover at a much finer granularity, redirecting only the specific services affected to a healthy region without moving the entire platform. This will materially reduce both the scope and duration of customer impact during regional events. Faster detection and triage. We are reviewing how the morning impact was detected and how quickly customer communication began, and tightening internal alerting on cloud dependency health so early signals reach our on-call engineers — and our status page — faster than they did on Friday. Improved handling of unhealthy service instances. During the morning impact, a small number of instances entered a degraded state and kept receiving a share of customer traffic before being recycled. We are tightening the health-check and pool-eviction settings so persistently unhealthy instances are removed from rotation automatically rather than continuing to serve requests. We are committed to providing a platform you can depend on. These actions are how we learn from this incident and earn back your full confidence. Thank you for your patience and continued partnership.
Your product runs on more than Netchex
Add the vendors you depend on and get one message when any of them reports trouble. Free for 3 vendors, no card.
Amazon Web Services