Rise Up Outage History

Started Oct 21, 2026, 02:23 PM UTC· resolved Oct 21, 2026, 03:53 PM UTC

1. identified Oct 21, 2026, 02:23 PM UTC

   On October 21, the PHP project disclosed critical security vulnerabilities affecting PHP, a technology used across the industry and within the Rise Up platform. Our infrastructure team was notified in advance and has been actively working on this issue. Preventive protective measures have already been deployed across our environments, and security updates are being rolled out progressively as they become available. We do not expect any service disruption as a result of these updates. We will post an update once the rollout is complete.

2. resolved Oct 21, 2026, 03:53 PM UTC

   Following the critical PHP security vulnerabilities disclosed on October 21, our infrastructure team has completed the deployment of security updates across the systems serving the Rise Up platform. Protective measures were applied ahead of the official disclosure, and the security fixes were rolled out as soon as they became available, with no service disruption. We continue to monitor our environments closely. No further action is required from our customers.

Read the full incident report →

Started Oct 07, 2026, 12:18 PM UTC· resolved Oct 07, 2026, 01:18 PM UTC

1. resolved Oct 07, 2026, 02:12 PM UTC

   On October 7, between 14:18 and 14:35 CEST, and again between 15:03 and 15:18 CEST, some users experienced errors or pages failing to load on the web platform, and part of the API requests returned errors. The cause was an overload of our internal messaging system, triggered by an unusually large volume of background tasks. The platform has been operating normally since 15:18 CEST. Some background processing, such as automatic enrollments and their notifications, is delayed and is expected to catch up by this evening. If you received API errors during these windows, please check the result of the affected requests before retrying them. We apologize for the inconvenience and are putting safeguards in place to prevent this from happening again.

Read the full incident report →

Started Sep 25, 2026, 04:00 PM UTC· resolved Sep 28, 2026, 07:55 AM UTC

1. resolved Sep 28, 2026, 08:27 AM UTC

   Between Friday, September 25 at 18:00 and Monday, September 28 at 09:55 (CEST), some users on certain custom subdomains intermittently saw broken styling or features that didn't load, because some CSS/JS assets weren't delivered correctly. A deployment issue caused this, and it was fixed this morning. All affected subdomains have been working normally since then. We're making changes to our deployment process so this doesn't happen again.

Read the full incident report →

Started Aug 19, 2026, 04:15 PM UTC· resolved Aug 19, 2026, 06:18 PM UTC

1. investigating Aug 19, 2026, 04:15 PM UTC

   The Rise Up platform is currently unavailable. Our engineering team has been alerted and is actively investigating the root cause. We will provide an update as soon as possible.

2. identified Aug 19, 2026, 04:53 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Aug 19, 2026, 04:56 PM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Aug 19, 2026, 06:18 PM UTC

   This incident has been resolved. An automated system update applied to our web proxy layer at 18:04 resulted in a configuration bug that blocked internal traffic to our authentication service until 18:51.

Read the full incident report →

Started Jul 15, 2026, 08:00 AM UTC· resolved Jul 16, 2026, 06:37 AM UTC

1. resolved Oct 14, 2026, 08:36 AM UTC

   On July 14 and on the morning of July 15, some users of platforms hosted on Azure experienced errors or pages failing to load on the web platform and mobile app, and part of the API requests returned errors. The impact occurred during the following windows (CEST): July 14 - 11:18 - 12:00, 12:23 - 13:10, 13:43 - 14:20 and 14:43 - 15:15: partial unavailability - 15:15 - 17:35: platform unavailable - 17:35 - 23:27: degraded performance with intermittent errors - 23:27 - 02:35 (July 15): platform unavailable July 15 - 07:55 - 08:20: platform unavailable The cause was an issue with our internal messaging system on this environment, which reached one of its resource limits and stopped accepting new messages. Requests depending on it were held, which progressively saturated the application servers. The platform has been operating normally since July 15 at 08:37 CEST, and delayed background processing caught up shortly after. We apologize for the inconvenience and have put safeguards in place to prevent this from happening again.

Read the full incident report →

Started Jul 05, 2026, 10:00 PM UTC· resolved Jul 06, 2026, 10:03 AM UTC

1. resolved Jul 06, 2026, 12:24 PM UTC

   Between 12:03 and 14:23 (CEST) on July 6, some users were unable to sign in via SSO on a subset of platforms. Sign-in attempts during this window would not complete and could return users to the login screen. Users with an already active session were not affected, and no data was impacted. The issue was introduced by a release deployed at 12:03 that affected how authentication tokens are handled at the end of the SSO sign-in flow. The cause was identified and a fix was deployed, fully restoring SSO sign-ins. No action is required - affected users can now sign in normally.

Read the full incident report →

Started Jun 15, 2026, 07:32 AM UTC· resolved Jun 15, 2026, 08:06 AM UTC

1. resolved Jun 19, 2026, 03:06 PM UTC

   Between approximately 09:32 and 10:06 CEST, some users may have experienced loading errors or a blank page when accessing the platform. The issue stemmed from a configuration change affecting the delivery of certain static resources. Users with an active session were not impacted. The change was identified and rolled back, and service was fully restored at 10:06 CEST. We apologize for any inconvenience.

Read the full incident report →

Started May 18, 2026, 09:57 AM UTC· resolved May 18, 2026, 10:51 AM UTC

1. investigating May 18, 2026, 09:57 AM UTC

   We are seeing partial outage on the web platform. We are currently working on it to identify the root cause.

2. identified May 18, 2026, 10:08 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring May 18, 2026, 10:12 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved May 18, 2026, 10:51 AM UTC

   Earlier today we experienced elevated load that impacted platform performance. The issue has been resolved and additional safeguards have been put in place.

Read the full incident report →

Started Mar 26, 2026, 07:12 AM UTC· resolved Mar 26, 2026, 08:05 AM UTC

1. investigating Mar 26, 2026, 07:12 AM UTC

   We encounter an issue on Azure platform. We are currently working on it to identify the root cause.

2. investigating Mar 26, 2026, 07:40 AM UTC

   We are continuing to investigate this issue. Only a small part of requests are affected.

3. monitoring Mar 26, 2026, 07:49 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Mar 26, 2026, 08:05 AM UTC

   This incident has been resolved.

Read the full incident report →

Started Mar 09, 2026, 09:37 AM UTC· resolved Mar 09, 2026, 10:41 AM UTC

1. investigating Mar 09, 2026, 09:37 AM UTC

   We encounter an issue on web platform where some requests can timeout. We are currently working on it to identify the root cause.

2. identified Mar 09, 2026, 09:40 AM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Mar 09, 2026, 09:46 AM UTC

   A fix has been implemented and we are monitoring the results.

4. resolved Mar 09, 2026, 10:41 AM UTC

   Between 10:10 and 10:26 CET our application experienced degraded performance and intermittent unavailability due to an unexpectedly long-running database operation. This led to an accumulation of backend connections and affected some users ability to access the application. We identified and mitigated the issue by adjusting configuration parameters and restarting affected services. The platform is now fully operational, and we continue to monitor to ensure stability. We're implementing additional safeguards to prevent similar behavior in the future.

Read the full incident report →

Started Feb 09, 2026, 08:32 AM UTC· resolved Feb 09, 2026, 08:32 AM UTC

1. resolved Feb 09, 2026, 09:09 AM UTC

   Incident timeframe: 2026-02-07 00:00 to 2026-02-09 08:19:10 (UTC+0) During the above window, some users may have experienced intermittent errors or failures when accessing certain learning content stored in our backend storage. Impact varied depending on the type of content and how it was accessed. The incident was caused by an internal configuration issue affecting our storage integration layer. This led to certain requests for stored content failing under specific conditions. The configuration has been corrected and normal access to all affected content was fully restored at 2026-02-09 08:19:10 (UTC+0). We are adding automated checks and monitoring around our storage integration configuration and expiration parameters to detect similar issues earlier and prevent reoccurrence.

Read the full incident report →

Started Jan 26, 2026, 12:24 PM UTC· resolved Jan 26, 2026, 02:54 PM UTC

1. investigating Jan 26, 2026, 12:24 PM UTC

   We encounter an issue on web platform where some requests can timeout. We are currently working on it to identify the root cause.

2. identified Jan 26, 2026, 12:26 PM UTC

   The issue has been identified and a fix is being implemented.

3. monitoring Jan 26, 2026, 12:38 PM UTC

   A fix has been implemented and we are monitoring the results.

4. identified Jan 26, 2026, 01:44 PM UTC

   Another concurrent issue contributing to the slowness of the platform is identified and a fix is being implemented.

5. monitoring Jan 26, 2026, 01:49 PM UTC

   A fix has been implemented and we are monitoring the results.

6. resolved Jan 26, 2026, 02:54 PM UTC

   Between 12:05 and 13:38 CET, a deployment issue initially caused slowness on the platform. After resolution, we identified that steps and preview steps were unavailable during this timeframe. The incorrect configuration introduced during deployment was corrected. Furthermore, between 14:24 and 14:50 CET, an unexpected CPU spike usage affected platform performance. We manually scaled ressources to resolve it quickly and will work on better resource scaling anticipation. All services are now fully operational.

Read the full incident report →

Started Dec 23, 2025, 02:01 PM UTC· resolved Dec 23, 2025, 03:13 PM UTC

1. investigating Dec 23, 2025, 02:01 PM UTC

   We have reports of navigation issue during course edition, preventing administrators from moving between steps. We are currently investigating.

2. identified Dec 23, 2025, 02:16 PM UTC

   The issue has been identified and a fix is being implemented.

3. resolved Dec 23, 2025, 03:13 PM UTC

   We’ve identified the issue as related to a recent code update affecting navigation during course editing. A fix has been deployed, and this incident is now fully resolved.

Read the full incident report →

Started Nov 25, 2025, 01:04 PM UTC· resolved Nov 25, 2025, 07:18 PM UTC

1. investigating Nov 25, 2025, 01:04 PM UTC

   Datalab dashboards show empty values, we are investigating.

2. identified Nov 25, 2025, 01:26 PM UTC

   The issue has been identified and a fix is being implemented.

3. resolved Nov 25, 2025, 07:18 PM UTC

   This incident was due to a misconfiguration of the way we forwarded values to our data lab. It’s now fully fixed.

Read the full incident report →

Started Oct 29, 2025, 03:46 PM UTC· resolved Oct 30, 2025, 07:44 AM UTC

1. identified Oct 31, 2025, 09:26 AM UTC

   We are seeing a connection issue since 4:46pm affecting customers hosted on Microsoft Azure. Web platform is not accessible. Our engineering team is working with Azure support to identify the root cause and restore service as quickly as possible. Preliminary informations points to an issue with Azure CDN services.

2. monitoring Oct 31, 2025, 09:26 AM UTC

   Azure has confirmed a widespread outage affecting their CDN services globally. They have completed the deployment of a stable configuration and are progressively restoring traffic through healthy nodes. Early signs of recovery are being observed, and full service restoration is expected within the next few hours.

3. resolved Oct 31, 2025, 09:27 AM UTC

   Azure confirmed end of incident at 01:05 last night. We'll update this status page event with Azure Post Incident Review once available, generally within 14 days.

4. postmortem Nov 12, 2025, 09:59 AM UTC

   **Post Incident Review \(PIR\) - Azure Front Door - Connectivity issues across multiple regions** **What happened?** Between 15:41 UTC on 29 October and 00:05 UTC on 30 October 2025, customers and Microsoft services leveraging Azure Front Door \(AFD\) and Azure Content Delivery Network \(CDN\) experienced connection timeout errors and Domain Name System \(DNS\) resolution issues. From 18:30 UTC on 29 October 2025, as the system recovered gradually, some customers started to see availability improve – albeit with increased latency – until the system fully stabilized by 00:05 UTC on 30 October 2025. Affected Azure services included, but were not limited to: Azure Active Directory B2C, Azure AI Video Indexer, Azure App Service, Azure Communication Services, Azure Databricks, Azure Healthcare APIs, Azure Maps, Azure Marketplace, Azure Media Services, Azure Portal, Azure Sphere Security Service, Azure SQL Database, and Azure Static Web Apps. Other Microsoft services were also impacted, including Microsoft 365 \(see: MO1181369\), the Microsoft Communication Registry website, Microsoft Copilot for Security, Microsoft Defender \(External Attack Surface Management\), Microsoft Dragon Copilot, Microsoft Dynamics 365 and Power Platform \(see: MX1181378\), Microsoft Entra ID \(Mobility Management Policy Service, Identity & Access Management, and User Management\), Microsoft Purview, Microsoft Sentinel \(Threat Intelligence\), Visual Studio App Center, and customers’ ability to open support cases \(both in the Azure Portal and by phone\). **What went wrong and why?** Azure Front Door \(AFD\) and Azure Content Delivery Network \(CDN\) route traffic using globally distributed edge sites supporting customers as well as Microsoft services including various management portals. The AFD control plane generates customer configuration metadata that the data plane consumes for all customer-initiated operations including purge and Web Application Firewall \(WAF\) on the AFD platform. Since customer applications hosted on AFD and CDN can be accessed by their end users from anywhere in the world, these changes are deployed globally across all its edge sites to provide a consistent user experience. A specific sequence of customer configuration changes, performed across two different control plane build versions, resulted in incompatible customer configuration metadata being generated. These customer configuration changes themselves were valid and non-malicious – however they produced metadata that, when deployed to edge site servers, exposed a latent bug in the data plane. This incompatibility triggered a crash during asynchronous processing within the data plane service. This defect escaped detection due to a gap in our pre-production validation, since not all features are validated across different control plane build versions. Azure Front Door employs multiple deployment stages, and a configuration protection system to ensure safe propagation of customer configurations. This system validates configurations at each deployment stage and advances only after receiving positive health signals from the data plane. Once deployments are rolled out successfully, the configuration propagation system also updates a ‘Last Known Good’ \(LKG\) snapshot \(a periodic snapshot of healthy customer configurations\) so that deployments can be automatically rolled back in case of any issues. The configuration protection system waits for approximately a minute between each stage, completing on an average within 5-10 minutes globally. During this incident, the incompatible customer configuration change was made at 15:35 UTC, and was applied to the data plane in a pre-production stage at 15:36 UTC. Our configuration propagation monitoring continued to receive healthy signals – although the problematic metadata was present, it had not caused any issues. Because the data plane crash surfaced asynchronously, after approximately five minutes, the configuration passed through the protection safeguards and propagated to later stages. This configuration \(with the incompatible metadata\) completed propagation to a majority of edge sites by 15:39 UTC. Since the incompatible customer configuration metadata was deployed successfully to the majority of fleet with positive health signal, the LKG was also updated with this configuration. The data plane impact began in phases starting with our preproduction environment at 15:41 UTC, and replicated across all edge sites globally by 15:45 UTC. As the data plane impact started, the configuration protection system detected this and stopped all new and inflight customer configuration changes from being propagated at 15:43 UTC. The incompatible customer configuration was processed by edge servers, causing crashes across our various edge sites. This also impacted AFD’s internal DNS service, hosted on the edge sites of Azure Front Door, resulting in intermittent DNS resolution errors for a subset of AFD customer requests. This sequence of events was the trigger for the global impact on the AFD platform. This AFD incident on 29 October was not directly related to the previous AFD incident, from 9 October. Both incidents were broadly related to configuration propagation risk \(inherent to a global Content Delivery Network, in which route/WAF/origin changes must be quickly deployed worldwide\) but while the failure mode was similar, the underlying defects were different. Azure Front Door’s configuration protection system is designed to validate configurations and proceed only after receiving positive health signals from the data plane. During the AFD incident on 9 October \(Tracking ID: QNBQ-5W8\) that protection system worked as intended, but was later bypassed by our engineering team during a manual cleanup operation. During this AFD incident on 29 October \(Tracking ID: YKYN-BWZ\) the incompatible customer configuration metadata progressed through the protection system, before the delayed asynchronous processing task resulted in the crash. Some of the learnings and repair items from the earlier incident are applicable to this incident as well, and are included in the list of repairs below. **How did we respond?** The issue started at 15:41 UTC and was detected by monitoring at 15:48 UTC, prompting our investigation. By 15:43 UTC the configuration protection system activated in response to widespread data plane issues, and automatically blocked all new and in-flight configuration changes from being deployed worldwide. Since the latest ‘last known good’ \(LKG\) version was updated with the conflicting metadata, we chose not to revert to it. To ensure system stability, we decided not to rollback to prior versions of the LKG either. Instead, we opted to edit the latest LKG, by removing the problematic customer configurations manually. We also opted to block all customer configuration changes from propagating to the data plane at 17:30 UTC so that, as we mitigate, we would not reintroduce this issue. At 17:40 UTC we began deploying the updated LKG configuration across the global fleet. Recovery required reloading all customer configurations at every edge site and rebalancing traffic gradually, to avoid overload conditions as Edge sites returned to service. This deliberate, phased recovery was necessary to stabilize the system while restoring scale and ensuring no recurrence of the issue. Many downstream services that use AFD were able to failover to prevent further customer impact, including Microsoft Entra and Intune portals, and Azure Active Directory B2C. In a more complex example, Azure Portal leveraged its standard recovery process to successfully transition away from AFD during the incident. Users of the Portal would have seen limited impact during this failover process and then been able to use the Portal without issue. Unfortunately, some services within the Portal did not have an established fallback strategy and therefore parts of the Portal experience continued to experience failures even after Portal recovery \(for example, Marketplace\). Timeline of major incident milestones: * 15:35 UTC on 29 October 2025 – Corrupt metadata first introduced, as described above. * 15:41 UTC on 29 October 2025 – Customer impact began, triggered by the resulting crashes. * 15:43 UTC on 29 October 2025 – Configuration protection system activated in response to issues. * 15:48 UTC on 29 October 2025 – Investigation commenced following monitoring alerts being triggered. * 16:15 UTC on 29 October 2025 – Focus of the investigation became examining AFD configuration changes. * 16:18 UTC on 29 October 2025 – Initial communication posted to our public status page. * 16:20 UTC on 29 October 2025 – Targeted communications to impacted customers sent to Azure Service Health. * 17:10 UTC on 29 October 2025 – Began updating the ‘last known good’ LKG configuration to remove problematic configurations manually. * 17:26 UTC on 29 October 2025 – Azure Portal failed away from Azure Front Door. * 17:30 UTC on 29 October 2025 – Blocked all customer configuration propagation to the data plane, in preparation for deploying the new configuration. * 17:40 UTC on 29 October 2025 – Initiated the deployment of our updated ‘last known good’ configuration. * 17:50 UTC on 29 October 2025 – Last known good configuration available to all Edge sites, which began gradually reloading LKG configuration. * 18:30 UTC on 29 October 2025 – AFD DNS servers recovered, allowing us to rebalance traffic manually to a small number of healthy Edge sites. Customers began seeing improvements in availability. * 20:20 UTC on 29 October 2025 – As a sufficient number of Edge sites had recovered, we switched to automatic traffic management – customers continued to see availability improve. * 00:05 UTC on 30 October 2025 – AFD impact confirmed mitigated for customers, as availability and latency had returned to pre-incident levels. Post mitigation, we temporarily blocked all AFD customer configuration changes at the Azure Resource Manager \(ARM\) level to ensure the safety of the data plane. We also implemented additional safeguards including \(i\) fixing the control plane and data plane defects, \(ii\) removing asynchronous processing from the data plane, \(iii\) introducing an additional ‘pre-canary’ stage to test customer configuration \(iv\) extending the bake time during each stage of the configuration propagation, and \(v\) improvements to the data plane recovery time from approximately 4.5 hours to approximately one hour. We began draining the customer configuration queue from 2 November 2025. Once these safeguards were fully implemented, this restriction was removed on 5 November 2025. The introduction of new stages in the configuration propagation pipeline was coupled with additional ‘bake time’ between stages – which has resulted in an increase in configuration propagation time, for all operations including create, update, delete, WAF operations on AFD platform, and cache purges. We continue to work on platform enhancements to ensure a robust configuration delivery pipeline and further reduce the propagation time. For more details on these temporary propagation delays, refer to [http://aka.ms/AFD\_FAQ](http://aka.ms/AFD_FAQ). **How are we making incidents like this less likely or less impactful?** To prevent issues like this, and improve deployment safety... * We have fixed both the original control plane incompatibility, and the data plane bug described above. \(Completed\) * We are now enforcing complete synchronous processing of each customer configuration, before advancing to production stages. \(Completed\) * We have implemented additional stages in our phased configuration rollout, including extended bake time to help detect configuration related issues. \(Completed\) * We are decoupling configuration processing in data plane servers from active traffic-serving instances to isolated worker process instances, thereby removing the risk of any configuration defect impacting data plane processing. \(Estimated completion: January 2026\) * Once our pre-validation of this configuration pipeline is in place, we will work towards reducing the propagation time from 45 minutes to approximately 15 minutes. \(Estimated completion: January 2026\) * We are enhancing our testing and validation framework to ensure backwards compatibility with configurations generated across previous versions of the control plane build. \(Estimated completion: February 2026\) To reduce the blast radius of potential future issues… * We have migrated critical first-party infrastructure \(including Azure Portal, Azure Communication Services, Marketplace, Linux Software Repository for Microsoft Products, Support ticket creation\) into an active-active solution with fail away. \(Completed\) * In the longer term, we are enhancing our customer configuration and traffic isolation, to ensure that no impact to any other customers from single customers’ traffic or configuration issue – utilizing ‘micro cell’ segmentation of the AFD data plane. \(Estimated completion: June 2026\) To be able to recover more quickly from issues… * We have made changes to accelerate our data plane recovery time, by leveraging the local customer configuration caching more effectively – to restore customer configurations within one hour. \(Completed\) * We are making investments to reduce data plane recovery time further, to restore customer configurations within approximately 10 minutes. \(Estimated completion: March 2026\) To improve our communications and support… * We have addressed delays in delivering alerts via Azure Service Health to impacted customers, by making immediate improvements to increase resource thresholds. \(Completed\) * We will expand the automated customer alerts sent via Azure Service Health, to include similar classes of service degradation – to notify impacted customers more quickly. \(Estimated completion: November 2025\) * Finally, we will resolve the technical and staffing challenges that prevented Premier and Unified customers from being able to create a support request, by ensuring that we can failover to our backups systems more quickly. \(Estimated completion: November 2025\)

Read the full incident report →

Started Aug 06, 2025, 07:00 AM UTC· resolved Aug 06, 2025, 07:30 AM UTC

1. resolved Aug 06, 2025, 08:09 AM UTC

   At 9:00 AM (Paris time), we were informed that several pages were not loading correctly. After investigation, our technical team identified an issue affecting specific pages and deployed a fix. The service has been fully restored since 10:00 AM (Paris time). We are continuing to monitor the situation as a precaution.

Read the full incident report →

Started May 09, 2025, 12:04 PM UTC· resolved May 09, 2025, 02:58 PM UTC

1. investigating May 09, 2025, 12:04 PM UTC

   We encounter an issue on web platform. We are currently working on it to identify the root cause.

2. monitoring May 09, 2025, 12:08 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved May 09, 2025, 02:58 PM UTC

   This incident has been resolved.

Read the full incident report →

Started May 02, 2025, 02:26 PM UTC· resolved May 02, 2025, 08:40 PM UTC

1. investigating May 02, 2025, 02:26 PM UTC

   Problem is still present for some clients.

2. investigating May 02, 2025, 02:26 PM UTC

   We are continuing to investigate this issue.

3. resolved May 02, 2025, 08:40 PM UTC

   This incident has been resolved.

Read the full incident report →

Started Apr 10, 2025, 12:41 PM UTC· resolved Apr 10, 2025, 12:47 PM UTC

1. investigating Apr 10, 2025, 12:41 PM UTC

   We encounter an issue on web platform. We have identified the root cause and are working on it.

2. monitoring Apr 10, 2025, 12:44 PM UTC

   A fix has been implemented and we are monitoring the results.

3. resolved Apr 10, 2025, 12:47 PM UTC

   This incident has been resolved. There was an issue with the way we handle auto-scaling where it didn’t scale quickly enough. We modified the auto-scaling configuration to ensure that it doesn’t happen again.

Read the full incident report →