Workspot Outage History

Started May 21, 2026, 04:05 PM UTC· resolved Jun 01, 2026, 09:47 PM UTC

1. identified May 21, 2026, 04:05 PM UTC

   Dear Customer, Workspot has identified an issue affecting the availability of Virtual Desktops hosted in the Azure East US region. Microsoft has confirmed ongoing Azure Compute capacity constraints in this region. Customers attempting to create, update, resume, scale, start, or restart Virtual Machines may experience allocation failures. Current Impact: - VM lifecycle operations may intermittently fail - Running desktops are not impacted unless restarted or redeployed How to Check if You Are Impacted: If your subscription is affected, details will be visible under: -->Azure Portal → Service Health → Active / Maintenance Events Workaround: - Redeploy the Virtual Machine from Workspot Control to help bring it online Recommendations: - Try a different Availability Zone within East US - Use an alternate Azure region with lower demand Mitigation Options: - Disable Pause/Resume (VM runs continuously; may increase cost) - On-demand Capacity Reservation: https://learn.microsoft.com/azure/virtual-machines/capacity-reservation-overview - Reserved Instances (cost benefit + prioritized capacity): https://learn.microsoft.com/azure/cost-management-billing/reservations/save-compute-costs-reservations

2. identified May 22, 2026, 12:45 PM UTC

   Update from Microsoft: Current Status: Due to an increase in demand for compute resources in the region, there are capacity constraints on available compute infrastructure. This in turn resulted in service management operation and allocation failures for VM resources. We are currently working on platform levers to restore healthy buffers in relation to capacity. We will continue to monitor the situation and supply updates accordingly. An update will be provided on 28 May 2026, or as events warrant.

3. resolved Jun 01, 2026, 09:47 PM UTC

   Update from Microsoft: Current Status: Resolved What went wrong and why? Due to an increase in demand for compute resources in a particular zone in the region, there were capacity constraints on available compute infrastructure. This in turn resulted in service management operation and allocation failures for VM resources. How did we respond? We’ve load balanced the capacity demands and restored the capacity buffers to a healthy state. With the congestion resolved, services should now function as usual. What happens next? • Ensuring capacity for our customers is a top priority for Microsoft. We sincerely apologize for any impact this may have caused, but be assured, we are continuously improving tools around monitoring usage, forecasting stock out, maintaining healthy buffers and improving cycle times to reduce this type of issue from happening.

Read the full incident report →

Started May 05, 2026, 12:08 PM UTC· resolved May 06, 2026, 04:32 AM UTC

1. investigating May 05, 2026, 12:08 PM UTC

   We are observing intermittent dynos restarts reported by our platform provider, Heroku. At this time, we have not observed any application downtime. This activity is not related to any security incident. Our engineering team is actively investigating the issue. We will continue to provide updates as we learn more.

2. investigating May 05, 2026, 12:35 PM UTC

   We have not observed any dyno restarts in the last 30 minutes. The engineering team is closely monitoring the environment and continues to investigate the underlying cause of the earlier dyno restarts. We will provide further updates as more information becomes available.

3. investigating May 05, 2026, 01:06 PM UTC

   We have not observed any dyno restarts since the incident was reported, nor any application downtime. The engineering team continues to monitor the environment closely and is investigating the underlying cause of the earlier dyno restarts. We will provide further updates as more information becomes available.

4. investigating May 05, 2026, 01:37 PM UTC

   No new issues have been observed. As a mitigation measure, the engineering team has implemented changes to improve platform stability. The team is closely monitoring the environment and continues to investigate the underlying cause of the earlier instability. The next update will be shared in one hour. We will provide further updates sooner if there are any changes.

5. investigating May 05, 2026, 02:33 PM UTC

   There are no new updates at this time. The platform remains stable, and the engineering team continues to monitor the environment. If there are no changes, the next update will be shared in the next 2 hours. We will provide further updates sooner if needed.

6. investigating May 05, 2026, 04:33 PM UTC

   No new update at this time. The platform remains stable, and the engineering team continues to actively monitor the environment. If there are no changes, the next update will be shared in the next two hours. We will provide an update sooner if there is any change in status.

7. investigating May 05, 2026, 07:01 PM UTC

   There are no new updates at this time. The platform remains stable, and the engineering team continues to monitor the environment. If there are no changes, the next update will be shared in the next 2 hours. We will provide further updates sooner if needed.

8. resolved May 06, 2026, 04:32 AM UTC

   Workspot investigated the restart of the Control Dynos reported by our platform provider. The system is stable and we are closing the incident on status, but continuing to actively monitor and investigate the root cause. We will share the RCA and updates as soon as they become available. Thank you.

Read the full incident report →

Started Apr 24, 2026, 01:31 PM UTC· resolved Apr 28, 2026, 06:52 AM UTC

1. investigating Apr 24, 2026, 01:31 PM UTC

   Workspot has identified an issue affecting accessibility of virtual desktops hosted in the Azure East US region. Microsoft has confirmed an ongoing Azure platform incident impacting a subset of Virtual Machines and Virtual Machine Scale Sets in this region. As a result, some customers may experience loss of network connectivity or inability to access their VMs. This Azure incident aligns with the connectivity issues currently being observed. Workspot is monitoring the situation closely and is in communication with Microsoft regarding the incident status. We will provide updates as they become available.

2. investigating Apr 24, 2026, 01:38 PM UTC

   Impact Statement: Starting at 11:39 UTC on 24 Apr 2026, a subset of customers using Virtual Machines and Virtual Machine Scale Sets in East US may experience issues with resources hosted in this region. Current Status: We are aware of this issue and are actively investigating. The next update will be provided as events warrant. IMPACTED SERVICE(S) AND REGION(S): Service Name Region Virtual Machine Scale Sets -->East US Virtual Machines --> East Asia

3. investigating Apr 24, 2026, 02:38 PM UTC

   Update from the Microsoft: Service: Virtual Machine Scale Sets, Virtual Machines Region: East US, East Asia Impact Statement: Starting at 11:39 UTC on 24 Apr 2026, a subset of customers using Virtual Machines and Virtual Machine Scale Sets in East US may experience issues with resources hosted in this region. Current Status:  We are aware of this issue and are actively investigating. The next update will be provided as events warrant. IMPACTED SERVICE(S) AND REGION(S): Service Name Region Virtual Machine Scale Sets -->East US Virtual Machines --> East Asia

4. investigating Apr 24, 2026, 03:52 PM UTC

   Microsoft has shared an update on the ongoing Azure platform incident impacting Virtual Machines and Virtual Machine Scale Sets in the East US region. Microsoft identified the issue through automated monitoring, which detected an unusual increase in failures within the service supporting virtual machine operations in East US. Microsoft is currently investigating the underlying cause and is actively working toward mitigation. Current Status: The issue remains in progress and under investigation by Microsoft. Customers may continue to experience VM connectivity or access issues in East US, and related impact may also be seen in East Asia. Microsoft indicated that the next update will be shared within approximately 60 minutes, or sooner if there are meaningful developments. Workspot continues to monitor the situation closely and will provide further updates as new information becomes available from Microsoft.

5. investigating Apr 24, 2026, 05:06 PM UTC

   Current Status: At this time, no new update has been received from Microsoft beyond their previous communication. The incident remains under investigation on their side. We are continuing to monitor the situation closely and are tracking customer impact, which currently includes multiple reported cases of virtual desktop connectivity issues and VM accessibility failures. Next Update: We will provide an update as soon as Microsoft shares further information or once there is any material change in status.

6. investigating Apr 24, 2026, 05:56 PM UTC

   Microsoft has shared below update on the ongoing Azure platform incident impacting Virtual Machines and Virtual Machine Scale Sets in the East US region. CURRENT STATUS: We detected this issue through automated monitoring after identifying an unusual drop in service success rates. The issue was caused by a recent change, and rollback actions are in progress across availability zones (AZs). Customers may begin to see recovery if using resources in AZ03, with Availability Zone 01 currently in progress. Please refer to this documentation to understand the logical to physical availability zone mapping for your subscription. However, there is no confirmed estimated time for full resolution at this moment. The next update will be shared within the next 60 minutes, or as events warrant.

7. investigating Apr 24, 2026, 11:38 PM UTC

   Microsoft has shared below update on the ongoing Azure platform incident impacting Virtual Machines and Virtual Machine Scale Sets in the East US region. Current Status: Rollback is complete for Availability Zone (AZ) 01 and AZ03, and is still in progress for AZ02. Mitigation is expected to complete within the next 1 hour. Customers should observe improvements to their services, and downstream services are also starting to recover. NEXT STEPS: The next update will be shared within 60 minutes, or as events warrant.

8. monitoring Apr 25, 2026, 04:15 AM UTC

   Update: Microsoft has shared below update on the ongoing Azure platform incident impacting Virtual Machines and Virtual Machine Scale Sets in the East US region. Current Status: Resolved What happened? Between 11:39 UTC on 24th April and 00:15 UTC on 25 April 2026, a platform issue resulted in an impact to a subset of Azure services in the East US region. Impacted customers experienced failures or delays when provisioning, scaling, or updating Azure resources. Intermittent connectivity issues may also be seen on existing running workloads in Virtual Machines and Azure Virtual Desktop sessions. This issue is now mitigated. An update with more information will be provided shortly. Next Steps: The next update will be shared once Microsoft provides the RCA. In the meantime, Workspot will continue to monitor impacted pools and virtual desktops to ensure stability and identify any residual issues.

9. resolved Apr 28, 2026, 06:52 AM UTC

   Update: We have received the Preliminary Post Incident Review(PIR) from Microsoft for the Azure East US control plane issue that impacted your environment on April 24. Here is a summary of what happened and why: STATUS: RCA COMMUNICATION: This is our Preliminary PIR to share what we know so far. After our internal retrospective is completed (generally within 14 days) we will publish a Final PIR with additional details. What happened? Between 11:30 UTC on 24 April and 00:15 UTC on 25 April 2026, customers may have experienced failures or delays when attempting to provision, scale, or update resources in East US. Beyond this, a smaller subset of impacted customers may have experienced intermittent connectivity issues on existing workloads (including Virtual Machines and Azure Virtual Desktop sessions) for scenarios dependent on unhealthy internal service dependencies. The issue initially began with impact to a subset of customers in a single Availability Zone (physical AZ-01) but as demand shifted, similar symptoms were observed impacting a subset of customers in AZ-02 and AZ-03. While none of these zones were impacted for the full duration of the incident, customers experienced periods of impact in each zone for portions of the incident. The following services were among those affected: Azure Application Gateway, Azure App Service, Azure Batch, Azure Cache for Redis, Azure Data Explorer, Azure Data Factory, Azure Databricks, Azure Health Data Services, Azure Kubernetes Service (AKS), Azure Red Hat OpenShift, Azure Service Fabric, Azure Synapse Analytics, Azure Virtual Desktop, Azure Virtual Machines, Azure Virtual Network Manager, Azure VMware Solution, Oracle Database@Azure, Virtual Machine Scale Sets – and potentially additional services that were dependent on new compute allocations in the region. Note: Logical availability zones assigned to customer subscriptions may map to different physical availability zones. Customers can use the Locations API to understand this mapping: https://learn.microsoft.com/rest/api/resources/subscriptions/list-locations?HTTP#availabilityzonemappings. What went wrong and why? The Azure PubSub service is a key component of the networking control plane, acting as an intermediary between resource providers and networking agents on Azure hosts. Resource providers, such as the Network Resource Provider, publish customer configurations during Virtual Machine or networking create, update, or delete operations. Networking agents (subscribers) on the hosts retrieve these configurations to program the hosts networking stack. Additionally, the service functions as a cache, ensuring efficient retrieval of configurations during VM reboots or restarts. This capability is essential for deployments, resource allocation, and traffic management in Azure Virtual Network (VNet) environments. During normal platform operations, one partition of this PubSub service in AZ-01 became unhealthy and automatically attempted to fail over to a secondary replica. The failover did not complete successfully, resulting in a partial loss of control plane availability within AZ-01. We intervened to investigate and attempted a manual failover of the primary partition, but this attempt was also unsuccessful. Shortly afterward, we observed a similar condition in AZ-03, which led to a partial loss of control plane availability in AZ-03 as well. As the investigation progressed, we suspected that a previously deployed update to a regional control plane dependency had introduced a latent regression. This issue did not surface during earlier validation and only manifested when failover conditions were triggered under sustained production load. As part of our mitigation efforts, we identified a version from the prior week that represented a Last Known Good (LKG) state. We first applied this rollback in AZ-03, which successfully restored control plane service health in that zone. Based on this, we began rolling back the affected components in AZ-01. By design, rollback operations are executed in stages by Azure Fabric controllers using update domains to ensure platform safety, while recovery proceeds incrementally. While mitigation was in progress, the platform was unable to maintain two healthy instances of the PubSub service across availability zones simultaneously, which is a requirement for normal replication and control plane operations. This resulted in a loss of quorum of the service. As the system attempted to rebalance, impact shifted between availability zones, leading to periods of degraded behavior across multiple zones. Similar failure patterns began to appear in AZ-02 and again in AZ-03, expanding the scope of impact across the region. For AZ-02 we initiated and completed a rollback, and although AZ-03 had previously shown recovery following the rollback, subsequent instability indicated that the rollback in that zone had not fully completed, because of an orchestration fault. As impact reemerged, rollback operations in AZ-03 were restarted and then completed, fully restoring service health. How did we respond? • 11:30 UTC on 24 April 2026 – Customer impact began. We observed failures or delays when customers attempted to provision, scale, or update resources in the affected region. • 11:38 UTC on 24 April 2026 – We detected an issue in AZ-01. A control plane partition became unhealthy and automatic failover attempts did not complete successfully. • 11:38–13:40 UTC on 24 April 2026 – We attempted manual failover in AZ-01. These efforts did not successfully restore service. • 13:40 UTC on 24 April 2026 – We identified a recently deployed update as the likely cause of the issue. • 13:50 UTC on 24 April 2026 – We began observing similar symptoms in AZ-03, indicating the issue was affecting multiple availability zones. • 14:07 UTC on 24 April 2026 – We initiated rollback to a previously known good version in AZ-03. • 15:03 UTC on 24 April 2026 – We observed significant recovery in AZ-03. Control plane availability exceeded 99%. • 15:04 UTC on 24 April 2026 – We initiated rollback actions to AZ-01. • 18:52 UTC on 24 April 2026 – We observed significant improvement in AZ-01 as rollback progressed. • 19:02 UTC on 24 April 2026 – We confirmed AZ-01 had recovered to greater than 99% availability, while the rollback continued in the background. • 19:05 UTC on 24 April 2026 – We observed similar symptoms in AZ-02 as load redistributed across the region. • 19:10 UTC on 24 April 2026 – We initiated rollback to a known good version in AZ-02. • 21:02 UTC on 24 April 2026 – We observed instability reappear in AZ-03. We determined this was because the rollback had not yet completed across all update domains. Consequently, we manually unblocked the rollback across remaining update domains in AZ03 to ensure stable recovery. • 22:39 UTC on 24 April 2026 – We confirmed rollback was fully completed in AZ-03. • 23:22 UTC on 24 April 2026 – We confirmed rollback was fully completed in AZ-02, completing PubSub mitigation across all affected zones. • 00:15 UTC on 25 April 2026 – We validated downstream service recovery and PubSub health across all zones in the region. How are we making incidents like this less likely or less impactful? • We have assessed the risk of occurrence in other high volume regions, and have taken steps to rollback this PubSub service in these regions out of an abundance of caution. (Completed) • We are investing in improving our test coverage surrounding the failure cases and load patterns that contributed to this incident, to catch issues like this one before they reach production. (Estimated completion: TBD) • We are working to reduce rollback complexity, to be able to mitigate issues like this more quickly in future. (Estimated completion: TBD) • This is our Preliminary PIR to share what we know so far. After our internal retrospective is completed (generally within 14 days) we will publish a Final PIR with additional details. How can customers make incidents like this less impactful? • Consider using Availability Zones (AZs) to run your services across physically separate locations within an Azure region. To help services be more resilient to localized failures like this one (which predominantly impacted zones at different times) many Azure services support zonal, zone-redundant, and/or always-available configurations: https://docs.microsoft.com/azure/availability-zones/az-overview • For mission-critical workloads, customers should consider a multi-region geodiversity strategy to avoid impact from incidents like this one that impacted a single region: https://learn.microsoft.com/azure/architecture/patterns/geodes and https://learn.microsoft.com/azure/well-architected/design-guides/regions-availability-zones • More generally, consider evaluating the reliability of your applications using guidance from the Azure Well-Architected Framework and its interactive Well-Architected Review: https://aka.ms/AzPIR/WAF • The impact times above represent the full incident duration, so are not specific to any individual customer. Actual impact to service availability varied between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring • Finally, consider ensuring that the right people in your organization will be notified about any future service issues – by configuring Azure Service Health alerts. These can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/AzPIR/Alerts

Read the full incident report →

Started Mar 20, 2026, 06:09 PM UTC· resolved Mar 24, 2026, 05:17 AM UTC

1. investigating Mar 20, 2026, 06:09 PM UTC

   Pingdom reported production was down for Five minutes(4:08 PM UTC - 4:13 PM UTC). There was a 5 minute intermittent service outage by a subservice provider resetting their system. The service automatically recovered. We will share the RCA once we receive from the subservice provider.

2. resolved Mar 24, 2026, 05:17 AM UTC

   The system has been stable and functioning normally for the past 48 hours. Our internal investigation found no application-level anomalies. We are closing this incident while the root cause analysis (RCA) from our infrastructure provider is still ongoing. We will share updates as soon as they become available. Thank you for your patience.

Read the full incident report →

Started Feb 02, 2026, 09:38 AM UTC· resolved Feb 04, 2026, 03:28 AM UTC

1. investigating Feb 02, 2026, 09:38 AM UTC

   We are currently experiencing intermittent availability issues with the Workspot Control service. The engineering team is actively working on this issue with top priority. We will share more details as they become available. This is an application‑level issue and not a security incident. We apologize for any inconvenience this may cause.

2. identified Feb 02, 2026, 10:02 AM UTC

   Following the recent 26.1 release, we have identified an issue impacting the Workspot Control service. A newly introduced application feature is not behaving as expected, resulting in an increased volume of system requests. This has led to request queuing and intermittent restarts of the Control service. Our engineering team is actively working on testing and implementing a fix with the highest priority to stabilize the service.

3. identified Feb 02, 2026, 11:12 AM UTC

   The issue has been identified, and corrective measures are in place. Our engineering team is reviewing remaining impact areas and completing final validation to ensure service stability before rollout.

4. identified Feb 02, 2026, 12:43 PM UTC

   Our engineering team is still reviewing the impact areas and completing validation to ensure service stability. We will share more details as they become available.

5. identified Feb 02, 2026, 02:58 PM UTC

   Our engineering team continues to review the impact areas and complete validation to ensure service stability. There are no new updates since our last communication.

6. identified Feb 02, 2026, 04:15 PM UTC

   Control is actively accepting requests; however, responses may be intermittent due to a high volume of queued requests. - Control login: Login requests may occasionally time out. Retrying may be successful. - User login: Most users can establish a session. If an issue occurs, please retry. Our Engineering team is also exploring a potential rollback option while ensuring data consistency. We are working to determine an ETA and will provide an update as soon as more information becomes available. We appreciate your patience as we continue to improve service stability.

7. identified Feb 02, 2026, 04:31 PM UTC

   The Engineering team is planning to roll back the release deployed over the weekend. We will continue to share status updates approximately every 30 minutes. An ETA is currently being assessed, and we will provide further updates as more information becomes available. We appreciate your patience as we work to restore service stability.

8. identified Feb 02, 2026, 05:00 PM UTC

   The Engineering team is continuing to assess the rollback plan and review the code changes. An ETA is being evaluated based on the outcome of the review and associated testing. We appreciate your patience as we work to restore service stability.

9. identified Feb 02, 2026, 05:31 PM UTC

   There is no new update at this time. The Engineering team continues to review the rollback plan and assess next steps. We will share another update as soon as more information becomes available. We appreciate your continued patience as we work to restore service stability.

10. identified Feb 02, 2026, 06:04 PM UTC

    There is no new update at this time. The Engineering team continues to review the rollback plan and assess next steps. We will share another update as soon as more information becomes available. We appreciate your continued patience as we work to restore service stability.

11. identified Feb 02, 2026, 06:33 PM UTC

    There is no new update at this time. The Engineering team continues to review the rollback plan and assess next steps. We will share another update as soon as more information becomes available. We appreciate your continued patience as we work to restore service stability.

12. identified Feb 02, 2026, 07:09 PM UTC

    Engineering team has made internal progress reviewing the rollback code, and testing activities are ongoing. We will share another update as soon as more information becomes available. We appreciate your continued patience as we work to restore service stability.

13. identified Feb 02, 2026, 07:56 PM UTC

    There is no new update at this time. Engineering team has made internal progress reviewing the rollback code, and testing activities are ongoing. We will share another update as soon as more information becomes available. We appreciate your continued patience as we work to restore service stability.

14. identified Feb 02, 2026, 08:22 PM UTC

    Engineering team is estimating 90 minutes to complete rollback testing and then a short maintenance window to rollback at approximately 1030 PM UTC. We will update as timeline firm up. We appreciate your patience as we work to restore service stability.

15. monitoring Feb 02, 2026, 09:41 PM UTC

    Engineering team has completed the rollback, and all services have now been successfully restored. We are continuing to monitor the environment and will keep you posted with any further updates. We greatly appreciate your patience and cooperation.

16. monitoring Feb 03, 2026, 11:00 AM UTC

    Following the rollback performed, no new issues have been reported. Customers have confirmed that the Workspot service is stable. Workspot Control is operating as expected, and all requests are being handled properly.

17. monitoring Feb 04, 2026, 03:22 AM UTC

    We are continuing to monitor for any further issues.

18. resolved Feb 04, 2026, 03:28 AM UTC

    The service has been fully restored after the rollback was performed. We appreciate your patience. A detailed Root Cause Analysis (RCA) will be shared once available.

Read the full incident report →

Started Nov 14, 2025, 03:57 PM UTC· resolved Nov 14, 2025, 06:30 PM UTC

1. investigating Nov 14, 2025, 03:57 PM UTC

   We are currently experiencing intermittent issues with Workspot Control logins and non-persistent VM logins. Our team is actively investigating the issue and has escalated the matter by raising a priority ticket with the platform vendor. We are working closely with them to resolve this as quickly as possible. Further updates will be shared as soon as more information becomes available.

2. monitoring Nov 14, 2025, 05:20 PM UTC

   Update: We are seeing improvement in both Workspot Control logins and non-persistent VM logins. At this time, we are closely monitoring the situation to ensure stability. Our investigation is ongoing, and we will share additional details as they become available.

3. resolved Nov 14, 2025, 06:30 PM UTC

   This issue has been resolved, and customers can now log in to Workspot Control and access non-persistent VMs without any issues. The platform is stable, and normal operations have resumed. We will continue to monitor the environment to ensure stability. We will share a post-incident summary once our investigation ends and relevant details are available.

Read the full incident report →

Started Oct 20, 2025, 05:11 PM UTC· resolved Oct 31, 2025, 06:51 AM UTC

1. monitoring Oct 20, 2025, 05:11 PM UTC

   We would like to inform you that Amazon Web Services (AWS) has been experiencing some operational issues. At this time, there is no operational impact to Workspot customers or services. Our engineering and operations teams are closely monitoring the situation to ensure continued stability and performance across all Workspot systems. In the event of any changes or potential service impact, we will provide timely updates. Latest Updates from AWS: --> Increased Error Rates and Latencies Oct 20 10:03 AM PDT: We continue to apply mitigation steps for network load balancer health and recovering connectivity for most AWS services. Lambda is experiencing function invocation errors because an internal subsystem was impacted by the network load balancer health checks. We are taking steps to recover this internal Lambda system. For EC2 launch instance failures, we are in the process of validating a fix and will deploy to the first AZ as soon as we have confidence we can do so safely. We will provide an update by 10:45 AM PDT. Oct 20 9:13 AM PDT: We have taken additional mitigation steps to aid the recovery of the underlying internal subsystem responsible for monitoring the health of our network load balancers and are now seeing connectivity and API recovery for AWS services. We have also identified and are applying next steps to mitigate throttling of new EC2 instance launches. We will provide an update by 10:00 AM PDT. Link --> https://health.aws.amazon.com/health/status We will continue to monitor the situation and provide updates as necessary. Thank you for your patience and cooperation.

2. resolved Oct 31, 2025, 06:51 AM UTC

   This incident has been resolved, and no Workspot customers have reported experiencing the issue. Here is the summary of the AWS incident: https://aws.amazon.com/message/101925/

Read the full incident report →

Started Oct 15, 2025, 06:58 PM UTC· resolved Oct 31, 2025, 06:45 AM UTC

1. identified Oct 15, 2025, 06:58 PM UTC

   Issue Summary : A subset of users attempting to log in to the Workspot Windows Client are experiencing authentication failures after entering the MFA verification code. Root Cause : The issue has been traced to the installation of the Preview Patch KB5065789 or the October 2025 Windows Cumulative Update. This update appears to interfere with the authentication flow used by the Workspot Windows Client. Workaround : >>Option 1: To restore normal login functionality, affected users should uninstall KB5065789 and the October 2025 Cumulative update from their local Windows device and perform a system reboot. This action has been confirmed to resolve the issue in multiple cases. >>Option 2: As an alternative, users can access Workspot resources using the Web Client, which remains fully functional and unaffected by this issue. Next Steps : We recommend IT administrators review devices for the presence of KB5065789 and coordinate its removal where login issues are reported. These workarounds will allow users to log in and continue working until we provide a permanent resolution. Thank you for your patience and understanding. We truly appreciate your cooperation.

2. identified Oct 16, 2025, 03:44 AM UTC

   Microsoft has confirmed that the issue is caused by recent Windows updates — specifically KB5065789, KB5066835, and KB5066131 — which interfere with the HTTP.sys component used in server-side applications. This affects environments running Windows Server 2025 and Windows 11, and disrupts the authentication flow used by the Workspot Windows Client. Resolution Steps (as per Microsoft): Microsoft recommends the following actions to mitigate the issue: - Step 1: On the affected machine, go to Settings > Windows Update and click “Check for updates.” - Step 2: Restart the machine, regardless of whether new updates are found. This reboot helps restore normal functionality. For air-gapped environments, Microsoft has provided a manual rollback installer: Download Known Issue Rollback MSI https://download.microsoft.com/download/16d61dc0-7e94-4cd2-ba3c-4f59dece8488/Windows 11 24H2, Windows 11 25H2 and Windows Server 2025 KB5066835 251015_22001 Known Issue Rollback.msi Next Steps: We recommend IT administrators: - Review devices for the presence of the affected KB updates. - Apply the Microsoft-recommended mitigation steps. - Use the Web Client as a temporary workaround if needed. We will continue to monitor the situation and provide updates as necessary. Thank you for your patience and cooperation.

3. resolved Oct 31, 2025, 06:45 AM UTC

   This issue has been resolved in accordance with the Microsoft article: https://learn.microsoft.com/en-us/windows/release-health/status-windows-11-25h2#issue-details. The problem is addressed in the October Windows non-security preview update (KB5067036) and all subsequent updates. Microsoft recommends installing the latest Windows updates, as they include important improvements and fixes, including this resolution. Additionally, we have not received any further reports of this issue from customers who have applied the recommended Microsoft updates.

Read the full incident report →

Started Sep 10, 2025, 02:43 PM UTC· resolved Sep 17, 2025, 04:05 AM UTC

1. identified Sep 10, 2025, 02:43 PM UTC

   We are aware that some customers are experiencing issues with VM provisioning and resuming operations in the East US and East US 2 Azure regions. Our team has already engaged Microsoft Azure Support with a Severity 1 ticket. According to Microsoft, the issue is related to unhealthy dependencies that are causing failures. They are actively investigating to determine the root cause and identify possible mitigation steps. We will share further updates as soon as Microsoft provides more information.

2. identified Sep 10, 2025, 02:43 PM UTC

   We are continuing to work with Microsoft

3. identified Sep 10, 2025, 03:40 PM UTC

   As per the latest update from Microsoft, they have observed that two of the three impacted zones have returned to a healthy state, and failure rates are now trending downward. Customers may see signs of recovery over time. This failure issue is limited to the East US2 region.

4. identified Sep 10, 2025, 04:50 PM UTC

   Below is the summary provided by Microsoft regarding the ongoing issue in the East US 2 region: "Current Status: We detected the issue through automated monitoring following a spike in failure rates. The root cause has been traced to a backend service responsible for managing resource placement, which is experiencing performance degradation. This has led to delays and failures in resource creation and management. Our engineering teams have attempted several recovery actions, including restarting key service components and shifting workloads away from affected infrastructure. However, these efforts have not yet fully resolved the issue due to system-level constraints. Mitigation Actions Taken: • Restarted critical service components to restore functionality. • Attempted to reroute workloads from affected infrastructure. • Initiated multiple recovery cycles for the impacted backend service. Active Recovery Efforts: • Zone 3 (Az03) is showing signs of improvement after targeted recovery actions. System performance has stabilized and is being closely monitored. • Zone 2 (Az02) is undergoing similar recovery steps. While new resource deployments remain restricted, existing resources are beginning to recover. • We are working to redistribute workloads to healthier zones (Az01 and Az03). However, limited capacity in these zones is causing throttling and delays. • In parallel, we are exploring emergency capacity expansion to alleviate resource constraints and accelerate recovery. Note that the 'logical' zones used by each customer subscription may correspond to different physical zones - customers can use the Locations API to understand this mapping, to confirm which resources run in this physical Availability Zone(AZ). Next Steps: We continue to monitor all zones and prioritize recovery in Az02 to restore capacity in the region."

5. identified Sep 10, 2025, 07:00 PM UTC

   Below is the summary provided by Microsoft regarding the ongoing issue in the East US 2 region: "Current Status: We detected the issue through automated monitoring following a spike in failure rates. We have identified a performance issue in a core infrastructure component responsible for managing resource placement. This is causing delays and failures in virtual machine provisioning. The issue stems from severe transaction delays and high system load in two zones of the region. Active Recovery Efforts: • Zone 2 (Az02) - Gradually re-enabling traffic (~50%) to Zone 2 using controlled allocation strategies. This should be introducing more capacity for the region and allowing higher allocation success rates. • Zone 3 (Az03) and Zone 1 (Az01) – Is recovered but due to partial traffic enabled for Zone 2, customers may still see allocation failures here. As Zone 2 traffic increases, allocation success should increase in all zones. Note that the 'logical' zones used by each customer subscription may correspond to different physical zones - customers can use the Locations API to understand this mapping, to confirm which resources run in this physical AZ. The next update will be provided within 60 minutes, or sooner if significant progress is made. "

6. identified Sep 10, 2025, 10:26 PM UTC

   As per Microsoft, this issue has been mitigated. Here is the incident summary, shared by Microsoft SUMMARY OF IMPACT: What happened? Between 09:12 UTC and 18:50 UTC on 10 September 2025, a platform issue resulted in an impact to multiple Azure services in the East US 2 region, more specifically two zones (Az02 and Az03). Impacted customers may have experienced error notifications when performing service management operations - such as create, delete, update, scaling, start or stop - for resources hosted in this region. The primary impacted service affected was Virtual Machines or Virtual Machines Scale Sets, but this would have resulted in issues for services dependent upon such Compute resources, such as Azure Databricks, Azure Kubernetes Service, Azure Synapse Analytics, Backup, and Data Factory. Customers that still see failed or unhealthy resources should attempt to update or redeploy the resource. What do we know so far? Our investigation identified that the issue impacting resource provisioning in East US 2 was linked to a failure in the platform component responsible for managing resource placement. The system is designed to recover quickly from transient issues, but in this case, the prolonged performance degradation caused recovery mechanisms themselves to become a source of instability. The incident was primarily driven by a combination of platform recovery behavior and sustained performance degradation. While customer-generated load remained within expected limits, internal platform services began retrying failed operations aggressively when performance issues emerged. These retries, intended to support resilience, instead created a surge in internal system activity. How did we respond? • 09:12 UTC on 10 September 2025 – Customer impact began. • 09:13 UTC on 10 September 2025– Our monitoring systems observed a rise in failure rates, triggering an alert and prompting our team to initiate an investigation. • 12:08 UTC on 10 September 2025 – We identified unhealthy dependencies in core infrastructure components as initial contributing factors. • 13:34 UTC on 10 September 2025 – Began mitigation efforts that included - Restarted critical service components to restore functionality, reroute workloads from affected infrastructure, initiated multiple recovery cycles for the impacted backend service, on recovery, internal workloads processed through backlogs to get to the current healthy state, and executed capacity operations to free up resources. • 18:50 UTC on 10 September 2025 – After a period of monitoring to validate the health of services, we were confident that the control plane service was restored, and no further impact was observed to downstream services for this issue.

7. monitoring Sep 11, 2025, 03:49 PM UTC

   A fix has been implemented and we are monitoring the results.

8. resolved Sep 17, 2025, 04:05 AM UTC

   Microsoft has informed us that the outage in the region has been mitigated. Please, find the summary below, as per Microsoft: STATUS: Mitigated 9/10/2025 9:55:15 PM UTC SUMMARY OF IMPACT: What happened? Between 09:12 UTC and 18:50 UTC on 10 September 2025, a platform issue resulted in an impact on multiple Azure services in the East US 2 region, more specifically, two zones (Az02 and Az03). Impacted customers may have experienced error notifications when performing service management operations, such as creating, deleting, updating, scaling, starting, or stopping, for resources hosted in this region. The primary impacted service was Virtual Machines or Virtual Machine Scale Sets, but this would have resulted in issues for services dependent on such Compute resources, such as Azure Databricks, Azure Kubernetes Service, Azure Synapse Analytics, Backup, and Data Factory. Customers who still see failed or unhealthy resources should attempt to update or redeploy the resource. What do we know so far? Our investigation identified that the issue impacting resource provisioning in East US 2 was linked to a failure in the platform component responsible for managing resource placement. The system is designed to recover quickly from transient issues, but in this case, the prolonged performance degradation caused recovery mechanisms themselves to become a source of instability. The incident was primarily driven by a combination of platform recovery behavior and sustained performance degradation. While customer-generated load remained within expected limits, internal platform services began retrying failed operations aggressively when performance issues emerged. These retries, intended to support resilience, instead created a surge in internal system activity. How did we respond? 09:12 UTC on 10 September 2025 – Customer impact began. 09:13 UTC on 10 September 2025– Our monitoring systems observed a rise in failure rates, triggering an alert and prompting our team to initiate an investigation. 12:08 UTC on 10 September 2025 – We identified unhealthy dependencies in core infrastructure components as initial contributing factors. 13:34 UTC on 10 September 2025 – Began mitigation efforts that included - Restarted critical service components to restore functionality, rerouted workloads from affected infrastructure, initiated multiple recovery cycles for the impacted backend service, on recovery, internal workloads processed through backlogs to get to the current healthy state, and executed capacity operations to free up resources. 18:50 UTC on 10 September 2025 – After a period of monitoring to validate the health of services, we were confident that the control plane service was restored, and no further impact was observed on downstream services for this issue. What happens next? Our team will be completing an internal retrospective to understand the incident in more detail. We will publish a Preliminary Post Incident Review (PIR) within approximately 72 hours to share more details on what happened and how we responded. After our internal retrospective is completed, generally within 14 days, we will publish a Final Post Incident Review with any additional details and learnings. To get notified when that happens, and/or to stay informed about future Azure service issues, make sure that you configure and maintain Azure Service Health alerts – these can trigger emails, SMS, push notifications, webhooks, and more: https://aka.ms/ash-alerts For more information on Post Incident Reviews, refer to https://aka.ms/AzurePIRs The impact times above represent the full incident duration, so they are not specific to any individual customer. Actual impact to service availability may vary between customers and resources – for guidance on implementing monitoring to understand granular impact: https://aka.ms/AzPIR/Monitoring Finally, for broader guidance on preparing for cloud incidents, refer to https://aka.ms/incidentreadiness Stay informed about your Azure services - Visit Azure Service Health to get your personalized view of possible impacted Azure resources, downloadable Issue Summaries, and engineering updates. - Set up service health alerts to stay notified of future service issues, planned maintenance, or health advisories.

Read the full incident report →

Started Jun 12, 2025, 07:10 PM UTC· resolved Jun 12, 2025, 09:03 PM UTC

1. identified Jun 12, 2025, 07:10 PM UTC

   GCP is reporting 11:46 PDT "Multiple GCP products are experiencing impact due to Identity and Access Management Service Issue". https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW

2. monitoring Jun 12, 2025, 07:15 PM UTC

   GCP update 12:09 PDT "Our engineers are continuing to mitigate the issue and we have confirmation that the issue is recovered in some locations."

3. monitoring Jun 12, 2025, 07:32 PM UTC

   GCP: 12:30 PDT All locations except us-central1 have fully recovered. us-central1 is mostly recovered. We do not have an ETA for full recovery in us-central1.

4. monitoring Jun 12, 2025, 07:44 PM UTC

   GCP 1241 PDT: "Our engineers have identified the root cause and have applied appropriate mitigations. While our engineers have confirmed that the underlying dependency is recovered in all locations except us-central1, we are aware that customers are still experiencing varying degrees of impact on individual google cloud products. All the respective engineering teams are actively engaged and working on service recovery. We do not have an ETA for full service recovery."

5. monitoring Jun 12, 2025, 08:19 PM UTC

   GCP 13:16 PDT We have identified the root cause and applied appropriate mitigations. Our infrastructure has recovered in all regions except us-central1.

6. resolved Jun 12, 2025, 09:03 PM UTC

   GCP 14:00 PDT We have implemented mitigation for the issue in us-central1 and multi-region/us and we are seeing signs of recovery. Please see GCP's status page for more details. https://status.cloud.google.com/incidents/ow5i3PPK96RduMcb1SsW

Read the full incident report →

Started Jun 10, 2025, 07:34 AM UTC· resolved Jun 12, 2025, 07:46 PM UTC

1. investigating Jun 10, 2025, 07:34 AM UTC

   We are currently investigating this issue. We have escalated it by raising a priority ticket with the platform vendor and are working with them to resolve it as quickly as possible. We will provide updates as we learn more.

2. investigating Jun 10, 2025, 08:35 AM UTC

   Update: The issue is still ongoing, and our team is actively working on it with the highest priority. We will share further updates as we receive more information.

3. investigating Jun 10, 2025, 10:33 AM UTC

   Update: The issue is ongoing, and our team is still working closely with the platform vendor to resolve it. We understand the significant impact this has on our customers. We remain committed to restoring service as quickly as possible and will continue to provide updates as we receive more information.

4. investigating Jun 10, 2025, 12:20 PM UTC

   Update: The issue is still ongoing. We are actively following-up with platform vendor. We will keep you updated in regular interval or as soon as we have more information.

5. investigating Jun 10, 2025, 02:32 PM UTC

   Heroku (Salesforce) platform is working through intermittent issue since 0603 UTC affecting multiple Heroku customers. For Workspot, this has impacted certain services that resume VMs and broker user connections to desktops. At this time, some customers will have access to persistent desktops. Users are able to connect to persistent desktops that were powered on (listed as ready, online or disconnected) at the time of impact. Engineering is working on: 1. Verifying that Admins may be able to power on "paused" VMs so that users can connect by selecting "Reboot" VMs from the Control UI or API service. 2. Verifying disabling pause/resume service so no VMs will pause or stop running until Heroku is recovered. 3. Verifying access to non persistent desktop and applications. Workspot Support and Operations will be continuing to communicate updates via status.workspot.com every 60 minutes.

6. investigating Jun 10, 2025, 03:56 PM UTC

   Update: Engineering is still working on: 1. Verifying that Admins may be able to power on "paused" VMs so that users can connect by selecting "Reboot" VMs from the Control UI or API service. 2. Verifying disabling pause/resume service so no VMs will pause or stop running until Heroku is recovered. 3. Verifying access to non persistent desktop and applications.

7. investigating Jun 10, 2025, 05:35 PM UTC

   Heroku (Salesforce) platform is still working through resolving the intermittent issues since 0603 UTC, affecting multiple Heroku customers. Workspot is applying mitigations to workaround and limit the impact of the Heroku outage. Some customers are seeing the restoration of their service. We are still monitoring the environment as Heroku is still working through the process. Workspot Support and Operations will be continuing to communicate updates via status.workspot.com every 60 minutes.

8. monitoring Jun 10, 2025, 06:18 PM UTC

   Heroku (Salesforce) platform has been working through intermittent issues since 0603 UTC, affecting multiple Heroku customers. Workspot has applied mitigations to limit the impact of the Heroku outage. Customers are seeing the restoration of the service for persistent, non-persistent, and applications. Please try accessing your desktops now. Let us know if you have any issues connecting to your desktops. Customer admins don’t have to reboot the VMs at this time per earlier instructions. We are still monitoring the environment as Heroku is still working through the process. Workspot Support and Operations will be continuing to communicate updates via status.workspot.com every 60 minutes.

9. monitoring Jun 11, 2025, 05:32 AM UTC

   Heroku (Salesforce) has resolved their issue. We are still monitoring the environment to ensure the Heroku updates are stable. Our internal systems are showing green on functionality and metrics. Our customers are reporting that services have been restored. We’ll continue to monitor and update as more information and RCA is made available to us.

10. resolved Jun 12, 2025, 07:46 PM UTC

    Workspot systems are fully recovered from this incident.

11. postmortem Jun 17, 2025, 03:46 PM UTC

    Heroku Salesforce posted the [official summary](https://www.heroku.com/blog/summary-of-june-10-outage/) and preliminary [Root Cause Analysis \(RCA\)](https://help.salesforce.com/s/articleView?language=en_US&id=005036674&type=1). Workspot is reviewing the Heroku findings and will provide customers update in a few days.

Read the full incident report →

Started May 29, 2025, 02:03 PM UTC· resolved May 30, 2025, 12:30 PM UTC

1. identified May 29, 2025, 02:03 PM UTC

   Issue Summary: Start Time: 09:15 UTC on 29 May 2025 Impact: Errors may occur during service management operations (create, delete, update, scale, start, stop) for VMs. Cause: A sudden spike in usage has caused backend VM components to hit operational limits, resulting in delays and failures. Current Status: Microsoft is mitigating the issue by failing over to a healthy backend instance. Please monitor the updates in Azure subscription under Service Health, if your resources are in East US region.

2. resolved May 30, 2025, 12:30 PM UTC

   As per Microsoft, this issue has been resoved. Here is the incident summary, shared by Microsoft: Issue Summary: Between 09:07 UTC and 16:25 UTC on 29 May 2025, a platform issue resulted in an impact to the following services in the East US region: - Virtual Machines & Virtual Machine Scale Sets: Error notifications when performing service management operations - such as create, delete, update, scaling, start, stop - for resources hosted in this region. This impact was restricted to a single Availability Zone (AZ01), Physical AZ01. Retries may have been successful. - Azure Synapse Analytics: Issues while executing Spark jobs through Synapse Pipelines or Notebooks, encountering the error code "CLUSTER_CREATION_TIMED_OUT". Retries may have been successful. - Azure Data Factory: Activity or Pipeline run failures and delays due to dataflow activity failures. MS Response Timeline: - 09:07 UTC: Customer impact began. - 09:12 UTC: Auto-recovery attempts started, including load-shedding and failover. - 09:15 UTC: Service monitoring detected spikes in VM failures; investigation began. - 11:45 UTC: Platform engineers terminated problematic service instances to free compute resources. - 12:53 UTC: Services started processing backlogged VM requests, with some customers still seeing timeouts and throttling. - 13:15 UTC: Engineers redirected VM deployment traffic to alternate management services to speed recovery. - 13:48 UTC: Failover progress noted, backlog began draining. - 13:58 UTC: Azure Data Factory service restored. - 14:09 UTC: Azure Synapse Analytics service restored. - 16:25 UTC: All services fully restored; customer impact mitigated.

Read the full incident report →

Started Apr 07, 2025, 03:52 PM UTC· resolved Apr 08, 2025, 04:59 AM UTC

1. investigating Apr 07, 2025, 03:52 PM UTC

   Workspot Service is in maintenance mode

2. monitoring Apr 07, 2025, 05:48 PM UTC

   The Engineering team has identified the issue and reverted the Control changes implemented over the weekend at approximately 16:00 UTC. Since the rollback, no further issues have been observed. We continue to actively monitor the environment.

3. resolved Apr 08, 2025, 04:59 AM UTC

   This incident has been resolved. No issues have been observed since the rollback was performed.

Read the full incident report →